MPI backend and GPU tensor error

Belegkarnil: <p>Hi,<br>
I cannot use CUDA tensor with MPI</p>
<p>I have compiled OpenMPI with</p>
<blockquote>
<p>./configure --prefix=/usr/local --with-cuda=${CUDA_HOME} --with-hwloc=internal --with-libevent=internal --with-ucc=/usr/local --with-ucx=/usr/local --enable-mpi-thread-multiple --enable-mpi-cx</p>
</blockquote>
<p>and then pyTorch (torch-2.1.0a0+git0af3203)</p>
<blockquote>
<p>export CUDA_NVCC_EXECUTABLE=“/usr/local/cuda-12/bin/nvcc”<br>
export CUDA_HOME=“/usr/local/cuda-12”<br>
export CUDNN_INCLUDE_PATH=“/usr/local/cuda-12/include/”<br>
export CUDNN_LIBRARY_PATH=“/usr/local/cuda-12/lib64/”<br>
export LIBRARY_PATH=“/usr/local/cuda-12/lib64”<br>
export USE_CUDA=1<br>
export USE_CUDNN=1<br>
export USE_MKLDNN=1<br>
export ATEN_NO_TEST=1<br>
export BUILD_ONNX_PYTHON=1<br>
export USE_TENSORRT=0<br>
export USE_FFMPEG=1<br>
export USE_DISTRIBUTED=1<br>
export USE_GLOO=1<br>
export USE_MPI=1<br>
export USE_TENSORPIPE=1<br>
export USE_OPENCV=1<br>
export USE_UCC=1<br>
export USE_SYSTEM_UCC=1<br>
export USE_NUMPY=1<br>
export USE_OPENCL=0<br>
export USE_MPI=1<br>
export USE_ZMQ=1<br>
export USE_FFTW=1<br>
export USE_MPS=1<br>
export BUILD_TEST=0<br>
export CMAKE_CUDA_COMPILER=“/usr/local/cuda-12/bin/nvcc”<br>
python setup.py bdist_wheel</p>
</blockquote>
<p>After that, I tried the example from <a href="https://github.com/Stonesjtu/pytorch-learning/blob/master/build-with-mpi.md" rel="noopener nofollow ugc">https://github.com/Stonesjtu/pytorch-learning/blob/master/build-with-mpi.md</a> which is</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
dist.init_process_group(backend='mpi')
t = torch.zeros(5,5).fill_(dist.get_rank()).cuda()
dist.all_reduce(t)
</code></pre>
<p>and I got the error <strong>RuntimeError: No backend type associated with device type cuda</strong><br>
But if I remove the .cuda() it works. Nevertheless, I would like to use MPI with my GPUs.<br>
I checked with ldd (ldd /usr/local/lib/python3.10/site-packages/torch/*.so | grep -E "libopen-rte|libmpi):</p>
<blockquote>
<pre><code>   libmpi_cxx.so.40 =&gt; /lib/x86_64-linux-gnu/libmpi_cxx.so.40 (0x00007ff6f0d6e000)
   libmpi.so.40 =&gt; /usr/local/lib/libmpi.so.40 (0x00007ff6f0c2f000)
   libopen-rte.so.40 =&gt; /usr/local/lib/libopen-rte.so.40 (0x00007ff6dc5d6000)
</code></pre>
</blockquote>
<p>I checked MPI with CUDA (<a href="https://gist.githubusercontent.com/K-Wu/6c353273aafe9a4eaaa344a8b74475b6/raw/bd7ea29cdd72801442ad6d69a88be5fee67cf534/mpi_cuda_awareness_check.cpp" rel="noopener nofollow ugc">https://gist.githubusercontent.com/K-Wu/6c353273aafe9a4eaaa344a8b74475b6/raw/bd7ea29cdd72801442ad6d69a88be5fee67cf534/mpi_cuda_awareness_check.cpp</a> AND <a href="https://github.com/NVIDIA/nccl-tests" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/nccl-tests: NCCL Tests</a>) which seems to work.</p>
<p>Can you help me, please?</p>

wconstab: <p>I’m not sure if we generally test/support this use case - i’m not saying it should not work, but i wonder if you have a reason to prefer MPI+Cuda instead of NCCL+Cuda?  NCCL is the comm lib that is commonly used with cuda for pytorch.</p>

Belegkarnil: <p>Ok but MPI also support NCCL. It a simple and easy way to make a code compatible to distribute a task for CPU and GPU.<br>
Moreover it’s better when it comes to distribute on CPU and GPU at the same time.</p>

kwen2501: <aside class="quote no-group" data-username="Belegkarnil" data-post="1" data-topic="185745">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/edb3f5/48.png" class="avatar"> Belegkarnil:</div>
<blockquote>
<p>RuntimeError: No backend type associated with device type cuda</p>
</blockquote>
</aside>
<p>Hi, from the above error, it may be worth trying:</p>
<pre><code class="lang-auto">dist.init_process_group(backend="cuda:mpi")
</code></pre>
<p>to force a mapping from CUDA tensor to MPI.</p>

Belegkarnil: <p>Thanks !</p>
<p>For those with the same issue, you also have to define RANK and WORLD_SIZE for rendez-vous.<br>
With OpenMPI, something like</p>
<pre><code class="lang-auto">import os
os.environ['RANK'] = os.environ['OMPI_COMM_WORLD_RANK']
os.environ['WORLD_SIZE'] = os.environ['OMPI_COMM_WORLD_SIZE']
torch.distributed.init_process_group(backend='cpu:mpi,cuda:mpi')
</code></pre>