Distributed Training Only Works When InfiniBand Is Disabled

username: MatCorr
cooked: <p>We are trying to run a training script in distributed form through Singularity and Slurm. Here’s a generic version of the script that triggers the code:</p>
<pre><code class="lang-auto">srun singularity run --nv \
      -B/files \
      image.sif \
      torchrun \
        --nnodes 3 \
        --nproc_per_node 8 \
        --rdzv_id $RANDOM \
        --rdzv_backend c10d \
        --rdzv_endpoint $head_node_ip:29500 \
        training.py
</code></pre>
<p>And here’s how we submit the script above to Slurm:</p>
<p><code>$ sbatch --nodelist=list_of_nodes_to_use train.slurm</code></p>
<p>It’s pretty run-of-the-mill stuff for distributed training, I assume, but we are running into the following problem: the scripts above only work when disabling the use of InifiBand via the command <code>export NCCL_IB_DISABLE=1</code>.</p>
<p>When that’s not done, we get the following error:</p>
<pre><code class="lang-auto">RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1269, internal error, NCCL version 2.14.3
ncclInternalError: Internal check failed.
Last error:
Proxy Call to rank 0 failed (Connect)
</code></pre>
<p>I assume there’s something missing (a library or specific configuration, perhaps) that will allow NCCL to work properly, but I am not sure what. Can anyone shed some light on this issue?</p>
<p>The image (.sif) we are using has Ubuntu 18.04.6 LTS and torch 1.13.1.</p>

username: ptrblck
cooked: <p>Could you rerun your script with <code>NCCL_DEBUG=INFO</code> and post the logs here?<br>
Also, is this error reproducible in the latest PyTorch release?</p>

username: MatCorr
cooked: <p>Thanks for the reply.</p>
<p>Here’s what I got running the script with <code>NCCL_DEBUG=INFO</code>.</p>
<pre><code class="lang-auto">Last error:
Net : Connect to &lt;IP&gt; failed : No route to hostNCCL error in: /opt/conda/conda-bld/pytorch_1670525552843/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1269, internal error, NCCL version 2.14.3
ncclInternalError: Internal check failed.
</code></pre>
<p>There are also some other messages that might be interesting further up on the log. This one:</p>
<p><code>misc/socket.cc:456 NCCL WARN Net : Connect to &lt;IP&gt; failed : No route to host</code></p>
<p>And this one:</p>
<p><code>NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</code></p>
<p>I will try adding that plugin to the image and get back to you as soon as possible, but I believe we have already tried that solution and got the same error.</p>
<p>As for being reproducible in the latest Pytorch version, I am not sure. We don’t have any images with that Pytorch version, but I can try and get back to you as well.</p>

username: rajachan
cooked: <aside class="quote no-group" data-username="MatCorr" data-post="3" data-topic="176575">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/fbc32d/48.png" class="avatar"> MatCorr:</div>
<blockquote>
<p><code>NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</code></p>
<p>I will try adding that plugin to the image and get back to you as soon as possible, but I believe we have already tried that solution and got the same error.</p>
</blockquote>
</aside>
<p>This is just information, not an error. In the absence of any network plugins, you should be able to fallback to the internal ib or socket implementations. Given this is running on an IB cluster, NET/IB should be kicking in. <code>NCCL_DEBUG=TRACE</code> might be a bit more verbose and point to the issue.</p>

username: MatCorr
cooked: <p>This is what I got with <code>NCCL_DEBUG=TRACE</code>. There are loads of these errors in the log. I am assuming one for each GPU.</p>
<pre><code class="lang-auto">RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1670525552843/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1269, internal error, NCCL version 2.14.3
ncclInternalError: Internal check failed.
Last error:
Proxy Call to rank 6 failed (Connect)
</code></pre>
<p>These also seem noteworthy, even if they might not be errors per se:</p>
<p><code>proxy.cc:1119 NCCL WARN [Proxy Service 9] Failed to execute operation Connect from rank 9, retcode 2</code></p>
<p><code>misc/ibvwrap.cc:311 NCCL WARN Call to ibv_create_qp failed with error Operation not supported</code><br>
`</p>

username: rajachan
cooked: <aside class="quote no-group" data-username="MatCorr" data-post="5" data-topic="176575">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/fbc32d/48.png" class="avatar"> MatCorr:</div>
<blockquote>
<p>misc/ibvwrap.cc:311 NCCL WARN Call to ibv_create_qp failed with error Operation not supported</p>
</blockquote>
</aside>
<p>I think this is the core issue. It is likely your container does not have the right permissions setup to talk to the IB device’s command interface via syscalls. I am not too familiar with Singularlity’s internals for IB devices, but you should have the /dev/infiniband/uverbsX device mapped into the container for the IB verbs API to send commands down to IB devices.</p>

username: rajachan
cooked: <p>This might help: <a href="https://singularity-userdoc.readthedocs.io/en/latest/bind_paths_and_mounts.html#system-defined-bind-points" class="inline-onebox" rel="noopener nofollow ugc">Bind Paths and Mounts — Singularity container 2.5.1 documentation</a></p>
<p>Make sure <code>/dev</code> is available in the container, and is writable. Perhaps you have an admin-level singularity config that excludes that binding?</p>

username: MatCorr
cooked: <p>Ok, so we got this to work and here’s how we did it.</p>
<p>It turns out the version of the MLNX_OFED package was different in our image and in our hardware. We updated MLNX_OFED in the former to match the one in the latter.</p>
<p>We also had to make some adjustments to these two environment variables.</p>
<p>export NCCL_SOCKET_IFNAME=bond0<br>
export NCCL_IB_HCA=mlx5_0:1</p>
<p>We set their values in the slurm script.</p>