NCCL error of PyTorch 2.1.0 when using multiple gpus

username: Galaxy-Husky
cooked: <p>Hi,</p>
<p>I encountered some NCCL error when using pytorch version 2.1.0 with multiple gpus.<br>
When I downgraded pytorch to 2.0.1, the error disappeared. The following is a minimal example:</p>
<pre><code class="lang-python">export NCCL_DEBUG=INFO

import torch

torch.cuda.nccl.all_gather([torch.zeros(5).cuda()], [torch.zeros(5).cuda()])
</code></pre>
<h2><a name="error-message-1" class="anchor" href="#error-message-1"></a>Error Message</h2>
<pre><code class="lang-auto">JMUSE-AS-2124GQ-NART:2311505:2311505 [0] NCCL INFO cudaDriverVersion 12000
JMUSE-AS-2124GQ-NART:2311505:2311505 [0] NCCL INFO Bootstrap : Using enxb03af2b6059f:169.254.3.1&lt;0&gt;
JMUSE-AS-2124GQ-NART:2311505:2311505 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
JMUSE-AS-2124GQ-NART:2311505:2311505 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation
NCCL version 2.18.5+cuda11.8
JMUSE-AS-2124GQ-NART:2311505:2311519 [0] NCCL INFO NET/IB : No device found.
JMUSE-AS-2124GQ-NART:2311505:2311519 [0] NCCL INFO NET/Socket : Using [0]enxb03af2b6059f:169.254.3.1&lt;0&gt; [1]enp97s0f0:192.168.187.57&lt;0&gt; [2]tun0:10.8.0.1&lt;0&gt; [3]br-cf914d50c295:172.18.0.1&lt;0&gt; [4]br-0eebd337ccf4:172.19.0.1&lt;0&gt; [5]tun1:198.18.192.11&lt;0&gt; [6]veth232c90d:fe80::985b:78ff:feeb:62d9%veth232c90d&lt;0&gt; [7]vethb4455b0:fe80::dc02:2dff:fe34:ab7f%vethb4455b0&lt;0&gt; [8]veth83972a3:fe80::7056:a7ff:fec4:94a5%veth83972a3&lt;0&gt; [9]veth8319398:fe80::440c:daff:fe3c:354c%veth8319398&lt;0&gt; [10]veth70b5a2c:fe80::1ccc:70ff:feb2:af31%veth70b5a2c&lt;0&gt; [11]veth67387c5:fe80::7c78:6fff:fe57:c5bf%veth67387c5&lt;0&gt; [12]veth7be2777:fe80::8408:f5ff:fe93:685d%veth7be2777&lt;0&gt; [13]vethb8dc075:fe80::9405:e4ff:fe22:dfa2%vethb8dc075&lt;0&gt;
JMUSE-AS-2124GQ-NART:2311505:2311519 [0] NCCL INFO Using network Socket
JMUSE-AS-2124GQ-NART:2311505:2311519 [0] NCCL INFO comm 0x560e80708b00 rank 0 nranks 1 cudaDev 0 nvmlDev 0 busId 1000 commId 0x12a81fddad48cd92 - Init START

JMUSE-AS-2124GQ-NART:2311505:2311519 [0] graph/xml.h:85 NCCL WARN Attribute busid of node nic not found
JMUSE-AS-2124GQ-NART:2311505:2311519 [0] NCCL INFO graph/xml.cc:585 -&gt; 3
JMUSE-AS-2124GQ-NART:2311505:2311519 [0] NCCL INFO graph/xml.cc:767 -&gt; 3
JMUSE-AS-2124GQ-NART:2311505:2311519 [0] NCCL INFO graph/topo.cc:655 -&gt; 3
JMUSE-AS-2124GQ-NART:2311505:2311519 [0] NCCL INFO init.cc:840 -&gt; 3
JMUSE-AS-2124GQ-NART:2311505:2311519 [0] NCCL INFO init.cc:1358 -&gt; 3
JMUSE-AS-2124GQ-NART:2311505:2311519 [0] NCCL INFO group.cc:65 -&gt; 3 [Async thread]
JMUSE-AS-2124GQ-NART:2311505:2311505 [0] NCCL INFO group.cc:406 -&gt; 3
JMUSE-AS-2124GQ-NART:2311505:2311505 [0] NCCL INFO group.cc:96 -&gt; 3
JMUSE-AS-2124GQ-NART:2311505:2311505 [0] NCCL INFO init.cc:1691 -&gt; 3
Traceback (most recent call last):
  File "/mnt/ramdisk0/test.py", line 2, in &lt;module&gt;
    torch.cuda.nccl.all_gather([torch.zeros(5).cuda()], [torch.zeros(5).cuda()])
  File "/home/ping/mambaforge/envs/pytorch-nccl/lib/python3.11/site-packages/torch/cuda/nccl.py", line 121, in all_gather
    torch._C._nccl_all_gather(inputs, outputs, streams, comms)
RuntimeError: NCCL Error 3: internal error - please report this issue to the NCCL developers
</code></pre>
<p>For detailed environment, please see <a href="https://github.com/pytorch/pytorch/issues/113245" class="inline-onebox" rel="noopener nofollow ugc">NCCL error of PyTorch 2.1.0 when using multiple gpus · Issue #113245 · pytorch/pytorch · GitHub</a>.</p>