NCCL Network is unreachable / Connection refused when initializing DDP

matthewdeng: <p>Hi, I’m trying to run a simple distributed PyTorch job across using GPU/NCCL across 2 <code>g4dn.xlarge</code> nodes. The process group seems to initialize fine, but when trying to wrap the model in DDP there is a NCCL connection error.</p>
<p><strong>Failure point:</strong></p>
<pre><code class="lang-python"> model = DistributedDataParallel(model, device_ids=[rank], output_device=rank)
</code></pre>
<p><strong>Environment:</strong></p>
<ul>
<li>
<strong>Torch:</strong> 1.9.0+cu111</li>
<li>
<strong>NCCL:</strong> 2.7.8</li>
</ul>
<p><strong>Logs with <code>NCCL_DEBUG=INFO</code>:</strong></p>
<pre><code class="lang-auto">Rank 0:
[0m ip-172-31-0-137:14605:14673 [0] NCCL INFO Bootstrap : Using [0]ens5:172.31.0.137&lt;0&gt; [1]veth75fa783:fe80::14ad:64ff:fe64:b31e%veth75fa783&lt;0&gt;
[0m ip-172-31-0-137:14605:14673 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
[0m 
[0m ip-172-31-0-137:14605:14673 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
[0m ip-172-31-0-137:14605:14673 [0] NCCL INFO NET/Socket : Using [0]ens5:172.31.0.137&lt;0&gt; [1]veth75fa783:fe80::14ad:64ff:fe64:b31e%veth75fa783&lt;0&gt;
[0m ip-172-31-0-137:14605:14673 [0] NCCL INFO Using network Socket
[0m NCCL version 2.7.8+cuda11.1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 00/04 :    0   1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 01/04 :    0   1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 02/04 :    0   1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 03/04 :    0   1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1|-1-&gt;0-&gt;1/-1/-1 [1] 1/-1/-1-&gt;0-&gt;-1|-1-&gt;0-&gt;1/-1/-1 [2] -1/-1/-1-&gt;0-&gt;1|1-&gt;0-&gt;-1/-1/-1 [3] -1/-1/-1-&gt;0-&gt;1|1-&gt;0-&gt;-1/-1/-1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 00 : 1[1e0] -&gt; 0[1e0] [receive] via NET/Socket/0
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 00 : 0[1e0] -&gt; 1[1e0] [send] via NET/Socket/0

Rank 1:
[0m ip-172-31-86-69:1538:1569 [0] NCCL INFO Bootstrap : Using [0]ens5:172.31.86.69&lt;0&gt; [1]veth0bd9843:fe80::6c37:83ff:fe11:cbd6%veth0bd9843&lt;0&gt;
[0m ip-172-31-86-69:1538:1569 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
[0m 
[0m ip-172-31-86-69:1538:1569 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
[0m ip-172-31-86-69:1538:1569 [0] NCCL INFO NET/Socket : Using [0]ens5:172.31.86.69&lt;0&gt; [1]veth0bd9843:fe80::6c37:83ff:fe11:cbd6%veth0bd9843&lt;0&gt;
[0m ip-172-31-86-69:1538:1569 [0] NCCL INFO Using network Socket
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0|0-&gt;1-&gt;-1/-1/-1 [1] -1/-1/-1-&gt;1-&gt;0|0-&gt;1-&gt;-1/-1/-1 [2] 0/-1/-1-&gt;1-&gt;-1|-1-&gt;1-&gt;0/-1/-1 [3] 0/-1/-1-&gt;1-&gt;-1|-1-&gt;1-&gt;0/-1/-1
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO Channel 00 : 0[1e0] -&gt; 1[1e0] [receive] via NET/Socket/0
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO NET/Socket: Using 2 threads and 8 sockets per thread
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO Channel 00 : 1[1e0] -&gt; 0[1e0] [send] via NET/Socket/0
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO Channel 01 : 0[1e0] -&gt; 1[1e0] [receive] via NET/Socket/1
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO Channel 01 : 1[1e0] -&gt; 0[1e0] [send] via NET/Socket/1
[0m 
[0m ip-172-31-86-69:1538:1572 [0] include/socket.h:403 NCCL WARN Connect to fe80::14ad:64ff:fe64:b31e%7&lt;54043&gt; failed : Network is unreachable
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO transport/net_socket.cc:313 -&gt; 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO include/net.h:21 -&gt; 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO transport/net.cc:161 -&gt; 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO transport.cc:68 -&gt; 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO init.cc:766 -&gt; 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO init.cc:840 -&gt; 2
[0m ip-172-31-86-69:1538:1572 [0] NCCL INFO group.cc:73 -&gt; 2 [Async thread]

Rank 0:
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 01 : 1[1e0] -&gt; 0[1e0] [receive] via NET/Socket/1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Channel 01 : 0[1e0] -&gt; 1[1e0] [send] via NET/Socket/1
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO Call to connect returned Connection refused, retrying
[0m 
[0m ip-172-31-0-137:14605:14688 [0] include/socket.h:403 NCCL WARN Connect to 172.31.86.69&lt;37817&gt; failed : Connection refused
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO bootstrap.cc:95 -&gt; 2
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO bootstrap.cc:363 -&gt; 2
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO transport.cc:59 -&gt; 2
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO init.cc:766 -&gt; 2
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO init.cc:840 -&gt; 2
[0m ip-172-31-0-137:14605:14688 [0] NCCL INFO group.cc:73 -&gt; 2 [Async thread]
</code></pre>
<p><strong>Final Output:</strong></p>
<pre><code class="lang-auto">  File "/home/ray/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 496, in __init__
    dist._verify_model_across_ranks(self.process_group, parameters)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8
ncclSystemError: System call (socket, malloc, munmap, etc) failed.                                                                                                                                        
</code></pre>
<p>A few other things worth mentioning:</p>
<ul>
<li>The script runs fine with Gloo.</li>
<li>The script runs when downgrading to Torch 1.60 with CUDA 10.2 and NCCL 2.4.8.</li>
</ul>
<p>Does anyone have any ideas or suggestions on how to debug this? Thanks in advance!</p>

rvarm1: <p>Hi, this might be a bug if it is working fine with an older version of torch/cuda.</p>
<p>Could you file an issue to <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a> with a detailed repro so that we can investigate? Thank you!</p>

matthewdeng: <p>I was able to get past this issue by setting <code>os.environ["NCCL_SOCKET_IFNAME"]="ens5"</code>. However, it’s still not clear to me why this is needed since this was working on an older version, so I created an issue here: <a href="https://github.com/pytorch/pytorch/issues/68893" class="inline-onebox" rel="noopener nofollow ugc">NCCL Network is unreachable / Connection refused when initializing DDP · Issue #68893 · pytorch/pytorch · GitHub</a>.</p>