Unexpected hang up when using DistributedDataParallel on two machines

IcarusWizard: <p>Hi everyone. I am working on training models across multiple machines. Following the instruction from the documents, I write following codes:</p>
<ul>
<li>On machine 1</li>
</ul>
<pre><code class="lang-python">import torch

torch.distributed.init_process_group(backend='nccl', world_size=2, rank=0, init_method='tcp://172.16.0.246:3456')

net = torch.nn.Linear(256, 128).cuda()
net = torch.nn.parallel.DistributedDataParallel(net, [0], 0)
</code></pre>
<ul>
<li>On machine 2</li>
</ul>
<pre><code class="lang-python">import torch

torch.distributed.init_process_group(backend='nccl', world_size=2, rank=1, init_method='tcp://172.16.0.246:3456')

net = torch.nn.Linear(256, 128).cuda()
net = torch.nn.parallel.DistributedDataParallel(net, [0], 0)
</code></pre>
<p>In which <code>172.16.0.246</code> is the IP of machine 1. However, the code hang up unexpectedly when calling function <code>_distributed_broadcast_coalesced</code> in the initialization of DistributedDataParallel.</p>
<p>Is there anyone knows what did I do wrong?</p>

mrshenli: <p>Hey <a class="mention" href="/u/icaruswizard">@IcarusWizard</a>, what error did you see? Does <code>gloo</code> backend work for you?</p>
<p>Can you run the following command to check if the <code>hostname</code> can resolve to the expected IP on both machines?</p>
<pre><code class="lang-auto">getent hosts `hostname`
</code></pre>
<p>If the resolved IP is wrong, you can set <a href="https://pytorch.org/docs/stable/distributed.html#common-environment-variables" rel="nofollow noopener"><code>NCCL_SOCKET_IFNAME </code></a> env var to point to the right nic (e.g., eth0).</p>

IcarusWizard: <p>Hey Shen!</p>
<p>The strange thing is that there is actually no error. The code stuck at <code>_distributed_broadcast_coalesced</code> and cannot be terminated by <code>Ctrl+C</code>.</p>
<p>I have tried <code>gloo</code>, and it works smoothly which may suggest it is not an issue related to firewall. I also have set <code>GLOO_SOCKET_IFNAME</code> and <code>NCCL_SOCKET_IFNAME</code> to the correct interface on both machine.</p>
<p>And for the command you suggested, it returns <code>127.0.1.1       icarus-Polixir</code> on machine 1, and</p>
<pre><code class="lang-auto">fe80::b62e:99ff:fe72:d1a1 polixir-G291-Z20-00
fe80::98a3:19ff:fe05:3c61 polixir-G291-Z20-00
fe80::42:adff:fe62:bb24 polixir-G291-Z20-00
fe80::d01c:dff:fe28:8b6f polixir-G291-Z20-00
fe80::1c3d:c8ff:fe62:76cc polixir-G291-Z20-00
</code></pre>
<p>on machine 2. I don’t know if it’s related to the issue.</p>

mrshenli: <p>If <code>NCCL_SOCKET_IFNAME</code> points to the correct interface, it should be fine even if the hostname resolves to wrong address, as the latter is a fallback of the former.</p>
<p>And as it has already reached the broadcast op in DDP, I would assume the rendezvous in <code>init_process_group</code> was successful. Could you please confirm this by adding the following code right after <code>init_process_group</code> and see if it also hang at this allreduce?</p>
<pre><code class="lang-python">print("rendezvous done")
tmp = torch.ones(2, 2)
torch.distributed.all_reduce(tmp)
print(tmp)
</code></pre>
<p>Another thing that we could try is to set the following env vars and see if there is any NCCL logs that stand out.</p>
<pre><code class="lang-auto">export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
</code></pre>

IcarusWizard: <p>It stuck on the <code>all_reduce</code> operation too. I get some new information from NCCL logs.</p>
<p>On machine 1:</p>
<pre><code class="lang-auto">rendezvous done
icarus-Polixir:637574:637574 [0] NCCL INFO Bootstrap : Using [0]wlp0s20f3:172.16.0.246&lt;0&gt;
icarus-Polixir:637574:637574 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

icarus-Polixir:637574:637574 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
icarus-Polixir:637574:637574 [0] NCCL INFO NET/Socket : Using [0]wlp0s20f3:172.16.0.246&lt;0&gt;
NCCL version 2.4.8+cuda10.2
icarus-Polixir:637574:637587 [0] NCCL INFO Setting affinity for GPU 0 to ff
icarus-Polixir:637574:637587 [0] NCCL INFO CUDA Dev 0[0], Socket NIC distance :  PHB
icarus-Polixir:637574:637587 [0] NCCL INFO Channel 00 :    0   1
icarus-Polixir:637574:637587 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for GPU 0[0] / HCA 0 (distance 2 &gt;= 2)
icarus-Polixir:637574:637587 [0] NCCL INFO Ring 00 : 1 -&gt; 0 [receive] via NET/Socket/0
icarus-Polixir:637574:637587 [0] NCCL INFO NET/Socket: Using 1 threads and 1 sockets per thread
icarus-Polixir:637574:637587 [0] NCCL INFO Ring 00 : 0 -&gt; 1 [send] via NET/Socket/0
</code></pre>
<p>On machine 2:</p>
<pre><code class="lang-auto">rendezvous done
polixir-G291-Z20-00:2672:2672 [0] NCCL INFO Bootstrap : Using [0]enp129s0f1:172.16.16.122&lt;0&gt;
polixir-G291-Z20-00:2672:2672 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
polixir-G291-Z20-00:2672:2672 [0] NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE ; OOB enp129s0f1:172.16.16.122&lt;0&gt;
polixir-G291-Z20-00:2672:2763 [0] NCCL INFO Setting affinity for GPU 0 to ffff0000,00000000,ffff0000,00000000
polixir-G291-Z20-00:2672:2763 [0] NCCL INFO CUDA Dev 0[4], IB NIC distance :  SYS
polixir-G291-Z20-00:2672:2763 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for GPU 0[4] / HCA 0 (distance 4 &gt;= 2)
polixir-G291-Z20-00:2672:2763 [0] NCCL INFO Ring 00 : 0 -&gt; 1 [receive] via NET/IB/0
polixir-G291-Z20-00:2672:2763 [0] NCCL INFO Ring 00 : 1 -&gt; 0 [send] via NET/IB/0
polixir-G291-Z20-00:2672:2763 [0] NCCL INFO NET/IB: Dev 0 Port 1 qpn 2358 mtu 3 GID 0 (80FE/A1D172FEFF992EB6)
</code></pre>
<p>First thing I noticed is that it trying to find <code>libnccl-net.so</code>. However I cannot find this file in official release of NCCL. I have tried to run this two scripts both on machine 1, and it works just fine. Thus I think it may not be the source of problem.</p>
<p>Any other thoughts?</p>

mrshenli: <p>Did you install NCCL yourself or the one bundled within PyTorch?</p>
<p>Not sure if I interpreted the logs correctly, but looks like machine 1 is trying to use TCP while machine 2 is trying to use IB? What if you set <code>NCCL_IB_DISABLE</code> on both machine? <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-ib-disable" rel="nofollow noopener">https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-ib-disable</a></p>

IcarusWizard: <p>IT WORKS after disabling IB. Seems like IB is a hardware related feature, and the web interface on machine 1 simply doesn’t support it.</p>
<p>I am using the NCCL bundled with PyTorch. I have also tried to install NCCL myself, but it makes no different.</p>
<p>Thanks a lot for your help!</p>