Simple multi-node NCCL script hangs at barrier

adrianwaelchli: <p>Hi there</p>
<p>I’m trying to run the following simple script on two machines:</p>
<pre><code class="lang-python">import torch.distributed
import time
import argparse

def main():

    parser = argparse.ArgumentParser()
    parser.add_argument("--local_rank", type=int)
    parser.add_argument("--global_rank", type=int)
    args = parser.parse_args()

    torch.cuda.set_device(args.local_rank)

    print("init")
    torch.distributed.init_process_group(
        backend="nccl",
        init_method="tcp://10.10.10.22:1191",
        world_size=2,
        rank=args.global_rank,
    )
    time.sleep(5)
    print("barrier")
    torch.distributed.barrier()  # HANGS HERE


if __name__ == "__main__":
    main()
</code></pre>
<p>I have two machines with the IPs 10.10.10.22 (master) and 10.10.10.25 with port 1191 open on both.</p>
<p>On master:</p>
<pre><code class="lang-auto">export NCCL_SOCKET_IFNAME=eno1
export NCCL_DEBUG_SUBSYS=ALL
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=1
python test.py --local_rank 0 --global_rank 0
</code></pre>
<pre><code class="lang-auto">lambda-server4:1793990:1793990 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno1
lambda-server4:1793990:1793990 [0] NCCL INFO NCCL_SOCKET_IFNAME set to eno1
lambda-server4:1793990:1793990 [0] NCCL INFO Bootstrap : Using [0]eno1:10.10.10.22&lt;0&gt;
lambda-server4:1793990:1793990 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
lambda-server4:1793990:1793990 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
lambda-server4:1793990:1793990 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno1
lambda-server4:1793990:1793990 [0] NCCL INFO NCCL_SOCKET_IFNAME set to eno1
lambda-server4:1793990:1793990 [0] NCCL INFO NET/Socket : Using [0]eno1:10.10.10.22&lt;0&gt;
lambda-server4:1793990:1793990 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda11.1
</code></pre>
<p>On the other machine:</p>
<pre><code class="lang-auto">export NCCL_SOCKET_IFNAME=enp49s0f1
export NCCL_DEBUG_SUBSYS=ALL
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=1
python test.py --local_rank 0 --global_rank 1
</code></pre>
<pre><code class="lang-auto">hyperplane1:1255526:1255526 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to enp49s0f1
hyperplane1:1255526:1255526 [1] NCCL INFO NCCL_SOCKET_IFNAME set to enp49s0f1
hyperplane1:1255526:1255526 [1] NCCL INFO Bootstrap : Using [0]enp49s0f1:10.10.10.25&lt;0&gt;
hyperplane1:1255526:1255526 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
hyperplane1:1255526:1255526 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
hyperplane1:1255526:1255526 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to enp49s0f1
hyperplane1:1255526:1255526 [1] NCCL INFO NCCL_SOCKET_IFNAME set to enp49s0f1
hyperplane1:1255526:1255526 [1] NCCL INFO NET/Socket : Using [0]enp49s0f1:10.10.10.25&lt;0&gt;
hyperplane1:1255526:1255526 [1] NCCL INFO Using network Socket

hyperplane1:1266304:1266392 [0] NCCL INFO Call to connect returned Connection timed out, retrying
hyperplane1:1266304:1266392 [0] NCCL INFO Call to connect returned Connection timed out, retrying

hyperplane1:1266304:1266392 [0] include/socket.h:403 NCCL WARN Connect to 10.10.10.22&lt;49177&gt; failed : Connection timed out
hyperplane1:1266304:1266392 [0] NCCL INFO bootstrap.cc:95 -&gt; 2
hyperplane1:1266304:1266392 [0] NCCL INFO bootstrap.cc:309 -&gt; 2
hyperplane1:1266304:1266392 [0] NCCL INFO init.cc:555 -&gt; 2
hyperplane1:1266304:1266392 [0] NCCL INFO init.cc:840 -&gt; 2
hyperplane1:1266304:1266392 [0] NCCL INFO group.cc:73 -&gt; 2 [Async thread]
</code></pre>
<p>The program hangs at the barrier call and I don’t know why. I get past the init_process_group call on both machines so I assume the connection between the two servers is fine, but at the barrier it times out.</p>
<p>Does anyone see the problem here? I have probably missed a configuration step, but I don’t know what.</p>
<p>PyTorch 1.8.1<br>
NCCL version 2.7.8+cuda11.1</p>

rvarm1: <p>Hi, what is the output of <code>ifconfig</code> on both your machines? If you’re using the <code>ifconfig</code> output to set the NCCL_SOCKET_IFNAME variables on each node, you could try setting NCCL_SOCKET_IFNAME=eno1, enp49s0f1 as per the comments on this issue: <a href="https://github.com/NVIDIA/nccl/issues/286" class="inline-onebox" rel="noopener nofollow ugc">How to set NCCL_SOCKET_IFNAME · Issue #286 · NVIDIA/nccl · GitHub</a>, and in general make sure that the two interfaces can talk to each other according to your network setup.</p>
<p>In addition, can you try changing <code>barrier()</code> to <code>allreduce</code> some tensors (allocated on the appropriate GPU) and check whether that works as expected?</p>

adrianwaelchli: <p>Hi thanks for your input.</p>
<p>ifconfig returns me multiple interfaces. I picked the one that has the IP address assigned I use to login (10.10.10.22 and 10.10.10.25).<br>
First machine:</p>
<pre><code class="lang-auto">br-5443622090a7: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
        inet6 fe80::42:1aff:fec8:448c  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether 02:42:1a:c8:44:8c  txqueuelen 0  (Ethernet)
        RX packets 178271  bytes 4991588 (4.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 39  bytes 5694 (5.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:e5ff:fe0a:b382  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether 02:42:e5:0a:b3:82  txqueuelen 0  (Ethernet)
        RX packets 76931  bytes 4131399 (4.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 86761  bytes 625271597 (625.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno1: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        inet 10.10.10.22  netmask 255.255.255.0  broadcast 10.10.10.255
        inet6 fe80::3eec:efff:fe03:ed46  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether 3c:ec:ef:03:ed:46  txqueuelen 1000  (Ethernet)
        RX packets 206957342  bytes 291080003217 (291.0 GB)
        RX errors 0  dropped 3340746  overruns 0  frame 0
        TX packets 47119321  bytes 7465744617 (7.4 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xc1320000-c133ffff

lo: flags=73&lt;UP,LOOPBACK,RUNNING&gt;  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10&lt;host&gt;
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 31035316  bytes 3161891159 (3.1 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 31035316  bytes 3161891159 (3.1 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth212871c: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        inet6 fe80::e400:bff:feb2:acb0  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether e6:00:0b:b2:ac:b0  txqueuelen 0  (Ethernet)
        RX packets 339096  bytes 21260368 (21.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 359220  bytes 21820500 (21.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth406ee85: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        inet6 fe80::d827:24ff:fe8c:3c0f  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether da:27:24:8c:3c:0f  txqueuelen 0  (Ethernet)
        RX packets 1807  bytes 131066 (131.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2265  bytes 13311970 (13.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethb0ca500: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        inet6 fe80::384f:23ff:fe37:a7b9  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether 3a:4f:23:37:a7:b9  txqueuelen 0  (Ethernet)
        RX packets 338768  bytes 21241184 (21.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 359348  bytes 21826112 (21.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
</code></pre>
<p>Second machine:</p>
<pre><code class="lang-auto">docker0: flags=4099&lt;UP,BROADCAST,MULTICAST&gt;  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:e5ff:fe55:c1a2  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether 02:42:e5:55:c1:a2  txqueuelen 0  (Ethernet)
        RX packets 990861  bytes 50900887 (50.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1872198  bytes 4152860383 (4.1 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp49s0f1: flags=4163&lt;UP,BROADCAST,RUNNING,MULTICAST&gt;  mtu 1500
        inet 10.10.10.25  netmask 255.255.255.0  broadcast 10.10.10.255
        inet6 fe80::3eec:efff:fe1e:dd5b  prefixlen 64  scopeid 0x20&lt;link&gt;
        ether 3c:ec:ef:1e:dd:5b  txqueuelen 1000  (Ethernet)
        RX packets 82711207  bytes 110825052923 (110.8 GB)
        RX errors 0  dropped 489968  overruns 0  frame 0
        TX packets 25754173  bytes 2401860983 (2.4 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73&lt;UP,LOOPBACK,RUNNING&gt;  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10&lt;host&gt;
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 36992814  bytes 2445844459 (2.4 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 36992814  bytes 2445844459 (2.4 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
</code></pre>
<p>Tried setting <code>NCCL_SOCKET_IFNAME=eno1, enp49s0f1</code> on both servers but it didn’t help unfortunately.</p>
<p>I replaced the barrier with an allreduce like so:</p>
<pre><code class="lang-python">    x = torch.tensor([args.global_rank], dtype=torch.float, device=torch.device("cuda", 0))
    torch.distributed.all_reduce(x)
    print(x)
</code></pre>
<p>but it hangs the same way as with barrier.</p>
<p>The two network interfaces can talk to each other, I verified that I can listen on one machine and send a message through telnet to the other machine:</p>
<pre><code class="lang-auto"># on one server
nc -l 1191
# on the other server
telnet 10.10.10.22 1191
</code></pre>
<p>and this works both ways.</p>

aykamko: <p>I ran into this same issue and found a quick solution in this thread on Github: <a href="https://github.com/NVIDIA/nccl/issues/631" class="inline-onebox" rel="noopener nofollow ugc">Question about nccl p2p disable · Issue #631 · NVIDIA/nccl · GitHub</a></p>
<p><strong>TL;DR:</strong> Run your script with <code>NCCL_P2P_DISABLE=1</code> as an environment variable. However, this may slow down your program since communication will happen through Shared Memory instead of direct GPU-to-GPU.</p>
<p>Longer explanation: I have IO Virtualization enabled in the BIOS on this machine. This is also known as PCI Access Control Services, ACS, VT-d, or IOMMU. (Lots of names for the same thing???) This NVIDIA documentation gives more info: <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#gpu-direct" class="inline-onebox" rel="noopener nofollow ugc">Troubleshooting — NCCL 2.16.2 documentation</a></p>
<p>The other way to fix this is to <em>disable</em> IO Virtualization, which may or may not be desired.</p>

adrianwaelchli: <p>Thanks for the suggestions <a class="mention" href="/u/aykamko">@aykamko</a>. It looks like there can be several reasons leading to a hang like here. It’s been a long time since I posted this but if I remember correctly, I had to open and allow some ports for NCCL to communicate, and it was the reason for the hang. I think this troubleshooting setting here eventually solved it for me: <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#ip-ports" class="inline-onebox" rel="noopener nofollow ugc">Troubleshooting — NCCL 2.16.2 documentation</a></p>