'unhandled system error' when training with multi nodes

username: miyanoo
cooked: <p>I met an error when I use DDP for multi node (two node, two GPUs each) training and ‘nccl’ backend (It runs perfect when I use ‘gloo’). The environment is Ubuntu 16.04+python3.5+pytorch1.5.0+cuda10.1.<br>
My code is based on the demo code on the offical website to test distributed training.</p>
<pre><code class="lang-auto">def setup(rank, world_size):
    os.environ['NCCL_DEBUG'] = 'INFO'
    os.environ['NCCL_SOCKET_IFNAME'] = 'eno1'
    os.environ['NCCL_IB_DISABLE'] = '1'
    dist.init_process_group(
        "nccl", rank=rank, init_method='tcp://162.105.146.176:22222', world_size=world_size)

class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 10)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(10, 5)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))
 ...
</code></pre>
<p>The error code when using ‘nccl’ is as following,</p>
<pre><code class="lang-auto">ptwop-176:1755:1755 [0] NCCL INFO Bootstrap : Using [0]eno1:162.105.146.176&lt;0&gt;
ptwop-176:1755:1755 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

ptwop-176:1755:1755 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
ptwop-176:1755:1755 [0] NCCL INFO NET/Socket : Using [0]eno1:162.105.146.176&lt;0&gt;
NCCL version 2.4.8+cuda10.1
ptwop-176:1756:1756 [1] NCCL INFO Bootstrap : Using [0]eno1:162.105.146.176&lt;0&gt;
ptwop-176:1756:1756 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).

ptwop-176:1756:1756 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
ptwop-176:1756:1756 [1] NCCL INFO NET/Socket : Using [0]eno1:162.105.146.176&lt;0&gt;
ptwop-176:1755:1870 [0] NCCL INFO Setting affinity for GPU 0 to 5555,55555555
ptwop-176:1756:1871 [1] NCCL INFO Setting affinity for GPU 1 to 5555,55555555

ptwop-176:1756:1871 [1] include/socket.h:390 NCCL WARN Connect to 162.105.146.178&lt;35007&gt; failed : No route to host
ptwop-176:1756:1871 [1] NCCL INFO bootstrap.cc:100 -&gt; 2

ptwop-176:1756:1871 [1] NCCL INFO bootstrap.cc:337 -&gt; 2
ptwop-176:1755:1869 [0] include/socket.h:390 NCCL WARN Connect to 162.105.146.178&lt;54473&gt; failed : No route to host
ptwop-176:1756:1871 [1] NCCL INFO init.cc:695 -&gt; 2
ptwop-176:1755:1869 [0] NCCL INFO bootstrap.cc:100 -&gt; 2
ptwop-176:1756:1871 [1] NCCL INFO init.cc:951 -&gt; 2
ptwop-176:1755:1869 [0] NCCL INFO bootstrap.cc:226 -&gt; 2
ptwop-176:1756:1871 [1] NCCL INFO misc/group.cc:69 -&gt; 2 [Async thread]
Traceback (most recent call last):
  File "test.py", line 73, in &lt;module&gt;
    run_demo(demo_basic, 2, 3)
  File "test.py", line 45, in run_demo
    join=True)
  File "/home/xukun/setsuna/lib/python3.5/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/xukun/setsuna/lib/python3.5/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
    while not context.join():
  File "/home/xukun/setsuna/lib/python3.5/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/xukun/setsuna/lib/python3.5/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/home/xukun/graph/multi_node/test.py", line 57, in demo_basic
    ddp_model = DDP(model.to(rank), device_ids=[rank])
  File "/home/xukun/setsuna/lib/python3.5/site-packages/torch/nn/parallel/distributed.py", line 285, in __init__
    self.broadcast_bucket_size)
  File "/home/xukun/setsuna/lib/python3.5/site-packages/torch/nn/parallel/distributed.py", line 483, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:514, unhandled system error, NCCL version 2.4.8
</code></pre>
<p>What can I do to avoid this error?</p>

username: Kiuk_Chung
cooked: <p>From this log message: <code>ptwop-176:1755:1869 [0] include/socket.h:390 NCCL WARN Connect to 162.105.146.178&lt;54473&gt; failed : No route to host</code>. I’m assuming the other node’s ip is <code>162.105.146.178</code>? Could you validate the following:</p>
<ol>
<li>See if the issue reproduces on a single-node multi-gpu setup.</li>
<li>Can you ping <code>162.105.146.178</code> from <code>162.105.146.176</code>?</li>
</ol>

username: miyanoo
cooked: <p>Yes, the nodes’ ips are <code>162.105.146.178</code> and <code>162.105.146.176</code>.</p>
<ol>
<li>The issue cannot reproduce on a single-node multi-gpu setup, and everything runs well.</li>
<li>The two nodes can ping each other successfully, and it runs well when I change the communication backend to “gloo” from “nccl” with nearly no code changed (minus some modification to make Tensors and model on cpu).</li>
</ol>

username: Kiuk_Chung
cooked: <p>Thanks for validating. Which version of NCCL are you using? And could you validate that NCCL has been successfully installed on both nodes by running: <a href="https://github.com/NVIDIA/nccl-tests" rel="nofollow noopener">https://github.com/NVIDIA/nccl-tests</a>.</p>

username: Kiuk_Chung
cooked: <p>Here’s one way to see if nccl is installed on the node:</p>
<pre><code class="lang-auto">locate nccl| grep "libnccl.so" | tail -n1 | sed -r 's/^.*\.so\.//'
</code></pre>