Torch distributed not working on two machines [nccl backend]

username: nash
cooked: <p>Hi,</p>
<p>I am running a simple application on two machines with 2 gpus each, it is throwing me an error. The application works fine on a single machine with 2gpus.</p>
<p>The NCCL info error info in here</p>
<p>dml4:26072:26072 [1] NCCL INFO Bootstrap : Using [0]XXXXXX&lt;0&gt; [1]enp0s20f0u1u6:169.254.95.120&lt;0&gt; [2]virbr0:192.168.122.1&lt;0&gt;<br>
dml4:26071:26071 [0] NCCL INFO Bootstrap : Using [0]XXXXX&lt;0&gt; [1]enp0s20f0u1u6:169.254.95.120&lt;0&gt; [2]virbr0:XXXXX&lt;0&gt;<br>
dml4:26072:26072 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).<br>
dml4:26071:26071 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).<br>
dml4:26072:26072 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE ; OOB enp88s0:9.1.44.100&lt;0&gt;<br>
dml4:26071:26071 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE ; OOB enp88s0:9.1.44.100&lt;0&gt;<br>
dml4:26072:26240 [1] NCCL INFO Setting affinity for GPU 1 to ffff,f00000ff,fff00000<br>
dml4:26071:26242 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff00,000fffff<br>
dml4:26072:26240 [1] NCCL INFO CUDA Dev 1[1], IB NIC distance :  SYS<br>
dml4:26071:26242 [0] NCCL INFO CUDA Dev 0[0], IB NIC distance :  NODE<br>
dml4:26071:26242 [0] NCCL INFO Ring 00 : 1 -&gt; 2 [receive] via NET/IB/0<br>
dml4:26071:26242 [0] NCCL INFO Ring 00 : 2[0] -&gt; 3[1] via direct shared memory<br>
dml4:26072:26240 [1] NCCL INFO Ring 00 : 3 -&gt; 0 [send] via NET/IB/0</p>
<p>dml4:26072:26240 [1] misc/ibvwrap.cc:252 NCCL WARN Call to ibv_reg_mr failed<br>
dml4:26072:26240 [1] NCCL INFO transport/net_ib.cc:601 -&gt; 2<br>
dml4:26072:26240 [1] NCCL INFO include/net.h:24 -&gt; 2<br>
dml4:26072:26240 [1] NCCL INFO transport/net.cc:360 -&gt; 2<br>
dml4:26072:26240 [1] NCCL INFO init.cc:669 -&gt; 2<br>
dml4:26072:26240 [1] NCCL INFO init.cc:815 -&gt; 2<br>
dml4:26072:26240 [1] NCCL INFO init.cc:951 -&gt; 2<br>
dml4:26072:26240 [1] NCCL INFO misc/group.cc:69 -&gt; 2 [Async thread]</p>
<p>dml4:26071:26242 [0] misc/ibvwrap.cc:252 NCCL WARN Call to ibv_reg_mr failed<br>
dml4:26071:26242 [0] NCCL INFO transport/net_ib.cc:601 -&gt; 2<br>
dml4:26071:26242 [0] NCCL INFO include/net.h:24 -&gt; 2<br>
dml4:26071:26242 [0] NCCL INFO transport/net.cc:388 -&gt; 2<br>
dml4:26071:26242 [0] NCCL INFO init.cc:679 -&gt; 2<br>
dml4:26071:26242 [0] NCCL INFO init.cc:815 -&gt; 2<br>
dml4:26071:26242 [0] NCCL INFO init.cc:951 -&gt; 2<br>
dml4:26071:26242 [0] NCCL INFO misc/group.cc:69 -&gt; 2 [Async thread]<br>
Traceback (most recent call last):<br>
File “conv_dist.py”, line 118, in <br>
main()<br>
File “conv_dist.py”, line 51, in main<br>
mp.spawn(train, nprocs=args.gpus, args=(args,), join=True)<br>
File “/work/tools/envs/dine2/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 200, in spawn<br>
return start_processes(fn, args, nprocs, join, daemon, start_method=‘spawn’)<br>
File “work/tools/envs/dine2/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 158, in start_processes<br>
while not context.join():<br>
File “/work/tools/envs/dine2/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 119, in join<br>
raise Exception(msg)<br>
Exception:</p>
<p>– Process 0 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/work/tools/envs/dine2/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 20, in _wrap<br>
fn(i, *args)<br>
File “/us4j4248/pt_dist/conv_dist.py”, line 75, in train<br>
model = DDP(model, device_ids=[gpu])<br>
File “/work/tools/envs/dine2/lib/python3.6/site-packages/torch/nn/parallel/distributed.py”, line 285, in <strong>init</strong><br>
self.broadcast_bucket_size)<br>
File “/work/tools/envs/dine2/lib/python3.6/site-packages/torch/nn/parallel/distributed.py”, line 496, in _distributed_broadcast_coalesced<br>
dist._broadcast_coalesced(self.process_group, tensors, buffer_size)<br>
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1591914838379/work/torch/lib/c10d/ProcessGroupNCCL.cpp:514, unhandled system error, NCCL version 2.4.8</p>
<p>ps: I have removed the ip addresses above.</p>
<p>Thanks</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/nash">@nash</a>, could you please share a minimum repro of this error?</p>

username: nash
cooked: <p>Hey Shen,<br>
I am running a simple application using &lt;torch.distributed.launch&gt; on two machines each having 2 gpus. It throws me the above error.<br>
CUDA 10.2<br>
Pytorch 1.5.1<br>
NCCL backend as<br>
<em>libnccl-devel-2.5.6-1+cuda10.2.x86_64</em><br>
<em>libnccl-2.5.6-1+cuda10.2.x86_64</em><br>
<em>libnccl-static-2.5.6-1+cuda10.2.x86_64</em></p>
<p>But torch.cuda.nccl.version() shows me 2.4.<br>
Ran NCCL tests – they are working fine.</p>
<pre><code class="lang-auto">def train(args):
    current_env = os.environ.copy()
    dist.init_process_group(backend='nccl', init_method='env://')
    model = ConvNet()
    torch.cuda.set_device(args.local_rank)
    model.cuda(args.local_rank)
    batch_size = 256
    # define loss function (criterion) and optimizer
    criterion = nn.CrossEntropyLoss().cuda(args.local_rank)
    optimizer = torch.optim.SGD(model.parameters(), 1e-4)

    model = DDP(model, device_ids=[args.local_rank])
    # Data loading code
    train_dataset = torchvision.datasets.MNIST(root='./data',
                                               train=True,
                                               transform=transforms.ToTensor(),
                                               download=True)

    train_sampler = torch.utils.data.distributed.DistributedSampler(
        train_dataset, num_replicas=int(current_env["WORLD_SIZE"]), rank=args.local_rank)

    train_loader = torch.utils.data.DataLoader(
    	dataset=train_dataset,
        batch_size=batch_size,
        shuffle=False,         
        num_workers=0,
        pin_memory=True,
        sampler=train_sampler)  
</code></pre>
<p>Plz let me know if you need any more info.<br>
Thanks</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/nash">@nash</a>, thanks for sharing the code. The code looks correct to me, except the rank argument in <code>DistributedSampler</code> might need to be global rank (i.e., <code>current_env["RANK"]</code>) instead of the local rank? But this is not the cause of the error, as the error was thrown in DDP ctor when it tries to run broadcast.</p>
<p>Regarding the error:</p>
<ol start="0">
<li>Curious, does <code>gloo</code> backend work?</li>
<li>Could you please print out the env vars set by the launching script? Sth as what <a href="https://github.com/pytorch/examples/tree/master/distributed/ddp#argument-passing-convention" rel="nofollow noopener">this example</a> does to <code>env_dict</code>.</li>
<li>Could you try run the following command on both nodes? If the return IP is not what you intended, you might need to set either <code>GLOO_SOCKET_IFNAME</code> or <code>NCCL_SOCKET_IFNAME</code> as mentioned <a href="https://pytorch.org/docs/stable/distributed.html#choosing-the-network-interface-to-use" rel="nofollow noopener">here</a> depending on which backend you are using.<pre><code class="lang-auto">getent hosts `hostname`
</code></pre>
</li>
</ol>

username: nash
cooked: <p>Hey <a class="mention" href="/u/mrshenli">@mrshenli</a> thanks for your response.</p>
<ol>
<li>Gloo backend works perfectly fine with this code. NCCL backend throws the error.</li>
</ol>
<p>2</p>
<pre><code class="lang-auto">here are the outputs of environ based on Gloo backend.
Node 0 output
Initializing process group with: {'MASTER_ADDR': 'a.b.c.d', 'MASTER_PORT': '12354', 'RANK': '1', 'WORLD_SIZE': '4'}
[3196] Initializing process group with: {'MASTER_ADDR': 'a.b.c.d', 'MASTER_PORT': '12354', 'RANK': '0', 'WORLD_SIZE': '4'}
[3196] world_size = 4, rank = 0, backend=gloo
[3197] world_size = 4, rank = 1, backend=gloo
Node 1 output
Initializing process group with: {'MASTER_ADDR': 'a.b.c.d', 'MASTER_PORT': '12354', 'RANK': '2', 'WORLD_SIZE': '4'}
[89966] Initializing process group with: {'MASTER_ADDR': 'a.b.c.d', 'MASTER_PORT': '12354', 'RANK': '3', 'WORLD_SIZE': '4'}
[89966] world_size = 4, rank = 3, backend=gloo
[89965] world_size = 4, rank = 2, backend=gloo
</code></pre>
<ol start="3">
<li>the return is the domain name addresses of each host (a.b.c.d) etc. I am not setting anywhere SOCKET_IFNAME</li>
</ol>
<p>My system has NCCL 2.5 while torch (torch.cuda.nccl.version()) shows 2.4.8<br>
Could this be the problem?<br>
How can I upgrade NCCL version in torch.</p>
<p>Thanks.</p>

username: mrshenli
cooked: <p>If Gloo works fine then it means all the env vars and configs should be correct.</p>
<blockquote>
<p>How can I upgrade NCCL version in torch.</p>
</blockquote>
<p>That will require modify pytorch NCCL submodule and recompile. Like this <a href="https://github.com/pytorch/pytorch/pull/40622" class="inline-onebox" rel="noopener nofollow ugc">updating nccl to 2.7.3 by agolynski · Pull Request #40622 · pytorch/pytorch · GitHub</a>. You can pull this PR can compile from it, which should be using NCCL 2.7.3.</p>
<p>Another option is to set <code>export USE_SYSTEM_NCCL=1</code>, and then compile from source, then it should be using the 2.5 that you installed on the machine.</p>

username: nash
cooked: <p>Thanks <a class="mention" href="/u/mrshenli">@mrshenli</a>.<br>
As you mentioned that pytorch has NCCL precompiled and both nodes use the same version of NCCL.<br>
Does that mean NCCL version is not the problem?</p>
<p>Did you notice this “misc/ibvwrap.cc:252 NCCL WARN Call to ibv_reg_mr failed” in the logs.</p>
<p>I tried to build torch from source, I hit another roadblock there as well.<br>
“Performing Test SUPPORT_GLIBCXX_USE_C99 - Failed”</p>
<p>thanks.</p>

username: lcw
cooked: <p>An error in ibv (i.e., InfiniBand verbs) indicates problems with GPU Direct, which NCCL tries to use for RDMA but which Gloo doesn’t. You can try to confirm that this is indeed the issue by running with the <code>NCCL_IB_DISABLE=1</code> env var. That may work but would probably end up being slower. In that case you might want to follow the instructions here to troubleshoot InfiniBand issues: <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#gpu-direct" rel="nofollow noopener">https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#gpu-direct</a></p>

username: nash
cooked: <p>Hi <a class="mention" href="/u/lcw">@lcw</a>, thanks. Indeed it is NCCL issue, and setting NCCL_IB_DISABLE =1 works fine.</p>