torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1168, unhandled system error, NCCL version 2.17.1

devsy: <p>While I was running DDP tutorial (<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.0.1+cu117 documentation</a>),<br>
on two machines, I got this error</p>
<p>Traceback (most recent call last):<br>
…<br>
ddp_model = DDP(model, device_ids=[device_id])<br>
File “/home/user/miniconda3/envs/torch2/lib/python3.10/site-packages/torch/nn/parallel/distributed.py”, line 791, in <strong>init</strong><br>
_verify_param_shape_across_processes(self.process_group, parameters)<br>
File “/home/user/miniconda3/envs/torch2/lib/python3.10/site-packages/torch/distributed/utils.py”, line 265, in _verify_param_shape_across_processes<br>
return dist._verify_params_across_processes(process_group, tensors, logger)<br>
torch.distributed.DistBackendError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1168, unhandled system error, NCCL version 2.17.1<br>
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.<br>
Last error:<br>
socketStartConnect: Connect to 192.168.0.4&lt;57117&gt; failed : Software caused connection abort</p>
<p>and I don’t understand the error.<br>
Each machine is equipped with nvlink connected GPUs, and two machines can communicate over TCP.<br>
I set <strong>NCCL_SOCKET_IFNAME</strong> correctly after checking ifconfig.<br>
To run the program i used following command</p>
<pre><code class="lang-auto">torchrun --nnodes=2 --nproc_per_node=$NUM_GPUS --rdzv_id=100 --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:29400 ddp_test_elastic.py
</code></pre>
<p>When I ping the other machine, it works alright.<br>
Can I get some hints to resolve this issue?<br>
Thanks!</p>

ptrblck: <p>Could you rerun your workload with <code>NCCL_DEBUG=INFO</code> and post the logs here, please?</p>

devsy: <p>Hello, this is the log I got.</p>
<p>mew1:387101:387101 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno8303</p>
<p>mew1:387101:387101 [0] NCCL INFO Bootstrap : Using eno8303:192.168.0.4&lt;0&gt;</p>
<p>mew1:387101:387101 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory</p>
<p>mew1:387101:387101 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation</p>
<p>mew1:387101:387101 [0] NCCL INFO cudaDriverVersion 12000</p>
<p>NCCL version 2.17.1+cuda11.7</p>
<p>mew1:387101:387152 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno8303</p>
<p>mew1:387101:387152 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [RO]; OOB eno8303:192.168.0.4&lt;0&gt;</p>
<p>mew1:387101:387152 [0] NCCL INFO Using network IB</p>
<p>mew1:387101:387151 [0] bootstrap.cc:126 NCCL WARN Bootstrap Root : mismatch in rank count from procs 4 : 6</p>
<p>mew1:387102:387102 [1] NCCL INFO cudaDriverVersion 12000</p>
<p>mew1:387102:387102 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno8303</p>
<p>mew1:387102:387102 [1] NCCL INFO Bootstrap : Using eno8303:192.168.0.4&lt;0&gt;</p>
<p>mew1:387102:387102 [1] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory</p>
<p>mew1:387102:387102 [1] NCCL INFO NET/Plugin : No plugin found, using internal implementation</p>
<p>mew1:387102:387160 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno8303</p>
<p>mew1:387103:387103 [2] NCCL INFO cudaDriverVersion 12000</p>
<p>mew1:387103:387103 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno8303</p>
<p>mew1:387103:387103 [2] NCCL INFO Bootstrap : Using eno8303:192.168.0.4&lt;0&gt;</p>
<p>mew1:387103:387103 [2] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory</p>
<p>mew1:387103:387103 [2] NCCL INFO NET/Plugin : No plugin found, using internal implementation</p>
<p>mew1:387104:387104 [3] NCCL INFO cudaDriverVersion 12000</p>
<p>mew1:387104:387104 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno8303</p>
<p>mew1:387104:387104 [3] NCCL INFO Bootstrap : Using eno8303:192.168.0.4&lt;0&gt;</p>
<p>mew1:387104:387104 [3] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory</p>
<p>mew1:387104:387104 [3] NCCL INFO NET/Plugin : No plugin found, using internal implementation</p>
<p>mew1:387103:387161 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno8303</p>
<p>mew1:387104:387162 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eno8303</p>
<p>mew1:387103:387161 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [RO]; OOB eno8303:192.168.0.4&lt;0&gt;</p>
<p>mew1:387103:387161 [2] NCCL INFO Using network IB</p>
<p>mew1:387103:387161 [2] misc/socket.cc:480 NCCL WARN socketStartConnect: Connect to 192.168.0.4&lt;35443&gt; failed : Software caused connection abort</p>
<p>mew1:387103:387161 [2] NCCL INFO misc/socket.cc:561 → 2</p>
<p>mew1:387103:387161 [2] NCCL INFO misc/socket.cc:615 → 2</p>
<p>mew1:387103:387161 [2] NCCL INFO bootstrap.cc:270 → 2</p>
<p>mew1:387103:387161 [2] NCCL INFO init.cc:630 → 2</p>
<p>mew1:387103:387161 [2] NCCL INFO init.cc:1114 → 2</p>
<p>mew1:387103:387161 [2] NCCL INFO group.cc:64 → 2 [Async thread]</p>
<p>mew1:387103:387103 [2] NCCL INFO group.cc:422 → 2</p>
<p>mew1:387103:387103 [2] NCCL INFO group.cc:106 → 2</p>
<p>mew1:387103:387103 [0] NCCL INFO comm 0x5fb6c6e0 rank 2 nranks 4 cudaDev 2 busId ca000 - Abort COMPLETE</p>
<p>I will appreciate any help on this. Thanks a lot!</p>

ptrblck: <aside class="quote no-group" data-username="devsy" data-post="3" data-topic="187506">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/c4cdca/48.png" class="avatar"> devsy:</div>
<blockquote>
<p>mew1:387103:387161 [2] misc/socket.cc:480 NCCL WARN socketStartConnect: Connect to 192.168.0.4&lt;35443&gt; failed : Software caused connection abort</p>
</blockquote>
</aside>
<p>It seems the actual connection fails. Did you make sure the nodes can access each other?</p>

devsy: <p>Will there be any quick way to check?<br>
When I checked with ping, they could send and receive properly.</p>

ptrblck: <p>Are you using WSL and could it be a firewall/routing issue as described <a href="https://github.com/NVIDIA/nccl/issues/929#issuecomment-1658248370">here</a>?</p>

devsy: <p>well i’m using a bare metal linux server and i confirmed the linux firewall is not running on mine. if it’s a routing problem, packets can’t be able to reached by ping as well. can they…?</p>