The connection to the C10d store has failed

omidnr: <p>I am trying to run my code on two servers having one GPU, I am trying a very simple code (that I already tested on a single machine with 2 GPU and works fine) I added some codes for the global rank and local rank to run on multi node form. but I am getting this error:<br>
I use these commands to run my code:<br>
considering first one for master node.</p>
<ul>
<li>
<p>$  torchrun --nproc_per_node=1 --nnodes=2 --node_rank=0 --rdzv_id=456 --rdzv_backend=c10d --rdzv_endpoint=10.30.7.22:29603 ddp-cifar100-multinode.py --epochs 10 --batch-size 16</p>
</li>
<li>
<p>$torchrun --nproc_per_node=1 --nnodes=2 --node_rank=1 --rdzv_id=456 --rdzv_backend=c10d --rdzv_endpoint=10.30.7.22:29603 ddp-cifar100-multinode.py --epochs 10 --batch-size 16</p>
</li>
</ul>
<p>ERROR:torch.distributed.elastic.multiprocessing.errors.error_handler:{<br>
“message”: {<br>
“message”: “RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.”,<br>
“extraInfo”: {<br>
“py_callstack”: “Traceback (most recent call last):\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 156, in _create_tcp_store\n    host, port, is_master=is_server, timeout=timedelta(seconds=read_timeout)\nRuntimeError: connect() timed out. Original timeout was 60000 ms.\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py", line 345, in wrapper\n    return f(*args, **kwargs)\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/run.py", line 719, in main\n    run(args)\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/run.py", line 713, in run\n    )(*cmd_args)\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 131, in <strong>call</strong>\n    return launch_agent(self._config, self._entrypoint, list(args))\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 228, in launch_agent\n    rdzv_handler = rdzv_registry.get_rendezvous_handler(rdzv_parameters)\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/registry.py", line 64, in get_rendezvous_handler\n    return handler_registry.create_handler(params)\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/api.py", line 253, in create_handler\n    handler = creator(params)\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/registry.py", line 35, in _create_c10d_handler\n    backend, store = create_backend(params)\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 250, in create_backend\n    store = _create_tcp_store(params)\n  File "/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py", line 177, in _create_tcp_store\n    ) from exc\ntorch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.\n”,<br>
“timestamp”: “1715699987”<br>
}<br>
}<br>
}<br>
Traceback (most recent call last):<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py”, line 156, in _create_tcp_store<br>
host, port, is_master=is_server, timeout=timedelta(seconds=read_timeout)<br>
RuntimeError: connect() timed out. Original timeout was 60000 ms.</p>
<p>The above exception was the direct cause of the following exception:</p>
<p>Traceback (most recent call last):<br>
File “/home/omid/omid/omid_env/bin/torchrun”, line 11, in <br>
sys.exit(main())<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 345, in wrapper<br>
return f(*args, **kwargs)<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/run.py”, line 719, in main<br>
run(args)<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/run.py”, line 713, in run<br>
)(*cmd_args)<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/launcher/api.py”, line 131, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/launcher/api.py”, line 228, in launch_agent<br>
rdzv_handler = rdzv_registry.get_rendezvous_handler(rdzv_parameters)<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/registry.py”, line 64, in get_rendezvous_handler<br>
return handler_registry.create_handler(params)<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/api.py”, line 253, in create_handler<br>
handler = creator(params)<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/registry.py”, line 35, in _create_c10d_handler<br>
backend, store = create_backend(params)<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py”, line 250, in create_backend<br>
store = _create_tcp_store(params)<br>
File “/home/omid/omid/omid_env/lib/python3.6/site-packages/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py”, line 177, in _create_tcp_store<br>
) from exc<br>
torch.distributed.elastic.rendezvous.api.RendezvousConnectionError: The connection to the C10d store has failed. See inner exception for details.</p>

Andy_Yang: <p>Hi omid<br>
I had the same problem with you, and dont know how to solve it.<br>
I just follow the tutorial from pytorch forum, but it’s just freezed.<br>
If you find solution other place, please let me know.</p>

omidnr: <p>I am still facing this problem the solution is very strange, I have three different servers just fixed one of them. I downgraded everything such as Python 3.8.10 Cuda 10.1 and pytorch 1.9 so It worked but the problem is I have done the same for other two but still same error! I am trying to solve it asap and let you know if I find anything useful. And I found out that my code is fine and the problem is configuration not the code. so try to be sure about your code or pick something that is already tested</p>

Andy_Yang: <p>yea, you are right. It’s the configuration of the machines.<br>
I just asked our cluster engineering team to send me some successed run example on the same cluster.<br>
And I find it’s worked after I change the docker image and some NCCL setting.<br>
So I recommend you check this too.</p>

omidnr: <p>exactly now i fixed two machines but I am facing a problem of NCCL and it says that there is an issue with nccl! should I change version of something? can you tell me what steps you have done to fix your problem? because now my code is running but I have two errors that one of them is nccl and the other says rank1 got 8 params while rank0 got 0 !</p>

H-Huang: <p>It does seem like a host configuration issue where the hosts cannot talk to each other.</p>
<p>One way to single out errors between NCCL and pytorch distributed is to create a sample script that just creates a <a href="https://pytorch.org/docs/master/distributed.html#torch.distributed.TCPStore" rel="noopener nofollow ugc">Store</a>. This is what is used to bootstrap the process groups and then nccl is initialized afterwards. If the store is able to be initialized correctly, then it would be a NCCL issue. However, based on the stack trace it doesn’t look like a NCCL issue.</p>

omidnr: <p>Thank you I fixed that problem (at least it looks like) but I have issues with nccl ! I am getting this error. everything works fine on single machine but when going to multi node I am getting this error:</p>
<p>Files already downloaded and verified<br>
smartedge:949125:949125 [0] NCCL INFO cudaDriverVersion 12040<br>
smartedge:949125:949125 [0] NCCL INFO Bootstrap : Using eno8303:10.30.2.58&lt;0&gt;<br>
smartedge:949125:949125 [0] NCCL INFO NET/Plugin : dlerror=libnccl-net.so: cannot open shared object file: No such file or directory No plugin found (libnccl-net.so), using internal implementation<br>
smartedge:949125:949125 [0] NCCL INFO init.cc:1732 Cuda Host Alloc Size 4 pointer 0x7ff1c3200000<br>
smartedge:949125:949227 [0] NCCL INFO NET/IB : Using [0]={[0] mlx5_0:1/RoCE, [1] mlx5_1:1/RoCE} [RO]; OOB eno8303:10.30.2.58&lt;0&gt;<br>
smartedge:949125:949227 [0] NCCL INFO Using non-device net plugin version 0<br>
smartedge:949125:949227 [0] NCCL INFO Using network IB<br>
smartedge:949125:949227 [0] NCCL INFO comm 0x6c3f9a0 rank 1 nranks 2 cudaDev 0 nvmlDev 0 busId 12000 commId 0x4fb96858dddc545 - Init START<br>
smartedge:949125:949227 [0] NCCL INFO NET/IB : GPU Direct RDMA Disabled for HCA 0 ‘mlx5_0+mlx5_1’<br>
smartedge:949125:949227 [0] NCCL INFO === System : maxBw 12.0 totalBw 12.0 ===<br>
smartedge:949125:949227 [0] NCCL INFO CPU/0 (1/1/2)<br>
smartedge:949125:949227 [0] NCCL INFO + PCI[12.0] - PCI/D000 (15b3197800000000)<br>
smartedge:949125:949227 [0] NCCL INFO               + PCI[24.0] - NIC/F000<br>
smartedge:949125:949227 [0] NCCL INFO                             + NET[25.0] - NET/0 (b0e48f00032db048/1/25.000000)<br>
smartedge:949125:949227 [0] NCCL INFO               + PCI[24.0] - PCI/10000 (15b3197800000000)<br>
smartedge:949125:949227 [0] NCCL INFO                             + PCI[12.0] - GPU/12000 (1)<br>
smartedge:949125:949227 [0] NCCL INFO ==========================================<br>
smartedge:949125:949227 [0] NCCL INFO GPU/12000 :GPU/12000 (0/5000.000000/LOC) CPU/0 (3/12.000000/PHB) NET/0 (6/12.000000/PHB)<br>
smartedge:949125:949227 [0] NCCL INFO NET/0 :GPU/12000 (6/12.000000/PHB) CPU/0 (3/12.000000/PHB) NET/0 (0/5000.000000/LOC)<br>
smartedge:949125:949227 [0] NCCL INFO Setting affinity for GPU 0 to 55555555,55555555,55555555,55555555<br>
smartedge:949125:949227 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 1, bw 5.000000/5.000000, type LOC/PHB, sameChannels 1<br>
smartedge:949125:949227 [0] NCCL INFO  0 : NET/0 GPU/1 NET/0<br>
smartedge:949125:949227 [0] NCCL INFO Pattern 3, crossNic 0, nChannels 1, bw 10.000000/5.000000, type LOC/PHB, sameChannels 1<br>
smartedge:949125:949227 [0] NCCL INFO  0 : NET/0 GPU/1 NET/0<br>
smartedge:949125:949227 [0] NCCL INFO comm 0x6c3f9a0 rank 1 nRanks 2 nNodes 2 localRanks 1 localRank 0 MNNVL 0<br>
smartedge:949125:949227 [0] NCCL INFO Tree 0 : 0 → 1 → -1/-1/-1<br>
smartedge:949125:949227 [0] NCCL INFO Tree 1 : -1 → 1 → 0/-1/-1<br>
smartedge:949125:949227 [0] NCCL INFO Ring 00 : 0 → 1 → 0<br>
smartedge:949125:949227 [0] NCCL INFO Ring 01 : 0 → 1 → 0<br>
smartedge:949125:949227 [0] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] 0/-1/-1-&gt;1-&gt;-1<br>
smartedge:949125:949227 [0] NCCL INFO P2P Chunksize set to 131072<br>
smartedge:949125:949227 [0] NCCL INFO UDS: Creating service thread comm 0x6c3f9a0 rank 1<br>
smartedge:949125:949227 [0] NCCL INFO misc/utils.cc:235 memory stack hunk malloc(65536)<br>
smartedge:949125:949227 [0] NCCL INFO channel.cc:40 Cuda Alloc Size 1152 pointer 0x7ff1a8e00000<br>
smartedge:949125:949227 [0] NCCL INFO channel.cc:43 Cuda Alloc Size 32 pointer 0x7ff1a9000000<br>
smartedge:949125:949227 [0] NCCL INFO channel.cc:54 Cuda Alloc Size 8 pointer 0x7ff1a9200000<br>
smartedge:949125:949227 [0] NCCL INFO channel.cc:40 Cuda Alloc Size 1152 pointer 0x7ff1a9400000<br>
smartedge:949125:949227 [0] NCCL INFO channel.cc:43 Cuda Alloc Size 32 pointer 0x7ff1a9600000<br>
smartedge:949125:949227 [0] NCCL INFO channel.cc:54 Cuda Alloc Size 8 pointer 0x7ff1a9800000<br>
smartedge:949125:949230 [0] NCCL INFO Mem Realloc old size 0, new size 8 pointer 0x7ff1a4004f10<br>
smartedge:949125:949230 [0] NCCL INFO Allocated 5767524 bytes of shared memory in /dev/shm/nccl-8t3VWy<br>
smartedge:949125:949230 [0] NCCL INFO New proxy recv connection 0 from local rank 0, transport 2<br>
smartedge:949125:949230 [0] NCCL INFO proxyProgressAsync opId=0x7ff1b35ada30 op.type=1 op.reqBuff=0x7ff1a4004ed0 op.respSize=16 done<br>
smartedge:949125:949227 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x7ff1b35ada30<br>
smartedge:949125:949227 [0] NCCL INFO resp.opId=0x7ff1b35ada30 matches expected opId=0x7ff1b35ada30<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Init res=0<br>
smartedge:949125:949227 [0] NCCL INFO Connected to proxy localRank 0 → connection 0x7ff1a4004f30<br>
smartedge:949125:949230 [0] NCCL INFO proxyProgressAsync opId=0x7ff1b35ada30 op.type=3 op.reqBuff=0x7ff1a4008e30 op.respSize=128 done<br>
smartedge:949125:949227 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x7ff1b35ada30<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Setup res=0<br>
smartedge:949125:949227 [0] NCCL INFO resp.opId=0x7ff1b35ada30 matches expected opId=0x7ff1b35ada30<br>
smartedge:949125:949227 [0] NCCL INFO Channel 00/0 : 0[0] → 1[0] [receive] via NET/IB/0<br>
smartedge:949125:949230 [0] NCCL INFO New proxy recv connection 1 from local rank 0, transport 2<br>
smartedge:949125:949230 [0] NCCL INFO proxyProgressAsync opId=0x7ff1b35ada30 op.type=1 op.reqBuff=0x7ff1a400e280 op.respSize=16 done<br>
smartedge:949125:949227 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x7ff1b35ada30<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Init res=0<br>
smartedge:949125:949227 [0] NCCL INFO resp.opId=0x7ff1b35ada30 matches expected opId=0x7ff1b35ada30<br>
smartedge:949125:949227 [0] NCCL INFO Connected to proxy localRank 0 → connection 0x7ff1a4004fa8<br>
smartedge:949125:949230 [0] NCCL INFO proxyProgressAsync opId=0x7ff1b35ada30 op.type=3 op.reqBuff=0x7ff1a400e2c0 op.respSize=128 done<br>
smartedge:949125:949227 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x7ff1b35ada30<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Setup res=0<br>
smartedge:949125:949227 [0] NCCL INFO resp.opId=0x7ff1b35ada30 matches expected opId=0x7ff1b35ada30<br>
smartedge:949125:949227 [0] NCCL INFO Channel 01/0 : 0[0] → 1[0] [receive] via NET/IB/0<br>
smartedge:949125:949230 [0] NCCL INFO New proxy send connection 2 from local rank 0, transport 2<br>
smartedge:949125:949230 [0] NCCL INFO proxyProgressAsync opId=0x7ff1b35ada30 op.type=1 op.reqBuff=0x7ff1a4013710 op.respSize=16 done<br>
smartedge:949125:949227 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x7ff1b35ada30<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Init res=0<br>
smartedge:949125:949227 [0] NCCL INFO resp.opId=0x7ff1b35ada30 matches expected opId=0x7ff1b35ada30<br>
smartedge:949125:949227 [0] NCCL INFO Connected to proxy localRank 0 → connection 0x7ff1a4005020<br>
smartedge:949125:949230 [0] NCCL INFO proxyProgressAsync opId=0x7ff1b35ada30 op.type=3 op.reqBuff=0x7ff1a4013750 op.respSize=0 done<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Setup res=0<br>
smartedge:949125:949227 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x7ff1b35ada30<br>
smartedge:949125:949227 [0] NCCL INFO resp.opId=0x7ff1b35ada30 matches expected opId=0x7ff1b35ada30<br>
smartedge:949125:949227 [0] NCCL INFO Channel 00/0 : 1[0] → 0[0] [send] via NET/IB/0<br>
smartedge:949125:949230 [0] NCCL INFO New proxy send connection 3 from local rank 0, transport 2<br>
smartedge:949125:949230 [0] NCCL INFO proxyProgressAsync opId=0x7ff1b35ada30 op.type=1 op.reqBuff=0x7ff1a4018a70 op.respSize=16 done<br>
smartedge:949125:949227 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x7ff1b35ada30<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Init res=0<br>
smartedge:949125:949227 [0] NCCL INFO resp.opId=0x7ff1b35ada30 matches expected opId=0x7ff1b35ada30<br>
smartedge:949125:949227 [0] NCCL INFO Connected to proxy localRank 0 → connection 0x7ff1a4005098<br>
smartedge:949125:949230 [0] NCCL INFO proxyProgressAsync opId=0x7ff1b35ada30 op.type=3 op.reqBuff=0x7ff1a4018ab0 op.respSize=0 done<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Setup res=0<br>
smartedge:949125:949227 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x7ff1b35ada30<br>
smartedge:949125:949227 [0] NCCL INFO resp.opId=0x7ff1b35ada30 matches expected opId=0x7ff1b35ada30<br>
smartedge:949125:949227 [0] NCCL INFO Channel 01/0 : 1[0] → 0[0] [send] via NET/IB/0<br>
smartedge:949125:949227 [0] NCCL INFO sendConnect ncclProxyCallAsync opId=0x7ff1b379c000<br>
smartedge:949125:949227 [0] NCCL INFO recvConnect ncclProxyCallAsync opId=0x7ff1b379c190 &amp;recv-&gt;proxyConn=0x7ff1b379c198 connectInfo=0x7ff1b39d3e40<br>
smartedge:949125:949227 [0] NCCL INFO sendConnect ncclProxyCallAsync opId=0x7ff1b39ce630<br>
smartedge:949125:949227 [0] NCCL INFO recvConnect ncclProxyCallAsync opId=0x7ff1b39ce7c0 &amp;recv-&gt;proxyConn=0x7ff1b39ce7c8 connectInfo=0x7ff1b39d3ec0<br>
smartedge:949125:949230 [0] NCCL INFO transport/net_ib.cc:773 Ib Alloc Size 181536 pointer 0x7ff1a4024000<br>
smartedge:949125:949230 [0] NCCL INFO NET/IB: NCCL Dev 0 IbDev 0 Port 1 qpn 252 mtu 5 query_ece={supported=1, vendor_id=0x15b3, options=0x30000002, comp_mask=0x0} GID 0 (80FE/B0E48FFEFF2DB04A) fifoRkey=0x4d999 fifoLkey=0x4d999<br>
smartedge:949125:949230 [0] NCCL INFO NET/IB: NCCL Dev 0 IbDev 1 Port 1 qpn 496 mtu 5 query_ece={supported=1, vendor_id=0x15b3, options=0x30000002, comp_mask=0x0} GID 0 (80FE/B1E48FFEFF2DB04A) fifoRkey=0x87f7b fifoLkey=0x87f7b<br>
smartedge:949125:949230 [0] NCCL INFO transport/net_ib.cc:867 Ib Alloc Size 3336 pointer 0x7ff1a40dc000<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Connect res=0<br>
smartedge:949125:949230 [0] NCCL INFO transport/net_ib.cc:978 Ib Alloc Size 164416 pointer 0x7ff1a40e3000<br>
smartedge:949125:949230 [0] NCCL INFO transport/net_ib.cc:991 Ib Alloc Size 3336 pointer 0x7ff1a410d000</p>
<p>smartedge:949125:949230 [0] transport/net_ib.cc:1017 NCCL WARN NET/IB : Local mergedDev mlx5_0+mlx5_1 has a different number of devices=2 as remote rocep13s0f0 1<br>
smartedge:949125:949230 [0] NCCL INFO transport/net_ib.cc:1132 Ib Alloc Size 3336 pointer 0x7ff1a41d9000<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Connect res=0</p>
<p>smartedge:949125:949230 [0] transport/net_ib.cc:891 NCCL WARN NET/IB : Local mergedDev=mlx5_0+mlx5_1 has a different number of devices=2 as remoteDev=rocep13s0f0 nRemDevs=1<br>
smartedge:949125:949230 [0] NCCL INFO Call to ibv_set_ece failed with error Operation not supported errno 95</p>
<p>smartedge:949125:949230 [0] misc/ibvwrap.cc:206 NCCL WARN Call to ibv_modify_qp failed with error Invalid argument errno 22<br>
smartedge:949125:949230 [0] NCCL INFO transport/net_ib.cc:725 → 2<br>
smartedge:949125:949230 [0] NCCL INFO transport/net_ib.cc:931 → 2<br>
smartedge:949125:949230 [0] NCCL INFO transport/net.cc:683 → 2<br>
smartedge:949125:949230 [0] NCCL INFO proxyProgressAsync opId=0x7ff1b379c000 op.type=4 op.reqBuff=0x7ff1a401ddd0 op.respSize=21040 done<br>
smartedge:949125:949227 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x7ff1b379c000<br>
smartedge:949125:949227 [0] NCCL INFO Queuing opId=0x7ff1b379c000 respBuff=0x7ff1b39ff070 respSize=21040<br>
smartedge:949125:949227 [0] NCCL INFO ncclPollProxyResponse Dequeued cached opId=0x7ff1b379c000<br>
smartedge:949125:949227 [0] NCCL INFO transport/net.cc:304 → 2<br>
smartedge:949125:949227 [0] NCCL INFO transport.cc:165 → 2<br>
smartedge:949125:949227 [0] NCCL INFO init.cc:1222 → 2<br>
smartedge:949125:949227 [0] NCCL INFO init.cc:1501 → 2<br>
smartedge:949125:949227 [0] NCCL INFO group.cc:64 → 2 [Async thread]<br>
smartedge:949125:949125 [0] NCCL INFO group.cc:418 → 2<br>
smartedge:949125:949125 [0] NCCL INFO group.cc:95 → 2<br>
smartedge:949125:949230 [0] NCCL INFO transport/net_ib.cc:773 Ib Alloc Size 181536 pointer 0x7ff1a41db000<br>
smartedge:949125:949230 [0] NCCL INFO NET/IB: NCCL Dev 0 IbDev 0 Port 1 qpn 255 mtu 5 query_ece={supported=1, vendor_id=0x15b3, options=0x30000002, comp_mask=0x0} GID 0 (80FE/B0E48FFEFF2DB04A) fifoRkey=0x4efae fifoLkey=0x4efae<br>
smartedge:949125:949230 [0] NCCL INFO NET/IB: NCCL Dev 0 IbDev 1 Port 1 qpn 499 mtu 5 query_ece={supported=1, vendor_id=0x15b3, options=0x30000002, comp_mask=0x0} GID 0 (80FE/B1E48FFEFF2DB04A) fifoRkey=0x89897 fifoLkey=0x89897<br>
smartedge:949125:949230 [0] NCCL INFO transport/net_ib.cc:867 Ib Alloc Size 3336 pointer 0x7ff1a427f000<br>
smartedge:949125:949230 [0] NCCL INFO proxy.cc:1425 → 3<br>
smartedge:949125:949230 [0] NCCL INFO Received and initiated operation=Connect res=3</p>
<h1><a name="smartedge949125949230-0-proxycc1567-nccl-warn-proxy-service-1-failed-to-execute-operation-connect-from-rank-1-retcode-3-rank1-traceback-most-recent-call-last-rank1-file-ddp-cifar100-multinodepy-line-142-in-module-rank1-mainargsepochs-argsbatch_size-rank1-file-ddp-cifar100-multinodepy-line-128-in-main-rank1-trainer-trainermodel-train_data-optimizer-save_every-rank1-file-ddp-cifar100-multinodepy-line-59-in-__init__-rank1-selfmodel-ddpmodel-device_idsselflocal_rank-rank1-file-homeadminsssaenv1libpython38site-packagestorchnnparalleldistributedpy-line-798-in-__init__-rank1-_verify_param_shape_across_processesselfprocess_group-parameters-rank1-file-homeadminsssaenv1libpython38site-packagestorchdistributedutilspy-line-269-in-_verify_param_shape_across_processes-rank1-return-dist_verify_params_across_processesprocess_group-tensors-logger-rank1-torchdistributeddistbackenderror-nccl-error-in-torchcsrcdistributedc10dprocessgroupncclcpp1970-unhandled-system-error-run-with-nccl_debuginfo-for-details-nccl-version-2205-rank1-ncclsystemerror-system-call-eg-socket-malloc-or-external-library-call-failed-or-device-error-rank1-last-error-rank1-call-to-ibv_modify_qp-failed-with-error-invalid-argument-errno-22-smartedge949125949231-0-nccl-info-proxy-service-uds-exit-stop-1-abortflag-1-smartedge949125949233-0-nccl-info-comm-0x6c3f9a0-rank-1-nranks-2-cudadev-0-busid-12000-abort-complete-e0602-193741145897-140028373555008-torchdistributedelasticmultiprocessingapipy826-failed-exitcode-1-local_rank-0-pid-949125-of-binary-homeadminsssaenv1binpython3-traceback-most-recent-call-last-file-homeadminsssaenv1bintorchrun-line-8-in-module-sysexitmain-file-homeadminsssaenv1libpython38site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-347-in-wrapper-return-fargs-kwargs-file-homeadminsssaenv1libpython38site-packagestorchdistributedrunpy-line-879-in-main-runargs-file-homeadminsssaenv1libpython38site-packagestorchdistributedrunpy-line-870-in-run-elastic_launch-file-homeadminsssaenv1libpython38site-packagestorchdistributedlauncherapipy-line-132-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-homeadminsssaenv1libpython38site-packagestorchdistributedlauncherapipy-line-263-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1" class="anchor" href="#smartedge949125949230-0-proxycc1567-nccl-warn-proxy-service-1-failed-to-execute-operation-connect-from-rank-1-retcode-3-rank1-traceback-most-recent-call-last-rank1-file-ddp-cifar100-multinodepy-line-142-in-module-rank1-mainargsepochs-argsbatch_size-rank1-file-ddp-cifar100-multinodepy-line-128-in-main-rank1-trainer-trainermodel-train_data-optimizer-save_every-rank1-file-ddp-cifar100-multinodepy-line-59-in-__init__-rank1-selfmodel-ddpmodel-device_idsselflocal_rank-rank1-file-homeadminsssaenv1libpython38site-packagestorchnnparalleldistributedpy-line-798-in-__init__-rank1-_verify_param_shape_across_processesselfprocess_group-parameters-rank1-file-homeadminsssaenv1libpython38site-packagestorchdistributedutilspy-line-269-in-_verify_param_shape_across_processes-rank1-return-dist_verify_params_across_processesprocess_group-tensors-logger-rank1-torchdistributeddistbackenderror-nccl-error-in-torchcsrcdistributedc10dprocessgroupncclcpp1970-unhandled-system-error-run-with-nccl_debuginfo-for-details-nccl-version-2205-rank1-ncclsystemerror-system-call-eg-socket-malloc-or-external-library-call-failed-or-device-error-rank1-last-error-rank1-call-to-ibv_modify_qp-failed-with-error-invalid-argument-errno-22-smartedge949125949231-0-nccl-info-proxy-service-uds-exit-stop-1-abortflag-1-smartedge949125949233-0-nccl-info-comm-0x6c3f9a0-rank-1-nranks-2-cudadev-0-busid-12000-abort-complete-e0602-193741145897-140028373555008-torchdistributedelasticmultiprocessingapipy826-failed-exitcode-1-local_rank-0-pid-949125-of-binary-homeadminsssaenv1binpython3-traceback-most-recent-call-last-file-homeadminsssaenv1bintorchrun-line-8-in-module-sysexitmain-file-homeadminsssaenv1libpython38site-packagestorchdistributedelasticmultiprocessingerrors__init__py-line-347-in-wrapper-return-fargs-kwargs-file-homeadminsssaenv1libpython38site-packagestorchdistributedrunpy-line-879-in-main-runargs-file-homeadminsssaenv1libpython38site-packagestorchdistributedrunpy-line-870-in-run-elastic_launch-file-homeadminsssaenv1libpython38site-packagestorchdistributedlauncherapipy-line-132-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-homeadminsssaenv1libpython38site-packagestorchdistributedlauncherapipy-line-263-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1"></a>smartedge:949125:949230 [0] proxy.cc:1567 NCCL WARN [Proxy Service 1] Failed to execute operation Connect from rank 1, retcode 3<br>
[rank1]: Traceback (most recent call last):<br>
[rank1]:   File “ddp-cifar100-multinode.py”, line 142, in <br>
[rank1]:     main(args.epochs, args.batch_size)<br>
[rank1]:   File “ddp-cifar100-multinode.py”, line 128, in main<br>
[rank1]:     trainer = Trainer(model, train_data, optimizer, save_every)<br>
[rank1]:   File “ddp-cifar100-multinode.py”, line 59, in <strong>init</strong><br>
[rank1]:     self.model = DDP(model, device_ids=[self.local_rank])<br>
[rank1]:   File “/home/adminsssa/env1/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 798, in <strong>init</strong><br>
[rank1]:     _verify_param_shape_across_processes(self.process_group, parameters)<br>
[rank1]:   File “/home/adminsssa/env1/lib/python3.8/site-packages/torch/distributed/utils.py”, line 269, in _verify_param_shape_across_processes<br>
[rank1]:     return dist._verify_params_across_processes(process_group, tensors, logger)<br>
[rank1]: torch.distributed.DistBackendError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1970, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.20.5<br>
[rank1]: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.<br>
[rank1]: Last error:<br>
[rank1]: Call to ibv_modify_qp failed with error Invalid argument errno 22<br>
smartedge:949125:949231 [0] NCCL INFO [Proxy Service UDS] exit: stop 1 abortFlag 1<br>
smartedge:949125:949233 [0] NCCL INFO comm 0x6c3f9a0 rank 1 nranks 2 cudaDev 0 busId 12000 - Abort COMPLETE<br>
E0602 19:37:41.145897 140028373555008 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 949125) of binary: /home/adminsssa/env1/bin/python3<br>
Traceback (most recent call last):<br>
File “/home/adminsssa/env1/bin/torchrun”, line 8, in <br>
sys.exit(main())<br>
File “/home/adminsssa/env1/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 347, in wrapper<br>
return f(*args, **kwargs)<br>
File “/home/adminsssa/env1/lib/python3.8/site-packages/torch/distributed/run.py”, line 879, in main<br>
run(args)<br>
File “/home/adminsssa/env1/lib/python3.8/site-packages/torch/distributed/run.py”, line 870, in run<br>
elastic_launch(<br>
File “/home/adminsssa/env1/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 132, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/home/adminsssa/env1/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 263, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2><a name="ddp-cifar100-multinodepy-failed-2" class="anchor" href="#ddp-cifar100-multinodepy-failed-2"></a>ddp-cifar100-multinode.py FAILED</h2>
<h2><a name="failures-no_other_failures-3" class="anchor" href="#failures-no_other_failures-3"></a>Failures:<br>
&lt;NO_OTHER_FAILURES&gt;</h2>
<p>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2024-06-02_19:37:41<br>
host      : smartedge<br>
rank      : 1 (local_rank: 0)<br>
exitcode  : 1 (pid: 949125)<br>
error_file: &lt;N/A&gt;<br>
traceback : To enable traceback see: <a href="https://pytorch.org/docs/stable/elastic/errors.html" class="inline-onebox" rel="noopener nofollow ugc">Error Propagation — PyTorch 2.3 documentation</a></p>

H-Huang: <blockquote>
<p>smartedge:949125:949230 [0] NCCL INFO Call to ibv_set_ece failed with error Operation not supported errno 95</p>
<p>smartedge:949125:949230 [0] misc/ibvwrap.cc:206 NCCL WARN Call to ibv_modify_qp failed with error Invalid argument errno 22</p>
</blockquote>
<p>These errors look concerning but I don’t know what the resolution would be. It looks like something with infiniband (ibv are infiniband verbs), so maybe look into how that is set up or you can try disabling infiniband to see if that is the culprit. NCCL experts may know more so you can make an issue in that repository (<a href="https://github.com/NVIDIA/nccl/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · NVIDIA/nccl · GitHub</a>)</p>

omidnr: <p>I tried to disable also the InfiniBand but still didn’t work, even I tried to set specific interface that I was sure that is connected directly but still the same problem! I updated everything and all the versions are same in both server but I am a bit lost what can I do to fix this nccl problem</p>

111568: <p>Hi, I met the same problem, did you solve this one?</p>

omidnr: <p>Unfortunately that problem didn’t solve because it might be problem of the whole server environment that i can not change everything and make two exactly equal as they were physical servers, but I made another <strong>solution</strong> to make the connection as two servers got problem recognizing each other, so I used docker swarm simply to make two nodes and made a overlay network between them (also it is default for it but you can customize ) then after making this change now two servers can communicate to each other without any problem! I hope it will help you</p>