torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:3368, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.25.1

username: Wznnnnn
cooked: <p>Hello exeryone!<br>
When using  <code>deepspeed</code> training job, I get this error:</p>
<pre><code class="lang-auto">[rank0]: torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:3368, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.25.1
[rank0]: ncclUnhandledCudaError: Call to CUDA function failed.
[rank0]: Last error:
[rank0]: Cuda failure 1 'invalid argument'
[rank0]:[W708 22:58:33.597508565 ProcessGroupNCCL.cpp:1497] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
</code></pre>
<p>This code works fine on a single GPU but gives this error in multi-gpu setup.<br>
Any suggestions are appreciated.</p>
<p><strong>Setup</strong><br>
Two RTX 5090 on one machine</p>
<p><strong>Software</strong><br>
<code>torch == 2.7.0.dev20250310+cu128</code><br>
<code>deepspeed == 0.17.0</code></p>
<p><strong>Additional Information</strong><br>
When setting <code>NCCL_DEBUG=INFO</code><br>
<strong>Log</strong></p>
<pre><code class="lang-auto">[2025-07-08 22:50:26,009] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-07-08 22:50:26,798] [INFO] [comm.py:675:init_distributed] cdb=None
[2025-07-08 22:50:26,798] [INFO] [comm.py:706:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2025-07-08 22:50:27,173] [INFO] [comm.py:675:init_distributed] cdb=None
You are using a model of type qwen2 to instantiate a model of type janus_qwen2. This is not supported for all configurations of models and can yield errors.
[2025-07-08 22:50:27,328] [INFO] [config.py:744:__init__] Config mesh_device None world_size = 2
You are using a model of type qwen2 to instantiate a model of type janus_qwen2. This is not supported for all configurations of models and can yield errors.
[2025-07-08 22:50:27,434] [INFO] [config.py:744:__init__] Config mesh_device None world_size = 2
autodl-container-cab44f99fc-82aab726:19858:19858 [0] NCCL INFO Bootstrap: Using eth0:172.17.0.8&lt;0&gt;
autodl-container-cab44f99fc-82aab726:19858:19858 [0] NCCL INFO cudaDriverVersion 12080
autodl-container-cab44f99fc-82aab726:19858:19858 [0] NCCL INFO NCCL version 2.25.1+cuda12.2
autodl-container-cab44f99fc-82aab726:19858:19858 [0] NCCL INFO Comm config Blocking set to 1
autodl-container-cab44f99fc-82aab726:19859:19859 [1] NCCL INFO cudaDriverVersion 12080
autodl-container-cab44f99fc-82aab726:19859:19859 [1] NCCL INFO Bootstrap: Using eth0:172.17.0.8&lt;0&gt;
autodl-container-cab44f99fc-82aab726:19859:19859 [1] NCCL INFO NCCL version 2.25.1+cuda12.2
autodl-container-cab44f99fc-82aab726:19859:19859 [1] NCCL INFO Comm config Blocking set to 1
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. Using internal network plugin.
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO NET/IB : No device found.
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO NET/IB : Using [RO]; OOB eth0:172.17.0.8&lt;0&gt;
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.8&lt;0&gt;
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO Using network Socket
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO ncclCommInitRankConfig comm 0xc360500 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId 5a000 commId 0x2bd6dadc0d6b3c51 - Init START
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. Using internal network plugin.
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO NET/IB : No device found.
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO NET/IB : Using [RO]; OOB eth0:172.17.0.8&lt;0&gt;
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.8&lt;0&gt;
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO Using network Socket
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO ncclCommInitRankConfig comm 0xdbcdea0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 49000 commId 0x2bd6dadc0d6b3c51 - Init START
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO RAS client listening socket at ::1&lt;28028&gt;
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO RAS client listening socket at ::1&lt;28028&gt;
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO Bootstrap timings total 0.001394 (create 0.000035, send 0.000088, recv 0.000357, ring 0.000132, delay 0.000000)
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO Bootstrap timings total 0.893562 (create 0.000028, send 0.000097, recv 0.866650, ring 0.000020, delay 0.000000)
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO Setting affinity for GPU 0 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO Setting affinity for GPU 1 to 0fffffff,ffffff00,00000000,000fffff,ffffffff
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO comm 0xdbcdea0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO Channel 00/04 : 0 1
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO Channel 01/04 : 0 1
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO Channel 02/04 : 0 1
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO Channel 03/04 : 0 1
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO comm 0xc360500 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] -1/-1/-1-&gt;0-&gt;1 [2] 1/-1/-1-&gt;0-&gt;-1 [3] -1/-1/-1-&gt;0-&gt;1
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO P2P Chunksize set to 131072
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO Check P2P Type intraNodeP2pSupport 0 directMode 0
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] 0/-1/-1-&gt;1-&gt;-1 [2] -1/-1/-1-&gt;1-&gt;0 [3] 0/-1/-1-&gt;1-&gt;-1
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO P2P Chunksize set to 131072
autodl-container-cab44f99fc-82aab726:19858:20699 [0] NCCL INFO [Proxy Service] Device 0 CPU core 38
autodl-container-cab44f99fc-82aab726:19859:20701 [1] NCCL INFO [Proxy Service] Device 1 CPU core 6
autodl-container-cab44f99fc-82aab726:19858:20700 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 143
autodl-container-cab44f99fc-82aab726:19859:20702 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 8
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO CC Off, workFifoBytes 1048576
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so libnccl-net.so. Using internal tuner plugin.
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO ncclCommInitRankConfig comm 0xdbcdea0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 49000 commId 0x2bd6dadc0d6b3c51 - Init COMPLETE
autodl-container-cab44f99fc-82aab726:19858:20209 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 34.18 (kernels 34.14, alloc 0.00, bootstrap 0.00, allgathers 0.03, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so libnccl-net.so. Using internal tuner plugin.
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO ncclCommInitRankConfig comm 0xc360500 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId 5a000 commId 0x2bd6dadc0d6b3c51 - Init COMPLETE
autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 34.18 (kernels 33.27, alloc 0.00, bootstrap 0.89, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
autodl-container-cab44f99fc-82aab726:19858:20703 [0] NCCL INFO Channel 00 : 0[0] -&gt; 1[1] via SHM/direct/direct
autodl-container-cab44f99fc-82aab726:19859:20704 [1] NCCL INFO Channel 00 : 1[1] -&gt; 0[0] via SHM/direct/direct
autodl-container-cab44f99fc-82aab726:19858:20703 [0] NCCL INFO Channel 01 : 0[0] -&gt; 1[1] via SHM/direct/direct
autodl-container-cab44f99fc-82aab726:19859:20704 [1] NCCL INFO Channel 01 : 1[1] -&gt; 0[0] via SHM/direct/direct
autodl-container-cab44f99fc-82aab726:19858:20703 [0] NCCL INFO Channel 02 : 0[0] -&gt; 1[1] via SHM/direct/direct
autodl-container-cab44f99fc-82aab726:19859:20704 [1] NCCL INFO Channel 02 : 1[1] -&gt; 0[0] via SHM/direct/direct
autodl-container-cab44f99fc-82aab726:19858:20703 [0] NCCL INFO Channel 03 : 0[0] -&gt; 1[1] via SHM/direct/direct
autodl-container-cab44f99fc-82aab726:19859:20704 [1] NCCL INFO Channel 03 : 1[1] -&gt; 0[0] via SHM/direct/direct
autodl-container-cab44f99fc-82aab726:19858:20703 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
autodl-container-cab44f99fc-82aab726:19859:20704 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1

autodl-container-cab44f99fc-82aab726:19858:19858 [0] enqueue.cc:1500 NCCL WARN Cuda failure 1 'invalid argument'

autodl-container-cab44f99fc-82aab726:19859:19859 [1] enqueue.cc:1500 NCCL WARN Cuda failure 1 'invalid argument'
autodl-container-cab44f99fc-82aab726:19858:19858 [0] NCCL INFO group.cc:242 -&gt; 1
autodl-container-cab44f99fc-82aab726:19858:19858 [0] NCCL INFO group.cc:470 -&gt; 1
autodl-container-cab44f99fc-82aab726:19859:19859 [1] NCCL INFO group.cc:242 -&gt; 1
autodl-container-cab44f99fc-82aab726:19858:19858 [0] NCCL INFO group.cc:573 -&gt; 1
autodl-container-cab44f99fc-82aab726:19859:19859 [1] NCCL INFO group.cc:470 -&gt; 1
autodl-container-cab44f99fc-82aab726:19858:19858 [0] NCCL INFO enqueue.cc:2229 -&gt; 1
autodl-container-cab44f99fc-82aab726:19859:19859 [1] NCCL INFO group.cc:573 -&gt; 1
autodl-container-cab44f99fc-82aab726:19859:19859 [1] NCCL INFO enqueue.cc:2229 -&gt; 1
[2025-07-08 22:51:01,771] [INFO] [partition_parameters.py:348:__exit__] finished initializing model - num_params = 1, num_elems = 0.31B
[rank0]: torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:3368, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.25.1
[rank0]: ncclUnhandledCudaError: Call to CUDA function failed.
[rank0]: Last error:
[rank0]: Cuda failure 1 'invalid argument'
[rank1]: torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:3368, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.25.1
[rank1]: ncclUnhandledCudaError: Call to CUDA function failed.
[rank1]: Last error:
[rank1]: Cuda failure 1 'invalid argument'
[rank0]:[W708 22:51:02.109986298 ProcessGroupNCCL.cpp:1497] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
autodl-container-cab44f99fc-82aab726:19858:20792 [0] NCCL INFO misc/socket.cc:64 -&gt; 3
autodl-container-cab44f99fc-82aab726:19858:20792 [0] NCCL INFO misc/socket.cc:80 -&gt; 3
autodl-container-cab44f99fc-82aab726:19858:20792 [0] NCCL INFO misc/socket.cc:828 -&gt; 3
autodl-container-cab44f99fc-82aab726:19858:20699 [0] NCCL INFO misc/socket.cc:880 -&gt; 3
autodl-container-cab44f99fc-82aab726:19858:20792 [0] NCCL INFO misc/socket.cc:64 -&gt; 3
autodl-container-cab44f99fc-82aab726:19858:20792 [0] NCCL INFO misc/socket.cc:80 -&gt; 3
autodl-container-cab44f99fc-82aab726:19858:20792 [0] NCCL INFO misc/socket.cc:828 -&gt; 3
autodl-container-cab44f99fc-82aab726:19859:20794 [1] NCCL INFO misc/socket.cc:64 -&gt; 3
autodl-container-cab44f99fc-82aab726:19858:20699 [0] NCCL INFO misc/socket.cc:880 -&gt; 3
autodl-container-cab44f99fc-82aab726:19859:20794 [1] NCCL INFO misc/socket.cc:80 -&gt; 3
autodl-container-cab44f99fc-82aab726:19859:20794 [1] NCCL INFO misc/socket.cc:828 -&gt; 3
autodl-container-cab44f99fc-82aab726:19859:20794 [1] NCCL INFO misc/socket.cc:64 -&gt; 3
autodl-container-cab44f99fc-82aab726:19859:20794 [1] NCCL INFO misc/socket.cc:80 -&gt; 3
autodl-container-cab44f99fc-82aab726:19859:20794 [1] NCCL INFO misc/socket.cc:828 -&gt; 3
autodl-container-cab44f99fc-82aab726:19859:20701 [1] NCCL INFO misc/socket.cc:880 -&gt; 3
autodl-container-cab44f99fc-82aab726:19858:20792 [0] NCCL INFO comm 0xdbcdea0 rank 0 nranks 2 cudaDev 0 busId 49000 - Abort COMPLETE
autodl-container-cab44f99fc-82aab726:19859:20794 [1] NCCL INFO comm 0xc360500 rank 1 nranks 2 cudaDev 1 busId 5a000 - Abort COMPLETE
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Wznnnnn" data-post="1" data-topic="221360">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wznnnnn/48/77621_2.png" class="avatar"> Wznnnnn:</div>
<blockquote>
<p><code>autodl-container-cab44f99fc-82aab726:19859:20210 [1] NCCL INFO ncclMaxSharedMem 82240 exceeds device/fn maxSharedMem 79856</code></p>
</blockquote>
</aside>
<p>Could you test the 2.8.0RC binaries or any nightly (with CUDA 12.9) as this should have been fixed already in NCCL 2.27.x?</p>

username: Wznnnnn
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the quick reply!</p>
<p>I rebuilt <code>PyTorch 2.9.0.dev</code> with <code>CUDA 12.9</code>, and that indeed resolved the NCCL issue!</p>
<p>The only downside is that <code>flash-attn2</code> hasn’t been updated to support <code>PyTorch versions beyond 2.7</code>. So while I’m now able to run multi-GPU training on the 5090s, I currently can’t take advantage of <code>flash-attn2</code> for acceleration <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=14" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>