Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels 

Shakhizat_Nurgaliyev: <p>Greeting to all,</p>
<p>Could you please help me to fix the issue described below? I was performing multi-node and multi-GPU training</p>
<p>I ran the following command on the first(master) node:</p>
<pre><code class="lang-auto">export NCCL_DEBUG=INFO
export TORCH_DISTRIBUTED_DEBUG=INFO                  
export NCCL_IB_HCA=mlx5_0                 
export NCCL_SOCKET_IFNAME=ibp12s0
export NCCL_IGNORE_CPU_AFFINITY=1
export NCCL_P2P_LEVEL=NVL

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python3 -m torch.distributed.launch --master_addr="172.16.0.25" --master_port 1234 --nproc_per_node=8 --nnodes=2 --node_rank=0  training.py
</code></pre>
<p>Then, on the second node, I ran the following command:</p>
<pre><code class="lang-auto">export NCCL_DEBUG=INFO
export TORCH_DISTRIBUTED_DEBUG=INFO                  
export NCCL_IB_HCA=mlx5_0                 
export NCCL_SOCKET_IFNAME=ibp12s0
export NCCL_IGNORE_CPU_AFFINITY=1
export NCCL_P2P_LEVEL=NVL

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python3 -m torch.distributed.launch --master_addr="172.16.0.25" --master_port 1234 --nproc_per_node=8 --nnodes=2 --node_rank=1 training.py

</code></pre>
<p>The error messages on the master node are as follows:</p>
<pre><code class="lang-auto">[rank6]:[E ProcessGroupNCCL.cpp:523] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800319 milliseconds before timing out.
[rank2]:[E ProcessGroupNCCL.cpp:523] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800469 milliseconds before timing out.
[rank4]:[E ProcessGroupNCCL.cpp:523] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800804 milliseconds before timing out.
[rank3]:[E ProcessGroupNCCL.cpp:523] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800815 milliseconds before timing out.
[rank5]:[E ProcessGroupNCCL.cpp:523] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800821 milliseconds before timing out.
[rank7]:[E ProcessGroupNCCL.cpp:523] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800824 milliseconds before timing out.
[rank1]:[E ProcessGroupNCCL.cpp:523] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800835 milliseconds before timing out.
[rank6]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank6]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank6]:[E ProcessGroupNCCL.cpp:1182] [Rank 6] NCCL watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800319 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff1ccef4d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional&lt;std::chrono::duration&lt;long, std::ratio&lt;1l, 1000l&gt; &gt; &gt;) + 0x1e6 (0x7ff1829df6e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7ff1829e2c3d in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7ff1829e3839 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: &lt;unknown function&gt; + 0xdc253 (0x7ff1cc6b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: &lt;unknown function&gt; + 0x94ac3 (0x7ff1d263bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: &lt;unknown function&gt; + 0x126850 (0x7ff1d26cd850 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [Rank 6] NCCL watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800319 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff1ccef4d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional&lt;std::chrono::duration&lt;long, std::ratio&lt;1l, 1000l&gt; &gt; &gt;) + 0x1e6 (0x7ff1829df6e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7ff1829e2c3d in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7ff1829e3839 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: &lt;unknown function&gt; + 0xdc253 (0x7ff1cc6b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: &lt;unknown function&gt; + 0x94ac3 (0x7ff1d263bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: &lt;unknown function&gt; + 0x126850 (0x7ff1d26cd850 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7ff1ccef4d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0xdf6b11 (0x7ff182739b11 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: &lt;unknown function&gt; + 0xdc253 (0x7ff1cc6b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: &lt;unknown function&gt; + 0x94ac3 (0x7ff1d263bac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: &lt;unknown function&gt; + 0x126850 (0x7ff1d26cd850 in /lib/x86_64-linux-gnu/libc.so.6)

[rank1]:[E ProcessGroupNCCL.cpp:537] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E ProcessGroupNCCL.cpp:543] To avoid data inconsistency, we are taking the entire process down.
[rank1]:[E ProcessGroupNCCL.cpp:1182] [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800835 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f65ad781d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional&lt;std::chrono::duration&lt;long, std::ratio&lt;1l, 1000l&gt; &gt; &gt;) + 0x1e6 (0x7f65631df6e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7f65631e2c3d in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f65631e3839 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: &lt;unknown function&gt; + 0xdc253 (0x7f65aceb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: &lt;unknown function&gt; + 0x94ac3 (0x7f65b2eedac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: &lt;unknown function&gt; + 0x126850 (0x7f65b2f7f850 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=60906, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1800000) ran for 1800835 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:525 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f65ad781d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional&lt;std::chrono::duration&lt;long, std::ratio&lt;1l, 1000l&gt; &gt; &gt;) + 0x1e6 (0x7f65631df6e6 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::workCleanupLoop() + 0x19d (0x7f65631e2c3d in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x119 (0x7f65631e3839 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: &lt;unknown function&gt; + 0xdc253 (0x7f65aceb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #5: &lt;unknown function&gt; + 0x94ac3 (0x7f65b2eedac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: &lt;unknown function&gt; + 0x126850 (0x7f65b2f7f850 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1186 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f65ad781d87 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0xdf6b11 (0x7f6562f39b11 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: &lt;unknown function&gt; + 0xdc253 (0x7f65aceb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: &lt;unknown function&gt; + 0x94ac3 (0x7f65b2eedac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: &lt;unknown function&gt; + 0x126850 (0x7f65b2f7f850 in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>
<p>Best regards,<br>
Shakhizat</p>

baiyuting: <aside class="quote no-group" data-username="Shakhizat_Nurgaliyev" data-post="1" data-topic="197215">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shakhizat_nurgaliyev/48/67712_2.png" class="avatar"> Shakhizat_Nurgaliyev:</div>
<blockquote>
<p><code>ction&gt; + 0x126850 (</code></p>
</blockquote>
</aside>
<p>did you solve it? I met the same question, and I am confused why  NumelIn=1, NumelOut=1</p>