Error: pytorch ddp NCCL ALLGATHER timeout

username: nero1
cooked: <p>PyTorch DDP training, 2 nodes * 8 GPU, at validation step 1440 (approximately 3 and a half hours), reported NCCL timeout error with type ALLGATHER.</p>
<p>Error info</p>
<pre><code class="lang-auto">2025-01-22T15:47:36.977Z [rank3]:[E ProcessGroupNCCL.cpp:563] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3472, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=1800000) ran for 1800058 milliseconds before timing out.
2025-01-22T15:47:36.980Z [rank3]:[E ProcessGroupNCCL.cpp:1537] [PG 0 Rank 3] Timeout at NCCL work: 3472, last enqueued NCCL work: 3472, last completed NCCL work: 3471.
2025-01-22T15:47:36.980Z [rank3]:[E ProcessGroupNCCL.cpp:577] [Rank 3] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
2025-01-22T15:47:36.980Z [rank3]:[E ProcessGroupNCCL.cpp:583] [Rank 3] To avoid data inconsistency, we are taking the entire process down.
2025-01-22T15:47:36.980Z [rank3]:[E ProcessGroupNCCL.cpp:1414] [PG 0 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3472, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=1800000) ran for 1800058 milliseconds before timing out.
2025-01-22T15:47:36.980Z Exception raised from checkTimeout at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
2025-01-22T15:47:36.980Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0x99 (0x7fd57c798e89 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2025-01-22T15:47:36.980Z frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional&lt;std::chrono::duration&lt;long, std::ratio&lt;1l, 1000l&gt; &gt; &gt;) + 0x1e1 (0x7fd5190ae121 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2025-01-22T15:47:36.980Z frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fd5190b54e0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2025-01-22T15:47:36.980Z frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10f (0x7fd5190b63ff in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2025-01-22T15:47:36.980Z frame #4: &lt;unknown function&gt; + 0xdc253 (0x7fd629ab0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
2025-01-22T15:47:36.980Z frame #5: &lt;unknown function&gt; + 0x94ac3 (0x7fd633b44ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2025-01-22T15:47:36.980Z frame #6: clone + 0x44 (0x7fd633bd5a04 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2025-01-22T15:47:36.980Z none
2025-01-22T15:47:36.981Z terminate called after throwing an instance of 'c10::DistBackendError'
2025-01-22T15:47:36.981Z   what():  [PG 0 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3472, OpType=ALLGATHER, NumelIn=1, NumelOut=16, Timeout(ms)=1800000) ran for 1800058 milliseconds before timing out.
2025-01-22T15:47:36.981Z Exception raised from checkTimeout at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):
2025-01-22T15:47:36.981Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0x99 (0x7fd57c798e89 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2025-01-22T15:47:36.981Z frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional&lt;std::chrono::duration&lt;long, std::ratio&lt;1l, 1000l&gt; &gt; &gt;) + 0x1e1 (0x7fd5190ae121 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2025-01-22T15:47:36.981Z frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7fd5190b54e0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2025-01-22T15:47:36.981Z frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10f (0x7fd5190b63ff in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2025-01-22T15:47:36.981Z frame #4: &lt;unknown function&gt; + 0xdc253 (0x7fd629ab0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
2025-01-22T15:47:36.981Z frame #5: &lt;unknown function&gt; + 0x94ac3 (0x7fd633b44ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2025-01-22T15:47:36.981Z frame #6: clone + 0x44 (0x7fd633bd5a04 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2025-01-22T15:47:36.981Z none
2025-01-22T15:47:36.981Z Exception raised from ncclCommWatchdog at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):
2025-01-22T15:47:36.981Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0x99 (0x7fd57c798e89 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
2025-01-22T15:47:36.981Z frame #1: &lt;unknown function&gt; + 0x103534e (0x7fd5190dd34e in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2025-01-22T15:47:36.981Z frame #2: &lt;unknown function&gt; + 0xcb0e25 (0x7fd518d58e25 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
2025-01-22T15:47:36.981Z frame #3: &lt;unknown function&gt; + 0xdc253 (0x7fd629ab0253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
2025-01-22T15:47:36.981Z frame #4: &lt;unknown function&gt; + 0x94ac3 (0x7fd633b44ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2025-01-22T15:47:36.981Z frame #5: clone + 0x44 (0x7fd633bd5a04 in /usr/lib/x86_64-linux-gnu/libc.so.6)
2025-01-22T15:47:36.981Z none
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d9c2822718f6d0edaf2f1034b220fae1457600d.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d9c2822718f6d0edaf2f1034b220fae1457600d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d9c2822718f6d0edaf2f1034b220fae1457600d_2_690x445.png" alt="image" data-base62-sha1="1WoLPvACmRkHZQpAfWP01NR8aUB" width="690" height="445" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d9c2822718f6d0edaf2f1034b220fae1457600d_2_690x445.png, https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d9c2822718f6d0edaf2f1034b220fae1457600d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d9c2822718f6d0edaf2f1034b220fae1457600d.png 2x" data-dominant-color="312F2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">840×542 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>CPU usage and mem usage<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/029a9fa6f7ea58b0b690775e199ec3b55418920a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/029a9fa6f7ea58b0b690775e199ec3b55418920a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/029a9fa6f7ea58b0b690775e199ec3b55418920a_2_690x128.jpeg" alt="image" data-base62-sha1="n2exFojtzLx3RnAXi2AvZbGuW6" width="690" height="128" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/029a9fa6f7ea58b0b690775e199ec3b55418920a_2_690x128.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/029a9fa6f7ea58b0b690775e199ec3b55418920a_2_1035x192.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/029a9fa6f7ea58b0b690775e199ec3b55418920a_2_1380x256.jpeg 2x" data-dominant-color="31332A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1430×266 71.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>GPU usage and GPU mem usage<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/28ced4cbb7ee26ca8b0a6e78c32249ed62802a1f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/28ced4cbb7ee26ca8b0a6e78c32249ed62802a1f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28ced4cbb7ee26ca8b0a6e78c32249ed62802a1f_2_690x104.jpeg" alt="image" data-base62-sha1="5P0eqt1rVysuMEcmrLypOYSec23" width="690" height="104" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28ced4cbb7ee26ca8b0a6e78c32249ed62802a1f_2_690x104.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28ced4cbb7ee26ca8b0a6e78c32249ed62802a1f_2_1035x156.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/8/28ced4cbb7ee26ca8b0a6e78c32249ed62802a1f_2_1380x208.jpeg 2x" data-dominant-color="4E4249"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1418×214 62.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I set NCCL_DEBUG=INFO, and the logs of other ranks seem to be normal, only rank[2] shows connection closed from other ranks, but only this information is available. Can anyone help me? Thanks.</p>
<pre><code class="lang-auto">cnwla-a800-p01072:1267938:1278813 [2] NCCL INFO [Service thread] Connection closed by localRank 5
cnwla-a800-p01072:1267938:1278813 [2] NCCL INFO [Service thread] Connection closed by localRank 7
cnwla-a800-p01072:1267938:1278813 [2] NCCL INFO [Service thread] Connection closed by localRank 3
cnwla-a800-p01072:1267938:1278813 [2] NCCL INFO [Service thread] Connection closed by localRank 6
cnwla-a800-p01072:1267938:1278813 [2] NCCL INFO [Service thread] Connection closed by localRank 1
cnwla-a800-p01072:1267938:1278813 [2] NCCL INFO [Service thread] Connection closed by localRank 4
</code></pre>