RuntimeError: NCCL communicator was aborted

fermat97: <p>Hi, I am using DDP on a single node with <code>NCCL</code> backend. After a couple of training epochs I got the following warning:</p>
<pre><code class="lang-auto">[E ProcessGroupNCCL.cpp:587] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1803308 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:587] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1803187 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:587] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1803385 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:587] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1803386 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:587] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1803385 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:587] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1802504 milliseconds before timing out.
</code></pre>
<p>and then I got the following traceback on each of the GPUs:</p>
<pre><code class="lang-auto">  File "/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/venv/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 878, in forward
    self._sync_params()
  File "/venv/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1379, in _sync_params
    self._distributed_broadcast_coalesced(
  File "/venv/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1334, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(
RuntimeError: NCCL communicator was aborted on rank 1.  Original reason for failure was: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1803385 milliseconds before timing out.
</code></pre>
<p>I am using torch version <code>1.10.0+cu102</code> with <code>python3.9</code>. Any idea?</p>

pritamdamania87: <p>This probably indicates that there might have been some sort of CUDA/NCCL deadlock causing these timeouts. There are a few ways to debug this:</p>
<ol>
<li>Set environment variable <code>NCCL_DEBUG=INFO</code>, this will print NCCL debugging information.</li>
<li>Set environment variable <code>TORCH_DISTRIBUTED_DETAIL=DEBUG</code>, this will add significant additional overhead but will give you an exact error if there are mismatched collectives.</li>
</ol>

rvarm1: <p>You can also try passing <code>broadcast_buffers=False</code> to DDP, although note that this will disable buffer synchronization which might affect model quality if you wanted to ensure all buffers are synchronized.</p>

fermat97: <p>Thank you <a class="mention" href="/u/pritamdamania87">@pritamdamania87</a> <a class="mention" href="/u/rvarm1">@rvarm1</a> for your replies.</p>
<p><a class="mention" href="/u/pritamdamania87">@pritamdamania87</a> I am running with the <code>env variables</code> you mentioned to see the debugging log. Since it is randomly generated at some epochs during the training.</p>
<p><a class="mention" href="/u/rvarm1">@rvarm1</a> is it a standard solution to avoid such problems?</p>

Zhang_Kin: <p>I also met this problem on torch 1.10.0+cu113 with python 3.7 in docker when running at 3090, I didn’t find solution to solve this still.</p>

LaCandela: <p>I am also having this problem w/o solution…<br>
using Docker<br>
Python 3.9.7<br>
torch 1.10.2<br>
Ubuntu 20.04.3 LTS<br>
8 V100 nVidia GPU in DELL DGX<br>
using torch.distributed.launch</p>

Yanli_Zhao: <aside class="quote no-group" data-username="pritamdamania87" data-post="2" data-topic="136630">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/48/28891_2.png" class="avatar"> pritamdamania87:</div>
<blockquote>
<p>Set environment variable <code>TORCH_DISTRIBUTED_DETAIL=DEBUG</code>, this will add significant additional overhead but will give you an exact error if there are mismatched collectives.</p>
</blockquote>
</aside>
<p>For NCCL timeout, mostly means there are some desync issues, would you please try to set environment variable <code>TORCH_DISTRIBUTED_DETAIL=DEBUG</code>, this will add significant additional overhead but will give you an exact error if there are mismatched collectives.</p>

amsword: <p>is there a way to disable the following timeout and allow it hanging there forever? Then, there is a way to inspect what is going on for each rank for further debugging.</p>
<pre><code class="lang-auto">[1,158]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 158] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1803931 milliseconds before timing out.
[1,105]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 105] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804038 milliseconds before timing out.
[1,110]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 110] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804067 milliseconds before timing out.
[1,207]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 207] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804115 milliseconds before timing out.
[1,248]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 248] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804127 milliseconds before timing out.
[1,202]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 202] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804206 milliseconds before timing out.
[1,106]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 106] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804245 milliseconds before timing out.
[1,104]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 104] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804245 milliseconds before timing out.
[1,251]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 251] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804274 milliseconds before timing out.
[1,205]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 205] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804314 milliseconds before timing out.
[1,203]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 203] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804386 milliseconds before timing out.
[1,252]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 252] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804396 milliseconds before timing out.
[1,254]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 254] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804410 milliseconds before timing out.
[1,255]&lt;stderr&gt;:[E ProcessGroupNCCL.cpp:566] [Rank 255] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1804451 milliseconds before timing out.
</code></pre>