NCCL Timeout when fail one node - Torchrun Elastic

username: Leandro_Libutti
cooked: <p>Hi, I´m running two nodes (on the same machine) where each one has an assigned gpu. I´m testing the scale-down and when I fail one node with SIGTERM signal I get hung the other node.  After a few minutes, the NCCL watchdog throws the following error:</p>
<blockquote>
<p>[rank0]:[E ProcessGroupNCCL.cpp:563] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3917, OpType=ALLREDUCE, NumelIn=513000, NumelOut=513000, Timeout(ms)=600000) ran for 600016 milliseconds before timing out.<br>
[rank0]:[E ProcessGroupNCCL.cpp:1537] [PG 0 Rank 0] Timeout at NCCL work: 3917, last enqueued NCCL work: 3928, last completed NCCL work: 3916.<br>
[rank0]:[E ProcessGroupNCCL.cpp:577] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.<br>
[rank0]:[E ProcessGroupNCCL.cpp:583] [Rank 0] To avoid data inconsistency, we are taking the entire process down.<br>
[rank0]:[E ProcessGroupNCCL.cpp:1414] [PG 0 Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3917, OpType=ALLREDUCE, NumelIn=513000, NumelOut=513000, Timeout(ms)=600000) ran for 600016 milliseconds before timing out.<br>
Exception raised from checkTimeout at …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6880ecf897 in /home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span> c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional&lt;std::chrono::duration&lt;long, std::ratio&lt;1l, 1000l&gt; &gt; &gt;) + 0x1d2 (0x7f6834a68c62 in /home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#2:</span> c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f6834a6da80 in /home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#3:</span> c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f6834a6edcc in /home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#4:</span>  + 0xd44a3 (0x7f68804d44a3 in /lib/x86_64-linux-gnu/libstdc++.so.6)<br>
frame <span class="hashtag-raw">#5:</span>  + 0x89134 (0x7f6881d2f134 in /lib/x86_64-linux-gnu/libc.so.6)<br>
frame <span class="hashtag-raw">#6:</span>  + 0x1097dc (0x7f6881daf7dc in /lib/x86_64-linux-gnu/libc.so.6)</p>
<p>terminate called after throwing an instance of ‘c10::DistBackendError’<br>
what():  [PG 0 Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3917, OpType=ALLREDUCE, NumelIn=513000, NumelOut=513000, Timeout(ms)=600000) ran for 600016 milliseconds before timing out.<br>
Exception raised from checkTimeout at …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:565 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6880ecf897 in /home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span> c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional&lt;std::chrono::duration&lt;long, std::ratio&lt;1l, 1000l&gt; &gt; &gt;) + 0x1d2 (0x7f6834a68c62 in /home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#2:</span> c10d::ProcessGroupNCCL::watchdogHandler() + 0x1a0 (0x7f6834a6da80 in /home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#3:</span> c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7f6834a6edcc in /home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#4:</span>  + 0xd44a3 (0x7f68804d44a3 in /lib/x86_64-linux-gnu/libstdc++.so.6)<br>
frame <span class="hashtag-raw">#5:</span>  + 0x89134 (0x7f6881d2f134 in /lib/x86_64-linux-gnu/libc.so.6)<br>
frame <span class="hashtag-raw">#6:</span>  + 0x1097dc (0x7f6881daf7dc in /lib/x86_64-linux-gnu/libc.so.6)</p>
<p>Exception raised from ncclCommWatchdog at …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1418 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f6880ecf897 in /home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span>  + 0xe32119 (0x7f68346f2119 in /home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag-raw">#2:</span>  + 0xd44a3 (0x7f68804d44a3 in /lib/x86_64-linux-gnu/libstdc++.so.6)<br>
frame <span class="hashtag-raw">#3:</span>  + 0x89134 (0x7f6881d2f134 in /lib/x86_64-linux-gnu/libc.so.6)<br>
frame <span class="hashtag-raw">#4:</span>  + 0x1097dc (0x7f6881daf7dc in /lib/x86_64-linux-gnu/libc.so.6)</p>
<p>Fatal Python error: Aborted</p>
<p>Thread 0x00007f6763bd36c0 (most recent call first):<br>
</p>
<p>Thread 0x00007f676dbd76c0 (most recent call first):<br>
</p>
<p>Thread 0x00007f6881ca52c0 (most recent call first):<br>
File “/home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/autograd/graph.py”, line 744 in _engine_run_backward<br>
File “/home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/autograd/<strong>init</strong>.py”, line 267 in backward<br>
File “/home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/_tensor.py”, line 525 in backward<br>
File “/home/llibutti/pytorch_env/torchx_examples/resnet_elastic1.py”, line 82 in _run_batch<br>
File “/home/llibutti/pytorch_env/torchx_examples/resnet_elastic1.py”, line 97 in _run_epoch<br>
File “/home/llibutti/pytorch_env/torchx_examples/resnet_elastic1.py”, line 110 in train<br>
File “/home/llibutti/pytorch_env/torchx_examples/resnet_elastic1.py”, line 148 in main<br>
File “/home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 347 in wrapper<br>
File “/home/llibutti/pytorch_env/torchx_examples/resnet_elastic1.py”, line 160 in </p>
<h1>Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, PIL._imaging, PIL._imagingft (total: 22)<br>
E0708 16:00:27.766000 140702371304128 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: -6) local_rank: 0 (pid: 977782) of binary: /home/llibutti/pytorch_env/pytorch_env/bin/python3<br>
Traceback (most recent call last):<br>
File “/home/llibutti/pytorch_env/pytorch_env/bin/torchrun”, line 8, in <br>
sys.exit(main())<br>
^^^^^^<br>
File “/home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 347, in wrapper<br>
return f(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^<br>
File “/home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/distributed/run.py”, line 879, in main<br>
run(args)<br>
File “/home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/distributed/run.py”, line 870, in run<br>
elastic_launch(<br>
File “/home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/distributed/launcher/api.py”, line 132, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/llibutti/pytorch_env/pytorch_env/lib/python3.11/site-packages/torch/distributed/launcher/api.py”, line 263, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2>resnet_elastic1.py FAILED</h2>
<h2>Failures:<br>
&lt;NO_OTHER_FAILURES&gt;</h2>
<h1>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2024-07-08_16:00:27<br>
host      : marbore.dacya.ucm.es<br>
rank      : 0 (local_rank: 0)<br>
exitcode  : -6 (pid: 977782)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 6 (SIGABRT) received by PID 977782</h1>
</blockquote>
<p>The commands launched to execute are as follows:</p>
<p><code>torchrun --nnodes=1:2 --nproc-per-node=1 --rdzv_id=22  --rdzv_backend=etcd-v2 --rdzv_endpoint=localhost:2379 --node_rank=0 --rdzv-conf=is_host=1 resnet_elastic1.py 10 1</code></p>
<p><code>torchrun --nnodes=1:2 --nproc-per-node=1 --rdzv_id=22  --rdzv_backend=etcd-v2 --rdzv_endpoint=localhost:2379 --node_rank=1 --rdzv-conf=is_host=0 resnet_elastic1.py 10 1</code></p>
<p>I understand that the problem is in the ALLREDUCE of the backward (in DDP model) that is not performed because a node failed. How do I restart torchrun so that it uses only alive node at that point?</p>