Multi GPU training timeout error: WorkNCCL(OpType=ALLGATHER, Timeout

username: jameelhassan
cooked: <p>I am running the codebase in <a href="https://github.com/salesforce/MUST" class="inline-onebox" rel="noopener nofollow ugc">GitHub - salesforce/MUST: PyTorch code for MUST</a> as it is, in 16 V100 GPUs.</p>
<p>After training, during the evaluation, I get the following error. The error occurs for the Imagenet dataset. For a smaller dataset such as Caltech101, it does not happen. Increasing the timeout limit beyond 1800 seconds just delays in which iteration the error occurs during the evaluation.</p>
<p>Solving this would be a great start to the new year <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<h1>
<a name="test-209010010-eta-15332-acc1-733203-766279-ema_acc1-777734-764956-time-08568-data-00-1" class="anchor" href="#test-209010010-eta-15332-acc1-733203-766279-ema_acc1-777734-764956-time-08568-data-00-1"></a>Test: [ 2090/10010] eta: 1:53:32 acc1: 73.3203 (76.6279) ema_acc1: 77.7734 (76.4956) time: 0.8568 data: 0.0</h1>
<p>001 max mem: 14135<br>
[E ProcessGroupNCCL.cpp:587] [Rank 12] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1800172 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 15] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1800768 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
[E ProcessGroupNCCL.cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 9] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1801901 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 8] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1801862 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 6] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1801679 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1802173 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 11] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1802073 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1801667 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1801777 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 7] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1801821 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 10] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1802399 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1802025 milliseconds before timing out.<br>
[E ProcessGroupNCCL.cpp:587] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1802703 milliseconds before timing out.<br>
Test: [ 2100/10010] eta: 1:53:23 acc1: 79.5312 (76.6506) ema_acc1: 81.9922 (76.5298) time: 0.8566 data: 0.0<br>
001 max mem: 14135<br>
[E ProcessGroupNCCL.cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
[E ProcessGroupNCCL.cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
[E ProcessGroupNCCL.cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
[E ProcessGroupNCCL.cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
[E ProcessGroupNCCL.cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
[E ProcessGroupNCCL.cpp:341] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1546 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1549 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1553 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1554 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1555 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1556 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1558 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1562 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1563 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1564 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1565 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1566 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1570 closing signal SIGTERM<br>
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 1574 closing signal SIGTERM<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 12 (pid: 1567) of binary: /<br>
nfs/users/ext_jameel.hassan/anaconda3/envs/must/bin/python<br>
Traceback (most recent call last): File “/nfs/users/ext_jameel.hassan/anaconda3/envs/must/lib/python3.8/runpy.py”, line 194, in _run_module_as_mai n<br>
return _run_code(code, main_globals, None,<br>
File “/nfs/users/ext_jameel.hassan/anaconda3/envs/must/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/nfs/users/ext_jameel.hassan/anaconda3/envs/must/lib/python3.8/site-packages/torch/distributed/run.py”, l<br>
ine 723, in<br>
main()<br>
File “/nfs/users/ext_jameel.hassan/anaconda3/envs/must/lib/python3.8/site-packages/torch/distributed/elastic/mu<br>
ltiprocessing/errors/<strong>init</strong>.py”, line 345, in wrapper<br>
return f(*args, **kwargs)<br>
File “/nfs/users/ext_jameel.hassan/anaconda3/envs/must/lib/python3.8/site-packages/torch/distributed/run.py”, l<br>
ine 719, in main<br>
run(args)<br>
File “/nfs/users/ext_jameel.hassan/anaconda3/envs/must/lib/python3.8/site-packages/torch/distributed/run.py”, l<br>
ine 710, in run<br>
elastic_launch(<br>
File “/nfs/users/ext_jameel.hassan/anaconda3/envs/must/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 131, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/nfs/users/ext_jameel.hassan/anaconda3/envs/must/lib/python3.8/site-packages/torch/distributed/launcher/api.py”, line 259, in launch_agent<br>
raise ChildFailedError(torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</p>
<h2>
<a name="trainpy-failed-2" class="anchor" href="#trainpy-failed-2"></a>train.py FAILED</h2>
<h2>
<a name="failures-3" class="anchor" href="#failures-3"></a>Failures:</h2>
<p>[1]:<br>
time : 2023-01-02_15:24:36<br>
host : HOST<br>
rank : 15 (local_rank: 15)<br>
exitcode : -6 (pid: 1575)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 6 (SIGABRT) received by PID 1575</p>
<h1>
<a name="root-cause-first-observed-failure-4" class="anchor" href="#root-cause-first-observed-failure-4"></a>Root Cause (first observed failure):</h1>
<p>[0]:<br>
time : 2023-01-02_15:24:36<br>
host : HOST<br>
rank : 12 (local_rank: 12)<br>
exitcode : -6 (pid: 1567)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 6 (SIGABRT) received by PID 1567</p>

username: jameelhassan
cooked: <p>Found the bug. I was using the train images for validation which caused the timeout. I believe that is because the evaluation is run on a single GPU, and when the time limit of 30mins is reached it kills the process.</p>

username: Qi_Yan
cooked: <p>Thanks for your hints!</p>

username: nero1
cooked: <aside class="quote no-group" data-username="jameelhassan" data-post="2" data-topic="169435" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/71c47a/48.png" class="avatar"> jameelhassan:</div>
<blockquote>
<p>Found the bug. I was using the train images for validation which caused the timeout. I believe that is because the evaluation is run on a single GPU, and when the time limit of 30mins is reached it kills the process.</p>
</blockquote>
</aside>
<p>Hi, I encountered the same issue. Could you explain why validation causes timeouts?</p>

username: jameelhassan
cooked: <p>I am not certain since I do not recall. I guess that the validation is done during training after X epochs. However, the validation script is run on a single GPU, during which time, the rest of the GPUs (set for training) are idle. Since the training images are used for validation – which are way more, it takes a lot longer, causing the idle GPUs to be idle for too long, hence killing the process.</p>

username: nero1
cooked: <p>Thanks! My training images are also used for validation, which takes a long time and NCCL timeout error occurred in this period. However, my validation runs on all GPUs, I think all GPUs are busy <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20">.</p>