NCCL timed out when using the torch.distributed.run

username: nikiguo93
cooked: <p>Hi, when I use the DDP to train my model, after 1 epoch, I got the folowing error message:<br>
[E ProcessGroupNCCL.cpp:325] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1803203 milliseconds before timing out.</p>
<p>Can anybody help me in solving that problem?</p>

username: ptrblck
cooked: <p>Could you rerun your script via:</p>
<pre><code class="lang-auto">export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL
export TORCH_DISTRIBUTED_DEBUG=INFO
</code></pre>
<p>and see if you would receive any warnings or errors?</p>

username: nikiguo93
cooked: <p>Hi, after adding those I got the following information:</p>
<p>File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 1229, in _process_data<br>
data.reraise()<br>
File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/_utils.py”, line 425, in reraise<br>
raise self.exc_type(msg)<br>
IsADirectoryError: Caught IsADirectoryError in DataLoader worker process 1.<br>
Original Traceback (most recent call last):<br>
File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py”, line 287, in _worker_loop<br>
data = fetcher.fetch(index)<br>
File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/home/nikiguo93/scratch_multitask_Imagenet.py”, line 109, in <strong>getitem</strong><br>
TheImage = Image.open(self.ImageNames[idx]).convert(“RGB”)<br>
File “/usr/lib/python3/dist-packages/PIL/Image.py”, line 2904, in open<br>
fp = builtins.open(filename, “rb”)<br>
IsADirectoryError: [Errno 21] Is a directory: ‘/local_scratch/nikiguo93/train/hpc-work/Codebase’</p>
<p>torch.Size([300, 17, 100])<br>
for train: torch.Size([300, 17, 100])<br>
for train: torch.Size([300, 17, 100])<br>
for train: tensor(0.4474, device=‘cuda:3’, grad_fn=) tensor(6.9942, device=‘cuda:3’, grad_fn=)<br>
tensor(42.7889, device=‘cuda:3’, grad_fn=)<br>
smicro01:681176:681927 [3] NCCL INFO AllReduce: opCount 485 sendbuff 0x7f5c10000000 recvbuff 0x7f5c10000000 count 11037324 datatype 6 op 0 root 0 comm 0x7f5f3c002e10 [nranks=4] stream 0x8070110<br>
smicro01:681176:681927 [3] NCCL INFO AllReduce: opCount 486 sendbuff 0x7f5ab8b00000 recvbuff 0x7f5ab8b00000 count 10289152 datatype 6 op 0 root 0 comm 0x7f5f3c002e10 [nranks=4] stream 0x8070110<br>
tensor(0.4443, device=‘cuda:0’, grad_fn=) tensor(7.0288, device=‘cuda:0’, grad_fn=)<br>
tensor(0.4410, device=‘cuda:1’, grad_fn=) tensor(42.5709, device=‘cuda:0’, grad_fn=)<br>
tensor(7.0794, device=‘cuda:1’, grad_fn=)<br>
tensor(42.3626, device=‘cuda:1’, grad_fn=)<br>
smicro01:681173:681928 [0] NCCL INFO AllReduce: opCount 485 sendbuff 0x7fc842000000 recvbuff 0x7fc842000000 count 11037324 datatype 6 op 0 root 0 comm 0x7fcb64002e10 [nranks=4] stream 0x8d29010<br>
smicro01:681174:681919 [1] NCCL INFO AllReduce: opCount 485 sendbuff 0x7f1418000000 recvbuff 0x7f1418000000 count 11037324 datatype 6 op 0 root 0 comm 0x7f1734002e10 [nranks=4] stream 0x699a060<br>
smicro01:681173:681928 [0] NCCL INFO AllReduce: opCount 486 sendbuff 0x7fc6e4b00000 recvbuff 0x7fc6e4b00000 count 10289152 datatype 6 op 0 root 0 comm 0x7fcb64002e10 [nranks=4] stream 0x8d29010<br>
smicro01:681174:681919 [1] NCCL INFO AllReduce: opCount 486 sendbuff 0x7f12c0b00000 recvbuff 0x7f12c0b00000 count 10289152 datatype 6 op 0 root 0 comm 0x7f1734002e10 [nranks=4] stream 0x699a060<br>
smicro01:681176:681927 [3] NCCL INFO AllReduce: opCount 487 sendbuff 0x7f5aa8000000 recvbuff 0x7f5aa8000000 count 7660736 datatype 6 op 0 root 0 comm 0x7f5f3c002e10 [nranks=4] stream 0x8070110<br>
smicro01:681176:681927 [3] NCCL INFO AllReduce: opCount 488 sendbuff 0x7f5a27648000 recvbuff 0x7f5a27648000 count 53120 datatype 7 op 0 root 0 comm 0x7f5f3c002e10 [nranks=4] stream 0x8070110<br>
smicro01:681173:681928 [0] NCCL INFO AllReduce: opCount 487 sendbuff 0x7fc6d4000000 recvbuff 0x7fc6d4000000 count 7660736 datatype 6 op 0 root 0 comm 0x7fcb64002e10 [nranks=4] stream 0x8d29010<br>
smicro01:681174:681919 [1] NCCL INFO AllReduce: opCount 487 sendbuff 0x7f12b0000000 recvbuff 0x7f12b0000000 count 7660736 datatype 6 op 0 root 0 comm 0x7f1734002e10 [nranks=4] stream 0x699a060<br>
smicro01:681173:681928 [0] NCCL INFO AllReduce: opCount 488 sendbuff 0x7fc653648000 recvbuff 0x7fc653648000 count 53120 datatype 7 op 0 root 0 comm 0x7fcb64002e10 [nranks=4] stream 0x8d29010<br>
smicro01:681174:681919 [1] NCCL INFO AllReduce: opCount 488 sendbuff 0x7f122f648000 recvbuff 0x7f122f648000 count 53120 datatype 7 op 0 root 0 comm 0x7f1734002e10 [nranks=4] stream 0x699a060<br>
[ERROR] 2022-06-04 17:20:41,039 api: failed (exitcode: 1) local_rank: 2 (pid: 681175) of binary: /usr/bin/python3<br>
[ERROR] 2022-06-04 17:20:41,039 local_elastic_agent: [default] Worker group failed<br>
[INFO] 2022-06-04 17:20:41,040 api: Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish<br>
/home/nikiguo93/.local/lib/python3.9/site-packages/torch/distributed/elastic/utils/store.py:70: FutureWarning: This is an experimental API and will be changed in future.<br>
warnings.warn(<br>
[INFO] 2022-06-04 17:20:41,041 api: Done waiting for other agents. Elapsed: 0.0005986690521240234 seconds<br>
{“name”: “torchelastic.worker.status.TERMINATED”, “source”: “WORKER”, “timestamp”: 0, “metadata”: {“run_id”: “none”, “global_rank”: 0, “group_rank”: 0, “worker_id”: “681173”, “role”: “default”, “hostname”: “<a href="http://smicro01.physik.fu-berlin.de" rel="noopener nofollow ugc">smicro01.physik.fu-berlin.de</a>”, “state”: “TERMINATED”, “total_run_time”: 1602, “rdzv_backend”: “static”, “raw_error”: null, “metadata”: “{“group_world_size”: 1, “entry_point”: “python3”, “local_rank”: [0], “role_rank”: [0], “role_world_size”: [4]}”, “agent_restarts”: 0}}<br>
{“name”: “torchelastic.worker.status.TERMINATED”, “source”: “WORKER”, “timestamp”: 0, “metadata”: {“run_id”: “none”, “global_rank”: 1, “group_rank”: 0, “worker_id”: “681174”, “role”: “default”, “hostname”: “<a href="http://smicro01.physik.fu-berlin.de" rel="noopener nofollow ugc">smicro01.physik.fu-berlin.de</a>”, “state”: “TERMINATED”, “total_run_time”: 1602, “rdzv_backend”: “static”, “raw_error”: null, “metadata”: “{“group_world_size”: 1, “entry_point”: “python3”, “local_rank”: [1], “role_rank”: [1], “role_world_size”: [4]}”, “agent_restarts”: 0}}<br>
{“name”: “torchelastic.worker.status.FAILED”, “source”: “WORKER”, “timestamp”: 0, “metadata”: {“run_id”: “none”, “global_rank”: 2, “group_rank”: 0, “worker_id”: “681175”, “role”: “default”, “hostname”: “<a href="http://smicro01.physik.fu-berlin.de" rel="noopener nofollow ugc">smicro01.physik.fu-berlin.de</a>”, “state”: “FAILED”, “total_run_time”: 1602, “rdzv_backend”: “static”, “raw_error”: “{“message”: “”}”, “metadata”: “{“group_world_size”: 1, “entry_point”: “python3”, “local_rank”: [2], “role_rank”: [2], “role_world_size”: [4]}”, “agent_restarts”: 0}}<br>
{“name”: “torchelastic.worker.status.TERMINATED”, “source”: “WORKER”, “timestamp”: 0, “metadata”: {“run_id”: “none”, “global_rank”: 3, “group_rank”: 0, “worker_id”: “681176”, “role”: “default”, “hostname”: “<a href="http://smicro01.physik.fu-berlin.de" rel="noopener nofollow ugc">smicro01.physik.fu-berlin.de</a>”, “state”: “TERMINATED”, “total_run_time”: 1602, “rdzv_backend”: “static”, “raw_error”: null, “metadata”: “{“group_world_size”: 1, “entry_point”: “python3”, “local_rank”: [3], “role_rank”: [3], “role_world_size”: [4]}”, “agent_restarts”: 0}}<br>
{“name”: “torchelastic.worker.status.SUCCEEDED”, “source”: “AGENT”, “timestamp”: 0, “metadata”: {“run_id”: “none”, “global_rank”: null, “group_rank”: 0, “worker_id”: null, “role”: “default”, “hostname”: “<a href="http://smicro01.physik.fu-berlin.de" rel="noopener nofollow ugc">smicro01.physik.fu-berlin.de</a>”, “state”: “SUCCEEDED”, “total_run_time”: 1602, “rdzv_backend”: “static”, “raw_error”: null, “metadata”: “{“group_world_size”: 1, “entry_point”: “python3”}”, “agent_restarts”: 0}}<br>
/home/nikiguo93/.local/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py:354: UserWarning:</p>
<hr>
<pre><code>           CHILD PROCESS FAILED WITH NO ERROR_FILE
</code></pre>
<hr>
<p>CHILD PROCESS FAILED WITH NO ERROR_FILE<br>
Child process 681175 (local_rank 2) FAILED (exitcode 1)<br>
Error msg: Process failed with exitcode 1<br>
Without writing an error file to &lt;N/A&gt;.<br>
While this DOES NOT affect the correctness of your application,<br>
no trace information about the error will be available for inspection.<br>
Consider decorating your top level entrypoint function with<br>
torch.distributed.elastic.multiprocessing.errors.record. Example:</p>
<p>from torch.distributed.elastic.multiprocessing.errors import record</p>
<p><span class="mention">@record</span><br>
def trainer_main(args):<br>
# do train</p>
<hr>
<p>warnings.warn(_no_error_file_warning_msg(rank, failure))<br>
Traceback (most recent call last):<br>
File “/usr/lib/python3.9/runpy.py”, line 197, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/usr/lib/python3.9/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/distributed/run.py”, line 637, in <br>
main()<br>
File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/distributed/run.py”, line 629, in main<br>
run(args)<br>
File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/distributed/run.py”, line 621, in run<br>
elastic_launch(<br>
File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 116, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 348, in wrapper<br>
return f(*args, **kwargs)<br>
File “/home/nikiguo93/.local/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 245, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</p>
<hr>
<h1>
<a name="scratch_multitask_imagenetpy-failed-1" class="anchor" href="#scratch_multitask_imagenetpy-failed-1"></a>scratch_multitask_Imagenet.py FAILED</h1>
<h1>
<a name="root-cause-0-time-2022-06-04_172039-rank-2-local_rank-2-exitcode-1-pid-681175-error_file-na-msg-process-failed-with-exitcode-1-2" class="anchor" href="#root-cause-0-time-2022-06-04_172039-rank-2-local_rank-2-exitcode-1-pid-681175-error_file-na-msg-process-failed-with-exitcode-1-2"></a>Root Cause:<br>
[0]:<br>
time: 2022-06-04_17:20:39<br>
rank: 2 (local_rank: 2)<br>
exitcode: 1 (pid: 681175)<br>
error_file: &lt;N/A&gt;<br>
msg: “Process failed with exitcode 1”</h1>
<p>Other Failures:<br>
&lt;NO_OTHER_FAILURES&gt;</p>
<hr>
<hr>
<p>Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.</p>

username: ptrblck
cooked: <p>The error is caused by:</p>
<pre><code class="lang-python">File “/home/nikiguo93/scratch_multitask_Imagenet.py”, line 109, in getitem
TheImage = Image.open(self.ImageNames[idx]).convert(“RGB”)
File “/usr/lib/python3/dist-packages/PIL/Image.py”, line 2904, in open
fp = builtins.open(filename, “rb”)
IsADirectoryError: [Errno 21] Is a directory: ‘/local_scratch/nikiguo93/train/hpc-work/Codebase’
</code></pre>
<p>as the dataset cannot open this file.<br>
The DDP run just re-raises this error so make sure you are able to properly load all samples in the dataset.</p>