NCCL error when running distributed training

ruka: <p>My code used to work in PyTorch 1.6. Recently it was upgraded to 1.9. When I try to do training under distributed mode(but actually I only have 1 PC with 2 GPUs, not several PCs), following error happens, sorry for the long log, I’ve never seen it before and totally lost.</p>
<blockquote>
<p>$ python -m torch.distributed.run --standalone --nnodes=1 --nproc_per_node=2 train_dist_2.py<br>
[INFO] 2021-08-13 18:21:14,035 run: Running torch.distributed.run with args: [‘/usr/lib/python3.9/site-packages/torch/distributed/run.py’, ‘–standalone’, ‘–nnodes=1’, ‘–nproc_per_node=2’, ‘train_dist_2.py’]<br>
[INFO] 2021-08-13 18:21:14,036 run:</p>
<hr>
<p>Rendezvous info:<br>
–rdzv_backend=c10d --rdzv_endpoint=localhost:29400 --rdzv_id=5c6a0ec7-2728-407d-8d25-7dde979518e6</p>
<hr>
<p>[INFO] 2021-08-13 18:21:14,036 run: Using nproc_per_node=2.</p>
<hr>
<p>Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.</p>
<hr>
<p>[INFO] 2021-08-13 18:21:14,036 api: Starting elastic_operator with launch configs:<br>
entrypoint       : train_dist_2.py<br>
min_nodes        : 1<br>
max_nodes        : 1<br>
nproc_per_node   : 2<br>
run_id           : 5c6a0ec7-2728-407d-8d25-7dde979518e6<br>
rdzv_backend     : c10d<br>
rdzv_endpoint    : localhost:29400<br>
rdzv_configs     : {‘timeout’: 900}<br>
max_restarts     : 3<br>
monitor_interval : 5<br>
log_dir          : None<br>
metrics_cfg      : {}</p>
<p>[INFO] 2021-08-13 18:21:14,059 c10d_rendezvous_backend: Process 25097 hosts the TCP store for the C10d rendezvous backend.<br>
[INFO] 2021-08-13 18:21:14,060 local_elastic_agent: log directory set to: /tmp/torchelastic_ra_2ujgp/5c6a0ec7-2728-407d-8d25-7dde979518e6_5pfgxyog<br>
[INFO] 2021-08-13 18:21:14,060 api: [default] starting workers for entrypoint: python<br>
[INFO] 2021-08-13 18:21:14,060 api: [default] Rendezvous’ing worker group<br>
[INFO] 2021-08-13 18:21:14,060 dynamic_rendezvous: The node ‘cnn_25097_0’ attempts to join the next round of the rendezvous ‘5c6a0ec7-2728-407d-8d25-7dde979518e6’.<br>
[INFO] 2021-08-13 18:21:14,277 dynamic_rendezvous: The node ‘cnn_25097_0’ has joined round 0 of the rendezvous ‘5c6a0ec7-2728-407d-8d25-7dde979518e6’ as rank 0 in a world of size 1.<br>
/usr/lib/python3.9/site-packages/torch/distributed/elastic/utils/store.py:52: FutureWarning: This is an experimental API and will be changed in future.<br>
warnings.warn(<br>
[INFO] 2021-08-13 18:21:14,278 api: [default] Rendezvous complete for workers. Result:<br>
restart_count=0<br>
master_addr=cnn<br>
master_port=36965<br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0, 1]<br>
role_ranks=[0, 1]<br>
global_ranks=[0, 1]<br>
role_world_sizes=[2, 2]<br>
global_world_sizes=[2, 2]</p>
<p>[INFO] 2021-08-13 18:21:14,278 api: [default] Starting worker group<br>
[INFO] 2021-08-13 18:21:14,278 <strong>init</strong>: Setting worker0 reply file to: /tmp/torchelastic_ra_2ujgp/5c6a0ec7-2728-407d-8d25-7dde979518e6_5pfgxyog/attempt_0/0/error.json<br>
[INFO] 2021-08-13 18:21:14,278 <strong>init</strong>: Setting worker1 reply file to: /tmp/torchelastic_ra_2ujgp/5c6a0ec7-2728-407d-8d25-7dde979518e6_5pfgxyog/attempt_0/1/error.json<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I20210813 18:21:23.793745 25104 ProcessGroupNCCL.cpp:480] [Rank 1] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 1800000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
NCCL_DEBUG: UNSET<br>
I20210813 18:21:23.793756 25154 ProcessGroupNCCL.cpp:580] [Rank 1] NCCL watchdog thread started!<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I20210813 18:21:23.801612 25103 ProcessGroupNCCL.cpp:480] [Rank 0] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 1800000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
NCCL_DEBUG: UNSET<br>
I20210813 18:21:23.801617 25157 ProcessGroupNCCL.cpp:580] [Rank 0] NCCL watchdog thread started!<br>
Device check. Running model on cuda</p>
<p>Device check. Running model on cuda</p>
<p>Traceback (most recent call last):<br>
Traceback (most recent call last):<br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 285, in <br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 285, in <br>
main(args)main(args)</p>
<p>File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 135, in main<br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 135, in main<br>
model = nn.parallel.DistributedDataParallel(model, device_ids = [args.local_rank])model = nn.parallel.DistributedDataParallel(model, device_ids = [args.local_rank])</p>
<p>File “/usr/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
File “/usr/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)dist._verify_model_across_ranks(self.process_group, parameters)</p>
<p>RuntimeErrorRuntimeError: : NCCL error in: …/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage, NCCL version 20.9.9<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).NCCL error in: …/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage, NCCL version 20.9.9<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).</p>
<p>I20210813 18:21:24.402045 25154 ProcessGroupNCCL.cpp:582] [Rank 1] NCCL watchdog thread terminated normally<br>
I20210813 18:21:24.403731 25157 ProcessGroupNCCL.cpp:582] [Rank 0] NCCL watchdog thread terminated normally<br>
[ERROR] 2021-08-13 18:21:29,303 api: failed (exitcode: 1) local_rank: 0 (pid: 25103) of binary: /usr/bin/python<br>
[ERROR] 2021-08-13 18:21:29,303 local_elastic_agent: [default] Worker group failed<br>
[INFO] 2021-08-13 18:21:29,303 api: [default] Worker group FAILED. 3/3 attempts left; will restart worker group<br>
[INFO] 2021-08-13 18:21:29,303 api: [default] Stopping worker group<br>
[INFO] 2021-08-13 18:21:29,303 api: [default] Rendezvous’ing worker group<br>
[INFO] 2021-08-13 18:21:29,303 dynamic_rendezvous: The node ‘cnn_25097_0’ attempts to join the next round of the rendezvous ‘5c6a0ec7-2728-407d-8d25-7dde979518e6’.<br>
[INFO] 2021-08-13 18:21:29,422 dynamic_rendezvous: The node ‘cnn_25097_0’ has joined round 1 of the rendezvous ‘5c6a0ec7-2728-407d-8d25-7dde979518e6’ as rank 0 in a world of size 1.<br>
[INFO] 2021-08-13 18:21:29,423 api: [default] Rendezvous complete for workers. Result:<br>
restart_count=1<br>
master_addr=cnn<br>
master_port=53181<br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0, 1]<br>
role_ranks=[0, 1]<br>
global_ranks=[0, 1]<br>
role_world_sizes=[2, 2]<br>
global_world_sizes=[2, 2]</p>
<p>[INFO] 2021-08-13 18:21:29,423 api: [default] Starting worker group<br>
[INFO] 2021-08-13 18:21:29,423 <strong>init</strong>: Setting worker0 reply file to: /tmp/torchelastic_ra_2ujgp/5c6a0ec7-2728-407d-8d25-7dde979518e6_5pfgxyog/attempt_1/0/error.json<br>
[INFO] 2021-08-13 18:21:29,423 <strong>init</strong>: Setting worker1 reply file to: /tmp/torchelastic_ra_2ujgp/5c6a0ec7-2728-407d-8d25-7dde979518e6_5pfgxyog/attempt_1/1/error.json<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I20210813 18:21:38.944895 25196 ProcessGroupNCCL.cpp:480] [Rank 1] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 1800000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
NCCL_DEBUG: UNSET<br>
I20210813 18:21:38.944903 25245 ProcessGroupNCCL.cpp:580] [Rank 1] NCCL watchdog thread started!<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I20210813 18:21:38.954780 25195 ProcessGroupNCCL.cpp:480] [Rank 0] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 1800000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
NCCL_DEBUG: UNSET<br>
I20210813 18:21:38.954794 25248 ProcessGroupNCCL.cpp:580] [Rank 0] NCCL watchdog thread started!<br>
Device check. Running model on cuda</p>
<p>Device check. Running model on cuda</p>
<p>Traceback (most recent call last):<br>
Traceback (most recent call last):<br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 285, in <br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 285, in <br>
main(args)main(args)</p>
<p>File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 135, in main<br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 135, in main<br>
model = nn.parallel.DistributedDataParallel(model, device_ids = [args.local_rank])model = nn.parallel.DistributedDataParallel(model, device_ids = [args.local_rank])</p>
<p>File “/usr/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
File “/usr/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)dist._verify_model_across_ranks(self.process_group, parameters)</p>
<p>RuntimeErrorRuntimeError: : NCCL error in: …/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage, NCCL version 20.9.9<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).NCCL error in: …/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage, NCCL version 20.9.9<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).</p>
<p>I20210813 18:21:39.523105 25248 ProcessGroupNCCL.cpp:582] [Rank 0] NCCL watchdog thread terminated normally<br>
I20210813 18:21:39.523314 25245 ProcessGroupNCCL.cpp:582] [Rank 1] NCCL watchdog thread terminated normally<br>
[ERROR] 2021-08-13 18:21:44,447 api: failed (exitcode: 1) local_rank: 0 (pid: 25195) of binary: /usr/bin/python<br>
[ERROR] 2021-08-13 18:21:44,447 local_elastic_agent: [default] Worker group failed<br>
[INFO] 2021-08-13 18:21:44,447 api: [default] Worker group FAILED. 2/3 attempts left; will restart worker group<br>
[INFO] 2021-08-13 18:21:44,447 api: [default] Stopping worker group<br>
[INFO] 2021-08-13 18:21:44,447 api: [default] Rendezvous’ing worker group<br>
[INFO] 2021-08-13 18:21:44,447 dynamic_rendezvous: The node ‘cnn_25097_0’ attempts to join the next round of the rendezvous ‘5c6a0ec7-2728-407d-8d25-7dde979518e6’.<br>
[INFO] 2021-08-13 18:21:44,448 dynamic_rendezvous: The node ‘cnn_25097_0’ has joined round 2 of the rendezvous ‘5c6a0ec7-2728-407d-8d25-7dde979518e6’ as rank 0 in a world of size 1.<br>
[INFO] 2021-08-13 18:21:44,449 api: [default] Rendezvous complete for workers. Result:<br>
restart_count=2<br>
master_addr=cnn<br>
master_port=35757<br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0, 1]<br>
role_ranks=[0, 1]<br>
global_ranks=[0, 1]<br>
role_world_sizes=[2, 2]<br>
global_world_sizes=[2, 2]</p>
<p>[INFO] 2021-08-13 18:21:44,449 api: [default] Starting worker group<br>
[INFO] 2021-08-13 18:21:44,449 <strong>init</strong>: Setting worker0 reply file to: /tmp/torchelastic_ra_2ujgp/5c6a0ec7-2728-407d-8d25-7dde979518e6_5pfgxyog/attempt_2/0/error.json<br>
[INFO] 2021-08-13 18:21:44,449 <strong>init</strong>: Setting worker1 reply file to: /tmp/torchelastic_ra_2ujgp/5c6a0ec7-2728-407d-8d25-7dde979518e6_5pfgxyog/attempt_2/1/error.json<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I20210813 18:21:52.939616 25287 ProcessGroupNCCL.cpp:480] [Rank 1] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 1800000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
NCCL_DEBUG: UNSET<br>
I20210813 18:21:52.939630 25330 ProcessGroupNCCL.cpp:580] [Rank 1] NCCL watchdog thread started!<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I20210813 18:21:52.949156 25286 ProcessGroupNCCL.cpp:480] [Rank 0] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 1800000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
NCCL_DEBUG: UNSET<br>
I20210813 18:21:52.949156 25333 ProcessGroupNCCL.cpp:580] [Rank 0] NCCL watchdog thread started!<br>
Device check. Running model on cuda</p>
<p>Device check. Running model on cuda</p>
<p>Traceback (most recent call last):<br>
Traceback (most recent call last):<br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 285, in <br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 285, in <br>
main(args)main(args)</p>
<p>File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 135, in main<br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 135, in main<br>
model = nn.parallel.DistributedDataParallel(model, device_ids = [args.local_rank])model = nn.parallel.DistributedDataParallel(model, device_ids = [args.local_rank])</p>
<p>File “/usr/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
File “/usr/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)dist._verify_model_across_ranks(self.process_group, parameters)</p>
<p>RuntimeError: RuntimeErrorNCCL error in: …/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage, NCCL version 20.9.9<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).:<br>
NCCL error in: …/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage, NCCL version 20.9.9<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).<br>
I20210813 18:21:53.513854 25330 ProcessGroupNCCL.cpp:582] [Rank 1] NCCL watchdog thread terminated normally<br>
I20210813 18:21:53.513996 25333 ProcessGroupNCCL.cpp:582] [Rank 0] NCCL watchdog thread terminated normally<br>
[ERROR] 2021-08-13 18:21:54,472 api: failed (exitcode: 1) local_rank: 0 (pid: 25286) of binary: /usr/bin/python<br>
[ERROR] 2021-08-13 18:21:54,472 local_elastic_agent: [default] Worker group failed<br>
[INFO] 2021-08-13 18:21:54,472 api: [default] Worker group FAILED. 1/3 attempts left; will restart worker group<br>
[INFO] 2021-08-13 18:21:54,472 api: [default] Stopping worker group<br>
[INFO] 2021-08-13 18:21:54,472 api: [default] Rendezvous’ing worker group<br>
[INFO] 2021-08-13 18:21:54,472 dynamic_rendezvous: The node ‘cnn_25097_0’ attempts to join the next round of the rendezvous ‘5c6a0ec7-2728-407d-8d25-7dde979518e6’.<br>
[INFO] 2021-08-13 18:21:54,473 dynamic_rendezvous: The node ‘cnn_25097_0’ has joined round 3 of the rendezvous ‘5c6a0ec7-2728-407d-8d25-7dde979518e6’ as rank 0 in a world of size 1.<br>
[INFO] 2021-08-13 18:21:54,474 api: [default] Rendezvous complete for workers. Result:<br>
restart_count=3<br>
master_addr=cnn<br>
master_port=44399<br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0, 1]<br>
role_ranks=[0, 1]<br>
global_ranks=[0, 1]<br>
role_world_sizes=[2, 2]<br>
global_world_sizes=[2, 2]</p>
<p>[INFO] 2021-08-13 18:21:54,474 api: [default] Starting worker group<br>
[INFO] 2021-08-13 18:21:54,474 <strong>init</strong>: Setting worker0 reply file to: /tmp/torchelastic_ra_2ujgp/5c6a0ec7-2728-407d-8d25-7dde979518e6_5pfgxyog/attempt_3/0/error.json<br>
[INFO] 2021-08-13 18:21:54,474 <strong>init</strong>: Setting worker1 reply file to: /tmp/torchelastic_ra_2ujgp/5c6a0ec7-2728-407d-8d25-7dde979518e6_5pfgxyog/attempt_3/1/error.json<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I20210813 18:22:03.975812 25356 ProcessGroupNCCL.cpp:480] [Rank 1] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 1800000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
NCCL_DEBUG: UNSET<br>
I20210813 18:22:03.975847 25408 ProcessGroupNCCL.cpp:580] [Rank 1] NCCL watchdog thread started!<br>
WARNING: Logging before InitGoogleLogging() is written to STDERR<br>
I20210813 18:22:03.977819 25411 ProcessGroupNCCL.cpp:580] [Rank 0] NCCL watchdog thread started!<br>
I20210813 18:22:03.977841 25355 ProcessGroupNCCL.cpp:480] [Rank 0] ProcessGroupNCCL initialized with following options:<br>
NCCL_ASYNC_ERROR_HANDLING: 1<br>
NCCL_BLOCKING_WAIT: 0<br>
TIMEOUT(ms): 1800000<br>
USE_HIGH_PRIORITY_STREAM: 0<br>
NCCL_DEBUG: UNSET<br>
Device check. Running model on cuda</p>
<p>Device check. Running model on cuda</p>
<p>Traceback (most recent call last):<br>
Traceback (most recent call last):<br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 285, in <br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 285, in <br>
main(args)main(args)</p>
<p>File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 135, in main<br>
File “/home/xxx/Desktop/MobileFaceNet-PyTorch/train_dist_2.py”, line 135, in main<br>
model = nn.parallel.DistributedDataParallel(model, device_ids = [args.local_rank])model = nn.parallel.DistributedDataParallel(model, device_ids = [args.local_rank])</p>
<p>File “/usr/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
File “/usr/lib/python3.9/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)dist._verify_model_across_ranks(self.process_group, parameters)</p>
<p>RuntimeErrorRuntimeError: : NCCL error in: …/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage, NCCL version 20.9.9<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).NCCL error in: …/torch/lib/c10d/ProcessGroupNCCL.cpp:911, invalid usage, NCCL version 20.9.9<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).</p>
<p>I20210813 18:22:04.551221 25408 ProcessGroupNCCL.cpp:582] [Rank 1] NCCL watchdog thread terminated normally<br>
I20210813 18:22:04.554879 25411 ProcessGroupNCCL.cpp:582] [Rank 0] NCCL watchdog thread terminated normally<br>
[ERROR] 2021-08-13 18:22:09,499 api: failed (exitcode: 1) local_rank: 0 (pid: 25355) of binary: /usr/bin/python<br>
[ERROR] 2021-08-13 18:22:09,499 local_elastic_agent: [default] Worker group failed<br>
[INFO] 2021-08-13 18:22:09,500 api: Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish<br>
/usr/lib/python3.9/site-packages/torch/distributed/elastic/utils/store.py:70: FutureWarning: This is an experimental API and will be changed in future.<br>
warnings.warn(<br>
[INFO] 2021-08-13 18:22:09,500 api: Done waiting for other agents. Elapsed: 0.00022149085998535156 seconds<br>
{“name”: “torchelastic.worker.status.FAILED”, “source”: “WORKER”, “timestamp”: 0, “metadata”: {“run_id”: “5c6a0ec7-2728-407d-8d25-7dde979518e6”, “global_rank”: 0, “group_rank”: 0, “worker_id”: “25355”, “role”: “default”, “hostname”: “cnn”, “state”: “FAILED”, “total_run_time”: 55, “rdzv_backend”: “c10d”, “raw_error”: “{"message": ""}”, “metadata”: “{"group_world_size": 1, "entry_point": "python", "local_rank": [0], "role_rank": [0], "role_world_size": [2]}”, “agent_restarts”: 3}}<br>
{“name”: “torchelastic.worker.status.FAILED”, “source”: “WORKER”, “timestamp”: 0, “metadata”: {“run_id”: “5c6a0ec7-2728-407d-8d25-7dde979518e6”, “global_rank”: 1, “group_rank”: 0, “worker_id”: “25356”, “role”: “default”, “hostname”: “cnn”, “state”: “FAILED”, “total_run_time”: 55, “rdzv_backend”: “c10d”, “raw_error”: “{"message": ""}”, “metadata”: “{"group_world_size": 1, "entry_point": "python", "local_rank": [1], "role_rank": [1], "role_world_size": [2]}”, “agent_restarts”: 3}}<br>
{“name”: “torchelastic.worker.status.SUCCEEDED”, “source”: “AGENT”, “timestamp”: 0, “metadata”: {“run_id”: “5c6a0ec7-2728-407d-8d25-7dde979518e6”, “global_rank”: null, “group_rank”: 0, “worker_id”: null, “role”: “default”, “hostname”: “cnn”, “state”: “SUCCEEDED”, “total_run_time”: 55, “rdzv_backend”: “c10d”, “raw_error”: null, “metadata”: “{"group_world_size": 1, "entry_point": "python"}”, “agent_restarts”: 3}}<br>
[INFO] 2021-08-13 18:22:09,501 dynamic_rendezvous: The node ‘cnn_25097_0’ has closed the rendezvous ‘5c6a0ec7-2728-407d-8d25-7dde979518e6’.<br>
/usr/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py:354: UserWarning:</p>
<hr>
<pre><code>           CHILD PROCESS FAILED WITH NO ERROR_FILE                
</code></pre>
<hr>
<p>CHILD PROCESS FAILED WITH NO ERROR_FILE<br>
Child process 25355 (local_rank 0) FAILED (exitcode 1)<br>
Error msg: Process failed with exitcode 1<br>
Without writing an error file to &lt;N/A&gt;.<br>
While this DOES NOT affect the correctness of your application,<br>
no trace information about the error will be available for inspection.<br>
Consider decorating your top level entrypoint function with<br>
torch.distributed.elastic.multiprocessing.errors.record. Example:</p>
<p>from torch.distributed.elastic.multiprocessing.errors import record</p>
<p><span class="mention">@record</span><br>
def trainer_main(args):<br>
# do train</p>
<hr>
<p>warnings.warn(_no_error_file_warning_msg(rank, failure))<br>
Traceback (most recent call last):<br>
File “/usr/lib/python3.9/runpy.py”, line 197, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/usr/lib/python3.9/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/usr/lib/python3.9/site-packages/torch/distributed/run.py”, line 637, in <br>
main()<br>
File “/usr/lib/python3.9/site-packages/torch/distributed/run.py”, line 629, in main<br>
run(args)<br>
File “/usr/lib/python3.9/site-packages/torch/distributed/run.py”, line 621, in run<br>
elastic_launch(<br>
File “/usr/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 116, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/usr/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/<strong>init</strong>.py”, line 348, in wrapper<br>
return f(*args, **kwargs)<br>
File “/usr/lib/python3.9/site-packages/torch/distributed/launcher/api.py”, line 245, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</p>
<hr>
<pre><code>     train_dist_2.py FAILED        
</code></pre>
<h1>=======================================<br>
Root Cause:<br>
[0]:<br>
time: 2021-08-13_18:22:09<br>
rank: 0 (local_rank: 0)<br>
exitcode: 1 (pid: 25355)<br>
error_file: &lt;N/A&gt;<br>
msg: “Process failed with exitcode 1”</h1>
<p>Other Failures:<br>
[1]:<br>
time: 2021-08-13_18:22:09<br>
rank: 1 (local_rank: 1)<br>
exitcode: 1 (pid: 25356)<br>
error_file: &lt;N/A&gt;<br>
msg: “Process failed with exitcode 1”</p>
<hr>
</blockquote>
<p>Any help is appreciated</p>

ptrblck: <p>You could rerun the script with <code>export NCCL_DEBUG=INFO</code> and check the logs for NCCL errors.</p>

rvandeghen: <p>Hi,</p>
<p>I’m using a cluster with multiples nodes with different GPUs (2080, Quadro, Tesla, …) and I get an error only with the quadro cards.<br>
Here is the output with <code>NCCL_DEBUG=INFO</code> for a node without problem:</p>
<pre><code class="lang-auto">| distributed init (rank 1): env://
| distributed init (rank 0): env://
| distributed init (rank 2): env://
| distributed init (rank 3): env://
compute-05:2612510:2612510 [0] NCCL INFO Bootstrap : Using enp4s0f0:&lt;0&gt;
compute-05:2612510:2612510 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
compute-05:2612510:2612510 [0] NCCL INFO NET/IB : No device found.
compute-05:2612510:2612510 [0] NCCL INFO NET/Socket : Using [0]enp4s0f0:&lt;0&gt;
compute-05:2612510:2612510 [0] NCCL INFO Using network Socket
NCCL version 2.10.3+cuda11.3
compute-05:2612511:2612511 [1] NCCL INFO Bootstrap : Using enp4s0f0:&lt;0&gt;
compute-05:2612513:2612513 [3] NCCL INFO Bootstrap : Using enp4s0f0:&lt;0&gt;
compute-05:2612511:2612511 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
compute-05:2612513:2612513 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
compute-05:2612512:2612512 [2] NCCL INFO Bootstrap : Using enp4s0f0:&lt;0&gt;
compute-05:2612511:2612511 [1] NCCL INFO NET/IB : No device found.
compute-05:2612513:2612513 [3] NCCL INFO NET/IB : No device found.
compute-05:2612512:2612512 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
compute-05:2612511:2612511 [1] NCCL INFO NET/Socket : Using [0]enp4s0f0:&lt;0&gt;
compute-05:2612511:2612511 [1] NCCL INFO Using network Socket
compute-05:2612512:2612512 [2] NCCL INFO NET/IB : No device found.
compute-05:2612513:2612513 [3] NCCL INFO NET/Socket : Using [0]enp4s0f0:&lt;0&gt;
compute-05:2612513:2612513 [3] NCCL INFO Using network Socket
compute-05:2612512:2612512 [2] NCCL INFO NET/Socket : Using [0]enp4s0f0:&lt;0&gt;
compute-05:2612512:2612512 [2] NCCL INFO Using network Socket
compute-05:2612510:2612567 [0] NCCL INFO Channel 00/02 :    0   1   2   3
compute-05:2612511:2612568 [1] NCCL INFO Trees [0] 2/-1/-1-&gt;1-&gt;0 [1] 2/-1/-1-&gt;1-&gt;0
compute-05:2612510:2612567 [0] NCCL INFO Channel 01/02 :    0   1   2   3
compute-05:2612510:2612567 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] 1/-1/-1-&gt;0-&gt;-1
compute-05:2612511:2612568 [1] NCCL INFO Setting affinity for GPU 1 to 070007
compute-05:2612510:2612567 [0] NCCL INFO Setting affinity for GPU 0 to 070007
compute-05:2612512:2612570 [2] NCCL INFO Trees [0] 3/-1/-1-&gt;2-&gt;1 [1] 3/-1/-1-&gt;2-&gt;1
compute-05:2612513:2612569 [3] NCCL INFO Trees [0] -1/-1/-1-&gt;3-&gt;2 [1] -1/-1/-1-&gt;3-&gt;2
compute-05:2612511:2612568 [1] NCCL INFO Channel 00 : 1[3000] -&gt; 2[81000] via direct shared memory
compute-05:2612512:2612570 [2] NCCL INFO Channel 00 : 2[81000] -&gt; 3[82000] via direct shared memory
compute-05:2612510:2612567 [0] NCCL INFO Channel 00 : 0[2000] -&gt; 1[3000] via direct shared memory
compute-05:2612511:2612568 [1] NCCL INFO Channel 01 : 1[3000] -&gt; 2[81000] via direct shared memory
compute-05:2612512:2612570 [2] NCCL INFO Channel 01 : 2[81000] -&gt; 3[82000] via direct shared memory
compute-05:2612510:2612567 [0] NCCL INFO Channel 01 : 0[2000] -&gt; 1[3000] via direct shared memory
compute-05:2612513:2612569 [3] NCCL INFO Channel 00 : 3[82000] -&gt; 0[2000] via direct shared memory
compute-05:2612513:2612569 [3] NCCL INFO Channel 01 : 3[82000] -&gt; 0[2000] via direct shared memory
compute-05:2612511:2612568 [1] NCCL INFO Connected all rings
compute-05:2612510:2612567 [0] NCCL INFO Connected all rings
compute-05:2612512:2612570 [2] NCCL INFO Connected all rings
compute-05:2612511:2612568 [1] NCCL INFO Channel 00 : 1[3000] -&gt; 0[2000] via direct shared memory
compute-05:2612513:2612569 [3] NCCL INFO Connected all rings
compute-05:2612511:2612568 [1] NCCL INFO Channel 01 : 1[3000] -&gt; 0[2000] via direct shared memory
compute-05:2612513:2612569 [3] NCCL INFO Channel 00 : 3[82000] -&gt; 2[81000] via direct shared memory
compute-05:2612513:2612569 [3] NCCL INFO Channel 01 : 3[82000] -&gt; 2[81000] via direct shared memory
compute-05:2612510:2612567 [0] NCCL INFO Connected all trees
compute-05:2612510:2612567 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
compute-05:2612510:2612567 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
compute-05:2612512:2612570 [2] NCCL INFO Channel 00 : 2[81000] -&gt; 1[3000] via direct shared memory
compute-05:2612512:2612570 [2] NCCL INFO Channel 01 : 2[81000] -&gt; 1[3000] via direct shared memory
compute-05:2612513:2612569 [3] NCCL INFO Connected all trees
compute-05:2612513:2612569 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
compute-05:2612513:2612569 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
compute-05:2612511:2612568 [1] NCCL INFO Connected all trees
compute-05:2612511:2612568 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
compute-05:2612511:2612568 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
compute-05:2612512:2612570 [2] NCCL INFO Connected all trees
compute-05:2612512:2612570 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/512
compute-05:2612512:2612570 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
compute-05:2612512:2612570 [2] NCCL INFO comm 0x7f3054002fb0 rank 2 nranks 4 cudaDev 2 busId 81000 - Init COMPLETE
compute-05:2612513:2612569 [3] NCCL INFO comm 0x7f686c002fb0 rank 3 nranks 4 cudaDev 3 busId 82000 - Init COMPLETE
compute-05:2612511:2612568 [1] NCCL INFO comm 0x7f94fc002fb0 rank 1 nranks 4 cudaDev 1 busId 3000 - Init COMPLETE
compute-05:2612510:2612567 [0] NCCL INFO comm 0x7f8410002fb0 rank 0 nranks 4 cudaDev 0 busId 2000 - Init COMPLETE
compute-05:2612510:2612510 [0] NCCL INFO Launch mode Parallel
</code></pre>
<p>and the output with the quadro:</p>
<pre><code class="lang-auto">| distributed init (rank 2): env://
| distributed init (rank 0): env://
| distributed init (rank 1): env://
| distributed init (rank 3): env://
compute-11:1358800:1358800 [0] NCCL INFO Bootstrap : Using eno1:&lt;0&gt;
compute-11:1358800:1358800 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
compute-11:1358800:1358800 [0] NCCL INFO NET/IB : Using [0]i40iw1:1/RoCE ; OOB eno1:&lt;0&gt;
compute-11:1358800:1358800 [0] NCCL INFO Using network IB
NCCL version 2.10.3+cuda11.3
compute-11:1358801:1358801 [1] NCCL INFO Bootstrap : Using eno1:&lt;0&gt;
compute-11:1358801:1358801 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
compute-11:1358803:1358803 [3] NCCL INFO Bootstrap : Using eno1:&lt;0&gt;
compute-11:1358802:1358802 [2] NCCL INFO Bootstrap : Using eno1:&lt;0&gt;
compute-11:1358802:1358802 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
compute-11:1358803:1358803 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
compute-11:1358801:1358801 [1] NCCL INFO NET/IB : Using [0]i40iw1:1/RoCE ; OOB eno1:&lt;0&gt;
compute-11:1358801:1358801 [1] NCCL INFO Using network IB
compute-11:1358802:1358802 [2] NCCL INFO NET/IB : Using [0]i40iw1:1/RoCE ; OOB eno1:&lt;0&gt;
compute-11:1358802:1358802 [2] NCCL INFO Using network IB
compute-11:1358803:1358803 [3] NCCL INFO NET/IB : Using [0]i40iw1:1/RoCE ; OOB eno1:&lt;0&gt;
compute-11:1358803:1358803 [3] NCCL INFO Using network IB
compute-11:1358800:1358860 [0] NCCL INFO Channel 00/04 :    0   1   2   3
compute-11:1358800:1358860 [0] NCCL INFO Channel 01/04 :    0   3   2   1
compute-11:1358800:1358860 [0] NCCL INFO Channel 02/04 :    0   1   2   3
compute-11:1358800:1358860 [0] NCCL INFO Channel 03/04 :    0   3   2   1
compute-11:1358801:1358862 [1] NCCL INFO Trees [0] 2/-1/-1-&gt;1-&gt;0 [1] -1/-1/-1-&gt;1-&gt;2 [2] 2/-1/-1-&gt;1-&gt;0 [3] -1/-1/-1-&gt;1-&gt;2
compute-11:1358800:1358860 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] 3/-1/-1-&gt;0-&gt;-1 [2] 1/-1/-1-&gt;0-&gt;-1 [3] 3/-1/-1-&gt;0-&gt;-1
compute-11:1358801:1358862 [1] NCCL INFO Setting affinity for GPU 1 to 07,00000007
compute-11:1358800:1358860 [0] NCCL INFO Setting affinity for GPU 0 to 07,00000007
compute-11:1358802:1358865 [2] NCCL INFO Trees [0] 3/-1/-1-&gt;2-&gt;1 [1] 1/-1/-1-&gt;2-&gt;3 [2] 3/-1/-1-&gt;2-&gt;1 [3] 1/-1/-1-&gt;2-&gt;3
compute-11:1358802:1358865 [2] NCCL INFO Setting affinity for GPU 2 to 07,00000007
compute-11:1358803:1358866 [3] NCCL INFO Trees [0] -1/-1/-1-&gt;3-&gt;2 [1] 2/-1/-1-&gt;3-&gt;0 [2] -1/-1/-1-&gt;3-&gt;2 [3] 2/-1/-1-&gt;3-&gt;0
compute-11:1358803:1358866 [3] NCCL INFO Setting affinity for GPU 3 to 07,00000007
compute-11:1358800:1358860 [0] NCCL INFO Channel 00 : 0[1a000] -&gt; 1[1c000] via P2P/IPC
compute-11:1358801:1358862 [1] NCCL INFO Channel 00 : 1[1c000] -&gt; 2[1d000] via P2P/IPC
compute-11:1358802:1358865 [2] NCCL INFO Channel 00 : 2[1d000] -&gt; 3[1e000] via P2P/IPC
compute-11:1358803:1358866 [3] NCCL INFO Channel 00 : 3[1e000] -&gt; 0[1a000] via P2P/IPC
compute-11:1358800:1358860 [0] NCCL INFO Channel 02 : 0[1a000] -&gt; 1[1c000] via P2P/IPC
compute-11:1358801:1358862 [1] NCCL INFO Channel 02 : 1[1c000] -&gt; 2[1d000] via P2P/IPC
compute-11:1358802:1358865 [2] NCCL INFO Channel 02 : 2[1d000] -&gt; 3[1e000] via P2P/IPC

compute-11:1358801:1358862 [1] transport/p2p.cc:136 NCCL WARN Cuda failure 'API call is not supported in the installed CUDA driver'
compute-11:1358801:1358862 [1] NCCL INFO transport/p2p.cc:238 -&gt; 1
compute-11:1358801:1358862 [1] NCCL INFO transport.cc:111 -&gt; 1
compute-11:1358801:1358862 [1] NCCL INFO init.cc:778 -&gt; 1
compute-11:1358801:1358862 [1] NCCL INFO init.cc:904 -&gt; 1
compute-11:1358801:1358862 [1] NCCL INFO group.cc:72 -&gt; 1 [Async thread]
compute-11:1358803:1358866 [3] NCCL INFO Channel 02 : 3[1e000] -&gt; 0[1a000] via P2P/IPC

compute-11:1358802:1358865 [2] transport/p2p.cc:136 NCCL WARN Cuda failure 'API call is not supported in the installed CUDA driver'
compute-11:1358802:1358865 [2] NCCL INFO transport/p2p.cc:238 -&gt; 1
compute-11:1358802:1358865 [2] NCCL INFO transport.cc:111 -&gt; 1
compute-11:1358802:1358865 [2] NCCL INFO init.cc:778 -&gt; 1
compute-11:1358802:1358865 [2] NCCL INFO init.cc:904 -&gt; 1
compute-11:1358802:1358865 [2] NCCL INFO group.cc:72 -&gt; 1 [Async thread]

compute-11:1358803:1358866 [3] transport/p2p.cc:136 NCCL WARN Cuda failure 'API call is not supported in the installed CUDA driver'
compute-11:1358803:1358866 [3] NCCL INFO transport/p2p.cc:238 -&gt; 1
compute-11:1358803:1358866 [3] NCCL INFO transport.cc:111 -&gt; 1
compute-11:1358803:1358866 [3] NCCL INFO init.cc:778 -&gt; 1
compute-11:1358803:1358866 [3] NCCL INFO init.cc:904 -&gt; 1
compute-11:1358803:1358866 [3] NCCL INFO group.cc:72 -&gt; 1 [Async thread]

compute-11:1358800:1358860 [0] transport/p2p.cc:136 NCCL WARN Cuda failure 'API call is not supported in the installed CUDA driver'
compute-11:1358800:1358860 [0] NCCL INFO transport/p2p.cc:238 -&gt; 1
compute-11:1358800:1358860 [0] NCCL INFO transport.cc:111 -&gt; 1
compute-11:1358800:1358860 [0] NCCL INFO init.cc:778 -&gt; 1
compute-11:1358800:1358860 [0] NCCL INFO init.cc:904 -&gt; 1
compute-11:1358800:1358860 [0] NCCL INFO group.cc:72 -&gt; 1 [Async thread]
</code></pre>
<p>and my error is:<br>
<code>RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1639180487213/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled cuda error, NCCL version 21.0.3 ncclUnhandledCudaError: Call to CUDA function failed.</code><br>
Any idea why the behaviors are different and how to solve it ?</p>

ptrblck: <aside class="quote no-group" data-username="rvandeghen" data-post="4" data-topic="129301">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rvandeghen/48/23881_2.png" class="avatar"> rvandeghen:</div>
<blockquote>
<p><code>NCCL WARN Cuda failure 'API call is not supported in the installed CUDA driver'</code></p>
</blockquote>
</aside>
<p>This could point towards a driver issue on your machine, so try to update the driver to match your CUDA toolkit as described <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver">here</a>.</p>

rvandeghen: <p>The driver version and CUDA version are the same for all nodes so I don’t think this is where the problem comes from. Have you heard about something which is directly related to Quadro cards ?</p>

ptrblck: <p>No, I’m not aware of Quadro-specific issues.</p>
<aside class="quote no-group" data-username="rvandeghen" data-post="6" data-topic="129301">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rvandeghen/48/23881_2.png" class="avatar"> rvandeghen:</div>
<blockquote>
<p>The driver version and CUDA version are the same for all nodes so I don’t think this is where the problem comes from.</p>
</blockquote>
</aside>
<p>This sounds a bit concerning as you are pointing to identical system setups where one node apparently crashes?<br>
Are the NCCL tests running fine on all nodes (in particular the problematic and a healthy one)?</p>

rvandeghen: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="129301">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Are the NCCL tests running fine on all nodes (in particular the problematic and a healthy one)?</p>
</blockquote>
</aside>
<p>From the outputs, the first difference happens here:</p>
<aside class="quote no-group" data-username="rvandeghen" data-post="4" data-topic="129301">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rvandeghen/48/23881_2.png" class="avatar"> rvandeghen:</div>
<blockquote>
<pre><code class="lang-auto">compute-05:2612511:2612568 [1] NCCL INFO Channel 00 : 1[3000] -&gt; 2[81000] via direct shared memory
compute-05:2612512:2612570 [2] NCCL INFO Channel 00 : 2[81000] -&gt; 3[82000] via direct shared memory
compute-05:2612510:2612567 [0] NCCL INFO Channel 00 : 0[2000] -&gt; 1[3000] via direct shared memory
compute-05:2612511:2612568 [1] NCCL INFO Channel 01 : 1[3000] -&gt; 2[81000] via direct shared memory
compute-05:2612512:2612570 [2] NCCL INFO Channel 01 : 2[81000] -&gt; 3[82000] via direct shared memory
compute-05:2612510:2612567 [0] NCCL INFO Channel 01 : 0[2000] -&gt; 1[3000] via direct shared memory
compute-05:2612513:2612569 [3] NCCL INFO Channel 00 : 3[82000] -&gt; 0[2000] via direct shared memory
compute-05:2612513:2612569 [3] NCCL INFO Channel 01 : 3[82000] -&gt; 0[2000] via direct shared memory
</code></pre>
</blockquote>
</aside>
<p>for the healthy node and here:</p>
<aside class="quote no-group" data-username="rvandeghen" data-post="4" data-topic="129301">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rvandeghen/48/23881_2.png" class="avatar"> rvandeghen:</div>
<blockquote>
<pre><code class="lang-auto">compute-11:1358800:1358860 [0] NCCL INFO Channel 00 : 0[1a000] -&gt; 1[1c000] via P2P/IPC
compute-11:1358801:1358862 [1] NCCL INFO Channel 00 : 1[1c000] -&gt; 2[1d000] via P2P/IPC
compute-11:1358802:1358865 [2] NCCL INFO Channel 00 : 2[1d000] -&gt; 3[1e000] via P2P/IPC
compute-11:1358803:1358866 [3] NCCL INFO Channel 00 : 3[1e000] -&gt; 0[1a000] via P2P/IPC
compute-11:1358800:1358860 [0] NCCL INFO Channel 02 : 0[1a000] -&gt; 1[1c000] via P2P/IPC
compute-11:1358801:1358862 [1] NCCL INFO Channel 02 : 1[1c000] -&gt; 2[1d000] via P2P/IPC
compute-11:1358802:1358865 [2] NCCL INFO Channel 02 : 2[1d000] -&gt; 3[1e000] via P2P/IPC
</code></pre>
</blockquote>
</aside>
<p>for the problematic one.<br>
From my understanding, the problematic node has a problem with the <code>transport/p2p.cc</code> but this is not needed with the healthy one which uses <code>direct shared memory</code></p>
<p>I will run the <a href="https://github.com/NVIDIA/nccl-tests" rel="noopener nofollow ugc">tests</a> and get back with the results.</p>

rvandeghen: <p>Hi, here are some more informations regarding the cluster and my environment:</p>
<pre><code class="lang-auto">----------------------  ----------------------------------------------------------------------------------------------
sys.platform            linux
Python                  3.9.7 (default, Sep 16 2021, 13:09:58) [GCC 7.5.0]
numpy                   1.21.2
PyTorch                 1.10.1 @/home/rvandeghen/anaconda3/envs/SNv3-detection/lib/python3.9/site-packages/torch
PyTorch debug build     False
GPU available           Yes
GPU 0                   Quadro RTX 6000 (arch=7.5)
Driver version          450.57
CUDA_HOME               /home/rvandeghen/anaconda3/envs/SNv3-detection
Pillow                  8.4.0
torchvision             0.11.2 @/home/rvandeghen/anaconda3/envs/SNv3-detection/lib/python3.9/site-packages/torchvision
torchvision arch flags  3.5, 5.0, 6.0, 7.0, 7.5, 8.0, 8.6
cv2                     4.5.4
----------------------  ----------------------------------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 11.6
  - Built with CUDA Runtime 11.3
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.2
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always-faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
</code></pre>
<p>Maybe you have more information with the cuda and cudnn version ?</p>

YK711: <p>I encounter this error:</p>
<pre><code class="lang-auto">RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1659484809662/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, invalid usage, NCCL version 2.10.3 ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).
</code></pre>
<p>I deleted this line, then it worked…</p>
<pre><code class="lang-auto">os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3"
</code></pre>