NCCL error when training data in GCP 

username: Youngwook_Lee
cooked: <p>Please provide the following information when requesting support.</p>
<p>• Hardware : GCP (Google Cloud Platform) → a100 40gb 8EA → ubuntu20.04<br>
• torch=1.9.0+cu111<br>
• torchvision=0.10.0+cu111<br>
• torchaudio=0.9.0<br>
• python=3.8<br>
• mmcv-full==1.6.0</p>
<p>When I do the deep learning training as shown below, the error occurs.<br>
I think there is an error in <code>graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci_bus/fffffff/..../fffffff:fff.</code></p>
<blockquote>
<p>a100-40g-8ea-aisa-northeast1-a:233417:233417 [0] NCCL INFO Bootstrap : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233417:233417 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>a100-40g-8ea-aisa-northeast1-a:233417:233417 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
a100-40g-8ea-aisa-northeast1-a:233417:233417 [0] NCCL INFO NET/Socket : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233417:233417 [0] NCCL INFO Using network Socket<br>
NCCL version 2.7.8+cuda11.1<br>
a100-40g-8ea-aisa-northeast1-a:233421:233421 [4] NCCL INFO Bootstrap : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233421:233421 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>a100-40g-8ea-aisa-northeast1-a:233421:233421 [4] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
a100-40g-8ea-aisa-northeast1-a:233421:233421 [4] NCCL INFO NET/Socket : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233421:233421 [4] NCCL INFO Using network Socket<br>
a100-40g-8ea-aisa-northeast1-a:233422:233422 [5] NCCL INFO Bootstrap : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233422:233422 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>a100-40g-8ea-aisa-northeast1-a:233422:233422 [5] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
a100-40g-8ea-aisa-northeast1-a:233422:233422 [5] NCCL INFO NET/Socket : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233422:233422 [5] NCCL INFO Using network Socket<br>
a100-40g-8ea-aisa-northeast1-a:233420:233420 [3] NCCL INFO Bootstrap : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233420:233420 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>a100-40g-8ea-aisa-northeast1-a:233420:233420 [3] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
a100-40g-8ea-aisa-northeast1-a:233420:233420 [3] NCCL INFO NET/Socket : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233420:233420 [3] NCCL INFO Using network Socket<br>
a100-40g-8ea-aisa-northeast1-a:233423:233423 [6] NCCL INFO Bootstrap : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233423:233423 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>a100-40g-8ea-aisa-northeast1-a:233423:233423 [6] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
a100-40g-8ea-aisa-northeast1-a:233423:233423 [6] NCCL INFO NET/Socket : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233423:233423 [6] NCCL INFO Using network Socket<br>
a100-40g-8ea-aisa-northeast1-a:233419:233419 [2] NCCL INFO Bootstrap : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233419:233419 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>a100-40g-8ea-aisa-northeast1-a:233419:233419 [2] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
a100-40g-8ea-aisa-northeast1-a:233419:233419 [2] NCCL INFO NET/Socket : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233419:233419 [2] NCCL INFO Using network Socket<br>
a100-40g-8ea-aisa-northeast1-a:233418:233418 [1] NCCL INFO Bootstrap : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233418:233418 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>a100-40g-8ea-aisa-northeast1-a:233418:233418 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
a100-40g-8ea-aisa-northeast1-a:233418:233418 [1] NCCL INFO NET/Socket : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233418:233418 [1] NCCL INFO Using network Socket<br>
a100-40g-8ea-aisa-northeast1-a:233424:233424 [7] NCCL INFO Bootstrap : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233424:233424 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>a100-40g-8ea-aisa-northeast1-a:233424:233424 [7] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
a100-40g-8ea-aisa-northeast1-a:233424:233424 [7] NCCL INFO NET/Socket : Using [0]ens8:10.146.0.3&lt;0&gt;<br>
a100-40g-8ea-aisa-northeast1-a:233424:233424 [7] NCCL INFO Using network Socket</p>
<p>a100-40g-8ea-aisa-northeast1-a:233422:233973 [5] graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci_bus/fffffff/…/…/fffffff:ff:f<br>
a100-40g-8ea-aisa-northeast1-a:233422:233973 [5] NCCL INFO graph/xml.cc:648 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233422:233973 [5] NCCL INFO graph/xml.cc:665 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233422:233973 [5] NCCL INFO graph/topo.cc:523 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233422:233973 [5] NCCL INFO init.cc:581 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233422:233973 [5] NCCL INFO init.cc:840 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233422:233973 [5] NCCL INFO group.cc:73 → 2 [Async thread]<br>
Traceback (most recent call last):<br>
File “tools/train.py”, line 301, in<br>
main()<br>
File “tools/train.py”, line 289, in main<br>
train_model(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 344, in train_model<br>
train_detector(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 226, in train_detector<br>
model = MMDistributedDataParallel(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.</p>
<p>a100-40g-8ea-aisa-northeast1-a:233423:233974 [6] graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci_bus/fffffff/…/…/fffffff:ff:f<br>
a100-40g-8ea-aisa-northeast1-a:233423:233974 [6] NCCL INFO graph/xml.cc:648 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233423:233974 [6] NCCL INFO graph/xml.cc:665 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233423:233974 [6] NCCL INFO graph/topo.cc:523 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233423:233974 [6] NCCL INFO init.cc:581 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233423:233974 [6] NCCL INFO init.cc:840 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233423:233974 [6] NCCL INFO group.cc:73 → 2 [Async thread]<br>
Traceback (most recent call last):<br>
File “tools/train.py”, line 301, in<br>
main()<br>
File “tools/train.py”, line 289, in main<br>
train_model(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 344, in train_model<br>
train_detector(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 226, in train_detector<br>
model = MMDistributedDataParallel(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.</p>
<p>a100-40g-8ea-aisa-northeast1-a:233420:233975 [3] graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci_bus/fffffff/…/…/fffffff:ff:f<br>
a100-40g-8ea-aisa-northeast1-a:233420:233975 [3] NCCL INFO graph/xml.cc:648 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233420:233975 [3] NCCL INFO graph/xml.cc:665 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233420:233975 [3] NCCL INFO graph/topo.cc:523 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233420:233975 [3] NCCL INFO init.cc:581 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233420:233975 [3] NCCL INFO init.cc:840 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233420:233975 [3] NCCL INFO group.cc:73 → 2 [Async thread]<br>
Traceback (most recent call last):<br>
File “tools/train.py”, line 301, in<br>
main()<br>
File “tools/train.py”, line 289, in main<br>
train_model(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 344, in train_model<br>
train_detector(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 226, in train_detector<br>
model = MMDistributedDataParallel(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.</p>
<p>a100-40g-8ea-aisa-northeast1-a:233418:233977 [1] graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci_bus/fffffff/…/…/fffffff:ff:f<br>
a100-40g-8ea-aisa-northeast1-a:233418:233977 [1] NCCL INFO graph/xml.cc:648 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233418:233977 [1] NCCL INFO graph/xml.cc:665 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233418:233977 [1] NCCL INFO graph/topo.cc:523 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233418:233977 [1] NCCL INFO init.cc:581 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233418:233977 [1] NCCL INFO init.cc:840 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233418:233977 [1] NCCL INFO group.cc:73 → 2 [Async thread]<br>
Traceback (most recent call last):<br>
File “tools/train.py”, line 301, in</p>
<p>a100-40g-8ea-aisa-northeast1-a:233419:233976 [2] graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci_bus/fffffff/…/…/fffffff:ff:f<br>
a100-40g-8ea-aisa-northeast1-a:233419:233976 [2] NCCL INFO graph/xml.cc:648 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233419:233976 [2] NCCL INFO graph/xml.cc:665 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233419:233976 [2] NCCL INFO graph/topo.cc:523 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233419:233976 [2] NCCL INFO init.cc:581 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233419:233976 [2] NCCL INFO init.cc:840 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233419:233976 [2] NCCL INFO group.cc:73 → 2 [Async thread]<br>
main()<br>
File “tools/train.py”, line 289, in main<br>
train_model(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 344, in train_model<br>
train_detector(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 226, in train_detector<br>
model = MMDistributedDataParallel(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.<br>
Traceback (most recent call last):<br>
File “tools/train.py”, line 301, in<br>
main()<br>
File “tools/train.py”, line 289, in main<br>
train_model(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 344, in train_model<br>
train_detector(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 226, in train_detector<br>
model = MMDistributedDataParallel(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.</p>
<p>a100-40g-8ea-aisa-northeast1-a:233417:233972 [0] graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci_bus/fffffff/…/…/fffffff:ff:f<br>
a100-40g-8ea-aisa-northeast1-a:233417:233972 [0] NCCL INFO graph/xml.cc:648 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233417:233972 [0] NCCL INFO graph/xml.cc:665 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233417:233972 [0] NCCL INFO graph/topo.cc:523 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233417:233972 [0] NCCL INFO init.cc:581 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233417:233972 [0] NCCL INFO init.cc:840 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233417:233972 [0] NCCL INFO group.cc:73 → 2 [Async thread]<br>
Traceback (most recent call last):<br>
File “tools/train.py”, line 301, in<br>
main()<br>
File “tools/train.py”, line 289, in main<br>
train_model(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 344, in train_model<br>
train_detector(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 226, in train_detector<br>
model = MMDistributedDataParallel(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.</p>
<p>a100-40g-8ea-aisa-northeast1-a:233424:233980 [7] graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci_bus/fffffff/…/…/fffffff:ff:f<br>
a100-40g-8ea-aisa-northeast1-a:233424:233980 [7] NCCL INFO graph/xml.cc:648 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233424:233980 [7] NCCL INFO graph/xml.cc:665 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233424:233980 [7] NCCL INFO graph/topo.cc:523 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233424:233980 [7] NCCL INFO init.cc:581 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233424:233980 [7] NCCL INFO init.cc:840 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233424:233980 [7] NCCL INFO group.cc:73 → 2 [Async thread]<br>
Traceback (most recent call last):<br>
File “tools/train.py”, line 301, in</p>
<p>a100-40g-8ea-aisa-northeast1-a:233421:233978 [4] graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci_bus/fffffff/…/…/fffffff:ff:f<br>
main()a100-40g-8ea-aisa-northeast1-a:233421:233978 [4] NCCL INFO graph/xml.cc:648 → 2</p>
<p>File “tools/train.py”, line 289, in main<br>
a100-40g-8ea-aisa-northeast1-a:233421:233978 [4] NCCL INFO graph/xml.cc:665 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233421:233978 [4] NCCL INFO graph/topo.cc:523 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233421:233978 [4] NCCL INFO init.cc:581 → 2<br>
a100-40g-8ea-aisa-northeast1-a:233421:233978 [4] NCCL INFO init.cc:840 → 2<br>
train_model(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 344, in train_model<br>
a100-40g-8ea-aisa-northeast1-a:233421:233978 [4] NCCL INFO group.cc:73 → 2 [Async thread]<br>
train_detector(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 226, in train_detector<br>
model = MMDistributedDataParallel(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.<br>
Traceback (most recent call last):<br>
File “tools/train.py”, line 301, in<br>
main()<br>
File “tools/train.py”, line 289, in main<br>
train_model(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 344, in train_model<br>
train_detector(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet3d/apis/train.py”, line 226, in train_detector<br>
model = MMDistributedDataParallel(<br>
File “/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/torch/nn/parallel/distributed.py”, line 496, in <strong>init</strong><br>
dist._verify_model_across_ranks(self.process_group, parameters)<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:911, unhandled system error, NCCL version 2.7.8<br>
ncclSystemError: System call (socket, malloc, munmap, etc) failed.<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 233417) of binary: /home/morai_developer/anaconda3/envs/cmt/bin/python<br>
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 3/3 attempts left; will restart worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous’ing worker group<br>
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:<br>
restart_count=1<br>
master_addr=127.0.0.1<br>
master_port=29500<br>
group_rank=0<br>
group_world_size=1<br>
local_ranks=[0, 1, 2, 3, 4, 5, 6, 7]<br>
role_ranks=[0, 1, 2, 3, 4, 5, 6, 7]<br>
global_ranks=[0, 1, 2, 3, 4, 5, 6, 7]<br>
role_world_sizes=[8, 8, 8, 8, 8, 8, 8, 8]<br>
global_world_sizes=[8, 8, 8, 8, 8, 8, 8, 8]</p>
<p>INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_fnf6mo4m/none__7wlltgj/attempt_1/0/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_fnf6mo4m/none__7wlltgj/attempt_1/1/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker2 reply file to: /tmp/torchelastic_fnf6mo4m/none__7wlltgj/attempt_1/2/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker3 reply file to: /tmp/torchelastic_fnf6mo4m/none__7wlltgj/attempt_1/3/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker4 reply file to: /tmp/torchelastic_fnf6mo4m/none__7wlltgj/attempt_1/4/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker5 reply file to: /tmp/torchelastic_fnf6mo4m/none__7wlltgj/attempt_1/5/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker6 reply file to: /tmp/torchelastic_fnf6mo4m/none__7wlltgj/attempt_1/6/error.json<br>
INFO:torch.distributed.elastic.multiprocessing:Setting worker7 reply file to: /tmp/torchelastic_fnf6mo4m/none__7wlltgj/attempt_1/7/error.json<br>
/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.<br>
warnings.warn(<br>
projects.mmdet3d_plugin<br>
/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.<br>
warnings.warn(<br>
projects.mmdet3d_plugin<br>
/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.<br>
warnings.warn(<br>
projects.mmdet3d_plugin<br>
/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.<br>
warnings.warn(<br>
projects.mmdet3d_plugin<br>
/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.<br>
warnings.warn(<br>
projects.mmdet3d_plugin<br>
/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.<br>
warnings.warn(<br>
projects.mmdet3d_plugin<br>
/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.<br>
warnings.warn(<br>
projects.mmdet3d_plugin<br>
/home/morai_developer/anaconda3/envs/cmt/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.<br>
warnings.warn(<br>
projects.mmdet3d_plugin</p>
</blockquote>
<h2><a name="below-are-the-results-of-nvidia-smi-1" class="anchor" href="#below-are-the-results-of-nvidia-smi-1"></a><strong>Below are the results of <code>nvidia-smi</code>.</strong></h2>
<p>±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 525.125.06 Driver Version: 525.125.06 CUDA Version: 12.0 |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br>
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br>
| | | MIG M. |<br>
|===============================+======================+======================|<br>
| 0 NVIDIA A100-SXM… Off | 00000000:00:04.0 Off | 0 |<br>
| N/A 30C P0 53W / 400W | 112MiB / 40960MiB | 0% Default |<br>
| | | Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
| 1 NVIDIA A100-SXM… Off | 00000000:00:05.0 Off | 0 |<br>
| N/A 29C P0 54W / 400W | 7MiB / 40960MiB | 0% Default |<br>
| | | Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
| 2 NVIDIA A100-SXM… Off | 00000000:00:06.0 Off | 0 |<br>
| N/A 29C P0 53W / 400W | 7MiB / 40960MiB | 0% Default |<br>
| | | Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
| 3 NVIDIA A100-SXM… Off | 00000000:00:07.0 Off | 0 |<br>
| N/A 30C P0 58W / 400W | 7MiB / 40960MiB | 0% Default |<br>
| | | Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
| 4 NVIDIA A100-SXM… Off | 00000000:80:00.0 Off | 0 |<br>
| N/A 29C P0 57W / 400W | 7MiB / 40960MiB | 0% Default |<br>
| | | Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
| 5 NVIDIA A100-SXM… Off | 00000000:80:01.0 Off | 0 |<br>
| N/A 31C P0 57W / 400W | 7MiB / 40960MiB | 0% Default |<br>
| | | Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
| 6 NVIDIA A100-SXM… Off | 00000000:80:02.0 Off | 0 |<br>
| N/A 28C P0 56W / 400W | 7MiB / 40960MiB | 0% Default |<br>
| | | Disabled |<br>
±------------------------------±---------------------±---------------------+<br>
| 7 NVIDIA A100-SXM… Off | 00000000:80:03.0 Off | 0 |<br>
| N/A 30C P0 52W / 400W | 7MiB / 40960MiB | 0% Default |<br>
| | | Disabled |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes: |<br>
| GPU GI CI PID Type Process name GPU Memory |<br>
| ID ID Usage |<br>
|=============================================================================|<br>
| 0 N/A N/A 2477 G /usr/lib/xorg/Xorg 95MiB |<br>
| 0 N/A N/A 2585 G /usr/bin/gnome-shell 12MiB |<br>
| 1 N/A N/A 2477 G /usr/lib/xorg/Xorg 4MiB |<br>
| 2 N/A N/A 2477 G /usr/lib/xorg/Xorg 4MiB |<br>
| 3 N/A N/A 2477 G /usr/lib/xorg/Xorg 4MiB |<br>
| 4 N/A N/A 2477 G /usr/lib/xorg/Xorg 4MiB |<br>
| 5 N/A N/A 2477 G /usr/lib/xorg/Xorg 4MiB |<br>
| 6 N/A N/A 2477 G /usr/lib/xorg/Xorg 4MiB |<br>
| 7 N/A N/A 2477 G /usr/lib/xorg/Xorg 4MiB |<br>
±----------------------------------------------------------------------------+</p>
<h2><a name="below-is-the-result-of-nvcc-v-2" class="anchor" href="#below-is-the-result-of-nvcc-v-2"></a><strong>Below is the result of <code>nvcc-V</code>.</strong></h2>
<p>nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2022 NVIDIA Corporation<br>
Built on Wed_Sep_21_10:33:58_PDT_2022<br>
Cuda compilation tools, release 11.8, V11.8.89<br>
Build cuda_11.8.r11.8/compiler.31833905_0</p>

username: H-Huang
cooked: <p>I see you are using cuda 11.8, but nccl 2.7.8 which is not tested with that cuda version. Can you try upgrading your nccl version?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f9547ea655ed7f01e4bb8087f04c4c8c9e3479c.png" data-download-href="https://discuss.pytorch.org/uploads/default/3f9547ea655ed7f01e4bb8087f04c4c8c9e3479c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f9547ea655ed7f01e4bb8087f04c4c8c9e3479c_2_690x243.png" alt="image" data-base62-sha1="94tUDCZlKGGbagPCUP9C3vWayZu" width="690" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f9547ea655ed7f01e4bb8087f04c4c8c9e3479c_2_690x243.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f9547ea655ed7f01e4bb8087f04c4c8c9e3479c_2_1035x364.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f9547ea655ed7f01e4bb8087f04c4c8c9e3479c_2_1380x486.png 2x" data-dominant-color="F2F2F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1404×496 50.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>You can try the nccl version that comes with PyTorch nightlies or just upgrade to nccl 2.17</p>