RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370128159/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8

username: Nishanth_D
cooked: <p>Hello, I am getting the above error and this is the log after using export NCCL_DEBUG=INFO.</p>
<p>1dfff1d89025:310:310 [0] NCCL INFO Bootstrap : Using [0]eth0:172.17.0.2&lt;0&gt;<br>
1dfff1d89025:310:310 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>1dfff1d89025:310:310 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
1dfff1d89025:310:310 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.2&lt;0&gt;<br>
1dfff1d89025:310:310 [0] NCCL INFO Using network Socket<br>
NCCL version 2.7.8+cuda10.2</p>
<p>1dfff1d89025:310:337 [0] graph/xml.cc:332 NCCL WARN Could not find real path of /sys/class/pci_bus/0000:01/…/…/0000:01:00.0<br>
1dfff1d89025:310:337 [0] NCCL INFO graph/xml.cc:469 → 2<br>
1dfff1d89025:310:337 [0] NCCL INFO graph/xml.cc:660 → 2<br>
1dfff1d89025:310:337 [0] NCCL INFO graph/topo.cc:523 → 2<br>
1dfff1d89025:310:337 [0] NCCL INFO init.cc:581 → 2<br>
1dfff1d89025:310:337 [0] NCCL INFO init.cc:840 → 2<br>
1dfff1d89025:310:337 [0] NCCL INFO group.cc:73 → 2 [Async thread]<br>
Traceback (most recent call last):<br>
File “/home/RNNPose/tools/eval.py”, line 579, in <br>
fire.Fire()<br>
File “/opt/miniconda3/envs/py37/lib/python3.7/site-packages/fire/core.py”, line 141, in Fire<br>
component_trace = _Fire(component, args, parsed_flag_args, context, name)<br>
File “/opt/miniconda3/envs/py37/lib/python3.7/site-packages/fire/core.py”, line 471, in _Fire<br>
target=component.<strong>name</strong>)<br>
File “/opt/miniconda3/envs/py37/lib/python3.7/site-packages/fire/core.py”, line 681, in _CallAndUpdateTrace<br>
component = fn(*varargs, **kwargs)<br>
File “/home/RNNPose/tools/eval.py”, line 227, in multi_proc_train<br>
args=( params,) )<br>
File “/opt/miniconda3/envs/py37/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 199, in spawn<br>
return start_processes(fn, args, nprocs, join, daemon, start_method=‘spawn’)<br>
File “/opt/miniconda3/envs/py37/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 157, in start_processes<br>
while not context.join():<br>
File “/opt/miniconda3/envs/py37/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 118, in join<br>
raise Exception(msg)<br>
Exception:</p>
<p>– Process 0 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/opt/miniconda3/envs/py37/lib/python3.7/site-packages/torch/multiprocessing/spawn.py”, line 19, in _wrap<br>
fn(i, *args)<br>
File “/home/RNNPose/tools/eval.py”, line 255, in train_worker<br>
apex_opt_level=params.apex_opt_level<br>
File “/home/RNNPose/tools/eval.py”, line 317, in eval<br>
backend=“nccl”, init_method=dist_url, world_size=get_world(use_dist), rank=get_rank(use_dist))<br>
File “/opt/miniconda3/envs/py37/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 455, in init_process_group<br>
barrier()<br>
File “/opt/miniconda3/envs/py37/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 1960, in barrier<br>
work = _default_pg.barrier()<br>
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370128159/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8</p>
<p>I am running the program inside the docker container.<br>
Python: 3.7.10<br>
Torch: 1.7.1<br>
CUDA: 10.2</p>
<p>Solutions Tried so far:</p>
<ol>
<li>using –ipc=host</li>
<li>export NCCL_IB_DISABLE=1</li>
<li>torch.cuda.set_device(rank)</li>
</ol>
<p>Any help is appreciated and let me know if needed more details. Thanks in advance.</p>

username: weifengpy
cooked: <aside class="quote no-group" data-username="Nishanth_D" data-post="1" data-topic="208124">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nishanth_d/48/72149_2.png" class="avatar"> Nishanth_D:</div>
<blockquote>
<p>NCCL WARN Could not find real path of</p>
</blockquote>
</aside>
<p>Thanks for reporting. the following error seems to be fatal<br>
<code>NCCL WARN Could not find real path of /sys/class/pci_bus</code></p>
<p>This is reported on NCCL 2.7 before by other users. NCCL experts suggested ugprading NCCL version to workdaround the topology detection <a href="https://github.com/NVIDIA/nccl/issues/573#issuecomment-965247426" class="inline-onebox" rel="noopener nofollow ugc">NCCL WARN Could not find real path of... · Issue #573 · NVIDIA/nccl · GitHub</a></p>