NCCL error right after init_process_group in barrier()

xuef: <p>Try to run this simple code in a 2 gpu environment, and got NCCL RuntimeError error. Appreciate your help!!</p>
<pre><code class="lang-auto">import torch
import os
import torch.distributed as dist

if __name__ == '__main__':
    print(f"RANK: {os.environ.get('RANK')} | WORLD_SIZE: {os.environ.get('WORLD_SIZE')} | LOCAL_RANK: {os.environ.get('LOCAL_RANK')}")
    print(f"MASTER_ADDR: {os.environ.get('MASTER_ADDR')} | MASTER_PORT: {os.environ.get('MASTER_PORT')}")
    rank = int(os.environ["RANK"])
    world_size = int(os.environ["WORLD_SIZE"])
    dist.init_process_group(backend='nccl', init_method='env://', rank=rank, world_size=world_size)
    dist.barrier()
</code></pre>
<pre><code class="lang-auto">$ NCCL_DEBUG=INFO torchrun --nnodes=1 --nproc_per_node=2 --master_port=29520 try_distributed.py 
</code></pre>
<p>RANK: 0 | WORLD_SIZE: 2 | LOCAL_RANK: 0<br>
MASTER_ADDR: 127.0.0.1 | MASTER_PORT: 29520<br>
RANK: 1 | WORLD_SIZE: 2 | LOCAL_RANK: 1<br>
MASTER_ADDR: 127.0.0.1 | MASTER_PORT: 29520<br>
mlgpu5:847:847 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.4&lt;0&gt;<br>
mlgpu5:847:847 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>mlgpu5:847:847 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
mlgpu5:847:847 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.4&lt;0&gt;<br>
mlgpu5:847:847 [0] NCCL INFO Using network Socket<br>
NCCL version 2.10.3+cuda10.2<br>
mlgpu5:848:848 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.4&lt;0&gt;<br>
mlgpu5:848:848 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>mlgpu5:848:848 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
mlgpu5:848:848 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.4&lt;0&gt;<br>
mlgpu5:848:848 [0] NCCL INFO Using network Socket</p>
<p>mlgpu5:848:863 [0] init.cc:521 NCCL WARN Duplicate GPU detected : rank 1 and rank 0 both on CUDA device 15000</p>
<p>mlgpu5:847:862 [0] init.cc:521 NCCL WARN Duplicate GPU detected : rank 0 and rank 1 both on CUDA device 15000<br>
mlgpu5:848:863 [0] NCCL INFO init.cc:904 → 5<br>
mlgpu5:847:862 [0] NCCL INFO init.cc:904 → 5<br>
mlgpu5:848:863 [0] NCCL INFO group.cc:72 → 5 [Async thread]<br>
mlgpu5:847:862 [0] NCCL INFO group.cc:72 → 5 [Async thread]<br>
Traceback (most recent call last):<br>
File “/repo/explore/try_distributed.py”, line 11, in <br>
Traceback (most recent call last):<br>
File “/repo/explore/try_distributed.py”, line 11, in <br>
dist.barrier()<br>
File “/opt/miniconda3/envs/torch_v2/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py”, line 2709, in barrier<br>
dist.barrier()<br>
File “/opt/miniconda3/envs/torch_v2/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py”, line 2709, in barrier<br>
work = default_pg.barrier(opts=opts)<br>
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1634272068185/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, invalid usage, NCCL version 21.0.3<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).<br>
work = default_pg.barrier(opts=opts)<br>
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1634272068185/work/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, invalid usage, NCCL version 21.0.3<br>
ncclInvalidUsage: This usually reflects invalid usage of NCCL library (such as too many async ops, too many collectives at once, mixing streams in a group, etc).</p>

xuef: <aside class="quote no-group" data-username="xuef" data-post="1" data-topic="170551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/ecd19e/48.png" class="avatar"> xuef:</div>
<blockquote>
<p>NCCL WARN Duplicate GPU detected</p>
</blockquote>
</aside>
<p>I find out the problem here. Apparently one of the GPUs is not visible due to a setting issue. Once I resolved that. It works. So the duplicate GPU WARN needs to be attended.</p>