I encountered the following NCCL issues while using PyTorch DDP for distributed training

username: tz_Wu
cooked: <p>“The error message is as follows.”<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/87518db47f3c84980610e293380013a0f89cf687.png" data-download-href="https://discuss.pytorch.org/uploads/default/87518db47f3c84980610e293380013a0f89cf687" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/7/87518db47f3c84980610e293380013a0f89cf687.png" alt="image" data-base62-sha1="jj5bxzObQlO39dlXlfufYiSQj9d" width="690" height="390" data-dominant-color="202020"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1599×906 38.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
ncll debug information is as follows.<br>
autodl-container-c1cd47b9b8-7085d22d:4018:4018 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.11&lt;0&gt;<br>
autodl-container-c1cd47b9b8-7085d22d:4018:4018 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4018:4018 [0] misc/ibvwrap.cc:212 NCCL WARN Call to ibv_open_device failed</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4018:4018 [0] transport/net_ib.cc:149 NCCL WARN NET/IB : Unable to open device mlx5_0</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4018:4018 [0] misc/ibvwrap.cc:212 NCCL WARN Call to ibv_open_device failed</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4018:4018 [0] transport/net_ib.cc:149 NCCL WARN NET/IB : Unable to open device mlx5_1<br>
autodl-container-c1cd47b9b8-7085d22d:4018:4018 [0] NCCL INFO NET/IB : No device found.<br>
autodl-container-c1cd47b9b8-7085d22d:4018:4018 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.11&lt;0&gt;<br>
autodl-container-c1cd47b9b8-7085d22d:4018:4018 [0] NCCL INFO Using network Socket<br>
NCCL version 2.10.3+cuda11.3<br>
autodl-container-c1cd47b9b8-7085d22d:4020:4020 [0] NCCL INFO Bootstrap : Using eth0:172.17.0.11&lt;0&gt;<br>
autodl-container-c1cd47b9b8-7085d22d:4020:4020 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4020:4020 [0] misc/ibvwrap.cc:212 NCCL WARN Call to ibv_open_device failed</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4020:4020 [0] transport/net_ib.cc:149 NCCL WARN NET/IB : Unable to open device mlx5_0</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4020:4020 [0] misc/ibvwrap.cc:212 NCCL WARN Call to ibv_open_device failed</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4020:4020 [0] transport/net_ib.cc:149 NCCL WARN NET/IB : Unable to open device mlx5_1<br>
autodl-container-c1cd47b9b8-7085d22d:4020:4020 [0] NCCL INFO NET/IB : No device found.<br>
autodl-container-c1cd47b9b8-7085d22d:4020:4020 [0] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.11&lt;0&gt;<br>
autodl-container-c1cd47b9b8-7085d22d:4020:4020 [0] NCCL INFO Using network Socket<br>
autodl-container-c1cd47b9b8-7085d22d:4019:4019 [1] NCCL INFO Bootstrap : Using eth0:172.17.0.11&lt;0&gt;<br>
autodl-container-c1cd47b9b8-7085d22d:4019:4019 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4019:4019 [1] misc/ibvwrap.cc:212 NCCL WARN Call to ibv_open_device failed</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4019:4019 [1] transport/net_ib.cc:149 NCCL WARN NET/IB : Unable to open device mlx5_0</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4019:4019 [1] misc/ibvwrap.cc:212 NCCL WARN Call to ibv_open_device failed</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4019:4019 [1] transport/net_ib.cc:149 NCCL WARN NET/IB : Unable to open device mlx5_1<br>
autodl-container-c1cd47b9b8-7085d22d:4019:4019 [1] NCCL INFO NET/IB : No device found.<br>
autodl-container-c1cd47b9b8-7085d22d:4019:4019 [1] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.11&lt;0&gt;<br>
autodl-container-c1cd47b9b8-7085d22d:4019:4019 [1] NCCL INFO Using network Socket<br>
autodl-container-c1cd47b9b8-7085d22d:4021:4021 [1] NCCL INFO Bootstrap : Using eth0:172.17.0.11&lt;0&gt;<br>
autodl-container-c1cd47b9b8-7085d22d:4021:4021 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4021:4021 [1] misc/ibvwrap.cc:212 NCCL WARN Call to ibv_open_device failed</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4021:4021 [1] transport/net_ib.cc:149 NCCL WARN NET/IB : Unable to open device mlx5_0</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4021:4021 [1] misc/ibvwrap.cc:212 NCCL WARN Call to ibv_open_device failed</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4021:4021 [1] transport/net_ib.cc:149 NCCL WARN NET/IB : Unable to open device mlx5_1<br>
autodl-container-c1cd47b9b8-7085d22d:4021:4021 [1] NCCL INFO NET/IB : No device found.<br>
autodl-container-c1cd47b9b8-7085d22d:4021:4021 [1] NCCL INFO NET/Socket : Using [0]eth0:172.17.0.11&lt;0&gt;<br>
autodl-container-c1cd47b9b8-7085d22d:4021:4021 [1] NCCL INFO Using network Socket</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4018:4348 [0] init.cc:521 NCCL WARN Duplicate GPU detected : rank 0 and rank 2 both on CUDA device b1000</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4020:4349 [0] init.cc:521 NCCL WARN Duplicate GPU detected : rank 2 and rank 0 both on CUDA device b1000</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4019:4350 [1] init.cc:521 NCCL WARN Duplicate GPU detected : rank 1 and rank 3 both on CUDA device b2000<br>
autodl-container-c1cd47b9b8-7085d22d:4018:4348 [0] NCCL INFO init.cc:904 → 5</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4021:4351 [1] init.cc:521 NCCL WARN Duplicate GPU detected : rank 3 and rank 1 both on CUDA device b2000<br>
autodl-container-c1cd47b9b8-7085d22d:4020:4349 [0] NCCL INFO init.cc:904 → 5<br>
autodl-container-c1cd47b9b8-7085d22d:4019:4350 [1] NCCL INFO init.cc:904 → 5<br>
autodl-container-c1cd47b9b8-7085d22d:4018:4348 [0] NCCL INFO group.cc:72 → 5 [Async thread]<br>
autodl-container-c1cd47b9b8-7085d22d:4021:4351 [1] NCCL INFO init.cc:904 → 5<br>
autodl-container-c1cd47b9b8-7085d22d:4020:4349 [0] NCCL INFO group.cc:72 → 5 [Async thread]<br>
autodl-container-c1cd47b9b8-7085d22d:4019:4350 [1] NCCL INFO group.cc:72 → 5 [Async thread]<br>
autodl-container-c1cd47b9b8-7085d22d:4021:4351 [1] NCCL INFO group.cc:72 → 5 [Async thread]</p>

username: ptrblck
cooked: <p>The code fails due to:</p>
<aside class="quote no-group" data-username="tz_Wu" data-post="1" data-topic="183717">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tz_wu/48/61997_2.png" class="avatar"> tz_Wu:</div>
<blockquote>
<p>autodl-container-c1cd47b9b8-7085d22d:4018:4348 [0] init.cc:521 NCCL WARN Duplicate GPU detected : rank 0 and rank 2 both on CUDA device b1000</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4020:4349 [0] init.cc:521 NCCL WARN Duplicate GPU detected : rank 2 and rank 0 both on CUDA device b1000</p>
<p>autodl-container-c1cd47b9b8-7085d22d:4019:4350 [1] init.cc:521 NCCL WARN Duplicate GPU detected : rank 1 and rank 3 both on CUDA device b2000</p>
</blockquote>
</aside>
<p>as it seems you are trying to use the same device on different ranks.</p>