Process stuck by the dist.barrier() using DDP after dist.init_process_group

KY_Tang: <p>I tried  to use <code>DistributedDataParallel</code> for multi-GPUs training  on one node and my code works fine on <code>Host-A</code> but when I run my code on <code>Host-B</code>, it is stuck by the <code>torch.distributed.barrier()</code> after <code>torch.distributed.init_process_group</code>. And the GPU-Util are both 100%.</p>
<p>Here is my code for initialize the <code>DistributedDataParallel</code>:</p>
<pre><code class="lang-auto">args.local_rank = int(os.environ['LOCAL_RANK'])

print(f"Local Rank {args.local_rank} | Using distributed mode by DistributedDataParallel")

torch.distributed.init_process_group(backend='nccl',
                                     world_size=args.world_size, rank=args.local_rank)

print('before barrier')

torch.distributed.barrier()

print('after barrier')
</code></pre>
<p>On <code>Host-B</code>, when run the following command which is the same on <code>Host-A</code>:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=0,1 torchrun --standalone --nnodes=1 --nproc_per_node=2 test_ddp.py
</code></pre>
<p>And it got the following output, both two processes are stuck by the <code>torch.distributed.barrier()</code>, The GPU-Util of two GPUs are both 100%.</p>
<pre><code class="lang-auto">[2024-03-09 19:37:32,243] torch.distributed.run: [WARNING] master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.
[2024-03-09 19:37:32,243] torch.distributed.run: [WARNING] 
[2024-03-09 19:37:32,243] torch.distributed.run: [WARNING] *****************************************
[2024-03-09 19:37:32,243] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-03-09 19:37:32,243] torch.distributed.run: [WARNING] *****************************************
Local Rank 0 | Using distributed mode by DistributedDataParallel
Local Rank 1 | Using distributed mode by DistributedDataParallel
before barrier
before barrier
</code></pre>
<p>But the same code works fine on <code>Host-A</code>. Is there any problem with my code? Or Is there a problem with <code>Host-B</code>?</p>
<p>The information of <code>Host-A</code> and <code>Host-B</code> is listed bellow:</p>
<p><code>Host-A</code>: Installed with 2 RTX-4090 GPUs, and the driver version is 535.129.03 and CUDA version is 12.2</p>
<p><code>Host-B</code>: Installed with 8 RTX-3090 GPUs, and the driver version is 545.23.08 and CUDA version is 12.3<br>
But for the <code>Host-B</code>, in the <code>/urs/local</code> , serval different versions of CUDA are installed but CUDA-12 is not installed. Is this the problem with <code>Host-B</code>?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0aa8b6c3b89653930e7abc52423abce61363dc10.png" data-download-href="https://discuss.pytorch.org/uploads/default/0aa8b6c3b89653930e7abc52423abce61363dc10" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0aa8b6c3b89653930e7abc52423abce61363dc10.png" alt="image" data-base62-sha1="1wif2D1u7qgRrzlASzEqVTuk9u8" width="690" height="45" data-dominant-color="20292C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1087×72 6.75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The version of <code>Pytorch</code> and <code>Python</code> is the same on two Hosts.</p>

cheeks: <p>I have the same issue on Cuda 12.2 machine. RTX A6000 gpus. The code below can recreate the issue. <code>torchrun --nnodes=1 --nproc_per_node=gpu ddp_test.py --distributed</code></p>
<pre><code class="lang-auto">import argparse  
import os
import torch.distributed as dist

parser = argparse.ArgumentParser()
parser.add_argument("--distributed", action="store_true")
args = parser.parse_args()
args.local_rank = int(os.environ['LOCAL_RANK'])

print(f"Local Rank {args.local_rank} | Using distributed mode by DistributedDataParallel")

dist.init_process_group(backend='nccl')
print(f'before barrier. Rank: {args.local_rank} {dist.get_rank()}. World size {dist.get_world_size()}')
dist.barrier()
print(f'after barrier. Rank: {args.local_rank}')
</code></pre>
<p>Logs show that all the ranks reach the barrier and then get stuck.</p>
<pre><code class="lang-auto">[2025-01-24 15:35:42,293] torch.distributed.run: [WARNING] 
[2025-01-24 15:35:42,293] torch.distributed.run: [WARNING] *****************************************
[2025-01-24 15:35:42,293] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2025-01-24 15:35:42,293] torch.distributed.run: [WARNING] *****************************************
Local Rank 1 | Using distributed mode by DistributedDataParallel
Local Rank 2 | Using distributed mode by DistributedDataParallel
Local Rank 0 | Using distributed mode by DistributedDataParallel
Local Rank 3 | Using distributed mode by DistributedDataParallel
before barrier. Rank: 1 1. World size 4
before barrier. Rank: 3 3. World size 4
before barrier. Rank: 2 2. World size 4
before barrier. Rank: 0 0. World size 4
</code></pre>

cheeks: <p>Issue turned out to be the NVLink setup between the GPUs on the machine i was using. I had to disable the NVLink usage and instead use PCI. The following exports resolved this for me.</p>
<p>export NCCL_P2P_DISABLE=1 export NCCL_IB_DISABLE=1 export NCCL_SOCKET_IFNAME=lo</p>