Why does init_process_group hang with world size > 1?

username: rdoshi
cooked: <p>I am running the following code on a workstation with two RTX 4090s:</p>
<pre><code class="lang-auto">import os
import torch
import torch.distributed as dist

# Set environment variables
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'

# Initialize the distributed environment
dist.init_process_group(backend="nccl", world_size=2, rank=0)
</code></pre>
<p>The line:</p>
<pre><code class="lang-auto">dist.init_process_group(backend="nccl", world_size=2, rank=0)
</code></pre>
<p>causes the code to hang with no errors. When I set world_size=1 it does not hang. When I dug deeper I found that an instantiation of a member of the TCPStore class in the source causes the freeze. I ran this line in isolation on my end:</p>
<pre><code class="lang-auto">    rank = 0  # This would vary for each process
    is_master = (rank == 0)
    world_size = 2
    hostname = 'localhost'  # Replace with the hostname of the master node
    port = 12355
    store = dist.TCPStore(hostname, port, world_size, is_master)
</code></pre>
<p>And it hangs in the same way.</p>
<p>Here is the output of nvidia-smi on my system:</p>
<pre><code class="lang-auto">+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:41:00.0  On |                  Off |
|  0%   50C    P8               9W / 450W |    156MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off | 00000000:42:00.0 Off |                  Off |
|  0%   43C    P8               8W / 450W |     13MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2029      G   /usr/bin/gnome-shell                        149MiB |
|    1   N/A  N/A      2029      G   /usr/bin/gnome-shell                          6MiB |
+---------------------------------------------------------------------------------------+
</code></pre>

username: Essoz
cooked: <p>I remember reading about some issues with IOMMU and p2p communication on RTX4090 cards. Can you try the solution here <a href="https://discuss.pytorch.org/t/torch-distributed-init-process-group-hangs-with-4-gpus-with-backend-nccl-but-not-gloo/149061/6" class="inline-onebox">`torch.distributed.init_process_group` hangs with 4 gpus with `backend="NCCL"` but not `"gloo"` - #6 by ritwik.m07</a> ?</p>

username: rdoshi
cooked: <p>I actually tried this solution and it didn’t work. I also ran this code on a workstation with two A6000s and I’m getting the same behavior where it hangs</p>

username: fduwjj
cooked: <aside class="quote no-group" data-username="rdoshi" data-post="1" data-topic="193686">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rdoshi/48/66242_2.png" class="avatar"> rdoshi:</div>
<blockquote>
<p><code>rank=0</code></p>
</blockquote>
</aside>
<p>If I understand this correctly, rank=0 actually needs to be rank=actual rank?</p>