Dist.init_process_group hangs silently

alchemi5t: <p>Hello,</p>
<p>I am trying to get a multi gpu training sess on, but the process keeps hanging on dist.init_process_group( no error or any kind of INFO message, even with NCCL_DEBUG=INFO). Any insights on how to fix this?</p>
<pre><code class="lang-auto">Fri Sep  6 15:27:48 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.26       Driver Version: 430.26       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN V             Off  | 00000000:18:00.0 Off |                  N/A |
| 38%   54C    P8    30W / 250W |      0MiB / 12066MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN V             Off  | 00000000:3B:00.0 Off |                  N/A |
| 44%   61C    P8    40W / 250W |      0MiB / 12066MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN Xp            Off  | 00000000:86:00.0 Off |                  N/A |
| 25%   44C    P8    10W / 250W |      2MiB / 12196MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  TITAN Xp            Off  | 00000000:AF:00.0  On |                  N/A |
| 25%   45C    P5    20W / 250W |   1242MiB / 12193MiB |      2%      Default |
+-------------------------------+----------------------+----------------------+
</code></pre>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
</code></pre>
<pre><code class="lang-auto">torch==0.4.1   (ALSO tried with 1.1)
torchaudio==0.2
torchsummary==1.5.1
torchtext==0.4.0
torchvision==0.4.0
</code></pre>

Brando_Miranda: <p>where you able to solve this? I also have mine hanging…</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/66498045/how-to-solve-dist-init-process-group-from-hanging-or-deadlocks" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    <img alt="Charlie Parker" src="https://i.stack.imgur.com/2B6rV.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/66498045/how-to-solve-dist-init-process-group-from-hanging-or-deadlocks" target="_blank" rel="noopener nofollow ugc">How to solve dist.init_process_group from hanging (or deadlocks)?</a>
</h4>

<div class="tags">
  <strong>python, machine-learning, pytorch, gpu</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    Charlie Parker
  </a>
  on <a href="https://stackoverflow.com/questions/66498045/how-to-solve-dist-init-process-group-from-hanging-or-deadlocks" target="_blank" rel="noopener nofollow ugc">06:49PM - 05 Mar 21 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>