DistributedDataParallel deadlock

username: Evangelos_Kazakos
cooked: <p>Hello,</p>
<p>Im trying to use DistributedDataParallel to train a model in a cluster with nodes that each has 2 GPUs K80 and in total Im using 4 GPUs. Im based on this example for the distributed training: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/imagenet/main.py</a>.</p>
<p>My problem is that possibly it gets into a deadlock as it never terminates and never returns an error message, the terminal is totally freezing. I ve seen in the documentation of PyTorch the following:</p>
<p>If you plan on using this module with a <code>nccl</code> backend or a <code>gloo</code><br>
backend (that uses Infiniband), together with a DataLoader that uses<br>
multiple workers, please change the multiprocessing start method to<br>
<code>forkserver</code> (Python 3 only) or <code>spawn</code>. Unfortunately<br>
Gloo (that uses Infiniband) and NCCL2 are not fork safe, and you will<br>
likely experience deadlocks if you don’t change this setting.</p>
<p>and indeed im using gloo backend since DistributedDataParallel supports only gloo. But I tried with a single workers and I still have the same problem.</p>
<p>Also I ve seen in this topic <a href="https://discuss.pytorch.org/t/distributed-data-parallel-freezes-without-error-message/8009/3">Distributed data parallel freezes without error message</a> that PyTorch developers are currently working in the NCCL2 problem.</p>
<p>Does anybody know if currently there is any solution to the problem? Is there any way to avoid the deadlock and make the distributed training to work? Thank you!</p>

username: nadia
cooked: <p>I am facing the same issue but with nccl backend.<br>
It freezes with no error message on <code>model = torch.nn.parallel.DistributedDataParallel(model)</code> line.</p>

username: Evangelos_Kazakos
cooked: <p>I still dont have a solution for it. As Im trying to use DistributedDataParallel along with DataLoader that uses multiple workers, I tried setting the multiprocessing start method to ‘spawn’ and ‘forkserver’  (as it is suggested in the PyTorch documntation) but Im still experiencing a deadlock.</p>

username: Sandeep_Subramanian
cooked: <p>Sorry for bumping this but I have the same problem that <a class="mention" href="/u/nadia">@nadia</a> described. I’ve been able to run the first example described in <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/dist_tuto.html</a> with the gloo and nccl backends, which makes me believe this might not be related to an NCCL deadlock, but things just freeze when I do <code>model = torch.nn.parallel.DistributedDataParallel(model)</code>.</p>

username: Dessix
cooked: <p>You mentioned “A single worker” - setting it to “0 workers” runs it in-process, which should not produce the deadlock.</p>

username: mohsaied
cooked: <p>getting the same hang with the pytorch imagenet distributed example<br>
freezes at the line: torch.nn.parallel.DistributedDataParallel(model)</p>
<p>any solutions?</p>

username: Shoaib_Ahmed_Siddiqu
cooked: <p>I have opened up a new issue on this to discuss since I am facing the same problem.<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/22834" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/shoaibahmed" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/3884843?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/22834" target="_blank" rel="nofollow noopener">Issue: Script freezes with no output when using DistributedDataParallel</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/shoaibahmed" target="_blank" rel="nofollow noopener">shoaibahmed</a>
	on <a href="https://github.com/pytorch/pytorch/issues/22834" target="_blank" rel="nofollow noopener">2019-07-13</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
I was trying to evaluate the performance of the system with static data but different models, batch sizes and AMP...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: distributed</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Shoaib_Ahmed_Siddiqu
cooked: <aside class="quote no-group" data-username="Shoaib_Ahmed_Siddiqu" data-post="7" data-topic="14809">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shoaib_ahmed_siddiqu/48/9118_2.png" class="avatar"> Shoaib_Ahmed_Siddiqu:</div>
<blockquote>
<p>I have opened up a new issue on this to discuss since I am facing the same problem.</p>
</blockquote>
</aside>
<p>The more people contribute there on the discussion, the better it will be. This is a significant problem, and it should be visible to the maintainers of PyTorch through the raised issue.</p>

username: valayDave
cooked: <p>I am facing the same problem. DDP with mp.spawn is freezing randomly.</p>

username: valayDave
cooked: <p>I actually found a solution to my problem of why the workers were freezing at the end of training. I realised that the Data Partition from which mini-batches were created had an imbalanced between workers. Due to this one worker went into deadlock because of gradient syncing.</p>
<p>So one needs to ensure the same number of minibatches by altering batch size to fit with the workers.</p>

username: mrshenli
cooked: <p>Hopefully, we can have a solution for this in the next release based on this design: <a href="https://github.com/pytorch/pytorch/issues/38174" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/38174</a></p>