Process got stuck when set find_unused_parameters=True in DDP

oliver_ss: <p><strong>Description of the problem</strong><br>
Hi,</p>
<p>I tried to train the model with DDP and a strange question occurred to me.</p>
<p>The training process worked really well if DataParallel was used instead of DistributedDataParallel.</p>
<p>When I wrapped the model with DDP,  the exception below was raised:</p>
<pre><code class="lang-auto">RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_para
meters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If
 you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return va
lue of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module whe
n reporting this issue (e.g. list, dict, iterable).
</code></pre>
<ol>
<li>
<p>I have tried to set “find_unused_parameters=True”, but the training process would get stuck. Specifically, the loss failed to go backward.</p>
</li>
<li>
<p>I have also checked all the parameters in my model with the method proposed here: <a href="https://discuss.pytorch.org/t/how-to-find-the-unused-parameters-in-network/63948" class="inline-onebox">How to find the unused parameters in network</a><br>
I am really sure all the parameters have changed after several iterations, but I would still meet the “RuntimeError” as above.</p>
</li>
</ol>
<p>Ps: The output of the model is a list of tensors.</p>
<p>I am really confused and wondering whether there exists any tips to help solve this problem.</p>
<p>Thank you!</p>

agolynski: <p>Hi,</p>
<p>Could you maybe add a source code with reproduction of the problem?</p>
<p>One more suggestion. I’ve recently looked at a problem<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/47562" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/47562" target="_blank" rel="noopener nofollow ugc">Wrong gradients when using DistributedDataParallel and autograd.grad.</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-11-07" data-time="17:37:48" data-timezone="UTC">05:37PM - 07 Nov 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-11-19" data-time="17:37:56" data-timezone="UTC">05:37PM - 19 Nov 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/TropComplique" target="_blank" rel="noopener nofollow ugc">
          <img alt="TropComplique" src="https://avatars0.githubusercontent.com/u/19353433?v=4" class="onebox-avatar-inline" width="20" height="20">
          TropComplique
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
The gradient reduction across workers doesn't work when:
gradient penalty is used as a loss,
a bias is used in the last...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">oncall: distributed</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
where loss function didn’t depend on all the parameters and it caused some issues for the gradient computation. What helped is to look at the autograd graph via <a href="https://github.com/szagoruyko/pytorchviz" rel="noopener nofollow ugc">https://github.com/szagoruyko/pytorchviz</a>
<p>Could you create such a graph from your model?</p>

oliver_ss: <p>Hi,</p>
<p>Thanks for your help. When I tried to make a toy example for you to reproduce the problem, I found that the toy example really worked well, which was strange.</p>
<p>I finally found that the problem lies in the loss function. Specifically, <strong>the output of the model is not fully used when calculating the loss</strong> as for the detection or instance segmentation task, only positive samples are used to get the loss. So if a certain branch does not have any corresponding groundtruth in a certain step, the error of unused parameters would be thrown.</p>
<p>To handle this problem, simply set the “find_unused_parameters=True” seems not to work, as least in my case. <strong>A simple solution</strong> I could find is adding a safety loss, this is, fetch all the outputs of my model and multiply them by 0. This would give all the parameters a 0 gradient but it might waster some cuda memory.</p>

mrshenli: <aside class="quote no-group" data-username="oliver_ss" data-post="3" data-topic="106078">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver_ss/48/30101_2.png" class="avatar"> oliver_ss:</div>
<blockquote>
<p>To handle this problem, simply set the “find_unused_parameters=True” seems not to work, as least in my case.</p>
</blockquote>
</aside>
<p>Yep, this is true. Because that mode in DDP would require full access to all outputs and then traverse the graph from those outputs to find unused parameters. That’s also the reason that the error message says: “(2) making sure all <code>forward</code> function outputs participate in calculating loss.”</p>
<blockquote>
<p><strong>A simple solution</strong> I could find is adding a safety loss, this is, fetch all the outputs of my model and multiply them by 0. This would give all the parameters a 0 gradient but it might waster some cuda memory.</p>
</blockquote>
<p>Another option might be only returning outputs that participate in computing loss. Other outputs can be stored in some model attributes, and retrieve them separately after the forward pass.</p>

Fredrik_Opeide: <aside class="quote no-group" data-username="oliver_ss" data-post="3" data-topic="106078">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver_ss/48/30101_2.png" class="avatar"> oliver_ss:</div>
<blockquote>
<p>fetch all the outputs of my model and multiply them by 0</p>
</blockquote>
</aside>
<p>Brilliant! I used this on my object detection model that i was training with DDL, and a safety loss on the box regression (regression*0).mean() solved my issues.<br>
Thanks a lot.</p>