Distributed Data Parallel with Multiple Losses

username: maralm
cooked: <p>Hi,</p>
<p>I am using DistributedDataParallel with nccl. I have two losses which are averaged before calling backward. But backward doesn’t work. Here is the part of the code that is problematic:</p>
<pre><code class="lang-auto">self.inputs['qa_in'][i] = Variable (self.inputs['qa_in'][i].data, requires_grad=True)

self.outputs['qa_outputs'][i] = self.qa_outputs(self.inputs['qa_in'][i])

start_logits, end_logits = self.outputs['qa_outputs'][i].split(1, dim=-1)
start_logits = start_logits.squeeze(-1)
end_logits = end_logits.squeeze(-1)

ignored_index = start_logits.size(1)
        
start_positions_ubatches[i].clamp_(0, ignored_index)
end_positions_ubatches[i].clamp_(0, ignored_index)

loss_fct = CrossEntropyLoss(ignore_index=ignored_index)

start_loss = loss_fct(start_logits, start_positions_ubatches[i])
end_loss = loss_fct(end_logits, end_positions_ubatches[i])

self.outputs['loss_out'][i] = (start_loss + end_loss) / 2
self.outputs['loss_out'][i].backward( retain_graph=True)
</code></pre>
<p>and I get the following error:</p>
<p>File “/home/suncast/venv3/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: Expected to mark a variable ready only once. This error is caused by use of a module parameter outside the <code>forward</code> function. The return value of the <code>forward</code> function is inspected by the distributed data parallel wrapper to figure out if any of the module’s parameters went unused. If this is the case, it knows they won’t receive gradients in a backward pass. If any of those parameters are then used outside <code>forward</code>, this error condition is triggered. You can disable unused parameter detection by passing the keyword argument <code>find_unused_parameters=False</code> to <code>torch.nn.parallel.DistributedDataParallel</code>. (mark_variable_ready at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:342)</p>
<p>I think the problem is the self.qa_outputs parameters are used twice in backward but I don’t know how to solve this. I don’t have any problem without distributed.</p>

username: osalpekar
cooked: <p>Have you tried disabling unused parameter detection by passing <code>find_unused_parameters = False</code> to <code>torch.nn.parallel.DistributedDataParallel</code>?</p>

username: maralm
cooked: <p>Yes. I get the following error when I set it to False:</p>
<p>File “/home/suncast/venv3/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: has_marked_unused_parameters_ INTERNAL ASSERT FAILED at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:290, please report a bug to PyTorch.</p>

username: maralm
cooked: <p>Anyone knows how to solve this?</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/maralm">@maralm</a></p>
<p>From your post, it is unclear which part is the DDP model. My assumption is that:</p>
<ol>
<li><code>self.inputs['qa_in'][i]</code>: this is input to DDP forward</li>
<li><code>self.qa_outputs</code>: this is your DDP model</li>
<li><code>self.outputs['qa_outputs'][i]</code>: this is your DDP outputs</li>
</ol>
<blockquote>
<p>I think the problem is the self.qa_outputs parameters are used twice in backward but I don’t know how to solve this. I don’t have any problem without distributed.</p>
</blockquote>
<p>This should be fine, the autograd engine should be able to manage backward inputs and dependencies from <code>start_loss</code> and <code>end_loss</code> properly.</p>
<p>Two questions:</p>
<ol>
<li>Does it work if you directly call <code>self.outputs['qa_outputs'][i].sum().backward()</code> after line 3?</li>
<li>Does any of the model parameters or outputs participates in other forward/backward passes?</li>
</ol>
<p>It will be very helpful for us to debug if you could share a minimum repro example. As we don’t know what happens outside of the posted code snippet, we can only make assumptions.</p>

username: maralm
cooked: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>Thanks for your reply.<br>
Your assumption is correct and self.qa_outputs is just a linear layer.</p>
<p>Regarding your questions:</p>
<ol>
<li>No it doesn’t work with that.</li>
<li>No, I am trying to just run a forward layer and compute backward with autograd.backward on that layer instead of running loss.backward().</li>
</ol>
<p>Basically, I have a large model and when I run it in a conventional way for forward and backward(loss.backward()), it works fine. But, I have a new implementation which runs backward layer by layer using autograd.backward. Using that, the algorithm works fine on a single gpu but face this error in distributed. I tried it on a different model which doesn’t have multiple losses and it is fine. In this case that I add multiple losses, the error comes.</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="maralm" data-post="6" data-topic="71648">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/e36b37/48.png" class="avatar"> maralm:</div>
<blockquote>
<p>No, I am trying to just run a forward layer and compute backward with autograd.backward on that layer instead of running loss.backward().</p>
</blockquote>
</aside>
<p>I see. DDP does not work for this case yet. Currently, all outputs you get from <code>DistributedDataPararlel.forward()</code> must participate in the same backward pass, otherwise, it would mess up with DDP’s internal communication state. Hope this can help explain that: <a href="https://pytorch.org/docs/master/notes/ddp.html#internal-design" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch master documentation</a></p>

username: mrshenli
cooked: <blockquote>
<p>I tried it on a different model which doesn’t have multiple losses and it is fine. In this case that I add multiple losses, the error comes.</p>
</blockquote>
<p>I might have misunderstand the use case. Adding up multiple losses should work, and this is different from running layer-by-layer backward, right? Would I be correct if I assume the code snippet you shared above is adding two losses together instead of doing layer-by-layer backward?</p>
<p>It would be helpful if you could share a minimum repro for this error. Thanks!</p>

username: maralm
cooked: <p>No this is the same issue.<br>
To simplify, let’s assume that I want to find the gradients for the last layer of the network only which includes a linear classifier and loss (using autograd.backward()). If I use a linear layer with single loss, DDP works with autograd but when I add two losses, it gives that error.</p>

username: wuchiz
cooked: <p>Did you solve this problem? I have met the same problem.</p>

username: xannex
cooked: <p>Same problem here. Any solutions?</p>