Why backpropagating twice is painfully slow?

kirk86: <p>Hi folks,<br>
I have the following pseudocode:</p>
<pre><code class="lang-auto">for epochs
   for minibatch
       inputs.requires_grad_()
       logits = model(inputs)
       loss1 = loss_fn(logits, y)
       loss1.backward(retain_graph=True)  # backward once
       loss2 = loss_fn(x, y)
       loss = loss1 + loss2 
       loss.backward()  # backward twice
       opt.step()
</code></pre>
<p>Which runs extremely slowly, when watching my gpu volatility its almost always at 0%.<br>
Am I doing sth wrong? Or, maybe you have any ideas to speed things up?</p>
<p>Thanks!</p>

ZimoNitrome: <p>Is there a reason why you need to backprop <code>loss1</code> first and again when <code>loss</code> have been calculated?</p>
<p>The common structure when using multiple objective functions is to simply just backprop both of them at once:</p>
<pre><code class="lang-python">for epochs
   for minibatch
       inputs.requires_grad_()
       logits = model(inputs)
       loss1 = loss_fn(logits, y)
       # oss1.backward(retain_graph=True)    &lt;-- Removed this line
       loss2 = loss_fn(x, y)
       loss = loss1 + loss2 
       loss.backward() 
       opt.step()
</code></pre>
<p>And balance the different losses with constants i.e.:</p>
<pre><code class="lang-python">loss = 2*loss1 + 1*loss2
</code></pre>
<p>If <code>loss1</code> is more important than <code>loss2</code>.</p>

kirk86: <aside class="quote no-group" data-username="ZimoNitrome" data-post="2" data-topic="113968">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p>Is there a reason why you need to backprop <code>loss1</code> first and again when <code>loss</code> have been calculated?</p>
</blockquote>
</aside>
<p>Yes there’s a reason for that, the 1st backprop gives me access to <code>inputs.grad</code> which is used then to produce <code>loss2</code>.</p>
<aside class="quote no-group" data-username="ZimoNitrome" data-post="2" data-topic="113968">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zimonitrome/48/27591_2.png" class="avatar"> ZimoNitrome:</div>
<blockquote>
<p><code> # oss1.backward(retain_graph=True)    &lt;-- Removed this line</code></p>
</blockquote>
</aside>
<p>Removing it will obliterate completely the purpose of <code>loss2</code></p>

ptrblck: <p>How are you profiling these operations? If you are using the GPU, remember that you would need to synchronize the code to get valid profiles, as CUDA operations are executed asynchronously.<br>
If you’ve already taken care of it, I guess the backward operation in <code>loss2</code> adds the overhead.</p>

kirk86: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
thanks for the remarks.<br>
Just to be sure that I’m not doing anything stupid could you please provide a MWE for this</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="113968">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>you would need to synchronize the code to get valid profiles</p>
</blockquote>
</aside>