How to use the backward functions for multiple losses?

username: han
cooked: <p>Hi,</p>
<p>I am playing with the <a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py" rel="nofollow noopener">DCGAN</a> code in pytorch examples .</p>
<p>Replacing <a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py#L224" rel="nofollow noopener">errD_real.backward()</a> and <a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py#L234" rel="nofollow noopener">errD_fake.backward()</a> with errD.backward() after Line 236 results in failure (get the nonsense output) in the training.</p>
<p>where errD = errD_real + errD_fake, but errD.backward() is not equal to errD_real.backward() and errD_fake.backward()</p>
<p>I think both of them should give me the same gradients, but in fact it does not. Does anyone know where the issue comes from? If the model has multiple losses, do we really need to call the backward function for each loss rather than “sum them up and then call the total_loss.backward”?</p>

username: han
cooked: <p>To be specific, changes  like the following will result in failure in training DCGAN:<br>
output = netD(input)<br>
errD_real = criterion(output, label)<br>
<del> errD_real.backward()</del><br>
D_x = output.data.mean()<br>
# train with fake<br>
noise.data.resize_(batch_size, nz, 1, 1)<br>
noise.data.normal_(0, 1)<br>
fake = netG(noise)<br>
label.data.fill_(fake_label)<br>
output = netD(fake.detach())<br>
errD_fake = criterion(output, label)<br>
<del>errD_fake.backward()</del><br>
D_G_z1 = output.data.mean()<br>
errD = errD_real + errD_fake<br>
<strong>errD.backward()</strong><br>
optimizerD.step()</p>
<p>Does anyone know where the issue comes from?</p>

username: Nabarun_Goswami
cooked: <p>I am wondering the same thing. Even while updating the Generator, if I have more terms in the Generator loss apart from the errG in the DCGAN example, should I call backward() for both the components separately or combine them and call backwards once?</p>
<p>What exactly happens on calling backwards() multiple times, does it calculate the gradients twice with respect to the same weights based on the criterion or does the second call calculate gradients on top of the gradients calculated in the first call.</p>
<p>Looking for answers!</p>
<p>Cheers,<br>
Nabarun</p>

username: tymokvo
cooked: <p>I imagine the error that is thrown is from trying to backprop Variables that belong to different graph structures. errD_real belongs to the graph of netD(input) where errD_fake belongs to netD(netG(noise)) (though it has been detached to only affect netD) so you’re trying to get the derivative wrt different inputs into different functions.</p>

username: tom
cooked: <p>Hello <a class="mention" href="/u/nabarun_goswami">@Nabarun_Goswami</a>,</p>
<p>to try to clear this up, in the DCGAN example, you have (think about mathematical functions here, I left out everything not relevant).</p>
<p>loss = criterion(netD(real, params))+criterion(netD(fake, params))</p>
<p>Spelling out the chain rule for the gradient of the loss w.r.t. the params:</p>
<p>∇<sub>params</sub> loss = ∇<sub>params</sub> netD(real, params)* ∇<sub>netD</sub> loss(netD (real,params)) +  ∇<sub>params</sub> netD(fake, params)* ∇<sub>netD</sub> loss(netD (fake,params)),</p>
<p>note how ∇<sub>params</sub> netD is evaluated at two different points, namely (real, params) and (fake, params).</p>
<p>The way backpropagation works is to evaluate the gradients <em>at the locations of the last forward pass</em>.<br>
In theory, you could also copy the network, make the parameters shared and then just add the loss to achieve the same. Then the backprop at real would go through one copy and the one at fake through the other.</p>
<p>Now, this is exactly why (I imagine, I didn’t design it) pytorch actually <em>adds</em> to the .grad on backward to allow the following:</p>
<ol>
<li>You zero gradients. <small>(Ha, I forgot that often enough to need a benefit of needing to that myself.)</small>
</li>
<li>You evaluate netD and criterion at the point real.</li>
<li>You backprop to compute derivatives at the point real (=the last evaluated point). The .grads are added to zero from step 1.</li>
<li>You evaluate netD and criterion at the point fake.</li>
<li>You backprop to compute derivatives at the point fake (=the last evaluated point). The .grads are added to the .grads you had from step 3.</li>
</ol>
<p>You have now computed the gradient of loss, but manually split it into the two summands.</p>
<p>If you just added the two parts to the loss and did backward, the netD would not know about step 2. anymore because step 4. overwrote things.</p>
<p>As seen in the Wasserstein GAN code and friends, you can also supply a -1 tensor to the .backward to emulate terms subtracted from the loss.</p>
<p>Now. If you just call .backward twice, there are two possibilities</p>
<ul>
<li>with keep_graph=True (or keep_variables=True in pytorch &lt;=0.1.12) in the first call, you will do the same as in 3 and five: You backprop twice to compute derivatives at the last evaluated point. The .grads are added to the .grads you had, so you end up computing twice the gradient at the last evaluated point.</li>
<li>without keep_graph=True in the first call, pytorch can throw away the processed bit’s information from forward, and will give an error meaning “forward info is gone, you used it and didn’t tell me to keep it”.</li>
</ul>
<p>I hope this helps you to decide what is the fit for your project.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Cysu
cooked: <p>I think calling backward once on errD_real + errD_fake should be the same as calling it twice separately. The problem mainly attributes to reusing the same label tensor for real and fake data (<a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py" rel="nofollow noopener">https://github.com/pytorch/examples/blob/master/dcgan/main.py</a>, L219, L232). If creating two label tensors, the problem should be resolved. Nevertheless, calling backward twice is recommended as it could save some GPU memory.</p>

username: SimonW
cooked: <p>This is <strong>not</strong> correct. Pytorch will <strong>not</strong> overwrite previous values. Being the same module/parameter or not is not related to how the dynamic graph is built. It will be different links in the graph. The variables saved for backward pass will be stored differently. See the following simple example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; l1 = nn.Linear(3, 3)
&gt;&gt;&gt; l1.weight.data.fill_(0)
&gt;&gt;&gt; l1.bias.data.fill_(0)
&gt;&gt;&gt; x = Variable(torch.ones(2, 3))
&gt;&gt;&gt;
&gt;&gt;&gt; # backward one loss only
&gt;&gt;&gt; loss1 = (l1(x) - 1).abs().sum()
&gt;&gt;&gt; loss1.backward()
&gt;&gt;&gt; l1.weight.grad
Variable containing:
-2 -2 -2
-2 -2 -2
-2 -2 -2
[torch.FloatTensor of size 3x3]

&gt;&gt;&gt; 
&gt;&gt;&gt; # backward the other loss only
&gt;&gt;&gt; l1.weight.grad = None
&gt;&gt;&gt; loss2 = (l1(x) + 1).abs().sum()
&gt;&gt;&gt; loss2.backward()
&gt;&gt;&gt; l1.weight.grad
Variable containing:
 2  2  2
 2  2  2
 2  2  2
[torch.FloatTensor of size 3x3]

&gt;&gt;&gt; 
&gt;&gt;&gt; # backward both losses together
&gt;&gt;&gt; l1.weight.grad = None
&gt;&gt;&gt; loss1 = (l1(x) - 1).abs().sum()
&gt;&gt;&gt; loss2 = (l1(x) + 1).abs().sum()
&gt;&gt;&gt; (loss1+loss2).backward()
&gt;&gt;&gt; l1.weight.grad
Variable containing:
 0  0  0
 0  0  0
 0  0  0
[torch.FloatTensor of size 3x3]

</code></pre>

username: tom
cooked: <p>Indeed, upon rereading this paragraph:</p>
<aside class="quote no-group" data-username="tom" data-post="5" data-topic="1826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>
The way backpropagation works is to evaluate the gradients at the locations of the last forward pass.<br>
In theory, you could also copy the network, make the parameters shared and then just add the loss to achieve the same. Then the backprop at real would go through one copy and the one at fake through the other.
</p>
</blockquote>
</aside>
<p>seems to be misleading (but I cannot edit that post anymore), thank you for pointing that out.</p>
<p>Indeed, what doing the backward in two pieces saves is the saving two calculation graphs at the same time, this has nothing to do with the network (weights).</p>
<p>Best regards</p>
<p>Thomas</p>

username: Sten_Sootla
cooked: <p>Given that the changes made by the OP to DCGAN don’t change the flow of the gradients, what could be the reason for the nonsensical outputs he obtained?</p>

username: Weifeng
cooked: <p>Good catch!<br>
It’s the in-place operation <code>label.data.fill_(fake_label)</code> that breaks the real part sub-graph.<br>
Adding the link for reference.<br>
<a href="https://pytorch.org/docs/stable/notes/autograd.html#in-place-operations-with-autograd" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/autograd.html#in-place-operations-with-autograd</a></p>

username: STU
cooked: <p>so what’s the real reason for this issue?</p>

username: liangbright
cooked: <p>#%% test<br>
import torch<br>
import torch.nn as nn<br>
#%%<br>
layer = nn.Linear(3, 3)<br>
layer.weight.data.fill_(0.1)<br>
layer.weight.grad=None</p>
<h1></h1>
<p>x1 = torch.ones((3,3), requires_grad=True)<br>
y1=layer(x1)<br>
loss1= torch.sum(y1)<br>
loss1.backward()<br>
print(x1.grad)<br>
print(layer.weight.grad)</p>
<h1></h1>
<p>x2 = torch.ones((3,3), requires_grad=True)<br>
x2.data.fill_(0.2)<br>
y2=layer(2*x2)<br>
loss2= torch.sum(y2)<br>
loss2.backward()<br>
print(x2.grad)<br>
print(layer.weight.grad)<br>
#%%<br>
‘’’<br>
tensor([[0.3000, 0.3000, 0.3000],<br>
[0.3000, 0.3000, 0.3000],<br>
[0.3000, 0.3000, 0.3000]])<br>
tensor([[3., 3., 3.],<br>
[3., 3., 3.],<br>
[3., 3., 3.]])<br>
tensor([[0.6000, 0.6000, 0.6000],<br>
[0.6000, 0.6000, 0.6000],<br>
[0.6000, 0.6000, 0.6000]])<br>
tensor([[4.2000, 4.2000, 4.2000],<br>
[4.2000, 4.2000, 4.2000],<br>
[4.2000, 4.2000, 4.2000]])<br>
‘’’<br>
#%%<br>
layer = nn.Linear(3, 3)<br>
layer.weight.data.fill_(0.1)</p>
<h1></h1>
<p>x1 = torch.ones((3,3), requires_grad=True)<br>
x2 = torch.ones((3,3), requires_grad=True)<br>
x2.data.fill_(0.2)</p>
<h1></h1>
<p>y1=layer(x1)<br>
y2=layer(2*x2)</p>
<h1></h1>
<p>loss1= torch.sum(y1)<br>
loss1.backward()<br>
print(x1.grad)<br>
print(layer.weight.grad)</p>
<h1></h1>
<p>loss2= torch.sum(y2)<br>
loss2.backward()<br>
print(x2.grad)<br>
print(layer.weight.grad)<br>
#%%<br>
‘’’<br>
tensor([[0.3000, 0.3000, 0.3000],<br>
[0.3000, 0.3000, 0.3000],<br>
[0.3000, 0.3000, 0.3000]])<br>
tensor([[3., 3., 3.],<br>
[3., 3., 3.],<br>
[3., 3., 3.]])<br>
tensor([[0.6000, 0.6000, 0.6000],<br>
[0.6000, 0.6000, 0.6000],<br>
[0.6000, 0.6000, 0.6000]])<br>
tensor([[4.2000, 4.2000, 4.2000],<br>
[4.2000, 4.2000, 4.2000],<br>
[4.2000, 4.2000, 4.2000]])<br>
‘’’<br>
#%%<br>
layer = nn.Linear(3, 3)<br>
layer.weight.data.fill_(0.1)</p>
<h1></h1>
<p>x1 = torch.ones((3,3), requires_grad=True)<br>
x2 = torch.ones((3,3), requires_grad=True)<br>
x2.data.fill_(0.2)</p>
<h1></h1>
<p>y1=layer(x1)<br>
y2=layer(2*x2)</p>
<h1></h1>
<p>loss1= torch.sum(y1)<br>
loss2= torch.sum(y2)<br>
loss=loss1+loss2<br>
loss.backward()</p>
<h1></h1>
<p>print(x1.grad)<br>
print(x2.grad)<br>
print(layer.weight.grad)<br>
#%%<br>
‘’’<br>
tensor([[0.3000, 0.3000, 0.3000],<br>
[0.3000, 0.3000, 0.3000],<br>
[0.3000, 0.3000, 0.3000]])<br>
tensor([[0.6000, 0.6000, 0.6000],<br>
[0.6000, 0.6000, 0.6000],<br>
[0.6000, 0.6000, 0.6000]])<br>
tensor([[4.2000, 4.2000, 4.2000],<br>
[4.2000, 4.2000, 4.2000],<br>
[4.2000, 4.2000, 4.2000]])<br>
‘’’</p>

username: yingda.yin
cooked: <p>Good catch!<br>
So I think it’s the correct answer. Thanks.</p>

username: Atcold
cooked: <aside class="quote no-group" data-username="tom" data-post="5" data-topic="1826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<ul>
<li>with keep_graph=True (or keep_variables=True in pytorch &lt;=0.1.12) in the first call, you will do the same as in 3 and five: You backprop twice to compute derivatives at the last evaluated point. The .grads are added to the .grads you had, so you end up computing twice the gradient at the last evaluated point.</li>
<li>without keep_graph=True in the first call, pytorch can throw away the processed bit’s information from forward, and will give an error meaning “forward info is gone, you used it and didn’t tell me to keep it”.</li>
</ul>
</blockquote>
</aside>
<p>Pointing out that <code>keep_graph</code> is now called <code>retain_graph</code>.</p>

username: karl7
cooked: <aside class="quote no-group" data-username="Cysu" data-post="6" data-topic="1826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cysu/48/113_2.png" class="avatar"> Cysu:</div>
<blockquote>
<p>calling backward twice is recommended as it could save some GPU memory</p>
</blockquote>
</aside>
<p>sorry, but may i ask why does it save memory?</p>
<aside class="quote no-group" data-username="Cysu" data-post="6" data-topic="1826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cysu/48/113_2.png" class="avatar"> Cysu:</div>
<blockquote>
<p>The problem mainly attributes to reusing the same label tensor</p>
</blockquote>
</aside>
<p>Another question, do you mean the code in referenced link (L219,L232) is wrong because it can cause the problem (different result between twice backward and once backward)? But why…? why same label tensor can cause this difference?</p>
<p>The last thing, more important, i didn’t use any inplace operations in my code like <code>label.fill_(fake_label)</code>, but i also encountered the problem that:<br>
the training result of twice backward is different from add the loss together then use 1 backward…</p>
<p>the later one will cause nan gradient and nan loss in the training, i don’t know why…</p>
<p>every time, i input 1 pair (2 different input with same label) to my network, to be clear:</p>
<pre><code class="lang-auto"># method 1
out1 = model(input1)
loss1 = criterion(out1,label)
loss1.backward()
optimizer.step()
optimizer.zero_grad()

out2 = model(input2)
loss2 = criterion(out2,label)
loss2.backward()
optimizer.step()
optimizer.zero_grad()

# method 2
out1 = model(input1)
out2 = model(input2)
loss1 = criterion(out1,label)
loss2 = criterion(out2,label)
lossAdd = loss1 + loss2
lossAdd.backward()
optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>in my experiment, method 1 and method 2 have different results in training, and method 2 can cause Nan problems after several training epochs (training can not converge). May i ask why does this happen?</p>
<p>is the reason that i used same label tesor?<br>
is because input1 and input2 are too different from each other so can not add their loss together?</p>
<p>Very look forward to your reply…</p>
<p><a class="mention" href="/u/han">@han</a> <a class="mention" href="/u/nabarun_goswami">@Nabarun_Goswami</a>  <a class="mention" href="/u/tymokvo">@tymokvo</a> <a class="mention" href="/u/tom">@tom</a> <a class="mention" href="/u/cysu">@Cysu</a> <a class="mention" href="/u/simonw">@SimonW</a> <a class="mention" href="/u/sten_sootla">@Sten_Sootla</a> <a class="mention" href="/u/weifeng">@Weifeng</a> <a class="mention" href="/u/atcold">@Atcold</a> <a class="mention" href="/u/yingda.yin">@yingda.yin</a></p>
<p>sorry to interrupt so many people with so old post, but i really got stuck here for more than one month, really really need help here…</p>

username: B.W_Zhang
cooked: <p>Not professional but in my opinion, what your method 1 doing is: ‘compute grad 1 → optimize along grad 1 → compute grad 2 → optimize along grad 2’, so when the second forward is performed the model params are already updated once, whereas in your method 2, it’s ‘compute grad 1 → compute and add grad 2 to grad 1 to form grad_total → optimize along grad_total’, there’s only one optimization step going towards the integrated gradient instead of two separated steps. I think this difference is like when you have two dataset, method 1 is alternatively feeding a batch of one dataset at a time and method 2 is mixing two batches one for each dataset and feeding them together.<br>
As for why backward twice saves memory, because once the loss is backward, memories of tensors involved in computing its forward step are freed (that’s why you cannot backward twice with one forward) whereas if you were to accumulate the loss and backward together, the memories are still retained when performing the second forward step.</p>

username: Imahn
cooked: <p>Hello <a class="mention" href="/u/tom">@tom</a>,</p>
<p>I’d have a quick follow-up comment to this part:</p>
<aside class="quote no-group quote-modified" data-username="tom" data-post="8" data-topic="1826">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>Indeed, what doing the backward in two pieces saves is the saving two calculation graphs at the same time, […].</p>
</blockquote>
</aside>
<p>I took the code that <a class="mention" href="/u/simonw">@SimonW</a> provided, and tested their memory consumption by moving both the input tensor <code>x</code> and the NN onto a CUDA device, and then simply doing <code>torch.cuda.memory_allocated(device)</code>. For me, it turns out that the code with</p>
<pre data-code-wrap="python"><code class="lang-python">loss = (l1(x) - 1).abs().sum() + (l1(x) + 1).abs().sum()
loss.backward()
</code></pre>
<p>consumes in total <code>3.0</code> kB (also accounting that <code>x</code> and the NN were moved onto the device), whereas</p>
<pre data-code-wrap="python"><code class="lang-python">loss_first_term = (l1(x) - 1).abs().sum()
loss_first_term.backward()

loss_second_term = (l1(x) + 1).abs().sum()
loss_second_term.backward()
</code></pre>
<p>consumes <code>3.5</code> kB… It seems at odds with your statement, or had you meant it in terms of the running time?</p>

username: tom
cooked: <p>Depending on what you are doing in l1 there will likely be lower order effects. For this function, you’d probably assign l1(x) to an intermediate and be done with it in one step…</p>