How can Pytorch calculate total loss like Tensorflow?

WinKawaks: <p>In Tensorflow, we can do backward like this:</p>
<pre><code class="lang-auto">loss = criterion(output, target)
tf.add_loss(loss)
total_loss = tf.losses.get_total_loss()
optimizer = tf.train.MomentumOptimizer(learning_rate=init_lr, momentum=0.9).minimize(total_loss, global_step=global_step)
</code></pre>
<p>But in Pytorch, I can’t find any methods like <em>add_loss</em> or <em>get_total_loss</em>. Then I try to use a custom loss function like this:</p>
<pre><code class="lang-auto">class Loss_func(nn.Module):
    def __init__(self):
        super(Loss_func, self).__init__()
        self.totalLoss = 0
        return

    def forward(self, output, target):
        temp_loss = criterion(output, target)
        self.totalLoss = self.totalLoss + temp_loss
        return self.totalLoss
</code></pre>
<p>I found this error:</p>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</p>
</blockquote>
<p>I changed the backward method like this:</p>
<pre><code class="lang-auto">loss.backward(retain_graph=True)
</code></pre>
<p>A new error occurs:</p>
<blockquote>
<p>Warning: Error detected in AddmmBackward. Traceback of forward call that caused the error:<br>
File “/vol/research/Jigsaw/jigsaw_torch.py”, line 231, in train<br>
features, outputs = model(batchImg.cuda())<br>
File “/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/vol/research/Jigsaw/jigsaw_torch.py”, line 86, in forward<br>
x = self.fc(x)<br>
File “/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/lib/python3.8/site-packages/torch/nn/modules/linear.py”, line 93, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/lib/python3.8/site-packages/torch/nn/functional.py”, line 1690, in linear<br>
ret = torch.addmm(bias, input, weight.t())<br>
(function _print_stack)<br>
Traceback (most recent call last):<br>
File “/vol/research/Jigsaw/jigsaw_torch.py”, line 439, in <br>
main()<br>
File “/vol/research/Jigsaw/jigsaw_torch.py”, line 435, in main<br>
train(trainSet, testSet, tripletSet, model, opt, lossFunc, epoch)<br>
File “/vol/research/sketch/Jigsaw/jigsaw_torch.py”, line 234, in trainAndValidate<br>
loss.backward(retain_graph=True)<br>
File “/lib/python3.8/site-packages/torch/tensor.py”, line 221, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 130, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1024, 81]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
</blockquote>
<p>The model is <strong>GoogleNet</strong>. Does this error mean that the <em>self.fc = nn.Linear(1024, num_classes)</em> in GoogleNet doesn’t support calculating backward twice?</p>
<p>How can I use total loss to calculate backward?</p>

Nikronic: <p>Hi,</p>
<p>If we assume <code>TF.get_total_loss</code> computes just adddition of multiple loss terms then we have these scenarios:</p>
<p>But the error you are getting is from calling <code>loss.backward</code> for multiple times.</p>
<p>Lets say you have following scenario (scenario <span class="hashtag">#1</span>):</p>
<pre><code class="lang-auto">input = #sth
ground_truth = #sth
output = model(input)
loss1 = criterion(output, ground_truth)
loss2 = criterion(changed_output, another_ground_truth)
# etc
total_loss = loss1 + loss2 + ...
total_loss.backward()  # this works 
</code></pre>
<p>Now scenario <span class="hashtag">#2:</span></p>
<pre><code class="lang-auto">input = #sth
ground_truth = #sth
output = model(input)
loss1 = criterion(output, ground_truth)
loss1.backward()  #now graph buffers have been freed, you cannot do backward on the same forward 
loss2 = criterion(changed_output, another_ground_truth)
loss2.backward()  # error, trying to do backward second time
# etc
total_loss = loss1 + loss2 + ...
total_loss.backward()  # this obviously does not work
</code></pre>
<p>You have not showed how you compute <code>loss.backward</code> or how many times you call it, but if you set <code>retain_graph=True</code> for previously used backwards, such as <code>loss1</code> and <code>loss2</code> in scenario <span class="hashtag">#2</span>, then you should be able to do another backward using <code>total_loss</code>.</p>
<p>TF docs are not clear but I think they are using first scenario.</p>
<p>Bests</p>

Yongjie_Shi: <p>I have a question.</p>
<pre><code class="lang-auto">input1 = #sth
input2 = # sth
label1 = # sth
label2 = # sth
output1 = model(input1)
output2 = model(input2)
loss1 = criterion(output1, label1)
loss2 = criterion(output2, label2)
total = loss1+loss2
total_loss.backward()
</code></pre>
<p>Does total loss consider the effect of loss1? I mean will</p>
<pre><code class="lang-auto">output2 = model(input2)
</code></pre>
<p>overwrite the gradient or any  information about input1 ?</p>

Nikronic: <p>Short answer is yes. Autograd will accumulate gradients unless user explicitly changes (zeroes) them.</p>
<p>I think this tutorial <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#optional-reading-vector-calculus-using-autograd" class="inline-onebox" rel="noopener nofollow ugc">A Gentle Introduction to torch.autograd — PyTorch Tutorials 1.8.1+cu102 documentation</a> is a good demonstration. You can think of your problem as a simple equation <code>c = a + b</code> where <code>a</code> and <code>b</code> have their own computational graph (i.e. operations).</p>

Yongjie_Shi: <p>Thanks for your help~ <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>