Obtaining gradients w.r.t. summand in a linear combination of loss functions

f_hensel: <p>Hi, this is my first post here and I’m not very familiar with pytorch yet. My situation is as follows:<br>
I’m training a simple (sequential, dense) neural network where the loss consists of a sum of two terms, say:<br>
<code>loss = loss1 + loss2</code>. I would like to investigate the contributions to the gradient of the network weights of each loss term separately (during the training). Unfortunately this does not seem to be straightforward, i.e. something along the lines of the following does not work:</p>
<pre><code class="lang-auto">loss1.backwards()
grad1 = model.layername.weight.grad.detach().clone()
loss2.backwards()
grad2 = model.layername.weight.grad.detach().clone()
</code></pre>
<p>I’ve also tried setting <code>retain_graph=True</code> as well as <code>optimizer.zero_grad()</code> in between.<br>
However I either get that <code>grad1</code> is exactly equal to <code>grad2</code>, which makes no sense since <code>loss1</code> and <code>loss2</code> are obtained through completely different functions, or I get that <code>grad2</code> is just a zero-tensor (which also does not make any sense since the optimization w.r.t. the total loss <code>loss</code> yields very different results that the optimization w.r.t. <code>loss1</code> alone).</p>
<p>What am I doing wrong, resp. how can I obtain the individual gradients w.r.t. to each summand of the total loss function in order to analyze/compare then?<br>
Thank you very much for any help!</p>

ptrblck: <p>Your code snippet should generally work (besides the typo in <code>backward()</code> as seen here:</p>
<pre><code class="lang-python"># setup
model = nn.Linear(1, 1, bias=False)
x = torch.randn(1, 1)

# zero out grads
out = model(x)

loss1 = out.mean()
loss2 = ((out - 1)**2).mean()

loss1.backward(retain_graph=True)
grad1 = model.weight.grad.clone()
model.zero_grad()
loss2.backward()
grad2 = model.weight.grad.clone()

print(grad1)
# tensor([[-1.9150]])
print(grad2)
# tensor([[4.5179]])

# clone each grad
model.zero_grad()
out = model(x)
loss1 = out.mean()
loss2 = ((out - 1)**2).mean()

loss1.backward(retain_graph=True)
grad1 = model.weight.grad.clone()
loss2.backward()
grad2 = model.weight.grad.clone()

print(grad1)
# tensor([[-1.9150]])
print(grad2)
# tensor([[2.6030]]) # = 4.5179 + (-1.9150)

# single loss
model.zero_grad()
out = model(x)
loss1 = out.mean()
loss2 = ((out - 1)**2).mean()
loss = loss1 + loss2
loss.backward()
print(model.weight.grad)
# tensor([[2.6030]])
</code></pre>

f_hensel: <p>Thank you very much for your answer and the example code <a class="mention" href="/u/ptrblck">@ptrblck</a>!<br>
This helped me greatly in figuring out what the issue was in my code. In fact, I accidentally had a <code>.detach()</code> in part of the computation of <code>loss2</code>, that’s why it’s gradient vanished (however it still had an effect on other parts of the network, which is presumably why I got very different results when training w.r.t. <code>loss</code> and <code>loss1</code>).</p>
<p>In case I’ll ever encounter a similar situation: Is there a way to analyse the effect of the back-propagation step by step in order to ‘debug’ or detect undesired behavior (such as disconnectedness of the graph)?<br>
It can be quite difficult to detect such bugs if the loss function is a complicated custom function (as in my case).</p>
<p>I very much appreciate you taking the time to answer my question!</p>

ptrblck: <p>Good to hear it’s working now!</p>
<p>To debug the computation graph you could visualize it via e.g. <a href="https://github.com/szagoruyko/pytorchviz"><code>torchviz</code></a>.</p>

f_hensel: <p>Great, I’ll take a look at <code>torchviz</code>. Thanks again!</p>