About torch.autograd.grad

username: bellos1203
cooked: <p>Hi, I have a quick question about <code>torch.autograd.grad</code> function.</p>
<p>First, let me explain my situation.<br>
My model consists of 2 consecutive networks, A &amp; B, where the output of A (<code>output_A</code>) directly becomes the input of B.<br>
After calculating the loss function using the outputs of B, my model computes the gradient of the loss with respect to the output of A to obtain grad-cam, by performing <code>grads = torch.autograd.grad(loss, output_A, retain_graph=True).detach()</code>.<br>
At this stage, my model modifies the <code>output_A</code> using grad cam (denote <code>output_A'</code>) and feeds the modified <code>output_A'</code> to model B again and computes the same loss once again.<br>
Finally, both of the <code>loss</code> computed from <code>output_A</code> and the <code>loss'</code> computed from <code>output_A'</code> are backpropagated to the model using <code>(loss+loss').backward()</code> and <code>optimizer.step()</code>.</p>
<p>By implementing this way, the model works and I’m able to train the model.<br>
But my question is that, is it ok to perform <code>grads = torch.autograd.grad(loss, output_A, retain_graph=True).detach()</code> during forward pass and then backprop the final <code>(loss+loss').backward()</code>?<br>
In other words, doesn’t <code>grads = torch.autograd.grad(loss, output_A, retain_graph=True).detach()</code> harm the <code>loss.backward()</code> part when computing <code>(loss+loss').backward()</code>?<br>
I think it is ok since torch.autograd.grad function just manually computes the gradients, but I just want to be sure about it.</p>
<p>Thank you for your consideration in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: srishti-git1110
cooked: <p><code>.detach()</code> will “return” (rather than modifying inplace) a tensor that is detached from the computation graph. This will break the graph and might cause error/wrong gradient calculations during the backward pass.</p>

username: bellos1203
cooked: <p>Thanks for your reply!</p>
<p>However, aren’t <code>grads = torch.autograd.grad(loss, output_A, retain_graph=True).detach()</code> and <code>(loss+loss').backward()</code> separate procedure?<br>
Does the computed gradient information still left in the computation graph after calling <code>torch.autograd.grad(loss, output_A, retain_graph=True)</code>?</p>
<p>Hmm, I will try without <code>detach()</code> and update my result after comparing both cases (w/ or w/o <code>detach()</code>).</p>

username: srishti-git1110
cooked: <p>Right! For you use-case, using <code>detach()</code> should not cause any errors.</p>
<aside class="quote no-group" data-username="bellos1203" data-post="3" data-topic="163603">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/f04885/48.png" class="avatar"> bellos1203:</div>
<blockquote>
<p>I will try without <code>detach()</code> and update my result after comparing both cases (w/ or w/o <code>detach()</code>).</p>
</blockquote>
</aside>
<p>Sounds good! Feel free to post if you see any errors.</p>

username: bellos1203
cooked: <p>After conducting the experiments for both cases, the difference in the results is quite negligible. It seems <code>detach()</code> does not affect the learning procedure for my case.</p>