Doubt regarding effect of requires_grad in loss backprop

username: Sai_Niranjan
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a6403759a1a8eec20a080097bfdd70fdd257a0d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6a6403759a1a8eec20a080097bfdd70fdd257a0d" title="img_train"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a6403759a1a8eec20a080097bfdd70fdd257a0d_2_690x249.jpeg" alt="img_train" data-base62-sha1="fbaTavK3bPko7VBl3F4XzhkL74h" width="690" height="249" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a6403759a1a8eec20a080097bfdd70fdd257a0d_2_690x249.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6a6403759a1a8eec20a080097bfdd70fdd257a0d_2_1035x373.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a6403759a1a8eec20a080097bfdd70fdd257a0d.jpeg 2x" data-dominant-color="F8F0F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">img_train</span><span class="informations">1143×414 46.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have a doubt regarding what’s going on behind the implementation of the architecture shown in the figure.</p>
<p>In the figure outputs 1 and 2 come from inputs 1 and 2 respectively.</p>
<p>Assume  that we detach output 1 and set requires_grad = False for F1 . It is observed empirically that loss.backward() works however based on the documentation gradient computation will stop after reaching F1 in a backward pass . So how exactly are the gradients being updated ?</p>
<p>Will there be any difference if we set requires_grad = True for F1 ?</p>
<p>Suppose now that output 1 is not detached and the loss is backpropagated this time .</p>
<p>Firstly , how exactly is the resultant gradient for the loss computed ?</p>
<p>Is it the sum of the partial derivatives we get with respect to outputs 1 and 2 or is it calculated in some other manner ?</p>
<p>Lastly if we set requires_grad = False for process 1 and input 1 and do the loss computation again will the gradients for output 2 have the same value as when output 1 was detached ?</p>

username: googlebot
cooked: <p>If F1 is a layer/module with at least one parameter, you can view it as a function: out = F1(input, F1.param). I’m assuming you’re asking about applying it multiple times. Basically, input.detach() or F1.param.requires_grad=False disables only one of backprop paths; if you freeze a parameter, dOut/dInput backprop may continue, dOut/dParam stops.</p>
<aside class="quote no-group" data-username="Sai_Niranjan" data-post="1" data-topic="91937">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sai_niranjan/48/27575_2.png" class="avatar"> Sai_Niranjan:</div>
<blockquote>
<p>Firstly , how exactly is the resultant gradient for the loss computed ?</p>
<p>Is it the sum of the partial derivatives we get with respect to outputs 1 and 2 or is it calculated in some other manner ?</p>
</blockquote>
</aside>
<p>for any X with requires_grad==True, as dLoss/dX = dLoss/dOut * dOut/dX<br>
yes, you sum gradients if backprop reaches the same node (i.e. F1.param) multiple times.</p>
<aside class="quote no-group" data-username="Sai_Niranjan" data-post="1" data-topic="91937">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sai_niranjan/48/27575_2.png" class="avatar"> Sai_Niranjan:</div>
<blockquote>
<p>Lastly if we set requires_grad = False for process 1 and input 1 and do the loss computation again will the gradients for output 2 have the same value as when output 1 was detached ?</p>
</blockquote>
</aside>
<p>Not sure I understand this, dLoss(out2,…)/dOut2 won’t depend on out1 if you don’t combine out1 and out2.</p>

username: Sai_Niranjan
cooked: <p>A little bit of clarification for my last doubt what I meant was is detaching output 1 equivalent to setting requires_grad = False in input 1 for example ?</p>

username: googlebot
cooked: <p>Without output1.detach(), F1.param may still train regardless of input tensor mode.</p>

username: Sai_Niranjan
cooked: <p>Hi , regarding your explanation that dOut/dInput backprop may continue while the other path is stopped. I did a little experiment , initialize a simple linear layered NN with some random input , output and target for loss computation. If I wrap the model in torch.nograd then backward() will show an error as no gradients can be passed . Now , suppose if I set requires_grad = False backprop is computed and I get some value a I get the same value even when I set requires_grad = True . Acc to the forums (<a href="https://discuss.pytorch.org/t/no-grad-vs-requires-grad/21272/2" class="inline-onebox">No_grad() vs requires_grad</a>) , requires_grad makes sense only for individual tensors and not for model as a whole as backprop will take place regardless . Thus , I am confused by your explanation that dOut/dparam will stop</p>

username: googlebot
cooked: <p>that gradient is only used to set param.grad (leaf variable): param.grad = dLoss/dParam = (dLoss/dOut) * (d/<strong>dParam</strong> F1(input, param)). param.requires_grad = False blocks the second term and you’ll have no param.grad. Backprop path for input is not affected, as it proceeds via d/<strong>dInput</strong> F1(input,param)</p>