Can.detach() work for parts of the layer weights?

username: Fawaz_Sammani
cooked: <p>Hi.For a related problem, i would like to update parts of the weights, and keep the rest frozen. For example,  Consider the below network, where the red weights are weights i want to freeze and not update during backpropagation. However, I can only set the <code>requires_grad  = False</code> on a layer weights, not on some weights of a layer. I’ve used <code>.detach()</code> for that specific weight, and PyTorch gives no error, but the particular weight still has the <code>requires_grad = false</code>. Is there any way to specify the update to happen on some weights of a particular layer before calling <code>.backward()</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/355e17188cd64c742b80b644f4e85e42160c2f59.png" data-download-href="https://discuss.pytorch.org/uploads/default/355e17188cd64c742b80b644f4e85e42160c2f59" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/355e17188cd64c742b80b644f4e85e42160c2f59_2_690x243.png" alt="Capture" data-base62-sha1="7C6TnPFmZSMtWJSPgQO65SjCPoZ" width="690" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/355e17188cd64c742b80b644f4e85e42160c2f59_2_690x243.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/355e17188cd64c742b80b644f4e85e42160c2f59_2_1035x364.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/355e17188cd64c742b80b644f4e85e42160c2f59_2_1380x486.png 2x" data-dominant-color="F6F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">1914×676 43.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: isalirezag
cooked: <p>interesting question. im curious to know how this can be possible?</p>

username: lugiavn
cooked: <p>No, detach() is used if you don’t want to do backward on that variable</p>
<p>To freeze some weights, you can add them to a different param groups in the optimizer, and set the learning rate there to be 0.<br>
You can also manually set the grad of those weight to 0 before calling step()</p>

username: Fawaz_Sammani
cooked: <p>Hi.Thanks for your answer. I can’t go with the second option because I want to freeze them before doing the backpropagation. However, for the first option, it dosen’t work as well. PyTorch still tells me that it can’t optimize these parameters. This is my code, and below is the error i get:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.layer1 = nn.Linear(3,5, bias = False)
    def forward(self, x):
        return self.layer1(x)
    
net = Net()
total_num = sum(p.numel() for p in net.parameters() if p.requires_grad)
print(total_num)                               # 15 parameters in total
print(list(net.parameters())[0][1:].numel())   # 12 parameters to optimize
param = list(list(net.parameters())[0][1:])
optim = torch.optim.SGD(param, lr = 0)
</code></pre>
<p><code>ValueError: can't optimize a non-leaf Tensor</code></p>
<p>Is there any other way?<br>
Thank you in advance!</p>

username: rasbt
cooked: <p>I don’t think it’s possible necessarily because the whole layer is a weight matrix and the <code>requires_grad</code> attribute is for that whole variable. What you can do though is to</p>
<ol>
<li>After the forward pass, make a copy of the weights</li>
<li>do backprop as usual</li>
<li>apply the gradients to update the weights as ussual</li>
<li>overwrite the weights you want to have frozen using the weights from the copy in (1) – you can simply use a mask for that, should be straightforward.</li>
</ol>
<p>While it sounds like an ugly workaround, this is probably the most efficient approach, I assume, as you don’t have to add some custom checks during the forward or backward passes but just do a copy -&gt; mask -&gt; overwrite</p>

username: Fawaz_Sammani
cooked: <p>Hi <a href="https://discuss.pytorch.org/u/rasbt">Sebastian Raschka</a>,<br>
Thanks for your answer. Yes it should work that way,but I’d like to save training time and not perform backprop on the weights i want to freeze. So the whole thing needs to happen before <code>loss.backward()</code>. If i do as you mentioned, or i set the gradients to zero before <code>optimizer.step()</code>, it still works, but there is no point because the backpropgation on all weights is happening anyways when calling <code>loss.backward()</code>.<br>
It seems that this is not possible in PyTorch by any means.</p>
<p>If there is anyone from the community that has a solution to this, it would be great if you share it here!</p>

username: rasbt
cooked: <p>Then your only option might be to unvectorize your weights. Like I said, the problem is that you use weight matrices where the weight matrix as a whole counts as a parameter. If you want to not backprop certain weights, you need to unroll the weights into a for loop such that each weight_{i, j} is a network parameter with its own “requires_grad”</p>
<p>Or, after the backward() call and before the .step() call, you can go into layer.weight.grad and zero the weights that you want to not updated (I think this would be equivalent in most cases).</p>
<p>Btw I don’t see that as a Python or PyTorch limitation but rather that we vectorize for loops and partial derivative computations using matrix calculus. So this problem you have would be the same with pen&amp;paper and other toolkits if you were to use matrix calculus.</p>

username: Fawaz_Sammani
cooked: <p>Hi. Thanks for your answer.<br>
Not calculating the gradients using a matrix multiplication was my last option. So it seems i’ll go with it.<br>
Thanks for your kind reply! Highly appreciated.</p>

username: oezguensi
cooked: <p>There is an “easy” way of doing this. By actually specifying parameter-specific learning rates and setting them to 0 as already suggested in the other comments.</p>
<p>As you already found out you can’t specify each specific parameter in the optimizer like so: <code>{'params': model.conv.weight[0, 0, 0, 0], 'lr': 0}</code>. This raises the <code>ValueError: can't optimize a non-leaf Tensor</code>.<br>
But you can specify a tensor for the learning rate that has the same shape as the parameter group: <code>{'params': model.conv.weight, 'lr': tensor.zeros_like(model.conv.weight)}</code>. If you do so you will encounter another problem: The optimizer e.g. Adam doesn’t allow a tensor as learning rate as <code>addcdiv_()</code> expects a scalar for its argument <code>value</code>. Therefore you need to change this last line in the <code>step</code> method to: <code>p.data.add_((-step_size * (exp_avg / denom)))</code></p>

username: Fawaz_Sammani
cooked: <p>Hi <a class="mention" href="/u/oezguensi">@oezguensi</a>. Thanks for your reply. Yes that solves the problem. But in this case, I still need to do <code>loss.backward()</code> and perform backprop for all the weights. Then, I can do the update as you specified. My main question was about <strong>not</strong> performing backprop for <em>some</em> of the weights within the layer. I really appreciate your help though!</p>

username: Manu_Sheoran
cooked: <p>Hey! have you found an efficient solution?<br>
I’m also facing a similar problem, where I need to freeze my weights to a fixed value. I want to save training time by neglecting the backprop on the weights that I freeze.<br>
For now, I’m overwriting the weights again and again to the fixed value in every forward pass. But that’s not an efficient way.</p>