How do I freeze the specific weights in a layer?

Drezal_Loh: <p>I am new to PyTorch.<br>
I am trying to freeze specific weights in my model. How do I do this?<br>
Here is the output of param.grad</p>
<blockquote>
<p>for param in model.parameters():<br>
param.grad</p>
<p>tensor([[ 1.2880e+00,  1.9992e+00, -1.8617e-01,  9.7155e-04,  6.1021e-02,<br>
-1.6990e-01, -1.6990e-01],<br>
[ 1.2238e+00,  1.6739e+00, -1.5423e-01,  6.8004e-04,  4.8646e-02,<br>
-1.3877e-01, -1.3877e-01],<br>
[-1.4395e-01,  2.6961e-01, -3.0931e-01,  1.2276e-03,  4.1116e-02,<br>
-9.4576e-02, -9.4576e-02],<br>
[-1.4087e+00, -1.7222e+00, -1.0639e-01,  1.6479e-04, -1.9364e-02,<br>
7.0537e-02,  7.0537e-02],<br>
[-3.8626e+00, -5.9818e+00, -7.5635e-01,  1.4528e-04, -8.4420e-02,<br>
2.8996e-01,  2.8996e-01],<br>
[ 4.5561e-01,  2.6636e+00,  3.0582e-01,  1.1337e-03,  8.0234e-02,<br>
-2.2834e-01, -2.2834e-01],<br>
[ 9.2553e+00, -2.3199e+00, -4.4024e+00,  3.2237e-03,  1.1196e-01,<br>
-2.6179e-01, -2.6179e-01],<br>
[ 5.0038e-01, -2.0895e-01, -3.9626e-01,  4.9325e-04,  2.0533e-02,<br>
-5.1361e-02, -5.1361e-02]])<br>
tensor([[-0.0604, -0.0332, -0.1097,  0.1066,  0.0263, -0.0495, -0.0896, -0.0305],<br>
[-0.0236, -0.0129, -0.0855,  0.0561,  0.0050, -0.0431, -0.0806, -0.0351],<br>
[-0.0619,  0.0210, -0.0466,  0.1049,  0.0903,  0.0441, -0.0294,  0.0410],<br>
[ 0.0039,  0.0439,  0.0502, -0.0047,  0.0523,  0.0802,  0.0768,  0.0853],<br>
[ 0.1193,  0.1690, -0.0268, -0.0964,  0.0923,  0.1122,  0.3309,  0.3556],<br>
[-0.1557, -0.0918,  0.0501,  0.1837,  0.1018,  0.0462, -0.1316, -0.0694],<br>
[ 0.2985,  0.2872, -0.2841, -0.2909, -0.2793, -0.2900,  0.0360, -0.2768],<br>
[ 0.0167,  0.0288, -0.0294, -0.0127, -0.0085, -0.0154,  0.0204,  0.0048]])<br>
tensor([-0.1699, -0.1388, -0.0946,  0.0705,  0.2900, -0.2283, -0.2618, -0.0514])<br>
tensor([-0.1699, -0.1388, -0.0946,  0.0705,  0.2900, -0.2283, -0.2618, -0.0514])<br>
tensor([[-3.5904e+00, -3.2880e+00,  3.6735e+00,  3.5596e+00,  3.4067e+00,<br>
3.9122e+00,  1.3480e+00,  3.1120e+00],<br>
[ 1.0125e+00,  1.0352e+00, -3.3861e-01, -1.0392e+00, -2.5707e-02,<br>
3.5781e-03,  2.3016e+00,  1.6572e+00],<br>
[ 1.4970e+00,  1.7948e+00, -1.7893e+00, -1.2388e+00, -7.7342e-01,<br>
-1.2879e+00,  4.3067e-01, -1.0760e-01],<br>
[-6.6468e-02, -1.3030e-01,  1.3785e-01,  1.6426e-02, -2.2650e-02,<br>
4.3137e-02, -3.3531e-02, -8.2672e-02],<br>
[-3.2205e-02, -4.9927e-02,  8.5638e-02,  1.8152e-02,  4.9936e-02,<br>
7.4584e-02,  1.0584e-01,  1.1505e-01]])<br>
tensor([ 3.7052,  1.1885, -1.4792,  0.1808,  0.1393])</p>
</blockquote>
<p>The particular weights I wish to freeze(set grad to zero) is the one in the 1st tensor, [:,5:].</p>
<p>Thanks</p>

Alexey_Demyanchuk: <p>Hey. It is not 100% clear for me what are you trying to achieve, but I answer as far as I understand.<br>
Pytorch weights tensors all have attribute <code>requires_grad</code>. If set to False weights of this ‘layer’ will not be updated during optimization process, simply frozen.<br>
You can do it in this manner, all 0th weight tensor is frozen:</p>
<pre><code class="lang-auto">for i, param in enumerate(m.parameters()):
    if i == 0:
        param.requires_grad = False
</code></pre>
<p>I am not aware of the method how you can do <code>requires_grad = False</code> for the slice of the weights. At least I can’t do it without pytorch complaining.</p>
<p>Anyway, you can zero some slice of the gradients before optimization step, so this exact slice of weights don’t changed after optimization step. Here is a dummy example:</p>
<pre><code class="lang-auto">import torch
m = torch.nn.Linear(4, 2)
opt = torch.optim.Adam(m.parameters())
x = torch.rand((1,4))
y = torch.tensor([[0, 1]], dtype=torch.float32)
crit = torch.nn.BCEWithLogitsLoss()

out = m(x)
loss = crit(y, out)
loss.backward()

for i, param in enumerate(m.parameters()):
    if i == 0:
        param.grad[:,1:] = torch.zeros_like(param.grad[:,1:])

opt.step()
print('after optimizer step')
for param in m.parameters():
    print(param)
</code></pre>
<p>After doing this, I can see that weights tensor[0] slice[:,1:] didn’t change after optimizer step.</p>
<p>Hope it helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

KFrank: <p>Hi Alexey and Drezal!</p>
<aside class="quote no-group" data-username="Alexey_Demyanchuk" data-post="2" data-topic="104722" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexey_demyanchuk/48/25254_2.png" class="avatar"> Alexey_Demyanchuk:</div>
<blockquote>
<p>I am not aware of the method how you can do <code>requires_grad = False</code> for the slice of the weights.</p>
</blockquote>
</aside>
<p>As Alexey notes, you can’t apply <code>requires_grad</code> to only part of a<br>
tensor.</p>
<aside class="quote no-group">
<blockquote>
<p>Anyway, you can zero some slice of the gradients before optimization step, so this exact slice of weights don’t changed after optimization step.</p>
</blockquote>
</aside>
<p>Please note that this approach – zeroing out parts of <code>.grad</code> before<br>
calling <code>opt.step()</code> – doesn’t work in general.  Some optimizers (e.g.,<br>
when using momentum or weight decay) will change the weights<br>
even if <code>.grad</code> is zero.</p>
<p>The more general approach will be to copy the weights you want<br>
frozen, call <code>opt.step()</code> and then rewrite the frozen weights back<br>
into your <code>param</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

Drezal_Loh: <p>Dear Alexey and KFrank,</p>
<p>I have tried both methods and they both worked!<br>
Thank you for noting that zeroing out parts of .grad sometimes will not work.</p>
<p>I am using MSELoss() with ADAM Optimizer.</p>
<p>Regards,<br>
Eric</p>

Alexey_Demyanchuk: <aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="104722">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Please note that this approach – zeroing out parts of <code>.grad</code> before<br>
calling <code>opt.step()</code> – doesn’t work in general. Some optimizers (e.g.,<br>
when using momentum or weight decay) will change the weights<br>
even if <code>.grad</code> is zero.</p>
</blockquote>
</aside>
<p>Thanks, this is really important note!</p>

sakimarquis: <p>That’s a super important notes! The adam has the momentum part, and it indeed doesn’t work well. Is there any way to clear up all the historical gradient stuff?</p>

ndvbd: <p>What about multiplying the tensor with a constant (requires_grad=False) mask?</p>