Freezing part of the Layer Weights

username: Lok
cooked: <p>Hello Everyone,</p>
<p>How could I freeze some parts of the layer weights to zero and not the entire layer.<br>
I tried below code, but it doesn’t freeze the specific parts(1:10 array in 2nd dimension) of the layer weights.<br>
I am new to ML &amp; started with Pytorch. Appreciate any help. Thanks.</p>
<p>for child in model_ft.children():<br>
print(“Freezing Parameters(1-&gt;10) on the Convolution Layer”,child)<br>
for param in child.parameters():<br>
param.data[:,1:10,:,:].zero_()<br>
param.data[:,1:10,:,:].requires_grad = False</p>
<p>optimizer_ft = OPTIM.SGD(filter(lambda p: p.requires_grad, model_ft.parameters()), lr=0.001, momentum=0.9)</p>

username: sonsang
cooked: <p>I think freezing specific parts of a parameter is not possible in PyTorch because requires_grad flag is set on each Parameter(collection of weights), not each weight.</p>
<p>One possible approach is manually zeroing gradient before you call optimizer function.</p>
<p>After you calculate gradient using backward() function, call</p>
<pre><code class="lang-python">param.grad[:, 1:10, :, :] = 0
</code></pre>
<p>to achieve you want.</p>
<p>Further, you can automatize this method using backward_hook in PyTorch.</p>
<p>You can check this link if you want.<br>
<a href="http://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html</a></p>

username: Lok
cooked: <p>Thanks Sonsang,<br>
This works, I can see the required weights are set to zero after training, but the training accuracy has come down. Initializing(xavier_normal) the non-zero weights didn’t help.<br>
Not sure if I am missing something else in my setup.</p>
<p>Does this approach work same as the freezing the weights to zero in the beginning?</p>

username: sonsang
cooked: <p>Do you mean that training accuracy is going down while training, or the training accuracy decreased after you zeroing-out some gradients?</p>
<p>In the first case, I’m not sure what is happening.</p>
<p>Maybe full code will help the understanding of that situation.</p>
<p>In the second case, it is natural because you use less parameters than the original model.</p>
<p>Thank you.</p>

username: Lok
cooked: <p>It’s the second case, I understand. Thank you.</p>

username: Brando_Miranda
cooked: <p>Just to double check, how is this mathematically correct? Since the chain rule involves products (and sums) couldn’t it lead to earlier layers having gradient zero even though they should not? Consider just a simple 3 number a<em>b</em>c and the derivative would be the product of each a’*b’*c’. Then set b’=0 yields the wrong derivative when it should be a’*b’, no?</p>

username: sonsang
cooked: <p>It does not mean that we set the gradient of b to zero.</p>
<pre><code class="lang-python">param.grad[:, 1:10, :, :] = 0
</code></pre>
<p>This line indicates that we just ignore the calculated gradient w.r.t loss so that the parameters are not updated.</p>
<p>You do not have to worry about your example because all gradients are calculated with backward() call.</p>
<p>We modify the gradient after the backward(), so other parameters are safe.</p>

username: Huige_Cheng
cooked: <aside class="quote no-group" data-username="sonsang" data-post="2" data-topic="9457">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sonsang/48/1528_2.png" class="avatar"> sonsang:</div>
<blockquote>
<p>param.grad[:, 1:10, :, :] = 0</p>
</blockquote>
</aside>
<p>This works, thanks a lot. But what if I want to set different learning rate to the first 10 rows of embedding ?<br>
Can I just grad *= 0.1 like this ?</p>

username: sonsang
cooked: <p>Sure. You can do anything you want with those gradients between <code>backward()</code> and <code>optimizer.step()</code> calls.</p>

username: 3yanlis1bos
cooked: <p>Your suggestion works, and might really be useful in a lot of cases, but it may not have the same effect with (requires_grad=false)</p>
<p>It works since it sets the selected gradients to zero and ensure thats there will be no changes for some certain weights, but the gradients will still be calculated for all the weights, including the ones that have been frozen.</p>
<p>I am concerned about the outcome of that since I am not sure if it would create the same kind of result as setting a requires_grad flag to false.</p>
<p>Because I think =not sure tho= that the loss calculation might be omitting the participation of frozen parameters in the inference when we set requires_grad to false and this would create a focus on the parameters with requires_grad set to true during loss calculation and may eventually let these parameters to be penalized as they are responsible for all the losses of the network.</p>
<p>Please correct me if I am wrong, I know I am assuming a lot of things but, maybe these thoughts can help someone.</p>

username: tengerye
cooked: <p>From the perspective of mathematics, they are the same. Freezing means no updating. Equivalently, the gradients to be updated on are zeros.</p>

username: Giang_Nguyen
cooked: <aside class="quote no-group" data-username="sonsang" data-post="2" data-topic="9457">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sonsang/48/1528_2.png" class="avatar"> sonsang:</div>
<blockquote>
<p>backward</p>
</blockquote>
</aside>
<p>I run like this and no error.<br>
Can I use like this? This way doesnt requires manually zeroing grad like your method:</p>
<pre><code class="lang-auto">decoder.linear.weight[:100].require_grad = False
</code></pre>

username: ANKUR_GUPTA1
cooked: <p>I have this model</p>
<pre><code class="lang-auto">class my_model(nn.Module):
  def __init__(self):
    super(my_model,self).__init__()
    self.conv1 = nn.Conv2d(3,16,kernel_size=3,stride=1,padding=1)
    self.conv2 = nn.Conv2d(16,32,kernel_size=3,stride=1,padding=1)
    self.conv3 = nn.Conv2d(32,64,kernel_size=3,stride=1,padding=1)
    self.pool = nn.MaxPool2d(2, 2)
    self.fc1 = nn.Linear(4*4*64,64)
    self.fc2 = nn.Linear(64,10)
  def forward(self,inp):
    ab = self.pool(F.relu(self.conv1(inp)))
    ab = self.pool(F.relu(self.conv2(ab)))
    ab = self.pool(F.relu(self.conv3(ab)))
    ab = ab.view(ab.shape[0],-1)
    ab = F.relu(self.fc1(ab))
    ab = F.relu(self.fc2(ab))
    return ab
</code></pre>
<p>can you tell how can i set the gradients of first filter of the conv1 layer to be zero<br>
using backward hook</p>

username: uhmbg
cooked: <p>What does this mean, that the first 100 will be frozen? I proved this, and it’s certainly not wrong, but I don’t really understand what weights are frozen, because they are not the first 100?</p>

username: Roy_Paik
cooked: <p>Hi, Let me ask you a quick question:<br>
When I have a model which has total 5 layer, if I want to freeze 3rd layer from the beginning layer, then the gradient of 3rd layer will affect updating 2nd and 1st layer, but the effect of 3rd layer’s gradients is consistent, and thus there won’t be problem, is my understanding correct?</p>

username: tengerye
cooked: <p>Yes, you only need to remember this: calculating the gradients and update the weights using the computed gradients are two different operations.</p>

username: galidor
cooked: <p>Hi, I see this issue is going on for a really long time, so I wrote <a href="https://github.com/galidor/PyTorchPartialLayerFreezing" rel="noopener nofollow ugc">this</a> simple repo in PyTorch. I hope someone may find it useful. It supports only partial freezing of Conv2d, Conv3d and Linear layers, but these are the most common anyway. If you want me to extend it to other layers, feel free to send me a message. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: kwea123
cooked: <p>Hi, this does not always work as intended. It depends on the <strong>optimizer.</strong> For example, if you look at some optimizer like <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/radam.py#L190" rel="noopener nofollow ugc">radam</a>, you can see that even if the current gradient is set to zero, due to the exponential average of previous results, <strong>the parameters will still be updated</strong> (although only by a little).</p>
<p>To achieve exactly what we wish, I can only come up with the naive implementation where we save the parameter value before opt.step() and reset it back after.</p>

username: Vezen_BU
cooked: <p>Hi! In your case, is using a “mask” matrix and a temporal matrix possible? You record the current values of the entries you do not want to update in the “temporal matrix” and use the “mask matrix” to indicate which entries they are.</p>