Parameters with requires_grad = False are updated during training

LeeDoYup: <p>Hello. I’am trying to freeze front layers during training.</p>
<p>Before starting optimization, the optimizer is constructed by</p>
<pre><code class="lang-python">optimizer = torch.optim.SGD(net.parameters(), lr, ...)
</code></pre>
<p>Then, during training, i changed the front layers’ <code>requires_grad=False</code>.<br>
Specifically,</p>
<pre><code class="lang-python">for epoch in range(total_epoch):
  if epoch == freeze_epoch:
    net.conv1.weight.requires_grad = False
  train_one_epoch() #update network in an epoch

</code></pre>
<p>However, I found that the weights of the front layers are still updated.<br>
I also double-checked that the front layer’s <code>requires_grad=False</code> after changing it.</p>
<ol>
<li>Why the gradients of front layers are still computed and updated?</li>
<li>How can i implement the same effect in other way (without wrapping each layer as module -&gt; because layers to freeze are not pre-defined ) ??</li>
</ol>
<p>p.s.</p>
<pre><code class="lang-python">def train_one_epoch():
  for data, target in train_loader:
    ...
    optimizer.zero_grad()
    loss = ...
    loss.backward()
    optimizer.step()
</code></pre>
<p>Thank you.</p>

KFrank: <p>Hi Doyup!</p>
<aside class="quote no-group" data-username="LeeDoYup" data-post="1" data-topic="90096" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leedoyup/48/27013_2.png" class="avatar"> LeeDoYup:</div>
<blockquote>
<p>However, I found that the weights of the front layers are still updated.</p>
</blockquote>
</aside>
<p>If you use <code>momentum</code> or <code>weight_decay</code> (both supported by <code>SGD</code>),<br>
your weights can still be updated even if their gradients are zero.</p>
<aside class="quote no-group">
<blockquote>
<ol>
<li>Why the gradients of front layers are still computed and updated?</li>
</ol>
</blockquote>
</aside>
<p>Have you checked that your <code>requires_grad = False</code> gradients are,<br>
in fact, being calculated and given non-zero values?  Or do you just<br>
see that the weights are being changed by <code>optimizer.step()</code>?</p>
<p>Best.</p>
<p>K. Frank</p>

LeeDoYup: <p>Thanks for your fast reply !!!</p>
<p>I also checked that the <code>requires_grad=False</code> but, still there are some values in <code>weight.grad</code>.</p>
<p>During training, the gradients are still computed, despite the <code>require_grad=False</code>.</p>

KFrank: <p>Hi Doyup!</p>
<aside class="quote no-group" data-username="LeeDoYup" data-post="3" data-topic="90096" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leedoyup/48/27013_2.png" class="avatar"> LeeDoYup:</div>
<blockquote>
<p>I also checked that the <code>requires_grad=False</code> but, still there are some values in <code>weight.grad</code>.</p>
</blockquote>
</aside>
<p>First, are you using non-zero <code>momentum</code> or <code>weight_decay</code>?</p>
<p>I should have asked my question about non-zero gradients more<br>
carefully:</p>
<p>I’m reasonably sure that <code>SGD</code> implements <code>weight_decay</code> (and<br>
maybe <code>momentum</code>) by modifying the gradients before applying<br>
the actual update step – even if the gradients are zero, or have<br>
<code>requires_grad = False</code>.</p>
<p>So, I should aks:  Are the gradients in question zero after calling<br>
<code>loss.backward()</code>, but <em>before</em> calling <code>optimizer.step()</code>?</p>
<p>Best.</p>
<p>K. Frank</p>

LeeDoYup: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="90096">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>First, are you using non-zero <code>momentum</code> or <code>weight_decay</code> ?</p>
</blockquote>
</aside>
<p>First of all, i am using the momentum in optimizer.<br>
So i understand, the optimizer could update the parameters after i changed the <code>requires_grad=False</code>.</p>
<p>When i check the gradient is “after” calling <code>optimizer.step()</code>.<br>
As you mentioned, if the optimizer with momentum modify the gradients of parameters with <code>requires_grad=False</code>, the phenomenon is understanble.</p>
<p>So i will check the gradients before optimizer.step() and will additionally attach the reply.</p>

LeeDoYup: <p>I found that</p>
<pre><code class="lang-auto">optimizer.zero_grad()
loss = torch.mean(torch.sum(-onehot*pred, dim=1)) #CE loss
loss.backwrad()
</code></pre>
<p>until this, the <code>conv1.weight.grad</code> has zero gradients.<br>
However, after i procede</p>
<pre><code class="lang-auto">optimizer.step()
</code></pre>
<p>the <code>conv1.weight.grad</code> was updated.</p>
<p>Thank you so much !!!</p>

LeeDoYup: <p>Then, <a class="mention" href="/u/kfrank">@KFrank</a>,<br>
Could you let me know how can we change the gradient histories of freezing layers to prevent gradient update after optimizer.step()</p>

PetroFranchuk: <p>Hi <a class="mention" href="/u/leedoyup">@LeeDoYup</a>,</p>
<p>Seems like I have also faced with similar problem. I was trying to use pretrained ResNet with changing few last layers, but it was not training (loss was increasing, while accuracy was near same).</p>
<p><a href="https://discuss.pytorch.org/t/model-loss-not-decreasing-via-transfer-learning/47910">Here</a> you can see how you can specify which layers in the model should be optimized, by checking whether they are freezed or not.</p>
<p>Hope that will help you.<br>
BR,<br>
Petro</p>

ian_malcolm: <p>My experience with AdamW told me that once you pass the parameters into an optimizer and train them for a step, then the only flag that determines whether to calculate grad, is NOT</p>
<blockquote>
<p>param.requires_grad</p>
</blockquote>
<p>but</p>
<blockquote>
<p>param.grad is None</p>
</blockquote>
<p>If you don’t want to re-instantiate the optimizer and just want to tell the existing optimizer not to calculate the grad of some parameters, you have to manually set param.grad to None.</p>
<p>For the same reason, if you want to resume the training of some parameters, requires_grad = True will not  work. Instead you have to do use param.grad = torch.zeros_like(param)</p>
<p>For your reference, my pytorch version is 1.7.1</p>

nmakes: <p>Interesting! Just curious, are there any plans to standardize this behavior? Or has this behavior already changed in the later versions?</p>
<p>For instance, isn’t this property desirable? : Regardless of the model’s state, the <code>requires_grad</code> flag should dictate the gradient computation and the update by the optimizer.</p>

chenyx.cs: <p>I think there two ways:</p>
<ol>
<li>set learning rate to 0, because params’s update is up to both learning rate and grad,  if set lr to 0, params will not change</li>
<li>create new optimizer without params which you dont want update</li>
</ol>