Freezing model but not sure it's working

bfeeny: <p>I am doing some transfer learning, and set a function inside my NN class as so:</p>
<pre><code class="lang-auto">    def freeze(self):    
        for name, child in self.arch.named_children():
            if name != "_fc":
                for param in child.parameters():
                    param.requires_grad = False
        for name, param in self.arch.named_parameters():
            if("bn" in name):
                param.requires_grad = True
</code></pre>
<p>Probably a bit convoluted I know, but I was basically just wanting to freeze everything except all BatchNorms and my classifier.</p>
<p>I instantiate my model like so:</p>
<pre><code class="lang-auto">   arch = EfficientNet.from_pretrained(param['arch'])   
    model = Net(arch=arch, n_meta_features=len(meta_features))  # New model for each fold 
    model.freeze()
    model = nn.DataParallel(model)
    model = model.to(device)
</code></pre>
<p>I think I must be doing something wrong however, because I would have suspected my optimizer to throw an error, as i was calling it like so:</p>
<pre><code class="lang-auto">    optim = torch.optim.AdamW(model.parameters(), lr=param['lr'], amsgrad=True) 
</code></pre>
<p>instead of like so:</p>
<pre><code class="lang-auto">    optim = torch.optim.AdamW(filter(lambda p: p.requires_grad, net.parameters()), lr=param['lr'], amsgrad=True) 

</code></pre>
<p>as it’s my understanding that the optimizer will throw an error if it encounters frozen parameters.  I also noticed that my training time (about 1min 47sec per epoch) is essentially unchanged as well, and I would have suspected it to train faster.</p>
<p>Is there something I am doing wrong?</p>

Hmrishav_Bandyopadhy: <p>Hi,<br>
I believe that the optimizer assigns None to all those places were you have set <code>requires_grad=False</code>. As a result, the gradients of all those parameters are not updating. An optimizer works with frozen params even when you do <code>optim = torch.optim.AdamW(model.parameters(), lr=param['lr'], amsgrad=True)</code>,<br>
as essentially, the params have just requires_grad set as False. My suggestion is unfreeze the params and you will see a difference in training time <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>For more info, check out <a href="https://discuss.pytorch.org/t/trying-to-understand-optimizer-and-relation-to-requires-grad/7994">this</a></p>

bfeeny: <p>If the optimizer is smart enough so that it looks at the <code>requires_grad</code>, why do people pass lambda functions into the optimizer, why not just pass in <code>model.parameters()</code>?</p>
<p>As for my training time, because I was setting all the BatchNorm to be unfrozen, and those are scattered literally all throughout the model, backdrop still had to be done, even though it wasn’t updating the parameters, since everything with backpropogation is chained/dependent.  I tried to just freeze the classifier and saw a difference in speed, but since the actual updates to the weights is super quick, its the actual calculations that take all the time, that is why I wasn’t seeing a speed increase.</p>
<p>So I guess I am still unclear though.  Is there any difference or benefit to passing in model.parameters() vs passing in<code> filter(lambda p: p.requires_grad, model.parameters())</code> when you have parts of your model frozen?</p>

ayalaa2: <p>Assuming that you set all the <code>requires_grad</code> flags before training at all, there is very little difference in using <code>filter</code>.</p>
<p>In the optimizer code, it iterates over each parameter and if the gradient is <code>None</code>, it continues. If <code>requires_grad=False</code>, the parameter’s gradient will never be updated thus it will always remains <code>None</code>. For this example, I think filter adds very little performance boost.</p>

bfeeny: <p>Thank you, it makes sense.  Perhaps the behavior of at least some optimizers was different in versions past?  I had read that the optimizer would throw an error if it encountered frozen parameters, that is why I was concerned that perhaps my parameters were not set right.</p>

LeeDoYup: <p>same question in</p><aside class="quote quote-modified" data-post="1" data-topic="90096">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leedoyup/48/27013_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/parameters-with-requires-grad-false-are-updated-during-training/90096">Parameters with requires_grad = False are updated during training</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hello. I’am trying to freeze front layers during training. 
Before starting optimization, the optimizer is constructed by 
optimizer = torch.optim.SGD(net.parameters(), lr, ...)

Then, during training, i changed the front layers’ requires_grad=False. 
Specifically, 
for epoch in range(total_epoch):
  if epoch == freeze_epoch:
    net.conv1.weight.requires_grad = False
  train_one_epoch() #update network in an epoch


However, I found that the weights of the front layers are still updated. 
I als…
  </blockquote>
</aside>

<p>momentum, weight decay affect updating params with <code>requires_grad=False</code></p>

bfeeny: <p>Thanks, so it seems the lambda function has merits.</p>