How to pass certain layers weights in the optimizer

username: n0obcoder
cooked: <p>lets say i have a model</p>
<p>from torchvision import models<br>
model = models.resnet18(pretrained=True)</p>
<p>Now i freeze all the layers</p>
<p>for param in model.parameters():<br>
param.requires_grad = False</p>
<p>and then pass all the models parameters in the optimizer</p>
<p>optimizer = optim.Adam(model.parameters() , lr=0.1)</p>
<p>would the optimzer do nothing, because all the model parameters have requires_grad = False?</p>
<p>thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: saransh
cooked: <p>Hey!</p>
<p>So, what kind of task do you want? You are absolutely correct as all the parameters have their requires_grad set to False.</p>
<p>If you only want to fine tune, you can just change the last layer according to number of output classes that you have.</p>
<pre><code class="lang-auto">model = models.resnet18(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
num_filters = model.fc.in_features
model.fc = nn.Linear(num_filters, &lt;no of output classes&gt;)
model = model.to(device)
</code></pre>
<p>Rest of the pipeline remains the same.</p>
<p>However, if you want to actually turn on the training for particular layers, it can be a bit tricky</p>
<p>One way of actually keeping a track is you can print the different modules in the resnet model using the following code</p>
<pre><code class="lang-auto">for child in model_ft.children():
     print(child)
</code></pre>
<p>You will be able to analyze the various blocks.</p>
<p>One step further would be actually choosing the parameters and enabling the requires_grad for that particular layer.</p>
<pre><code class="lang-auto">for child in model_ft.children():
     for parameter in child.parameters():
        parameter.requires_grad = True
</code></pre>
<p>However you have to carefully make sure that you are enabling the correct parameter.</p>

username: n0obcoder
cooked: <p>So once the requires_grad have been set to True/False for the specific layers, we can simply pass the whole model in the optimizer, right?</p>
<p>like</p>
<p>optimizer = optim.Adam(model.parameters() , lr=0.1)  ???</p>
<p>even i all the parameters have been passed in the optimizer, only thoses parameters will get updated, which have their requires_grad set to True…</p>
<p>Pls correct me if i am wrong and thanks yaar, for the explanation !!! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: saransh
cooked: <p>Yes, that is correct</p>

username: n0obcoder
cooked: <p>thanks saransh. that was very helpful ! : )</p>

username: saransh
cooked: <p>I am glad that I could be of help</p>

username: n0obcoder
cooked: <p>can u pls tell me how to set requires_grad = False or True depending on the layer name of the model.</p>
<p>i know that the following code sets all the parameters to False or True</p>
<p>for param in model.parameters():<br>
param.requires_grad = False</p>
<p>i want to set requires_grad based on the layer name…</p>
<p>how can that be done.</p>

username: ecdrid
cooked: <p>search for named_parameters…</p>
<p>here’s how it’s done,</p>
<pre><code class="lang-auto">print('Training these layers')
for name,param in model.named_parameters():
    if param.requires_grad is True:
        print(name, param.requires_grad)
</code></pre>
<p>I hope you can flip the <code>requires_grad</code> as per the need…</p>
<p>Secondly don’t pass <code>model.parameters()</code> like that rather filter the ones which have <code>requires_grad</code> as <code>True</code>… (looks cool)</p>

username: n0obcoder
cooked: <p>thanks for showing me the named_parameters thing. but can u pls explain in greater details how can i pass the parameters in the optimizer, once i have set the requires_grad as per my needs?</p>

username: saransh
cooked: <p>please refer to : <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a></p>
<p>Everything remains the same, except that you actually selectively switch on / off the layers from the pretrained model whose gradients are to be considered in the backward pass.</p>
<p>Try to code it up and see for yourself, you don’t need to manually specify for a particular layer to be used by your optimizer, everything is taken care off by PyTorch.</p>

username: ecdrid
cooked: <p>Yep Exactly! Do what saransh said!</p>

username: n0obcoder
cooked: <p>can u pls tell me when where and why do we use save and load the optimizers state_dict ? I mean if i wanna resume models training for a few more epochs, do i really need to load the optimizers state_dict or can i just initialize the optimizer once again ? what exactly does the optimizers state_dict contain ?</p>

username: n0obcoder
cooked: <pre><code class="lang-auto">optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=0.0006, momentum=0.9)
</code></pre>
<p>is this what you were talking about? if yes, pls also explain what would happen if pass the parameters as<br>
<code>optimizer = torch.optim.SGD(model.parameters(), lr=0.0006, momentum=0.9)</code> ???</p>

username: ecdrid
cooked: <p>Both are same and the one thing, it’s just what ones prefer over other!</p>

username: n0obcoder
cooked: <p>so the only difference is the way the instruction is written right? both are essentially doing the same thing. there wouldnt be any difference in the output</p>