How can I disable all layers gradient expect the last layer in Pytorch?

Shisho_Sama: <p>Hello All, I’m trying to fine-tune a resnet18 model.</p>
<p>I want to freeze all layers except the last one. I did</p>
<pre><code class="lang-auto">resnet18 = models.resnet18(pretrained=True)
resnet18.fc = nn.Linear(512, 10) 
for param in resnet18.parameters():
    param.requires_grad = False 
</code></pre>
<p>However, doing</p>
<pre><code class="lang-auto">for param in resnet18.fc.parameters():
    param.requires_grad = True
</code></pre>
<p>Fails. How can I set a specific layers parameters to have requires_grad to True?</p>
<p>Thank you all in advance</p>
<p>Note:</p>
<p>I specifically don’t want to swap the order of assigning a new layer with setting all the grads to false</p>
<p>I want to learn how this specific thing can be done.</p>

ptrblck: <p>Your code works.<br>
After running your code snippets, you can print the <code>requires_grad</code> attributes:</p>
<pre><code class="lang-python">for name, param in resnet18.named_parameters():
    print(name, param.requires_grad)
</code></pre>
<p>which shows, that <code>fc.weight</code> and <code>fc.bias</code> both require the gradient.<br>
You will also get a valid gradients in these layers:</p>
<pre><code class="lang-python">resnet18(torch.randn(1, 3, 224, 224)).mean().backward()
for name, param in resnet18.named_parameters():
    print(name, param.grad)
</code></pre>

Shisho_Sama: <p>Thank you very much, but the code I gave produces an error. it says:<br>
<code>fc doesnt have any attribute named parameters()</code><br>
so instead I did :</p>
<pre><code class="lang-auto">for _, param in resnet18.fc._parameters.items(): 
    print(param.requires_grad)
    param.requires_grad = True
</code></pre>
<p>and interestingly for  this  to work I have to do :</p>
<pre><code class="lang-auto">for module in resnet18.modules():
    if module._get_name() != 'Linear':
        print('layer: ',module._get_name())
        for param in module.parameters():
            param.requires_grad_(False)
    elif module._get_name() == 'Linear':
        for param in module.parameters():
            param.requires_grad_(True)
</code></pre>
<p>again if I just do :</p>
<pre><code class="lang-auto">for module in resnet18.modules():
    if module._get_name() != 'Linear':
        print('layer: ',module._get_name())
        for param in module.parameters():
            param.requires_grad_(False)
</code></pre>
<p>and print</p>
<pre><code class="lang-auto">for param in resnet18.parameters():
    print(param.requires_grad)
</code></pre>
<p>all parameters are set as False!<br>
This is really puzzeling.</p>

ptrblck: <p>I would recommend to stick to the <code>named_parameters</code> approach, as in your approach <code>resnet18.modules()</code> will also return <code>fc.weight</code> and <code>fc.bias</code>, which do not contain the <code>'Linear'</code> name in it.</p>
<aside class="quote no-group" data-username="Shisho_Sama" data-post="3" data-topic="53043">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shisho_sama/48/6926_2.png" class="avatar"> Shisho_Sama:</div>
<blockquote>
<p>but the code I gave produces an error. it says:<br>
<code>fc doesnt have any attribute named parameters()</code></p>
</blockquote>
</aside>
<p>Does this code raise this error:</p>
<pre data-code-wrap="python"><code class="lang-python">for name, param in resnet18.fc.named_parameters():
    print(name, param.requires_grad)
</code></pre>
<p>If so, could you post your pytorch and torchvision versions, as I would like to have a look at it?</p>

Shisho_Sama: <p>Its very weird! both your code and also!<code>fc.parameters()</code> are now working just fine!!!<br>
This has got me confused for two days! and now they are just working fine!<br>
I don’t know what could have caused this! or I may have pretty much made a mistake!<br>
By the way I am running Pytorch 1.0!<br>
Any way thanks a gazillion times that was a tremendous help.<br>
By the way do you mind if I ask you to kindly have a look <a href="https://discuss.pytorch.org/t/why-doesnt-pytorch-provide-properties-for-class-names/47220/4">here</a> as well?</p>

ptrblck: <p>Good to hear, it’s working now!<br>
If you are running a Jupyter notebook, make sure to run all previous cells, as it’s easy to forget about old variables etc. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Shisho_Sama: <p>Yes, it was on Jupyter,<br>
One more thing I was experimenting with different ways of doing this and wrote this :</p>
<pre><code class="lang-auto">for k, p in resnet18.fc._parameters.items():
    p.requires_grad = True
</code></pre>
<p>which works but I tried to changed it again and wrote it this way this time:</p>
<p><code>(p.requires_grad_(True) for k,p in resnet18.fc._parameters.items())</code></p>
<p>which failed miserably!<br>
I expected this to also work since I’m using the inplace operator (<code>requires_grad_</code>) but it doesnt! do you know why this is not working?</p>

ptrblck: <p>In your second code snippet you are creating a Python <code>Generator</code>, which will be lazily evaluated.<br>
Your code works, if you execute the generator or use a list comprehension instead.</p>
<pre><code class="lang-python">resnet18 = models.resnet18()
for param in resnet18.fc.parameters():
    print(param.requires_grad)

# 1
gen = (p.requires_grad_(False) for k,p in resnet18.fc._parameters.items())
next(gen)
next(gen)
# 2
list((p.requires_grad_(False) for k,p in resnet18.fc._parameters.items()))
# 3    
[p.requires_grad_(False) for k,p in resnet18.fc._parameters.items()]

for param in resnet18.fc.parameters():
    print(param.requires_grad)
</code></pre>

Shisho_Sama: <p>Thanks a quintilion times sir <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
God bless you and have a fantastic weekend</p>