How to freeze the network in some layers, in fully connected network?

Mari: <p>Hi,<br>
How can I freeze network in initial layers except two last layers? Do I need to implement filter in optimizer for those fixed layers? if yes how?</p>
<p>class Net_k(nn.Module):</p>
<pre><code>    #The __init__ function stack the layers of the 
    #network Sequentially 
    def __init__(self):
        super(Net_k, self).__init__()
        self.main = nn.Sequential(
            nn.Linear(input_n,h_nk),
            Swish(),
            nn.Linear(h_nk,h_nk),
            Swish(),
            nn.Linear(h_nk,h_nk),
            Swish(),
            nn.Linear(h_nk,h_nk),
            Swish(),
            nn.Linear(h_nk,h_nk),
            Swish(),
            nn.Linear(h_nk,h_nk),
            Swish(),
            nn.Linear(h_nk,h_nk),
            Swish(),
            nn.Linear(h_nk,h_nk),
            Swish(), 
            nn.Linear(h_nk,h_nk),
            Swish(),
            nn.Linear(h_nk,h_nk),
            Swish(),
            nn.Linear(h_nk,1),
            Swish(),
            
        )
   
    def forward(self,x):
        output = self.main(x)
        return  output
</code></pre>
<p>optimizer_k = optim.Adam(net_k.parameters(), lr=learning_rate, betas = (0.9,0.99),eps = 10**-15)</p>

ptrblck: <aside class="quote no-group" data-username="Mari" data-post="1" data-topic="130793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/51bf81/48.png" class="avatar"> Mari:</div>
<blockquote>
<p>How can I freeze network in initial layers except two last layers?</p>
</blockquote>
</aside>
<p>You can set the <code>.requires_grad</code> attribute of the parameters of the first layers to <code>False</code> to freeze them.</p>
<aside class="quote no-group" data-username="Mari" data-post="1" data-topic="130793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/51bf81/48.png" class="avatar"> Mari:</div>
<blockquote>
<p>Do I need to implement filter in optimizer for those fixed layers? if yes how?</p>
</blockquote>
</aside>
<p>You don’t necessarily need to filter out frozen parameters, as they won’t be updated, but could do it via e.g <code>torch.optim.SG(filter(lambda p: p.requires_grad, model.parameters()), lr=...)</code>.</p>

Mari: <p>Thanks ptrblck.<br>
Can you please provide me an example of how I can use requires_grad to freeze a layer?</p>

ptrblck: <p>Sure, this code snippet freezes the parameters of the first layer:</p>
<pre><code class="lang-python">model = models.resnet18()
# freeze parameters of first layer
for param in model.conv1.parameters():
    param.requires_grad = False
</code></pre>

Mari: <p>If I want to freeze all layers except the last one is this correct to write:</p>
<p>for parameter in Net_k.parameters():<br>
parameter.requires_grad = False<br>
for parameter in Net_k[-1].parameters():<br>
parameter.requires_grad = True</p>

ptrblck: <p>I would, if <code>Net_k[-1]</code> returns the last layer, e.g. if it’s defined as an <code>nn.Sequential</code> module. However, if you are using a custom <code>nn.Module</code>, you would need to access the last layer directly via <code>model.last_layer_name.parameters()</code>,</p>

Mari: <p>The network I defined is an nn.Sequential module like above. Then I think this is correct.</p>