Freezing all parameters in NN

Mari: <p>I want to freeze all parameters in network ks as a sub class in Net2_kc, but when i print parameters it changes everytime. Can someone tell me what the problem is?</p>
<pre><code class="lang-auto">#transfer learning
net2_kc.ks.load_state_dict(torch.load(path+"heat_conductivity_from_fluent_solution_one_layer_200"+".pt"))


class Net2_kc(nn.Module):  

        #The __init__ function stack the layers of the 
        #network Sequentially 
        def __init__(self):
            super(Net2_kc, self).__init__()
            self.ks = ks()
            
           
            self.final = nn.Sequential(
                nn.Linear(1,1),  #Doing just 2,1 gives very bad results!!!!!!!!!
            )


        def forward(self,x): 
            ks = self.ks(x)
            output2 = self.final(ks)
            return  output2
  
for parameter in Net2_kc().ks.main.parameters():
         parameter.requires_grad = False            
         print("param", parameter) 

optimizer_kc = optim.Adam(filter(lambda p: p.requires_grad,         net2_kc.parameters()), lr=learning_rate,betas = (0.9,0.99),eps = 10**-15)
</code></pre>

eqy: <p>It is a bit hard to see what is going on without the full model definition, but it looks like in this line <code>for parameter in Net2_kc().ks.main.parameters():</code> a separate, new network is being created with <code>Net2_kc()</code> so that you may not actually be changing the <code>requires_grad</code> field of the model that you are using.</p>
<p>Additionally, to simplify things, you could also try just wrapping the relevant part of your model with <code>torch.no_grad</code>: <a href="https://pytorch.org/docs/stable/generated/torch.no_grad.html" class="inline-onebox" rel="noopener nofollow ugc">no_grad — PyTorch 1.11.0 documentation</a></p>

Mari: <p>Thanks for the answer.<br>
yes right, network ks is defined as a seperate network:</p>
<pre><code class="lang-auto">class ks(nn.Module):

        #The __init__ function stack the layers of the 
        #network Sequentially 
        def __init__(self):
            super(ks, self).__init__()
            self.main = nn.Sequential(
                nn.Linear(input_n,h_nk),
                Swish(),
                nn.Linear(h_nk,h_nk),
                Swish(),

                nn.Linear(h_nk,1),
            )
        
        def forward(self,x):
            output = self.main(x)
            
            return  output  
</code></pre>
<p>i did not use torch.no_grad, how can i use it if i want to freeze all parameters in network ks?</p>

eqy: <p>For example, if you don’t want <code>main</code>'s parameters to be affected, you could try changing that part to</p>
<pre><code class="lang-auto">with torch.no_grad():
    output = self.main(x)
</code></pre>
<p>But I’m not entirely familiar with your use case so the manual <code>requires_grad</code> setting should also work if the bug is fixed.</p>

Mari: <p>do i need to define this as a part of forward function?</p>
<pre><code class="lang-auto">def forward(self,x):
       with torch.no_grad():
          output = self.main(x)

    return  output 
</code></pre>

eqy: <p>Yes, something like that, but it depends on which part of the model you want to freeze; if it is part of the sequential module then your original approach might be cleaner.</p>

Mari: <p>I still could not solve this issue, any more suggestion?</p>

eqy: <p>Have you retried the original approach with the fix to avoid defining a new model?</p>
<p>I would also check what parameters are passed to the optimizer.</p>