Correct way of freezing layers

username: asura
cooked: <p>Hello,</p>
<p>I have a model <code>M</code> and I am cloning it <code>M.clone()</code><br>
Now, I want to freeze certain layers of <code>M.clone()</code>. When I set <code>requires_grad=False</code>, it results in this error:</p>
<p><code>RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().</code></p>
<p>How to freeze the layers of <code>M.clone()</code> in that case? I want to ensure that when I backpropagate using the loss computed on a batch using <code>M.clone()</code>, I compute the gradients of <code>M</code></p>
<p>A small script:</p>
<pre><code class="lang-python">model = ResNet()
optimizer = Adam(model.parameters())
cloned_model = model.clone() # .clone() is a custom method that creates a copy of the model
for p in cloned_model.features.parameters():
     p.require_grad = False
error = loss(cloned_model(data), labels)
error.backward()
optimizer.step()
</code></pre>
<p>Thank you!</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/asura">@asura</a><br>
Some small changes</p>
<pre><code class="lang-auto">model = ResNet()
optimizer = Adam(model.parameters())
cloned_model = model.clone()
for p in cloned_model.parameters():
     p.requires_grad = False
error = loss(cloned_model(data), labels)
error.backward()
optimizer.step()
</code></pre>

username: asura
cooked: <p>Thanks <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> ! However, by adding <code>cloned_model.features</code>, I want to specify that I don’t want to freeze all layers. Regarding my above question, can you suggest anything?</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/asura">@asura</a></p>
<p>That can also be done</p>
<ul>
<li>Resnet has a lot of layers. If you want to freeze specific internal resnet layers, then you will have to do it manually<br>
e.g.</li>
</ul>
<pre><code class="lang-auto">for p in cloned_model.layer1.parameters():
     p.requires_grad = False
</code></pre>
<ul>
<li>If you want the entire resent to be frozen and only allow the linear layer after resnet<br>
<a href="https://stackoverflow.com/questions/46610732/how-to-freeze-some-layers-when-fine-tune-resnet50" class="inline-onebox" rel="noopener nofollow ugc">neural network - how to freeze some layers when fine tune resnet50 - Stack Overflow</a>
</li>
</ul>

username: asura
cooked: <p>Thanks <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a>, Resnet() is just an example model. You can consider any model (say a CNN with 4 layers). My question lies here itself. Doing <code>cloned_model.layer1.parameters()</code> results in the RuntimeError I mentioned above.</p>

username: asura
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for your reply in the other thread.<br>
I had one more issue related to the loss problem. Can you please give some insights on this? Once again, thanks a lot!</p>

username: ptrblck
cooked: <p><code>.clone()</code> is a tensor method and is undefined for <code>nn.Module</code>s so your code should already fail in <code>model.clone()</code> unless it’s a custom method which isn’t posted here.</p>

username: asura
cooked: <p>My bad, it is indeed a custom method that creates a copy of the module whose parameters are created using the <code>torch.clone()</code> method.</p>

username: ptrblck
cooked: <p>Depending how this custom method is implemented, you might want to call <code>detach()</code> on the parameters additionally or just use <code>copy.deepcopy(model)</code>.</p>

username: asura
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>, however, if I use either of the methods, will I be able to compute gradients on the main model using <code>error.backward()</code>?</p>

username: ptrblck
cooked: <p>No, since you want to freeze the cloned model. If you want to keep the gradient history intact, don’t freeze the parameters of the cloned model:</p>
<pre><code class="lang-python">weight = nn.Parameter(torch.randn(2, 2))
weight_cloned = weight.clone()

y = weight_cloned * 2
y.mean().backward()
print(weight.grad)
&gt; tensor([[0.5000, 0.5000],
          [0.5000, 0.5000]])
</code></pre>

username: asura
cooked: <p>Got it. Thanks a lot for your help!</p>