Correct way to freeze layers

username: nabihach
cooked: <p>I have some confusion regarding the correct way to freeze layers.<br>
Suppose I have the following NN: layer1, layer2, layer3<br>
I want to freeze the weights of layer2, and only update layer1 and layer3.<br>
Based on other threads, I am aware of the following ways of achieving this goal.</p>
<p>Method 1:</p>
<ul>
<li>optim = {layer1, layer3}</li>
<li>compute loss</li>
<li>loss.backward()</li>
<li>optim.step()</li>
</ul>
<p>Method 2:</p>
<ul>
<li>layer2_requires_grad=False</li>
<li>optim = {all layers with requires_grad = True}</li>
<li>compute loss</li>
<li>loss.backward()</li>
<li>optim.step()</li>
</ul>
<p>Method 3:</p>
<ul>
<li>optim = {layer1, layer2, layer3}</li>
<li>layer2_old_weights = layer2.weight (this saves layer2 weights to a variable)</li>
<li>compute loss</li>
<li>loss.backward()</li>
<li>optim.step()</li>
<li>layer2.weight = layer2_old_weights (this sets layer2 weights to old weights)</li>
</ul>
<p>Method 4:</p>
<ul>
<li>optim = {layer1, layer2, layer3}</li>
<li>compute loss</li>
<li>loss.backward()</li>
<li>set layer2 gradients to 0</li>
<li>optim.step()</li>
</ul>
<p>My questions:</p>
<ol>
<li>Should we get different results for each method?</li>
<li>Is any of these methods wrong?</li>
<li>Is there a preferred method?</li>
</ol>

username: kelam_goutam
cooked: <p>I would like to do it the following way -</p>
<pre><code class="lang-auto"># we want to freeze the fc2 layer this time: only train fc1 and fc3
net.fc2.weight.requires_grad = False
net.fc2.bias.requires_grad = False

# passing only those parameters that explicitly requires grad
optimizer = optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), lr=0.1)

# then do the normal execution of loss calculation and backward propagation

#  unfreezing the fc2 layer for extra tuning if needed
net.fc2.weight.requires_grad = True
net.fc2.bias.requires_grad = True

# add the unfrozen fc2 weight to the current optimizer
optimizer.add_param_group({'params': net.fc2.parameters()})
</code></pre>

username: nabihach
cooked: <p><a class="mention" href="/u/kelam_goutam">@kelam_goutam</a> I believe your way is the same as Method 2 described above. Can you please explain why you prefer this over others?</p>

username: kelam_goutam
cooked: <p>I feel method 3 and 4 are waste of computation. Why to compute gradients for the layers which you dont want to update. I think method 1 would be ideal as in method 2 we need to explicitly mark the layers parameters as False and then its our responsibility to mark them as True if we need to update those layers. However I preferred the Method 2 thinking that using this way it was easier to freeze weights of any layer in case of a huge network as optimizer will automatically gather which all layers to update.</p>
<p>warm regards,<br>
Goutam Kelam</p>

username: enolerobotti
cooked: <p>If you can change the contents of forward method of a layer, you can use self.eval() and with torch.no_grad():</p>
<pre><code class="lang-auto">if self.frozen_pretrained_weights:
  self.eval()
    with torch.no_grad():
      output = self.encoders(batch)
  else:
    output = self.encoders(batch)
</code></pre>