How the pytorch freeze network in some layers, only the rest of the training?

earnp: <p>How the pytorch freeze network in some layers, only the rest of the training?</p>

L0SG: <p>Each parameters of the model have requires_grad flag:<br>
<a href="http://pytorch.org/docs/master/notes/autograd.html" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/master/notes/autograd.html</a></p>
<p>For resnet example in the doc, this loop will freeze all layers</p>
<pre><code>for param in model.parameters():
    param.requires_grad = False
</code></pre>
<p>For partially unfreezing some of the last layers, we can identify parameters we want to unfreeze in this loop. setting the flag to True will suffice.</p>

SpandanMadan: <p>I faced this just a few days ago, so I’m sure this code should be up to date. Here’s my answer for Resnet, but this answer can be used for literally any model.</p>
<p>The basic idea is that all models have a function model.children() which returns it’s layers. Within each layer, there are parameters (or weights), which can be obtained using .param() on any children (i.e. layer). Now, every parameter has an attribute called requires_grad which is by default True. True means it will be backpropagrated and hence to freeze a layer you need to set requires_grad to False for all parameters of a layer. This can be done like this -</p>
<pre><code>model_ft = models.resnet50(pretrained=True)
ct = 0
for child in model_ft.children():
ct += 1
if ct &lt; 7:
    for param in child.parameters():
        param.requires_grad = False
</code></pre>
<p>This freezes layers 1-6 in the total 10 layers of Resnet50. Hope this helps!</p>

James_Chen: <p>I am wondering whether to set .eval() for those frozen layers since it may still update its running mean and running var.</p>

SpandanMadan: <p>Try this. Reduce your learning rate drastically. Try viewing your gradients wrt input and see if there’s any place where they are blowing up(going to inf). If yes,see why this happens.</p>

varghese_alex: <p>Hi Spandan;</p>
<p>I try to replicate your code on Resnet 18. Kind of completed the code. My aim was to freeze all layers in the network except the classification layer and the layer/block preceding it. Could you please let me know your thoughts if this is right</p>
<p>import torch<br>
import torchvision</p>
<p>model = torchvision.models.resnet18(pretrained=True)</p>
<p>lt=8<br>
cntr=0</p>
<p>for child in model.children():<br>
cntr+=1</p>
<pre><code>if cntr &lt; lt:
	print child
	for param in child.parameters():
		param.requires_grad = False
</code></pre>
<p>num_ftrs = model.fc.in_features<br>
model.fc = nn.Linear(num_ftrs,2)</p>
<p>criterion = nn.CrossEntropyLoss()</p>
<p>optimizer_ft = optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=0.001, momentum=0.9)</p>

vmirly1: <p>Setting <code>.requires_grad = False</code> should work for convolution and FC layers. But how about networks that have instanceNormalization? Is setting <code>.requires_grad = False</code> enough for normalization layers too?</p>

Brando_Miranda: <p>does this work even if the network has already been trained? Say I have loaded a pre-trained net on X and I want to freeze layer Y (say 2nd layer to make the example concrete). How do I do that exactly?</p>

L0SG: <p><a href="https://gist.github.com/L0SG/2f6d81e4ad119c4f798ab81fa8d62d3f" rel="nofollow noopener">This snippet</a> may clarify how to do it.</p>
<p>Set requires_grad to false you want to freeze:</p>
<pre><code class="lang-auto"># we want to freeze the fc2 layer
net.fc2.weight.requires_grad = False
net.fc2.bias.requires_grad = False
</code></pre>
<p>Then set the optimizer like the following:</p>
<pre><code class="lang-auto">optimizer = optim.SGD(filter(lambda p: p.requires_grad, net.parameters()), lr=0.1)
</code></pre>
<p>Alternatively, you can only add the parameters you want to train to the optimizer:<br>
<a href="https://discuss.pytorch.org/t/to-require-grad-or-to-not-require-grad/5726">https://discuss.pytorch.org/t/to-require-grad-or-to-not-require-grad/5726</a></p>
<p>But I think the above method is more straight-forward.</p>

milani: <p><a class="mention" href="/u/l0sg">@L0SG</a>  What if we want to unfreeze those layers later in the code? For example, I want to fine tune heads first, then tune the other layers as well. We will need to instantiate a new optimizer, right? If so, does it affect the optimization?</p>

L0SG: <p><a href="http://pytorch.org/docs/stable/optim.html#torch.optim.Optimizer.add_param_group" rel="nofollow noopener">optimizer.add_param_group</a> would be what you want. This will unfreeze the previously frozen layer, by adding the parameters to new dict elements of param_groups (list) of the optimizer:</p>
<pre><code class="lang-auto"># let's unfreeze the fc2 layer this time for extra tuning
net.fc2.weight.requires_grad = True
net.fc2.bias.requires_grad = True

# add the unfrozen fc2 weight to the current optimizer
optimizer.add_param_group({'params': net.fc2.parameters()})

</code></pre>

Brando_Miranda: <p>if you only do:</p>
<pre><code># we want to freeze the fc2 layer
net.fc2.weight.requires_grad = False
net.fc2.bias.requires_grad = False
</code></pre>
<p>without the second part does it still work? i.e. without:</p>
<pre><code>optimizer = optim.SGD(filter(lambda p: p.requires_grad, net.parameters()), lr=0.1)</code></pre>

L0SG: <p>Yes, it does work when you add the parameters with <code>requires_grad=True</code> to the optimizer then setting to <code>False</code> after. You can also find out yourself by commenting out</p>
<pre><code class="lang-auto">optimizer = optim.SGD(filter(lambda p: p.requires_grad, net.parameters()), lr=0.1)
</code></pre>
<p>In the snippet above, since the previous optimizer contains all parameters including the fc2 with the changed <code>requires_grad</code> flag.</p>
<p>Note that the above snippet assumed a common “train =&gt; save =&gt; load =&gt; freeze parts” scenario.</p>

cysmith: <p>Snippet for recursively freezing a portion of your graph.</p>
<pre><code class="lang-auto">def dfs_freeze(model):
    for name, child in model.named_children():
        for param in child.parameters():
            param.requires_grad = False
        dfs_freeze(child)
</code></pre>
<p>Then:</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=opt.lr, amsgrad=True)
</code></pre>

daisy: <p><a class="mention" href="/u/l0sg">@L0SG</a>, When I unfreeze the previously frozen layers and also want to change the learning rate and the Momentum, but the parameters of these layers don’t have ‘momentum_bufer’ in optimizer’s state, do you have any suggestions to this issue?</p>

herleeyandi: <p>Hello I am still confuse about freeze the weight in Pytorch looks like very hard to do. Suppose I want to make a loss function which filtering the loss using the initialized kernel. I am using <code>nn.conv2D</code> to do my job but I don’t want the weight being updated(freeze). The loss function basically the simple network let said the A network is the main network that will be updated, and B Network which is the network for computing the loss function. For this task I enabled the grad for A and disable for B. During the training my program will taking that loss from B, then backpropagate into the main network A (where the weight should be update). However I always ended with this.<br>
<code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code><br>
Anyone have the better way to do this?, can I just use <code>modelB.eval()</code> ? or <code>modelB.Train(False)</code>?</p>

lugiavn: <p>You can try setting the learning rate for those to 0. Optimizer support different learning rate for each param groups, it can be adjusted online, look that up.<br>
Note that layers (such that BatchNorm) do the learning in forward phase, so you have to set them to eval() too</p>

rohun: <p>If we set the requires_grad to false for a particular layer, do we have to leave it out of the optimizer?<br>
Such as, this -&gt;</p>
<pre><code class="lang-auto">optimizer = optim.SGD(filter(lambda p: p.requires_grad, net.parameters()), lr=0.1)
</code></pre>
<p>Is the benefit in the speedup? Or is it wrong if one does not filter it out of the optimizer?<br>
Thanks</p>

L0SG: <p>The error from adding <em>already frozen</em> parameters to the optimizer is an <a href="https://github.com/pytorch/pytorch/issues/1266" rel="nofollow noopener">explicit design choice to avoid silly mistakes</a> (one could mistakenly add the frozen one and assume that the parameter is being trained). But manually freezing parameters <em>after</em> declaring the optimizer works as intended without re-defining the optimizer. <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088/14?u=l0sg">(my own replay above in this thread)</a></p>

rohun: <p>Thanks for the clarification!<br>
As a follow-up to my point regarding the speed up - I am not observing a speedup when I freeze the initial 80% of the network. I expected the training to be faster, since it only has to update 20% and to be lighter, since it only has to store the information to execute a backward for 20% of the network.<br>
Is there a speedup expected in this scenario?</p>