Finetuning Torchvision Models - Tutorial

Etienne-Meunier: <p>Hi, I had a question regarding the official tutorial on <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="nofollow noopener">Finetuning Torchvision Models</a>, in that tutorial they set all the parameters of the network except the new ones ( from the new classifier layer ) to <code>requires_grad = False</code> and then they use this code to build the Optimizer :</p>
<p>`params_to_update = model_ft.parameters()<br>
print(“Params to learn:”)<br>
if feature_extract:<br>
params_to_update = []<br>
for name,param in model_ft.named_parameters():<br>
if param.requires_grad == True:<br>
params_to_update.append(param)<br>
print("\t",name)<br>
else:<br>
for name,param in model_ft.named_parameters():<br>
if param.requires_grad == True:<br>
print("\t",name)</p>
<p>optimizer_ft = optim.SGD(params_to_update, lr=0.001, momentum=0.9)<br>
`</p>
<p>( The else here is for finetuning ). But settings <code>requires_grad = False</code> is not enough to not train the parameters even if we pass them to the Optimizer? In my understanding of the autograd module, both cases here are doing the same thing ( training only the parameters <code>with requires_grad = True</code> ) even if they don’t have the same parameters in the Optimizer. Am I missing something ?</p>
<p>Thank you very much for your help !</p>

ptrblck: <p>If your optimizer uses weight decay, the parameters might be updated even without having a gradient, which is why they are also filtered out of the parameter list passed to the optimizer.</p>

Etienne-Meunier: <p>Oh right, super interesting, thank you very much for this answer !</p>

Etienne-Meunier: <p>On a second thinking, in this case using weight decay + requires_grad = False and still having the parameters in the optimizer ( what they did in the fine tuning case ) will imply that the of the “frozen” part of the network are going to get reduced at every epoch even if they are not changed by backprop, would it be not good for the network ?</p>

ptrblck: <p>I’m not sure, if weight decay on frozen parameters is beneficial or not and you would have to run some experiments on it.<br>
Just for the sake of completeness, weight decay won’t be applied if the <code>.grad</code> attribute is <code>None</code> as seen <a href="https://github.com/pytorch/pytorch/blob/bf4a28175d773e2d51c49eecf1817528418c0f54/torch/optim/sgd.py#L89" rel="nofollow noopener">here</a>. If you set the <code>requires_grad</code> attribute to <code>False</code> from the beginning, these parameters won’t be updated, as their <code>.grad</code> will never be set to a valid value.</p>
<p>However, if you’ve applied some training to these parameters before setting <code>requires_grad=False</code> and only zero out their gradients, weight decay will manipulate these in the next optimizer steps.</p>

antoon: <p>Excuse If I’m bumping this old topic. Actually fine tuning only works for me if I set requires_grad to False. If I set it to true, accuracy will always be low. Maybe should I import the weights of the imported network in some way to make it work?</p>

ptrblck: <aside class="quote no-group" data-username="antoon" data-post="6" data-topic="55475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/977dab/48.png" class="avatar"> antoon:</div>
<blockquote>
<p>Maybe should I import the weights of the imported network in some way to make it work?</p>
</blockquote>
</aside>
<p>You should use the pretrained model or load a <code>state_dict</code> for an already trained model, if you want to fine tune it.<br>
Which parameters are you freezing and is the training as well as validation accuracy low or just one of them?</p>

antoon: <p>Loss is high, validation accuracy low( example: Epoch 50: Validation Loss =4.9031, VA=0.0100) . I use the pretrained model like this inside the init function of my model:</p>
<pre><code class="lang-auto">        alexnet = models.alexnet(pretrained=True)
        self.features = alexnet.features
        model_dict=self.state_dict()
        pretrained_dict=alexnet.state_dict()
        pretrained_dict= {k: v for k, v in pretrained_dict.items() if k in model_dict}
        model_dict.update(pretrained_dict)
        self.load_state_dict(model_dict)
</code></pre>