Is it necessary to disable gradient computation in fine-tuning?

mrityu: <p>Hey.</p>
<p>Generally, when we fine-tune a classifier by keeping a pre-trained model as a feature-extractor only, we set the <code>requires_grad = False</code>  for the pre-trained block and only train the newly added FC layer.<br>
For eg., see the code snippet below:-</p>
<pre><code class="lang-auto"># Setting up the model
# Note that the parameters of imported models are set to requires_grad=True by default

res_mod = models.resnet34(pretrained=True)
for param in res_mod.parameters():
    param.requires_grad = False

num_ftrs = res_mod.fc.in_features
res_mod.fc = nn.Linear(num_ftrs, 2)

res_mod = res_mod.to(device)
criterion = nn.CrossEntropyLoss()

# Here's another change: instead of all parameters being optimized
# only the params of the final layers are being optimized

optimizer_ft = optim.SGD(res_mod.fc.parameters(), lr=0.001, momentum=0.9)

exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
</code></pre>
<p>My question is, is it necessary to set <code>requires_grad=False</code>  in fine-tuning because we are anyways specifying the parameters in <code>optimizer_ft</code> which needs to update i.e. the last FC layer params?</p>
<p>I know it will be a computational disaster and shouldn’t be done this way, but I am just asking out of curiosity.</p>

OrielBanne: <p>are you trying to freeze a pre trained net and train an FC layer for, say, classification? so the net gives you features it learned elsewhere and FC learns the classification task?<br>
fine tuning includes  re-train to a model that is pre-trained, you should not freeze a model if you would like to fine tune it</p>

mrityu: <p>I am asking if its “necessary” to do:-</p>
<pre><code class="lang-auto">for param in res_mod.parameters():
    param.requires_grad = False
</code></pre>
<p>if I want to “freeze” a part of a model. According to what I know, if we do this:-</p>
<pre><code class="lang-auto">optimizer_ft = optim.SGD(res_mod.fc.parameters(), lr=0.001, momentum=0.9)
</code></pre>
<p>then also we are only updating the weights of the ‘FC’ layer(or any other layer for that matter) and not the rest of the model. But yes, I agree it will be computationally expensive. I am asking this just to check if my understanding of how the optimizers in PyTorch work is correct or not.</p>

OrielBanne: <p>yes - freezing any part of the model requires freezing the grads. otherwise, every optimizer stepyou make will change all the weights</p>

KFrank: <p>Hi Mrityunjay!</p>
<aside class="quote no-group" data-username="mrityu" data-post="3" data-topic="134218" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrityu/48/33916_2.png" class="avatar"> mrityu:</div>
<blockquote>
<p>we are only updating the weights of the ‘FC’ layer(or any other layer for that matter) and not the rest of the model.</p>
</blockquote>
</aside>
<p>Yes, you are correct.  When you call <code>optimizer_ft.step()</code>, the<br>
optimizer will only update the weights that were specified in that<br>
optimizer.</p>
<p>Even, if other weights – not in the optimizer – have had their<br>
gradients computed, those other weights will not be updated.</p>
<aside class="quote no-group">
<blockquote>
<p>But yes, I agree it will be computationally expensive.</p>
</blockquote>
</aside>
<p>Yes.  Just to confirm, computing gradients for other weights not<br>
in the optimizer will add (unnecessary) computational cost.  You<br>
will pay the cost – mostly in memory – of building the “computation<br>
graph” for those weights during the forward pass, and pay the<br>
cost – mostly computational – of computing their gradients during<br>
the backward pass.  But then you throw those gradients away,<br>
wasting that effort.  (But doing so won’t break anything, except<br>
maybe cause you to run out of memory.)</p>
<aside class="quote no-group" data-username="OrielBanne" data-post="4" data-topic="134218" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/orielbanne/48/23185_2.png" class="avatar"> OrielBanne:</div>
<blockquote>
<p>yes - freezing any part of the model requires freezing the grads. otherwise, every optimizer stepyou make will change all the weights</p>
</blockquote>
</aside>
<p>Hi Oriel!</p>
<p>No, if by “freezing the grads” you mean setting <code>requires_grad = False</code><br>
for them, this is not true.  Even if you compute the gradients for various<br>
weights (for which <code>requires_grad = True</code>), they will not be updated if<br>
they are not “part of” the optimizer on which you call <code>optimizer.step()</code>.</p>
<p>For example, it is perfectly possible to use one optimizer, <code>optA</code>, to<br>
optimize one part, “part A,” of a model and use another, <code>optB</code> to<br>
optimize another part, “part B.”  You could compute the gradients<br>
for all of your weights, both those in “part A” and “part B” of your<br>
model, with a single backward pass.  But when you call <code>optA.step()</code><br>
you will only update the weights in “part A,” leaving “part B” unchanged,<br>
and only update the weights in “part B” when you call <code>optB.step()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

OrielBanne: <p>Thanks Frank, that’s awesome!</p>