Changing `requires_grad` during training

MinkyuChoi: <p>I understand that the tensors with <code>requires_grad=False</code> do not calculate gradients. I wonder if I can change the <code>requires_grad</code> flag during training so as to train/freeze parts of network during training. I want to train some parameters if some conditions are met and freeze them if conditions are not matched. An example code is like below. When I change this flag, the optimizer would reflect it automatically? or do I have to do something to let the optimizer know about it?</p>
<pre><code>....
optimizer = optim.Adam(model.parameters(), lr = lr)
....
for epoch in range(total_epochs):
    .... training goes on ....

    if certain_condition:
        certain_parameter.requires_grad=False 
        # 'certain_parameter' is model parameter in model.parameters(). 
    else:
        certain_parameter.requires_grad=True

    .... training goes on ....</code></pre>

albanD: <p>Hi,</p>
<p>This will work to freeze the gradients to 0.<br>
Be careful though that many optimizer will still update the weights even for a gradient of 0! If you have regularization or momentum for example.</p>

yifita: <p>thanks! I had the same question.<br>
what do you suggest in case one wants to <em>completely</em> freeze the parameters?</p>

albanD: <p>The best way is not to give them to the optimizer in the first place I think.<br>
Or if you want to update them sometimes, have two optimizers, one that you step() at every iteration and one that you step() only when you want to update these special elements.</p>

DoubtWang: <p>However, Is there an easier way When encountering the following situation?<br>
model has three parameters (parameter_1, parameter_2  and parameter_3),<br>
first, model need update all parameters.<br>
During the training process, parameter_1 and parameter_2 will be updated sometimes, sometimes parameter_1 and parameter_3.<br>
So, we need three optimizers:<br>
optimizer_1—&gt;parameter_1, parameter_2  and parameter_3<br>
optimizer_2—&gt;parameter_1 and parameter_2<br>
optimizer_3—&gt;parameter_1 and parameter_3<br>
But how to deal with when there are more parameters ?<br>
I hope my expression is clear.</p>

albanD: <p>Hi,</p>
<p>You don’t want to have parameters shared in multiple optimizers as internal states kept for example by Adam or momentum will be wrong.<br>
You can create one optimizer for each set of paremeters and .step() all the ones that need to be updated at that iterations.</p>

DoubtWang: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="35694">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>You can create one optimizer for each set of paremeters and .step() all the ones that need to be updated at that iterations.</p>
</blockquote>
</aside>
<p>I know your means.<br>
However,<br>
the number of optimizers will increase when the set of paremeters is too large, which will cause some complexity.</p>

baromri: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="35694">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>This will work to freeze the gradients to 0.<br>
Be careful though that many optimizer will still update the weights even for a gradient of 0! If you have regularization or momentum for example.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/alband">@albanD</a>, I add the same issue and when setting <code>param.grad = None</code> it force the optimizer to skip this param update at step. Thus, also optimizers with momentum will keep those layers params fixed.</p>