Trying to understand Optimizer and relation to requires_grad

username: ImgPrcSng
cooked: <p>Consider the case where I have constructed the optimizer as follows</p>
<pre><code class="lang-python">model = load_some_pretrained_model()
optimizer = torch.optim.some_optim(model.parameters(), arg1, arg2)

# Later I Set required_grad of some params as False.
for param in model.parameters():
    if param in arbit_list:
        param.requires_grad = False

# ... Some more code for doing training
optimizer.step()
</code></pre>
<p>I wanted to confirm if the changes to requires_grad is reflected in the optimizer.step() as well. I do understand the graph is reconstructed every time the code is run, thus it makes sense for the optimizer to check before calling a step on the parameter. But just in case there was some optimization to not check it every time …</p>

username: albanD
cooked: <p>Hi,</p>
<p>What happens is that if you set <code>requires_grad=False</code>, the gradient will not be computed and thus <code>param.grad=None</code>.<br>
As you can see for example in the SGD optimizer <a href="https://github.com/pytorch/pytorch/blob/c27aaf67cd133c501c34d91cf77855fdae4a58ae/torch/optim/sgd.py#L81-L82">here</a> the check is done every time.</p>

username: ozancaglayan
cooked: <p>This does not seem to work if the <code>requires_grad</code> is turned off after at least one forward/backward/step since in that case <code>p.grad</code> is not <code>None</code> anymore but 0. Or am I misunderstanding something?</p>

username: hari_krishnan1
cooked: <p>Can someone clarify this? I have same doubt as above</p>
<pre><code class="lang-auto">This does not seem to work if the requires_grad is turned off after at least one forward/backward/step since in that case p.grad is not None anymore but 0. Or am I misunderstanding something?
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>Yes this is kind of a grey area for the optimizers: they handle differently a gradient of <code>None</code> vs a Tensor full of 0s.<br>
The first will perform no update at all.<br>
The second will perform the update with a gradient of <code>0</code>, which can be non-zero if there is weight decay for example.</p>

username: hari_krishnan1
cooked: <p>Can you change the behaviour to check for <code>if not p.requires_grad:</code> ?</p>
<p>Consider the scenario where you are training a GAN and you have a generator and discriminator.</p>
<p>Now the front of the discriminator is a pretrained CNN (like <code>resnet18(pretrained=True)</code> ) and we don’t want to train this. We only want to train the final Linear layer of the discriminator.</p>
<p>However setting <code>cnn.requires_grad=False</code> will not work since generator output (fake images) requires gradient. The gradient has to flow through the frozen CNN back into the generator in order to train it.</p>
<p>Right now for this scenario you need to manually select parameters from the discriminator and pass it to the optimizer to avoid changing the weights of the pretrained CNN.</p>

username: albanD
cooked: <p>If you set the requires_grad field of these parameters to False before doing anything with them. Then their .grad field will be None. And they won’t be updated.</p>
<blockquote>
<p>Can you change the behaviour to check for <code>if not p.requires_grad:</code> ?</p>
</blockquote>
<p>This would be a major breaking change. I don’t think we can do that <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>