Does optimizer skip update of parameters with requires_grad=False?

ado_sar: <p>Do all PyTorch optimizers ignore parameters with <code>requires_grad=False</code> even if these parameters are explicitly passed during the instantiation of optimizer?</p>
<p>From the source code of <a href="https://github.com/pytorch/pytorch/blob/main/torch/optim/optimizer.py" rel="noopener nofollow ugc"><code>optimizer.py</code></a>, it can be seen the <code>zero_grad</code> does not have any effect if <code>p.grad is not None</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">for p in group["params"]:
    if p.grad is not None:
        if set_to_none:
            p.grad = None
        else:
            if p.grad.grad_fn is not None:
                p.grad.detach_()
            else:
                p.grad.requires_grad_(False)
            if not foreach or p.grad.is_sparse:
                p.grad.zero_()
            else:
                assert per_device_and_dtype_grads is not None
                per_device_and_dtype_grads[p.grad.device][
                    p.grad.dtype
                ].append(p.grad)
</code></pre>
<p>Moreover, the <code>step</code> method (shown for <code>Adam</code> as example) doesn’t also have an effect if <code>p.grad is None</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">def step(self, closure=None):
    """Performs a single optimization step.

    Arguments:
        closure (callable, optional): A closure that reevaluates the model
            and returns the loss.
    """
    loss = None
    if closure is not None:
        loss = closure()


    for group in self.param_groups:
        for p in group['params']:
            if p.grad is None:
                continue
</code></pre>
<p>If that is the case for all optimizers, then setting <code>mymodel.param.requires_grad_(False)</code> would be enough for ensuring that the model parameters are not updated by the optimizer. Can someone from PyTorch dev team verify that this is the general behavior?</p>

ptrblck: <aside class="quote no-group" data-username="ado_sar" data-post="1" data-topic="216205">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ado_sar/48/59241_2.png" class="avatar"> ado_sar:</div>
<blockquote>
<p>If that is the case for all optimizers, then setting <code>mymodel.param.requires_grad_(False)</code> would be enough for ensuring that the model parameters are not updated by the optimizer.</p>
</blockquote>
</aside>
<p>No, it’s generally not enough since the <code>optimizer</code> does not check for <code>param.requires_grad</code> but for the actual <code>.grad</code> attribute.</p>
<p>Even if the parameter is frozen (via <code>requires_grad = False</code>) it can still have a valid <code>.grad</code> attribute (including zeros) and might be updated. Here is a simple example:</p>
<pre data-code-wrap="python"><code class="lang-python">lin = nn.Linear(1, 1, bias=False)
optimizer = torch.optim.Adam(lin.parameters(), lr=1.)
x = torch.randn(1, 1)

# update
print(lin.weight)
# Parameter containing:
# tensor([[-0.8696]], requires_grad=True)
out = lin(x)
out.mean().backward()
optimizer.step()
print(lin.weight)
# Parameter containing:
# tensor([[-1.8696]], requires_grad=True)
optimizer.zero_grad()

# freeze
lin.weight.requires_grad = False
out = lin(x)
# out.mean().backward() # would fail
print(lin.weight.grad)
# None
# manually set the .grad
lin.weight.grad = torch.full_like(lin.weight, 1000.)
optimizer.step()
print(lin.weight)
# Parameter containing:
# tensor([[-2.6148]])

# Adam has runnin internal stats and will also update the param with a zero gradient
lin.weight.grad = torch.zeros_like(lin.weight)
optimizer.step()
print(lin.weight)
# Parameter containing:
# tensor([[-3.1908]])

# setting the .grad to None will skip the update
optimizer.zero_grad(set_to_none=True) # set_to_none=True is the default
optimizer.step()
print(lin.weight)
# Parameter containing:
# tensor([[-3.1908]])
</code></pre>

ado_sar: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the answer and the examples! In my mind I had the “typical scenario” where the user doesn’t interact with the <code>.grad</code> attribute manually after it has been set <code>requires_grad=False</code> (e.g. it freezes part of the model). For example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch.nn import Linear
from torch.optim import Adam
from torch.utils.data import DataLoader

model = Linear(1, 1)
optimizer = Adam(model.parameters(), weight_decay=0.1)
x = torch.randn(10, 1)
loader = DataLoader(x, batch_size=2)

requires_grad = False
model.bias.requires_grad_(requires_grad)
print(f'\Requires grad: {requires_grad}\n')

print('Before training')
print(list(model.named_parameters()))

for batch in loader:
    optimizer.zero_grad()
    out = model(x).sum()
    out.backward()
    optimizer.step()

print('After training')
print(list(model.named_parameters()))
for name, p in model.named_parameters():
    print(name, p.grad)
</code></pre>
<p>I think the safest choice is to filter the parameters passed to the optimizer?</p>
<pre data-code-wrap="python"><code class="lang-python">optimizer = Adam(filter(lambda p: p.requires_grad, parameters()))
</code></pre>

ptrblck: <aside class="quote no-group" data-username="ado_sar" data-post="3" data-topic="216205">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ado_sar/48/59241_2.png" class="avatar"> ado_sar:</div>
<blockquote>
<p>In my mind I had the “typical scenario” where the user doesn’t interact with the <code>.grad</code> attribute manually</p>
</blockquote>
</aside>
<p>Yes, typical scenarios might work as you’ve described. I’m just pointing out limitations for potentially hard-to-debug issues.</p>
<aside class="quote no-group" data-username="ado_sar" data-post="3" data-topic="216205">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ado_sar/48/59241_2.png" class="avatar"> ado_sar:</div>
<blockquote>
<p>I think the safest choice is to filter the parameters passed to the optimizer?</p>
</blockquote>
</aside>
<p>Yes, I’m an advocate for an explicit usage of all APIs avoiding potential side-effects.</p>