What's the expected behaviour of calling optimizer.step() AFTER optimizer.zero_grad()

username: Valiox
cooked: <p>I realized I had a bug in my model with multiple optimizers, because some optimizers had overlapping param groups (I assume that is never supposed to happen).</p>
<p>I’ve been debugging things around and I noticed this weird (somewhat unrelated) behaviour:</p>
<p>Here I take one parameter from one optimizer:</p>
<pre><code class="lang-auto">In[25]: optim.param_groups[0]["params"][0]
Out[25]: 
Parameter containing:
tensor([[ 0.0239,  0.0519,  0.0257,  ..., -0.0024,  0.0691, -0.0380]],
       device='cuda:0', requires_grad=True)
</code></pre>
<p>I zero out the gradient and step (again, not something you’re supposed to do, but that’s what was happening in my project for a while so I’d like to understand the effect it had):</p>
<pre><code class="lang-auto">In[26]: optim.zero_grad()
In[27]: optim.param_groups[0]["params"][0].grad
Out[27]: tensor([[0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')
In[28]: optim.step()
</code></pre>
<p>Even though the gradient was 0, the tensor values changed a bit:</p>
<pre><code class="lang-auto">In[29]: optim.param_groups[0]["params"][0]
Out[29]: 
Parameter containing:
tensor([[ 0.0238,  0.0518,  0.0258,  ..., -0.0024,  0.0691, -0.0380]],
       device='cuda:0', requires_grad=True)
</code></pre>
<p>And the gradient is no longer 0, but very small values:</p>
<pre><code class="lang-auto">In[30]: optim.param_groups[0]["params"][0].grad
Out[30]: 
tensor([[ 2.3881e-08,  5.1892e-08,  2.5746e-08,  ..., -2.3582e-09,
          6.9058e-08, -3.8007e-08]], device='cuda:0')
</code></pre>
<p>I this normal behaviour? Why were the tensor values changed at all?</p>

username: KFrank
cooked: <p>Hello Valiox!</p>
<aside class="quote no-group quote-modified" data-username="Valiox" data-post="1" data-topic="74359" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/valiox/48/19212_2.png" class="avatar"> Valiox:</div>
<blockquote>
<p>I zero out the gradient and step<br>
…<br>
Even though the gradient was 0, the tensor values changed a bit:</p>
</blockquote>
</aside>
<p>One possibility – this would depend on the details of what you’re<br>
doing – is that some optimizers have memory, e.g., <code>SGD</code> with<br>
momentum or <code>Adam</code>.  Therefore, even though the <em>current</em><br>
gradient is zero, the optimizer might be remembering previous<br>
steps with a non-zero gradient, and therefore still be changing<br>
the parameters.</p>
<aside class="quote no-group">
<blockquote>
<p>And the gradient is no longer 0, but very small values:</p>
</blockquote>
</aside>
<p>I don’t think that momentum use in an optimizer changes the<br>
gradient.  (Hypothetically, it could be implemented that way,<br>
but I don’t think it is.)</p>
<p>However, at the risk of being completely wrong, I seem to recall<br>
that <em>weight decay</em> might be implemented by actually changing<br>
the gradient.  So turning on weight decay in your optimizer could<br>
explain both the changing parameters and the non-zero gradient.<br>
(!!! Possible misinformation alert !!!)</p>
<aside class="quote no-group">
<blockquote>
<p>I this normal behaviour? Why were the tensor values changed at all?</p>
</blockquote>
</aside>
<p>I could see your parameter values changing (under the right<br>
conditions) if your optimizer uses momentum.  My speculation<br>
about the non-zero gradient is more suspect.</p>
<p>Would it be possible to re-run your test using plain-vanilla<br>
<code>torch.optim.SGD</code>, making sure that <code>momentum</code> and<br>
<code>weight_decay</code> are turned off?  (I believe that they are off<br>
by default.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Valiox
cooked: <p>Hello KFrank,</p>
<p>You are absolutely correct. My optimizer was Adam with weight decay, and using SGD without weight decay gives the “expected” behaviour:</p>
<pre><code class="lang-auto">#optim = Adam(params, lr, weight_decay=1e-6)
optim = SGD(params, lr)
...

In[2]: optim.param_groups[0]["params"][0]
Out[2]: 
Parameter containing:
tensor([[ 0.0913,  0.0789,  0.0728,  ..., -0.0352,  0.0444, -0.0342],
        [ 0.0287,  0.0179,  0.0200,  ..., -0.0781, -0.0859, -0.0117],
        [-0.0233,  0.0586, -0.0877,  ..., -0.0872,  0.0094, -0.0664],
        ...,
        [-0.0286,  0.0854, -0.0020,  ..., -0.0824,  0.0082,  0.0802],
        [-0.0504, -0.0254,  0.0182,  ..., -0.0826,  0.0746,  0.0196],
        [-0.0585, -0.0425, -0.0545,  ...,  0.0924,  0.0455, -0.0788]],
       device='cuda:0', requires_grad=True)
In[3]: optim.zero_grad()
In[4]: optim.param_groups[0]["params"][0].grad
Out[4]: 
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')
In[5]: optim.step()
In[6]: optim.param_groups[0]["params"][0]
Out[6]: 
Parameter containing:
tensor([[ 0.0913,  0.0789,  0.0728,  ..., -0.0352,  0.0444, -0.0342],
        [ 0.0287,  0.0179,  0.0200,  ..., -0.0781, -0.0859, -0.0117],
        [-0.0233,  0.0586, -0.0877,  ..., -0.0872,  0.0094, -0.0664],
        ...,
        [-0.0286,  0.0854, -0.0020,  ..., -0.0824,  0.0082,  0.0802],
        [-0.0504, -0.0254,  0.0182,  ..., -0.0826,  0.0746,  0.0196],
        [-0.0585, -0.0425, -0.0545,  ...,  0.0924,  0.0455, -0.0788]],
       device='cuda:0', requires_grad=True)
In[7]: optim.param_groups[0]["params"][0].grad
Out[7]: 
tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.]], device='cuda:0')
</code></pre>
<p>Thank you!</p>