Some questions about the Adam optimizer

Youlong_Ding: <p>I use the full batch gradient decent method with Adam optimizer, obtaining the following result<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e335e41dfd7f2130301a4de69b0a8a5976b9c29.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1e335e41dfd7f2130301a4de69b0a8a5976b9c29" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e335e41dfd7f2130301a4de69b0a8a5976b9c29_2_476x500.jpeg" alt="1" data-base62-sha1="4janyyN8iAT7n12TLoOkLr8oCCZ" width="476" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e335e41dfd7f2130301a4de69b0a8a5976b9c29_2_476x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e335e41dfd7f2130301a4de69b0a8a5976b9c29.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e335e41dfd7f2130301a4de69b0a8a5976b9c29.jpeg 2x" data-dominant-color="E9EFF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">484×508 70.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
as seen in the picture, the direction of the first update is not along the direction of the gradient, but during the first update, there are no momentum acculumated, why the direction of first update is the same as gradient.</p>

KFrank: <p>Hi Youlong!</p>
<aside class="quote no-group" data-username="Youlong_Ding" data-post="1" data-topic="129291" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/youlong_ding/48/38706_2.png" class="avatar"> Youlong_Ding:</div>
<blockquote>
<p>there are no momentum acculumated, why the direction of first update is the same as gradient.</p>
</blockquote>
</aside>
<p>Looking at the original <a href="https://arxiv.org/pdf/1412.6980.pdf" rel="noopener nofollow ugc">Adam paper</a> linked to in pytorch’s<br>
<a href="https://pytorch.org/docs/1.9.0/generated/torch.optim.Adam.html" rel="noopener nofollow ugc"><code>Adam</code> documentation</a>, I believe this is the way <code>Adam</code> is<br>
supposed to work. (I have not looked at pytorch’s <code>Adam</code><br>
implementation.)</p>
<p>Quoting from the abstract of the Adam paper:</p>
<blockquote>
<p>The method … is invariant to diagonal rescaling of the gradients</p>
</blockquote>
<p>Note, that the direction of the gradient in some sense “washes out”<br>
is portrayed as a desirable feature of <code>Adam</code>.</p>
<p>(Some suggestive terminology:  The “m” in “Adam” refers to “moment”<br>
rather than “momentum.”  It is true that <code>Adam</code> "accumulates moments,<br>
but in its very first step it moves a a direction that scales like<br>
<code>gradient / sqrt (gradient**2)</code>, which is to say, it does <em>not</em><br>
(necessarily) move in the direction of the gradient.)</p>
<p>We can verify that this behavior is displayed by pytorch’s <code>Adam</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; def fs (t):
...     return  (t * t * torch.tensor ([1.0, 1.0])).sum()
...
&gt;&gt;&gt; def fa (t):
...     return  (t * t * torch.tensor ([2.0, 0.5])).sum()
...
&gt;&gt;&gt; tss = torch.tensor ([1.0, 1.0], requires_grad = True)
&gt;&gt;&gt; tsa = torch.tensor ([1.0, 1.0], requires_grad = True)
&gt;&gt;&gt; tas = torch.tensor ([1.0, 1.0], requires_grad = True)
&gt;&gt;&gt; taa = torch.tensor ([1.0, 1.0], requires_grad = True)
&gt;&gt;&gt; sgds = torch.optim.SGD ([tss], lr = 0.1)
&gt;&gt;&gt; sgda = torch.optim.SGD ([tsa], lr = 0.1)
&gt;&gt;&gt; adas = torch.optim.Adam ([tas])
&gt;&gt;&gt; adaa = torch.optim.Adam ([taa])
&gt;&gt;&gt; fs (tss).backward()
&gt;&gt;&gt; tss.grad
tensor([2., 2.])
&gt;&gt;&gt; fa (tsa).backward()
&gt;&gt;&gt; tsa.grad
tensor([4., 1.])
&gt;&gt;&gt; fs (tas).backward()
&gt;&gt;&gt; tas.grad
tensor([2., 2.])
&gt;&gt;&gt; fa (taa).backward()
&gt;&gt;&gt; taa.grad
tensor([4., 1.])
&gt;&gt;&gt; sgds.step()
&gt;&gt;&gt; tss
tensor([0.8000, 0.8000], requires_grad=True)
&gt;&gt;&gt; sgda.step()
&gt;&gt;&gt; tsa
tensor([0.6000, 0.9000], requires_grad=True)
&gt;&gt;&gt; adas.step()
&gt;&gt;&gt; tas
tensor([0.9990, 0.9990], requires_grad=True)
&gt;&gt;&gt; adaa.step()
&gt;&gt;&gt; taa
tensor([0.9990, 0.9990], requires_grad=True)
</code></pre>
<p>Here, <code>fa (t)</code> is a function that has a larger gradient in the <code>t[0]</code><br>
direction than in the <code>t[1]</code> direction (while <code>fs (s)</code> is symmetrical).<br>
But you can see that <code>Adam</code> takes a step directly toward the origin<br>
(the minimum), and this is not in the direction of the gradient of<br>
<code>fa (t)</code>.  (By way of comparison, the steps taken by <code>SGD</code> are in<br>
the direction of the gradient.)</p>
<p>Best.</p>
<p>K. Frank</p>