SGD with momentum pseudocode error?

belsten: <p>Hello,</p>
<p>I’m interested in using SGD with momentum. In looking at the pseudocode on the SGD documentation site (<a href="https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD" class="inline-onebox" rel="noopener nofollow ugc">SGD — PyTorch 2.6 documentation</a>), I’m confused as to why the momentum parameter (<code>momentum&gt;0</code>) is never touches the gradient when <code>nesterov=False</code> and <code>dampening&gt;=1</code>.</p>
<p>Below is a screenshot of the pseudocode where the red arrows indicate (to my understanding) what lines would be executed should you define SGD to be:</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
</code></pre>
<p>Based on the pseudocode, it seems like momentum is not used at all under these parameter settings. Am I missing something or is the pseudocode incorrect?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/2/629513af133b3b796dd1e2ea0beb07388c42d8ef.png" data-download-href="https://discuss.pytorch.org/uploads/default/629513af133b3b796dd1e2ea0beb07388c42d8ef" title="Screenshot 2025-03-23 at 2.01.28 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/629513af133b3b796dd1e2ea0beb07388c42d8ef_2_544x500.png" alt="Screenshot 2025-03-23 at 2.01.28 PM" data-base62-sha1="e46bEafuHdNuQSpMfcVlBsf74sv" width="544" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/629513af133b3b796dd1e2ea0beb07388c42d8ef_2_544x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/629513af133b3b796dd1e2ea0beb07388c42d8ef_2_816x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/2/629513af133b3b796dd1e2ea0beb07388c42d8ef_2_1088x1000.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2025-03-23 at 2.01.28 PM</span><span class="informations">1408×1294 39.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Belsten!</p>
<aside class="quote no-group" data-username="belsten" data-post="1" data-topic="218178">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/belsten/48/76304_2.png" class="avatar"> belsten:</div>
<blockquote>
<p>Based on the pseudocode, it seems like momentum is not used at all under these parameter settings. Am I missing something or is the pseudocode incorrect?</p>
</blockquote>
</aside>
<p>The pseudocode looks sensible to me and is not obviously incorrect.  (I don’t know<br>
whether it agrees in detail with pytorch’s implementation.)</p>
<p>As I read it, on the first iteration (<code>t = 1</code>) the momentum, <code>mu</code>, is indeed not used.  But<br>
this makes sense because <code>mu</code> is basically a moving-average parameter.  On the first<br>
iteration, the “moving-average gradient” is just set equal to the gradient, because there<br>
is not yet a previous value with which to average it.</p>
<p>On subsequent iterations (<code>t &gt; 1</code>), we have <code>b_t = mu b_t + (1 - tau) g_t</code>.  <code>mu</code> is<br>
used and is specifying how much how much of the previous value of <code>b_t</code> is mixed into<br>
the new value of <code>b_t</code>.</p>
<p>Is it possible that you are misreading <code>tau</code>, the dampening parameter, for <code>t</code>, the iteration<br>
index used in the pseudocode?</p>
<p>Best.</p>
<p>K. Frank</p>