Loss calculation of SGD with Nesterov

gilbertocunha: <p>So from my knowledge nesterov momentum should look something like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/925fe7dcae1231602dca45c8ee063dd4794775ed.png" data-download-href="https://discuss.pytorch.org/uploads/default/925fe7dcae1231602dca45c8ee063dd4794775ed" title="nesterov"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/925fe7dcae1231602dca45c8ee063dd4794775ed.png" alt="nesterov" data-base62-sha1="kSTbDBQLCMmCm9z70Bq1bHiU46F" width="690" height="127" data-dominant-color="000000"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nesterov</span><span class="informations">1113×206 8.21 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>According to this formula, the loss function should be calculated not according to our model’s parameters, but with a prediction of the future model’s parameters.</p>
<p>And using a torch SGD optimizer with Nesterov should look like the following:</p>
<pre data-code-wrap="python"><code class="lang-python">optimizer = torch.optim.SGD(..., nesterov=True)
optimizer.zero_grad()
loss_fn(model(input), target).backward()
optimizer.step()
</code></pre>
<p>My question is how can <code>optimizer.step()</code> apply the correct parameter update (in the Nesterov case) if the loss is calculated according to the model’s present parameters and not “future” parameter predictions as in the equation above? I already looked at the source code and I still do not understand how this was done.</p>
<p>Does PyTorch call the <code>forward</code> method of the model and somehow overwrites its parameters in <code>model(input)</code>? If that is the case, how is that done?</p>

albanD: <p>Hi,</p>
<p>I am not 100% sure but I think the trick is to have the parameters actually contain <code>\theta_t - \mu v_t</code>. That way a regular forward/backawrd evaluates the right value and you only need to update the step formula to take this into account.</p>

gilbertocunha: <p>Thanks for the response</p>
<p>I guess it does make some sense that way, but those parameters would have to be changed before the model(input) instruction, or while that instruction is being evaluated, I guess.</p>
<p>Also not sure if the parameters should not be reverted back before calling <code>backward()</code>, though, would that not have an impact on the gradient calculation?</p>

albanD: <aside class="quote no-group" data-username="gilbertocunha" data-post="3" data-topic="110814">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gilbertocunha/48/27690_2.png" class="avatar"> gilbertocunha:</div>
<blockquote>
<p>would that not have an impact on the gradient calculation?</p>
</blockquote>
</aside>
<p>You compute the gradient wrt to the weights, you added just a constant value to them, so no it wouldn’t impact the gradient computation.</p>

gilbertocunha: <p>good point, silly me ahaha</p>
<p>Thank you so much for your help!<br>
Even if this isn’t the way it is implementated in the source code it certainly seems like an implementable solution!</p>

ado_sar: <p>Can you elaborate how PyTorch calculates the gradient for future model’s parameters?</p>
<p>By reading the pseudocode:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6f593183533783774faf5836b149dd942e195ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/b6f593183533783774faf5836b149dd942e195ec" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6f593183533783774faf5836b149dd942e195ec_2_501x499.png" alt="image" data-base62-sha1="q6x2wDVAjRPbApCZl09pPStNPqc" width="501" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6f593183533783774faf5836b149dd942e195ec_2_501x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6f593183533783774faf5836b149dd942e195ec_2_751x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6f593183533783774faf5836b149dd942e195ec.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">795×792 21.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t see how PyTorch achieve this.</p>