Need explanations of adamw implementation

leengsmile: <p>The <a href="https://arxiv.org/pdf/1711.05101.pdf" rel="noopener nofollow ugc">Adamw paper</a> says the Adam with weight decay looks like<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64bda9967d821668f20f0cffe852c241f080c8f2.png" data-download-href="https://discuss.pytorch.org/uploads/default/64bda9967d821668f20f0cffe852c241f080c8f2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64bda9967d821668f20f0cffe852c241f080c8f2_2_690x336.png" alt="image" data-base62-sha1="enc60LXtSiLw8QENFyaZXMiaUkG" width="690" height="336" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64bda9967d821668f20f0cffe852c241f080c8f2_2_690x336.png, https://discuss.pytorch.org/uploads/default/original/3X/6/4/64bda9967d821668f20f0cffe852c241f080c8f2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/4/64bda9967d821668f20f0cffe852c241f080c8f2.png 2x" data-dominant-color="EEEDEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">861×420 124 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And the corresponding <a href="https://github.com/pytorch/pytorch/blob/a500491cbc51c093b5990a69c6066479c13e360c/torch/optim/adamw.py#L71" rel="noopener nofollow ugc">pytorch implementation</a> is</p>
<pre><code class="lang-auto"># Perform stepweight decay
p.data.mul_(1 - group['lr'] * group['weight_decay'])
</code></pre>
<p>I’m stuck by how line 12 in <strong>Algorithm 2</strong>(adamw) comes to the pytorch version.</p>
<p>I googled for a while and found that fast.ai published a post <a href="https://www.fast.ai/2018/07/02/adam-weight-decay/" rel="noopener nofollow ugc">AdamW and Super-convergence is now the fastest way to train neural nets<br>
</a>, where it concluded that <code>Adamw</code> might be implemented in some way like</p>
<pre><code class="lang-auto">loss.backward()
for group in optimizer.param_groups():
    for param in group['params']:
        param.data = param.data.add(-wd * group['lr'], param.data)
optimizer.step()
</code></pre>
<p>Am I missing something in order to derive from Algorithm 2 to the pytorch implementation?</p>
<p>Thank you for any elaborations.</p>

vincentqb: <p>There was a discussion about this <a href="https://github.com/pytorch/pytorch/pull/21250#issuecomment-520559993" rel="nofollow noopener">here</a>. Is this what you were looking for?</p>

leengsmile: <p>Thank you! Your github comments really helps me understand the “hidden” thoughts, though I scratched my head for hours before I realize the thoughts. To verify the implementation,  line 12 could be further expanded as</p>
<p>theta_t &lt;- theta_{t-1} - eta_{t} * apha \frac{.}{.} -  eta_t * lambda * theta_{t-1}</p>
<p>group[“lr”] = eta_{t} * apha because group[“lr”] might be lr scheduler, and thus  group[“weight_decay”] = lambda / apha, when multiplied, the alphas get cancelled.</p>
<p>(Sorry for the ugly plain-text equations).</p>
<p>The actual weight decay parameter is scaled by initial learning rate (1/alpha). And we should take that in to consideration when tuning models.</p>
<p>Correct me if I’m wrong.</p>