Weight_decay in torch.Adam

maridia: <p>In the current <a href="https://pytorch.org/docs/stable/optim.html" rel="noopener nofollow ugc">pytorch docs for torch.Adam</a>, the following is written:</p>
<p>"Implements Adam algorithm.</p>
<p>It has been proposed in <a href="https://arxiv.org/abs/1412.6980" rel="noopener nofollow ugc">Adam: A Method for Stochastic Optimization</a>. The implementation of the L2 penalty follows changes proposed in <a href="https://arxiv.org/abs/1711.05101" rel="noopener nofollow ugc">Decoupled Weight Decay Regularization</a>."</p>
<p>This would lead me to believe that the current implementation of Adam is essentially equivalent to AdamW.  The fact that torch.AdamW exists as a separate optimizer leads me to believe that this isn’t true.  Also, after looking at the source code of torch.Adam, I don’t see any difference from a standard L2 penalty implementation.  So is the documentation incorrect and the “changes proposed in [Decoupled Weight Decay Regularization]” are actually absent from torch.Adam?</p>

ptrblck: <p>This might be indeed a documentation issue, as I cannot see any point of using <code>AdamW</code> in this case.<br>
Would you mind creating a GitHub issue so that we can track it and would you be interested in fixing the doc?</p>

maridia: <p>I’ve made a new issue here:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/48793" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/48793" target="_blank" rel="noopener nofollow ugc">Weight_decay in torch.Adam</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-12-03" data-time="17:06:20" data-timezone="UTC">05:06PM - 03 Dec 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/maridia" target="_blank" rel="noopener nofollow ugc">
          <img alt="maridia" src="https://avatars1.githubusercontent.com/u/40644568?v=4" class="onebox-avatar-inline" width="20" height="20">
          maridia
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">📚 Documentation
In the current pytorch docs for torch.Adam, the following is written:
"Implements Adam algorithm.
It has been proposed in Adam: A Method...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I’m not sure what is involved in fixing the doc myself as I’ve never opened a pull request before.</p>

ptrblck: <p>Thanks for creating the issue! I’m sure we can guide you through the PR. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>