Why don't we add AdamW to the official optimizer sets?

jinserk: <p>Hi all,</p>
<p>I’m usually using <a href="https://github.com/egg-west/AdamW-pytorch" rel="nofollow noopener">AdamW optimizer implemented by egg-west</a>, since it is obviously and definitely effective when I train models. So I wonder why PyTorch doesn’t include AdamW or SGDR in our official optimizer sets. Is there any specific reason that AdamW or SGDR has some unclear issues in theory or in their implementation?</p>
<p>Thanks,<br>
Jinserk</p>

Tony-Y: <p><a href="https://www.fast.ai/2018/07/02/adam-weight-decay/" class="onebox" target="_blank" rel="nofollow noopener">https://www.fast.ai/2018/07/02/adam-weight-decay/</a></p>
<p>This article says “fast.ai” is only the library that implemented the fix.</p>

Tony-Y: <aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/pull/3740" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
      <a href="https://github.com/jingweiz" target="_blank" rel="nofollow noopener">
    <img alt="jingweiz" src="https://avatars2.githubusercontent.com/u/9096283?v=4" class="thumbnail onebox-avatar" width="90" height="90">
  </a>

<h4>
  <a href="https://github.com/pytorch/pytorch/pull/3740" target="_blank" rel="nofollow noopener">Fixing Weight Decay Regularization in Adam </a>
</h4>

<div class="date">
  by <a href="https://github.com/jingweiz" target="_blank" rel="nofollow noopener">jingweiz</a>
  on <a href="https://github.com/pytorch/pytorch/pull/3740" target="_blank" rel="nofollow noopener">07:02PM - 16 Nov 17 UTC</a>
</div>

<div class="github-commit-stats">
  <strong>16 commits</strong>
  changed <strong>8 files</strong>
  with <strong>398 additions</strong>
  and <strong>0 deletions</strong>.
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I found a PR for AdamW.</p>
<p>(Edit)<br>
<aside class="onebox githubpullrequest">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/pull/4429" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
      <a href="https://github.com/kashif" target="_blank" rel="nofollow noopener">
    <img alt="kashif" src="https://avatars0.githubusercontent.com/u/8100?v=4" class="thumbnail onebox-avatar" width="90" height="90">
  </a>

<h4>
  <a href="https://github.com/pytorch/pytorch/pull/4429" target="_blank" rel="nofollow noopener">Decoupled Weight Decay Regularization in optimizers (added adamw and sgdw among others)</a>
</h4>

<div class="date">
  by <a href="https://github.com/kashif" target="_blank" rel="nofollow noopener">kashif</a>
  on <a href="https://github.com/pytorch/pytorch/pull/4429" target="_blank" rel="nofollow noopener">05:54PM - 01 Jan 18 UTC</a>
</div>

<div class="github-commit-stats">
  <strong>43 commits</strong>
  changed <strong>9 files</strong>
  with <strong>208 additions</strong>
  and <strong>36 deletions</strong>.
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I think this PR is better than the above one.</p>

Tony-Y: <p>TORCH.OPTIM.LR_SCHEDULER.CosineAnnealingWarmRestarts is a scheduler for SGDR.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/66e6571eb871bafe8d03419c17be260a9b06c32f/torch/optim/lr_scheduler.py#L656" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/66e6571eb871bafe8d03419c17be260a9b06c32f/torch/optim/lr_scheduler.py#L656" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/66e6571eb871bafe8d03419c17be260a9b06c32f/torch/optim/lr_scheduler.py#L656</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="646" style="counter-reset: li-counter 645 ;">
<li>                    momentum = max_momentum - base_height * self.scale_fn(cycle)</li>
<li>                else:</li>
<li>                    momentum = max_momentum - base_height * self.scale_fn(self.last_epoch)</li>
<li>                momentums.append(momentum)</li>
<li>            for param_group, momentum in zip(self.optimizer.param_groups, momentums):</li>
<li>                param_group['momentum'] = momentum</li>
<li>
</li>
<li>        return lrs</li>
<li>
</li>
<li>
</li>
<li class="selected">class CosineAnnealingWarmRestarts(_LRScheduler):</li>
<li>    r"""Set the learning rate of each parameter group using a cosine annealing</li>
<li>    schedule, where :math:`\eta_{max}` is set to the initial lr, :math:`T_{cur}`</li>
<li>    is the number of epochs since the last restart and :math:`T_{i}` is the number</li>
<li>    of epochs between two warm restarts in SGDR:</li>
<li>
</li>
<li>    .. math::</li>
<li>        \eta_t = \eta_{min} + \frac{1}{2}(\eta_{max} - \eta_{min})(1 +</li>
<li>        \cos(\frac{T_{cur}}{T_{i}}\pi))</li>
<li>
</li>
<li>    When :math:`T_{cur}=T_{i}`, set :math:`\eta_t = \eta_{min}`.</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


jinserk: <p>Thanks <a class="mention" href="/u/tony-y">@Tony-Y</a> for the informations! I didn’t know the PR exists and am surprised that it has been more than a year to verify. I knew that the Fast.ai’s solution but wanted to ask why the official PyTorch doesn’t have one since the AdamW is quite effective. I’ll follow up the PRs as well. Thank you again!</p>