Should a Learning Rate Scheduler adjust the learning rate at each optimization step (batch) or at each epoch?

username: Cyber_punk
cooked: <p>In <a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" rel="noopener nofollow ugc">PyTorch doc</a>, it suggests</p>
<blockquote>
<p><code>torch.optim.lr_scheduler</code> provides several methods to adjust the learning rate based on the number of <strong>epochs</strong>.</p>
</blockquote>
<p>However, from other sources it looks like the learning rate should be adjusted in every optimization step (batch):</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://kikaben.com/transformers-training-details/">
  <header class="source">
      <img src="https://kikaben.com/favicon.png" class="site-icon" width="" height="">

      <a href="https://kikaben.com/transformers-training-details/" target="_blank" rel="noopener nofollow ugc">KiKaBeN</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://kikaben.com/transformers-training-details/images/thumbnail.png" class="thumbnail">

<h3><a href="https://kikaben.com/transformers-training-details/" target="_blank" rel="noopener nofollow ugc">KiKaBeN - Transformer Training Details</a></h3>

  <p>Optimizer, Scheduler, Loss Function</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox stackexchange" data-onebox-src="https://datascience.stackexchange.com/questions/103022/warmup-steps-in-deep-learning">
  <header class="source">

      <a href="https://datascience.stackexchange.com/questions/103022/warmup-steps-in-deep-learning" target="_blank" rel="noopener nofollow ugc">datascience.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://datascience.stackexchange.com/users/104153/ss-varshini" target="_blank" rel="noopener nofollow ugc">
    <img alt="SS Varshini" src="https://lh4.googleusercontent.com/-N8a_5EdQluE/AAAAAAAAAAI/AAAAAAAAAAA/AMZuucmw6eeftb4oUftCeTfVtyrMC1A1ow/photo.jpg?sz=256" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://datascience.stackexchange.com/questions/103022/warmup-steps-in-deep-learning" target="_blank" rel="noopener nofollow ugc">Warmup steps in deep learning</a>
</h4>

<div class="tags">
  <strong>optimization</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://datascience.stackexchange.com/users/104153/ss-varshini" target="_blank" rel="noopener nofollow ugc">
    SS Varshini
  </a>
  on <a href="https://datascience.stackexchange.com/questions/103022/warmup-steps-in-deep-learning" target="_blank" rel="noopener nofollow ugc">07:51AM - 11 Oct 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><a href="https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial6/Transformers_and_MHAttention.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial6/Transformers_and_MHAttention.html</a></p>
<p>My question is: <strong>Should the learning rate in a Learning Rate Scheduler be adjusted per optimization step (batch) or per epoch?</strong></p>
<p>Is there a definitive answer to this, or it depends on the model?<br>
For transformer models, it looks like the learning rate is adjusted by batches. There are a few thousands of steps so it cannot be epochs, right?</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://nn.labml.ai/optimizers/noam.html">
  <header class="source">
      <img src="https://nn.labml.ai/icon.png" class="site-icon" width="64" height="64">

      <a href="https://nn.labml.ai/optimizers/noam.html" target="_blank" rel="noopener nofollow ugc">Noam optimizer from Attention is All You Need paper</a>
  </header>

  <article class="onebox-body">
    <img width="400" height="400" src="https://avatars1.githubusercontent.com/u/64068543?s=400&amp;v=4" class="thumbnail onebox-avatar">

<h3><a href="https://nn.labml.ai/optimizers/noam.html" target="_blank" rel="noopener nofollow ugc">Noam optimizer from Attention is All You Need paper</a></h3>

  <p>This is a tutorial/implementation of Noam optimizer. Noam optimizer has a warm-up period and then an exponentially decaying learning rate.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bca402f233a48718249cd1241f4b13fd14a41e9a.png" alt="noam_lr" data-base62-sha1="qUN9LlVwmWNWXUJuTfuiwDSdEEO" width="640" height="480"></p>