Scheduler.step() after each epoch or after each minibatch

username: jpainam
cooked: <p>Hi, I defined a <code>exp_lr_scheduler</code> like</p>
<pre><code class="lang-python">exp_lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=40, gamma=0.1)
</code></pre>
<p>But was wondering what is the best way to use it? After each epoch or after each minibacth<br>
USE CASE 1.</p>
<pre><code class="lang-python">for epoch in range(num_epoch):
  scheduler.step()
  for img, labels in train_loader:
    .....
    optimizer.zero_grad()
    optimizer.step()
</code></pre>
<p>This one give this warning</p>
<pre><code class="lang-python">UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`.  
In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()
</code></pre>
<p>USE CASE 2</p>
<pre><code class="lang-python">for epoch in range(num_epoch):
  for img, labels in train_loader:
    .....
    optimizer.zero_grad()
    optimizer.step()
  # At the end of the epoch
  scheduler.step()
</code></pre>
<p>This way, no warning, since the <code>optimizer.step()</code> is called before <code>scheduler.step()</code>. But <code>scheduler.step()</code> still run for once per epoch<br>
USE CASE 3</p>
<pre><code class="lang-python">for epoch in range(num_epoch):
  for img, labels in train_loader:
    .....
    scheduler.step()  # run for each minibach
    optimizer.zero_grad()
    optimizer.step()
</code></pre>
<p>Here, <code>scheduler.step()</code> runs for each minibatch.<br>
What do you think is the right way of using such scheduler.<br>
Thank you.</p>

username: Dwight_Foster
cooked: <p>Use case <span class="hashtag">#2</span> is the best. That is the one recommended by the pytorch docs. You could use 3 if you set the stepsize really high but usually doing it in minibatches will lower the learning rate too fast.</p>

username: cgr71ii
cooked: <p>Hi!</p>
<p>Is always the best solution to update the LR scheduler when the epoch finishes? There are examples like <code>get_linear_schedule_with_warmup</code> from HuggingFace that it seems to be applied at batch-level (github huggingface/transformers/issues/2375, <a href="https://github.com/huggingface/transformers/blob/594ca6deadb6bb79451c3093641e3c9e5dcfa446/examples/contrib/run_openai_gpt.py#L252" class="inline-onebox" rel="noopener nofollow ugc">transformers/run_openai_gpt.py at 594ca6deadb6bb79451c3093641e3c9e5dcfa446 · huggingface/transformers · GitHub</a>). Moreover, I’ve tried to check the fairseq code and, apparently, they apply the LR scheduler once per batch (default configuration if I’m not wrong) instead of once per epoch (<a href="https://github.com/facebookresearch/fairseq/blob/cba35cdbca8385fe00045796ea731c0296f5434b/fairseq/trainer.py#L1009" class="inline-onebox" rel="noopener nofollow ugc">fairseq/trainer.py at cba35cdbca8385fe00045796ea731c0296f5434b · facebookresearch/fairseq · GitHub</a> → train_step() is invoked by train() which is invoked by fairseq_train CLI, and train_step() calls set_num_updates() per batch, which updates the number of steps and the LR scheduler), but I’m not really sure since I don’t know very good the code.</p>
<p>I’ve tried to look for the answer, but I’m not sure what is the best solution yet. Or even if it depends on the specific LR scheduler or task…</p>

username: R-N
cooked: <p>Bump. I’d like to know too. I see recent large models with very large datasets tend to be trained by step count instead of epoch, and they train for only few epoch overall due to dataset size. If that’s the case then the scheduler step should be called at every step or every n step, no? Is there a rule of thumb, guideline, or something? Or should it just be a hyperparameter to tune?</p>