Correct way to use(order) scheduler in pytorch 1.6

John_J_Watson: <p>I am looking for some guidance on the correct way to use/order <code>scheduler.step()</code> within epochs. So, ofcourse, the official guidance says (<a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" class="inline-onebox" rel="noopener nofollow ugc">torch.optim — PyTorch 2.1 documentation</a>):</p>
<p>PATTERN:0</p>
<blockquote>
<blockquote>
<blockquote>
<p>scheduler = …</p>
</blockquote>
</blockquote>
</blockquote>
<pre><code class="lang-auto">&gt;&gt;&gt; for epoch in range(100):
&gt;&gt;&gt;     train(...)
&gt;&gt;&gt;     validate(...)
&gt;&gt;&gt;     scheduler.step()
</code></pre>
<p>But then I also see code like this (see for example: <a href="https://www.deeplearningwizard.com/deep_learning/boosting_models_pytorch/lr_scheduling/" class="inline-onebox" rel="noopener nofollow ugc">Learning Rate Scheduling - Deep Learning Wizard</a>):</p>
<p>PATTERN:1</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    # Decay Learning Rate
    scheduler.step()
    # Print Learning Rate
    print('Epoch:', epoch,'LR:', scheduler.get_lr())
    for i, (images, labels) in enumerate(train_loader):
        # Load images
        images = images.view(-1, 28*28).requires_grad_()
</code></pre>
<p>I also see this ^ similar pattern in quite a few github repos.</p>
<p>So, I did some small scale experiments using these two patterns in Pytorch1.6 and I got slightly better results using PATTERN:1.</p>
<p>So, my questions are:<br>
[1] Does the order matter?<br>
[2] Does the order depend on <code>step size</code> or <code>epoch number</code> or <code>optimizer</code>?</p>
<p>Also, I have another problem: I have two envs with pytorch 1.6 and I set PATTERN:1 for both. In one of the envs I get the warning:</p>
<pre><code class="lang-auto">UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/ok/ok0/ok1/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:351: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.
  "please use `get_last_lr()`.", UserWarning)
</code></pre>
<p>but in the other env, I dont <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"> Cant think why this could be!</p>
<p>Thank you!</p>

tom: <aside class="quote no-group" data-username="John_J_Watson" data-post="1" data-topic="97229">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_j_watson/48/16421_2.png" class="avatar"> John_J_Watson:</div>
<blockquote>
<p>[1] Does the order matter?<br>
[2] Does the order depend on <code>step size</code> or <code>epoch number</code> or <code>optimizer</code> ?</p>
</blockquote>
</aside>
<ol>
<li>Yes, use the pattern recommended by the documentation.</li>
<li>No.</li>
</ol>
<p>The background is that people found out that it doing the LR step first gives unintuitive maths w.r.t. how/when the learning rate changes, so it is strictly recommended to do the LR step last. As always, advice on the internet may be outdated (seems to be the case here, you could help out with filing an issue or PR on their github) or even bad advice (probably not the case here, <a class="mention" href="/u/ritchieng">@ritchieng</a> has been around for a long time and does know his stuff pretty well in my experience, but so I sometimes found stack overflow had really strange recommendations).</p>
<p>Regarding the warning: This might be some more general configuration about warnings on your system. If you believe it is because of PyTorch, you could double-check the versions (<code>torch.__version__</code>) and see in <code>torch.optim</code> to see which path it uses and see if <code>lr_scheduler.py</code> has the warning.</p>
<p>Best regards</p>
<p>Thomas</p>