StepLR can't match the rule?

Danil_Hu: <p>Hi, I’m training my network with Pytorch. And I’ve met a curious problem.<br>
See, I’m going to use StepLR as my lr changing rule, and <strong>I expect that lr=0.0002 when epoch &lt; 30 and lr = 0.00002 while 30&lt; epoch &lt; 40(the entire epoch is 40)</strong>. So I set the function as follow, my optimizer is Adam:</p>
<pre data-code-wrap="python"><code class="lang-python">optimizer = optim.Adam(model.parameters(), lr=0.0002)
shceduler = optim.lr_sheduler.StepLR(optimizer, 30, gamma=0.1)
</code></pre>
<p>Learning rate is printed as below format:</p>
<pre data-code-wrap="python"><code class="lang-python">print("Training: Epoch[{:0&gt;3}/{0&gt;3}] Iteration[{:0&gt;3}/{0&gt;3}] Loss: {:.4f} lr={:.8f}".format(epoch + 1, iterations, i + 1, len(trainLoader), loss_avg, sheduler.get_lr()[0]))
</code></pre>
<p><strong>But I’ve got lr=0.000002 as epoch &gt; 30</strong>, I’ve check many times and don’t know why should this be happening<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b315e0f117555680ac5f60d73ebc649bc40d5a4.png" data-download-href="https://discuss.pytorch.org/uploads/default/3b315e0f117555680ac5f60d73ebc649bc40d5a4" title="tmp3"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b315e0f117555680ac5f60d73ebc649bc40d5a4.png" alt="tmp3" data-base62-sha1="8rDWhcUsyGDujvL9yYYFXfIzihS" width="674" height="500" data-dominant-color="193F49"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">tmp3</span><span class="informations">1219×903 226 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Can somebody tell what is the wrong here? pls</p>

Quinn: <p>Check your question again, I think you can substitue <code>get_last_lr()</code> for <code>get_lr()</code>. <code>get_lr()</code> couldn’t give you the lastest learning rate.</p>
<p>To get the last learning rate computed by the scheduler, please use <code>get_last_lr()</code>.</p>
<hr>
<p>Orignal reply:</p>
<p>Hi Danil, I think you haven’t accurately used <strong>StepLR</strong> which should be initialized and <strong>step</strong>(scheduler.step()). Here is a simple usage about StepLR.</p>
<h2>
<a name="h-1-you-need-schedulerstep-1" class="anchor" href="#h-1-you-need-schedulerstep-1"></a>1. you need scheduler.step()</h2>
<pre><code class="lang-auto">import torch
from torch import optim, nn
class NN(nn.Module):
    def __init__(self):
        super(NN, self).__init__()
        self.layer = nn.Linear(10,10)
    def forward(self, x):
        return self.layer(x)
model = NN()
optimizer = optim.Adam(model.parameters(), lr=0.0002)
scheduler = optim.lr_scheduler.StepLR(optimizer, 30, gamma=0.1)
for epoch in range(40):
    scheduler.step() # let scheduler's timer add 1!
    print("Training: Epoch[{:0&gt;3}] lr={:.8f}".format(epoch + 1, scheduler.get_last_lr()[0]))
</code></pre>
<h2>
<a name="h-2-you-may-need-schedulerget_last_lr-2" class="anchor" href="#h-2-you-may-need-schedulerget_last_lr-2"></a>2. you may need scheduler.get_last_lr()</h2>
<p>To get the last learning rate computed by the scheduler, please use <code>get_last_lr()</code>.</p>
<p>I hope this reply is helpful for you.</p>

Danil_Hu: <p>Oh.  you mean that:</p>
<ul>
<li>FIsrt, I should change <code>get_lr()</code> to <code>get_last_lr()</code>?</li>
<li>Second, I should put the <code>sheduler.step()</code> ahead of <code>optimizer.step()</code>?</li>
</ul>
<p>Emm I’ve put the <code>sheduler.step()</code> after finishing each epoch.<br>
What I’ve written is as below</p>
<pre><code class="lang-python">for epoch in range(40):
    print(epoch, 'lr={:6f}'.format(scheduler.get_lr()[0])
    for i, image in enumerate(trainLoader):
        ...
        optimizer.step()
        if i % 200 == 0 and i !=0:
            print("Training: Epoch[{:0&gt;3}/{0&gt;3}] Iteration[{:0&gt;3}/{0&gt;3}] Loss: {:.4f} lr={:.8f}".format(epoch + 1, iterations, i + 1, len(trainLoader), loss_avg, sheduler.get_lr()[0]))
           ...
    sheduler.step()

</code></pre>
<p>And I should modify it as this?</p>
<pre><code class="lang-python">for epoch in range(40):
    sheduler.step()
    print(epoch, 'lr={:6f}'.format(scheduler.get_last_lr()[0])
    for i, image in enumerate(trainLoader):
        ...
        optimizer.step()
        if i % 200 == 0 and i !=0:
            print("Training: Epoch[{:0&gt;3}/{0&gt;3}] Iteration[{:0&gt;3}/{0&gt;3}] Loss: {:.4f} lr={:.8f}".format(epoch + 1, iterations, i + 1, len(trainLoader), loss_avg, sheduler.get_last_lr()[0]))
</code></pre>

Quinn: <p>Answer of the first is <code>get_last_lr()</code>.<br>
Answer of the second is that both are right.</p>

Danil_Hu: <p>Got it! Thanks, I’ll try it. <img src="https://discuss.pytorch.org/images/emoji/apple/saluting_face.png?v=12" title=":saluting_face:" class="emoji" alt=":saluting_face:" loading="lazy" width="20" height="20"></p>