lr_schedular.StepLR() seems to be changing the LR wrongly in the first .step() call

username: dasayan05
cooked: <p>I was running this very simple (toy) code to observe the change in LR with epochs when there is a schedular. I used the simple <code>StepLR()</code>.</p>
<pre><code class="lang-auto">import torch
print('PyTorch version: {0}'.format(torch.__version__))

model = Net(n_in, n_out) # 'Net' is a simple MLP
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2)
schedular = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.1)

print('Initial LR : {0:.8f}'.format(schedular.get_lr()[0]))
for e in range(8):
    schedular.step()
    print('Epoch {0}, LR: {1:.8f}'.format(e, schedular.get_lr()[0]))
    for i in range(5):
        optimizer.zero_grad()

        y = model(x) # x is proper sized input tensor
        l = y.mean()
        l.backward()
        
        optimizer.step()
</code></pre>
<p>When I ran this code with <strong>PyTorch 1.1.0</strong>, it produced</p>
<pre><code class="lang-auto">PyTorch version: 1.1.0
Initial LR : 0.01000000
Epoch 0, LR: 0.00010000
Epoch 1, LR: 0.00001000
Epoch 2, LR: 0.00000100
Epoch 3, LR: 0.00000010
Epoch 4, LR: 0.00000001
Epoch 5, LR: 0.00000000
Epoch 6, LR: 0.00000000
Epoch 7, LR: 0.00000000
</code></pre>
<p><strong>My question is, why is the first invocation of <code>schedular.step()</code> multiplies the LR with 0.01 and not 0.1 (i.e., gamma) ? Although, the later calls to <code>.step()</code> are okay.</strong></p>
<p>When I went further to investigate, I found out that this behaviour is different in PyTorch version <strong>0.4.0</strong> which produced this output. This seems to be correct.</p>
<pre><code class="lang-auto">PyTorch version: 0.4.0
Initial LR : 0.10000000
Epoch 0, LR: 0.01000000
Epoch 1, LR: 0.00100000
Epoch 2, LR: 0.00010000
Epoch 3, LR: 0.00001000
Epoch 4, LR: 0.00000100
Epoch 5, LR: 0.00000010
Epoch 6, LR: 0.00000001
Epoch 7, LR: 0.00000000
</code></pre>

username: ptrblck
cooked: <p>This seems to be related to <a href="https://github.com/pytorch/pytorch/issues/20138" rel="nofollow noopener">this issue</a>.<br>
If you print the optimizer’s learning rate, you should see that it’s behaving as expected:</p>
<pre><code class="lang-python">model = nn.Linear(1, 1) # 'Net' is a simple MLP
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2)
schedular = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.1)

print('Initial LR : {0:.8f}'.format(schedular.get_lr()[0]))
for e in range(8):
    schedular.step()
    print('Epoch {0}, LR: {1:.8f}, opt LR {2:.8f}'.format(e, schedular.get_lr()[0],
          optimizer.param_groups[0]['lr']))
</code></pre>

username: Mughees
cooked: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I ran this code on colab and the output is not consistent. <a href="https://colab.research.google.com/drive/15yjCBbiiHCRJ63aFUdoukN4kjwi9eClA" rel="nofollow noopener">link to colab</a></p>
<p>import torch<br>
print(“pytorch version”,torch. <strong>version</strong> )<br>
import torch.nn as nn<br>
model = nn.Linear(1, 1) # ‘Net’ is a simple MLP<br>
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)<br>
schedular = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones = [4,7], gamma=0.1)</p>
<p>print(‘Initial LR : {0:.8f}’.format(schedular.get_lr()[0]))<br>
for e in range(8):<br>
schedular.step()<br>
print(‘Epoch {0}, LR: {1:.8f}, opt LR {2:.8f}’.format(e, schedular.get_lr()[0],<br>
optimizer.param_groups[0][‘lr’]))</p>
<p>pytorch version 1.4.0<br>
Initial LR : 0.10000000<br>
Epoch 0, LR: 0.10000000, opt LR 0.10000000<br>
Epoch 1, LR: 0.10000000, opt LR 0.10000000<br>
Epoch 2, LR: 0.10000000, opt LR 0.10000000<br>
<strong>Epoch 3, LR: 0.00100000, opt LR 0.01000000</strong><br>
Epoch 4, LR: 0.01000000, opt LR 0.01000000<br>
Epoch 5, LR: 0.01000000, opt LR 0.01000000<br>
<strong>Epoch 6, LR: 0.00010000, opt LR 0.00100000</strong><br>
Epoch 7, LR: 0.00100000, opt LR 0.00100000</p>

username: Mughees
cooked: <p>even this is inconsistent at Google colab. i am receiving output with step_size = 2 as,</p>
<p>pytorch version 1.4.0<br>
Initial LR : 0.10000000<br>
Epoch 0, LR: 0.10000000, opt LR 0.10000000<br>
Epoch 1, LR: 0.00100000, opt LR 0.01000000<br>
Epoch 2, LR: 0.01000000, opt LR 0.01000000<br>
Epoch 3, LR: 0.00010000, opt LR 0.00100000<br>
Epoch 4, LR: 0.00100000, opt LR 0.00100000<br>
Epoch 5, LR: 0.00001000, opt LR 0.00010000<br>
Epoch 6, LR: 0.00010000, opt LR 0.00010000<br>
Epoch 7, LR: 0.00000100, opt LR 0.00001000</p>

username: cslxiao
cooked: <p>You should use <code>get_last_lr()</code> in pytorch 1.4 instead.<br>
refer to <a href="https://github.com/pytorch/pytorch/pull/26423" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/26423</a></p>