Confusion with LR Scheduler get_lr()

SreenivasVRao: <pre><code class="lang-auto">test = torch.autograd.Variable(torch.randn([5,5]), requires_grad=True)
optimizer = torch.optim.Adam([test], lr = 0.0001)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.1)
lr_scheduler.get_lr()
</code></pre>
<p>This code snippet gives me a learning rate of 0.001</p>
<p>I want to start the optimizer with LR = 1e-4, and schedule a decay of 0.1 every 100 epochs.<br>
It seems like the LR Scheduler starts me off at 1e-3 instead.<br>
Can someone clarify?</p>
<p>This happens on Torch 0.3.1 and 0.4.0.</p>

ptrblck: <p>It’s a bit strange to see the initial learning rate seems to be too high, but it’s consistent with the example in the docs:</p>
<pre><code class="lang-python">scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
for epoch in range(100): 
    scheduler.step() 
    train(...) 
    validate(...)
</code></pre>
<p>As you call <code>scheduler</code> first, the learning rate will be set to <code>1e-4</code> for the next <code>step_size</code> epochs.</p>

SreenivasVRao: <pre><code class="lang-auto">test = torch.autograd.Variable(torch.randn([5,5]), requires_grad=True)
optimizer = torch.optim.Adam([test], lr = 0.0001)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.1)
lr_scheduler.step()
lr_scheduler.get_lr()
</code></pre>
<p>Yeah. The behaviour is weird. If I call <code>step()</code> before <code>get_lr()</code>, the output looks correct.</p>

IAmIronMan: <p>I am also finding the same issue while using lr scheduler. I want to start it from 1e-05 but it starts from 1e-03.<br>
Another thing I found out that, if I increase the step size, then it will starts from 1e-04<br>
Any one else got this issue. How they Solved it ?</p>

ptrblck: <p>Could you post a minimal code snippet to reproduce your issue?</p>

IAmIronMan: <h1><a name="training-settings-1" class="anchor" href="#training-settings-1"></a>Training Settings</h1>
<pre><code>base_lr = 0.000001 # Hyper parameters
max_lr = 1 # Hyper parameters
</code></pre>
<blockquote>
<p>def main():<br>
print(‘…Main Function starts…’)<br>
optimizer = optim.SGD(model.parameters(), lr=base_lr, momentum=momentum)<br>
scheduler = optim.lr_scheduler.CyclicLR(optimizer=optimizer,base_lr=base_lr,max_lr=max_lr,step_size_up=100,step_size_down=100)</p>
</blockquote>
<h1><a name="training-function-2" class="anchor" href="#training-function-2"></a>Training function</h1>
<blockquote>
<p>def train(model, optimizer, scheduler,plot_loss,plot_lr, training_dataloader, batch_size, grids, grids_values, bounding_boxes, classes, lambda_coord, lambda_noobject):<br>
model.train()<br>
batch_loss = 0<br>
for batch_index, batched_sample in enumerate(training_dataloader):<br>
print(‘Current LR : {}’.format(scheduler.get_lr()))<br>
batched_image = torch.tensor(batched_sample[‘image’], requires_grad=True, dtype=torch.float)<br>
batched_label = torch.tensor(batched_sample[‘label’], requires_grad=True, dtype=torch.float)<br>
batched_output = model(batched_image)<br>
batched_output = batched_output.view(batch_size, grids, grids_values) # Convert the output size into [N X GRIDS(225 X 225) X (5 * B + C)]<br>
loss = yolo_loss(batched_output, batched_label, grids, bounding_boxes, classes, lambda_coord, lambda_noobject)<br>
batch_loss = batch_loss + loss.item()<br>
optimizer.zero_grad()<br>
loss.backward()<br>
optimizer.step()<br>
scheduler.step()</p>
</blockquote>
<p><strong>For this settings, the output is :</strong> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4967517e2ec6fd0496e83b79a49716a1a02c5dfe.png" data-download-href="https://discuss.pytorch.org/uploads/default/4967517e2ec6fd0496e83b79a49716a1a02c5dfe" title="pytorchdisscus1"><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4967517e2ec6fd0496e83b79a49716a1a02c5dfe.png" alt="pytorchdisscus1" data-base62-sha1="atmdg1PA5Ah3ZIEaSyDWuEUMiSy" width="690" height="127" data-dominant-color="424242"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorchdisscus1</span><span class="informations">1235×228 14 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Changing the setting to step_size_up = 500 and nothing else, output is :</strong><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/86c379cfee92bc8365d2fe2571a181b37990048c.png" data-download-href="https://discuss.pytorch.org/uploads/default/86c379cfee92bc8365d2fe2571a181b37990048c" title="pytorchdisscus2"><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/86c379cfee92bc8365d2fe2571a181b37990048c.png" alt="pytorchdisscus2" data-base62-sha1="jeaMJF3DpKiLvIX3nKWmAUO0TY8" width="689" height="147" data-dominant-color="414141"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorchdisscus2</span><span class="informations">1240×265 15.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol>
<li>It is not starting from the base_lr</li>
<li>By changing the step size, why the lr scheduler is starting from different base_lr.</li>
<li>To work around this issue, I have to go through 2 cycles - one from 1-e06 to 1-e04 and 1-e04 to1. Is there any range issue with lr scheduler</li>
<li>Is there any bug or am I implementing it in the wrong way ?</li>
</ol>

ptrblck: <p>I might be misunderstanding the output, but the learning rate is starting from the <code>base_lr=1e-6</code> in both screenshots.</p>

IAmIronMan: <p>Correct me, If I am wrong. It is showing at the first print base_lr = 1e-06, when the lr scheduler is not called. After calling lr scheduler, it jumps to 0.01 and scaling with (+0.01). Rather it should starts from 1e-06 after calling lr scheduler and should scale accordingly.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/9354cb8f3172d49311ef436c614a8de37eb205d3.png" alt="pytorchdisscus3" data-base62-sha1="l1lRnqUby0qWZsRroXogkh82FZp" width="606" height="370"></p>