What does scheduler.step() do?

voqtuyen: <p>Why do we have to call scheduler.step() every epoch like in the tutorial by pytorch:</p>
<h4>Observe that all parameters are being optimized</h4>
<p>optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)</p>
<h4>Decay LR by a factor of 0.1 every 7 epochs</h4>
<p>exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)</p>
<p><a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a><br>
What if we don’t call it?</p>

ptrblck: <p>If you don’t call it, the learning rate won’t be changed and stays at the initial value.</p>

voqtuyen: <p>But it still follows something like:<br>
lr = 0.001     if epoch &lt; 7<br>
lr = 0.0001    if 7 &lt;= epoch &lt; 14<br>
lr = 0.00001   if 14 &lt;= epoch &lt; 21</p>
<p>And for epoch = 7, if we call scheduler.step(), lr will be updated from 0.001 to 0.0001<br>
for epoch = 8, if we call scheduler.step(), lr will be still 0.0001, right?</p>

ptrblck: <p>It won’t follow these scheme, if you don’t call <code>scheduler.step()</code> in each epoch.<br>
Here is a small example:</p>
<pre><code class="lang-python">optimizer = optim.SGD([torch.randn(1, requires_grad=True)], lr=1e-3)
exp_lr_scheduler = optim.lr_scheduler.StepLR(optimizer,
                                             step_size=7, gamma=0.1)

for epoch in range(1, 25):
    exp_lr_scheduler.step()
    print('Epoch {}, lr {}'.format(
        epoch, optimizer.param_groups[0]['lr']))

&gt; Epoch 1, lr 0.001
Epoch 2, lr 0.001
Epoch 3, lr 0.001
Epoch 4, lr 0.001
Epoch 5, lr 0.001
Epoch 6, lr 0.001
Epoch 7, lr 0.0001
Epoch 8, lr 0.0001
Epoch 9, lr 0.0001
Epoch 10, lr 0.0001
Epoch 11, lr 0.0001
Epoch 12, lr 0.0001
Epoch 13, lr 0.0001
Epoch 14, lr 1e-05
Epoch 15, lr 1e-05
Epoch 16, lr 1e-05
Epoch 17, lr 1e-05
Epoch 18, lr 1e-05
Epoch 19, lr 1e-05
Epoch 20, lr 1e-05
Epoch 21, lr 1.0000000000000002e-06
Epoch 22, lr 1.0000000000000002e-06
Epoch 23, lr 1.0000000000000002e-06
Epoch 24, lr 1.0000000000000002e-06
</code></pre>
<p>To get this behavior, you should call it in every epoch, not only if the current epoch equals the step size.</p>

krishna511: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Sir where should we use this schedular.step() method , after mode.eval()<br>
or in each epoch. My editor is throwing warning saying you should use it after optimizer.step()<br>
Also can I input step_size as [15,30,45] in sets, unfortunately that too is giving error, single number in step_size is working fine.</p>

ptrblck: <p>You should call it after the <code>optimizer.step()</code> operation as given in the warning message.<br>
To pass multiple milestones, you could use <code>torch.optim.lr_scheduler.MultiStepLR</code>.</p>

krishna511: <p>Got it Sir.Thanks again. One more thing sir is this LR schedular here is same concept  as earlystop in model training in case of other frameworks, or is there something else for this?</p>

slmatrix: <p>No. <code>torch.optim.lr_scheduler</code> is used to adjust only the hyperparameter of learning rate in a model.</p>
<p><strong>Early stopping</strong> refers to another hyperparameter, the number of train epochs. It is the stopping of training when loss reaches a plateau. Typically, models are over-parameterized meaning that if you keep training for “too many” epochs, they will begin to overfit on junk info (noise) from the dataset. To prevent overfitting, use <strong>early stopping</strong>; this means, when the loss stops decreasing on the <em>validation set</em>, stop training.</p>
<p>Jason has a great post on early stopping <a href="https://machinelearningmastery.com/early-stopping-to-avoid-overtraining-neural-network-models/" rel="noopener nofollow ugc">here</a>.</p>