What is the proper way of resuming a scheduler?

Eta_C: <p>PyTorch Version: 1.11<br>
I want to resume my learning rate after my training is terminated.<br>
Here’s a toy example:</p>
<pre><code class="lang-python">import torch.nn as nn
from torch.optim import SGD
from torch.optim.lr_scheduler import CosineAnnealingLR

n = 100
net = nn.Conv2d(3, 3, 1)
opt = SGD(net.parameters(), 0.1)
s = CosineAnnealingLR(opt, n, last_epoch=-1)
ckpt = "s.pt"

s_lr = []
for _ in range(s.last_epoch, n):
    lr = s.get_last_lr()
    s_lr.extend(lr)
    s.step()
    if _ == 10:
        torch.save(s.state_dict(), ckpt)
</code></pre>
<p>Now I resume it by two ways. Note that my program is terminated. All object should be instanced again.</p>
<pre><code class="lang-python"># Method 1
n = 100
net = nn.Conv2d(3, 3, 1)
opt = SGD(net.parameters(), 0.1)
ckpt = "s.pt"
state_dict = torch.load(ckpt)
s = CosineAnnealingLR(opt, n, last_epoch=state_dict["last_epoch"])
s.load_state_dict(state_dict)
s_lr = []
for _ in range(s.last_epoch, n):
    lr = s.get_last_lr()
    s_lr.extend(lr)
    s.step()
print(s_lr[:3])
</code></pre>
<p>It throws an error <code>param 'initial_lr' is not specified in param_groups[0] when resuming an optimizer</code>. And</p>
<pre><code class="lang-python"># Method 2
n = 100
net = nn.Conv2d(3, 3, 1)
opt = SGD(net.parameters(), 0.1)
ckpt = "s.pt"
state_dict = torch.load(ckpt)
s = CosineAnnealingLR(opt, n, last_epoch=-1)
s.load_state_dict(state_dict)
s_lr = []
for _ in range(s.last_epoch, n):
    lr = s.get_last_lr()
    s_lr.extend(lr)
    s.step()
print(s_lr[:3])
</code></pre>
<p>It print a wrong learning rate <code>[0.09704403844771128, 0.09942787402278414, 0.09880847171860509]</code>, which <code>0.099 &gt; 0.097</code>.</p>
<p>What is the proper way of resuming a scheduler?</p>
<p>NOTE: I have checked <a href="https://discuss.pytorch.org/t/what-is-the-proper-way-of-using-last-epoch-in-a-lr-scheduler/20297" class="inline-onebox">What is the proper way of using last_epoch in a lr_scheduler?</a> but it could not solve my problem.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Eta_C: <p>Does anyone have some suggesstions?</p>

ptrblck: <p>The first approach should work if you also restore the optimizer and resume from the 10th epoch, since you have also stored the <code>state_dict</code> of the scheduler in the 10th step:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.nn as nn
from torch.optim import SGD
from torch.optim.lr_scheduler import CosineAnnealingLR

n = 100
net = nn.Conv2d(3, 3, 1)
opt = SGD(net.parameters(), 0.1)
s = CosineAnnealingLR(opt, n, last_epoch=-1)
ckpt = "s.pt"

s_lr = []
for _ in range(s.last_epoch, n):
    opt.step()
    lr = s.get_last_lr()
    s_lr.extend(lr)
    s.step()
    if _ == 10:
        torch.save(s.state_dict(), ckpt)

plt.plot(s_lr)
opt_sd = opt.state_dict()


s_lr = np.array(s_lr)

# Method 1
n = 100
net = nn.Conv2d(3, 3, 1)
opt = SGD(net.parameters(), 0.1)
opt.load_state_dict(opt_sd)
ckpt = "s.pt"
state_dict = torch.load(ckpt)
s = CosineAnnealingLR(opt, n, last_epoch=state_dict["last_epoch"]-1)
s.load_state_dict(state_dict)
s_lr_res = []
for _ in range(s.last_epoch, n):
    opt.step()
    lr = s.get_last_lr()
    s_lr_res.extend(lr)
    s.step()


s_lr_res = np.array(s_lr_res)
plt.plot(np.arange(100), s_lr)
plt.plot(np.arange(100), np.concatenate((np.zeros(11), s_lr_res)))

print(s_lr[11:] - s_lr_res)
# [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
#  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
#  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
#  0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
</code></pre>
<p>Output plot:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a2d7c1d883771d496f39f7ea8d02dd7c69d38934.png" alt="image" data-base62-sha1="nezBGMmmjt97GWGIpsHTStdLPUw" width="378" height="248"></p>

mifan002: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>I can reproduce your code and see the same plot using CosineAnnealingLR. But when I tried with PolynomialLR and setting its power=2.0, the plot is strange: the resuming lr curve did not overlap on the old curve, as the attached plot shows. When I enter the debug mode in pycharm, I observed that the resuming scheduler’s last epoch changed to 0 after the first step().</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/73d434471c12a04430b5d6c65a502af12c33c7a1.png" alt="polynomialLR_resuming_curve" data-base62-sha1="gwFynANJb2ncYpUciMaj87WMtPj" width="640" height="480"><br>
What could be the reason for this?</p>
<p>Following is the code I used, I just copied your code snippet and replace the CosineAnnealingLR with PolynomialLR and set its “power” to 2.0.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.optim import SGD
from torch.optim.lr_scheduler import CosineAnnealingLR
from torch.optim.lr_scheduler import PolynomialLR
import matplotlib.pyplot as plt
import numpy as np


n = 100
net = nn.Conv2d(3, 3, 1)
opt = SGD(net.parameters(), 0.1)
s = PolynomialLR(opt, n, last_epoch=-1, power=2.0)
ckpt = "s.pt"

s_lr = []
for _ in range(s.last_epoch, n):
    opt.step()
    lr = s.get_last_lr()
    s_lr.extend(lr)
    s.step()
    if _ == 10:
        torch.save(s.state_dict(), ckpt)

plt.plot(s_lr)
plt.show()
opt_sd = opt.state_dict()


s_lr = np.array(s_lr)

# Method 1
n = 100
net = nn.Conv2d(3, 3, 1)
opt = SGD(net.parameters(), 0.1)
opt.load_state_dict(opt_sd)
ckpt = "s.pt"
state_dict = torch.load(ckpt)
s = PolynomialLR(opt, n, last_epoch=state_dict["last_epoch"]-1, power=2.0)
s.load_state_dict(state_dict)
s_lr_res = []
for _ in range(s.last_epoch, n):
    opt.step()
    lr = s.get_last_lr()
    s_lr_res.extend(lr)
    s.step()


s_lr_res = np.array(s_lr_res)
plt.plot(np.arange(100), s_lr)
plt.plot(np.arange(100), np.concatenate((np.zeros(11), s_lr_res)))
plt.show()
print(s_lr[11:] - s_lr_res)
</code></pre>