Sequential LR schedulers

Guru_Swaroop: <p>I am not able to build a custom learning rate scheduler using SequentialLR. To elaborate, I am trying to combine CosineAnnealingLR followed by ConstantLR (as per the code below).</p>
<p>I have set the initial learning rate of 0.1</p>
<ul>
<li>I was expecting to have a cosine decay starting from the learning rate of 0.1 and decay to 1e-4 for first 72 epochs</li>
<li>Thereafter, for last 18 epochs, I would like to have a fixed learning rate of 1e-4</li>
</ul>
<pre><code class="lang-auto">import torch
import math
from torch.optim.lr_scheduler import SequentialLR, CosineAnnealingLR, ConstantLR
from matplotlib import pyplot as plt

class TinyModel(torch.nn.Module):
    def __init__(self):
        super(TinyModel, self).__init__()
        self.linear2 = torch.nn.Linear(10, 10)
        self.softmax = torch.nn.Softmax()

    def forward(self, x):
        x = self.linear1(x)
        x = self.softmax(x)
        return x

# Set the parameters for learning
num_epochs = 90

model = TinyModel()

optimizer = torch.optim.SGD(
    params=model.parameters(),
    lr=0.1,
    momentum=0.9,
    weight_decay=5e-5,
)

num_steps_optimizer1 = math.ceil(num_epochs * 0.8)
num_steps_optimizer2 = num_epochs - num_steps_optimizer1

scheduler1 = CosineAnnealingLR(optimizer, T_max=num_steps_optimizer1, eta_min=1e-4)
scheduler2 = ConstantLR(optimizer, factor=1e-3, total_iters=num_steps_optimizer2)
scheduler = SequentialLR(optimizer, schedulers=[scheduler1, scheduler2], milestones=[num_steps_optimizer1+1])

lr_schedule = []
for epoch in range(num_epochs):
    # train_epoch(...)
    scheduler.step()
    lr_schedule.extend(scheduler.get_last_lr())

# plotting the simulated LR schedule
plt.figure(figsize=(5, 3))
plt.plot(range(len(lr_schedule)), lr_schedule)
plt.xlabel('Epochs')
plt.ylabel('Learning rate')
plt.tight_layout()
plt.show()
</code></pre>
<p>The issue with my implementation is that, the initial learning rate of the CosineAnnealingLR is being determined by the factor that I set for ConstantLR.</p>
<p>More precisely, I observe the following:<br>
with the setting,</p>
<pre><code class="lang-auto">scheduler1 = CosineAnnealingLR(optimizer, T_max=num_steps_optimizer1, eta_min=1e-4)
scheduler2 = ConstantLR(optimizer, factor=1e-3, total_iters=num_steps_optimizer2)
</code></pre>
<p>I notice the following plot:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/5081f378403360ce4407bff9100237ec6ddb1e59.png" alt="image" data-base62-sha1="bucCdLsN8V42LCWtUV6fPHeTtZn" width="490" height="290"></p>
<p>and, with the following setting (changed the eta_min in scheduler1),</p>
<pre><code class="lang-auto">scheduler1 = CosineAnnealingLR(optimizer, T_max=num_steps_optimizer1, eta_min=1e-5)
scheduler2 = ConstantLR(optimizer, factor=1e-3, total_iters=num_steps_optimizer2)
</code></pre>
<p>I notice the following plot:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8dcc8086fda98acdbb6c7f621afeb049898a36c8.png" alt="image" data-base62-sha1="kepsX3omHeftl82ZWMAkt6TDBeo" width="490" height="290"></p>
<p>So therefore, I conclude that the <code>factor</code> parameter in <code>scheduler2</code> is influencing the initial learning rate for <code>scheduler1</code>.</p>
<p>I would appreciate any insights and guidance on how to overcome this issue.</p>

ptrblck: <p>I don’t believe <code>scheduler2</code> has any influence in <code>scheduler1</code> and you would see the same effect by removing it:</p>
<pre data-code-wrap="python"><code class="lang-python">scheduler1 = CosineAnnealingLR(optimizer, T_max=num_steps_optimizer1, eta_min=1e-5, verbose=True, last_epoch=-1)

lr_schedule = []
for epoch in range(num_epochs*2):
    optimizer.step()
    scheduler1.step()
    lr_schedule.extend([optimizer.param_groups[0]['lr']])

# plotting the simulated LR schedule
plt.figure(figsize=(5, 3))
plt.plot(range(len(lr_schedule)), lr_schedule)
plt.xlabel('Epochs')
plt.ylabel('Learning rate')
plt.tight_layout()
plt.show()
</code></pre>
<p>which shows:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d49843b58c3223e1f093372b6519313e6836b11.png" alt="image" data-base62-sha1="fANygEtkCvyTmiU7kNuMT8XT2lX" width="352" height="208"></p>
<p>To be honest, I don’t fully understand why the learning rate starts at <code>eta_min</code> since it doesn’t seem to fit the posted formula in the docs:</p>
<pre data-code-wrap="python"><code class="lang-python">nmax = 0.1
nmin = 1e-4
Tmax = 72

a = []
for Tcur in range(num_epochs*2):
    a.append(nmin + 0.5 * (nmax - nmin) * (1 + np.cos(Tcur/Tmax * np.pi)))
    
plt.plot(a)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e10e0f290c9f74cd05f188f9e38a40d22053b6b.png" alt="image" data-base62-sha1="4hYualR9hXJZYY8FEc1atQvG3Jp" width="378" height="248"></p>
<p>But maybe I’m using it wrong and <a class="mention" href="/u/alband">@albanD</a> can correct me.</p>

albanD: <p>I’m afraid I don’t know <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"><br>
We would need to dive into the code.</p>