How to increase the learning rate without using cyclical learning rates?

username: Imahn
cooked: <p>Hi,</p>
<p>I’d have a quick question: In the <a href="https://arxiv.org/pdf/1512.03385.pdf" rel="noopener nofollow ugc">ResNet paper</a>, in section 4.2, an architecture is described for the CIFAR10 dataset. When they describe using a very deep ResNet, they write:</p>
<blockquote>
<p>So we use 0.01 to warm up the training until the training error is below<br>
80% (about 400 iterations), and then go back to 0.1 and continue training.</p>
</blockquote>
<p>Therefore, I would like to have the following learning rate schedule:</p>
<ul>
<li>LR = 10^{-2} for the first two epochs,</li>
<li>LR = 10^{-1} for the rest of the training.</li>
</ul>
<p>However, I haven’t found a way to achieve exactly this (cyclical learning rates would only gradually increase the learning rate, not at once). Any help would be appreciated. (-:</p>

username: mMagmer
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.optim.lr_scheduler import ConstantLR , ReduceLROnPlateau, SequentialLR


model = nn.Sequential(nn.Linear(2,50),
                      nn.Sigmoid(),
                      nn.Linear(50,50),
                      nn.Sigmoid(),
                      nn.Linear(50,2)
                      )


criteria = nn.CrossEntropyLoss()
optim =  torch.optim.SGD(model.parameters(),lr=.1,momentum=.9)

def get_toyData(N1,N2):
    labels = torch.cat([torch.ones(N1),torch.zeros(N2)])
    data = torch.cat([torch.randn(N1,2),torch.randn(N2,2)+1.5])
    return data , labels 

N1 , N2 = 1000,1000 
train_data , train_labels = get_toyData(N1,N2)

dataset = torch.utils.data.TensorDataset(train_data,train_labels)
dl = torch.utils.data.DataLoader(dataset,batch_size=10,
                                 shuffle=True)

scheduler1 = ConstantLR(optim, factor=0.1, total_iters=40)
#scheduler2 = ExponentialLR(optim, gamma=0.9)
scheduler2 = ReduceLROnPlateau(optim, 'min')
    
for i in range(100):
    totalLoss = 0
    train_error = 0
    for x,t in dl:
        optim.zero_grad()
        out=model(x)
        loss = criteria(out,t.long())
        loss.backward()
        optim.step()
        totalLoss += loss.item()
    if i &lt;40:
        
        scheduler1.step()
    else:
        scheduler2.step(totalLoss)
    print(i , optim.state_dict()['param_groups'][0]['lr'])
        
    #print(i,totalLoss,train_error/1300)
</code></pre>
<blockquote>
<p>Assuming optimizer uses lr = 0.1 for all groups<br>
lr = 0.01     if epoch &lt; 40<br>
lr = 0.1    if epoch =40<br>
and ReduceLROnPlateau by .1  if epoch &gt; 40</p>
</blockquote>

username: Imahn
cooked: <p>First of all: Thanks! My apologies, but there’s one thing I forgot to mention in my original post, I’m afraid. Let me quote from the ResNet paper, section 4.2 again:</p>
<blockquote>
<p>So we use 0.01 to warm up the training until the training error is below<br>
80% (about 400 iterations), and then go back to 0.1 and continue training. The rest of the learning schedule is as done previously.</p>
</blockquote>
<p>By “as done previously”, this is meant:</p>
<blockquote>
<p>We start with a learning rate of 0.1, divide it by 10 at 32k and 48k iterations, and<br>
terminate training at 64k iterations, […].</p>
</blockquote>
<p>Basically, something like this should be our learning rate schedule:<br>
lr = 0.01 if num_iters &lt; 400,<br>
lr = 0.1 if 400 &lt;= num_iters &lt; 32k,<br>
lr = 0.01 if 32k &lt;= num_iters &lt; 48k,<br>
lr = 0.001 if 48k &lt;= num_iters &lt; 60k.</p>
<p>I’d also be happy to use epochs instead of number of iterations, but I’m not sure how to achieve either. )-:</p>

username: mMagmer
cooked: <p>hi,</p>
<pre><code class="lang-auto">scheduler1 = ConstantLR(optim, factor=0.1, total_iters=400)
scheduler2 = MultiStepLR(optim, milestones=[32000,48000], gamma=0.1)

# in training inner loop
for j, x,t in enumerate(dl):
        optim.zero_grad()
        out=model(x)
        loss = criteria(out,t.long())
        loss.backward()
        optim.step()
        totalLoss += loss.item()
        if i*len(dl) + j &lt;400:
            scheduler1.step()
            scheduler2.step()
        else:
            scheduler2.step()
</code></pre>
<p>I’m not sure  that <code>if</code> statement is necessary anymore.<br>
also, You can use <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.LambdaLR.html#torch.optim.lr_scheduler.LambdaLR" rel="noopener nofollow ugc">LAMBDALR</a> to do anything you want.</p>

username: Imahn
cooked: <p>Hi! Actually, I think that the if-statement is indeed not needed:</p>
<pre><code class="lang-python">scheduler1 = ConstantLR(optimizer, factor=0.01, total_iters=400)
scheduler2 = MultiStepLR(optimizer, milestones=[32000, 48000], gamma=0.1)
chained_scheduler = ChainedScheduler([scheduler1, scheduler2])

for (batch_idx, data, labels) in enumerate(train_loader): 
     ...
    chained_scheduler.step()
</code></pre>
<p>If I can ask a brief follow-up question: I had actually tried using the LambdaLR for this problem as well, but at the end, I didn’t really know how to implement it… I’d be happy if you showed me how! (-:</p>

username: mMagmer
cooked: <p>this should work.</p>
<pre><code class="lang-auto">def _lr_lambda(current_step):
        """
        _lr_lambda returns a multiplicative factor given an interger parameter epochs.
        """

        if current_step &lt; 400:
            _lr =.1
        elif current_step &lt; 32000:
            _lr = 1
        elif current_step &lt; 48000:
            _lr = .1
        else:
            _lr = .01

        return _lr

scheduler = LambdaLR(optimizer, _lr_lambda, last_epoch)
</code></pre>