A problem occured when resuming an optimizer

username: dreamyun
cooked: <pre><code class="lang-auto">model=Net()
optimizer1=torch.optim.SGD(model.features.parameters(),lr=0.1,momentum=0.9,dampening=0.9)
scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer1,
                                                       gamma=0.999,
                                                       last_epoch=100)
</code></pre>
<p>Afther runing, a error is shown like this:&gt;---------------------------------------------------------------------------<br>
KeyError                                  Traceback (most recent call last)<br>
 in ()<br>
3 scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer1,<br>
4                                                        gamma=0.999,<br>
----&gt; 5                                                        last_epoch=100)</p>
<p>~/anaconda3/envs/pytorch4/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in <strong>init</strong>(self, optimizer, gamma, last_epoch)<br>
180     def <strong>init</strong>(self, optimizer, gamma, last_epoch=-1):<br>
181         self.gamma = gamma<br>
–&gt; 182         super(ExponentialLR, self).<strong>init</strong>(optimizer, last_epoch)<br>
183<br>
184     def get_lr(self):</p>
<p>~/anaconda3/envs/pytorch4/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in <strong>init</strong>(self, optimizer, last_epoch)<br>
18                 if ‘initial_lr’ not in group:<br>
19                     raise KeyError("param ‘initial_lr’ is not specified "<br>
—&gt; 20                                    “in param_groups[{}] when resuming an optimizer”.format(i))<br>
21         self.base_lrs = list(map(lambda group: group[‘initial_lr’], optimizer.param_groups))<br>
22         self.step(last_epoch + 1)</p>
<p>KeyError: “param ‘initial_lr’ is not specified in param_groups[0] when resuming an optimizer”</p>

username: ptrblck
cooked: <p>You are trying to initialize a new optimizer and initialize the scheduler to another <code>last_epoch</code>.<br>
As the optimizer wasn’t used in the scheduler from the beginning, the <code>param_group</code> <code>initial_lr</code> is missing.<br>
What is your exact use case?<br>
Would you like to use the scheduler as if it was already used for 100 epochs?<br>
If so you could set <code>last_epoch=-1</code> in the instantiation and call the scheduler 100 times in a dummy for loop.</p>

username: Sergius_Liu
cooked: <p>To continue that question, when we initialize a scheduler like</p>
<blockquote>
<p>scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer1,<br>
gamma=0.999,<br>
last_epoch=100)</p>
</blockquote>
<p>‘Last_epoch’ is an argument for users which means we can specify it as any number instead of -1.<br>
If we can’t even assign it to other numbers when initialize, isn’t this arg redundant?<br>
I prefer a design that can automatically specify epoch state with ‘last_epoch’ arg.</p>

username: glenn.jocher
cooked: <p>Getting this error my self in <a href="https://github.com/ultralytics/yolov3" rel="nofollow noopener">https://github.com/ultralytics/yolov3</a>. The band-aid ‘solution’ was to define the attribute after the scheduler is already defined. I’m not sure if the scheduler is actually properly initialized to the correct LR, but the code runs without errors in the second example below:</p>
<pre><code class="lang-auto">    # ERROR
    scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lf, last_epoch=start_epoch - 1)

    # NO ERROR
    scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lf)
    scheduler.last_epoch = start_epoch - 1
</code></pre>
<p>Original error message is:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 423, in &lt;module&gt;
    train()  # train normally
  File "train.py", line 152, in train
    scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lf, last_epoch=start_epoch - 1)
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py", line 189, in __init__
    super(LambdaLR, self).__init__(optimizer, last_epoch)
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py", line 41, in __init__
    "in param_groups[{}] when resuming an optimizer".format(i))
KeyError: "param 'initial_lr' is not specified in param_groups[0] when resuming an optimizer"
</code></pre>

username: Howardchen
cooked: <p>Hi <a class="mention" href="/u/glenn.jocher">@glenn.jocher</a>,  I’ve also encountered the same error in <a href="https://github.com/ultralytics/yolov3" rel="nofollow noopener">https://github.com/ultralytics/yolov3</a> when resuming training. But I examined the source code of torch.optim.lr_scheduler.LambdaLR, and found that the reason your code works is that LambdaLR takes the default value last_epoch = -1 and reset the lr. Thus lr isn’t loaded from last.pt, but reset as a new optimizer.<br>
<a href="https://s0pytorch0org.icopy.site/docs/0.2.0/_modules/torch/optim/lr_scheduler.html" rel="nofollow noopener">https://s0pytorch0org.icopy.site/docs/0.2.0/_modules/torch/optim/lr_scheduler.html</a></p>
<pre><code class="lang-auto">class LambdaLR(_LRScheduler):
    """Sets the learning rate of each parameter group to the initial lr
    times a given function. When last_epoch=-1, sets initial lr as lr.
    ......
</code></pre>
<p>Why initial_lr is missing in last.pt is because the optimizer will not be saved in the last epoch. I think that the mechanism of resume training is designed for the scheme that the training is interrupted, not for continuing after the last epoch.<br>
If we want to continue training after the last epoch, we may modify the line<br>
<code>'optimizer': None if final_epoch else optimizer.state_dict()</code><br>
to<br>
<code>'optimizer': optimizer.state_dict()</code><br>
in train.py.<br>
I’m not sure but it seems to work for me.</p>
<pre><code class="lang-auto"># Save training results
        save = (not opt.nosave) or (final_epoch and not opt.evolve)
        if save:
            with open(results_file, 'r') as f:
                # Create checkpoint
                chkpt = {'epoch': epoch,
                         'best_fitness': best_fitness,
                         'training_results': f.read(),
                         'model': model.module.state_dict() if hasattr(model, 'module') else model.state_dict(),
                         'optimizer': None if final_epoch else optimizer.state_dict()}
</code></pre>

username: glenn.jocher
cooked: <p>Ah, yes, thanks for the feedback! Yes you are correct, --resume is really only intended for accidentally stopped training. i.e. you train to 300, but your computer shuts down at 100. You can use the same exact training command you originally used plus --resume to finish the training to 300.</p>
<p>If you train to 300/300, and then decide you want to train to 400, you are out of luck, because the LR scheduler has already reduced to near zero, and defining a new number of --epochs will create a nonlinearity in the LR scheduler. In this case you should restart your training from the beginning with --epochs 400.</p>
<p>And to answer your last point, we actually remove the optimizer on purpose after complaints about file sizes, as the optimizer will double the size of the weightfile, since the file is now carrying gradients for each parameter in addition to the parameters themselves.</p>