"When last_epoch=-1, sets initial lr as lr."

kuraga: <p>Good day!</p>
<p>Most of learning rate schedulers have in their documentation:</p>
<blockquote>
<p>When last_epoch=-1, sets initial lr as lr.</p>
</blockquote>
<p>But what does it mean exactly?</p>
<p>Thanks!</p>

ptrblck: <p>The <code>last_epoch</code> argument is used to resume the training after <code>N</code> epochs as seen in this example:</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=1.)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=2, gamma=0.5, last_epoch=-1)

for _ in range(6):
    print(optimizer.param_groups[0]["lr"])
    optimizer.step()
    scheduler.step()
# 1.0
# 1.0
# 0.5
# 0.5
# 0.25
# 0.25
print("==============")
opt_sd = optimizer.state_dict()

# resume training
model = nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=1.)
optimizer.load_state_dict(opt_sd)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=2, gamma=0.5, last_epoch=6)

for _ in range(6):
    print(optimizer.param_groups[0]["lr"])
    optimizer.step()
    scheduler.step()
    
# 0.125
# 0.0625
# 0.0625
# 0.03125
# 0.03125
# 0.015625
</code></pre>

kuraga: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks very much!<br>
(UPD 2.)<br>
Before the next questions, is it expected that <code>0.125</code> appears once?<br>
Should we have read “last epoch” as “last finished epoch” and have written <code>last_epoch=5</code>?</p>

ptrblck: <p>Setting <code>last_epoch=5</code> will continue with:</p>
<pre data-code-wrap="python"><code class="lang-python">0.0625
0.0625
0.03125
0.03125
0.015625
0.015625
</code></pre>
<p>and entirely skip <code>0.125</code>. <a class="mention" href="/u/alband">@albanD</a> do you know what the issue in my code snippet is?</p>

kuraga: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<ol>
<li>Seems like the <code>last_epoch</code> argument just omits <code>last_epoch+1</code> steps on the scheduler object initialization.</li>
</ol>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L52-L52">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L52-L52" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L52-L52" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L52-L52</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="52" style="counter-reset: li-counter 51 ;">
          <li>self.last_epoch = last_epoch</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<ol start="2">
<li>And doesn’t matter how many steps were done before the scheduler (re-)initialization:</li>
</ol>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L42-L51">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L42-L51" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L42-L51" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L42-L51</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="42" style="counter-reset: li-counter 41 ;">
          <li># Initialize epoch and base learning rates</li>
          <li>if last_epoch == -1:</li>
          <li>    for group in optimizer.param_groups:</li>
          <li>        group.setdefault('initial_lr', group['lr'])</li>
          <li>else:</li>
          <li>    for i, group in enumerate(optimizer.param_groups):</li>
          <li>        if 'initial_lr' not in group:</li>
          <li>            raise KeyError("param 'initial_lr' is not specified "</li>
          <li>                           f"in param_groups[{i}] when resuming an optimizer")</li>
          <li>self.base_lrs = [group['initial_lr'] for group in optimizer.param_groups]</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>So, on any <code>last_epoch</code> argument value, this “scrolling” (“omitting”) starts with “initial learning rate” (literally), i.e. stepping doesn’t change it.</p>
<p>(To check this and the above one, set <code>step_size=3</code> and variate the <code>last_epoch</code> argument in your code’s second part.)</p>
<ol start="3">
<li>Keep in mind, the scheduler object initialization does the <code>.step()</code> itself:</li>
</ol>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L85-L91">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L85-L91" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L85-L91" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/v2.2.1/torch/optim/lr_scheduler.py#L85-L91</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="85" style="counter-reset: li-counter 84 ;">
          <li>    self._initial_step()</li>
          <li></li>
          <li>def _initial_step(self):</li>
          <li>    """Initialize step counts and performs a step"""</li>
          <li>    self.optimizer._step_count = 0</li>
          <li>    self._step_count = 0</li>
          <li>    self.step()</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<ol start="4">
<li>Keep in mind, numeratiion starts with <code>0</code>.</li>
</ol>
<p>(Programatically, these steps occur in the order: 2., 1., 3.)</p>
<hr>
<p>The Question is: does</p>
<blockquote>
<p>When last_epoch=-1, sets initial lr as lr.</p>
</blockquote>
<p>tell about pp. 1. or 2.?</p>
<p>What does it say about instead?</p>
<p>Does the documentation tell about these three statements?</p>
<p>Thanks!</p>

kuraga: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , <a class="mention" href="/u/alband">@albanD</a> , any thoughts? Thanks.</p>

kuraga: <p>Continued at <a href="https://github.com/pytorch/pytorch/issues/120735#issuecomment-1974862083" class="inline-onebox" rel="noopener nofollow ugc">`torch.optim.lr_scheduler.LRScheduler`: lack of documentation · Issue #120735 · pytorch/pytorch · GitHub</a>.</p>