Understanding step lr schedule

James_W: <p>I am looking at some code from fb here:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/main_moco.py#L372">
  <header class="source">

      <a href="https://github.com/facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/main_moco.py#L372" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/main_moco.py#L372" target="_blank" rel="noopener nofollow ugc">facebookresearch/moco/blob/78b69cafae80bc74cd1a89ac3fb365dc20d157d3/main_moco.py#L372</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="362" style="counter-reset: li-counter 361 ;">
          <li>        entries = [self.prefix + self.batch_fmtstr.format(batch)]</li>
          <li>        entries += [str(meter) for meter in self.meters]</li>
          <li>        print('\t'.join(entries))</li>
          <li>
          </li>
<li>    def _get_batch_fmtstr(self, num_batches):</li>
          <li>        num_digits = len(str(num_batches // 1))</li>
          <li>        fmt = '{:' + str(num_digits) + 'd}'</li>
          <li>        return '[' + fmt + '/' + fmt.format(num_batches) + ']'</li>
          <li>
          </li>
<li>
          </li>
<li class="selected">def adjust_learning_rate(optimizer, epoch, args):</li>
          <li>    """Decay the learning rate based on schedule"""</li>
          <li>    lr = args.lr</li>
          <li>    if args.cos:  # cosine lr schedule</li>
          <li>        lr *= 0.5 * (1. + math.cos(math.pi * epoch / args.epochs))</li>
          <li>    else:  # stepwise lr schedule</li>
          <li>        for milestone in args.schedule:</li>
          <li>            lr *= 0.1 if epoch &gt;= milestone else 1.</li>
          <li>    for param_group in optimizer.param_groups:</li>
          <li>        param_group['lr'] = lr</li>
          <li>
      </li>
</ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>which uses stepwise lr scheduler as follows (ignoring cos lr scheduler):</p>
<pre><code class="lang-auto">def adjust_learning_rate(optimizer, epoch, args):
    """Decay the learning rate based on schedule"""
    lr = args.lr
    for milestone in args.schedule:
        lr *= 0.1 if epoch &gt;= milestone else 1.
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr
</code></pre>
<p>and the corresponding scheduler is this:</p>
<pre><code class="lang-auto">parser.add_argument('--schedule', default=[120, 160], nargs='*', type=int,
                    help='learning rate schedule (when to drop lr by 10x)')
</code></pre>
<p>my understanding from the above is that at epochs 120 and 160, the learning rate should be dropped by 10x.</p>
<p>However, I think what these lines do:</p>
<pre><code class="lang-auto">    for milestone in args.schedule:
        lr *= 0.1 if epoch &gt;= milestone else 1.
</code></pre>
<p>is to drop lr at 121,122,123… It kind of does not make sense to me <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"> or am I missing something here? I think it should be:</p>
<pre><code class="lang-auto">    for milestone in args.schedule:
        lr *= 0.1 if epoch == milestone else 1.
</code></pre>
<p>I wonder if I am missing something fundamental here?</p>

James_W: <p>apologies. I figured out why. I read the code wrong - it is using the lr from the args to slash the lr in epochs!</p>