Why step twice in object detection tutorial?

emcp: <p>I noticed in the detection tutorial that we have our learning rate stepped in possibly two places… and I wanted to understand why please</p>
<p><a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<pre><code class="lang-auto">    # construct an optimizer
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005,
                                momentum=0.9, weight_decay=0.0005)
    # and a learning rate scheduler
    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                                   step_size=3,
                                                   gamma=0.1)

    # let's train it for 10 epochs
    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)
</code></pre>
<p>and then within the detection reference engine we have…</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/references/detection/engine.py#L49-L50" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/references/detection/engine.py#L49-L50" target="_blank" rel="nofollow noopener">pytorch/vision/blob/master/references/detection/engine.py#L49-L50</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="49" style="counter-reset: li-counter 48 ;">
<li>if lr_scheduler is not None:</li>
<li>    lr_scheduler.step()</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">def train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq):
    model.train()
    metric_logger = utils.MetricLogger(delimiter="  ")
    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=1, fmt='{value:.6f}'))
    header = 'Epoch: [{}]'.format(epoch)

    lr_scheduler = None
    if epoch == 0:
        warmup_factor = 1. / 1000
        warmup_iters = min(1000, len(data_loader) - 1)

        lr_scheduler = utils.warmup_lr_scheduler(optimizer, warmup_iters, warmup_factor)

    for images, targets in metric_logger.log_every(data_loader, print_freq, header):
   
...

        optimizer.zero_grad()
        losses.backward()
        optimizer.step()

        if lr_scheduler is not None:
            lr_scheduler.step()

...
</code></pre>
<p>Any ideas why?</p>
<p>In the classification engine there is no such LR stepping nor warmup being performed</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/references/classification/train.py#L19" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/references/classification/train.py#L19" target="_blank" rel="nofollow noopener">pytorch/vision/blob/master/references/classification/train.py#L19</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="9" style="counter-reset: li-counter 8 ;">
<li>from torchvision import transforms</li>
<li>
</li><li>import utils</li>
<li>
</li><li>try:</li>
<li>    from apex import amp</li>
<li>except ImportError:</li>
<li>    amp = None</li>
<li>
</li><li>
</li><li class="selected">def train_one_epoch(model, criterion, optimizer, data_loader, device, epoch, print_freq, apex=False):</li>
<li>    model.train()</li>
<li>    metric_logger = utils.MetricLogger(delimiter="  ")</li>
<li>    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=1, fmt='{value}'))</li>
<li>    metric_logger.add_meter('img/s', utils.SmoothedValue(window_size=10, fmt='{value}'))</li>
<li>
</li><li>    header = 'Epoch: [{}]'.format(epoch)</li>
<li>    for image, target in metric_logger.log_every(data_loader, print_freq, header):</li>
<li>        start_time = time.time()</li>
<li>        image, target = image.to(device), target.to(device)</li>
<li>        output = model(image)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>The schedulers are not identical, i.e. a <code>StepLR</code> will be used after each epoch and a <code>warmup_lr_scheduler</code>, which can be found <a href="https://github.com/pytorch/vision/blob/7a2d06186ed592311dc372b861dfa23b1d24597c/references/detection/utils.py#L239-L247">here</a>, will be used inside the training loop.</p>

emcp: <p>care to give a bit of background on why this is?</p>
<p>Is it that the training and scoring/evaluating both use StepLR. What does a warm_lr_scheduler do?  Had not see this anywhere before</p>

Seo: <p>The warmup technique is used during the first phase of the training. It consists in small learning rate and a scheduler that increases it everytime until a certain number of steps (warmup steps). So you need 2 different scheduler, as you can see from the code you posted, the warmup scheduler seems to be used only during the first epoch.</p>
<p>reference: <a href="https://datascience.stackexchange.com/questions/55991/in-the-context-of-deep-learning-what-is-training-warmup-steps" rel="nofollow noopener">https://datascience.stackexchange.com/questions/55991/in-the-context-of-deep-learning-what-is-training-warmup-steps</a></p>