Importance of Optimizers when continuing training

username: Epoching
cooked: <p>How important is it to use the same optimizer when continuing training?</p>
<ul>
<li>
<p>If I train a model with an Adam Optimizer wrapped in a OneCycle LR Schedule, for let’s say ~10 epochs.</p>
</li>
<li>
<p>Then decide to fine-tune the model and I make a new optimizer with different settings (e.g. learning rate), and train for let’s say another ~10 epochs.</p>
</li>
</ul>
<p>Example:</p>
<pre><code class="lang-auto"># initial model, opt, and scheduler.
my_model = torchvision.models.resnet18()
my_optim = optim.Adam(lr=1e-3)
my_lr_scheduler = optim.lr_scheduler.OneCycleLR(optimizer=my_optim, max_lr=3e-2)

# Fit/train for 10 epochs
fit()

# Fine-tune model with a smaller lr.
my_optim = optim.Adam(lr=1e-5)
my_lr_scheduler = optim.lr_scheduler.OneCycleLR(optimizer=my_optim, max_lr=3e-4)

# Fit/train for 10 more epochs
fit()
</code></pre>
<p>Will this perform worse compared to changing the original optimizer’s hyperparameters manually via <code>my_optim.param_groups[0]['lr']</code>.</p>
<p>I noticed that an optimizer’s state dict, <code>my_optim.state_dict()</code>, holds two keys: <code>state</code> and <code>param_groups</code>. Making a new optimizer in the example above will make the <code>param_groups</code> hold the right values, but the <code>state</code> is lost. The below example maintains <code>state</code></p>
<p>Example:</p>
<pre><code class="lang-auto"># initial model, opt, and scheduler.
my_model = torchvision.models.resnet18()
my_optim = optim.Adam(lr=1e-3)
my_lr_scheduler = optim.lr_scheduler.OneCycleLR(optimizer=my_optim, max_lr=3e-2)

# Fit/train for 10 epochs
fit()

# Fine-tune model with a smaller lr.
my_optim.param_groups[0]['lr'] = 1e-5
my_lr_scheduler = optim.lr_scheduler.OneCycleLR(optimizer=my_optim, max_lr=3e-4)

# Fit/train for 10 more epochs
fit()
</code></pre>
<p>Is there any resources out there showing the importance of whether or not to reuse Optimizers and their states?</p>

username: Epoching
cooked: <p>Fun little experiment to follow up with my original question: <a href="https://amarsaini.github.io/Optimizer-Benchmarks/" rel="nofollow noopener">Optimizer Benchmarks</a></p>
<p>Main conclusions from my project page/first blog:</p>
<ul>
<li>OneCycle LR &gt; Constant LR</li>
<li>
<em>Making a new optimizer</em> <strong>vs.</strong> <em>Preserving <code>state</code> and re-using the same optimizer</em> both achieve very similar performance. i.e. Discarding an optimizer’s <code>state</code> didn’t really hurt the model’s performance, with or without an LR Scheduler. Maybe the state is learned quickly.</li>
</ul>
<p><strong>Note: Conclusions are based on the Adam optimizer and OneCycle LR Scheduler. I haven’t experimented with other optimizers to see if dropping their <code>state</code> is more impactful</strong></p>
<p><em>Edit Note: I’m not proposing to always throw optimizers away, I still believe the general guideline is to use the same optimizer and keep the history, as explained below. Kindly share resources if anyone found results showing the importance of using the same optimizer <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></em></p>

username: morourke
cooked: <p>The internal workings of some optimization algorithms (some examples are Adam, its variants, BFGS, and its variants) have “memory-based” components that generally improve their performance over very naive direct gradient descent procedures. Adam uses running averages over quantities computed in previous iterations, and L-BFGS stores previous gradients in order to compute an approximate Hessian (which improves the ability of the algorithms to “step” in the correct direction).</p>
<p>When using these types of algorithms, it is detrimental to stop the optimization and then initialize a completely new optimizer, (even if it is of the same algorithm) because the “memory” effects are thrown away.</p>
<p>I’m not an expert on saving the <code>state</code> of an optimizer, but with my moderate experience using L-BFGS I do believe that initializing a new optimizer using the <code>state</code> of the original optimizer will allow you to retain the “memory” from the old one. This should hopefully lead to very similar performance to the case in which you keep the old optimizer running.</p>
<p>These are just general guidelines for how the algorithms themselves work. I’m not sure exactly how they apply to the specific problem instance that you posted above.</p>

username: Epoching
cooked: <p>I completely agree with how it’s detrimental to create new optimizers, throwing away history. I’m a bit surprised to not see much slow down/performance hinderance when doing this. It’s definitely important to continue using the same optimizer, but I’m curious to get an idea for <em>how</em> important it is <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>

username: ptrblck
cooked: <p>Often you would see a spike in your loss, if you don’t restore an optimizer with some internal states (and we have some posts about this behavior here in the discussion board).</p>
<p>Even with this spike your final loss might end up in the same value range as a complete training, but you might increase the training time.</p>

username: Epoching
cooked: <p>I see, makes sense. I’ll look around the forums further! Thanks for the tips as usual</p>

username: Wei_Wong
cooked: <p>Hi, optimizer of previous saved model cannot change learning rate for next training phrase, for example the loss is decreasing, I want to change a smaller learning rate.  Is there a way of saving model that can accomplish this? How can I save a model like pytorch pretrained model for example mobilenetv2, that I can train my model on this base, save model and use new optimizer or new learning rate like we do on pretrained model?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Wei_Wong" data-post="7" data-topic="64788">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wei_wong/48/13278_2.png" class="avatar"> Wei_Wong:</div>
<blockquote>
<p>How can I save a model like pytorch pretrained model for example mobilenetv2, that I can train my model on this base, save model and use new optimizer or new learning rate like we do on pretrained model?</p>
</blockquote>
</aside>
<p>The work flow would be the same. Once you’ve stored the <code>state_dict</code> of the pretrained model, you could create a new model instance and load the <code>state_dict</code> to it.</p>