Adaptive learning rate

davidenitti: <p>How do I change the learning rate of an optimizer during the training phase?</p>
<p>thanks</p>

trypag: <p>See next comment to match <a class="mention" href="/u/apaszke">@apaszke</a> observation</p>

apaszke: <p>No! This doesn’t work anymore! The example has been updated to match the new semantics of <code>state_dict</code>.</p>
<p>If you want to change the LR we recommend reconstructing the optimizer with new parameters.</p>

trypag: <p>Oh, ok sorry ! things move fast</p>
<pre><code class="lang-python">def adjust_learning_rate(optimizer, epoch):
    """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
    lr = args.lr * (0.1 ** (epoch // 30))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

</code></pre>

ecolss: <p>Learning rate decay is a common need during model training, right?<br>
So we don’t have this in current Pytorch optim?<br>
As you recommend, I wonder reconstructing the optimizer with new parameters would bring in some performance overhead, although it would be very small compared to the whole training time?</p>

apaszke: <p><a class="mention" href="/u/ecolss">@ecolss</a> creating optimizer is nearly free (order of ms), especially that you do it at most once per epoch, and you only often have only few epochs in jobs that take a number of days to complete. Don’t optimize prematurely.</p>

ruotianluo: <p><a class="mention" href="/u/apaszke">@apaszke</a> However if you create a new optimizer, the previous momemtum (or other states) will be discarded.</p>

lysuhin: <p>That’s the point. Maybe there’s a good way to save current state of optimizer, call the constructor for optimizer with the new value of <code>lr</code> and assign old state values to the new one. In this case, we will have just a copy of old optimizer with <code>lr</code> adjusted.</p>

mderakhshani: <p>Have you found any way to update learning rate? It is so critical for me to update the <code>lr</code>.</p>

FuriouslyCurious: <p><a class="mention" href="/u/apaszke">@apaszke</a> <a class="mention" href="/u/smth">@smth</a></p>
<p>Keras provides two functions which are fairly straightforward to implement, and everyone loves them:</p>
<p>This one reduces LR when gradient is stuck on a plateau for past “X=patience” epochs:</p>
<blockquote>
<p><code>ReduceLROnPlateau(monitor='loss_value', factor=np.sqrt(0.1), cooldown=0, patience=10, min_lr=0.5e-6, verbose=1)</code></p>
</blockquote>
<p>This one stops you from burning up your Amazon AWS $$$ credits if your model is not learning anything after “X=patience” epochs:</p>
<blockquote>
<p><code>EarlyStopping(monitor='accuracy_value', min_delta=0.0001, patience=15, verbose=1)</code></p>
</blockquote>
<p>Would PyTorch be open to adding something like this?</p>

ecolss: <p>Any update on this thread about learning rate decay?</p>

will: <p><a class="mention" href="/u/apaszke">@apaszke</a> the concern here is definitely about losing the running averages, etc maintained by the optimizer. As you say the cost of constructing it is negligible.</p>
<p>In the models I’m training right now I see an increase in the loss when I construct a new optimizer to decay the learning rate.</p>

Rongzhao_Zhan: <p>You can change lr in this way as <a class="mention" href="/u/trypag">@trypag</a> mentioned[quote=“trypag, post:4, topic:320”]<br>
for param_group in optimizer.param_groups:<br>
&lt;\t&gt;param_group[‘lr’] = lr<br>
[/quote]<br>
no state loss anymore</p>

chsasank: <p>Hi,</p>
<p>I have asked <a class="mention" href="/u/apaszke">@apaszke</a> about this on slack. See <a href="https://pytorch.slackarchive.io/beginner/page-2/ts-1491917484294546" rel="noopener nofollow ugc">this</a>.</p>
<blockquote>
<p><strong>chsasank</strong> Apr 11, 2017 18:11<br>
Hi <a class="mention" href="/u/apaszke">@apaszke</a> , can you clear how to change LR. You have given different answers:</p>
<p>Is there any way to decay the learning rate for optimisers? (slack)</p>
<blockquote>
<p>check out the imagenet example (This uses param_groups)</p>
</blockquote>
<p><a href="https://discuss.pytorch.org/t/adaptive-learning-rate/320/3" class="inline-onebox">Adaptive learning rate - #3 by apaszke</a></p>
<blockquote>
<p>If you want to change the LR we recommend reconstructing the optimizer with new parameters.</p>
</blockquote>
</blockquote>
<blockquote>
<p><strong>apaszke</strong> Apr 11, 2017 19:01<br>
both ways are ok. second one is simpler, but will clear momentum buffers + if you use ada* optimizers your model might blow up if your default lr is too large<br>
now I’m leaning towards the first one</p>
</blockquote>
<p>I hope this clears things up.</p>

apaszke: <p>It’s likely that the lr was lowered by the optimizer and the new one doesn’t know about it, so it applies too large updates for a few iterations. You can always safely modify the <code>parameter_groups</code> dict, but I guess we’ll need to figure out a better way.</p>

will: <p>Yeah, a cleaner method would be appreciated. I’m using <a class="mention" href="/u/trypag">@trypag</a>’s method now, but it seems brittle to modify the internals of the optimizer and redundant for everyone to have to write this themselves.</p>

lliu25: <p>How about more general situations where it is desired to adaptively adjust any parameters defined in the model? Is it practical to reconstruct optim in every loop? Maybe a mechanism like placeholder?</p>
<p>Thanks!</p>

smth: <p><a class="mention" href="/u/lliu25">@lliu25</a> in that case, define a python function that changes the parameter_groups as you wish (after every iteration or epoch). Placeholders are unnecessary.</p>

lliu25: <p><a class="mention" href="/u/smth">@smth</a> True.  That does put placeholder extraneous.</p>

ncullen93: <p>hey not to spam but I implemented these two callbacks (and more) which are available at <a href="https://github.com/ncullen93/torchsample" rel="nofollow noopener">this repository</a>, and can be used in any kind of training loop. See <a href="https://discuss.pytorch.org/t/supermodule-for-keras-like-training-with-callbacks-constraints-and-progress-bar/2075">this thread</a> for further discussion on it</p>