Tricks for being able to use a higher learning rate

username: Luca_Pamparana
cooked: <p>I am currently training a model where the loss explodes if I use a higher learning rate. So the highest learning rate I can use is like 1e-3. The loss even goes to NaN after the first iteration, which was a bit surprising to me.</p>
<p>I thought I had some bug but the model does converge after a long time and I get meaningful predictions from it.</p>
<p>I am currently not using any regularization and no weight decay.</p>
<p>I was wondering if there is some advise on what I could try and do to push the learning rate up.</p>

username: thecho7
cooked: <p>High learning rate cannot ensure the fast training.</p>
<p>Anyway, going to NaN is not good at all. Does your model normalize the features by using Batchnorm or something? And activation functions are needed in most of cases.</p>

username: KFrank
cooked: <p>Hi Luca!</p>
<aside class="quote no-group" data-username="Luca_Pamparana" data-post="1" data-topic="142179" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luca_pamparana/48/14310_2.png" class="avatar"> Luca_Pamparana:</div>
<blockquote>
<p>So the highest learning rate I can use is like 1e-3. The loss even goes to NaN after the first iteration,</p>
</blockquote>
</aside>
<p>Just to be clear, this all depends on the details of your model and data.</p>
<p>Sometimes if your training is initially unstable (your “loss explodes”) it<br>
can be worth training with a small learning rate for a “warm-up” period<br>
of 10 or 100 iterations or epochs and then see if you can increase the<br>
learning rate.</p>
<p>The intuition is that your randomly-initialized weights start out, by<br>
happenstance, on the side of a steep “hill,” so gradient descent has<br>
you taking excessively large steps that just move you to other bad<br>
places in configuration space – jumping across “valleys,” and such.</p>
<p>If you train with a small learning rate for a while, you move downhill<br>
into the “valley” where the weights are more sensible, the gradients<br>
are not so large, and you can safely increase your learning rate.</p>
<p>(Note, it is also often recommended – whether or not you start out<br>
with a smaller warm-up learning rate – to train for a while with a larger<br>
learning rate so that you minimize your loss with respect to the coarse<br>
topography of the loss landscape, and then train further with with a<br>
smaller learning rate so that you can minimize your loss within the<br>
finer feature of the loss landscape.  In general, this is what learning-rate<br>
schedulers are about.)</p>
<p>Although using “momentum” is not the same as increasing the learning<br>
rate, it does have some similarities.  Try using momentum with your<br>
optimizer and see if you training progresses more quickly.  You could<br>
also experiment with the <a href="https://pytorch.org/docs/1.10.1/generated/torch.optim.Adam.html" rel="noopener nofollow ugc">Adam</a> (“adaptive momentum”) optimizer.  It<br>
can be less stable than plain gradient descent but in many problems<br>
it can speed up training dramatically.  (You will need to adjust your<br>
learning rate if you add momentum or switch to <code>Adam</code> because the<br>
learning rate and momentum terms interact with one another.)</p>
<p>If it’s not too hard for you to modify your model, you could try adding<br>
<a href="https://pytorch.org/docs/1.10.1/generated/torch.nn.BatchNorm2d.html" rel="noopener nofollow ugc">BatchNorm</a> to some of your layers.  I haven’t experimented with this<br>
myself, but some of the lore says that using BatchNorm will let you<br>
increase your learning rate.</p>
<p>Good luck!</p>
<p>K. Frank</p>