Get the best learning rate automatically

username: shirui-japina
cooked: <p>It is very difficult to adjust the best hyper-parameters in the process of studying the deep learning model.<img src="https://discuss.pytorch.org/images/emoji/apple/cold_face.png?v=9" title=":cold_face:" class="emoji" alt=":cold_face:"><img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=9" title=":scream:" class="emoji" alt=":scream:"><br>
Is there some great function in PyTorch to get the best learning rate?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

username: albanD
cooked: <p>I’m sure that if you find such function, your easily get the best paper award at NeurIPS !<br>
This is an active area of research so no solution exists <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: shirui-japina
cooked: <p>That’s right.<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"><br>
Thanks for the best answer.<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

username: dejanbatanjac
cooked: <p>Hi <a class="mention" href="/u/shirui-japina">@shirui-japina</a>,</p>
<p>There is actually a guy called <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Smith%2C+L+N" rel="noopener nofollow ugc">Leslie N. Smith</a> who created this <a href="https://arxiv.org/abs/1506.01186" rel="noopener nofollow ugc">paper</a>.</p>
<p>Based on this paper, some other guy created the <a href="https://docs.fast.ai/callbacks.lr_finder.html" rel="noopener nofollow ugc">learning rate finder</a>.</p>
<p>What, it does, it measures the loss for the different learning rates and plots the diagram as this one:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc36be7a2d3c061ef52a97aa577848cd46f828a1.png" alt="image" data-base62-sha1="vq6kAy3la8BBmahxe5xmAhfH3O1" width="424" height="274"></p>
<p>It shows up (empirically) that the best learning rate is a value that is approximately in the middle of the sharpest downward slope.</p>
<p>However, the modern practice is to <em>alter</em> the learning rate while training described in <a href="https://docs.fast.ai/callbacks.one_cycle.html" rel="noopener nofollow ugc">here</a>.</p>
<p>At the end you would probable do learning rate annealing.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f49d27f459ac3e25e65b67f902d391bbc456d051.png" data-download-href="https://discuss.pytorch.org/uploads/default/f49d27f459ac3e25e65b67f902d391bbc456d051" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f49d27f459ac3e25e65b67f902d391bbc456d051_2_690x249.png" alt="image" data-base62-sha1="yTXcU1aOZYu7iMKMaHSFtHEPs2t" width="690" height="249" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f49d27f459ac3e25e65b67f902d391bbc456d051_2_690x249.png, https://discuss.pytorch.org/uploads/default/original/3X/f/4/f49d27f459ac3e25e65b67f902d391bbc456d051.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/4/f49d27f459ac3e25e65b67f902d391bbc456d051.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">730×264 16.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
[first image is the learning rate second is the momentum in time]</p>
<p>It really depends on optimization algorithm but the techniques from fastai work well on Adam and SGD.</p>
<p>Lastly some newer optimizer algorithms don’t even care about the learning rate ;).</p>

username: shirui-japina
cooked: <p>Thanks for your help.<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: shirui-japina
cooked: <p>I have thought about it for a long time and got the following questions.</p>
<blockquote>
<p>There is actually a guy called <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Smith%2C+L+N" rel="noopener nofollow ugc">Leslie N. Smith </a> who created this <a href="https://arxiv.org/abs/1506.01186" rel="noopener nofollow ugc">paper </a>.</p>
</blockquote>
<ul>
<li>
<p>But in the paper, there is only graph, which <strong>learning rate as the x-axis</strong>, and <strong>accuracy rate as the y-axis</strong>. Not like the graph you mentioned, why is the reason?</p>
</li>
<li>
<p>And at the stage of <em><strong>Choose a good learning rate</strong></em>, there is <a href="https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html" rel="noopener nofollow ugc">Sylvain Gugger’s post</a> explaining how the <a href="https://docs.fast.ai/callbacks.lr_finder.html" rel="noopener nofollow ugc">learning rate finder</a> works:</p>
</li>
</ul>
<blockquote>
<p>Over an epoch begin your SGD with a very low learning rate (like 10−810−8) but change it (by multiplying it by a certain factor for instance) at each mini-batch until it reaches a very high value (like 1 or 10). Record the loss each time at each iteration and once you’re finished, plot those losses against the learning rate.</p>
</blockquote>
<p>My question about here is:</p>
<ul>
<li>
<p>if <em><strong>change it (by multiplying it by a certain factor for instance) at each mini-batch</strong></em>,<br>
the change of loss will because of: 1. input data(each mini-batch data is different); 2. learning rate.<br>
How can we make sure the change of loss is just because of learning rate?</p>
</li>
<li>
<p>during the <a href="https://docs.fast.ai/callbacks.lr_finder.html" rel="noopener nofollow ugc">learning rate finder</a> working, will the weights in the model be updated?<br>
If yes, The loss value will become smaller because of the model learning itself.<br>
How can we know the change of loss is because of learning rate but not model learning itself?</p>
</li>
<li>
<p>What is the value of loss in <a href="https://docs.fast.ai/callbacks.lr_finder.html" rel="noopener nofollow ugc">learning rate finder</a>? Is just the average of the model output nodes?</p>
</li>
</ul>
<p>I also asked <a href="https://forums.fast.ai/t/learning-rate-finders-mathematical-theory/56738" rel="noopener nofollow ugc">here</a>, but I’m not sure if I can get an answer.</p><aside class="onebox discoursetopic" data-onebox-src="https://forums.fast.ai/t/learning-rate-finders-mathematical-theory/56738">
  <header class="source">
      <img src="https://forums.fast.ai/uploads/default/optimized/3X/b/3/b395de7a2ba00b82865031c97a8cadf3d80e71e5_2_32x32.png" class="site-icon" width="" height="">

      <a href="https://forums.fast.ai/t/learning-rate-finders-mathematical-theory/56738" target="_blank" rel="noopener nofollow ugc" title="07:11PM - 17 October 2019">fast.ai Course Forums – 17 Oct 19</a>
  </header>

  <article class="onebox-body">
    <img src="https://forums.fast.ai/uploads/default/original/3X/b/b/bb86399eaa2363a96163c3a7b757f88a403319ea.png" class="thumbnail" width="" height="">

<div class="title-wrapper">
  <h3><a href="https://forums.fast.ai/t/learning-rate-finders-mathematical-theory/56738" target="_blank" rel="noopener nofollow ugc">Learning Rate Finder's mathematical theory</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #0088CC;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">fastai</span>
        </span>
      </span>
  </div>
</div>

  <p>My understanding is,  by Learning Rate Finder, we can get an image ‘learning rate’ vs ‘loss’ like this one.    And by observing this image, we can get the best learning rate. Just as described here.  And it said,   For a more intuitive...</p>

  <p>
    <span class="label1">Reading time: 2 mins 🕑</span>
      <span class="label2">Likes: 3 ❤</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: dejanbatanjac
cooked: <aside class="quote no-group" data-username="shirui-japina" data-post="6" data-topic="58269">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/48/15285_2.png" class="avatar"> shirui-japina:</div>
<blockquote>
<p>Not like the graph you mentioned, why is the reason?</p>
</blockquote>
</aside>
<p>The paper L. Smith created did some things premature because he had a trip, and no enough time to polish the paper. I think this was the main reason.</p>
<p>I think measuring loss is better adoption in here. These are two antipodes, If loss is small the accuracy big, and opposite. If one goes up, the other goes down like so:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30c0597495e5385b465227afe2e2272d5d2fd597.png" data-download-href="https://discuss.pytorch.org/uploads/default/30c0597495e5385b465227afe2e2272d5d2fd597" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30c0597495e5385b465227afe2e2272d5d2fd597_2_398x500.png" alt="image" data-base62-sha1="6Xh1zqCZkHaiDv5aFOjhsbZeqUv" width="398" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30c0597495e5385b465227afe2e2272d5d2fd597_2_398x500.png, https://discuss.pytorch.org/uploads/default/original/3X/3/0/30c0597495e5385b465227afe2e2272d5d2fd597.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/0/30c0597495e5385b465227afe2e2272d5d2fd597.png 2x" data-dominant-color="EEF1F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">540×677 87.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<hr>
<p><a href="https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html" rel="noopener nofollow ugc">Sylvain Gugger’s post</a> was an early version of the algorithm, since then they adopted it a bit (they are using hooks now), but the principles are the same.</p>
<p>One of the rules S. Googer explained:</p>
<blockquote>
<p>The first one is that we won’t really plot the loss of each mini-batch, but some smoother version of it.</p>
</blockquote>
<p>So how to make it smooth? They used exponentially weighed averages.<br>
This is the same what people do to predict <a href="https://en.wikipedia.org/wiki/Moving_average" rel="noopener nofollow ugc">stocks</a>. EWA works as a filter and you basically use <a href="https://pytorch.org/docs/stable/torch.html#torch.lerp" rel="noopener nofollow ugc">lerp</a> in PyTorch to calc this.</p>
<p><sub>PS. The original version of the algorithm doesn’t use <code>lerp</code> though.</sub></p>
<blockquote>
<p>during the <a href="https://docs.fast.ai/callbacks.lr_finder.html" rel="noopener nofollow ugc">learning rate finder</a> working, will the weights in the model be updated?<br>
If yes, The loss value will become smaller because of the model learning itself.<br>
How can we know the change of loss is because of learning rate but not model learning itself?</p>
</blockquote>
<p>They save the model and then reload the initial state when done, check the <code>on_train_end</code>.</p>
<pre><code class="lang-auto">class LRFinder(LearnerCallback):
    "Causes `learn` to go on a mock training from `start_lr` to `end_lr` for `num_it` iterations."
    def __init__(self, learn:Learner, start_lr:float=1e-7, end_lr:float=10, num_it:int=100, stop_div:bool=True):
        super().__init__(learn)
        self.data,self.stop_div = learn.data,stop_div
        self.sched = Scheduler((start_lr, end_lr), num_it, annealing_exp)

    def on_train_begin(self, pbar, **kwargs:Any)-&gt;None:
        "Initialize optimizer and learner hyperparameters."
        setattr(pbar, 'clean_on_interrupt', True)
        self.learn.save('tmp')
        self.opt = self.learn.opt
        self.opt.lr = self.sched.start
        self.stop,self.best_loss = False,0.
        return {'skip_validate': True}

    def on_batch_end(self, iteration:int, smooth_loss:TensorOrNumber, **kwargs:Any)-&gt;None:
        "Determine if loss has runaway and we should stop."
        if iteration==0 or smooth_loss &lt; self.best_loss: self.best_loss = smooth_loss
        self.opt.lr = self.sched.step()
        if self.sched.is_done or (self.stop_div and (smooth_loss &gt; 4*self.best_loss or torch.isnan(smooth_loss))):
            #We use the smoothed loss to decide on the stopping since it's less shaky.
            return {'stop_epoch': True, 'stop_training': True}

    def on_train_end(self, **kwargs:Any)-&gt;None:
        "Cleanup learn model weights disturbed during LRFinder exploration."
        self.learn.load('tmp', purge=False)
        if hasattr(self.learn.model, 'reset'): self.learn.model.reset()
        for cb in self.callbacks:
            if hasattr(cb, 'reset'): cb.reset()
        print('LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.')
</code></pre>
<blockquote>
<p>if <em><strong>change it (by multiplying it by a certain factor for instance) at each mini-batch</strong></em> ,<br>
the change of loss will because of: 1. input data(each mini-batch data is different); 2. learning rate.<br>
How can we make sure the change of loss is just because of learning rate?</p>
</blockquote>
<p>Well, what is loss in PyTorch. It is a tensor representing a scalar value.<br>
You can write it as: <code>loss = loss_fn(y_hat, y)</code> where <code>loss_fn</code> is also know as criterion, <code>y_hat</code> is know as output and <code>y</code> is know as target. So:</p>
<p><code>loss = criterion(output, target)</code></p>
<p>But most importantly it is <strong>one batch loss</strong>.<br>
So the most primitive LR finder may work just on a same batch. First one you take from <code>DataLoader</code>.</p>
<p>However, these batch data is very close if you check std and mean, particularly if the batch size is large enough, so I think this is the reason to slide trough batches as S. Googer wrote in algorithm.</p>

username: shirui-japina
cooked: <blockquote>
<p>The paper L. Smith created did some things premature because he had a trip, and no enough time to polish the paper. I think this was the main reason.</p>
</blockquote>
<p>Ok…</p>
<blockquote>
<p>I think measuring loss is better adoption in here.</p>
</blockquote>
<p>Yes, maybe it’s something like when training the model, it is better to observe the loss value.</p>
<blockquote>
<p>They save the model and then reload the initial state when done, check the  <code>on_train_end</code> .</p>
</blockquote>
<p>So that means no updating for the weights in the model, during the <a href="https://docs.fast.ai/callbacks.lr_finder.html" rel="noopener nofollow ugc">learning rate finder</a> working, Is it right?</p>
<blockquote>
<p>However, these batch data is very close if you check std and mean, particularly if the batch size is large enough, so I think this is the reason to slide trough batches as S. Googer wrote in algorithm.</p>
</blockquote>
<p>I did not verify with my own model and input data.<br>
But that, <em><strong>these batch data is very close if you check std and mean, particularly if the batch size is large enough</strong></em>, may be right.</p>

username: dejanbatanjac
cooked: <aside class="quote no-group" data-username="shirui-japina" data-post="8" data-topic="58269">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/48/15285_2.png" class="avatar"> shirui-japina:</div>
<blockquote>
<p>So that means no updating for the weights in the model, during the <a href="https://docs.fast.ai/callbacks.lr_finder.html" rel="noopener nofollow ugc">learning rate finder</a> working, Is it right?</p>
</blockquote>
</aside>
<p>No, the weights are updated in the algorithm you can see this from the :</p>
<pre><code class="lang-auto">loss.backward()
optimizer.step()
</code></pre>
<p>part of the algorithm S. Googer created.</p>
<p><sub>However, I am not saying it is not possible to create LR finder even without updating the weights, but this would not be one cycle learning paper.</sub></p>
<blockquote>
<p>I did not verify with my own model and input data.</p>
</blockquote>
<p>Yeah, good point. We need to check everything, else we don’t know anything.<br>
You may try this exact S. Googer algorithm from his article, but just instead go trough different batches, try on a single (first) batch.</p>
<p>Hope you will get more feedback from the fastai blog.</p>

username: shirui-japina
cooked: <blockquote>
<p>No, the weights are updated in the algorithm</p>
</blockquote>
<p>Thanks… I understand it wrong.<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>part of the algorithm S. Googer created.</p>
<p>However, I am not saying it is not possible to create LR finder even without updating the weights, but this would not be one cycle learning paper.</p>
</blockquote>
<p>I got it. Maybe can do some research on this part.<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>Hope you will get more feedback from the fastai blog.</p>
</blockquote>
<p>Thanks for your reply and help for me so many times.<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

username: bdhammel
cooked: <p>Hi <a class="mention" href="/u/shirui-japina">@shirui-japina</a> I wrote a blog post which might provide a bit more insight on picking learning rates, hope it helps: <a href="http://www.bdhammel.com/learning-rates/" rel="nofollow noopener">http://www.bdhammel.com/learning-rates/</a></p>
<p>TL;DR: Just use the learning rate finder.</p>
<p>For even more in-depth understanding, I <em>strongly</em> recommend giving this blog post a read:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://296830-909578-raikfcquaxqncofqfm.stackpathdns.com/wp-content/uploads/2019/06/cropped-Favicon-32x32.jpg" class="site-icon" width="16" height="16">
      <a href="https://myrtle.ai/how-to-train-your-resnet/" target="_blank" title="05:11PM - 24 September 2018" rel="nofollow noopener">Myrtle – 24 Sep 18</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:132/79;"><img src="https://296830-909578-raikfcquaxqncofqfm.stackpathdns.com/wp-content/uploads/2018/09/intro.jpg" class="thumbnail"></div>

<h3><a href="https://myrtle.ai/how-to-train-your-resnet/" target="_blank" rel="nofollow noopener">How to Train Your ResNet</a></h3>

<p>The introduction to a series of posts investigating how to train Residual networks efficiently on the CIFAR10 image classification dataset. By the fourth post, we can train to the 94% accuracy threshold of the DAWNBench competition in 79 seconds on a...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
in particular, post <span class="hashtag">#5</span> on hyperparameters</p>