About LR range Test y OneCycleScheduler

username: Jona
cooked: <p>Hi everyone,</p>
<p>I read the paper about LR range test</p><aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1506.01186.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1506.01186.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1506.01186.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1506.01186.pdf" target="_blank" rel="noopener nofollow ugc">1506.01186.pdf</a></h3>

  <p class="filesize">965.07 KB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>And the one about OneCyclePolicy</p><aside class="onebox pdf" data-onebox-src="https://arxiv.org/pdf/1708.07120.pdf">
  <header class="source">

      <a href="https://arxiv.org/pdf/1708.07120.pdf" target="_blank" rel="noopener nofollow ugc">arxiv.org</a>
  </header>

  <article class="onebox-body">
    <a href="https://arxiv.org/pdf/1708.07120.pdf" target="_blank" rel="noopener nofollow ugc"><span class="pdf-onebox-logo"></span></a>

<h3><a href="https://arxiv.org/pdf/1708.07120.pdf" target="_blank" rel="noopener nofollow ugc">1708.07120.pdf</a></h3>

  <p class="filesize">2.88 MB</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I copied this implementation of the LR range test</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html">
  <header class="source">

      <a href="https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html" target="_blank" rel="noopener nofollow ugc" title="04:15PM - 20 March 2018">Another data science student's blog – 20 Mar 18</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://sgugger.github.io/how-do-you-find-a-good-learning-rate.html" target="_blank" rel="noopener nofollow ugc">How Do You Find A Good Learning Rate</a></h3>

  <p>This is the main hyper-parameter to set when we train a neural net, but how do you determine the best value? Here's a technique to quickly decide on one.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I’ve also documented myself about de OneCyclePolicy</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://sgugger.github.io/the-1cycle-policy.html">
  <header class="source">

      <a href="https://sgugger.github.io/the-1cycle-policy.html" target="_blank" rel="noopener nofollow ugc" title="03:23PM - 07 April 2018">Another data science student's blog – 7 Apr 18</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://sgugger.github.io/the-1cycle-policy.html" target="_blank" rel="noopener nofollow ugc">The 1cycle policy</a></h3>

  <p>Properly setting the hyper-parameters of a neural network can be challenging, fortunately, there are some recipe that can help.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><a href="https://towardsdatascience.com/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6" class="onebox" target="_blank" rel="noopener nofollow ugc">https://towardsdatascience.com/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6</a></p>
<p>It seems to be something many people wonder:</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/54553388/have-i-implemented-implemenation-of-learning-rate-finder-correctly">
  <header class="source">

      <a href="https://stackoverflow.com/questions/54553388/have-i-implemented-implemenation-of-learning-rate-finder-correctly" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/470184/blue-sky" target="_blank" rel="noopener nofollow ugc">
    <img alt="blue-sky" src="https://i.stack.imgur.com/DzXRW.jpg?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/54553388/have-i-implemented-implemenation-of-learning-rate-finder-correctly" target="_blank" rel="noopener nofollow ugc">Have I implemented implemenation of learning rate finder correctly?</a>
</h4>

<div class="tags">
  <strong>deep-learning, computer-vision, pytorch, mnist</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/470184/blue-sky" target="_blank" rel="noopener nofollow ugc">
    blue-sky
  </a>
  on <a href="https://stackoverflow.com/questions/54553388/have-i-implemented-implemenation-of-learning-rate-finder-correctly" target="_blank" rel="noopener nofollow ugc">12:12PM - 06 Feb 19 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/59996859/how-to-use-pytorch-onecyclelr-in-a-training-loop-and-optimizer-scheduler-intera">
  <header class="source">

      <a href="https://stackoverflow.com/questions/59996859/how-to-use-pytorch-onecyclelr-in-a-training-loop-and-optimizer-scheduler-intera" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/5688752/genec" target="_blank" rel="noopener nofollow ugc">
    <img alt="GeneC" src="https://www.gravatar.com/avatar/9586a34fd19b11311ee8ec1771d5fc31?s=256&amp;d=identicon&amp;r=PG&amp;f=y&amp;so-version=2" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/59996859/how-to-use-pytorch-onecyclelr-in-a-training-loop-and-optimizer-scheduler-intera" target="_blank" rel="noopener nofollow ugc">How to use Pytorch OneCycleLR in a training loop (and optimizer/scheduler interactions)?</a>
</h4>

<div class="tags">
  <strong>pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/5688752/genec" target="_blank" rel="noopener nofollow ugc">
    GeneC
  </a>
  on <a href="https://stackoverflow.com/questions/59996859/how-to-use-pytorch-onecyclelr-in-a-training-loop-and-optimizer-scheduler-intera" target="_blank" rel="noopener nofollow ugc">01:31AM - 31 Jan 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p><a href="https://mc.ai/super-convergence-with-just-pytorch/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://mc.ai/super-convergence-with-just-pytorch/</a></p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/62917353/how-does-one-use-torch-optim-lr-scheduler-onecyclelr">
  <header class="source">

      <a href="https://stackoverflow.com/questions/62917353/how-does-one-use-torch-optim-lr-scheduler-onecyclelr" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/13311131/ayushman-buragohain" target="_blank" rel="noopener nofollow ugc">
    <img alt="Ayushman Buragohain" src="https://lh3.googleusercontent.com/a-/AOh14Gi6uw0zxYGkNuotqsRN55kA491YrCdcgj-V7q1y9A=k-s256" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/62917353/how-does-one-use-torch-optim-lr-scheduler-onecyclelr" target="_blank" rel="noopener nofollow ugc">How does one use torch.optim.lr_scheduler.OneCycleLR()?</a>
</h4>

<div class="tags">
  <strong>python-3.x, deep-learning, pytorch, torchvision</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/13311131/ayushman-buragohain" target="_blank" rel="noopener nofollow ugc">
    Ayushman Buragohain
  </a>
  on <a href="https://stackoverflow.com/questions/62917353/how-does-one-use-torch-optim-lr-scheduler-onecyclelr" target="_blank" rel="noopener nofollow ugc">02:33PM - 15 Jul 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="quote quote-modified" data-post="1" data-topic="59939">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cthnguyen/48/14557_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/lr-parameter-in-optimizer-larger-than-max-lr-in-onecyclelr/59939">LR parameter in optimizer larger than max_lr in OneCycleLr</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Hello, 
I made a little mistake in my code (I think at least …), where I put the following : 
# Optimizer
wd = 0.01
optimizer = optim.Adam(model.parameters(), lr=10e-4, weight_decay=wd)

# LR scheduler
scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=1e-4, steps_per_epoch=len(train_loader), epochs=epochs, pct_start=0.3) # only available on pytorch v1.3.0

Notice that in the optimizer I put a learning rate of 10e-4, whereas I put the max_lr on OneCycleLR to 1e-4. So in this situation w…
  </blockquote>
</aside>

<p>Suppose I have implemented correctly a LR Test</p>
<ul>
<li>How is the correct orchestation between the optimizer and the one cycle scheduler? Because, the optimizer has the param <code>lr</code>, and the scheduler <code>max_lr</code>?</li>
<li>Can somebody provide me some example?, since I’ve been looking for it with no success</li>
</ul>
<p>Thanks</p>