Cyclical learning rate

smu226: <p>Hello! I am a bit confused about using <code>torch.optim.lr_scheduler.CyclicLR</code>. I see that by default it has 2000 steps up and 2000 down. This means (if i understand it right), that for this built-in case I need 4000 iterations in my code. So if I have more or less then that, I have to adjust the numbers by hand, such that the sum of steps up + steps down = number of iterations, right? Also, what happens if I keep it like this but I have more than 4000 iterations, do I get an error or the rest of the code is ran with the final value of the LR i.e. the smallest one. Then, I know that in the paper that introduced the cyclical LR, after the down step, the LR was going even further down by few order of magnitude, for something like 10% of the total number of iterations. How can I set this parameter? I see no parameter for this percentage? Thank you!</p>

basingse: <p>You will not get any error if you don’t keep <code>steps up + steps down = number of iterations</code>. Paper suggest to do this (<code>step_size = 2*iterations</code>) to get good results. The answer to second question is , if you use <code>mode = 'triangular2'</code> your ampltiude decreases by half evey time. You can also use <code>exp_range</code>.<br>
You can read more this <a href="https://github.com/bckenstler/CLR#policies" rel="nofollow noopener">here</a>.</p>

smu226: <p>Thank you! For the second question, I was actually wondering if I can do something like this: <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b2665bb1da2fd29501475ec2618c20c0a9408155.png" data-download-href="https://discuss.pytorch.org/uploads/default/b2665bb1da2fd29501475ec2618c20c0a9408155" title="37"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b2665bb1da2fd29501475ec2618c20c0a9408155_2_690x476.png" alt="37" data-base62-sha1="psci37xyRZGk9QHNzoaOIEijgPP" width="690" height="476" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b2665bb1da2fd29501475ec2618c20c0a9408155_2_690x476.png, https://discuss.pytorch.org/uploads/default/original/2X/b/b2665bb1da2fd29501475ec2618c20c0a9408155.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/b2665bb1da2fd29501475ec2618c20c0a9408155.png 2x" data-dominant-color="F9FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">37</span><span class="informations">961×664 72.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So after the “normal” cycle is over, the LR gets lower by a few orders of magnitude for a few more iterations.</p>

basingse: <aside class="quote no-group quote-modified" data-username="smu226" data-post="1" data-topic="53525" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smu226/48/9063_2.png" class="avatar"> smu226:</div>
<blockquote>
<p>So after the “normal” cycle is over, the LR gets lower by a few orders of magnitude for a few more iterations.</p>
</blockquote>
</aside>
<p>what happens after getting lowered ? , it will start new cycle ?</p>

smu226: <p>No, it stops. It’s the one cycle policy from Leslie Smith’s paper. The plot shows the evolution of the LR over the whole training</p>

basingse: <p>I am sorry but I don’t know how to implement this in pytorch.</p>