Cosine Learning Rate Decay

username: Jacky_Wang
cooked: <p>Hi, guys. I am trying to replicate the <code>torch.optim.lr_scheduler.CosineAnnealingLR</code>. Which looks like:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4de6d50a138c7a17184d147099687881d51eb9d1.png" alt="image" data-base62-sha1="b79jTxtoYnrboQT8HIjw4Z1ciLn" width="452" height="347"><br>
However, if I implement the formula mentioned in the docs, which is:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e1700a20a02fd92141ae8657d40beabd035da67.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e1700a20a02fd92141ae8657d40beabd035da67" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e1700a20a02fd92141ae8657d40beabd035da67.png" alt="image" data-base62-sha1="4ibBC5G6eZRDqeIsCz2TEHSwA9F" width="690" height="125" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">791×144 12.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It is simply up-moved cosine function, instead of the truncated one above.</p>
<pre><code class="lang-auto">import numpy as np
from matplotlib import pyplot as plt
import math
lmin=0.001
lmax=0.01
tmax=50
x=[i for i in range(200)]
y=[lmin+0.5*(lmax-lmin)*(1+math.cos(i*math.pi/tmax)) for i in range(200)]
for i in range(200):
  if (i/tmax)%2==1:
    y[i+1]=y[i]+0.5*(lmax-lmin)*(1-math.cos(1/tmax))
    # pass
plt.plot(x,y)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67d95e04769512b4994af232e0cc17cf609768e7.png" alt="image" data-base62-sha1="eOGSYimYomawQOg4aPit8pqNcCb" width="481" height="250"></p>
<p>I wonder if there’s anything wrong with my code?</p>

username: ptrblck
cooked: <p>You might want to use <code>CosineAnnealingWarmRestarts</code> as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">optimizer = torch.optim.SGD([nn.Parameter(torch.randn(1, 1))], lr=1.)
T_0 = 10

scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0)

lrs = []
for epoch in range(50):
    scheduler.step(epoch)
    print(scheduler.get_last_lr())
    lrs.append(scheduler.get_last_lr())

lrs = np.array(lrs)
plt.plot(lrs)
</code></pre>
<p>Output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2cbe647466acbeaf91595da5e2622176ba28aafd.png" alt="image" data-base62-sha1="6nOVceCiimiIxyndDvAL7t34PiB" width="372" height="248"></p>

username: Jacky_Wang
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="145276">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">optimizer = torch.optim.SGD([nn.Parameter(torch.randn(1, 1))], lr=1.)
T_0 = 10

scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0)

lrs = []
for epoch in range(50):
    scheduler.step(epoch)
    print(scheduler.get_last_lr())
    lrs.append(scheduler.get_last_lr())

lrs = np.array(lrs)
plt.plot(lrs)
</code></pre>
</blockquote>
</aside>
<p>Thank you, Mr. Patrick. I finally figured out that <code>T_cur</code> represents the epochs since last restart, instead of the accumulated epochs. In my code,</p>
<pre><code class="lang-auto">y=[lmin+0.5*(lmax-lmin)*(1+math.cos(i*math.pi/tmax)) for i in range(200)]
</code></pre>
<p>should be changed to</p>
<pre><code class="lang-auto">y=[lmin+0.5*(lmax-lmin)*(1+math.cos((i%tmax)*math.pi/tmax)) for i in range(200)]
</code></pre>
<p>By the way, do you think it would be a good idea to gradually decay <code>eta_max</code> during training (maybe directly revert to the original <code>eta_max</code> might break the suboptimal to much)?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Jacky_Wang" data-post="3" data-topic="145276">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jacky_wang/48/29933_2.png" class="avatar"> Jacky_Wang:</div>
<blockquote>
<p><code>By the way, do you think it would be a good idea to gradually decay </code>eta_max<code>during training (maybe directly revert to the original</code>eta_max<code> might break the suboptimal to much)?</code></p>
</blockquote>
</aside>
<p>Oh, I don’t know as I’m not experienced enough with these learning rate scheduler schemes, so let’s wait for an expert to chime in. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: Jacky_Wang
cooked: <p>Thanks Sir for the reply and the help. I will try to carry out experiments to verify my conjecture <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: L-Reichardt
cooked: <p>An interesting questions. I found a <a href="https://linuxtut.com/en/b554744859c014f55145/" rel="noopener nofollow ugc">link online which describes chained learning rate schedulers</a>. Maybe this can be used to realize a decaying <code>eta_max</code> by combining CosineWR with something like Exponential decay.</p>