Weird learning rate pattern under combination of Adam and CosAnnealingLR at small Tmax

Seikegn: <p>Hi guys, I recently systematically compared training Point Cloud Transformer (Guo2020) for 500 epochs with Adam optimizer and with CosineAnnealingLR at varied Tmax. The selection of Tmax is [1,10,100,500]*num_batches. Before training, I checked the expected apparent learning rate patterns (as the value of optimizer.param_groups[0][‘lr’]), they look as follows.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1adfd28b97d7842e3327dc29357b0cfe22bda33a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1adfd28b97d7842e3327dc29357b0cfe22bda33a" title="Check"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1adfd28b97d7842e3327dc29357b0cfe22bda33a_2_517x345.jpeg" alt="Check" data-base62-sha1="3PJWZXGh0u3dwsUSWHLHRe9cH9M" width="517" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1adfd28b97d7842e3327dc29357b0cfe22bda33a_2_517x345.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1adfd28b97d7842e3327dc29357b0cfe22bda33a_2_775x517.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1adfd28b97d7842e3327dc29357b0cfe22bda33a_2_1034x690.jpeg 2x" data-dominant-color="F2F6F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Check</span><span class="informations">3218×2152 591 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The code snippet to reproduce the result is:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

num_batches = n
num_epochs = 500
learning_rate = 5e-5
tensor_sample = torch.tensor([1.0],requires_grad=True)
optimizer = torch.optim.Adam([tensor_sample],lr=learning_rate,betas=(0.9,0.99))
factor = 1 #10,100,500
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, num_batches*factor, eta_min=0, last_epoch = -1)

for epoch in range(num_epochs):
    for batch in range(num_batches):
        scheduler.step()
    record(epoch, optimizer.param_groups[0]['lr'])
</code></pre>
<p>In the actual training process, the optimizer and schduler are set in the very same way, and also optimizer.param_groups[0][‘lr’] was recorded. However, in the results it showed that when factor is as low as 1, the apparent learning rate displayed an interesting but unexpected pattern, while higher factors displayed very the same patterns as before. Shown below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2899ff835afcfed8d4b903723417157f60164d8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f2899ff835afcfed8d4b903723417157f60164d8" title="result"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2899ff835afcfed8d4b903723417157f60164d8_2_517x292.jpeg" alt="result" data-base62-sha1="yBAphzlKGnFy79AwFmYQaYKVyly" width="517" height="292" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2899ff835afcfed8d4b903723417157f60164d8_2_517x292.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2899ff835afcfed8d4b903723417157f60164d8_2_775x438.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2899ff835afcfed8d4b903723417157f60164d8_2_1034x584.jpeg 2x" data-dominant-color="F8F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">result</span><span class="informations">4110×2326 636 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any idea on the cause? Thanks.</p>