learning rate not restored properly when using a CosineAnnealingLR scheduler
🐛 Bug
When resuming training from a saved checkpoint, learning rate is not restored. It causes the learning rate to follow incorrect curve. The issue is most prominent when using a multiplicative LR scheduler (ie. torch.optim.lr_scheduler.CosineAnnealingLR, instead of assigning a new value it multiplies the old one by a specific factor), but I've encountered similar behavior with other schedulers too. An example below:

To Reproduce
Create a model with torch.optim.lr_scheduler.CosineAnnealingLR. Train for several epochs, checkpoint, resume training from the checkpoint. The learning rate should then follow similar (incorrect) curve as in the example above.