Request for Feature that Switches Optimizer After Validation Loss Stops Improving
🚀 Feature
It will be awesome if we can configure Lightning to switch from one optimizer to the second optimizer (e.g. Adam to SGD) when the validation loss stops improving for a certain number of validation epochs.
Kind of like EarlyStopping and its patience parameter, but switches optimizer instead of stop training.
Motivation

Adam is known to converge faster than SGD, while SGD is typically considered to generalize better than Adam. This approach of switching optimizers in the middle of training is starting to become more common.
Pitch

Alternatives

Additional context


#6796 (comment)