Reduce LR On Plateau after validation epoch when val_check_interval < 1
🚀 Feature
Add the possibility to update learning rates after every validation epochs instead of training steps/epochs.
Motivation
I am currently training a model on a very large dataset for which I like to use val_check_interval < 1.0 to have intermediary validation steps during a given epoch.
I also like to use ReduceLROnPlateau and monitor a metric computed on the validation set.
Currently it seems the ReduceLROnPlateau only works if you call it at the end of an epoch (interval="epoch",val_check_interval=1.0) or during training steps modulo the inputed frequency). This leads to updating the LR between validation epochs, when the validation metric is not modified.
If your patience is lower than the number of steps between two validation epochs the learning rate will be updated.
Pitch
To enable this we could add an interval flag to the current ones : "step", "epoch" which could be named "val_epoch".
After on_validation_epoch_end, trainer.optimizer_connector.update_learning_rates(interval="val_epoch") would be called.
Alternatives
Currently, to update the learning rates after the validation epochs called during a training epoch I use a Callback :
class UpdateLRAfterValidation(Callback):
    def __init__(self):
        pass
    def on_validation_epoch_end(self, trainer, pl_module):
        trainer.optimizer_connector.update_learning_rates(interval="epoch")

Although I am not sure if this is called after my lightning module's on_validation_epoch_end during which I log the validation metrics.