Improve manual_optimization
🚀 Feature

Motivation
The current manual optimization is not flexible enough.
From discussion with @justusschock and @awaelchli, here is the proposal:

manual_backward shouldn't take any optimizers as they aren't used internally expect in Horovod. To solve this, in Horovod, call synchronise on all optimizers.
In LightningOptimizer, optimizer shoudn't follow accumulated_grad_batches.


Pitch

Alternatives

Additional context