How to train two optimizer with one loss?
I have read the GAN Demo, it is for two losses.
Suppose I have two modules A and B. The training_step is:
def training_step(self, batch, batch_idx, optimizer_idx):
        x, y = batch
        a1, a2 = self.A(x[0])
        preds = self.B(x[1], a1, a2)
        loss = loss_fn(preds, y)
        return loss

How to train A and B with different optimizer or learning rate?
The best is to use manual optimization:
https://pytorch-lightning.readthedocs.io/en/latest/common/optimizers.html#manual-optimization
Then you have control over each optimizer step.
You can get the optimzers with self.optimizers()
and for backward you simply have to replace loss.backward() with self.manual_backward(loss)
Make sure you use the latest version of Lightning for that for optimal support.
Hope this helps