question: How to train two optimizer with one loss?
context: I have read the GAN Demo, it is for two losses.
  Suppose I have two modules A and B. The training_step is:
  def training_step(self, batch, batch_idx, optimizer_idx):
          x, y = batch
          a1, a2 = self.A(x[0])
          preds = self.B(x[1], a1, a2)
          loss = loss_fn(preds, y)
          return loss
  
  How to train A and B with different optimizer or learning rate?
answer: The best is to use manual optimization:
  https://pytorch-lightning.readthedocs.io/en/latest/common/optimizers.html#manual-optimization
  Then you have control over each optimizer step.
  You can get the optimzers with self.optimizers()
  and for backward you simply have to replace loss.backward() with self.manual_backward(loss)
  Make sure you use the latest version of Lightning for that for optimal support.
  Hope this helps