Multiple optimizers but only one loss
Hey! I have a question regarding this library. Really, like how it forces me to structure my code better. I encountered one problem I did not know how to solve based on the documentation.
Let's say I have two optimizers for two parts of the network, e.g. my configure_optimizers() looks like this:
    def configure_optimizers(self):
        optimizer_encoder = optim.Adam(self.encoder.parameters(), ...)
        optimizer_decoder = optim.Adam(self.decoder.parameters(), ...)
        return [optimizer_encoder, optimizer_decoder]

now in the training loop I forward pass the encoder, then the decoder and compute my loss based on the output:
    def training_step(self, batch, batch_nb, optimizer_idx):
        inp, gt = ...

        encoding = self.encoder(inp)
        pred = self.decoder(encoding)

        loss = F.mse_loss(pred, gt)
        return {'loss': loss}

Since I have two optimizers I have to respect that this function is called two times with different optimizer_idx however I just have one loss to backprop. How would I go about this?
What have you tried?
I tried something like this
    def training_step(self, batch, batch_nb, optimizer_idx):
        if optimizer_idx == 1:
              return {}
        inp, gt = ...

        encoding = self.encoder(inp)
        pred = self.decoder(encoding)

        loss = F.mse_loss(pred, gt)
        return {'loss': loss}

However, this leads to an error since no loss key is present in trainer.py:1392.
in that case, just pass in both sets of params to a single optimizer