question: predict with multiple GPUs doesn't aggregate the predictions even with on_predict_end or on_predict_batch_end
context: I train a model with 2 GPUs， when running predictions = trainer.predict(model, datamodule) in a script, what I get is not a single variable predictions as desired, instead, I get two independent variables on two GPUs. I tried to use on_predict_end and on_predict_batch_end to aggregate the predictions on different GPUs, but it seems they change the behavior of the method trainer.predict， i.e. I can't get an aggregated predicitons.  What should I do to aggregate predictions defined in predict_step when using multiple GPUs?
  
  To see the issue, simple create a LightningModule with the predict_step method
  class Model(pl.LightningModule):
      ...
      def predict_step(self, batch, batch_idx, dataloader_idx):
          y = self(x)
          return {"predict":y}
   
  m = Modle(...)
  and use any pl.DataModule with a predict_dataloader method
  trainer = pl.Trainer(gpus=2, accelerator=ddp)
  predictions = trainer.predict(model=m, datamodule=dm)
  but we'll get two predictions on two GPUs(they are actually the outputs of two scripts that pytorch_lightning creates for us anyway...), and we can't use it as a single object for later processing in the script.
answer: You can either sync them by writing them both to disk (we have a PredictionWriter for that) or you can use self.all_gather inside your predict_step to sync across GPUs (note that this can lead to GPU OOM!)