question: How to gather results on multiple GPUs while testing? ddp
context: ❓ Questions and Help
  What is your question?
  I want to test summarization model from huggingface summarization example on multiple GPUs . My problem is how could I collect test results  on different GPUs , since test_epoch_end only processes epoch for a single GPU.
  For more information, the model is trained with ddp backend.
  Code
   def test_epoch_end(self, outputs):
          output_test_predictions_file = os.path.join(self.hparams.output_dir, "test_predictions.txt")
          output_test_targets_file = os.path.join(self.hparams.output_dir, "test_targets.txt")
          # write predictions and targets for later rouge evaluation.
          with open(output_test_predictions_file, "w+") as p_writer, open(output_test_targets_file, "w+") as t_writer:
              for output_batch in outputs:
                  p_writer.writelines(s + "\n" for s in output_batch["preds"])
                  t_writer.writelines(s + "\n" for s in output_batch["target"])
              p_writer.close()
              t_writer.close()
  
          return self.test_end(outputs)
  
  What have you tried?
  For now, I can only use single GPU to get result of whole dataset.
  What's your environment?
  
  OS: Unbuntu 18.04
  Packaging pip
  Version: 0.7.6
answer: Use torch.distributed.all_gather to gather and merge the outputs from all GPUs.
  And you should remove the redundant examples due to the ddp_sampler adds extra examples to work with multi GPUS. (https://pytorch.org/docs/stable/_modules/torch/utils/data/distributed.html#DistributedSampler)
  Here is the workaround snippet used in my own project.
  def gather_distributed(*tensors):
      output_tensors = []
      for tensor in tensors:
          tensor_list = [torch.ones_like(tensor) for _ in range(dist.get_world_size())]
          dist.all_gather(tensor_list, tensor)
          output_tensors.append(torch.cat(tensor_list))
      return output_tensors
  
  
  def deduplicate_and_sort(index, *tensors):
      reverse_index = torch.zeros_like(index)
      for ri, i in enumerate(index):
          reverse_index[i] = ri
      reverse_index = reverse_index[:index.max() + 1]
      output_tensors = [tensor.index_select(0, reverse_index) for tensor in tensors]
      return output_tensors
  
  In the above code, you need the index of each example to remove redundant examples and sort outputs in order.
  Notice that the index should consist of consecutive integers (e.g., 0,1,2,...N).