metrics in multi-gpu settings
Question regarding the logs in ddp mode:
Logs (in metrics.csv) only seem to contain metrics of the model/batches on the gpu with lowest index, although they are computed for each gpu according to validation_end.


(Minor) issue regarding the dp/ddp2 mode:
The documentation suggests averaging the metrics as
if self.trainer.use_dp or self.trainer.use_ddp2:
metric_value = torch.mean(metric_value)
However, this might be (slightly) wrong if the batchsize is not divisible by the number of gpus.