question: What's the difference between `on_step` and `on_epoch` of `pl_module.log`
context: https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html?highlight=on_epoch%20#pytorch_lightning.core.lightning.LightningModule.log.params.on_epoch
  
  I'm using horovod to train the model. I wonder if on_step and on_epoch average the metrics across all GPUs automatically. In other words, do we need to explicitly average the metrics in functions like training_epoch_end and validation_epoch_end?
answer: Dear @marsggbo,
  When using self.log(..., on_step=True), this will compute the metric per step locally as synchronisation adds performance hit.
  When using self.log(..., on_step=True, sync_dist=True), this will compute the metric per step across GPUS.
  When using self.log(..., on_epoch=True), this will compute the metrics across GPUS and epoch batches automatically.
  Best,
  T.C