What's the difference between `on_step` and `on_epoch` of `pl_module.log`
https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html?highlight=on_epoch%20#pytorch_lightning.core.lightning.LightningModule.log.params.on_epoch

I'm using horovod to train the model. I wonder if on_step and on_epoch average the metrics across all GPUs automatically. In other words, do we need to explicitly average the metrics in functions like training_epoch_end and validation_epoch_end?
Dear @marsggbo,
When using self.log(..., on_step=True), this will compute the metric per step locally as synchronisation adds performance hit.
When using self.log(..., on_step=True, sync_dist=True), this will compute the metric per step across GPUS.
When using self.log(..., on_epoch=True), this will compute the metrics across GPUS and epoch batches automatically.
Best,
T.C