question: logging question in DDP
context: Question regarding logging from lightning module in DDP model
  For example, here is a validation step function that computes accuracy.
  def validation_step(self, batch, batch_idx):
         x, y = batch
         logits = self(x)
         acc = acc_fun(logits,y)     
         self.log('val_acc', acc)
  So what happens in DDP, is the logged value averaged across GPUs? At the end of every epoch?
answer: It is averaged if you set sync_dist=True (https://pytorch-lightning.readthedocs.io/en/stable/lightning_module.html#pytorch_lightning.core.lightning.LightningModule.log.params.sync_dist)
  You can change the op from average (mean) to something else with sync_dist_op.
  Given that you are logging inside of validation_step, it will be done at the end of the epoch by default. You can change this behaviour via on_step and on_epoch.
  Please have a look at the docs page I linked. Lots of useful info there 😄