logging question in DDP
Question regarding logging from lightning module in DDP model
For example, here is a validation step function that computes accuracy.
def validation_step(self, batch, batch_idx):
       x, y = batch
       logits = self(x)
       acc = acc_fun(logits,y)     
       self.log('val_acc', acc)
So what happens in DDP, is the logged value averaged across GPUs? At the end of every epoch?
It is averaged if you set sync_dist=True (https://pytorch-lightning.readthedocs.io/en/stable/lightning_module.html#pytorch_lightning.core.lightning.LightningModule.log.params.sync_dist)
You can change the op from average (mean) to something else with sync_dist_op.
Given that you are logging inside of validation_step, it will be done at the end of the epoch by default. You can change this behaviour via on_step and on_epoch.
Please have a look at the docs page I linked. Lots of useful info there 😄