Question about the log system
Hello, I have some question about the self.log function and batch_size during the trainer.
If I have two GPUs, and I want to train my model with batch_size 16 per GPU and I use DDP, so what's the number of batch_size in Datamodule and what's the number of batch_size in self.log, If I want to calculate my metrics correctly?
hey @exiawsh
it should stay as batch_size for a single device only. With DDP if you set batch_size=7, then each device gets the batch of batch_size=7, and effective batch_size increases with the number of devices. Now if you want to log by accumulating metrics across devices, you need to set sync_dist=True. Check out the section here: https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html#automatic-logging