accumulate_grad_batches and DDP
Hi!
When I use no multi-gpu settings and just a single GPU and the following parameters:
batch_size = 16
accumulate_grad_batches=2
my effective batch size is 32. My question is how accumulate_grad_batches and DDP interact.
If I am using 2 GPUS that are on the same machine and i use the parameters
batch_size = 16
accumulate_grad_batches=2
accelerator='ddp'
gpus=[0,1]

is my effective batch size now 64?
Thanks for the help!
Yes: https://pytorch-lightning.readthedocs.io/en/latest/advanced/multi_gpu.html#batch-size