question: accumulate_grad_batches and DDP
context: Hi!
  When I use no multi-gpu settings and just a single GPU and the following parameters:
  batch_size = 16
  accumulate_grad_batches=2
  my effective batch size is 32. My question is how accumulate_grad_batches and DDP interact.
  If I am using 2 GPUS that are on the same machine and i use the parameters
  batch_size = 16
  accumulate_grad_batches=2
  accelerator='ddp'
  gpus=[0,1]
  
  is my effective batch size now 64?
  Thanks for the help!
answer: Yes: https://pytorch-lightning.readthedocs.io/en/latest/advanced/multi_gpu.html#batch-size