Should the total epoch size be less when using multi-gpu DDP?
If I have 100 training examples and 100 validation examples, and I run on a single gpu with a batch size of 10, the tqdm bar will show 20 epoch iterations. If I run on 2 gpus with ddp and the same batch size, the tqdm bar will still show 20 epoch iterations, but isnt the effective batch size now 20 instead of 10 because theres 2 gpus? Shouldnt the total number of iterations be half?
Thanks for any clarification.
Hi @jipson7 ,
First of all: You're right, that's how it should be.
We tried to reproduce this, but for us this produced the following (correct) output. Do you have a minimal reproduction example?
Epoch 0: 100%|███████████████████████████████████████████████████████| 10/10 [00:00<00:00, 17.23it/s, loss=-43.6, v_num=272]
seen train: 5
seen train: 5