Why MultiGPU dp seems slower?
❓ Questions and Help
Having 2 gpus with DP seems to be slowers than using just 1. Is it normal?
My intuition is that if you are using 2 GPUs and the batch is being splitted into 2 batches, this should be faster. But when I tested the same code using 1 vs >1 my epoch time increased
Code
Minimalist Implementation of a BERT Sentence Classifier
What have you tried?
I also tried to run ddp but my code seems to break with a TypeError: cannot serialize '_io.TextIOWrapper' object error. I searched online but I couldn't find the reason...
What's your environment?

OS: Linux
you should double your batch size.
dp still has overhead in communication, so it won't be linear scaling.
also try ddp