question: Why MultiGPU dp seems slower?
context: ❓ Questions and Help
  Having 2 gpus with DP seems to be slowers than using just 1. Is it normal?
  My intuition is that if you are using 2 GPUs and the batch is being splitted into 2 batches, this should be faster. But when I tested the same code using 1 vs >1 my epoch time increased
  Code
  Minimalist Implementation of a BERT Sentence Classifier
  What have you tried?
  I also tried to run ddp but my code seems to break with a TypeError: cannot serialize '_io.TextIOWrapper' object error. I searched online but I couldn't find the reason...
  What's your environment?
  
  OS: Linux
answer: you should double your batch size.
  dp still has overhead in communication, so it won't be linear scaling.
  also try ddp