DistributedDataParallel taking twice more time then DataParallel

shrutishrestha: <p>I used implementation of DataParallel from<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html</a></p>
<p>and DistributedDataParallel from <a href="https://pytorch.org/docs/master/notes/ddp.html#:~:text=reveals%20implementation%20details.-,Example,-Let%20us%20start" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch master documentation</a></p>
<p>As shown in the DataParallel docs<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#:~:text=2%20GPUs,2%2C%20you%20will%20see%3A" class="inline-onebox" rel="noopener nofollow ugc">Optional: Data Parallelism — PyTorch Tutorials 1.12.0+cu102 documentation</a> I printed using DataParallel and DistributedDataParallel, the results are as shown below the batch size is 4 and images have 1 channel with 512x512 size.</p>
<p>------------------ DataParallel ------------------</p>
<p>In Model: input size torch.Size([2, 1, 512, 512])<br>
In Model: input size torch.Size([2, 1, 512, 512])<br>
Outside: input size torch.Size([4, 1, 512, 512]) output_size torch.Size([4, 1, 512, 512])</p>
<p>In Model: input size torch.Size([2, 1, 512, 512])<br>
In Model: input size torch.Size([2, 1, 512, 512])<br>
Outside: input size torch.Size([4, 1, 512, 512]) output_size torch.Size([4, 1, 512, 512])</p>
<p>In Model: input size torch.Size([1, 1, 512, 512])<br>
Outside: input size torch.Size([1, 1, 512, 512]) output_size torch.Size([1, 1, 512, 512])</p>
<p>------------------ Distributed Data Parallel ------------------<br>
In Model: input size torch.Size([4, 1, 512, 512])<br>
Outside: input size torch.Size([4, 1, 512, 512]) output_size torch.Size([4, 1, 512, 512])</p>
<p>In Model: input size torch.Size([4, 1, 512, 512])<br>
Outside: input size torch.Size([4, 1, 512, 512]) output_size torch.Size([4, 1, 512, 512])</p>
<p>In Model: input size torch.Size([4, 1, 512, 512])<br>
Outside: input size torch.Size([4, 1, 512, 512]) output_size torch.Size([4, 1, 512, 512])</p>
<p>In Model: input size torch.Size([4, 1, 512, 512])<br>
Outside: input size torch.Size([4, 1, 512, 512]) output_size torch.Size([4, 1, 512, 512])</p>
<p>In Model: input size torch.Size([1, 1, 512, 512])<br>
Outside: input size torch.Size([1, 1, 512, 512]) output_size torch.Size([1, 1, 512, 512])</p>
<p>In Model: input size torch.Size([1, 1, 512, 512])<br>
Outside: input size torch.Size([1, 1, 512, 512]) output_size torch.Size([1, 1, 512, 512])</p>
<p>As seen above, the dataparallel is taking 2-2 batch in each of the gpus whereas the DsitributedDataPrallel is taking 4 batch in each of the gpus.</p>
<p>So this DsitributedDataPrallel is taking exactly double the time as taken by DataParallel. Should I implement all Partition, DataPartitioner, partition_dataset, average_gradients like functions shown in the link (<a href="https://github.com/seba-1511/dist_tuto.pth/blob/gh-pages/train_dist.py" class="inline-onebox" rel="noopener nofollow ugc">dist_tuto.pth/train_dist.py at gh-pages · seba-1511/dist_tuto.pth · GitHub</a>)to make the DistributedDataParallel work?</p>
<p>And doing the things specified in this tutorial only (<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.12.0+cu102 documentation</a>) doesnot help??</p>

ptrblck: <p>Your use cases are not equal as you are doubling the batch size in the DDP use case.<br>
Note that the data loading in DDP is done on each rank separately and you would usually use a <code>DistributedSampler</code> to avoid drawing duplicated samples.<br>
Could you explain where you are currently stuck?</p>