How can I train a model with multiple data loaders using DistributedDataParallel?

ysleer: <p>HI, Thank you for your interest in my writing.</p>
<p>I’ve seen a lot of posts on the pytorch forum but couldn’t find a solution to what I was wondering about.</p>
<p>Can 3 dataloaders be applied to DistributedDataParallel??</p>
<p>The process I think is as follows.</p>
<p>GPU1 GPU2 GPU3<br>
model model model<br>
dataloader1 dataloader2 dataloader3</p>
<p>I want to backpropagation by averaging the loss values of each dataloader.</p>
<p>Are there any possible methods or examples??</p>
<p>Each data loader has different image size and batch size, so concatDataset cannot be used.</p>
<p>If there is a way, please comment.</p>
<p>Thank you.</p>

mrshenli: <blockquote>
<p>The process I think is as follows.</p>
<p>GPU1 GPU2 GPU3<br>
model model model<br>
dataloader1 dataloader2 dataloader3</p>
</blockquote>
<p>This is possible and is actually the recommended use case for DDP. For dataloader, you can use the <a href="https://pytorch.org/docs/master/data.html#torch.utils.data.distributed.DistributedSampler" rel="noopener nofollow ugc">DistributedSampler</a> and set <code>num_replicas</code> properly to match DDP world size.</p>
<blockquote>
<p>I want to backpropagation by averaging the loss values of each dataloader.</p>
</blockquote>
<p>This is a bit tricky. With DDP, outputs and losses are local to each process. DDP synchronizes models by synchronizing gradients. More details are available in this <a href="http://www.vldb.org/pvldb/vol13/p3005-li.pdf" rel="noopener nofollow ugc">PyTorch Distributed paper</a>.</p>
<blockquote>
<p>Each data loader has different image size and batch size, so concatDataset cannot be used.</p>
</blockquote>
<p>I see. Independent dataloaders could still work with DDP, but you will need to be aware of the implications on model accuracy. Is this the reason why you have to average loss instead of gradients?</p>

ysleer: <p>Thanks for the answer.</p>
<p>sorry.</p>
<p>I didn’t understand your answer to “Is this the reason why you have to average loss instead of gradients?”</p>
<p>My question doesn’t seem to be clear.</p>
<p>To be more simply, I want to training one model using multiple dataloader.</p>
<p>All I want to do is update using the average of the losses calculated from multiple dataloader.</p>
<p>I just want to use various loss values using various image size and batch size.</p>
<p>Please tell me what you want to know specifically about my question.</p>

mrshenli: <blockquote>
<p>I just want to use various loss values using various image size and batch size.</p>
</blockquote>
<p>If this is all you need, you don’t need to average loss. You can let each DDP process creates its own independent data loader and then produce a local loss. Then, during the backward pass, DDP will synchronize gradients for you.</p>

ysleer: <p>Okay.</p>
<p>Do you have any examples or documentation for reference?</p>
<p>Thank you.</p>

mrshenli: <p>Yep, here is a starter example: <a href="https://pytorch.org/docs/stable/notes/ddp.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch 1.8.0 documentation</a></p>
<p>You can replace the <code>torch.randn(20, 10).to(rank)</code> random input tensor by input and labels from a dataloader <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html#iterate-through-the-dataloader" rel="noopener nofollow ugc">example</a></p>
<p>Here is a complete list of DDP tutorials: <a href="https://pytorch.org/tutorials/beginner/dist_overview.html#torch-nn-parallel-distributeddataparallel" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Distributed Overview — PyTorch Tutorials 1.8.0 documentation</a></p>
<p>It should work, but will have different accuracy implications to different applications, depending on your model, data, batch size, loss function, etc. I would suggest taking a look at the <a href="http://www.vldb.org/pvldb/vol13/p3005-li.pdf" rel="noopener nofollow ugc">DDP paper</a> or at lest the <a href="https://pytorch.org/docs/master/notes/ddp.html#internal-design" rel="noopener nofollow ugc">design notes</a> and verify whether DDP’s gradient averaging algorithm is OK for your application.</p>