Can DDP divide dataset unevenly to workers?

zizhao.mo: <p>Hi,</p>
<p>Is there any native method in DDP to divide a given dataset unevenly? For example, 60% of Cifar10 data are distributed to the first worker in each epoch while the other 40% are run through by another worker.</p>
<p>Thanks for any comments in advance.</p>

rvarm1: <p>Hi, there is no native support for this at the moment as <code>DistributedSampler</code> does not take in a custom sampler so it will always do the default which is divide evenly.</p>
<p>One option is to implement your own data sampling across workers and use DDP’s uneven inputs join context manager: <a href="https://pytorch.org/tutorials/advanced/generic_join.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Training with Uneven Inputs Using the Join Context Manager — PyTorch Tutorials 1.11.0+cu102 documentation</a> to make sure DDP does not hang due to uneven dataset size.</p>
<p>Curious about your use case though, what is the aim of having 60/40 skew of data across workers?</p>

zizhao.mo: <p>Hi, thanks for your response.</p>
<p>My motivation is that, I want the faster GPU process more data and let the slower GPU process fewer data. The reason behind is the different capabilities across GPUs. For example, batch size = 60 costs GPU1 the same time as batch size = 40 for GPU2. For this reason, I want to divide the data 6:4 to alleviate the speed gap.</p>

huahuanZ: <p>Design a custom batch_sampler could make that.</p>
<pre><code class="lang-auto"># this is a super simple example, you should impl your own one.
class MyDistributedSampler(DistributedSampler):
    # customize the __iter__ method
    def __iter__(self):
        if self.rank == 0: # data 0-59 on node 0
            return [list(range(60))]
        if self.rank == 1: # data 60-99 on node 1
            return [list(range(60, 100))]

# use batch_sampler to init the dataloader
train_sampler = MyDistributedSampler(...)
trainloader = DataLoader(tr_set, batch_sampler=train_sampler, ...)
</code></pre>
<p>And note that you should do addtional normalization to the loss since batch sizes on nodes are not the same. See “Alternatives” at <a href="https://github.com/pytorch/pytorch/issues/67253" rel="noopener nofollow ugc">Support different batch size across GPUs with DDP · Issue #67253 · pytorch/pytorch (github.com)</a></p>

zizhao.mo: <p>Thank you very much. This is exactly what I am finding.</p>

Hong_Wang: <p>Thanks you and I have a question. When we instantiate a DataLoader object. The batch size is fixed among workers when we use torch DDP. If we define a uneven sampler and 60 data is on node 0 while 40 data on node 1, the two nodes will end up having different iterations. For example, if batch size is 5, node 0 will have 12 iterations while node 1 has 8 iterations. There will be an error. How can we solve this issue?</p>

huahuanZ: <p>To clarify we’re discussing the same thing, there are two levels of “uneven data”:</p>
<ol>
<li>uneven samples in mini-batch, e.g. at step=0, GPU0 is fed with a mini-batch of 10 samples and for GPU1 it is 5 samples.</li>
<li>uneven mini-batches across devices, e.g. for GPU0, 1 epoch runs for 20 steps (mini-batches); and for GPU1 it runs for 10 steps (mini-batches)</li>
</ol>
<p>For situation 1, DDP is still synchronous and wouldn’t be blocked.</p>
<p>For 2, which I believe is what you are interested, there are multiple ways to address that, for example, one could repeat samples for the GPU worker with less mini-batches; or drop the rest of mini-batches once a GPU worker runs more steps than others.<br>
And pytorch has natively supported a api to address such issue, please check this guide <a href="https://pytorch.org/tutorials/advanced/generic_join.html?highlight=join" class="inline-onebox" rel="noopener nofollow ugc">Distributed Training with Uneven Inputs Using the Join Context Manager — PyTorch Tutorials 2.3.0+cu121 documentation</a></p>