How to handle non-determinism in DistributedSampler?

username: paganpasta
cooked: <p>As per my requirement the batch composition is <strong>non-deterministic</strong> which is easily handled through a custom <code>Batchsampler</code> on a single-GPU or <code>DataParallel</code> implementation. For a <code>DistributedDataParallel</code> scenario, I can’t think of anyway to fix the issue.</p>
<p>I wish to seek discussion regarding the potential solution I have in mind for managing it for DDP.<br>
<strong>Potential Solution</strong><br>
Since only <strong>1</strong> sequence of indices should be generated is it wise to restrict sampling to rank=0 and use <code>torch.distributed.broadcast</code> to collect samples produced by rank=0 onto corresponding ranks?<br>
Something like</p>
<pre><code class="lang-auto">import torch
from torch.utils.data.sampler import SequentialSampler
#Simulate different ordering of indices with a non-deterministic batch sampler
samples = torch.randperm(12).tolist()
print('Starting samples for rank:', args.rank, samples)

###Sync'd solution
s = torch.cuda.Stream()
samples = torch.LongTensor(samples).to(args.rank)
handle=torch.distributed.broadcast(samples, src=0, async_op=True)
handle.wait()
with torch.cuda.stream(s):
    s.wait_stream(torch.cuda.default_stream())
print('Samples in rank:',args.rank, samples)

==output==
Starting samples for rank: 1 [6, 0, 8, 11, 3, 4, 7, 5, 2, 9, 10, 1]
Starting samples for rank: 0 [3, 5, 6, 9, 7, 11, 10, 1, 8, 0, 2, 4]
Samples in rank: 1 tensor([ 3,  5,  6,  9,  7, 11, 10,  1,  8,  0,  2,  4], device='cuda:1')
Samples in rank: 0 tensor([ 3,  5,  6,  9,  7, 11, 10,  1,  8,  0,  2,  4], device='cuda:0')
</code></pre>
<p>I can effectively move this inside <code>CustomDistributedSampler</code> to broadcast rank:0’s indices across all devices and use a slice similar to PyTorch’s distributed sampler to return indices.</p>
<p>Any issues with the above implementation? Is there a better looking and/or optimized solution?</p>

username: paganpasta
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Any suggestions</p>

username: ptrblck
cooked: <p>I don’t fully understand your use case since you are broadcasting the same indices.<br>
Is this intended, i.e. do you want to load the same samples on each rank?<br>
Note that the <code>DistributedSampler</code> chunks the indices of the <code>Dataset</code> so that sample duplication would be avoided.</p>

username: paganpasta
cooked: <p>Sorry for the confusion.<br>
I intend to send same value of indices to all ranks. I am using a custom sampler for which even seeding everything leads to non-determinism in indices generation. This makes using the default DistributedSampler useless. At the moment, instead of finding and fixing the non-determinism I am trying to broadcast the indices so that all DistributedSampler instances sample from identical indices list.</p>

username: ptrblck
cooked: <p>OK, I see and your approach seems to work for this. However, note that I’ve described that using the same indices in each rank would not be the standard approach. Usually you would use a chunk of the indices on each rank so that they can train on a separate subset.<br>
If your dataset is the same on each rank, the model’s state as well, and you are using the same indices, you would just repeat the same operations on each rank.<br>
Since I’m not familiar with your use case, this might be indeed what you want, so see this as a side note.</p>

username: paganpasta
cooked: <p>Yup, noted. Thanks for the confirmation.</p>