Distributed Data Sampler

Rakshith_V: <p>I have been using Speechbrain’s Distributed sampler wrapper :</p>
<p>class DistributedSamplerWrapper(DistributedSampler):<br>
“”“This wrapper allows using any sampler with Distributed Data Parallel (DDP) correctly.<br>
Passing blindly the sampler to each DDP process will cause to have access<br>
within each process to all the data in the dataset instead of only a subset<br>
of it which is unique to each process.  This wrapper prevents this and<br>
allows to use only a subset of the original data for each process.<br>
NOTE<br>
----<br>
This is is automatically applied to any sampler in the Brain class when DDP<br>
training is used.<br>
“””</p>
<pre><code>def __init__(self, sampler, *args, **kwargs):
    # DistributedSampler only calls len() on dataset
    # so a sampler is fine to pass there, as well.
    super().__init__(dataset=sampler, *args, **kwargs)
    self.sampler = sampler

def __iter__(self):
    # It is easiest to use a random access interface to the wrapped
    # sampler's indices, so we just fetch all indices from the wrapped
    # sampler
    sampler_indices = list(self.sampler.__iter__())
   print(sampler_indices)
    indices_of_indices = super().__iter__()
    # Itemgetter fetches the wrapped sampler indices from the positions
    # pointed to by DistributedSampler
    return iter(itemgetter(*indices_of_indices)(sampler_indices))

def set_epoch(self, epoch):
    """Pass set_epoch() through to DistributedSampler and the wrapper one"""
    super().set_epoch(epoch)
    if hasattr(self.sampler, "set_epoch"):
        self.sampler.set_epoch(epoch)
</code></pre>
<p>######################################<br>
I’m trying to print sampler_indices variable, but it was printing twice- so,what could be the reason for that ?</p>

ptrblck: <p>I guess you might be running in a <code>DistributedDataParallel</code> setup where each process uses a single device. If so, then each process will also initialize an own sampler and the <code>print</code> statement should be visible <code>num_processes</code> times.</p>

Rakshith_V: <p>When I use 2 device, it’s printing 4 times and 2 times when device is one</p>

ptrblck: <p>How many workers are you using and how many calls are you expecting to <code>__iter__</code>?</p>

Rakshith_V: <p>As you have rightly mentioned earlier, I’m using a DDP set-up. I have 2 GPU in a single mode.</p>
<p>I’m expecting to call iter once. In my case I will be having 2 process, so each will call iter once and that returns an iterator.</p>
<p>What’s happening is -<br>
For single process,</p>
<ol>
<li>Variable name gets printed</li>
<li>Iterator is not returned</li>
<li>Variable name gets printed again</li>
<li>Iterator is returned</li>
</ol>
<p>I feel like I’m probably missing something here, though overall outcome (obtaining different subset of samples for each process) is as expected</p>

ptrblck: <p>Thanks for the update. That’s strange and I cannot reproduce the issue using the native <code>DistributedSampler</code>.<br>
If I add a debug <code>print</code> statement <a href="https://github.com/pytorch/pytorch/blob/0708630d9f30210b0a51d13558ad173da630ef95/torch/utils/data/distributed.py#L118">here</a> to print the <code>indices</code> I see <code>nproc_per_node</code> outputs. I.e. using 2 GPUs via <code>--nproc_per_node=2</code> prints the <code>indices</code> 2 times, <code>--nproc_per_node=8</code> 8 times etc.</p>

Rakshith_V: <p>Thanks for the help.<br>
I will update in this thread, if I’m able to find the reasoning behind this behavior</p>

ptrblck: <p>That sounds good. One debugging step could be to check if you are seeing the expected behavior using the native <code>DistributedSampler</code>. Could you add the <code>print</code> statements into the same line of code in your local installation and rerun your code?</p>

Rakshith_V: <pre><code class="lang-auto">   indices = indices[self.rank:self.total_size:self.num_replicas]
    print(indices)
    assert len(indices) == self.num_samples
</code></pre>
<p>I tried as you suggested , indices are getting printed twice for single process.</p>
<p>This is my dataloader<br>
sampler = DistributedSamplerWrapper(DynamicBatch(path)) # Wrapper code I have pasted earlier<br>
DataLoader(dataset,collate_fn=collate_wrapper,batch_sampler=sampler,num_workers=16)</p>

ghjuliasialelli: <p>any progress fixing this? i’m encountering the same issue</p>

yf225: <p>Do you have a simple script to reproduce this behavior? If so, would you like to create a Github issue on <a href="https://github.com/pytorch/pytorch/issues?q=sort%3Aupdated-desc+is%3Aissue+is%3Aopen" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a>? Thanks!</p>