DistributedSampler not shuffling dataset

username: kristianeschenburg
cooked: <p>Hi</p>
<p>I’m trying to use the DistributedSampler class with the argument <code>shuffle=True</code>.  However, it seems that my batches are not being shuffled across epochs and across devices.  See the attached file “TrainingPrintOut.png”: I’m printing out the number of nodes per batch assigned to GPU device with rank 0 – across two different epochs, each batch assigned to device with rank 0 has the same number of nodes for each corresponding iterant, which (correct me if I’m wrong) indicates that the batches / data aren’t being shuffled.  One can also see this effect in the training performance metrics (see second attached file, “TrainingAccuracy.png”), which show consistent periodic spikes and valleys in the estimated metrics as a function of batch.</p>
<p>Here is the code for how I’m loading the datasets, DataLoaders, and Distributed Samplers:</p>
<pre data-code-wrap="python"><code class="lang-python">
DATASET_MAP = {'D1': datasets.D1_Dataset,
               'D2': datasets.D2_Dataset}

# load datasets
datasets = {subset: DATASET_MAP[args.dataset](root=root_dir,
                                                  data_subset=subset,
                                                  transform=xfm,
                                                  in_memory=args.in_memory,
                                                  feature_type=args.feature_type,
                                                  n_samples=sample_sizes[subset],
                                                  verbose=0)
                for subset in ['train', 'val']}

# set up data samplers
samplers = {subset: DistributedSampler(datasets[subset],
                                       rank=local_rank,
                                       shuffle=True)
            for subset in ['train', 'val']}

# set up dataloaders
loaders = {subset: DataLoader(datasets[subset],
                              batch_size=args.batch_size,
                              num_workers=0,
                              sampler=samplers[subset])
           for subset in ['train', 'val']}
</code></pre>
<p>Can anyone shed light on how to actually shuffle the dataset when using DistributedSampler?</p>
<p>Thanks.</p>
<p>Kristian<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/4979d935d7e74d75152418549d761388aa8cb589.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4979d935d7e74d75152418549d761388aa8cb589" title="TrainingAccuracy"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4979d935d7e74d75152418549d761388aa8cb589_2_690x324.jpeg" alt="TrainingAccuracy" data-base62-sha1="atZUHlyWKoLczSwcnGXvnvdiRFD" width="690" height="324" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4979d935d7e74d75152418549d761388aa8cb589_2_690x324.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4979d935d7e74d75152418549d761388aa8cb589_2_1035x486.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4979d935d7e74d75152418549d761388aa8cb589_2_1380x648.jpeg 2x" data-dominant-color="EDEEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">TrainingAccuracy</span><span class="informations">1920×904 127 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb772efef632e2a1d019f60f90bd192bce35ef7f.png" data-download-href="https://discuss.pytorch.org/uploads/default/eb772efef632e2a1d019f60f90bd192bce35ef7f" title="TrainingPrintOut"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb772efef632e2a1d019f60f90bd192bce35ef7f.png" alt="TrainingPrintOut" data-base62-sha1="xB1yqaabYlWbxyGn8MMTl8QwB3N" width="690" height="438" data-dominant-color="272A2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">TrainingPrintOut</span><span class="informations">1770×1126 141 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: kristianeschenburg
cooked: <p>Calling <code>sampler.set_epoch(epoch)</code> fixed the issue for more and now the model trains with randomness in the sampled batches.  Naturally, I did not see the big warning message at the bottom of the <a href="https://pytorch.org/docs/stable/data.html#randomness-in-multi-process-data-loading:~:text=Default%3A%20False.-,WARNING,-In%20distributed%20mode" rel="noopener nofollow ugc">page</a>!</p>