How to shuffle an iterable dataset

Ahmed_m: <p>Hi,</p>
<p>I am using the IterableDataset class in order to avoid loading the whole data to memory. However, I cannot shuffle the dataset in that case. Is there any trick for shuffling an iterable dataset?</p>
<p>Thanks in advance!</p>

tom: <p>No.<br>
The typical thing people are doing is to use a conventional dataset that will load data in the <code>__getitem__</code> method.<br>
Look at TorchVision’s DatasetFolder-dataset (used via ImageFolder for Imagenet) for inspiration.</p>
<p>The deeper reason is that suffling requires, by definition, “random access” of some sort. That’s what classic datasets are for.</p>
<p>Best regards</p>
<p>Thomas</p>

Ahmed_m: <p>Thanks!<br>
But in that sense, do you think that there are neural networks that can be trained using a non-shuffled dataset so that iterable dataset class can be applied? Because AFAIK any neural network training with mini-batches requires to shuffle the dataset.</p>

tom: <p>Well, so it depends a bit. There are some things like language models where the text is decidedly not shuffled. It probably is not too good to feed a <em>sorted</em> (by categories) dataset into a classification network, but quite likely, it is not always necessary to have completely random order.<br>
That said, I’d probably use a classic dataset unless you know you cannot use it (i.e. take the common route if you can).</p>
<p>Best regards</p>
<p>Thomas</p>

SimonW: <p>also there are cases where your data is generated / sampled on the fly: from distribution, from simulation, where iterable datasets are more suitable.</p>

sharvil: <p>I think the standard approach to shuffling an iterable dataset is to introduce a shuffle buffer into your pipeline. Here’s the class I use to shuffle an iterable dataset:</p>
<pre><code class="lang-python">class ShuffleDataset(torch.utils.data.IterableDataset):
  def __init__(self, dataset, buffer_size):
    super().__init__()
    self.dataset = dataset
    self.buffer_size = buffer_size

  def __iter__(self):
    shufbuf = []
    try:
      dataset_iter = iter(self.dataset)
      for i in range(self.buffer_size):
        shufbuf.append(next(dataset_iter))
    except:
      self.buffer_size = len(shufbuf)

    try:
      while True:
        try:
          item = next(dataset_iter)
          evict_idx = random.randint(0, self.buffer_size - 1)
          yield shufbuf[evict_idx]
          shufbuf[evict_idx] = item
        except StopIteration:
          break
      while len(shufbuf) &gt; 0:
        yield shufbuf.pop()
    except GeneratorExit:
      pass
</code></pre>
<p>You’d wrap your existing iterable dataset like this:</p>
<pre><code class="lang-python">dataset = MyIterableDataset()
dataset = ShuffleDataset(dataset, 1024)  # shuffle buffer size depends on your application
</code></pre>

jastern33: <p>The algorithm described above is now implemented natively in Pytorch, as BufferedShuffleDataset, documented <a href="https://pytorch.org/docs/stable/data.html" rel="noopener nofollow ugc">here</a>.</p>

cerisara: <p>Is the BufferedShuffleDataset gone ? Or may be replaced by another class ? I can’t find it in the latest doc.</p>

jastern33: <aside class="quote no-group" data-username="cerisara" data-post="8" data-topic="64130">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cerisara/48/428_2.png" class="avatar"> cerisara:</div>
<blockquote>
<p>BufferedShuffleDataset</p>
</blockquote>
</aside>
<p>Yeah, I’m not sure where it went. I don’t see any issues on github about removing it. It shows up in the <a href="https://pytorch.org/docs/1.8.0/data.html" rel="noopener nofollow ugc">1.8.0 docs</a>, but it’s gone in the current version.</p>

ptrblck: <p>It seems the removal wasn’t documented and no deprecation warning was used.<br>
I could find <a href="https://github.com/pytorch/pytorch/issues/66116">this</a> and <a href="https://github.com/pytorch/pytorch/issues/62310">this</a> issue, which seems to point towards</p>
<pre><code class="lang-python">torch.utils.data.datapipes.iter.combinatorics.ShuffleIterDataPipe
</code></pre>
<p>as its replacement.</p>

Olaurent: <p>For those interested, it is now <code>ShufflerIterDataPipe</code> from</p>
<pre><code class="lang-auto">torch.utils.data.datapipes.iter.combinatorics
</code></pre>
<p>Or it might have been a typo from the start (missing an “r”).</p>