How does Dataloader choose batch?

PhysicsIsFun: <p>Greetings!</p>
<p>I came across the (apparently not so trivial question?) of how to subsample without replacement, i.e. from a set of {1,2,…,N} choose k&lt;N elements without replacement, where each integer has the same probability.</p>
<p>I was wondering how the Torch Dataloader handles this? There seem to be algorithms of order N and others of order k.</p>
<p>One naive idea is to shuffle the whole number array and then choose the first k elements to be the samples (order N).<br>
Then there is Yates algorithm:</p>
<ul>
<li>choose a random int in the set</li>
<li>swap last int in the set with sampled int</li>
<li>repeat for the penultimate int in the set<br>
Continue until one has k sampled ints for the last k elements in the set.</li>
</ul>
<p>This is of order k.</p>

KFrank: <p>Hi Physics!</p>
<aside class="quote no-group" data-username="PhysicsIsFun" data-post="1" data-topic="151800" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>I was wondering how the Torch Dataloader handles this? There seem to be algorithms of order N and others of order k.</p>
</blockquote>
</aside>
<p>I haven’t verified this, but I believe that with <code>shuffle = True</code>, the default<br>
<code>Dataloader</code> shuffles the (indices for the) entire <code>Dataset</code>, and then returns<br>
the samples in the <code>Dataset</code> one by one or batch by batch.  There is no<br>
“optimization” for sampling only <code>k</code> of the total <code>N</code> samples in the <code>Dataset</code><br>
because the expectation is that you will iterate over the entire <code>Dataset</code><br>
(and, in fact, do so multiple times as you train over multiple epochs).</p>
<aside class="quote no-group">
<blockquote>
<p>One naive idea is to shuffle the whole number array and then choose the first k elements to be the samples (order N).</p>
</blockquote>
</aside>
<p>Which is, in essence, what I believe <code>Dataloader</code> does, for example, in<br>
the case where <code>Dataset</code> contains <code>N</code> samples, but you choose to sample<br>
a single batch containing only <code>k</code> samples.</p>
<aside class="quote no-group">
<blockquote>
<p>Then there is Yates algorithm:</p>
<ul>
<li>choose a random int in the set</li>
<li>swap last int in the set with sampled int</li>
<li>repeat for the penultimate int in the set<br>
Continue until one has k sampled ints for the last k elements in the set.</li>
</ul>
<p>This is of order k.</p>
</blockquote>
</aside>
<p>Note, as a quibble, forming the initial set of <code>N</code> integers to which you apply<br>
this algorithm is an order-<code>N</code> operation.</p>
<p>Also, as a practical matter, the difference between order-<code>k</code> and order-<code>N</code><br>
will be fully irrelevant compared to the real work of processing the samples<br>
in any realistic use case.</p>
<p>Best.</p>
<p>K. Frank</p>

ID56: <p>PyTorch has map-style datasets (ones with <code>__getitem__()</code> and <code>__len__()</code>) and iterable-style datasets (implements <code>__iter__()</code> instead). By a torch dataset, we generally mean a map-style dataset, as that is the most commonly used type.</p>
<p>You can go and read this part in the docs: <a href="https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler" rel="noopener nofollow ugc">Data Loading Order and Sampler</a>. I will highlight the important part:</p>
<blockquote>
<p>A sequential or shuffled sampler will be automatically constructed based on the <code>shuffle</code> argument to a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc"><code>DataLoader</code></a>. Alternatively, users may use the <code>sampler</code> argument to specify a custom <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Sampler" rel="noopener nofollow ugc"><code>Sampler</code></a> object that at each time yields the next index/key to fetch.</p>
</blockquote>
<p>We can write our custom samplers if needed, but you are asking about the default sampler, I believe. Depending on <code>shuffle=False</code> and <code>shuffle=True</code>, the underlying sampler is either a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.SequentialSampler" rel="noopener nofollow ugc">SequentialSampler</a> or a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.RandomSampler" rel="noopener nofollow ugc">RandomSampler</a>. You can check this inside <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html#DataLoader" rel="noopener nofollow ugc">DataLoader’s source code</a> very quickly. I’ll again highlight the relevant part:</p>
<pre><code class="lang-auto">        if sampler is None:  # give default samplers
            if self._dataset_kind == _DatasetKind.Iterable:
                # See NOTE [ Custom Samplers and IterableDataset ]
                sampler = _InfiniteConstantSampler()
            else:  # map-style
                if shuffle:
                    sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
                else:
                    sampler = SequentialSampler(dataset)
</code></pre>
<p>You can now go check <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/sampler.html#RandomSampler" rel="noopener nofollow ugc">RandomSampler’s source code</a> to see exactly what is going on. As I see a call to <code>torch.randperm(n)</code>, where n = length of dataset, it is apparent that the whole number array is shuffled first (or rather, a random ordered number array is constructed).</p>