Sampling with(out) replacement in DataLoader?

username: theory_buff
cooked: <p>I have a simplistic dataset – a torch tensor of size <code>[N, D+1]</code> : <code>N</code> samples of size <code>D+1</code>, denoting an input-output pair. I am using this tensor as input to the <code>torch.utils.data.DataLoader()</code> to sample rows for minibatch SGD optimization. (<code>shuffle = True</code> in the current setting).</p>
<p>My question is – by default, how does DataLoader sample from a dataset? Is it with replacement across batches, or without?</p>
<p>The documentation seems to have these details for specific samplers like <code>torch.utils.data.RandomSampler()</code>, however the default sampler for <code>torch.utils.data.DataLoader()</code> is <code>None</code>. (<a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">torch.utils.data : PyTorch documentation</a>)</p>
<p>For example, for <code>N=5</code> and <code>batch_size=2</code> for the <code>DataLoader</code>, can a sequence of samples in an epoch be (1,2), (1,3), (4)? (i.e. 1 is being sampled repeatedly due to sampling with replacement, if it exists by default), or would it be something like (1,2), (4,5), (3)?</p>
<p>Thanks!</p>

username: nivek
cooked: <p>I think the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.RandomSampler" rel="noopener nofollow ugc">documentation of <code>RamdomSampler</code></a> should answer your question:</p>
<p>Samples elements randomly. If without replacement, then sample from a shuffled dataset. If with replacement, then user can specify <code>num_samples</code> to draw.</p>
<p>Let me know if any part of that is unclear.</p>

username: theory_buff
cooked: <p>Does the <code>DataLoader</code> use <code>RandomSampler</code> by default? (default value of the <code>sampler</code> argument of <code>Dataloader</code> is <code>None</code>)</p>
<p>Also at the link you mentioned, what exactly does “on-demand” mean in the following: <code>samples are drawn on-demand with replacement</code></p>

username: srishti-git1110
cooked: <p>Hi <a class="mention" href="/u/theory_buff">@theory_buff</a>,<br>
Firstly, since you are using <code>shuffle=True</code>, you will not need to specify any value for the argument <code>sampler</code> in your Dataloader.</p>
<p>Check <a href="https://wandb.ai/srishti-gureja-wandb/posts/How-to-Eliminate-the-Data-Processing-Bottleneck-with-PyTorch--VmlldzoyNDMxNzM1#the-mighty-dataloader" rel="noopener nofollow ugc">this</a> out to get an idea about the mechanism of how actually the data is sampled across DataLoader worker processes when <code>shuffle=True</code>.</p>
<p>And so, in a <code>shuffle=True</code> setting, I don’t think there is any duplication of samples across batches.<br>
Why I think so?<br>
The DataLoader by default applies sharding in each worker process to ensure no duplication is happening.</p>
<p>Additionally, <a href="https://wandb.ai/srishti-gureja-wandb/posts/How-to-Eliminate-the-Data-Processing-Bottleneck-with-PyTorch--VmlldzoyNDMxNzM1#let's-code!" rel="noopener nofollow ugc">see this</a> (scroll down a bit) to get an idea of how duplication happens (when we use <code>DataPipes</code> instead of <code>Dataset</code>) but for sharding_filter( ). That said, I’ll restate my point : since sharding is by default applied across worker processes, I don’t think there should be duplication “by default”.</p>
<p>Feel free to point me to any resources that state otherwise.</p>

username: nivek
cooked: <blockquote>
<p>Does the <code>DataLoader</code> use <code>RandomSampler</code> by default? (default value of the <code>sampler</code> argument of <code>Dataloader</code> is <code>None</code>)</p>
</blockquote>
<p>As <a href="https://pytorch.org/docs/stable/data.html#data-loading-order-and-sampler" rel="noopener nofollow ugc">discussed in this section</a>, " a sequential or shuffled sampler will be automatically constructed based on the <code>shuffle</code> argument to a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc"><code>DataLoader</code></a>".</p>
<p>It uses a sequential sampler when <code>False</code>, shuffled sampler when <code>shuffle=True</code>, unless you pass in a custom sampler.</p>
<blockquote>
<p>Also at the link you mentioned, what exactly does “on-demand” mean in the following: <code>samples are drawn on-demand with replacement</code></p>
</blockquote>
<p>On-demand means the new index is created lazily (like a generator), in contrast with generated ahead of time and saved in a buffer.</p>

username: theory_buff
cooked: <p>Thanks <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a> , <a class="mention" href="/u/nivek">@nivek</a> .</p>