Multi-process data loading and prefetching

username: claudiacorreia60
cooked: <p>From what I understand the worker processes of the Dataloader fetch batches instead of fetching samples. Is there a way of fetching samples instead of batches?</p>
<p>Also, when setting <code>num_workers &gt; 0</code>, by default each worker prefetches 2 samples in advance. I don’t understand exactly how this works. Each worker prepares a batch and reads 2 samples in advance for the next batch? Do the workers prefetch samples for the next training epoch before it starts?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The Dataloader fetches batches so that it can perform all the preprocessing and creation on the batch on the worker process and have as few things as possible to do in the main process once the batch is ready.<br>
Why would you want workers to load samples only?</p>
<p>Each worker prefetches 2 batches in advance to make sure that when the main process asks for the next batch, there is always one ready.<br>
Note that if you use nightly build, you can control that number with the <code>prefetch_factor</code> argument to the dataloader (doc here: <a href="https://pytorch.org/docs/master/data.html#torch.utils.data.DataLoader">https://pytorch.org/docs/master/data.html#torch.utils.data.DataLoader</a>)</p>

username: claudiacorreia60
cooked: <p>Hi,</p>
<p>Thank you very much for your answer!</p>
<p>Ok, I now understand why the workers fetch batches instead of samples.</p>
<p>Do the workers fetch batches for the next epoch before it starts, or the batches of an epoch only start being fetched when the epoch starts?</p>
<p>On another subject, I noticed that when I choose <code>batch_size=64</code>, worker 1 reads the first 64 indexes, worker 2 reads the next 64 indexes, and so on. Is there a way of having workers reading interleaved indexes?<br>
For example, when <code>num_workers = 3</code>:</p>
<ul>
<li>worker 1 reads indexes 1, 4, 7, 10…</li>
<li>worker 2 reads indexes 2, 5, 8, 11…</li>
<li>worker 3 reads indexes 3, 6, 9, 12…</li>
</ul>
<p>Thank you!</p>

username: albanD
cooked: <aside class="quote no-group" data-username="claudiacorreia60" data-post="3" data-topic="98997">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/claudiacorreia60/48/29671_2.png" class="avatar"> claudiacorreia60:</div>
<blockquote>
<p>Do the workers fetch batches for the next epoch before it starts, or the batches of an epoch only start being fetched when the epoch starts?</p>
</blockquote>
</aside>
<p>It only start with the epoch, when the iterator is created. Meaning, when you do:</p>
<pre><code class="lang-auto">for sample in dataloader:
</code></pre>
<p>You can specify a <code>sampler</code> (<a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader">doc</a>) when you create the Dataloader that is reponsible to draw the samples.<br>
You can control the sampler to force a certain pattern in the content of the batch (and thus what the workers will load).</p>

username: claudiacorreia60
cooked: <p>Ok, I will try that.</p>
<p>Thank you very much for your help!</p>