Prefetch_factor not working

username: sachinruk
cooked: <p>I had an issue at work where the questions was if I should stream the data from an S3 bucket via the <code>Dataset</code> class, or if I should download first and simply read it in. I was hoping that increasing <code>prefetch_factor</code> in dataloaders would increase the speed when streaming it via S3, and possibly even be an alternative to downloading.</p>
<p>In order to stream data instead of opening from disk (as is done in many tutorials) the dataset class was setup as the following:</p>
<pre><code class="lang-python">class Data(Dataset):
    def __init__(self, prefix, transform):
        self.prefix = "https://aft-vbi-pds.s3.amazonaws.com/bin-images"
        self.transform = transform
        
    def __len__(self):
        return 999
    
    def __getitem__(self, i):
        response = requests.get(self.prefix + f"/{i+1}.jpg")
        img = Image.open(BytesIO(response.content))
        return self.transform(img)
</code></pre>
<p>As shown in the experiments <a href="https://www.kaggle.com/sachin/pytorch-prefetch-experiment" rel="noopener nofollow ugc">done in this kaggle kernel</a>, <code>prefetch_factor</code> flag did not speed things in a meaningful manner. The results are summarisd below. For each iteration the following code snippet was run, where model is simply resnet18.</p>
<pre><code class="lang-python">with torch.inference_mode():
    for img_batch in tqdm(dl):
        out = model(img_batch.to(device))
</code></pre>
<div class="md-table">
<table>
<thead>
<tr>
<th>Settings</th>
<th>Time Elapsed</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>num_workers = 2</code></td>
<td>04:02</td>
</tr>
<tr>
<td>
<code>num_workers = 2</code>, <code>prefetch_factor=8</code>
</td>
<td>03:57</td>
</tr>
<tr>
<td><code>num_workers = 8</code></td>
<td>1:01</td>
</tr>
<tr>
<td>
<code>num_workers = 8</code>, <code>prefetch_factor=8</code>
</td>
<td>1:01</td>
</tr>
</tbody>
</table>
</div><p>All other parameters such as <code>batch_size=32</code>, <code>pin_memory=True</code> was held constant across all iterations.</p>
<p>Note that the reason we had 2 workers was due to the fact that this was the number given by <code>multiprocessing.cpu_count()</code>. However, going past that number in the last iteration and setting it at 8 worked, and gave the following ugly (repeated) warnings: <code>Exception ignored in: Exception ignored in: &lt;function _MultiProcessingDataLoaderIter.__del__</code>.</p>
<p>Any thoughts as to why <code>prefetch_factor</code> doesn’t work?</p>

username: ptrblck
cooked: <p>I’m not sure I understand why the <code>prefetch_factor</code> should speed up the data loading.<br>
The docs explain this argument as:</p>
<blockquote>
<p>Number of samples loaded in advance by each worker.</p>
</blockquote>
<p>If your workers are not fast enough to preload data, increasing the <code>prefetch_factor</code> won’t help.</p>

username: sachinruk
cooked: <p>I see, I was hoping that it worked something like the <a href="https://www.tensorflow.org/guide/data_performance#prefetching" rel="noopener nofollow ugc">tensorflow prefetch</a> where it would load the next batch while the GPU was busy.</p>
<p>I might be wrong, but shouldn’t the fact that I increased num_workers (even though mp.cpu_count() is 2) and sped it up mean that there would be room for improvement via prefetching?</p>

username: ptrblck
cooked: <p>The <code>DataLoader</code> is already prefetching the data (the default for the <code>prefetch_factor</code> is set to 2) and (as you’ve pointed out) increasing the number of workers seems to improve the loading speed.<br>
You could profile the data loading in isolation to see how the “vertical/horizontal” scaling plays out in your system (i.e. increasing the parallel workload via <code>num_workers</code> vs. storing more batches sequentially via increasing the <code>prefetch_factor</code>). Based on your experiments the latter doesn’t help and I don’t think you can universally expect a speedup by increasing the <code>prefetch_factor</code>.</p>

username: Vinayaka_Hegde
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I don’t understand the exact use of the <code>prefetch factor</code> argument.</p>
<p>Acc. to the docs, it is <code>Number of batches loaded in advance by each worker.</code> My question is - since prefetching is a one-time thing done by the workers at the start of data loading, it might give a slight speedup for the initial batches to be loaded, but in the long run, the data loading by the workers would fail to catchup with the GPU speed and there would be stalls (when there are no more elements in the data queue) until the worker fetches the next batch.</p>
<p>therefore, <code>num_workers</code> argument makes sense as more the workers, the faster we would be loading into the data queue for the GPU to consume the batches, but <code>prefetch_factor</code> does not make sense to me.</p>
<p>Could you please help ? Thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Vinayaka_Hegde" data-post="5" data-topic="144077">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vinayaka_hegde/48/56332_2.png" class="avatar"> Vinayaka_Hegde:</div>
<blockquote>
<p>My question is - since prefetching is a one-time thing done by the workers at the start of data loading, it might give a slight speedup for the initial batches to be loaded, but in the long run, the data loading by the workers would fail to catchup with the GPU speed and there would be stalls (when there are no more elements in the data queue) until the worker fetches the next batch.</p>
</blockquote>
</aside>
<p>That’s not the case since each worker will start loading the next batch once the number of batches waiting in the queue is not meeting the prefetch factor.<br>
<a href="https://discuss.pytorch.org/t/when-does-a-pytorch-dataset-worker-pull-new-data/153286/2">This code snippet</a> illustrates it.<br>
As you can see the creation of the iterator will directly load <code>prefetch_factor * num_workers = 2 * 8 = 16</code> batches, so <code>nb_batches * batch_size = 16 * 2 = 32</code> samples.<br>
The <code>next</code> call then returns a batch and you can see that one of the workers directly starts to load the next batch (2 samples) to fill the queue.</p>

username: Vinayaka_Hegde
cooked: <p>Ok makes sense now. So, the main use of <code>prefetch_factor</code> is to ensure that we always have <code>prefetch_factor * num_workers</code> number of batches in the data queue for the GPU to consume right?</p>

username: ptrblck
cooked: <p>Yes, this is the idea. So even if some training iterations might be faster (for some reason) the queue could still be filled and you should not see any slowdown due to the data loading and the workers will try to fill the queue again.</p>

username: Croydon-Brixton
cooked: <p>Thank you for the input <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<p>I was wondering whether the prefetching happens in a round-robin fashion (i.e. the GPU queries worker 1, then 2, then 3, …) or whether it pulls in entirely at random order from the buffer?</p>
<p>The reason I am asking is because I have a process where different samples have very different loading times (e.g. one sample might take 4s while another is instant). Would the worker that’s pulling the slow 4s example hold up the GPU at some point, or could the GPU work with instant samples from the other workers until this sample is ready?</p>

username: ptrblck
cooked: <p>By default the worker order should be guaranteed for determinism. You could check it by creating a toy example and letting each worker return samples with its own ID.</p>