Iterating batch in DataLoader takes more time after every nth iteration

aswamy: <p>My data is both image and the compressed(.npz) files that contain 3d points and labels for each 3d points(contains a minimum of 64<em>64</em>64 points and each file size varies from 10MB to 45MB). num_workers=4, batch_size=32, points_sampled for every batch is 2048.</p>
<p>Total number of parameters: 13414465</p>
<p>t<strong>rain batch 1 load time 21.43259024620056s</strong><br>
train batch 2 load time 0.031423091888427734s<br>
train batch 3 load time 0.004406452178955078<br>
train batch 4 load time 0.004347562789916992<br>
<strong>train batch 5 load time 18.13344931602478</strong><br>
train batch 6 load time 0.004399538040161133<br>
train batch 7 load time 0.03353142738342285<br>
train batch 8 load time 0.004467010498046875<br>
<strong>train batch 9 load time 16.202253103256226</strong><br>
train batch 10 load time 0.8377358913421631</p>
<p>I understand that input file is too large and I load it from a network location, this could cause the slowdown.<br>
Questions are…</p>
<ol>
<li>Why is it slow only after <span class="hashtag">#num_workers</span> iteration?</li>
<li>How can we make it faster loading?</li>
</ol>
<p>Thanks<br>
Anil</p>

ptrblck: <ol>
<li>
<p>Each worker preloads a complete batch. If all start at the same time, they might finish close to each other and in your case 4 batches might be ready. Your actual model workload seems to be small in comparison to the data loading so that the training using these 4 batches finishes quickly. Meanwhile the workers already started to load the new batches, but cannot keep up with the model training so you have clearly a data loading bottleneck in your code.</p>
</li>
<li>
<p>Have a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a> for a general explanation and some advice.</p>
</li>
</ol>

aswamy: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for the answer. Does this mean that using maximum possible num_workers is efficient? assuming some data loading bottleneck remains.</p>

ptrblck: <p>I think you would get the best performance using a “sane” number of workers, which depend on your overall system. E.g. 4 workers might work well for local workstations, but might not be enough for bigger servers.</p>

aswamy: <p>Experimented and understood, thank you!</p>

yzholdfish: <p>the default value of prefetch_factor is 2, why the iteration is num_workers instead of prefetch_factor * num_workers.i’m confused</p>

ptrblck: <p>The <code>prefetch_factor</code> doesn’t accelerate the data loading so if the workers are too slow and each worker can finish to load a single batch, the slowdown will happen after these 4 batches are consumed as described in the previous post.</p>

yzholdfish: <p>thank you for your answer, In the data loader source code, prefetch_ factor acts on _reset function.I increased the prefetch_factor, resulting in more obvious periodicity. As you say, prefetch_factor cannot accelerate the data loading then what is it real role.</p>

ptrblck: <aside class="quote no-group" data-username="yzholdfish" data-post="8" data-topic="101881">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yzholdfish/48/49941_2.png" class="avatar"> yzholdfish:</div>
<blockquote>
<p>As you say, prefetch_factor cannot accelerate the data loading then what is it real role.</p>
</blockquote>
</aside>
<p>It’s role is to create a “buffer” by adding samples to a queue so that the training doesn’t stall assuming:</p>
<ul>
<li>the data loading is fast enough and</li>
<li>the model training workload is large enough compared to the data loading.</li>
</ul>
<p>It seems the previous example might still be unclear so try to imagine the timeline if your workload uses:</p>
<ul>
<li>10 seconds to load a single batch</li>
<li>0.25s second to train the model (forward/backward/step) using a single batch.</li>
</ul>
<p>In this case, the initial “warmup” time will take ~10s to allow each worker to load a single batch.<br>
The queue will now be filled with 4 ready batches. The workers will immediately start loading the next batch (as the prefetching kicks in) which will take another ~10s.<br>
The model finishes the training using all 4 batches in 1s (4x0.25s) and has to wait for the next data to be ready. Since that’s not the case, you will see a stall in the training regardless how high the <code>prefetch_factor</code> is set.</p>

yzholdfish: <p>a good example helps i understand your what you mean.<br>
but i am still confused that prefetch_factor’s action position.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65085e2d9e9fddd4f5cf4b5cb4801d5b7c131030.jpeg" alt="image" data-base62-sha1="epM9sGoVJ568S7ihAme9sDVZanK" width="482" height="252"><br>
the experiments’ factor: num_process =16, prefetch_factor =2<br>
the first 32 batches took less time, i think prefetch_factor works,but after that the iteration is 16 = num_workers<br>
so im confused that it just work in the initial.Or it works in the whole training, i understand in the wrong way.</p>

ptrblck: <p>Sorry, the image is unfortunately too small and I’m unsure what exactly it shows. Could you upload a larger version somewhere?</p>

yzholdfish: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18a7ee5022b2bbb37d36430ab15b2880f17a7498.png" data-download-href="https://discuss.pytorch.org/uploads/default/18a7ee5022b2bbb37d36430ab15b2880f17a7498" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18a7ee5022b2bbb37d36430ab15b2880f17a7498_2_690x362.png" alt="image" data-base62-sha1="3w7ftugru80qhjcnYuNpYLAQe0w" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18a7ee5022b2bbb37d36430ab15b2880f17a7498_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18a7ee5022b2bbb37d36430ab15b2880f17a7498_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18a7ee5022b2bbb37d36430ab15b2880f17a7498_2_1380x724.png 2x" data-dominant-color="FDFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1570×824 46.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
xlabel batch_index<br>
ylabel dataloader_itertime</p>

Bernard_Han: <p>Coming back at this in 2024 – we are facing a similar issue and I’m confused too because there should have been about prefetch_factor * num_workers batches fetched in advance before step 0 starts. However, I’m seeing slowness every <span class="hashtag-raw">#num_workers</span> steps.</p>
<p>Should data loading be the bottleneck, shouldn’t we expect to see slowness every prefetch_factor * num_workers step but not every num_workers steps?</p>
<p>I can draft a more detailed post too. Thanks in advance.</p>

ptrblck: <aside class="quote no-group" data-username="Bernard_Han" data-post="13" data-topic="101881">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bernard_han/48/72568_2.png" class="avatar"> Bernard_Han:</div>
<blockquote>
<p>Should data loading be the bottleneck, shouldn’t we expect to see slowness every prefetch_factor * num_workers step but not every num_workers steps?</p>
</blockquote>
</aside>
<p>Not necessarily.</p>
<ul>
<li>Imagine the current iteration needs to wait for an input batch to be ready and you are thus seeing a wait time.</li>
<li>After the queue is filled with a data sample, the training iteration starts and the workers start loading the next batch.</li>
<li>The training iterations continue until the queue is empty again and waits. Let’s assume all workers have added their batch into the queue.</li>
<li>If the training iteration finishes before any of the workers was able to push a new batch into the queue, you will see a wait time every <code>num_workers</code>.</li>
</ul>
<p>The <code>prefetch_factor</code> defines how many batches each worker is is allowed to prefetch, but does not mean we have to wait in each iteration for the queue to be filled again (this wouldn’t be useful since you would fall back to a <code>prefetch_factor=1</code> use case).</p>