Is there any data chache for dataloader?

xugaoqi1993: <p>I found a problem that when I trained with a big dataset(same pictures size. same pre-process, only difference is that there are more pictures), even I set the same batch size and run iterations of dataloader witouht model training, the dataloader of a small dataset can provide data much faster(about 20 batches per second) while the dataloader of a big dataset can only provide 1batches per second.<br>
I ran some experiments about this issues and found that after I roboot the server, I run 10 epoch of iterations of dataloader of the small dataset, the first epoch is also slow(about 1 batch per second). The next 9 epoch is really fast(about 20 batches per second).<br>
I assume there is some data cache when we use the dataloader. However this cache seems to be useless for the dataloader with a  big dataset(130G,1.2million images) because after the first epoch, the speeed of the iterations of dataloader is still slow.<br>
Why is it happen? How can I solve this problem? Please help me.</p>
<p>The iterations of the dataloader with a big dataset:<br>
dataset: 130G 1.2million images<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d98ed0eb00f4f8b8184bcf960dec67eaadbdbfd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9d98ed0eb00f4f8b8184bcf960dec67eaadbdbfd" title="微信图片_20221108154359"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d98ed0eb00f4f8b8184bcf960dec67eaadbdbfd_2_690x204.jpeg" alt="微信图片_20221108154359" data-base62-sha1="muaBwPJyD1ruOgOZHWHr8Qx5U4Z" width="690" height="204" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d98ed0eb00f4f8b8184bcf960dec67eaadbdbfd_2_690x204.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9d98ed0eb00f4f8b8184bcf960dec67eaadbdbfd_2_1035x306.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/d/9d98ed0eb00f4f8b8184bcf960dec67eaadbdbfd.jpeg 2x" data-dominant-color="2E2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20221108154359</span><span class="informations">1336×395 32.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The iterations of the dataloader with a big dataset:<br>
dataset: 13G 120,000 images<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab0ce1c2ac4c90c6154292079e163aa29360f92c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ab0ce1c2ac4c90c6154292079e163aa29360f92c" title="微信图片_20221108154359"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab0ce1c2ac4c90c6154292079e163aa29360f92c_2_690x239.jpeg" alt="微信图片_20221108154359" data-base62-sha1="opbfit8PhDcwodvBd7CLOXsCtmQ" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab0ce1c2ac4c90c6154292079e163aa29360f92c_2_690x239.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/ab0ce1c2ac4c90c6154292079e163aa29360f92c_2_1035x358.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/b/ab0ce1c2ac4c90c6154292079e163aa29360f92c.jpeg 2x" data-dominant-color="2F2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20221108154359</span><span class="informations">1245×432 35.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Two dataset are similar, so two custom dataset class are similar.</p>

ptrblck: <p>Maybe you are seeing a speedup for the tiny dataset due to your system cache.<br>
You could drop it via:</p>
<pre><code class="lang-python">echo 3 &gt; /proc/sys/vm/drop_caches
# or
sync &amp;&amp; sudo /sbin/sysctl vm.drop_caches=3
</code></pre>
<p>and see if the small dataset also slows down again.</p>