Dataloader lmdb very slow when num_workers > 1 or DDP

bring728: <p>The dataset I use is about 500,000 images. It took a long time to load the data, so I made an lmdb file for it. And I made a dataloader using lmdb. (reference : <a href="https://github.com/thecml/pytorch-lmdb" class="inline-onebox" rel="noopener nofollow ugc">GitHub - thecml/pytorch-lmdb: A simple Lightning Memory-Mapped Database (LMDB) converter for ImageFolder datasets in PyTorch. Using LMDB over a regular file structure improves I/O performance significantly. Works on both Windows and Linux. Comes with latest Python support.</a>)</p>
<p>And using this, I trained the network.</p>
<p><strong>1 - experimenting with num_workers=0 on pytorch single gpu(process)</strong><br>
it takes only 1~2ms to load the batch for the area cached at the beginning of iteration. Then, after some iteration, it takes about 0.2 seconds to load because caching is not done.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/532677a8a8ede2266a1eb172e6b5d90a8781dc93.png" data-download-href="https://discuss.pytorch.org/uploads/default/532677a8a8ede2266a1eb172e6b5d90a8781dc93" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/532677a8a8ede2266a1eb172e6b5d90a8781dc93_2_690x241.png" alt="image" data-base62-sha1="bRA2Rf03NNMdlDkGHXcQdjl4Ofp" width="690" height="241" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/532677a8a8ede2266a1eb172e6b5d90a8781dc93_2_690x241.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/532677a8a8ede2266a1eb172e6b5d90a8781dc93_2_1035x361.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/3/532677a8a8ede2266a1eb172e6b5d90a8781dc93.png 2x" data-dominant-color="2F2E2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1059×371 82.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When num_workers = 1, it behaved the same as num_workers = 0.</p>
<p><strong>2 - experimenting with num_workers &gt; 1</strong><br>
similarly, it takes only 1ms to load data at the beginning of the iteration, but after the iteration is repeated to some extent, it takes 80 seconds or tens of seconds, not 0.2 seconds, as in the image below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1caf7a7f6a8a793a8e3958fc498d9f007928129.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e1caf7a7f6a8a793a8e3958fc498d9f007928129" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1caf7a7f6a8a793a8e3958fc498d9f007928129_2_690x274.jpeg" alt="image" data-base62-sha1="wdshLmntFCfaEM9UuYPRYlGNiop" width="690" height="274" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1caf7a7f6a8a793a8e3958fc498d9f007928129_2_690x274.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1caf7a7f6a8a793a8e3958fc498d9f007928129_2_1035x411.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1caf7a7f6a8a793a8e3958fc498d9f007928129.jpeg 2x" data-dominant-color="322F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1051×418 65.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When using DDP, even if num_worker is 0, it often takes tens of seconds instead of 0.2 seconds.<br>
It’s probably a problem when accessing the lmdb file in multi-process, how can I solve it?</p>
<p>For reference, even when reading an image file without using lmdb, there was still a problem that it was fast at the beginning of the iteration but became too slow after a while. I am trying to use lmdb to solve this.</p>
<p>One characteristic is that CPU utilization drops considerably when it’s too slow. Conversely, when it is fast (when data is read quickly), CPU utilization is high.</p>

ptrblck: <aside class="quote no-group" data-username="bring728" data-post="1" data-topic="153183">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bring728/48/50128_2.png" class="avatar"> bring728:</div>
<blockquote>
<p>One characteristic is that CPU utilization drops considerably when it’s too slow. Conversely, when it is fast (when data is read quickly), CPU utilization is high.</p>
</blockquote>
</aside>
<p>Could you check if your system is throttling the clock frequencies e.g. due to overheating?<br>
Based on your current results it seems as if the multi-processing approach works at the beginning and then suffers from a slowdown.</p>

bring728: <p>Thank you for answer. For those who will read this later, the problem was with the HDD. Regardless of whether you use lmdb or the number of num_workers, accessing the data in the HDD itself was too slow. When data is stored on SSD, the data loading speed is more than 10 times faster.</p>
<p>So, in my opinion, if possible, it is better to avoid storing data on HDD during deep learning training.</p>

orena1: <p>you should not use an HDD for anything pytorch related.</p>