DataLoader parameters have no or little impact to training speed

username: qoob
cooked: <p>Hey there, I’m training a DeTR model with around 0.9M images, each paired with a XML file that contains multiple annotations for that image. I upgraded my GPU from GTX 980 Ti to RTX 4090 and only ~doubled performance, when I expected much more. After asking around many have agreed that I’m likely bottlenecked at the data-loading part of my training script.</p>
<p>I’ve initialized the data as <code>torch.utils.data.Datasets</code>, and it is passed to a <code>torch.utils.data.DataLoader</code>. In hopes of removing the bottleneck I’ve now tested out multiple combinations of settings for <code>DataLoader</code> without success.</p>
<ul>
<li>
<code>pin_memory</code> set to True and False, seems to make no difference whatsoever</li>
<li>
<code>num_workers</code> in a range from 0 to 12 (amount of CPU cores in my Intel i7-8700K). So far the best results are with 1.</li>
<li>
<code>prefetch_factor</code> with setting [null, 2, 8, 64, 256, 1024]. 2 seems to be the quickest, although I have a lot of GPU and CPU RAM available at that setting and training is slow.</li>
<li>
<code>persist_workers</code> set to True and False. Seems to make no difference.</li>
<li>
<code>batch_size</code> I’ve kept at 2 for these tests, because that is what the authors used. Each epoch is around 400K steps with this setting.</li>
</ul>
<p>About the hardware and what happens during training:</p>
<ul>
<li>All the data is stored on a NVMe SSD, from where everything is also installed (&gt;1Gb/s reads and writes). During training I don’t see much stress on the SSD.</li>
<li>RAM hovers between 10-64Gb/64Gb at different setups. Only configuration that has capped RAM was <code>num_workers=12</code> combined with <code>prefetch_factor=256</code>. In those cases RAM loads to around 64GB and then quickly starts to offload, even before ~1000 steps have passed. I don’t understand why data loads up to RAM and slowly is removed from it.</li>
<li>CPU is usually at 30-50%. It goes to CPU only in configurations where a lot of data is loaded to RAM and stays there for that period, and then quickly drops back down to 30-50 %. Changing <code>num_workers</code> doesn’t impact CPU.</li>
<li>My RTX 4090 is hovering around 50-100 % in spikes, not even close to full capacity. In terms of GPU RAM in use, at this <code>batch_size</code> I only consume between 5-9 Gb. I can go up to <code>batch_size=8</code> and then GPU RAM is at +20Gb/24Gb.</li>
</ul>
<p>I would expect the above parameters to improve training speed, but nothing seems to be helping.</p>
<p>Here is a <a href="https://github.com/microsoft/table-transformer/blob/02a2a4eded9ccf70be356fad405bc9555b4ad551/src/table_datasets.py#L613" rel="noopener nofollow ugc">link to what __getitem__ does</a></p>
<p>Here is an <a href="https://github.com/microsoft/table-transformer/blob/02a2a4eded9ccf70be356fad405bc9555b4ad551/src/table_datasets.py#L460" rel="noopener nofollow ugc">example of the transforms that are done to images</a> at each pass.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="qoob" data-post="1" data-topic="181750">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/qoob/48/61143_2.png" class="avatar"> qoob:</div>
<blockquote>
<p>After asking around many have agreed that I’m likely bottlenecked at the data-loading part of my training script.</p>
</blockquote>
</aside>
<p>Did you profile the code to verify it or is this just a guess?</p>

username: qoob
cooked: <p>This is just a guess and I did profile the code afterwards. To be frank from the results I can’t tell much, but maybe someone else is better at interpreting them (below).</p>
<p>For example when I look for <code>TightAnnotationCrop</code> which is done to every single image, the profiler says it’s called only once. <code>read_pascal_voc</code> is used to load the annotations from XML files and that is called for 104K times (once for each batch), taking a total of 12 seconds / 11 hours. But what I don’t understand is that it says <code>cumtime</code> is 63, meaning that the script is done loading XML files after the first minute of operation, before training starts.</p>
<p>Another thing I noticed after posting is that there are indeed 12 Python processes running after setting <code>num_workers</code> to 12, and each process consumes around 0.2Gb RAM, and uses ~1-5% CPU to load something on the disk, at around 0.5 - 5 Mb/s.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/914313e995279563ecad2c14bddf20fc59d441c0.png" data-download-href="https://discuss.pytorch.org/uploads/default/914313e995279563ecad2c14bddf20fc59d441c0" title="Zoom to train_one_epoch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914313e995279563ecad2c14bddf20fc59d441c0_2_690x381.png" alt="Zoom to train_one_epoch" data-base62-sha1="kJ2WH7EEQFUTPJGix6puVuWo4q4" width="690" height="381" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914313e995279563ecad2c14bddf20fc59d441c0_2_690x381.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914313e995279563ecad2c14bddf20fc59d441c0_2_1035x571.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/1/914313e995279563ecad2c14bddf20fc59d441c0_2_1380x762.png 2x" data-dominant-color="474E53"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Zoom to train_one_epoch</span><span class="informations">1865×1031 34.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Made an album of images with more captions: <a href="https://imgur.com/a/KiNm3C9" rel="noopener nofollow ugc">https://imgur.com/a/KiNm3C9</a></p>