Reduce Idleness Between Batch Loads

johnorford: <p>12 workers supply the GPU with data. When doing inference, the GPU is shows:</p>
<ul>
<li>100% GPU utilisation</li>
<li>all GPU RAM is used</li>
</ul>
<p>HTop shows more-or-less that:</p>
<ul>
<li>none of the CPUs are bottlenecked.</li>
</ul>
<p>But in between batch loading, there are moments when the GPU % drops to zero.</p>
<p>I suspect this is the main reason for low GPU utilisation.</p>
<p>Is there any way to reduce this idleness?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/381c280af068e169a5810c0ed93a1591d7e38bc8.png" data-download-href="https://discuss.pytorch.org/uploads/default/381c280af068e169a5810c0ed93a1591d7e38bc8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/381c280af068e169a5810c0ed93a1591d7e38bc8_2_690x226.png" alt="image" data-base62-sha1="80n3UTZi7GFF5mOU0ZlSNIRUhhm" width="690" height="226" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/381c280af068e169a5810c0ed93a1591d7e38bc8_2_690x226.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/381c280af068e169a5810c0ed93a1591d7e38bc8_2_1035x339.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/381c280af068e169a5810c0ed93a1591d7e38bc8_2_1380x452.png 2x" data-dominant-color="E9E8EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2132×700 114 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have many more avenues to improve performance, but this seems blindingly obvious, but I am unsure how to improve this.</p>
<p>Thanks!</p>

nlgranger: <p>Can you describe the structure of the batch and the size of the tensors?<br>
I assume you have memory pinning enabled and <code>non_blocking=True</code>?</p>

johnorford: <p>Hi,</p>
<p>Here are some more details:</p>
<p>Dataloader:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f45e975b0d126388bc92abba23b672ddc4f78a61.png" alt="image" data-base62-sha1="yRNa8LJzaAh9O0FOgMLCRLc7itH" width="517" height="111"></p>
<p>Collate function:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d53fe207a1779d199828152de8661f696f0bbe87.png" alt="image" data-base62-sha1="uquzk7irsEoIOMHrMhjLOjwlX2D" width="432" height="268"></p>
<p><code>non_blocking</code> etc.:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da3362603a2d255628dadd86d9580174e351c7ad.png" alt="image" data-base62-sha1="v8ib5A9PmgLHNF2K1B34CEfgpyt" width="468" height="199"></p>
<p><code>img</code> size (actually the <code>stack</code> of images): <code>torch.Size([2010,3,256,256])</code></p>

johnorford: <p>OK, I realised I didn’t read the <code>workers</code> argument correctly! All good now.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji only-emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>