Can someone explain how data starvation could be happening here?

username: Vedant_Roy
cooked: <p>Here’s a graph:</p>
<ul>
<li>green = each time a batch is produced by (1 of 8) workers</li>
<li>blue = each time the GPU starts processing a batch</li>
<li>orange = each time the GPU finishes processing a batch</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5ea304576c33f601e783eedd23471a23eb9bd63f.png" data-download-href="https://discuss.pytorch.org/uploads/default/5ea304576c33f601e783eedd23471a23eb9bd63f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5ea304576c33f601e783eedd23471a23eb9bd63f_2_607x500.png" alt="image" data-base62-sha1="dvc8XfwNrZcRPT8oDaLe6rzqoP5" width="607" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5ea304576c33f601e783eedd23471a23eb9bd63f_2_607x500.png, https://discuss.pytorch.org/uploads/default/original/3X/5/e/5ea304576c33f601e783eedd23471a23eb9bd63f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/e/5ea304576c33f601e783eedd23471a23eb9bd63f.png 2x" data-dominant-color="E4E5E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">844×695 111 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We can tell there’s no data loader starvation here because:<br>
As soon as a orange dot appears (which indicates the GPU has finished processing), then a blue dot appears next (indicating the GPU is starting to process the next batch).</p>
<p>Now, here’s a graph with a slower dataloader:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/1928ad49b89a601ee2cc9ee252deb5409b92484b.png" data-download-href="https://discuss.pytorch.org/uploads/default/1928ad49b89a601ee2cc9ee252deb5409b92484b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/1928ad49b89a601ee2cc9ee252deb5409b92484b_2_630x500.png" alt="image" data-base62-sha1="3Az5n0wuiRge61I6x2KMaUBmWhB" width="630" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/1928ad49b89a601ee2cc9ee252deb5409b92484b_2_630x500.png, https://discuss.pytorch.org/uploads/default/original/3X/1/9/1928ad49b89a601ee2cc9ee252deb5409b92484b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/9/1928ad49b89a601ee2cc9ee252deb5409b92484b.png 2x" data-dominant-color="E6E6E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">849×673 111 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We can see that there’s data starvation, namely, because sometimes there is a long gap between an orange dot and the next blue dot.</p>
<p>However, while it is clear the issue is data loader starvation (I know because the fast data loader is identical to the slow data loader, except it just generates random data instead of performing expensive augmentations), I’m not sure <strong>why</strong>.</p>
<p>For example, it is clear that there is a long gap between batch 45 and batch 46. But, we see batch 46 is generated long before it is actually consumed. So I would expect batch 46 to be consumed immediately after batch 45.</p>
<p>I’m not sure what could be causing such a long delay between batch 45 &amp; 46. (And I’m using these 2 batches as an example).</p>
<p>Does anyone know what could be going on here?</p>