Why are all PyTorch dataloader proccesses in S state (interruptible sleep) except one?

PabloMessina: <p><strong>Short version</strong>: I’m using PyTorch dataloading library to load data in parallel for training a deep learning model. When I look at the CPU usage with <code>htop</code>, I see a bunch of processes running the same python script but only one of them is in R state (<em>running or runnable</em>) using 150% CPU, whereas all the others are in S state (<em>interruptible sleep</em>), using around 8% CPU each.</p>
<p><strong>More details</strong>:</p>
<p>I’m training a VQA model with a bunch of auxiliary tasks. In terms of data, this means I’m loading several lists, numpy arrays, dictionaries and images. These are queried on demand by datasets via the <code>__getitem__</code> function, which in turn are used by the dataloader processes.</p>
<p>This is what <em>htop</em> looks like:</p>
<p><a href="https://i.stack.imgur.com/MSQJK.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/MSQJK.png" alt="enter image description here" width="690" height="446"></a></p>
<p>This is what <em>nvtop</em> looks like:</p>
<p><a href="https://i.stack.imgur.com/IlU4H.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/IlU4H.png" alt="enter image description here" width="690" height="362"></a></p>
<p>As you can see, only one process is in R state, everything else is in S state.</p>
<p>In this particular example, I’m using two dataloaders for training (concretely, <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/datasets/vqa.py#L296" rel="noopener nofollow ugc">this one</a> and <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/datasets/vqa.py#L879" rel="noopener nofollow ugc">this one</a>), and I’m merging them <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/train_vqa.py#L754" rel="noopener nofollow ugc">here</a> using <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/datasets/dataloading_utils.py#L165" rel="noopener nofollow ugc">this function</a>.</p>
<p>I also build a hierarchy of nested datasets <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/datasets/vqa.py#L291" rel="noopener nofollow ugc">here</a> using <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/datasets/dataloading_utils.py#L62" rel="noopener nofollow ugc">these classes</a>, but the “leaf” atomic dataset class is defined <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/datasets/vqa.py#L67" rel="noopener nofollow ugc">here</a>. I don’t know if having a hierarchy of nested datasets could be (part of) the problem.</p>
<p>The data itself is loaded into memory from disk and prepared <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/datasets/mimiccxr/mimiccxr_vqa_dataset_management.py#L392" rel="noopener nofollow ugc">here</a>, <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/datasets/mimiccxr/mimiccxr_vqa_dataset_management.py#L609" rel="noopener nofollow ugc">here</a>, <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/datasets/mimiccxr/mimiccxr_vqa_dataset_management.py#L623" rel="noopener nofollow ugc">here</a>, and <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/datasets/mimiccxr/mimiccxr_vqa_dataset_management.py#L665" rel="noopener nofollow ugc">here</a>.</p>
<p>I’m profoundly ignorant as to how PyTorch implements multiprocess dataloading and how the data is shared and read across the different processes under the hood. Is there something I’m probably doing wrong that is causing all processes except one to be in <em>interruptible sleep</em> (S) state? How can I have all processes loading data at roughly 100% CPU usage (in R state) to make the most of Pytorch’s parallel dataloading?</p>
<p><strong>Extra details</strong></p>
<p>For this example, I run <a href="https://github.com/PabloMessina/MedVQA/blob/69a095a7d11cc6de0774fab01f38486119395407/medvqa/train_vqa.py#L1679" rel="noopener nofollow ugc">this script</a> with argument --num-workers=3.</p>
<p>This script was invoked inside a jupyter notebook cell using the !syntax (!python myscript.py), which is itself running inside a tmux terminal. Just letting those extra details be known in case they may be part of the problem.</p>

ptrblck: <p>I’m unsure why you would expect to see all workers running the entire time?<br>
Each worker will preload batches based on the specified <code>prefetch_factor</code> and then load the next one on demand. Did you profile your code and see that your GPU is starving, i.e. that you are running into a data loading bottleneck?</p>

PabloMessina: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="172149">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Did you profile your code and see that your GPU is starving, i.e. that you are running into a data loading bottleneck?</p>
</blockquote>
</aside>
<p>I haven’t. What is the easiest way to do that?</p>
<p>The closest I’ve done to profiling my code is monitoring the output of the htop and nvtop commands. nvtop shows 0% usage most of the time, with some spikes in usage at certain intervals, as you can see in the nvtop screenshot above, which gives me the impression that the GPU is starving, but I’m not sure.</p>

ptrblck: <p>You could use the native PyTorch profiler as described <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html">here</a> or Nsight Systems as described <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">here</a>.</p>