Higher CPU usage with DataLoader num_workers=0

claudiacorreia60: <p>I was training AlexNet on the ImageNet dataset and decided to vary the <code>num_workers</code> argument of the Dataloader, to see the impact it had. When analyzing the CPU usage, I found that the usage is higher with <code>num_workers=0</code> than with <code>num_workers</code> set to <code>2</code>, <code>4</code>, or <code>8</code> (the results are presented in the graph). I know the difference is not much, but is there a reason for this to happen?<br>
I don’t know if it’s important, but the experiments were run with <code>pin_memory=True</code>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a741bd9545aaf120df0953e28f9eb675da9c4c9f.png" data-download-href="https://discuss.pytorch.org/uploads/default/a741bd9545aaf120df0953e28f9eb675da9c4c9f" title="alexnet-pytorch-avg-cpu"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a741bd9545aaf120df0953e28f9eb675da9c4c9f_2_690x414.png" alt="alexnet-pytorch-avg-cpu" data-base62-sha1="nRCAkPdxfGvh8WiGr1clN9L8JLV" width="690" height="414" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a741bd9545aaf120df0953e28f9eb675da9c4c9f_2_690x414.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a741bd9545aaf120df0953e28f9eb675da9c4c9f_2_1035x621.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a741bd9545aaf120df0953e28f9eb675da9c4c9f_2_1380x828.png 2x" data-dominant-color="E3E3E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">alexnet-pytorch-avg-cpu</span><span class="informations">1500×900 18.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Note: The results shown in the graph are the average of 5 runs</p>

ptrblck: <p>How did you plot the CPU usage? Is is the peak usage of a single core or some kind of mean values of all cores? A usage of 6% also seems to be quite low.</p>

claudiacorreia60: <p>I used <code>dstat</code> to collect the CPU usage. The values presented in the plot correspond to the system CPU + user CPU. The experiments were performed on a machine with a 2.40 GHz and 20 cores CPU, I think that is the reason why the usage is so low.</p>

ptrblck: <p>Thanks for the information. I just tested a quick resnet training on ImageNet using 5 workers vs. 0 workers on an <code>AMD EPYC 7742 64-Core Processor</code> and see that the 5 workers use:</p>
<pre><code class="lang-auto">--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
 12   1  86   1   0|   0    32k|  31M 2301k|   0     0 |  81k   86k
[...]
</code></pre>
<p>while no additional workers (loading in the main process) uses:</p>
<pre><code class="lang-auto"> 3   0  97   0   0|   0    56k| 259k  303k|   0     0 |  13k   14k
[...]
</code></pre>

claudiacorreia60: <p>Ok, so the problem must be on my setup. Did you run with <code>pin_memory=True</code>? Additionally, I was using <code>torch.nn.DataParallel</code> to train with 4 GPUs, and the dataset was stored in <code> torch.Tensor</code>s with the images already pre-processed, to avoid having to do the pre-processing during training. Do you think any of these factors may have caused the differences in CPU usage?</p>
<p>Thank you very much for your help!</p>

ptrblck: <p>Yes, I used <code>pin_memory=True</code> and all 8x GA100s on this node. Also I loaded and augmented the ImageNet data on-the-fly.<br>
The difference would be that I’ve used <code>DistributedDataParallel</code> with a single process per GPU, while you are using <code>nn.DataParallel</code>, which might cause a different CPU usage (unsure, as I haven’t verified it).<br>
Could you try to use DDP and check, if the usage is still the same (or remove <code>DataParallel</code>)?</p>

claudiacorreia60: <p>Thanks for the information you provided. Unfortunately, I no longer have access to the infrastructure where I ran the experiments, so I cannot see what happens with DDP. However, do you have an idea why <code>nn.DataParallel</code> may be causing the differences in CPU usage?</p>

ptrblck: <p>No, I don’t have a good idea what might be causing it. However, based on the initial description of the issue, it would be interesting to see the CPU workload for the <code>DataLoader</code> only in your setup to check, if we see any unexpected behavior there.</p>