How to choose the value of the num_workers of Dataloader

crissallan: <p>I run models on a machine with 8 core CPU and NVIDIA v100, how should I choose the num_workers to make the data be loaded efficiently.</p>

albanD: <p>Hi,</p>
<p>Unfortunately, there is no absolute true value. It will depend on your setup.<br>
For low number of workers, you should see an improvement whenever you add more of them. At some point, it will stop improving and just keep the same performances even with more workers. You should use the lowest number that gives you good performances.</p>

crissallan: <p>Many thanks! Actually, I’m think there is a bottle neck of my training process and I’m trying to find it out. I am training a 3-layers MLP model but the training is irrationally slow.<br>
Though I did a preprocessing of data in the training loop, I think it won’t influence the training time too much, because the preprocessing should be accelerated by CUDA. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/48e6fd1a3420891742a3919e44843f917740fcdb.png" data-download-href="https://discuss.pytorch.org/uploads/default/48e6fd1a3420891742a3919e44843f917740fcdb" title="09"><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/48e6fd1a3420891742a3919e44843f917740fcdb.png" alt="09" data-base62-sha1="aoVgFpTxblsH5AyJ5StA18a0Odd" width="690" height="260" data-dominant-color="2F2F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">09</span><span class="informations">1278×482 31.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> So I think whether the data loader is working efficiently. The data loader load a np.array object in the .<strong>getitem</strong> method.</p>
<p>The most weird thing is the training seems be slower epoch by epoch, though I use a torch.cuda.empty_cache, it doesn’t help much.<br>
A preload is also implement, so the GPU is not starving.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/7103416f3f402a5ff3675dbfe1c238b78515bd69.png" data-download-href="https://discuss.pytorch.org/uploads/default/7103416f3f402a5ff3675dbfe1c238b78515bd69" title="48"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7103416f3f402a5ff3675dbfe1c238b78515bd69_2_456x500.png" alt="48" data-base62-sha1="g7KVC4A4eogrG4415PPvRIf2yYV" width="456" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7103416f3f402a5ff3675dbfe1c238b78515bd69_2_456x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/7103416f3f402a5ff3675dbfe1c238b78515bd69_2_684x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/7/7103416f3f402a5ff3675dbfe1c238b78515bd69_2_912x1000.png 2x" data-dominant-color="353536"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">48</span><span class="informations">1168×1278 152 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

crissallan: <p>Oh another question is that is num_workers related with multi-threading or multi-processing?</p>

alex.veuthey: <p>Yes, <code>num_workers</code> is the total number of processes used in data loading.</p>
<p>I’ve found <a href="https://discuss.pytorch.org/t/guidelines-for-assigning-num-workers-to-dataloader/813/5">here</a> the general recommandation of using 4 workers per GPU, and I’ve found that it works really well with my own setup, but that might not be universal… <a class="mention" href="/u/alband">@albanD</a>’s method (adding more until it peaks) is probably the best way to find what works for you.</p>

albanD: <p>You should not use <code>empty_cache()</code> as it can only slow down your code.<br>
If the GPU is used at 99%, then there isn’t much more you can do I’m afraid.<br>
You can try the <a href="https://pytorch.org/docs/stable/bottleneck.html" rel="nofollow noopener"><code>bottleneck</code></a> tool maybe to get more information.</p>