Num_workers > 0 makes image loading slower rather than faster

yuval: <p>I am training a resnet-18 network in pytorch with a standard pytorch dataloader. When I increase the number of workers from 0 (no parallelization) to any number (even just to 1), the time for loading a batch increases by a large factor (~x5), instead of decreasing. My CPU utilization is low, and the images are loaded from my local SSD.</p>
<p>Any idea how to overcome this issue?</p>

JuanFMontesinos: <p>How many cores do you use? Main thread is usually faster. Aditionally, how are you measuring that time? Multiprocessing should be slower at the beggining until it reaches its max speed.</p>

yuval: <p>I have 12 cores.<br>
With num_workers=0, the utilization is 5-6 cores@100% and the rest <span class="mention">@10-15</span>%<br>
With num_workers=1, the utilization is <span class="mention">@50-60</span>% on all cores.<br>
With num_workers=2, the utilization is <span class="mention">@60-70</span>% on all cores.</p>
<p>I am measuring time by profiling with NVIDIA nsight-systems<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://developer.nvidia.com/sites/all/themes/devzone_base/favicon.ico" class="site-icon" width="48" height="48">
      <a href="https://developer.nvidia.com/nsight-systems" target="_blank" title="05:16PM - 12 March 2018" rel="nofollow noopener">NVIDIA Developer – 12 Mar 18</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail">

<h3><a href="https://developer.nvidia.com/nsight-systems" target="_blank" rel="nofollow noopener">NVIDIA Nsight Systems</a></h3>

<p>NVIDIA® Nsight™ Systems is a system-wide performance analysis tool designed to visualize an application’s algorithms, help you identify the largest opportunities to optimize, and tune to scale efficiently across any quantity or size of CPUs and GPUs;...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

JuanFMontesinos: <p>In theory mum_workers 0 runs everything on the main thread which should  be faster than using a single worker. Di you find the same behaviour using a great amount of cores like 10 or 20? (Note that you usually set amount of threads not cores which is usually twice as many cores as your cou has)</p>

yuval: <p>I managed to solve the problem.<br>
It is related to the fact that my code used:</p>
<pre><code class="lang-auto">           for i in range(len(train_loader)):
               X, y1, y2 = train_loader.__iter__().__next__()
               &lt;process batch&gt;
</code></pre>
<p>Replacing it with the following solved the problem:</p>
<pre><code class="lang-auto">           for i, (X, y1, y2) in enumerate(train_loader):
               &lt;process batch&gt;
</code></pre>
<p>Thanks for the assistance</p>

kinwai_cheuk: <p>I am facing the same problem. I didn’t know writing the for loop differently would affect the data loading speed so much, and I have been spending hours trying to figure out what is wrong with higher number of workers.</p>

MrCrHaM: <p>But why though? What causes the difference? They are basically the same IMO.</p>

ahmdtaha: <p>This stackoverflow post answers your question</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/53280967/pytorch-nextitertraining-loader-extremely-slow-simple-data-cant-num-worke/53293991#53293991">
  <header class="source">

      <a href="https://stackoverflow.com/questions/53280967/pytorch-nextitertraining-loader-extremely-slow-simple-data-cant-num-worke/53293991#53293991" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1714410/shai" target="_blank" rel="noopener nofollow ugc">
    <img alt="Shai" src="https://www.gravatar.com/avatar/a254f486ab076ab1af26162a975e90f9?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/53280967/pytorch-nextitertraining-loader-extremely-slow-simple-data-cant-num-worke/53293991#53293991" target="_blank" rel="noopener nofollow ugc">PyTorch next(iter(training_loader)) extremely slow, simple data, can't num_workers?</a>
</h4>

<div class="tags">
  <strong>python, performance, machine-learning, iterator</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/1714410/shai" target="_blank" rel="noopener nofollow ugc">
    Shai
  </a>
  on <a href="https://stackoverflow.com/questions/53280967/pytorch-nextitertraining-loader-extremely-slow-simple-data-cant-num-worke/53293991#53293991" target="_blank" rel="noopener nofollow ugc">05:58AM - 14 Nov 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


MrCrHaM: <p>Thanks a lot for the info <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>