CPU usage far too high and training inefficient

username: Raphikowski
cooked: <p>Hi all,</p>
<p>I am training my model on the CPU. A very strange behaviour occured (that I could solve) but I thought I would bring it up because I cannot imagine that this is a desired behaviour:</p>
<p>So when I just train my model on the CPU on my PC with 24 cores, all 24 cores being used 100% even though my model is rather small (thats why I dont train it on the GPU). And most of the workload is also Kernel usage. The training time per epoch requires about 2.5 seconds. I have version 1.0.1.post2 on that PC.</p>
<p>So to make it train faster I pushed it to a server with 80 cores. There, however, I got the exact same behaviour: When training all 80 cores were used with 100% work load. The time per epoch took again about 2.5 seconds on average. On that server I use pytorch version 1.1.0.</p>
<p>Reading through some threads<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/25010" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/rwightman" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/5702664?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/25010" target="_blank" rel="nofollow noopener">Issue: Very high CPU utilization with pin_memory=True and num_workers &gt; 0</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/rwightman" target="_blank" rel="nofollow noopener">rwightman</a>
	on <a href="https://github.com/pytorch/pytorch/issues/25010" target="_blank" rel="nofollow noopener">2019-08-22</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/facebook-github-bot" target="_blank" rel="nofollow noopener">facebook-github-bot</a>
	on <a href="https://github.com/pytorch/pytorch/issues/25010" target="_blank" rel="nofollow noopener">2019-08-23</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
There seems to be an issue with CPU utilization when using a DataLoader with pin_memory=True and num_workers &gt; 0.
I revisited...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: cpu</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: dataloader</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">topic: multithreading</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/24809" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/ggaemo" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/8081512?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/24809" target="_blank" rel="nofollow noopener">Issue: cpu usage is too high on the main thread after pytorch version 1.1 (and 1.2) (not data loader workers )</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/ggaemo" target="_blank" rel="nofollow noopener">ggaemo</a>
	on <a href="https://github.com/pytorch/pytorch/issues/24809" target="_blank" rel="nofollow noopener">2019-08-17</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/VitalyFedyunin" target="_blank" rel="nofollow noopener">VitalyFedyunin</a>
	on <a href="https://github.com/pytorch/pytorch/issues/24809" target="_blank" rel="nofollow noopener">2019-08-19</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">I am using python 3.7 CUDA 10.1 and pytorch 1.2
When I am running pytorch on GPU, the cpu usage of the...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: cpu</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I tried torch.set_num_threads(1) and this not just cut the CPU usage to one core (as expected) but the training also is much faster: About 1 seconds per epoch now.</p>
<p>So, I am not sure if this behaviour is really desired, as it seems like spreading the workload over all CPU cores not just requires all ressources but it also is much slower.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Unfortunately, this is a known limitation. We use openMP to parallelize cpu work and by default it uses all the available cores. So for machines with many cores, it is sometimes necessary to manually reduce the number of cores it can use <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: Raphikowski
cooked: <p>okay, I see. But do you have any clue why the code is running so much slower when running on 80 CPUs as opposed to when running on one CPU?</p>

username: albanD
cooked: <p>It is most likely because it tries to parallelize many “small” operations on many cores. So each core has almost nothing to do, but the overhead of communication between the cores is very large.</p>

username: albanD
cooked: <p>By the way, this is in the process of being solved, you can track the progress here: <a href="https://github.com/pytorch/pytorch/issues/24080" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/24080</a>.</p>

username: Raphikowski
cooked: <p>great, thanks a lot!</p>

username: phisad
cooked: <p>Seems there is no progress anymore on this?</p>
<p>I just today run into the same problem having 64 cores. A simple training was pushing all of them to 100% slowing down the training time immensely.</p>
<p>The fix was to set <code>torch.set_num_threads(1)</code> at the beginning.</p>