Use single thread on Intel CPU

sbuschjaeger: <p>Hi,</p>
<p>I have been playing around with the C++ Frontend for PyTorch on my Laptop (Intel® Core™ i7-4600U) and were able to include PyTorch into my CPP app by following the MNIST example (<a href="https://github.com/goldsborough/examples/blob/cpp/cpp/mnist/mnist.cpp" rel="nofollow noopener">https://github.com/goldsborough/examples/blob/cpp/cpp/mnist/mnist.cpp</a>).<br>
My app already utilizes parallelization to some degree, so I would like to run training / inference on a single thread. However, I was not able to tell PyTorch that - it would always use all cores. Here is what I did so far:</p>
<ul>
<li>I could not find any “torch.set_num_threads(1)” so I searched the PyTorch sources for anything related and found at::set_num_threads(1);</li>
<li>Setting the openmp environment variable “OMP_NUM_THREADS=1”</li>
<li>Setting the MKL environment variable “MKL_NUM_THREADS=1”</li>
<li>Further investigation of the source code revealed, that caffee2 uses a ThreadPool which is initialized with cpuinfo_get_processors_count() from the cpuinfo lib (<a href="https://github.com/pytorch/cpuinfo" rel="nofollow noopener">https://github.com/pytorch/cpuinfo</a>). I did not find a way to set this from the outside</li>
</ul>
<p>note, that I did not compile PyTorch myself, but used the library provided on the website. There seem to be a lot of different frameworks involved such as openmp,mkl,mkldnn etc. Thus I am a little confused on how to force PyTorch to use one thread. Any ideas?</p>
<p>Thanks!</p>

afshin67: <p><a class="mention" href="/u/sbuschjaeger">@sbuschjaeger</a><br>
Did you find any solution for this?<br>
Now I see that there is <code>torch::set_num_threads(1);</code>, but it does not work.</p>

yf225: <p>When you try <code>OMP_NUM_THREADS=1</code>, does <code>htop</code> say that you are still using all CPU cores?</p>

afshin67: <p><a class="mention" href="/u/yf225">@yf225</a><br>
Yes, I added <code>omp_set_num_threads(1)</code> (which has precedence over <code>OMP_NUM_THREADS=1</code>) in the beginning of the main function and still it uses all the CPU cores.<br>
I also removed <code>omp_set_num_threads(1)</code> from the code, and entered <code>OMP_NUM_THREADS=1</code> in the command line before running the mnist, and still it uses all of the CPU cores.</p>

bsting: <p><a class="mention" href="/u/afshin67">@afshin67</a> did you manage limit the thread to 1? having same problem with you, after calling omp_set_num_threads(1) or torch::set_num_threads(1), all cores still being occupied. my libtorch downloaded from PyTorch official website too.</p>

bsting: <p>My PC has 8 cores, results of running torch::get_num_threads()  and omp_get_max_threads() are difference. Also, calling torch::set_num_threads(1) made no effect on omp_get_max_threads()</p>
<p>std::cout &lt;&lt; torch::get_num_threads() &lt;&lt; std::endl;<br>
-1<br>
std::cout &lt;&lt; omp_get_max_threads() &lt;&lt; std::endl;<br>
8</p>
<p>torch::set_num_threads(1);</p>
<p>std::cout &lt;&lt; omp_get_max_threads() &lt;&lt; std::endl;<br>
8<br>
std::cout &lt;&lt; torch::get_num_threads() &lt;&lt; std::endl;<br>
1</p>
<p>omp_set_num_threads(1);</p>
<p>std::cout &lt;&lt; omp_get_max_threads() &lt;&lt; std::endl;<br>
1</p>

afshin67: <p>No, I still have the problem and reported it on the github page too:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/19213" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/oroojlooy" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/20797260?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/19213" target="_blank" rel="nofollow noopener">Issue: torch::set_num_threads does not work </a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/oroojlooy" target="_blank" rel="nofollow noopener">oroojlooy</a>
	on <a href="https://github.com/pytorch/pytorch/issues/19213" target="_blank" rel="nofollow noopener">2019-04-12</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
I added torch::set_num_threads(2) in the beginning of the mnist code and still the executable uses all of the available threads,...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">high priority</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: openmp</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">topic: multithreading</span>
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

bsting: <p>Managed to limit the thread usage to 1 by using latest nightly built libtorch and below code</p>
<p><span class="hashtag">#include</span> “ATen/Parallel.h”<br>
at::set_num_threads(1);</p>

afshin67: <p>It did not work for me. Did you try it with <code>mnist</code>?</p>

bsting: <p>I didn’t try it with mnist. Do you test with nightly built libtorch? it only works with the nightly built version.</p>

afshin67: <p>Yes, I used the downloaded the nightly_build and tested with that.</p>

waleedfarrukhgini: <p>Hi, was there any progress on this problem? We have a similar problem with Pytorch 1.1 with C++ frontend. On Python, I am able to set the thread limit to 1.</p>

Mr.Crab: <p>i did what you say, but i can not compile the C++ code for the “at::set_num_threads(1);” is wrong, and how should i to set environment?</p>

fisakhan: <p>How do you set the number of threads to 1 in python? Did anyone solve this problem and force PyTorch to use only 1 threads?</p>
<p>The following setting make the number of threads to 1 but as soon as I use an object detection model in torch, the number of threads reaches to 8.</p>
<p>os.environ[“OMP_NUM_THREADS”] = “1”<br>
os.environ[“MKL_NUM_THREADS”] = “1”<br>
torch.set_num_threads(1)<br>
torch.set_num_interop_threads(1)<br>
os.system(“top -H -b -n1 | grep project | wc -l”)</p>

JimQ: <p>I’m getting the same problem on my C++ torch project in Windows (I’m building as a DLL and calling it from a dotnetcore console app). I have set both the ‘OMP_NUM_THREADS’ and the ‘MKL_NUM_THREADS’ environment variables to 1. I have also tried ‘<span class="hashtag">#include</span> “ATen/Parallel.h”’ and ‘at::set_num_threads(1);’ at the top of my code. Still running the app spawns 14 threads and maxes out all 8 cores. Has anyone solved this on Windows?</p>