Difficulty limiting thread usage in Pytorch

username: Caius
cooked: <p>Hi,</p>
<p>I am attempting to run my code on a HPC cluster. I want to limit the number of threads used to the number of cpus I demand. I find, however, that during the backward step thread usage shoots up.<br>
As example: I demand 1 CPU, but during the backward step, 4 threads are used (and afterwards they remain used).</p>
<p>I have set the following environment variables at the start of the code:</p>
<blockquote>
<p>import os<br>
os.environ[“OMP_NUM_THREADS”] = “1”<br>
os.environ[“OPENBLAS_NUM_THREADS”] = “1”<br>
os.environ[“MKL_NUM_THREADS”] = “1”<br>
os.environ[“VECLIB_MAXIMUM_THREADS”] = “1”<br>
os.environ[“NUMEXPR_NUM_THREADS”] = “1”</p>
</blockquote>
<p>and also set</p>
<blockquote>
<p>import torch<br>
torch.set_num_threads(1)</p>
</blockquote>
<p>But thread usage during the backward step remains at 4.<br>
Another thing is that the command in the linux terminal (with PID meaning process id)</p>
<blockquote>
<p>ps -o nlwp {PID}</p>
</blockquote>
<p>and the method</p>
<blockquote>
<p>torch.get_num_threads()</p>
</blockquote>
<p>return different results: the former command tells me my process is using 4 threads, the latter says it only sees 1 thread. I am inclined to believe the former.</p>
<p>Help would be greatly appreciated.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I guess your cluster has 4 GPUs?<br>
The autograd engine also uses threading to be able to send work to the GPUs fast enough during the backward.<br>
Is that a problem in your setup?</p>

username: Yaroslav_Bulatov
cooked: <p>You probably just want to limit concurrency of compute-heavy stuff, so set OMP_NUM_THREADS=1 and num_workers=1 . But thread switching in general is fast, you won’t get much benefit from getting rid of threads that sit idle most of the time. You can see what all the threads are doing “sudo gdb -p  … thread apply all bt”</p>

username: Caius
cooked: <p><a class="mention" href="/u/alband">@albanD</a><br>
My cluster is divided up into several nodes, the current node I am utilizing does not have any GPUs.</p>
<p><a class="mention" href="/u/yaroslav_bulatov">@Yaroslav_Bulatov</a><br>
I am currently not using a dataloader so I don’t believe num_workers=1 is relevant to my case.<br>
The main reason why I want to limit the number of threads is because in the past my jobs have overloaded the compute nodes due to using more threads than the number of CPUs.</p>

username: albanD
cooked: <p>If you don’t have GPUs, then the autograd won’t use extra threads.<br>
Are you using the jit?</p>

username: Caius
cooked: <p>No, this is the first time I heard of it.</p>

username: albanD
cooked: <p>hmmm not sure where these threads could be coming from then…<br>
<a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> might have an idea?</p>
<p>Another solution would be to start your job with gdb and when 4 threads are used, interrupt it and check the stack trace for each thread. That should tell us why it happens.</p>

username: Caius
cooked: <p>Here is the trace for the threads:</p>
<p>(gdb) thread apply all bt</p>
<blockquote>
<p>Thread 4 (Thread 0x7facf308c700 (LWP 16191)):<br>
<span class="hashtag-raw">#0</span>  0x00007fad7454bd1f in accept4 () from /lib64/libc.so.6<br>
<span class="hashtag-raw">#1</span>  0x00007facf32b6a5a in ?? () from /lib64/libcuda.so.1<br>
<span class="hashtag-raw">#2</span>  0x00007facf32a85bd in ?? () from /lib64/libcuda.so.1<br>
<span class="hashtag-raw">#3</span>  0x00007facf32b8118 in ?? () from /lib64/libcuda.so.1<br>
<span class="hashtag-raw">#4</span>  0x00007fad74f2ae65 in start_thread () from /lib64/libpthread.so.0<br>
<span class="hashtag-raw">#5</span>  0x00007fad7454a88d in clone () from /lib64/libc.so.6</p>
</blockquote>
<blockquote>
<p>Thread 3 (Thread 0x7facf288b700 (LWP 16192)):<br>
<span class="hashtag-raw">#0</span>  0x00007fad74f2e9f5 in pthread_cond_wait@<span class="mention">@GLIBC_2.3.2</span> () from /lib64/libpthread.so.0<br>
<span class="hashtag-raw">#1</span>  0x00007fad644e582c in std::condition_variable::wait(std::unique_lock<a>std::mutex</a>&amp;) ()<br>
from /lib64/libstdc++.so.6<br>
<span class="hashtag-raw">#2</span>  0x00007fad1dc8ce23 in torch::autograd::ReadyQueue::pop() ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch.so<br>
<span class="hashtag-raw">#3</span>  0x00007fad1dc8ed7c in torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autogr                                                     ad::GraphTask&gt; const&amp;, bool) ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch.so<br>
<span class="hashtag-raw">#4</span>  0x00007fad1dc88979 in torch::autograd::Engine::thread_init(int) ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch.so<br>
<span class="hashtag-raw">#5</span>  0x00007fad64d9408a in torch::autograd::python::PythonEngine::thread_init(int) ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so<br>
<span class="hashtag-raw">#6</span>  0x00007fad659afdef in execute_native_thread_routine ()<br>
—Type  to continue, or q  to quit—return<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-li                                                     nux-gnu.so<br>
<span class="hashtag-raw">#7</span>  0x00007fad74f2ae65 in start_thread () from /lib64/libpthread.so.0<br>
<span class="hashtag-raw">#8</span>  0x00007fad7454a88d in clone () from /lib64/libc.so.6</p>
</blockquote>
<blockquote>
<p>Thread 2 (Thread 0x7facf208a700 (LWP 16193)):<br>
<span class="hashtag-raw">#0</span>  0x00007fad74f2e9f5 in pthread_cond_wait@<span class="mention">@GLIBC_2.3.2</span> () from /lib64/libpthread.so.0<br>
<span class="hashtag-raw">#1</span>  0x00007fad644e582c in std::condition_variable::wait(std::unique_lock<a>std::mutex</a>&amp;) ()<br>
from /lib64/libstdc++.so.6<br>
<span class="hashtag-raw">#2</span>  0x00007fad1dc8ce23 in torch::autograd::ReadyQueue::pop() ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch.so<br>
<span class="hashtag-raw">#3</span>  0x00007fad1dc8ed7c in torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autogr                                                     ad::GraphTask&gt; const&amp;, bool) ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch.so<br>
<span class="hashtag-raw">#4</span>  0x00007fad1dc88979 in torch::autograd::Engine::thread_init(int) ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch.so<br>
<span class="hashtag-raw">#5</span>  0x00007fad64d9408a in torch::autograd::python::PythonEngine::thread_init(int) ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so<br>
<span class="hashtag-raw">#6</span>  0x00007fad659afdef in execute_native_thread_routine ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-li                                                     nux-gnu.so<br>
<span class="hashtag-raw">#7</span>  0x00007fad74f2ae65 in start_thread () from /lib64/libpthread.so.0<br>
<span class="hashtag-raw">#8</span>  0x00007fad7454a88d in clone () from /lib64/libc.so.6<br>
—Type  to continue, or q  to quit—return</p>
</blockquote>
<blockquote>
<p>Thread 1 (Thread 0x7fad75853740 (LWP 16174)):<br>
<span class="hashtag-raw">#0</span>  0x00007fad744d2424 in memalign () from /lib64/libc.so.6<br>
<span class="hashtag-raw">#1</span>  0x00007fad744d404c in posix_memalign () from /lib64/libc.so.6<br>
<span class="hashtag-raw">#2</span>  0x00007fad1960ab4a in c10::alloc_cpu(unsigned long) ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libc10.so<br>
<span class="hashtag-raw">#3</span>  0x00007fad1960c5fa in c10::DefaultCPUAllocator::allocate(unsigned long) const ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libc10.so<br>
<span class="hashtag-raw">#4</span>  0x00007fad1b93c78a in at::native::empty_cpu(c10::ArrayRef, c10::TensorOptions cons                                                     t&amp;, c10::optional<a>c10::MemoryFormat</a>) ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch.so<br>
<span class="hashtag-raw">#5</span>  0x00007fad1bb8530b in at::CPUType::(anonymous namespace)::empty(c10::ArrayRef, c10                                                     ::TensorOptions const&amp;, c10::optional<a>c10::MemoryFormat</a>) ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch.so<br>
<span class="hashtag-raw">#6</span>  0x00007fad1bbccb37 in c10::detail::wrap_kernel_functor_unboxed_&lt;c10::detail::WrapRuntime                                                     KernelFunctor_&lt;at::Tensor (<em>)(c10::ArrayRef, c10::TensorOptions const&amp;, c10::optional&lt;                                                     c10::MemoryFormat&gt;), at::Tensor, c10::guts::typelist::typelist&lt;c10::ArrayRef, c10::Ten                                                     sorOptions const&amp;, c10::optional<a>c10::MemoryFormat</a> &gt; &gt;, at::Tensor (c10::ArrayRef, c1                                                     0::TensorOptions const&amp;, c10::optional<a>c10::MemoryFormat</a>)&gt;::call(c10::OperatorKernel</em>, c10:                                                     :ArrayRef, c10::TensorOptions const&amp;, c10::optional<a>c10::MemoryFormat</a>) ()<br>
from /home/nfs/USER/.local/lib/python3.6/site-packages/torch/lib/libtorch.so<br>
<span class="hashtag-raw">#7</span>  0x00007fad1db8c795 in torch::autograd::VariableType::(anonymous namespace)::empty(c10::A</p>
</blockquote>
<p>It does seem like it is autograd that is using extra threads, despite the fact that there is no GPU to make use of.</p>

username: albanD
cooked: <p>So thread 4 one seems to be a cuda driver thread. Not sure we can do anything about this ? (cc <a class="mention" href="/u/ptrblck">@ptrblck</a>)</p>
<p>thread 1 is your main worker thread</p>
<p>thread 3/4 are autograd worker thread (not sure why there are 2), but the CPU worker thread should run only while thread 1 will be blocked on waiting for it to finish. So that should use more than one core.</p>

username: VitalyFedyunin
cooked: <p>Just in case <code>OMP_NUM_THREADS</code> controls ONLY number of operator threads, and does nothing with Autograd threads or Data Loading threads.</p>