PyTorch changes the process affinity

jiashenC: <p>I am running a simple while loop for inference on CPU. I want the inference to run only on 1 CPU, but it seems like PyTorch changes the process affinity. I can see from htop that jobs are being scheduled on multiple CPUs that I do not expect.</p>
<p>My python program.</p>
<pre><code class="lang-auto">import os
import torch
from torchvision.models import resnet50

os.sched_setaffinity(os.getpid(), {0})
print('Set PID-{} to affinity: {}'.format(os.getpid(), os.sched_getaffinity(os.getpid())))

net = resnet50()
net.eval()
while True:
    with torch.no_grad():
        net(torch.randn(1, 3, 224, 224))
</code></pre>
<h2><a name="output-from-affinity-setting-and-my-htop-monitoring-screen-shot-2020-02-22-at-30951-pm690x19upload2vtdblsbhg7v1ffvtkvdfad1jl6png-1" class="anchor" href="#output-from-affinity-setting-and-my-htop-monitoring-screen-shot-2020-02-22-at-30951-pm690x19upload2vtdblsbhg7v1ffvtkvdfad1jl6png-1"></a>Output from affinity setting and my htop monitoring.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/1/119f348d4472553268a6c72b973c81f7db0cb2fc.png" data-download-href="https://discuss.pytorch.org/uploads/default/119f348d4472553268a6c72b973c81f7db0cb2fc" title="Screen Shot 2020-02-22 at 3.09.51 PM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/1/119f348d4472553268a6c72b973c81f7db0cb2fc.png" alt="Screen Shot 2020-02-22 at 3.09.51 PM" data-base62-sha1="2vTdbLsbhg7V1fFvTKVDFad1jL6" width="690" height="19" data-dominant-color="032D38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-02-22 at 3.09.51 PM</span><span class="informations">939×27 1.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></h2>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50ca482d861a5ee461b3221f94c490227f62ffdb.png" data-download-href="https://discuss.pytorch.org/uploads/default/50ca482d861a5ee461b3221f94c490227f62ffdb" title="Screen Shot 2020-02-22 at 3.10.07 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50ca482d861a5ee461b3221f94c490227f62ffdb_2_492x500.png" alt="Screen Shot 2020-02-22 at 3.10.07 PM" data-base62-sha1="bwHAfBe9M2i5X0CKJUxXmcaRX1V" width="492" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50ca482d861a5ee461b3221f94c490227f62ffdb_2_492x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50ca482d861a5ee461b3221f94c490227f62ffdb_2_738x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/50ca482d861a5ee461b3221f94c490227f62ffdb.png 2x" data-dominant-color="0B3739"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-02-22 at 3.10.07 PM</span><span class="informations">949×964 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this a bug or misuse from my side?</p>

albanD: <p>Hi,</p>
<p>By default, we use multi-threading to speed up cpu computations. Maybe with <code>torch.set_num_threads(1)</code> you will get the behavior you want?</p>

jiashenC: <p>Thanks for the reply. I have tried this approach, but unfortunately it still schedules the job on multiple CPUs.</p>

albanD: <p>Actually, your original code works fine for me:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9bb197c3209886f3eb2662b01a855b4c2b72ed1.png" data-download-href="https://discuss.pytorch.org/uploads/default/e9bb197c3209886f3eb2662b01a855b4c2b72ed1" title="Screen Shot 2020-02-24 at 10.00.25 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9bb197c3209886f3eb2662b01a855b4c2b72ed1_2_690x299.png" alt="Screen Shot 2020-02-24 at 10.00.25 AM" data-base62-sha1="xlG6N7Vw3ysTCCvO8Xf7pz1MOpb" width="690" height="299" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9bb197c3209886f3eb2662b01a855b4c2b72ed1_2_690x299.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9bb197c3209886f3eb2662b01a855b4c2b72ed1_2_1035x448.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9bb197c3209886f3eb2662b01a855b4c2b72ed1_2_1380x598.png 2x" data-dominant-color="1B211E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-02-24 at 10.00.25 AM</span><span class="informations">2374×1030 320 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jiashenC: <p>I see. Would you like to share your CPU information by <code>lscpu</code>?</p>

albanD: <p>It’s a skylake running on a centos-like machine.</p>
<pre><code class="lang-auto">Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              24
On-line CPU(s) list: 0-23
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           24
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               94
Model name:          Intel Core Processor (Skylake)
Stepping:            3
CPU MHz:             2394.477
BogoMIPS:            4788.95
Virtualization:      VT-x
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            4096K
L3 cache:            16384K
NUMA node0 CPU(s):   0-23
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat
</code></pre>

jiashenC: <p>I see. The only difference is the machine I am currently using has just 2 sockets, but more threads per core / more cores per sockets.</p>
<p>In detail, 2 sockets, 22 cores per socket, and 2 threads per core. Total of 88 CPU(s).</p>

albanD: <p>That is surprising…<br>
Does setting the affinity works with other libraries like <code>numpy</code> ?</p>

jiashenC: <p>Yes, <code>numpy</code> works seamlessly with the affinity.</p>
<p>My simple program.</p>
<pre><code class="lang-auto">import os
import numpy as np

os.sched_setaffinity(os.getpid(), {0, 1, 2})
print('Set PID-{} to affinity: {}'.format(os.getpid(),
                                          os.sched_getaffinity(os.getpid())))

while True:
    a = np.random.random_sample([100, 100])
    b = np.random.random_sample([100, 100])
    c = np.dot(a, b)
</code></pre>
<p>Without setting affinity.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df74f35e541fbc3735cf8094aa512cb0a0c05556.png" data-download-href="https://discuss.pytorch.org/uploads/default/df74f35e541fbc3735cf8094aa512cb0a0c05556" title="Screen Shot 2020-02-24 at 10.43.25 AM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df74f35e541fbc3735cf8094aa512cb0a0c05556.png" alt="Screen Shot 2020-02-24 at 10.43.25 AM" data-base62-sha1="vSN2zG1CLB22UL4NvWIvohdGalU" width="690" height="297" data-dominant-color="083236"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-02-24 at 10.43.25 AM</span><span class="informations">1172×505 47.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With affinity 0.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ea5423f6a01ce1b4cfb2d4a0a8c28db74324aa9.png" data-download-href="https://discuss.pytorch.org/uploads/default/8ea5423f6a01ce1b4cfb2d4a0a8c28db74324aa9" title="Screen Shot 2020-02-24 at 10.43.58 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ea5423f6a01ce1b4cfb2d4a0a8c28db74324aa9_2_690x297.png" alt="Screen Shot 2020-02-24 at 10.43.58 AM" data-base62-sha1="klTRHjBFPByWPOtQ1kamqlSgve9" width="690" height="297" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ea5423f6a01ce1b4cfb2d4a0a8c28db74324aa9_2_690x297.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/e/8ea5423f6a01ce1b4cfb2d4a0a8c28db74324aa9_2_1035x445.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/e/8ea5423f6a01ce1b4cfb2d4a0a8c28db74324aa9.png 2x" data-dominant-color="022E38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-02-24 at 10.43.58 AM</span><span class="informations">1172×505 34.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With affinity 0, 1, 2.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96303f024e6b9c8d433e7829e6a38f9cfd4253f1.png" data-download-href="https://discuss.pytorch.org/uploads/default/96303f024e6b9c8d433e7829e6a38f9cfd4253f1" title="Screen Shot 2020-02-24 at 10.44.20 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96303f024e6b9c8d433e7829e6a38f9cfd4253f1_2_690x297.png" alt="Screen Shot 2020-02-24 at 10.44.20 AM" data-base62-sha1="lqCZdEhF82F2A4zN3OkX1BUnJ9D" width="690" height="297" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96303f024e6b9c8d433e7829e6a38f9cfd4253f1_2_690x297.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96303f024e6b9c8d433e7829e6a38f9cfd4253f1_2_1035x445.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/6/96303f024e6b9c8d433e7829e6a38f9cfd4253f1.png 2x" data-dominant-color="032E38"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-02-24 at 10.44.20 AM</span><span class="informations">1172×505 35.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Interestingly, that does not work on my machine for numpy, and uses multiple cores and changes <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>I guess some BLAS libraries might not be following this.<br>
You can try running different ops in the while loop. Just the random numbers vs only the dot for example.</p>

jiashenC: <p>LOL, very interesting.</p>
<p>For <code>numpy</code>, here is a post about how to disable the OpenBLAS affinity resetting behavior. <a href="https://stackoverflow.com/questions/15639779/why-does-multiprocessing-use-only-a-single-core-after-i-import-numpy" rel="nofollow noopener">https://stackoverflow.com/questions/15639779/why-does-multiprocessing-use-only-a-single-core-after-i-import-numpy</a></p>
<p>But, even with this, my issue still persists with PyTorch.</p>

albanD: <p>Well, if you use a binary package, we ship with MKL <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> So maybe a similar MKL flag is needed?</p>

Yaroslav_Bulatov: <p>MKL has extra handling for CPU parallelism. The following two things always worked for me to restrict MKL usage to 1 core:</p>
<pre><code class="lang-auto">1. taskset 0x1 python myscript.py
2. OMP_NUM_THREADS=1 python myscript.py
</code></pre>