PyTorch CPU Slowdown: OSX vs. other Linux

hyrox: <p>I have installed PyTorch on several different machines, and I am consistently seeing the OS X CPU-only performance outperforming PyTorch on other platforms, even though the other machines have better CPUs.</p>
<p>I was using <a href="https://discuss.pytorch.org/t/blas-performance-on-macos-vs-linux-vs-lua-torch/744">this</a> post as a point of reference, as the symptoms sound similar. However, I ultimately found that while installing with the iomp5 flag (which now appears to be integrated into the master branch) does indeed produce faster matrix multiplication on the server, overall performance still seems to be slower.</p>
<p>My target application is kind of a monster, so my first step at isolating the issue was trying out some of the PyTorch <a href="https://github.com/pytorch/examples" rel="nofollow noopener">examples</a>. For example, I ran the mnist example on two different machines, and the example finished 10 epochs almost twice as fast on my MacBook Pro than on the server. Here are the specifications for each machine (very similar to the post linked above):</p>
<p>macOS: Sierra, CPU: Intel i7-4770HQ (8) @ 2.20GHz, 16GB RAM<br>
CentOS 7.0, CPU: Intel Xeon E5-2695v2 (48) @ 2.40GHz, 384GB RAM</p>
<p>I tried installing with conda and from source, but the results are the same.</p>
<p>Has anyone seen similar behavior?</p>
<p>What would be some useful further steps to isolate the cause of the performance differences?</p>
<p>Are there more useful benchmarking scripts to use?</p>
<p>Thanks.</p>

smth: <p>If you have a Xeon server with 48 hyperthreads (2 sockets, 24 physical cores), you will almost surely have worse perf by default than i7 with 8 cores.</p>
<p>The reason here is that Intel MKL and Intel OpenMP which PyTorch uses to parallelize it’s CPU code, try to use all possible cores by default.<br>
However, using all cores is not great in many cases. And its especially not great when you have 12 cores split across 2 sockets, and oversubscribed with hyperthreads.</p>
<p>Try to run your python program like this on your Linux machine:</p>
<pre><code class="lang-python">OMP_NUM_THREADS=8 MKL_NUM_THREADS=8 python foo.py
</code></pre>
<p>Try different values than <code>8</code> from 1 to 12. Going more than 12 will surely be bad perf, unless you have very large workloads.</p>

FuriouslyCurious: <p><em>Edit: Never mind - MKL_DYNAMIC and OMP_DYNAMIC would not help so removed it from my post</em></p>
<p>I was wondering about PyTorch internals: if I am using a server with larger number of cores (64), would you recommend setting OMP_NESTED flag to true or false?</p>
<p><a href="https://software.intel.com/en-us/mkl-linux-developer-guide-mkl-dynamic" class="onebox" target="_blank" rel="nofollow noopener">https://software.intel.com/en-us/mkl-linux-developer-guide-mkl-dynamic</a></p>

smth: <p>nested should be false.</p>

yilunzhang: <p><a class="mention" href="/u/smth">@smth</a> Thank you for your solution for improvement. I had the same problem with OP and this largely improved inference speed on the Xeon server that I’m using. However, it is still significantly slower than on a Mac. Are there any other methods that you can think of that can make the gap smaller? Thanks!</p>

gabrieldernbach: <p>How do you do your benchmark? Just in case :  reading from disk can be a bottleneck if you compare your osx ssd with the server hdd</p>