Very low performance on 64-bit ARM servers compared to Xeon servers (CPU-only)

dloghin: <p>I recently run MLPerf Benchmark on a few 64-bit ARM servers (e.g., AWS Graviton3, AWS Graviton2, Ampere Altra) and on a Xeon server from AWS (on a c6id.8xlarge instance). In particular, I run the Image Classification benchmark from MLPerf version 2.0 with ssd-mobilenet model on COCO dataset with images of size 300x300. The benchmark outputs the results in “queries per second” (qps). The benchmark supports Tensorflow, PyTorch, and ONNX. Surprisingly, PyTorch exhibits very low qps on ARM, while Tensorflow exhibits expected results. Graviton3 is the most advanced ARM-based server, so I will show you the results (qps) for this compared to Xeon:</p>
<pre><code class="lang-auto">                         Graviton3 (c7g.8xlarge)    Xeon (c6id.8xlarge)
   PyTorch (1.13.0a0)      0.9                      34.7
   Tensorflow (2.10.0)    79.9                      83.9
</code></pre>
<p>I run the benchmark in Docker (version 20.10.23) on Ubuntu OS (20.045 LTS). In Docker, I use Python 3.7.13 and PyTorch 1.13.0a0.</p>
<p>Steps to reproduce:</p>
<ul>
<li>
<p>Download the patch for MLPerf from here: <a href="https://github.com/dloghin/arm-cloud-bench/blob/main/mlperf/mlperf-arm64.patch" rel="noopener nofollow ugc">https://github.com/dloghin/arm-cloud-bench/blob/main/mlperf/mlperf-arm64.patch</a></p>
</li>
<li>
<p>Download the pre-processed COCO dataset (please message me for the link or follow the tutorial from MLPerf to generate them). Once you have the archive, run:</p>
</li>
</ul>
<p>tar xf coco-300.tar.bz2<br>
mkdir -p /home/$USER/data<br>
mv coco-300 /home/$USER/data/</p>
<ul>
<li>Prepare MLPerf Docker image and run the benchmark:</li>
</ul>
<p>git clone <a href="https://github.com/mlcommons/inference.git" rel="noopener nofollow ugc">https://github.com/mlcommons/inference.git</a><br>
cd inference<br>
git checkout v2.0<br>
git apply &lt; mlperf-arm64.patch<br>
docker build -f Dockerfile.cpu -t mlperf-cpu .<br>
docker run -v /home/$USER/data/coco-300:/data -it mlperf-cpu</p>
<ul>
<li>Then, in Docker, run:</li>
</ul>
<p>cd /tmp/inference/vision/classification_and_detection<br>
export DATA_DIR=/data/coco-300<br>
./run_local.sh pytorch ssd-mobilenet cpu<br>
…<br>
TestScenario.SingleStream qps=0.79, mean=1.2648, time=1295.256, queries=1024, tiles=50.0:1.2092,80.0:1.2159,90.0:1.4371,95.0:1.6403,99.0:1.9828,99.9:2.2888</p>
<p>I profiled the execution with perf (Linux tool) and it seems that PyTorch spends 99% of the time in <em>omp get num procs()</em> which is really strange. Does anyone have any idea why is this happening?</p>
<p>Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="dloghin" data-post="1" data-topic="171698">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dloghin/48/56975_2.png" class="avatar"> dloghin:</div>
<blockquote>
<p>I profiled the execution with perf (Linux tool) and it seems that PyTorch spends 99% of the time in <em>omp get num procs()</em> which is really strange.</p>
</blockquote>
</aside>
<p>Did you try to play around with e.g. <code>OMP_NUM_THREADS</code> to change the OpenMP threading behavior?</p>

dloghin: <p>Yes, I tried with <code>OMP_NUM_THREADS</code> set to 8, 16, and 32 (and also not set) – the results are the same: low throughput (below 1 qps).</p>

SuoXC: <p>I encountered the same issue, and eventually found that I needed to use <code>set_num_threads</code> to set the parallel degree to match the cpuset of the Docker container. This adjustment restored performance to normal levels. But I don’t know why.</p>