Pytorch on Aarch64 -- Neon instructions

username: Wilson_Cole
cooked: <p>Hi everyone,</p>
<p>I have a question regarding native ARM NEON support for Pytorch on ARMv8 architectures.</p>
<p>(<a href="https://github.com/ARM-software/Tool-Solutions/tree/main/docker/pytorch-aarch64#image-contents" class="inline-onebox" rel="noopener nofollow ugc">Tool-Solutions/docker/pytorch-aarch64 at main · ARM-software/Tool-Solutions · GitHub</a>)</p>
<p>I can see in the pytorch docker build that OpenBLAS is the default backend for AArch64, and on the OpenBLAS website it seems that ARM Neon intrinsics are not available by default.</p>
<p>(<a href="https://github.com/OpenMathLib/OpenBLAS#:~:text=A9%20and%20A15" class="inline-onebox" rel="noopener nofollow ugc">GitHub - OpenMathLib/OpenBLAS: OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version.</a>)</p>
<p>My question is, are ARM NEON intrinsics enabled by default for floating point inference on the latest Pytorch wheels? Is there an easy way to check if my inference is currently using NEON instructions?</p>
<p>If NEON instructions are not enabled by default? Is there any latest pytorch wheels which do have NEON support? Or are there any build instructions which I can access that can use to build pytorch with the Arm Compute Library enabled?</p>
<p>Thanks everyone so much. I couldn’t really find any helpful documentation for this.</p>
<p>P.S. I also have tried to build Pytorch with ACL support myself but I keep running into issues with linking ACL …</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b0143a8ab276037efd50ad46d161790c3460351.png" data-download-href="https://discuss.pytorch.org/uploads/default/8b0143a8ab276037efd50ad46d161790c3460351" title="Screenshot 2024-05-17 at 11.14.49 AM"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b0143a8ab276037efd50ad46d161790c3460351.png" alt="Screenshot 2024-05-17 at 11.14.49 AM" data-base62-sha1="jPH4MtonRpJweUTEhLYARKlN0Bj" width="689" height="208" data-dominant-color="0C0C0C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-05-17 at 11.14.49 AM</span><span class="informations">1290×389 18.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Noureddine_Bouhali
cooked: <p>Hello,<br>
The docker builds for pytorch have two backends for blas: either onednn with support for ACL or Openblas without ACL. You can find both in : <a href="https://hub.docker.com/r/armswdev/pytorch-arm-neoverse" rel="noopener nofollow ugc">https://hub.docker.com/r/armswdev/pytorch-arm-neoverse</a><br>
Docker pull the correct version is required then.<br>
P.S: inference on ARM currently supports FP32 AFAIK, but int8 inference doesn’t seem to be fully supported…</p>

username: cfRod
cooked: <aside class="quote no-group" data-username="Wilson_Cole" data-post="1" data-topic="202960">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wilson_cole/48/70100_2.png" class="avatar"> Wilson_Cole:</div>
<blockquote>
<p>I can see in the pytorch docker build that OpenBLAS is the default backend for AArch64, and on the OpenBLAS website it seems that ARM Neon intrinsics are not available by default.</p>
</blockquote>
</aside>
<p>Hi, our optimised backed is based on ACL.  Could you build with the oneDNN + ACL backend? Alternatively our upstream pip wheels should come with ACL. If you pass ONEDNN_VERBOSE=profile_exec,profile_externals, you should be able to see onednn verbose logs calling into ACL kernels.</p>
<p>Regarding Arm NEON intrinsics, we should have that enabled for eager mode : <a href="https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/cpu/vec/vec256" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/cpu/vec/vec256 at main · pytorch/pytorch · GitHub</a></p>
<p>We are also going enable SLEEF via torch compile as well. How are you running your code? Eager mode or Torch compile?</p>
<p>Regarding the issue you are facing building with ACL. Have you set the ACL_ROOT_DIR ?</p><aside class="onebox githubblob" data-onebox-src="https://github.com/ARM-software/Tool-Solutions/blob/f8c06121b27dc997fbf69302de5a3660d69adbe2/docker/pytorch-aarch64/Dockerfile#L175C5-L175C17">
  <header class="source">

      <a href="https://github.com/ARM-software/Tool-Solutions/blob/f8c06121b27dc997fbf69302de5a3660d69adbe2/docker/pytorch-aarch64/Dockerfile#L175C5-L175C17" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/ARM-software/Tool-Solutions/blob/f8c06121b27dc997fbf69302de5a3660d69adbe2/docker/pytorch-aarch64/Dockerfile#L175C5-L175C17" target="_blank" rel="noopener nofollow ugc">ARM-software/Tool-Solutions/blob/f8c06121b27dc997fbf69302de5a3660d69adbe2/docker/pytorch-aarch64/Dockerfile#L175C5-L175C17</a></h4>



    <pre class="onebox"><code class="lang-">
      <ol class="start lines" start="175" style="counter-reset: li-counter 174 ;">
          <li>ENV ACL_ROOT_DIR=$PROD_DIR/ComputeLibrary</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Can you raise an issue on our Tool-Solutions Github repo, if you are still facing this issue?</p>
<p>Thanks<br>
Crefeda</p>