RuntimeError: quantized engine QNNPACK is not supported

username: dalseeroh
cooked: <p>Hello,</p>
<p>I am trying to deploy my CNN inpainting model to Android mobile app, so I am following this tutorial.<br>
<a href="https://pytorch.org/tutorials/recipes/quantization.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/quantization.html</a></p>
<pre><code class="lang-auto">model = PartialConvUNet()
backend = 'qnnpack'

model.qconfig = torch.quantization.get_default_qconfig(backend)
torch.backends.quantized.engine = backend
model_static_quantized = torch.quantization.prepare(model, inplace=False)
model_static_quantized = torch.quantization.convert(model_static_quantized, inplace=False)
print_size_of_model(model_static_quantized)
</code></pre>
<p>And I run this, I get this error.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:\PycharmProjects\MyCNN\mymodel.py", line 219, in &lt;module&gt;
    torch.backends.quantized.engine = 'qnnpack'
  File "C:\Anaconda3\envs\deeplearning\lib\site-packages\torch\backends\quantized\__init__.py", line 29, in __set__
    torch._C._set_qengine(_get_qengine_id(val))
RuntimeError: quantized engine QNNPACK is not supported
</code></pre>
<p>If I set<br>
<code>backend='fbgemm'</code><br>
It works, but this is the backend for the x86 server, which will not be compatible to Android environment, correct?</p>

username: Vasiliy_Kuznetsov
cooked: <p>This probably means that the machine you are doing quantization on does not support QNNPACK.  Could you share what machine and envoronment you are using, and what PyTorch version?</p>

username: dalseeroh
cooked: <p>Hello Vasiliy,</p>
<p>OS: Windows 10 x64<br>
CPU: Intel I9-10885H 2.40GHz<br>
GPU: NVIDIA GeForce 1650Ti<br>
PyTorch Version: 1.10.0.dev20210629</p>

username: kimishpatel
cooked: <aside class="quote no-group" data-username="dalseeroh" data-post="1" data-topic="136524">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dalseeroh/48/29798_2.png" class="avatar"> dalseeroh:</div>
<blockquote>
<p>inpainting model to Android mobile app</p>
</blockquote>
</aside>
<p>Are you building pytorch for windows on your machine or are you cross compiling for android? I see that you want to deploy on android so there qnnpack is definitely supported. But on windows for OSS I am not sure. I will check and get back to you.</p>

username: dalseeroh
cooked: <p>Yes. I wrote the model in PyTorch on Windows machine. I am following this deployment workflow.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/mobile/home/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/mobile/home/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/mobile/home/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Write a model → Quantize → Script/trace → Optimize → Maven(?)</p>

username: dalseeroh
cooked: <p>According to this article: <a href="https://spell.ml/blog/pytorch-quantization-X8e7wBAAACIAHPhT" rel="noopener nofollow ugc">https://spell.ml/blog/pytorch-quantization-X8e7wBAAACIAHPhT</a></p>
<p>I needa ARM CPU…</p>
<blockquote>
<p><em>“Since these libraries are architecture-dependent, <strong>static quantization must be performed on a machine with the same architecture as your deployment target.</strong> If you are using FBGEMM, you must perform the calibration pass on an x86 CPU (usually not a problem); if you are using QNNPACK, calibration needs to happen on an ARM CPU (this is quite a bit harder).”</em></p>
</blockquote>

username: Vasiliy_Kuznetsov
cooked: <p>hi <a class="mention" href="/u/dalseeroh">@dalseeroh</a> ,</p>
<blockquote>
<p><em>“Since these libraries are architecture-dependent, <strong>static quantization must be performed on a machine with the same architecture as your deployment target.</strong> If you are using FBGEMM, you must perform the calibration pass on an x86 CPU (usually not a problem); if you are using QNNPACK, calibration needs to happen on an ARM CPU (this is quite a bit harder).”</em></p>
</blockquote>
<p>That quote does not seem to be accurate.  In general, you can use QNNPACK on x86, this is a widely used functionality on Meta where models are calibrated on linux machines with QNNPACK for inference on arm.  I think the issues you are hitting on your machine might be specific to your environment.</p>