Which machine should I do the INT8 quantization for mobile deployment?

username: dalseeroh
cooked: <p>Hello,</p>
<p>I recently wrote a custom image inpainting model, an altered form of UNet. Now I want to demo run this model on Android environment, which will be Samsung Galaxy S10. After a few days of research, I got to know that I need to quantize the model for speedy mobile performance. I chose to follow ‘Post training static quantization’ in the link: <a href="https://pytorch.org/tutorials/recipes/quantization.html" class="inline-onebox" rel="noopener nofollow ugc">Quantization Recipe — PyTorch Tutorials 1.10.0+cu102 documentation</a>. But I just encountered a roadblock with a question that, which machine should I run this code below? I wrote this model in x64 Windows 10. Should I run this in pycharm IDE and save the model and then import the model in Android studio? Can someone give me a little bit of guidance to me?</p>
<pre><code class="lang-auto">backend = "qnnpack"
model.qconfig = torch.quantization.get_default_qconfig(backend)
torch.backends.quantized.engine = backend
model_static_quantized = torch.quantization.prepare(model, inplace=False)
model_static_quantized = torch.quantization.convert(model_static_quantized, inplace=False)
</code></pre>
<p><strong>Development Environment:</strong><br>
OS: Windows 10 x64<br>
CPU: Intel I9-10885H 2.40GHz<br>
GPU: NVIDIA GeForce 1650Ti<br>
PyTorch Version: 1.10.0.dev20210629</p>
<p><strong>Target Environment:</strong><br>
Android OS: 4.4+<br>
Device: Samsung Galaxy S10</p>

username: Karthik_Ganesan
cooked: <p>Have you taken a look at this page: <a href="https://pytorch.org/mobile/android/" class="inline-onebox" rel="noopener nofollow ugc">Android | PyTorch</a>?</p>

username: HDCharles
cooked: <p>A lot of deployment is also done through torchscript, you can get some more info here: <a href="https://pytorch.org/tutorials/recipes/torchscript_inference.html" class="inline-onebox" rel="noopener nofollow ugc">TorchScript for Deployment — PyTorch Tutorials 1.10.0+cu102 documentation</a></p>

username: dalseeroh
cooked: <p>I am trying to do quantization-aware-training before torch scripting to reduce the model size and to keep the accuracy. “My question is:” qnnpack only runs on arm architecture. Where should I run this if I am using Windows machine with Intel CPU?</p>

username: dalseeroh
cooked: <p>Of course yes. I am still trying to look for an answer of: Where should I run ‘qnnpack’ backend if I plan to do quantization-aware-training? Jetson Nano? Coral Devboard? Raspberry Pi? Cross compile?</p>

username: Vasiliy_Kuznetsov
cooked: <aside class="quote no-group" data-username="dalseeroh" data-post="4" data-topic="137461">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dalseeroh/48/29798_2.png" class="avatar"> dalseeroh:</div>
<blockquote>
<p>“My question is:” qnnpack only runs on arm architecture. Where should I run this if I am using Windows machine with Intel CPU?</p>
</blockquote>
</aside>
<p>QNNPACK is most performant on arm, but it does run on x86 with the same numerics.  You can use any machine which is compiled with qnnpack to calibrate your model.</p>