Pointers to bring quantized models to device

escorciav: <p>Dear team &amp; community,</p>
<p>Can anyone describe their workflow to bring quantized models onto devices (mobile, edge devices, etc.)?</p>
<ol>
<li>AFAIU the quantization API does not support exporting onto ONNX.</li>
</ol>
<p><strong>2. What do people often do then?</strong></p>
<ol start="3">
<li>Currently, the road feels incomplete. no?</li>
</ol>
<ul>
<li>3a. Where can we read the roadmap?</li>
<li>3b. What’s missing to achieve something similar to (e.g., TFLite) in Pytorch?</li>
<li>3c. Can you please mention the existing players (raising starts <img src="https://discuss.pytorch.org/images/emoji/apple/star2.png?v=12" title=":star2:" class="emoji" alt=":star2:" loading="lazy" width="20" height="20"> , and partners) such that people can better decide how to organize their projects?</li>
</ul>
<p>Thanks in advance &amp; keep it up!<br>
-Victor</p>
<p>P.D. Take my words with a dose of honey, and fast-fwd criticism.  I believe AI will be ubiquitous as long as it’s accessible in users’ pockets (e.g., mobile devices &amp; owned hardware), not just done in big cluster farms (e.g., via APIs). Thus, I truly appreciate that Pytorch opened up space for quantization.</p>

111357: <p>How about exporting the model into <code>ONNX</code> first and then quantizing the <code>ONNX</code> model by <a href="https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/quantization/README.md" rel="noopener nofollow ugc">Onnxruntime</a>.</p>

escorciav: <p>Thanks for chiming in <a class="mention" href="/u/111357">@111357</a></p>
<p>I read somewhere on the web (Forums, Github, etc.), <a href="https://discuss.pytorch.org/t/pointers-to-bring-quantized-models-to-device/174791/2">your suggested path</a>. <em>Feel free to provide more details about your experience (e.g.,. linking a Github project or mentioning the models) walking along that trail <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></em></p>
<p>For the current model (YOLOv5-x), I’m working on people suggested the following pipeline:</p>
<p>pytorch → onnx → SNPE/DLC (Qualcomm specific SDK &amp; hardware)</p>
<p>Thus, I’m doing most of the development in Pytorch (ML pipeline &amp; Quantization biz).</p>
<p>BTW, a colleague also suggested cutting any middleman, i.e., jumping directly to hardware SDK (DLC in my case). I didn’t as it’s my first venturing in that frontier.</p>

111357: <p>Hi, I recommend the <a href="https://github.com/quic/aimet" rel="noopener nofollow ugc">AIMET</a> which is a tool supported by Qualcomm.<br>
Here is an <a href="https://github.com/quic/aimet/blob/develop/Examples/torch/quantization/qat.ipynb" rel="noopener nofollow ugc">example</a> that quantized a torch model</p>

supriyar: <p>hi there, we have a recipe for how to run quantized models on device <a href="https://pytorch.org/tutorials/recipes/ptmobile_recipes_summary.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/ptmobile_recipes_summary.html</a></p>
<p>We are working on building an executor stack for exported PyTorch programs to run on device. The release plan to OSS is still in progress, but expect to hear something later this year. cc <a class="mention" href="/u/martin_yuan">@Martin_Yuan</a>, <a class="mention" href="/u/raziel">@raziel</a></p>

raziel: <aside class="quote no-group quote-modified" data-username="supriyar" data-post="5" data-topic="174791">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/db5fbb/48.png" class="avatar"> supriyar:</div>
<blockquote>
<p>hi there, we have a recipe for how to run quantized models on device <a href="https://pytorch.org/tutorials/recipes/ptmobile_recipes_summary.html" class="inline-onebox" rel="noopener nofollow ugc">Summary of PyTorch Mobile Recipes — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
</blockquote>
</aside>
<p>That’s right.</p>
<p>Moving forward…</p>
<p>We’re building a new on-device stack within PyTorch, and quantization is a core concern just as is building the ecosystem to target the heterogeneous hardware landscape.</p>
<p>Right now I cannot share a lot more than what we presented last December in the PyTorch Conference: <a href="https://youtu.be/XJJzJbDEAic" rel="noopener nofollow ugc">https://youtu.be/XJJzJbDEAic</a></p>

escorciav: <p>Thanks! I’ll definitely take a look at AIMET in the future.</p>
<p><strong>Have you used the <a href="https://quic.github.io/aimet-pages/releases/latest/api_docs/torch_quantsim.html#aimet_torch.quantsim.QuantizationSimModel.export" rel="noopener nofollow ugc"><code>QuantizationSimModel.export</code></a>, mentioned at the end of QAT tutorial?</strong></p>
<p>Namely,</p>
<ul>
<li>Did you manage to export a useful ONNX file?</li>
<li>How did your pipeline with SNPE (Snapdragon SDK) look-alike with the quantized onnx file?<br>
Apparently, SNPE only generates DLC files out of float32 models. I won’t be surprised if it’s just a matter of casting weights to float32.</li>
</ul>

escorciav: <p>Update…</p>
<ul>
<li>A few colleagues of mine have used AIMET. It might be a good fit for research (assuming you’re doing something similar to Qualcomm AI papers).</li>
<li>Should you go for it then? We’ve found sharp edges to bring stuff onto mobile or on-device <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=12" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:" loading="lazy" width="20" height="20"> . Refer to comments &amp; feedback in their Github.</li>
<li>Does anyone know the roadmap of PyTorch quantization API?</li>
</ul>

jerryzh168: <p>Here are some tutorials for the new quantization API:</p>
<ul>
<li><a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2.0 Export Post Training Static Quantization — PyTorch Tutorials 2.0.1+cu117 documentation</a></li>
<li><a href="https://pytorch.org/tutorials/prototype/pt2e_quantizer.html" class="inline-onebox" rel="noopener nofollow ugc">How to Write a Quantizer for PyTorch 2.0 Export Quantization — PyTorch Tutorials 2.0.1+cu117 documentation</a></li>
</ul>
<p>and PyTorch mobile will also release executorch (native pytorch platform/runtime for edge models) in PyTorch conference: <a href="https://events.linuxfoundation.org/pytorch-conference/program/schedule/" class="inline-onebox" rel="noopener nofollow ugc">Schedule | Linux Foundation Events</a>, it’s 3 weeks away.</p>