How to perform quantization of a model in PyTorch?

r3krut: <p>Hello everyone!</p>
<p>I have trained the model <code>MobileNetV2 + SSD Lite</code> in PyTorch from ‘<a href="https://github.com/qfgaohao/pytorch-ssd/blob/master/vision/ssd/mobilenet_v2_ssd_lite.py" rel="nofollow noopener">https://github.com/qfgaohao/pytorch-ssd/blob/master/vision/ssd/mobilenet_v2_ssd_lite.py</a>’. Now, I want use it in Raspberry Pi3.</p>
<p>I converted ‘.pth’ model into Caffe2 model through <code>ONNX</code> representation and I got two files: init_net.pb and predict_net.pb for Caffe2 framework.</p>
<p>As far as I know, to accelerate the model on mobile systems such as Rpi3(B/B+) I should use the <code>QNNPACK</code> lib which allows make the low-precision inference using operators with <code>int8</code> data type.</p>
<p>How to perform quantization of this model?<br>
How can I make low-precision inference using <code>QNNPACK</code>?<br>
Maybe there are some tutorials about it?</p>
<p>Thnx.</p>

jfix: <p>Hi <a class="mention" href="/u/r3krut">@r3krut</a>,</p>
<p>This category is for Glow, which is a different PyTorch backend from Caffe2 (which <a href="https://github.com/pytorch/QNNPACK#end-to-end-benchmarking" rel="nofollow noopener">"natively integrates <code>QNNPACK</code>"</a>). Glow primarily targets neural network accelerators, though it does have a CPU backend and supports automatic profiling + quantization. If you want to use <code>QNNPACK</code>, I believe all you need to do is make sure your model (e.g. <code>predict_net</code>) is using operators such as <code>Int8Conv</code>, <code>Int8FC</code>, etc. and your Caffe2 model would use it.</p>
<p>However, if your network is not quantized and/or you don’t want to install Caffe2 on your Raspberry Pi, you could use try to use Glow to profile your model, quantize it, and then save what we call an ahead-of-time compiled “bundle”, which is just a binary to copy to your Raspberry Pi3 to run (see docs <a href="https://github.com/pytorch/glow/blob/master/docs/Quantization.md#how-to-perform-nn-conversion" rel="nofollow noopener">here</a> and <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md#creating-standalone-executable-bundles" rel="nofollow noopener">here</a>). Note that it may not perform as well as <code>QNNPACK</code>; we are more focused on accelerator backends right now.</p>
<p>Thanks,<br>
Jordan</p>

Stella: <p>Is it possible to do a quantization of .pb in pytorch and get a quantized .pb directly? Like bazel quantization tools ? Any tutorial will be appreciated!</p>

r3krut: <p>Thnx for answer <a class="mention" href="/u/jfix">@jfix</a> , but I’m a bit confused.<br>
At the moment my model <code>predict_net.pb</code> does not use operators such as <code>Int8Conv</code> and etc.<br>
How do I force my model to use these operations?<br>
Should I change operators such as <code>Conv</code> to <code>Int8Conv</code> manually?<br>
Which file should I change? <code>predict_net.pb</code> or <code>predcit_net.pbtxt</code>?</p>
<p>Example of some <code>op</code> from my <code>predict_net.pbtxt</code>:</p>
<pre><code class="lang-auto">op {
  input: "0"
  input: "1"
  output: "497"
  name: ""
  type: "Conv"
  arg {
    name: "strides"
    ints: 2
    ints: 2
  }
  arg {
    name: "pads"
    ints: 1
    ints: 1
    ints: 1
    ints: 1
  }
  arg {
    name: "kernels"
    ints: 3
    ints: 3
  }
  arg {
    name: "group"
    i: 1
  }
  arg {
    name: "dilations"
    ints: 1
    ints: 1
  }
}
</code></pre>
<p>Do I have to make changes here or not? Do replace from <code>type: "Conv"</code> on <code>type: "Int8Conv"</code>?<br>
Help me deal with this, please. Thnx.</p>

jfix: <p>So it looks like your model is only in float right now. You cannot just simply replace <code>Conv</code> with <code>In8tConv</code> etc. – in order to use quantization you need to know the quantization parameters to use for each operator. In Glow we call this <code>scale</code> and <code>offset</code>; in Caffe2 it’s called <code>Y_scale</code> and <code>Y_zero_point</code>. These are usually based on actual values you expect to flow through your graph.</p>
<p>If you don’t know what the scales/offsets should be (likely the case), one option would be to use Glow’s profiling and quantization to quantize automatically. Like I said in my previous comment:</p>
<blockquote>
<p>However, if your network is not quantized and/or you don’t want to install Caffe2 on your Raspberry Pi, you could use try to use Glow to profile your model, quantize it, and then save what we call an ahead-of-time compiled “bundle”, which is just a binary to copy to your Raspberry Pi3 to run (see docs <a href="https://github.com/pytorch/glow/blob/master/docs/Quantization.md#how-to-perform-nn-conversion" rel="noopener nofollow ugc">here </a>and <a href="https://github.com/pytorch/glow/blob/master/docs/AOT.md#creating-standalone-executable-bundles" rel="noopener nofollow ugc">here </a>). Note that it may not perform as well as  <code>QNNPACK</code> ; we are more focused on accelerator backends right now.</p>
</blockquote>
<p>Again, this is not <code>QNNPACK</code>; Glow does not use it. If you are interested in using <code>QNNPACK</code> and Caffe2 on your Raspberry Pi then you could try asking the question in a separate category.</p>

jfix: <p>We always first import Caffe2 or ONNX protos, and generate them into Glow IR, and then profile/quantize the Glow IR from there. However once you’re in Glow IR, there is no current way to generate anything back out to Caffe2/ONNX/PyTorch protos, whether quantized or not.</p>
<p>If this might fit your needs, you can always follow <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html" rel="nofollow noopener">this tutorial</a> to get ONNX or Caffe2 from your PyTorch model, which you can then import to Glow.</p>

symashayak: <p>Hi,</p>
<p>Is there an example of taking a float32 ONNX graph, quantizing it with Glow and generate Glow IR? Is there a Python interface to Glow profile/quantization?</p>
<p>Thanks,<br>
Sikandar</p>

jfix: <p>You can follow the instructions <a href="https://github.com/pytorch/glow/blob/master/docs/Quantization.md#how-to-perform-nn-conversion" rel="nofollow noopener">here</a> on how to gather a profile of a model and then quantize the model. You just need an ONNX proto to load into Glow – see the page on Testing <a href="https://github.com/pytorch/glow/blob/master/docs/Testing.md#model-loader" rel="nofollow noopener">here</a> which discusses how to load a model using one of our example proto model Loaders. We have some limtited support for python via PyTorch through the ONNXIFI interface – you can find info <a href="https://github.com/pytorch/glow/blob/master/docs/Onnxifi.md" rel="nofollow noopener">here</a>. Otherwise it’s relatively straightforward to run the Testing Loader examples I linked to above after you’ve built in C++ and quantize/run your model.</p>

eric4337: <p>Hi <a class="mention" href="/u/jfix">@jfix</a>, it seems glow ./bin/image-classifier, currently support image classification type model only?<br>
Is there a way to quantize Image Generator type model? or other kind of model? Thanks.</p>

jfix: <p>Hi <a class="mention" href="/u/eric4337">@eric4337</a>, we currently also have a NMT model driver called <code>text-translator</code>, but it’s for pre-unrolled NMT models. We also have a <code>model-runner</code> driver but it’s very simple and just for testing, for models without any non-Constant inputs.</p>
<p>If you want to try other models you need to create your own driver, probably based on <code>tools/loader/ImageClassifier.cpp</code> if you’re interested in image-based models. This mostly would mean you are able to correctly load in and out the inputs/outputs based on their expected shape(s) and datatype(s). Also, depending on the model you may need to add additional operator support.</p>
<p>Once you have those things done you can quantize the model.</p>

AlexKoff88: <p>Hi,</p>
<p>Look what is available here: <a href="https://github.com/opencv/openvino_training_extensions/tree/develop/pytorch_toolkit/nncf" rel="nofollow noopener">https://github.com/opencv/openvino_training_extensions/tree/develop/pytorch_toolkit/nncf</a>.</p>
<p>This is a Quantization Aware Training in PyTorch with ability to export the quantized model to ONNX.</p>

blueskywwc: <p>Hi,<br>
Can you export the quantized model to onnx? thanks!</p>