How to use a quantized model on INT8 harware?

babak_hss: <p>Recently I used pytorch quantization-aware training to quantize my model.<br>
The result still has good accuracy, and it uses per channel scales.</p>
<p>However, our hardware colleagues told me that because it has FP scales and zero-points in channels, the hardware <strong>should still support FP</strong> in order to implement it.</p>
<p>They also argued that in each internal stage, the values (in-channels) should be dequantized and converted to FP and quantized again for the next layer.</p>
<p>I’m not sure if this argument is valid or if such limitation is applied on such quantization?</p>

hx89: <p>For the first argument you are right, since scales and zero-points are FP, hardware need to support FP for the computation.</p>
<p>The second argument may not be true, for static quantization the output of the previous layer can be fed into next layer without dequantizing to FP. Maybe they are thinking about dynamic quantization, which keeps tensors between two layers in FP.</p>

babak_hss: <p>Are these per channel scales determined for quantizing the network weights or for the tensors (layer outputs)?</p>
<p>If it’s the first case, then can’t we compute INT8 weights offline (using the obtained scales) and upload them to the hardware without the need for having FP support? I only use the hardware for the inference phase.</p>
<p>If these scales are required for quantizing the internal tensors (layers outputs/inputs), then doesn’t it mean that they have to be dequantized first in each stage to be quantized again with a different scale?</p>

dassima: <p>For static quantization, you can fed directly the outputs to the next layer, but how do you add the zero points if they are 32FP? or when exactly in the flow?</p>

dassima: <p>if it helps, I found out that they use the scales to convert to 8 int the weights and data (but each feature map, kernel, input has its own scale) and after that do the multiplications in 8 int, they accumulate the result to 32int for each layer and requantize to 8 int using another scale which you can find it only when you acces the model as a dictionary.</p>

babak_hss: <aside class="quote no-group" data-username="dassima" data-post="5" data-topic="70605">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dassima/48/20474_2.png" class="avatar"> dassima:</div>
<blockquote>
<p>they accumulate the result to 32int for each layer and requantize to 8 int using another scale</p>
</blockquote>
</aside>
<p>So, does this requantization step in each layer (to convert its output to INT8 with another scale) requires the hardware to support FP32 arithmetic?</p>
<p>Also, how expensive would this conversion be compared to doing everything in FP32?</p>

dassima: <p>I think the answer is yes if you want to do in hardware the whole inference at once… And I do not know how expensive it is in comparison. I think it actually depends on the piece of hardware</p>

raghuramank100: <p>Yes, in our implementations we do a floating point multiply to requantize from 32 bit to 8 bit. You can also do this with integer arithmetic if needed. We found that on ARM and x86 CPUs, doing the requantization with fp32 is more efficient.</p>

babak_hss: <p>Is there other possibilities like having INT8/INT32 scale factors or having INT32 but non-per channel quantization?</p>
<p>The per-channel quantization is great regarding the accuracy, but I do not need that much of accuracy and instead need to perform all the calculations in INT8 or even INT32.</p>

jerryzh168: <p>I see, we don’t have this right now, you will need to write a new Quantizer to enable this: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/quantized/Quantizer.h" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/quantized/Quantizer.h</a></p>

mg1371: <p>Is there a way to fake quantize scale factors or zero points (instead of defining new Quantizers)?</p>

jerryzh168: <p>if you only need this in quantization aware training, you’ll need to define your own fake quantize module(<a href="https://github.com/pytorch/pytorch/blob/v1.3.1/torch/quantization/fake_quantize.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/v1.3.1/torch/quantization/fake_quantize.py</a>) and fake quantize op<br>
(<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/fake_quant_per_tensor_affine.cpp" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/fake_quant_per_tensor_affine.cpp</a>)</p>