Is there any way to avoid zero_point subtraction in quantization inference?

Alex_Sung: <p>Hi there,<br>
I’m new to here.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6e75b5c6c4f386aa37d05acbbba79909edb44bd.png" data-download-href="https://discuss.pytorch.org/uploads/default/d6e75b5c6c4f386aa37d05acbbba79909edb44bd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6e75b5c6c4f386aa37d05acbbba79909edb44bd_2_690x135.png" alt="image" data-base62-sha1="uF7Re5nJiY9rSaTmOhjcSg4BxzD" width="690" height="135" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6e75b5c6c4f386aa37d05acbbba79909edb44bd_2_690x135.png, https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6e75b5c6c4f386aa37d05acbbba79909edb44bd.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6e75b5c6c4f386aa37d05acbbba79909edb44bd.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">888×174 24.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>During quantization, we’ll calculate <code>scale</code> and <code>zero_point</code> and follow the above formula while performing inference. In my project, <strong>I need to avoid that <code>zero_point</code> subtraction.</strong></p>
<p>I know my issue is similar to <strong>symmetric quantization.</strong> But I don’t know how to implement it on Pytorch.</p>
<p>I’ve tried using Pytorch quantization API, and there seems to be no function that meets my requirement. I’ve tried using <code>qscheme=torch.per_tensor_symmetric</code>, and set <code>dtype=qint8</code>.</p>
<p>However, during inference, the model couldn’t work. Since the activations need to be <code>quint8</code>, while I’ve set it to <code>qint8</code> to make zero_point=0.</p>
<p>Besides, I’ve also implemented QAT from scratch without using quantization API, and manully setting zero_poinrt=0, but the performance dropped a lot, and I don’t know where was wrong.</p>
<p>Could any one give me some suggestion? Thanks a lot! <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"></p>

jerryzh168: <p>dtype=qint8 for activation is currently not supported in the native backend (server and mobile cpu) of pytorch. Could you specify which bakcend/device (server cpu/mobile cpu/cuda etc.) are you targeting?</p>
<p>For the QAT question, are you setting zero_point to 0 after training? I don’t think we can do that</p>

Alex_Sung: <p>Hi,<br>
None of these backend are my targets. I need to implement the neural network on <strong>chip</strong>, so Multiply-Add operation only is preferred. Subtraction operation may be a burden for me.</p>
<p>For QAT, I think I’m just performing symmetric quantization, mapping floating point value to -128~127, where setting z=0 is okay.<br>
When there’s ReLU, <code>FakeQuantize</code> floating point to 0~255.<br>
When there’s no ReLU, <code>FakeQuantize</code> floating point to -128-127.<br>
<code>(FakeQuantize = quantize then dequantize)</code></p>
<p>My QAT work from scratch, self-define FakeQuantize, Q-convolution and so on, can do well on easy task like MNINST, but I couldn’t get it work as well as FP32 version of my object tracking task.</p>
<p>Therefore, I’m turning back to pytorch to see if there’s any API to set z=0.</p>

jerryzh168: <p>just to clarify, the goal is to do QAT and set zero_point to 0? can you just set dtype to <code>qint8</code> and qscheme to <code>per_tensor_symmetric</code> when you use <code>FakeQuantize</code>?</p>