Quantization : How to avoid zero subtraction at run time in dequantization process

username: vanajareedy0426
cooked: <p>I’m applying quantization for my model which is fp32. I’m having convolution layers in my model. Once quantization is done, I have int8 weights. I’m using those int8 weights and in quantize process I need to subtract the zero_point from every weight and multiply the output with scale.</p>
<p>Since the zero_point subtraction is in the loop (need to do it for every weight , We are getting high CPU . Now, we want to avoid that zero subtraction in the loop and plan to place store the weight - zero_point value as weights and only need to do scale multiplication in the dequantization process.</p>
<p>But, when I tried this facing some issues with output mismatch. is there any way to avoid this zero_point subtraction in run time? if not what is the reason? please help</p>
<p>reference link which we are following <a href="https://leimao.github.io/article/Neural-Networks-Quantization/" class="inline-onebox" rel="noopener nofollow ugc">Lei Mao's Log Book – Quantization for Neural Networks</a></p>
<p>Thanks in advance!</p>

username: HDCharles
cooked: <p>Can you give a repro of the issue? I’m not sure exactly what you are trying to do, the quantization APIs should be handling all of that for you.</p>

username: vanajareedy0426
cooked: <p>we are not using quantization APIs , I’m doing quantization using C code.<br>
The peace of code I’m using for quantization is this</p>
<pre><code class="lang-auto">
struct QuantizationParams {
   float scale;
   unsigned char zero_point;
};
void Quantize(const QuantizationParams&amp; qparams, float* src, unsigned char* dst, int size) {

   for (std::size_t i = 0; i &lt; size; i++) {
      const float real_val = src[i];
      const float transformed_val = qparams.zero_point + real_val / qparams.scale;
      const float clamped_val = std::max(0.f, std::min(255.f, transformed_val));
      dst[i] = static_cast&lt;std::uint8_t&gt;(std::round(clamped_val));
   }
}
</code></pre>
<p>let’s say Im having a function with input and weight . And i’m using quantized weight for the computation . weight is unsigned char , zero point is unsigned char and scale is float .<br>
Then my function implementation is something like given below , and it is working absolutely fine</p>
<pre><code class="lang-auto">
for (i_l = 0; i_l &lt;= size1; i_l += size2)
   {
      A_lp = &amp;QA_lp[i_l];
      for (j_l = 0; j_l &lt; size5; j_l++)
      {
         B_lp = (unsigned char*)&amp;B[j_l * size3];
         sum_l = 0;
         for (k_l = 0; k_l &lt; size4; k_l ++)
         {
            sum_l += (A_lp[k_l] ) * (B_lp[k_l] - ZERO_POINT);
         }
		 *C++ = (float) sum_l *SCALE
}
 
</code></pre>
<p>sum_l += (A_lp[k_l] ) * (B_lp[k_l] - ZERO_POINT);<br>
Now , what I’m trying is , the ZERO_POINT subtraction i want to do it before it self and save it as my weights.<br>
Now my weight will be weight - ZERO_POINT .And my function code will be<br>
{<br>
sum_l += (A_lp[k_l] ) * (B_lp[k_l] );<br>
}<br>
*C++ = (float) sum_l *SCALE<br>
}</p>
<p>I’m trying to avoid that subtraction in the loop .And here my data type of weights (weight - ZERO_POINT) is char.<br>
When I’m doing this, the output is not matching with the original one. What will be reason ?</p>
<p>Please help me with this.</p>