Casting from 32b to 8 bit after accumulation in a multiplication

dassima: <p>Hi,</p>
<p>I was reading about the QNNpack and FBGEMM configuration which explains really well the way to multiply, but I am left with the question what is happening after you multiply 8b x 8b and accumulate the result on 16 or 32 bits…how do you cast to 8 bits to get to another convolution?</p>

jerryzh168: <p>cc <a class="mention" href="/u/dskhudia">@dskhudia</a> <a class="mention" href="/u/jianyuhuang">@jianyuhuang</a></p>

jianyuhuang: <p>In FBGEMM, we have requantization as a post-processing step to convert the 32 bit to 8 bit after the accumulation. The requantization basically does the following op in the language of numpy:</p>
<pre><code class="lang-auto">X_q = (np.round(X / X_scale) + X_zp).clip(0, 255).astype(np.uint8)
</code></pre>
<p>The modularized requantization wrapper is here:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/FBGEMM/blob/master/include/fbgemm/OutputProcessing-inl.h#L59-L185" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/FBGEMM/blob/master/include/fbgemm/OutputProcessing-inl.h#L59-L185" target="_blank" rel="nofollow noopener">pytorch/FBGEMM/blob/master/include/fbgemm/OutputProcessing-inl.h#L59-L185</a></h4>
<pre class="onebox"><code class="lang-h"><ol class="start lines" start="59" style="counter-reset: li-counter 58 ;">
<li>template &lt;</li>
<li>    bool FUSE_RELU,</li>
<li>    QuantizationGranularity Q_GRAN,</li>
<li>    typename BIAS_TYPE,</li>
<li>    typename outT,</li>
<li>    typename inT,</li>
<li>    typename nextOPType&gt;</li>
<li>template &lt;inst_set_t instSet&gt;</li>
<li>inline int</li>
<li>ReQuantizeOutput&lt;FUSE_RELU, Q_GRAN, BIAS_TYPE, outT, inT, nextOPType&gt;::f(</li>
<li>    outT* out,</li>
<li>    const inT* inp,</li>
<li>    const block_type_t&amp; block,</li>
<li>    int ld_out,</li>
<li>    int ld_in) const {</li>
<li>  static_assert(</li>
<li>      std::is_same&lt;inT, int32_t&gt;::value,</li>
<li>      "input data type must be of int32_t type");</li>
<li>  int ncol_per_group = ncols_ / groups_;</li>
<li>  assert(</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/FBGEMM/blob/master/include/fbgemm/OutputProcessing-inl.h#L59-L185" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
And inside it a efficient AVX2 kernel is implemented:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/FBGEMM/blob/master/src/QuantUtilsAvx2.cc#L533-L554" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/FBGEMM/blob/master/src/QuantUtilsAvx2.cc#L533-L554" target="_blank" rel="nofollow noopener">pytorch/FBGEMM/blob/master/src/QuantUtilsAvx2.cc#L533-L554</a></h4>
<pre class="onebox"><code class="lang-cc"><ol class="start lines" start="533" style="counter-reset: li-counter 532 ;">
<li>/*</li>
<li> * Convert scaled FP32 result to int32_t using CVTPS2DQ instruction.</li>
<li> * CVTPS2DQ instruction rounds result according to nearest FP32 value</li>
<li> * with ties to even (assuming default MXCSR rounding mode). However,</li>
<li> * when conversion overflows, it produces INT32_MIN as a result. For</li>
<li> * large positive inputs the result of conversion can become negative,</li>
<li> * which affects the final requantization result. Note that on x86</li>
<li> * SSE2 we have e.g. int32_t(float(INT32_MAX)) == INT32_MIN! This</li>
<li> * happens because float(INT32_MAX) rounds to 2**31, which overflows</li>
<li> * int32_t when it is converted back to integer.</li>
<li> *</li>
<li> * Thankfully, we can prove that overflow never happens in this</li>
<li> * requantization scheme. The largest positive input is INT32_MAX</li>
<li> * (2**31 - 1), which turns into 2**31 when converted to float. The</li>
<li> * largest scale value is 0x1.FFFFFEp-1. When multiplied together, the</li>
<li> * result is 2147483520 (compare to INT32_MAX = 2147483647), which</li>
<li> * fits into int32_t without overflow.</li>
<li> */</li>
<li>__m256i x_rounded_v = _mm256_cvtps_epi32(x_scaled_v);</li>
<li>__m256i y_rounded_v = _mm256_cvtps_epi32(y_scaled_v);</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/FBGEMM/blob/master/src/QuantUtilsAvx2.cc#L533-L554" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

dassima: <p>ok, this makes sense now. But this raises 3 questions for me when I computed an example:</p>
<ol>
<li>How do I access the scale of the re-quantization? I just managed to see how the scale of converting from 32float to 8int looks like:<br>
BASIC EXAMPLE:</li>
</ol>
<pre><code class="lang-auto">input: tensor([[[[ 1.0074,  2.0148,  3.0222,  4.0297],
                 [ 5.0371,  6.0445,  7.0519,  8.0593],
                 [ 8.9408,  9.9482, 10.9556, 11.9630],
                 [12.9704, 13.9779, 14.9853, 15.9927]]]], size=(1, 1, 4, 4),
                 dtype=torch.quint8, quantization_scheme=torch.per_tensor_affine,
                 scale=0.125926584005, zero_point=0)
Weight:  &lt;bound method Conv2d.weight of QuantizedConv2d(1, 1, kernel_size=(3, 3), stride=(1, 1), 
          scale=4.51094579697, zero_point=0)
output: tensor([[[[347.3428, 392.4523],
                  [527.7806, 572.8901]]]], size=(1, 1, 2, 2), dtype=torch.quint8,
                  quantization_scheme=torch.per_tensor_affine, scale=4.51094579697,
                  zero_point=0)
</code></pre>
<p>PRINTING OUT THE DICTIONARY OF 8INT MODEL:</p>
<pre><code class="lang-auto">[(u'conv1.weight', tensor([[[[0.9882, 1.9765, 2.9647],
                             [4.0235, 5.0118, 6.0000],
                             [6.9882, 7.9765, 8.9647]]]], size=(1, 1, 3, 3), dtype=torch.qint8,
                             quantization_scheme=torch.per_channel_affine,
                             scale=tensor([0.0706], dtype=torch.float64), zero_point=tensor([0]),
                             axis=0)),
(u'conv1.scale', tensor(4.5109)),
(u'conv1.zero_point', tensor(0)), 
(u'conv1.bias', tensor([0.], requires_grad=True)), 
(u'quant.scale', tensor([0.1259])), 
(u'quant.zero_point', tensor([0]))]
</code></pre>
<p>Because what I have is:</p>
<ul>
<li>scale 0.12 to convert input from 32float to 8int</li>
<li>scale 0.0706 to convert weights from 32float to 8int</li>
<li>scale 4.51 to convert output from 32float to 8int (just to see how all the values are in int)</li>
</ul>
<p><strong>I was expecting the scale of requantization to be something like 8.33 for this example.</strong><br>
2. Are this scales trainable parameters or are computed using basic numpy operations?</p>
<ol start="2">
<li>In the FBGEMM documentation on Facebook Engineering I’ve read that the accumulation is done on 16b… so then from where is the difference?</li>
</ol>

onesnow123q: <aside class="quote no-group" data-username="dassima" data-post="4" data-topic="69512">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dassima/48/20474_2.png" class="avatar"> dassima:</div>
<blockquote>
<p>on Facebook Engineering I’ve read that the accumulation is done on 16b… so then from where is the difference?</p>
</blockquote>
</aside>
<p>HELLO, if you already know the principle？？I am confused about it too.</p>