FBGEMM with PyTorch Mobile

username: pshashk
cooked: <p>Is it possible to run a model with fbgemm qconfig on mobile? Or is it x86 only? Simply plugging such model into demo app triggers qnnpack assert here <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qconv_prepack.cpp#L223" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qconv_prepack.cpp#L223</a><br>
It seems like FBGEMM support was disabled by this commit for some reason <a href="https://github.com/pytorch/pytorch/commit/6fead9afd4cdc6306fb0e2180ca625160b59ea71" rel="nofollow noopener">https://github.com/pytorch/pytorch/commit/6fead9afd4cdc6306fb0e2180ca625160b59ea71</a></p>

username: pshashk
cooked: <p>I wasn’t able to get good results from QNNPACK compatible per tensor quantization qconfig. Target metric value relative to fp32 model:<br>
<code>get_default_qconfig('fbgemm')</code> -&gt; 99.8%<br>
<code>get_default_qconfig('qnnpack')</code> -&gt; 58.5%<br>
<code>default_qconfig</code> -&gt; 54.4%<br>
Is there any way to reduce that gap without changing architecture?</p>

username: raghuramank100
cooked: <p>FBGEMM is supported only for x86. You can get very good accuracies for qnnpack also.<br>
Please make sure that when you set:</p>
<pre><code class="lang-auto">qconfig = torch.quantization.get_default_qconfig('qnnpack')
</code></pre>
<p>You also do:</p>
<pre><code class="lang-auto">torch.backends.quantized.engine = 'qnnpack'
</code></pre>
<p>before running the model.<br>
The poorer accuracy numbers are likely due to FBGEMM saturating for large weight/activation values, due to this issue:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/intel/mkl-dnn/blob/f38fecf5b76421fe277cfb15ec1d5090f1d30c07/doc/advanced/int8_computations.md#1-inputs-of-mixed-type-u8-and-s8" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/intel/mkl-dnn/blob/f38fecf5b76421fe277cfb15ec1d5090f1d30c07/doc/advanced/int8_computations.md#1-inputs-of-mixed-type-u8-and-s8" target="_blank" rel="nofollow noopener">intel/mkl-dnn/blob/f38fecf5b76421fe277cfb15ec1d5090f1d30c07/doc/advanced/int8_computations.md#1-inputs-of-mixed-type-u8-and-s8</a></h4>
<pre><code class="lang-md">Int8 Computation Aspects {#dev_guide_int8_computations}
=======================================================

&gt; This document uses **int8** to denote 8-bit integer no matter whether it is
&gt; signed or unsigned. To emphasize the signedness of the data type
&gt; **u8** (`uint8_t`) or **s8** (`int8_t`) are used. In particular, if a
&gt; primitive has two inputs the types would be written using "/". For instance:
&gt; - int8 GEMM denotes any integer GEMM with 8-bit integer inputs, while
&gt; - u8/s8 GEMM denotes dnnl_gemm_u8s8s32() only.

The operation primitives that work with the int8 data type
(#dnnl::memory::data_type::s8 and #dnnl::memory::data_type::u8)
typically use s32 (`int32_t`) as an intermediate data type
(#dnnl::memory::data_type::s32) to avoid integer overflows.

For instance, the int8 average [pooling](@ref dev_guide_pooling) primitive
accumulates the int8 input values in a window to an s32 accumulator, then
divides the result by the window size, and then stores the result back to the
int8 destination:

</code></pre>

  This file has been truncated. <a href="https://github.com/intel/mkl-dnn/blob/f38fecf5b76421fe277cfb15ec1d5090f1d30c07/doc/advanced/int8_computations.md#1-inputs-of-mixed-type-u8-and-s8" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: pshashk
cooked: <p>Thanks. With engine set preparation, calibration, and conversion of the model work fine. But evaluation triggers errors like: <code>Error in QNNPACK: failed to create convolution with 0.1966128 input scale, 1.698165 kernel scale, and 0.2075303 output scale: convolution scale 1.608829 is greater or equal to 1.0</code>. The cause seems to be in the SE block implemented via 1x1 convolution that receives 1x1 input. I probably should have used <code>Linear</code> anyway, but maybe it will be useful to someone.</p>

username: pshashk
cooked: <p>Ok, I’ve managed to get good result from QNNPACK. Maybe <code>torch.backends.quantized.engine</code> should be mentioned somewhere on <a href="https://pytorch.org/docs/master/quantization.html" rel="nofollow noopener">quantization page</a>?</p>

username: supriyar
cooked: <p>Great that this worked! We will make sure to mention this on our quantization page. Thanks for the suggestion! cc <a class="mention" href="/u/raghuramank100">@raghuramank100</a></p>