Qnnpack using activation dtype int8 is not runnable

TingfengTang: <p>hi, <a class="mention" href="/u/jerryzh168">@jerryzh168</a>, when i convert a model using</p>
<pre><code class="lang-auto">qconfig=default_symmetric_qnnpack_qat_qconfig
</code></pre>
<p>the jit model can not run using libtorch2.01(c++) on win10 vs2019</p>
<p>but if I use the following qconfig</p>
<pre><code class="lang-auto">qconfig=get_default_qat_qconfig(backend='qnnpack', version=1)
</code></pre>
<p>it will work.</p>
<p>the code i used is below:</p>
<pre><code class="lang-auto">import torch

class Model(torch.nn.Module):
    def __init__(self):
        super(Model,self).__init__()
        self.conv_test=torch.nn.Conv2d(1,3,(3,3))
    def forward(self,x):
        x=self.conv_test(x)
        return x
input=torch.randn(1,1,15,15)
model=Model()

import torch.ao.quantization.quantize_fx as quantize_fx
import copy

model_to_quantize=copy.deepcopy(model)
model_to_quantize.train()

from torch.ao.quantization.qconfig_mapping import QConfigMapping
from torch.ao.quantization.qconfig import default_symmetric_qnnpack_qat_qconfig,get_default_qat_qconfig
qconfig=get_default_qat_qconfig(backend='qnnpack', version=1)
# qconfig=default_symmetric_qnnpack_qat_qconfig
qconfig_mapping = QConfigMapping().set_global(qconfig)
backend_config=torch.ao.quantization.backend_config.qnnpack.get_qnnpack_backend_config()
torch.backends.quantized.engine = 'qnnpack'

model_prepared = quantize_fx.prepare_qat_fx(model_to_quantize,qconfig_mapping=qconfig_mapping,example_inputs=input,backend_config=backend_config)
model_prepared(input)
model_prepared(input)
model_quantized = quantize_fx.convert_fx(model_prepared,qconfig_mapping=qconfig_mapping,backend_config=backend_config)

model_trace=torch.jit.trace(model_quantized,input)
torch.jit.save(model_trace,'conv_layer.pth')
</code></pre>
<p>pytorch version : 2.0.1+cu117</p>

jerryzh168: <p>what is the error message?</p>

TingfengTang: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29a39ac902cfccdc0fbeb3cb5d9aac1ee255ae19.png" data-download-href="https://discuss.pytorch.org/uploads/default/29a39ac902cfccdc0fbeb3cb5d9aac1ee255ae19" title="截图_libotrch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29a39ac902cfccdc0fbeb3cb5d9aac1ee255ae19.png" alt="截图_libotrch" data-base62-sha1="5Wm638EglwY6eU6NJ46bU0iqDfb" width="690" height="255" data-dominant-color="D2E4D5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截图_libotrch</span><span class="informations">980×363 38 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
the source code i use is:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/4486fe22e04f8de4bb4792aed6b106ba82a38f8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/4486fe22e04f8de4bb4792aed6b106ba82a38f8a" title="截图_libtorch2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/4486fe22e04f8de4bb4792aed6b106ba82a38f8a.png" alt="截图_libtorch2" data-base62-sha1="9MdGGOMz1lYtbDj1HBFLIIueqQq" width="690" height="382" data-dominant-color="C8E2CB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截图_libtorch2</span><span class="informations">863×478 34.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

jerryzh168: <p>this looks like a lint error? does it actually results in real errors?</p>

TingfengTang: <p>It’s actually a runtime error.<br>
I have also run the jit model using C++ on Linux_x86_64 platform, the error produced by  the following qconfig is more clear:</p>
<pre><code class="lang-auto">qconfig=default_symmetric_qnnpack_qat_qconfig
</code></pre>
<p>error message:</p>
<pre><code class="lang-auto">(torch2.01) /libTorchTest/build# ./example-app 
loading model success
thread number of the machine is:48
terminate called after throwing an instance of 'std::runtime_error'
  what():  The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/torch/fx/graph_module.py", line 15, in forward
    conv_test_input_scale_0 = self.conv_test_input_scale_0
    input = torch.quantize_per_tensor(x, conv_test_input_scale_0, conv_test_input_zero_point_0, 12)
    _0 = torch.dequantize((conv_test).forward(input, ))
                           ~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    return _0
  File "code/__torch__/torch/ao/nn/quantized/modules/conv.py", line 86, in forward
    input: Tensor) -&gt; Tensor:
    _packed_params = self._packed_params
    conv_test = ops.quantized.conv2d(input, _packed_params, 0.017023416236042976, -16)
                ~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    return conv_test

Traceback of TorchScript, original code (most recent call last):
/home/sfadmin/anaconda3/envs/torch2.01/lib/python3.9/site-packages/torch/_ops.py(502): __call__
/home/sfadmin/anaconda3/envs/torch2.01/lib/python3.9/site-packages/torch/ao/nn/quantized/modules/conv.py(469): forward
/home/sfadmin/anaconda3/envs/torch2.01/lib/python3.9/site-packages/torch/nn/modules/module.py(1488): _slow_forward
/home/sfadmin/anaconda3/envs/torch2.01/lib/python3.9/site-packages/torch/nn/modules/module.py(1501): _call_impl
&lt;eval_with_key&gt;.6(8): forward
/home/sfadmin/anaconda3/envs/torch2.01/lib/python3.9/site-packages/torch/nn/modules/module.py(1488): _slow_forward
/home/sfadmin/anaconda3/envs/torch2.01/lib/python3.9/site-packages/torch/nn/modules/module.py(1501): _call_impl
/home/sfadmin/anaconda3/envs/torch2.01/lib/python3.9/site-packages/torch/fx/graph_module.py(271): __call__
/home/sfadmin/anaconda3/envs/torch2.01/lib/python3.9/site-packages/torch/fx/graph_module.py(662): call_wrapped
/home/sfadmin/anaconda3/envs/torch2.01/lib/python3.9/site-packages/torch/jit/_trace.py(1056): trace_module
/home/sfadmin/anaconda3/envs/torch2.01/lib/python3.9/site-packages/torch/jit/_trace.py(794): trace
quant_2.py(33): &lt;module&gt;
RuntimeError: quantized::conv2d (ONEDNN): data type of input should be QUint8.

Aborted (core dumped)
</code></pre>
<p>It seems that the qconfig of symmetric activation can not be executed on x86_64. but the python script works well on both qconfigs.</p>
<p>all the code I used is as follow:<br>
python script:</p>
<pre><code class="lang-auto">import torch

class Model(torch.nn.Module):
    def __init__(self):
        super(Model,self).__init__()
        self.conv_test=torch.nn.Conv2d(1,3,(3,3))
    def forward(self,x):
        x=self.conv_test(x)
        return x
input=torch.randn(1,1,15,15)
model=Model()

import torch.ao.quantization.quantize_fx as quantize_fx
import copy

model_to_quantize=copy.deepcopy(model)
model_to_quantize.train()

from torch.ao.quantization.qconfig_mapping import QConfigMapping,_get_symmetric_qnnpack_qconfig_mapping
from torch.ao.quantization.qconfig import default_symmetric_qnnpack_qat_qconfig,get_default_qat_qconfig
# qconfig=get_default_qat_qconfig(backend='qnnpack', version=1)
qconfig=default_symmetric_qnnpack_qat_qconfig
qconfig_mapping = QConfigMapping().set_global(qconfig)
backend_config=torch.ao.quantization.backend_config.qnnpack.get_qnnpack_backend_config()
torch.backends.quantized.engine = 'qnnpack'

model_prepared = quantize_fx.prepare_qat_fx(model_to_quantize,qconfig_mapping=qconfig_mapping,example_inputs=input,backend_config=backend_config)
model_prepared(input)
model_prepared(input)
model_quantized = quantize_fx.convert_fx(model_prepared,qconfig_mapping=qconfig_mapping,backend_config=backend_config)


model_trace=torch.jit.trace(model_quantized,input)
torch.jit.save(model_trace,'conv_layer_3.pth')

model_trace_reload=torch.jit.load('conv_layer_3.pth')

result_ref=model_quantized(input)
result_model_trace_reload=model_trace_reload(input)
print(torch.allclose(result_ref,result_model_trace_reload))
</code></pre>
<p>the c++ test code:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include "torch/torch.h"
#include "torch/script.h"
int main(int argc, char* argv[])
{
    torch::jit::script::Module module;
    try{
        module=torch::jit::load("./conv_layer_3.pth");
    }
    catch(const c10::Error&amp; e){
        std::cerr&lt;&lt;"error loading model:"&lt;&lt;e.what()&lt;&lt;std::endl;
        throw;
    }
    std::cout&lt;&lt;"loading model success"&lt;&lt;std::endl;
    std::cout&lt;&lt;"thread number of the machine is:"&lt;&lt;torch::get_num_threads()&lt;&lt;std::endl;
    torch::set_num_threads(1);
    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.push_back(torch::randn({1,1,15,15}));
    at::Tensor output=module.forward(inputs).toTensor();
    std::cout&lt;&lt;output&lt;&lt;std::endl;
    std::cout&lt;&lt;"calc finished"&lt;&lt;std::endl;
}
</code></pre>
<p>CMakeList.txt:</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.18 FATAL_ERROR)
project(example-app)

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(example-app test.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 17)
</code></pre>

jerryzh168: <aside class="quote no-group" data-username="TingfengTang" data-post="5" data-topic="185243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tingfengtang/48/41874_2.png" class="avatar"> TingfengTang:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: quantized::conv2d (ONEDNN): data type of input should be QUint8.
</code></pre>
</blockquote>
</aside>
<p>looks like the error is: <code>RuntimeError: quantized::conv2d (ONEDNN): data type of input should be QUint8.</code></p>
<p>“It seems that the qconfig of symmetric activation can not be executed on x86_64”, I think x86_64 is expecting activation dtype to be QUInt8, not QInt8 from the “get_qnnpack_backend_config”</p>
<p>are you trying to use qnnpack in jit runtime?</p>

TingfengTang: <p>Yes, I am trying to use it in C++ backend on x86_64 platform.<br>
I see that QInt8 input and output is supported by  “get_qnnpack_backend_config” and can be executed in python script but failed in C++ environment.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/backend_config/qnnpack.py#L95-L100">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/backend_config/qnnpack.py#L95-L100" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/backend_config/qnnpack.py#L95-L100" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/torch/ao/quantization/backend_config/qnnpack.py#L95-L100</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="95" style="counter-reset: li-counter 94 ;">
          <li>qnnpack_weighted_op_qint8_symmetric_dtype_config = DTypeConfig(</li>
          <li>    input_dtype=qnnpack_act_qint8_scale_min_2_neg_12,</li>
          <li>    output_dtype=qnnpack_act_qint8_scale_min_2_neg_12,</li>
          <li>    weight_dtype=qnnpack_weight_qint8_neg_127_to_127_scale_min_2_neg_12,</li>
          <li>    bias_dtype=torch.float,</li>
          <li>)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


jerryzh168: <p>in that case you should use fbgemm/x86 configs, not qnnpack configs I think: <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/qconfig.py#L313" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/qconfig.py#L313</a></p>

TingfengTang: <p>Is there a future support on qint8 input/output on x86_64? it’s inefficient to test large data using qnnpack backend devides</p>

jerryzh168: <p>I think you can ask <a class="mention" href="/u/leslie-fang-intel">@leslie-fang-intel</a> <a class="mention" href="/u/jiong_gong">@Jiong_Gong</a></p>

Jiong_Gong: <p>For x86, we support quint8 with asymmetric quantization for activations (weight is qint8 with symmetric per-tensor/per-channel). Are you able to use this configuration instead?</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/backend_config/x86.py">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/backend_config/x86.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/backend_config/x86.py" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/main/torch/ao/quantization/backend_config/x86.py</a></h4>


      <pre><code class="lang-py">import torch
from ._common_operator_config_utils import (
    _get_binary_op_configs,
    _get_bn_configs,
    _get_cat_config,
    _get_conv_configs,
    _get_default_op_configs,
    _get_embedding_op_configs,
    _get_fixed_qparams_op_configs,
    _get_linear_configs,
    _get_rnn_op_configs,
    _get_share_qparams_op_configs,
    _get_tensor_info_op_configs,
)
from .backend_config import BackendConfig, DTypeConfig

__all__ = [
    "get_x86_backend_config",
]

</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/quantization/backend_config/x86.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


TingfengTang: <p>From the x86.py, I know that it’s activation support dtype is quint8 for conv2d on x86 backend. Is there a future support of qint8 activation for  x86_64?  QNNPACK has the qint8 activation config, but the exported model is not usable on my windows environment.</p>

rick1: <aside class="quote no-group" data-username="jerryzh168" data-post="6" data-topic="185243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/48/15217_2.png" class="avatar"> jerryzh168:</div>
<blockquote>
<p>use qnnpack in jit runtime</p>
</blockquote>
</aside>
<p>hello , I also meet a error " RuntimeError: quantized::conv2d (ONEDNN): data type of input should be QUint8. " when i try to export a  S8S8(activation qint8) quantize model to onnx, if there are any solution for this?</p>

TingfengTang: <p>may be this will help to you <a href="https://github.com/pytorch/pytorch/issues/106598" class="inline-onebox" rel="noopener nofollow ugc">jit model of qnnpack's int8 activation can not be executed on x86_64 using libtorch but python script works well · Issue #106598 · pytorch/pytorch · GitHub</a></p>

rick1: <p>i modyfied my code by adding:</p>
<pre><code class="lang-auto">q_backend = "qnnpack"  # qnnpack  or fbgemm
torch.backends.quantized.engine = q_backend
</code></pre>
<p>and i got:<br>
quantized::linear (xnnpack): Unsupported config for dtype KQInt8<br>
it seems change backend is make sense , but how do i make the config support?<br>
my config is</p>
<pre><code class="lang-auto">   qconfig = torch.ao.quantization.qconfig.QConfig(
    activation=torch.ao.quantization.observer.HistogramObserver.with_args(
        qscheme=torch.per_tensor_symmetric, dtype=torch.qint8),
    weight=torch.ao.quantization.observer.default_per_channel_weight_observer
)
    qconfig_mapping = (QConfigMapping()
        .set_global(qconfig)  
       
    )
</code></pre>

TingfengTang: <p>it seems that pytorch version below 2.0 does not support qint8 input. you can update your pytorch to  the latest version and have a try following the link above.</p>

rick1: <p>thankyou for reply, i have relove it by change my backend but qnnpack  does not currently support per-channel fully connected op… <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=12" title=":rofl:" class="emoji" alt=":rofl:" loading="lazy" width="20" height="20">it is hard to convert a fx quantize model to onnx</p>