[ONNX] Quantized fused Conv2d won't trace

username: zetyquickly
cooked: <p><span class="hashtag">#onnx</span> <a class="hashtag" href="/c/jit/13">#<span>jit</span></a> <a class="hashtag" href="/c/quantization/17">#<span>quantization</span></a></p>
<p>Hi, I am very confused.</p>
<p>While tracing to ONNX my quantized model faced an error. This happens with fused <code>QuantizedConvReLU2d</code>. I use <code>OperatorExportTypes.ONNX_ATEN_FALLBACK</code>.<br>
Pytorch version is <code>1.6.0.dev20200520</code></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "./tools/caffe2_converter.py", line 115, in &lt;module&gt;
    caffe2_model = export_caffe2_model(cfg, model, first_batch)
  File "/root/some_detectron2/detectron2/export/api.py", line 157, in export_caffe2_model
    return Caffe2Tracer(cfg, model, inputs).export_caffe2()
  File "/root/some_detectron2/detectron2/export/api.py", line 95, in export_caffe2
    predict_net, init_net = export_caffe2_detection_model(model, inputs)
  File "/root/some_detectron2/detectron2/export/caffe2_export.py", line 144, in export_caffe2_detection_model
    onnx_model = export_onnx_model(model, (tensor_inputs,))
  File "/root/some_detectron2/detectron2/export/caffe2_export.py", line 63, in export_onnx_model
    export_params=True,
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/__init__.py", line 172, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 92, in export
    use_external_data_format=use_external_data_format)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 530, in _export
    fixed_batch_size=fixed_batch_size)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 366, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 319, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/jit/__init__.py", line 284, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 577, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/jit/__init__.py", line 372, in forward
    self._force_outplace,
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/jit/__init__.py", line 358, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 575, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 561, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/root/some_detectron2/detectron2/export/caffe2_modeling.py", line 319, in forward
    features = self._wrapped_model.backbone(images.tensor)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 575, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 561, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/root/DensePose_ADASE/densepose/modeling/quantize_caffe2.py", line 166, in new_forward
    p5, p4, p3, p2 = self.bottom_up(x)  # top-&gt;down
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 575, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 561, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/timm/models/efficientnet.py", line 350, in forward
    x = self.conv_stem(x)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 575, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 561, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/intrinsic/quantized/modules/conv_relu.py", line 71, in forward
    input, self._packed_params, self.scale, self.zero_point)
RuntimeError: Tried to trace &lt;__torch__.torch.classes.quantized.Conv2dPackedParamsBase object at 0x5600474e9670&gt; but it is not part of the active trace. Modules that are called during a trace must be registered 
as submodules of the thing being traced.
</code></pre>
<p>May presense of <code>pre_forward</code> hooks in <code>self.bottom_up(x)</code> (but not the <code>self.conv_stem(x)</code>) affect tracing such way?<br>
Model were QAT with preserving hooks from commit <a href="https://github.com/pytorch/pytorch/pull/37233" rel="nofollow noopener">https://github.com/pytorch/pytorch/pull/37233</a><br>
Also PT -&gt; ONNX -&gt; Caffe2 exporting works on this very model without quantization patching</p>

username: zetyquickly
cooked: <p>P.S. here’s also a warning</p>
<pre><code class="lang-auto">/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/quantized/modules/utils.py:10: UserWarning: 0quantize_tensor_per_tensor_affine current rounding mode is not set to round-to-nearest-ties-to-e
ven (FE_TONEAREST). This will cause accuracy issues in quantized models. (Triggered internally at  /opt/conda/conda-bld/pytorch_1589958443755/work/aten/src/ATen/native/quantized/affine_quantizer.cpp:25.)
  float(wt_scale), int(wt_zp), torch.qint8)
</code></pre>

username: jerryzh168
cooked: <p>cc <a class="mention" href="/u/james_reed">@James_Reed</a> is this related to TorchBind object?</p>

username: jerryzh168
cooked: <p><a class="mention" href="/u/zetyquickly">@zetyquickly</a> could you give a minimal repo of the issue?</p>

username: zetyquickly
cooked: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> <a class="mention" href="/u/james_reed">@James_Reed</a></p>
<p>I’ve prepared a repro. It might be not minimal but it mocks the pipeline I use.<br>
Two files:</p>
<pre><code class="lang-auto"># network.py

import torch

class ConvModel(torch.nn.Module):
    def __init__(self):
        super(ConvModel, self).__init__()
        self.conv_stem = torch.nn.Conv2d(
            3, 5, 2, bias=True
        ).to(dtype=torch.float)

        self.bn1 = torch.nn.BatchNorm2d(5)
        self.act1 = torch.nn.ReLU()

    def forward(self, x):
        x = self.conv_stem(x)
        x = self.bn1(x)
        x = self.act1(x)
        return x
</code></pre>
<pre><code class="lang-auto"># actions.py

import torch
import io
import onnx
from torch.onnx import OperatorExportTypes


def ConvModel_decorate(cls):

    def fuse(self):
        torch.quantization.fuse_modules(
            self, 
            ['conv_stem', 'bn1', 'act1'], 
            inplace=True
        )

    cls.fuse = fuse
    return cls

def fuse_modules(module):
    module_output = module
    if callable(getattr(module_output, "fuse", None)):
        module_output.fuse()
    for name, child in module.named_children():
        new_child = fuse_modules(child)
        if new_child is not child:
            module_output.add_module(name, new_child)
    return module_output

def create_and_update_model():
    import network
    network.ConvModel = ConvModel_decorate(network.ConvModel)
    model = network.ConvModel()
    backend = 'qnnpack'
    model = fuse_modules(model)
    model.qconfig = torch.quantization.get_default_qat_qconfig(backend)
    torch.backends.quantized.engine = backend
    torch.quantization.prepare_qat(model, inplace=True)
    model.apply(torch.nn.intrinsic.qat.freeze_bn_stats)
    return model

def QAT(model):
    N = 100
    for idx in range(N):
        input_tensor = torch.rand(1, 3, 6, 6)
        model(input_tensor)
    return model

if __name__ == '__main__':
    model = create_and_update_model()
    model = QAT(model)
    torch.quantization.convert(model, inplace=True)
    
    model.eval()
    inputs = torch.rand(1, 3, 6, 6)
    # Export the model to ONNX
    with torch.no_grad():
        with io.BytesIO() as f:
            torch.onnx.export(
                model,
                inputs,
                f,
                opset_version=11,
                operator_export_type=OperatorExportTypes.ONNX_ATEN_FALLBACK,
                verbose=True,  # NOTE: uncomment this for debugging
                export_params=True,
            )
            onnx_model = onnx.load_from_string(f.getvalue())
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">(pytorch-gpu) root@ca7d6f51c4c7:~/some_detectron2# /root/anaconda2/envs/pytorch-gpu/bin/python /root/some_detectron2/min_repro/actions.py
/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/quantized/modules/utils.py:10: UserWarning: 0quantize_tensor_per_tensor_affine current rounding mode is not set to round-to-nearest-ties-to-even (FE_TONEAREST). This will cause accuracy issues in quantized models. (Triggered internally at  /opt/conda/conda-bld/pytorch_1589958443755/work/aten/src/ATen/native/quantized/affine_quantizer.cpp:25.)
  float(wt_scale), int(wt_zp), torch.qint8)
/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py:243: UserWarning: `add_node_names' can be set to True only when 'operator_export_type' is `ONNX`. Since 'operator_export_type' is not set to 'ONNX', `add_node_names` argument will be ignored.
  "`{}` argument will be ignored.".format(arg_name, arg_name))
/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py:243: UserWarning: `do_constant_folding' can be set to True only when 'operator_export_type' is `ONNX`. Since 'operator_export_type' is not set to 'ONNX', `do_constant_folding` argument will be ignored.
  "`{}` argument will be ignored.".format(arg_name, arg_name))
Traceback (most recent call last):
  File "/root/some_detectron2/min_repro/actions.py", line 65, in &lt;module&gt;
    export_params=True,
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/__init__.py", line 172, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 92, in export
    use_external_data_format=use_external_data_format)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 530, in _export
    fixed_batch_size=fixed_batch_size)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 366, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 319, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/jit/__init__.py", line 284, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 577, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/jit/__init__.py", line 372, in forward
    self._force_outplace,
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/jit/__init__.py", line 358, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 575, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 561, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/root/some_detectron2/min_repro/network.py", line 14, in forward
    x = self.conv_stem(x)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 575, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 561, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/intrinsic/quantized/modules/conv_relu.py", line 71, in forward
    input, self._packed_params, self.scale, self.zero_point)
RuntimeError: Tried to trace &lt;__torch__.torch.classes.quantized.Conv2dPackedParamsBase object at 0x564c572bd980&gt; but it is not part of the active trace. Modules that are called during a trace must be registered as submodules of the thing being traced.
</code></pre>
<pre><code class="lang-auto"></code></pre>

username: James_Reed
cooked: <p>Hi <a class="mention" href="/u/zetyquickly">@zetyquickly</a>,</p>
<p>First, your model does not run with the given inputs. The quantized model expects a quantized input, but <code>inputs</code> in your script is float-valued. QuantWrapper can be used to force quantization/dequantization for inputs/outputs of the model, respectively:</p>
<pre><code class="lang-auto">@@ -31,7 +31,7 @@ def fuse_modules(module):
 def create_and_update_model():
     import network
     network.ConvModel = ConvModel_decorate(network.ConvModel)
-    model = network.ConvModel()
+    model = torch.quantization.QuantWrapper(network.ConvModel())
</code></pre>
<p>Second, there’s a strange difference in behavior here between when ONNX is tracing the model and when we use the standalone TorchScript tracer. Tracing the model works fine when we use the standalone tracer. To workaround this issue, you can do this:</p>
<pre><code class="lang-auto">@@ -54,16 +54,19 @@ if __name__ == '__main__':
     
     model.eval()
     inputs = torch.rand(1, 3, 6, 6)
+    traced = torch.jit.trace(model, (inputs,))
+
     # Export the model to ONNX
     with torch.no_grad():
         with io.BytesIO() as f:
             torch.onnx.export(
-                model,
+                traced,
                 inputs,
                 f,
                 opset_version=11,
                 operator_export_type=OperatorExportTypes.ONNX_ATEN_FALLBACK,
                 verbose=True,  # NOTE: uncomment this for debugging
                 export_params=True,
+                example_outputs=traced(inputs)
             )
             onnx_model = onnx.load_from_string(f.getvalue())
</code></pre>
<p>We will investigate this difference in tracing</p>

username: zetyquickly
cooked: <p>Thank you <a class="mention" href="/u/james_reed">@James_Reed</a></p>
<p>I had knew that we should trace a model before passing it to ONNX <code>export</code>, <a href="https://discuss.pytorch.org/t/onnx-tried-to-trace-submodule-but-it-is-not-part-of-the-active-trace/80320/3">but</a>.<br>
For now I know it for sure</p>
<p>Could you please help me reveal what’s going on with traced model during exporting when I see the following:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "./tools/caffe2_converter.py", line 115, in &lt;module&gt;
    caffe2_model = export_caffe2_model(cfg, model, first_batch)
  File "/root/some_detectron2/detectron2/export/api.py", line 157, in export_caffe2_model
    return Caffe2Tracer(cfg, model, inputs).export_caffe2()
  File "/root/some_detectron2/detectron2/export/api.py", line 95, in export_caffe2
    predict_net, init_net = export_caffe2_detection_model(model, inputs)
  File "/root/some_detectron2/detectron2/export/caffe2_export.py", line 147, in export_caffe2_detection_model
    onnx_model = export_onnx_model(model, (tensor_inputs,))
  File "/root/some_detectron2/detectron2/export/caffe2_export.py", line 66, in export_onnx_model
    example_outputs=traced(inputs[0])
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/__init__.py", line 172, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 92, in export
    use_external_data_format=use_external_data_format)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 530, in _export
    fixed_batch_size=fixed_batch_size)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 384, in _model_to_graph
    fixed_batch_size=fixed_batch_size, params_dict=params_dict)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 171, in _optimize_graph
    torch._C._jit_pass_onnx_unpack_quantized_weights(graph, params_dict)
RuntimeError: quantized::conv2d_relu expected scale to be 7th input
</code></pre>
<p>How could it be that layer has lost its parameters?</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="zetyquickly" data-post="7" data-topic="82244">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zetyquickly/48/22489_2.png" class="avatar"> zetyquickly:</div>
<blockquote>
<p><code>expected scale to be 7th input</code></p>
</blockquote>
</aside>
<p>we did some refactor in this PR: <a href="https://github.com/pytorch/pytorch/pull/35923/files" class="inline-onebox" rel="noopener nofollow ugc">[quant] ConvPackedParams with TorchBind by jerryzh168 · Pull Request #35923 · pytorch/pytorch · GitHub</a> that removed some arguments from <code>quantized::conv2d</code> related ops. Does it work for <code>quantized::conv2d</code>?</p>

username: zetyquickly
cooked: <p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> thankyou, this allowed us to take a step forward!</p>
<p>Changed fusing configuration.<br>
Now all <code>QuantizedConvReLU2d</code> to <code>QuantizedConv2d + QuantizedReLU</code>. Don’t know whether it’s work but it <strong>produces</strong> a graph but it’s inconsistent. It causes an error that I’ve seen already.<br>
Something wrong with produced ONNX graph.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "./tools/caffe2_converter.py", line 115, in &lt;module&gt;
    caffe2_model = export_caffe2_model(cfg, model, first_batch)
  File "/root/some_detectron2/detectron2/export/api.py", line 157, in export_caffe2_model
    return Caffe2Tracer(cfg, model, inputs).export_caffe2()
  File "/root/some_detectron2/detectron2/export/api.py", line 95, in export_caffe2
    predict_net, init_net = export_caffe2_detection_model(model, inputs)
  File "/root/some_detectron2/detectron2/export/caffe2_export.py", line 147, in export_caffe2_detection_model
    onnx_model = export_onnx_model(model, (tensor_inputs,))
  File "/root/some_detectron2/detectron2/export/caffe2_export.py", line 66, in export_onnx_model
    example_outputs=traced(inputs[0])
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/__init__.py", line 172, in export
    custom_opsets, enable_onnx_checker, use_external_data_format)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 92, in export
    use_external_data_format=use_external_data_format)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/onnx/utils.py", line 557, in _export
    _check_onnx_proto(proto)
RuntimeError: Attribute 'kernel_shape' is expected to have field 'ints'

==&gt; Context: Bad node spec: input: "735" input: "98" input: "99" output: "743" op_type: "Conv" attribute { name: "dilations" ints: 1 ints: 1 type: INTS } attribute { name: "group" i: 1 type: INT } attribute { na
me: "kernel_shape" type: INTS } attribute { name: "pads" ints: 1 ints: 1 ints: 1 ints: 1 type: INTS } attribute { name: "strides" ints: 1 ints: 1 type: INTS }
</code></pre>
<p>This is very location where quantized output is dequantized and fed into <code>Conv</code> of <code>RPN</code>.<br>
Here are the bits of a graph output:</p>
<pre><code class="lang-auto">...
%98 : Long(1:1),
%99 : Long(1:1),
...
%620 : QUInt8(1:1638400, 64:25600, 128:200, 200:1) = _caffe2::Int8Relu[Y_scale=0.045047003775835037, Y_zero_point=119](%619), scope: __module._wrapped_model.backbone/__module._wrapped_model.backbone.p2_out/__module._wrapped_model.backbone.p2_out.2 # /root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/quantized/functional.py:381:0
...
%735 : Float(1:1638400, 64:25600, 128:200, 200:1) = _caffe2::Int8Dequantize(%620), scope: __module._wrapped_model.backbone/__module._wrapped_model.backbone.dequant_out # /root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/quantized/modules/__init__.py:74:0
...
%743 : Float(1:1638400, 64:25600, 128:200, 200:1) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=annotate(List[int], []), pads=[1, 1, 1, 1], strides=[1, 1]](%735, %98, %99), scope: __module._wrapped_model.proposal_generator/__module._wrapped_model.proposal_generator.rpn_head/__module._wrapped_model.proposal_generator.rpn_head.conv # /root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/conv.py:374:0
</code></pre>
<p>Important that eager mode model without quantization smoothly passes through this convertion pipeline and it is not data dependent.<br>
If you are interested this is detectron2 export to Caffe2 <a href="https://github.com/facebookresearch/detectron2/tree/master/detectron2/export" rel="nofollow noopener">pipeline</a></p>

username: supriyar
cooked: <aside class="quote no-group" data-username="zetyquickly" data-post="7" data-topic="82244">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zetyquickly/48/22489_2.png" class="avatar"> zetyquickly:</div>
<blockquote>
<p><code>RuntimeError: quantized::conv2d_relu expected scale to be 7th input</code></p>
</blockquote>
</aside>
<p>Please re-try with pytorch nightly build, we recently fixed this so you shouldn’t be seeing this error anymore.</p>
<aside class="quote no-group" data-username="zetyquickly" data-post="9" data-topic="82244">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zetyquickly/48/22489_2.png" class="avatar"> zetyquickly:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: Attribute 'kernel_shape' is expected to have field 'ints'
</code></pre>
</blockquote>
</aside>
<p>Seems like the conv layer is not quantized so it produces <code>onnx::Conv</code> as opposed to the <code>_caffe2::Int8Conv</code> operator. Currently the onnx export path to caffe2 does not support partially quantized model, so it expects the entire pytorch model to be able to get quantized.</p>

username: zetyquickly
cooked: <p>Thank you very much <a class="mention" href="/u/supriyar">@supriyar</a>,</p>
<p>I am still eager to find a solution. I’ve tried your assumptions, installed fresh build and tried again. Re-run QAT on model (just to make sure) and exporting process.</p>
<p>Now it says that <code>MaxPool</code> cannot be created.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "./tools/caffe2_converter.py", line 114, in &lt;module&gt;
    caffe2_model = export_caffe2_model(cfg, model, first_batch)
  File "/root/some_detectron2/detectron2/export/api.py", line 157, in export_caffe2_model
    return Caffe2Tracer(cfg, model, inputs).export_caffe2()
  File "/root/some_detectron2/detectron2/export/api.py", line 95, in export_caffe2
    predict_net, init_net = export_caffe2_detection_model(model, inputs)
  File "/root/some_detectron2/detectron2/export/caffe2_export.py", line 151, in export_caffe2_detection_model
    onnx_model = export_onnx_model(model, (tensor_inputs,))
  File "/root/some_detectron2/detectron2/export/caffe2_export.py", line 53, in export_onnx_model
    traced = torch.jit.trace(model, inputs, strict=False)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/jit/__init__.py", line 900, in trace
    check_tolerance, strict, _force_outplace, _module_class)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/jit/__init__.py", line 1054, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, strict, _force_outplace)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 575, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 561, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/root/some_detectron2/detectron2/export/caffe2_modeling.py", line 319, in forward
    features = self._wrapped_model.backbone(images.tensor)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 575, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 561, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/root/DensePose_ADASE/densepose/modeling/quantize.py", line 205, in new_forward
    return {"p2": p2_out, "p3": p3_out, "p4": p4_out, "p5": p5_out, "p6": self.top_block(p5_out)[0]}
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 575, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 561, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/root/some_detectron2/detectron2/modeling/backbone/fpn.py", line 177, in forward
    return [F.max_pool2d(x, kernel_size=1, stride=2, padding=0)]
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/_jit_internal.py", line 210, in fn
    return if_false(*args, **kwargs)
  File "/root/anaconda2/envs/pytorch-gpu/lib/python3.7/site-packages/torch/nn/functional.py", line 576, in _max_pool2d
    input, kernel_size, stride, padding, dilation, ceil_mode)
RuntimeError: createStatus == pytorch_qnnp_status_success INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1590649859799/work/aten/src/ATen/native/quantized/cpu/qpool.cpp":313, please report a bug to PyTorch. failed to create QNNPACK MaxPool operator

</code></pre>
<p><a href="https://github.com/pytorch/pytorch/blob/352731bd6edefbec707fbef3662f4a60934b8443/aten/src/ATen/native/quantized/cpu/qpool.cpp#L330" rel="nofollow noopener">Path to the source</a></p>
<p>Looks weird, why didn’t it happen earlier?</p>
<p><strong>UPD</strong>: It looks like nested <code>F.max_pool2d</code> won’t quantize. Test showed that it works with <code>float32</code> after <code>convert</code></p>

username: zetyquickly
cooked: <aside class="quote no-group" data-username="supriyar" data-post="10" data-topic="82244">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/db5fbb/48.png" class="avatar"> supriyar:</div>
<blockquote>
<p>Seems like the conv layer is not quantized so it produces <code>onnx::Conv</code> as opposed to the <code>_caffe2::Int8Conv</code> operator.</p>
</blockquote>
</aside>
<p>Is it possible to find a workaround for now? Do I understand correctly, that it is impossible to have a network with quantize dequantize during inference in Caffe2 export?</p>
<p><strong>UPD:</strong> what if we just make all Convs are quantized for ONNX.export not fail</p>

username: supriyar
cooked: <aside class="quote no-group" data-username="zetyquickly" data-post="12" data-topic="82244">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zetyquickly/48/22489_2.png" class="avatar"> zetyquickly:</div>
<blockquote>
<p><strong>UPD:</strong> what if we just make all Convs are quantized for ONNX.export not fail</p>
</blockquote>
</aside>
<p>If all convs in the network are quantized it should work and you will see <code>_caffe2::Int8Conv</code> ops in the converted network.</p>
<aside class="quote no-group" data-username="zetyquickly" data-post="12" data-topic="82244">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zetyquickly/48/22489_2.png" class="avatar"> zetyquickly:</div>
<blockquote>
<p>Do I understand correctly, that it is impossible to have a network with quantize dequantize during inference in Caffe2 export?</p>
</blockquote>
</aside>
<p>You would have quantize dequantize at the start and end of the network. Which implies all the network ops are quantized.</p>

username: zetyquickly
cooked: <p>Thank you <a class="mention" href="/u/supriyar">@supriyar</a> , I see</p>
<p>I have another question about non-quant operations in network</p>
<p>What do you think if we register C10 export like it is done <a href="https://github.com/pytorch/pytorch/blob/master/caffe2/operators/roi_align_op.cc#L302" rel="nofollow noopener">here</a><br>
would it be possible to patch non-quantized operators from <code>torch.nn.ConvTranspose2d</code> to <code>torch.ops._caffe2.ConvTranspose2d</code> to use them as is? Or is it better to implement quantized version of <code>nn.ConvTranspose2d</code>?</p>
<p><a class="mention" href="/u/jerryzh168">@jerryzh168</a> I could implement a PR of such functionality if it is valid</p>

username: jerryzh168
cooked: <p>I think we are already working on quantized version of conv2d transpose, cc <a class="mention" href="/u/zafar">@Zafar</a></p>

username: blueskywwc
cooked: <p>Hello, can you now export the quantized model to Caffe2, and then export Caffe2 to ncnn? Thank you!</p>

username: zetyquickly
cooked: <p>Hello, <a class="mention" href="/u/blueskywwc">@blueskywwc</a></p>
<p>First of all I haven’t managed to export quantized network to Caffe2. I do not know is it possible to export it to ncnn. Just a suggestion, maybe it is better to export model to ONNX and than to ncnn</p>

username: blueskywwc
cooked: <p>hello,<a class="mention" href="/u/jerryzh168">@jerryzh168</a>   When will it be possible to support the conversion of the quantified model to onnx, I hope there is a reference time, thank you！</p>

username: jerryzh168
cooked: <p>we are not working on onnx conversions, feel free to submit PRs to add the support.</p>