Repository: apache/tvm

Issue:
[TVM][RELAY] Incomplete quantize annotate
I changed the `compile` function in [deploy ssd gluoncv](https://docs.tvm.ai/tutorials/frontend/deploy_ssd_gluoncv.html#sphx-glr-tutorials-frontend-deploy-ssd-gluoncv-py) tutorial to have default quantization pass

```
def compile(target):
    net, params = relay.frontend.from_mxnet(block, {"data": dshape})
    with tvm.relay.quantize.qconfig():
        qgraph = relay.quantize.quantize(net, params)

    with relay.build_config(opt_level=3):
        graph, lib, params = relay.build(qgraph, target)

    return graph, lib, params
```

and then

```
for target, ctx in target_list:
    if target == "cuda":
        print("GPU not supported yet, skip.")
        continue
    graph, lib, params = compile(target)
    class_IDs, scores, bounding_boxs = run(graph, lib, params, ctx)
```

failed with

```
TVMError                                  Traceback (most recent call last)
<ipython-input-23-f6d8f7869cc8> in <module>
      3         print("GPU not supported yet, skip.")
      4         continue
----> 5     graph, lib, params = compile(target)
      6     class_IDs, scores, bounding_boxs = run(graph, lib, params, ctx)

<ipython-input-22-f3b5a820bcdd> in compile(target)
     17                      debug_enabled_ops=None):
     18 
---> 19         qgraph = relay.quantize.quantize(net, params)
     20 
     21     with relay.build_config(opt_level=3):

~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/relay/quantize/quantize.py in quantize(graph, params, dataset)
    284         graph = _build.optimize(graph, params=params)
    285 
--> 286     graph = annotate(graph)
    287     graph = calibrate(graph, dataset)
    288     graph = realize(graph)

~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/relay/quantize/quantize.py in annotate(graph)
    175     """
    176     _set_conv_counter(0)  # reset counter
--> 177     return _quantize.annotate(graph)
    178 
    179 

tvm/_ffi/_cython/./function.pxi in tvm._ffi._cy3.core.FunctionBase.__call__()

tvm/_ffi/_cython/./function.pxi in tvm._ffi._cy3.core.FuncCall()

tvm/_ffi/_cython/./function.pxi in tvm._ffi._cy3.core.FuncCall3()

tvm/_ffi/_cython/./base.pxi in tvm._ffi._cy3.core.CALL()

TVMError: TVMCall CFunc Error:
Traceback (most recent call last):
  File "tvm/_ffi/_cython/./function.pxi", line 38, in tvm._ffi._cy3.core.tvm_callback
  File "/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/relay/quantize/_annotate.py", line 96, in frewrite_with_guard
    return func(ref_call, new_args, ctx)
  File "/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/relay/quantize/_annotate.py", line 176, in multiply_rewrite
    raise ValueError
ValueError
```

1. How to fix the compilation for quantization?
2. Fix error handling of [`multiply_rewrite`](https://github.com/dmlc/tvm/blob/master/python/tvm/relay/quantize/_annotate.py#L176).

* [TVM master](https://github.com/dmlc/tvm/tree/c0a5a9be2fdc6afa8020a3a9732721251c28cb97)
* mxnet-cu9.0==1.4
* gluoncv==0.4 pre release

cc @ZihengJiang @vinx13 


Discussion:
@ZihengJiang we missed the case where lhs_kind is QInput and rhs_kind is None in multiply_rewrite
@ZihengJiang any good news on this?
Fix: #2848 
@vinx13 thanks! now there's this issue quantizing "yolo3_mobilenet1.0_coco" loaded from gluoncv:

```
TVMError  Traceback (most recent call last)
<ipython-input-23-f6d8f7869cc8> in <module>
      3         print("GPU not supported yet, skip.")
      4         continue
----> 5     graph, lib, params = compile(target)
      6     class_IDs, scores, bounding_boxs = run(graph, lib, params, ctx)

<ipython-input-21-5901f3423aef> in compile(target, quantize)
      4     if quantize:
      5         with tvm.relay.quantize.qconfig(skip_k_conv=0):
----> 6             net = relay.quantize.quantize(net, params)
      7 
      8     with relay.build_config(opt_level=3):

~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/relay/quantize/quantize.py in quantize(graph, params, dataset)
    284         graph = _build.optimize(graph, params=params)
    285 
--> 286     graph = annotate(graph)
    287     graph = calibrate(graph, dataset)
    288     graph = realize(graph)

~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/relay/quantize/quantize.py in annotate(graph)
    175     """
    176     _set_conv_counter(0)  # reset counter
--> 177     return _quantize.annotate(graph)
    178 
    179 

tvm/_ffi/_cython/./function.pxi in tvm._ffi._cy3.core.FunctionBase.__call__()

tvm/_ffi/_cython/./function.pxi in tvm._ffi._cy3.core.FuncCall()

tvm/_ffi/_cython/./function.pxi in tvm._ffi._cy3.core.FuncCall3()

tvm/_ffi/_cython/./base.pxi in tvm._ffi._cy3.core.CALL()

TVMError: Traceback (most recent call last):
  [bt] (8) ~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/libtvm.so(+0xa9759b) [0x7f4a605ce59b]
  [bt] (7) ~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/libtvm.so(+0xc84471) [0x7f4a607bb471]
  [bt] (6) ~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/libtvm.so(+0xc85953) [0x7f4a607bc953]
  [bt] (5) ~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/libtvm.so(+0xa8d837) [0x7f4a605c4837]
  [bt] (4) ~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/libtvm.so(+0xa92d6d) [0x7f4a605c9d6d]
  [bt] (3) ~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/libtvm.so(+0xa94cf9) [0x7f4a605cbcf9]
  [bt] (2) ~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/libtvm.so(+0xa9759b) [0x7f4a605ce59b]
  [bt] (1) ~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/libtvm.so(+0xc847f4) [0x7f4a607bb7f4]
  [bt] (0) ~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/libtvm.so(+0xf262b4) [0x7f4a60a5d2b4]
  File "tvm/_ffi/_cython/./function.pxi", line 38, in tvm._ffi._cy3.core.tvm_callback
  File "~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/relay/quantize/_annotate.py", line 96, in frewrite_with_guard
    return func(ref_call, new_args, ctx)
  File "~/anaconda3/lib/python3.7/site-packages/tvm-0.6.dev0-py3.7-linux-x86_64.egg/tvm/relay/quantize/_annotate.py", line 258, in concatenate_rewrite
    assert k is None
TVMError: AssertionError
```
@ehsanmok I also met this before. We need to quantize all fields in case that there are float fields in concat. I will make a pr soon.

Resolving patch:
diff --git a/python/tvm/relay/quantize/_annotate.py b/python/tvm/relay/quantize/_annotate.py
--- a/python/tvm/relay/quantize/_annotate.py
+++ b/python/tvm/relay/quantize/_annotate.py
@@ -253,11 +253,10 @@ def concatenate_rewrite(ref_call, new_args, ctx):
 
     # make sure the inputs of concatenate are all normal
     # expression or annotate expression
-    if kind_list[0] is None:
-        for k in kind_list:
-            assert k is None
+    if all([k is None for k in kind_list]):
         return None
-    for k in kind_list:
-        assert k is not None
+    for i, k in enumerate(kind_list):
+        if k is None:
+            expr_list[i] = attach_simulated_quantize(expr_list[i], QAnnotateKind.ACTIVATION)
     expr = _forward_op(ref_call, [_expr.Tuple(expr_list)])
     return QAnnotateExpr(expr, QAnnotateKind.ACTIVATION)


Test patch: