Repository: apache/tvm

Issue:
[Bug] [ONNX] QLinearMatMul works with binary output only
I'm using [`from_onnx`](https://github.com/apache/tvm/blob/44fe7ef816565f43380c50e0b43fd626fad9d029/python/tvm/relay/frontend/onnx.py#L5063) to convert my model 
[dense_h32_w32_c3_sNone_pNone_kNone.pt_quant.onnx.zip](https://github.com/apache/tvm/files/7851265/dense_h32_w32_c3_sNone_pNone_kNone.pt_quant.onnx.zip) into tvm relay. It looks like `b_scale` in [`QLinearMatMul`](https://github.com/apache/tvm/blob/44fe7ef816565f43380c50e0b43fd626fad9d029/python/tvm/relay/frontend/onnx.py#L3760) expects a scalar scale not a vector of scales. 
I'm not sure if this is a problem with onnx creating a vector for the weights, or if QLinearMatMul should support scale vectors.

Info about the model:
1. Input: (batch, channel, height, width) -> (1, 3, 32, 32)
2. Global Average pool: (batch, channel, 1, 1) -> (1, 3, 1, 1)
3. Reshape: (batch, channel) -> (1, 3)
4. Dense Layer: (batch, channel, 20) -> (3, 20) 
    > Dense Layer is where the error occurs 

> You might notice when you debug `b_scale`'s shape that it has shape=20. 

### Expected behavior

No errors converting my model [dense_h32_w32_c3_sNone_pNone_kNone.pt_quant.onnx.zip](https://github.com/apache/tvm/files/7851265/dense_h32_w32_c3_sNone_pNone_kNone.pt_quant.onnx.zip) into tvm relay using  [`from_onnx`](https://github.com/apache/tvm/blob/44fe7ef816565f43380c50e0b43fd626fad9d029/python/tvm/relay/frontend/onnx.py#L5063).


### Actual behavior

```
assert num_elem == 1, "Cannot squeeze tensor shape {} to scalar form.".format(x_shape)
E       AssertionError: Cannot squeeze tensor shape (20,) to scalar form.
```


Discussion:
Code to reproduce:
```
import onnx
from tvm import relay
m = onnx.load("dense_h32_w32_c3_sNone_pNone_kNone.pt_quant.onnx")
relay.frontend.from_onnx(m)
```
Error:
```
AssertionError: Cannot squeeze tensor shape (20,) to scalar form.
```

Hello,
@cconvey The current implementation of QLinearMatMul in TVM may not support scale vectors. The behaviour for now is parsing the input scale as scalar. See PR https://github.com/apache/tvm/pull/8952. 
Thanks @alnah005 @FZYUAN-1 for creating this Issue and pinging me!

As far as I can tell, the input model is valid ONNX, and you're bumping into a limitation in the current version of the ONNX->TVM importer code.

I'm afraid I don't have the bandwidth to (personally) fix this a the moment, but if someone else wants to create a PR I'd be very happy to review it.

> Thanks @alnah005 @FZYUAN-1 for creating this Issue and pinging me!
> 
> As far as I can tell, the input model is valid ONNX, and you're bumping into a limitation in the current version of the ONNX->TVM importer code.
> 
> I'm afraid I don't have the bandwidth to (personally) fix this a the moment, but if someone else wants to create a PR I'd be very happy to review it.

Thanks for your reply!
I will fix this in next several days. @alnah005 @denise-k 

Resolving patch:
diff --git a/python/tvm/relay/frontend/onnx.py b/python/tvm/relay/frontend/onnx.py
--- a/python/tvm/relay/frontend/onnx.py
+++ b/python/tvm/relay/frontend/onnx.py
@@ -3804,18 +3804,16 @@ def _impl_v10(cls, inputs, attr, params):
         #
         # This function attempts to present 'x' in a form that meets both of those
         # requirements.
-        def try_resolve_to_const_scalar(x, dtype_override=None):
+        def try_resolve_to_const(x, dtype_override=None):
             x2 = try_resolve_var_to_const(x, params)
-            x3 = ensure_scalar_shape(x2)
-
+            num_elem = np.prod(infer_shape(x))
+            if num_elem == 1:
+                x2 = ensure_scalar_shape(x2)
             x_dtype = infer_type(x).checked_type.dtype
             if (dtype_override is not None) and (dtype_override != x_dtype):
-                x4 = _op.cast(x3, dtype_override)
-            else:
-                x4 = x3
-
-            x5 = fold_constant(x4)
-            return x5
+                x2 = _op.cast(x2, dtype_override)
+            x3 = fold_constant(x2)
+            return x3
 
         # Unpack the inputs and obtain some type info...
         a, a_scale, a_zp, b, b_scale, b_zp, y_scale, y_zp = inputs
@@ -3855,14 +3853,14 @@ def try_resolve_to_const_scalar(x, dtype_override=None):
         )
 
         # _qnn.op.dense requires the zero-point values to have dtype int32.
-        a_scale_scalar = try_resolve_to_const_scalar(a_scale)
-        a_zp_scalar = try_resolve_to_const_scalar(a_zp, "int32")
+        a_scale_scalar = try_resolve_to_const(a_scale)
+        a_zp_scalar = try_resolve_to_const(a_zp, "int32")
 
-        b_scale_scalar = try_resolve_to_const_scalar(b_scale)
-        b_zp_scalar = try_resolve_to_const_scalar(b_zp, "int32")
+        b_scale_scalar = try_resolve_to_const(b_scale)
+        b_zp_scalar = try_resolve_to_const(b_zp, "int32")
 
-        y_scale_scalar = try_resolve_to_const_scalar(y_scale)
-        y_zp_scalar = try_resolve_to_const_scalar(y_zp, "int32")
+        y_scale_scalar = try_resolve_to_const(y_scale)
+        y_zp_scalar = try_resolve_to_const(y_zp, "int32")
 
         # TODO: Confirm that we're using 'num_hidden_units' correctly / as intended with
         # the '_qnn.op.dense' instance below.


Test patch: