Cannot export quantized model to onnx: cannot call qscheme on UnknownQuantizer
Labels: module: onnx, oncall: quantization, triaged

### 🐛 Describe the bug

I am trying to convert a quantized model to onnx. It fails with the error as shown below. This issue may be related to  #100654 because a pretrained pytorch model is quantized and converted to onnx without error.

[Code](https://gist.github.com/chinmayjog13/c66226681d538382014a9140e0bc8de1)

**Output**- contains quantized model graph and the error message

```
(v_pytorch2) chinmay@mcfluid03:~/tf_face_recognition$ python test_quantized_model.py 
/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/ao/quantization/observer.py:214: UserWarning: Please use quant_min and quant_max to specify the range for observers.                     reduce_range will be deprecated in a future release of PyTorch.
  warnings.warn(
prepared model for calibration
num imgs for calibration: 1000
batch 0 done
/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/ao/quantization/utils.py:302: UserWarning: must run observer before calling calculate_qparams. Returning default values.
  warnings.warn(
GraphModule(
  (conv1): QuantizedConv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), scale=0.04243697598576546, zero_point=64, padding=(1, 1))
  (prelu): QuantizedPReLU()
  (layer1): Module(
    (0): Module(
      (conv1): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=0.048839956521987915, zero_point=70, padding=(1, 1))
      (prelu): QuantizedPReLU()
      (conv2): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), scale=0.02982931397855282, zero_point=56, padding=(1, 1))
      (downsample): Module(
        (0): QuantizedConv2d(64, 64, kernel_size=(1, 1), stride=(2, 2), scale=0.026763780042529106, zero_point=72)
      )
    )
    (1): Module(
      (conv1): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=0.03681090474128723, zero_point=88, padding=(1, 1))
      (prelu): QuantizedPReLU()
      (conv2): QuantizedConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), scale=0.030850104987621307, zero_point=61, padding=(1, 1))
    )
  )
  (layer2): Module(
    (0): Module(
      (conv1): QuantizedConv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), scale=0.028876036405563354, zero_point=68, padding=(1, 1))
      (prelu): QuantizedPReLU()
      (conv2): QuantizedConv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), scale=0.016839243471622467, zero_point=68, padding=(1, 1))
      (downsample): Module(
        (0): QuantizedConv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), scale=0.0114758824929595, zero_point=64)
      )
    )
    (1): Module(
      (conv1): QuantizedConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), scale=0.02233685925602913, zero_point=82, padding=(1, 1))
      (prelu): QuantizedPReLU()
      (conv2): QuantizedConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), scale=0.023190662264823914, zero_point=66, padding=(1, 1))
    )
  )
  (layer3): Module(
    (0): Module(
      (conv1): QuantizedConv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), scale=0.02156665176153183, zero_point=67, padding=(1, 1))
      (prelu): QuantizedPReLU()
      (conv2): QuantizedConv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), scale=0.01482717040926218, zero_point=64, padding=(1, 1))
      (downsample): Module(
        (0): QuantizedConv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), scale=0.0048522488214075565, zero_point=65)
      )
    )
    (1): Module(
      (conv1): QuantizedConv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), scale=0.01981436274945736, zero_point=80, padding=(1, 1))
      (prelu): QuantizedPReLU()
      (conv2): QuantizedConv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), scale=0.014433014206588268, zero_point=63, padding=(1, 1))
    )
  )
  (layer4): Module(
    (0): Module(
      (conv1): QuantizedConv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), scale=0.019120512530207634, zero_point=79, padding=(1, 1))
      (prelu): QuantizedPReLU()
      (conv2): QuantizedConv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), scale=0.011832070536911488, zero_point=70, padding=(1, 1))
      (downsample): Module(
        (0): QuantizedConv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), scale=0.002516211476176977, zero_point=61)
      )
    )
    (1): Module(
      (conv1): QuantizedConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), scale=0.03342185169458389, zero_point=87, padding=(1, 1))
      (prelu): QuantizedPReLU()
      (conv2): QuantizedConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), scale=0.017271075397729874, zero_point=63, padding=(1, 1))
    )
  )
  (dropout): QuantizedDropout(p=0, inplace=True)
  (fc): QuantizedLinear(in_features=25088, out_features=512, scale=0.06431952863931656, zero_point=63, qscheme=torch.per_channel_affine)
)



def forward(self, x):
    conv1_input_scale_0 = self.conv1_input_scale_0
    conv1_input_zero_point_0 = self.conv1_input_zero_point_0
    quantize_per_tensor = torch.quantize_per_tensor(x, conv1_input_scale_0, conv1_input_zero_point_0, torch.quint8);  x = conv1_input_scale_0 = conv1_input_zero_point_0 = None
    conv1 = self.conv1(quantize_per_tensor);  quantize_per_tensor = None
    prelu = self.prelu(conv1);  conv1 = None
    layer1_0_conv1 = getattr(self.layer1, "0").conv1(prelu)
    layer1_0_prelu = getattr(self.layer1, "0").prelu(layer1_0_conv1);  layer1_0_conv1 = None
    layer1_0_conv2 = getattr(self.layer1, "0").conv2(layer1_0_prelu);  layer1_0_prelu = None
    layer1_0_downsample_0 = getattr(getattr(self.layer1, "0").downsample, "0")(prelu);  prelu = None
    layer1_0_scale_0 = self.layer1_0_scale_0
    layer1_0_zero_point_0 = self.layer1_0_zero_point_0
    add_8 = torch.ops.quantized.add(layer1_0_conv2, layer1_0_downsample_0, layer1_0_scale_0, layer1_0_zero_point_0);  layer1_0_conv2 = layer1_0_downsample_0 = layer1_0_scale_0 = layer1_0_zero_point_0 = None
    layer1_1_conv1 = getattr(self.layer1, "1").conv1(add_8)
    layer1_1_prelu = getattr(self.layer1, "1").prelu(layer1_1_conv1);  layer1_1_conv1 = None
    layer1_1_conv2 = getattr(self.layer1, "1").conv2(layer1_1_prelu);  layer1_1_prelu = None
    layer1_1_scale_0 = self.layer1_1_scale_0
    layer1_1_zero_point_0 = self.layer1_1_zero_point_0
    add_9 = torch.ops.quantized.add(layer1_1_conv2, add_8, layer1_1_scale_0, layer1_1_zero_point_0);  layer1_1_conv2 = add_8 = layer1_1_scale_0 = layer1_1_zero_point_0 = None
    layer2_0_conv1 = getattr(self.layer2, "0").conv1(add_9)
    layer2_0_prelu = getattr(self.layer2, "0").prelu(layer2_0_conv1);  layer2_0_conv1 = None
    layer2_0_conv2 = getattr(self.layer2, "0").conv2(layer2_0_prelu);  layer2_0_prelu = None
    layer2_0_downsample_0 = getattr(getattr(self.layer2, "0").downsample, "0")(add_9);  add_9 = None
    layer2_0_scale_0 = self.layer2_0_scale_0
    layer2_0_zero_point_0 = self.layer2_0_zero_point_0
    add_10 = torch.ops.quantized.add(layer2_0_conv2, layer2_0_downsample_0, layer2_0_scale_0, layer2_0_zero_point_0);  layer2_0_conv2 = layer2_0_downsample_0 = layer2_0_scale_0 = layer2_0_zero_point_0 = None
    layer2_1_conv1 = getattr(self.layer2, "1").conv1(add_10)
    layer2_1_prelu = getattr(self.layer2, "1").prelu(layer2_1_conv1);  layer2_1_conv1 = None
    layer2_1_conv2 = getattr(self.layer2, "1").conv2(layer2_1_prelu);  layer2_1_prelu = None
    layer2_1_scale_0 = self.layer2_1_scale_0
    layer2_1_zero_point_0 = self.layer2_1_zero_point_0
    add_11 = torch.ops.quantized.add(layer2_1_conv2, add_10, layer2_1_scale_0, layer2_1_zero_point_0);  layer2_1_conv2 = add_10 = layer2_1_scale_0 = layer2_1_zero_point_0 = None
    layer3_0_conv1 = getattr(self.layer3, "0").conv1(add_11)
    layer3_0_prelu = getattr(self.layer3, "0").prelu(layer3_0_conv1);  layer3_0_conv1 = None
    layer3_0_conv2 = getattr(self.layer3, "0").conv2(layer3_0_prelu);  layer3_0_prelu = None
    layer3_0_downsample_0 = getattr(getattr(self.layer3, "0").downsample, "0")(add_11);  add_11 = None
    layer3_0_scale_0 = self.layer3_0_scale_0
    layer3_0_zero_point_0 = self.layer3_0_zero_point_0
    add_12 = torch.ops.quantized.add(layer3_0_conv2, layer3_0_downsample_0, layer3_0_scale_0, layer3_0_zero_point_0);  layer3_0_conv2 = layer3_0_downsample_0 = layer3_0_scale_0 = layer3_0_zero_point_0 = None
    layer3_1_conv1 = getattr(self.layer3, "1").conv1(add_12)
    layer3_1_prelu = getattr(self.layer3, "1").prelu(layer3_1_conv1);  layer3_1_conv1 = None
    layer3_1_conv2 = getattr(self.layer3, "1").conv2(layer3_1_prelu);  layer3_1_prelu = None
    layer3_1_scale_0 = self.layer3_1_scale_0
    layer3_1_zero_point_0 = self.layer3_1_zero_point_0
    add_13 = torch.ops.quantized.add(layer3_1_conv2, add_12, layer3_1_scale_0, layer3_1_zero_point_0);  layer3_1_conv2 = add_12 = layer3_1_scale_0 = layer3_1_zero_point_0 = None
    layer4_0_conv1 = getattr(self.layer4, "0").conv1(add_13)
    layer4_0_prelu = getattr(self.layer4, "0").prelu(layer4_0_conv1);  layer4_0_conv1 = None
    layer4_0_conv2 = getattr(self.layer4, "0").conv2(layer4_0_prelu);  layer4_0_prelu = None
    layer4_0_downsample_0 = getattr(getattr(self.layer4, "0").downsample, "0")(add_13);  add_13 = None
    layer4_0_scale_0 = self.layer4_0_scale_0
    layer4_0_zero_point_0 = self.layer4_0_zero_point_0
    add_14 = torch.ops.quantized.add(layer4_0_conv2, layer4_0_downsample_0, layer4_0_scale_0, layer4_0_zero_point_0);  layer4_0_conv2 = layer4_0_downsample_0 = layer4_0_scale_0 = layer4_0_zero_point_0 = None
    layer4_1_conv1 = getattr(self.layer4, "1").conv1(add_14)
    layer4_1_prelu = getattr(self.layer4, "1").prelu(layer4_1_conv1);  layer4_1_conv1 = None
    layer4_1_conv2 = getattr(self.layer4, "1").conv2(layer4_1_prelu);  layer4_1_prelu = None
    layer4_1_scale_0 = self.layer4_1_scale_0
    layer4_1_zero_point_0 = self.layer4_1_zero_point_0
    add_15 = torch.ops.quantized.add(layer4_1_conv2, add_14, layer4_1_scale_0, layer4_1_zero_point_0);  layer4_1_conv2 = add_14 = layer4_1_scale_0 = layer4_1_zero_point_0 = None
    flatten = torch.flatten(add_15, 1);  add_15 = None
    dropout = self.dropout(flatten);  flatten = None
    fc = self.fc(dropout);  dropout = None
    dequantize_41 = fc.dequantize();  fc = None
    return dequantize_41
    
# To see more debug info, please use `graph_module.print_readable()`
Converted model
================ Diagnostic Run torch.onnx.export version 2.0.0 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Traceback (most recent call last):
  File "test_quantized_model.py", line 193, in <module>
    torch.onnx.export(quantized_model, img, output, opset_version=18,
  File "/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/onnx/utils.py", line 506, in export
    _export(
  File "/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/onnx/utils.py", line 1548, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/onnx/utils.py", line 1117, in _model_to_graph
    graph = _optimize_graph(
  File "/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/onnx/utils.py", line 665, in _optimize_graph
    graph = _C._jit_pass_onnx(graph, operator_export_type)
  File "/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/onnx/utils.py", line 1891, in _run_symbolic_function
    return symbolic_fn(graph_context, *inputs, **attrs)
  File "/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py", line 6829, in prim_constant
    return g.op("Constant", value_t=symbolic_helper._node_get(node, "value"))
  File "/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/onnx/_internal/jit_utils.py", line 86, in op
    return _add_op(self, opname, *raw_args, outputs=outputs, **kwargs)
  File "/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/onnx/_internal/jit_utils.py", line 245, in _add_op
    node = _create_node(
  File "/home/chinmay/anaconda3/envs/v_pytorch2/lib/python3.8/site-packages/torch/onnx/_internal/jit_utils.py", line 306, in _create_node
    _C._jit_pass_onnx_node_shape_type_inference(node, params_dict, opset_version)
RuntimeError: false INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1678402379298/work/aten/src/ATen/quantized/Quantizer.cpp":444, please report a bug to PyTorch. cannot call qscheme on UnknownQuantizer
```

### Versions

Collecting environment information...
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.16 (default, Mar  2 2023, 03:21:46)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe
GPU 2: NVIDIA A100 80GB PCIe
GPU 3: NVIDIA A100 80GB PCIe
GPU 4: NVIDIA A100 80GB PCIe
GPU 5: NVIDIA A100 80GB PCIe
GPU 6: NVIDIA A100 80GB PCIe
GPU 7: NVIDIA A100 80GB PCIe

Nvidia driver version: 515.105.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.5.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.5.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   43 bits physical, 48 bits virtual
CPU(s):                          64
On-line CPU(s) list:             0-63
Thread(s) per core:              1
Core(s) per socket:              32
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           49
Model name:                      AMD EPYC 7502 32-Core Processor
Stepping:                        0
Frequency boost:                 enabled
CPU MHz:                         1498.611
CPU max MHz:                     3354.4919
CPU min MHz:                     1500.0000
BogoMIPS:                        5000.33
Virtualization:                  AMD-V
L1d cache:                       2 MiB
L1i cache:                       2 MiB
L2 cache:                        32 MiB
L3 cache:                        256 MiB
NUMA node0 CPU(s):               0-31
NUMA node1 CPU(s):               32-63
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca

Versions of relevant libraries:
[pip3] numpy==1.24.3
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.0
[pip3] torchvision==0.15.0
[pip3] triton==2.0.0
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py38h7f8727e_0  
[conda] mkl_fft                   1.3.1            py38hd3c417c_0  
[conda] mkl_random                1.2.2            py38h51133e4_0  
[conda] numpy                     1.24.3           py38h14f4228_0  
[conda] numpy-base                1.24.3           py38h31eccc5_0  
[conda] pytorch                   2.0.0           py3.8_cuda11.7_cudnn8.5.0_0    pytorch
[conda] pytorch-cuda              11.7                 h778d358_3    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                2.0.0                py38_cu117    pytorch
[conda] torchtriton               2.0.0                      py38    pytorch
[conda] torchvision               0.15.0               py38_cu117    pytorch

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel