Torchscript model Runtime Error after quantization
Labels: oncall: jit, oncall: quantization, triaged

### 🐛 Describe the bug

When applying quantization to a CNN - ReLu - Batch Norm model, and converting it to Torchscript format, the inference fails.

Reproductible code:

```python
import torch.nn as nn
import torch
from torch.quantization import get_default_qconfig
from torch.quantization.quantize_fx import prepare_fx, convert_fx

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(in_channels=1, 
                    out_channels=64, 
                    kernel_size=(8, 20), 
                    stride=(1, 1),
                    padding=0)
        self.relu = nn.ReLU()
        self.bn = nn.BatchNorm2d(num_features=64)

    def forward(self, x):
        # x = [batch_size, channel=1, height=10, width=20]
        x = self.conv(x)
        x = self.relu(x)
        x = self.bn(x)
        return x
    
model = MyModel()

## Quantize model (post-training Static quantization FX Graph mode)
torch.backends.quantized.engine = "qnnpack"

model.to("cpu")
model.eval()

prepared_model = prepare_fx(model, {"": get_default_qconfig("qnnpack")})

with torch.no_grad(): # calibrate using random data
    data = torch.rand((7,1,10,20))
    prepared_model(data)
model_quantized = convert_fx(prepared_model)

## Prediction using scripted model an error
model_quantized_scripted = torch.jit.script(model_quantized) # script model
model_quantized_scripted(torch.rand((7,1,10,20))) # raises an error
```

The error raised:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [1], in <module>
     39 ## Prediction using scripted model raises and error
     40 model_quantized_scripted = torch.jit.script(model_quantized) # script model
---> 41 model_quantized_scripted(torch.rand((7,1,10,20)))

File ~/.pyenv/versions/3.8.11/envs/torch-commands/lib/python3.8/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
   1106 # If we don't have any hooks, we want to skip the rest of the logic in
   1107 # this function, and just call forward.
   1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110     return forward_call(*input, **kwargs)
   1111 # Do not call functions when jit is used
   1112 full_backward_hooks, non_full_backward_hooks = [], []

RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "<eval_with_key>.46", line 9, in forward
    quantize_per_tensor = torch.quantize_per_tensor(x, conv_input_scale_0, conv_input_zero_point_0, torch.quint8);  x = conv_input_scale_0 = conv_input_zero_point_0 = None
    conv = self.conv(quantize_per_tensor);  quantize_per_tensor = None
    bn = self.bn(conv);  conv = None
         ~~~~~~~ <--- HERE
    dequantize = bn.dequantize();  bn = None
    return dequantize
  File "/Users/myname/.pyenv/versions/3.8.11/envs/torch-commands/lib/python3.8/site-packages/torch/nn/quantized/modules/batchnorm.py", line 17, in forward
    def forward(self, input):
        return torch.ops.quantized.batch_norm2d(input, self.weight, self.bias, self.running_mean,
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
                                                self.running_var, self.eps, self.scale, self.zero_point)
RuntimeError: Cannot input a tensor of dimension other than 0 as a scalar argument

```

### Versions

CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.0.1 (x86_64)
GCC version: Could not collect
Clang version: 13.0.0 (clang-1300.0.29.3)
CMake version: version 3.22.1
Libc version: N/A

Python version: 3.8.11 (default, Jan 12 2022, 11:42:46)  [Clang 13.0.0 (clang-1300.0.29.3)] (64-bit runtime)
Python platform: macOS-12.0.1-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.5
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] Could not collect

cc @EikanWang @jgong5 @wenzhe-nrv @sanchitintel @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @Xia-Weiwen @leslie-fang-intel