Large Discrepancies between PyTorch and ONNXRuntime Inference 
Labels: module: onnx, triaged

### 🐛 Describe the bug

I am getting large discrepancies between PyTorch inference and ONNXRuntime inference for a number of models. I am unsure why this is the case. See below:

| model               |   greatest abs diff. |
|:--------------------|---------------------:|
| model_78_3459169476 |          9.22337e+18 |
| model_68_72227799   |     164704           |
| model_6_419668759   |     111273           |
| model_96_728966504  |      26586.2         |
| model_30_213242519  |       4178           |
| model_15_2262524614 |       2369.29        |
| model_29_103575771  |       1258.29        |
| model_78_2707514266 |        481.639       |
| model_60_674754574  |         66.582       |
| model_91_3302636491 |         41           |
| model_50_1243263724 |         29           |
| model_74_3620935663 |         12           |
| model_6_3867095439  |          8.49595     |
| model_25_1123596368 |          5.14588     |
| model_23_3959668477 |          5           |

I generated these models using the [NNSmith Fuzzing Tool](https://github.com/ise-uiuc/nnsmith) and found the discrepancies using the `find_mismatch` function. Due to the way, NNSmith creates models I am unable to provide a minimal reproduction but linked are pickled objects that are loaded in the reproduction script ([here](https://drive.google.com/drive/folders/1sMWNhIZ4PutSlZVaVsehnzvBp59pmTaA?usp=sharing)).

# Reproduction:
## Dependencies:
- [NNSmith](https://github.com/ise-uiuc/nnsmith) : `pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade`
- Model Link: ([Original Model and Exported Repro](https://drive.google.com/drive/folders/1sMWNhIZ4PutSlZVaVsehnzvBp59pmTaA?usp=sharing))

## Script:
```python
import pickle
from pathlib import Path

import torch
from nnsmith.materialize import Model, Oracle
from torch.onnx.verification import find_mismatch

path = Path("./model_78_3459169476")

# Get the paths for pickles and weights
gir_path: Path = path / "gir.pkl"
oracle_path: Path = path / "oracle.pkl"
weights_path: Path = path / "model.pth"

# Load the model from pickle
with gir_path.open("rb") as f:
    gir = pickle.load(f)
model_type = Model.init("torch", "cpu")
model = model_type.from_gir(gir)

# Load weights from weight path.
model.torch_model.load_state_dict(torch.load(weights_path), strict=False)

# Load oracle
oracle = Oracle.load(oracle_path)

model_args = tuple([torch.from_numpy(val) for key, val in oracle.input.items()])

print(f"Testing: {str(path)}")
graph_info = find_mismatch(
    model.torch_model,
    model_args,
    opset_version=16,
    keep_initializers_as_inputs=False,
)

repro_path = path / "model_repro"
graph_info.export_repro(repro_path)
```

### Versions

PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.5.2 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.0.40.1)
CMake version: version 3.27.5
Libc version: N/A

Python version: 3.10.13 (main, Sep 11 2023, 08:16:02) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-13.5.2-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1

Versions of relevant libraries:
[pip3] numpy==1.25.2
[pip3] pytorch-lightning==2.0.3
[pip3] torch==2.0.1
[pip3] torch-tb-profiler==0.4.1
[pip3] torchaudio==2.0.0
[pip3] torchdata==0.6.1
[pip3] torchmetrics==0.11.4
[pip3] torchtext==0.15.2
[pip3] torchvision==0.15.0
[pip3] torchviz==0.0.2
[conda] numpy                     1.25.2          py310h3b2db8e_0  
[conda] numpy-base                1.25.2          py310ha9811e2_0  
[conda] pytorch-lightning         2.0.3                    pypi_0    pypi
[conda] torch                     2.0.1                    pypi_0    pypi
[conda] torch-tb-profiler         0.4.1                    pypi_0    pypi
[conda] torchaudio                2.0.0                 py310_cpu    pytorch
[conda] torchdata                 0.6.1                    pypi_0    pypi
[conda] torchmetrics              0.11.4                   pypi_0    pypi
[conda] torchtext                 0.15.2                   pypi_0    pypi
[conda] torchvision               0.15.0                py310_cpu    pytorch
[conda] torchviz                  0.0.2                    pypi_0    pypi