Converting mBART to ONNX format
Author: rumeshmadhusanka
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.16.2
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.10.2+cu102 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: No


Models:
- MBART: @patil-suraj

## To reproduce
I'm trying to export mBART50 to ONNX on a Colab Notebook.
```
pip install transformers[onnx] sentencepiece -q
pip install torch --upgrade -q
python -m transformers.onnx --model=facebook/mbart-large-50 --feature seq2seq-lm-with-past onnx/
```
I'm getting the following error:

> Downloading: 100% 531/531 [00:00<00:00, 496kB/s]
> Downloading: 100% 1.38k/1.38k [00:00<00:00, 1.20MB/s]
> Downloading: 100% 4.83M/4.83M [00:00<00:00, 48.2MB/s]
> Downloading: 100% 649/649 [00:00<00:00, 456kB/s]
> Downloading: 100% 2.28G/2.28G [00:47<00:00, 51.6MB/s]
> Using framework PyTorch: 1.10.2+cu102
> Overriding 1 configuration item(s)
> 	- use_cache -> True
> /usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py:90: UserWarning: 'enable_onnx_checker' is deprecated and ignored. It will be removed in the next PyTorch release. To proceed despite ONNX checker failures, catch torch.onnx.ONNXCheckerError.
>   warnings.warn("'enable_onnx_checker' is deprecated and ignored. It will be removed in "
> /usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py:103: UserWarning: `use_external_data_format' is deprecated and ignored. Will be removed in next PyTorch release. The code will work as it is False if models are not larger than 2GB, Otherwise set to False because of size limits imposed by Protocol Buffers.
>   warnings.warn("`use_external_data_format' is deprecated and ignored. Will be removed in next "
> /usr/local/lib/python3.7/dist-packages/transformers/models/mbart/modeling_mbart.py:223: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
>   if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
> /usr/local/lib/python3.7/dist-packages/transformers/models/mbart/modeling_mbart.py:229: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
>   if attention_mask.size() != (bsz, 1, tgt_len, src_len):
> /usr/local/lib/python3.7/dist-packages/transformers/models/mbart/modeling_mbart.py:260: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
>   if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
> /usr/local/lib/python3.7/dist-packages/transformers/models/mbart/modeling_mbart.py:889: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
>   if input_shape[-1] > 1:
> /usr/local/lib/python3.7/dist-packages/transformers/models/mbart/modeling_mbart.py:91: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
>   if past_key_values_length > 0:
> Validating ONNX model...
> tcmalloc: large alloc 1073741824 bytes == 0x563a06b34000 @  0x7f3d33962b6b 0x7f3d33982379 0x7f3c45cf91dc 0x7f3c461c75a4 0x7f3c4603d385 0x7f3c4629e225 0x7f3c4602fd99 0x7f3c46027e76 0x7f3c45cc088e 0x7f3c45c762b8 0x7f3c45c41252 0x56388cae6f78 0x56388cb5aa6d 0x56388cb5502f 0x56388cae7aba 0x56388cb56108 0x56388cb5502f 0x56388cae836c 0x56388cb297b9 0x56388cb266d4 0x56388cae6c29 0x56388cb5ae61 0x56388cae79da 0x56388cb55eae 0x56388cae79da 0x56388cb55eae 0x56388cb5502f 0x56388cb54d43 0x56388cb531b0 0x56388cae6229 0x56388cae6120
> tcmalloc: large alloc 2147483648 bytes == 0x563a5c1dc000 @  0x7f3d33962b6b 0x7f3d33982379 0x7f3c45cf91dc 0x7f3c461c75a4 0x7f3c4603d385 0x7f3c4629e225 0x7f3c4602fd99 0x7f3c46027e76 0x7f3c45cc088e 0x7f3c45c762b8 0x7f3c45c41252 0x56388cae6f78 0x56388cb5aa6d 0x56388cb5502f 0x56388cae7aba 0x56388cb56108 0x56388cb5502f 0x56388cae836c 0x56388cb297b9 0x56388cb266d4 0x56388cae6c29 0x56388cb5ae61 0x56388cae79da 0x56388cb55eae 0x56388cae79da 0x56388cb55eae 0x56388cb5502f 0x56388cb54d43 0x56388cb531b0 0x56388cae6229 0x56388cae6120
> 	-[✓] ONNX model output names match reference model ({'present.5.encoder.key', 'present.4.decoder.key', 'present.6.decoder.value', 'present.0.decoder.value', 'present.8.decoder.value', 'present.11.encoder.key', 'present.0.encoder.key', 'present.11.encoder.value', 'present.3.encoder.key', 'present.0.encoder.value', 'present.7.decoder.key', 'present.5.encoder.value', 'present.0.decoder.key', 'present.4.encoder.key', 'present.2.decoder.value', 'present.10.encoder.key', 'present.3.decoder.value', 'present.11.decoder.value', 'present.6.encoder.value', 'present.5.decoder.key', 'present.8.encoder.value', 'present.10.decoder.value', 'present.5.decoder.value', 'present.7.decoder.value', 'present.2.encoder.key', 'present.10.encoder.value', 'present.1.decoder.key', 'present.9.decoder.value', 'present.9.decoder.key', 'present.10.decoder.key', 'logits', 'present.2.encoder.value', 'present.4.decoder.value', 'present.8.decoder.key', 'present.1.encoder.key', 'present.9.encoder.value', 'present.1.decoder.value', 'present.4.encoder.value', 'present.11.decoder.key', 'present.1.encoder.value', 'present.6.encoder.key', 'present.3.encoder.value', 'present.7.encoder.value', 'present.8.encoder.key', 'present.2.decoder.key', 'present.3.decoder.key', 'present.7.encoder.key', 'present.6.decoder.key', 'present.9.encoder.key'})
> 	- Validating ONNX Model output "logits":
> 		-[✓] (2, 2, 250054) matches (2, 2, 250054)
> 		-[x] values not close enough (atol: 1e-05)
> Traceback (most recent call last):
>   File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
>     "__main__", mod_spec)
>   File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
>     exec(code, run_globals)
>   File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/__main__.py", line 77, in <module>
>     main()
>   File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/__main__.py", line 70, in main
>     validate_model_outputs(onnx_config, tokenizer, model, args.output, onnx_outputs, args.atol)
>   File "/usr/local/lib/python3.7/dist-packages/transformers/onnx/convert.py", line 230, in validate_model_outputs
>     "Outputs values doesn't match between reference model and ONNX exported model: "
> ValueError: Outputs values doesn't match between reference model and ONNX exported model: Got max absolute difference of: 3.5762786865234375e-05

Is there a workaround/fix?


Comments:
cc @lewtun 

Hey @rumeshmadhusanka thanks for raising this issue! The error indicates that the numerical agreement between the raw and ONNX models is larger than ~3.6e-5. 

We try to pick good defaults in the library, but in some cases you get this disagreement. You can tweak the required agreement by setting the `--atol` argument in the CLI, e.g.

```bash
python -m transformers.onnx --model=facebook/mbart-large-50 --feature seq2seq-lm-with-past --atol=5e-5 onnx/
```

We find that agreement in the range 1e-5 to 1e-3 is generally fine - hope that helps!