Got `ONNXRuntimeError` when try to run BART in ONNX format
Author: ryangawei
Pull request: no
Labels: name: WIP

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Using GPU in script?: Yes

### Who can help
@mfuntowicz

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.



Models:

- 

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## To reproduce

I was using Google Colab and trying to export model `facebook/bart-large-cnn` to the onnx format. I ran the command `python -m transformers.onnx -m=facebook/bart-large-cnn onnx/bart-large-cnn`, and the outputs seem okay.

```
2021-07-22 23:14:33.821472: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Using framework PyTorch: 1.9.0+cu102
Overriding 1 configuration item(s)
	- use_cache -> False
/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py:212: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py:218: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py:249: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py:863: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_shape[-1] > 1:
tcmalloc: large alloc 1625399296 bytes == 0x5595ce83a000 @  0x7f1780d9f887 0x7f177f695c29 0x7f177f696afb 0x7f177f696bb4 0x7f177f696f9c 0x7f17670dcbb7 0x7f17670dd064 0x7f175b75ba1c 0x7f176bf8eaff 0x7f176b949b88 0x55949fda8bf8 0x55949fe1c6f2 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fe16c35 0x55949fda973a 0x55949fe1bf40 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fda965a 0x55949fe17b0e 0x55949fda965a 0x55949fe17b0e 0x55949fe16c35 0x55949fe16933 0x55949fe14da0 0x55949fda7ea9 0x55949fda7da0 0x55949fe1bbb3
tcmalloc: large alloc 1625399296 bytes == 0x55962f654000 @  0x7f1780d9f887 0x7f177f695c29 0x7f177f696afb 0x7f177f696bb4 0x7f177f696f9c 0x7f17670dcbb7 0x7f17670dd064 0x7f175b75ba1c 0x7f176bf8ecab 0x7f176b949b88 0x55949fda8bf8 0x55949fe1c6f2 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fe16c35 0x55949fda973a 0x55949fe1bf40 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fda965a 0x55949fe17b0e 0x55949fda965a 0x55949fe17b0e 0x55949fe16c35 0x55949fe16933 0x55949fe14da0 0x55949fda7ea9 0x55949fda7da0 0x55949fe1bbb3
tcmalloc: large alloc 1625399296 bytes == 0x5595ce83a000 @  0x7f1780d9d1e7 0x55949fdd9a18 0x55949fda4987 0x7f176bf8ece2 0x7f176b949b88 0x55949fda8bf8 0x55949fe1c6f2 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fe16c35 0x55949fda973a 0x55949fe1bf40 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fda965a 0x55949fe17b0e 0x55949fda965a 0x55949fe17b0e 0x55949fe16c35 0x55949fe16933 0x55949fe14da0 0x55949fda7ea9 0x55949fda7da0 0x55949fe1bbb3 0x55949fe16c35 0x55949fda973a 0x55949fe17b0e 0x55949fe16c35 0x55949fce8eb1
tcmalloc: large alloc 1625399296 bytes == 0x55962f654000 @  0x7f1780d9f887 0x7f177f695c29 0x7f177f695d47 0x7f177f6977a5 0x7f176bd60368 0x7f176bfbc844 0x7f176b949b88 0x55949fda8010 0x55949fda7da0 0x55949fe1bbb3 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fe16c35 0x55949fda973a 0x55949fe1bf40 0x55949fe16c35 0x55949fda973a 0x55949fe1893b 0x55949fda965a 0x55949fe17b0e 0x55949fda965a 0x55949fe17b0e 0x55949fe16c35 0x55949fe16933 0x55949fe14da0 0x55949fda7ea9 0x55949fda7da0 0x55949fe1bbb3 0x55949fe16c35 0x55949fda973a
Validating ONNX model...
	-[✓] ONNX model outputs' name match reference model ({'last_hidden_state', 'encoder_last_hidden_state'}
	- Validating ONNX Model output "last_hidden_state":
		-[✓] (2, 8, 1024) matchs (2, 8, 1024)
		-[✓] all values close (atol: 0.0001)
	- Validating ONNX Model output "encoder_last_hidden_state":
		-[✓] (2, 8, 1024) matchs (2, 8, 1024)
		-[✓] all values close (atol: 0.0001)
All good, model saved at: onnx/bart-large-cnn/model.onnx
```
Then I tried to execute the model in `onnxruntime`,

```
import onnxruntime as ort

ort_session = ort.InferenceSession('onnx/bart-large-cnn/model.onnx')

# Got input_ids and attention_mask using tokenizer

outputs = ort_session.run(None, {'input_ids': input_ids.detach().cpu().numpy(), 'attention_mask': attention_mask.detach().cpu().numpy()})
```
And I got the error,

```
---------------------------------------------------------------------------
RuntimeException                          Traceback (most recent call last)
<ipython-input-30-380e6a0e1085> in <module>()
----> 1 outputs = ort_session.run(None, {'input_ids': input_ids.detach().cpu().numpy(), 'attention_mask': attention_mask.detach().cpu().numpy()})

/usr/local/lib/python3.7/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py in run(self, output_names, input_feed, run_options)
    186             output_names = [output.name for output in self._outputs_meta]
    187         try:
--> 188             return self._sess.run(output_names, input_feed, run_options)
    189         except C.EPFail as err:
    190             if self._enable_fallback:

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_109' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:42 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, std::vector<long int>&, bool) gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{2}, requested shape:{1,1}
```

I see that BART is recently supported for ONNX in the latest release, but there isn't any code to exactly explain how to run the inference in `onnxruntime`. Maybe I'm doing something wrong here, so any help will be appreciated!


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->


Comments:
I can reproduce in latest `transformers` with latest onnx runtime. 

FYI this error seems to be linked to the dimension of the input; if you use a batch size 2 it should work.

As seen with @mfuntowicz offline, we'll be working on a fix in the coming weeks cc @michaelbenayoun 

@LysandreJik Thank you for the follow-up. I'll pay attention to any updates.

Can reproduce with `valhalla/distilbart-mnli-12-1` in `4.10.0`. @LysandreJik 
The export is essentially dependent on the number of hypotheses it was exported with, as far as I can tell.

Any update on this? Can reproduce the same for facebook/bart-large-mnli. Works only with a batch size of 2 during inference. @LysandreJik @mfuntowicz 

transformers.__version__ == 4.20.0.dev0
onnxruntime.__version__ == 1.11.1

exported facebook/bart-base successfully , following instructions on -
https://github.com/huggingface/transformers/tree/main/examples/research_projects/onnx/summarization

script output - 

2022-05-16 16:06:57 | INFO | __main__ |  [run_onnx_exporter.py:163] Model outputs from torch and ONNX Runtime are similar.
2022-05-16 16:06:57 | INFO | __main__ |  [run_onnx_exporter.py:164] Success.

however, loading the exported model fails after it hangs forever (timing out), using this script - 

```
import torch
from onnxruntime import InferenceSession, SessionOptions, GraphOptimizationLevel

options = SessionOptions() # initialize session options
options.graph_optimization_level = GraphOptimizationLevel.ORT_ENABLE_ALL

session = InferenceSession(
    'optimized_BART.onnx',
    sess_options=options,
    providers=["CPUExecutionProvider"]
)

session.disable_fallback()
```


(py39) user@Avis-MacBook-Pro-2 summarization % ls -lht
-rw-r--r--  1 user  staff   680M May 16 16:06 optimized_BART.onnx

exported model size about 680MB 

any advice on this? 

transformers.__version__ == 4.20.0.dev0
onnxruntime.__version__ == 1.11.1

onnx bart fails to load (hangs forever) when passing options to InferenceSession()

avoid these -
options.graph_optimization_level = GraphOptimizationLevel.ORT_ENABLE_ALL

otherwise loading the model hangs forever.
upon keyboard interrupt, I am getting tons of these warnings - 

2022-05-16 15:57:35.009102 [W:onnxruntime:, graph.cc:3559 CleanUnusedInitializersAndNodeArgs] Removing initializer '1772'. It is not used by any node and should be removed from the model.
2022-05-16 15:57:36.410981 [W:onnxruntime:, constant_folding.cc:202 ApplyImpl] Unsupported output type of N11onnxruntime22SequenceTensorTypeBaseE. Can't constant fold SequenceEmpty node 'SequenceEmpty_5330'
2022-05-16 15:57:36.416645 [W:onnxruntime:, constant_folding.cc:202 ApplyImpl] Unsupported output type of N11onnxruntime22SequenceTensorTypeBaseE. Can't constant fold SequenceEmpty node 'SequenceEmpty_808'
2022-05-16 15:57:36.416741 [W:onnxruntime:, constant_folding.cc:202 ApplyImpl] Unsupported output type of N11onnxruntime22SequenceTensorTypeBaseE. Can't constant fold SequenceEmpty node 'SequenceEmpty_1'
2022-05-16 15:57:36.446512 [W:onnxruntime:, constant_folding.cc:202 ApplyImpl] Unsupported output type of N11onnxruntime22SequenceTensorTypeBaseE. Can't constant fold SequenceEmpty node 'SequenceEmpty_5128'
2022-05-16 15:57:37.813252 [W:onnxruntime:, graph.cc:3559 CleanUnusedInitializersAndNodeArgs] Removing initializer '3149'. It is not used by any node and should be removed from the model.
2022-05-16 15:57:37.813269 [W:onnxruntime:, graph.cc:3559 CleanUnusedInitializersAndNodeArgs] Removing initializer '2153'. It is not used by any node and should be removed from the model.
....

loaded the onnx model successfully without options.graph_optimization_level. 
fails to get a prediction :(



```
import onnxruntime as ort
import numpy as np

ort_session = ort.InferenceSession(
    'optimized_BART.onnx')

print(f'inputs: {[i.name for i in ort_session.get_inputs()]}')

feed_dict = summarizer.tokenizer(text)
feed_dict['num_beams'] = 4
feed_dict['max_length'] = 120
feed_dict['decoder_start_token_id'] = 2
feed_dict = {k: np.int64([v]) for k, v in feed_dict.items()}

for key in feed_dict:
    print(f'feed_dict key: {key}, shape: {feed_dict[key].shape}')

pred = session.run(None, feed_dict)
```` 

### printout - 

inputs: ['input_ids', 'attention_mask', 'num_beams', 'max_length', 'decoder_start_token_id']
feed_dict key: input_ids, shape: (1, 228)
feed_dict key: attention_mask, shape: (1, 228)
feed_dict key: num_beams, shape: (1,)
feed_dict key: max_length, shape: (1,)
feed_dict key: decoder_start_token_id, shape: (1,)

InvalidArgument                           Traceback (most recent call last)
Input In [39], in <cell line: 11>()
      8 for key in feed_dict:
      9     print(f'feed_dict key: {key}, shape: {feed_dict[key].shape}')
---> 11 pred = session.run(['output_ids'], feed_dict)

File ~/envs/py39/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:192, in Session.run(self, output_names, input_feed, run_options)
    190     output_names = [output.name for output in self._outputs_meta]
    191 try:
--> 192     return self._sess.run(output_names, input_feed, run_options)
    193 except C.EPFail as err:
    194     if self._enable_fallback:

InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: attention_mask for the following indices
 index: 1 Got: 228 Expected: 13
 Please fix either the inputs or the model.


fails to export facebook/bart-large-cnn or , following instructions on -
https://github.com/huggingface/transformers/tree/main/examples/research_projects/onnx/summarization

(py39) user@Avis-MacBook-Pro-2 summarization % python run_onnx_exporter.py --model_name_or_path facebook/bart-large-cnn
Traceback (most recent call last):
  File "~/src/transformers/examples/research_projects/onnx/summarization/run_onnx_exporter.py", line 207, in <module>
    main()
  File "~/src/transformers/examples/research_projects/onnx/summarization/run_onnx_exporter.py", line 184, in main
    model, tokenizer = load_model_tokenizer(args.model_name_or_path, device)
  File "~/src/transformers/examples/research_projects/onnx/summarization/run_onnx_exporter.py", line 93, in load_model_tokenizer
    huggingface_model = model_dict[model_name].from_pretrained(model_name).to(device)
KeyError: 'facebook/bart-large-cnn'

same error when trying to export model lidiya/bart-base-samsum

any advice would be greatly appreciated. thanks.