Is it possible to use ONNX for summarisation with BART yet?
Author: HenryDashwood
Pull request: no

Hi there, I've been trying out the latest iteration of ONNX support for BART because it looks like it now supports past key values which I think means I could generate summaries from an ONNX version of my BART models.

I seem to be able to get the model converting nicely as you can see here
```python
import numpy as np
from transformers import AutoTokenizer
from onnxruntime import InferenceSession

model_ckpt = "/home/ubuntu/article-summarisation/checkpoints/bart/2021_08_20_15_08_00/model"
onnx_path = "/home/ubuntu/article-summarisation/checkpoints/bart/2021_08_20_15_08_00/onnx"

tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-cnn")
tokenizer.save_pretrained(model_ckpt)

!python -m transformers.onnx --model={model_ckpt} --feature="seq2seq-lm-with-past" {onnx_path} --atol 1e-04
```
```
Using framework PyTorch: 1.10.1+cu102
Overriding 1 configuration item(s)
	- use_cache -> True
/home/ubuntu/.pyenv/versions/3.9.9/envs/py399/lib/python3.9/site-packages/torch/onnx/utils.py:90: UserWarning: 'enable_onnx_checker' is deprecated and ignored. It will be removed in the next PyTorch release. To proceed despite ONNX checker failures, catch torch.onnx.ONNXCheckerError.
  warnings.warn("'enable_onnx_checker' is deprecated and ignored. It will be removed in "
/home/ubuntu/.pyenv/versions/3.9.9/envs/py399/lib/python3.9/site-packages/torch/onnx/utils.py:103: UserWarning: `use_external_data_format' is deprecated and ignored. Will be removed in next PyTorch release. The code will work as it is False if models are not larger than 2GB, Otherwise set to False because of size limits imposed by Protocol Buffers.
  warnings.warn("`use_external_data_format' is deprecated and ignored. Will be removed in next "
/home/ubuntu/.pyenv/versions/3.9.9/envs/py399/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py:217: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):
/home/ubuntu/.pyenv/versions/3.9.9/envs/py399/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py:223: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attention_mask.size() != (bsz, 1, tgt_len, src_len):
/home/ubuntu/.pyenv/versions/3.9.9/envs/py399/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py:254: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
/home/ubuntu/.pyenv/versions/3.9.9/envs/py399/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py:889: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if input_shape[-1] > 1:
/home/ubuntu/.pyenv/versions/3.9.9/envs/py399/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py:88: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if past_key_values_length > 0:
Validating ONNX model...
	-[✓] ONNX model output names match reference model ({'present.10.decoder.key', 'present.11.decoder.value', 'present.6.decoder.value', 'present.1.decoder.value', 'present.11.decoder.key', 'present.11.encoder.key', 'present.11.encoder.value', 'present.10.encoder.value', 'present.9.decoder.value', 'present.9.encoder.value', 'present.6.encoder.key', 'present.3.encoder.value', 'present.3.decoder.key', 'logits', 'present.4.decoder.value', 'present.4.encoder.value', 'present.8.encoder.value', 'present.7.decoder.key', 'present.7.encoder.value', 'present.0.encoder.key', 'present.5.decoder.key', 'present.9.encoder.key', 'present.2.decoder.key', 'present.10.encoder.key', 'present.3.decoder.value', 'present.1.encoder.value', 'present.10.decoder.value', 'present.9.decoder.key', 'present.0.encoder.value', 'present.2.decoder.value', 'present.5.decoder.value', 'present.8.encoder.key', 'present.5.encoder.value', 'present.0.decoder.value', 'present.5.encoder.key', 'present.7.encoder.key', 'present.2.encoder.value', 'present.6.decoder.key', 'present.7.decoder.value', 'present.4.decoder.key', 'present.1.encoder.key', 'present.2.encoder.key', 'present.0.decoder.key', 'present.1.decoder.key', 'present.3.encoder.key', 'present.4.encoder.key', 'present.8.decoder.key', 'present.8.decoder.value', 'present.6.encoder.value'})
	- Validating ONNX Model output "logits":
		-[✓] (2, 2, 50265) matches (2, 2, 50265)
		-[✓] all values close (atol: 0.0001)
	- Validating ONNX Model output "present.0.decoder.key":
		-[✓] (2, 16, 7, 64) matches (2, 16, 7, 64)
		-[✓] all values close (atol: 0.0001)
		-[✓] all values close (atol: 0.0001)
	- Validating ONNX Model output "present.11.encoder.value":
		-[✓] (2, 16, 8, 64) matches (2, 16, 8, 64)
		-[✓] all values close (atol: 0.0001)
All good, model saved at: /home/ubuntu/article-summarisation/checkpoints/bart/2021_08_20_15_08_00/onnx/model.onnx
```

But it all falls apart when I run anything
```python
ort_session = InferenceSession(f"{onnx_path}/model.onnx")
inputs = tokenizer("My name is Henry", return_tensors="pt")
onnx_outputs = ort_session.run(None, dict(inputs))
```
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [5], in <module>
----> 1 onnx_outputs = ort_session.run(None, dict(inputs))

File ~/.pyenv/versions/3.9.9/envs/py399/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:188, in Session.run(self, output_names, input_feed, run_options)
    186 # the graph may have optional inputs used to override initializers. allow for that.
    187 if num_inputs < num_required_inputs:
--> 188     raise ValueError("Model requires {} inputs. Input Feed contains {}".format(num_required_inputs, num_inputs))
    189 if not output_names:
    190     output_names = [output.name for output in self._outputs_meta]

ValueError: Model requires 52 inputs. Input Feed contains 2
```

Am I missing something, or is this still beyond the scope of what's possible? I've spent the day playing round with different approaches which throw up different errors but this feels like the closest I've got to success.

Comments:
I haven't run the script, but reading off of your code, is there a reason why you put `None` in `onnx_outputs = ort_session.run(None, dict(inputs))`? My understanding is that you would need something like `outputs = ort_session.run(["last_hidden_state"], dict(inputs))` instead, where the string key is obtained from `BartOnnxConfig(BartConfig()).outputs.keys()`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@HenryDashwood ,
Check by creating if all input criteria's are met. since you are using seq2seq-lm-with-past feature, it means you will have to pass decoder hidden state values too in input.