Onnx T5 for Generation
Author: hSterz
Pull request: no

## Environment info

`adapter-transformers` version: 2.1.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.5
- PyTorch version (GPU?): 1.8.1+cpu (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
 @patrickvonplaten, @patil-suraj
## Information
I want to use the to Onnx converted T5 model for generation, but I can only pass decoder_input_ids with a sequence length of 1. 

## To reproduce

Steps to reproduce the behavior:

1. Convert the T5 model to onnx: ` python -m transformers.onnx --model=t5-base --feature=seq2seq-lm onnx/t5-base/ `
2. Load the onnx model with `onnxruntime`: ` session = onnxruntime.InferenceSession('onnx/t5-base/model.onnx')`
3. Pass the model an input with a decoder sequence with more than one element: 
```
tokenizer = AutoTokenizer.from_pretrained("t5-base")
encoder_input = tokenizer("This is some text.", return_tensors="np")
decoder_inputs = tokenizer("bla bla", return_tensors="np")
print(decoder_inputs)

model_input = {
    "input_ids": encoder_input["input_ids"], 
    "attention_mask": encoder_input["attention_mask"], 
    "decoder_input_ids": decoder_inputs["input_ids"], 
    "decoder_attention_mask": decoder_inputs["attention_mask"]
}
outputs = session.run([], model_input)
```

## Expected behavior

I would expect there to be a way to pass multiple decoder_input_ids to the model to generate text. How is this intended to be done?


Comments:
Gently pinging @michaelbenayoun here

Is there any news concerning this?

Also gently pinging @lewtun and @echarlaix - should we try to move all Onnx issues to `optimum` instead?

That is because for now it is expected to be used with `past_key_values`, meaning that only the last `decoder_input_ids` are neeeded. We are currently working on changing this and it should be merged by the end of the year.

> Also gently pinging @lewtun and @echarlaix - should we try to move all Onnx issues to `optimum` instead?

From discussions with @mfuntowicz, my understanding is that we will keep the ONNX export functionality in `transformers`, and put all optimisation features (e.g. ONNX Runtime) in `optimum`. 

If that's still the plan, I suggest we only move optimization-related issues to the `optimum` repo for now.


Btw @hSterz, if you don't want to wait for the refactoring of the T5 ONNX conversion in `transformers`, there's a nice project called `fastt5` (https://github.com/Ki6an/fastT5) which does this and even includes a `generate()` method for the optimized model :)

> That is because for now it is expected to be used with `past_key_values`, meaning that only the last `decoder_input_ids` are neeeded. We are currently working on changing this and it should be merged by the end of the year.

I see. What would I use for the first token that is generated as past key values?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

As a side note, the export to ONNX is now officially handled in [Optimum](https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model).

Feel free to try it out:

```
optimum-cli export onnx --model t5-small --for-ort --task seq2seq-lm-with-past t5_onnx/
```

Giving:

```
.
└── t5_onnx
    ├── config.json
    ├── decoder_model.onnx
    ├── decoder_with_past_model.onnx
    ├── encoder_model.onnx
    ├── special_tokens_map.json
    ├── spiece.model
    ├── tokenizer_config.json
    └── tokenizer.json
```

Try `optimum-cli export onnx --help` for more.