Exporting T5 to ONNX
Author: jdsirota
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): t5-small (T5ForConditionalGeneration)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
The problem arises when running `convert_graph_to_onnx.py`

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x ] my own task or dataset: (give details below)
I am using a T5ForConditionalGeneration for machine translation.
## To reproduce

Steps to reproduce the behavior:

1. Run `python transformers/convert_graph_to_onnx.py --framework pt --model t5-small --tokenizer t5-small --opset 12 t5-small.onnx`

```
ONNX opset version set to: 12
Loading pipeline (model: t5-small, tokenizer: t5-small)
/Users/joshuasirota/onnx_env/lib/python3.6/site-packages/transformers/modeling_auto.py:798: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
  FutureWarning,
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 242M/242M [01:09<00:00, 3.50MB/s]
Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at t5-small and are newly initialized: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using framework PyTorch: 1.5.1
Error while converting the model: You have to specify either decoder_input_ids or decoder_inputs_embeds
```

## Expected behavior

An ONNX export should be created

## Environment info

- `transformers` version: 3.0.2
- Platform: Darwin-18.6.0-x86_64-i386-64bit
- Python version: 3.6.5
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Could you please provide your modified script?

fwiw, I'm seeing the same error when trying to export a T5 model for translation with both PyTorch and TensorFlow and the latest version of Transformers (3.3.1).

```txt
python3 path/to/convert_graph_to_onnx.py --model t5-base translation_en_to_de.onnx --pipeline translation_en_to_de --framework pt

====== Converting model to ONNX ======
ONNX opset version set to: 11
Loading pipeline (model: t5-base, tokenizer: t5-base)
Using framework PyTorch: 1.6.0
Error while converting the model: You have to specify either decoder_input_ids or decoder_inputs_embeds
```

```txt
python3 path/to/convert_graph_to_onnx.py --model t5-base translation_en_to_de.onnx --pipeline translation_en_to_de --framework tf

====== Converting model to ONNX ======
ONNX opset version set to: 11
Loading pipeline (model: t5-base, tokenizer: t5-base)
/usr/local/lib/python3.8/site-packages/transformers/modeling_tf_auto.py:690: FutureWarning: The class `TFAutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `TFAutoModelForCausalLM` for causal language models, `TFAutoModelForMaskedLM` for masked language models and `TFAutoModelForSeq2SeqLM` for encoder-decoder models.
  warnings.warn(
2020-09-30 16:29:58.866842: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-09-30 16:29:58.883705: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe05e6c8780 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-30 16:29:58.883726: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-09-30 16:29:58.893052: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
All model checkpoint weights were used when initializing TFT5ForConditionalGeneration.

All the weights of TFT5ForConditionalGeneration were initialized from the model checkpoint at t5-base.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFT5ForConditionalGeneration for predictions without further training.
/!\ Please note TensorFlow doesn't support exporting model > 2Gb /!\
Using framework TensorFlow: 2.3.1, keras2onnx: 1.7.0
Error while converting the model: You have to specify either inputs or inputs_embeds
```

Has anyone solved the issue??


Having this issue too, has anyone found a workaround?

I had the same issue but[ this](https://stackoverflow.com/a/66117248/13273054) post gave an insight on what was causing the error.

I had the same issue. Has anyone solved the issue?
Thanks!

@suyuzhang @Anku5hk @howlinghuffy @ankane @LowinLi please have a look at the [fastT5 ](https://github.com/Ki6an/fastT5) library. it **exports** any t5 model to onnx, quantized it, runs it on onnxruntime. you can speed up the t5 models up to  5x and can reduce the model size to 3x. for more info check out the repo [here ](https://github.com/Ki6an/fastT5).

Looks appropriate, Thanks!

I had the same issue. Has anyone solved the issue?
Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.