Recommended way of exporting encoder-decoder model to ONNX with `transformers[onnx]`
Author: gomerudo
Pull request: no

I am looking for a way to export an encoder-decoder to ONNX to run inference. I followed the guide at [Exporting Transformers Models](https://huggingface.co/docs/transformers/serialization) but that only shows an example of an encoder-only model. Trying to accomplish this for the specific case of the [Helsinki-NLP/Opus-MT model for Spanish to English](https://huggingface.co/Helsinki-NLP/opus-mt-es-en), I did the following:

1. I exported the model with the following command: `python -m transformers.onnx --model=Helsinki-NLP/opus-mt-es-en --feature=seq2seq-lm --atol=2e-05 workspace/onnx/opus-mt-es-en `

The output of the model was successful.

2. Then, as in the docs, I tried running inference on the model with a code similar to the one below.

```
from transformers import AutoTokenizer
from onnxruntime import InferenceSession

tokenizer = ...
session = InferenceSession("onnx/model.onnx")

inputs = tokenizer("Probando el uso de Marian despues de haberlo exportando a ONNX", return_tensors="np", padding=True)
outputs = session.run(output_names=["logits"], input_feed=dict(inputs))
```

This yields the following exception:

`ValueError: Model requires 4 inputs. Input Feed contains 2`. 

------

I tried a similar thing with T5, and the same exception was raised. After some debugging, I realized that any encoder-decoder architecture expects the following 4 arguments: `input_ids`, `attention_mask`, `decoder_input_ids`, `decoder_attention_mask`. 

After thorough reading of the transformer's code, my understanding is that any model in transformers inherits from `PreTrainedModel`, which obviously defines models that are intended for training. This implies that the associated Config requires inputs that are used during training and that explains the need for 4 arguments instead of 2, contrary to the trivial case of encoder-only models that is documented in website.

However, when working with a transformer model (no export to ONNX), one is able to use the `generate()` function that is added on top of seq2seq models by `GenerationMixin`. This function is the helper to perform seq2seq during inference.

------

The question is the following:

Is there a way (or maybe a recommended workaround) to export an encoder-decoder model to ONNX, such that it behaves as the `generate()` function from `GenerationMixin` and not as the `forward()` method in `PreTrainedModel`?

-----

I know that a possible workaround would be to export both the encoder and the decoder separately and programmatically connect the input/outputs of each Individual `InferenceSession`. Other than that, I cannot figure out an obvious solution to this problem using the out-of-the-box methods in `transformers`.

Any help will be highly appreciated :)


Comments:
Hey @gomerudo thanks for raising this issue and taking the time to clearly explain the problem! 

Your analysis is spot on regarding the need for 4 arguments instead of 2. We mention the role of `output_names` in the guide, but we should also mention that one can get the input names via:

```python
from transformers.models.marian import MarianConfig, MarianOnnxConfig

config = MarianConfig()
onnx_config = MarianOnnxConfig(config)
# Returns ['input_ids', 'attention_mask', 'decoder_input_ids', 'decoder_attention_mask']
print(list(onnx_config.inputs.keys()))
```

I'll update the guide to clarify this :)

Currently, the best solution is to export the encoder and decoder separately (e.g. as done in the `fastt5` project [here](https://github.com/Ki6an/fastT5/blob/2f73bd57ca3bab226952679b4381049eb09721a4/fastT5/onnx_models.py#L140)). We're investigating an approach to support text generation in our `optimum` [library](https://github.com/huggingface/optimum), but there's no timeline I can share yet

Hi @gomerudo, did you succeed at making this work?

@lewtun I used `fastT5` to export a T5 checkpoint to ONNX. The export finished to completion, and I was able to make inference using the `generate()` function, but was unable to extract hidden states. Please see code snippet below. 

```
from fastT5 import export_and_get_onnx_model
from transformers import AutoTokenizer


model_path = "path/to/model/checkpoint"
model = export_and_get_onnx_model(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
input_terms = ["this is a sample input"]

token = tokenizer(input_terms, max_length=512 * 2, padding=True, truncation=True, return_tensors='pt')
tokens = model.generate(input_ids=token['input_ids'].to('cpu'),
                        attention_mask=token['attention_mask'].to('cpu'),
                        return_dict_in_generate=True,
                        max_length=512 * 2,
                        num_beams=1,
                        output_scores=True,
                        output_hidden_states=True)
```

The resulting dictionary returned by `model.generate()` is as follows:

```
GreedySearchEncoderDecoderOutput(
   sequences=tensor([[  0, 119, 114, 102, 108, 111, 108, 125, 120, 112, 100, 101,  35,  53, 
        ...
        ...),  
   encoder_attentions=None, 
   encoder_hidden_states=None, 
   decoder_attentions=None, 
   cross_attentions=None, 
   decoder_hidden_states=(None, None,...
        ...
        None)
)
```

Any idea why this is happening? How can I get the hidden states? Using the model before conversion with the above `generate()` function call generates the hidden states for both encoder and decoder.

Hey @vsoesanto I think that kind of question would be best asked on the `fastT5` repo as I believe they wrap the model classes in a special way. This might explain why you can no longer see the hidden states with their `generate()` method

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

HI all! Any updates on this? I've also tried to convert the OPUS-MT Marian Models to ONNX without success. I'm aware of the option to convert the encoder and decoder separately, but the issue is implementing Beam Search for the decoding process. Any changes the `optimum` package will be able to handle seq2seq models in the near future? Thanks in advance :) cc @lewtun 

> HI all! Any updates on this? I've also tried to convert the OPUS-MT Marian Models to ONNX without success. I'm aware of the option to convert the encoder and decoder separately, but the issue is implementing Beam Search for the decoding process. Any changes the `optimum` package will be able to handle seq2seq models in the near future? Thanks in advance :) cc @lewtun

Hello Pedrogov! Did you successfully convert the encoder-decoder model to onnx format? I have encountered this problem now, and I plan to convert the transformer model written by pytorch into onnx, which feels very tricky. If you have achieved success, can you lend me a reference? Thanks in advance！

Hi @doubletfly! No, I have not yet converted neither the encoder nor the decoder to ONNX. The main issue I issue I faced was that the models seem to generate text using Beam Search instead of Greedy, so this would need to be implemented by hand in order to replace the model.

@doubletfly @Pedrohgv this worked for me, at least for the m2m_100: https://github.com/huggingface/optimum/blob/b8ea77029bb6fffddc175a540cc29f15efa0fe68/docs/source/onnxruntime/modeling_ort.mdx#export-and-inference-of-sequence-to-sequence-models

> Hi @doubletfly! No, I have not yet converted neither the encoder nor the decoder to ONNX. The main issue I issue I faced was that the models seem to generate text using Beam Search instead of Greedy, so this would need to be implemented by hand in order to replace the model.

Okay！Thanks for your reply.  :)

> @doubletfly @Pedrohgv this worked for me, at least for the m2m_100: https://github.com/huggingface/optimum/blob/b8ea77029bb6fffddc175a540cc29f15efa0fe68/docs/source/onnxruntime/modeling_ort.mdx#export-and-inference-of-sequence-to-sequence-models

That's good, i'm going to study it, thanks for your advice. :）

@malloc-naski that's great, I tried Optimum once and it didn't work for the OPUS models, but it was some time ago and I don't remember exactly what it was I was doing. If I have time, I'll test this. Thank you!

Hi @Pedrohgv @doubletfly @malloc-naski we've recently added support for seq2seq models in `optimum`, so you can now do text generation with ONNX models directly via the `pipeline()` function from `transformers` 😎 

Here's a simple demo:

```python
# Install `optimum` from source with ONNX Runtime backend
%pip install git+https://github.com/huggingface/optimum.git#egg=optimum[onnxruntime]

from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de")
# `from_transformers=True` downloads the PyTorch weights and converts them to ONNX format
model = ORTModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-en-de", from_transformers=True)
onnx_translation = pipeline("translation_en_to_de", model=model, tokenizer=tokenizer)

text = "My name is Lewis and I live in Switzerland."
pred = onnx_translation(text)
# Returns [{'translation_text': 'Mein Name ist Lewis und ich lebe in der Schweiz'}]
pred
```

Hope this helps!

@lewtun Thank you very much! 

> Hi @Pedrohgv @doubletfly @malloc-naski we've recently added support for seq2seq models in `optimum`, so you can now do text generation with ONNX models directly via the `pipeline()` function from `transformers` 😎
> 
> Here's a simple demo:
> 
> ```python
> # Install `optimum` from source with ONNX Runtime backend
> %pip install git+https://github.com/huggingface/optimum.git#egg=optimum[onnxruntime]
> 
> from transformers import AutoTokenizer, pipeline
> from optimum.onnxruntime import ORTModelForSeq2SeqLM
> 
> tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de")
> # `from_transformers=True` downloads the PyTorch weights and converts them to ONNX format
> model = ORTModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-en-de", from_transformers=True)
> onnx_translation = pipeline("translation_en_to_de", model=model, tokenizer=tokenizer)
> 
> text = "My name is Lewis and I live in Switzerland."
> pred = onnx_translation(text)
> # Returns [{'translation_text': 'Mein Name ist Lewis und ich lebe in der Schweiz'}]
> pred
> ```
> 
> Hope this helps!

Hi!
This works well. But I have problems when trying to load a saved onnx model.
The error is as follows:

`ValueError: Model requires 49 inputs. Input Feed contains 25`

Any suggestions?

Hi @ahmedbr! Can you provide an example script to reproduce this issue please?

https://colab.research.google.com/drive/14Gmc9xkkvJNCf7EiPYp6cjhThwAux7M6?usp=sharing
 I am trying to get the resulted text from this model but I only get two results 
last_hidden_state with shape (1, 2, 512) and onnx::MatMul_949 with shape of (1, 1500, 512)
how can I get token ids from this two outputs 
this onnx was made by transformers.onnx as one output

I tried using optimum too and it gave me several models :
    "decoder_model.onnx",
    "decoder_model_quantized.onnx",
    "encoder_model.onnx",
    "decoder_model_merged.onnx",
    "decoder_with_past_model.onnx",
    "encoder_model_quantized.onnx",
    "decoder_model_merged_quantized.onnx",
    "decoder_with_past_model_quantized.onnx"

here is the code i used this models in :
https://colab.research.google.com/drive/167UnyEdzPfXFWg81m32efvBNqTHxR37Q?usp=sharing

I used the encoder and decoder and I was able to get logits that gave
valid token ids but only 2 tokens, the first I am sure it's correct

@AMF777 Optimum is the recommended tool to export models to ONNX as `transformers.onnx` is no longer maintained.

Trying to make the different exported decoders clearer:
- `decoder_model.onnx` is the decoder without key-value cache. It is usually used in the first generation iteration.
- `decoder_with_past_model.onnx` is the decoder with key-value cache. It is used after the first generation iteration, when a cache can be set to avoid recomputing intermediate key and values.
- `decoder_model_merged` is the merge of the two previous decoders, so it will automatically switch between both. That's the one you should use.

Let me know if that helps.

Thank you very much @regisss  it helped 

@regisss 
hey i was working with a s2s model called [Pix2struct](https://huggingface.co/docs/transformers/v4.31.0/en/model_doc/pix2struct)
and since the model is an encoder_decoder type 
Using this Optimum  converter i was able to generate onnx encoders and decoders but i now have a problem of matching the inputs
```python
import onnx
import torch
import onnxruntime
model_encoder = onnx.load("encoder_model.onnx")
model_decoder = onnx.load("decoder_model.onnx")
onnx.checker.check_model(model_encoder)
onnx.checker.check_model(model_decoder)
ort_session_encoder = onnxruntime.InferenceSession("encoder_model.onnx")
ort_session_decoder = onnxruntime.InferenceSession("decoder_model.onnx")
print("Model encoder loaded")
print("Model decoder loaded")
# Get input and output names
'''
Encoder input name 0 : flattened_patches
Encoder input name 1 : attention_mask
**********
Encoder output name 0 : last_hidden_state
**********
Decoder input name 0 : encoder_attention_mask
Decoder input name 1 : input_ids
Decoder input name 2 : decoder_attention_mask
Decoder input name 3 : encoder_hidden_states
**********
'''
```
Im somewhat new to this, if you don't mind me could you tell me how it can be done?

@rakshith111 It's better to open a dedicated issue in Optimum: https://github.com/huggingface/optimum/issues
Please provide a code snippet that enables us to reproduce this issue if possible.

> @rakshith111 It's better to open a dedicated issue in Optimum: https://github.com/huggingface/optimum/issues Please provide a code snippet that enables us to reproduce this issue if possible.

ok, I will create a detailed issue once I make sure I'm not missing anything. 
Thanks for your reply

> > 最好在 Optimum 中打开一个专用问题：https://github.com/huggingface/optimum/issues 请提供一个代码片段，以便我们尽可能重现此问题。
> 
> 好的，一旦我确保我没有遗漏任何东西，我将创建一个详细的问题。感谢您的回复

@rakshith111 ,hi、Have you managed to solve this problem? Could you please share it?

> [@AMF777](https://github.com/AMF777) Optimum is the recommended tool to export models to ONNX as `transformers.onnx` is no longer maintained.
> 
> Trying to make the different exported decoders clearer:
> 
> * `decoder_model.onnx` is the decoder without key-value cache. It is usually used in the first generation iteration.
> * `decoder_with_past_model.onnx` is the decoder with key-value cache. It is used after the first generation iteration, when a cache can be set to avoid recomputing intermediate key and values.
> * `decoder_model_merged` is the merge of the two previous decoders, so it will automatically switch between both. That's the one you should use.
> 
> Let me know if that helps.

Thanks, one issue is that is takes a lot of disk space on target device:

1. `decoder_model.onnx` and `decoder_with_past_model.onnx` duplicate many parameters.
2. If the PyTorch model shares the embedding layer with both the encoder and the decoder, then both the `encoder_model.onnx` and `decoder_model.onnx` contains the embedding layer (and in some model, the embedding layer represent the majority of the model size, or close to it).

Is there any way to make the ONNX files smaller?