Add MarianMT to models exportable with ONNX
Author: Maxinho96
Pull request: no

# 🚀 Feature request

Add the support to convert the MarianMT model with the `transformers.onnx` package documented [here](https://huggingface.co/transformers/serialization.html). The conversion now returns `marian () is not supported yet. Only [...] are supported. If you want to support (marian) please propose a PR or open up an issue.`

## Motivation

MarianMT is one of the best translation models in the hub because of the extensive number of pretrained language pairs, but it can be slow for real-time use cases. A conversion to ONNX combined with quantization could significantly improve inference time.


Comments:
Hi!

To support `MarianMT` with ONNX , we'll need to add `MarianOnnxConfig`, similar `BartOnnxConfig`

https://github.com/huggingface/transformers/blob/8bbb53e20b7873ba7f63be70d4d798e0c3568bfa/src/transformers/models/bart/configuration_bart.py#L183

Marian is very similar to BARTs, so adding a similar config as Bart should work. Would you like to open a PR to add this?

cc @michaelbenayoun 

Hi @patil-suraj !

I would be glad to contribute with a PR, but there are some points I don't understand.

For example in BART, the config define these inputs:
```
OrderedDict(
[
    ("input_ids", {0: "batch", 1: "sequence"}),
    ("attention_mask", {0: "batch", 1: "sequence"}),
]
```

but BartModel takes many other inputs in its forward() method:

```
input_ids=None,
attention_mask=None,
decoder_input_ids=None,
decoder_attention_mask=None,
head_mask=None,
decoder_head_mask=None,
cross_attn_head_mask=None,
encoder_outputs=None,
past_key_values=None,
inputs_embeds=None,
decoder_inputs_embeds=None,
use_cache=None,
output_attentions=None,
output_hidden_states=None,
return_dict=None,
```

and the same happens for the outputs.
```
OrderedDict(
[
    ("last_hidden_state", {0: "batch", 1: "sequence"}),
    ("past_keys", {0: "batch", 2: "sequence"}),
    ("encoder_last_hidden_state", {0: "batch", 1: "sequence"}),
]
```
```
last_hidden_state=decoder_outputs.last_hidden_state,
past_key_values=decoder_outputs.past_key_values,
decoder_hidden_states=decoder_outputs.hidden_states,
decoder_attentions=decoder_outputs.attentions,
cross_attentions=decoder_outputs.cross_attentions,
encoder_last_hidden_state=encoder_outputs.last_hidden_state,
encoder_hidden_states=encoder_outputs.hidden_states,
encoder_attentions=encoder_outputs.attentions,
```

What is the criterion to choose the keys to keep?

@michaelbenayoun is the onnx pro, so I will let him answer :) 

@patil-suraj I'm trying with a basic configuration but I get `ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds` when `convert()` is called, because it calls `MarianMTModel.forward()` that doesn't work with the output generated by `MarianTokenizer`. The tokenizer only creates `input_ids` and `attention_mask`, which is fine when `MarianMTModel.generate()` is called, but apparently is not sufficient for `MarianMTModel.forward()`, which is weird. Is this a bug or I need to generate the `decoder_input_ids` in some way inside `generate_dummy_inputs`?

I have gone past that issue by adding the code to automatically generate `decoder_input_ids` in `MarianModel.forward()`

Now I am able to correctly convert the model to ONNX, but when doing inference I have 2 problems:

1. Dynamic batching doesn't work, it only works with the batch_size used during conversion.
2. The model doesn't generate correct sentences, only the first word is correct.

Here's the inference code:
```
tokenizer = MarianTokenizer.from_pretrained(tokenizer_name)

inputs = tokenizer(text=[sample_text] * 2, return_tensors="np", padding=True)

ort_session = ort.InferenceSession(model_name, ort.SessionOptions())

outputs = ort_session.run(["logits"], dict(inputs))
outputs = np.argmax(outputs[0], axis=-1)

words = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(words)
```

Could you please take a look at the PR and see what is causing the issues?
@patil-suraj @michaelbenayoun 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@lewtun and @NielsRogge have also expressed their interest in helping out on ONNX issues so pinging them here

Thanks for the ping @LysandreJik - happy to look into the issue :)

Hey @Maxinho96 do you mind if I continue working on your branch in #13854? This will allow your contribution to be accounted for once we eventually merge the support for MarianMT models :)

> Hey @Maxinho96 do you mind if I continue working on your branch in #13854? This will allow your contribution to be accounted for once we eventually merge the support for MarianMT models :)

Sure, thank you 🙏