Repository: huggingface/optimum

Issue:
MT5 Support 
Big Thanks for the awesome Optimum library, 
MT5 is a Massive Transformers And Supports Many Languages, but it has a slow inference time
for speedup MT5 inference time, We need some post-training methods, and Optimum will help more!

Please Add MT5 Support for Optimum! Thanks


Discussion:
Hi @MiladMolazadeh, thanks for the suggestion! Are you interested in opening a PR to add MT5?

Basically to add the support for a new model there are __3 steps__ to follow and __2 PR__ in two repos will be needed:

__[transformers]__
------------------------
1. Add OnnxConfig in transformers -> which enables the export to ONNX format(in the case of MT5, it shall be the same as the ONNXConfig of T5). 
  * [Detailed guide on adding unsupported models](https://huggingface.co/docs/transformers/serialization#exporting-a-model-for-an-unsupported-architecture)
  * You can find more discussion [here](https://github.com/huggingface/transformers/issues/16308)
2. Register supported tasks of the model into [`transformers.onnx.FeaturesManager`](https://github.com/huggingface/transformers/blob/main/src/transformers/onnx/features.py), by far you will be able to export the model to ONNX / use `ORTModels` for inference / use `ORTQuantizer` for quantization.

__[Optimum]__
------------------------
3. If you are also interested in using `ORTOptimizer` to do some graph optimizations, then you shall open a PR in optimum to add the support of MT5 in [`ORTConfigManager`](https://github.com/huggingface/optimum/blob/main/optimum/onnxruntime/utils.py#L60)

Please feel free to reach out if have any further questions.


Hi, I'm working this morning on the MT5 Onnx Config implementation. It should be added during this week! @JingyaHuang 

> Hi @MiladMolazadeh, thanks for the suggestion! Are you interested in opening a PR to add MT5?
Quick update: `MT5OnnxConfig` has been merged to the main branch. @MiladMolazadeh 

You can now convert an MT5 model to Onnx and use it with 🤗 Optimum.

@JingyaHuang Adding MT5 to `ORTConfigManager` is as simple as adding a key/value pair to the dict you linked below. Or does it involve more stuff? I'm looking for adding it too.
Hi, greate work merging the feature on MT5 models!.

I have tried adding the key/value pair to the dict and I got the next error converting the model to Onnx:
`RuntimeError: Sizes of tensors must match except in dimension 2. Expected size 85 but got size 64 for tensor number 1 in the list.`

It works with BART model so I don´t really know where the problem is. Any idea?

Edit: I have seen this comment in the PR: https://github.com/huggingface/optimum/pull/341#issuecomment-1227202684
and wondered if onnx conversion it's worth it at this moment

Resolving patch:
diff --git a/optimum/onnxruntime/utils.py b/optimum/onnxruntime/utils.py
--- a/optimum/onnxruntime/utils.py
+++ b/optimum/onnxruntime/utils.py
@@ -69,6 +69,7 @@ class ORTConfigManager:
         "electra": ("num_attention_heads", "hidden_size", "bert"),
         "gpt2": ("n_head", "n_embd", "gpt2"),
         "gpt_neo": ("num_heads", "hidden_size", "gpt2"),
+        "mt5": ("num_heads", "d_model", "bart"),
         "marian": ("encoder_attention_heads", "d_model", "bart"),
         "roberta": ("num_attention_heads", "hidden_size", "bert"),
         "xlm-roberta": ("num_attention_heads", "hidden_size", "bert"),


Test patch:
diff --git a/tests/onnxruntime/test_optimization.py b/tests/onnxruntime/test_optimization.py
--- a/tests/onnxruntime/test_optimization.py
+++ b/tests/onnxruntime/test_optimization.py
@@ -35,8 +35,8 @@ class ORTOptimizerTest(unittest.TestCase):
 
     # Contribution note: Please add test models in alphabetical order. Find test models here: https://huggingface.co/hf-internal-testing.
     SUPPORTED_ARCHITECTURES_WITH_MODEL_ID = (
-        (ORTModelForSequenceClassification, "hf-internal-testing/tiny-random-bert"),
         (ORTModelForSequenceClassification, "hf-internal-testing/tiny-random-bart"),
+        (ORTModelForSequenceClassification, "hf-internal-testing/tiny-random-bert"),
         (ORTModelForSequenceClassification, "hf-internal-testing/tiny-random-big_bird"),
         (ORTModelForSequenceClassification, "hf-internal-testing/tiny-random-distilbert"),
         (ORTModelForSequenceClassification, "hf-internal-testing/tiny-random-electra"),
@@ -77,6 +77,7 @@ def test_compare_original_model_with_optimized_model(self, model_cls, model_name
         (ORTModelForSeq2SeqLM, "hf-internal-testing/tiny-random-bart", True),
         (ORTModelForSeq2SeqLM, "hf-internal-testing/tiny-random-marian", False),
         (ORTModelForSeq2SeqLM, "hf-internal-testing/tiny-random-marian", True),
+        (ORTModelForSeq2SeqLM, "hf-internal-testing/tiny-random-onnx-mt5", False),
     )
 
     @parameterized.expand(SUPPORTED_SEQ2SEQ_ARCHITECTURES_WITH_MODEL_ID)