Repository: huggingface/optimum

Issue:
Inference worse with onnxruntime-gpu than native pytorch for seq2seq model
### System Info

```shell
Optimum: 1.4.1.dev0
torch: 1.12.1+cu116
onnx: 1.12.0
onnxruntime-gpu: 1.12.1
python: 3.8.13
CUDA: 11.6
cudnn: 8.4.1
RTX 3090
```


### Who can help?

@JingyaHuang @echarlaix 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I compared inference on GPU of a native torch `Helsinki-NLP/opus-mt-fr-en` model with respect to the optimized onnx model thanks to Optimum library. So, I have defined a fastAPI microservice based on two classes below for GPU both torch and optimized ONNX, repsectively:

```
class Seq2SeqModel:
    tokenizer: Optional[MarianTokenizer]
    model: Optional[MarianMTModel]

    def load_model(self):
        """Loads the model"""
        # model_id="Helsinki-NLP/opus-mt-fr-en"
        model_path = Path("./app/artifacts/HF")
        tokenizer = AutoTokenizer.from_pretrained(model_path)
        model = AutoModelForSeq2SeqLM.from_pretrained(model_path).to("cuda")
        self.tokenizer = tokenizer
        self.model = model

    async def predict(self, input: PredictionInput) -> PredictionOutput:
        """Runs a prediction"""
        if not self.tokenizer or not self.model:
            raise RuntimeError("Model is not loaded")
        tokens = self.tokenizer(input.text, return_tensors="pt").to("cuda")
        translated = self.model.generate(**tokens, num_beams=beam_size)
        return PredictionOutput(translated_text=self.tokenizer.decode(translated[0], skip_special_tokens=True))

class OnnxOptimizedSeq2SeqModel:
    tokenizer: Optional[MarianTokenizer]
    model: Optional[ORTModelForSeq2SeqLM]

    def load_model(self):
        """Loads the model"""
        # model_id="Helsinki-NLP/opus-mt-fr-en"
        onnx_path = Path("./app/artifacts/OL_1")
        tokenizer = AutoTokenizer.from_pretrained(onnx_path)
        optimized_model = ORTModelForSeq2SeqLM.from_pretrained(
            onnx_path,
            encoder_file_name="encoder_model_optimized.onnx",
            decoder_file_name="decoder_model_optimized.onnx",
            decoder_file_with_past_name="decoder_with_past_model_optimized.onnx",
            provider="CUDAExecutionProvider"
        )
        self.tokenizer = tokenizer
        self.model = optimized_model

app = FastAPI()
seq2seq_model = Seq2SeqModel()
onnx_optimized_seq2seq_model = OnnxOptimizedSeq2SeqModel()
beam_size = 3

@app.on_event("startup")
async def startup():
    seq2seq_model.load_model()
    onnx_optimized_seq2seq_model.load_model()

@app.post("/prediction")
async def prediction(
    output: PredictionOutput = Depends(seq2seq_model.predict),
) -> PredictionOutput:
    return output

@app.post("/prediction_onnx_optimized")
async def prediction(
    output: PredictionOutput = Depends(onnx_optimized_seq2seq_model.predict),
) -> PredictionOutput:
    return output
```

### Expected behavior

When load testing the model on my local computer, I was surprised by two things:

1. The performance on GPU of the optimized ONNX model is worse than the native torch (maybe linked to #365 and #396?) :

![GPU_optimized_onnxruntime](https://user-images.githubusercontent.com/77435960/192752820-ceaa7cf4-1cf7-48b6-8734-a219d14fe99d.png)
![GPU_torch](https://user-images.githubusercontent.com/77435960/192752824-0d73c3f2-dadb-4d7f-a380-b8d46e32fed7.png)

2. When running this fastAPI service into a docker image I got the following warning:

`2022-09-28 08:20:21.214094612 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:566 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.`

Does this mean the `CUDAExecutionProvider` is not working even if I set it in?:

```
        optimized_model = ORTModelForSeq2SeqLM.from_pretrained(
            onnx_path,
            encoder_file_name="encoder_model_optimized.onnx",
            decoder_file_name="decoder_model_optimized.onnx",
            decoder_file_with_past_name="decoder_with_past_model_optimized.onnx",
            provider="CUDAExecutionProvider"
        )
```
What could be caused that? I saw in https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html that CUDA 11.6 is not mentionned, could it be this?


Discussion:
Hi @Matthieu-Tinycoaching , thank you for the report! Check my answer for your question 2. in the linked issue in onnxruntime repo.

Could you post the report of `pip list | grep onnx` and `pip list | grep optimum`?

On my end, I run `onnxruntime-gpu=1.12.1`,  CUDA `cuda_11.7.r11.7`.

Maybe try

```
export CUDA_PATH=/usr/local/cuda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.7/lib64
```

with the right path for CUDA.

For question 1, I am currently looking into it, stay tuned.
Answering your question in the issue in onnxruntime repo: Yes, passing `provider="CUDAExecutionProvider"` is necessary, see the Optimum documentation: https://huggingface.co/docs/optimum/main/en/onnxruntime/modeling_ort#optimum.onnxruntime.ORTModel.from_pretrained

Given that you can load the model with CUDAExecutionProvider with my code snippet, I am not sure what goes wrong in yours. It's likely not an issue with CUDA/onnxruntime install. How were your `encoder_model_optimized.onnx` and such saved?
Hi @fxmarty,

Thanks for helping. For the question 2. it seems that my GPU is working properly... Please find below the steps I used to export and optimize ONNX models:

```
## Convert native model to ONNX

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer
from pathlib import Path

model_id="Helsinki-NLP/opus-mt-fr-en"
onnx_path = Path("/home/matthieu/Deployment/ONNX_runtime/GPU/V3/Optimum/opus-mt-fr-en/onnx1.12.0_onxxruntime-gpu1.12.1/OL_1")

# load vanilla transformers and convert to onnx
model = ORTModelForSeq2SeqLM.from_pretrained(model_id, from_transformers=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# save onnx checkpoint and tokenizer
model.save_pretrained(onnx_path)
tokenizer.save_pretrained(onnx_path)
```

```
## Optimize the model for GPU using ORTOptimizer

from optimum.onnxruntime import ORTOptimizer
from optimum.onnxruntime.configuration import OptimizationConfig

# Create ORTOptimizer
optimizer = ORTOptimizer.from_pretrained(model)

# Define the optimization strategy by creating the appropriate configuration
optimization_config = OptimizationConfig(optimization_level=1,
                                        optimize_for_gpu=True,
                                        fp16=True
                                        )

# Optimize the model
optimizer.optimize(save_dir=onnx_path, optimization_config=optimization_config)
```

Please find below the output of the following commands:

```
pip list | grep onnx
onnx                      1.12.0
onnxruntime-gpu           1.12.1
onnxruntime-tools         1.7.0
```

```
pip list | grep optimum
optimum                   1.4.1.dev0
```

1 / Did I do something wrong? Could you reproduce my results using fastAPI?

2/ Is there a way to get the prediction output from combination ONNX InferenceSession with the 3 ONNX optimized model instead of using Optimum? So, I could check with another deployment framework than fastAPI?

3 / How long does it takes to export the ONNX model from torch? I try to convert the model from my first code block above by adding `provider="CUDAExecutionProvider"` flag an it has still run from 8 minutes...
Concerning 3/: The `from_transformers` argument is used in case we load the model from a transformers checkpoint. In case your model is already stored as onnx, you don't need to pass the argument, or pass `from_transformers=False`. You can refer to https://huggingface.co/docs/optimum/main/en/onnxruntime/modeling_ort#optimum.onnxruntime.ORTModel.from_pretrained for details. Having the conversion run for 8 minutes is not normal (unless you are downloading a model not cached from the Hub).

2/: not sure what you mean. Do you mean, could you use the `encoder_model_optimized.onnx`, `decoder_model_optimized.onnx` independently of Optimum?

In the first place, you don't get anymore the warning `2022-09-28 08:20:21.214094612 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:566 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.`?
Hi @fxmarty,

OK I regenerate all onnx models based on this script:

```
import onnxruntime
from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import MarianTokenizer, pipeline
from pathlib import Path
from optimum.onnxruntime import ORTOptimizer
from optimum.onnxruntime.configuration import OptimizationConfig

onnx_path = Path("/home/matthieu/Deployment/ONNX_runtime/GPU/V3/Optimum/opus-mt-fr-en/github_issue_onnxruntime")

tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-fr-en")

options = onnxruntime.SessionOptions()
options.log_severity_level = 0  # verbose, to see which execution provider is used

#### 1. Convert an opus-mt model to ONNX for inference ####

# load vanilla transformers and convert to onnx
ort_model = ORTModelForSeq2SeqLM.from_pretrained(
    "Helsinki-NLP/opus-mt-fr-en",
    from_transformers=True,
    provider="CUDAExecutionProvider",
    session_options=options,
)

print(ort_model.providers)

# save onnx checkpoint and tokenizer
ort_model.save_pretrained(onnx_path)
tokenizer.save_pretrained(onnx_path)

#### 2. Optimize model for GPU using ORTOptimizer ####

# Create ORTOptimizer
optimizer = ORTOptimizer.from_pretrained(ort_model)

# Define the optimization strategy by creating the appropriate configuration
optimization_config = OptimizationConfig(optimization_level=1,
                                        optimize_for_gpu=True,
                                        fp16=True
                                        )

# Optimize the model
optimizer.optimize(save_dir=onnx_path, optimization_config=optimization_config)
```
And I got both triplets (`encoder_model.onnx`, `decoder_with_past_model.onnx`, `decoder_model.onnx`) and (`encoder_model_optimized.onnx`, `decoder_with_past_model_optimized.onnx`, `decoder_model_optimized.onnx`) in the `onnx_path` folder with GPU optimization.

Then, I tried to compare load performance between native torch model and onnx models (w/wo optimizations) using fastAPI:

```
import os
from typing import List, Optional, Tuple
from fastapi import FastAPI, Depends
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, pipeline
from transformers.pipelines.text2text_generation import TranslationPipeline
from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers.models.marian.tokenization_marian import MarianTokenizer
from transformers.models.marian.modeling_marian import MarianMTModel
from pathlib import Path
# from easynmt import EasyNMT

class PredictionInput(BaseModel):
    text: str

class PredictionOutput(BaseModel):
    translated_text: str

class Seq2SeqModel:
    tokenizer: Optional[MarianTokenizer]
    model: Optional[MarianMTModel]

    def load_model(self):
        """Loads the model"""
        # model_id="Helsinki-NLP/opus-mt-fr-en"
        model_path = Path("./app/artifacts/HF")
        tokenizer = AutoTokenizer.from_pretrained(model_path)
        model = AutoModelForSeq2SeqLM.from_pretrained(model_path).to("cuda")
        self.tokenizer = tokenizer
        self.model = model

    async def predict(self, input: PredictionInput) -> PredictionOutput:
        """Runs a prediction"""
        if not self.tokenizer or not self.model:
            raise RuntimeError("Model is not loaded")
        tokens = self.tokenizer(input.text, return_tensors="pt").to("cuda")
        translated = self.model.generate(**tokens, num_beams=beam_size)
        return PredictionOutput(translated_text=self.tokenizer.decode(translated[0], skip_special_tokens=True))

class OnnxSeq2SeqModel:
    tokenizer: Optional[MarianTokenizer]
    model: Optional[ORTModelForSeq2SeqLM]

    def load_model(self):
        """Loads the model"""
        # model_id="Helsinki-NLP/opus-mt-fr-en"
        onnx_path = Path("./app/artifacts/OL_1")
        tokenizer = AutoTokenizer.from_pretrained(onnx_path)
        optimized_model = ORTModelForSeq2SeqLM.from_pretrained(
            onnx_path,
            encoder_file_name="encoder_model.onnx",
            decoder_file_name="decoder_model.onnx",
            decoder_file_with_past_name="decoder_with_past_model.onnx",
            provider="CUDAExecutionProvider"
        )
        self.tokenizer = tokenizer
        self.model = optimized_model

    async def predict(self, input: PredictionInput) -> PredictionOutput:
        """Runs a prediction"""
        if not self.tokenizer or not self.model:
            raise RuntimeError("Model is not loaded")
        tokens = self.tokenizer(input.text, return_tensors="pt").to("cuda")
        translated = self.model.generate(**tokens, num_beams=beam_size)
        return PredictionOutput(translated_text=self.tokenizer.decode(translated[0], skip_special_tokens=True))

class OnnxOptimizedSeq2SeqModel:
    tokenizer: Optional[MarianTokenizer]
    model: Optional[ORTModelForSeq2SeqLM]

    def load_model(self):
        """Loads the model"""
        # model_id="Helsinki-NLP/opus-mt-fr-en"
        onnx_path = Path("./app/artifacts/OL_1")
        tokenizer = AutoTokenizer.from_pretrained(onnx_path)
        optimized_model = ORTModelForSeq2SeqLM.from_pretrained(
            onnx_path,
            encoder_file_name="encoder_model_optimized.onnx",
            decoder_file_name="decoder_model_optimized.onnx",
            decoder_file_with_past_name="decoder_with_past_model_optimized.onnx",
            provider="CUDAExecutionProvider"
        )
        self.tokenizer = tokenizer
        self.model = optimized_model

    async def predict(self, input: PredictionInput) -> PredictionOutput:
        """Runs a prediction"""
        if not self.tokenizer or not self.model:
            raise RuntimeError("Model is not loaded")
        tokens = self.tokenizer(input.text, return_tensors="pt").to("cuda")
        translated = self.model.generate(**tokens, num_beams=beam_size)
        return PredictionOutput(translated_text=self.tokenizer.decode(translated[0], skip_special_tokens=True))

class OnnxOptimizedSeq2SeqPipeline:
    pipeline: Optional[TranslationPipeline]

    def load_pipeline(self):
        """Loads the pipeline"""
        # model_id="Helsinki-NLP/opus-mt-fr-en"
        onnx_path = Path("./app/artifacts/OL_1")
        tokenizer = AutoTokenizer.from_pretrained(onnx_path)
        optimized_model = ORTModelForSeq2SeqLM.from_pretrained(
            onnx_path,
            encoder_file_name="encoder_model_optimized.onnx",
            decoder_file_name="decoder_model_optimized.onnx",
            decoder_file_with_past_name="decoder_with_past_model_optimized.onnx",
            provider="CUDAExecutionProvider"
        )
        self.pipeline = pipeline("translation_fr_to_en", model=optimized_model, tokenizer=tokenizer, num_beams=beam_size, device=0)

    async def predict(self, input: PredictionInput) -> PredictionOutput:
        """Runs a prediction"""
        if not self.pipeline:
            raise RuntimeError("Pipeline is not loaded")
        translated = self.pipeline(input.text)
        return PredictionOutput(translated_text=translated[0]['translation_text'])

app = FastAPI()
seq2seq_model = Seq2SeqModel()
onnx_seq2seq_model = OnnxSeq2SeqModel()
onnx_optimized_seq2seq_model = OnnxOptimizedSeq2SeqModel()
onnx_optimized_seq2seq_pipeline = OnnxOptimizedSeq2SeqPipeline()
beam_size = 3

@app.on_event("startup")
async def startup():
    seq2seq_model.load_model()
    onnx_seq2seq_model.load_model()
    onnx_optimized_seq2seq_model.load_model()
    onnx_optimized_seq2seq_pipeline.load_pipeline()

@app.post("/prediction")
async def prediction(
    output: PredictionOutput = Depends(seq2seq_model.predict),
) -> PredictionOutput:
    return output

@app.post("/prediction_onnx")
async def prediction(
    output: PredictionOutput = Depends(onnx_seq2seq_model.predict),
) -> PredictionOutput:
    return output

@app.post("/prediction_onnx_optimized")
async def prediction(
    output: PredictionOutput = Depends(onnx_optimized_seq2seq_model.predict),
) -> PredictionOutput:
    return output

@app.post("/prediction_onnx_optimized_pipeline")
async def prediction(
    output: PredictionOutput = Depends(onnx_optimized_seq2seq_pipeline.predict),
) -> PredictionOutput:
    return output
```

![GPU_onn_optimized_pipeline](https://user-images.githubusercontent.com/77435960/192833078-ce55da6f-24df-4f66-98b3-08aa58cf971e.png)
![GPU_onnx_optimized](https://user-images.githubusercontent.com/77435960/192833085-3a2ba3a0-7f4e-4efd-939b-79abb65228b0.png)
![GPU_onnx](https://user-images.githubusercontent.com/77435960/192833086-9b918bae-b5ab-4c21-aef9-7878cc8f7e9a.png)
![GPU_native_torch](https://user-images.githubusercontent.com/77435960/192833090-c3dc4826-3283-4871-8bd4-a7d3dcb2389e.png)

1 / And best results are the last one for the native pytorch model. Could you reproduce these misperformance of onnx models (with or wihtout optimization) regarding native model?

2/ Do you mean, could you use the encoder_model_optimized.onnx, decoder_model_optimized.onnx independently of Optimum?

Yes, is it possible to sequentially call the triplet of onnx models?

3 / `In the first place, you don't get anymore the warning 2022-09-28 08:20:21.214094612 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:566 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.?`

I'll test tomorrow with docker image, but since performance are still worse with onnx, I suppose it stil there...
Hi @Matthieu-Tinycoaching,

For your concern about the inference speed, it could be caused by the overhead on data copying between CPU and GPU, as ONNX Runtime put inputs and outputs on the CPU by default. I just merged the PR which add the [IO binding support](https://github.com/huggingface/optimum/pull/421) to avoid the issue, do you want to test with our main branch(no code change as `use_io_binding=True` by default) to see if this can solve the problem?

You can build optimum from source with
```
python -m pip install git+https://github.com/huggingface/optimum.git#egg=optimum[onnxruntime-gpu]
```



In my case on T5 seq2seq model,

| model | use_io_binding | num_beams | average time for 10 loops [sec] |
| :---: | :---: | :---: | :---: |
| pytorch |  | 1 | 0.89 |
| onnx | True | 1 | 0.54 |
| onnx | False | 1 | 0.77 |
| pytorch |  | 5 | 1.17 |
| onnx | True | 5 | 3.17 |
| onnx | False | 5 | 2.13 |

When `num_beams > 1`, the average time is even slower than before. (`use_io_binding=False`)  
All results was generated by using these functions:
```python
def load_pytorch(saved_path, device="cuda:0"):
    tokenizer = T5Tokenizer.from_pretrained("google/mt5-base")
    net = MT5ForConditionalGeneration.from_pretrained("google/mt5-base").to(device)
    net.eval()
    state_dict = torch.load(saved_path, map_location=device)["model_state_dict"]
    net.load_state_dict(state_dict)
    return tokenizer, net


def load_onnx(saved_path, device="cuda:0"):
    tokenizer = T5Tokenizer.from_pretrained(saved_path)
    net = ORTModelForSeq2SeqLM.from_pretrained(
        saved_path,
        encoder_file_name="encoder_model.onnx",
        decoder_file_name="decoder_model.onnx",
        decoder_with_past_file_name="decoder_with_past_model.onnx",
        use_io_binding=False,
        provider="CUDAExecutionProvider",
    ).to(device)
    return tokenizer, net


def inference(tokenizer, net, sentence, device="cuda:0"):
    input_ids = tokenizer.encode(sentence)
    input_ids = torch.tensor(input_ids).to(dtype=torch.int64, device=device)
    input_ids = torch.unsqueeze(input_ids, dim=0)
    output_ids = net.generate(input_ids, max_length=256, num_beams=5)
    translated = tokenizer.decode(output_ids[0], skip_special_tokens=True)
    return translated
```



Hi @soocheolnoh,

Thanks for testing the IO binding feature and sharing your results! 

I have done a quick test with beam search `num_beams=5` on the checkpoint `facebook/m2m100_418M`(the snippet was borrowed from #365 )
```python
# -*- coding: utf-8 -*-

import logging
import time

import torch
from transformers import AutoTokenizer
from tqdm import tqdm


logging.basicConfig(level=logging.INFO)


model_checkpoint = "facebook/m2m100_418M"

loop = 100


chinese_text = "机器学习是人工智能的一个分支。人工智能的研究历史有着一条从以“推理”为重点，到以“知识”为重点，再到以“学习”为重点的自然、清晰的脉络。显然，机器学习是实现人工智能的一个途径，即以机器学习为手段解决人工智能中的问题。机器学习在近30多年已发展为一门多领域交叉学科，涉及概率论、统计学、逼近论、凸分析（英语：Convex analysis）、计算复杂性理论等多门学科。机器学习理论主要是设计和分析一些让计算机可以自动“学习”的算法。机器学习算法是一类从数据中自动分析获得规律，并利用规律对未知数据进行预测的算法。因为学习算法中涉及了大量的统计学理论，机器学习与推断统计学联系尤为密切，也被称为统计学习理论。算法设计方面，机器学习理论关注可以实现的，行之有效的学习算法。很多推论问题属于无程序可循难度，所以部分的机器学习研究是开发容易处理的近似算法。机器学习已广泛应用于数据挖掘、计算机视觉、自然语言处理、生物特征识别、搜索引擎、医学诊断、检测信用卡欺诈、证券市场分析、DNA序列测序、语音和手写识别、战略游戏和机器人等领域"
logging.info(f"chinese_text is {chinese_text}")
logging.info(f"chinese_text length is {len(chinese_text)}")

device = torch.device("cuda:0")
logging.info(f"This test will use device: {device}")


def get_transformer_model():
    from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

    model = M2M100ForConditionalGeneration.from_pretrained(model_checkpoint).to(device)
    tokenizer = M2M100Tokenizer.from_pretrained(model_checkpoint)
    return (model, tokenizer)


def get_optimum_onnx_model():
    from optimum.onnxruntime import ORTModelForSeq2SeqLM

    model = ORTModelForSeq2SeqLM.from_pretrained(model_checkpoint, from_transformers=True).to(device)
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
    return (model, tokenizer)


def translate():

    # (model, tokenizer) = get_optimum_onnx_model()
    (model, tokenizer) = get_transformer_model()

    # Warm-up
    for i in range(10):
        encoded_zh = tokenizer(chinese_text, return_tensors="pt").to(device)
        generated_tokens = model.generate(
            **encoded_zh,
            forced_bos_token_id=tokenizer.get_lang_id("en"),
            max_length=256,
            num_beams=5,
        )
        result = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
        logging.debug(f"#{i}: {result}")

    start = time.time()
    for i in tqdm(range(loop)):
        encoded_zh = tokenizer(chinese_text, return_tensors="pt").to(device)
        generated_tokens = model.generate(
            **encoded_zh,
            forced_bos_token_id=tokenizer.get_lang_id("en"),
            max_length=256,
            num_beams=5,
        )
    end = time.time()
    total_time = end - start

    logging.info(f"total: {total_time}")
    logging.info(f"loop: {loop}")
    logging.info(f"avg(s): {total_time / loop}")
    logging.info(f"throughput(translation/s): {loop / total_time}")


if __name__ == "__main__":
    translate()
```

Here are the results that I got with a T4, ORTModel with IO binding V.S. PyTorch:
|   | PyTorch | Optimum(w/. io)  |
|---|---|---|
| total(s)  | 442.00  | 323.40  |
| loop | 100  | 100  | 
| avg(s) |  4.420 | 3.234  |
| throughput(translation/s) | 0.2262  | 0.3092  |

Can you test the snippet to see if you can get something similar on your end, or share your entire script so that I can try to reproduce your experiment?
@JingyaHuang Thanks for your response!

I'd try your script and I got similar results. (pytorch: 4.074 sec, optimum: 2.7825 sec for average time)
And I also ran the following script for mT5 model: (currently I'm using)

```python
# -*- coding: utf-8 -*-

import logging
import time

import torch
from tqdm import tqdm
from transformers import AutoTokenizer

logging.basicConfig(level=logging.INFO)


# model_checkpoint = "facebook/m2m100_418M"
model_checkpoint = "K024/mt5-zh-ja-en-trimmed"

loop = 100


chinese_text = "zh2en: 机器学习是人工智能的一个分支。人工智能的研究历史有着一条从以“推理”为重点，到以“知识”为重点，再到以“学习”为重点的自然、清晰的脉络。显然，机器学习是实现人工智能的一个途径，即以机器学习为手段解决人工智能中的问题。机器学习在近30多年已发展为一门多领域交叉学科，涉及概率论、统计学、逼近论、凸分析（英语：Convex analysis）、计算复杂性理论等多门学科。机器学习理论主要是设计和分析一些让计算机可以自动“学习”的算法。机器学习算法是一类从数据中自动分析获得规律，并利用规律对未知数据进行预测的算法。因为学习算法中涉及了大量的统计学理论，机器学习与推断统计学联系尤为密切，也被称为统计学习理论。算法设计方面，机器学习理论关注可以实现的，行之有效的学习算法。很多推论问题属于无程序可循难度，所以部分的机器学习研究是开发容易处理的近似算法。机器学习已广泛应用于数据挖掘、计算机视觉、自然语言处理、生物特征识别、搜索引擎、医学诊断、检测信用卡欺诈、证券市场分析、DNA序列测序、语音和手写识别、战略游戏和机器人等领域"
logging.info(f"chinese_text is {chinese_text}")
logging.info(f"chinese_text length is {len(chinese_text)}")

device = torch.device("cuda:0")
logging.info(f"This test will use device: {device}")


def get_transformer_model():
    # from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

    # model = M2M100ForConditionalGeneration.from_pretrained(model_checkpoint).to(device)
    # tokenizer = M2M100Tokenizer.from_pretrained(model_checkpoint)

    from transformers import MT5ForConditionalGeneration, T5Tokenizer

    model = MT5ForConditionalGeneration.from_pretrained(model_checkpoint).to(device)
    tokenizer = T5Tokenizer.from_pretrained(model_checkpoint)
    return (model, tokenizer)


def get_optimum_onnx_model():
    from optimum.onnxruntime import ORTModelForSeq2SeqLM
    from transformers import T5Tokenizer

    model = ORTModelForSeq2SeqLM.from_pretrained(model_checkpoint, from_transformers=True).to(device)
    tokenizer = T5Tokenizer.from_pretrained(model_checkpoint)
    return (model, tokenizer)


def translate():

    (model, tokenizer) = get_optimum_onnx_model()
    # (model, tokenizer) = get_transformer_model()

    # Warm-up
    for i in range(10):
        encoded_zh = tokenizer(chinese_text, return_tensors="pt").to(device)
        generated_tokens = model.generate(
            **encoded_zh,
            # forced_bos_token_id=tokenizer.get_lang_id("en"),
            max_length=256,
            num_beams=5,
        )
        result = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
        logging.debug(f"#{i}: {result}")

    start = time.time()
    for i in tqdm(range(loop)):
        encoded_zh = tokenizer(chinese_text, return_tensors="pt").to(device)
        generated_tokens = model.generate(
            **encoded_zh,
            # forced_bos_token_id=tokenizer.get_lang_id("en"),
            max_length=256,
            num_beams=5,
        )
    end = time.time()
    total_time = end - start

    logging.info(f"total: {total_time}")
    logging.info(f"loop: {loop}")
    logging.info(f"avg(s): {total_time / loop}")
    logging.info(f"throughput(translation/s): {loop / total_time}")


if __name__ == "__main__":
    translate()
```

The result is (done by V100):
| | pytorch | optimum |
| :---: | :---: | :---: |
| total | 289.4229 | 298.1077 |
| loop | 100 | 100 |
| avg(s) | 2.8942 | 2.9811 |
| throughput | 0.3455 | 0.3354 |

When I ran the pytorch model, I had no warnings, but in optimum I got the following warnings:
```
UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.

TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_mask.shape[1] < attention_mask.shape[1]:
```
 
Also the translation results are different like (optimum result seems to be strange):
- `pytorch` Machine learning is a branch of artificial intelligence. The history of artificial intelligence has a natural, clear line from "rationality" to "knowledge" to "learning" which is clearly a way to achieve artificial intelligence. Machine learning has been developed for nearly 30 years as a cross-disciplinary discipline involving probability theory, statistics, proximity theory, convex analysis, and computational complexity theory. Machine learning theory primarily designs and analyzes algorithms that allow computers to "learn" automatically.
- `optimum` Machine learning is a branch of artificial intelligence. It learning has been  learning has been   learning has learning is a  learning Machine  learning algorithms are learning algorithms are algorithms that automatically analyze patterns from data and use patterns to predict unknown data. Machine  algorithms are a learning algorithms,  learning algorithms are algorithms that are   learning is  learning  learning is learning is learning is learning is learning is learning is learning  learning  learning  learning is Machine learning Machine learning is a type  Machine learning learning algorithms are algorithm learning algorithms   learning  learning Machine learning is learning Machine Machine learning learning  learning  learning algorithms are algorithm learning algorithms  learning algorithms are algorithms that automatically analyze patterns from data and use patterns to predict unknown data. Since learning is a class of algorithms   which learning algorithms that are  learning is   learning algorithms are learning algorithms are algorithms  learning algorithms, learning algorithms are algorithms  learning algorithms  learning algorithm

I think the reason of the different between the average times could be just the results' different (because of the length of output tokens?) or other reasons. But I'm not sure why the result using opimum is strange.
Hi @soocheolnoh, thanks for testing.

From my side, for the mt5 model, the generated results are different w/. V.S. w/o. IO binding, which is not normal as IO Binding is not supposed to change the result(only the place to put the data should be different), it might be a bug on the post-processing of the outputs. I will take a closer look, and fix the beam search ASAP.



Resolving patch:
diff --git a/optimum/onnxruntime/modeling_ort.py b/optimum/onnxruntime/modeling_ort.py
--- a/optimum/onnxruntime/modeling_ort.py
+++ b/optimum/onnxruntime/modeling_ort.py
@@ -477,7 +477,7 @@ def prepare_output_buffer(self, batch_size, sequence_length, hidden_size, output
         torch_type = TypeHelper.ort_type_to_torch_type(ort_type)
 
         output_shape = (batch_size, sequence_length, hidden_size)
-        output_buffer = torch.empty(np.prod(output_shape), dtype=torch_type, device=self.device)
+        output_buffer = torch.empty(np.prod(output_shape), dtype=torch_type, device=self.device).contiguous()
 
         return output_shape, output_buffer
 
@@ -490,6 +490,7 @@ def prepare_io_binding(
         io_binding = self.model.io_binding()
 
         # bind input ids
+        input_ids = input_ids.contiguous()
         io_binding.bind_input(
             "input_ids",
             input_ids.device.type,
@@ -499,6 +500,7 @@ def prepare_io_binding(
             input_ids.data_ptr(),
         )
         # bind attention mask
+        attention_mask = attention_mask.contiguous()
         io_binding.bind_input(
             "attention_mask",
             attention_mask.device.type,
@@ -510,6 +512,7 @@ def prepare_io_binding(
 
         if token_type_ids is not None:
             # bind token type ids
+            token_type_ids = token_type_ids.contiguous()
             io_binding.bind_input(
                 "token_type_ids",
                 token_type_ids.device.type,
@@ -648,7 +651,7 @@ def prepare_logits_buffer(self, batch_size, sequence_length, output_name: str):
         torch_type = TypeHelper.ort_type_to_torch_type(ort_type)
 
         logits_shape = (batch_size, sequence_length)
-        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device)
+        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device).contiguous()
 
         return logits_shape, logits_buffer
 
@@ -661,6 +664,7 @@ def prepare_io_binding(
         io_binding = self.model.io_binding()
 
         # bind input ids
+        input_ids = input_ids.contiguous()
         io_binding.bind_input(
             "input_ids",
             input_ids.device.type,
@@ -670,6 +674,7 @@ def prepare_io_binding(
             input_ids.data_ptr(),
         )
         # bind attention mask
+        attention_mask = attention_mask.contiguous()
         io_binding.bind_input(
             "attention_mask",
             attention_mask.device.type,
@@ -681,6 +686,7 @@ def prepare_io_binding(
 
         if token_type_ids is not None:
             # bind token type ids
+            token_type_ids = token_type_ids.contiguous()
             io_binding.bind_input(
                 "token_type_ids",
                 token_type_ids.device.type,
@@ -848,7 +854,7 @@ def prepare_logits_buffer(self, batch_size, num_labels):
         torch_type = TypeHelper.ort_type_to_torch_type(ort_type)
 
         logits_shape = (batch_size, num_labels)
-        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device)
+        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device).contiguous()
 
         return logits_shape, logits_buffer
 
@@ -861,6 +867,7 @@ def prepare_io_binding(
         io_binding = self.model.io_binding()
 
         # bind input ids
+        input_ids = input_ids.contiguous()
         io_binding.bind_input(
             "input_ids",
             input_ids.device.type,
@@ -870,6 +877,7 @@ def prepare_io_binding(
             input_ids.data_ptr(),
         )
         # bind attention mask
+        attention_mask = attention_mask.contiguous()
         io_binding.bind_input(
             "attention_mask",
             attention_mask.device.type,
@@ -881,6 +889,7 @@ def prepare_io_binding(
 
         if token_type_ids is not None:
             # bind token type ids
+            token_type_ids = token_type_ids.contiguous()
             io_binding.bind_input(
                 "token_type_ids",
                 token_type_ids.device.type,
@@ -1017,7 +1026,7 @@ def prepare_logits_buffer(self, batch_size, sequence_length, num_labels):
         torch_type = TypeHelper.ort_type_to_torch_type(ort_type)
 
         logits_shape = (batch_size, sequence_length, num_labels)
-        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device)
+        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device).contiguous()
 
         return logits_shape, logits_buffer
 
@@ -1030,6 +1039,7 @@ def prepare_io_binding(
         io_binding = self.model.io_binding()
 
         # bind input ids
+        input_ids = input_ids.contiguous()
         io_binding.bind_input(
             "input_ids",
             input_ids.device.type,
@@ -1039,6 +1049,7 @@ def prepare_io_binding(
             input_ids.data_ptr(),
         )
         # bind attention mask
+        attention_mask = attention_mask.contiguous()
         io_binding.bind_input(
             "attention_mask",
             attention_mask.device.type,
@@ -1050,6 +1061,7 @@ def prepare_io_binding(
 
         if token_type_ids is not None:
             # bind token type ids
+            token_type_ids = token_type_ids.contiguous()
             io_binding.bind_input(
                 "token_type_ids",
                 token_type_ids.device.type,
@@ -1181,7 +1193,7 @@ def prepare_logits_buffer(self, batch_size, num_choices):
         torch_type = TypeHelper.ort_type_to_torch_type(ort_type)
 
         logits_shape = (batch_size, num_choices)
-        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device)
+        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device).contiguous()
 
         return logits_shape, logits_buffer
 
@@ -1194,6 +1206,7 @@ def prepare_io_binding(
         io_binding = self.model.io_binding()
 
         # bind input ids
+        input_ids = input_ids.contiguous()
         io_binding.bind_input(
             "input_ids",
             input_ids.device.type,
@@ -1203,6 +1216,7 @@ def prepare_io_binding(
             input_ids.data_ptr(),
         )
         # bind attention mask
+        attention_mask = attention_mask.contiguous()
         io_binding.bind_input(
             "attention_mask",
             attention_mask.device.type,
@@ -1214,6 +1228,7 @@ def prepare_io_binding(
 
         if token_type_ids is not None:
             # bind token type ids
+            token_type_ids = token_type_ids.contiguous()
             io_binding.bind_input(
                 "token_type_ids",
                 token_type_ids.device.type,
@@ -1355,7 +1370,7 @@ def prepare_logits_buffer(self, batch_size, sequence_length):
         torch_type = TypeHelper.ort_type_to_torch_type(ort_type)
 
         logits_shape = (batch_size, sequence_length, self.config.vocab_size)
-        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device)
+        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device).contiguous()
 
         return logits_shape, logits_buffer
 
@@ -1367,6 +1382,7 @@ def prepare_io_binding(
         io_binding = self.model.io_binding()
 
         # bind input_ids
+        input_ids = input_ids.contiguous()
         io_binding.bind_input(
             "input_ids",
             input_ids.device.type,
@@ -1376,6 +1392,7 @@ def prepare_io_binding(
             input_ids.data_ptr(),
         )
         # bind attention mask
+        attention_mask = attention_mask.contiguous()
         io_binding.bind_input(
             "attention_mask",
             attention_mask.device.type,
@@ -1530,7 +1547,7 @@ def prepare_logits_buffer(self, batch_size):
         torch_type = TypeHelper.ort_type_to_torch_type(ort_type)
 
         logits_shape = (batch_size, self.config.num_labels)
-        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device)
+        logits_buffer = torch.empty(np.prod(logits_shape), dtype=torch_type, device=self.device).contiguous()
 
         return logits_shape, logits_buffer
 
@@ -1541,6 +1558,7 @@ def prepare_io_binding(
         io_binding = self.model.io_binding()
 
         # bind pixel values
+        pixel_values = pixel_values.contiguous()
         io_binding.bind_input(
             "pixel_values",
             pixel_values.device.type,
diff --git a/optimum/onnxruntime/modeling_seq2seq.py b/optimum/onnxruntime/modeling_seq2seq.py
--- a/optimum/onnxruntime/modeling_seq2seq.py
+++ b/optimum/onnxruntime/modeling_seq2seq.py
@@ -576,7 +576,7 @@ def prepare_output_buffer(self, batch_size, sequence_length):
 
         hidden_size = getattr(self.config, ORTConfigManager.get_hidden_size_name(self.config.model_type))
         output_shape = (batch_size, sequence_length, hidden_size)
-        output_buffer = torch.empty(np.prod(output_shape), dtype=torch_type, device=self._device)
+        output_buffer = torch.empty(np.prod(output_shape), dtype=torch_type, device=self._device).contiguous()
 
         return output_shape, output_buffer
 
@@ -588,6 +588,7 @@ def prepare_io_binding(
         io_binding = self.session.io_binding()
 
         # bind input ids
+        input_ids = input_ids.contiguous()
         io_binding.bind_input(
             "input_ids",
             input_ids.device.type,
@@ -598,6 +599,7 @@ def prepare_io_binding(
         )
         if "attention_mask" in self.input_names:
             # bind attention mask
+            attention_mask = attention_mask.contiguous()
             io_binding.bind_input(
                 "attention_mask",
                 attention_mask.device.type,
@@ -707,10 +709,10 @@ def prepare_output_buffer(
         torch_type = TypeHelper.ort_type_to_torch_type(ort_type)
         if output_name == "loss":
             output_shape = (1,)
-            output_buffer = torch.empty(1, dtype=torch_type, device=self._device)
+            output_buffer = torch.empty(1, dtype=torch_type, device=self._device).contiguous()
         elif output_name == "logits":
             output_shape = (batch_size, sequence_length, self.config.vocab_size)
-            output_buffer = torch.empty(np.prod(output_shape), dtype=torch_type, device=self._device)
+            output_buffer = torch.empty(np.prod(output_shape), dtype=torch_type, device=self._device).contiguous()
         elif "key_values" in output_name:
             num_heads = getattr(self.config, ORTConfigManager.get_num_heads_name(self.config.model_type))
             hidden_size = getattr(self.config, ORTConfigManager.get_hidden_size_name(self.config.model_type))
@@ -722,7 +724,7 @@ def prepare_output_buffer(
             else:
                 output_shape = (batch_size, num_heads, encoder_sequence_length, embed_size_per_head)
 
-            output_buffer = torch.empty(np.prod(output_shape), dtype=torch_type, device=self._device)
+            output_buffer = torch.empty(np.prod(output_shape), dtype=torch_type, device=self._device).contiguous()
 
         return output_shape, output_buffer
 
@@ -737,6 +739,7 @@ def prepare_io_binding(
         io_binding = self.session.io_binding()
 
         # bind input ids
+        input_ids = input_ids.contiguous()
         io_binding.bind_input(
             "input_ids",
             input_ids.device.type,
@@ -747,6 +750,7 @@ def prepare_io_binding(
         )
 
         # bind encoder attention mask
+        encoder_attention_mask = encoder_attention_mask.contiguous()
         io_binding.bind_input(
             "encoder_attention_mask",
             encoder_attention_mask.device.type,
@@ -758,6 +762,7 @@ def prepare_io_binding(
 
         # bind encoder hidden states
         if "encoder_hidden_states" in self.session_input_names:
+            encoder_hidden_states = encoder_hidden_states.contiguous()
             io_binding.bind_input(
                 "encoder_hidden_states",
                 encoder_hidden_states.device.type,
@@ -770,6 +775,7 @@ def prepare_io_binding(
         # bind past key values
         if past_key_values is not None:
             for input_name, past_key_value in zip(self.key_value_input_names, past_key_values):
+                past_key_value = past_key_value.contiguous()
                 io_binding.bind_input(
                     input_name,
                     past_key_value.device.type,
@@ -781,6 +787,7 @@ def prepare_io_binding(
 
         # bind labels
         if "labels" in self.session_input_names:
+            labels = labels.contiguous()
             io_binding.bind_input(
                 "labels",
                 labels.device.type,


Test patch:
diff --git a/tests/onnxruntime/test_modeling.py b/tests/onnxruntime/test_modeling.py
--- a/tests/onnxruntime/test_modeling.py
+++ b/tests/onnxruntime/test_modeling.py
@@ -566,7 +566,7 @@ def test_compare_to_io_binding(self, model_arch):
         io_model = ORTModelForQuestionAnswering.from_pretrained(model_id, from_transformers=True, use_io_binding=True)
 
         tokenizer = get_preprocessor(model_id)
-        tokens = tokenizer("This is a sample output", return_tensors="pt")
+        tokens = tokenizer(["This is a sample output"] * 2, return_tensors="pt")
         onnx_outputs = onnx_model(**tokens)
         io_outputs = io_model(**tokens)
 
@@ -700,7 +700,7 @@ def test_compare_to_io_binding(self, model_arch):
         )
 
         tokenizer = get_preprocessor(model_id)
-        tokens = tokenizer("This is a sample output", return_tensors="pt")
+        tokens = tokenizer(["This is a sample output"] * 2, return_tensors="pt")
         onnx_outputs = onnx_model(**tokens)
         io_outputs = io_model(**tokens)
 
@@ -809,7 +809,7 @@ def test_compare_to_io_binding(self, model_arch):
         )
 
         tokenizer = get_preprocessor(model_id)
-        tokens = tokenizer("This is a sample output", return_tensors="pt")
+        tokens = tokenizer(["This is a sample output"] * 2, return_tensors="pt")
         onnx_outputs = onnx_model(**tokens)
         io_outputs = io_model(**tokens)
 
@@ -913,7 +913,7 @@ def test_compare_to_io_binding(self, model_arch):
         io_model = ORTModelForFeatureExtraction.from_pretrained(model_id, from_transformers=True, use_io_binding=True)
 
         tokenizer = get_preprocessor(model_id)
-        tokens = tokenizer("This is a sample output", return_tensors="pt")
+        tokens = tokenizer(["This is a sample output"] * 2, return_tensors="pt")
         onnx_outputs = onnx_model(**tokens)
         io_outputs = io_model(**tokens)
 
@@ -1116,7 +1116,7 @@ def test_compare_to_io_binding(self, model_arch):
         io_model = ORTModelForCausalLM.from_pretrained(model_id, from_transformers=True, use_io_binding=True)
 
         tokenizer = get_preprocessor(model_id)
-        tokens = tokenizer("This is a sample output", return_tensors="pt")
+        tokens = tokenizer(["This is a sample output"] * 2, return_tensors="pt")
         onnx_outputs = onnx_model(**tokens)
         io_outputs = io_model(**tokens)
 
@@ -1246,7 +1246,7 @@ def test_compare_to_io_binding(self, *args, **kwargs):
         preprocessor = get_preprocessor(model_id)
         url = "http://images.cocodataset.org/val2017/000000039769.jpg"
         image = Image.open(requests.get(url, stream=True).raw)
-        inputs = preprocessor(images=image, return_tensors="pt")
+        inputs = preprocessor(images=[image] * 2, return_tensors="pt")
         onnx_outputs = onnx_model(**inputs)
         io_outputs = io_model(**inputs)
 
@@ -1410,9 +1410,9 @@ def test_compare_to_io_binding(self, model_arch):
         io_model = ORTModelForSeq2SeqLM.from_pretrained(model_id, from_transformers=True, use_io_binding=True)
 
         tokenizer = get_preprocessor(model_id)
-        tokens = tokenizer("This is a sample output", return_tensors="pt")
+        tokens = tokenizer(["This is a sample output"] * 2, return_tensors="pt")
         decoder_start_token_id = onnx_model.config.decoder_start_token_id if model_arch != "mbart" else 2
-        decoder_inputs = {"decoder_input_ids": torch.ones((1, 1), dtype=torch.long) * decoder_start_token_id}
+        decoder_inputs = {"decoder_input_ids": torch.ones((2, 1), dtype=torch.long) * decoder_start_token_id}
 
         onnx_outputs = onnx_model(**tokens, **decoder_inputs)
         io_outputs = io_model(**tokens, **decoder_inputs)
@@ -1436,8 +1436,8 @@ def test_compare_generation_to_io_binding(self, model_arch):
 
         tokenizer = get_preprocessor(model_id)
         tokens = tokenizer("This is a sample output", return_tensors="pt")
-        onnx_outputs = onnx_model.generate(**tokens)
-        io_outputs = io_model.generate(**tokens)
+        onnx_outputs = onnx_model.generate(**tokens, num_beams=5)
+        io_outputs = io_model.generate(**tokens, num_beams=5)
 
         # compare tensor outputs
         self.assertTrue(torch.equal(onnx_outputs, io_outputs))