ONNX conversion from VisionEncoderDecoderModel with different dimensions
Author: entropy2333
Pull request: no

### System Info

- `transformers` version: 4.23.0.dev0
- Platform: Linux-4.4.0-87-generic-x86_64-with-glibc2.23
- Python version: 3.9.13
- Huggingface_hub version: 0.10.0
- PyTorch version (GPU?): 1.12.1 (True)

### Who can help?

@NielsRogge, @patrickvonplaten

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am trying to convert a VisionEncoderDecoder model to ONNX using the feature that has been recently merged https://github.com/huggingface/transformers/pull/19254.
However, when two pretrained models whose model dimensions are different, It reproduces errors as below.

## Model Load & Save

```python
from transformers import VisionEncoderDecoderModel, BertTokenizer, AutoFeatureExtractor

encoder_name_or_path = "hf-internal-testing/tiny-random-vit"
decoder_name_or_path = "fnlp/bart-base-chinese"
model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained(
    encoder_name_or_path,
    decoder_name_or_path,
)
tokenizer = BertTokenizer.from_pretrained(decoder_name_or_path)
feature_extractor = AutoFeatureExtractor.from_pretrained(encoder_name_or_path)

output_dir = "outputs"
model.save_pretrained(output_dir)
feature_extractor.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)
```

## Model Structure

```
VisionEncoderDecoderModel(
  (encoder): SwinModel(...)
  (decoder): BartForCausalLM(...)
  (enc_to_dec_proj): Linear(in_features=32, out_features=768, bias=True)
)
```

There exists a new linear layer to project encoder hidden states in [modeling_vision_encoder_decoder.py#L217](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py#L217)

```python
# encoder outputs might need to be projected to different dimension for decoder
if (
    self.encoder.config.hidden_size != self.decoder.config.hidden_size
    and self.decoder.config.cross_attention_hidden_size is None
):
    self.enc_to_dec_proj = nn.Linear(self.encoder.config.hidden_size, self.decoder.config.hidden_size)
```

## Conversion to ONNX

```bash
python -m transformers.onnx --model=outputs/ --feature=vision2seq-lm onnx/ --atol 1e-3
```

Output:
```bash
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/backup2/mkf/transformers/src/transformers/onnx/__main__.py", line 180, in <module>
    main()
  File "/backup2/mkf/transformers/src/transformers/onnx/__main__.py", line 118, in main
    onnx_inputs, onnx_outputs = export(
  File "/backup2/mkf/transformers/src/transformers/onnx/convert.py", line 339, in export
    return export_pytorch(preprocessor, model, config, opset, output, tokenizer=tokenizer, device=device)
  File "/backup2/mkf/transformers/src/transformers/onnx/convert.py", line 192, in export_pytorch
    onnx_export(
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/onnx/__init__.py", line 350, in export
    return utils.export(
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/onnx/utils.py", line 163, in export
    _export(
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/onnx/utils.py", line 1074, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/onnx/utils.py", line 727, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/onnx/utils.py", line 602, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/onnx/utils.py", line 517, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/jit/_trace.py", line 1175, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/jit/_trace.py", line 127, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/jit/_trace.py", line 118, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/backup2/mkf/transformers/src/transformers/models/bart/modeling_bart.py", line 1851, in forward
    outputs = self.model.decoder(
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/backup2/mkf/transformers/src/transformers/models/bart/modeling_bart.py", line 1104, in forward
    layer_outputs = decoder_layer(
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/backup2/mkf/transformers/src/transformers/models/bart/modeling_bart.py", line 439, in forward
    hidden_states, cross_attn_weights, cross_attn_present_key_value = self.encoder_attn(
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/backup2/mkf/transformers/src/transformers/models/bart/modeling_bart.py", line 201, in forward
    key_states = self._shape(self.k_proj(key_value_states), -1, bsz)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/user/anaconda3/envs/swinocr/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (16x32 and 768x768)
```

### Expected behavior

It seems that the existing ONNX conversion for EncoderDecoderModel only converts the encoder and decoder, and ignores this linear layer.
If I change the model to [microsoft/trocr-base-handwritten](https://huggingface.co/microsoft/trocr-base-handwritten), which has a similar structure and the same dimensions (i.e. no linear layer), the conversion works.

```bash
python -m transformers.onnx --model=microsoft/trocr-base-handwritten --feature=vision2seq-lm trocr_onnx/ --atol 1e-3
```

Thanks a lot for looking into it :)

Comments:
I'm try to fix your problem in #19819

Cc @mht-sharma 

I've come across an issue with the ONNX conversion of TrOCR-base. I'm not sure if they are entirely related, but I've managed to convert the models with huggingface.onnx into actual onnx files. After conversion I obtain an `encoder.onnx` and a `decoder.onnx` file, which seems to be as it should be.

Then, for processing an input image I do the following:
1. Send image through the processor: 
`processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-str")`
`processor_output = processor(img)`
2. Send output of processor through encoder
`encoder_output = encoder(processor_output )`
3. Send output of encoder through the decoder.
`decoder_output = decoder(trace_tensor)`

The conversion happens like this:
```
python -m transformers.onnx   --model=microsoft/trocr-base-str
                              --feature=vision2seq-lm models/onnx
                              --atol 1e-3
```

This yields 2 errors. 1 after running the encoder (the error that occurs breaks the process). A second one occurs with the decoder. For this I have created a tensor that mimics the input shapes of the batch that the decoder expects.

ERROR ENCODER:
```
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_937' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\cpu\tensor\reshape_helper.h:41 onnxruntime::ReshapeHelper::ReshapeHelper gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{2,100,100,64}, requested shape:{1,9216,128}
```

ERROR DECODER:
```
File "C:\Users\Miniconda3\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 196, in run
    raise ValueError("Model requires {} inputs. Input Feed contains {}".format(num_required_inputs, num_inputs))
ValueError: Model requires 3 inputs. Input Feed contains 1
```

@NielsRogge @mht-sharma any clues on this?

> I've come across an issue with the ONNX conversion of TrOCR-base. I'm not sure if they are entirely related, but I've managed to convert the models with huggingface.onnx into actual onnx files. After conversion I obtain an `encoder.onnx` and a `decoder.onnx` file, which seems to be as it should be.
> 
> Then, for processing an input image I do the following:
> 
> 1. Send image through the processor:
>    `processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-str")`
>    `processor_output = processor(img)`
> 2. Send output of processor through encoder
>    `encoder_output = encoder(processor_output )`
> 3. Send output of encoder through the decoder.
>    `decoder_output = decoder(trace_tensor)`
> 
> The conversion happens like this:
> 
> ```
> python -m transformers.onnx   --model=microsoft/trocr-base-str
>                               --feature=vision2seq-lm models/onnx
>                               --atol 1e-3
> ```
> 
> This yields 2 errors. 1 after running the encoder (the error that occurs breaks the process). A second one occurs with the decoder. For this I have created a tensor that mimics the input shapes of the batch that the decoder expects.
> 
> ERROR ENCODER:
> 
> ```
> onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_937' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\cpu\tensor\reshape_helper.h:41 onnxruntime::ReshapeHelper::ReshapeHelper gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{2,100,100,64}, requested shape:{1,9216,128}
> ```
> 
> ERROR DECODER:
> 
> ```
> File "C:\Users\Miniconda3\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 196, in run
>     raise ValueError("Model requires {} inputs. Input Feed contains {}".format(num_required_inputs, num_inputs))
> ValueError: Model requires 3 inputs. Input Feed contains 1
> ```
> 
> @NielsRogge @mht-sharma any clues on this?

Hi @Fritskee, the decoder ONNX model expects 3 inputs during inference, namely: `input_ids`, `attention_mask` &  `encoder_hidden_states` (output from the encoder model). Hence, the above error. You need to provide the appropriate start token id and attention mask as input to the decoder to start the sequence.

Encoder error: Please ensure if you have followed appropriate steps to generate the input. For the above model, the following steps worked for me: 

```python
# load image from the IIIT-5k dataset
url = 'https://i.postimg.cc/ZKwLg2Gw/367-14.png'
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
pixel_values = processor(images=image, return_tensors="pt").pixel_values
```

For more info check: [trocr-base-str](https://huggingface.co/microsoft/trocr-base-str)

Let me know if there are additional issues with the model.


@mht-sharma Thanks for you assistance.
I just tried the suggested code of yours for the encoder part. The issue remains exactly the same.

To make sure we're talking about the same thing, I also used the image that you pulled from the web. I'm now running the following code:
```py
import requests
import numpy as np
import onnxruntime as onnxrt
from PIL import Image
from transformers import TrOCRProcessor
import config as c

class OnnxModel():
    def __init__(self, model_path):
        self.model = onnxrt.InferenceSession(model_path)

    def __call__(self, img):
        onnx_inputs = {self.model.get_inputs()[0].name: np.asarray(img, dtype='float32')}
        onnx_output = self.model.run(None, onnx_inputs)[0]
        return onnx_output

if __name__ == "__main__":
    processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-str")
    encoder = OnnxModel(c.encoder_path)

    url = 'https://i.postimg.cc/ZKwLg2Gw/367-14.png'
    image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

    pixel_values = processor(images=image, return_tensors="pt").pixel_values
    encoder_output = encoder(pixel_values)
```

Running this code, gives me exactly the same error. Namely:
`onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_937' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\cpu\tensor\reshape_helper.h:41 onnxruntime::ReshapeHelper::ReshapeHelper gsl::narrow_cast<int64_t>(input_shape.Size()) == size was false. The input tensor cannot be reshaped to the requested shape. Input shape:{2,100,100,64}, requested shape:{1,9216,128}`

I also looked at the link of trocr-base-str that you provided. In there I see that they do `model.generate(pixel_values)`, but this is not possible with ONNX, since the 'InferenceSession' object that is used with onnx has no attribute 'generate'. 

@Fritskee Thanks for trying the suggestion.

I have tried your code and it works for me. I would mention the steps I use for exporting and running the inference.

1. Clone the transformers and install it from source.
2. Export the model
```python
python -m transformers.onnx --model=microsoft/trocr-base-str  --feature=vision2seq-lm models_trocr_base --atol 1e-3
```
3. Ran inference using your code. I updated the following line in your code because of version changes.
```python
self.model = onnxrt.InferenceSession(model_path, providers=["CPUExecutionProvider"])
```

Also please find the onnx, torch and onnxruntime version I am using.
```bash
onnx==1.12.0
onnxruntime==1.12.1
torch==1.12.1
```

For running inference using the 2 models, as of now, you'll have to roll your own generation loop with onnxruntime. An alternative would be to implement an ORTModelForVisionSeq2Seq in optimum, similar to how Whisper is being implemented: https://github.com/huggingface/optimum/pull/420/files#diff-77c4bfa5fbc9262eda15bbbc01d9796a0daa33e6725ca41e1cfe600a702d0bfc


@mht-sharma I just created a new conda env, matched your package versions and installed hf/transformers from source. Did the job! I now get a `(1, 577, 768)` tensor at the output of the encoder.

Thanks for your assistance! Will try to figure out the decoder part now!

@mht-sharma I am facing one final issue with the decoder, which is also a shape issue. 

I'm doing the following to infere with ONNX. I checked the code of [optimum](https://github.com/huggingface/optimum/pull/420/files#diff-77c4bfa5fbc9262eda15bbbc01d9796a0daa33e6725ca41e1cfe600a702d0bfc) as per your suggestion. From this I implemented my callable functionality of the `OnnxDecoder`. I also checked the keys of the `input_names` of the onnx names and all this is correct.

However, at the end I keep on getting an issue with size mismatches. The way I currently understand it, this cannot be solved since the (1, 577, 1024) shape contains the prime number 577. This makes it impossible to find an integer dimension that can match the shape (x, -1, 16, 64). Additionally, the `input_ids` shape is limited to 514 in either dimension. Thus unless I use dimension 1, I cannot create a working dimension. But then using dimension 1 makes the runtime way too long.

Any clue on this one? 

```py
import requests
import numpy as np
import onnxruntime as onnxrt
from PIL import Image
from transformers import TrOCRProcessor
import config as c

class OnnxDecoder():
    def __init__(self, model_path):
        self.model = onnxrt.InferenceSession(model_path, providers=["CPUExecutionProvider"])
        self.input_names = {input_key.name: idx for idx, input_key in enumerate(self.model.get_inputs())}

    def __call__(self, input_ids: torch.LongTensor,
                 encoder_hidden_states: torch.FloatTensor,
                 attention_mask: torch.LongTensor):
        onnx_inputs = {"input_ids": input_ids.cpu().detach().numpy()}

        if "attention_mask" in self.input_names:
            onnx_inputs["attention_mask"] = attention_mask.cpu().detach().numpy()

        if "encoder_hidden_states" in self.input_names:
            onnx_inputs["encoder_hidden_states"] = encoder_hidden_states.cpu().detach().numpy()

        onnx_output = self.model.run(None, onnx_inputs)
        return onnx_output

if __name__ == "__main__":
    processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-str")
    encoder = OnnxModel(c.encoder_path)

    url = 'https://i.postimg.cc/ZKwLg2Gw/367-14.png'
    image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

    pixel_values = processor(images=image, return_tensors="pt").pixel_values
    encoder_output = encoder(pixel_values)
    decoder_output = decoder(input_ids=torch.LongTensor(np.random.rand(512, 512)),
                     encoder_hidden_states=torch.FloatTensor(encoder_output),
                     attention_mask=torch.LongTensor(np.random.rand(512, 512)))
```

ERROR:
`onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_623' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\cpu\tensor\reshape_helper.h:36 onnxruntime::ReshapeHelper::ReshapeHelper size != 0 && (input_shape.Size() % size) == 0 was false. The input tensor cannot be reshaped to the requested shape. Input shape:{1,577,1024}, requested shape:{512,-1,16,64}`

I am aware that using tensors with random values will result in a wrong output, but at this moment I'm just trying it to get to run and check the inference speed of the onnx model. 

Hello @Fritskee the issue is because of the sample input. The `input_ids` and `attention_mask` expects input of size `<batch_size, sequence_length>`. 

In your above snippet you have created an input of batch size 512, however, the `encoder_hidden_states` are of batch size 1, hence the error.  Try creating input with batch size 1 and it should work.

Additionally, please use `torch.ones` to generate the `attention_mask` input, with same shape as input_ids.

> Hello @Fritskee the issue is because of the sample input. The `input_ids` and `attention_mask` expects input of size `<batch_size, sequence_length>`.
> 
> In your above snippet you have created an input of batch size 512, however, the `encoder_hidden_states` are of batch size 1, hence the error. Try creating input with batch size 1 and it should work.
> 
> Additionally, please use `torch.ones` to generate the `attention_mask` input, with same shape as input_ids.

@mht-sharma This is indeed the error that I made. Thanks for pointing that out! 
I do notice that I am getting better inference times with the Huggingface pytorch model, than with the ONNX model. Which is something I've never encountered. Generally ONNX always outperforms PyTorch for inference. ONNX runs in 2.5 sec, PyTorch runs in 1.8 sec. Both on the same CPU.

Hello @Fritskee I am able to observe speedup on both cpu and gpu with the model. Could you please share your inference / benchmarking code if possible for testing? 

> Hello @Fritskee I am able to observe speedup on both cpu and gpu with the model. Could you please share your inference / benchmarking code if possible for testing?

Apologies for the late reply @mht-sharma, I use following code to run inference of the **ONNX model:**
```py
if __name__ == "__main__":
    image = Image.open(r"C:\Users\local_img.png").convert("RGB")

    processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-str")
    encoder = OnnxEncoder(c.encoder_path)
    decoder = OnnxDecoder(c.decoder_path)

    start = time.time()
    pixel_values = processor(images=image, return_tensors="pt").pixel_values
    encoder_output = encoder(pixel_values)
    decoder_output = decoder(input_ids=torch.LongTensor(np.random.rand(1,384)),
                                 encoder_hidden_states=torch.FloatTensor(encoder_output),
                                 attention_mask=torch.LongTensor(np.ones((1,384))))
    end = time.time()
```

This code is used to run inference with the **PyTorch/HuggingFace model:**
```py
if __name__ == "__main__":
    image = Image.open(r"C:\Users\local_img.png").convert("RGB")

    processor = TrOCRProcessor.from_pretrained('microsoft/trocr-base-str')
    model = VisionEncoderDecoderModel.from_pretrained(
        r"C:\Users\Downloads\text-model\text-model\pytorch_model.bin",
        config=r"C:\Users\Downloads\text-model\text-model\config.json")

    start = time.time()
    pixel_values = processor(images=image, return_tensors="pt").pixel_values
    generated_ids = model.generate(pixel_values)
    model_output = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
    end = time.time()
```

I got an avg. inference time of **3.339415764808655 seconds** over 100 runs for the **ONNX model**.
Similarly, I got an avg. inference time of **2.542718324661255** over 100 runs for the **torch/HF model**.
Both ran on the exact same machine, on CPU, while no other processes were running in the background. 

  Hi @Fritskee , for ORT inference you'll have to roll your own generation loop with ONNX Runtime to run the inference. The above code runs decoder with SL 384 with one forward pass which will give you incorrect results.

You can wrap your ORTEncode and ORTDecoder in a ORTModelForVision2Seq

```python

class ORTModelForVision2Seq(VisionEncoderDecoderModel):
    def __init__(self, *args, **kwargs):
        config = AutoConfig.from_pretrained(model_name)
        super().__init__(config)
        self._device = "cpu"

        self.encoder = ORTEncoder()
        self.decoder = ORTDecoder()

    def forward(
        self,
        pixel_values: Optional[torch.FloatTensor] = None,
        decoder_input_ids: Optional[torch.LongTensor] = None,
        encoder_outputs: Optional[Tuple[Tuple[torch.Tensor]]] = None,
        **kwargs,
    ) -> Seq2SeqLMOutput:

        # Encode if needed : first prediction pass
        if encoder_outputs is None:
            encoder_outputs = self.encoder(pixel_values=pixel_values)

        # Decode
        decoder_attention_mask = decoder_input_ids.new_ones(decoder_input_ids.shape)
        decoder_outputs = self.decoder(
            input_ids=decoder_input_ids,
            attention_mask=decoder_attention_mask,
            encoder_hidden_states=encoder_outputs.last_hidden_state,
        )

        return Seq2SeqLMOutput(
            logits=decoder_outputs.logits,
        )

    def prepare_inputs_for_generation(self, input_ids, attention_mask=None, encoder_outputs=None, **kwargs):

        return {
            "decoder_input_ids": input_ids,
            "decoder_atttention_mask": input_ids,
            "encoder_outputs": encoder_outputs,
        }

model = ORTModelForVision2Seq()

start = time.time()
model.config.decoder_start_token_id = 2
model.config.pad_token_id = processor.tokenizer.pad_token_id
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.vocab_size = model.config.decoder.vocab_size

generated_ids = model.generate(pixel_values.to(device))
end = time.time()
```

The class would be soon implemented in the `optimum` soon for easier inference. Stay tuned!



@mht-sharma Thanks for the example! I tried implementing it. For the further implementation I looked at the [optimum/pipelines.py](https://github.com/huggingface/optimum/blob/816268d7c3aba0de98f2d74db06344e76f071535/optimum/pipelines.py) and at [optimum/onnxruntime/modeling_seq2seq.py](https://github.com/huggingface/optimum/blob/816268d7c3aba0de98f2d74db06344e76f071535/optimum/onnxruntime/modeling_seq2seq.py). 

Basically I took the examples from `modeling_seq2seq.py` for the `ORTEncoder` and `ORTDecoder`, and I took your example from above and initialize the `ORTModelForVision2Seq(VisionEncoderDecoderModel)` like this:
```py
class ORTModelForVision2Seq(VisionEncoderDecoderModel):
    def __init__(self, *args, **kwargs):
        config = AutoConfig.from_pretrained('microsoft/trocr-base-str')
        super().__init__(config)
        self._device = "cpu"
        self.encoder = ORTEncoder(onnxruntime.InferenceSession(c.encoder_path, providers=["CPUExecutionProvider"]), device='cpu')
        self.decoder = ORTDecoder(onnxruntime.InferenceSession(c.decoder_path, providers=["CPUExecutionProvider"]), device='cpu')
```
The encoder_path is the path to the file of `encoder.onnx` and the path to the decoder file is the path to `decoder.onnx`.

For your example, the ORTEncoder is initialized like this:
```py
class ORTEncoder:
    """
    Encoder model for ONNX Runtime inference.
    Arguments:
        session (`onnxruntime.InferenceSession`):
            The ONNX Runtime inference session associated to the encoder.
    """

    def __init__(
        self, session: onnxruntime.InferenceSession, device: torch.device, main_input_name: str = "input_ids"
    ):
        self.session = session
        self._device = device
        self.main_input_name = main_input_name
        self.input_names = {input_key.name: idx for idx, input_key in enumerate(self.session.get_inputs())}
        self.output_names = {output_key.name: idx for idx, output_key in enumerate(self.session.get_outputs())}
```

When I initialize the Onnx InferenceSessions as shown in the first code block of this message, I get the following error: 
`self.encoder = ORTEncoder(onnxruntime.InferenceSession(c.encoder_path, providers=["CPUExecutionProvider"]), device='cpu')
  File "C:\Users\FrCa\Miniconda3\envs\onnxfix\lib\site-packages\torch\nn\modules\module.py", line 1242, in __setattr__
    raise TypeError("cannot assign '{}' as child module '{}' "
TypeError: cannot assign '__main__.ORTEncoder' as child module 'encoder' (torch.nn.Module or None expected)
python-BaseException`

The `ORTEncoder` seems to expect a path to a Pytorch model for its session, which seems odd. I am currently passing the onnx converted encoder to `ORTEncoder`, but due to the error, I have also tried passing the equivalent `.pth` model Additionally, I also tried passing None (which doesn't make much sense, but it says it is a possibility). Both of them also give errors.

**EDIT**:

I did find that by not adding the superclass of `VisionEncoderDecoderModel`, the model can initialize both the ORTEncoder and ORTDecoder. However, this causes the code to break, because the model does need the `config` attribute to work with the example that is provided here.
```py
class ORTModelForVision2Seq():
    def __init__(self, *args, **kwargs):
        self._device = "cpu"
        self.encoder = ORTEncoder(onnxruntime.InferenceSession(c.encoder_path, providers=["CPUExecutionProvider"]), device='cpu')
        self.decoder = ORTDecoder(onnxruntime.InferenceSession(c.decoder_path, providers=["CPUExecutionProvider"]), device='cpu')
```
`model.config.decoder_start_token_id = 2
AttributeError: 'ORTModelForVision2Seq' object has no attribute 'config'`



Hi @Fritskee , apologies for the late reply. You need to inherit `ORTEncoder` and `ORTDecoder` from `torch.nn.Module` to avoid the issue.

Hi @umanniyaz, please open a new issue instead

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.