ONNX encoder decoder exchange invoke issue
Author: umanniyaz
Pull request: no

### System Info

TR-OCR  Model
Encoder - BeiT  -encoder.onnx
Decoder - Roberta large- decoder.onnx

System config:
Intel i9 11 gen
Nvidia Quadro RTX 4000 Max Q design - 16GB

Dependencies version: 
onnx == 1.12.0
onnx-runtime == 1.13.1
torch == 1.13.0
transformers == 4.24.0
torchvision ==0.14.0

Issue:
Unable to start ONNX inference sessions with Tr-OCR ONNX conversions-> encoder.onnx & decoder.onnx with **ORTModelForVision2Seq** , model.generate() -> raising this error:

```
model.generate(pixel_values.to('cpu'))
``` 

Traceback (most recent call last):  
```
File "<string>", line 1, in <module>   File "C:\Users\110769\Anaconda3\envs\ocr2\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context     return func(*args, **kwargs)   File "C:\Users\110769\Anaconda3\envs\ocr2\lib\site-packages\transformers\generation_utils.py", line 1339, in generate model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation( File "C:\Users\110769\Anaconda3\envs\ocr2\lib\site-packages\transformers\generation_utils.py", line 583, in _prepare_encoder_decoder_kwargs_for_generation model_kwargs["encoder_outputs"]: ModelOutput = encoder(**encoder_kwargs) File "C:\Users\110769\Anaconda3\envs\ocr2\lib\site-packages\torch\nn\modules\module.py", line 1188, in _call_impl if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks File "C:\Users\110769\Anaconda3\envs\ocr2\lib\site-packages\torch\nn\modules\module.py", line 1265, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'ORTEncoder' object has no attribute '_backward_hooks'
```

Whole Code Snippet :
```
class ORTEncoder(nn.Module):
    """
    Encoder model for ONNX Runtime inference.
    Arguments:
        session (`onnxruntime.InferenceSession`):
            The ONNX Runtime inference session associated to the encoder.
    """
    def __init__(
        self, session: onnxrt.InferenceSession, device: torch.device, main_input_name: str = "input_ids"
    ):
        self.session = session
        self._device = device
        self.main_input_name = main_input_name
        self.input_names = {input_key.name: idx for idx, input_key in enumerate(self.session.get_inputs())}
        self.output_names = {output_key.name: idx for idx, output_key in enumerate(self.session.get_outputs())}
class ORTDecoder(nn.Module):
    """
    Encoder model for ONNX Runtime inference.
    Arguments:
        session (`onnxruntime.InferenceSession`):
            The ONNX Runtime inference session associated to the encoder.
    """
    def __init__(
        self, session: onnxrt.InferenceSession, device: torch.device, main_input_name: str = "input_ids"
    ):
        self.session = session
        self._device = device
        self.main_input_name = main_input_name
        self.input_names = {input_key.name: idx for idx, input_key in enumerate(self.session.get_inputs())}
        self.output_names = {output_key.name: idx for idx, output_key in enumerate(self.session.get_outputs())}
class ORTModelForVision2Seq(VisionEncoderDecoderModel):
    def __init__(self, *args, **kwargs):
        config = AutoConfig.from_pretrained('microsoft/trocr-base-printed')
        super().__init__(config)
        self._device = "cpu"
        self.encoder = ORTEncoder(onnxrt.InferenceSession(encoder_path,providers=["CPUExecutionProvider"]),device='cpu')
        self.decoder = ORTDecoder(onnxrt.InferenceSession(decoder_path,providers=["CPUExecutionProvider"]),device='cpu')
    def forward(
        self,
        pixel_values: Optional[torch.FloatTensor] = None,
        decoder_input_ids: Optional[torch.LongTensor] = None,
        encoder_outputs: Optional[Tuple[Tuple[torch.Tensor]]] = None,
        **kwargs,
    ) -> Seq2SeqLMOutput:
        # Encode if needed : first prediction pass
        if encoder_outputs is None:
            encoder_outputs = self.encoder(pixel_values=pixel_values)
        # Decode
        decoder_attention_mask = decoder_input_ids.new_ones(decoder_input_ids.shape)
        decoder_outputs = self.decoder(
            input_ids=decoder_input_ids,
            attention_mask=decoder_attention_mask,
            encoder_hidden_states=encoder_outputs.last_hidden_state,
        )
        return Seq2SeqLMOutput(
            logits=decoder_outputs.logits,
        )
    def prepare_inputs_for_generation(self, input_ids, attention_mask=None, encoder_outputs=None, **kwargs):
        return {
            "decoder_input_ids": input_ids,
            "decoder_atttention_mask": input_ids,
            "encoder_outputs": encoder_outputs,
        }
model = ORTModelForVision2Seq()
start = time.time()
img = Image.open(r'PATH').convert("RGB")
processor = TrOCRProcessor.from_pretrained('microsoft/trocr-base-printed')
pixel_values = processor(images=img, return_tensors="pt").pixel_values
model.config.decoder_start_token_id = 2
model.config.pad_token_id = processor.tokenizer.pad_token_id
model.config.eos_token_id = processor.tokenizer.sep_token_id
model.config.vocab_size = model.config.decoder.vocab_size
generated_ids = model.generate(pixel_values.to(device))
end = time.time()
```

How can I run encoder/decoder with wrapped ORT instead invoking two concurrent sessions for Encoder and decoder in loop

@mht-sharma  @NielsRogge 

encoder_path -> takes encoder ONNX
decoder_path -> take decdoer ONNX

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce:

1. Run above code snippet model.generate() invokes error


### Expected behavior

ONNX inference session should be invoked using ORTVisionseq2seq ,when given encoder and decoder. Also Model.generate() not a valid function for generating IDS

Comments:
@NielsRogge @mht-sharma 

New issue raised here: (https://github.com/huggingface/transformers/issues/20644)

Added a draft PR in optimum for easing inference. https://github.com/huggingface/optimum/pull/588

What about this : 
**Adds ORTModelForVision2Seq for inference (In progress...)**

Like I said I TrOCR model in encoder_onnx and decoder_onnx , how can I invoke these two models together on ONNX runtime for fast inference.

@mht-sharma 


Hi @umanniyaz could you try the following code for inference for testing.

https://gist.github.com/mht-sharma/f38c670930ac7df413c07327e692ee39

> What about this : **Adds ORTModelForVision2Seq for inference (In progress...)**
> 
> Like I said I TrOCR model in encoder_onnx and decoder_onnx , how can I invoke these two models together on ONNX runtime for fast inference.
> 
> @mht-sharma

I am waiting for a few PRs to merge before this and would work on the inference. Should be available by next week.

> Hi @umanniyaz could you try the following code for inference for testing.
> 
> https://gist.github.com/mht-sharma/f38c670930ac7df413c07327e692ee39

@mht-sharma Inference using these helper classes is still bad, I don't see any decrease in latency, plus performance in terms of text recognition decreases

Hi @umanniyaz could you share which device are you using for inference. If you are using GPU, you need to add the iobinding to observe the speedup. 

For CPU inference, it may differ on the kind of CPU you are using. Currently the ORT inference uses the torch for the generation, hence, there can be a resource crunch between the torch and ORT which may lead to a slowdown. You may need to set the appropriate `intra-op-threads` and `torch threads` to observe the speedup.  https://github.com/microsoft/onnxruntime/issues/13808

> Hi @umanniyaz could you share which device are you using for inference. If you are using GPU, you need to add the iobinding to observe the speedup.
> 
> For CPU inference, it may differ on the kind of CPU you are using. Currently the ORT inference uses the torch for the generation, hence, there can be a resource crunch between the torch and ORT which may lead to a slowdown. You may need to set the appropriate `intra-op-threads` and `torch threads` to observe the speedup. [microsoft/onnxruntime#13808](https://github.com/microsoft/onnxruntime/issues/13808)

@mht-sharma I am using Intel i9 10th generation using this class for a Django REST API - CPU,not using GPU ,my GPU configuration stands Nvidia Quadro RTX-4000 Max Q design.

Can you provide with CPU and with GPU implementation,i just need to see where it speeds up.

Is onnx runtime for task image-to-text in Optimum pipelines for coming anytime soon

@mht-sharma @NielsRogge  I just tried the above things in you inference_testing code for GPU with iobinding and on CPU by adding intra_op_threads for parrallel execution and noticed change in inference , but the accuracy of TR-OCR on changing to respective Encoder_model.onnx and Decoder_model.onnx suffers,it gives bad results than Original,like whitespaces between text are ignored and CER in text recognition increases in case of using above ONNX models

![image](https://user-images.githubusercontent.com/33204214/209306924-80f95525-6910-430f-9917-0b2dab293fe5.png)


Hi @umanniyaz , 

`Is ONNX runtime for task image-to-text in Optimum pipelines for coming anytime soon` - Things got little delayed due to the NYE. I would work on the ONNXRuntime pipeline in optimum in the coming week.

The decrease in accuracy may not be because of adding `iobinding` or `intra_op_threads`. Let me know if it is otherwise. The drop in accuracy is on both CPU and GPU (CUDAExecutionProvider)?

Could you share which `atol` you have used for the ONNX export.

@mht-sharma There is a consistent decrease in accuracy irrespective of CPU, GPU intra op threads or iobinding, For onnx export I have utilised your latest PR on Vision EncoderDecoder Model conversion as mentioned,you can send the TrOCR conversion again,further  i used this:

python -m transformers.onnx --model=microsoft/trocr-base-printed  --feature=vision2seq-lm models_trocr_base --atol 1e-3


Note: Need to use Tr-OCR-Base Printed 

> Hi @umanniyaz ,
> 
> `Is ONNX runtime for task image-to-text in Optimum pipelines for coming anytime soon` - Things got little delayed due to the NYE. I would work on the ONNXRuntime pipeline in optimum in the coming week.
> 
> The decrease in accuracy may not be because of adding `iobinding` or `intra_op_threads`. Let me know if it is otherwise. The drop in accuracy is on both CPU and GPU (CUDAExecutionProvider)?
> 
> Could you share which `atol` you have used for the ONNX export.

Hi @mht-sharma --atol 1e-3 gives inaccurate results then actual model,using 1e-4 onwards not feasible can you please tell value for atol for converting models

> Hi @umanniyaz ,
> 
> `Is ONNX runtime for task image-to-text in Optimum pipelines for coming anytime soon` - Things got little delayed due to the NYE. I would work on the ONNXRuntime pipeline in optimum in the coming week.
> 
> The decrease in accuracy may not be because of adding `iobinding` or `intra_op_threads`. Let me know if it is otherwise. The drop in accuracy is on both CPU and GPU (CUDAExecutionProvider)?
> 
> Could you share which `atol` you have used for the ONNX export.

Any updates on this? @mht-sharma 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> @mht-sharma @NielsRogge I just tried the above things in you inference_testing code for GPU with iobinding and on CPU by adding intra_op_threads for parrallel execution and noticed change in inference , but the accuracy of TR-OCR on changing to respective Encoder_model.onnx and Decoder_model.onnx suffers,it gives bad results than Original,like whitespaces between text are ignored and CER in text recognition increases in case of using above ONNX models
> 
> ![image](https://user-images.githubusercontent.com/33204214/209306924-80f95525-6910-430f-9917-0b2dab293fe5.png)

Mr @umanniyaz Can you share me your code inference trocr onnx with iobinding, thanks a lot for your help !!!