ONNX conversion from VisionEncoderDecoderModel?
Author: kangsan0420
Pull request: no


# Description

I'd like to convert a VisionEncoderDecoder model to ONNX using the feature that has been recently merged #19254  
However, It reproduces errors as below.  
What am I missing?

# Environment
```python
import transformers
import torch
import sys

!echo "OS: $(cat /etc/issue)"
!echo "Arch.: $(arch)"
print('python:', sys.version)
print('transformers:', transformers.__version__)
print('torch', torch.__version__)
```
OS: Ubuntu 20.04.4 LTS \n \l
Arch.: x86_64
python: 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0]
transformers: 4.23.1
torch 1.12.1+cu102

# Reproduce
## Model Loading
```python
model = torch.load('221002_203253.pt') # TrOCR model that I have trained.
model.save_pretrained('trocr')

# To show they're same type.
model2 = transformers.VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")
type(model), type(model2)
```
Output:  
<div class="lm-Widget p-Widget jp-OutputArea jp-Cell-outputArea" style=""><div class="lm-Widget p-Widget lm-Panel p-Panel jp-OutputArea-child"><div class="lm-Widget p-Widget jp-OutputPrompt jp-OutputArea-prompt"></div><div class="lm-Widget p-Widget jp-RenderedText jp-mod-trusted jp-OutputArea-output" data-mime-type="application/vnd.jupyter.stderr"><pre>Some weights of VisionEncoderDecoderModel were not initialized from the model checkpoint at microsoft/trocr-base-handwritten and are newly initialized: ['encoder.pooler.dense.bias', 'encoder.pooler.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
</pre></div></div><div class="lm-Widget p-Widget lm-Panel p-Panel jp-OutputArea-child jp-OutputArea-executeResult"><div class="lm-Widget p-Widget jp-RenderedText jp-mod-trusted jp-OutputArea-output" data-mime-type="text/plain"><pre>(transformers.models.vision_encoder_decoder.modeling_vision_encoder_decoder.VisionEncoderDecoderModel,
 transformers.models.vision_encoder_decoder.modeling_vision_encoder_decoder.VisionEncoderDecoderModel)</pre></div></div></div>

## Check Support
```python
try:
    transformers.onnx.FeaturesManager.check_supported_model_or_raise(model)
except Exception as e:
    print(type(e), e)
    
print('='*100)
    
try:
    transformers.onnx.FeaturesManager.check_supported_model_or_raise(model2)
except Exception as e:
    print(type(e), e)
```
Output:  
<pre>&lt;class 'ValueError'&gt; vision-encoder-decoder doesn't support feature default. Supported values are: {'vision2seq-lm': functools.partial(&lt;bound method OnnxConfig.from_model_config of &lt;class 'transformers.models.vision_encoder_decoder.configuration_vision_encoder_decoder.VisionEncoderDecoderOnnxConfig'&gt;&gt;, task='vision2seq-lm')}
====================================================================================================
&lt;class 'ValueError'&gt; vision-encoder-decoder doesn't support feature default. Supported values are: {'vision2seq-lm': functools.partial(&lt;bound method OnnxConfig.from_model_config of &lt;class 'transformers.models.vision_encoder_decoder.configuration_vision_encoder_decoder.VisionEncoderDecoderOnnxConfig'&gt;&gt;, task='vision2seq-lm')}
</pre>

## Conversion to ONNX
```shell
python3 -m transformers.onnx -m trocr onnx/
```
Output:  
<pre>Local PyTorch model found.
Framework not requested. Using torch to export to ONNX.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/transformers/onnx/__main__.py", line 180, in &lt;module&gt;
    main()
  File "/usr/local/lib/python3.8/dist-packages/transformers/onnx/__main__.py", line 72, in main
    model = FeaturesManager.get_model_from_feature(
  File "/usr/local/lib/python3.8/dist-packages/transformers/onnx/features.py", line 666, in get_model_from_feature
    model = model_class.from_pretrained(model, cache_dir=cache_dir)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py", line 466, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class &lt;class 'transformers.models.vision_encoder_decoder.configuration_vision_encoder_decoder.VisionEncoderDecoderConfig'&gt; for this kind of AutoModel: AutoModel.
Model type should be one of AlbertConfig, BartConfig, BeitConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, CanineConfig, CLIPConfig, CodeGenConfig, ConditionalDetrConfig, ConvBertConfig, ConvNextConfig, CTRLConfig, CvtConfig, Data2VecAudioConfig, Data2VecTextConfig, Data2VecVisionConfig, DebertaConfig, DebertaV2Config, DecisionTransformerConfig, DeformableDetrConfig, DeiTConfig, DetrConfig, DistilBertConfig, DonutSwinConfig, DPRConfig, DPTConfig, ElectraConfig, ErnieConfig, EsmConfig, FlaubertConfig, FlavaConfig, FNetConfig, FSMTConfig, FunnelConfig, GLPNConfig, GPT2Config, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, GroupViTConfig, HubertConfig, IBertConfig, ImageGPTConfig, LayoutLMConfig, LayoutLMv2Config, LayoutLMv3Config, LEDConfig, LevitConfig, LongformerConfig, LongT5Config, LukeConfig, LxmertConfig, M2M100Config, MarianConfig, MarkupLMConfig, MaskFormerConfig, MBartConfig, MCTCTConfig, MegatronBertConfig, MobileBertConfig, MobileViTConfig, MPNetConfig, MT5Config, MvpConfig, NezhaConfig, NystromformerConfig, OpenAIGPTConfig, OPTConfig, OwlViTConfig, PegasusConfig, PegasusXConfig, PerceiverConfig, PLBartConfig, PoolFormerConfig, ProphetNetConfig, QDQBertConfig, ReformerConfig, RegNetConfig, RemBertConfig, ResNetConfig, RetriBertConfig, RobertaConfig, RoFormerConfig, SegformerConfig, SEWConfig, SEWDConfig, Speech2TextConfig, SplinterConfig, SqueezeBertConfig, SwinConfig, Swinv2Config, T5Config, TapasConfig, TimeSeriesTransformerConfig, TrajectoryTransformerConfig, TransfoXLConfig, UniSpeechConfig, UniSpeechSatConfig, VanConfig, VideoMAEConfig, ViltConfig, VisionTextDualEncoderConfig, VisualBertConfig, ViTConfig, ViTMAEConfig, ViTMSNConfig, Wav2Vec2Config, Wav2Vec2ConformerConfig, WavLMConfig, WhisperConfig, XCLIPConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, YolosConfig, YosoConfig.
</pre>

Comments:
Hi,

1) did you use Transformers from the main branch?
2) you should probably do the following

```
model_ckpt = "path_to_your_checkpoint"
!python -m transformers.onnx --model={model_ckpt} --feature=vision2seq-lm onnx/ --atol 1e-3
```

@NielsRogge ` --feature=vision2seq-lm` worked for me. Thank you!

@NielsRogge . I would like to get the inference script after onnx conversion of VisionEncoderDecoder model. Any suggestion please?


@kangsan0420 looking at 
```
model_ckpt = "path_to_your_checkpoint"
!python -m transformers.onnx --model={model_ckpt} --feature=vision2seq-lm onnx/ --atol 1e-3
```
if downloading from the pertained trocr model, where is the path to the checkpoint?

@NielsRogge 
I have fine tuned the trocr small printed model on a custom single line text dataset. After training I converted the model to onnx format using the following  [PR](https://github.com/huggingface/transformers/pull/19254#issue-1392234601). Converting the model to onnx results in drastic decrease in accuracy. 

Also, if I use the pretrained trocr small printed model and convert the same to onnx using exactly same procedure, there is  a very small change in accuracy.

Can someone explain why is there so much change in accuracy.

Please reply if you need additional information.
Thanks.

hello guys, exist an onnx model for TrOcr 
someone answer me please
and thank you

> @NielsRogge . I would like to get the inference script after onnx conversion of VisionEncoderDecoder model. Any suggestion please?

@NielsRogge thanks I have been able to convert my Donut model to ONNX format. Any idea how I can proceed to perform inference for the onnx model ?

@Mir-Umar @Kamilya2020 Please open issues in the Optimum repository: https://github.com/huggingface/optimum