Can't load pretrained TrOCR model
Author: nmntz
Pull request: no

@NielsRogge I get this error when I try to load a local TrOCR checkpoint.
```python
>>> processor = TrOCRProcessor.from_pretrained("./checkpoint-2")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/users/gpupro/gpu_tazi/.local/lib/python3.8/site-packages/transformers/processing_utils.py", line 186, in from_pretrained
    args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/usr/users/gpupro/gpu_tazi/.local/lib/python3.8/site-packages/transformers/processing_utils.py", line 230, in _get_arguments_from_pretrained
    args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
  File "/usr/users/gpupro/gpu_tazi/.local/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 544, in from_pretrained
    tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
  File "/usr/users/gpupro/gpu_tazi/.local/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 564, in __getitem__
    raise KeyError(key)
KeyError: <class 'transformers.models.vision_encoder_decoder.configuration_vision_encoder_decoder.VisionEncoderDecoderConfig'>
```
This is the content of my checkpoint folder:
```
checkpoint-2
  |-trainer_state.json
  |-preprocessor_config.json
  |-training_args.bin
  |-scaler.pt
  |-optimizer.pt
  |-scheduler.pt
  |-pytorch_model.bin
  |-rng_state.pth
  |-config.json
```

Yet, loading a TrOCR checkpoint from the hub works just fine.

Comments:
I have exactly the same problem. If anyone finds a solution, it would be appreciated. If I manage to solve it, I will post it here ASAP.

EDIT: 
I have not been able to follow the code trace exactly but I believe the error is as follows. When the model is created from the `hub.py` file, the model class is `DeitConfig`, however, when the model is created from a configuration file, the model appears as `VisionEncoderDecoderConfig`. In the `transformers.models.auto.auto_factory file`, it is validated that the model is among the tokenizers defined in `TOKENIZER_MAPPING` and if it is not, the error is raised. That is, the behavior of loading the same file from the hub and from local differs and that is what is causing this problem.


Could be related to https://github.com/huggingface/transformers/issues/14884

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I have the same problem, Is any solution to this?

@NouamaneTazi  If the new version 4.19 still doesn't work with your local checkpoint, could you please upload your checkpoint/config files to your HF Hub repo, and provide a link to it?

@emigomez Do you use a local checkpoint or a checkpoint from the Hub? Could you try with v4.19?

Hi @emigomez , if the solution proposed by @ydshieh doesn't work for you, what I did was to train the model as in the tutorials and, once trained, load the model from the generated checkpoint:

```
model = VisionEncoderDecoderModel.from_pretrained(/path/to/local/checkpoint)
```

And the preprocessor I load it from the hub (using the same checkpoint of the pretrained model), for example, if you have finetuned the model from `microsoft/trocr-base-printed`:
```
preprocessor = TrOCRProcessor.from_pretrained(`microsoft/trocr-base-printed`).
```
This way the combination model + preprocessor works for me, allowing me to use the models to predict. 

I hope it helps you!


Hi @ydshieh  I am using my local checkpoints obtained after fine tuning, and I am using the v4.19 yes
I have generated this pretrained model with     
```
    trainer = Seq2SeqTrainer(.....)
    trainer.train()
    trainer.save_model("./models")
```
    
Hi @CuarteroAlvaro I was trying to make the inference with my model with:
```
    MODEL_PATH = "./models/checkpoints/checkpoint_10000/" # option 1
    MODEL_PATH = "./models/" # option 2
    processor = TrOCRProcessor.from_pretrained(MODEL_PATH) 
    model = VisionEncoderDecoderModel.from_pretrained(MODEL_PATH)   
```
    
My error appears when I was trying to load the processor, so as you suggested I load the one used in my training:
`    processor = TrOCRProcessor.from_pretrained('microsoft/trocr-base-printed') `
    
This is the complete code that I'm using to infer:
```
from transformers import TrOCRProcessor, VisionEncoderDecoderModel
import requests 
from PIL import Image
import time
import torch

MODEL_PATH = "./models/checkpoints/checkpoint-10000/" # option 1
MODEL_PATH = "./models/" # option 2

processor = TrOCRProcessor.from_pretrained('microsoft/trocr-base-printed') 
model = VisionEncoderDecoderModel.from_pretrained(MODEL_PATH)

device = 'cuda' if torch.cuda.is_available() else 'cpu'
print("Running in device:", device)
model.to(device)

image = Image.open("0_0.tif").convert("RGB")
timeini = time.time()

pixel_values = processor(image, return_tensors="pt").pixel_values 
generated_ids = model.generate(pixel_values)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] 

timeend = time.time() - timeini
print("\nResult: ", generated_text)
print("Execution time: ", timeend)
```

With both  MODEL_PATH options I obtain the next error:
```
$ python trocr_infer.py 
Running in device: cuda
Traceback (most recent call last):
  File "trocr_infer.py", line 21, in <module>
    generated_ids = model.generate(pixel_values)
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\generation_utils.py", line 1172, in generate
    model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\generation_utils.py", line 525, in _prepare_encoder_decoder_kwargs_for_generation
    model_kwargs["encoder_outputs"]: ModelOutput = encoder(**encoder_kwargs)
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\models\vit\modeling_vit.py", line 572, in forward
    embedding_output = self.embeddings(
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\models\vit\modeling_vit.py", line 135, in forward
    embeddings = self.patch_embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding)
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\models\vit\modeling_vit.py", line 191, in forward
    x = self.projection(pixel_values).flatten(2).transpose(1, 2)
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\conv.py", line 447, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\Users\MSI\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\conv.py", line 443, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor
```

And these are the files that I have under the previous folders:
models/:
`checkpoints/  config.json  preprocessor_config.json  pytorch_model.bin  runs/  training_args.bin`
models/checkpoints/checkpoint-10000/:
`config.json  optimizer.pt  preprocessor_config.json  pytorch_model.bin  rng_state.pth  scaler.pt  scheduler.pt  trainer_state.json  training_args.bin`

Do you know how to fix this problem? 

Thank you both for your quick reply!!



@emigomez Would you mind to share a complete code snippet of your training script and arguments, so we can reproduce the issue quickly in order to identify the cause?

Without training, I couldn't reproduce the issue (by just loading/saving/loading-again)

```
from transformers import TrOCRProcessor, VisionEncoderDecoderModel

processor = TrOCRProcessor.from_pretrained('microsoft/trocr-base-printed')
model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-base-printed')
model.save_pretrained("./local_checkpoint")
loaded_model = VisionEncoderDecoderModel.from_pretrained('./local_checkpoint')
print(loaded_model)
```

The problem was that my model is on the GPU, but my data is on the CPU. So, I need to send my data to GPU changing on the previous code that I have shared:

`generated_ids = model.generate(pixel_values.to(device))`

And that is working.

Thank you!!

Yes, i was just writing the answer!



@ydshieh, The problem appears when trying to load the preprocessor from a local checkpoint. I think this code will reproduce the issue:

```python
from transformers import TrOCRProcessor, VisionEncoderDecoderModel

processor = TrOCRProcessor.from_pretrained('microsoft/trocr-base-printed')
model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-base-printed')
model.save_pretrained("./local_checkpoint")
loaded_preprocessor = TrOCRProcessor.from_pretrained('./local_checkpoint')
loaded_model = VisionEncoderDecoderModel.from_pretrained('./local_checkpoint')
print(loaded_preprocessor, loaded_model)

```

Hi, @CuarteroAlvaro 

In you code snippet, there is 1 line missing
```
processor.save_pretrained('./local_checkpoint')  # <-- this is required 
```
The following will work
```
from transformers import TrOCRProcessor, VisionEncoderDecoderModel

processor = TrOCRProcessor.from_pretrained('microsoft/trocr-base-printed')
model = VisionEncoderDecoderModel.from_pretrained('microsoft/trocr-base-printed')
model.save_pretrained("./local_checkpoint")
processor.save_pretrained('./local_checkpoint')  # <-- this is required 
loaded_preprocessor = TrOCRProcessor.from_pretrained('./local_checkpoint') 
loaded_model = VisionEncoderDecoderModel.from_pretrained('./local_checkpoint')
print(loaded_preprocessor, loaded_model)
```

But I can't reproduce the error shown in @NouamaneTazi 's original issue.
```
KeyError: <class 'transformers.models.vision_encoder_decoder.configuration_vision_encoder_decoder.VisionEncoderDecoderConfig'>
```

I don't seem to be able to reproduce the problem, so I'll mark this as resolved! Thanks for your help 🤗