TrOCR wrong output after fine-tuning on specific tokenizer
Author: magistermilitum
Pull request: no

### System Info

I get funny outputs like that: '叡 叡 叡ělělěllioliolio 揮 揮 揮ത്തിത്തിത്തി ὴ ὴ ὴ Dans Dans Dans Wesley Wesley Wesley contaba contaba contaba ......

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have following Nielsr and replacing the decoder of the pretrained trocr-base-handwritten with a custom one as follows:

```python
processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
processor.tokenizer = AutoTokenizer.from_pretrained("magistermilitum/bert_medieval_multilingual") #or any other bert or roberta
processor.save_pretrained('./processor')

model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")
model.decoder = AutoModelForCausalLM.from_pretrained("magistermilitum/bert_medieval_multilingual", is_decoder=True, add_cross_attention=True)

model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
model.config.pad_token_id = processor.tokenizer.pad_token_id
model.config.vocab_size = model.config.decoder.vocab_size

model.save_pretrained("trainer_trocr/model")
```


In this fashion training works fine and at the end I have a model achiving CER = 0.09 and WER = 0.17 on the development set, which are overall good results (in any case better that using only trocr-base-handwritten). But, when using the generated model on inference there is a possible miss-alignement as using in this way: 

```python
device=torch.device("cpu")
processor = TrOCRProcessor.from_pretrained("./processor")
model = VisionEncoderDecoderModel.from_pretrained("trainer_trocr/model").to(device)

### load image 
url = "D:\\trocr\\1538983940018_363.png"
image = Image.open(url).convert("RGB")

pixel_values = processor(image, return_tensors="pt").pixel_values
generated_ids = model.generate(pixel_values)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

print(generated_ids)
print("OCR output:", generated_text)
```


I get funny outputs like that: 
```
叡 叡 叡ělělěllioliolio 揮 揮 揮ത്തിത്തിത്തി ὴ ὴ ὴ Dans Dans Dans Wesley Wesley Wesley contaba contaba contaba ......
```

Even worse, each time i pass the inference the output change but always using this patttern of 3 by 3 characters. Maybe i have not using  inference from a tokenizer-modified model in the proper way but i have not be able to find any readme indication about this.

### Expected behavior

a logical output

Comments:
cc @NielsRogge 

This was resolved here: https://github.com/huggingface/transformers/issues/19329#issuecomment-1869144991

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.