Tr-OCR Large Checkpoint model diverges
Author: nogifeet
Pull request: no

### System Info

Transformers Version -- 4.35.2
Python Version -- 3.10.12  [GCC 11.4.0]
Environment -- Google Colab 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When using the large checkpoint, the initial max_token_length is 20, which is not ideal as the tokens could be larger for different images. When playing with this parameter we notice that the model starts diverging and producing repeated tokens in some of the sample images.

Please use the below sample image
![e04-083-00](https://github.com/huggingface/transformers/assets/72322393/b8c20b9e-bc8b-441a-a417-2131e0af78c6)


    from transformers import TrOCRProcessor, VisionEncoderDecoderModel
    import requests
    from PIL import Image
    
    processor = TrOCRProcessor.from_pretrained("microsoft/trocr-large-handwritten")
    model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-large-handwritten")
    
    image = Image.open("/content/e04-083-00.png").convert("RGB")
    
    pixel_values = processor(image, return_tensors="pt").pixel_values
    generated_ids = model.generate(pixel_values,use_cache=True,max_new_tokens=100)
    
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
    
    print(generated_text)
    // The edges of the transoms should be bevelled to be edges to the edges of the edges of the edges of the edges of the 
    edges of the edges of the edges of the edges of the edges of the edges of the edges of the edges of the edges of 
    the edges of the edges of the edges of the edges of the edges of the edges of the edges of the edges of the edges 
    of the edges of the edges of the edges of the edges of the edges of the edges of the
    
    print(len(generated_text))
    // 427

### Expected behavior

I would expect the large checkpoint to behave similarly or even better than the base checkpoint...

    from transformers import TrOCRProcessor, VisionEncoderDecoderModel
    import requests
    from PIL import Image
    
    processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten")
    model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")
    
    # load image from the IAM dataset
    image = Image.open("/content/e04-083-00.png").convert("RGB")
    
    pixel_values = processor(image, return_tensors="pt").pixel_values
    generated_ids = model.generate(pixel_values,use_cache=True,max_new_tokens=100)
    
    generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
    
    print(generated_text)
    // The edges of the transoms should be bevelled to
    
    print(len(generated_text))
    // 47

Comments:
Hi @nogifeet, thanks for raising this issue! 

Hmmm, I'm not sure what's happening here. However, I don't think there's an issue in the code itself. 

Clearly the larger model isn't generating the eos token. However, the processing and generation configs for the models are equivalent, and the only difference in the models' configs is their encoder hidden size (expected). 

Following the paper, both models have been trained to predict eos and the only difference should be the size of the vision encoder, which matches with the difference in the hidden sizes in the model configs.  

The only thing that's slightly unexpected is the models are adding a pooling layer, which is randomly initialized (but this would affect both) cc @NielsRogge 

In terms of your experiments - does the large model always have this behaviour or only occasionally i.e. do we ever observe it correctly generating the eos token? 

Hello @amyeroberts. There are examples where the eos is generated successfully and the example mentioned is actually an outlier. I can provide more examples from the IAM dataset if you want.

I would also like to point out that I don't think it's a problem isolated to the large checkpoint, I have observed them in the base checkpoints.


@nogifeet OK, in this case I'd recommend looking into [different decoding strategies](https://huggingface.co/docs/transformers/generation_strategies#decoding-strategies) when generating to try and force more sensible sequences. Some more information can be found here: https://huggingface.co/blog/how-to-generate. 

@amyeroberts Is the handling of decoder layer nodes with shape limitations, managed internally by Hugging Face to prevent undefined behavior?

> Is the handling of decoder layer nodes with shape limitations, managed internally by Hugging Face to prevent undefined behavior?

Sorry, I don't understand. What do you mean by "decoder layer nodes with shape limitations"? 
You should be able to call `model.generate` and all the handling of autoregressive calls and generation logic should be handled for you. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I have the same issues fine-tuning a base model on a custom dataset. If i up the max_length parameter the model starts to diverge, and the more epochs i train, the more frequent the repeating behaviour at the end of the generated string become