Loaded Donut models lost their predictive power 
Author: Edgar454
Pull request: no

### System Info

Lately , every time i load a model it seems to have lost all it has learned. I checked if the state dict are identical after and before and yes they are
Moreover i have already use this code in the past and didn’t experience any problem. It’s the first time i encountering it

Code to save
```
def on_train_end(self, trainer, pl_module):
        print(f"Pushing model to the hub after training")
        pl_module.processor.push_to_hub("Edgar404/donut-shivi-cheques_320",
                                    commit_message=f"Training done")
        pl_module.model.push_to_hub("Edgar404/donut-shivi-cheques_320",
                                    commit_message=f"Training done")
```

Code used to load :

```
from transformers import DonutProcessor, VisionEncoderDecoderModel

processor = DonutProcessor.from_pretrained("Edgar404/donut-shivi-cheques_320")
model = VisionEncoderDecoderModel.from_pretrained("Edgar404/donut-shivi-cheques_320")
model = model.to(device)
```

@SunMarc @muellerz @zunn

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Train the model using the Donut Demo notebook on cord
2. Save the model
3. Try to use this model for inference

### Expected behavior

**Expected Behavior :** Loaded model shoul keep the same accuracy  before and after the save
**Actual Behavior :** The loaded model generate incoherent outputs as if it was never trained



Comments:
It's an issue relate to transformers version 41.1
I retrograded it to version 40.0 and it worked

Hi @Edgar454, just to clarify, if you load the same saved with different versions of transformers the performance is different? 

@Edgar454 did you train with version 4.41.1? I am running into the same issue with my DONUT model (using transformers 4.41.1). When loading the trained model I am receiving a garbage output along the lines of: `










` Despite the training curves indicating that the model is fine-tuning to the new task. 


Edit:
- The model needs to be trained with the transformers package reverted back to 4.40. It does not work if you train with version 4.41 and then attempt inference with version 4.40. Hope this helps. 

@sguerin13 If there's a degradation between versions then it's a regression we should try and track down. So that we can look into this, could you provide some more information? 

Specifically:
* Could you share a script, showing how you're training the model? 
* If you train a model in v4.40 and then load in v4.41, do you see a similar degradation? 
* If you train a model in v4.41 and then load in v4.40, do you see an improvement in performance? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.