Error in reproducing results with checkpoint saved by Trainer.
Author: saxenarohit
Pull request: no

### System Info

I am trying to use the official example summarization.py
https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization.py

The model trains and the prediction results are fine (rouge-1 34) .
But when I try to load the model from the checkpoint, the results seem to be quite off (rouge-1 18).
I can see multiple threads online facing similar issues but there seems to be no concrete solution.

I am loading model using AutoModelForSeq2SeqLM.from_pretrained() with checkpoint path
I am saving 2 checkpoints and have tried both.

I now feel there is some error in saving the checkpoint.

Transformers version `4.37.2`

Thanks.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction


Sample configuration which I am using.
```
{
     "model_name_or_path": "google/pegasus-x-base",
     "learning_rate" : 4e-4,
     "lr_scheduler_type" : "linear",
     "warmup_ratio":0.01,
     "num_train_epochs" : 30,
     "output_dir": "./exp/pegasus_x_base",
     "overwrite_output_dir": false,
     "do_train": false,
     "do_eval": false,
     "do_predict": true,
     "evaluation_strategy": "steps",
     "per_device_train_batch_size": 8,
     "per_device_eval_batch_size": 8,
     "predict_with_generate": true,
     "logging_strategy":"steps",
     "logging_steps":1,
     "evaluation_strategy": "steps",
     "eval_steps": 50,
     "save_total_limit":2,
     "load_best_model_at_end": true,
     "metric_for_best_model":"rouge1",
     "gradient_accumulation_steps" : 1,
     "seed": 42,
     "weight_decay":0.01,
     "generation_max_length":1024,
     "generation_num_beams":4,
     "max_source_length":16384,
     "gradient_checkpointing":true,
     "fp16":true
 }
```

The model ends with prediction results below
```
***** predict metrics *****
12530   predict_gen_len            =      128.0
12531   predict_loss               =     8.9858
12532   predict_rouge1             =    34.7833
12533   predict_rouge2             =     6.3896
12534   predict_rougeL             =    18.9697
12535   predict_rougeLsum          =    30.8148
```

Loading checkpoint results

```
'predict_gen_len': 127.9,
'predict_loss': 38.49708938598633, 
'predict_rouge1': 18.6118, 
'predict_rouge2': 4.2491, 
'predict_rougeL': 10.4743, 
'predict_rougeLsum': 17.4561, 
```

### Expected behavior

Results after loading the model from the checkpoint should match the results of prediction as the last step during training.

Comments:
Hey, sorry it's gonna be super hard to debug without knowing how you trained the model, where it was saved , the logs etc etc. This probably means that you are not loading the best model, maybe the latest 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.