trainer is not reproducible 
Author: jackfeinmann5
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.1
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 1.9 
- Tensorflow version (GPU?): - 
- Using GPU in script?: - 
- Using distributed or parallel set-up in script?: - 

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

trainer: @sgugger

## Information

Model I am using T5-small model and I am testing the original run_translation.py codes [1] for reproducibility when we need to restart the codes from the previously saved checkpoints (I only have access to gpus for a short time and I need to restart the codes). 



## To reproduce

Steps to reproduce the behavior:

1) Please kindly run this command:
```
python run_translation.py     --model_name_or_path t5-small     --do_train     --do_eval     --source_lang en     --target_lang ro     --source_prefix "translate English to Romanian: "     --dataset_name wmt16     --dataset_config_name ro-en     --output_dir /temp/jack/tst-translation     --per_device_train_batch_size=4     --per_device_eval_batch_size=4     --overwrite_output_dir     --predict_with_generate --max_steps 100 --eval_step 10 --evaluation_strategy steps  --max_train_samples 100 --max_eval_samples 100  --save_total_limit 1  --load_best_model_at_end  --metric_for_best_model bleu  --greater_is_better true
```

 then kindly break the codes in this points:

```
{'eval_loss': 1.3589547872543335, 'eval_bleu': 10.9552, 'eval_gen_len': 18.05, 'eval_runtime': 4.0518, 'eval_samples_per_second': 24.68, 'eval_steps_per_second': 6.17, 'epoch': 0.8}
 20%|██████████████████████████████▍                                                                                                                         | 20/100 [00:11<00:21,  3.70it/s[INFO|trainer.py:1919] 2021-07-29 17:22:43,852 >> Saving model checkpoint to /temp/jack/tst-translation/checkpoint-20                                              
[INFO|configuration_utils.py:379] 2021-07-29 17:22:43,857 >> Configuration saved in /temp/jack/tst-translation/checkpoint-20/config.json
[INFO|modeling_utils.py:997] 2021-07-29 17:22:44,351 >> Model weights saved in /temp/jack/tst-translation/checkpoint-20/pytorch_model.bin
[INFO|tokenization_utils_base.py:2006] 2021-07-29 17:22:44,355 >> tokenizer config file saved in /temp/jack/tst-translation/checkpoint-20/tokenizer_config.json
[INFO|tokenization_utils_base.py:2012] 2021-07-29 17:22:44,357 >> Special tokens file saved in /temp/jack/tst-translation/checkpoint-20/special_tokens_map.json
 29%|████████████████████████████████████████████                                                                                                            | 29/100 [00:14<00:22,  3.20it/s][INFO|trainer.py:2165] 2021-07-29 17:22:46,444 >> ***** Running Evaluation *****
[INFO|trainer.py:2167] 2021-07-29 17:22:46,444 >>   Num examples = 100
[INFO|trainer.py:2170] 2021-07-29 17:22:46,444 >>   Batch size = 4
```

break here please 

```
{'eval_loss': 1.3670727014541626, 'eval_bleu': 10.9234, 'eval_gen_len': 18.01, 'eval_runtime': 3.9468, 'eval_samples_per_second': 25.337, 'eval_steps_per_second': 6.334, 'epoch': 2.4}
[INFO|trainer.py:1919] 2021-07-29 17:24:01,570 >> Saving model checkpoint to /temp/jack/tst-translation/checkpoint-60                                                       
[INFO|configuration_utils.py:379] 2021-07-29 17:24:01,576 >> Configuration saved in /temp/jack/tst-translation/checkpoint-60/config.json            | 60/100 [00:23<00:11,  3.42it/s]
[INFO|modeling_utils.py:997] 2021-07-29 17:24:02,197 >> Model weights saved in /temp/jack/tst-translation/checkpoint-60/pytorch_model.bin
[INFO|tokenization_utils_base.py:2006] 2021-07-29 17:24:02,212 >> tokenizer config file saved in /temp/jack/tst-translation/checkpoint-60/tokenizer_config.json
[INFO|tokenization_utils_base.py:2012] 2021-07-29 17:24:02,218 >> Special tokens file saved in /temp/jack/tst-translation/checkpoint-60/special_tokens_map.json
[INFO|trainer.py:1995] 2021-07-29 17:24:03,216 >> Deleting older checkpoint [/temp/jack/tst-translation/checkpoint-50] due to args.save_total_limit

         [INFO|trainer.py:2165] 2021-07-29 17:24:03,810 >> ***** Running Evaluation *****██████████████████████████████▉                                               | 69/100 [00:26<00:09,  3.37it/s]
[INFO|trainer.py:2167] 2021-07-29 17:24:03,810 >>   Num examples = 100
[INFO|trainer.py:2170] 2021-07-29 17:24:03,810 >>   Batch size = 4
```


break here please and then run the codes please from here till the end.

```
final train metrics
***** train metrics *****
  epoch                    =        4.0
  train_loss               =     0.1368
  train_runtime            = 0:00:27.13
  train_samples            =        100
  train_samples_per_second =     14.741
  train_steps_per_second   =      3.685
07/29/2021 17:25:08 - INFO - __main__ - *** Evaluate ***
[INFO|trainer.py:2165] 2021-07-29 17:25:08,774 >> ***** Running Evaluation *****
[INFO|trainer.py:2167] 2021-07-29 17:25:08,774 >>   Num examples = 100
[INFO|trainer.py:2170] 2021-07-29 17:25:08,774 >>   Batch size = 4
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [00:08<00:00,  2.92it/s]
***** eval metrics *****
  epoch                   =        4.0
  eval_bleu               =    24.3863
  eval_gen_len            =      32.84
  eval_loss               =     1.3565
  eval_runtime            = 0:00:09.08
  eval_samples            =        100
  eval_samples_per_second =     11.005
  eval_steps_per_second   =      2.751
```


the final metrics when running the codes without breaks:

```
***** train metrics *****
  epoch                    =        4.0
  train_loss               =     0.3274
  train_runtime            = 0:01:04.19
  train_samples            =        100
  train_samples_per_second =      6.231
  train_steps_per_second   =      1.558
07/29/2021 17:00:12 - INFO - __main__ - *** Evaluate ***
[INFO|trainer.py:2165] 2021-07-29 17:00:12,315 >> ***** Running Evaluation *****
[INFO|trainer.py:2167] 2021-07-29 17:00:12,315 >>   Num examples = 100
[INFO|trainer.py:2170] 2021-07-29 17:00:12,315 >>   Batch size = 4
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25/25 [00:08<00:00,  2.97it/s]
***** eval metrics *****
  epoch                   =        4.0
  eval_bleu               =    24.3863
  eval_gen_len            =      32.84
  eval_loss               =     1.3565
  eval_runtime            = 0:00:08.95
  eval_samples            =        100
  eval_samples_per_second =     11.164
  eval_steps_per_second   =      2.791
```

the training loss between the two runs with and without break would be different. 
I kindly appreciate having a look, this is required for me to be able to use the great huggingface codes. and I would like to appreciate a lot your great work and colleague on this second to none, great work you are doing. thanks a lot.

## Expected behavior
 to see the same training loss when the user trains the codes without any break and when we train the codes with breaking in between.

Comments:
The average training loss is indeed not saved and thus you will have a different one restarting from a checkpoint. It's also not a useful metric in most cases, which is why we don't bother. You will notice however that your eval BLEU is exactly the same, so the training yielded the same model at the end.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.