How to load the best performance checkpoint after  training？
Author: Gpwner
Pull request: no

When I was training MLM with `https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_mlm.py`
, I have already set the `--load_best_model_at_end True`.But when  finish the training，I check the trainer_state.json file and I found these mssage:
```
  "best_metric": null,
  "best_model_checkpoint": null,
  "epoch": 100.0,
  "global_step": 559300,
  "is_hyper_param_search": false,
  "is_local_process_zero": true,
  "is_world_process_zero": true,
```
as shown above，"best_model_checkpoint" is null.


Here is my question，how to load the best performance checkpoint？If I have ONLY one checkpoint，is 
it the best performance checkpoint？Thanks in advance!

@sgugger 

Comments:
Could you give us the whole command you executed? We can't reproduce the problem without it.

@sgugger Here is my whole command:
`python run_mlm.py --model_name_or_path model/roberta-base  --do_train --do_eval --output_dir mlm_new --max_seq_length 128 --load_best_model_at_end True --save_steps 2000 --overwrite_output_dir True  --per_device_train_batch_size 32 --train_file data/wikitext-train.txt --validation_file data/wikitext-validation.txt`



Where the wikitext-train.txt and wikitext-validation.txt  are txt files ,something like ：

` Three other applications make up nearly all the rest of the consumption . One of these uses is as a stabilizer and a catalyst for the production of polyethyleneterephthalate . Another application is to serve as a fining agent to remove microscopic bubbles in glass , mostly for TV screens ; this is achieved by the interaction of antimony ions with oxygen , interfering the latter from forming bubbles . The third major application is the use as pigment . 
 Antimony is being increasingly used in the semiconductor industry as a dopant for heavily doped n @-@ type silicon wafers in the production of diodes , infrared detectors , and Hall @-@ effect devices . In the 1950s , tiny beads of a lead @-@ antimony alloy were used to dope the emitters and collectors of n @-@ p @-@ n alloy junction transistors with antimony . Indium antimonide is used as a material for mid @-@ infrared detectors . `


IN MY case ,I  just change these two files to my domain corpus txt files.

another Question，what is the difference between `load_best_model_at_end ` and Early Stop？



Is there at least one evaluation during the training? I don't know the size of your dataset. Passing `--evaluation_strategy epoch` would ensure there is one per epoch at least.

As for your second question, `loas_best_model_at_end` loads the best model seen during the training. Early stopping, stops when the loss/metrics does not improve.

@sgugger 
1.YES，there is one evaluation at the end，But `best_model_checkpoint` is still null:
```
[INFO|trainer.py:2107] 2021-06-03 02:40:32,693 >> ***** Running Evaluation *****
[INFO|trainer.py:2109] 2021-06-03 02:40:32,693 >>   Num examples = 17979
[INFO|trainer.py:2112] 2021-06-03 02:40:32,693 >>   Batch size = 128
100%|█████████████████████████████████████████████████████████████████████████████████████| 141/141 [00:48<00:00,  2.92it/s]
[INFO|trainer_pt_utils.py:907] 2021-06-03 02:41:21,274 >> ***** eval metrics *****
[INFO|trainer_pt_utils.py:912] 2021-06-03 02:41:21,274 >>   epoch                     =        3.0
[INFO|trainer_pt_utils.py:912] 2021-06-03 02:41:21,274 >>   eval_loss                 =     1.4715
[INFO|trainer_pt_utils.py:912] 2021-06-03 02:41:21,274 >>   eval_mem_cpu_alloc_delta  =        0MB
[INFO|trainer_pt_utils.py:912] 2021-06-03 02:41:21,274 >>   eval_mem_cpu_peaked_delta =        0MB
[INFO|trainer_pt_utils.py:912] 2021-06-03 02:41:21,274 >>   eval_mem_gpu_alloc_delta  =        0MB
[INFO|trainer_pt_utils.py:912] 2021-06-03 02:41:21,274 >>   eval_mem_gpu_peaked_delta =     3928MB
[INFO|trainer_pt_utils.py:912] 2021-06-03 02:41:21,274 >>   eval_runtime              = 0:00:48.49
[INFO|trainer_pt_utils.py:912] 2021-06-03 02:41:21,274 >>   eval_samples              =      17979
[INFO|trainer_pt_utils.py:912] 2021-06-03 02:41:21,274 >>   eval_samples_per_second   =    370.704
[INFO|trainer_pt_utils.py:912] 2021-06-03 02:41:21,274 >>   perplexity                =     4.3558

```

After passing `--evaluation_strategy epoch`  ,`best_model_checkpoint` become normal and it is not null now.
```
  "best_metric": 1.4732710123062134,
  "best_model_checkpoint": "mlm_new1/checkpoint-426",
  "epoch": 3.0,
```

2.Does  `loads the best model seen during the training` mean the code will load the best model  in memory or  save it in disk? In my origin case (without passing `--evaluation_strategy epoch`) ，I Have only one checkpoint.Is it the best checkpoint or the last checkpoint？


No the evaluation at the end does not count, it's not part of the `train` method. If there is no evaluation during the training phase, there can't be a best model to load, it's as simple as that.

The `load_best_model_at_end` just keeps track of the best model as you evaluate it and will reload at the end the checkpoint that had the best evaluation score.

@sgugger I think if this process can  automatically save the best performance and  the last checkpoints ,that will be great.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.