Repository: huggingface/transformers

Issue:
Add automatic best model loading to Trainer
# What does this PR do?

This PR cleans up a bit the part that saves the training state inside `Trainer` and adds an API that can track which was the best model during any of the evaluation phases to load it back at the end.

When fine-tuning a model on a dataset that can easily overfit the model, it's quite common to have the last model not be the best one (in terms of metrics). This PR adds a `TrainingArgument` named `load_best_model_at_end` that triggers the following behavior:
- `save_steps` gets ignored and the model is saved every time there is an evaluation (determined by `evaluation_strategy` and `eval_steps`)
- It keeps track in a `TrainerState` of when the best model was encountered (that state is saved along the checkpoints so it can work with resuming a training)
- The best model is determined by the new `TrainingArgument`s `metric_for_best_model` (defaults to the loss) and `greater_is_better` (default to False for the loss, True otherwise).
- The best model is loaded once the training is finished.

In passing I've added some tests of the saving API in Trainer and made sure it can handle both `PreTrainedModel` and regular `nn.Module` (a feature asked in #6901). Both are now tested in the CI, as is the new API.

Fixes #6901

Those newly introduced arguments and APIs can then be leveraged to have early stopping supported in `Trainer`.


Discussion:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7431?src=pr&el=h1) Report
> Merging [#7431](https://codecov.io/gh/huggingface/transformers/pull/7431?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f62f2ffdcc2df75cf01438bebc7ae281d921d21d?el=desc) will **increase** coverage by `0.53%`.
> The diff coverage is `76.74%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/7431/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/7431?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #7431      +/-   ##
==========================================
+ Coverage   78.17%   78.71%   +0.53%     
==========================================
  Files         181      181              
  Lines       35800    35858      +58     
==========================================
+ Hits        27986    28224     +238     
+ Misses       7814     7634     -180     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/7431?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `63.23% <73.43%> (+7.80%)` | :arrow_up: |
| [src/transformers/trainer\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3V0aWxzLnB5) | `63.30% <80.00%> (+2.66%)` | :arrow_up: |
| [src/transformers/training\_args.py](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzLnB5) | `91.72% <100.00%> (+0.45%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_electra.py](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9lbGVjdHJhLnB5) | `24.25% <0.00%> (-73.56%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_lxmert.py](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9seG1lcnQucHk=) | `22.14% <0.00%> (-72.41%)` | :arrow_down: |
| [src/transformers/modeling\_rag.py](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yYWcucHk=) | `25.32% <0.00%> (-51.72%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG0ucHk=) | `58.52% <0.00%> (-34.74%)` | :arrow_down: |
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `61.53% <0.00%> (-33.66%)` | :arrow_down: |
| [src/transformers/modeling\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19tYXJpYW4ucHk=) | `60.00% <0.00%> (-30.00%)` | :arrow_down: |
| [src/transformers/modeling\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19sb25nZm9ybWVyLnB5) | `74.14% <0.00%> (-18.70%)` | :arrow_down: |
| ... and [25 more](https://codecov.io/gh/huggingface/transformers/pull/7431/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/7431?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7431?src=pr&el=footer). Last update [f62f2ff...738935a](https://codecov.io/gh/huggingface/transformers/pull/7431?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

IMO this closes #4186
@sgugger how does this work together with `save_total_limit` ? If it is set might it happen that the best model gets deleted?

well - see here https://github.com/huggingface/transformers/issues/7556
The best model is not deleted with `save_total_limit`. It is always put at the top of the list after sorting the chceckpoints.

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -1219,7 +1219,7 @@ def _sorted_checkpoints(self, checkpoint_prefix=PREFIX_CHECKPOINT_DIR, use_mtime
         checkpoints_sorted = [checkpoint[1] for checkpoint in checkpoints_sorted]
         # Make sure we don't delete the best model.
         if self.state.best_model_checkpoint is not None:
-            best_model_index = checkpoints_sorted.index(self.state.best_model_checkpoint)
+            best_model_index = checkpoints_sorted.index(str(Path(self.state.best_model_checkpoint)))
             checkpoints_sorted[best_model_index], checkpoints_sorted[-1] = (
                 checkpoints_sorted[-1],
                 checkpoints_sorted[best_model_index],


Test patch: