Add patience argument to Trainer
Author: thesamuel
Pull request: yes

This closes #4894.

# Summary
Often, we want to stop training if loss does not improve for a number of epochs. This PR adds a "patience" argument, which is a limit on the number of times we can get a non-improving eval loss before stopping training early.

It is implemented by other NLP frameworks, such as AllenNLP (see [trainer.py](https://github.com/allenai/allennlp/blob/master/allennlp/training/trainer.py#L95) and [metric_tracker.py](https://github.com/allenai/allennlp/blob/1a8a12cd1b065d74fec3d2e80105a684736ff709/allennlp/training/metric_tracker.py#L6)).

# Motivation
This feature allows faster fine-tuning by breaking the training loop early and avoids users the toil of checking metrics on Tensorboard.

# Caveats
Often, models are evaluated once per epoch, but run_lm_finetuning.py has an option to evaluate after a set number of model update steps (dictated by `--logging_steps` if `--evaluate_during_training` is true). Because of this, I've elected to tie patience to the number of evaluations without improvement in loss.

Comments:
This supercedes https://github.com/huggingface/transformers/pull/2840, where I added patience to the outdated `run_language_modeling.py` script.

Looking good! Can you add a reference to your original post that this closes https://github.com/huggingface/transformers/issues/4894? Thanks

Hello, when this feature will be merged? I would like to use it. Thank you.

> Hello, when this feature will be merged? I would like to use it. Thank you.

There are some changes requested that @thesamuel should fix before this can be merged.

Bump. Early stopping is critical for an automated Trainer that reliably gives us the best model. Current way of figuring out the training stopping point seems to be specifying a static train_epochs but the training duration a model can take depends on way too many factors like learning rate, data complexity, model, model size, optimizer and so on that it is unreasonable to ask the user to specify the epochs in advance.
I believe the current assumption is that people train with very small learning rates so that the loss always seems to keep decreasing very slowly but according to my experience (and on my data) it is a sub-optimal schedule which takes too much time. I see that training with higher learning rates with larger batch sizes and stopping at the early stopping point results in an equally good if not better models. Although this requires use of early stopping.

I would like to use this early stopping on downstream training.
The current implementation only stops training by monitoring loss. IMO it should also be possible to monitor other metrics like F1 and ROC-AUC. 

I also would like to add a feature that stores the model each time when the monitored metric improves and then optionaly loads the model after training. Then later evaluation can be done on this "best" model.

@thesamuel @julien-c @kevin-yauris what do you think?

I plan to work on this once I'm finished with the Funnel Transformer model @PhilipMay (so end of this week, beginning of the next).

> 
> 
> I plan to work on this once I'm finished with the Funnel Transformer model @PhilipMay (so end of this week, beginning of the next).

@sgugger That would be awsome. Maybe you want to get some inspiration from the FARM training loop which is pretty nice IMO:

https://github.com/deepset-ai/FARM/blob/master/farm/train.py#L262-L370

I just found this PR that was already merged: #7431
I think it solved this...

Not quite, but it makes implementing it easier.

> Not quite, but it makes implementing it easier.

Yes - you are right. The patience part is still missing. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@sgugger Should we keep this open? You wrote in this thread you will work on this if you find the time, but I am not sure if you plan to use another PR for that.

There has been a PR merged adding the `EarlyStoppingCallback` (#8581) so I think this can be closed now.

Thanks @cbrochtrup @sgugger! Sorry I didn't get around to this...

You're welcome, happy to help!