Early stopping patience does not work when resuming from checkpoint
Author: Ubadub
Pull request: no

### System Info

- `transformers` version: 4.33.3
- Platform: Linux-4.18.0-348.23.1.el8_5.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.16
- Huggingface_hub version: 0.16.2
- Safetensors version: 0.3.1
- Accelerate version: 0.24.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.0.post101 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@muellerzr and @pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Fundamentally the issue is that the `early_stopping_patience_counter` is not persisted when checkpointing. Consequently, it is [always (re)set to 0](https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/trainer_callback.py#L564) when initializing `Trainer`, including when resuming from checkpoint. This means that if, for example, you never train your model for `early_stopping_patience`-many evaluation steps at once before stopping and resuming from checkpoint, early stopping will never happen.

An auxiliary issue is that even if you train your model for longer than `early_stopping_patience`-many evaluation steps, and training correctly stops, if you happen to then re-initiate training from a checkpoint, training will resume [even though the run ended with `self.control.should_training_stop == True`](https://github.com/huggingface/transformers/blob/f4f57f9dfa68948a383c352a900d588f63f6290a/src/transformers/trainer_callback.py#L154). This is because this variable is also not persisted to the `trainer_state.json` file when checkpointing. This issue was reported in #10290, but was never resolved before the issue was closed as stale.

To reproduce the main issue, simply initiate a training run and set `early_stopping_patience` to a value of your choice, then interrupt training before the run gets there. Reinitiate training with `resume_from_checkpoint=True`. Rinse and repeat until `best_metric` increases for `early_stopping_patience`-many evaluation calls.

To reproduce the auxiliary issue, don't interrupt your run until it stops due to early stopping. When it is complete, reinitiate training with `resume_from_checkpoint=True`.

### Expected behavior

Early stopping patience should work exactly the same when stopping and resuming runs from a checkpoint as when training continuously without interruption.

Comments:
IMO, the problem can be generalised through the TrainingArguments parameter `save_only_model` ([link](https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/training_args.py#L307)):

Its definition states: "...when this is true, you won't be able to resume training from checkpoint.".
As the current [TrainerState](https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/trainer_callback.py#L35) implementation stands, we are not _truly_ able to resume training from a checkpoint even on setting `save_only_model=False`.

As pointed by @Ubadub here, Trainer's callbacks' states are not persisted along with models (even on setting `save_only_model=False`). To fix this issue (and the auxiliary issue #10290, pointed above), we need the capability to persist callbacks and load them when using `resume_from_checkpoint`.

In short, Trainer's callbacks should be a part of the [TrainerState](https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/trainer_callback.py#L35) object.

I can help in this implementation if this analysis seems reasonable.


Cheers!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I forgot to reply to this in time, but I am confirming that this issue is still present and should not be closed. @muellerzr and @pacman100 could one of you reopen this issue?

@tanmay17061's suggested solution makes a lot of sense to me. I can attempt a go at solving this myself, but likely not for a few more weeks at the earliest. 

Hi all, #29666 should solve the issue. I chose to write a relatively-scalable way for us to save callback data inside the `TrainerState` :) 

You can test it with `pip install git+https://github.com/huggingface/transformers@muellerzr-checkpoint-callbacks`

Thanks for your patience!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.