EarlyStoppingCallback Not Working with Accelerate
Author: superleesa
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-5.15.0-1042-gcp-x86_64-with-glibc2.38
- Python version: 3.11.7
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: Yes

### Who can help?

_No response_

### Information

- [] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Train llama2 (I used a fine-tuned llama2 for Japanese) with accelerate (I'm training with DDP using four GPUs), using a script that uses Trainer API and EarlyStoppingCallback


1. please download the scripts i used [finetune.py and data_loader.py](https://github.com/superleesa/dump)
1. on terminal run `accelerate launch finetune.py`

note: i ran this without any accelerate configurations

### Expected behavior

I'm fine-tuning llama2 with accelerate with DDP using four GPUs.  I used a script that uses Trainer API and EarlyStoppingCallback.  Whenever I run the code and after few iterations of training, I get following error:

```
[E ProcessGroupNCCL.cpp:475] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3523, OpType=ALLREDUCE, NumelIn=7828736, NumelOut=7828736, Timeout(ms)=1800000) ran for 1800228 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:475] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3523, OpType=ALLREDUCE, NumelIn=7828736, NumelOut=7828736, Timeout(ms)=1800000) ran for 1800550 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:475] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3523, OpType=ALLREDUCE, NumelIn=7828736, NumelOut=7828736, Timeout(ms)=1800000) ran for 1800903 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:475] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3524, OpType=BROADCAST, NumelIn=33554432, NumelOut=33554432, Timeout(ms)=1800000) ran for 1800371 milliseconds before timing out.

972ee88fbdea:1599:1757 [0] NCCL INFO [Service thread] Connection closed by localRank 0
972ee88fbdea:1599:1724 [0] NCCL INFO comm 0x1c635a70 rank 0 nranks 4 cudaDev 0 busId 40 - Abort COMPLETE

[E ProcessGroupNCCL.cpp:489] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:495] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:916] [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3524, OpType=BROADCAST, NumelIn=33554432, NumelOut=33554432, Timeout(ms)=1800000) ran for 1800371 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 0] NCCL watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=3524, OpType=BROADCAST, NumelIn=33554432, NumelOut=33554432, Timeout(ms)=1800000) ran for 1800371 milliseconds before timing out.

[2024-01-15 15:57:12,861] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1600 closing signal SIGTERM
[2024-01-15 15:57:12,861] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1601 closing signal SIGTERM
[2024-01-15 15:57:12,861] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1602 closing signal SIGTERM
[2024-01-15 15:57:12,976] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 
```

Some insights:
- I first searched online for the error and some people suggested disabling NCCL P2P and increasing the timeout threshold but both of them did not work.
- then, **when i removed theEarlyStoppingCallback the code worked fine, so i found that it must be related to EarlyStoppingCallback**. Therefore, although I use lots of other components in the code including LoRA and 4 bit quantization, the error must be related to EarlyStoppingCallback.
- because the error only happens whenever the training loop should stop with the earlystopping callback (i.e. the validation loss does not decrease for the specified "persistance" number of times), i suspected that this is caused by the training not being able to quit from loop.
- i read the source code and found that when the early stopping counter stops, control.should_training_stop is to set True.  however, within the training loop in the Trainer class, i believe this is not handled properly for training that uses multiple GPUs.
- particularly, i suspect it's not breaking the training loop and there might be an improper use of the break statement in combination with how Accelerate works.
- i assume this is basically the same problem as [discussed here](https://discuss.huggingface.co/t/early-stopping-for-eval-loss-causes-timeout/51349). only the difference is that he is using his own training script but i'm using the Trainer.
- to fix this, i think we need to use accelerate.set_breakpoint and accelerate.check_breakpoint functions within the [_inner_training_loop function in the Trainer class](https://github.com/huggingface/transformers/blob/7e0ddf89f483f53107870cddabb2e1cc93069705/src/transformers/trainer.py#L1933C1-L1934C26), as they were used to fix the problem in the discussion link above.
- **so i believe the real problem here is not the EarlyStoppingCallback but the condition to exit the training loop.**

Comments:
Actually, the problem occurred even after removing the EarlyStoppingCallback..... 

cc @muellerzr @pacman100 

The weirdest part is that the error always occurs at a training step between 229~235. I was thinking that this is due to a deadlock caused by evaluation / saving but even if I set the the evaluation & saving step to 300 (i.e. the first evaluation & saving step happens at 300 which is after the error occurs), the error still occurs, implying that they are not likely the cause. 

Also, I tried this with both DDP and Deepspeed but the same problem happens for the both.

Gentle ping  @muellerzr @pacman100

Hmmm.... I couldn't reproduce this. The `break` should happen on every version regardless because the loss will be the same across all nodes after syncing, which is when this triggers, and why there shouldn't be a failure. 

You mention deadlock, can you share your `TrainingArguments`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.