Process hangs when evaluating the model before finishing an epoch using `accelerate` in a multi-GPU environment (no trainer).
Author: beyondguo
Pull request: no

### System Info

- `transformers` version: 4.34.1
- Platform: Linux-3.10.0-1160.31.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.8
- Huggingface_hub version: 0.18.0
- Safetensors version: 0.4.0
- Accelerate version: 0.31.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: fp16
        - use_cpu: False
        - debug: False
        - num_processes: 2
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm using 2-gpu to train a roberta model, and I want to calculate the valication loss after `eval_steps`, here's the core code:
```python
...
model, optimizer, train_dataloader, val_dataloader = accelerator.prepare(model, optimizer, train_dataloader, val_dataloader)
...
best_eval_loss = float('inf')

# Training loop:
for epoch in range(num_epochs): 
    for batch in train_dataloader:
        model.train()
        outputs = model(**batch)
        loss = outputs.loss 
        accelerator.backward(loss)
        optimizer.step()
        optimizer.zero_grad()
        progress_bar.update(1)

        # validation when reach 'eval_steps'
        if progress_bar.n > 10 and progress_bar.n % args.eval_steps == 0:
            model.eval()
            losses = []
            for val_batch in val_dataloader:
                # ----------------> stuck here!
                with torch.no_grad():
                    outputs = model(**val_batch)
                e_loss = outputs.loss
                losses.append(accelerator.gather_for_metrics(e_loss.repeat(val_bs)))
            losses = torch.cat(losses)
            eval_loss = torch.mean(losses)
            print("eval_loss", eval_loss.item())
            
            # save the current best model
            if eval_loss.item() < best_eval_loss:
                print("Current best model!, Steps:", progress_bar.n, "Eval loss:", eval_loss.item())
                # save
                ...
```
The process just got stuck when computing the first validation batch.  Two GPUs are both 100% utilization, so I guess the process is hanging. 

After a long time, an error will occur:
```
[E ProcessGroupNCCL.cpp:828] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=2315, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1808543 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:828] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=2315, OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1808546 milliseconds before timing out.
...
Traceback (most recent call last):
  File "rolling_model_train.py", line 289, in <module>
    train()
  File "rolling_model_train.py", line 195, in train
    losses.append(accelerator.gather_for_metrics(loss.repeat(val_bs)))
  File "/home/guoby/app/Anaconda3-2021.05/envs/news/lib/python3.8/site-packages/accelerate/accelerator.py", line 2217, in gather_for_metrics
[E ProcessGroupNCCL.cpp:455] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:460] To avoid data inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=2315, OpType=ALLGATHER, Timeout(ms)=1800000) ran for 1808546 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:455] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:460] To avoid data inconsistency, we are taking the entire process down.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=2315, OpType=ALLREDUCE, Timeout(ms)=1800000) ran for 1808543 milliseconds before timing out.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 89866) of binary: /home/guoby/app/Anaconda3-2021.05/envs/news/bin/python
Traceback (most recent call last):
  File "/home/guoby/app/Anaconda3-2021.05/envs/news/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/guoby/app/Anaconda3-2021.05/envs/news/lib/python3.8/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/home/guoby/app/Anaconda3-2021.05/envs/news/lib/python3.8/site-packages/accelerate/commands/launch.py", line 977, in launch_command
    multi_gpu_launcher(args)
  File "/home/guoby/app/Anaconda3-2021.05/envs/news/lib/python3.8/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
    distrib_run.run(args)
  File "/home/guoby/app/Anaconda3-2021.05/envs/news/lib/python3.8/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/guoby/app/Anaconda3-2021.05/envs/news/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/guoby/app/Anaconda3-2021.05/envs/news/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
======================================================
rolling_model_train.py FAILED
------------------------------------------------------
Failures:
[1]:
  time      : 2024-06-20_15:02:22
  rank      : 1 (local_rank: 1)
  exitcode  : -6 (pid: 89867)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 89867
------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-06-20_15:02:22
  rank      : 0 (local_rank: 0)
  exitcode  : -6 (pid: 89866)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 89866
======================================================
```

Note that, **if I move the validation code after each epoch, instead during one epoch, the program runs fine!** Which is like this:
```python
# Training loop:
for epoch in range(num_epochs): 
    for batch in train_dataloader:
        model.train()
        outputs = model(**batch)
        loss = outputs.loss 
        accelerator.backward(loss)
        optimizer.step()
        optimizer.zero_grad()
        progress_bar.update(1)

    # validation:
    model.eval()
    losses = []
    for val_batch in val_dataloader:
        with torch.no_grad():
            outputs = model(**val_batch)
        e_loss = outputs.loss
        losses.append(accelerator.gather_for_metrics(e_loss.repeat(val_bs)))
    losses = torch.cat(losses)
    eval_loss = torch.mean(losses)
    print("eval_loss", eval_loss.item())

```

That is strange! Could anyone help me?

### Expected behavior

Able to evaluate model after a training step in multi-gpu env.

Comments:
I've tried to disable some NCCL features (https://github.com/huggingface/accelerate/issues/2174) but dosen't work.

cc @muellerzr @SunMarc 

Solved! It is becaues I set the progress_bar enbaled only on the main process:
```
progress_bar = tqdm(range(num_epochs*len(train_dataloader)), total=total_steps, disable= not accelerator.is_main_process)
```
So this caused only the main process enter into the evaluate part, so the `gather_for_metrics` is waiting for the other process forever....

Nothing wrong with `transformers` or `accelerate`, lol.


Make sense ! Thanks for the explanation !