Distil* hanging on torch.distributed.barrier()
Author: higopires
Pull request: no

### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.14.21-150400.24.55-default-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 1.13.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

@VictorSanh @sgugger 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm trying to run [Distil*](https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation) project with a custom dataset. After the preprocessing steps, I enter the following command to start the training (single-node multi-GPU):

```
CUDA_VISIBLE_DEVICES=0,1,2 N_GPU_NODE=3 N_NODES=1 NODE_RANK=0 python -m torch.distributed.launch \
    --nproc_per_node=3 \
    train.py \
    	--force \
    	--dump_path serialization_dir/my_first_training \
    	--data_file ./data/binarized_text.roberta-base.pickle \
    	--student_type roberta \
    	--student_config ./training_configs/distilroberta-base.json \
    	--student_pretrained_weights ~/higo/distilbert/serialization_dir/tf_roberta_048131723.pth \
    	--teacher_type roberta \
    	--teacher_name roberta-base \
    	--mlm \
    	--temperature 2.0 \
    	--alpha_ce 5.0 \
    	--alpha_mlm 2.0 \
    	--alpha_clm 0.0 \
    	--alpha_mse 0.0 \
    	--alpha_cos 1.0 \
    	--token_counts ./data/token_counts.binarized_text.roberta-base.pickle \
    	--freeze_pos_embs \
    	--freeze_token_type_embds \
    	--n_epoch 4 \
    	--batch_size 8 \
    	--gradient_accumulation_steps 256 \
    	--learning_rate 2e-4 \
    	--n_gpu 3 \
    	--seed 42
```
When the script arrives [here](https://github.com/huggingface/transformers/blob/a074a5d34d6411fb00e83a2ed30acf23d8c976b5/examples/research_projects/distillation/distiller.py#L343), it get stuck, the train does not start and after a given span of time, I got a timeout error. I tried to set higher timeout values (up to 3 hours), with no result. At this given point, my `nvidia-smi` is shown like this:

```
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100 80GB PCIe           Off| 00000000:4F:00.0 Off |                    0 |
| N/A   34C    P0               71W / 300W|   2479MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100 80GB PCIe           Off| 00000000:52:00.0 Off |                    0 |
| N/A   34C    P0               69W / 300W|   1851MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA A100 80GB PCIe           Off| 00000000:CE:00.0 Off |                    0 |
| N/A   36C    P0               68W / 300W|   1827MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA A100 80GB PCIe           Off| 00000000:D1:00.0 Off |                    0 |
| N/A   42C    P0               66W / 300W|  63151MiB / 81920MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A     69824      C   .../home/u021274/higo/myenv/bin/python     2476MiB |
|    1   N/A  N/A     69825      C   .../home/u021274/higo/myenv/bin/python     1848MiB |
|    2   N/A  N/A     69826      C   .../home/u021274/higo/myenv/bin/python     1824MiB |
|    3   N/A  N/A     97463      C   python3                                   63148MiB |
+---------------------------------------------------------------------------------------+
```

(Process at GPU 3 is from another researcher)

To me, the little amount of allocated memory seems odd. I honestly don't have a clue of what can be happening. Checked some other threads, but nothing helped to make things clear.

### Expected behavior

Start of single-node, multi-GPU distributed training.

Comments:
This is probably a setup error in the environment, as it means that one of the processes does not properly ping the others at the barrier line, and then it hangs forever.

Is there a way to solve this setup error?

i have same problem

I run into this problem recently.

> 

for me it was gpu communication issue
i used multi gpu from gpu cluster server.
but gpu cannot communicate each other so they cannot find other gpu comes to barrier() function. then they wait forever.
it was gpu environment setting issue
So I have contacted the cluster manager.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.