Loading tokenized dataset stucks for multi-gpu but works for multi-node single-gpu on some computing nodes
Author: BiEchi
Pull request: no

### System Info

[2023-10-14 20:15:10,690] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.32.0.dev0
- Platform: Linux-3.10.0-1160.31.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.0
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Single-node Multi-GPU

### Who can help?

@ArthurZucker

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I run the `run_mlm.py` example provided in the latest `transformers` release, and the process stucks at this bunch of code: https://github.com/huggingface/transformers/blob/21dc5859421cf0d7d82d374b10f533611745a8c5/examples/pytorch/language-modeling/run_mlm.py#L513

For example, if I run using 2 GPUs on a single node using this code:

```

function pretrain_cont(){
    python -m torch.distributed.launch \
        --nproc_per_node=8 \
        --nnodes=1 \
        --node_rank=0 \
        --master_port=19500 \
        --use_env \
        transformers/examples/pytorch/language-modeling/run_mlm.py \
        --config_name $MODEL \
        --tokenizer_name $TOKENIZER \
        --dataset_name $PT_DATASET \
        --max_steps $MAX_STEPS \
        --preprocessing_num_workers 32 \
        --ddp_timeout 180000 \
        --save_strategy steps \
        --save_steps 0.05 \
        --fp16 \
        --cache_dir $CACHE_DIR \
        --per_device_train_batch_size $BATCH_SIZE \
        --per_device_eval_batch_size $BATCH_SIZE \
        --gradient_accumulation_steps $ACCUMULATE \
        --adam_epsilon $PT_ADAM_EPS \
        --adam_beta1 $PT_ADAM_BETA1 \
        --adam_beta2 $PT_ADAM_BETA2 \
        --weight_decay $PT_ADAM_WEIGHT_DECAY \
        --warmup_steps $WARMUP_STEPS \
        --learning_rate $PT_PEAK_LR \
        --lr_scheduler_type $PT_LR_DECAY \
        --max_seq_length 512 \
        --do_train \
        --do_eval \
        --output_dir $OUTPUT_DIR \
        --overwrite_output_dir
}
```

The tokenization process will get stuck at the line I mentioned above and give such results (as I've tokenized before, it simply loads the preprocessed dataset):

```
...
.cache/User___bert_pretrain_datasets/default/0.0.0/9107755b15521c04/cache-f6e24321e136c7cb_*_of_00032.arrow
Concatenating 32 shards
10/14/2023 20:15:51 - INFO - datasets.arrow_dataset - Concatenating 32 shards
```

### Expected behavior

The same code works on all of other of my machines (it doesn't get stuck), but don't work on one of my machine, so I'm writing to ask how I can debug this issue. Looking forward to your support!

Comments:
Hey! Thanks for reporting. 
I am not really sure we can debug this for you, but if you are looking for tips on how to debug this I would recommend you to ask this on [the forum](https://discuss.huggingface.co/)! 🤗 

I've solved it by prepending a NCCL environment varaible to disable the p2p communication. Thanks!

Thanks for sharing the fix 😉