Keep getting ChildFailedError Error in distributed Eval/Train
Author: IdoAmit198
Pull request: no

### System Info

- `transformers` version: 4.25.1   
- Platform: Linux-4.15.0-200-generic-x86_64-with-glibc2.29
- Python version: 3.8.10 
- Huggingface_hub version: 0.11.1 
- PyTorch version (GPU?): 1.12.1+cu113 (True)     
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed  
- JaxLib version: not installed 
- Using GPU in script?: Yes, multiple GPUs on a single node.
- Using distributed or parallel set-up in script?: Yes. Running my script with `torchrun`.

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm running a slightly modified [run_clm.py script](https://github.com/huggingface/transformers/blob/v4.24.0/examples/pytorch/language-modeling/run_clm.py) with vary number of A100 GPUs (2-8) on a single node, and keep getting the ChildFailedError right after the training/evaluation ends.
I’m running [GPT2 (smallest model)](https://huggingface.co/gpt2) on the [OpenWebText dataset](https://huggingface.co/datasets/openwebtext).

### An example how I run my code from a shell script is as follow:
> GPU=0,1,2,3,4,5
export TORCH_CPP_LOG_LEVEL=INFO NCCL_DEBUG=INFO
export CUDA_VISIBLE_DEVICES=$GPU
>
> torchrun \
    --standalone \
    --nnodes=1 \
    --nproc_per_node=${NUM_GPU} \
    run_clm.py \
    --model_name_or_path ${MODEL} \
    --dataset_name ${DS_NAME} \
    --preprocessing_num_workers 16 \
    --logging_steps 5000 \
    --save_steps ${SAVE_STEPS} \
    --do_eval \
    --per_device_eval_batch_size ${EVAL_BATCH} \
    --seed ${RANDOM} \
    --evaluation_strategy steps \
    --logging_dir ${OUTPUT_DIR} \
    --output_dir ${OUTPUT_DIR} \
    --overwrite_output_dir \

### And getting the follow error:
> 100%|██████████| 3155/3155 [43:38<00:00,  1.20it/s]WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 650 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 651 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 652 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 653 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 654 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 655 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 0 (pid: 649) of binary: /venv/bin/python3
Traceback (most recent call last):
  File "/venv/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/venv/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/torch/distributed/run.py", line 761, in main
    run(args)
  File "/venv/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run
    elastic_launch(
  File "/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
./code/gpt2/Model-Compression-Research-Package/examples/transformers/language-modeling/run_clm.py FAILED
Failures:
  <NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
  time      : 2022-12-26_10:59:55
  host      : distributed-05-pt266-zvdww
  rank      : 0 (local_rank: 0)
  exitcode  : -9 (pid: 649)
  error_file: <N/A>
  traceback : Signal 9 (SIGKILL) received by PID 649
============================================================

### Full log is attached:
[eval_log.txt](https://github.com/huggingface/transformers/files/10303349/eval_log.txt)

### Notes:
1. The error occurs in training and in evaluation.
2. I tried to run using torchrun and using torch.distributed.launch and faced the same issue.
3. The number of samples in my training/eval doesn’t affect and the issue remain.
4. I track my memory usage and OOM is not the case here (kinda wish it was).
5. The error occurs only in distributed setup. When not using distributed, or when using it with a single GPU, the problem doesn't pop.
6. The error doesn't reproduce in much smaller dataset, such as wikitext-2. In this case both train and eval works in distributed setup.


### Expected behavior

Expect evaluation/training to finish successfully, log results (sample per second, loss, perplexity, etc..) and save json files of results, as I achieve in non-distributed setup.

Comments:
Update: after reading this [thread](https://github.com/facebookresearch/detectron2/issues/3319), tried to add the following exports:
>export NCCL_IB_DISABLE=1
export NCCL_P2P_DISABLE=1

But no luck.
The warnings regards `NCCL_P2P` disappeared and the same error remains.

Thanks for your report. I have no idea why PyTorch gobbles the error message, but without any clue in the logs there is little we can do to investigate if you don't share the script you are running.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.