Keep getting ChildFailedError Error
Labels: oncall: distributed

### 🐛 Describe the bug

I'm running a slightly modified [run_clm.py script](https://github.com/huggingface/transformers/blob/v4.24.0/examples/pytorch/language-modeling/run_clm.py) with vary number of A100 GPUs (4-8) on a single node, and keep getting the ChildFailedError right after the training/evaluation ends.
I’m running [GPT2 (smallest model)](https://huggingface.co/gpt2) on the [OpenWebText dataset](https://huggingface.co/datasets/openwebtext).

### I'm running my code from a shell script as follow:
> GPU=1,2,3,4,5
export TORCH_CPP_LOG_LEVEL=INFO NCCL_DEBUG=INFO
export CUDA_VISIBLE_DEVICES=$GPU
>
> torchrun \
    --standalone \
    --nnodes=1 \
    --nproc_per_node=${NUM_GPU} \
    run_clm.py \
    --model_name_or_path ${MODEL} \
    --dataset_name ${DS_NAME} \
    --preprocessing_num_workers 16 \
    --logging_steps 5000 \
    --save_steps ${SAVE_STEPS} \
    --do_eval \
    --per_device_eval_batch_size ${EVAL_BATCH} \
    --seed ${RANDOM} \
    --evaluation_strategy steps \
    --logging_dir ${OUTPUT_DIR} \
    --output_dir ${OUTPUT_DIR} \
    --overwrite_output_dir \
    --ddp_timeout 324000 \

### And getting the follow error:
> 100%|██████████| 2209/2209 [39:03<00:00,  1.06s/it]WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4041 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4042 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4043 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 4045 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 3 (pid: 4044) of binary: /venv/bin/python3
Traceback (most recent call last):
  File "/venv/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/venv/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/torch/distributed/run.py", line 719, in main
    run(args)
  File "/venv/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
    elastic_launch(
  File "/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
./code/gpt2/Model-Compression-Research-Package/examples/transformers/language-modeling/run_clm.py FAILED
Failures:
  <NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
  time      : 2022-12-21_21:57:29
  host      : december-ds-2h4b6-5hpkj
  rank      : 3 (local_rank: 3)
  exitcode  : -9 (pid: 4044)
  error_file: <N/A>
  traceback : Signal 9 (SIGKILL) received by PID 4044
============================================================

### Full log is attached:
[debug_flags_log.txt](https://github.com/pytorch/pytorch/files/10282140/debug_flags_log.txt)

### Notes:
1. The error occurs in training and in evaluation.
2. In order to eliminate the option of timeout I deliberately fixed very high value for timeout.
3. I tried to run using torchrun and using torch.distributed.launch and faced the same issue.
4. The number of samples in my training/eval doesn’t affect and the issue remain.
5. I track my memory usage and OOM is not the case here (kinda wish it was).
6. The error occurs only in distributed setup. When not using distributed, or when using it with a single GPU, the problem doesn't pop.

Would really appreciate any help on this issue since I'm really stuck on my research till then :(

### Versions

According to running `pip freeze --local`:

**torch==1.10.2+cu113
torchaudio==0.10.2+cu113
torchvision==0.11.3+cu113
nvidia-ml-py==11.495.46
multiprocess==0.70.14
packaging==21.3
transformers==4.25.1
huggingface-hub==0.11.1**
absl-py==1.0.0
accelerate==0.15.0
aiohttp==3.8.3
aiosignal==1.3.1
async-timeout==4.0.2
attrs==22.1.0
blessed==1.19.1
cachetools==5.0.0
certifi==2021.10.8
charset-normalizer==2.0.12
click==8.1.3
datasets==2.7.1
deepspeed==0.6.0
dill==0.3.6
docker-pycreds==0.4.0
evaluate==0.4.0
fairscale==0.4.6
filelock==3.8.2
frozenlist==1.3.3
fsspec==2022.11.0
gitdb==4.0.10
GitPython==3.1.29
google-auth==2.6.0
google-auth-oauthlib==0.4.6
gpustat==1.0.0
grpcio==1.44.0
hjson==3.0.2
idna==3.3
importlib-metadata==4.11.2
joblib==1.2.0
Markdown==3.3.6
model-compression-research @ file:///
multidict==6.0.3
ninja==1.10.2.3
nltk==3.8
numpy==1.22.3
oauthlib==3.2.0
pandas==1.5.2
pathtools==0.1.2
Pillow==9.0.1
pkg_resources==0.0.0
promise==2.3
protobuf==3.19.4
psutil==5.9.0
py-cpuinfo==8.0.0
pyarrow==10.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.7
python-dateutil==2.8.2
pytz==2022.7
PyYAML==6.0
regex==2022.10.31
requests==2.27.1
requests-oauthlib==1.3.1
responses==0.18.0
rsa==4.8
scikit-learn==1.2.0
scipy==1.9.3
sentencepiece==0.1.97
sentry-sdk==1.12.0
setproctitle==1.3.2
shortuuid==1.0.11
six==1.16.0
sklearn==0.0.post1
smmap==5.0.0
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
threadpoolctl==3.1.0
tokenizers==0.13.2
tqdm==4.63.0
typing_extensions==4.1.1
urllib3==1.26.13
wandb==0.13.7
wcwidth==0.2.5
Werkzeug==2.0.3
xxhash==3.1.0
yarl==1.8.2
zipp==3.7.0


cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu