In distributed get SIGTERM and run crash
Labels: oncall: distributed

### 🐛 Describe the bug

Hi everyone,
I'm tying to fine-tune [GPT2-small](https://huggingface.co/gpt2) on the [OpenWebText dataset](https://huggingface.co/datasets/openwebtext) (A big dataset consist of 40GB+), by running a slightly modified HuggingFace [run_clm.py script](https://github.com/huggingface/transformers/blob/v4.24.0/examples/pytorch/language-modeling/run_clm.py) in distributed.
However, I get SIGTERM in different states of the training/inference without further explanation.
Didn't find a similar reported issue with a working solution yet so decided to post and ask for your help.

### Scheme of how I run my code:
    torchrun \
    --standalone \
    --nnodes=1 \
    --nproc_per_node=${NUM_GPU} \
    run_clm.py
    --ddp_timeout 3240000 \
    --ddp_find_unused_parameters False \
    --arg_1 --arg_2 ... arg_n

## My errors:
### Error on fine-tune:
When I tried to fine-tune my model, I suffered from error mid-way while tokenizing my loaded dataset after successfully tokenized chunk of data, but errored in the next chunk:
> Running tokenizer on dataset:   1%|          | 1/81 [00:04<06:15,  4.70s/ba]WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 464 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 465 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 466 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 467 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 468 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 469 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 0 (pid: 463) of binary: /venv/bin/python3
Traceback (most recent call last):
  File "/venv/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/venv/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
    return f(*args, **kwargs)
  File "/venv/lib/python3.8/site-packages/torch/distributed/run.py", line 719, in main
    run(args)
  File "/venv/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
    elastic_launch(
  File "/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/venv/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
./code/gpt2/Model-Compression-Research-Package/examples/transformers/language-modeling/run_clm.py FAILED
Failures:
  <NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
  time      : 2022-12-12_08:13:08
  host      : openwebtext-pcfz2-2zz8p
  rank      : 0 (local_rank: 0)
  exitcode  : -9 (pid: 463)
  error_file: <N/A>
  traceback : Signal 9 (SIGKILL) received by PID 463
============================================================

### Error on inference:
I was running my model in inference mode only, it took 3.5 hours to load, process the validation dataset (about 310MB of data) and run inference on those examples, and after it finished all steps, errored again the same non-informative SIGTERM error.

### Notes:
- I'm using high value for `ddp_timeout` in order to avoid timeout when loading and processing my huge dataset. 
- I track my memory usage and it doesn't seem like OOM errors.
- It doesn't seem to error due to timeout, since I dealt with it when placing high values for `ddp_timeout`
- The error doesn't appear in single GPU.
- The error appear much sooner when I use all my 8 GPUs (still 1 node), rather than when I use only few of them.

Would really appreciate any help on the issue!

### Versions

Ubuntu version = 20.04.4 LTS (Focal Fossa)

Python = 3.8.10
transformers = 4.24.0
torch = 1.10.2+cu113
datasets = 2.7.0
tokenizers = 0.13.2
accelerate = 0.14.0
evaluate = 0.3.0
numpy = 1.22.3