BrokenPipe Error while training GPT-2 from scratch with run_clm.py
Author: sanprit
Pull request: no

### System Info

Transformers version:  v4.24.0
Python version: 3.9.13 
system: EC2 with g5.12xlarge, Ubuntu, Pytorch (1.12.1)
Data size : ~250GB 


### Who can help?

@patil-suraj, @patrickvonplaten, @LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`python3 -m torch.distributed.launch --nproc_per_node 4 transformers/examples/pytorch/language-modeling/2_run_clm.py     --tokenizer_name hf_tokenizer_model     --model_type gpt2     --train_file  train_file.txt     --validation_split_percentage 10    --per_device_train_batch_size 32     --per_device_eval_batch_size 8     --gradient_accumulation_steps 8     --eval_accumulation_steps 8     --block_size 512     --do_train     --do_eval     --fp16     --output_dir MODELS     --logging_dir logs     --cache_dir cache     --overwrite_output_dir yes     --overwrite_cache yes     --num_train_epochs 10     --no_cuda False     --learning_rate 1e-5     --save_on_each_node False     --seed 42     --disable_tqdm False     --config_overrides="n_layer=12,vocab_size=96000,eos_token_id=0,bos_token_id=0"     --auto_find_batch_size True      --logging_strategy steps     --save_strategy steps     --evaluation_strategy steps     --logging_steps 25000     --save_steps 25000     --eval_steps 25000     --preprocessing_num_workers 24     --dataloader_num_workers 24     --save_total_limit  15`


```
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/pool.py", line 136, in worker
    put((job, i, (False, wrapped)))
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/queues.py", line 380, in put
    self._writer.send_bytes(obj)
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/connection.py", line 208, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/connection.py", line 419, in _send_bytes
    self._send(header + buf)
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/connection.py", line 376, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Running tokenizer on dataset #2: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 38169/38170 [3:20:47<00:00,  3.17ba/s]
Process ForkPoolWorker-3:
Traceback (most recent call last):00%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 38169/38170 [3:20:47<00:00,  5.32ba/s]
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/pool.py", line 131, in worker
    put((job, i, result))
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/queues.py", line 380, in put
    self._writer.send_bytes(obj)
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/connection.py", line 208, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/connection.py", line 419, in _send_bytes
    self._send(header + buf)
  File "/opt/conda/envs/pytorch/lib/python3.9/site-packages/multiprocess/connection.py", line 376, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

```

This happend when tokenization was just about to complete

### Expected behavior

After training, Grouping of texts (tokens) should start

Comments:
Hey @sanprit,

What exactly is `2_run_clm.py`? Was this a typo and supposed to be `run_clm.py`? Also we would need to be able to re-run this command to reproduce the error => how can we access `train_file.txt`?

2_run_clm.py is run_clm.py only with some logging comments added in it. Regarding train_file it is too large for me to share, can you please sun it on random 230 GB data

Now, I have diagnosed the problem, "why I was getting broken pipe error". While tokenising and creating groups of the texts, in midway it requires huge disk size to store the cache, Although It deletes major part in the end. When working/testing with small dataset, I was checking only the final cache size, and I was extrapolating that size for big data. But keeping disk size by extrapolating the final cache size is not true because in the midway it creates very high memory cache.

one observation: Even for 20MB of text data, final cache size is only 138GB but in midway of tokenising and grouping, it forms 500GB cache, so we will need minimum 500GB disk. else broken pipe error will occur

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.