Trainer very slow to start training
Author: alexcoca
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.19.2
- Platform: Linux-4.18.0-372.19.1.el8_6.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.6.0
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes



### Who can help?

@sgugger , this is related to the trainer

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have adapted the run_summarisation.py script to perform dialogue state tracking. I use T5 base model, starting from Google's v1_1 pretrained model. My dataset contains 9,608,136 examples, which are split into 6 json files.

My dataset is loaded using:

```
    raw_datasets = load_dataset(extension, data_files=data_files, cache_dir=model_args.cache_dir, field='data')

```
Above `data_files` is a list of `6` json files, each containing 1/6 of the data.

Then it is processed using:

```
  train_dataset = train_dataset.map(
      preprocess_function,
      batched=True,
      num_proc=data_args.preprocessing_num_workers,
      remove_columns=column_names,
      load_from_cache_file=not data_args.overwrite_cache,
      desc="Running tokenizer on train dataset",
  )
```

After this, I initialize the trainer and call train:

```        train_result = trainer.train(resume_from_checkpoint=checkpoint)
```

In the past, I noticed that there is about a half an hour delay between the method call and when the training actually starts. However, without any change to the code the trainer seems to start very slowly. My logs look as follows:

```
09/29/2022 03:30:18 - INFO - __main__ - Initialising trainer...
09/29/2022 03:30:18 - INFO - __main__ - Starting training...
[INFO|trainer.py:470] 2022-09-29 03:30:19,248 >> max_steps is given, it will override any value given in num_train_epochs
09/29/2022 03:30:19 - INFO - __main__ - Starting training...
[INFO|trainer.py:1419] 2022-09-29 04:41:11,893 >> ***** Running training *****
[INFO|trainer.py:1420] 2022-09-29 04:41:11,893 >>   Num examples = 9608136
[INFO|trainer.py:1421] 2022-09-29 04:41:11,893 >>   Num Epochs = 2
[INFO|trainer.py:1419] 2022-09-29 04:41:11,894 >> ***** Running training *****
[INFO|trainer.py:1422] 2022-09-29 04:41:11,893 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:1420] 2022-09-29 04:41:11,894 >>   Num examples = 9608136
[INFO|trainer.py:1421] 2022-09-29 04:41:11,894 >>   Num Epochs = 2
[INFO|trainer.py:1423] 2022-09-29 04:41:11,893 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:1422] 2022-09-29 04:41:11,894 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:1424] 2022-09-29 04:41:11,893 >>   Gradient Accumulation steps = 2
[INFO|trainer.py:1423] 2022-09-29 04:41:11,894 >>   Total train batch size (w. parallel, distributed & accumulation) = 256
[INFO|trainer.py:1425] 2022-09-29 04:41:11,893 >>   Total optimization steps = 40000
[INFO|trainer.py:1424] 2022-09-29 04:41:11,894 >>   Gradient Accumulation steps = 2
[INFO|trainer.py:1425] 2022-09-29 04:41:11,894 >>   Total optimization steps = 40000
```

Notice it takes over 1 hour to actually start running the training loop. The expected training time is 12 hrs on 8 A100 gpus across two nodes. However, 4h and 20 minutes later I had trained 342 steps (usually there should be more than one step per second). 

I have no idea what is actually going on, why is the trainer so slow to start? The processing of the data seems to be happening rather quickly. I would expect there is a delay at startup because presumably the large dataset has to be put in a table for all the  workers to access. Why is this so slow? 

@sgugger needless to say I can make the code and data available to you for a deep investigation. 

### Expected behavior

A lag between the `.train` call and the training running is expected because the arrow table has to be built. However, I would expect that following the initial lag, the training happens rather quickly, but this does not happen.

Comments:
Hi there. You should probably try the [forums](https://discuss.huggingface.co/) to have the community help you debug the code. Without a clear and concise reproducer, there is nothing I can to fix it on my side

@sgugger this issue has been flagged before but there is no answer. I'm 100% the code is correct (I wrote an entire paper based on experiments I did on this code).  Hence why I was keen to make the code available to you so that we can figure out why the trainer gets slowed down so much.

Some posts I checked:

https://discuss.huggingface.co/t/it-takes-so-long-before-the-model-start-training-wav2vec2-fine-tuning/5384 

This is not helpful at all. 

I also checked this post (https://github.com/huggingface/datasets/issues/4394) ... My data is already split in six shards, about 1.7GB each so I am already following best practice?

For posterity, crazy slow training and waiting time for training to start can also be due to slow I/O, so people struggling should check their storage works fine. 

> For posterity, crazy slow training and waiting time for training to start can also be due to slow I/O, so people struggling should check their storage works fine.

I ran into the same problem, but I only used 64 pieces of data as my dataset. Did you find any other reason？