Issue: Very slow data loading on large dataset
State: closed

I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ?

When you load a text file for the first time with `nlp`, the file is converted into Apache Arrow format. Arrow allows to use memory-mapping, which means that you can load an arbitrary large dataset.

Note that as soon as the conversion has been done once, the next time you'll load the dataset it will be much faster.

However for a 1TB dataset, the conversion can indeed take time. You could try to load parts of it in parallel, and then use `nlp.concatenate_datasets` to get your full dataset.

Humm, we can give a look at these large scale datasets indeed.

Do you mind sharing a few stats on your dataset so I can try to test on a similar one?

In particular some orders of magnitudes for the number of files, number of lines per files, line lengths.

@lhoestq Yes, I understand that the first time requires more time. The concatenate_datasets seems to be a workaround, but I believe a multi-processing method should be integrated into load_dataset to make it easier and more efficient for users.

@thomwolf Sure, here are the statistics:
Number of lines: 4.2 Billion
Number of files: 6K
Number of tokens: 800 Billion
The number of lines is distributed equally across these 6k files.
The line length varies between 100 tokens to 40k tokens.


@agemagician you can give a try at a multithreaded version if you want (currently on the #548).

To test it, you just need to copy the new `text` processing script which is [here](https://github.com/huggingface/nlp/blob/07d92a82b7594498ff702f3cca55c074e2052257/datasets/text/text.py) somewhere on your drive and give it's local path instead of `text` to `load_dataset`. E.g. in your example:
```python
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('./datasets/text.py',   # path to where you've dowloaded the multi-threaded text loading script
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

I have already generated the dataset, but now I tried to reload it and it is still very slow.

I also have installed your commit and it is slow, even after the dataset was already generated.
`pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257`

It uses only a single thread.

Did I miss something ?

As mentioned in #548 , each time you call `load_dataset` with `data_files=`, they are hashed to get the cache directory name. Hashing can be too slow with 1TB of data. I feel like we should have a faster way of getting a hash that identifies the input data files

I believe this is really a very important feature, otherwise, we will still have the issue of too slow loading problems even if the data cache generation is fast.

Hmm ok then maybe it's the hashing step indeed.

Let's see if we can improve this as well.

(you will very likely have to regenerate your dataset if we change this part of the lib though since I expect modifications on this part of the lib to results in new hashes)

Also, @agemagician you have to follow the step I indicate in my previous message [here](https://github.com/huggingface/nlp/issues/546#issuecomment-684648927) to use the new text loading script.

Just doing `pip install git+https://github.com/huggingface/nlp.git@07d92a82b7594498ff702f3cca55c074e2052257` like you did won't use the new script (they are not inside the library but hosted on our hub).

No problem, I will regenerate it. This will make us see if we solved both issues and now both the data generation step, as well as the hashing step, is fast.

Any news for the hashing ?

I'm working on it today :)

Ok so now the text files won't be hashed.

I also updated #548 to include this change.
Let us know if it helps @agemagician :)

Perfect thanks for your amazing work.

Right now, for caching 18Gb data, it is taking 1 hour 10 minute. Is that proper expected time? @lhoestq @agemagician 
In this rate (assuming large file will caching at the same rate) caching full mC4 (27TB) requires a month (~26 days). 


Hi ! Currently it is that slow because we haven't implemented parallelism for the dataset generation yet.
Though we will definitely work on this :)

For now I'd recommend loading the dataset shard by shard in parallel, and then concatenate them:
```python
# in one process, load first 100 files for english
shard1 = load_dataset("allenai/c4", data_files="multilingual/c4-en.tfrecord-000**.json.gz")
# in another process load next 100 files for english
shard2 = load_dataset("allenai/c4", data_files="multilingual/c4-en.tfrecord-001**.json.gz")

# finally
concatenate_datasets([shard1, shard2, ...])

Thanks for the help..!!!

Sorry to write on a closed issue but, has there been any progress on parallelizing the `load_dataset` function?

Hi ! No but this is in our plans (probably a few weeks)

I'm literally crying waiting for the trainer to restart from checkpoint. It's getting stuck at `get_train_dataloader` and I think this is to do with the same issue... has there been any progress on this?

> I'm literally crying waiting for the trainer to restart from checkpoint. It's getting stuck at get_train_dataloader and I think this is to do with the same issue...

Once the dataset is cached once, it's not regenerated again. Your issue seems different

hmmm, yes. I'll come back with details on this, fairly easy to reproduce. Takes about 30 minutes to get from checkpoint loading to starting training...

@lhoestq yo, any news in making it possible to download large datasets faster?

@lhoestq For some reason [setting num_proc](https://discuss.huggingface.co/t/how-can-i-multithreadedly-download-a-huggingface-dataset/56178/2?u=kopyl) does not work at all... My dataset has 58 parquet files and i was hoping passing `num_proc` to `load_dataset` would spawn 58 Python processes each downloading its own parquet so I can load my dataset in 1 minutes instead of 50...

It does spawn `num_proc` processes. Note that when you download in parallel you're often bounded by your bandwidth at one point, so 50 processes is unlikely to get you a x50 download speed up but a bit less

> It does spawn `num_proc` processes. Note that when you download in parallel you're often bounded by your bandwidth at one point, so 50 processes is unlikely to get you a x50 download speed up but a bit less

why then i see only 1 parquet file download progress bar at a time?

Ah indeed parallel downloads are not enabled yet for some datasets.
I opened https://github.com/huggingface/datasets/pull/6551 to fix this.


@lhoestq thank you very much, i though for a second i'm just tripping.