Title: Load Data Sets Too Slow In Train Seq2seq Model
State: closed

### Describe the bug


step 'Generating train split' in load_dataset is too slow：
![image](https://github.com/huggingface/datasets/assets/19569322/d9b08eee-95fe-4741-a346-b70416c948f8)


### Steps to reproduce the bug

Data： own data，16K16B Mono wav
Oficial Script:[ run_speech_recognition_seq2seq.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py)
Add Code：
    if data_args.data_path is not None:
        print(data_args.data_path)
        raw_datasets = load_dataset("audiofolder", data_dir=data_args.data_path, cache_dir=model_args.cache_dir)
        raw_datasets = raw_datasets.cast_column("audio", Audio(sampling_rate=16000))
        raw_datasets = raw_datasets["train"].train_test_split(test_size=0.005, shuffle=True)
 （change cache_dir to other path ，ex:/DATA/cache）

### Expected behavior

load data fast,at least 1000+
`Generating train split: 387875 examples [32:24:45, 1154.83 examples/s]`

### Environment info


- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.4.0-149-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.13.2
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Comments: Hi ! you can speed it up using multiprocessing by passing `num_proc=` to `load_dataset()`already did，but not useful for step Generating train split，it works in step "Resolving data files" & "Downloading data files" @mariosasko some advice ， thanks！I met the same problem,  terrible  experience@mariosasko We need more info about the issue to provide help. 

Can you interrupt the process (with `num_proc=None`) after the `load_dataset` call when the slowdown occurs? So we can know what part of the code is causing it.

The `audiofolder` \ `imagefolder` with metadata is not performant for large datasets. Luckily, we can make them much faster if drop the nested metadata files feature (not that useful). I plan to work on this soon.

In the meantime, it's better to use `Dataset.from_generator` (requires replacing the `load_dataset` calls in the transformers script with `Dataset.from_generator`) or write a dataset loading script for large datasets.Can you interrupt the process (with num_proc=None) after the load_dataset call when the slowdown occurs? So we can know what part of the code is causing it.
（I'll try this operation）
The audiofolder \ imagefolder with metadata is not performant for large datasets. Luckily, we can make them much faster if drop the nested metadata files feature (not that useful). I plan to work on this soon.
(My data is indeed a bit large, exceeding 10000 hours of audio data. Looking forward to your improvement work very much)

In the meantime, it's better to use Dataset.from_generator (requires replacing the load_dataset calls in the transformers script with Dataset.from_generator) or write a dataset loading script for large datasets.
（I want to use Dataset.from_generator instead of  load_dataset ，where can i found sample code to load audio&label dataset， I was to do asr task）Can you interrupt the process (with num_proc=None) after the load_dataset call when the slowdown occurs? So we can know what part of the code is causing it.
================================================================================
Here is the log：
[load_dataset.log](https://github.com/huggingface/datasets/files/12169362/load_dataset.log)
（The larger my training data, the slower it loads）
![image](https://github.com/huggingface/datasets/assets/19569322/381b73e4-0a54-4240-b95e-cb8164584047)

In the meantime, it's better to use Dataset.from_generator (requires replacing the load_dataset calls in the transformers script with Dataset.from_generator) or write a dataset loading script for large datasets.
================================================================================
I tried ‘Dataset. from_generator’ implements data loading, but the testing results show no improvementI have already solved this problem, referring to #5990 : read audio frist, then use data_generator to change format .