Title: Stuck in "Resolving data files..."
State: open

### Describe the bug

I have an image dataset with 300k images, the size of image is 768 * 768.

When I run `dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')` in second time, it takes 50 minutes to finish "Resolving data files" part, what's going on in this part?

From my understand, after Arrow files been created in the first run, the second run should not take time longer than one or two minutes.

### Steps to reproduce the bug

# Run following code two times
dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')

### Expected behavior

Fast dataset building

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.3

Comments:
Most likely, the data file inference logic is the problem here.

You can run the following code to verify this:
```python
import time
from datasets.data_files import get_data_patterns
start_time = time.time()
get_data_patterns("/path/to/img_dir")
end_time = time.time()
print(f"Elapsed time: {end_time - start_time:.2f}s")
```
 
We plan to optimize this for the next version (or version after that). In the meantime, specifying the split patterns manually should give better performance:
```python
ds = load_dataset("imagefolder", data_files={"train": "path/to/img_dir/train/**", ...}, split="train")
```

Hi, @mariosasko, you are right; data file inference logic is extremely slow.

I have done a similar test, that is I modify the source code of datasets/load.py to measure the cost of two suspicious operations:
```python
def get_module(self) -> DatasetModule:
        base_path = Path(self.data_dir or "").expanduser().resolve().as_posix()
        start = time.time()
        patterns = sanitize_patterns(self.data_files) if self.data_files is not None else get_data_patterns(base_path)
        print(f"patterns: {time.time() - start}")
        start = time.time()
        data_files = DataFilesDict.from_patterns(
            patterns,
            download_config=self.download_config,
            base_path=base_path,
        )
        print(f"data_files: {time.time() - start}")
```
It gaves:
patterns: 3062.2050700187683
data_files: 413.9576675891876

Thus, these two operations contribute to almost all of load time. What's going on in them?

Furthermore, what's my current workaround about this problem? Should I save it by `save_to_disk()` and load dataset through `load_from_disk`?

were you able to solve this issue?, I am facing the same issue

Still suffering from this issue. For me, I cannot download Emilia dataset, and stucked at 'Resolving data files' forever.
``` e = load_dataset('amphion/Emilia-Dataset', token=my_token)
Resolving data files:   0%|                                                                                  | 1/4343 [00:00<11:27,  6.32it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████| 4343/4343 [00:01<00:00, 3844.15it/s]
```