SalKhan12/Github_Issues huggingface dataset:
Issue with offline mode and partial dataset cached

### Describe the bug

Hi, 

a issue related to #4760 here when loading a single file from a dataset, unable to access it in offline mode afterwards

### Steps to reproduce the bug

```python
import os
# os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["HF_TOKEN"] = "xxxxxxxxxxxxxx"

import datasets

dataset_name = "uonlp/CulturaX"
data_files = "fr/fr_part_00038.parquet"

ds = datasets.load_dataset(dataset_name, split='train', data_files=data_files)
print(f"Dataset loaded   : {ds}")
```
Once the file has been cached, I rerun with the HF_HUB_OFFLINE activated an get this error : 
```
ValueError: Couldn't find cache for uonlp/CulturaX for config 'default-1e725f978350254e'
Available configs in the cache: ['default-2935e8cdcc21c613']
```

### Expected behavior

Should be able to access the previously cached files

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.4.0-215-generic-x86_64-with-glibc2.31
- Python version: 3.12.0
- `huggingface_hub` version: 0.27.0
- PyArrow version: 19.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


It seems the problem comes from builder.py / create_config_id()

On the first call, when the cache is empty we have
```
config_kwargs = {'data_files': {'train': ['hf://datasets/uonlp/CulturaX@6a8734bc69fefcbb7735f4f9250f43e4cd7a442e/fr/fr_part_00038.parquet']}}
```
leading to config_id beeing 'default-2935e8cdcc21c613'

then, on the second call, 
```
config_kwargs = {'data_files': 'fr/fr_part_00038.parquet'}
```
thus explaining why the hash is not the same, despite having the same parameter when calling load_dataset : data_files="fr/fr_part_00038.parquet"

Same behavior with version 3.5.1
Large memory use when loading large datasets from hub

### Describe the bug

When I load large parquet based datasets from the hub like `MLCommons/peoples_speech` using `load_dataset`, all my memory (500GB) is used and isn't released after loading, meaning that the process is terminated by the kernel if I try to load an additional dataset. This makes it impossible to train models using multiple large datasets.

### Steps to reproduce the bug

`uv run --with datasets==3.5.1 python`

```python
from datasets import load_dataset
load_dataset('MLCommons/peoples_speech', 'clean')
load_dataset('mozilla-foundation/common_voice_17_0', 'en')
```

### Expected behavior

I would expect that a lot less than 500GB of RAM would be required to load the dataset, or at least that the RAM usage would be cleared as soon as the dataset is loaded (and thus reside as a memory mapped file) such that other datasets can be loaded.

### Environment info

I am currently using the latest datasets==3.5.1 but I have had the same problem with multiple other versions.

Hi ! datasets are memory mapped from disk, so they don't fill out your RAM. Not sure what's the source of your memory issue.

What kind of system are you using ? and what kind of disk ?
TensorFlow RaggedTensor Support (batch-level)

### Feature request

Hi,

Currently datasets does not support RaggedTensor output on batch-level.
When building a Object Detection Dataset (with TensorFlow) I need to enable RaggedTensors as that's how BBoxes & classes are expected from the Keras Model POV.

Currently there's a error thrown saying that "Nested Data is not supported".

It'd be very helpful if this was fixed! :)

### Motivation

Enabling Object Detection pipelines for TensorFlow.

### Your contribution

With guidance I'd happily help making the PR.

The current implementation with DataCollator and later enforcing `np.array` is the problematic part (at the end of `np_get_batch` in `tf_utils.py`). As `numpy` don't support "Raggednes"

Keras doesn't support other inputs other than tf.data.Dataset objects ? it's a bit painful to have to support and maintain this kind of integration

Is there a way to use a `datasets.Dataset` with outputs formatted as tensors / ragged tensors instead ? like in https://huggingface.co/docs/datasets/use_with_tensorflow#dataset-format
Add custom fingerprint support to `from_generator`

This PR adds `dataset_id_suffix` parameter to 'Dataset.from_generator' function.

`Dataset.from_generator` function passes all of its arguments to `BuilderConfig.create_config_id`, including generator function itself. `BuilderConfig.create_config_id` function tries to hash all the args, which can take a large amount of time or even cause MemoryError if the dataset processed in a generator function is large enough.

This PR allows user to pass a custom fingerprint (`dataset_id_suffix`) to be used as a suffix in a dataset name instead of the one generated by hashing the args.

This PR is a possible solution of #7513 

This is great !

What do you think of passing `config_id=` directly to the builder instead of just the suffix ? This would be a power user argument though, or for internal use. And in from_generator the new argument can be `fingerprint=` as in `Dataset.__init__()`

The `config_id` can be defined using something like `config_id = "default-fingerprint=" + fingerprint`

I feel ike this could make the Dataset API more coherent if we avoid introducing a new argument while we can juste use `fingerprint=`
Faster downloads/uploads with Xet storage

![Image](https://github.com/user-attachments/assets/6e247f4a-d436-4428-a682-fe18ebdc73a9)

## Xet is out !

Over the past few weeks, Hugging Face’s [Xet Team](https://huggingface.co/xet-team) took a major step forward by [migrating the first Model and Dataset repositories off LFS and to Xet storage](https://huggingface.co/posts/jsulz/911431940353906).

See more information on the HF blog: https://huggingface.co/blog/xet-on-the-hub

You can already enable Xet on Hugging Face account to benefit from faster downloads and uploads :)

We finalized an official integration with the `huggingface_hub` library that means you get the benefits of Xet without any significant changes to your current workflow.


## Previous versions of `datasets`

For older versions of `datasets` you might see this warning in `push_to_hub()`:

```
Uploading files as bytes or binary IO objects is not supported by Xet Storage.
```

This means the `huggingface_hub` + Xet integration isn't enabled for your version of `datasets`.

You can fix this by updating to `datasets>=3.6.0` and `huggingface_hub>=0.31.0`

```
pip install -U datasets huggingface_hub
```

## Known issues

The Dataset Viewer may show errors like

```
Error code:   CreateCommitError
```

We are actively working on a fix.
In the meantime you can use an older version of `datasets` to upload datasets

## The future

Stay tuned for more Xet optimizations, especially on [Xet-optimized Parquet](https://huggingface.co/blog/improve_parquet_dedupe)
Incompatibile dill version (0.3.9) in datasets 2.18.0 - 3.5.0

### Describe the bug

Datasets 2.18.0 - 3.5.0 has a dependency on dill < 0.3.9. This causes errors with dill >= 0.3.9.

Could you please take a look into it and make it compatible?

### Steps to reproduce the bug

1. Install setuptools >= 2.18.0
2. Install dill >=0.3.9
3. Run pip check
4. Output:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires dill<0.3.9,>=0.3.0, but you have dill 0.3.9 which is incompatible.

### Expected behavior

Pip install both libraries without any errors

### Environment info

Datasets version: 2.18 - 3.5
Python: 3.11

Hi ! We can bump `dill` to 0.3.9 if we make sure it's deterministic and doesn't break the caching mechanism in `datasets`.

Would you be interested in opening a PR ? Then we can run the CI to see if it works

Hi!. Yeah I can do it. Should I make any changes besides dill versions?

There are probably some usage of internal functions from `dill` that we'll need to update in `datasets`

If you run `pytest tests/test_fingerprint.py` you should already have a good idea of what works and what doesn't.
But feel free to open a PR anyway, this way we can run the full CI and see the results


Hi, sorry for no response from my side. I will try to do it today.

Created pull request: [LINK](https://github.com/huggingface/datasets/pull/7535)
Tried to run tests by using command you have send and got few errors:

![Image](https://github.com/user-attachments/assets/acbf1feb-4dd1-416e-a118-c91abe0d188b)
BuilderConfig ParquetConfig(...) doesn't have a 'use_auth_token' key.

### Describe the bug

Trying to run the following fine-tuning script (based on this page [here](https://github.com/huggingface/instruction-tuned-sd)):

```
! accelerate launch /content/instruction-tuned-sd/finetune_instruct_pix2pix.py \
    --pretrained_model_name_or_path=${MODEL_ID} \
    --dataset_name=${DATASET_NAME} \
    --use_ema \
    --enable_xformers_memory_efficient_attention \
    --resolution=512 --random_flip \
    --train_batch_size=2 --gradient_accumulation_steps=4 --gradient_checkpointing \
    --max_train_steps=500 \
    --checkpointing_steps=25 --checkpoints_total_limit=1 \
    --learning_rate=5e-05 --max_grad_norm=1 --lr_warmup_steps=20 \
    --conditioning_dropout_prob=0.1 \
    --mixed_precision=fp16 \
    --seed=42 \
    --output_dir=${OUTPUT_DIR} \
    --original_image_column=before \
    --edit_prompt=prompt \
    --edited_image=after
```

but I keep getting the following error:

```
Traceback (most recent call last):
  File "/content/instruction-tuned-sd/finetune_instruct_pix2pix.py", line 1137, in <module>
    main()
  File "/content/instruction-tuned-sd/finetune_instruct_pix2pix.py", line 652, in main
    dataset = load_dataset(
              ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/load.py", line 1886, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
                                       ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 342, in __init__
    self.config, self.config_id = self._create_builder_config(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 590, in _create_builder_config
    raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
ValueError: BuilderConfig ParquetConfig(name='default', version=0.0.0, data_dir=None, data_files={'train': ['data/train-*']}, description=None, batch_size=None, columns=None, features=None, filters=None) doesn't have a 'use_auth_token' key.
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 10, in <module>
    sys.exit(main())
             ^^^^^^
```

Any ideas? `datasets` version should be `3.2.0`.

### Steps to reproduce the bug

Just running the script above.

### Expected behavior

No errors

### Environment info

Python 3.11.11

datasets==3.2.0

I encountered the same error, have you resolved it?

Hi ! `use_auth_token` has been deprecated and removed some time ago. You should use `token` instead in `load_dataset()`
Columns in the dataset obtained though load_dataset do not correspond to the one in the dataset viewer since 3.4.0

### Describe the bug

I have noticed that on my dataset named [BrunoHays/Accueil_UBS](https://huggingface.co/datasets/BrunoHays/Accueil_UBS), since the version 3.4.0, every column except audio is missing when I load the dataset.

Interestingly, the dataset viewer still shows the correct columns

### Steps to reproduce the bug

```python
from datasets import load_dataset
ds = load_dataset("BrunoHays/Accueil_UBS", streaming=True)
print(next(iter(ds["test"])).keys())
```
With datasets >= 3.4.0: 
-> dict_keys(['audio'])
With datasets == 3.3.2:
-> dict_keys(['audio', 'id', 'speaker', 'sentence', 'raw_sentence', 'start_timestamp', 'end_timestamp', 'overlap'])


### Expected behavior

All the columns should be present

### Environment info

- `datasets` version: 3.3.2
- Platform: macOS-14.6.1-x86_64-i386-64bit
- Python version: 3.10.15
- `huggingface_hub` version: 0.30.1
- PyArrow version: 16.1.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0
pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'

### Describe the bug

A dict with its keys are all str but get following error
```python
test_data=[{'input_ids':[1,2,3],'labels':[[Counter({2:1})]]}]
dataset = datasets.Dataset.from_list(test_data)
```
```bash
pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'
```

### Steps to reproduce the bug

.

### Expected behavior

.

### Environment info

datasets 3.3.2
Refactor `Dataset.map` to reuse cache files mapped with different `num_proc`

Fixes #7433

This refactor unifies `num_proc is None or num_proc == 1` and `num_proc > 1`; instead of handling them completely separately where one uses a list of kwargs and shards and the other just uses a single set of kwargs and `self`, by wrapping the `num_proc == 1` case in a list and making the difference just whether or not you use a pool, you set up either case to be able to load each other's cache files just by changing `num_shards`; `num_proc == 1` can sequentially load the shards of a dataset mapped `num_shards > 1` and map any missing shards

Other than the structural refactor, the main contribution of this PR is `existing_cache_file_map`, which uses a regex of `cache_file_name` and `suffix_template` to find existing cache files, grouped by their `num_shards`; using this data structure, we can reset `num_shards` to an existing set of cache files, and load them accordingly

@lhoestq please let me know what you think about this.

It looks like I can't change the merge target to #7435, so it will look like there is a bunch of extra stuff until #7435 is in main.

@lhoestq Thanks so much for reviewing #7435! Now that that's merged, I think this PR is ready!! Can you kick off CI when you get the chance?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7434). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Do you mind kicking off CI again?

The change I made to support windows paths in 637c1600fe7dd601eff571fda446937bd96c5c84 ended up breaking causing these tests in [tests/test_data_files.py](https://github.com/huggingface/datasets/actions/runs/13858546629/job/38781008643#step:10:6991). When I removed `glob_pattern_to_regex` in 583c28e7560b9d6db2e13048731f41ec8fa11361, none of the tests failed. So I'm thinking the `unicode_escape` may be handling the what `glob_pattern_to_regex` was doing.


@lhoestq will you have a chance to review this today?
pyarrow.lib.arrowinvalid: cannot mix list and non-list, non-null values with map function

### Describe the bug

Encounter pyarrow.lib.arrowinvalid error with map function in some example when loading the dataset

### Steps to reproduce the bug

```
from datasets import load_dataset
from PIL import Image, PngImagePlugin

dataset = load_dataset("leonardPKU/GEOQA_R1V_Train_8K")
system_prompt="You are a helpful AI Assistant"
def make_conversation(example):
    prompt = []

    prompt.append({"role": "system", "content": system_prompt})
    prompt.append(
            {
                "role": "user", 
                "content": [
                        {"type": "image"},
                        {"type": "text", "text": example["problem"]},
                        ]
            }
        )
    return {"prompt": prompt}

def check_data_types(example):
    for key, value in example.items():
        if key == 'image':
            if not isinstance(value, PngImagePlugin.PngImageFile):
                print(value)
        if key == "problem" or key == "solution":
            if not isinstance(value, str):
                print(value)

    return example

dataset = dataset.map(check_data_types)
dataset = dataset.map(make_conversation)
```

### Expected behavior

Successfully process the dataset with map

### Environment info

datasets==3.3.1

@lhoestq 

Can you try passing text: None for the image object ? Pyarrow expects all the objects to have the exact same type, in particular the dicttionaries in "content" should all have the keys "type" and "text"

The following modification on system prompt works, but it is different from the usual way to use it.
```
def make_conversation(example):
    prompt = []

    prompt.append({"role": "system", "content": [{"type": "text", "text": system_prompt}]})
    prompt.append(
            {
                "role": "user", 
                "content": [
                        {"type": "image"},
                        {"type": "text", "text": example["problem"]},
                        ]
            }
        )
    return {"prompt": prompt}
```

Good to know ! But yes Arrow / Parquet have this typing limitation (which is great to ensure data integrity, but constraining at the same time). It's is really blocking you, feel free to ping the arrow team / community if they plan to have a Union type or a JSON type
ImportError: To support decoding images, please install 'Pillow'.

### Describe the bug

Following this tutorial locally using a macboko and VSCode: https://huggingface.co/docs/diffusers/en/tutorials/basic_training
This line of code: for i, image in enumerate(dataset[:4]["image"]):
throws: ImportError: To support decoding images, please install 'Pillow'.

Pillow is installed.

### Steps to reproduce the bug

Run the tutorial


### Expected behavior

Images should be rendered

### Environment info

MacBook, VSCode

what's your `pip show Pillow` output

same issue.. my pip show Pillow output as below:

```
Name: pillow
Version: 11.1.0
Summary: Python Imaging Library (Fork)
Home-page: https://python-pillow.github.io/
Author: 
Author-email: "Jeffrey A. Clark" <aclark@aclark.net>
License: MIT-CMU
Location: [/opt/homebrew/lib/python3.10/site-packages](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/lib/python3.10/site-packages)
Requires: 
Required-by:
```

I encountered the same problem on Ubuntu system, my pip show Pillow output as below:

```
Name: pillow
Version: 10.4.0
Summary: Python Imaging Library (Fork)
Home-page: https://python-pillow.org/
Author: 
Author-email: "Jeffrey A. Clark" <[aclark@aclark.net](mailto:aclark@aclark.net)>
License: HPND
Location: /home/shunying/.local/lib/python3.8/site-packages
Requires: 
Required-by: 
```

Well, solved this by specifying the pip version to my conda virtual environment :)
Not available datasets[audio] on python 3.13

### Describe the bug

This is the error I got, it seems numba package does not support python 3.13

PS C:\Users\sergi\Documents> pip install datasets[audio]
Defaulting to user installation because normal site-packages is not writeable
Collecting datasets[audio]
  Using cached datasets-3.2.0-py3-none-any.whl.metadata (20 kB)
... (OTHER PACKAGES)
Collecting numba>=0.51.0 (from librosa->datasets[audio])
  Downloading numba-0.60.0.tar.gz (2.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 44.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [24 lines of output]
      Traceback (most recent call last):
        File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.496.0_x64__qbz5n2kfra8p0\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
          ~~~~^^
        File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.496.0_x64__qbz5n2kfra8p0\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.496.0_x64__qbz5n2kfra8p0\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "C:\Users\sergi\AppData\Local\Temp\pip-build-env-yauns_qh\overlay\Lib\site-packages\setuptools\build_meta.py", line 334, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\sergi\AppData\Local\Temp\pip-build-env-yauns_qh\overlay\Lib\site-packages\setuptools\build_meta.py", line 304, in _get_build_requires
          self.run_setup()
          ~~~~~~~~~~~~~~^^
      RuntimeError: Cannot install on Python version 3.13.1; only versions >=3.9,<3.13 are supported.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

### Steps to reproduce the bug

1. install python >=3.13
2. !pip install datasets[audio]

### Expected behavior

I needed datasets[audio] in the python 3.13

### Environment info

python 3.13.1

It looks like an issue with `numba` which can't be installed on 3.13 ? `numba` is a dependency of `librosa`, used to decode audio files
TypeError: Value.__init__() missing 1 required positional argument: 'dtype'

### Describe the bug

ds = load_dataset(
    "./xxx.py",
    name="default",
    split="train",
)

The datasets does not support debugging locally anymore...

### Steps to reproduce the bug

```
from datasets import load_dataset

ds = load_dataset(
    "./repo.py",
    name="default",
    split="train",
)

for item in ds:
    print(item)
```

It works fine for "username/repo", but it does not work for "./repo.py" when debugging locally...
Running above code template will report TypeError: Value.__init__() missing 1 required positional argument: 'dtype'

### Expected behavior

fix this bug

### Environment info

python 3.10 datasets==2.21

same error 
```
data = load_dataset('/opt/deepseek_R1_finetune/hf_datasets/openai/gsm8k', 'main')[split] 
```

> same error
> 
> ```
> data = load_dataset('/opt/deepseek_R1_finetune/hf_datasets/openai/gsm8k', 'main')[split] 
> ```

https://github.com/huggingface/open-r1/issues/204  this help me
`Dataset.save_to_disk` hangs when using num_proc > 1

### Describe the bug

Hi, I'm encountered a small issue when saving datasets that led to the saving taking up to multiple hours.
Specifically, [`Dataset.save_to_disk`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.save_to_disk) is a lot slower when using `num_proc>1` than when using `num_proc=1`

The documentation mentions that "Multiprocessing is disabled by default.", but there is no explanation on how to enable it.

### Steps to reproduce the bug

```
import numpy as np
from datasets import Dataset

n_samples = int(4e6)
n_tokens_sample = 100
data_dict = {
    'tokens' : np.random.randint(0, 100, (n_samples, n_tokens_sample)),
}

dataset = Dataset.from_dict(data_dict)
dataset.save_to_disk('test_dataset', num_proc=1)
dataset.save_to_disk('test_dataset', num_proc=4)
dataset.save_to_disk('test_dataset', num_proc=8)
```

This results in:
```
>>> dataset.save_to_disk('test_dataset', num_proc=1)
Saving the dataset (7/7 shards): 100%|██████████████| 4000000/4000000 [00:17<00:00, 228075.15 examples/s]
>>> dataset.save_to_disk('test_dataset', num_proc=4)
Saving the dataset (7/7 shards): 100%|██████████████| 4000000/4000000 [01:49<00:00, 36583.75 examples/s]
>>> dataset.save_to_disk('test_dataset', num_proc=8)
Saving the dataset (8/8 shards): 100%|██████████████| 4000000/4000000 [02:11<00:00, 30518.43 examples/s]
```

With larger datasets it can take hours, but I didn't benchmark that for this bug report.

### Expected behavior

I would expect using `num_proc>1` to be faster instead of slower than `num_proc=1`. 

### Environment info

- `datasets` version: 3.1.0
- Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.26.2
- PyArrow version: 18.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1

I've met the same situations.

Here's my logs:
num_proc = 64, I stop it early as it cost **too** much time.
```
Saving the dataset (1540/4775 shards):  32%|███▏      | 47752224/147853764 [15:32:54<132:28:34, 209.89 examples/s]
Saving the dataset (1540/4775 shards):  32%|███▏      | 47754224/147853764 [15:32:54<78:14:37, 355.37 examples/s] 
Saving the dataset (1540/4775 shards):  32%|███▏      | 47755224/147853764 [15:32:59<93:43:45, 296.65 examples/s]
```

num_proc = 1(Not set num_proc parameter)
```
Saving the dataset (1753/4775 shards):  37%|███▋      | 54301556/147853764 [24:46<38:33, 40440.93 examples/s]
Saving the dataset (1753/4775 shards):  37%|███▋      | 54306556/147853764 [24:46<39:34, 39392.01 examples/s]
Saving the dataset (1753/4775 shards):  37%|███▋      | 54311520/147853764 [24:46<38:56, 40030.53 examples/s]
```

I check the conditions of CPUs and Memory I/O, I found that disk I/O was blocked, but CPU and memory usage was high. There should be some bugs in the code.
`load_dataset` fails to load dataset saved by `save_to_disk`

### Describe the bug

This code fails to load the dataset it just saved:

```python
from datasets import load_dataset
from transformers import AutoTokenizer

MODEL = "google-bert/bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(MODEL)

dataset = load_dataset("yelp_review_full")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets.save_to_disk("dataset")

tokenized_datasets = load_dataset("dataset/")  # raises
```

It raises `ValueError: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('test'): ('json', {})}`.

I believe this bug is caused by the [logic that tries to infer dataset format](https://github.com/huggingface/datasets/blob/9af8dd3de7626183a9a9ec8973cebc672d690400/src/datasets/load.py#L556). It counts the most common file extension. However, a small dataset can fit in a single `.arrow` file and have two JSON metadata files, causing the format to be inferred as JSON:

```shell
$ ls -l dataset/test
-rw-r--r-- 1 sliedes sliedes 191498784 Jul  1 13:55 data-00000-of-00001.arrow
-rw-r--r-- 1 sliedes sliedes      1730 Jul  1 13:55 dataset_info.json
-rw-r--r-- 1 sliedes sliedes       249 Jul  1 13:55 state.json
```

### Steps to reproduce the bug

Execute the code above.

### Expected behavior

The dataset is loaded successfully.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-6.9.3-arch1-1-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0


In my case the error was:
```
ValueError: You are trying to load a dataset that was saved using `save_to_disk`. Please use `load_from_disk` instead.
```
Did you try `load_from_disk`?

More generally, any reason there is no API consistency between save_to_disk and push_to_hub ? 

Would be nice to be able to save_to_disk and then upload manually to the hub and load_dataset (which works in some situations but not all)...

I have the exact same problem !

`load_from_disk` managed to load the dataset, but the bug with `load_dataset` needs to be fixed.
Loading just one particular split is not possible for imagenet-1k

### Describe the bug

I'd expect the following code to download just the validation split but instead I get all data on my disk (train, test and validation splits)

`
from datasets import load_dataset

dataset = load_dataset("imagenet-1k", split="validation", trust_remote_code=True)
`

Is it expected to work like that? 

### Steps to reproduce the bug

1. Install the required libraries (python, datasets, huggingface_hub)
2. Login using huggingface cli
2. Run the code in the description

### Expected behavior

Just a single (validation) split should be downloaded.

### Environment info

python: 3.12.2
datasets: 2.18.0
huggingface_hub: 0.22.2

+1
PyArrow 'Memory mapping file failed: Cannot allocate memory' bug

### Describe the bug

Hello,

I've been struggling with a problem using Huggingface datasets caused by PyArrow memory allocation. I finally managed to solve it, and thought to document it since similar issues have been raised here before (https://github.com/huggingface/datasets/issues/5710, https://github.com/huggingface/datasets/issues/6176).

In my case, I was trying to load ~70k dataset files from disk using `datasets.load_from_disk(data_path)` (meaning 70k repeated calls to load_from_disk). This triggered an (uninformative) exception around 64k loaded files:
```
  File "pyarrow/io.pxi", line 1053, in pyarrow.lib.memory_map
  File "pyarrow/io.pxi", line 1000, in pyarrow.lib.MemoryMappedFile._open
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
OSError: Memory mapping file failed: Cannot allocate memory
```
Despite system RAM usage being very low. After a lot of digging around, I discovered that my Ubuntu machine had a limit on the maximum number of memory mapped files in `/proc/sys/vm/max_map_count` set to 65530, which was causing my data loader to crash. Increasing the limit in the file (`echo <new_mmap_size> | sudo tee /proc/sys/vm/max_map_count`) made the issue go away.

While this isn't a bug as such in either Datasets or PyArrow, this behavior can be very confusing to users. Maybe this should be mentioned in documentation? I suspect the other issues raised here about memory mapping OOM errors could actually be consequence of system configuration.

Br,
Lauri

### Steps to reproduce the bug

```
import numpy as np
import pyarrow as pa
import tqdm

# Write some data to disk
arr = pa.array(np.arange(100))
schema = pa.schema([
    pa.field('nums', arr.type)
])
with pa.OSFile('arraydata.arrow', 'wb') as sink:
    with pa.ipc.new_file(sink, schema=schema) as writer:
        batch = pa.record_batch([arr], schema=schema)
        writer.write(batch)

# Number of times to open the memory map
nums = 70000

# Read the data back
arrays = [pa.memory_map('arraydata.arrow', 'r') for _ in tqdm.tqdm(range(nums))]
```

### Expected behavior

No errors.

### Environment info

datasets: 2.18.0
pyarrow: 15.0.0

Thanks for a very clean explanation. This happened to me too, and I don't have sudo access to update the value. I wonder if there might be another workaround.

One option is to just have more data in each file - /proc/sys/vm/max_map_count limits the maximum number of concurrently open files, but I don't know if the size of a single file is restricted in any way. E.g. 5000 files with 1GB each is 5TB of data. https://huggingface.co/docs/datasets/v2.18.0/en/package_reference/main_classes#datasets.concatenate_datasets can come in handy.