josteece/github-issues huggingface dataset:
load_dataset() in 4.0.0 returns different audio samples compared to earlier versions breaking reproducibility
Opened by Manalelaidouni (closed)

### Describe the bug

In datasets 4.0.0 release, `load_dataset()` returns different audio samples compared to earlier versions, this breaks integration tests that depend on consistent sample data across different environments (first and second envs specified below).



### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(24000))
sample= ds[0]["audio"]["array"]
print(sample)

# sample in 3.6.0 
[0.00231914 0.00245417 0.00187414 ... 0.00061956 0.00101157 0.00076325]


# sample in 4.0.0 
array([0.00238037, 0.00220794, 0.00198703, ..., 0.00057983, 0.00085863,
       0.00115309], dtype=float32)
```

### Expected behavior

The same dataset should load identical samples across versions to maintain reproducibility.

### Environment info

First env:
- datasets version: 3.6.0
- Platform: Windows-10-10.0.26100-SP0
- Python: 3.11.0

Second env:
- datasets  version: 4.0.0
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python: 3.11.13

Comments:
Hi ! This is because `datasets` now uses the FFmpeg-based library `torchcodec` instead of the libsndfile-based library `soundfile` to decode audio data. Those two have different decoding implementations

I’m all for torchcodec, good luck with the migration!
Support downloading specific splits in load_dataset
Opened by ArjunJagdale (closed)

This PR builds on #6832 by @mariosasko.

May close - #4101, #2538

Discussion - https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130

---
### Note - This PR is under work and frequent changes will be pushed.

Comments:
I’ve completed the following steps to continue the partial split download support (from PR #6832):

I did changes on top of what has been done by mario. Here are some of those changes: 
- Restored support for writing multiple split shards:

- In _prepare_split_single, we now correctly replace JJJJJ and SSSSS placeholders in the fpath for job/shard IDs before creating the writer.

- Added os.makedirs(os.path.dirname(path), exist_ok=True) after placeholder substitution to prevent FileNotFoundError.

- Applied the fix to both split writers:

       1] self._generate_examples version (used by most modules).

       2] self._generate_tables version (used by IterableDatasetBuilder).

- Confirmed 109/113 tests passing, meaning the general logic is working across the board.

What’s still failing
4 integration tests fail:

`test_load_hub_dataset_with_single_config_in_metadata`

`test_load_hub_dataset_with_two_config_in_metadata`

`test_load_hub_dataset_with_metadata_config_in_parallel`

`test_reload_old_cache_from_2_15`

All are due to FileNotFoundError from uncreated output paths, which I'm currently finalizing by ensuring os.makedirs() is correctly applied before every writer instantiation.

I will update about these fixes after running tests!

@lhoestq this was just an update

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7695). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Local DIR wasn't doing well, dk actually what happened, will PR again! Sorry :)
xopen: invalid start byte for streaming dataset with trust_remote_code=True
Opened by sedol1339 (open)

### Describe the bug

I am trying to load YODAS2 dataset with datasets==3.6.0
```
from datasets import load_dataset
next(iter(load_dataset('espnet/yodas2', name='ru000', split='train', streaming=True, trust_remote_code=True)))
```
And get `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte`

The cause of the error is the following:

```
from datasets.utils.file_utils import xopen

filepath = 'https://huggingface.co/datasets/espnet/yodas2/resolve/c9674490249665d658f527e2684848377108d82c/data/ru000/text/00000000.json'
xopen(filepath, 'r').read()
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte
```

And the cause of this is the following:

```
import fsspec

fsspec.open(
    'hf://datasets/espnet/yodas2@c9674490249665d658f527e2684848377108d82c/data/ru000/text/00000000.json',
    mode='r',
    hf={'token': None, 'endpoint': 'https://huggingface.co'},
).open().read()
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte
```

Is it true that streaming=True loading is not supported anymore for trust_remote_code=True, even with datasets==3.6.0? This breaks backward compatibility.

### Steps to reproduce the bug

```
from datasets import load_dataset
next(iter(load_dataset('espnet/yodas2', name='ru000', split='train', streaming=True)))
```

### Expected behavior

No errors expected

### Environment info

datasets==3.6.0, ubuntu 24.04

Comments:
Hi ! it would be cool to convert this dataset to Parquet. This will make it work for `datasets>=4.0`, enable the Dataset Viewer and make it more reliable to load/stream (currently it uses a loading script in python and those are known for having issues sometimes)

using `datasets==3.6.0`, here is the command to convert it and open a Pull Request:

```
datasets-cli convert_to_parquet espnet/yodas2 --trust_remote_code
```

Though it's likely that the `UnicodeDecodeError` comes from the loading script. If the script has a bug, it must be fixed to be able to convert the dataset without errors
HDF5 support
Opened by klamike (closed)

This PR adds support for tabular HDF5 file(s) by converting each row to an Arrow table. It supports columns with the usual dtypes including up to 5-dimensional arrays as well as support for complex/compound types by using `Features(dict)`. All datasets within the HDF5 file should have rows on the first dimension (groups/subgroups are still allowed). Closes #3113.

Replaces #7625 which only supports a relatively small subset of HDF5.

Comments:
A few to-dos which I think can be left for future PRs (which I am happy to do/help with -- just this one is already huge 😄 ):
- [Enum types](https://docs.h5py.org/en/stable/special.html#enumerated-types)
- HDF5 [io](https://github.com/huggingface/datasets/tree/main/src/datasets/io)
- [dataset-viewer](https://github.com/huggingface/dataset-viewer) support (not sure if changes are needed with the way it is written now)

@lhoestq any interest in merging this? Let me know if I can do anything to make reviewing it easier!

Sorry for the delay, I'll review your PR soon :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7690). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review @lhoestq! Rebased on main and incorporated most of your suggestions.

I believe the only one left is the zero-dim handling with `table_cast`...

@lhoestq is 2c4bfba what you meant?

Awesome! Yes, I'm happy to help with the docs. Would appreciate any pointers, we can discuss in #7740.

It does look like there was a CI test failure, though it seems unrelated?
```
FAILED tests/test_dataset_dict.py::test_dummy_datasetdict_serialize_fs - ValueError: Protocol not known: mock
FAILED tests/test_arrow_dataset.py::test_dummy_dataset_serialize_fs - ValueError: Protocol not known: mock
```
Also, what do you think of the todos in https://github.com/huggingface/datasets/pull/7690#issuecomment-3105391677 ? In particular I think support in dataset-viewer would be nice.

Cool ! Yeah the failure is unrelated

Regarding the Viewer, it should work out of the box when it's updated with the next version of `datasets` :)
No module named "distributed"
Opened by yingtongxiong (open)

### Describe the bug

hello, when I run the command "from datasets.distributed import split_dataset_by_node", I always met the bug "No module named 'datasets.distributed" in different version like 4.0.0, 2.21.0 and so on. How can I solve this?

### Steps to reproduce the bug

1. pip install datasets
2. from datasets.distributed import split_dataset_by_node

### Expected behavior

expecting the command "from datasets.distributed import split_dataset_by_node" can be ran successfully

### Environment info

python: 3.12

Comments:
The error ModuleNotFoundError: No module named 'datasets.distributed' means your installed datasets library is too old or incompatible with the version of Library you are using(in my case it was BEIR). The datasets.distributed module was removed in recent versions of the datasets library.

Downgrade datasets to version 2.14.6 : ! pip install datasets==2.14.6


this code does run in `datasets` 4.0:
```python
from datasets.distributed import split_dataset_by_node
```

make sure you have a python version that is recent enough (>=3.9) to be able to install `datasets` 4.0

I do think the problem is caused by the python version, because I do have python version 3.12.5
Datasets keeps rebuilding the dataset every time i call the python script
Opened by CALEB789 (open)

### Describe the bug

Every time it runs, somehow, samples increase.
This can cause a 12mb dataset to have other built versions of 400 mbs+

<img width="363" height="481" alt="Image" src="https://github.com/user-attachments/assets/766ce958-bd2b-41bc-b950-86710259bfdc" />

### Steps to reproduce the bug


`from datasets import load_dataset
s = load_dataset('~/.cache/huggingface/datasets/databricks___databricks-dolly-15k')['train']
`
1. A dataset needs to be available in the .cache folder
2. Run the code multiple times, and every time it runs, more versions are created


### Expected behavior

The number of samples increases every time the script runs

### Environment info

- `datasets` version: 3.6.0
- Platform: Windows-11-10.0.26100-SP0
- Python version: 3.13.3
- `huggingface_hub` version: 0.32.3
- PyArrow version: 20.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0


Comments:
here is the code to load the dataset form the cache:

```python
s = load_dataset('databricks/databricks-dolly-15k')['train']
```

if you pass the location of a local directory it will create a new cache based on that directory content
fix audio cast storage from array + sampling_rate
Opened by lhoestq (closed)

fix https://github.com/huggingface/datasets/issues/7682

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7684). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Convert to string when needed + faster .zstd
Opened by lhoestq (closed)

for https://huggingface.co/datasets/allenai/olmo-mix-1124

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7683). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Probabilistic High Memory Usage and Freeze on Python 3.10
Opened by ryan-minato (open)

### Describe the bug

A probabilistic issue encountered when processing datasets containing PIL.Image columns using the huggingface/datasets library on Python 3.10. The process occasionally experiences a sudden and significant memory spike, reaching 100% utilization, leading to a complete freeze. During this freeze, the process becomes unresponsive, cannot be forcefully terminated, and does not throw any exceptions.

I have attempted to mitigate this issue by setting `datasets.config.IN_MEMORY_MAX_SIZE`, but it had no effect. In fact, based on the document of `load_dataset`, I suspect that setting `IN_MEMORY_MAX_SIZE` might even have a counterproductive effect.

This bug is not consistently reproducible, but its occurrence rate significantly decreases or disappears entirely when upgrading Python to version 3.11 or higher. Therefore, this issue also serves to share a potential solution for others who might encounter similar problems.

### Steps to reproduce the bug

Due to the probabilistic nature of this bug, consistent reproduction cannot be guaranteed for every run. However, in my environment, processing large datasets like timm/imagenet-1k-wds(whether reading, casting, or mapping operations) almost certainly triggers the issue at some point.
The probability of the issue occurring drastically increases when num_proc is set to a value greater than 1 during operations.

When the issue occurs, my system logs repeatedly show the following warnings:
```
WARN: very high memory utilization: 57.74GiB / 57.74GiB (100 %)
WARN: container is unhealthy: triggered memory limits (OOM)
WARN: container is unhealthy: triggered memory limits (OOM)
WARN: container is unhealthy: triggered memory limits (OOM)
```

### Expected behavior

The dataset should be read and processed normally without memory exhaustion or freezing. If an unrecoverable error occurs, an appropriate exception should be raised.

I have found that upgrading Python to version 3.11 or above completely resolves this issue. On Python 3.11, when memory usage approaches 100%, it suddenly drops before slowly increasing again. I suspect this behavior is due to an expected memory management action, possibly involving writing to disk cache, which prevents the complete freeze observed in Python 3.10.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.15.0-71-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.33.4
- PyArrow version: 20.0.0
- Pandas version: 2.3.1
- `fsspec` version: 2025.3.0

Comments:
metric glue breaks with 4.0.0
Opened by stas00 (closed)

### Describe the bug

worked fine with 3.6.0, and with 4.0.0 `eval_metric = metric.compute()` in HF Accelerate breaks.

The code that fails is:
https://huggingface.co/spaces/evaluate-metric/glue/blob/v0.4.0/glue.py#L84

```
def simple_accuracy(preds, labels):
    print(preds, labels)
    print(f"{preds==labels}")
    return float((preds == labels).mean())
```
data:
```
Column([1, 0, 0, 1, 1]) Column([1, 0, 0, 1, 0])
False
```

```
[rank0]:     return float((preds == labels).mean())
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: AttributeError: 'bool' object has no attribute 'mean'
```

Some behavior has changed in this new major release of `datasets` and requires updating HF accelerate and perhaps the glue metric code, all belong to HF.

### Environment info

datasets=4.0.0

Comments:
I released `evaluate` 0.4.5 yesterday to fix the issue - sorry for the inconvenience:

```
pip install -U evaluate
```

Thanks so much, @lhoestq!
To support decoding audio data, please install 'torchcodec'.
Opened by alpcansoydas (closed)


In the latest version of datasets==4.0.0, i cannot print the audio data on the Colab notebook. But it works on the 3.6.0 version.

!pip install -q -U datasets huggingface_hub fsspec

from datasets import load_dataset

downloaded_dataset = load_dataset("ymoslem/MediaSpeech", "tr", split="train")
print(downloaded_dataset["audio"][0])

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
[/tmp/ipython-input-4-90623240.py](https://localhost:8080/#) in <cell line: 0>()
----> 1 downloaded_dataset["audio"][0]

10 frames
[/usr/local/lib/python3.11/dist-packages/datasets/features/audio.py](https://localhost:8080/#) in decode_example(self, value, token_per_repo_id)
    170             from ._torchcodec import AudioDecoder
    171         else:
--> 172             raise ImportError("To support decoding audio data, please install 'torchcodec'.")
    173 
    174         if not self.decode:

ImportError: To support decoding audio data, please install 'torchcodec'. 

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python version: 3.11.13
- `huggingface_hub` version: 0.33.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2025.3.0

Comments:
Hi ! yes you should `!pip install -U datasets[audio]` to have the required dependencies.

`datasets` 4.0 now relies on `torchcodec` for audio decoding. The `torchcodec` AudioDecoder enables streaming from HF and also allows to decode ranges of audio

Same issues on Colab.

> !pip install -U datasets[audio] 

This works for me. Thanks.
Toxicity fails with datasets 4.0.0
Opened by serena-ruan (closed)

### Describe the bug

With the latest 4.0.0 release, huggingface toxicity evaluation module fails with error: `ValueError: text input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).`

### Steps to reproduce the bug

Repro:
```
>>> toxicity.compute(predictions=["This is a response"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/evaluate/module.py", line 467, in compute
    output = self._compute(**inputs, **compute_kwargs)
  File "/Users/serena.ruan/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-measurement--toxicity/2390290fa0bf6d78480143547c6b08f3d4f8805b249df8c7a8e80d0ce8e3778b/toxicity.py", line 135, in _compute
    scores = toxicity(predictions, self.toxic_classifier, toxic_label)
  File "/Users/serena.ruan/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-measurement--toxicity/2390290fa0bf6d78480143547c6b08f3d4f8805b249df8c7a8e80d0ce8e3778b/toxicity.py", line 103, in toxicity
    for pred_toxic in toxic_classifier(preds):
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/text_classification.py", line 159, in __call__
    result = super().__call__(*inputs, **kwargs)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1431, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1437, in run_single
    model_inputs = self.preprocess(inputs, **preprocess_params)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/pipelines/text_classification.py", line 183, in preprocess
    return self.tokenizer(inputs, return_tensors=return_tensors, **tokenizer_kwargs)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2867, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/Users/serena.ruan/miniconda3/envs/mlflow-310/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2927, in _call_one
    raise ValueError(
ValueError: text input must be of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).
```

### Expected behavior

This works before 4.0.0 release

### Environment info

- `datasets` version: 4.0.0
- Platform: macOS-15.5-arm64-arm-64bit
- Python version: 3.10.16
- `huggingface_hub` version: 0.33.0
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments:
Hi ! You can fix this by upgrading `evaluate`:

```
pip install -U evaluate
```

Thanks, verified evaluate 0.4.5 works!
Fix double sequence
Opened by lhoestq (closed)

```python
>>> Features({"a": Sequence(Sequence({"c": Value("int64")}))})
{'a': List({'c': List(Value('int64'))})}
```

instead of `{'a': {'c': List(List(Value('int64')))}}`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7672). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Mapping function not working if the first example is returned as None
Opened by dnaihao (closed)

### Describe the bug

https://github.com/huggingface/datasets/blob/8a19de052e3d79f79cea26821454bbcf0e9dcd68/src/datasets/arrow_dataset.py#L3652C29-L3652C37

Here we can see the writer is initialized on `i==0`. However, there can be cases where in the user mapping function, the first example is filtered out (length constraints, etc).

In this case, the writer would be a `None` type and the code will report `NoneType has no write function`.

A simple fix is available, simply change line 3652 from `if i == 0:` to `if writer is None:`

### Steps to reproduce the bug

Prepare a dataset

have this function

```
import datasets


def make_map_fn(split, max_prompt_tokens=3):
    def process_fn(example, idx):

        question = example['question']
        reasoning_steps = example['reasoning_steps']
        label = example['label']

        answer_format = ""
        for i in range(len(reasoning_steps)):
            system_message = "Dummy"

        all_steps_formatted = []

        content = f"""Dummy"""

        prompt = [
            {"role": "system", "content": system_message},
            {"role": "user", "content": content},
        ]
        tokenized = tokenizer.apply_chat_template(prompt, return_tensors="pt", truncation=False)
        if tokenized.shape[1] > max_prompt_tokens:
            return None  # skip overly long examples

        data = {
            "dummy": "dummy"
        }

        return data

    return process_fn

...
# load your dataset
...
train = train.map(function=make_map_fn('train'), with_indices=True)
```

### Expected behavior

The dataset mapping shall behave even when the first example is filtered out.

### Environment info

I am using `datasets==3.6.0` but I have observed this issue in the github repo too: https://github.com/huggingface/datasets/blob/8a19de052e3d79f79cea26821454bbcf0e9dcd68/src/datasets/arrow_dataset.py#L3652C29-L3652C37

Comments:
Hi, map() always expect an output.

If you wish to filter examples, you should use filter(), in your case it could be something like this:

```python
ds = ds.map(my_processing_function).filter(ignore_long_prompts)
```

Realized this! Thanks a lot, I will close this issue then.
How can I add my custom data to huggingface datasets
Opened by xiagod (open)

I want to add my custom dataset in huggingface dataset. Please guide me how to achieve that.

Comments:
Hey @xiagod 

The easiest way to add your custom data to Hugging Face Datasets is to use the built-in load_dataset function with your local files. Some examples include:

CSV files:
from datasets import load_dataset
dataset = load_dataset("csv", data_files="my_file.csv")

JSON or JSONL files:
from datasets import load_dataset
dataset = load_dataset("json", data_files="my_file.json")


Images stored in folders (e.g. data/train/cat/, data/train/dog/):
from datasets import load_dataset
dataset = load_dataset("imagefolder", data_dir="/path/to/pokemon")


These methods let you quickly create a custom dataset without needing to write a full script.

More information can be found in Hugging Face's tutorial "Create a dataset" or "Load" documentation here: 

https://huggingface.co/docs/datasets/create_dataset 

https://huggingface.co/docs/datasets/loading#local-and-remote-files



If you want to submit your dataset to the Hugging Face Datasets GitHub repo so others can load it follow this guide: 

https://huggingface.co/docs/datasets/upload_dataset