ahmedfarazsyk/datasets-github-issues huggingface dataset:
Datasets 4.0 map function causing column not found
Author: Darejkal
State: open

### Describe the bug

Column returned after mapping is not found in new instance of the dataset.

### Steps to reproduce the bug

Code for reproduction. After running get_total_audio_length, it is errored out due to `data` not having `duration`
```
def compute_duration(x):
    return {"duration": len(x["audio"]["array"]) / x["audio"]["sampling_rate"]}


def get_total_audio_length(dataset):
    data = dataset.map(compute_duration, num_proc=NUM_PROC)
    print(data)
    durations=data["duration"]
    total_seconds = sum(durations)
    return total_seconds
```

### Expected behavior

New datasets.Dataset instance should have new columns attached.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.33.2
- PyArrow version: 20.0.0
- Pandas version: 2.3.0
- `fsspec` version: 2023.12.2

Comments: Hi, I tried to reproduce this issue on the latest `main` branch but it seems to be working correctly now. My test script (which creates a dummy dataset and applies the `.map()` function) successfully creates and accesses the new column without a `KeyError`.

It's possible this was fixed by a recent commit. The maintainers might want to consider closing this issue.

Hi, have you tried on a large dataset (200GB+) perhaps? I will try my best to do a rerun with main branch when I have the time.

I ran it on a small dataset, maybe that’s why I didn’t hit the issue. If it still shows up on your side with the latest main, let me know. I can try it on a bigger set too.
Specify dataset columns types in typehint
Author: Samoed
State: open
Labels: name: enhancement

### Feature request

Make dataset optionaly generic to datasets usage with type annotations like it was done in `torch.Dataloader` https://github.com/pytorch/pytorch/blob/134179474539648ba7dee1317959529fbd0e7f89/torch/utils/data/dataloader.py#L131

### Motivation

In MTEB we're using a lot of datasets objects, but they're a bit poor in typehints. E.g. we can specify this for dataloder

```python
from typing import TypedDict
from torch.utils.data import DataLoader

class CorpusInput(TypedDict):
    title: list[str]
    body: list[str]


class QueryInput(TypedDict):
    query: list[str]
    instruction: list[str]

def queries_loader() -> DataLoader[QueryInput]:
    ...

def corpus_loader() -> DataLoader[CorpusInput]:
   ...
```
But for datasets we can only specify columns in type in comments 
```python
from datasets import Dataset

QueryDataset = Dataset
"""Query dataset should have `query` and `instructions` columns as `str` """
```

### Your contribution

I can create draft implementation
Cached dataset is not used when explicitly passing the cache_dir parameter
Author: padmalcom
State: open

### Describe the bug

Hi, we are pre-downloading a dataset using snapshot_download(). When loading this exact dataset with load_dataset() the cached snapshot is not used. In both calls, I provide the cache_dir parameter.

### Steps to reproduce the bug

```
from datasets import load_dataset, concatenate_datasets
from huggingface_hub import snapshot_download
    
def download_ds(name: str):
    snapshot_download(repo_id=name, repo_type="dataset", cache_dir="G:/Datasets/cache")

def prepare_ds():
    audio_ds = load_dataset("openslr/librispeech_asr", num_proc=4, cache_dir="G:/Datasets/cache")
    print(sfw_ds.features)

if __name__ == '__main__':
    download_ds("openslr/librispeech_asr")
    prepare_ds()
```

### Expected behavior

I'd expect that the cached version of the dataset is used. Instead, the same dataset is downloaded again to the default cache directory.

### Environment info

Windows 11
datasets==4.0.0
Python 3.12.11

Comments: Hi, I've investigated this issue and can confirm the bug. Here are my findings:

**1. Reproduction:**
I was able to reproduce the issue on the latest `main` branch. Using the provided code snippet, `snapshot_download` correctly populates the custom `cache_dir`, but `load_dataset` with the same `cache_dir` triggers a full re-download and re-processing of the dataset, ignoring the existing cache.

**2. Investigation:**
I traced the `cache_dir` parameter from `load_dataset` down to the `DatasetBuilder` class in `src/datasets/builder.py`. The root cause seems to be a mismatch between the cache path structure created by `snapshot_download` and the path structure expected by the `DatasetBuilder`.

Specifically, the `_relative_data_dir` method in `DatasetBuilder` constructs a path using `namespace___dataset_name` (with three underscores), while the cache from `snapshot_download` appears to use a `repo_id` based format like `datasets--namespace--dataset_name` (with double hyphens).

**3. Attempted Fix & Result:**
I attempted a fix by modifying the `_relative_data_dir` method to replace the path separator "/" in `self.repo_id` with "--", to align it with the `snapshot_download` structure.

This partially worked: `load_dataset` no longer re-downloads the files. However, it still re-processes them every time (triggering "Generating train split...", etc.) instead of loading the already processed Arrow files from the cache.

This suggests the issue is deeper than just the directory name and might be related to how the builder verifies the integrity or presence of the processed cache files.

I hope these findings are helpful for whoever picks up this issue.
Release 4.0.0 breaks usage patterns of with_format
Author: wittenator
State: closed

### Describe the bug

Previously it was possible to access a whole column that was e.g. in numpy format via `with_format` by indexing the column. Now this possibility seems to be gone with the new Column() class. As far as I see, this makes working on a whole column (in-memory) more complex, i.e. normalizing an in-memory dataset for which iterating would be too slow. Is this intended behaviour? I couldn't find much documentation on the intended usage of the new Column class yet.

### Steps to reproduce the bug

Steps to reproduce:
```
from datasets import load_dataset
dataset = load_dataset("lhoestq/demo1")
dataset = dataset.with_format("numpy")
print(dataset["star"].ndim)
```

### Expected behavior

Working on whole columns should be possible.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.8.0-63-generic-x86_64-with-glibc2.36
- Python version: 3.12.11
- `huggingface_hub` version: 0.34.3
- PyArrow version: 21.0.0
- Pandas version: 2.3.1
- `fsspec` version: 2025.3.0

Comments: This is a breaking change with 4.0 which introduced `Column` objects. To get the numpy array from a `Column` you can `col[i]`, `col[i:j]` or even `col[:]` if you want the full column as a numpy array:

```python
from datasets import load_dataset
dataset = load_dataset(...)
dataset = dataset.with_format("numpy")
print(dataset["star"][:].ndim)
```

Ah perfect, thanks for clearing this up. I would close this ticket then.
Can Not read installed dataset in dataset.load(.)
Author: HuangChiEn
State: open

Hi, folks, I'm newbie in huggingface dataset api.
As title, i'm facing the issue that the dataset.load api can not connect to the installed dataset.

code snippet :
<img width="572" height="253" alt="Image" src="https://github.com/user-attachments/assets/10f48aaf-d6ca-4239-b1cf-145d74f125d1" />

data path : 
"/xxx/joseph/llava_ds/vlm_ds"
it contains all video clips i want!
<img width="1398" height="261" alt="Image" src="https://github.com/user-attachments/assets/bf213b66-e344-4311-97e7-bc209677ae77" />

i run the py script by 
<img width="1042" height="38" alt="Image" src="https://github.com/user-attachments/assets/8b3fcee4-e1a6-41b8-bee1-91567b00d9d2" />

But bad happended, even i provide dataset path by "HF_HUB_CACHE", it still attempt to download data from remote side : 

<img width="1697" height="813" alt="Image" src="https://github.com/user-attachments/assets/baa6cff1-a724-4710-a8c4-4805459deffb" />

Any suggestion will be appreciated!!

Comments: You can download the dataset locally using [huggingface_hub.snapshot_download](https://huggingface.co/docs/huggingface_hub/v0.34.3/en/package_reference/file_download#huggingface_hub.snapshot_download) and then do

```python
dataset = load_dataset(local_directory_path)
```

> You can download the dataset locally using [huggingface_hub.snapshot_download](https://huggingface.co/docs/huggingface_hub/v0.34.3/en/package_reference/file_download#huggingface_hub.snapshot_download) and then do
> 
> dataset = load_dataset(local_directory_path)

It's good suggestion, but my server env is network restriction. It can not directly fetch data from huggingface. I spent lot of time to download and transfer it to the server.
So, I attempt to make load_dataset connect to my local dataset.  

Just Solved it few day before. Will post solution later...
also thanks folks quick reply..
[Docs] map() example uses undefined `tokenizer` — causes NameError
Author: Sanjaykumar030
State: open

## Description

The current documentation example for `datasets.Dataset.map()` demonstrates batched processing but uses a `tokenizer` object without defining or importing it. This causes an error every time it's copied.

Here is the problematic line:

```python
# process a batch of examples
>>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This assumes the user has already set up a tokenizer, which contradicts the goal of having self-contained, copy-paste-friendly examples. 

## Problem
Users who copy and run the example as-is will encounter:

```python
NameError: name 'tokenizer' is not defined
```

This breaks the flow for users and violates HuggingFace's documentation principle that examples should "work as expected" when copied directly. 

## Proposal
Update the example to include the required tokenizer setup using the Transformers library, like so:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
ds_tokenized = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This will help new users understand the workflow and apply the method correctly.

## Note
This PR complements ongoing improvements like #7700, which clarifies multiprocessing in .map(). My change focuses on undefined tokenizer — causes NameError






Comments: I've submitted PR #7704 which adds documentation to clarify the behavior of `map()` when returning `None`.
[doc] map.num_proc needs clarification
Author: sfc-gh-sbekman
State: open

https://huggingface.co/docs/datasets/v4.0.0/en/package_reference/main_classes#datasets.Dataset.map.num_proc
```
num_proc (int, optional, defaults to None) — Max number of processes when generating cache. Already cached
shards are loaded sequentially.
```

for batch:
```
num_proc (int, optional, defaults to None): The number of processes to use for multiprocessing. If None, no 
multiprocessing is used. This can significantly speed up batching for large datasets.
```

So what happens to `map.num_proc` - is it the same behavior as `batch.num_proc` - so only if `num_proc=None` then no multiprocessing is used?

Let's update the doc to be unambiguous.

**bonus**: we could make all of these behave similarly to `DataLoader.num_workers` - where `num_workers==0` implies no multiprocessing. I think that's the most intuitive, IMHO. 0 workers - the main process has to do all the work. `None` could be the same as `0`.

context: debugging a failing `map`

Thank you!
Broken link in documentation for "Create a video dataset"
Author: cleong110
State: open

The link to "the [WebDataset documentation](https://webdataset.github.io/webdataset)." is broken. 
https://huggingface.co/docs/datasets/main/en/video_dataset#webdataset 

<img width="2048" height="264" alt="Image" src="https://github.com/user-attachments/assets/975dd10c-aad8-42fc-9fbc-de0e2747a326" />

Comments: The URL is ok but it seems the webdataset website is down. There seems to be a related issue here: https://github.com/webdataset/webdataset/issues/155

Feel free to ask the authors there for an update. Otherwise happy to witch the link to the mirror shared in that issue
NotImplementedError when using streaming=True in Google Colab environment
Author: Aniket17200
State: open

### Describe the bug

When attempting to load a large dataset (like tiiuae/falcon-refinedweb or allenai/c4) using streaming=True in a standard Google Colab notebook, the process fails with a NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet. This issue persists even after upgrading datasets and huggingface_hub and restarting the session.

### Steps to reproduce the bug

Open a new Google Colab notebook.

(Optional but recommended) Run !pip install --upgrade datasets huggingface_hub and restart the runtime.

Run the following code:

Python

from datasets import load_dataset

try:
    print("Attempting to load a stream...")
    streaming_dataset = load_dataset('tiiuae/falcon-refinedweb', streaming=True)
    print("Success!")
except Exception as e:
    print(e)

### Expected behavior

The load_dataset command should return a StreamingDataset object without raising an error, allowing iteration over the dataset.

Actual Behavior
The code fails and prints the following error traceback:

[PASTE THE FULL ERROR TRACEBACK HERE]
(Note: Copy the entire error message you received, from Traceback... to the final error line, and paste it in this section.)

### Environment info

Platform: Google Colab

datasets version: [Run !pip show datasets in Colab and paste the version here]

huggingface_hub version: [Run !pip show huggingface_hub and paste the version here]

Python version: [Run !python --version and paste the version here]

Comments: Hi, @Aniket17200, try upgrading datasets using '!pip install -U datasets'. I hope this will resolve your issue.

Thank you @tanuj-rai, it's working great
load_dataset() in 4.0.0 returns different audio samples compared to earlier versions breaking reproducibility
Author: Manalelaidouni
State: closed

### Describe the bug

In datasets 4.0.0 release, `load_dataset()` returns different audio samples compared to earlier versions, this breaks integration tests that depend on consistent sample data across different environments (first and second envs specified below).



### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(24000))
sample= ds[0]["audio"]["array"]
print(sample)

# sample in 3.6.0 
[0.00231914 0.00245417 0.00187414 ... 0.00061956 0.00101157 0.00076325]


# sample in 4.0.0 
array([0.00238037, 0.00220794, 0.00198703, ..., 0.00057983, 0.00085863,
       0.00115309], dtype=float32)
```

### Expected behavior

The same dataset should load identical samples across versions to maintain reproducibility.

### Environment info

First env:
- datasets version: 3.6.0
- Platform: Windows-10-10.0.26100-SP0
- Python: 3.11.0

Second env:
- datasets  version: 4.0.0
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python: 3.11.13

Comments: Hi ! This is because `datasets` now uses the FFmpeg-based library `torchcodec` instead of the libsndfile-based library `soundfile` to decode audio data. Those two have different decoding implementations

I’m all for torchcodec, good luck with the migration!
xopen: invalid start byte for streaming dataset with trust_remote_code=True
Author: sedol1339
State: open

### Describe the bug

I am trying to load YODAS2 dataset with datasets==3.6.0
```
from datasets import load_dataset
next(iter(load_dataset('espnet/yodas2', name='ru000', split='train', streaming=True, trust_remote_code=True)))
```
And get `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte`

The cause of the error is the following:

```
from datasets.utils.file_utils import xopen

filepath = 'https://huggingface.co/datasets/espnet/yodas2/resolve/c9674490249665d658f527e2684848377108d82c/data/ru000/text/00000000.json'
xopen(filepath, 'r').read()
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte
```

And the cause of this is the following:

```
import fsspec

fsspec.open(
    'hf://datasets/espnet/yodas2@c9674490249665d658f527e2684848377108d82c/data/ru000/text/00000000.json',
    mode='r',
    hf={'token': None, 'endpoint': 'https://huggingface.co'},
).open().read()
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte
```

Is it true that streaming=True loading is not supported anymore for trust_remote_code=True, even with datasets==3.6.0? This breaks backward compatibility.

### Steps to reproduce the bug

```
from datasets import load_dataset
next(iter(load_dataset('espnet/yodas2', name='ru000', split='train', streaming=True)))
```

### Expected behavior

No errors expected

### Environment info

datasets==3.6.0, ubuntu 24.04

Comments: Hi ! it would be cool to convert this dataset to Parquet. This will make it work for `datasets>=4.0`, enable the Dataset Viewer and make it more reliable to load/stream (currently it uses a loading script in python and those are known for having issues sometimes)

using `datasets==3.6.0`, here is the command to convert it and open a Pull Request:

```
datasets-cli convert_to_parquet espnet/yodas2 --trust_remote_code
```

Though it's likely that the `UnicodeDecodeError` comes from the loading script. If the script has a bug, it must be fixed to be able to convert the dataset without errors
No module named "distributed"
Author: yingtongxiong
State: open

### Describe the bug

hello, when I run the command "from datasets.distributed import split_dataset_by_node", I always met the bug "No module named 'datasets.distributed" in different version like 4.0.0, 2.21.0 and so on. How can I solve this?

### Steps to reproduce the bug

1. pip install datasets
2. from datasets.distributed import split_dataset_by_node

### Expected behavior

expecting the command "from datasets.distributed import split_dataset_by_node" can be ran successfully

### Environment info

python: 3.12

Comments: The error ModuleNotFoundError: No module named 'datasets.distributed' means your installed datasets library is too old or incompatible with the version of Library you are using(in my case it was BEIR). The datasets.distributed module was removed in recent versions of the datasets library.

Downgrade datasets to version 2.14.6 : ! pip install datasets==2.14.6


this code does run in `datasets` 4.0:
```python
from datasets.distributed import split_dataset_by_node
```

make sure you have a python version that is recent enough (>=3.9) to be able to install `datasets` 4.0

I do think the problem is caused by the python version, because I do have python version 3.12.5
Datasets keeps rebuilding the dataset every time i call the python script
Author: CALEB789
State: open

### Describe the bug

Every time it runs, somehow, samples increase.
This can cause a 12mb dataset to have other built versions of 400 mbs+

<img width="363" height="481" alt="Image" src="https://github.com/user-attachments/assets/766ce958-bd2b-41bc-b950-86710259bfdc" />

### Steps to reproduce the bug


`from datasets import load_dataset
s = load_dataset('~/.cache/huggingface/datasets/databricks___databricks-dolly-15k')['train']
`
1. A dataset needs to be available in the .cache folder
2. Run the code multiple times, and every time it runs, more versions are created


### Expected behavior

The number of samples increases every time the script runs

### Environment info

- `datasets` version: 3.6.0
- Platform: Windows-11-10.0.26100-SP0
- Python version: 3.13.3
- `huggingface_hub` version: 0.32.3
- PyArrow version: 20.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0


Comments: here is the code to load the dataset form the cache:

```python
s = load_dataset('databricks/databricks-dolly-15k')['train']
```

if you pass the location of a local directory it will create a new cache based on that directory content
Probabilistic High Memory Usage and Freeze on Python 3.10
Author: ryan-minato
State: open

### Describe the bug

A probabilistic issue encountered when processing datasets containing PIL.Image columns using the huggingface/datasets library on Python 3.10. The process occasionally experiences a sudden and significant memory spike, reaching 100% utilization, leading to a complete freeze. During this freeze, the process becomes unresponsive, cannot be forcefully terminated, and does not throw any exceptions.

I have attempted to mitigate this issue by setting `datasets.config.IN_MEMORY_MAX_SIZE`, but it had no effect. In fact, based on the document of `load_dataset`, I suspect that setting `IN_MEMORY_MAX_SIZE` might even have a counterproductive effect.

This bug is not consistently reproducible, but its occurrence rate significantly decreases or disappears entirely when upgrading Python to version 3.11 or higher. Therefore, this issue also serves to share a potential solution for others who might encounter similar problems.

### Steps to reproduce the bug

Due to the probabilistic nature of this bug, consistent reproduction cannot be guaranteed for every run. However, in my environment, processing large datasets like timm/imagenet-1k-wds(whether reading, casting, or mapping operations) almost certainly triggers the issue at some point.
The probability of the issue occurring drastically increases when num_proc is set to a value greater than 1 during operations.

When the issue occurs, my system logs repeatedly show the following warnings:
```
WARN: very high memory utilization: 57.74GiB / 57.74GiB (100 %)
WARN: container is unhealthy: triggered memory limits (OOM)
WARN: container is unhealthy: triggered memory limits (OOM)
WARN: container is unhealthy: triggered memory limits (OOM)
```

### Expected behavior

The dataset should be read and processed normally without memory exhaustion or freezing. If an unrecoverable error occurs, an appropriate exception should be raised.

I have found that upgrading Python to version 3.11 or above completely resolves this issue. On Python 3.11, when memory usage approaches 100%, it suddenly drops before slowly increasing again. I suspect this behavior is due to an expected memory management action, possibly involving writing to disk cache, which prevents the complete freeze observed in Python 3.10.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.15.0-71-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.33.4
- PyArrow version: 20.0.0
- Pandas version: 2.3.1
- `fsspec` version: 2025.3.0
metric glue breaks with 4.0.0
Author: stas00
State: closed

### Describe the bug

worked fine with 3.6.0, and with 4.0.0 `eval_metric = metric.compute()` in HF Accelerate breaks.

The code that fails is:
https://huggingface.co/spaces/evaluate-metric/glue/blob/v0.4.0/glue.py#L84

```
def simple_accuracy(preds, labels):
    print(preds, labels)
    print(f"{preds==labels}")
    return float((preds == labels).mean())
```
data:
```
Column([1, 0, 0, 1, 1]) Column([1, 0, 0, 1, 0])
False
```

```
[rank0]:     return float((preds == labels).mean())
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: AttributeError: 'bool' object has no attribute 'mean'
```

Some behavior has changed in this new major release of `datasets` and requires updating HF accelerate and perhaps the glue metric code, all belong to HF.

### Environment info

datasets=4.0.0

Comments: I released `evaluate` 0.4.5 yesterday to fix the issue - sorry for the inconvenience:

```
pip install -U evaluate
```

Thanks so much, @lhoestq!