MarouaneSanhaji/github-issues huggingface dataset:
Document HDF5 support
State: closed

I think these are at least the main places where we should put content. Ideally it is not just repeated in the final version


ref #7690 


- [x] Wait for #7743 to land

@lhoestq any guidance on what else to add/feedback on what is there now? It seems a bit minimal, but I don't think it's worth doing an entire page on HDF5?
Replacement of "Sequence" feature with "List" breaks backward compatibility
State: open

PR #7634 replaced the Sequence feature with List in 4.0.0, so datasets saved with version 4.0.0 with that feature cannot be loaded by earlier versions. There is no clear option in 4.0.0 to use the legacy feature type to preserve backward compatibility.

Why is this a problem? I have a complex preprocessing and training pipeline dependent on 3.6.0; we manage a very large number of separate datasets that get concatenated during training. If just one of those datasets is saved with 4.0.0, they become unusable, and we have no way of "fixing" them. I can load them in 4.0.0 but I can't re-save with the legacy feature type, and I can't load it in 3.6.0 for obvious reasons.

Perhaps I'm missing something here, since the PR says that backward compatibility is preserved; if so, it's not obvious to me how.

Backward compatibility here means 4.0.0 can load datasets saved with older versions.

You will need 4.0.0 to load datasets saved with 4.0.0
Allow saving multi-dimensional ndarray with dynamic shapes
State: open

### Feature request

I propose adding a dedicated feature to the datasets library that allows for the efficient storage and retrieval of multi-dimensional ndarray with dynamic shapes. Similar to how Image columns handle variable-sized images, this feature would provide a structured way to store array data where the dimensions are not fixed.

A possible implementation could be a new Array or Tensor feature type that stores the data in a structured format, for example, 

```python
{
  "shape": (5, 224, 224), 
  "dtype": "uint8", 
  "data": [...]
}
```

This would allow the datasets library to handle heterogeneous array sizes within a single column without requiring a fixed shape definition in the feature schema.

### Motivation

I am currently trying to upload data from astronomical telescopes, specifically FITS files, to the Hugging Face Hub. This type of data is very similar to images but often has more than three dimensions. For example, data from the SDSS project contains five channels (u, g, r, i, z), and the pixel values can exceed 255, making the Pillow based Image feature unsuitable.

The current datasets library requires a fixed shape to be defined in the feature schema for multi-dimensional arrays, which is a major roadblock. This prevents me from saving my data, as the dimensions of the arrays can vary across different FITS files.

https://github.com/huggingface/datasets/blob/985c9bee6bfc345787a8b9dd316e1d4f3b930503/src/datasets/features/features.py#L613-L614

A feature that supports dynamic shapes would be incredibly beneficial for the astronomy community and other fields dealing with similar high-dimensional, variable-sized data (e.g., medical imaging, scientific simulations).

### Your contribution

I am willing to create a PR to help implement this feature if the proposal is accepted.

I agree this would be super valuable.

It looks like this was discussed a few years ago in https://github.com/huggingface/datasets/issues/5272#issuecomment-1550200824 but there were some issues. Those PRs are merged now and it looks like Arrow [officially supports](https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor) this so it's a good time to re-evaluate!

Happy to help with this, maybe we can think of adding a new type `Tensor` (instead of Array2D, 3D etc. which imply a fixed number of dims - we can keep them for backward compat anyways) that uses VariableShapeTensor (or FixedShapeTensor if the shape is provided maybe ? happy to discuss this)
docs: Add column overwrite example to batch mapping guide
State: closed

This PR adds a complementary example showing the **column-overwriting** pattern, which is both more direct and more flexible for many transformations.

### Proposed Change

The original `remove_columns` example remains untouched. Below it, this PR introduces an alternative approach that overwrites an existing column during batch mapping.

This teaches users a core `.map()` capability for in-place transformations without extra intermediate steps.

**New Example:**

> ```python
> >>> from datasets import Dataset
> >>> dataset = Dataset.from_dict({"a": [0, 1, 2]})
> # Overwrite "a" directly to duplicate each value
> >>> duplicated_dataset = dataset.map(
> ...     lambda batch: {"a": [x for x in batch["a"] for _ in range(2)]},
> ...     batched=True
> ... )
> >>> duplicated_dataset
> Dataset({
>     features: ['a'],
>     num_rows: 6
> })
> >>> duplicated_dataset["a"]
> [0, 0, 1, 1, 2, 2]
> ```


Hi @lhoestq, just a gentle follow-up on this PR.
Fixing __getitem__ of datasets which behaves inconsistent to documentation when setting _format_type to None
State: closed

Setting _format_type to None, should return plain python object but as of 4.0.0 returns Column. This fails in libs such as sentencetransformers (such as in generation of hard negatives) where plain python is expected.

this breaking change is actually expected, happy to help with a fix in sentencetransformers to account for this

Thank you for the context. I thought this was a mismatch do the documentation. Good to know it was intentional. No worries, can add a PR to sentence transformers.
Add the possibility of a backend for audio decoding
State: open

### Feature request

Add the possibility of a backend for audio decoding. Before version 4.0.0, soundfile was used, and now torchcodec is used, but the problem is that torchcodec requires ffmpeg, which is problematic to install on the same colab. Therefore, I suggest adding a decoder selection when loading the dataset.

### Motivation

I use a service for training models in which ffmpeg cannot be installed.

### Your contribution

I use a service for training models in which ffmpeg cannot be installed.

is there a work around im stuck

never mind just downgraded
OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory
State: open

> Hi  is there any solution for that eror i try to install this one 
pip install torch==1.12.1+cpu torchaudio==0.12.1+cpu -f https://download.pytorch.org/whl/torch_stable.html  
this is working fine but tell me how to install pytorch version that is fit for gpu 

Is this related to the "datasets" library? @SaleemMalikAI
NonMatchingSplitsSizesError and ExpectedMoreSplitsError
State: open

### Describe the bug

When loading dataset, the info specified by `data_files` did not overwrite the original info.


### Steps to reproduce the bug

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz", 
                        "validation": "en/c4-validation.00000-of-00008.json.gz"},
        )
```


```log
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=828589180707, num_examples=364868892, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=809262831, num_examples=356317, shard_lengths=[223006, 133311], dataset_name='c4')}, {'expected': SplitInfo(name='validation', num_bytes=825767266, num_examples=364608, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=102199431, num_examples=45576, shard_lengths=None, dataset_name='c4')}]
```

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
            split="train"
        )
```

```log
ExpectedMoreSplitsError: {'validation'}
```

### Expected behavior

No error

### Environment info

datasets 4.0.0

To load just one shard without errors, you should use data_files directly with split set to "train", but don’t specify "allenai/c4", since that points to the full dataset with all shards.

Instead, do this:
```
from datasets import load_dataset
from datasets import load_dataset

# Load only one shard of C4
traindata = load_dataset(
    "json",   # <-- use "json" since you’re directly passing JSON files
    data_files={"train": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz"},
    split="train"
)

print(traindata)
```
If you want both train and validation but only a subset of shards, do:
```
traindata = load_dataset(
    "json",
    data_files={
        "train": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz",
        "validation": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-validation.00000-of-00008.json.gz"
    }
)

print(traindata)
```

I just want to load a few files from allenai/c4.
If I do not specify allenai/c4, where will the files be loaded from?

My apologies, I’ve modified my previous answer.
You just need to specify the full path, for example:

https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz

<img width="1843" height="633" alt="Image" src="https://github.com/user-attachments/assets/b2922958-9d87-4b62-a00e-c5ca02e31c27" />

I hope this updated answer is helpful.
config paths that start with ./ are not valid as hf:// accessed repos, but are valid when accessed locally
State: open

### Describe the bug

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - images/xyz/*.jpg
```
will correctly download but

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```
will error with `FileNotFoundError` due to improper url joining. `load_dataset` on the same directory locally works fine.

### Steps to reproduce the bug

1. create a README.md with the front matter of the form

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```

2. `touch ./images/xyz/1.jpg`
3. Observe this directory loads with `load_dataset("filesystem_path", "some_config")` correctly.
4. Observe exceptions when you load this with `load_dataset("repoid/filesystem_path", "some_config")`

### Expected behavior

`./` prefix should be interpreted correctly

### Environment info

datasets 4.0.0
datasets 3.4.0

reproduce
fix(webdataset): don't .lower() field_name
State: closed

This fixes cases where keys have upper case identifiers

fixes: https://github.com/huggingface/datasets/issues/7732

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7726). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

CI failures are unrelated, merging :)
Don't remove `trust_remote_code` arg!!!
State: open

### Feature request

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!

Add `trust_remote_code` arg back please!

### Motivation

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!


### Your contribution

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!
Out of memory even though using load_dataset(..., streaming=True)
State: open

### Describe the bug

I am iterating over a large dataset that I load using streaming=True to avoid running out of memory. Unfortunately, I am observing that memory usage increases over time and I'm finally running in an oom.

### Steps to reproduce the bug

```
  ds = load_dataset("openslr/librispeech_asr", split="train.clean.360", streaming=True)
  for i,sample in enumerate(tqdm(ds)):
    target_file = os.path.join(NSFW_TARGET_FOLDER, f'audio{i}.wav')
    try:
      sf.write(target_file, sample['audio']['array'], samplerate=sample['audio']['sampling_rate'])
    except Exception as e:
      print(f"Could not write audio {i} in ds: {e}")
```


### Expected behavior

I'd expect to have a small memory footprint and memory being freed after each iteration of the for loop. Instead the memory usage is increasing. I tried to remove the logic to write the sound file and just print the sample but the issue remains the same.

### Environment info

Python 3.12.11
Ubuntu 24
datasets 4.0.0 and 3.6.0
Bad split error message when using percentages
State: open

### Describe the bug

Hi, I'm trying to download a dataset. To not load the entire dataset in memory, I split it as described [here](https://huggingface.co/docs/datasets/v4.0.0/loading#slice-splits) in 10% steps.

When doing so, the library returns this error:
    raise ValueError(f"Bad split: {split}. Available splits: {list(splits_generators)}")
ValueError: Bad split: train[0%:10%]. Available splits: ['train']

Edit: Same happens with a split like _train[:90000]_
### Steps to reproduce the bug

```
  for split in range(10):
    split_str = f"train[{split*10}%:{(split+1)*10}%]"
    print(f"Processing split {split_str}...")
    ds = load_dataset("user/dataset", split=split_str, streaming=True)
```


### Expected behavior

I'd expect the library to split my dataset in 10% steps.

### Environment info

python 3.12.11
ubuntu 24
dataset 4.0.0

I'd like to work on this: add clearer validation/messages for percent-based splits + tests

The most basic example is this code:
`load_dataset("openslr/librispeech_asr", split="train[10%:20%]")`

This results in this ValueError:
```
    raise ValueError(f'Unknown split "{split}". Should be one of {list(name2len)}.')
ValueError: Unknown split "train". Should be one of ['test.clean', 'test.other', 'train.clean.100', 'train.clean.360', 'train.other.500', 'validation.clean', 'validation.other'].
```
Datasets 4.0 map function causing column not found
State: open

### Describe the bug

Column returned after mapping is not found in new instance of the dataset.

### Steps to reproduce the bug

Code for reproduction. After running get_total_audio_length, it is errored out due to `data` not having `duration`
```
def compute_duration(x):
    return {"duration": len(x["audio"]["array"]) / x["audio"]["sampling_rate"]}


def get_total_audio_length(dataset):
    data = dataset.map(compute_duration, num_proc=NUM_PROC)
    print(data)
    durations=data["duration"]
    total_seconds = sum(durations)
    return total_seconds
```

### Expected behavior

New datasets.Dataset instance should have new columns attached.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.33.2
- PyArrow version: 20.0.0
- Pandas version: 2.3.0
- `fsspec` version: 2023.12.2

Hi, I tried to reproduce this issue on the latest `main` branch but it seems to be working correctly now. My test script (which creates a dummy dataset and applies the `.map()` function) successfully creates and accesses the new column without a `KeyError`.

It's possible this was fixed by a recent commit. The maintainers might want to consider closing this issue.

Hi, have you tried on a large dataset (200GB+) perhaps? I will try my best to do a rerun with main branch when I have the time.

I ran it on a small dataset, maybe that’s why I didn’t hit the issue. If it still shows up on your side with the latest main, let me know. I can try it on a bigger set too.
Specify dataset columns types in typehint
State: open

### Feature request

Make dataset optionaly generic to datasets usage with type annotations like it was done in `torch.Dataloader` https://github.com/pytorch/pytorch/blob/134179474539648ba7dee1317959529fbd0e7f89/torch/utils/data/dataloader.py#L131

### Motivation

In MTEB we're using a lot of datasets objects, but they're a bit poor in typehints. E.g. we can specify this for dataloder

```python
from typing import TypedDict
from torch.utils.data import DataLoader

class CorpusInput(TypedDict):
    title: list[str]
    body: list[str]


class QueryInput(TypedDict):
    query: list[str]
    instruction: list[str]

def queries_loader() -> DataLoader[QueryInput]:
    ...

def corpus_loader() -> DataLoader[CorpusInput]:
   ...
```
But for datasets we can only specify columns in type in comments 
```python
from datasets import Dataset

QueryDataset = Dataset
"""Query dataset should have `query` and `instructions` columns as `str` """
```

### Your contribution

I can create draft implementation
add support for pyarrow string view in features
State: closed


@lhoestq who do you think would be the best to have a look at this? Any pointers would be appreciated, thanks!

Hi ! what's the rationale for supporting string view ? I'm afraid it can complexify the typing logic without much value

Hi @lhoestq ! I mainly want to be able to create features by using `Features.from_arrow_schema(dataset_schema)` on an arrow dataset with string view columns, currently there is no easy way to do this, and string_view is becoming an increasingly common data type for string columns in arrow. Thanks for having a look!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7718). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cached dataset is not used when explicitly passing the cache_dir parameter
State: open

### Describe the bug

Hi, we are pre-downloading a dataset using snapshot_download(). When loading this exact dataset with load_dataset() the cached snapshot is not used. In both calls, I provide the cache_dir parameter.

### Steps to reproduce the bug

```
from datasets import load_dataset, concatenate_datasets
from huggingface_hub import snapshot_download
    
def download_ds(name: str):
    snapshot_download(repo_id=name, repo_type="dataset", cache_dir="G:/Datasets/cache")

def prepare_ds():
    audio_ds = load_dataset("openslr/librispeech_asr", num_proc=4, cache_dir="G:/Datasets/cache")
    print(sfw_ds.features)

if __name__ == '__main__':
    download_ds("openslr/librispeech_asr")
    prepare_ds()
```

### Expected behavior

I'd expect that the cached version of the dataset is used. Instead, the same dataset is downloaded again to the default cache directory.

### Environment info

Windows 11
datasets==4.0.0
Python 3.12.11

Hi, I've investigated this issue and can confirm the bug. Here are my findings:

**1. Reproduction:**
I was able to reproduce the issue on the latest `main` branch. Using the provided code snippet, `snapshot_download` correctly populates the custom `cache_dir`, but `load_dataset` with the same `cache_dir` triggers a full re-download and re-processing of the dataset, ignoring the existing cache.

**2. Investigation:**
I traced the `cache_dir` parameter from `load_dataset` down to the `DatasetBuilder` class in `src/datasets/builder.py`. The root cause seems to be a mismatch between the cache path structure created by `snapshot_download` and the path structure expected by the `DatasetBuilder`.

Specifically, the `_relative_data_dir` method in `DatasetBuilder` constructs a path using `namespace___dataset_name` (with three underscores), while the cache from `snapshot_download` appears to use a `repo_id` based format like `datasets--namespace--dataset_name` (with double hyphens).

**3. Attempted Fix & Result:**
I attempted a fix by modifying the `_relative_data_dir` method to replace the path separator "/" in `self.repo_id` with "--", to align it with the `snapshot_download` structure.

This partially worked: `load_dataset` no longer re-downloads the files. However, it still re-processes them every time (triggering "Generating train split...", etc.) instead of loading the already processed Arrow files from the cache.

This suggests the issue is deeper than just the directory name and might be related to how the builder verifies the integrity or presence of the processed cache files.

I hope these findings are helpful for whoever picks up this issue.
Concurrent IterableDataset push_to_hub
State: closed

Same as https://github.com/huggingface/datasets/pull/7708 but for `IterableDataset`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7710). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Release 4.0.0 breaks usage patterns of with_format
State: closed

### Describe the bug

Previously it was possible to access a whole column that was e.g. in numpy format via `with_format` by indexing the column. Now this possibility seems to be gone with the new Column() class. As far as I see, this makes working on a whole column (in-memory) more complex, i.e. normalizing an in-memory dataset for which iterating would be too slow. Is this intended behaviour? I couldn't find much documentation on the intended usage of the new Column class yet.

### Steps to reproduce the bug

Steps to reproduce:
```
from datasets import load_dataset
dataset = load_dataset("lhoestq/demo1")
dataset = dataset.with_format("numpy")
print(dataset["star"].ndim)
```

### Expected behavior

Working on whole columns should be possible.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.8.0-63-generic-x86_64-with-glibc2.36
- Python version: 3.12.11
- `huggingface_hub` version: 0.34.3
- PyArrow version: 21.0.0
- Pandas version: 2.3.1
- `fsspec` version: 2025.3.0

This is a breaking change with 4.0 which introduced `Column` objects. To get the numpy array from a `Column` you can `col[i]`, `col[i:j]` or even `col[:]` if you want the full column as a numpy array:

```python
from datasets import load_dataset
dataset = load_dataset(...)
dataset = dataset.with_format("numpy")
print(dataset["star"][:].ndim)
```

Ah perfect, thanks for clearing this up. I would close this ticket then.
Concurrent push_to_hub
State: closed

Retry the step that (download + update + upload) the README.md using `create_commit(..., parent_commit=...)` if there was a commit in the meantime. This should enable concurrent `push_to_hub()` since it won't overwrite the README.md metadata anymore.

Note: we fixed an issue server side to make this work:

<details>

DO NOT MERGE FOR NOW since it seems there is one bug that prevents this logic from working:

I'm using parent_commit to enable concurrent push_to_hub() in datasets for a retry mechanism, but for some reason I always run into a weird situation.
Sometimes create_commit(.., parent_commit=...) returns error 500 but the commit did happen on the Hub side without respecting parent_commit

e.g. request id
```
huggingface_hub.errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/lhoestq/tmp/commit/main (Request ID: Root=1-6888d8af-2ce517bc60c69cb378b51526;d1b17993-c5d0-4ccd-9926-060c45f9ed61)
```

fix coming in [internal](https://github.com/huggingface-internal/moon-landing/pull/14617)

</details>

close https://github.com/huggingface/datasets/issues/7600

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7708). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Reimplemented partial split download support (revival of #6832)
State: open

(revival of #6832)

https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130

Close https://github.com/huggingface/datasets/issues/4101, and more

---
### PR under work!!!!


 Mario’s Patch (in PR #6832):
```
def _make_split_generators_kwargs(self, prepare_split_kwargs):
    # Pass `pipeline` into `_split_generators()` from `prepare_split_kwargs` if
    # it's in the call signature of `_split_generators()`.
    # This allows for global preprocessing in beam.
    split_generators_kwargs = {}
    if "pipeline" in inspect.signature(self._split_generators).parameters:
        split_generators_kwargs["pipeline"] = prepare_split_kwargs["pipeline"]
    split_generators_kwargs.update(super()._make_split_generators_kwargs(prepare_split_kwargs))
    return split_generators_kwargs
```

In the latest main(in my fork and og repo's main):
```
def _make_split_generators_kwargs(self, prepare_split_kwargs):
    """Get kwargs for `self._split_generators()` from `prepare_split_kwargs`."""
    splits = prepare_split_kwargs.pop("splits", None)
    if self._supports_partial_generation():
        return {"splits": splits}
    return {}
```
It enables passing splits into _split_generators() only for builders that support it(if i am not wrong..). So ignored Beam logic for now!

Awesome ! btw we can modify the GeneratorBasedBuilder and ArrowBasedBuilder if needed now that custom loading scripts are not supported anymore :)

I'll review this in a bit
Can Not read installed dataset in dataset.load(.)
State: open

Hi, folks, I'm newbie in huggingface dataset api.
As title, i'm facing the issue that the dataset.load api can not connect to the installed dataset.

code snippet :
<img width="572" height="253" alt="Image" src="https://github.com/user-attachments/assets/10f48aaf-d6ca-4239-b1cf-145d74f125d1" />

data path : 
"/xxx/joseph/llava_ds/vlm_ds"
it contains all video clips i want!
<img width="1398" height="261" alt="Image" src="https://github.com/user-attachments/assets/bf213b66-e344-4311-97e7-bc209677ae77" />

i run the py script by 
<img width="1042" height="38" alt="Image" src="https://github.com/user-attachments/assets/8b3fcee4-e1a6-41b8-bee1-91567b00d9d2" />

But bad happended, even i provide dataset path by "HF_HUB_CACHE", it still attempt to download data from remote side : 

<img width="1697" height="813" alt="Image" src="https://github.com/user-attachments/assets/baa6cff1-a724-4710-a8c4-4805459deffb" />

Any suggestion will be appreciated!!

You can download the dataset locally using [huggingface_hub.snapshot_download](https://huggingface.co/docs/huggingface_hub/v0.34.3/en/package_reference/file_download#huggingface_hub.snapshot_download) and then do

```python
dataset = load_dataset(local_directory_path)
```

> You can download the dataset locally using [huggingface_hub.snapshot_download](https://huggingface.co/docs/huggingface_hub/v0.34.3/en/package_reference/file_download#huggingface_hub.snapshot_download) and then do
> 
> dataset = load_dataset(local_directory_path)

It's good suggestion, but my server env is network restriction. It can not directly fetch data from huggingface. I spent lot of time to download and transfer it to the server.
So, I attempt to make load_dataset connect to my local dataset.  

Just Solved it few day before. Will post solution later...
also thanks folks quick reply..
Fix map() example in datasets documentation: define tokenizer before use
State: closed

## Problem
The current datasets.Dataset.map() example in the documentation demonstrates batched processing using a tokenizer object without defining or importing it. This causes a NameError when users copy and run the example as-is, breaking the expected seamless experience.

## Correction
This PR fixes the issue by explicitly importing and initializing the tokenizer using the Transformers library (AutoTokenizer.from_pretrained("bert-base-uncased")), making the example self-contained and runnable without errors.
This will help new users understand the workflow and apply the method correctly.

Closes #7703 



Hi @lhoestq, just a gentle follow-up on this doc fix PR (#7704). Let me know if any changes are needed — happy to update.
Hope this improvement helps users run the example without confusion!

the modified file is the readme of the docs, not about map() specifically
[Docs] map() example uses undefined `tokenizer` — causes NameError
State: open

## Description

The current documentation example for `datasets.Dataset.map()` demonstrates batched processing but uses a `tokenizer` object without defining or importing it. This causes an error every time it's copied.

Here is the problematic line:

```python
# process a batch of examples
>>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This assumes the user has already set up a tokenizer, which contradicts the goal of having self-contained, copy-paste-friendly examples. 

## Problem
Users who copy and run the example as-is will encounter:

```python
NameError: name 'tokenizer' is not defined
```

This breaks the flow for users and violates HuggingFace's documentation principle that examples should "work as expected" when copied directly. 

## Proposal
Update the example to include the required tokenizer setup using the Transformers library, like so:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
ds_tokenized = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This will help new users understand the workflow and apply the method correctly.

## Note
This PR complements ongoing improvements like #7700, which clarifies multiprocessing in .map(). My change focuses on undefined tokenizer — causes NameError






I've submitted PR #7704 which adds documentation to clarify the behavior of `map()` when returning `None`.
num_proc=0 behave like None, num_proc=1 uses one worker (not main process) and clarify num_proc documentation
State: closed

Fixes issue #7700

This PR makes num_proc=0 behave like None in Dataset.map(), disabling multiprocessing.
It improves UX by aligning with DataLoader(num_workers=0) behavior.
The num_proc docstring is also updated to clearly explain valid values and behavior.

@SunMarc


I think we can support num_proc=0 and make it equivalent to `None` to make it simpler

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7702). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> I think we can support num_proc=0 and make it equivalent to `None` to make it simpler

Thank you @lhoestq for reviewing it. Please let me know if anything needs to be updated further.
Update fsspec max version to current release 2025.7.0
State: closed

Diffusers currently asks for a max fsspec version of `2025.3.0`. This change updates it to the current latest version. This change is mainly required to resolve conflicts with other packages in an environment. In my particular case, `aider-chat` which is a part of my environment installs `2025.5.1` which is incompatible with `datasets`.

@lhoestq I ran the test suite locally and while some tests were failing those failures are present on the main branch too. Could you please review and trigger the CI?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7701). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Which release will this be available in ? I'm running into this issue with `datasets=3.6.0`
[doc] map.num_proc needs clarification
State: open

https://huggingface.co/docs/datasets/v4.0.0/en/package_reference/main_classes#datasets.Dataset.map.num_proc
```
num_proc (int, optional, defaults to None) — Max number of processes when generating cache. Already cached
shards are loaded sequentially.
```

for batch:
```
num_proc (int, optional, defaults to None): The number of processes to use for multiprocessing. If None, no 
multiprocessing is used. This can significantly speed up batching for large datasets.
```

So what happens to `map.num_proc` - is it the same behavior as `batch.num_proc` - so only if `num_proc=None` then no multiprocessing is used?

Let's update the doc to be unambiguous.

**bonus**: we could make all of these behave similarly to `DataLoader.num_workers` - where `num_workers==0` implies no multiprocessing. I think that's the most intuitive, IMHO. 0 workers - the main process has to do all the work. `None` could be the same as `0`.

context: debugging a failing `map`

Thank you!
Broken link in documentation for "Create a video dataset"
State: open

The link to "the [WebDataset documentation](https://webdataset.github.io/webdataset)." is broken. 
https://huggingface.co/docs/datasets/main/en/video_dataset#webdataset 

<img width="2048" height="264" alt="Image" src="https://github.com/user-attachments/assets/975dd10c-aad8-42fc-9fbc-de0e2747a326" />

The URL is ok but it seems the webdataset website is down. There seems to be a related issue here: https://github.com/webdataset/webdataset/issues/155

Feel free to ask the authors there for an update. Otherwise happy to witch the link to the mirror shared in that issue
NotImplementedError when using streaming=True in Google Colab environment
State: open

### Describe the bug

When attempting to load a large dataset (like tiiuae/falcon-refinedweb or allenai/c4) using streaming=True in a standard Google Colab notebook, the process fails with a NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet. This issue persists even after upgrading datasets and huggingface_hub and restarting the session.

### Steps to reproduce the bug

Open a new Google Colab notebook.

(Optional but recommended) Run !pip install --upgrade datasets huggingface_hub and restart the runtime.

Run the following code:

Python

from datasets import load_dataset

try:
    print("Attempting to load a stream...")
    streaming_dataset = load_dataset('tiiuae/falcon-refinedweb', streaming=True)
    print("Success!")
except Exception as e:
    print(e)

### Expected behavior

The load_dataset command should return a StreamingDataset object without raising an error, allowing iteration over the dataset.

Actual Behavior
The code fails and prints the following error traceback:

[PASTE THE FULL ERROR TRACEBACK HERE]
(Note: Copy the entire error message you received, from Traceback... to the final error line, and paste it in this section.)

### Environment info

Platform: Google Colab

datasets version: [Run !pip show datasets in Colab and paste the version here]

huggingface_hub version: [Run !pip show huggingface_hub and paste the version here]

Python version: [Run !python --version and paste the version here]

Hi, @Aniket17200, try upgrading datasets using '!pip install -U datasets'. I hope this will resolve your issue.

Thank you @tanuj-rai, it's working great
load_dataset() in 4.0.0 returns different audio samples compared to earlier versions breaking reproducibility
State: closed

### Describe the bug

In datasets 4.0.0 release, `load_dataset()` returns different audio samples compared to earlier versions, this breaks integration tests that depend on consistent sample data across different environments (first and second envs specified below).



### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(24000))
sample= ds[0]["audio"]["array"]
print(sample)

# sample in 3.6.0 
[0.00231914 0.00245417 0.00187414 ... 0.00061956 0.00101157 0.00076325]


# sample in 4.0.0 
array([0.00238037, 0.00220794, 0.00198703, ..., 0.00057983, 0.00085863,
       0.00115309], dtype=float32)
```

### Expected behavior

The same dataset should load identical samples across versions to maintain reproducibility.

### Environment info

First env:
- datasets version: 3.6.0
- Platform: Windows-10-10.0.26100-SP0
- Python: 3.11.0

Second env:
- datasets  version: 4.0.0
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python: 3.11.13

Hi ! This is because `datasets` now uses the FFmpeg-based library `torchcodec` instead of the libsndfile-based library `soundfile` to decode audio data. Those two have different decoding implementations

I’m all for torchcodec, good luck with the migration!
Support downloading specific splits in load_dataset
State: closed

This PR builds on #6832 by @mariosasko.

May close - #4101, #2538

Discussion - https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130

---
### Note - This PR is under work and frequent changes will be pushed.

I’ve completed the following steps to continue the partial split download support (from PR #6832):

I did changes on top of what has been done by mario. Here are some of those changes: 
- Restored support for writing multiple split shards:

- In _prepare_split_single, we now correctly replace JJJJJ and SSSSS placeholders in the fpath for job/shard IDs before creating the writer.

- Added os.makedirs(os.path.dirname(path), exist_ok=True) after placeholder substitution to prevent FileNotFoundError.

- Applied the fix to both split writers:

       1] self._generate_examples version (used by most modules).

       2] self._generate_tables version (used by IterableDatasetBuilder).

- Confirmed 109/113 tests passing, meaning the general logic is working across the board.

What’s still failing
4 integration tests fail:

`test_load_hub_dataset_with_single_config_in_metadata`

`test_load_hub_dataset_with_two_config_in_metadata`

`test_load_hub_dataset_with_metadata_config_in_parallel`

`test_reload_old_cache_from_2_15`

All are due to FileNotFoundError from uncreated output paths, which I'm currently finalizing by ensuring os.makedirs() is correctly applied before every writer instantiation.

I will update about these fixes after running tests!

@lhoestq this was just an update

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7695). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Local DIR wasn't doing well, dk actually what happened, will PR again! Sorry :)
xopen: invalid start byte for streaming dataset with trust_remote_code=True
State: open

### Describe the bug

I am trying to load YODAS2 dataset with datasets==3.6.0
```
from datasets import load_dataset
next(iter(load_dataset('espnet/yodas2', name='ru000', split='train', streaming=True, trust_remote_code=True)))
```
And get `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte`

The cause of the error is the following:

```
from datasets.utils.file_utils import xopen

filepath = 'https://huggingface.co/datasets/espnet/yodas2/resolve/c9674490249665d658f527e2684848377108d82c/data/ru000/text/00000000.json'
xopen(filepath, 'r').read()
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte
```

And the cause of this is the following:

```
import fsspec

fsspec.open(
    'hf://datasets/espnet/yodas2@c9674490249665d658f527e2684848377108d82c/data/ru000/text/00000000.json',
    mode='r',
    hf={'token': None, 'endpoint': 'https://huggingface.co'},
).open().read()
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte
```

Is it true that streaming=True loading is not supported anymore for trust_remote_code=True, even with datasets==3.6.0? This breaks backward compatibility.

### Steps to reproduce the bug

```
from datasets import load_dataset
next(iter(load_dataset('espnet/yodas2', name='ru000', split='train', streaming=True)))
```

### Expected behavior

No errors expected

### Environment info

datasets==3.6.0, ubuntu 24.04

Hi ! it would be cool to convert this dataset to Parquet. This will make it work for `datasets>=4.0`, enable the Dataset Viewer and make it more reliable to load/stream (currently it uses a loading script in python and those are known for having issues sometimes)

using `datasets==3.6.0`, here is the command to convert it and open a Pull Request:

```
datasets-cli convert_to_parquet espnet/yodas2 --trust_remote_code
```

Though it's likely that the `UnicodeDecodeError` comes from the loading script. If the script has a bug, it must be fixed to be able to convert the dataset without errors
HDF5 support
State: closed

This PR adds support for tabular HDF5 file(s) by converting each row to an Arrow table. It supports columns with the usual dtypes including up to 5-dimensional arrays as well as support for complex/compound types by using `Features(dict)`. All datasets within the HDF5 file should have rows on the first dimension (groups/subgroups are still allowed). Closes #3113.

Replaces #7625 which only supports a relatively small subset of HDF5.

A few to-dos which I think can be left for future PRs (which I am happy to do/help with -- just this one is already huge 😄 ):
- [Enum types](https://docs.h5py.org/en/stable/special.html#enumerated-types)
- HDF5 [io](https://github.com/huggingface/datasets/tree/main/src/datasets/io)
- [dataset-viewer](https://github.com/huggingface/dataset-viewer) support (not sure if changes are needed with the way it is written now)

@lhoestq any interest in merging this? Let me know if I can do anything to make reviewing it easier!

Sorry for the delay, I'll review your PR soon :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7690). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review @lhoestq! Rebased on main and incorporated most of your suggestions.

I believe the only one left is the zero-dim handling with `table_cast`...

@lhoestq is 2c4bfba what you meant?

Awesome! Yes, I'm happy to help with the docs. Would appreciate any pointers, we can discuss in #7740.

It does look like there was a CI test failure, though it seems unrelated?
```
FAILED tests/test_dataset_dict.py::test_dummy_datasetdict_serialize_fs - ValueError: Protocol not known: mock
FAILED tests/test_arrow_dataset.py::test_dummy_dataset_serialize_fs - ValueError: Protocol not known: mock
```
Also, what do you think of the todos in https://github.com/huggingface/datasets/pull/7690#issuecomment-3105391677 ? In particular I think support in dataset-viewer would be nice.

Cool ! Yeah the failure is unrelated

Regarding the Viewer, it should work out of the box when it's updated with the next version of `datasets` :)
No module named "distributed"
State: open

### Describe the bug

hello, when I run the command "from datasets.distributed import split_dataset_by_node", I always met the bug "No module named 'datasets.distributed" in different version like 4.0.0, 2.21.0 and so on. How can I solve this?

### Steps to reproduce the bug

1. pip install datasets
2. from datasets.distributed import split_dataset_by_node

### Expected behavior

expecting the command "from datasets.distributed import split_dataset_by_node" can be ran successfully

### Environment info

python: 3.12

The error ModuleNotFoundError: No module named 'datasets.distributed' means your installed datasets library is too old or incompatible with the version of Library you are using(in my case it was BEIR). The datasets.distributed module was removed in recent versions of the datasets library.

Downgrade datasets to version 2.14.6 : ! pip install datasets==2.14.6


this code does run in `datasets` 4.0:
```python
from datasets.distributed import split_dataset_by_node
```

make sure you have a python version that is recent enough (>=3.9) to be able to install `datasets` 4.0

I do think the problem is caused by the python version, because I do have python version 3.12.5
Datasets keeps rebuilding the dataset every time i call the python script
State: open

### Describe the bug

Every time it runs, somehow, samples increase.
This can cause a 12mb dataset to have other built versions of 400 mbs+

<img width="363" height="481" alt="Image" src="https://github.com/user-attachments/assets/766ce958-bd2b-41bc-b950-86710259bfdc" />

### Steps to reproduce the bug


`from datasets import load_dataset
s = load_dataset('~/.cache/huggingface/datasets/databricks___databricks-dolly-15k')['train']
`
1. A dataset needs to be available in the .cache folder
2. Run the code multiple times, and every time it runs, more versions are created


### Expected behavior

The number of samples increases every time the script runs

### Environment info

- `datasets` version: 3.6.0
- Platform: Windows-11-10.0.26100-SP0
- Python version: 3.13.3
- `huggingface_hub` version: 0.32.3
- PyArrow version: 20.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0


here is the code to load the dataset form the cache:

```python
s = load_dataset('databricks/databricks-dolly-15k')['train']
```

if you pass the location of a local directory it will create a new cache based on that directory content
metric glue breaks with 4.0.0
State: closed

### Describe the bug

worked fine with 3.6.0, and with 4.0.0 `eval_metric = metric.compute()` in HF Accelerate breaks.

The code that fails is:
https://huggingface.co/spaces/evaluate-metric/glue/blob/v0.4.0/glue.py#L84

```
def simple_accuracy(preds, labels):
    print(preds, labels)
    print(f"{preds==labels}")
    return float((preds == labels).mean())
```
data:
```
Column([1, 0, 0, 1, 1]) Column([1, 0, 0, 1, 0])
False
```

```
[rank0]:     return float((preds == labels).mean())
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: AttributeError: 'bool' object has no attribute 'mean'
```

Some behavior has changed in this new major release of `datasets` and requires updating HF accelerate and perhaps the glue metric code, all belong to HF.

### Environment info

datasets=4.0.0

I released `evaluate` 0.4.5 yesterday to fix the issue - sorry for the inconvenience:

```
pip install -U evaluate
```

Thanks so much, @lhoestq!
To support decoding audio data, please install 'torchcodec'.
State: closed


In the latest version of datasets==4.0.0, i cannot print the audio data on the Colab notebook. But it works on the 3.6.0 version.

!pip install -q -U datasets huggingface_hub fsspec

from datasets import load_dataset

downloaded_dataset = load_dataset("ymoslem/MediaSpeech", "tr", split="train")
print(downloaded_dataset["audio"][0])

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
[/tmp/ipython-input-4-90623240.py](https://localhost:8080/#) in <cell line: 0>()
----> 1 downloaded_dataset["audio"][0]

10 frames
[/usr/local/lib/python3.11/dist-packages/datasets/features/audio.py](https://localhost:8080/#) in decode_example(self, value, token_per_repo_id)
    170             from ._torchcodec import AudioDecoder
    171         else:
--> 172             raise ImportError("To support decoding audio data, please install 'torchcodec'.")
    173 
    174         if not self.decode:

ImportError: To support decoding audio data, please install 'torchcodec'. 

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python version: 3.11.13
- `huggingface_hub` version: 0.33.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2025.3.0

Hi ! yes you should `!pip install -U datasets[audio]` to have the required dependencies.

`datasets` 4.0 now relies on `torchcodec` for audio decoding. The `torchcodec` AudioDecoder enables streaming from HF and also allows to decode ranges of audio

Same issues on Colab.

> !pip install -U datasets[audio] 

This works for me. Thanks.