Minimartzz/github-issues huggingface dataset:
Reimplemented partial split download support (revival of #6832)
State: open

(revival of #6832)

https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130

Close https://github.com/huggingface/datasets/issues/4101, and more

---
### PR under work!!!!


Comments:
 Mario’s Patch (in PR #6832):
```
def _make_split_generators_kwargs(self, prepare_split_kwargs):
    # Pass `pipeline` into `_split_generators()` from `prepare_split_kwargs` if
    # it's in the call signature of `_split_generators()`.
    # This allows for global preprocessing in beam.
    split_generators_kwargs = {}
    if "pipeline" in inspect.signature(self._split_generators).parameters:
        split_generators_kwargs["pipeline"] = prepare_split_kwargs["pipeline"]
    split_generators_kwargs.update(super()._make_split_generators_kwargs(prepare_split_kwargs))
    return split_generators_kwargs
```

In the latest main(in my fork and og repo's main):
```
def _make_split_generators_kwargs(self, prepare_split_kwargs):
    """Get kwargs for `self._split_generators()` from `prepare_split_kwargs`."""
    splits = prepare_split_kwargs.pop("splits", None)
    if self._supports_partial_generation():
        return {"splits": splits}
    return {}
```
It enables passing splits into _split_generators() only for builders that support it(if i am not wrong..). So ignored Beam logic for now!

Awesome ! btw we can modify the GeneratorBasedBuilder and ArrowBasedBuilder if needed now that custom loading scripts are not supported anymore :)

I'll review this in a bit

@lhoestq @ArjunJagdale is this still work in progress or is just a review missing? Anything I can help with here? This would indeed be a cool feature

I did a preliminary pass and it looks good but we should check the CI, could you run `make style` @ArjunJagdale so we can run the CI ?

Done! Also some parts may be incomplete because I had to focus on important exams and semester activities so couldn’t finish the work fully. I will still try my best.
Can Not read installed dataset in dataset.load(.)
State: open

Hi, folks, I'm newbie in huggingface dataset api.
As title, i'm facing the issue that the dataset.load api can not connect to the installed dataset.

code snippet :
<img width="572" height="253" alt="Image" src="https://github.com/user-attachments/assets/10f48aaf-d6ca-4239-b1cf-145d74f125d1" />

data path : 
"/xxx/joseph/llava_ds/vlm_ds"
it contains all video clips i want!
<img width="1398" height="261" alt="Image" src="https://github.com/user-attachments/assets/bf213b66-e344-4311-97e7-bc209677ae77" />

i run the py script by 
<img width="1042" height="38" alt="Image" src="https://github.com/user-attachments/assets/8b3fcee4-e1a6-41b8-bee1-91567b00d9d2" />

But bad happended, even i provide dataset path by "HF_HUB_CACHE", it still attempt to download data from remote side : 

<img width="1697" height="813" alt="Image" src="https://github.com/user-attachments/assets/baa6cff1-a724-4710-a8c4-4805459deffb" />

Any suggestion will be appreciated!!

Comments:
You can download the dataset locally using [huggingface_hub.snapshot_download](https://huggingface.co/docs/huggingface_hub/v0.34.3/en/package_reference/file_download#huggingface_hub.snapshot_download) and then do

```python
dataset = load_dataset(local_directory_path)
```

> You can download the dataset locally using [huggingface_hub.snapshot_download](https://huggingface.co/docs/huggingface_hub/v0.34.3/en/package_reference/file_download#huggingface_hub.snapshot_download) and then do
> 
> dataset = load_dataset(local_directory_path)

It's good suggestion, but my server env is network restriction. It can not directly fetch data from huggingface. I spent lot of time to download and transfer it to the server.
So, I attempt to make load_dataset connect to my local dataset.  

Just Solved it few day before. Will post solution later...
also thanks folks quick reply..
Fix map() example in datasets documentation: define tokenizer before use
State: closed

## Problem
The current datasets.Dataset.map() example in the documentation demonstrates batched processing using a tokenizer object without defining or importing it. This causes a NameError when users copy and run the example as-is, breaking the expected seamless experience.

## Correction
This PR fixes the issue by explicitly importing and initializing the tokenizer using the Transformers library (AutoTokenizer.from_pretrained("bert-base-uncased")), making the example self-contained and runnable without errors.
This will help new users understand the workflow and apply the method correctly.

Closes #7703 



Comments:
Hi @lhoestq, just a gentle follow-up on this doc fix PR (#7704). Let me know if any changes are needed — happy to update.
Hope this improvement helps users run the example without confusion!

the modified file is the readme of the docs, not about map() specifically
[Docs] map() example uses undefined `tokenizer` — causes NameError
State: open

## Description

The current documentation example for `datasets.Dataset.map()` demonstrates batched processing but uses a `tokenizer` object without defining or importing it. This causes an error every time it's copied.

Here is the problematic line:

```python
# process a batch of examples
>>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This assumes the user has already set up a tokenizer, which contradicts the goal of having self-contained, copy-paste-friendly examples. 

## Problem
Users who copy and run the example as-is will encounter:

```python
NameError: name 'tokenizer' is not defined
```

This breaks the flow for users and violates HuggingFace's documentation principle that examples should "work as expected" when copied directly. 

## Proposal
Update the example to include the required tokenizer setup using the Transformers library, like so:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
ds_tokenized = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This will help new users understand the workflow and apply the method correctly.

## Note
This PR complements ongoing improvements like #7700, which clarifies multiprocessing in .map(). My change focuses on undefined tokenizer — causes NameError






Comments:
I've submitted PR #7704 which adds documentation to clarify the behavior of `map()` when returning `None`.
num_proc=0 behave like None, num_proc=1 uses one worker (not main process) and clarify num_proc documentation
State: closed

Fixes issue #7700

This PR makes num_proc=0 behave like None in Dataset.map(), disabling multiprocessing.
It improves UX by aligning with DataLoader(num_workers=0) behavior.
The num_proc docstring is also updated to clearly explain valid values and behavior.

@SunMarc


Comments:
I think we can support num_proc=0 and make it equivalent to `None` to make it simpler

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7702). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> I think we can support num_proc=0 and make it equivalent to `None` to make it simpler

Thank you @lhoestq for reviewing it. Please let me know if anything needs to be updated further.
Update fsspec max version to current release 2025.7.0
State: closed

Diffusers currently asks for a max fsspec version of `2025.3.0`. This change updates it to the current latest version. This change is mainly required to resolve conflicts with other packages in an environment. In my particular case, `aider-chat` which is a part of my environment installs `2025.5.1` which is incompatible with `datasets`.

Comments:
@lhoestq I ran the test suite locally and while some tests were failing those failures are present on the main branch too. Could you please review and trigger the CI?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7701). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Which release will this be available in ? I'm running into this issue with `datasets=3.6.0`
[doc] map.num_proc needs clarification
State: open

https://huggingface.co/docs/datasets/v4.0.0/en/package_reference/main_classes#datasets.Dataset.map.num_proc
```
num_proc (int, optional, defaults to None) — Max number of processes when generating cache. Already cached
shards are loaded sequentially.
```

for batch:
```
num_proc (int, optional, defaults to None): The number of processes to use for multiprocessing. If None, no 
multiprocessing is used. This can significantly speed up batching for large datasets.
```

So what happens to `map.num_proc` - is it the same behavior as `batch.num_proc` - so only if `num_proc=None` then no multiprocessing is used?

Let's update the doc to be unambiguous.

**bonus**: we could make all of these behave similarly to `DataLoader.num_workers` - where `num_workers==0` implies no multiprocessing. I think that's the most intuitive, IMHO. 0 workers - the main process has to do all the work. `None` could be the same as `0`.

context: debugging a failing `map`

Thank you!

Comments:
Broken link in documentation for "Create a video dataset"
State: open

The link to "the [WebDataset documentation](https://webdataset.github.io/webdataset)." is broken. 
https://huggingface.co/docs/datasets/main/en/video_dataset#webdataset 

<img width="2048" height="264" alt="Image" src="https://github.com/user-attachments/assets/975dd10c-aad8-42fc-9fbc-de0e2747a326" />

Comments:
The URL is ok but it seems the webdataset website is down. There seems to be a related issue here: https://github.com/webdataset/webdataset/issues/155

Feel free to ask the authors there for an update. Otherwise happy to witch the link to the mirror shared in that issue
NotImplementedError when using streaming=True in Google Colab environment
State: open

### Describe the bug

When attempting to load a large dataset (like tiiuae/falcon-refinedweb or allenai/c4) using streaming=True in a standard Google Colab notebook, the process fails with a NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet. This issue persists even after upgrading datasets and huggingface_hub and restarting the session.

### Steps to reproduce the bug

Open a new Google Colab notebook.

(Optional but recommended) Run !pip install --upgrade datasets huggingface_hub and restart the runtime.

Run the following code:

Python

from datasets import load_dataset

try:
    print("Attempting to load a stream...")
    streaming_dataset = load_dataset('tiiuae/falcon-refinedweb', streaming=True)
    print("Success!")
except Exception as e:
    print(e)

### Expected behavior

The load_dataset command should return a StreamingDataset object without raising an error, allowing iteration over the dataset.

Actual Behavior
The code fails and prints the following error traceback:

[PASTE THE FULL ERROR TRACEBACK HERE]
(Note: Copy the entire error message you received, from Traceback... to the final error line, and paste it in this section.)

### Environment info

Platform: Google Colab

datasets version: [Run !pip show datasets in Colab and paste the version here]

huggingface_hub version: [Run !pip show huggingface_hub and paste the version here]

Python version: [Run !python --version and paste the version here]

Comments:
Hi, @Aniket17200, try upgrading datasets using '!pip install -U datasets'. I hope this will resolve your issue.

Thank you @tanuj-rai, it's working great
load_dataset() in 4.0.0 returns different audio samples compared to earlier versions breaking reproducibility
State: closed

### Describe the bug

In datasets 4.0.0 release, `load_dataset()` returns different audio samples compared to earlier versions, this breaks integration tests that depend on consistent sample data across different environments (first and second envs specified below).



### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(24000))
sample= ds[0]["audio"]["array"]
print(sample)

# sample in 3.6.0 
[0.00231914 0.00245417 0.00187414 ... 0.00061956 0.00101157 0.00076325]


# sample in 4.0.0 
array([0.00238037, 0.00220794, 0.00198703, ..., 0.00057983, 0.00085863,
       0.00115309], dtype=float32)
```

### Expected behavior

The same dataset should load identical samples across versions to maintain reproducibility.

### Environment info

First env:
- datasets version: 3.6.0
- Platform: Windows-10-10.0.26100-SP0
- Python: 3.11.0

Second env:
- datasets  version: 4.0.0
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python: 3.11.13

Comments:
Hi ! This is because `datasets` now uses the FFmpeg-based library `torchcodec` instead of the libsndfile-based library `soundfile` to decode audio data. Those two have different decoding implementations

I’m all for torchcodec, good luck with the migration!
Support downloading specific splits in load_dataset
State: closed

This PR builds on #6832 by @mariosasko.

May close - #4101, #2538

Discussion - https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130

---
### Note - This PR is under work and frequent changes will be pushed.

Comments:
I’ve completed the following steps to continue the partial split download support (from PR #6832):

I did changes on top of what has been done by mario. Here are some of those changes: 
- Restored support for writing multiple split shards:

- In _prepare_split_single, we now correctly replace JJJJJ and SSSSS placeholders in the fpath for job/shard IDs before creating the writer.

- Added os.makedirs(os.path.dirname(path), exist_ok=True) after placeholder substitution to prevent FileNotFoundError.

- Applied the fix to both split writers:

       1] self._generate_examples version (used by most modules).

       2] self._generate_tables version (used by IterableDatasetBuilder).

- Confirmed 109/113 tests passing, meaning the general logic is working across the board.

What’s still failing
4 integration tests fail:

`test_load_hub_dataset_with_single_config_in_metadata`

`test_load_hub_dataset_with_two_config_in_metadata`

`test_load_hub_dataset_with_metadata_config_in_parallel`

`test_reload_old_cache_from_2_15`

All are due to FileNotFoundError from uncreated output paths, which I'm currently finalizing by ensuring os.makedirs() is correctly applied before every writer instantiation.

I will update about these fixes after running tests!

@lhoestq this was just an update

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7695). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Local DIR wasn't doing well, dk actually what happened, will PR again! Sorry :)
xopen: invalid start byte for streaming dataset with trust_remote_code=True
State: open

### Describe the bug

I am trying to load YODAS2 dataset with datasets==3.6.0
```
from datasets import load_dataset
next(iter(load_dataset('espnet/yodas2', name='ru000', split='train', streaming=True, trust_remote_code=True)))
```
And get `UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte`

The cause of the error is the following:

```
from datasets.utils.file_utils import xopen

filepath = 'https://huggingface.co/datasets/espnet/yodas2/resolve/c9674490249665d658f527e2684848377108d82c/data/ru000/text/00000000.json'
xopen(filepath, 'r').read()
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte
```

And the cause of this is the following:

```
import fsspec

fsspec.open(
    'hf://datasets/espnet/yodas2@c9674490249665d658f527e2684848377108d82c/data/ru000/text/00000000.json',
    mode='r',
    hf={'token': None, 'endpoint': 'https://huggingface.co'},
).open().read()
>>> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 1: invalid start byte
```

Is it true that streaming=True loading is not supported anymore for trust_remote_code=True, even with datasets==3.6.0? This breaks backward compatibility.

### Steps to reproduce the bug

```
from datasets import load_dataset
next(iter(load_dataset('espnet/yodas2', name='ru000', split='train', streaming=True)))
```

### Expected behavior

No errors expected

### Environment info

datasets==3.6.0, ubuntu 24.04

Comments:
Hi ! it would be cool to convert this dataset to Parquet. This will make it work for `datasets>=4.0`, enable the Dataset Viewer and make it more reliable to load/stream (currently it uses a loading script in python and those are known for having issues sometimes)

using `datasets==3.6.0`, here is the command to convert it and open a Pull Request:

```
datasets-cli convert_to_parquet espnet/yodas2 --trust_remote_code
```

Though it's likely that the `UnicodeDecodeError` comes from the loading script. If the script has a bug, it must be fixed to be able to convert the dataset without errors
HDF5 support
State: closed

This PR adds support for tabular HDF5 file(s) by converting each row to an Arrow table. It supports columns with the usual dtypes including up to 5-dimensional arrays as well as support for complex/compound types by using `Features(dict)`. All datasets within the HDF5 file should have rows on the first dimension (groups/subgroups are still allowed). Closes #3113.

Replaces #7625 which only supports a relatively small subset of HDF5.

Comments:
A few to-dos which I think can be left for future PRs (which I am happy to do/help with -- just this one is already huge 😄 ):
- [Enum types](https://docs.h5py.org/en/stable/special.html#enumerated-types)
- HDF5 [io](https://github.com/huggingface/datasets/tree/main/src/datasets/io)
- [dataset-viewer](https://github.com/huggingface/dataset-viewer) support (not sure if changes are needed with the way it is written now)

@lhoestq any interest in merging this? Let me know if I can do anything to make reviewing it easier!

Sorry for the delay, I'll review your PR soon :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7690). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review @lhoestq! Rebased on main and incorporated most of your suggestions.

I believe the only one left is the zero-dim handling with `table_cast`...

@lhoestq is 2c4bfba what you meant?

Awesome! Yes, I'm happy to help with the docs. Would appreciate any pointers, we can discuss in #7740.

It does look like there was a CI test failure, though it seems unrelated?
```
FAILED tests/test_dataset_dict.py::test_dummy_datasetdict_serialize_fs - ValueError: Protocol not known: mock
FAILED tests/test_arrow_dataset.py::test_dummy_dataset_serialize_fs - ValueError: Protocol not known: mock
```
Also, what do you think of the todos in https://github.com/huggingface/datasets/pull/7690#issuecomment-3105391677 ? In particular I think support in dataset-viewer would be nice.

Cool ! Yeah the failure is unrelated

Regarding the Viewer, it should work out of the box when it's updated with the next version of `datasets` :)
No module named "distributed"
State: open

### Describe the bug

hello, when I run the command "from datasets.distributed import split_dataset_by_node", I always met the bug "No module named 'datasets.distributed" in different version like 4.0.0, 2.21.0 and so on. How can I solve this?

### Steps to reproduce the bug

1. pip install datasets
2. from datasets.distributed import split_dataset_by_node

### Expected behavior

expecting the command "from datasets.distributed import split_dataset_by_node" can be ran successfully

### Environment info

python: 3.12

Comments:
The error ModuleNotFoundError: No module named 'datasets.distributed' means your installed datasets library is too old or incompatible with the version of Library you are using(in my case it was BEIR). The datasets.distributed module was removed in recent versions of the datasets library.

Downgrade datasets to version 2.14.6 : ! pip install datasets==2.14.6


this code does run in `datasets` 4.0:
```python
from datasets.distributed import split_dataset_by_node
```

make sure you have a python version that is recent enough (>=3.9) to be able to install `datasets` 4.0

I do think the problem is caused by the python version, because I do have python version 3.12.5
Datasets keeps rebuilding the dataset every time i call the python script
State: open

### Describe the bug

Every time it runs, somehow, samples increase.
This can cause a 12mb dataset to have other built versions of 400 mbs+

<img width="363" height="481" alt="Image" src="https://github.com/user-attachments/assets/766ce958-bd2b-41bc-b950-86710259bfdc" />

### Steps to reproduce the bug


`from datasets import load_dataset
s = load_dataset('~/.cache/huggingface/datasets/databricks___databricks-dolly-15k')['train']
`
1. A dataset needs to be available in the .cache folder
2. Run the code multiple times, and every time it runs, more versions are created


### Expected behavior

The number of samples increases every time the script runs

### Environment info

- `datasets` version: 3.6.0
- Platform: Windows-11-10.0.26100-SP0
- Python version: 3.13.3
- `huggingface_hub` version: 0.32.3
- PyArrow version: 20.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0


Comments:
here is the code to load the dataset form the cache:

```python
s = load_dataset('databricks/databricks-dolly-15k')['train']
```

if you pass the location of a local directory it will create a new cache based on that directory content
Convert to string when needed + faster .zstd
State: closed

for https://huggingface.co/datasets/allenai/olmo-mix-1124

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7683). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.