noamaanMulla-03/datasets-issues huggingface dataset:
No module named "distributed"

### Describe the bug

hello, when I run the command "from datasets.distributed import split_dataset_by_node", I always met the bug "No module named 'datasets.distributed" in different version like 4.0.0, 2.21.0 and so on. How can I solve this?

### Steps to reproduce the bug

1. pip install datasets
2. from datasets.distributed import split_dataset_by_node

### Expected behavior

expecting the command "from datasets.distributed import split_dataset_by_node" can be ran successfully

### Environment info

python: 3.12


The error ModuleNotFoundError: No module named 'datasets.distributed' means your installed datasets library is too old or incompatible with the version of Library you are using(in my case it was BEIR). The datasets.distributed module was removed in recent versions of the datasets library.

Downgrade datasets to version 2.14.6 : ! pip install datasets==2.14.6


this code does run in `datasets` 4.0:
```python
from datasets.distributed import split_dataset_by_node
```

make sure you have a python version that is recent enough (>=3.9) to be able to install `datasets` 4.0

I do think the problem is caused by the python version, because I do have python version 3.12.5
Datasets keeps rebuilding the dataset every time i call the python script

### Describe the bug

Every time it runs, somehow, samples increase.
This can cause a 12mb dataset to have other built versions of 400 mbs+

<img width="363" height="481" alt="Image" src="https://github.com/user-attachments/assets/766ce958-bd2b-41bc-b950-86710259bfdc" />

### Steps to reproduce the bug


`from datasets import load_dataset
s = load_dataset('~/.cache/huggingface/datasets/databricks___databricks-dolly-15k')['train']
`
1. A dataset needs to be available in the .cache folder
2. Run the code multiple times, and every time it runs, more versions are created


### Expected behavior

The number of samples increases every time the script runs

### Environment info

- `datasets` version: 3.6.0
- Platform: Windows-11-10.0.26100-SP0
- Python version: 3.13.3
- `huggingface_hub` version: 0.32.3
- PyArrow version: 20.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0



here is the code to load the dataset form the cache:

```python
s = load_dataset('databricks/databricks-dolly-15k')['train']
```

if you pass the location of a local directory it will create a new cache based on that directory content
[doc] map.num_proc needs clarification

https://huggingface.co/docs/datasets/v4.0.0/en/package_reference/main_classes#datasets.Dataset.map.num_proc
```
num_proc (int, optional, defaults to None) — Max number of processes when generating cache. Already cached
shards are loaded sequentially.
```

for batch:
```
num_proc (int, optional, defaults to None): The number of processes to use for multiprocessing. If None, no 
multiprocessing is used. This can significantly speed up batching for large datasets.
```

So what happens to `map.num_proc` - is it the same behavior as `batch.num_proc` - so only if `num_proc=None` then no multiprocessing is used?

Let's update the doc to be unambiguous.

**bonus**: we could make all of these behave similarly to `DataLoader.num_workers` - where `num_workers==0` implies no multiprocessing. I think that's the most intuitive, IMHO. 0 workers - the main process has to do all the work. `None` could be the same as `0`.

context: debugging a failing `map`

Thank you!
[Docs] map() example uses undefined `tokenizer` — causes NameError

## Description

The current documentation example for `datasets.Dataset.map()` demonstrates batched processing but uses a `tokenizer` object without defining or importing it. This causes an error every time it's copied.

Here is the problematic line:

```python
# process a batch of examples
>>> ds = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This assumes the user has already set up a tokenizer, which contradicts the goal of having self-contained, copy-paste-friendly examples. 

## Problem
Users who copy and run the example as-is will encounter:

```python
NameError: name 'tokenizer' is not defined
```

This breaks the flow for users and violates HuggingFace's documentation principle that examples should "work as expected" when copied directly. 

## Proposal
Update the example to include the required tokenizer setup using the Transformers library, like so:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
ds_tokenized = ds.map(lambda example: tokenizer(example["text"]), batched=True)
```
This will help new users understand the workflow and apply the method correctly.

## Note
This PR complements ongoing improvements like #7700, which clarifies multiprocessing in .map(). My change focuses on undefined tokenizer — causes NameError







I've submitted PR #7704 which adds documentation to clarify the behavior of `map()` when returning `None`.
Row indexing a dataset with numpy integers

### Feature request

Allow indexing datasets with a scalar numpy integer type.

### Motivation

Indexing a dataset with a scalar numpy.int* object raises a TypeError. This is due to the test in `datasets/formatting/formatting.py:key_to_query_type`

``` python
def key_to_query_type(key: Union[int, slice, range, str, Iterable]) -> str:
    if isinstance(key, int):
        return "row"
    elif isinstance(key, str):
        return "column"
    elif isinstance(key, (slice, range, Iterable)):
        return "batch"
    _raise_bad_key_type(key)
```

In the row case, it checks if key is an int, which returns false when key is integer like but not a builtin python integer type. This is counterintuitive because a numpy array of np.int64s can be used for the batch case.

For example:

``` python
import numpy as np

import datasets

dataset = datasets.Dataset.from_dict({"a": [1, 2, 3, 4], "b": [5, 6, 7, 8]})

# Regular indexing
dataset[0]
dataset[:2]

# Indexing with numpy data types (expect same results)
idx = np.asarray([0, 1])
dataset[idx]  # Succeeds when using an array of np.int64 values
dataset[idx[0]]  # Fails with TypeError when using scalar np.int64
```

For the user, this can be solved by wrapping `idx[0]` in `int` but the test could also be changed in `key_to_query_type` to accept a less strict definition of int.

``` diff
+import numbers
+
def key_to_query_type(key: Union[int, slice, range, str, Iterable]) -> str:
+   if isinstance(key, numbers.Integral):
-   if isinstance(key, int):
        return "row"
    elif isinstance(key, str):
        return "column"
    elif isinstance(key, (slice, range, Iterable)):
        return "batch"
    _raise_bad_key_type(key)
```

Looking at how others do it, pandas has an `is_integer` definition that it checks which uses `is_integer_object` defined in `pandas/_libs/utils.pxd`:

``` cython
cdef inline bint is_integer_object(object obj) noexcept:
    """
    Cython equivalent of

    `isinstance(val, (int, np.integer)) and not isinstance(val, (bool, np.timedelta64))`

    Parameters
    ----------
    val : object

    Returns
    -------
    is_integer : bool

    Notes
    -----
    This counts np.timedelta64 objects as integers.
    """
    return (not PyBool_Check(obj) and isinstance(obj, (int, cnp.integer))
            and not is_timedelta64_object(obj))
```

This would be less flexible as it explicitly checks for numpy integer, but worth noting that they had the need to ensure the key is not a bool.

### Your contribution

I can submit a pull request with the above changes after checking that indexing succeeds with the numpy integer type. Or if there is a different integer check that would be preferred I could add that.

If there is a reason not to want this behavior that is fine too.


Labels: name: enhancement

Would be cool to be consistent when it comes to indexing with numpy objects, if we do accept numpy arrays we should indeed accept numpy integers. Your idea sounds reasonable, I'd also be in favor of adding a simple test as well
Caching shuffles by np.random.Generator results in unintiutive behavior

### Describe the bug

Create a dataset. Save it to disk. Load from disk. Shuffle, usning a `np.random.Generator`. Iterate. Shuffle again. Iterate. The iterates are different since the supplied np.random.Generator has progressed between the shuffles.

Load dataset from disk again. Shuffle and Iterate. See same result as before. Shuffle and iterate, and this time it does not have the same shuffling as ion previous run.

The motivation is I have a deep learning loop with 

```
for epoch in range(10):
    for batch in dataset.shuffle(generator=generator).iter(batch_size=32):
        .... # do stuff
```
where I want a new shuffling at every epoch. Instead I get the same shuffling.

### Steps to reproduce the bug

Run the code below two times. 

```python
import datasets
import numpy as np

generator = np.random.default_rng(0)
ds = datasets.Dataset.from_dict(mapping={"X":range(1000)})
ds.save_to_disk("tmp")
print("First loop: ", end="")
for _ in range(10):
    print(next(ds.shuffle(generator=generator).iter(batch_size=1))['X'], end=", ")
print("")

print("Second loop: ", end="")
ds = datasets.Dataset.load_from_disk("tmp")
for _ in range(10):
    print(next(ds.shuffle(generator=generator).iter(batch_size=1))['X'], end=", ")
print("")
```

The output is:

```
$ python main.py 
Saving the dataset (1/1 shards): 100%|███████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 495019.95 examples/s]
First loop: 459, 739, 72, 943, 241, 181, 845, 830, 896, 334, 
Second loop: 741, 847, 944, 795, 483, 842, 717, 865, 231, 840,
$ python main.py 
Saving the dataset (1/1 shards): 100%|████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 22243.40 examples/s]
First loop: 459, 739, 72, 943, 241, 181, 845, 830, 896, 334, 
Second loop: 741, 741, 741, 741, 741, 741, 741, 741, 741, 741, 
```

The second loop, on the second run, only spits out "741, 741, 741...." which is *not* the desired output

### Expected behavior

I want the dataset to shuffle at every epoch since I provide it with a generator for shuffling. 

### Environment info

Datasets version 2.21.0
Ubuntu linux.


I first thought this was a mistake of mine, and also posted on stack overflow. https://stackoverflow.com/questions/78913797/iterating-a-huggingface-dataset-from-disk-using-generator-seems-broken-how-to-d 

It seems to me the issue is the caching step in 

https://github.com/huggingface/datasets/blob/be5cff059a2a5b89d7a97bc04739c4919ab8089f/src/datasets/arrow_dataset.py#L4306-L4316

because the shuffle happens after checking the cache, the rng state won't advance if the cache is used. This is VERY confusing. Also not documented.

My proposal is that you remove the API for using a Generator, and only keep the seed-based API since that is functional and cache-compatible.

Second that, its very confusing.
Caching map result of DatasetDict.

Hi!

I'm currenty using the map function to tokenize a somewhat large dataset, so I need to use the cache to save ~25 mins.

Changing num_proc incduces the recomputation of the map, I'm not sure why and if this is excepted behavior? 

here it says, that cached files are loaded sequentially:

https://github.com/huggingface/datasets/blob/bb2664cf540d5ce4b066365e7c8b26e7f1ca4743/src/datasets/arrow_dataset.py#L3005-L3006

it seems like I can pass in a fingerprint, and load it directly:

https://github.com/huggingface/datasets/blob/bb2664cf540d5ce4b066365e7c8b26e7f1ca4743/src/datasets/arrow_dataset.py#L3108-L3125

**Environment Setup:**

- Python 3.11.9
- datasets  2.19.1 conda-forge
- Linux 6.1.83-1.el9.elrepo.x86_64

**MRE**
```python
fixed raw_datasets
fixed tokenize_function

tokenized_datasets = raw_datasets.map(
                        tokenize_function,
                        batched=True,
                        num_proc=9,
                        remove_columns=['text'],
                        load_from_cache_file= True,
                        desc="Running tokenizer on dataset line_by_line",
                    )


tokenized_datasets = raw_datasets.map(
                        tokenize_function,
                        batched=True,
                        num_proc=5,
                        remove_columns=['text'],
                        load_from_cache_file= True,
                        desc="Running tokenizer on dataset line_by_line",
                    )
```


Has to do with sharding as far as I am aware, num_proc splits the dataset in shards and each process picks up a shard.

@dimidagd why wouldn't you redistribute the processes (e.g. param_num_process to sequentially fill fingerprint_num_process) to make things work out? 

Perhaps I'm missing something important.

I assume data has to be split between the processes somehow, and the decision to split it to num_processes shards is very convenient :). As I understand, the idea behind individual shard caching is being able to recover part of the processing if one of the processes fails. It would make sense to do a hierarchical cache checking though, where irregardless of the amount of processes, there is a check on whether the entire .map result has been cached or not (basically num_proc=1). Downside is more disk space used to cache both the shards and the entire dataset.
Support downloading specific splits in load_dataset

This PR builds on #6832 by @mariosasko.

May close - #4101, #2538

Discussion - https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130

---
### Note - This PR is under work and frequent changes will be pushed.


I’ve completed the following steps to continue the partial split download support (from PR #6832):

I did changes on top of what has been done by mario. Here are some of those changes: 
- Restored support for writing multiple split shards:

- In _prepare_split_single, we now correctly replace JJJJJ and SSSSS placeholders in the fpath for job/shard IDs before creating the writer.

- Added os.makedirs(os.path.dirname(path), exist_ok=True) after placeholder substitution to prevent FileNotFoundError.

- Applied the fix to both split writers:

       1] self._generate_examples version (used by most modules).

       2] self._generate_tables version (used by IterableDatasetBuilder).

- Confirmed 109/113 tests passing, meaning the general logic is working across the board.

What’s still failing
4 integration tests fail:

`test_load_hub_dataset_with_single_config_in_metadata`

`test_load_hub_dataset_with_two_config_in_metadata`

`test_load_hub_dataset_with_metadata_config_in_parallel`

`test_reload_old_cache_from_2_15`

All are due to FileNotFoundError from uncreated output paths, which I'm currently finalizing by ensuring os.makedirs() is correctly applied before every writer instantiation.

I will update about these fixes after running tests!

@lhoestq this was just an update

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7695). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Local DIR wasn't doing well, dk actually what happened, will PR again! Sorry :)
feat(map): reuse unchanged columns when input_columns specified to reduce disk usage (#6013)

## Summary

This PR addresses [#6013](https://github.com/huggingface/datasets/issues/6013) by reusing unchanged columns from the original dataset in the `map()` method when `input_columns` is specified.

## What’s Implemented

- Injected logic at the end of `Dataset.map()` to:
  - Identify untouched columns not in `input_columns` or `remove_columns`
  - Select those columns from the original dataset
  - Concatenate them with the transformed result using `pyarrow.concat_tables`

## Example Behavior

```python
ds = Dataset.from_dict({"a": [1, 2], "b": [3, 4]})
ds2 = ds.map(lambda x: {"c": x["a"] + 10}, input_columns=["a"], remove_columns=["a"])
print(ds2.column_names)  # Output: ['b', 'c']
````

Column `b` is reused from the original dataset.

## Notes

* This keeps disk usage and caching minimal by avoiding full dataset duplication.
* Only triggered when `input_columns` is set.

---

cc @lhoestq @mariosasko for review 🙂
Support downloading specific splits in `load_dataset`

This PR builds on https://github.com/huggingface/datasets/pull/6639 to support downloading only the specified splits in `load_dataset`. For this to work, a builder's `_split_generators` need to be able to accept the requested splits (as a list) via a `splits` argument to avoid processing the non-requested ones. Also, the builder has to define a `_available_splits` method that lists all the possible `splits` values.

Close https://github.com/huggingface/datasets/issues/4101, close https://github.com/huggingface/datasets/issues/2538 (I'm probably missing some)

Should also make it possible to address https://github.com/huggingface/datasets/issues/6793


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6832). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Friendly ping on this! This feature would be really helpful and useful to me (and likely others with limited download speed and storage space!). Thanks so much!

No one is working on this atm afaik :/

No worries! I've patched the ImageNet dataset in: <https://huggingface.co/datasets/ILSVRC/imagenet-1k/blob/refs%2Fpr%2F20/imagenet-1k.py> 

Together with:

```python
dataset = load_dataset(
        "ILSVRC/imagenet-1k",
        split="validation",
        data_files={"val": "data/val_images.tar.gz"},
        revision="refs/pr/20",
        trust_remote_code=True,
        download_config=DownloadConfig(resume_download=True),
        verification_mode=VerificationMode.NO_CHECKS,
    )
```

It only downloads the validation set this way (NO_CHECKS is a bit annoying because I'd rather have md5 checks, but I guess I can't have everything) ^^' The patch is not perfect, but it does the job.



#self assign 
https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130
load_dataset() in 4.0.0 returns different audio samples compared to earlier versions breaking reproducibility

### Describe the bug

In datasets 4.0.0 release, `load_dataset()` returns different audio samples compared to earlier versions, this breaks integration tests that depend on consistent sample data across different environments (first and second envs specified below).



### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(24000))
sample= ds[0]["audio"]["array"]
print(sample)

# sample in 3.6.0 
[0.00231914 0.00245417 0.00187414 ... 0.00061956 0.00101157 0.00076325]


# sample in 4.0.0 
array([0.00238037, 0.00220794, 0.00198703, ..., 0.00057983, 0.00085863,
       0.00115309], dtype=float32)
```

### Expected behavior

The same dataset should load identical samples across versions to maintain reproducibility.

### Environment info

First env:
- datasets version: 3.6.0
- Platform: Windows-10-10.0.26100-SP0
- Python: 3.11.0

Second env:
- datasets  version: 4.0.0
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python: 3.11.13


Hi ! This is because `datasets` now uses the FFmpeg-based library `torchcodec` instead of the libsndfile-based library `soundfile` to decode audio data. Those two have different decoding implementations

I’m all for torchcodec, good luck with the migration!
Concurrent push_to_hub

Retry the step that (download + update + upload) the README.md using `create_commit(..., parent_commit=...)` if there was a commit in the meantime. This should enable concurrent `push_to_hub()` since it won't overwrite the README.md metadata anymore.

Note: we fixed an issue server side to make this work:

<details>

DO NOT MERGE FOR NOW since it seems there is one bug that prevents this logic from working:

I'm using parent_commit to enable concurrent push_to_hub() in datasets for a retry mechanism, but for some reason I always run into a weird situation.
Sometimes create_commit(.., parent_commit=...) returns error 500 but the commit did happen on the Hub side without respecting parent_commit

e.g. request id
```
huggingface_hub.errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/lhoestq/tmp/commit/main (Request ID: Root=1-6888d8af-2ce517bc60c69cb378b51526;d1b17993-c5d0-4ccd-9926-060c45f9ed61)
```

fix coming in [internal](https://github.com/huggingface-internal/moon-landing/pull/14617)

</details>

close https://github.com/huggingface/datasets/issues/7600


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7708). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`push_to_hub` is not concurrency safe (dataset schema corruption)

### Describe the bug

Concurrent processes modifying and pushing a dataset can overwrite each others' dataset card, leaving the dataset unusable.

Consider this scenario:
- we have an Arrow dataset
- there are `N` configs of the dataset
- there are `N` independent processes operating on each of the individual configs (e.g. adding a column, `new_col`)
- each process calls `push_to_hub` on their particular config when they're done processing
- all calls to `push_to_hub` succeed
- the `README.md` now has some configs with `new_col` added and some with `new_col` missing

Any attempt to load a config (using `load_dataset`) where `new_col` is missing will fail because of a schema mismatch between `README.md` and the Arrow files. Fixing the dataset requires updating `README.md` by hand with the correct schema for the affected config. In effect, `push_to_hub` is doing a `git push --force` (I found this behavior quite surprising).

We have hit this issue every time we run processing jobs over our datasets and have to fix corrupted schemas by hand.

Reading through the code, it seems that specifying a [`parent_commit`](https://github.com/huggingface/huggingface_hub/blob/v0.32.4/src/huggingface_hub/hf_api.py#L4587) hash around here https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L5794 would get us to a normal, non-forced git push, and avoid schema corruption. I'm not familiar enough with the code to know how to determine the commit hash from which the in-memory dataset card was loaded.

### Steps to reproduce the bug

See above.

### Expected behavior

Concurrent edits to disjoint configs of a dataset should never corrupt the dataset schema.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.15.0-118-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- `huggingface_hub` version: 0.30.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.2
- `fsspec` version: 2023.9.0


@lhoestq can you please take a look? I've submitted a PR that fixes this issue. Thanks.

Thanks for the ping ! As I said in https://github.com/huggingface/datasets/pull/7605 there is maybe a more general approach using retries :)

Dropping this due to inactivity; we've implemented push_to_hub outside of HF datasets that's concurrency safe. Feel free to use the code I provided as a starting point if there's still interest in addressing this issue.

Exploring another fix here: https://github.com/huggingface/datasets/issues/7600
Concurrent IterableDataset push_to_hub

Same as https://github.com/huggingface/datasets/pull/7708 but for `IterableDataset`


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7710). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.