noamaanMulla-03/datasets-issues huggingface dataset:
fix: resolve base_path conflict between builder_kwargs and config_kwargs

## What does this PR fix?

This PR fixes a bug where passing `base_path` to `load_dataset_builder()` or `load_dataset()` causes a TypeError because both `builder_kwargs` and `config_kwargs` can contain the same key.

## Problem

When users call:
```python
from datasets import load_dataset
load_dataset("rotten_tomatoes", base_path="./sample_data")
```

They get:
```
TypeError: type object got multiple values for keyword argument "base_path"
```

This happens because `base_path` can exist in both `builder_kwargs` (from the dataset module) and `config_kwargs` (from user input), causing a conflict when both are unpacked.

## Solution

- Pop `base_path` from `builder_kwargs` before merging with `config_kwargs`
- Explicitly pass `base_path` to the builder constructor

## Testing

The fix should resolve the issue demonstrated in #4910:
```python
from datasets import load_dataset
load_dataset("rotten_tomatoes", base_path="./sample_data")  # Now works!
```

Fixes #4910
Speed up local 'get_data_patterns' by avoiding repeated recursive scans

This PR speeds up `get_data_patterns` for local paths.

### Problem

For `load_dataset("imagefolder", data_dir=...)`, `get_data_patterns` was repeatedly scanning the same local directory tree for many split patterns (`train`, `test`, etc.).  
With large folders, this became very slow. This has also been reported before in earlier performance discussions/issues.

### Change

In `get_data_patterns` (local paths only):

- scan files once (`resolve_pattern("**", ...)`)
- then match split patterns in memory

Remote paths keep the old behavior.

No API changes.

### My Env
- Mac mini (M4), 16 GB RAM
- Python 3.13
- Datasets version: 4.5.0

### Benchmarks

`imagefolder` with local `.jpg` files

`data_dir_only`:
- 10k: `4.35s -> 1.40s` (3.10x)
- 100k: `33.48s -> 9.77s` (3.43x)
- 300k: `160.20s -> 35.81s` (4.47x)
- 1M: `1877.70s -> 164.17s` (11.44x) :fire:

`explicit_data_files`:
- 10k: `0.75s -> 0.79s`
- 100k: `7.23s -> 7.29s`
- 300k: `25.44s -> 24.26s`
- 1M: `115.85s -> 112.66s`

As expected, the improvement is on `data_dir_only` (auto pattern detection path).
Memory usage did not show a consistent regression in these runs and stayed within normal run-to-run variance for this benchmark setup.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8014). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> lgtm ! can you run `make style` to fix the CI before we merge ?

done. All good, no warnings.
Allow saving multi-dimensional ndarray with dynamic shapes

### Feature request

I propose adding a dedicated feature to the datasets library that allows for the efficient storage and retrieval of multi-dimensional ndarray with dynamic shapes. Similar to how Image columns handle variable-sized images, this feature would provide a structured way to store array data where the dimensions are not fixed.

A possible implementation could be a new Array or Tensor feature type that stores the data in a structured format, for example, 

```python
{
  "shape": (5, 224, 224), 
  "dtype": "uint8", 
  "data": [...]
}
```

This would allow the datasets library to handle heterogeneous array sizes within a single column without requiring a fixed shape definition in the feature schema.

### Motivation

I am currently trying to upload data from astronomical telescopes, specifically FITS files, to the Hugging Face Hub. This type of data is very similar to images but often has more than three dimensions. For example, data from the SDSS project contains five channels (u, g, r, i, z), and the pixel values can exceed 255, making the Pillow based Image feature unsuitable.

The current datasets library requires a fixed shape to be defined in the feature schema for multi-dimensional arrays, which is a major roadblock. This prevents me from saving my data, as the dimensions of the arrays can vary across different FITS files.

https://github.com/huggingface/datasets/blob/985c9bee6bfc345787a8b9dd316e1d4f3b930503/src/datasets/features/features.py#L613-L614

A feature that supports dynamic shapes would be incredibly beneficial for the astronomy community and other fields dealing with similar high-dimensional, variable-sized data (e.g., medical imaging, scientific simulations).

### Your contribution

I am willing to create a PR to help implement this feature if the proposal is accepted.

Labels: name: enhancement

I agree this would be super valuable.

It looks like this was discussed a few years ago in https://github.com/huggingface/datasets/issues/5272#issuecomment-1550200824 but there were some issues. Those PRs are merged now and it looks like Arrow [officially supports](https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor) this so it's a good time to re-evaluate!

Happy to help with this, maybe we can think of adding a new type `Tensor` (instead of Array2D, 3D etc. which imply a fixed number of dims - we can keep them for backward compat anyways) that uses VariableShapeTensor (or FixedShapeTensor if the shape is provided maybe ? happy to discuss this)

An update here [VariableShapeTensor C++ implementation](https://github.com/apache/arrow/issues/38007) was merged in Arrow and the relatively trivial [Python wrapper](https://github.com/apache/arrow/pull/40354) will likely make it into the next release (April-ish).

Wow exciting ! thanks @rok
Use temp files in push_to_hub to save memory

write parquet data to temp files on disk prior to upload to save memory

this is enabled for for datasets loaded using streaming=True/False


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7979). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

And they are never deleted?

Just to be clear @lhoestq, by not deleting the files on successful shard push they will accumulate and eventually fill the disk. By accumulating files, stream processing is effectively broken. Why stream files if you end up with all of them on disk?

Is there no review process in place for PRs? Even an automated one would have gotten this.

arf what an oversight, thanks for reporting I'll do a patch release asap

4.6.1 is released with a fix !
Improve error message for deprecated dataset scripts with migration guidance

Summary
This PR improves the RuntimeError message raised when deprecated local dataset scripts are detected.
Previously, the error message:
    "Dataset scripts are no longer supported"
did not provide actionable migration guidance.
Improvements
- Clarifies the new architecture
- Provides explicit migration steps
- Includes link to official documentation
- Adds a focused unit test to validate error clarity
This should reduce confusion for contributors migrating legacy dataset scripts.


Hi team! I’m just checking in on this PR to see if there is any additional information or documentation I can provide to help with the review process. Thanks for your time!
Allow passing a multiprocessing context to functions that support `num_proc`

### Feature request

Allow specifying [a multiprocessing context](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) to functions that support `num_proc` or use multiprocessing pools. For example, the following could be done:

```python
dataset = dataset.map(_func, num_proc=2, mp_context=multiprocess.get_context("spawn"))
```

Or at least the multiprocessing start method ("fork", "spawn", "fork_server" or `None`):

```python
dataset = dataset.map(_func, num_proc=2, mp_start_method="spawn")
```

Another option could be passing the `pool` as an argument.

### Motivation

By default, `multiprocess` (the `multiprocessing`-fork library that this repo uses) uses the "fork" start method for multiprocessing pools (for the default context). It could be changed by using `set_start_method`. However, this conditions the multiprocessing start method from all processing in a Python program that uses the default context, because [you can't call that function more than once](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods:~:text=set_start_method()%20should%20not%20be%20used%20more%20than%20once%20in%20the%20program.). My proposal is to allow using a different multiprocessing context, not to condition the whole Python program.

One reason to change the start method is that "fork" (the default) makes child processes likely deadlock if thread pools were created before (and also this is not supported by POSIX). For example, this happens when using PyTorch because OpenMP threads are used for CPU intra-op parallelism, which is enabled by default (e.g., for context see [`torch.set_num_threads`](https://pytorch.org/docs/stable/generated/torch.set_num_threads.html)). This can also be fixed by setting `torch.set_num_threads(1)` (or similarly by other methods) but this conditions the whole Python program as it can only be set once to guarantee its behavior (similarly to). There are noticeable performance differences when setting this number to 1 even when using GPU(s). Using, e.g., a "spawn" start method would solve this issue.

For more context, see:

* https://discuss.huggingface.co/t/dataset-map-stuck-with-torch-set-num-threads-set-to-2-or-larger/37984
* https://discuss.huggingface.co/t/using-num-proc-1-in-dataset-map-hangs/44310

### Your contribution

I'd be happy to review a PR that makes such a change. And if you really don't have the bandwidth for it, I'd consider creating one.

Labels: name: enhancement

Are there any updates on this? This is still a problem when using Datasets inside Ray processes.

Any updates on this? Getting
```
DeprecationWarning: This process (pid=269) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()
```
when using `num_proc` on python 3.12. I believe the default will be set to spawn for python 3.14, but was wondering if any fixes will be made for older versions of python?
[CUDA Tensors Not working in ~v4.5.0] set_format(type="torch", device="cuda") returns cpu

### Describe the bug

The problem is that when calling:
```ds.set_format(type="torch", columns = ["input", "labels"], device="cuda")```
The device type of the individual datapoints is now: `cpu` as opposed to `cuda:0`.

With `v4.0.0` it still works. With `v4.5.0` it doesn't work anymore.


Related Issue:
https://github.com/huggingface/datasets/issues/1762

### Steps to reproduce the bug

Steps to reproduce the bug:

```
ds.set_format(type="torch", columns = ["input", "labels"], device="cuda")
print(ds["train"][0]["input"].device) #outputs cpu
```

The above should be `cuda:0`.

### Expected behavior

`set_format` should be able to move data to GPU.

### Environment info

datasets==4.5.0


@lhoestq tagging you here as you were on the previous issue, hope that's fine. 

Thanks for reporting, let me take a look

I reverted this change which caused the issue #7961 , I'll do a new release soon but in the meantime feel free to install `datasets` from source

@lhoestq thanks a lot. I am actually checking older versions of datasets and it seems that it doesn't work with 4.2.0 as well.

(hopefully that's relevant).

This has been fixed in the latest version :)
Custom `dl_manager` in `load_dataset`

### Feature request
https://github.com/huggingface/datasets/blob/4.0.0/src/datasets/load.py#L1411-L1418

```

def load_dataset(
    ...
    dl_manager: Optional[DownloadManager] = None,  # add this new argument
    **config_kwargs,
) -> Union[DatasetDict, Dataset, IterableDatasetDict, IterableDataset]:

    ...

    # Create a dataset builder
    builder_instance = load_dataset_builder(
        path=path,
        name=name,
        data_dir=data_dir,
        data_files=data_files,
        cache_dir=cache_dir,
        features=features,
        download_config=download_config,
        download_mode=download_mode,
        revision=revision,
        token=token,
        storage_options=storage_options,
        **config_kwargs,
    )

    # Return iterable dataset in case of streaming
    if streaming:
        return builder_instance.as_streaming_dataset(split=split)

    # Note: This is the revised part
    if dl_manager is None:
        if download_config is None:
            download_config = DownloadConfig(
                cache_dir=builder_instance._cache_downloaded_dir,
                force_download=download_mode == DownloadMode.FORCE_REDOWNLOAD,
                force_extract=download_mode == DownloadMode.FORCE_REDOWNLOAD,
                use_etag=False,
                num_proc=num_proc,
                token=builder_instance.token,
                storage_options=builder_instance.storage_options,
            )  # We don't use etag for data files to speed up the process

        dl_manager = DownloadManager(
            dataset_name=builder_instance.dataset_name,
            download_config=download_config,
            data_dir=builder_instance.config.data_dir,
            record_checksums=(
                builder_instance._record_infos or verification_mode == VerificationMode.ALL_CHECKS
            ),
        )

    # Download and prepare data
    builder_instance.download_and_prepare(
        download_config=download_config,
        download_mode=download_mode,
        verification_mode=verification_mode,
        dl_manager=dl_manager,  # pass the new argument
        num_proc=num_proc,
        storage_options=storage_options,
    )

    ...

```

### Motivation

In my case, I'm hoping to deal with the cache files downloading manually (not using hash filenames and save to another location, or using potential existing local files).

### Your contribution

It's already implemented above. If maintainers think this should be considered, I'll open a PR.

Labels: name: enhancement

Any feedback from maintainers on this?
Make `with_format` correctly indicate that a `Dataset` is compatible with PyTorch's `Dataset` class

### Feature request

Currently `datasets` does not correctly indicate to the Python type-checker (e.g. `pyright` / `Pylance`) that the output of `with_format` is compatible with PyTorch's `Dataloader` since it does not indicate that the HuggingFace `Dataset` is compatible with the PyTorch `Dataset` class. It would be great if we could get the typing to work nicely.

### Motivation

To avoid casting types in our Python code.

### Your contribution

I would be happy to contribute a PR if this is something that may be accepted and could work with the current approach.
This doesn't have to be for just PyTorch, but I imagine that the same thing would be useful for `tensorflow` and such, but we only have a need for PyTorch at this stage.

Labels: name: enhancement

Does the torch `DataLoader` really require the dataset to be a subclass of `torch.utils.data.Dataset` ? Or is there a simpler type we could use ?

PS: also note that a dataset without `with_format()` can also be used in a torch `DataLoader` . Calling `with_format("torch")` simply makes the output of the dataset torch Tensors in an efficient way.
perf: use deque for async map task/index queues

## Problem

`arrow_dataset.py` and `iterable_dataset.py` drain `indices` and `tasks` lists front-to-back via `.pop(0)` during parallel async map operations. With `MAX_NUM_RUNNING_ASYNC_MAP_FUNCTIONS_IN_PARALLEL` items in flight, each `.pop(0)` is **O(n)** making the drain loop **O(n²)**.

## Solution

Switch both `indices` and `tasks` to `collections.deque` with `.popleft()` for **O(1)** front removal.

## Changes

- `src/datasets/arrow_dataset.py`: Import `deque`, type `tasks` and `indices` as deque, use `.popleft()`
- `src/datasets/iterable_dataset.py`: Import `deque`, type `tasks`, `indices`, and `_owned_loops_and_tasks` entry as deque, use `.popleft()`

## Testing

- Syntax verified via `ast.parse()` on both files


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8033). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Friendly ping — CI is green and this is ready for review. Happy to address any feedback. Thanks!
Representing dictionaries (json) objects as features

In the process of adding a new dataset to the hub, I stumbled upon the inability to represent dictionaries that contain different key names, unknown in advance (and may differ between samples), original asked in the [forum](https://discuss.huggingface.co/t/representing-nested-dictionary-with-different-keys/16442).

For instance:

```
sample1 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
}}
sample2 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
}}
sample3 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
  "d": {"id": 3, "text": "text4"},
}}
```

the `nps` field cannot be represented as a Feature while maintaining its original structure.
@lhoestq suggested to add JSON as a new feature type, which will solve this problem.


It seems like an alternative solution would be to change the original data format, which isn't an optimal solution in my case. Moreover, JSON is a common structure, that will likely to be useful in future datasets as well.

Labels: name: enhancement

This is indeed a problem for VLMs in 2025 too! Where we expect the inputs to be of the format:
```
content = [{
"type": "text",
"text": <text>
}, {
"type": "image",
"image": <image>
}]
```
CC: @lhoestq 

The community has also been using this format instead, with the images on the side: https://huggingface.co/docs/trl/en/dataset_formats#vision-datasets

```python
dataset = Dataset.from_dict({
    "prompt": [
        [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What color is the sky in the image?"}]}],
        [{"role": "user", "content": [{"type": "text", "text": "What is the capital of France?"}]}],
    ],
    "completion": [
        [{"role": "assistant", "content": [{"type": "text", "text": "It is blue."}]}],
        [{"role": "assistant", "content": [{"type": "text", "text": "Paris."}]}],
    ],
    "images": [
        [PIL.Image.open("path/to/sky_image1.png")],
        [],
    ],
})
```

like this one: https://huggingface.co/datasets/trl-lib/rlaif-v

Hi ! with #8027 you will be able to do 
```python
from datasets import Dataset, Features, Json

features = Features({"nps": Json()})
ds = Dataset.from_list([sample1, sample2, sample3], features=features)
# OR
# ds = Dataset.from_list([sample1, sample2, sample3], on_mixed_types="use_json")

print(ds.features)
# {'nps': Json()}

for nps in ds["nps"]:
    print(nps)
# {'a': {'id': 0, 'text': 'text1'}, 'b': {'id': 1, 'text': 'text2'}}
# {'a': {'id': 0, 'text': 'text1'}, 'b': {'id': 1, 'text': 'text2'}, 'c': {'id': 2, 'text': 'text3'}}
# {'a': {'id': 0, 'text': 'text1'}, 'b': {'id': 1, 'text': 'text2'}, 'c': {'id': 2, 'text': 'text3'}, 'd': {'id': 3, 'text': 'text4'}}
```
Add `Json()` type

`Json()` type is needed when some fields don't have fixed types, e.g. "tools" and "tool_calls" types in conversation + tool calling datasets like [dataclaw](https://huggingface.co/datasets?other=dataclaw) datasets. Cc @peteromallet and @woctordho @Nanbeige for viz

Examples of supported tool-calling / dataclaw datasets:
```python
ds = load_dataset("peteromallet/dataclaw-peteromallet")  #  happens to not need Json() since tool types are fixed
ds = load_dataset("woctordho/dataclaw")
ds = load_dataset("Nanbeige/ToolMind")
```

The `Json()` type is auto-applied when loading JSON files when mixed types are found. The `Json()`type is set to the list containing objects with mixed types, to end up with a `List(Json())` type for "tools" and "tool_calls" columns.

It is also possible to define a Dataset with a `Json()` type like this:

```python
>>> from datasets import load_dataset, Dataset, Features, Json, List
>>> example = {"a": [{"key": 0}, {"another-key": "another-type"}]}
>>> features = Features({"a": List(Json())})
>>> ds = Dataset.from_list([example], features=features)
>>> ds[0]
{'a': [{'key': 0}, {'another-key': 'another-type'}]}
```

close https://github.com/huggingface/datasets/issues/7869
close https://github.com/huggingface/datasets/issues/4120
close https://github.com/huggingface/datasets/issues/5827
close https://github.com/huggingface/datasets/issues/7418
related to https://github.com/huggingface/datasets/issues/7092
related to https://github.com/huggingface/datasets/issues/6162
related to https://github.com/huggingface/datasets/issues/2799
related to https://huggingface.co/datasets/PatoFlamejanteTV/LocalLLaMA/discussions/1


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8027). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
load json dataset interrupt when dtype cast problem occured

### Describe the bug

i have a json like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}, 
    ....
]
，which have several problematic rows data like row 2, then i load it with datasets.load_dataset('json', data_files=['xx.json'], split='train'), it will report like this:
Generating train split: 0 examples [00:00, ? examples/s]Failed to read file 'C:\Users\gawinjunwu\Downloads\test\data\a.json' with error <class 'pyarrow.lib.ArrowInvalid'>: Could not convert '2' with type str: tried to convert to int64
Traceback (most recent call last):
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1858, in _prepare_split_single
    for _, table in generator:
  File "D:\Python3.9\lib\site-packages\datasets\packaged_modules\json\json.py", line 146, in _generate_tables
    raise ValueError(f"Not able to read records in the JSON file at {file}.") from None
ValueError: Not able to read records in the JSON file at C:\Users\gawinjunwu\Downloads\test\data\a.json.     

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\gawinjunwu\Downloads\test\scripts\a.py", line 4, in <module>
    ds = load_dataset('json', data_dir='data', split='train')
  File "D:\Python3.9\lib\site-packages\datasets\load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1746, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1891, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset.
Could datasets skip those problematic data row? 

### Steps to reproduce the bug

prepare a json file like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}
]
then use datasets.load_dataset('json', dir_files=['xxx.json']) to load the json file

### Expected behavior

skip the problematic data row and load row1 and row3

### Environment info

python3.9


Indeed the JSON dataset builder raises an error when it encounters an unexpected type.

There's an old PR open to add away to ignore such elements though, if it can help: https://github.com/huggingface/datasets/pull/2838

Hi ! with #8027 you will be able to do 
```python
from datasets import load_dataset

ds = load_dataset("json", data_files="xxx.json", split="train")

print(ds.features)
# {'id': Value('string'), 'name': Json()}

print(list(ds["name"]))
# [1, 'Nan', 3]
```