noamaanMulla-03/datasets-issues huggingface dataset:
Refactor `Dataset.map` to reuse cache files mapped with different `num_proc`

Fixes #7433

This refactor unifies `num_proc is None or num_proc == 1` and `num_proc > 1`; instead of handling them completely separately where one uses a list of kwargs and shards and the other just uses a single set of kwargs and `self`, by wrapping the `num_proc == 1` case in a list and making the difference just whether or not you use a pool, you set up either case to be able to load each other's cache files just by changing `num_shards`; `num_proc == 1` can sequentially load the shards of a dataset mapped `num_shards > 1` and map any missing shards

Other than the structural refactor, the main contribution of this PR is `existing_cache_file_map`, which uses a regex of `cache_file_name` and `suffix_template` to find existing cache files, grouped by their `num_shards`; using this data structure, we can reset `num_shards` to an existing set of cache files, and load them accordingly


@lhoestq please let me know what you think about this.

It looks like I can't change the merge target to #7435, so it will look like there is a bunch of extra stuff until #7435 is in main.

@lhoestq Thanks so much for reviewing #7435! Now that that's merged, I think this PR is ready!! Can you kick off CI when you get the chance?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7434). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Do you mind kicking off CI again?

The change I made to support windows paths in 637c1600fe7dd601eff571fda446937bd96c5c84 ended up breaking causing these tests in [tests/test_data_files.py](https://github.com/huggingface/datasets/actions/runs/13858546629/job/38781008643#step:10:6991). When I removed `glob_pattern_to_regex` in 583c28e7560b9d6db2e13048731f41ec8fa11361, none of the tests failed. So I'm thinking the `unicode_escape` may be handling the what `glob_pattern_to_regex` was doing.


@lhoestq will you have a chance to review this today?

Any update?

> LGTM and sorry for the delay !
> 
> note that the CI failures are unrelated to this PR :)

Great job!

great job to @ringohoffman ! ;)
Improved type annotation

I've refined several type annotations throughout the codebase to align with current best practices and enhance overall clarity. Given the complexity of the code, there may still be areas that need further attention. I welcome any feedback or suggestions to make these improvements even better.

 - Fixes #7202 


@lhoestq  Could someone please take a quick look or let me know if there’s anything I should change? Thanks!

could you fix the conflicts ? I think some type annotations have been improved since your first commit

It should be good now.
I'm happy to add more annotations or refine further if needed—just let me know!
`Dataset.map` disable progress bar

I can't find anything to turn off the `tqdm` progress bars while running a preprocessing function using `Dataset.map`. I want to do akin to `disable_tqdm=True` in the case of `transformers`. Is there something like that?


Progress bar can be disabled like this:
```python
from datasets.utils.logging import set_verbosity_error
set_verbosity_error()
```

There is this line in `Dataset.map`:
```python
not_verbose = bool(logger.getEffectiveLevel() > WARNING)
```

So any logging level higher than `WARNING` turns off the progress bar.

From the linked issues above, an up-to-date solution is:

```python
from datasets.utils.logging import disable_progress_bar
disable_progress_bar()
```

https://github.com/huggingface/datasets/blob/c6e08fcfc3a04e53430c26fa7c07da4cb18d977d/src/datasets/utils/logging.py#L233-L236

Why not have a parameter in the function such as `progress: bool = True`?

+1. We shouldn't need to play with logging levels for a simple thing like this. For instance, trainers have an option `show_progress` that does exactly this.

Bump on this, such a simple QOL issue why can't we fix this?
Implementation of iteration over values of a column in an IterableDataset object

Refers to [this issue](https://github.com/huggingface/datasets/issues/7381).


A couple of questions:
1. I've noticed two strange things: 1) "Around 80% of the final dataset is made of the `en_dataset`" in https://huggingface.co/docs/datasets/stream, 2) "Click on "Pull request" to send your to the project maintainers" in https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md Are `en_dataset` and "your [???]" typos? If so, I can fix them in this PR.
2. Should I update https://huggingface.co/docs/datasets/stream or https://huggingface.co/docs/datasets/access#iterabledataset to include the new feature?

Great ! and chained indexing was easy indeed, thanks :)

regarding your questions:

> I've noticed two strange things: 1) "Around 80% of the final dataset is made of the en_dataset" in https://huggingface.co/docs/datasets/stream, 2) "Click on "Pull request" to send your to the project maintainers" in https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md Are en_dataset and "your [???]" typos? If so, I can fix them in this PR.

Oh good catch, both should be fixed indeed. Feel free to open a new PR for those docs fixes

> Should I update https://huggingface.co/docs/datasets/stream or https://huggingface.co/docs/datasets/access#iterabledataset to include the new feature?

Yep good idea, I think in both places, since /stream is supposed to be exhaustive, and /access already mentions accessing a specific column for `Dataset`

@lhoestq, thank you for the answers!

> Yep good idea, I think in both places, since /stream is supposed to be exhaustive, and /access already mentions accessing a specific column for Dataset

👍, I'll try to add something.

By the way, do you have any ideas about why the CI pipelines have failed? Essentially, I've already encountered these problems [here](https://github.com/huggingface/datasets/issues/7381#issuecomment-2863421974).
I think `check_code_quality` has failed due to the usage of `pre-commit`. The problem seems to be the old version of the ruff hook. I've tried `v0.11.8` (the one that was installed with `pip install -e ".[quality]"`) and `pre-commit` seems to work like `make style` now. However, I don't have any ideas about `pyav` since I don't know what it is...

I've updated /stream and /access, please check the style and clarity. By the way, I would like to add `IterableDataset.skip` near `IterableDataset.take` to mimic [slicing](https://huggingface.co/docs/datasets/access/#slicing). What do you think?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7564). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`concatenate_datasets` does not preserve Pytorch format for IterableDataset

### Describe the bug

When concatenating datasets with `concatenate_datasets`, I would expect the resulting combined dataset to be in the same format as the inputs (assuming it's consistent). This is indeed the behavior when combining `Dataset`, but not when combining `IterableDataset`. Specifically, when applying `concatenate_datasets` to a list of `IterableDataset` in Pytorch format (i.e. using `.with_format(Pytorch)`), the output `IterableDataset` is not in Pytorch format.

### Steps to reproduce the bug

```
import datasets
ds = datasets.Dataset.from_dict({"a": [1,2,3]})
iterable_ds = ds.to_iterable_dataset()
datasets.concatenate_datasets([ds.with_format("torch")]) # <- this preserves Pytorch format
datasets.concatenate_datasets([iterable_ds.with_format("torch")]) # <- this does NOT preserves Pytorch format
```

### Expected behavior

Pytorch format should be preserved when combining IterableDataset in Pytorch format.

### Environment info

datasets==3.5.0, Python 3.11.11, torch==2.2.2


Hi ! Oh indeed it would be cool to return the same format in that case. Would you like to submit a PR ? The function that does the concatenation is here:

https://github.com/huggingface/datasets/blob/90e5bf8a8599b625d6103ee5ac83b98269991141/src/datasets/iterable_dataset.py#L3375-L3380

Thank you for the pointer, @lhoestq ! See #7522
Enable streaming dataset to use the "all" split

Fixes #4637


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4698). All of your documentation changes will be reflected on that endpoint.

@albertvillanova 
Adding the validation split causes these two `assert_called_once` assertions to fail with `AssertionError: Expected 'ArrowWriter' to have been called once. Called 2 times`:

https://github.com/huggingface/datasets/blob/main/tests/test_builder.py#L548-L562

It might be better to create a new dummy generator for the streaming tests, WDYT? Alternatively we could test for `self.call_count` equalling 2.

@cakiki have you read my comment in the issue page?
https://github.com/huggingface/datasets/issues/4637#issuecomment-1175984812

Streaming with `split=all` seems to be working, will fix the failing test next

Not sure if marking the PR as "ready for review" actually notified you, so tagging @albertvillanova just in case :smiley_cat: 

cc @lhoestq 

Hi @cakiki, still interested in working on this? :) 

@albertvillanova So sorry; I have no idea how this slipped through the cracks. Yes, I'd still like to work on this. Is it okay if I DM you on slack?

Sure!! And nevermind!
load_dataset type stubs reject List[str] for split parameter, but runtime supports it

### Describe the bug

The [load_dataset](https://huggingface.co/docs/datasets/v3.6.0/en/package_reference/loading_methods#datasets.load_dataset) method accepts a `List[str]` as the split parameter at runtime, however, the current type stubs restrict the split parameter to `Union[str, Split, None]`. This causes type checkers like Pylance to raise `reportArgumentType` errors when passing a list of strings, even though it works as intended at runtime.

### Steps to reproduce the bug

1. Use load_dataset with multiple splits e.g.:
```
from datasets import load_dataset

ds_train, ds_val, ds_test = load_dataset(
    "Silly-Machine/TuPyE-Dataset",
    "binary",
    split=["train[:75%]", "train[75%:]", "test"]
)
```
2. Observe that code executes correctly at runtime and Pylance raises `Argument of type "List[str]" cannot be assigned to parameter "split" of type "str | Split | None"`

### Expected behavior

The type stubs for [load_dataset](https://huggingface.co/docs/datasets/v3.6.0/en/package_reference/loading_methods#datasets.load_dataset) should accept `Union[str, Split, List[str], None]` or more specific overloads for the split parameter to correctly represent runtime behavior. 

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
- Python version: 3.12.7
- `huggingface_hub` version: 0.32.0
- PyArrow version: 20.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0
Missing multilingual directions in IWSLT2017 dataset's processing script

### Describe the bug

Hi, 

Upon using `iwslt2017.py` in `IWSLT/iwslt2017` on the Hub for loading the datasets, I am unable to obtain the datasets for the language pairs `de-it`, `de-ro`, `de-nl`, `it-de`, `nl-de`, and `ro-de` using it. These 6 pairs do not show up when using `get_dataset_config_names()` to obtain the list of all the configs present in `IWSLT/iwslt2017`. This should not be the case since as mentioned in their original paper (please see https://aclanthology.org/2017.iwslt-1.1.pdf), the authors specify that "_this year we proposed the multilingual translation between any pair of languages from {Dutch, English, German, Italian, Romanian}..._" and because these datasets are indeed present in `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip`.

Best Regards,
Anand

### Steps to reproduce the bug

Check the output of `get_dataset_config_names("IWSLT/iwslt2017", trust_remote_code=True)`: only 24 language pairs are present and the following 6 config names are absent: `iwslt2017-de-it`, `iwslt2017-de-ro`, `iwslt2017-de-nl`, `iwslt2017-it-de`, `iwslt2017-nl-de`, and `iwslt2017-ro-de`.

### Expected behavior

The aforementioned 6 language pairs should also be present and hence, all these 6 language pairs' IWSLT2017 datasets must also be available for further use. 

I would suggest removing `de` from the `BI_LANGUAGES` list and moving it over to the `MULTI_LANGUAGES` list instead in `iwslt2017.py` to account for all the 6 missing language pairs (the same `de-en` dataset is present in both `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip` and `data/2017-01-trnted/texts/de/en/de-en.zip` but the `de-ro`, `de-nl`, `it-de`, `nl-de`, and `ro-de` datasets are only present in  `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip`: so, its unclear why the following comment: _`# XXX: Artificially removed DE from here, as it also exists within bilingual data`_ has been added as `L71` in `iwslt2017.py`). The `README.md` file in `IWSLT/iwslt2017`must then be re-created using `datasets-cli test path/to/iwslt2017.py --save_info --all_configs` to pass all split size verification checks for the 6 new language pairs which were previously non-existent. 

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.8.0-56-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


I have opened 2 PRs on the Hub: `https://huggingface.co/datasets/IWSLT/iwslt2017/discussions/7` and `https://huggingface.co/datasets/IWSLT/iwslt2017/discussions/8` to resolve this issue

cool ! I pinged the owners of the dataset on HF to merge your PRs :)
arrow_schema is not compatible with list

### Describe the bug

```
import datasets
f = datasets.Features({'x': list[datasets.Value(dtype='int32')]})
f.arrow_schema

Traceback (most recent call last):
  File "datasets/features/features.py", line 1826, in arrow_schema
    return pa.schema(self.type).with_metadata({"huggingface": json.dumps(hf_metadata)})
                     ^^^^^^^^^
  File "datasets/features/features.py", line 1815, in type
    return get_nested_type(self)
           ^^^^^^^^^^^^^^^^^^^^^
  File "datasets/features/features.py", line 1252, in get_nested_type
    return pa.struct(
           ^^^^^^^^^^
  File "pyarrow/types.pxi", line 5406, in pyarrow.lib.struct
  File "pyarrow/types.pxi", line 3890, in pyarrow.lib.field
  File "pyarrow/types.pxi", line 5918, in pyarrow.lib.ensure_type
TypeError: DataType expected, got <class 'list'>
```

The following works
```
f = datasets.Features({'x': datasets.LargeList(datasets.Value(dtype='int32'))})
```

### Expected behavior

according to https://github.com/huggingface/datasets/blob/458f45a22c3cc9aea5f442f6f519333dcfeae9b9/src/datasets/features/features.py#L1765 python list should be a valid type specification for features

### Environment info

- `datasets` version: 3.5.1
- Platform: macOS-15.5-arm64-arm-64bit
- Python version: 3.12.9
- `huggingface_hub` version: 0.30.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


Thanks for reporting, I'll look into it

Actually it looks like you just forgot parenthesis:

```diff
- f = datasets.Features({'x': list[datasets.Value(dtype='int32')]})
+ f = datasets.Features({'x': list([datasets.Value(dtype='int32')])})
```

or simply using the `[ ]` syntax:

```python
f = datasets.Features({'x':[datasets.Value(dtype='int32')]})
```

I'm closing this issue if you don't mind

Ah is that what the syntax is? I don't think I was able to find an actual example of it so I assumed it was in the same way that you specify types eg. `list[int]`. This is good to know, thanks.
ImportError: To support decoding images, please install 'Pillow'.

### Describe the bug

Following this tutorial locally using a macboko and VSCode: https://huggingface.co/docs/diffusers/en/tutorials/basic_training
This line of code: for i, image in enumerate(dataset[:4]["image"]):
throws: ImportError: To support decoding images, please install 'Pillow'.

Pillow is installed.

### Steps to reproduce the bug

Run the tutorial


### Expected behavior

Images should be rendered

### Environment info

MacBook, VSCode


what's your `pip show Pillow` output

same issue.. my pip show Pillow output as below:

```
Name: pillow
Version: 11.1.0
Summary: Python Imaging Library (Fork)
Home-page: https://python-pillow.github.io/
Author: 
Author-email: "Jeffrey A. Clark" <aclark@aclark.net>
License: MIT-CMU
Location: [/opt/homebrew/lib/python3.10/site-packages](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/lib/python3.10/site-packages)
Requires: 
Required-by:
```

I encountered the same problem on Ubuntu system, my pip show Pillow output as below:

```
Name: pillow
Version: 10.4.0
Summary: Python Imaging Library (Fork)
Home-page: https://python-pillow.org/
Author: 
Author-email: "Jeffrey A. Clark" <[aclark@aclark.net](mailto:aclark@aclark.net)>
License: HPND
Location: /home/shunying/.local/lib/python3.8/site-packages
Requires: 
Required-by: 
```

Well, solved this by specifying the pip version to my conda virtual environment :)

I have also encountered this. It's a strange thing that's happening.

Checking the code `datasets` it uses `importlib.util.find_spec("PIL")` to verify if `PIL` is installed. While both `pip show` and `importlib` work correctly, I still got the error.

In my case, restarting and redoing the `datasets` import helped. Seems weird to me.
ValueError: Invalid pattern: '**' can only be an entire path component [Colab]

### Describe the bug

I have a dataset on HF [here](https://huggingface.co/datasets/kambale/luganda-english-parallel-corpus) that i've previously used to train a translation model [here](https://huggingface.co/kambale/pearl-11m-translate).

now i changed a few hyperparameters to increase number of tokens for the model, increase Transformer layers, and all

however, when i try to load the dataset, this error keeps coming up.. i have tried everything.. i have re-written the code a hundred times, and this keep coming up

### Steps to reproduce the bug

Imports:

```bash
!pip install datasets huggingface_hub fsspec
```

Python code:

```python
from datasets import load_dataset

HF_DATASET_NAME = "kambale/luganda-english-parallel-corpus"

# Load the dataset
try:
    if not HF_DATASET_NAME or HF_DATASET_NAME == "YOUR_HF_DATASET_NAME":
        raise ValueError(
            "Please provide a valid Hugging Face dataset name."
        )

    dataset = load_dataset(HF_DATASET_NAME)

# Omitted code as the error happens on the line above 

except ValueError as ve:
    print(f"Configuration Error: {ve}")
    raise
except Exception as e:
    print(f"An error occurred while loading the dataset '{HF_DATASET_NAME}': {e}")
    raise e
```

now, i have tried going through this [issue](https://github.com/huggingface/datasets/issues/6737) and nothing helps 

### Expected behavior

loading the dataset successfully and perform splits (train, test, validation)

### Environment info

from the imports, i do not install specific versions of these libraries, so the latest or available version is installed

* `datasets` version: latest
* `Platform`: Google Colab
* `Hardware`: NVIDIA A100 GPU
* `Python` version: latest
* `huggingface_hub` version: latest
* `fsspec` version: latest


Could you please run the following code snippet in your environment and share the exact output? This will help check for any compatibility issues within the env itself. 

```
import datasets
import huggingface_hub
import fsspec

print("datasets version:", datasets.__version__)
print("huggingface_hub version:", huggingface_hub.__version__)
print("fsspec version:", fsspec.__version__)
```

```bash
datasets version: 2.14.4
huggingface_hub version: 0.31.4
fsspec version: 2025.3.2
```

Version 2.14.4 is not the latest version available, in fact it is from August 08, 2023 (you can check here: https://pypi.org/project/datasets/#history)

Use pip install datasets==3.6.0 to install a more recent version (from May 7, 2025)

I also had the same problem with Colab, after updating to the latest version it was solved.

I hope it helps

thank you @CleitonOERocha. it sure did help.

updating `datasets` to v3.6.0 and keeping `fsspec` on v2025.3.2 eliminates the issue.

Very helpful, thank you!
NotImplementedError: <class 'datasets.iterable_dataset.RepeatExamplesIterable'> doesn't implement num_shards yet

### Describe the bug

When using `.repeat()` on an `IterableDataset`, this error gets thrown. There is [this thread](https://discuss.huggingface.co/t/making-an-infinite-iterabledataset/146192/5) that seems to imply the fix is trivial, but I don't know anything about this codebase, so I'm opening this issue rather than attempting to open a PR.

### Steps to reproduce the bug

1. Create an `IterableDataset`.
2. Call `.repeat(None)` on it.
3. Wrap it in a pytorch `DataLoader`
4. Iterate over it.

### Expected behavior

This should work normally.

### Environment info

datasets: 3.5.0
Avoid multiple default config names

Fix duplicating default config names.

Currently, when calling `push_to_hub(set_default=True` with 2 different config names, both are set as default. 

Moreover, this will generate an error next time we try to push another default config name, raised by `MetadataConfigs.get_default_config_name`:
https://github.com/huggingface/datasets/blob/da1db8a5b89fc0badaa0f571b36e122e52ae8c61/src/datasets/arrow_dataset.py#L5757
https://github.com/huggingface/datasets/blob/da1db8a5b89fc0badaa0f571b36e122e52ae8c61/src/datasets/utils/metadata.py#L186-L188


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7585). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix regex library warnings

# PR Summary
This small PR resolves the regex library warnings showing starting Python3.11:
```python
DeprecationWarning: 'count' is passed as positional argument
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7576). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix broken link to albumentations

A few months back I rewrote all docs at [https://albumentations.ai/docs](https://albumentations.ai/docs), and some pages changed their links.

In this PR fixed link to the most recent doc in Albumentations about bounding boxes and it's format. 

Fix a few typos in the doc as well.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7593). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq ping
Add `IterableDataset.push_to_hub()`

Basic implementation, which writes one shard per input dataset shard.
This is to be improved later.

Close https://github.com/huggingface/datasets/issues/5665

PS: for image/audio datasets structured as actual image/audio files (not parquet), you can sometimes speed it up with `ds.decode(num_threads=...).push_to_hub(...)`


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7595). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.