AniketTec/github-issues huggingface dataset:
Title: load_dataset type stubs reject List[str] for split parameter, but runtime supports it
State: closed

### Describe the bug

The [load_dataset](https://huggingface.co/docs/datasets/v3.6.0/en/package_reference/loading_methods#datasets.load_dataset) method accepts a `List[str]` as the split parameter at runtime, however, the current type stubs restrict the split parameter to `Union[str, Split, None]`. This causes type checkers like Pylance to raise `reportArgumentType` errors when passing a list of strings, even though it works as intended at runtime.

### Steps to reproduce the bug

1. Use load_dataset with multiple splits e.g.:
```
from datasets import load_dataset

ds_train, ds_val, ds_test = load_dataset(
    "Silly-Machine/TuPyE-Dataset",
    "binary",
    split=["train[:75%]", "train[75%:]", "test"]
)
```
2. Observe that code executes correctly at runtime and Pylance raises `Argument of type "List[str]" cannot be assigned to parameter "split" of type "str | Split | None"`

### Expected behavior

The type stubs for [load_dataset](https://huggingface.co/docs/datasets/v3.6.0/en/package_reference/loading_methods#datasets.load_dataset) should accept `Union[str, Split, List[str], None]` or more specific overloads for the split parameter to correctly represent runtime behavior. 

### Environment info

- `datasets` version: 3.6.0
- Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
- Python version: 3.12.7
- `huggingface_hub` version: 0.32.0
- PyArrow version: 20.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0

Comments:
Title: Requesting a specific split (eg: test) still downloads all (train, test, val) data when streaming=False.
State: open

### Describe the bug

When using load_dataset() from the datasets library (in load.py), specifying a particular split (e.g., split="train") still results in downloading data for all splits when streaming=False. This happens during the builder_instance.download_and_prepare() call.
This behavior leads to unnecessary bandwidth usage and longer download times, especially for large datasets, even if the user only intends to use a single split.

### Steps to reproduce the bug

dataset_name = "skbose/indian-english-nptel-v0"
dataset = load_dataset(dataset_name, token=hf_token, split="test")


### Expected behavior

Optimize the download logic so that only the required split is downloaded when streaming=False when a specific split is provided.

### Environment info

Dataset: skbose/indian-english-nptel-v0
Platform: M1 Apple Silicon
Python verison: 3.12.9
datasets>=3.5.0

Comments:
Hi ! There was a PR open to improve this: https://github.com/huggingface/datasets/pull/6832 
but it hasn't been continued so far.

It would be a cool improvement though !

Been having this problem with datasets and dataloader for a while.
Title: arrow_schema is not compatible with list
State: closed

### Describe the bug

```
import datasets
f = datasets.Features({'x': list[datasets.Value(dtype='int32')]})
f.arrow_schema

Traceback (most recent call last):
  File "datasets/features/features.py", line 1826, in arrow_schema
    return pa.schema(self.type).with_metadata({"huggingface": json.dumps(hf_metadata)})
                     ^^^^^^^^^
  File "datasets/features/features.py", line 1815, in type
    return get_nested_type(self)
           ^^^^^^^^^^^^^^^^^^^^^
  File "datasets/features/features.py", line 1252, in get_nested_type
    return pa.struct(
           ^^^^^^^^^^
  File "pyarrow/types.pxi", line 5406, in pyarrow.lib.struct
  File "pyarrow/types.pxi", line 3890, in pyarrow.lib.field
  File "pyarrow/types.pxi", line 5918, in pyarrow.lib.ensure_type
TypeError: DataType expected, got <class 'list'>
```

The following works
```
f = datasets.Features({'x': datasets.LargeList(datasets.Value(dtype='int32'))})
```

### Expected behavior

according to https://github.com/huggingface/datasets/blob/458f45a22c3cc9aea5f442f6f519333dcfeae9b9/src/datasets/features/features.py#L1765 python list should be a valid type specification for features

### Environment info

- `datasets` version: 3.5.1
- Platform: macOS-15.5-arm64-arm-64bit
- Python version: 3.12.9
- `huggingface_hub` version: 0.30.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments:
Thanks for reporting, I'll look into it

Actually it looks like you just forgot parenthesis:

```diff
- f = datasets.Features({'x': list[datasets.Value(dtype='int32')]})
+ f = datasets.Features({'x': list([datasets.Value(dtype='int32')])})
```

or simply using the `[ ]` syntax:

```python
f = datasets.Features({'x':[datasets.Value(dtype='int32')]})
```

I'm closing this issue if you don't mind

Ah is that what the syntax is? I don't think I was able to find an actual example of it so I assumed it was in the same way that you specify types eg. `list[int]`. This is good to know, thanks.
Title: Fix regex library warnings
State: closed

# PR Summary
This small PR resolves the regex library warnings showing starting Python3.11:
```python
DeprecationWarning: 'count' is passed as positional argument
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7576). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Missing multilingual directions in IWSLT2017 dataset's processing script
State: open

### Describe the bug

Hi, 

Upon using `iwslt2017.py` in `IWSLT/iwslt2017` on the Hub for loading the datasets, I am unable to obtain the datasets for the language pairs `de-it`, `de-ro`, `de-nl`, `it-de`, `nl-de`, and `ro-de` using it. These 6 pairs do not show up when using `get_dataset_config_names()` to obtain the list of all the configs present in `IWSLT/iwslt2017`. This should not be the case since as mentioned in their original paper (please see https://aclanthology.org/2017.iwslt-1.1.pdf), the authors specify that "_this year we proposed the multilingual translation between any pair of languages from {Dutch, English, German, Italian, Romanian}..._" and because these datasets are indeed present in `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip`.

Best Regards,
Anand

### Steps to reproduce the bug

Check the output of `get_dataset_config_names("IWSLT/iwslt2017", trust_remote_code=True)`: only 24 language pairs are present and the following 6 config names are absent: `iwslt2017-de-it`, `iwslt2017-de-ro`, `iwslt2017-de-nl`, `iwslt2017-it-de`, `iwslt2017-nl-de`, and `iwslt2017-ro-de`.

### Expected behavior

The aforementioned 6 language pairs should also be present and hence, all these 6 language pairs' IWSLT2017 datasets must also be available for further use. 

I would suggest removing `de` from the `BI_LANGUAGES` list and moving it over to the `MULTI_LANGUAGES` list instead in `iwslt2017.py` to account for all the 6 missing language pairs (the same `de-en` dataset is present in both `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip` and `data/2017-01-trnted/texts/de/en/de-en.zip` but the `de-ro`, `de-nl`, `it-de`, `nl-de`, and `ro-de` datasets are only present in  `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip`: so, its unclear why the following comment: _`# XXX: Artificially removed DE from here, as it also exists within bilingual data`_ has been added as `L71` in `iwslt2017.py`). The `README.md` file in `IWSLT/iwslt2017`must then be re-created using `datasets-cli test path/to/iwslt2017.py --save_info --all_configs` to pass all split size verification checks for the 6 new language pairs which were previously non-existent. 

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.8.0-56-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments:
I have opened 2 PRs on the Hub: `https://huggingface.co/datasets/IWSLT/iwslt2017/discussions/7` and `https://huggingface.co/datasets/IWSLT/iwslt2017/discussions/8` to resolve this issue

cool ! I pinged the owners of the dataset on HF to merge your PRs :)
Title: No Samsum dataset
State: closed

### Describe the bug

https://huggingface.co/datasets/Samsung/samsum dataset not found error 404

Originated from https://github.com/meta-llama/llama-cookbook/issues/948

### Steps to reproduce the bug

go to website https://huggingface.co/datasets/Samsung/samsum
see the error

also downloading it with python throws 
```
Couldn't find 'Samsung/samsum' on the Hugging Face Hub either: FileNotFoundError: Samsung/samsum@f00baf5a7d4abfec6820415493bcb52c587788e6/samsum.py (repository not found)
```

### Expected behavior

Dataset exists

### Environment info

```
- `datasets` version: 3.2.0
- Platform: macOS-15.4.1-arm64-arm-64bit
- Python version: 3.12.2
- `huggingface_hub` version: 0.26.5
- PyArrow version: 16.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
```

Comments:
According to the following https://huggingface.co/posts/seawolf2357/424129432408590, as of now the dataset seems to be inaccessible.

@IgorKasianenko, would https://huggingface.co/datasets/knkarthick/samsum suffice for your purpose?


Thanks @SP1029 for the update!
That will work for now, using it as replacement. Is there a officially recommended way to maintain the CC licensed dataset under the organization account? 
Feel free to close this issue

> Is there an officially recommended way to maintain a CC-licensed dataset under an organizational account?

@IgorKasianenko, apologies, this is not my area of expertise.

> Please feel free to close this issue.

I have limited access and may not be able to do that. Since you opened it, you would be able to close it.

dataset_samsum = load_dataset("knkarthick/samsum")

is working
Title: add check if repo exists for dataset uploading
State: closed

Currently, I'm reuploading datasets for [`MTEB`](https://github.com/embeddings-benchmark/mteb/). Some of them have many splits (more than 20), and I'm encountering the error:
`Too many requests for https://huggingface.co/datasets/repo/create`.

It seems that this issue occurs because the dataset tries to recreate itself every time a split is uploaded. To resolve this, I've added a check to ensure that if the dataset already exists, it won't attempt to recreate it.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7565). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq Can you review, please? I don't think that errors in CI are related to my changes

Similar implementation done in https://github.com/huggingface/datasets/pull/7708
Title: Implementation of iteration over values of a column in an IterableDataset object
State: closed

Refers to [this issue](https://github.com/huggingface/datasets/issues/7381).

Comments:
A couple of questions:
1. I've noticed two strange things: 1) "Around 80% of the final dataset is made of the `en_dataset`" in https://huggingface.co/docs/datasets/stream, 2) "Click on "Pull request" to send your to the project maintainers" in https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md Are `en_dataset` and "your [???]" typos? If so, I can fix them in this PR.
2. Should I update https://huggingface.co/docs/datasets/stream or https://huggingface.co/docs/datasets/access#iterabledataset to include the new feature?

Great ! and chained indexing was easy indeed, thanks :)

regarding your questions:

> I've noticed two strange things: 1) "Around 80% of the final dataset is made of the en_dataset" in https://huggingface.co/docs/datasets/stream, 2) "Click on "Pull request" to send your to the project maintainers" in https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md Are en_dataset and "your [???]" typos? If so, I can fix them in this PR.

Oh good catch, both should be fixed indeed. Feel free to open a new PR for those docs fixes

> Should I update https://huggingface.co/docs/datasets/stream or https://huggingface.co/docs/datasets/access#iterabledataset to include the new feature?

Yep good idea, I think in both places, since /stream is supposed to be exhaustive, and /access already mentions accessing a specific column for `Dataset`

@lhoestq, thank you for the answers!

> Yep good idea, I think in both places, since /stream is supposed to be exhaustive, and /access already mentions accessing a specific column for Dataset

👍, I'll try to add something.

By the way, do you have any ideas about why the CI pipelines have failed? Essentially, I've already encountered these problems [here](https://github.com/huggingface/datasets/issues/7381#issuecomment-2863421974).
I think `check_code_quality` has failed due to the usage of `pre-commit`. The problem seems to be the old version of the ruff hook. I've tried `v0.11.8` (the one that was installed with `pip install -e ".[quality]"`) and `pre-commit` seems to work like `make style` now. However, I don't have any ideas about `pyav` since I don't know what it is...

I've updated /stream and /access, please check the style and clarity. By the way, I would like to add `IterableDataset.skip` near `IterableDataset.take` to mimic [slicing](https://huggingface.co/docs/datasets/access/#slicing). What do you think?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7564). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: NotImplementedError: <class 'datasets.iterable_dataset.RepeatExamplesIterable'> doesn't implement num_shards yet
State: closed

### Describe the bug

When using `.repeat()` on an `IterableDataset`, this error gets thrown. There is [this thread](https://discuss.huggingface.co/t/making-an-infinite-iterabledataset/146192/5) that seems to imply the fix is trivial, but I don't know anything about this codebase, so I'm opening this issue rather than attempting to open a PR.

### Steps to reproduce the bug

1. Create an `IterableDataset`.
2. Call `.repeat(None)` on it.
3. Wrap it in a pytorch `DataLoader`
4. Iterate over it.

### Expected behavior

This should work normally.

### Environment info

datasets: 3.5.0

Comments:
Title: fix regression
State: closed

reported in https://github.com/huggingface/datasets/pull/7557 (I just reorganized the condition)

wanted to apply this change to the original PR but github didn't let me apply it directly - merging this one instead

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7558). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: check for empty _formatting
State: closed

Fixes a regression from #7553 breaking shuffling of iterable datasets 
<img width="884" alt="Screenshot 2025-05-07 at 9 16 52 AM" src="https://github.com/user-attachments/assets/d2f43c5f-4092-4efe-ac31-a32cbd025fe3" />


Comments:
Thanks for reporting and for the fix ! I tried to reorganize the condition in your PR but didn't get the right permission so. I ended up merging https://github.com/huggingface/datasets/pull/7558 directly so I can make a release today - I hope you don't mind
Title: Add `--merge-pull-request` option for `convert_to_parquet`
State: closed

Closes #7527 

Note that this implementation **will only merge the last PR in the case that they get split up by `push_to_hub`**. See https://github.com/huggingface/datasets/discussions/7555 for more details.

Comments:
This is ready for a review, happy to make any changes. The main question for maintainers is how this should interact with #7555. If my suggestion there is accepted, this PR can be kept as is. If not, more changes are required to merge all the PR parts.

Closing since convert to parquet has been removed... https://github.com/huggingface/datasets/pull/7592#issuecomment-3073053138
Title: datasets downloads and generates all splits, even though a single split is requested (for dataset with loading script)
State: closed

### Describe the bug

`datasets` downloads and generates all splits, even though a single split is requested.  [This](https://huggingface.co/datasets/jordiae/exebench) is the dataset in question.  It uses a loading script.  I am not 100% sure that this is a bug, because maybe with loading scripts `datasets` must actually process all the splits?  But I thought loading scripts were designed to avoid this.

### Steps to reproduce the bug

See [this notebook](https://colab.research.google.com/drive/14kcXp_hgcdj-kIzK0bCG6taE-CLZPVvq?usp=sharing)

Or:
```python
from datasets import load_dataset
dataset = load_dataset('jordiae/exebench', split='test_synth', trust_remote_code=True)
```

### Expected behavior

I expected only the `test_synth` split to be downloaded and processed.

### Environment info

- `datasets` version: 3.5.1
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python version: 3.11.12
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2025.3.0

Comments:
Hi ! there has been some effort on allowing to download only a subset of splits in https://github.com/huggingface/datasets/pull/6832 but no one has been continuing this work so far. This would be a welcomed contribution though

Also note that loading script are often unoptimized, and we recommend using datasets in standard formats like Parquet instead.

Btw there is a CLI tool to convert a loading script to parquet:

```
datasets-cli convert_to_parquet <dataset-name> --trust_remote_code
```

Closing in favor of #6832
Title: Rebatch arrow iterables before formatted iterable
State: closed

close https://github.com/huggingface/datasets/issues/7538 and https://github.com/huggingface/datasets/issues/7475

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7553). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq Our CI found an issue with this changeset causing a regression with shuffling iterable datasets 
<img width="884" alt="Screenshot 2025-05-07 at 9 16 52 AM" src="https://github.com/user-attachments/assets/bf7d9c7e-cc14-47da-8da6-d1a345992d7c" />
Title: Enable xet in push to hub
State: closed

follows https://github.com/huggingface/huggingface_hub/pull/3035

related to https://github.com/huggingface/datasets/issues/7526

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7552). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Issue with offline mode and partial dataset cached
State: open

### Describe the bug

Hi, 

a issue related to #4760 here when loading a single file from a dataset, unable to access it in offline mode afterwards

### Steps to reproduce the bug

```python
import os
# os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["HF_TOKEN"] = "xxxxxxxxxxxxxx"

import datasets

dataset_name = "uonlp/CulturaX"
data_files = "fr/fr_part_00038.parquet"

ds = datasets.load_dataset(dataset_name, split='train', data_files=data_files)
print(f"Dataset loaded   : {ds}")
```
Once the file has been cached, I rerun with the HF_HUB_OFFLINE activated an get this error : 
```
ValueError: Couldn't find cache for uonlp/CulturaX for config 'default-1e725f978350254e'
Available configs in the cache: ['default-2935e8cdcc21c613']
```

### Expected behavior

Should be able to access the previously cached files

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.4.0-215-generic-x86_64-with-glibc2.31
- Python version: 3.12.0
- `huggingface_hub` version: 0.27.0
- PyArrow version: 19.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


Comments:
It seems the problem comes from builder.py / create_config_id()

On the first call, when the cache is empty we have
```
config_kwargs = {'data_files': {'train': ['hf://datasets/uonlp/CulturaX@6a8734bc69fefcbb7735f4f9250f43e4cd7a442e/fr/fr_part_00038.parquet']}}
```
leading to config_id beeing 'default-2935e8cdcc21c613'

then, on the second call, 
```
config_kwargs = {'data_files': 'fr/fr_part_00038.parquet'}
```
thus explaining why the hash is not the same, despite having the same parameter when calling load_dataset : data_files="fr/fr_part_00038.parquet"

Same behavior with version 3.5.1

Same issue when loading `google/IndicGenBench_flores_in` with `dataset==2.21.0` and `dataset==3.6.0` .




> It seems the problem comes from builder.py / create_config_id()
> 
> On the first call, when the cache is empty we have
> 
> ```
> config_kwargs = {'data_files': {'train': ['hf://datasets/uonlp/CulturaX@6a8734bc69fefcbb7735f4f9250f43e4cd7a442e/fr/fr_part_00038.parquet']}}
> ```
> 
> leading to config_id beeing 'default-2935e8cdcc21c613'
> 
> then, on the second call,
> 
> ```
> config_kwargs = {'data_files': 'fr/fr_part_00038.parquet'}
> ```
> 
> thus explaining why the hash is not the same, despite having the same parameter when calling load_dataset : data_files="fr/fr_part_00038.parquet"


I have identified that the issue indeed lies in the `data_files` within `config_kwargs`. 
The format and prefix of `data_files` differ depending on whether `HF_HUB_OFFLINE` is set, leading to different final `config_id` values. 
When I use other datasets without passing the `data_files` parameter, this issue does not occur.

A possible solution might be to standardize the formatting of `data_files` within the `create_config_id` function.
Title: Avoid global umask for setting file mode.
State: closed

This PR updates the method for setting the permissions on `cache_path` after calling `shutil.move`. The call to `shutil.move` may not preserve permissions if the source and destination are on different filesystems. Reading and resetting umask can cause race conditions, so directly read what permissions were set for the `temp_file` instead.

This fixes https://github.com/huggingface/datasets/issues/7536.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7547). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.