NyssaW/github-issues huggingface dataset:
Requesting a specific split (eg: test) still downloads all (train, test, val) data when streaming=False.

### Describe the bug

When using load_dataset() from the datasets library (in load.py), specifying a particular split (e.g., split="train") still results in downloading data for all splits when streaming=False. This happens during the builder_instance.download_and_prepare() call.
This behavior leads to unnecessary bandwidth usage and longer download times, especially for large datasets, even if the user only intends to use a single split.

### Steps to reproduce the bug

dataset_name = "skbose/indian-english-nptel-v0"
dataset = load_dataset(dataset_name, token=hf_token, split="test")


### Expected behavior

Optimize the download logic so that only the required split is downloaded when streaming=False when a specific split is provided.

### Environment info

Dataset: skbose/indian-english-nptel-v0
Platform: M1 Apple Silicon
Python verison: 3.12.9
datasets>=3.5.0

Comments:
Hi ! There was a PR open to improve this: https://github.com/huggingface/datasets/pull/6832 
but it hasn't been continued so far.

It would be a cool improvement though !

Been having this problem with datasets and dataloader for a while.
arrow_schema is not compatible with list

### Describe the bug

```
import datasets
f = datasets.Features({'x': list[datasets.Value(dtype='int32')]})
f.arrow_schema

Traceback (most recent call last):
  File "datasets/features/features.py", line 1826, in arrow_schema
    return pa.schema(self.type).with_metadata({"huggingface": json.dumps(hf_metadata)})
                     ^^^^^^^^^
  File "datasets/features/features.py", line 1815, in type
    return get_nested_type(self)
           ^^^^^^^^^^^^^^^^^^^^^
  File "datasets/features/features.py", line 1252, in get_nested_type
    return pa.struct(
           ^^^^^^^^^^
  File "pyarrow/types.pxi", line 5406, in pyarrow.lib.struct
  File "pyarrow/types.pxi", line 3890, in pyarrow.lib.field
  File "pyarrow/types.pxi", line 5918, in pyarrow.lib.ensure_type
TypeError: DataType expected, got <class 'list'>
```

The following works
```
f = datasets.Features({'x': datasets.LargeList(datasets.Value(dtype='int32'))})
```

### Expected behavior

according to https://github.com/huggingface/datasets/blob/458f45a22c3cc9aea5f442f6f519333dcfeae9b9/src/datasets/features/features.py#L1765 python list should be a valid type specification for features

### Environment info

- `datasets` version: 3.5.1
- Platform: macOS-15.5-arm64-arm-64bit
- Python version: 3.12.9
- `huggingface_hub` version: 0.30.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments:
Thanks for reporting, I'll look into it

Actually it looks like you just forgot parenthesis:

```diff
- f = datasets.Features({'x': list[datasets.Value(dtype='int32')]})
+ f = datasets.Features({'x': list([datasets.Value(dtype='int32')])})
```

or simply using the `[ ]` syntax:

```python
f = datasets.Features({'x':[datasets.Value(dtype='int32')]})
```

I'm closing this issue if you don't mind

Ah is that what the syntax is? I don't think I was able to find an actual example of it so I assumed it was in the same way that you specify types eg. `list[int]`. This is good to know, thanks.
Fix regex library warnings

# PR Summary
This small PR resolves the regex library warnings showing starting Python3.11:
```python
DeprecationWarning: 'count' is passed as positional argument
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7576). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Missing multilingual directions in IWSLT2017 dataset's processing script

### Describe the bug

Hi, 

Upon using `iwslt2017.py` in `IWSLT/iwslt2017` on the Hub for loading the datasets, I am unable to obtain the datasets for the language pairs `de-it`, `de-ro`, `de-nl`, `it-de`, `nl-de`, and `ro-de` using it. These 6 pairs do not show up when using `get_dataset_config_names()` to obtain the list of all the configs present in `IWSLT/iwslt2017`. This should not be the case since as mentioned in their original paper (please see https://aclanthology.org/2017.iwslt-1.1.pdf), the authors specify that "_this year we proposed the multilingual translation between any pair of languages from {Dutch, English, German, Italian, Romanian}..._" and because these datasets are indeed present in `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip`.

Best Regards,
Anand

### Steps to reproduce the bug

Check the output of `get_dataset_config_names("IWSLT/iwslt2017", trust_remote_code=True)`: only 24 language pairs are present and the following 6 config names are absent: `iwslt2017-de-it`, `iwslt2017-de-ro`, `iwslt2017-de-nl`, `iwslt2017-it-de`, `iwslt2017-nl-de`, and `iwslt2017-ro-de`.

### Expected behavior

The aforementioned 6 language pairs should also be present and hence, all these 6 language pairs' IWSLT2017 datasets must also be available for further use. 

I would suggest removing `de` from the `BI_LANGUAGES` list and moving it over to the `MULTI_LANGUAGES` list instead in `iwslt2017.py` to account for all the 6 missing language pairs (the same `de-en` dataset is present in both `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip` and `data/2017-01-trnted/texts/de/en/de-en.zip` but the `de-ro`, `de-nl`, `it-de`, `nl-de`, and `ro-de` datasets are only present in  `data/2017-01-trnmted/texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.zip`: so, its unclear why the following comment: _`# XXX: Artificially removed DE from here, as it also exists within bilingual data`_ has been added as `L71` in `iwslt2017.py`). The `README.md` file in `IWSLT/iwslt2017`must then be re-created using `datasets-cli test path/to/iwslt2017.py --save_info --all_configs` to pass all split size verification checks for the 6 new language pairs which were previously non-existent. 

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.8.0-56-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments:
I have opened 2 PRs on the Hub: `https://huggingface.co/datasets/IWSLT/iwslt2017/discussions/7` and `https://huggingface.co/datasets/IWSLT/iwslt2017/discussions/8` to resolve this issue

cool ! I pinged the owners of the dataset on HF to merge your PRs :)
No Samsum dataset

### Describe the bug

https://huggingface.co/datasets/Samsung/samsum dataset not found error 404

Originated from https://github.com/meta-llama/llama-cookbook/issues/948

### Steps to reproduce the bug

go to website https://huggingface.co/datasets/Samsung/samsum
see the error

also downloading it with python throws 
```
Couldn't find 'Samsung/samsum' on the Hugging Face Hub either: FileNotFoundError: Samsung/samsum@f00baf5a7d4abfec6820415493bcb52c587788e6/samsum.py (repository not found)
```

### Expected behavior

Dataset exists

### Environment info

```
- `datasets` version: 3.2.0
- Platform: macOS-15.4.1-arm64-arm-64bit
- Python version: 3.12.2
- `huggingface_hub` version: 0.26.5
- PyArrow version: 16.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
```

Comments:
According to the following https://huggingface.co/posts/seawolf2357/424129432408590, as of now the dataset seems to be inaccessible.

@IgorKasianenko, would https://huggingface.co/datasets/knkarthick/samsum suffice for your purpose?


Thanks @SP1029 for the update!
That will work for now, using it as replacement. Is there a officially recommended way to maintain the CC licensed dataset under the organization account? 
Feel free to close this issue

> Is there an officially recommended way to maintain a CC-licensed dataset under an organizational account?

@IgorKasianenko, apologies, this is not my area of expertise.

> Please feel free to close this issue.

I have limited access and may not be able to do that. Since you opened it, you would be able to close it.

dataset_samsum = load_dataset("knkarthick/samsum")

is working
add check if repo exists for dataset uploading

Currently, I'm reuploading datasets for [`MTEB`](https://github.com/embeddings-benchmark/mteb/). Some of them have many splits (more than 20), and I'm encountering the error:
`Too many requests for https://huggingface.co/datasets/repo/create`.

It seems that this issue occurs because the dataset tries to recreate itself every time a split is uploaded. To resolve this, I've added a check to ensure that if the dataset already exists, it won't attempt to recreate it.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7565). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq Can you review, please? I don't think that errors in CI are related to my changes

Similar implementation done in https://github.com/huggingface/datasets/pull/7708
Implementation of iteration over values of a column in an IterableDataset object

Refers to [this issue](https://github.com/huggingface/datasets/issues/7381).

Comments:
A couple of questions:
1. I've noticed two strange things: 1) "Around 80% of the final dataset is made of the `en_dataset`" in https://huggingface.co/docs/datasets/stream, 2) "Click on "Pull request" to send your to the project maintainers" in https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md Are `en_dataset` and "your [???]" typos? If so, I can fix them in this PR.
2. Should I update https://huggingface.co/docs/datasets/stream or https://huggingface.co/docs/datasets/access#iterabledataset to include the new feature?

Great ! and chained indexing was easy indeed, thanks :)

regarding your questions:

> I've noticed two strange things: 1) "Around 80% of the final dataset is made of the en_dataset" in https://huggingface.co/docs/datasets/stream, 2) "Click on "Pull request" to send your to the project maintainers" in https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md Are en_dataset and "your [???]" typos? If so, I can fix them in this PR.

Oh good catch, both should be fixed indeed. Feel free to open a new PR for those docs fixes

> Should I update https://huggingface.co/docs/datasets/stream or https://huggingface.co/docs/datasets/access#iterabledataset to include the new feature?

Yep good idea, I think in both places, since /stream is supposed to be exhaustive, and /access already mentions accessing a specific column for `Dataset`

@lhoestq, thank you for the answers!

> Yep good idea, I think in both places, since /stream is supposed to be exhaustive, and /access already mentions accessing a specific column for Dataset

👍, I'll try to add something.

By the way, do you have any ideas about why the CI pipelines have failed? Essentially, I've already encountered these problems [here](https://github.com/huggingface/datasets/issues/7381#issuecomment-2863421974).
I think `check_code_quality` has failed due to the usage of `pre-commit`. The problem seems to be the old version of the ruff hook. I've tried `v0.11.8` (the one that was installed with `pip install -e ".[quality]"`) and `pre-commit` seems to work like `make style` now. However, I don't have any ideas about `pyav` since I don't know what it is...

I've updated /stream and /access, please check the style and clarity. By the way, I would like to add `IterableDataset.skip` near `IterableDataset.take` to mimic [slicing](https://huggingface.co/docs/datasets/access/#slicing). What do you think?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7564). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
NotImplementedError: <class 'datasets.iterable_dataset.RepeatExamplesIterable'> doesn't implement num_shards yet

### Describe the bug

When using `.repeat()` on an `IterableDataset`, this error gets thrown. There is [this thread](https://discuss.huggingface.co/t/making-an-infinite-iterabledataset/146192/5) that seems to imply the fix is trivial, but I don't know anything about this codebase, so I'm opening this issue rather than attempting to open a PR.

### Steps to reproduce the bug

1. Create an `IterableDataset`.
2. Call `.repeat(None)` on it.
3. Wrap it in a pytorch `DataLoader`
4. Iterate over it.

### Expected behavior

This should work normally.

### Environment info

datasets: 3.5.0

Comments:
fix regression

reported in https://github.com/huggingface/datasets/pull/7557 (I just reorganized the condition)

wanted to apply this change to the original PR but github didn't let me apply it directly - merging this one instead

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7558). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
check for empty _formatting

Fixes a regression from #7553 breaking shuffling of iterable datasets 
<img width="884" alt="Screenshot 2025-05-07 at 9 16 52 AM" src="https://github.com/user-attachments/assets/d2f43c5f-4092-4efe-ac31-a32cbd025fe3" />


Comments:
Thanks for reporting and for the fix ! I tried to reorganize the condition in your PR but didn't get the right permission so. I ended up merging https://github.com/huggingface/datasets/pull/7558 directly so I can make a release today - I hope you don't mind
Add `--merge-pull-request` option for `convert_to_parquet`

Closes #7527 

Note that this implementation **will only merge the last PR in the case that they get split up by `push_to_hub`**. See https://github.com/huggingface/datasets/discussions/7555 for more details.

Comments:
This is ready for a review, happy to make any changes. The main question for maintainers is how this should interact with #7555. If my suggestion there is accepted, this PR can be kept as is. If not, more changes are required to merge all the PR parts.

Closing since convert to parquet has been removed... https://github.com/huggingface/datasets/pull/7592#issuecomment-3073053138
datasets downloads and generates all splits, even though a single split is requested (for dataset with loading script)

### Describe the bug

`datasets` downloads and generates all splits, even though a single split is requested.  [This](https://huggingface.co/datasets/jordiae/exebench) is the dataset in question.  It uses a loading script.  I am not 100% sure that this is a bug, because maybe with loading scripts `datasets` must actually process all the splits?  But I thought loading scripts were designed to avoid this.

### Steps to reproduce the bug

See [this notebook](https://colab.research.google.com/drive/14kcXp_hgcdj-kIzK0bCG6taE-CLZPVvq?usp=sharing)

Or:
```python
from datasets import load_dataset
dataset = load_dataset('jordiae/exebench', split='test_synth', trust_remote_code=True)
```

### Expected behavior

I expected only the `test_synth` split to be downloaded and processed.

### Environment info

- `datasets` version: 3.5.1
- Platform: Linux-6.1.123+-x86_64-with-glibc2.35
- Python version: 3.11.12
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2025.3.0

Comments:
Hi ! there has been some effort on allowing to download only a subset of splits in https://github.com/huggingface/datasets/pull/6832 but no one has been continuing this work so far. This would be a welcomed contribution though

Also note that loading script are often unoptimized, and we recommend using datasets in standard formats like Parquet instead.

Btw there is a CLI tool to convert a loading script to parquet:

```
datasets-cli convert_to_parquet <dataset-name> --trust_remote_code
```

Closing in favor of #6832
Rebatch arrow iterables before formatted iterable

close https://github.com/huggingface/datasets/issues/7538 and https://github.com/huggingface/datasets/issues/7475

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7553). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq Our CI found an issue with this changeset causing a regression with shuffling iterable datasets 
<img width="884" alt="Screenshot 2025-05-07 at 9 16 52 AM" src="https://github.com/user-attachments/assets/bf7d9c7e-cc14-47da-8da6-d1a345992d7c" />
Enable xet in push to hub

follows https://github.com/huggingface/huggingface_hub/pull/3035

related to https://github.com/huggingface/datasets/issues/7526

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7552). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue with offline mode and partial dataset cached

### Describe the bug

Hi, 

a issue related to #4760 here when loading a single file from a dataset, unable to access it in offline mode afterwards

### Steps to reproduce the bug

```python
import os
# os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["HF_TOKEN"] = "xxxxxxxxxxxxxx"

import datasets

dataset_name = "uonlp/CulturaX"
data_files = "fr/fr_part_00038.parquet"

ds = datasets.load_dataset(dataset_name, split='train', data_files=data_files)
print(f"Dataset loaded   : {ds}")
```
Once the file has been cached, I rerun with the HF_HUB_OFFLINE activated an get this error : 
```
ValueError: Couldn't find cache for uonlp/CulturaX for config 'default-1e725f978350254e'
Available configs in the cache: ['default-2935e8cdcc21c613']
```

### Expected behavior

Should be able to access the previously cached files

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.4.0-215-generic-x86_64-with-glibc2.31
- Python version: 3.12.0
- `huggingface_hub` version: 0.27.0
- PyArrow version: 19.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1


Comments:
It seems the problem comes from builder.py / create_config_id()

On the first call, when the cache is empty we have
```
config_kwargs = {'data_files': {'train': ['hf://datasets/uonlp/CulturaX@6a8734bc69fefcbb7735f4f9250f43e4cd7a442e/fr/fr_part_00038.parquet']}}
```
leading to config_id beeing 'default-2935e8cdcc21c613'

then, on the second call, 
```
config_kwargs = {'data_files': 'fr/fr_part_00038.parquet'}
```
thus explaining why the hash is not the same, despite having the same parameter when calling load_dataset : data_files="fr/fr_part_00038.parquet"

Same behavior with version 3.5.1

Same issue when loading `google/IndicGenBench_flores_in` with `dataset==2.21.0` and `dataset==3.6.0` .




> It seems the problem comes from builder.py / create_config_id()
> 
> On the first call, when the cache is empty we have
> 
> ```
> config_kwargs = {'data_files': {'train': ['hf://datasets/uonlp/CulturaX@6a8734bc69fefcbb7735f4f9250f43e4cd7a442e/fr/fr_part_00038.parquet']}}
> ```
> 
> leading to config_id beeing 'default-2935e8cdcc21c613'
> 
> then, on the second call,
> 
> ```
> config_kwargs = {'data_files': 'fr/fr_part_00038.parquet'}
> ```
> 
> thus explaining why the hash is not the same, despite having the same parameter when calling load_dataset : data_files="fr/fr_part_00038.parquet"


I have identified that the issue indeed lies in the `data_files` within `config_kwargs`. 
The format and prefix of `data_files` differ depending on whether `HF_HUB_OFFLINE` is set, leading to different final `config_id` values. 
When I use other datasets without passing the `data_files` parameter, this issue does not occur.

A possible solution might be to standardize the formatting of `data_files` within the `create_config_id` function.
Avoid global umask for setting file mode.

This PR updates the method for setting the permissions on `cache_path` after calling `shutil.move`. The call to `shutil.move` may not preserve permissions if the source and destination are on different filesystems. Reading and resetting umask can cause race conditions, so directly read what permissions were set for the `temp_file` instead.

This fixes https://github.com/huggingface/datasets/issues/7536.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7547). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Large memory use when loading large datasets to a ZFS pool

### Describe the bug

When I load large parquet based datasets from the hub like `MLCommons/peoples_speech` using `load_dataset`, all my memory (500GB) is used and isn't released after loading, meaning that the process is terminated by the kernel if I try to load an additional dataset. This makes it impossible to train models using multiple large datasets.

### Steps to reproduce the bug

`uv run --with datasets==3.5.1 python`

```python
from datasets import load_dataset
load_dataset('MLCommons/peoples_speech', 'clean')
load_dataset('mozilla-foundation/common_voice_17_0', 'en')
```

### Expected behavior

I would expect that a lot less than 500GB of RAM would be required to load the dataset, or at least that the RAM usage would be cleared as soon as the dataset is loaded (and thus reside as a memory mapped file) such that other datasets can be loaded.

### Environment info

I am currently using the latest datasets==3.5.1 but I have had the same problem with multiple other versions.

Comments:
Hi ! datasets are memory mapped from disk, so they don't fill out your RAM. Not sure what's the source of your memory issue.

What kind of system are you using ? and what kind of disk ?

Well, the fact of the matter is that my RAM is getting filled out by running the given example, as shown in [this video](https://streamable.com/usb0ql).

My system is a GPU server running Ubuntu. The disk is a SATA SSD attached to the server using a backplane. It is formatted with ZFS, mounted in /cache, and my HF_HOME is set to /cache/hf

I really need this fixed, so I am more than willing to test out various suggestions you might have, or write a PR if we can figure out what is going on.

I'm not super familiar with ZFS, but it looks like it loads the data in memory when the files are memory mapped, which is an issue.

Maybe it's a caching mechanism ? Since `datasets` accesses every memory mapped file to read a small part (the metadata of the arrow record batches), maybe ZFS brings the whole files in memory for quicker subsequent reads. This is an antipattern when it comes to lazy loading datasets of that size though

This is the answer.

I tried changing my HF_HOME to an NFS share, and no RAM is then consumed loading the dataset.

I will try to see if I can find a way to configure the ZFS pool to not cache the files (disabling the ARC/primary cache didn't work), and if I do write the solution in this issue. If I can't I guess I have to reformat my cache drive.
Networked Pull Through Cache

### Feature request

Introduce a HF_DATASET_CACHE_NETWORK_LOCATION configuration (e.g. an environment variable) together with a companion network cache service.

Enable a three-tier cache lookup for datasets:

1. Local on-disk cache
2. Configurable network cache proxy
3. Official Hugging Face Hub

### Motivation

- Distributed training & ephemeral jobs: In high-performance or containerized clusters, relying solely on a local disk cache either becomes a streaming bottleneck or incurs a heavy cold-start penalty as each job must re-download datasets.
- Traffic & cost reduction: A pull-through network cache lets multiple consumers share a common cache layer, reducing duplicate downloads from the Hub and lowering egress costs.
- Better streaming adoption: By offloading repeat dataset pulls to a locally managed cache proxy, streaming workloads can achieve higher throughput and more predictable latency.
- Proven pattern: Similar proxy-cache solutions (e.g. Harbor’s Proxy Cache for Docker images) have demonstrated reliability and performance at scale: https://goharbor.io/docs/2.1.0/administration/configure-proxy-cache/

### Your contribution

I’m happy to draft the initial PR for adding HF_DATASET_CACHE_NETWORK_LOCATION support in datasets and sketch out a minimal cache-service prototype.

I have limited bandwidth so I would be looking for collaborators if anyone else is interested. 

Comments:
Add try_original_type to DatasetDict.map

This PR resolves #7472 for DatasetDict 

The previously merged PR #7483 added `try_original_type` to ArrowDataset, but DatasetDict misses `try_original_type`

Cc: @lhoestq 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7544). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sure! I just committed the changes

@lhoestq 
Let me know if there are other things to do before merge or other places to add `try_original_type` argument
The memory-disk mapping failure issue of the map function（resolved, but there are some suggestions.）

### Describe the bug

## bug
When the map function processes a large dataset, it temporarily stores the data in a cache file on the disk. After the data is stored, the memory occupied by it is released. Therefore, when using the map function to process a large-scale dataset, only a dataset space of the size of `writer_batch_size` will be occupied in memory.

However, I found that the map function does not actually reduce memory usage when I used it. At first, I thought there was a bug in the program, causing a memory leak—meaning the memory was not released after the data was stored in the cache. But later, I used a Linux command to check for recently modified files during program execution and found that no new files were created or modified. This indicates that the program did not store the dataset in the disk cache.
## bug solved
After modifying the parameters of the map function multiple times, I discovered the `cache_file_name` parameter. By changing it, the cache file can be stored in the specified directory. After making this change, I noticed that the cache file appeared. Initially, I found this quite incredible, but then I wondered if the cache file might have failed to be stored in a certain folder. This could be related to the fact that I don't have root privileges.

So, I delved into the source code of the map function to find out where the cache file would be stored by default. Eventually, I found the function `def _get_cache_file_path(self, fingerprint):`, which automatically generates the storage path for the cache file. The output was as follows: `/tmp/hf_datasets-j5qco9ug/cache-f2830487643b9cc2.arrow`. My hypothesis was confirmed: the lack of root privileges indeed prevented the cache file from being stored, which in turn prevented the release of memory. Therefore, changing the storage location to a folder where I have write access resolved the issue.

### Steps to reproduce the bug

my code
`train_data = train_data.map(process_fun, remove_columns=['image_name', 'question_type', 'concern', 'question', 'candidate_answers', 'answer'])`

### Expected behavior

Although my bug has been resolved, it still took me nearly a week to search for relevant information and debug the program. However, if a warning or error message about insufficient cache file write permissions could be provided during program execution, I might have been able to identify the cause more quickly. Therefore, I hope this aspect can be improved. I am documenting this bug here so that friends who encounter similar issues can solve their problems in a timely manner.

### Environment info

python: 3.10.15
datasets: 3.5.0

Comments:
support pyarrow 20

fix

```
TypeError: ArrayExtensionArray.to_pylist() got an unexpected keyword argument 'maps_as_pydicts'
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7540). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix IterableDataset state_dict shard_example_idx counting

# Fix IterableDataset's state_dict shard_example_idx reporting

## Description

This PR fixes issue #7475 where the `shard_example_idx` value in `IterableDataset`'s `state_dict()` always equals the number of samples in a shard, even if only a few examples have been consumed.

The issue is in the `_iter_arrow` method of the `ArrowExamplesIterable` class where it updates the `shard_example_idx` state by the full length of the batch (`len(pa_table)`) even when we're only partway through processing the examples. 

## Changes

Modified the `_iter_arrow` method of `ArrowExamplesIterable` to:
1. Track the actual number of examples processed
2. Only increment the `shard_example_idx` by the number of examples actually yielded
3. Handle partial batches correctly

## How to Test

I've included a simple test case that demonstrates the fix:

```python
from datasets import Dataset

# Create a test dataset
ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=1)

# Iterate through part of the dataset
for idx, example in enumerate(ds):
    print(example)
    if idx == 2:  # Stop after 3 examples (0, 1, 2)
        state_dict = ds.state_dict()
        print("Checkpoint state_dict:", state_dict)
        break

# Before the fix, the output would show shard_example_idx: 6
# After the fix, it shows shard_example_idx: 3, correctly reflecting the 3 processed examples
```

## Implementation Details

1. Added logic to track the number of examples actually seen in the current shard
2. Modified the state update to only count examples actually yielded
3. Improved handling of partial batches and skipped examples

This fix ensures that checkpointing and resuming works correctly with exactly the expected number of examples, rather than skipping ahead to the end of the batch.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7539). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi ! FYI I made a PR to fix https://github.com/huggingface/datasets/issues/7538 and it also fixed https://github.com/huggingface/datasets/issues/7475, so if I'm not mistaken this PR is not needed anymore
`IterableDataset` drops samples when resuming from a checkpoint

When resuming from a checkpoint, `IterableDataset` will drop samples if `num_shards % world_size == 0` and the underlying example supports `iter_arrow` and needs to be formatted. 

In that case, the `FormattedExamplesIterable` fetches a batch of samples from the child iterable's `iter_arrow` and yields them one by one (after formatting). However, the child increments the `shard_example_idx` counter (in its `iter_arrow`) before returning the batch for the whole batch size, which leads to a portion of samples being skipped if the iteration (of the parent iterable) is stopped mid-batch. 

Perhaps one way to avoid this would be by signalling the child iterable which samples (within the chunk) are processed by the parent and which are not, so that it can adjust the `shard_example_idx` counter accordingly. This would also mean the chunk needs to be sliced when resuming, but this is straightforward to implement.

The following is a minimal reproducer of the bug:
```python
from datasets import Dataset
from datasets.distributed import split_dataset_by_node

ds = Dataset.from_dict({"n": list(range(24))})
ds = ds.to_iterable_dataset(num_shards=4)

world_size = 4
rank = 0    
ds_rank = split_dataset_by_node(ds, rank, world_size)

it = iter(ds_rank)
examples = []
for idx, example in enumerate(it):
    examples.append(example)
    if idx == 2:
        state_dict = ds_rank.state_dict()
        break

ds_rank.load_state_dict(state_dict)
it_resumed = iter(ds_rank)
examples_resumed = examples[:]

for example in it:
    examples.append(example)

for example in it_resumed:
    examples_resumed.append(example)

print("ORIGINAL ITER EXAMPLES:", examples)
print("RESUMED ITER EXAMPLES:", examples_resumed)
```

Comments:
Thanks for reporting ! I fixed the issue using RebatchedArrowExamplesIterable before the formatted iterable
`datasets.map(..., num_proc=4)` multi-processing fails

The following code fails in python 3.11+ 

```python
tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])  
```
Error log:
```bash
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/multiprocess/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.12/dist-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.12/dist-packages/multiprocess/pool.py", line 114, in worker
task = get()
^^^^^
File "/usr/local/lib/python3.12/dist-packages/multiprocess/queues.py", line 371, in get
return _ForkingPickler.loads(res)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 327, in loads
return load(file, ignore, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 313, in load
return Unpickler(file, ignore=ignore, **kwds).load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 525, in load
obj = StockUnpickler.load(self)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 659, in _create_code
if len(args) == 16: return CodeType(*args)
^^^^^^^^^^^^^^^
TypeError: code() argument 13 must be str, not int
```

After upgrading dill to the latest 0.4.0 with "pip install --upgrade dill", it can pass. So it seems that there is a compatibility issue between dill 0.3.4 and python 3.11+, because python 3.10 works fine. 


Is the dill deterministic issue mentioned in https://github.com/huggingface/datasets/blob/main/setup.py#L117) still valid? Any plan to unpin? 


Comments:
related: https://github.com/huggingface/datasets/issues/7510

we need to do more tests to see if latest `dill` is deterministic
TensorFlow RaggedTensor Support (batch-level)

### Feature request

Hi,

Currently datasets does not support RaggedTensor output on batch-level.
When building a Object Detection Dataset (with TensorFlow) I need to enable RaggedTensors as that's how BBoxes & classes are expected from the Keras Model POV.

Currently there's a error thrown saying that "Nested Data is not supported".

It'd be very helpful if this was fixed! :)

### Motivation

Enabling Object Detection pipelines for TensorFlow.

### Your contribution

With guidance I'd happily help making the PR.

The current implementation with DataCollator and later enforcing `np.array` is the problematic part (at the end of `np_get_batch` in `tf_utils.py`). As `numpy` don't support "Raggednes"

Comments:
Keras doesn't support other inputs other than tf.data.Dataset objects ? it's a bit painful to have to support and maintain this kind of integration

Is there a way to use a `datasets.Dataset` with outputs formatted as tensors / ragged tensors instead ? like in https://huggingface.co/docs/datasets/use_with_tensorflow#dataset-format

I'll give it a try when I get the time. But quite sure I already tested the `with_format` approach.

Keras when using TF as backend converts the datasets into `tf.data.Dataset`, much like you do.

Hi @Lundez! Thanks for raising this — very valid point, especially for Object Detection use-cases.

You're right that np_get_batch currently enforces numpy batching, which breaks RaggedTensor support due to its inability to handle nested structures. This likely needs a redesign to allow TensorFlow-native batching in specific formats.

Before diving into a code change though, could you confirm:

Does `.with_format("tensorflow")` (without batching) return a `tf.data.Dataset` that works if batching is deferred to `model.fit()`?

Have you tried something like:

```python
tf_dataset = dataset.with_format("tensorflow").to_tf_dataset(
    columns=["image", "labels"],
    label_cols=None,
    batch_size=None  # No batching here
)
model.fit(tf_dataset.batch(BATCH_SIZE))  # Use RaggedTensor batching here
```

If this works, it might be worth updating the documentation rather than changing batching logic inside datasets itself.

That said, happy to explore changes if batching needs to be supported natively for RaggedTensor. Just flagging that it’d require some careful design due to existing numpy assumptions.

Hi, we've had to move on for now. 

We have actually also moved to dense tensors to make it possible to xla complie the training. 

But I'll check when I'm back from vacation which is far into the future. 

Thanks
Add custom fingerprint support to `from_generator`

This PR adds `dataset_id_suffix` parameter to 'Dataset.from_generator' function.

`Dataset.from_generator` function passes all of its arguments to `BuilderConfig.create_config_id`, including generator function itself. `BuilderConfig.create_config_id` function tries to hash all the args, which can take a large amount of time or even cause MemoryError if the dataset processed in a generator function is large enough.

This PR allows user to pass a custom fingerprint (`dataset_id_suffix`) to be used as a suffix in a dataset name instead of the one generated by hashing the args.

This PR is a possible solution of #7513 

Comments:
This is great !

What do you think of passing `config_id=` directly to the builder instead of just the suffix ? This would be a power user argument though, or for internal use. And in from_generator the new argument can be `fingerprint=` as in `Dataset.__init__()`

The `config_id` can be defined using something like `config_id = "default-fingerprint=" + fingerprint`

I feel ike this could make the Dataset API more coherent if we avoid introducing a new argument while we can juste use `fingerprint=`

@lhoestq could you please re-review the changes I made?

@lhoestq ping
I also added a simple test for the `fingerprint` parameter

@lhoestq could you please review the PR? I implemented the requested changes and added a test for the added `fingerprint` arg

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7533). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Document the HF_DATASETS_CACHE environment variable in the datasets cache documentation


This pull request updates the Datasets documentation to include the `HF_DATASETS_CACHE` environment variable. While the current documentation only mentions `HF_HOME` for overriding the default cache directory, `HF_DATASETS_CACHE` is also a supported and useful option for specifying a custom cache location for datasets stored in Arrow format.

This addition is based on the discussion in (https://github.com/huggingface/datasets/issues/7457), where users noted the absence of this variable in the documentation despite its functionality. The update adds a new section to `cache.mdx` that explains how to use `HF_DATASETS_CACHE` with an example.

This change aims to improve clarity and help users better manage their cache directories when working in shared environments or with limited local storage.

Closes #7457.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7532). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Your clarification in your comment at https://github.com/huggingface/datasets/issues/7480#issuecomment-2833640084 sounds great, would you like to update this PR to include it ?

Hi @lhoestq, I’ve updated the documentation to reflect the clarifications discussed in #7480. Let me know if anything else is needed!
How to solve "Spaces stuck in Building" problems

### Describe the bug

Public spaces may stuck in Building after restarting, error log as follows:

build error
Unexpected job error

ERROR: failed to push spaces-registry.huggingface.tech/spaces/*:cpu-*-*: unexpected status from HEAD request to https://spaces-registry.huggingface.tech/v2/spaces/*/manifests/cpu-*-*: 401 Unauthorized

### Steps to reproduce the bug

Restart space / Factory rebuild cannot avoid it

### Expected behavior

Fix this problem

### Environment info

no requirements.txt can still happen
python gradio spaces

Comments:
I'm facing the same issue—Space stuck in "Building" even after restart and Factory rebuild. Any fix?


> I'm facing the same issue—Space stuck in "Building" even after restart and Factory rebuild. Any fix?

Also see https://github.com/huggingface/huggingface_hub/issues/3019

I'm facing the same issue. The build fails with the same error, and restarting won't help. Is there a fix or ETA? 

Same error happens today, Nov 10th.
audio folder builder cannot detect custom split name

### Describe the bug

when using audio folder builder (`load_dataset("audiofolder", data_dir="/path/to/folder")`), it cannot detect custom split name other than train/validation/test

### Steps to reproduce the bug

i have the following folder structure
```
my_dataset/
├── train/
│   ├── lorem.wav
│   ├── …
│   └── metadata.csv
├── test/
│   ├── ipsum.wav
│   ├── …
│   └── metadata.csv
├── validation/
│   ├── dolor.wav
│   ├── …
│   └── metadata.csv
└── custom/
    ├── sit.wav
    ├── …
    └── metadata.csv
```
using `ds = load_dataset("audiofolder", data_dir="/path/to/my_dataset")`

### Expected behavior

i got `ds` with only 3 splits train/validation/test, whenever i rename train/validation/test folder it also disappear if i re-create `ds`

### Environment info

- `datasets` version: 3.5.0
- Platform: Windows-11-10.0.26100-SP0
- Python version: 3.12.8
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
Data Studio Error: Convert JSONL incorrectly

### Describe the bug

Hi there,

I uploaded a dataset here https://huggingface.co/datasets/V-STaR-Bench/V-STaR, but I found that Data Studio incorrectly convert the "bboxes" value for the whole dataset. Therefore, anyone who downloaded the dataset via the API would get the wrong "bboxes" value in the data file.

Could you help me address the issue?

Many thanks,



### Steps to reproduce the bug

The JSONL file of [V_STaR_test_release.jsonl](https://huggingface.co/datasets/V-STaR-Bench/V-STaR/blob/main/V_STaR_test_release.jsonl) has the correct values of every "bboxes" for each sample.

But in the Data Studio, we can see that the values of "bboxes" have changed, and load the dataset via API will also get the wrong values.

### Expected behavior

Fix the bug to correctly download my dataset.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.14.0-427.22.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2023.10.0

Comments:
Hi ! Your JSONL file is incompatible with Arrow / Parquet. Indeed in Arrow / Parquet every dict should have the same keys, while in your dataset the bboxes have varying keys.

This causes the Data Studio to treat the bboxes as if each row was missing the keys from other rows.

Feel free to take a look at the docs on object segmentation to see how to format a dataset with bboxes: https://huggingface.co/docs/datasets/object_detection
Auto-merge option for `convert-to-parquet`

### Feature request

Add a command-line option, e.g. `--auto-merge-pull-request` that enables automatic merging of the commits created by the `convert-to-parquet` tool.

### Motivation

Large datasets may result in dozens of PRs due to the splitting mechanism. Each of these has to be manually accepted via the website.

### Your contribution

Happy to look into submitting a PR if this is of interest to maintainers.

Comments:
Alternatively, there could be an option to switch from submitting PRs to just committing changes directly to `main`.

Why not, I'd be in favor of `--merge-pull-request` to call `HfApi().merge_pull_request()` at the end of the conversion :) feel free to open a PR if you'd like

#self-assign

Closing since convert to parquet has been removed... https://github.com/huggingface/datasets/pull/7592#issuecomment-3073053138
Faster downloads/uploads with Xet storage

![Image](https://github.com/user-attachments/assets/6e247f4a-d436-4428-a682-fe18ebdc73a9)

## Xet is out !

Over the past few weeks, Hugging Face’s [Xet Team](https://huggingface.co/xet-team) took a major step forward by [migrating the first Model and Dataset repositories off LFS and to Xet storage](https://huggingface.co/posts/jsulz/911431940353906).

See more information on the HF blog: https://huggingface.co/blog/xet-on-the-hub

You can already enable Xet on Hugging Face account to benefit from faster downloads and uploads :)

We finalized an official integration with the `huggingface_hub` library that means you get the benefits of Xet without any significant changes to your current workflow.


## Previous versions of `datasets`

For older versions of `datasets` you might see this warning in `push_to_hub()`:

```
Uploading files as bytes or binary IO objects is not supported by Xet Storage.
```

This means the `huggingface_hub` + Xet integration isn't enabled for your version of `datasets`.

You can fix this by updating to `datasets>=3.6.0` and `huggingface_hub>=0.31.0`

```
pip install -U datasets huggingface_hub
```

## The future

Stay tuned for more Xet optimizations, especially on [Xet-optimized Parquet](https://huggingface.co/blog/improve_parquet_dedupe)



Comments:
Fix indexing in split commit messages

When a large commit is split up, it seems the commit index in the message is zero-based while the total number is one-based. I came across this running `convert-to-parquet` and was wondering why there was no `6-of-6` commit. This PR fixes that by adding one to the commit index, so both are one-based.

Current behavior:
<img width="463" alt="Screenshot 2025-04-17 at 1 00 17 PM" src="https://github.com/user-attachments/assets/7f3d389e-cb92-405d-a3c2-f2b1cdf0cb79" />

Comments:
Hi ! this is expected and is coherent with other naming conventions in `datasets` such as parquet shards naming
fix: Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames (#7517)

## Task
Support bytes-like objects (bytes and bytearray) in Features classes

### Description
The `Features` classes only accept `bytes` objects for binary data, but not `bytearray`. This leads to errors when using `IterableDataset.from_spark()` with Spark DataFrames as they contain `bytearray` objects, even though both `bytes` and `bytearray` are valid [*bytes-like objects* in Python](https://docs.python.org/3/glossary.html#term-bytes-like-object).

### Changes
- Updated `Features` classes to accept both `bytes` and `bytearray` types for binary data fields.

### Reasoning
- `bytes` and `bytearray` serve the same purpose for binary data, with the only difference being mutability.
- Modifying the Spark iterator to convert `bytearray` to `bytes` would be a workaround, not a true fix. I think the correct solution is to accept all bytes-like objects as input.
- This approach is more robust and future-proof since Python 3.12+ provides a [standard way to check for buffer protocol](https://docs.python.org/3/c-api/buffer.html#bufferobjects).

### Testing
- Added tests to cover `bytearray` inputs for image features.

### Related Issues
- Fixes: #7517 

Comments:
@lhoestq let me know if you prefer to change the spark iterator so it outputs `bytes`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7521). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.