konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
support pyarrow 20

fix

```
TypeError: ArrayExtensionArray.to_pylist() got an unexpected keyword argument 'maps_as_pydicts'
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7540). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix IterableDataset state_dict shard_example_idx counting

# Fix IterableDataset's state_dict shard_example_idx reporting

## Description

This PR fixes issue #7475 where the `shard_example_idx` value in `IterableDataset`'s `state_dict()` always equals the number of samples in a shard, even if only a few examples have been consumed.

The issue is in the `_iter_arrow` method of the `ArrowExamplesIterable` class where it updates the `shard_example_idx` state by the full length of the batch (`len(pa_table)`) even when we're only partway through processing the examples. 

## Changes

Modified the `_iter_arrow` method of `ArrowExamplesIterable` to:
1. Track the actual number of examples processed
2. Only increment the `shard_example_idx` by the number of examples actually yielded
3. Handle partial batches correctly

## How to Test

I've included a simple test case that demonstrates the fix:

```python
from datasets import Dataset

# Create a test dataset
ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=1)

# Iterate through part of the dataset
for idx, example in enumerate(ds):
    print(example)
    if idx == 2:  # Stop after 3 examples (0, 1, 2)
        state_dict = ds.state_dict()
        print("Checkpoint state_dict:", state_dict)
        break

# Before the fix, the output would show shard_example_idx: 6
# After the fix, it shows shard_example_idx: 3, correctly reflecting the 3 processed examples
```

## Implementation Details

1. Added logic to track the number of examples actually seen in the current shard
2. Modified the state update to only count examples actually yielded
3. Improved handling of partial batches and skipped examples

This fix ensures that checkpointing and resuming works correctly with exactly the expected number of examples, rather than skipping ahead to the end of the batch.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7539). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi ! FYI I made a PR to fix https://github.com/huggingface/datasets/issues/7538 and it also fixed https://github.com/huggingface/datasets/issues/7475, so if I'm not mistaken this PR is not needed anymore
`IterableDataset` drops samples when resuming from a checkpoint

When resuming from a checkpoint, `IterableDataset` will drop samples if `num_shards % world_size == 0` and the underlying example supports `iter_arrow` and needs to be formatted. 

In that case, the `FormattedExamplesIterable` fetches a batch of samples from the child iterable's `iter_arrow` and yields them one by one (after formatting). However, the child increments the `shard_example_idx` counter (in its `iter_arrow`) before returning the batch for the whole batch size, which leads to a portion of samples being skipped if the iteration (of the parent iterable) is stopped mid-batch. 

Perhaps one way to avoid this would be by signalling the child iterable which samples (within the chunk) are processed by the parent and which are not, so that it can adjust the `shard_example_idx` counter accordingly. This would also mean the chunk needs to be sliced when resuming, but this is straightforward to implement.

The following is a minimal reproducer of the bug:
```python
from datasets import Dataset
from datasets.distributed import split_dataset_by_node

ds = Dataset.from_dict({"n": list(range(24))})
ds = ds.to_iterable_dataset(num_shards=4)

world_size = 4
rank = 0    
ds_rank = split_dataset_by_node(ds, rank, world_size)

it = iter(ds_rank)
examples = []
for idx, example in enumerate(it):
    examples.append(example)
    if idx == 2:
        state_dict = ds_rank.state_dict()
        break

ds_rank.load_state_dict(state_dict)
it_resumed = iter(ds_rank)
examples_resumed = examples[:]

for example in it:
    examples.append(example)

for example in it_resumed:
    examples_resumed.append(example)

print("ORIGINAL ITER EXAMPLES:", examples)
print("RESUMED ITER EXAMPLES:", examples_resumed)
```

Thanks for reporting ! I fixed the issue using RebatchedArrowExamplesIterable before the formatted iterable
`datasets.map(..., num_proc=4)` multi-processing fails

The following code fails in python 3.11+ 

```python
tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])  
```
Error log:
```bash
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/multiprocess/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.12/dist-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.12/dist-packages/multiprocess/pool.py", line 114, in worker
task = get()
^^^^^
File "/usr/local/lib/python3.12/dist-packages/multiprocess/queues.py", line 371, in get
return _ForkingPickler.loads(res)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 327, in loads
return load(file, ignore, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 313, in load
return Unpickler(file, ignore=ignore, **kwds).load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 525, in load
obj = StockUnpickler.load(self)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 659, in _create_code
if len(args) == 16: return CodeType(*args)
^^^^^^^^^^^^^^^
TypeError: code() argument 13 must be str, not int
```

After upgrading dill to the latest 0.4.0 with "pip install --upgrade dill", it can pass. So it seems that there is a compatibility issue between dill 0.3.4 and python 3.11+, because python 3.10 works fine. 


Is the dill deterministic issue mentioned in https://github.com/huggingface/datasets/blob/main/setup.py#L117) still valid? Any plan to unpin? 


related: https://github.com/huggingface/datasets/issues/7510

we need to do more tests to see if latest `dill` is deterministic
TensorFlow RaggedTensor Support (batch-level)

### Feature request

Hi,

Currently datasets does not support RaggedTensor output on batch-level.
When building a Object Detection Dataset (with TensorFlow) I need to enable RaggedTensors as that's how BBoxes & classes are expected from the Keras Model POV.

Currently there's a error thrown saying that "Nested Data is not supported".

It'd be very helpful if this was fixed! :)

### Motivation

Enabling Object Detection pipelines for TensorFlow.

### Your contribution

With guidance I'd happily help making the PR.

The current implementation with DataCollator and later enforcing `np.array` is the problematic part (at the end of `np_get_batch` in `tf_utils.py`). As `numpy` don't support "Raggednes"

Keras doesn't support other inputs other than tf.data.Dataset objects ? it's a bit painful to have to support and maintain this kind of integration

Is there a way to use a `datasets.Dataset` with outputs formatted as tensors / ragged tensors instead ? like in https://huggingface.co/docs/datasets/use_with_tensorflow#dataset-format

I'll give it a try when I get the time. But quite sure I already tested the `with_format` approach.

Keras when using TF as backend converts the datasets into `tf.data.Dataset`, much like you do.

Hi @Lundez! Thanks for raising this — very valid point, especially for Object Detection use-cases.

You're right that np_get_batch currently enforces numpy batching, which breaks RaggedTensor support due to its inability to handle nested structures. This likely needs a redesign to allow TensorFlow-native batching in specific formats.

Before diving into a code change though, could you confirm:

Does `.with_format("tensorflow")` (without batching) return a `tf.data.Dataset` that works if batching is deferred to `model.fit()`?

Have you tried something like:

```python
tf_dataset = dataset.with_format("tensorflow").to_tf_dataset(
    columns=["image", "labels"],
    label_cols=None,
    batch_size=None  # No batching here
)
model.fit(tf_dataset.batch(BATCH_SIZE))  # Use RaggedTensor batching here
```

If this works, it might be worth updating the documentation rather than changing batching logic inside datasets itself.

That said, happy to explore changes if batching needs to be supported natively for RaggedTensor. Just flagging that it’d require some careful design due to existing numpy assumptions.

Hi, we've had to move on for now. 

We have actually also moved to dense tensors to make it possible to xla complie the training. 

But I'll check when I'm back from vacation which is far into the future. 

Thanks
Add custom fingerprint support to `from_generator`

This PR adds `dataset_id_suffix` parameter to 'Dataset.from_generator' function.

`Dataset.from_generator` function passes all of its arguments to `BuilderConfig.create_config_id`, including generator function itself. `BuilderConfig.create_config_id` function tries to hash all the args, which can take a large amount of time or even cause MemoryError if the dataset processed in a generator function is large enough.

This PR allows user to pass a custom fingerprint (`dataset_id_suffix`) to be used as a suffix in a dataset name instead of the one generated by hashing the args.

This PR is a possible solution of #7513 

This is great !

What do you think of passing `config_id=` directly to the builder instead of just the suffix ? This would be a power user argument though, or for internal use. And in from_generator the new argument can be `fingerprint=` as in `Dataset.__init__()`

The `config_id` can be defined using something like `config_id = "default-fingerprint=" + fingerprint`

I feel ike this could make the Dataset API more coherent if we avoid introducing a new argument while we can juste use `fingerprint=`

I looked into this issue and the original cause makes total sense — hashing a large generator is clearly inefficient and fragile for big datasets.

PR #7533 looks like a robust and flexible solution! It cleanly separates the fingerprinting responsibility by letting users pass `fingerprint=` (now `config_id=`), which avoids hashing heavy objects like generators but still preserves caching logic.


@lhoestq could you please re-review the changes I made?
Document the HF_DATASETS_CACHE environment variable in the datasets cache documentation


This pull request updates the Datasets documentation to include the `HF_DATASETS_CACHE` environment variable. While the current documentation only mentions `HF_HOME` for overriding the default cache directory, `HF_DATASETS_CACHE` is also a supported and useful option for specifying a custom cache location for datasets stored in Arrow format.

This addition is based on the discussion in (https://github.com/huggingface/datasets/issues/7457), where users noted the absence of this variable in the documentation despite its functionality. The update adds a new section to `cache.mdx` that explains how to use `HF_DATASETS_CACHE` with an example.

This change aims to improve clarity and help users better manage their cache directories when working in shared environments or with limited local storage.

Closes #7457.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7532). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Your clarification in your comment at https://github.com/huggingface/datasets/issues/7480#issuecomment-2833640084 sounds great, would you like to update this PR to include it ?

Hi @lhoestq, I’ve updated the documentation to reflect the clarifications discussed in #7480. Let me know if anything else is needed!
How to solve "Spaces stuck in Building" problems

### Describe the bug

Public spaces may stuck in Building after restarting, error log as follows:

build error
Unexpected job error

ERROR: failed to push spaces-registry.huggingface.tech/spaces/*:cpu-*-*: unexpected status from HEAD request to https://spaces-registry.huggingface.tech/v2/spaces/*/manifests/cpu-*-*: 401 Unauthorized

### Steps to reproduce the bug

Restart space / Factory rebuild cannot avoid it

### Expected behavior

Fix this problem

### Environment info

no requirements.txt can still happen
python gradio spaces

I'm facing the same issue—Space stuck in "Building" even after restart and Factory rebuild. Any fix?


> I'm facing the same issue—Space stuck in "Building" even after restart and Factory rebuild. Any fix?

Also see https://github.com/huggingface/huggingface_hub/issues/3019

I'm facing the same issue. The build fails with the same error, and restarting won't help. Is there a fix or ETA?
audio folder builder cannot detect custom split name

### Describe the bug

when using audio folder builder (`load_dataset("audiofolder", data_dir="/path/to/folder")`), it cannot detect custom split name other than train/validation/test

### Steps to reproduce the bug

i have the following folder structure
```
my_dataset/
├── train/
│   ├── lorem.wav
│   ├── …
│   └── metadata.csv
├── test/
│   ├── ipsum.wav
│   ├── …
│   └── metadata.csv
├── validation/
│   ├── dolor.wav
│   ├── …
│   └── metadata.csv
└── custom/
    ├── sit.wav
    ├── …
    └── metadata.csv
```
using `ds = load_dataset("audiofolder", data_dir="/path/to/my_dataset")`

### Expected behavior

i got `ds` with only 3 splits train/validation/test, whenever i rename train/validation/test folder it also disappear if i re-create `ds`

### Environment info

- `datasets` version: 3.5.0
- Platform: Windows-11-10.0.26100-SP0
- Python version: 3.12.8
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
Data Studio Error: Convert JSONL incorrectly

### Describe the bug

Hi there,

I uploaded a dataset here https://huggingface.co/datasets/V-STaR-Bench/V-STaR, but I found that Data Studio incorrectly convert the "bboxes" value for the whole dataset. Therefore, anyone who downloaded the dataset via the API would get the wrong "bboxes" value in the data file.

Could you help me address the issue?

Many thanks,



### Steps to reproduce the bug

The JSONL file of [V_STaR_test_release.jsonl](https://huggingface.co/datasets/V-STaR-Bench/V-STaR/blob/main/V_STaR_test_release.jsonl) has the correct values of every "bboxes" for each sample.

But in the Data Studio, we can see that the values of "bboxes" have changed, and load the dataset via API will also get the wrong values.

### Expected behavior

Fix the bug to correctly download my dataset.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.14.0-427.22.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2023.10.0

Hi ! Your JSONL file is incompatible with Arrow / Parquet. Indeed in Arrow / Parquet every dict should have the same keys, while in your dataset the bboxes have varying keys.

This causes the Data Studio to treat the bboxes as if each row was missing the keys from other rows.

Feel free to take a look at the docs on object segmentation to see how to format a dataset with bboxes: https://huggingface.co/docs/datasets/object_detection
Auto-merge option for `convert-to-parquet`

### Feature request

Add a command-line option, e.g. `--auto-merge-pull-request` that enables automatic merging of the commits created by the `convert-to-parquet` tool.

### Motivation

Large datasets may result in dozens of PRs due to the splitting mechanism. Each of these has to be manually accepted via the website.

### Your contribution

Happy to look into submitting a PR if this is of interest to maintainers.

Alternatively, there could be an option to switch from submitting PRs to just committing changes directly to `main`.

Why not, I'd be in favor of `--merge-pull-request` to call `HfApi().merge_pull_request()` at the end of the conversion :) feel free to open a PR if you'd like

#self-assign
Fix indexing in split commit messages

When a large commit is split up, it seems the commit index in the message is zero-based while the total number is one-based. I came across this running `convert-to-parquet` and was wondering why there was no `6-of-6` commit. This PR fixes that by adding one to the commit index, so both are one-based.

Current behavior:
<img width="463" alt="Screenshot 2025-04-17 at 1 00 17 PM" src="https://github.com/user-attachments/assets/7f3d389e-cb92-405d-a3c2-f2b1cdf0cb79" />

Hi ! this is expected and is coherent with other naming conventions in `datasets` such as parquet shards naming
fix: Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames (#7517)

## Task
Support bytes-like objects (bytes and bytearray) in Features classes

### Description
The `Features` classes only accept `bytes` objects for binary data, but not `bytearray`. This leads to errors when using `IterableDataset.from_spark()` with Spark DataFrames as they contain `bytearray` objects, even though both `bytes` and `bytearray` are valid [*bytes-like objects* in Python](https://docs.python.org/3/glossary.html#term-bytes-like-object).

### Changes
- Updated `Features` classes to accept both `bytes` and `bytearray` types for binary data fields.

### Reasoning
- `bytes` and `bytearray` serve the same purpose for binary data, with the only difference being mutability.
- Modifying the Spark iterator to convert `bytearray` to `bytes` would be a workaround, not a true fix. I think the correct solution is to accept all bytes-like objects as input.
- This approach is more robust and future-proof since Python 3.12+ provides a [standard way to check for buffer protocol](https://docs.python.org/3/c-api/buffer.html#bufferobjects).

### Testing
- Added tests to cover `bytearray` inputs for image features.

### Related Issues
- Fixes: #7517 

@lhoestq let me know if you prefer to change the spark iterator so it outputs `bytes`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7521). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update items in the dataset without `map`

### Feature request

I would like to be able to update items in my dataset without affecting all rows. At least if there was a range option, I would be able to process those items, save the dataset, and then continue.

If I am supposed to split the dataset first, that is not clear, since the docs suggest that any of those functions returns a new object, so I don't think I can do that.

### Motivation

I am applying an extremely time-consuming function to each item in my `Dataset`. Unfortunately, datasets only supports updating values via `map`, so if my computer dies in the middle of this long-running process, I lose all progress. This is far from ideal. I would like to use `datasets` throughout this processing, but this limitation is now forcing me to write my own dataset format just to do this intermediary operation.

It would be less intuitive but I suppose I could split and then concatenate the dataset before saving? But this feels very inefficient.

### Your contribution

I can test the feature.

Hello!

Have you looked at `Dataset.shard`? [Docs](https://huggingface.co/docs/datasets/en/process#shard)

Using this method you could break your dataset in N shards. Apply `map` on each shard and concatenate them back.
num_proc parallelization works only for first ~10s.

### Describe the bug

When I try to load an already downloaded dataset with num_proc=64, the speed is very high for the first 10-20 seconds acheiving 30-40K samples / s, and 100% utilization for all cores but it soon drops to <= 1000 with almost 0% utilization for most cores. 

### Steps to reproduce the bug

```
// download dataset with cli
!huggingface-cli download --repo-type dataset timm/imagenet-1k-wds --max-workers 32

from datasets import load_dataset
ds = load_dataset("timm/imagenet-1k-wds", num_proc=64)
```


### Expected behavior

100% core utilization throughout. 

### Environment info

Azure A100-80GB, 16 cores VM

![Image](https://github.com/user-attachments/assets/69d00fe3-d720-4474-9439-21e046d85034)

Hi, can you check if the processes are still alive ? It's a bit weird because `datasets` does check if processes crash and return an error in that case

Thank you for reverting quickly. I digged a bit, and realized my disk's IOPS is also limited - which is causing this. will check further and report if it's an issue of hf datasets' side or mine.
Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames

### Describe the bug

When using `IterableDataset.from_spark()` with a Spark DataFrame containing image data, the `Image` feature class fails to properly process this data type, causing an `AttributeError: 'bytearray' object has no attribute 'get'`

### Steps to reproduce the bug

1. Create a Spark DataFrame with a column containing image data as bytearray objects
2. Define a Feature schema with an Image feature
3. Create an IterableDataset using `IterableDataset.from_spark()`
4. Attempt to iterate through the dataset

```
from pyspark.sql import SparkSession
from datasets import Dataset, IterableDataset, Features, Image, Value

# initialize spark
spark = SparkSession.builder.appName("MinimalRepro").getOrCreate()

# create spark dataframe
data = [(0, open("image.png", "rb").read())]
df = spark.createDataFrame(data, "idx: int, image: binary")

# convert to dataset
features = Features({"idx": Value("int64"), "image": Image()})
ds = Dataset.from_spark(df, features=features)
ds_iter = IterableDataset.from_spark(df, features=features)

# iterate
print(next(iter(ds)))
print(next(iter(ds_iter)))
```

### Expected behavior

The features should work on `IterableDataset` the same way they work on `Dataset`

### Environment info

- `datasets` version: 3.5.0
- Platform: macOS-15.3.2-arm64-arm-64bit
- Python version: 3.12.7
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


Hi ! The `Image()` type accepts either
- a `bytes` object containing the image bytes
- a `str` object containing the image path
- a `PIL.Image` object

but it doesn't support `bytearray`, maybe you can convert to `bytes` beforehand ?

Hi @lhoestq, 
converting to bytes is certainly possible and would work around the error. However, the core issue is that `Dataset` and `IterableDataset` behave differently with the features.

I’d be happy to work on a fix for this issue.

I see, that's an issue indeed. Feel free to ping me if I can help with reviews or any guidance

If it can help, the code that takes a Spark DataFrame and iterates on the rows for `IterableDataset` is here: 

https://github.com/huggingface/datasets/blob/6a96bf313085d7538a999b929a550e14e1d406c9/src/datasets/packaged_modules/spark/spark.py#L49-L53

#self-assign
`concatenate_datasets` does not preserve Pytorch format for IterableDataset

### Describe the bug

When concatenating datasets with `concatenate_datasets`, I would expect the resulting combined dataset to be in the same format as the inputs (assuming it's consistent). This is indeed the behavior when combining `Dataset`, but not when combining `IterableDataset`. Specifically, when applying `concatenate_datasets` to a list of `IterableDataset` in Pytorch format (i.e. using `.with_format(Pytorch)`), the output `IterableDataset` is not in Pytorch format.

### Steps to reproduce the bug

```
import datasets
ds = datasets.Dataset.from_dict({"a": [1,2,3]})
iterable_ds = ds.to_iterable_dataset()
datasets.concatenate_datasets([ds.with_format("torch")]) # <- this preserves Pytorch format
datasets.concatenate_datasets([iterable_ds.with_format("torch")]) # <- this does NOT preserves Pytorch format
```

### Expected behavior

Pytorch format should be preserved when combining IterableDataset in Pytorch format.

### Environment info

datasets==3.5.0, Python 3.11.11, torch==2.2.2

Hi ! Oh indeed it would be cool to return the same format in that case. Would you like to submit a PR ? The function that does the concatenation is here:

https://github.com/huggingface/datasets/blob/90e5bf8a8599b625d6103ee5ac83b98269991141/src/datasets/iterable_dataset.py#L3375-L3380

Thank you for the pointer, @lhoestq ! See #7522
Incompatibile dill version (0.3.9) in datasets 2.18.0 - 3.5.0

### Describe the bug

Datasets 2.18.0 - 3.5.0 has a dependency on dill < 0.3.9. This causes errors with dill >= 0.3.9.

Could you please take a look into it and make it compatible?

### Steps to reproduce the bug

1. Install setuptools >= 2.18.0
2. Install dill >=0.3.9
3. Run pip check
4. Output:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires dill<0.3.9,>=0.3.0, but you have dill 0.3.9 which is incompatible.

### Expected behavior

Pip install both libraries without any errors

### Environment info

Datasets version: 2.18 - 3.5
Python: 3.11

Hi ! We can bump `dill` to 0.3.9 if we make sure it's deterministic and doesn't break the caching mechanism in `datasets`.

Would you be interested in opening a PR ? Then we can run the CI to see if it works

Hi!. Yeah I can do it. Should I make any changes besides dill versions?

There are probably some usage of internal functions from `dill` that we'll need to update in `datasets`

If you run `pytest tests/test_fingerprint.py` you should already have a good idea of what works and what doesn't.
But feel free to open a PR anyway, this way we can run the full CI and see the results


Hi, sorry for no response from my side. I will try to do it today.

Created pull request: [LINK](https://github.com/huggingface/datasets/pull/7535)
Tried to run tests by using command you have send and got few errors:

![Image](https://github.com/user-attachments/assets/acbf1feb-4dd1-416e-a118-c91abe0d188b)

Thanks for running the test ! So it appears we have two issues to fix:
1. 'log' is not defined: it seems an internal `dill` function has disappeared, so we should adapt the `datasets` code that was using it
2. there are some hashes mismatches, which means `dill` doesn't seem to output the same dump when passed the same ipython function twice, or the same function but located at a different line in a python file
Iterating over Image feature columns is extremely slow

We are trying to load datasets where the image column stores `PIL.PngImagePlugin.PngImageFile` images. However, iterating over these datasets is extremely slow. 
What I have found:
1. It is the presence of the image column that causes the slowdown. Removing the column from the dataset results in blazingly fast (as expected) times
2. It is ~2x faster to iterate when the column contains a single image as opposed to a list of images i.e., the feature is a Sequence of Image objects. We often need multiple images per sample, so we need to work with a list of images
3. It is ~17x faster to store paths to PNG files and load them using `PIL.Image.open`, as opposed to iterating over a `Dataset` with an Image column, and ~30x faster compared to `Sequence` of `Image`s. See a simple script below with an openly available dataset.


It would be great to understand the standard practices for storing and loading multimodal datasets (image + text). 

https://huggingface.co/docs/datasets/en/image_load seems a bit underdeveloped? (e.g., `dataset.decode` only works with `IterableDataset`, but it's not clear from the doc)

Thanks!

```python
from datasets import load_dataset, load_from_disk
from PIL import Image
from pathlib import Path

ds = load_dataset("getomni-ai/ocr-benchmark")

for idx, sample in enumerate(ds["test"]):
    image = sample["image"]
    image.save(f"/tmp/ds_files/images/image_{idx}.png")

ds.save_to_disk("/tmp/ds_columns")

# Remove the 'image' column
ds["test"] = ds["test"].remove_columns(["image"])

# Create image paths for each sample
image_paths = [f"images/image_{idx}.png" for idx in range(len(ds["test"]))]

# Add the 'image_path' column to the dataset
ds["test"] = ds["test"].add_column("image_path", image_paths)

# Save the updated dataset
ds.save_to_disk("/tmp/ds_files")
files_path = Path("/tmp/ds_files")
column_path = Path("/tmp/ds_columns")

# load and benchmark
ds_file = load_from_disk(files_path)
ds_column = load_from_disk(column_path)
import time


images_files = []
start = time.time()
for idx in range(len(ds_file["test"])):
    image_path = files_path / ds_file["test"][idx]["image_path"]
    image = Image.open(image_path)
    images_files.append(image)
end = time.time()
print(f"Time taken to load images from files: {end - start} seconds")

# Time taken to load images from files: 1.2364635467529297 seconds


images_column = []
start = time.time()
for idx in range(len(ds_column["test"])):
    images_column.append(ds_column["test"][idx]["image"])
end = time.time()
print(f"Time taken to load images from columns: {end - start} seconds")

# Time taken to load images from columns: 20.49347186088562 seconds
```

Hi ! Could it be because the `Image()` type in dataset does `image  = Image.open(image_path)` and also `image.load()` which actually loads the image data in memory ? This is needed to avoid too many open files issues, see https://github.com/huggingface/datasets/issues/3985

Yes, that seems to be it. For my purposes, I've cast the column to `Image(decode=False)`, and only load the images when necessary, which is much much faster
Front-end statistical data quantity deviation

### Describe the bug

While browsing the dataset at https://huggingface.co/datasets/NeuML/wikipedia-20250123, I noticed that a dataset with nearly 7M entries was estimated to be only 4M in size—almost half the actual amount. According to the post-download loading and the dataset_info (https://huggingface.co/datasets/NeuML/wikipedia-20250123/blob/main/train/dataset_info.json), the true data volume is indeed close to 7M. This significant discrepancy could mislead users when sorting datasets by row count. Why not directly retrieve this information from dataset_info?

Not sure if this is the right place to report this bug, but leaving it here for the team's awareness.

Hi ! the format of this dataset is not supported by the Dataset Viewer. It looks like this dataset was saved using `save_to_disk()` which is meant for local storage / easy reload without compression, not for sharing online.
`load_dataset` of size 40GB creates a cache of >720GB

Hi there,

I am trying to load a dataset from the Hugging Face Hub and split it into train and validation splits. Somehow, when I try to do it with `load_dataset`, it exhausts my disk quota. So, I tried manually downloading the parquet files from the hub and loading them as follows:

```python
 ds = DatasetDict(
        {
            "train": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=0, to=NUM_TRAIN, unit="abs"),  # type: ignore
            ),
            "validation": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=NUM_TRAIN, unit="abs"),  # type: ignore
            )
        }
    )

```

which still strangely creates 720GB of cache. In addition, if I remove the raw parquet file folder (`f"{local_dir}/{tok}"` in this example), I am not able to load anything. So, I am left wondering what this cache is doing. Am I missing something? Is there a solution to this problem?

Thanks a lot in advance for your help!

A related issue: https://github.com/huggingface/transformers/issues/10204#issue-809007443.

---

Python: 3.11.11
datasets: 3.5.0


Hi ! Parquet is a compressed format. When you load a dataset, it uncompresses the Parquet data into Arrow data on your disk. That's why you can indeed end up with 720GB of uncompressed data on disk. The uncompression is needed to enable performant dataset objects (especially for random access).

To save some storage you can instead load the dataset with `streaming=True`. This way you get an `IterableDataset` that reads the Parquet data iteratively without ever writing to disk.

PS: `ReadInstruction` might not be implemented for `streaming=True`, if it's the case you can use `ds.take()` and `ds.skip()` instead

Hi @lhoestq, thanks a lot for your answer. This makes perfect sense. I will try using the streaming mode. Closing the issue.
Nested Feature raises ArrowNotImplementedError: Unsupported cast using function cast_struct

### Describe the bug

`datasets.Features` seems to be unable to handle json file that contains fields of `list[dict]`.

### Steps to reproduce the bug

```json
// test.json
{"a": 1, "b": [{"c": 2, "d": 3}, {"c": 4, "d": 5}]}
{"a": 5, "b": [{"c": 7, "d": 8}, {"c": 9, "d": 10}]}
```

```python
import json
from datasets import Dataset, Features, Value, Sequence, load_dataset

annotation_feature = Features({
    "a": Value("int32"),
    "b": Sequence({
        "c": Value("int32"),
        "d": Value("int32"),
    }),
})
annotation_dataset = load_dataset(
    "json", 
    data_files="test.json",
    features=annotation_feature
)
```

```
ArrowNotImplementedError: Unsupported cast from list<item: struct<c: int32, d: int32>> to struct using function cast_struct

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[46], line 11
      2 from datasets import Dataset, Features, Value, Sequence, load_dataset
      4 annotation_feature = Features({
      5     "a": Value("int32"),
      6     "b": Sequence({
   (...)      9     }),
     10 })
---> 11 annotation_dataset = load_dataset(
     12     "json", 
     13     data_files="test.json",
     14     features=annotation_feature
     15 )
```

### Expected behavior

A `datasets.Datasets` instance should be initialized.

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.11.0-21-generic-x86_64-with-glibc2.39
- Python version: 3.11.11
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Solved by the default `load_dataset(features)` parameters. Do not use `Sequence` for the `list` in `list[any]` json schema, just simply use `[]`. For example, `"b": Sequence({...})` fails but `"b": [{...}]` works fine.
Make `with_format` correctly indicate that a `Dataset` is compatible with PyTorch's `Dataset` class

### Feature request

Currently `datasets` does not correctly indicate to the Python type-checker (e.g. `pyright` / `Pylance`) that the output of `with_format` is compatible with PyTorch's `Dataloader` since it does not indicate that the HuggingFace `Dataset` is compatible with the PyTorch `Dataset` class. It would be great if we could get the typing to work nicely.

### Motivation

To avoid casting types in our Python code.

### Your contribution

I would be happy to contribute a PR if this is something that may be accepted and could work with the current approach.
This doesn't have to be for just PyTorch, but I imagine that the same thing would be useful for `tensorflow` and such, but we only have a need for PyTorch at this stage.

Does the torch `DataLoader` really require the dataset to be a subclass of `torch.utils.data.Dataset` ? Or is there a simpler type we could use ?

PS: also note that a dataset without `with_format()` can also be used in a torch `DataLoader` . Calling `with_format("torch")` simply makes the output of the dataset torch Tensors in an efficient way.
Added cache dirs to load and file_utils

When adding "cache_dir" to datasets.load_dataset, the cache_dir gets lost in the function calls, changing the cache dir to the default path. This fixes a few of these instances.

hi ! the `hf_hub_download` cache_dir is a different cache directory than the one for `datasets`.

`hf_hub_download` uses the `huggingface_hub` cache which is located in by default in `~/.cache/huggingface/hub`, while `datasets` uses a different cache for Arrow files and map() results `~/.cache/huggingface/datasets`

Is there a way to change the default cache directory for both of these on calling load_dataset? Currently, cache_dir makes dealing with where I want files to go a bit confusing as the documentation doesn't mention it only relocates.../datasets and not .../hub.

You can set `HF_HOME` which is the common parent directory for those two caches. Or individually `HF_DATASETS_CACHE` and `HF_HUB_CACHE`

Got it. Can this be added to the documentation for load_dataset and related functions to avoid confusion with cache_dir?

done in https://github.com/huggingface/datasets/pull/7532 :)
How to convert videos to images?

### Feature request

Does someone know how to return the images from videos?

### Motivation

I am trying to use openpi(https://github.com/Physical-Intelligence/openpi) to finetune my Lerobot dataset(V2.0 and V2.1). I find that although the codedaset is v2.0, they are different. It seems like Lerobot V2.0 has two version, one is data include images infos and another one is separate to data and videos.

Does someone know how to return the images from videos?





Hi ! there is some documentation here on how to read video frames: https://huggingface.co/docs/datasets/video_load
Json builder: Allow features to override problematic Arrow types

### Feature request

In the JSON builder, use explicitly requested feature types before or while converting to Arrow.

### Motivation

Working with JSON datasets is really hard because of Arrow.  At the very least, it seems like it should be possible to work-around these problems by explicitly setting problematic columns's types.  But it seems like this is not possible because the features are only used *after* converting to arrow.

Here's a simple example where the Arrow error could potentially be avoided by converting the column to a string: https://colab.research.google.com/drive/16QHRdbUwKSrpwVfGwu8V8AHr8v2dv0dt?usp=sharing

### Your contribution

Maybe with some guidance.  I'm not very familiar with arrow or pandas.

Hi ! It would be cool indeed, currently the JSON data are generally loaded here: 

https://github.com/huggingface/datasets/blob/90e5bf8a8599b625d6103ee5ac83b98269991141/src/datasets/packaged_modules/json/json.py#L137-L140

Maybe we can pass a Arrow `schema` to avoid errors ?
Columns in the dataset obtained though load_dataset do not correspond to the one in the dataset viewer since 3.4.0

### Describe the bug

I have noticed that on my dataset named [BrunoHays/Accueil_UBS](https://huggingface.co/datasets/BrunoHays/Accueil_UBS), since the version 3.4.0, every column except audio is missing when I load the dataset.

Interestingly, the dataset viewer still shows the correct columns

### Steps to reproduce the bug

```python
from datasets import load_dataset
ds = load_dataset("BrunoHays/Accueil_UBS", streaming=True)
print(next(iter(ds["test"])).keys())
```
With datasets >= 3.4.0: 
-> dict_keys(['audio'])
With datasets == 3.3.2:
-> dict_keys(['audio', 'id', 'speaker', 'sentence', 'raw_sentence', 'start_timestamp', 'end_timestamp', 'overlap'])


### Expected behavior

All the columns should be present

### Environment info

- `datasets` version: 3.3.2
- Platform: macOS-14.6.1-x86_64-i386-64bit
- Python version: 3.10.15
- `huggingface_hub` version: 0.30.1
- PyArrow version: 16.1.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0


Hi, the dataset viewer shows all the possible columns and their types, but `load_dataset()` iterates through all the columns that you defined. It seems that you only have one column (‘audio’) defined in your dataset because when I ran `print(ds.column_names)`, the only name I got was “audio”. You need to clearly define all the other features of the dataset as columns to enable your original code to work. Furthermore, you can run this code to print out all the features of your dataset: 
```python
from datasets import load_dataset_builder
ds_builder = load_dataset_builder("BrunoHays/Accueil_UBS")
print(ds_builder.info.features)
```


@phoebecd 
Good catch, even in datasets<3.4.0, the only feature is "audio".
This datasets follows the [audio folder](https://huggingface.co/docs/datasets/en/audio_dataset#audiofolder) structure with metadata.csv.
Maybe I missed something or there is a bug when having and audio_folder with a metadata file

What do you think @lhoestq ?

I opened a PR to fix the issue :) https://huggingface.co/datasets/BrunoHays/Accueil_UBS/discussions/2

We expect the metadata file to be in the <split>/ folder now to allow one CSV metadata file per split. But in the PR I just added a manual configuration instead of moving the file and updating all the relative paths it contains.
Broken links in pdf loading documentation

### Describe the bug

Hi, just a couple of small issues I ran into while reading the docs for [loading pdf data](https://huggingface.co/docs/datasets/main/en/document_load):

1. The link for the [`Create a pdf dataset`](https://huggingface.co/docs/datasets/main/en/document_load#pdffolder) points to https://huggingface.co/docs/datasets/main/en/pdf_dataset instead of https://huggingface.co/docs/datasets/main/en/document_dataset and hence gives a 404 error.

2. At the top of the page, it's mentioned that to work with pdf datasets we need to have the `pdfplumber` package installed but the link to its installation guide points to `pytorch/vision` [installation instructions](https://github.com/pytorch/vision#installation) instead of `pdfplumber`'s [guide](https://github.com/jsvine/pdfplumber#installation)

I love the work on enabling pdf dataset support and these small tweaks would help everyone navigate the docs better. Thanks!

### Steps to reproduce the bug

The issue is on the [Load Document Data](https://huggingface.co/docs/datasets/main/en/document_load) page of the datasets docs.

### Expected behavior

1. For solving the first issue, I went through the [source .mdx code](https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx?plain=1#L188) of the datasets docs and found that the link is pointing to `./pdf_dataset` instead of `./document_dataset`

2. For the second issue, I went through the [source .mdx code](https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx?plain=1#L13) of the datasets docs and found that the link is `pytorch/vision` [installation instructions](https://github.com/pytorch/vision#installation) instead of `pdfplumber`'s [guide](https://github.com/jsvine/pdfplumber#installation)

Just replacing these two links should fix the bugs

### Environment info

datasets v3.5.0 (main at the time of writing)

thanks for reporting ! I fixed the links, the docs will be updated in the next release
push_to_hub does not upload videos

### Describe the bug

Hello,

I would like to upload a video dataset (some .mp4 files and some segments within them), i.e. rows correspond to subsequences from videos. Videos might be referenced by several rows.

I created a dataset locally and it references the videos and the video readers can read them correctly. I use push_to_hub() to upload the dataset to the hub.

Expectation: A user uses `load_dataset` and can load the videos.

However, the videos seem to be just referenced via paths on the computer and not uploaded to the hub. Therefore a target user cannot load the videos in the dataset.

### Steps to reproduce the bug

1. create a video dataset with paths e.g. { ["videos"]: [path1, path2, ...]}
2. dataset.push_to_hub
3. on a different computer (or same pc if relative paths are used in a different folder):
```
dataset = load_dataset("siplab/egosim", split="train")
video = dataset[0]["video_head"]
```

3. will fail

### Expected behavior

Expectation: A user uses `load_dataset` and can load the videos.

### Environment info

datasets                       3.1.0
Python 3.8.18

Hi ! the `Video` type is still experimental, and in particular `push_to_hub` doesn't upload videos at the moment (only the paths).

There is an open question to either upload the videos inside the Parquet files, or rather have them as separate files (which is great to enable remote seeking/streaming)
fix: loading of datasets from Disk(#7373)

Fixes dataset loading from disk by ensuring that memory maps and streams are properly closed. 


For more details, see https://github.com/huggingface/datasets/issues/7373.

@nepfaff Could you confirm if this fixes the issue for you? I checked Memray, and everything looked good on my end.

Install: `pip install git+https://github.com/sam-hey/datasets.git@fix/concatenate_datasets`


Will aim to get to this soon. I don't have a rapid testing pipeline setup but need to wait for some AWS nodes to become free

I now set up a small experiment:

```python
# Log initial RAM usage
    process = psutil.Process(os.getpid())
    initial_ram = process.memory_info().rss / (1024 * 1024)  # Convert to MB
    logging.info(f"Initial RAM usage: {initial_ram:.2f} MB")

    chunk_datasets = [
        Dataset.load_from_disk(dataset_path, keep_in_memory=False) for _ in range(N)
    ]
    combined_dataset = concatenate_datasets(chunk_datasets)

    # Log final RAM usage
    final_ram = process.memory_info().rss / (1024 * 1024)  # Convert to MB
    ram_diff = final_ram - initial_ram
    logging.info(f"Final RAM usage: {final_ram:.2f} MB")
    logging.info(f"RAM usage increase: {ram_diff:.2f} MB")
```

The RAM usage is linearly correlated with `N` on datasets master!

For my test dataset:
- N=5 => RAM usage increase: 26302.91 MB
- N=10 => RAM usage increase: 52315.18 MB
- N=20 => RAM usage increase: 104510.65 MB
- N=40 => RAM usage increase: 209166.30 MB

Unfortunately, your patch doesn't seem to change this:
```bash
pip install git+https://github.com/sam-hey/datasets.git@fix/concatenate_datasets
pip list | grep datasets
datasets                 3.5.1.dev0
```
Gives exactly the same RAM statistics.

**Edit:** The results are a bit flawed as the memory increase all seems to come from `Dataset.load_from_disk(dataset_path, keep_in_memory=False)` here (which I don't think should happen either?) and not from `concatenate_datasets`. This seems different from my large-scale setup that runs out of memory during `concatenate_datasets` but I don't seem to be able to replicate this here...

Thanks a lot, @nepfaff, for taking a look at this! It seems that `concatenate_datasets()` is fixed with this PR. I can also confirm that loading a large number of files requires significant memory. However, as I understand it, this is expected/a bug since the memory consumption stems from `pa.memory_map()`, which returns a memory-mapped file.

This behavior might be related to this bug: https://github.com/apache/arrow/issues/34423 

<img width="1728" alt="Screenshot 2025-04-03 at 16 01 11" src="https://github.com/user-attachments/assets/475691d8-3aba-4d7e-b8ef-5e7552c70b14" />


Great ! have you tested that it also fixes the memory issue in your case @iamollas ?

Happy to know that it works for you @sam-hey ! Looking forward to merging this

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7489). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Write pdf in map

Fix this error when mapping a PDF dataset

```
pyarrow.lib.ArrowInvalid: Could not convert <pdfplumber.pdf.PDF object at 0x13498ee40> with type PDF: did not recognize Python value type when inferring an Arrow data type
```

and also let map() outputs be lists of images or pdfs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7487). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.