Sulak2020/github-issues-multirepo-datasets-Sulakshana huggingface dataset:
Title: Update items in the dataset without `map`
State: open
Labels: name: enhancement
description: New feature or request

### Feature request

I would like to be able to update items in my dataset without affecting all rows. At least if there was a range option, I would be able to process those items, save the dataset, and then continue.

If I am supposed to split the dataset first, that is not clear, since the docs suggest that any of those functions returns a new object, so I don't think I can do that.

### Motivation

I am applying an extremely time-consuming function to each item in my `Dataset`. Unfortunately, datasets only supports updating values via `map`, so if my computer dies in the middle of this long-running process, I lose all progress. This is far from ideal. I would like to use `datasets` throughout this processing, but this limitation is now forcing me to write my own dataset format just to do this intermediary operation.

It would be less intuitive but I suppose I could split and then concatenate the dataset before saving? But this feels very inefficient.

### Your contribution

I can test the feature.

Comments: Hello!

Have you looked at `Dataset.shard`? [Docs](https://huggingface.co/docs/datasets/en/process#shard)

Using this method you could break your dataset in N shards. Apply `map` on each shard and concatenate them back.
Title: num_proc parallelization works only for first ~10s.
State: open

### Describe the bug

When I try to load an already downloaded dataset with num_proc=64, the speed is very high for the first 10-20 seconds acheiving 30-40K samples / s, and 100% utilization for all cores but it soon drops to <= 1000 with almost 0% utilization for most cores. 

### Steps to reproduce the bug

```
// download dataset with cli
!huggingface-cli download --repo-type dataset timm/imagenet-1k-wds --max-workers 32

from datasets import load_dataset
ds = load_dataset("timm/imagenet-1k-wds", num_proc=64)
```


### Expected behavior

100% core utilization throughout. 

### Environment info

Azure A100-80GB, 16 cores VM

![Image](https://github.com/user-attachments/assets/69d00fe3-d720-4474-9439-21e046d85034)

Comments: Hi, can you check if the processes are still alive ? It's a bit weird because `datasets` does check if processes crash and return an error in that caseThank you for reverting quickly. I digged a bit, and realized my disk's IOPS is also limited - which is causing this. will check further and report if it's an issue of hf datasets' side or mine.
Title: Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames
State: closed

### Describe the bug

When using `IterableDataset.from_spark()` with a Spark DataFrame containing image data, the `Image` feature class fails to properly process this data type, causing an `AttributeError: 'bytearray' object has no attribute 'get'`

### Steps to reproduce the bug

1. Create a Spark DataFrame with a column containing image data as bytearray objects
2. Define a Feature schema with an Image feature
3. Create an IterableDataset using `IterableDataset.from_spark()`
4. Attempt to iterate through the dataset

```
from pyspark.sql import SparkSession
from datasets import Dataset, IterableDataset, Features, Image, Value

# initialize spark
spark = SparkSession.builder.appName("MinimalRepro").getOrCreate()

# create spark dataframe
data = [(0, open("image.png", "rb").read())]
df = spark.createDataFrame(data, "idx: int, image: binary")

# convert to dataset
features = Features({"idx": Value("int64"), "image": Image()})
ds = Dataset.from_spark(df, features=features)
ds_iter = IterableDataset.from_spark(df, features=features)

# iterate
print(next(iter(ds)))
print(next(iter(ds_iter)))
```

### Expected behavior

The features should work on `IterableDataset` the same way they work on `Dataset`

### Environment info

- `datasets` version: 3.5.0
- Platform: macOS-15.3.2-arm64-arm-64bit
- Python version: 3.12.7
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


Comments: Hi ! The `Image()` type accepts either
- a `bytes` object containing the image bytes
- a `str` object containing the image path
- a `PIL.Image` object

but it doesn't support `bytearray`, maybe you can convert to `bytes` beforehand ?Hi @lhoestq, 
converting to bytes is certainly possible and would work around the error. However, the core issue is that `Dataset` and `IterableDataset` behave differently with the features.

I’d be happy to work on a fix for this issue.I see, that's an issue indeed. Feel free to ping me if I can help with reviews or any guidance

If it can help, the code that takes a Spark DataFrame and iterates on the rows for `IterableDataset` is here: 

https://github.com/huggingface/datasets/blob/6a96bf313085d7538a999b929a550e14e1d406c9/src/datasets/packaged_modules/spark/spark.py#L49-L53#self-assign
Title: unsloth/DeepSeek-R1-Distill-Qwen-32B server error
State: closed

### Describe the bug

hfhubhttperror: 500 server error: internal server error for url: https://huggingface.co/api/models/unsloth/deepseek-r1-distill-qwen-32b-bnb-4bit/commits/main (request id: root=1-67fe23fa-3a2150eb444c2a823c388579;de3aed68-c397-4da5-94d4-6565efd3b919) internal error - we're working hard to fix this as soon as possible!

### Steps to reproduce the bug

unsloth/DeepSeek-R1-Distill-Qwen-32B server error

### Expected behavior

Network repair

### Environment info

The web side is also unavailable
Title: `concatenate_datasets` does not preserve Pytorch format for IterableDataset
State: closed

### Describe the bug

When concatenating datasets with `concatenate_datasets`, I would expect the resulting combined dataset to be in the same format as the inputs (assuming it's consistent). This is indeed the behavior when combining `Dataset`, but not when combining `IterableDataset`. Specifically, when applying `concatenate_datasets` to a list of `IterableDataset` in Pytorch format (i.e. using `.with_format(Pytorch)`), the output `IterableDataset` is not in Pytorch format.

### Steps to reproduce the bug

```
import datasets
ds = datasets.Dataset.from_dict({"a": [1,2,3]})
iterable_ds = ds.to_iterable_dataset()
datasets.concatenate_datasets([ds.with_format("torch")]) # <- this preserves Pytorch format
datasets.concatenate_datasets([iterable_ds.with_format("torch")]) # <- this does NOT preserves Pytorch format
```

### Expected behavior

Pytorch format should be preserved when combining IterableDataset in Pytorch format.

### Environment info

datasets==3.5.0, Python 3.11.11, torch==2.2.2

Comments: Hi ! Oh indeed it would be cool to return the same format in that case. Would you like to submit a PR ? The function that does the concatenation is here:

https://github.com/huggingface/datasets/blob/90e5bf8a8599b625d6103ee5ac83b98269991141/src/datasets/iterable_dataset.py#L3375-L3380Thank you for the pointer, @lhoestq ! See #7522
Title: Do not hash `generator` in `BuilderConfig.create_config_id`
State: closed

`Dataset.from_generator` function passes all of its arguments to `BuilderConfig.create_config_id`, including generator function itself. `BuilderConfig.create_config_id` function tries to hash all the args, and hashing a `generator` can take a large amount of time or even cause MemoryError if the dataset processed in a generator function is large enough.

Maybe we should pop generator from `config_kwargs_to_add_to_suffix` before hashing to avoid it.

There is a more detailed description of the problem this PR solves in #7513
Title: Incompatibile dill version (0.3.9) in datasets 2.18.0 - 3.5.0
State: closed

### Describe the bug

Datasets 2.18.0 - 3.5.0 has a dependency on dill < 0.3.9. This causes errors with dill >= 0.3.9.

Could you please take a look into it and make it compatible?

### Steps to reproduce the bug

1. Install setuptools >= 2.18.0
2. Install dill >=0.3.9
3. Run pip check
4. Output:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires dill<0.3.9,>=0.3.0, but you have dill 0.3.9 which is incompatible.

### Expected behavior

Pip install both libraries without any errors

### Environment info

Datasets version: 2.18 - 3.5
Python: 3.11

Comments: Hi ! We can bump `dill` to 0.3.9 if we make sure it's deterministic and doesn't break the caching mechanism in `datasets`.

Would you be interested in opening a PR ? Then we can run the CI to see if it worksHi!. Yeah I can do it. Should I make any changes besides dill versions?There are probably some usage of internal functions from `dill` that we'll need to update in `datasets`

If you run `pytest tests/test_fingerprint.py` you should already have a good idea of what works and what doesn't.
But feel free to open a PR anyway, this way we can run the full CI and see the results
Hi, sorry for no response from my side. I will try to do it today.Created pull request: [LINK](https://github.com/huggingface/datasets/pull/7535)
Tried to run tests by using command you have send and got few errors:

![Image](https://github.com/user-attachments/assets/acbf1feb-4dd1-416e-a118-c91abe0d188b)Thanks for running the test ! So it appears we have two issues to fix:
1. 'log' is not defined: it seems an internal `dill` function has disappeared, so we should adapt the `datasets` code that was using it
2. there are some hashes mismatches, which means `dill` doesn't seem to output the same dump when passed the same ipython function twice, or the same function but located at a different line in a python file
Title: Iterating over Image feature columns is extremely slow
State: open

We are trying to load datasets where the image column stores `PIL.PngImagePlugin.PngImageFile` images. However, iterating over these datasets is extremely slow. 
What I have found:
1. It is the presence of the image column that causes the slowdown. Removing the column from the dataset results in blazingly fast (as expected) times
2. It is ~2x faster to iterate when the column contains a single image as opposed to a list of images i.e., the feature is a Sequence of Image objects. We often need multiple images per sample, so we need to work with a list of images
3. It is ~17x faster to store paths to PNG files and load them using `PIL.Image.open`, as opposed to iterating over a `Dataset` with an Image column, and ~30x faster compared to `Sequence` of `Image`s. See a simple script below with an openly available dataset.


It would be great to understand the standard practices for storing and loading multimodal datasets (image + text). 

https://huggingface.co/docs/datasets/en/image_load seems a bit underdeveloped? (e.g., `dataset.decode` only works with `IterableDataset`, but it's not clear from the doc)

Thanks!

```python
from datasets import load_dataset, load_from_disk
from PIL import Image
from pathlib import Path

ds = load_dataset("getomni-ai/ocr-benchmark")

for idx, sample in enumerate(ds["test"]):
    image = sample["image"]
    image.save(f"/tmp/ds_files/images/image_{idx}.png")

ds.save_to_disk("/tmp/ds_columns")

# Remove the 'image' column
ds["test"] = ds["test"].remove_columns(["image"])

# Create image paths for each sample
image_paths = [f"images/image_{idx}.png" for idx in range(len(ds["test"]))]

# Add the 'image_path' column to the dataset
ds["test"] = ds["test"].add_column("image_path", image_paths)

# Save the updated dataset
ds.save_to_disk("/tmp/ds_files")
files_path = Path("/tmp/ds_files")
column_path = Path("/tmp/ds_columns")

# load and benchmark
ds_file = load_from_disk(files_path)
ds_column = load_from_disk(column_path)
import time


images_files = []
start = time.time()
for idx in range(len(ds_file["test"])):
    image_path = files_path / ds_file["test"][idx]["image_path"]
    image = Image.open(image_path)
    images_files.append(image)
end = time.time()
print(f"Time taken to load images from files: {end - start} seconds")

# Time taken to load images from files: 1.2364635467529297 seconds


images_column = []
start = time.time()
for idx in range(len(ds_column["test"])):
    images_column.append(ds_column["test"][idx]["image"])
end = time.time()
print(f"Time taken to load images from columns: {end - start} seconds")

# Time taken to load images from columns: 20.49347186088562 seconds
```

Comments: Hi ! Could it be because the `Image()` type in dataset does `image  = Image.open(image_path)` and also `image.load()` which actually loads the image data in memory ? This is needed to avoid too many open files issues, see https://github.com/huggingface/datasets/issues/3985Yes, that seems to be it. For my purposes, I've cast the column to `Image(decode=False)`, and only load the images when necessary, which is much much faster
Title: Front-end statistical data quantity deviation
State: open

### Describe the bug

While browsing the dataset at https://huggingface.co/datasets/NeuML/wikipedia-20250123, I noticed that a dataset with nearly 7M entries was estimated to be only 4M in size—almost half the actual amount. According to the post-download loading and the dataset_info (https://huggingface.co/datasets/NeuML/wikipedia-20250123/blob/main/train/dataset_info.json), the true data volume is indeed close to 7M. This significant discrepancy could mislead users when sorting datasets by row count. Why not directly retrieve this information from dataset_info?

Not sure if this is the right place to report this bug, but leaving it here for the team's awareness.

Comments: Hi ! the format of this dataset is not supported by the Dataset Viewer. It looks like this dataset was saved using `save_to_disk()` which is meant for local storage / easy reload without compression, not for sharing online.
Title: `load_dataset` of size 40GB creates a cache of >720GB
State: closed

Hi there,

I am trying to load a dataset from the Hugging Face Hub and split it into train and validation splits. Somehow, when I try to do it with `load_dataset`, it exhausts my disk quota. So, I tried manually downloading the parquet files from the hub and loading them as follows:

```python
 ds = DatasetDict(
        {
            "train": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=0, to=NUM_TRAIN, unit="abs"),  # type: ignore
            ),
            "validation": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=NUM_TRAIN, unit="abs"),  # type: ignore
            )
        }
    )

```

which still strangely creates 720GB of cache. In addition, if I remove the raw parquet file folder (`f"{local_dir}/{tok}"` in this example), I am not able to load anything. So, I am left wondering what this cache is doing. Am I missing something? Is there a solution to this problem?

Thanks a lot in advance for your help!

A related issue: https://github.com/huggingface/transformers/issues/10204#issue-809007443.

---

Python: 3.11.11
datasets: 3.5.0


Comments: Hi ! Parquet is a compressed format. When you load a dataset, it uncompresses the Parquet data into Arrow data on your disk. That's why you can indeed end up with 720GB of uncompressed data on disk. The uncompression is needed to enable performant dataset objects (especially for random access).

To save some storage you can instead load the dataset with `streaming=True`. This way you get an `IterableDataset` that reads the Parquet data iteratively without ever writing to disk.

PS: `ReadInstruction` might not be implemented for `streaming=True`, if it's the case you can use `ds.take()` and `ds.skip()` insteadHi @lhoestq, thanks a lot for your answer. This makes perfect sense. I will try using the streaming mode. Closing the issue.
Title: Nested Feature raises ArrowNotImplementedError: Unsupported cast using function cast_struct
State: closed

### Describe the bug

`datasets.Features` seems to be unable to handle json file that contains fields of `list[dict]`.

### Steps to reproduce the bug

```json
// test.json
{"a": 1, "b": [{"c": 2, "d": 3}, {"c": 4, "d": 5}]}
{"a": 5, "b": [{"c": 7, "d": 8}, {"c": 9, "d": 10}]}
```

```python
import json
from datasets import Dataset, Features, Value, Sequence, load_dataset

annotation_feature = Features({
    "a": Value("int32"),
    "b": Sequence({
        "c": Value("int32"),
        "d": Value("int32"),
    }),
})
annotation_dataset = load_dataset(
    "json", 
    data_files="test.json",
    features=annotation_feature
)
```

```
ArrowNotImplementedError: Unsupported cast from list<item: struct<c: int32, d: int32>> to struct using function cast_struct

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[46], line 11
      2 from datasets import Dataset, Features, Value, Sequence, load_dataset
      4 annotation_feature = Features({
      5     "a": Value("int32"),
      6     "b": Sequence({
   (...)      9     }),
     10 })
---> 11 annotation_dataset = load_dataset(
     12     "json", 
     13     data_files="test.json",
     14     features=annotation_feature
     15 )
```

### Expected behavior

A `datasets.Datasets` instance should be initialized.

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.11.0-21-generic-x86_64-with-glibc2.39
- Python version: 3.11.11
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments: Solved by the default `load_dataset(features)` parameters. Do not use `Sequence` for the `list` in `list[any]` json schema, just simply use `[]`. For example, `"b": Sequence({...})` fails but `"b": [{...}]` works fine.
Title: Make `with_format` correctly indicate that a `Dataset` is compatible with PyTorch's `Dataset` class
State: open
Labels: name: enhancement
description: New feature or request

### Feature request

Currently `datasets` does not correctly indicate to the Python type-checker (e.g. `pyright` / `Pylance`) that the output of `with_format` is compatible with PyTorch's `Dataloader` since it does not indicate that the HuggingFace `Dataset` is compatible with the PyTorch `Dataset` class. It would be great if we could get the typing to work nicely.

### Motivation

To avoid casting types in our Python code.

### Your contribution

I would be happy to contribute a PR if this is something that may be accepted and could work with the current approach.
This doesn't have to be for just PyTorch, but I imagine that the same thing would be useful for `tensorflow` and such, but we only have a need for PyTorch at this stage.

Comments: Does the torch `DataLoader` really require the dataset to be a subclass of `torch.utils.data.Dataset` ? Or is there a simpler type we could use ?

PS: also note that a dataset without `with_format()` can also be used in a torch `DataLoader` . Calling `with_format("torch")` simply makes the output of the dataset torch Tensors in an efficient way.
Title: Added cache dirs to load and file_utils
State: closed

When adding "cache_dir" to datasets.load_dataset, the cache_dir gets lost in the function calls, changing the cache dir to the default path. This fixes a few of these instances.

Comments: hi ! the `hf_hub_download` cache_dir is a different cache directory than the one for `datasets`.

`hf_hub_download` uses the `huggingface_hub` cache which is located in by default in `~/.cache/huggingface/hub`, while `datasets` uses a different cache for Arrow files and map() results `~/.cache/huggingface/datasets`Is there a way to change the default cache directory for both of these on calling load_dataset? Currently, cache_dir makes dealing with where I want files to go a bit confusing as the documentation doesn't mention it only relocates.../datasets and not .../hub.You can set `HF_HOME` which is the common parent directory for those two caches. Or individually `HF_DATASETS_CACHE` and `HF_HUB_CACHE`Got it. Can this be added to the documentation for load_dataset and related functions to avoid confusion with cache_dir?done in https://github.com/huggingface/datasets/pull/7532 :)
Title: Extreme memory bandwidth.
State: open

### Describe the bug

When I use hf datasets on 4 GPU with 40 workers I get some extreme memory bandwidth of constant ~3GB/s. 

However, if I wrap the dataset in `IterableDataset`, this issue is gone and the data also loads way faster (4x faster training on 1 worker).

It seems like the workers don't share memory and basically duplicate the data 4x40.



### Steps to reproduce the bug

Trainer arguments:
```
    dataloader_pin_memory=True,
    dataloader_num_workers=40,
    dataloader_prefetch_factor=2,
    dataloader_persistent_workers=True,
```
Call trainer:
```
trainer = Trainer(
    model=model,
    args=train_args,
    train_dataset=load_from_disk('..').with_fromat('torch'),
)
```
The dataset has 600GB and consists of 1225 files.

### Expected behavior

The optimal bandwidth should be 100MB/s to keep up with GPU.

### Environment info

Linux
Python 3.11
datasets==3.2.0
Title: How to convert videos to images?
State: open
Labels: name: enhancement
description: New feature or request

### Feature request

Does someone know how to return the images from videos?

### Motivation

I am trying to use openpi(https://github.com/Physical-Intelligence/openpi) to finetune my Lerobot dataset(V2.0 and V2.1). I find that although the codedaset is v2.0, they are different. It seems like Lerobot V2.0 has two version, one is data include images infos and another one is separate to data and videos.

Does someone know how to return the images from videos?





Comments: Hi ! there is some documentation here on how to read video frames: https://huggingface.co/docs/datasets/video_load
Title: Json builder: Allow features to override problematic Arrow types
State: open
Labels: name: enhancement
description: New feature or request

### Feature request

In the JSON builder, use explicitly requested feature types before or while converting to Arrow.

### Motivation

Working with JSON datasets is really hard because of Arrow.  At the very least, it seems like it should be possible to work-around these problems by explicitly setting problematic columns's types.  But it seems like this is not possible because the features are only used *after* converting to arrow.

Here's a simple example where the Arrow error could potentially be avoided by converting the column to a string: https://colab.research.google.com/drive/16QHRdbUwKSrpwVfGwu8V8AHr8v2dv0dt?usp=sharing

### Your contribution

Maybe with some guidance.  I'm not very familiar with arrow or pandas.

Comments: Hi ! It would be cool indeed, currently the JSON data are generally loaded here: 

https://github.com/huggingface/datasets/blob/90e5bf8a8599b625d6103ee5ac83b98269991141/src/datasets/packaged_modules/json/json.py#L137-L140

Maybe we can pass a Arrow `schema` to avoid errors ?