helmo/github-issues huggingface dataset:
Update items in the dataset without `map`
State: open | Author: mashdragon (NONE)
Labels: name: enhancement

### Feature request

I would like to be able to update items in my dataset without affecting all rows. At least if there was a range option, I would be able to process those items, save the dataset, and then continue.

If I am supposed to split the dataset first, that is not clear, since the docs suggest that any of those functions returns a new object, so I don't think I can do that.

### Motivation

I am applying an extremely time-consuming function to each item in my `Dataset`. Unfortunately, datasets only supports updating values via `map`, so if my computer dies in the middle of this long-running process, I lose all progress. This is far from ideal. I would like to use `datasets` throughout this processing, but this limitation is now forcing me to write my own dataset format just to do this intermediary operation.

It would be less intuitive but I suppose I could split and then concatenate the dataset before saving? But this feels very inefficient.

### Your contribution

I can test the feature.

Comments: Hello!

Have you looked at `Dataset.shard`? [Docs](https://huggingface.co/docs/datasets/en/process#shard)

Using this method you could break your dataset in N shards. Apply `map` on each shard and concatenate them back.
num_proc parallelization works only for first ~10s.
State: open | Author: pshishodiaa (NONE)

### Describe the bug

When I try to load an already downloaded dataset with num_proc=64, the speed is very high for the first 10-20 seconds acheiving 30-40K samples / s, and 100% utilization for all cores but it soon drops to <= 1000 with almost 0% utilization for most cores. 

### Steps to reproduce the bug

```
// download dataset with cli
!huggingface-cli download --repo-type dataset timm/imagenet-1k-wds --max-workers 32

from datasets import load_dataset
ds = load_dataset("timm/imagenet-1k-wds", num_proc=64)
```


### Expected behavior

100% core utilization throughout. 

### Environment info

Azure A100-80GB, 16 cores VM

![Image](https://github.com/user-attachments/assets/69d00fe3-d720-4474-9439-21e046d85034)

Comments: Hi, can you check if the processes are still alive ? It's a bit weird because `datasets` does check if processes crash and return an error in that case

Thank you for reverting quickly. I digged a bit, and realized my disk's IOPS is also limited - which is causing this. will check further and report if it's an issue of hf datasets' side or mine.
Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames
State: closed | Author: giraffacarp (CONTRIBUTOR)

### Describe the bug

When using `IterableDataset.from_spark()` with a Spark DataFrame containing image data, the `Image` feature class fails to properly process this data type, causing an `AttributeError: 'bytearray' object has no attribute 'get'`

### Steps to reproduce the bug

1. Create a Spark DataFrame with a column containing image data as bytearray objects
2. Define a Feature schema with an Image feature
3. Create an IterableDataset using `IterableDataset.from_spark()`
4. Attempt to iterate through the dataset

```
from pyspark.sql import SparkSession
from datasets import Dataset, IterableDataset, Features, Image, Value

# initialize spark
spark = SparkSession.builder.appName("MinimalRepro").getOrCreate()

# create spark dataframe
data = [(0, open("image.png", "rb").read())]
df = spark.createDataFrame(data, "idx: int, image: binary")

# convert to dataset
features = Features({"idx": Value("int64"), "image": Image()})
ds = Dataset.from_spark(df, features=features)
ds_iter = IterableDataset.from_spark(df, features=features)

# iterate
print(next(iter(ds)))
print(next(iter(ds_iter)))
```

### Expected behavior

The features should work on `IterableDataset` the same way they work on `Dataset`

### Environment info

- `datasets` version: 3.5.0
- Platform: macOS-15.3.2-arm64-arm-64bit
- Python version: 3.12.7
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


Comments: Hi ! The `Image()` type accepts either
- a `bytes` object containing the image bytes
- a `str` object containing the image path
- a `PIL.Image` object

but it doesn't support `bytearray`, maybe you can convert to `bytes` beforehand ?

Hi @lhoestq, 
converting to bytes is certainly possible and would work around the error. However, the core issue is that `Dataset` and `IterableDataset` behave differently with the features.

I’d be happy to work on a fix for this issue.

I see, that's an issue indeed. Feel free to ping me if I can help with reviews or any guidance

If it can help, the code that takes a Spark DataFrame and iterates on the rows for `IterableDataset` is here: 

https://github.com/huggingface/datasets/blob/6a96bf313085d7538a999b929a550e14e1d406c9/src/datasets/packaged_modules/spark/spark.py#L49-L53

#self-assign
unsloth/DeepSeek-R1-Distill-Qwen-32B server error
State: closed | Author: Editor-1 (NONE)

### Describe the bug

hfhubhttperror: 500 server error: internal server error for url: https://huggingface.co/api/models/unsloth/deepseek-r1-distill-qwen-32b-bnb-4bit/commits/main (request id: root=1-67fe23fa-3a2150eb444c2a823c388579;de3aed68-c397-4da5-94d4-6565efd3b919) internal error - we're working hard to fix this as soon as possible!

### Steps to reproduce the bug

unsloth/DeepSeek-R1-Distill-Qwen-32B server error

### Expected behavior

Network repair

### Environment info

The web side is also unavailable
`concatenate_datasets` does not preserve Pytorch format for IterableDataset
State: closed | Author: francescorubbo (CONTRIBUTOR)

### Describe the bug

When concatenating datasets with `concatenate_datasets`, I would expect the resulting combined dataset to be in the same format as the inputs (assuming it's consistent). This is indeed the behavior when combining `Dataset`, but not when combining `IterableDataset`. Specifically, when applying `concatenate_datasets` to a list of `IterableDataset` in Pytorch format (i.e. using `.with_format(Pytorch)`), the output `IterableDataset` is not in Pytorch format.

### Steps to reproduce the bug

```
import datasets
ds = datasets.Dataset.from_dict({"a": [1,2,3]})
iterable_ds = ds.to_iterable_dataset()
datasets.concatenate_datasets([ds.with_format("torch")]) # <- this preserves Pytorch format
datasets.concatenate_datasets([iterable_ds.with_format("torch")]) # <- this does NOT preserves Pytorch format
```

### Expected behavior

Pytorch format should be preserved when combining IterableDataset in Pytorch format.

### Environment info

datasets==3.5.0, Python 3.11.11, torch==2.2.2

Comments: Hi ! Oh indeed it would be cool to return the same format in that case. Would you like to submit a PR ? The function that does the concatenation is here:

https://github.com/huggingface/datasets/blob/90e5bf8a8599b625d6103ee5ac83b98269991141/src/datasets/iterable_dataset.py#L3375-L3380

Thank you for the pointer, @lhoestq ! See #7522
Do not hash `generator` in `BuilderConfig.create_config_id`
State: closed | Author: simonreise (NONE)

`Dataset.from_generator` function passes all of its arguments to `BuilderConfig.create_config_id`, including generator function itself. `BuilderConfig.create_config_id` function tries to hash all the args, and hashing a `generator` can take a large amount of time or even cause MemoryError if the dataset processed in a generator function is large enough.

Maybe we should pop generator from `config_kwargs_to_add_to_suffix` before hashing to avoid it.

There is a more detailed description of the problem this PR solves in #7513
Incompatibile dill version (0.3.9) in datasets 2.18.0 - 3.5.0
State: open | Author: JGrel (NONE)

### Describe the bug

Datasets 2.18.0 - 3.5.0 has a dependency on dill < 0.3.9. This causes errors with dill >= 0.3.9.

Could you please take a look into it and make it compatible?

### Steps to reproduce the bug

1. Install setuptools >= 2.18.0
2. Install dill >=0.3.9
3. Run pip check
4. Output:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires dill<0.3.9,>=0.3.0, but you have dill 0.3.9 which is incompatible.

### Expected behavior

Pip install both libraries without any errors

### Environment info

Datasets version: 2.18 - 3.5
Python: 3.11

Comments: Hi ! We can bump `dill` to 0.3.9 if we make sure it's deterministic and doesn't break the caching mechanism in `datasets`.

Would you be interested in opening a PR ? Then we can run the CI to see if it works

Hi!. Yeah I can do it. Should I make any changes besides dill versions?

There are probably some usage of internal functions from `dill` that we'll need to update in `datasets`

If you run `pytest tests/test_fingerprint.py` you should already have a good idea of what works and what doesn't.
But feel free to open a PR anyway, this way we can run the full CI and see the results


Hi, sorry for no response from my side. I will try to do it today.

Created pull request: [LINK](https://github.com/huggingface/datasets/pull/7535)
Tried to run tests by using command you have send and got few errors:

![Image](https://github.com/user-attachments/assets/acbf1feb-4dd1-416e-a118-c91abe0d188b)

Thanks for running the test ! So it appears we have two issues to fix:
1. 'log' is not defined: it seems an internal `dill` function has disappeared, so we should adapt the `datasets` code that was using it
2. there are some hashes mismatches, which means `dill` doesn't seem to output the same dump when passed the same ipython function twice, or the same function but located at a different line in a python file
Front-end statistical data quantity deviation
State: open | Author: rangehow (NONE)

### Describe the bug

While browsing the dataset at https://huggingface.co/datasets/NeuML/wikipedia-20250123, I noticed that a dataset with nearly 7M entries was estimated to be only 4M in size—almost half the actual amount. According to the post-download loading and the dataset_info (https://huggingface.co/datasets/NeuML/wikipedia-20250123/blob/main/train/dataset_info.json), the true data volume is indeed close to 7M. This significant discrepancy could mislead users when sorting datasets by row count. Why not directly retrieve this information from dataset_info?

Not sure if this is the right place to report this bug, but leaving it here for the team's awareness.

Comments: Hi ! the format of this dataset is not supported by the Dataset Viewer. It looks like this dataset was saved using `save_to_disk()` which is meant for local storage / easy reload without compression, not for sharing online.
BuilderConfig ParquetConfig(...) doesn't have a 'use_auth_token' key.
State: open | Author: tteguayco (NONE)

### Describe the bug

Trying to run the following fine-tuning script (based on this page [here](https://github.com/huggingface/instruction-tuned-sd)):

```
! accelerate launch /content/instruction-tuned-sd/finetune_instruct_pix2pix.py \
    --pretrained_model_name_or_path=${MODEL_ID} \
    --dataset_name=${DATASET_NAME} \
    --use_ema \
    --enable_xformers_memory_efficient_attention \
    --resolution=512 --random_flip \
    --train_batch_size=2 --gradient_accumulation_steps=4 --gradient_checkpointing \
    --max_train_steps=500 \
    --checkpointing_steps=25 --checkpoints_total_limit=1 \
    --learning_rate=5e-05 --max_grad_norm=1 --lr_warmup_steps=20 \
    --conditioning_dropout_prob=0.1 \
    --mixed_precision=fp16 \
    --seed=42 \
    --output_dir=${OUTPUT_DIR} \
    --original_image_column=before \
    --edit_prompt=prompt \
    --edited_image=after
```

but I keep getting the following error:

```
Traceback (most recent call last):
  File "/content/instruction-tuned-sd/finetune_instruct_pix2pix.py", line 1137, in <module>
    main()
  File "/content/instruction-tuned-sd/finetune_instruct_pix2pix.py", line 652, in main
    dataset = load_dataset(
              ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/load.py", line 1886, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
                                       ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 342, in __init__
    self.config, self.config_id = self._create_builder_config(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 590, in _create_builder_config
    raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
ValueError: BuilderConfig ParquetConfig(name='default', version=0.0.0, data_dir=None, data_files={'train': ['data/train-*']}, description=None, batch_size=None, columns=None, features=None, filters=None) doesn't have a 'use_auth_token' key.
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 10, in <module>
    sys.exit(main())
             ^^^^^^
```

Any ideas? `datasets` version should be `3.2.0`.

### Steps to reproduce the bug

Just running the script above.

### Expected behavior

No errors

### Environment info

Python 3.11.11

datasets==3.2.0

Comments: I encountered the same error, have you resolved it?

Hi ! `use_auth_token` has been deprecated and removed some time ago. You should use `token` instead in `load_dataset()`
`load_dataset` of size 40GB creates a cache of >720GB
State: closed | Author: pietrolesci (NONE)

Hi there,

I am trying to load a dataset from the Hugging Face Hub and split it into train and validation splits. Somehow, when I try to do it with `load_dataset`, it exhausts my disk quota. So, I tried manually downloading the parquet files from the hub and loading them as follows:

```python
 ds = DatasetDict(
        {
            "train": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=0, to=NUM_TRAIN, unit="abs"),  # type: ignore
            ),
            "validation": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=NUM_TRAIN, unit="abs"),  # type: ignore
            )
        }
    )

```

which still strangely creates 720GB of cache. In addition, if I remove the raw parquet file folder (`f"{local_dir}/{tok}"` in this example), I am not able to load anything. So, I am left wondering what this cache is doing. Am I missing something? Is there a solution to this problem?

Thanks a lot in advance for your help!

A related issue: https://github.com/huggingface/transformers/issues/10204#issue-809007443.

---

Python: 3.11.11
datasets: 3.5.0


Comments: Hi ! Parquet is a compressed format. When you load a dataset, it uncompresses the Parquet data into Arrow data on your disk. That's why you can indeed end up with 720GB of uncompressed data on disk. The uncompression is needed to enable performant dataset objects (especially for random access).

To save some storage you can instead load the dataset with `streaming=True`. This way you get an `IterableDataset` that reads the Parquet data iteratively without ever writing to disk.

PS: `ReadInstruction` might not be implemented for `streaming=True`, if it's the case you can use `ds.take()` and `ds.skip()` instead

Hi @lhoestq, thanks a lot for your answer. This makes perfect sense. I will try using the streaming mode. Closing the issue.
Nested Feature raises ArrowNotImplementedError: Unsupported cast using function cast_struct
State: closed | Author: yaner-here (NONE)

### Describe the bug

`datasets.Features` seems to be unable to handle json file that contains fields of `list[dict]`.

### Steps to reproduce the bug

```json
// test.json
{"a": 1, "b": [{"c": 2, "d": 3}, {"c": 4, "d": 5}]}
{"a": 5, "b": [{"c": 7, "d": 8}, {"c": 9, "d": 10}]}
```

```python
import json
from datasets import Dataset, Features, Value, Sequence, load_dataset

annotation_feature = Features({
    "a": Value("int32"),
    "b": Sequence({
        "c": Value("int32"),
        "d": Value("int32"),
    }),
})
annotation_dataset = load_dataset(
    "json", 
    data_files="test.json",
    features=annotation_feature
)
```

```
ArrowNotImplementedError: Unsupported cast from list<item: struct<c: int32, d: int32>> to struct using function cast_struct

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[46], line 11
      2 from datasets import Dataset, Features, Value, Sequence, load_dataset
      4 annotation_feature = Features({
      5     "a": Value("int32"),
      6     "b": Sequence({
   (...)      9     }),
     10 })
---> 11 annotation_dataset = load_dataset(
     12     "json", 
     13     data_files="test.json",
     14     features=annotation_feature
     15 )
```

### Expected behavior

A `datasets.Datasets` instance should be initialized.

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.11.0-21-generic-x86_64-with-glibc2.39
- Python version: 3.11.11
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments: Solved by the default `load_dataset(features)` parameters. Do not use `Sequence` for the `list` in `list[any]` json schema, just simply use `[]`. For example, `"b": Sequence({...})` fails but `"b": [{...}]` works fine.
Make `with_format` correctly indicate that a `Dataset` is compatible with PyTorch's `Dataset` class
State: open | Author: benglewis (NONE)
Labels: name: enhancement

### Feature request

Currently `datasets` does not correctly indicate to the Python type-checker (e.g. `pyright` / `Pylance`) that the output of `with_format` is compatible with PyTorch's `Dataloader` since it does not indicate that the HuggingFace `Dataset` is compatible with the PyTorch `Dataset` class. It would be great if we could get the typing to work nicely.

### Motivation

To avoid casting types in our Python code.

### Your contribution

I would be happy to contribute a PR if this is something that may be accepted and could work with the current approach.
This doesn't have to be for just PyTorch, but I imagine that the same thing would be useful for `tensorflow` and such, but we only have a need for PyTorch at this stage.

Comments: Does the torch `DataLoader` really require the dataset to be a subclass of `torch.utils.data.Dataset` ? Or is there a simpler type we could use ?

PS: also note that a dataset without `with_format()` can also be used in a torch `DataLoader` . Calling `with_format("torch")` simply makes the output of the dataset torch Tensors in an efficient way.
Added cache dirs to load and file_utils
State: closed | Author: gmongaras (NONE)

When adding "cache_dir" to datasets.load_dataset, the cache_dir gets lost in the function calls, changing the cache dir to the default path. This fixes a few of these instances.

Comments: hi ! the `hf_hub_download` cache_dir is a different cache directory than the one for `datasets`.

`hf_hub_download` uses the `huggingface_hub` cache which is located in by default in `~/.cache/huggingface/hub`, while `datasets` uses a different cache for Arrow files and map() results `~/.cache/huggingface/datasets`

Is there a way to change the default cache directory for both of these on calling load_dataset? Currently, cache_dir makes dealing with where I want files to go a bit confusing as the documentation doesn't mention it only relocates.../datasets and not .../hub.

You can set `HF_HOME` which is the common parent directory for those two caches. Or individually `HF_DATASETS_CACHE` and `HF_HUB_CACHE`

Got it. Can this be added to the documentation for load_dataset and related functions to avoid confusion with cache_dir?

done in https://github.com/huggingface/datasets/pull/7532 :)
Extreme memory bandwidth.
State: open | Author: J0SZ (NONE)

### Describe the bug

When I use hf datasets on 4 GPU with 40 workers I get some extreme memory bandwidth of constant ~3GB/s. 

However, if I wrap the dataset in `IterableDataset`, this issue is gone and the data also loads way faster (4x faster training on 1 worker).

It seems like the workers don't share memory and basically duplicate the data 4x40.



### Steps to reproduce the bug

Trainer arguments:
```
    dataloader_pin_memory=True,
    dataloader_num_workers=40,
    dataloader_prefetch_factor=2,
    dataloader_persistent_workers=True,
```
Call trainer:
```
trainer = Trainer(
    model=model,
    args=train_args,
    train_dataset=load_from_disk('..').with_fromat('torch'),
)
```
The dataset has 600GB and consists of 1225 files.

### Expected behavior

The optimal bandwidth should be 100MB/s to keep up with GPU.

### Environment info

Linux
Python 3.11
datasets==3.2.0
How to convert videos to images?
State: open | Author: Loki-Lu (NONE)
Labels: name: enhancement

### Feature request

Does someone know how to return the images from videos?

### Motivation

I am trying to use openpi(https://github.com/Physical-Intelligence/openpi) to finetune my Lerobot dataset(V2.0 and V2.1). I find that although the codedaset is v2.0, they are different. It seems like Lerobot V2.0 has two version, one is data include images infos and another one is separate to data and videos.

Does someone know how to return the images from videos?





Comments: Hi ! there is some documentation here on how to read video frames: https://huggingface.co/docs/datasets/video_load
Json builder: Allow features to override problematic Arrow types
State: open | Author: edmcman (NONE)
Labels: name: enhancement

### Feature request

In the JSON builder, use explicitly requested feature types before or while converting to Arrow.

### Motivation

Working with JSON datasets is really hard because of Arrow.  At the very least, it seems like it should be possible to work-around these problems by explicitly setting problematic columns's types.  But it seems like this is not possible because the features are only used *after* converting to arrow.

Here's a simple example where the Arrow error could potentially be avoided by converting the column to a string: https://colab.research.google.com/drive/16QHRdbUwKSrpwVfGwu8V8AHr8v2dv0dt?usp=sharing

### Your contribution

Maybe with some guidance.  I'm not very familiar with arrow or pandas.

Comments: Hi ! It would be cool indeed, currently the JSON data are generally loaded here: 

https://github.com/huggingface/datasets/blob/90e5bf8a8599b625d6103ee5ac83b98269991141/src/datasets/packaged_modules/json/json.py#L137-L140

Maybe we can pass a Arrow `schema` to avoid errors ?