noamaanMulla-03/datasets-issues huggingface dataset:
Nested Feature raises ArrowNotImplementedError: Unsupported cast using function cast_struct

### Describe the bug

`datasets.Features` seems to be unable to handle json file that contains fields of `list[dict]`.

### Steps to reproduce the bug

```json
// test.json
{"a": 1, "b": [{"c": 2, "d": 3}, {"c": 4, "d": 5}]}
{"a": 5, "b": [{"c": 7, "d": 8}, {"c": 9, "d": 10}]}
```

```python
import json
from datasets import Dataset, Features, Value, Sequence, load_dataset

annotation_feature = Features({
    "a": Value("int32"),
    "b": Sequence({
        "c": Value("int32"),
        "d": Value("int32"),
    }),
})
annotation_dataset = load_dataset(
    "json", 
    data_files="test.json",
    features=annotation_feature
)
```

```
ArrowNotImplementedError: Unsupported cast from list<item: struct<c: int32, d: int32>> to struct using function cast_struct

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[46], line 11
      2 from datasets import Dataset, Features, Value, Sequence, load_dataset
      4 annotation_feature = Features({
      5     "a": Value("int32"),
      6     "b": Sequence({
   (...)      9     }),
     10 })
---> 11 annotation_dataset = load_dataset(
     12     "json", 
     13     data_files="test.json",
     14     features=annotation_feature
     15 )
```

### Expected behavior

A `datasets.Datasets` instance should be initialized.

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.11.0-21-generic-x86_64-with-glibc2.39
- Python version: 3.11.11
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


Solved by the default `load_dataset(features)` parameters. Do not use `Sequence` for the `list` in `list[any]` json schema, just simply use `[]`. For example, `"b": Sequence({...})` fails but `"b": [{...}]` works fine.
Add Biwi Kinect Head Pose Database

## Adding a Dataset
- **Name:** Biwi Kinect Head Pose Database
- **Description:** Over 15K images of 20 people recorded with a Kinect while turning their heads around freely. For each frame, depth and rgb images are provided, together with ground in the form of the 3D location of the head and its rotation angles.
- **Data:** [*link to the Github repository or current dataset location*](https://icu.ee.ethz.ch/research/datsets.html)
- **Motivation:** Useful pose estimation dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Labels: name: dataset request

name: vision

Official dataset location : https://icu.ee.ethz.ch/research/datsets.html
In the "Biwi Kinect Head Pose Database" section, I do not find any information regarding "Downloading the dataset." . Do we mail the authors regarding this ?

I found the dataset on Kaggle : [Link](https://www.kaggle.com/kmader/biwi-kinect-head-pose-database) , but since 🤗 does not host any of the datasets, this would require the user to provide their Kaggle username and API key to download. 

Any inputs on how we could proceed ? Thank you.
[ Need your inputs here, @lhoestq or @mariosasko ]

Hi @dnaveenr! Thanks for tackling this issue. This link should work: https://data.vision.ee.ethz.ch/cvl/gfanelli/kinect_head_pose_db.tgz

#self-assign

Added in https://github.com/huggingface/datasets/pull/3903, thanks @dnaveenr !

@mariosasko Hi,
this link [https://data.vision.ee.ethz.ch/cvl/gfanelli/kinect_head_pose_db.tgz]
seems not working now. Do you have alternatives?

It would be cool to find a copy of the data and host it directly in https://huggingface.co/datasets/ETHZurich/biwi_kinect_head_pose

> It would be cool to find a copy of the data and host it directly in https://huggingface.co/datasets/ETHZurich/biwi_kinect_head_pose

Hi, you mean I can download it by running python script in your link?

This dataset repository only contains a script that points to the URL of the file that doesn't exist anymore, so it's not usable anymore either :/

> This dataset repository only contains a script that points to the URL of the file that doesn't exist anymore, so it's not usable anymore either :/

Then what did you mean 
"It would be cool to find a copy of the data and host it directly in https://huggingface.co/datasets/ETHZurich/biwi_kinect_head_pose"?
I don't understand the meaning of 'host'

If we find a copy of the data we can upload the file in the dataset repository on HF, since the original data is not available anymore
Allow pathlib PoxisPath in Dataset.read_json

**Is your feature request related to a problem? Please describe.**
```
from pathlib import Path
from datasets import Dataset
ds = Dataset.read_json(Path('data.json'))
```
causes an error
```
AttributeError: 'PosixPath' object has no attribute 'decode'
```

**Describe the solution you'd like**

It should be able to accept PosixPath and read the json from inside.

Labels: name: enhancement

This same error will occur using `ds = datasets.load_dataset('json', data_files=['test.jsonl'])`

@cccntu I want to make a quick fix for this, but I am struggling to find where the json dataset builder is. Do you know?

@vvvm23 I think you mean think:
```python
ds = datasets.load_dataset('json', data_files=[Path('test.jsonl')])
```
And the place you want to modify is here:
```
utils/file_utils.py:64, in is_remote_url(url_or_filename)
     63 def is_remote_url(url_or_filename: str) -> bool:
---> 64     parsed = urlparse(url_or_filename)
     65     return parsed.scheme in ("http", "https", "s3", "gs", "hdfs", "ftp")
```

Probably just need to check first if `url_or_filename` is [PathLike](https://docs.python.org/3/library/os.html#os.PathLike) and return False early.

Btw, I tried installing from main, and ran my code above and got a different error. Probably because the API have changed.
`AttributeError: module 'datasets' has no attribute 'read_json'`


> @vvvm23 I think you mean think:

You are correct, thanks!

> Probably just need to check first if url_or_filename is [PathLike](https://docs.python.org/3/library/os.html#os.PathLike) and return False early.

Is PathLike sufficient, or should I check the file exists here? Or is that handled later?

I think here we just want to avoid passing Path to urlparse. A simpler solution is to add a str() call and convert the input to string before passing to the next step. No need to check anything.

Above PR should do your first suggestion. Hope that works for you, as I am going on holiday and won't be able to change much :wink: 

I just came across this problem with `DatasetDict.from_parquet()`. The error still seems to originate from the same place:
```
  File "/usr/local/lib/python3.12/dist-packages/datasets/utils/file_utils.py", line 88, in is_relative_path
    return urlparse(url_or_filename).scheme == "" and not os.path.isabs(url_or_filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
```
The PR looks like it would still solve everything. Is there a reason this got delayed?
unsloth/DeepSeek-R1-Distill-Qwen-32B server error

### Describe the bug

hfhubhttperror: 500 server error: internal server error for url: https://huggingface.co/api/models/unsloth/deepseek-r1-distill-qwen-32b-bnb-4bit/commits/main (request id: root=1-67fe23fa-3a2150eb444c2a823c388579;de3aed68-c397-4da5-94d4-6565efd3b919) internal error - we're working hard to fix this as soon as possible!

### Steps to reproduce the bug

unsloth/DeepSeek-R1-Distill-Qwen-32B server error

### Expected behavior

Network repair

### Environment info

The web side is also unavailable
How to convert videos to images?

### Feature request

Does someone know how to return the images from videos?

### Motivation

I am trying to use openpi(https://github.com/Physical-Intelligence/openpi) to finetune my Lerobot dataset(V2.0 and V2.1). I find that although the codedaset is v2.0, they are different. It seems like Lerobot V2.0 has two version, one is data include images infos and another one is separate to data and videos.

Does someone know how to return the images from videos?





Labels: name: enhancement

Hi ! there is some documentation here on how to read video frames: https://huggingface.co/docs/datasets/video_load
Front-end statistical data quantity deviation

### Describe the bug

While browsing the dataset at https://huggingface.co/datasets/NeuML/wikipedia-20250123, I noticed that a dataset with nearly 7M entries was estimated to be only 4M in size—almost half the actual amount. According to the post-download loading and the dataset_info (https://huggingface.co/datasets/NeuML/wikipedia-20250123/blob/main/train/dataset_info.json), the true data volume is indeed close to 7M. This significant discrepancy could mislead users when sorting datasets by row count. Why not directly retrieve this information from dataset_info?

Not sure if this is the right place to report this bug, but leaving it here for the team's awareness.


Hi ! the format of this dataset is not supported by the Dataset Viewer. It looks like this dataset was saved using `save_to_disk()` which is meant for local storage / easy reload without compression, not for sharing online.
Json builder: Allow features to override problematic Arrow types

### Feature request

In the JSON builder, use explicitly requested feature types before or while converting to Arrow.

### Motivation

Working with JSON datasets is really hard because of Arrow.  At the very least, it seems like it should be possible to work-around these problems by explicitly setting problematic columns's types.  But it seems like this is not possible because the features are only used *after* converting to arrow.

Here's a simple example where the Arrow error could potentially be avoided by converting the column to a string: https://colab.research.google.com/drive/16QHRdbUwKSrpwVfGwu8V8AHr8v2dv0dt?usp=sharing

### Your contribution

Maybe with some guidance.  I'm not very familiar with arrow or pandas.

Labels: name: enhancement

Hi ! It would be cool indeed, currently the JSON data are generally loaded here: 

https://github.com/huggingface/datasets/blob/90e5bf8a8599b625d6103ee5ac83b98269991141/src/datasets/packaged_modules/json/json.py#L137-L140

Maybe we can pass a Arrow `schema` to avoid errors ?
num_proc parallelization works only for first ~10s.

### Describe the bug

When I try to load an already downloaded dataset with num_proc=64, the speed is very high for the first 10-20 seconds acheiving 30-40K samples / s, and 100% utilization for all cores but it soon drops to <= 1000 with almost 0% utilization for most cores. 

### Steps to reproduce the bug

```
// download dataset with cli
!huggingface-cli download --repo-type dataset timm/imagenet-1k-wds --max-workers 32

from datasets import load_dataset
ds = load_dataset("timm/imagenet-1k-wds", num_proc=64)
```


### Expected behavior

100% core utilization throughout. 

### Environment info

Azure A100-80GB, 16 cores VM

![Image](https://github.com/user-attachments/assets/69d00fe3-d720-4474-9439-21e046d85034)


Hi, can you check if the processes are still alive ? It's a bit weird because `datasets` does check if processes crash and return an error in that case

Thank you for reverting quickly. I digged a bit, and realized my disk's IOPS is also limited - which is causing this. will check further and report if it's an issue of hf datasets' side or mine.
Broken links in pdf loading documentation

### Describe the bug

Hi, just a couple of small issues I ran into while reading the docs for [loading pdf data](https://huggingface.co/docs/datasets/main/en/document_load):

1. The link for the [`Create a pdf dataset`](https://huggingface.co/docs/datasets/main/en/document_load#pdffolder) points to https://huggingface.co/docs/datasets/main/en/pdf_dataset instead of https://huggingface.co/docs/datasets/main/en/document_dataset and hence gives a 404 error.

2. At the top of the page, it's mentioned that to work with pdf datasets we need to have the `pdfplumber` package installed but the link to its installation guide points to `pytorch/vision` [installation instructions](https://github.com/pytorch/vision#installation) instead of `pdfplumber`'s [guide](https://github.com/jsvine/pdfplumber#installation)

I love the work on enabling pdf dataset support and these small tweaks would help everyone navigate the docs better. Thanks!

### Steps to reproduce the bug

The issue is on the [Load Document Data](https://huggingface.co/docs/datasets/main/en/document_load) page of the datasets docs.

### Expected behavior

1. For solving the first issue, I went through the [source .mdx code](https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx?plain=1#L188) of the datasets docs and found that the link is pointing to `./pdf_dataset` instead of `./document_dataset`

2. For the second issue, I went through the [source .mdx code](https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx?plain=1#L13) of the datasets docs and found that the link is `pytorch/vision` [installation instructions](https://github.com/pytorch/vision#installation) instead of `pdfplumber`'s [guide](https://github.com/jsvine/pdfplumber#installation)

Just replacing these two links should fix the bugs

### Environment info

datasets v3.5.0 (main at the time of writing)


thanks for reporting ! I fixed the links, the docs will be updated in the next release
`load_dataset` of size 40GB creates a cache of >720GB

Hi there,

I am trying to load a dataset from the Hugging Face Hub and split it into train and validation splits. Somehow, when I try to do it with `load_dataset`, it exhausts my disk quota. So, I tried manually downloading the parquet files from the hub and loading them as follows:

```python
 ds = DatasetDict(
        {
            "train": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=0, to=NUM_TRAIN, unit="abs"),  # type: ignore
            ),
            "validation": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=NUM_TRAIN, unit="abs"),  # type: ignore
            )
        }
    )

```

which still strangely creates 720GB of cache. In addition, if I remove the raw parquet file folder (`f"{local_dir}/{tok}"` in this example), I am not able to load anything. So, I am left wondering what this cache is doing. Am I missing something? Is there a solution to this problem?

Thanks a lot in advance for your help!

A related issue: https://github.com/huggingface/transformers/issues/10204#issue-809007443.

---

Python: 3.11.11
datasets: 3.5.0



Hi ! Parquet is a compressed format. When you load a dataset, it uncompresses the Parquet data into Arrow data on your disk. That's why you can indeed end up with 720GB of uncompressed data on disk. The uncompression is needed to enable performant dataset objects (especially for random access).

To save some storage you can instead load the dataset with `streaming=True`. This way you get an `IterableDataset` that reads the Parquet data iteratively without ever writing to disk.

PS: `ReadInstruction` might not be implemented for `streaming=True`, if it's the case you can use `ds.take()` and `ds.skip()` instead

Hi @lhoestq, thanks a lot for your answer. This makes perfect sense. I will try using the streaming mode. Closing the issue.
Iterating over Image feature columns is extremely slow

We are trying to load datasets where the image column stores `PIL.PngImagePlugin.PngImageFile` images. However, iterating over these datasets is extremely slow. 
What I have found:
1. It is the presence of the image column that causes the slowdown. Removing the column from the dataset results in blazingly fast (as expected) times
2. It is ~2x faster to iterate when the column contains a single image as opposed to a list of images i.e., the feature is a Sequence of Image objects. We often need multiple images per sample, so we need to work with a list of images
3. It is ~17x faster to store paths to PNG files and load them using `PIL.Image.open`, as opposed to iterating over a `Dataset` with an Image column, and ~30x faster compared to `Sequence` of `Image`s. See a simple script below with an openly available dataset.


It would be great to understand the standard practices for storing and loading multimodal datasets (image + text). 

https://huggingface.co/docs/datasets/en/image_load seems a bit underdeveloped? (e.g., `dataset.decode` only works with `IterableDataset`, but it's not clear from the doc)

Thanks!

```python
from datasets import load_dataset, load_from_disk
from PIL import Image
from pathlib import Path

ds = load_dataset("getomni-ai/ocr-benchmark")

for idx, sample in enumerate(ds["test"]):
    image = sample["image"]
    image.save(f"/tmp/ds_files/images/image_{idx}.png")

ds.save_to_disk("/tmp/ds_columns")

# Remove the 'image' column
ds["test"] = ds["test"].remove_columns(["image"])

# Create image paths for each sample
image_paths = [f"images/image_{idx}.png" for idx in range(len(ds["test"]))]

# Add the 'image_path' column to the dataset
ds["test"] = ds["test"].add_column("image_path", image_paths)

# Save the updated dataset
ds.save_to_disk("/tmp/ds_files")
files_path = Path("/tmp/ds_files")
column_path = Path("/tmp/ds_columns")

# load and benchmark
ds_file = load_from_disk(files_path)
ds_column = load_from_disk(column_path)
import time


images_files = []
start = time.time()
for idx in range(len(ds_file["test"])):
    image_path = files_path / ds_file["test"][idx]["image_path"]
    image = Image.open(image_path)
    images_files.append(image)
end = time.time()
print(f"Time taken to load images from files: {end - start} seconds")

# Time taken to load images from files: 1.2364635467529297 seconds


images_column = []
start = time.time()
for idx in range(len(ds_column["test"])):
    images_column.append(ds_column["test"][idx]["image"])
end = time.time()
print(f"Time taken to load images from columns: {end - start} seconds")

# Time taken to load images from columns: 20.49347186088562 seconds
```


Hi ! Could it be because the `Image()` type in dataset does `image  = Image.open(image_path)` and also `image.load()` which actually loads the image data in memory ? This is needed to avoid too many open files issues, see https://github.com/huggingface/datasets/issues/3985

Yes, that seems to be it. For my purposes, I've cast the column to `Image(decode=False)`, and only load the images when necessary, which is much much faster
Remove conditions for Python < 3.9

This PR remove conditions for Python < 3.9.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7474). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks ! can you run `make style` to fix code formatting ? then we can merge

@lhoestq Done
Index error when data is large

### Describe the bug

At `save_to_disk` step, the `max_shard_size` by default is `500MB`. However, one row of the dataset might be larger than `500MB` then the saving will throw an index error. Without looking at the source code, the bug is due to wrong calculation of number of shards which i think is 
`total_size / min(max_shard_size, row_size)` which should be `total_size / max(max_shard_size, row_size)`

The fix is setting a larger `max_shard_size`

### Steps to reproduce the bug

1. create a dataset with large dense tensors per row
2. set a small `max_shard_size` say 1MB
3. `save_to_disk`

### Expected behavior

```
raise IndexError(f"Index {index} out of range for dataset of size {size}.")

IndexError: Index 10 out of range for dataset of size 10.
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2


I'm facing this problem while doing my translation of [mteb/stackexchange-clustering](https://huggingface.co/datasets/mteb/stackexchange-clustering). each row has lots of samples (up to 100k samples), because in this dataset, each row represent multiple clusters.
my hack is to setting `max_shard_size` to 20Gb or even larger
```py
final_dataset.push_to_hub(
        output_dataset, 
        private=True,
        max_shard_size="20GB"  # This will ensure appropriate sharding based on data size
    )
```
It will work, but depends on your data size.
Parquet stream-conversion fails to embed images/audio files from gated repos

it seems to be an issue with datasets not passing the token to embed_table_storage when generating a dataset

See https://github.com/huggingface/datasets-server/issues/2010

Labels: name: bug

I have created a project to stream audio in the datasets viewer on Hugging Face using Parquet.

https://github.com/pr0mila/ParquetToHuggingFace
Cannot create a dataset with relative audio path

### Describe the bug

Hello! I want to create a dataset of parquet files, with audios stored as separate .mp3 files. However, it says "No such file or directory" (see the reproducing code).

### Steps to reproduce the bug

Creating a dataset
```
from pathlib import Path
from datasets import Dataset, load_dataset, Audio

Path('my_dataset/audio').mkdir(parents=True, exist_ok=True)
Path('my_dataset/audio/file.mp3').touch(exist_ok=True)
Dataset.from_list(
    [{'audio': {'path': 'audio/file.mp3'}}]
).to_parquet('my_dataset/data.parquet')
```

Result:
```
# my_dataset
# ├── audio
# │   └── file.mp3
# └── data.parquet
```

Trying to load the dataset
```
dataset = (
    load_dataset('my_dataset', split='train')
    .cast_column('audio', Audio(sampling_rate=16_000))
)
dataset[0]

>>> FileNotFoundError: [Errno 2] No such file or directory: 'audio/file.mp3'
```

### Expected behavior

I expect the dataset to load correctly.

I've found 2 workarounds, but they are not very good:
1. I can specify an absolute path to the audio, however, when I move the folder or upload to HF it will stop working.
2. I can set `'path': 'file.mp3'`, and load with `load_dataset('my_dataset', data_dir='audio')` - it seems to work, but does this mean that anyone from Hugging Face who wants to use this dataset should also pass the `data_dir` argument, otherwise it won't work?

### Environment info

datasets 3.1.0, Ubuntu 24.04.1


Hello ! when you `cast_column` you need the paths to be absolute paths or relative paths to your working directory, not the original dataset directory.

Though I'd recommend structuring your dataset as an AudioFolder which automatically links a metadata.jsonl or csv to the audio files via relative paths **within** the dataset repository: https://huggingface.co/docs/datasets/v3.2.0/en/audio_load#audiofolder

@lhoestq thank you, but there are two problems with using AudioFolder:
1. It is said that AudioFolder requires metadata.csv. However, my datset is too large and contains nested and np.ndarray fields, so I can't use csv.
2. It is said that I need to load the dataset with `load_dataset("audiofolder", ...)`. However, if possible, I want my dataset to be loaded as usual with `load_dataset(dataset_name)` after I upload if to HF.

You can use metadata.jsonl if you have nested data :)

And actually if you have a dataset structured as an AudioFolder then `load_dataset(dataset_name)` does work after uploading to HF

I have created an audio dataset. In my repo, I have explained the steps and structure. An example dataset is also available in the repo. https://github.com/pr0mila/ParquetToHuggingFace