josteece/github-issues huggingface dataset:
`load_dataset` of size 40GB creates a cache of >720GB
Opened by pietrolesci (closed)

Hi there,

I am trying to load a dataset from the Hugging Face Hub and split it into train and validation splits. Somehow, when I try to do it with `load_dataset`, it exhausts my disk quota. So, I tried manually downloading the parquet files from the hub and loading them as follows:

```python
 ds = DatasetDict(
        {
            "train": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=0, to=NUM_TRAIN, unit="abs"),  # type: ignore
            ),
            "validation": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=NUM_TRAIN, unit="abs"),  # type: ignore
            )
        }
    )

```

which still strangely creates 720GB of cache. In addition, if I remove the raw parquet file folder (`f"{local_dir}/{tok}"` in this example), I am not able to load anything. So, I am left wondering what this cache is doing. Am I missing something? Is there a solution to this problem?

Thanks a lot in advance for your help!

A related issue: https://github.com/huggingface/transformers/issues/10204#issue-809007443.

---

Python: 3.11.11
datasets: 3.5.0


Comments:
Hi ! Parquet is a compressed format. When you load a dataset, it uncompresses the Parquet data into Arrow data on your disk. That's why you can indeed end up with 720GB of uncompressed data on disk. The uncompression is needed to enable performant dataset objects (especially for random access).

To save some storage you can instead load the dataset with `streaming=True`. This way you get an `IterableDataset` that reads the Parquet data iteratively without ever writing to disk.

PS: `ReadInstruction` might not be implemented for `streaming=True`, if it's the case you can use `ds.take()` and `ds.skip()` instead

Hi @lhoestq, thanks a lot for your answer. This makes perfect sense. I will try using the streaming mode. Closing the issue.
Nested Feature raises ArrowNotImplementedError: Unsupported cast using function cast_struct
Opened by yaner-here (closed)

### Describe the bug

`datasets.Features` seems to be unable to handle json file that contains fields of `list[dict]`.

### Steps to reproduce the bug

```json
// test.json
{"a": 1, "b": [{"c": 2, "d": 3}, {"c": 4, "d": 5}]}
{"a": 5, "b": [{"c": 7, "d": 8}, {"c": 9, "d": 10}]}
```

```python
import json
from datasets import Dataset, Features, Value, Sequence, load_dataset

annotation_feature = Features({
    "a": Value("int32"),
    "b": Sequence({
        "c": Value("int32"),
        "d": Value("int32"),
    }),
})
annotation_dataset = load_dataset(
    "json", 
    data_files="test.json",
    features=annotation_feature
)
```

```
ArrowNotImplementedError: Unsupported cast from list<item: struct<c: int32, d: int32>> to struct using function cast_struct

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[46], line 11
      2 from datasets import Dataset, Features, Value, Sequence, load_dataset
      4 annotation_feature = Features({
      5     "a": Value("int32"),
      6     "b": Sequence({
   (...)      9     }),
     10 })
---> 11 annotation_dataset = load_dataset(
     12     "json", 
     13     data_files="test.json",
     14     features=annotation_feature
     15 )
```

### Expected behavior

A `datasets.Datasets` instance should be initialized.

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.11.0-21-generic-x86_64-with-glibc2.39
- Python version: 3.11.11
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments:
Solved by the default `load_dataset(features)` parameters. Do not use `Sequence` for the `list` in `list[any]` json schema, just simply use `[]`. For example, `"b": Sequence({...})` fails but `"b": [{...}]` works fine.
Make `with_format` correctly indicate that a `Dataset` is compatible with PyTorch's `Dataset` class
Opened by benglewis (open)

### Feature request

Currently `datasets` does not correctly indicate to the Python type-checker (e.g. `pyright` / `Pylance`) that the output of `with_format` is compatible with PyTorch's `Dataloader` since it does not indicate that the HuggingFace `Dataset` is compatible with the PyTorch `Dataset` class. It would be great if we could get the typing to work nicely.

### Motivation

To avoid casting types in our Python code.

### Your contribution

I would be happy to contribute a PR if this is something that may be accepted and could work with the current approach.
This doesn't have to be for just PyTorch, but I imagine that the same thing would be useful for `tensorflow` and such, but we only have a need for PyTorch at this stage.

Comments:
Does the torch `DataLoader` really require the dataset to be a subclass of `torch.utils.data.Dataset` ? Or is there a simpler type we could use ?

PS: also note that a dataset without `with_format()` can also be used in a torch `DataLoader` . Calling `with_format("torch")` simply makes the output of the dataset torch Tensors in an efficient way.
Added cache dirs to load and file_utils
Opened by gmongaras (closed)

When adding "cache_dir" to datasets.load_dataset, the cache_dir gets lost in the function calls, changing the cache dir to the default path. This fixes a few of these instances.

Comments:
hi ! the `hf_hub_download` cache_dir is a different cache directory than the one for `datasets`.

`hf_hub_download` uses the `huggingface_hub` cache which is located in by default in `~/.cache/huggingface/hub`, while `datasets` uses a different cache for Arrow files and map() results `~/.cache/huggingface/datasets`

Is there a way to change the default cache directory for both of these on calling load_dataset? Currently, cache_dir makes dealing with where I want files to go a bit confusing as the documentation doesn't mention it only relocates.../datasets and not .../hub.

You can set `HF_HOME` which is the common parent directory for those two caches. Or individually `HF_DATASETS_CACHE` and `HF_HUB_CACHE`

Got it. Can this be added to the documentation for load_dataset and related functions to avoid confusion with cache_dir?

done in https://github.com/huggingface/datasets/pull/7532 :)
Extreme memory bandwidth.
Opened by J0SZ (open)

### Describe the bug

When I use hf datasets on 4 GPU with 40 workers I get some extreme memory bandwidth of constant ~3GB/s. 

However, if I wrap the dataset in `IterableDataset`, this issue is gone and the data also loads way faster (4x faster training on 1 worker).

It seems like the workers don't share memory and basically duplicate the data 4x40.



### Steps to reproduce the bug

Trainer arguments:
```
    dataloader_pin_memory=True,
    dataloader_num_workers=40,
    dataloader_prefetch_factor=2,
    dataloader_persistent_workers=True,
```
Call trainer:
```
trainer = Trainer(
    model=model,
    args=train_args,
    train_dataset=load_from_disk('..').with_fromat('torch'),
)
```
The dataset has 600GB and consists of 1225 files.

### Expected behavior

The optimal bandwidth should be 100MB/s to keep up with GPU.

### Environment info

Linux
Python 3.11
datasets==3.2.0



Comments:
How to convert videos to images?
Opened by Loki-Lu (open)

### Feature request

Does someone know how to return the images from videos?

### Motivation

I am trying to use openpi(https://github.com/Physical-Intelligence/openpi) to finetune my Lerobot dataset(V2.0 and V2.1). I find that although the codedaset is v2.0, they are different. It seems like Lerobot V2.0 has two version, one is data include images infos and another one is separate to data and videos.

Does someone know how to return the images from videos?





Comments:
Hi ! there is some documentation here on how to read video frames: https://huggingface.co/docs/datasets/video_load
Json builder: Allow features to override problematic Arrow types
Opened by edmcman (open)

### Feature request

In the JSON builder, use explicitly requested feature types before or while converting to Arrow.

### Motivation

Working with JSON datasets is really hard because of Arrow.  At the very least, it seems like it should be possible to work-around these problems by explicitly setting problematic columns's types.  But it seems like this is not possible because the features are only used *after* converting to arrow.

Here's a simple example where the Arrow error could potentially be avoided by converting the column to a string: https://colab.research.google.com/drive/16QHRdbUwKSrpwVfGwu8V8AHr8v2dv0dt?usp=sharing

### Your contribution

Maybe with some guidance.  I'm not very familiar with arrow or pandas.

Comments:
Hi ! It would be cool indeed, currently the JSON data are generally loaded here: 

https://github.com/huggingface/datasets/blob/90e5bf8a8599b625d6103ee5ac83b98269991141/src/datasets/packaged_modules/json/json.py#L137-L140

Maybe we can pass a Arrow `schema` to avoid errors ?
Columns in the dataset obtained though load_dataset do not correspond to the one in the dataset viewer since 3.4.0
Opened by bruno-hays (closed)

### Describe the bug

I have noticed that on my dataset named [BrunoHays/Accueil_UBS](https://huggingface.co/datasets/BrunoHays/Accueil_UBS), since the version 3.4.0, every column except audio is missing when I load the dataset.

Interestingly, the dataset viewer still shows the correct columns

### Steps to reproduce the bug

```python
from datasets import load_dataset
ds = load_dataset("BrunoHays/Accueil_UBS", streaming=True)
print(next(iter(ds["test"])).keys())
```
With datasets >= 3.4.0: 
-> dict_keys(['audio'])
With datasets == 3.3.2:
-> dict_keys(['audio', 'id', 'speaker', 'sentence', 'raw_sentence', 'start_timestamp', 'end_timestamp', 'overlap'])


### Expected behavior

All the columns should be present

### Environment info

- `datasets` version: 3.3.2
- Platform: macOS-14.6.1-x86_64-i386-64bit
- Python version: 3.10.15
- `huggingface_hub` version: 0.30.1
- PyArrow version: 16.1.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0


Comments:
Hi, the dataset viewer shows all the possible columns and their types, but `load_dataset()` iterates through all the columns that you defined. It seems that you only have one column (‘audio’) defined in your dataset because when I ran `print(ds.column_names)`, the only name I got was “audio”. You need to clearly define all the other features of the dataset as columns to enable your original code to work. Furthermore, you can run this code to print out all the features of your dataset: 
```python
from datasets import load_dataset_builder
ds_builder = load_dataset_builder("BrunoHays/Accueil_UBS")
print(ds_builder.info.features)
```


@phoebecd 
Good catch, even in datasets<3.4.0, the only feature is "audio".
This datasets follows the [audio folder](https://huggingface.co/docs/datasets/en/audio_dataset#audiofolder) structure with metadata.csv.
Maybe I missed something or there is a bug when having and audio_folder with a metadata file

What do you think @lhoestq ?

I opened a PR to fix the issue :) https://huggingface.co/datasets/BrunoHays/Accueil_UBS/discussions/2

We expect the metadata file to be in the <split>/ folder now to allow one CSV metadata file per split. But in the PR I just added a manual configuration instead of moving the file and updating all the relative paths it contains.
Broken links in pdf loading documentation
Opened by VyoJ (closed)

### Describe the bug

Hi, just a couple of small issues I ran into while reading the docs for [loading pdf data](https://huggingface.co/docs/datasets/main/en/document_load):

1. The link for the [`Create a pdf dataset`](https://huggingface.co/docs/datasets/main/en/document_load#pdffolder) points to https://huggingface.co/docs/datasets/main/en/pdf_dataset instead of https://huggingface.co/docs/datasets/main/en/document_dataset and hence gives a 404 error.

2. At the top of the page, it's mentioned that to work with pdf datasets we need to have the `pdfplumber` package installed but the link to its installation guide points to `pytorch/vision` [installation instructions](https://github.com/pytorch/vision#installation) instead of `pdfplumber`'s [guide](https://github.com/jsvine/pdfplumber#installation)

I love the work on enabling pdf dataset support and these small tweaks would help everyone navigate the docs better. Thanks!

### Steps to reproduce the bug

The issue is on the [Load Document Data](https://huggingface.co/docs/datasets/main/en/document_load) page of the datasets docs.

### Expected behavior

1. For solving the first issue, I went through the [source .mdx code](https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx?plain=1#L188) of the datasets docs and found that the link is pointing to `./pdf_dataset` instead of `./document_dataset`

2. For the second issue, I went through the [source .mdx code](https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx?plain=1#L13) of the datasets docs and found that the link is `pytorch/vision` [installation instructions](https://github.com/pytorch/vision#installation) instead of `pdfplumber`'s [guide](https://github.com/jsvine/pdfplumber#installation)

Just replacing these two links should fix the bugs

### Environment info

datasets v3.5.0 (main at the time of writing)

Comments:
thanks for reporting ! I fixed the links, the docs will be updated in the next release
push_to_hub does not upload videos
Opened by DominikVincent (open)

### Describe the bug

Hello,

I would like to upload a video dataset (some .mp4 files and some segments within them), i.e. rows correspond to subsequences from videos. Videos might be referenced by several rows.

I created a dataset locally and it references the videos and the video readers can read them correctly. I use push_to_hub() to upload the dataset to the hub.

Expectation: A user uses `load_dataset` and can load the videos.

However, the videos seem to be just referenced via paths on the computer and not uploaded to the hub. Therefore a target user cannot load the videos in the dataset.

### Steps to reproduce the bug

1. create a video dataset with paths e.g. { ["videos"]: [path1, path2, ...]}
2. dataset.push_to_hub
3. on a different computer (or same pc if relative paths are used in a different folder):
```
dataset = load_dataset("siplab/egosim", split="train")
video = dataset[0]["video_head"]
```

3. will fail

### Expected behavior

Expectation: A user uses `load_dataset` and can load the videos.

### Environment info

datasets                       3.1.0
Python 3.8.18

Comments:
Hi ! the `Video` type is still experimental, and in particular `push_to_hub` doesn't upload videos at the moment (only the paths).

There is an open question to either upload the videos inside the Parquet files, or rather have them as separate files (which is great to enable remote seeking/streaming)

im having the same issue (btw i mistook this to be xet error https://huggingface.co/spaces/xet-team/README/discussions/4 )

@jsulz suggested me to use `upload_folder` but it exceeds hf limits (>10k files per folder and >100k files in total)

from my reading of the docs, in my case i have to save as either parquet or webdataset and then use `upload_folder`

i tried `ds.to_parquet("...")` but the parquet file also doesnt contain video, as of `datasets` v4.0

so i think the only workaround for my case is webdataset

in that case you can create a VideoFolder dataset instead, see the docs at http://huggingface.co/docs/datasets/video_dataset#videofolder
(refactor) remove redundant logic in _check_valid_index_key
Opened by suzyahyah (open)

This PR contributes a minor refactor, in a small function in `src/datasets/formatting/formatting.py`. No change in logic. 

In the original code, there are separate if-conditionals for `isinstance(key, range)` and `isinstance(key, Iterable)`, with essentially the same logic. 

This PR combines these two using a single if statement.

**Considerations**

1. Although range in python is guaranteed to have integers, internally calling `int()` on an object that is already an int is negligible. (In python it returns the original object. It doesn't create a new integer object or perform any actual conversion)

2. Technically a range is already an Iterable, and we could just do `isinstance(key, Iterable)` but I explicitly did `isinstance(key, (range, Iterable))` just to be super obvious and consistent that both cases are handled because I see `slice, range, Iterable` everywhere in this `formatting.py`

3. This PR removes the `if len(key)>0` conditional. I think it is cleaner to have it this way for three reasons.
- There was originally no else statement and the code would have failed silently anyway.  
- The if len(key)>0 should be caught much earlier, rather than in `formatting.py`. 
- There are actually multiple cases where this would fail, if len(key)>0, if key is non numeric or float, or if key is a list of lists. It's clunky to state all this and the error be thrown during max or indexing. 


**Previous PR and Issues Checks**

1. No known PR or Issues (both closed or open) in hf datasets repository 

**Tests**

1. Tested using Dataset (load_dataset("wikitext", "wikitext-103-raw-v1")), Pytorch DataLoader, with a Pytorch BatchSampler (list of indexes returned instead of single index). 


Comments:
fix: loading of datasets from Disk(#7373)
Opened by sam-hey (open)

Fixes dataset loading from disk by ensuring that memory maps and streams are properly closed. 


For more details, see https://github.com/huggingface/datasets/issues/7373.

Comments:
@nepfaff Could you confirm if this fixes the issue for you? I checked Memray, and everything looked good on my end.

Install: `pip install git+https://github.com/sam-hey/datasets.git@fix/concatenate_datasets`


Will aim to get to this soon. I don't have a rapid testing pipeline setup but need to wait for some AWS nodes to become free

I now set up a small experiment:

```python
# Log initial RAM usage
    process = psutil.Process(os.getpid())
    initial_ram = process.memory_info().rss / (1024 * 1024)  # Convert to MB
    logging.info(f"Initial RAM usage: {initial_ram:.2f} MB")

    chunk_datasets = [
        Dataset.load_from_disk(dataset_path, keep_in_memory=False) for _ in range(N)
    ]
    combined_dataset = concatenate_datasets(chunk_datasets)

    # Log final RAM usage
    final_ram = process.memory_info().rss / (1024 * 1024)  # Convert to MB
    ram_diff = final_ram - initial_ram
    logging.info(f"Final RAM usage: {final_ram:.2f} MB")
    logging.info(f"RAM usage increase: {ram_diff:.2f} MB")
```

The RAM usage is linearly correlated with `N` on datasets master!

For my test dataset:
- N=5 => RAM usage increase: 26302.91 MB
- N=10 => RAM usage increase: 52315.18 MB
- N=20 => RAM usage increase: 104510.65 MB
- N=40 => RAM usage increase: 209166.30 MB

Unfortunately, your patch doesn't seem to change this:
```bash
pip install git+https://github.com/sam-hey/datasets.git@fix/concatenate_datasets
pip list | grep datasets
datasets                 3.5.1.dev0
```
Gives exactly the same RAM statistics.

**Edit:** The results are a bit flawed as the memory increase all seems to come from `Dataset.load_from_disk(dataset_path, keep_in_memory=False)` here (which I don't think should happen either?) and not from `concatenate_datasets`. This seems different from my large-scale setup that runs out of memory during `concatenate_datasets` but I don't seem to be able to replicate this here...

Thanks a lot, @nepfaff, for taking a look at this! It seems that `concatenate_datasets()` is fixed with this PR. I can also confirm that loading a large number of files requires significant memory. However, as I understand it, this is expected/a bug since the memory consumption stems from `pa.memory_map()`, which returns a memory-mapped file.

This behavior might be related to this bug: https://github.com/apache/arrow/issues/34423 

<img width="1728" alt="Screenshot 2025-04-03 at 16 01 11" src="https://github.com/user-attachments/assets/475691d8-3aba-4d7e-b8ef-5e7552c70b14" />


Great ! have you tested that it also fixes the memory issue in your case @iamollas ?

Happy to know that it works for you @sam-hey ! Looking forward to merging this

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7489). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support underscore int read instruction
Opened by lhoestq (closed)

close https://github.com/huggingface/datasets/issues/7481

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7488). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

you rock, Quentin - thank you!
Write pdf in map
Opened by lhoestq (closed)

Fix this error when mapping a PDF dataset

```
pyarrow.lib.ArrowInvalid: Could not convert <pdfplumber.pdf.PDF object at 0x13498ee40> with type PDF: did not recognize Python value type when inferring an Arrow data type
```

and also let map() outputs be lists of images or pdfs

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7487). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support skip_trying_type 
Opened by yoshitomo-matsubara (closed)

This PR addresses Issue #7472

cc: @lhoestq 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7483). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! Can you run `make style` to fix code formatting ?

I was also thinking of naming the argument `try_original_type` and have it `True` by default

@lhoestq 

Thank you for the suggestion! I renamed the argument with `True` by default and ran `make style`
Does it look good?

Thanks @lhoestq !

Let me know if there are anything that I can do for this PR. Otherwise, looking forward to seeing this update in the package soon!

CI failures are unrelated, merging :)

Great, thanks for your support!
I can't wait for the next release :)
deal with python `10_000` legal number in slice syntax
Opened by sfc-gh-sbekman (closed)

### Feature request

```
In [6]: ds = datasets.load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1000]")

In [7]: ds = datasets.load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1_000]")
[dozens of frames skipped]
File /usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py:444, in _str_to_read_instruction(spec)
    442 res = _SUB_SPEC_RE.match(spec)
    443 if not res:
--> 444     raise ValueError(f"Unrecognized instruction format: {spec}")
ValueError: Unrecognized instruction format: train_sft[:1_000]
```

It took me a while to understand what the problem was. But apparently `pyarrow` doesn't allow python numbers that may include `_` as in `1_000`. The `_` aids readability since `10_000_000` vs `10000000` is obviously easier to grasp of what the actual number is.

Feature request:

ideally `datasets` being a python module will do the right thing and convert python numbers into whatever pyarrow supports - in this case stripping `_`s.

Second best it'd err and tell the user that using numbers with `_` in split slices is not acceptible, so that the user won't have to deal with a huge pyarrow assert they know nothing about.

Thank you!

Comments:
should be an easy fix, I opened a PR