josteece/github-issues huggingface dataset:
Features.from_arrow_schema is destructive
Opened by BramVanroy (open)

### Describe the bug

I came across this, perhaps niche, bug where `Features` does not/cannot account for pyarrow's `nullable=False` option in Fields. Interestingly, I found that in regular "flat" fields this does not necessarily lead to conflicts, but when a non-nullable field is in a struct, an incompatibility arises.

It's not easy to explain in words, so the minimal example below should help I hope.

Note that I suggest a solution in the comments in the code, simply allowing `Dataset.to_parquet` to allow for a `schema` argument which, when provided, will override the default ds.features.arrow_schema.

### Steps to reproduce the bug

```python
import os
from datasets import Dataset, Features

import pyarrow as pa
import pyarrow.parquet as pq

# HF datasets is destructive when you call Features.from_arrow_schema(schema) on a schema 
# because it will not account for nullable and non-nullable fields in structs (it will always allow nullable)
# Reloading the same dataset with the original schema will raise an error because the schema is not the same anymore
non_nullable_schema = pa.schema(
    [
        pa.field("text", pa.string(), nullable=False),
        pa.field("meta",
            pa.struct(
                [
                    pa.field("date", pa.list_(pa.string()), nullable=False),
                ],
            ),
        ),

    ]
)
print("ORIGINAL SCHEMA")
print(non_nullable_schema)
print()

feats = Features.from_arrow_schema(non_nullable_schema)

print("FEATUR-IZED SCHEMA (nullable-restrictions are gone)")
print(feats.arrow_schema)
print()

ds = Dataset.from_dict(
    {
        "text": ["a", "b", "c"],
        "meta": [{"date": ["2021-01-01"]}, {"date": ["2021-01-02"]}, {"date": ["2021-01-03"]}],
    },
    features=feats,
)

fname = "tmp.parquet"

# This is not possible: TypeError: pyarrow.parquet.core.ParquetWriter() got multiple values for keyword argument 'schema'
# Though I believe this would be the easiest fix: allow schema to be passed to to_parquet and overwrite the schema in the dataset
# ds.to_parquet(fname, schema=non_nullable_schema)

ds.to_parquet(fname)

try:
    _ = pq.read_table(fname, schema=non_nullable_schema)
finally:
    os.unlink(fname)
```


### Expected behavior

- Non-destructive behavior when converting an arrow schema to Features; or
- the ability to override the default arrow schema with a custom one

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-427.20.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.10
- `huggingface_hub` version: 0.27.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
update fsspec 2025.3.0
Opened by peteski22 (closed)

It appears there have been two releases of fsspec since this dependency was last updated, it would be great if Datasets could be updated so that it didn't hold back the usage of newer fsspec versions in consuming projects.

PR based on https://github.com/huggingface/datasets/pull/7352

Comments:
Sorry for tagging you @lhoestq but since you merged the linked PR, I wondered if you might be able to help me get this triaged so it can be reviewed/rejected etc. 🙏🏼 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7478). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
What is the canonical way to compress a Dataset?
Opened by eric-czech (open)

Given that Arrow is the preferred backend for a Dataset, what is a user supposed to do if they want concurrent reads, concurrent writes AND on-disk compression for a larger dataset?

Parquet would be the obvious answer except that there is no native support for writing sharded, parquet datasets concurrently [[1](https://github.com/huggingface/datasets/issues/7047)].

Am I missing something?  

And if so, why is this not the standard/default way that `Dataset`'s work as they do in Xarray, Ray Data, Composer, etc.?

Comments:
I saw this post by @lhoestq: https://discuss.huggingface.co/t/increased-arrow-table-size-by-factor-of-2/26561/4 suggesting that there is at least some internal code for writing sharded parquet datasets non-concurrently.  This appears to be that code: https://github.com/huggingface/datasets/blob/94ccd1b4fada8a92cea96dc8df4e915041d695b6/src/datasets/arrow_dataset.py#L5380-L5397

Is there any fundamental reason (e.g. race conditions) that this kind of operation couldn't exist as a utility or method on a `Dataset` with a `num_proc` argument?  I am not seeing any other issues explicitly for that ask. 


We simply haven't implemented a method to save as sharded parquet locally yet ^^'

Right now the only sharded parquet export method is `push_to_hub()` which writes to HF. But we can have a local one as well. 

In the meantime the easiest way to export as sharded parquet locally is to `.shard()` and `.to_parquet()` (see code from my comment [here](https://github.com/huggingface/datasets/issues/7047#issuecomment-2233163406))

> In the meantime the easiest way to export as sharded parquet locally is to .shard() and .to_parquet()

Makes sense, BUT how can it be done concurrently?  I could of course use multiprocessing myself or a dozen other libraries for parallelizing single-node/local operations like that.

What I'm asking though is, what is the way to do this that is most canonical for `datasets` specifically?  I.e. what is least likely to causing pickling or other issues because it is used frequently internally by `datasets` and already likely tests for a lot of library-native edge-cases?

Everything in `datasets` is picklable :) and even better: since the data are memory mapped from disk, pickling in one process and unpickling in another doesn't do any copy - it instantaneously reloads the memory map.

So feel free to use the library you prefer to parallelize your operations.

(it's another story in distributed setups though, because in that case you either need to copy and send the data or setup a distributed filesystem)
Priotitize json
Opened by lhoestq (closed)

`datasets` should load the JSON data in https://huggingface.co/datasets/facebook/natural_reasoning, not the PDF

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7476). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove conditions for Python < 3.9
Opened by cyyever (closed)

This PR remove conditions for Python < 3.9.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7474). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks ! can you run `make style` to fix code formatting ? then we can merge

@lhoestq Done
Webdataset data format problem
Opened by edmcman (closed)

### Describe the bug

Please see https://huggingface.co/datasets/ejschwartz/idioms/discussions/1

Error code: FileFormatMismatchBetweenSplitsError

All three splits, train, test, and validation, use webdataset. But only the train split has more than one file. How can I force the other two splits to also be interpreted as being the webdataset format?  (I don't think there is currently a way, but happy to be told that I am wrong.)

### Steps to reproduce the bug

```
import datasets
datasets.load_dataset("ejschwartz/idioms")

### Expected behavior

The dataset loads.  Alternatively, there is a YAML syntax for manually specifying the format.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.28.1
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
I was able to work around it
Adding argument to `_get_data_files_patterns`
Opened by SangbumChoi (closed)

### Feature request

How about adding if the user already know about the pattern?

https://github.com/huggingface/datasets/blob/a256b85cbc67aa3f0e75d32d6586afc507cf535b/src/datasets/data_files.py#L252

### Motivation

While using this load_dataset people might use 10M of images for the local files.
However, due to searching all the appropriate file pattern in fsspec, purely searching this pattern takes more than 10 hours (real use-case).

### Your contribution

Yeah I can make this happen if this seems valid. @lhoestq WDYT?
such like
```
def _get_data_files_patterns(pattern_resolver: Callable[[str], list[str]], patterns: PATTERNS) -> dict[str, list[str]]:
```

Comments:
Hi ! The pattern can be specified in advance in YAML in the README.md of the dataset :)

For example

```
---
configs:
- config_name: default
  data_files:
  - split: train
    path: "train/*"
  - split: test
    path: "test/*"
---
```

See the docs at https://huggingface.co/docs/hub/en/datasets-manual-configuration

@lhoestq How can we choose in this case ? https://huggingface.co/datasets/datasets-examples/doc-image-5


choose what ? sorry I didn't get it ^^'
Custom split name with the web interface
Opened by vince62s (closed)

### Describe the bug

According the doc here: https://huggingface.co/docs/hub/datasets-file-names-and-splits#custom-split-name
it should infer the split name from the subdir of data or the beg of the name of the files in data.
When doing this manually through web upload it does not work. it uses "train" as a unique split.
example: https://huggingface.co/datasets/eole-nlp/estimator_chatml

### Steps to reproduce the bug

follow the link above

### Expected behavior

there should be two splits "mlqe" and "1720_da"

### Environment info

website

Comments:
function `load_dataset`  can't solve folder path with regex characters like "[]"
Opened by Hpeox (open)

### Describe the bug

When using the `load_dataset` function with a folder path containing regex special characters (such as "[]"), the issue occurs due to how the path is handled in the `resolve_pattern` function. This function passes the unprocessed path directly to `AbstractFileSystem.glob`, which supports regular expressions. As a result, the globbing mechanism interprets these characters as regex patterns, leading to a traversal of the entire disk partition instead of confining the search to the intended directory.

### Steps to reproduce the bug

just create a folder like `E:\[D_DATA]\koch_test`, then `load_dataset("parquet", data_dir="E:\[D_DATA]\\test", split="train")`
it will keep searching the whole disk.

I add two `print` in `glob` and `resolve_pattern` to see the path

### Expected behavior

it should load the dataset as in normal folders

### Environment info

- `datasets` version: 3.3.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments:
Hi ! Have you tried escaping the glob special characters `[` and `]` ?

btw note that`AbstractFileSystem.glob` doesn't support regex, instead it supports glob patterns as in the python library [glob](https://docs.python.org/3/library/glob.html)
load_dataset with streaming hangs on parquet datasets
Opened by The0nix (open)

### Describe the bug

When I try to load a dataset with parquet files (e.g. "bigcode/the-stack") the dataset loads, but python interpreter can't exit and hangs

### Steps to reproduce the bug

```python3
import datasets

print('Start')
dataset = datasets.load_dataset("bigcode/the-stack", data_dir="data/yaml", streaming=True, split="train")
it = iter(dataset)
next(it)
print('Finish')
```

The program prints finish but doesn't exit and hangs indefinitely.
I tried this on two different machines and several datasets.

### Expected behavior

The program exits successfully

### Environment info

datasets==3.4.1
Python 3.12.9.
MacOS and Ubuntu Linux

Comments:
Hi ! The issue comes from `pyarrow`, I reported it here: https://github.com/apache/arrow/issues/45214 (feel free to comment / thumb up).

Alternatively we can try to find something else than `ParquetFileFragment.to_batches()` to iterate on Parquet data and keep the option the pass `filters=`...
Fix local pdf loading
Opened by lhoestq (closed)

fir this error when accessing a local pdf

```
File ~/.pyenv/versions/3.12.2/envs/hf-datasets/lib/python3.12/site-packages/pdfminer/psparser.py:220, in PSBaseParser.seek(self, pos)
    218 """Seeks the parser to the given position."""
    219 log.debug("seek: %r", pos)
--> 220 self.fp.seek(pos)
    221 # reset the status for nextline()
    222 self.bufpos = pos

ValueError: seek of closed file
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7466). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Adds EXR format to store depth images in float32
Opened by ducha-aiki (open)

This PR adds the EXR feature to store depth images (or can be normals, etc) in float32.
It relies on [openexr_numpy](https://github.com/martinResearch/openexr_numpy/tree/main) to manipulate EXR images. 



Comments:
Hi ! I'mn wondering if this shouldn't this be an `Image()` type and decoded as a `PIL.Image` ?

This would make it easier to integrate with the rest of the HF ecosystem, and you could still get a numpy array using `ds = ds.with_format("numpy")` which sets all the images to be formatted as numpy arrays

@lhoestq do you mean to add the decoder, and exr extension to the image format? Yes, that probably would be better 

yes exactly
Document the HF_DATASETS_CACHE env variable
Opened by LSerranoPEReN (closed)

### Feature request

Hello,

I have a use case where my team is sharing models and dataset in shared directory to avoid duplication.
I noticed that the [cache documentation for datasets](https://huggingface.co/docs/datasets/main/en/cache) only mention the `HF_HOME` environment variable but never the `HF_DATASETS_CACHE`.

It should be nice to add `HF_DATASETS_CACHE` to datasets documentation if it's an intended feature.
If it's not, I think a depreciation warning would be appreciated.

### Motivation

This variable is fully working and similar to what `HF_HUB_CACHE` does for models, so it's nice to know that this exists. This seems to be a quick change to implement.

### Your contribution

I could contribute since this is only affecting a small portion of the documentation

Comments:
Strongly agree to this, in addition, I am also suffering to change the cache location similar to other issues (since I changed the environmental variables).
https://github.com/huggingface/datasets/issues/6886

`HF_DATASETS_CACHE` should be documented there indeed, feel free to open a PR :) 

Hey, I’d love to work on this issue! Could you assign it to me?

sure ! you can also comment #self-assign in an issue and a bot assigns you automatically :)
Problems with local dataset after upgrade from 3.3.2 to 3.4.0
Opened by andjoer (open)

### Describe the bug

I was not able to open a local saved dataset anymore that was created using an older datasets version after the upgrade yesterday from datasets 3.3.2 to 3.4.0

The traceback is 

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/arrow/arrow.py", line 67, in _generate_tables
    batches = pa.ipc.open_stream(f)
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 190, in open_stream
    return RecordBatchStreamReader(source, options=options,
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 52, in __init__
    self._open(source, options=options, memory_pool=memory_pool)
  File "pyarrow/ipc.pxi", line 1006, in pyarrow.lib._RecordBatchStreamReader._open
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Expected to read 538970747 metadata bytes, but only read 2126

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1855, in _prepare_split_single
    for _, table in generator:
  File "/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/arrow/arrow.py", line 69, in _generate_tables
    reader = pa.ipc.open_file(f)
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 234, in open_file
    return RecordBatchFileReader(
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 110, in __init__
    self._open(source, footer_offset=footer_offset,
  File "pyarrow/ipc.pxi", line 1090, in pyarrow.lib._RecordBatchFileReader._open
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not an Arrow file
```

### Steps to reproduce the bug

Load a dataset from a local folder with 

```
dataset = load_dataset(
                args.train_data_dir,
                cache_dir=args.cache_dir,
            )
```
as it is done for example in the training script for SD3 controlnet.

This is the minimal script to test it: 

```
from datasets import load_dataset

def main():
    dataset = load_dataset(
        "local_dataset",  
    )
    print(dataset)
    print("Sample data:", dataset["train"][0])

if __name__ == "__main__":
    main()
````

### Expected behavior

Work in 3.4.0 like in 3.3.2

### Environment info

- `datasets` version: 3.4.0
- Platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Comments:
Hi ! I just released 3.4.1 with a fix, let me know if it's working now !
Fix resuming after `ds.set_epoch(new_epoch)`
Opened by lhoestq (closed)

close https://github.com/huggingface/datasets/issues/7447

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7451). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add IterableDataset.decode with multithreading
Opened by lhoestq (closed)

Useful for dataset streaming for multimodal datasets, and especially for lerobot.
It speeds up streaming up to 20 times.

When decoding is enabled (default), media types are decoded:

* audio -> dict of "array" and "sampling_rate" and "path"
* image -> PIL.Image
* video -> torchvision.io.VideoReader

You can enable multithreading using `num_threads`. This is especially useful to speed up remote
data streaming. However it can be slower than `num_threads=0` for local data on fast disks.

PS: Disabling decoding is useful if you want to iterate on the paths or bytes of the media files
without actually decoding their content.

Example: Speed up streaming with multithreading:

```py
>>> import os
>>> from datasets import load_dataset
>>> from tqdm import tqdm
>>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True)
>>> num_threads = min(32, (os.cpu_count() or 1) + 4)
>>> ds = ds.decode(num_threads=num_threads)
>>> for _ in tqdm(ds):  # 20 times faster !
...     ...
```

why not multiprocessing ? decoding is done with the GIL released in soundfile/PIL/torchvision so multiprocessing would just use more memory

TODO
- [x] test
- [x] add to docs

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7450). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`datasets.disable_caching` doesn't work
Opened by UCC-team (open)

When I use `Dataset.from_generator(my_gen)` to load my dataset, it simply skips my changes to the generator function.
I tried `datasets.disable_caching`, but it doesn't work!

Comments:
cc

Yes I have the same issue.  It's a confusingly named function.  See [here](https://github.com/huggingface/datasets/blob/main/src/datasets/fingerprint.py#L115-L130)

```
...
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
    More precisely, if the caching is disabled:
    - cache files are always recreated
    - cache files are written to a temporary directory that is deleted when session closes
    - cache files are named using a random hash instead of the dataset fingerprint
```

Also, unfortunately the member variable `ds.cache_files` is not populated either.

I'll let you know if I find a solution.
pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'
Opened by rangehow (closed)

### Describe the bug

A dict with its keys are all str but get following error
```python
test_data=[{'input_ids':[1,2,3],'labels':[[Counter({2:1})]]}]
dataset = datasets.Dataset.from_list(test_data)
```
```bash
pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'
```

### Steps to reproduce the bug

.

### Expected behavior

.

### Environment info

datasets 3.3.2

Comments:
I think the Counter object you used in 'labels' may be the issue, since the {2:1} inside is the dict and 2 is the int

> I think the Counter object you used in 'labels' may be the issue, since the {2:1} inside is the dict and 2 is the int我认为您在 'labels' 中使用的 Counter 对象可能是问题所在，因为里面的 {2：1} 是 dict，而 2 是 int

Yes, that's the point.
Fix small bugs with async map
Opened by lhoestq (closed)

helpful for the next PR to enable parallel image/audio/video decoding and make multimodal datasets go brr (e.g. for lerobot)

- fix with_indices
- fix resuming with save_state_dict() / load_state_dict() - omg that wasn't easy
- remove unnecessary decoding in map() to enable parallelism in FormattedExampleIterable later

small bonus: keeping features in batch()

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7445). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
index error when num_shards > len(dataset)
Opened by eminorhan (open)

In `ds.push_to_hub()` and `ds.save_to_disk()`, `num_shards` must be smaller than or equal to the number of rows in the dataset, but currently this is not checked anywhere inside these functions. Attempting to invoke these functions with `num_shards > len(dataset)` should raise an informative `ValueError`.

I frequently work with datasets with a small number of rows where each row is pretty large, so I often encounter this issue, where the function runs until the shard index in `ds.shard(num_shards, indx)` goes out of bounds. Ideally, a `ValueError` should be raised before reaching this point (i.e. as soon as `ds.push_to_hub()` or `ds.save_to_disk()` is invoked with `num_shards > len(dataset)`).

It seems that adding something like:
```python
if len(self) < num_shards:
   raise ValueError(f"num_shards ({num_shards}) must be smaller than or equal to the number of rows in the dataset ({len(self)}). Please either reduce num_shards or increase max_shard_size to make sure num_shards <= len(dataset).")
```
to the beginning of the definition of the `ds.shard()` function [here](https://github.com/huggingface/datasets/blob/f693f4e93aabafa878470c80fd42ddb10ec550d6/src/datasets/arrow_dataset.py#L4728) would deal with this issue for both `ds.push_to_hub()` and `ds.save_to_disk()`, but I'm not exactly sure if this is the best place to raise the `ValueError` (it seems that a more correct way to do it would be to write separate checks for `ds.push_to_hub()` and `ds.save_to_disk()`). I'd be happy to submit a PR if you think something along these lines would be acceptable.

Comments:
Actually, looking at the code a bit more carefully, maybe an even better solution is to explicitly set `num_shards=len(self)` somewhere inside both `push_to_hub()` and `save_to_disk()` when these functions are invoked with `num_shards > len(dataset)`.