MarouaneSanhaji/github-issues huggingface dataset:
HF_DATASETS_CACHE ignored?
State: open

### Describe the bug

I'm struggling to get things to respect HF_DATASETS_CACHE.

Rationale: I'm on a system that uses NFS for homedir, so downloading to NFS is expensive, slow, and wastes valuable quota compared to local disk. Instead, it seems to rely mostly on HF_HUB_CACHE.

Current version: 3.2.1dev. In the process of testing 3.4.0

### Steps to reproduce the bug

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]

dump.py:
```python
from datasets import load_dataset
dataset = load_dataset("HuggingFaceFW/fineweb", name="sample-100BT", split="train")
```

Repro steps
```bash
# ensure no cache
$ mv ~/.cache/huggingface ~/.cache/huggingface.bak

$ export HF_DATASETS_CACHE=/tmp/roller/datasets
$ rm -rf ${HF_DATASETS_CACHE}
$ env | grep HF | grep -v TOKEN
HF_DATASETS_CACHE=/tmp/roller/datasets

$ python dump.py
# (omitted for brevity)

# (while downloading) 
$ du -hcs ~/.cache/huggingface/hub
18G     hub
18G     total

# (after downloading)
$ du -hcs ~/.cache/huggingface/hub
```

It's a shame because datasets supports s3 (which I could really use right now) but hub does not.

### Expected behavior

* ~/.cache/huggingface/hub stays empty
* /tmp/roller/datasets becomes full of stuff

### Environment info

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]

FWIW, it does eventually write to /tmp/roller/datasets when generating the final version.

Hey, I’d love to work on this issue but I am a beginner, can I work it with you?

Hi @lhoestq,
I'd like to look into this issue but I'm still learning. Could you share any quick pointers on the HF_DATASETS_CACHE behavior here? Thanks!

Hi ! `HF_DATASETS_CACHE` is only for the cache files of the `datasets` library, not for the `huggingface_hub` cache for files downloaded from the Hugging Face Hub.

You should either specify `HF_HOME` (parent cache path for everything HF) or both `HF_DATASETS_CACHE` and `HF_HUB_CACHE`



Thanks for clarifying, @lhoestq! To make sure I’ve got it right:

1. **HF_DATASETS_CACHE** only controls where the **datasets** library writes its own cache files (e.g. processed shards, Arrow files, etc.).
2. Anything downloaded via **huggingface_hub** (models, tokenizers, raw files) still goes into the Hub cache (by default `~/.cache/huggingface/hub`), unless you set **HF_HUB_CACHE** or the parent **HF_HOME**.

So if you want everything off NFS and onto local disk you have two options:

- **Set both**  
  ```bash
  export HF_DATASETS_CACHE=/tmp/roller/datasets  
  export HF_HUB_CACHE=/tmp/roller/hub
  ```
- **Or set**  
  ```bash
  export HF_HOME=/tmp/roller
  ```
  which will apply to both subdirectories.

Is that correct? And would it make sense to add a note to the docs clarifying the distinction (or even support S3 for the Hub cache in the future)? I’m happy to draft a small docs PR if that would help.

Hi, yes that's correct, thanks for the clarification ! A note in the docs would be welcome, thanks
update fsspec 2025.3.0
State: closed

It appears there have been two releases of fsspec since this dependency was last updated, it would be great if Datasets could be updated so that it didn't hold back the usage of newer fsspec versions in consuming projects.

PR based on https://github.com/huggingface/datasets/pull/7352

Sorry for tagging you @lhoestq but since you merged the linked PR, I wondered if you might be able to help me get this triaged so it can be reviewed/rejected etc. 🙏🏼 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7478). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
What is the canonical way to compress a Dataset?
State: open

Given that Arrow is the preferred backend for a Dataset, what is a user supposed to do if they want concurrent reads, concurrent writes AND on-disk compression for a larger dataset?

Parquet would be the obvious answer except that there is no native support for writing sharded, parquet datasets concurrently [[1](https://github.com/huggingface/datasets/issues/7047)].

Am I missing something?  

And if so, why is this not the standard/default way that `Dataset`'s work as they do in Xarray, Ray Data, Composer, etc.?

I saw this post by @lhoestq: https://discuss.huggingface.co/t/increased-arrow-table-size-by-factor-of-2/26561/4 suggesting that there is at least some internal code for writing sharded parquet datasets non-concurrently.  This appears to be that code: https://github.com/huggingface/datasets/blob/94ccd1b4fada8a92cea96dc8df4e915041d695b6/src/datasets/arrow_dataset.py#L5380-L5397

Is there any fundamental reason (e.g. race conditions) that this kind of operation couldn't exist as a utility or method on a `Dataset` with a `num_proc` argument?  I am not seeing any other issues explicitly for that ask. 


We simply haven't implemented a method to save as sharded parquet locally yet ^^'

Right now the only sharded parquet export method is `push_to_hub()` which writes to HF. But we can have a local one as well. 

In the meantime the easiest way to export as sharded parquet locally is to `.shard()` and `.to_parquet()` (see code from my comment [here](https://github.com/huggingface/datasets/issues/7047#issuecomment-2233163406))

> In the meantime the easiest way to export as sharded parquet locally is to .shard() and .to_parquet()

Makes sense, BUT how can it be done concurrently?  I could of course use multiprocessing myself or a dozen other libraries for parallelizing single-node/local operations like that.

What I'm asking though is, what is the way to do this that is most canonical for `datasets` specifically?  I.e. what is least likely to causing pickling or other issues because it is used frequently internally by `datasets` and already likely tests for a lot of library-native edge-cases?

Everything in `datasets` is picklable :) and even better: since the data are memory mapped from disk, pickling in one process and unpickling in another doesn't do any copy - it instantaneously reloads the memory map.

So feel free to use the library you prefer to parallelize your operations.

(it's another story in distributed setups though, because in that case you either need to copy and send the data or setup a distributed filesystem)
Remove conditions for Python < 3.9
State: closed

This PR remove conditions for Python < 3.9.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7474). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks ! can you run `make style` to fix code formatting ? then we can merge

@lhoestq Done
Webdataset data format problem
State: closed

### Describe the bug

Please see https://huggingface.co/datasets/ejschwartz/idioms/discussions/1

Error code: FileFormatMismatchBetweenSplitsError

All three splits, train, test, and validation, use webdataset. But only the train split has more than one file. How can I force the other two splits to also be interpreted as being the webdataset format?  (I don't think there is currently a way, but happy to be told that I am wrong.)

### Steps to reproduce the bug

```
import datasets
datasets.load_dataset("ejschwartz/idioms")

### Expected behavior

The dataset loads.  Alternatively, there is a YAML syntax for manually specifying the format.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.28.1
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

I was able to work around it
Adding argument to `_get_data_files_patterns`
State: closed

### Feature request

How about adding if the user already know about the pattern?

https://github.com/huggingface/datasets/blob/a256b85cbc67aa3f0e75d32d6586afc507cf535b/src/datasets/data_files.py#L252

### Motivation

While using this load_dataset people might use 10M of images for the local files.
However, due to searching all the appropriate file pattern in fsspec, purely searching this pattern takes more than 10 hours (real use-case).

### Your contribution

Yeah I can make this happen if this seems valid. @lhoestq WDYT?
such like
```
def _get_data_files_patterns(pattern_resolver: Callable[[str], list[str]], patterns: PATTERNS) -> dict[str, list[str]]:
```

Hi ! The pattern can be specified in advance in YAML in the README.md of the dataset :)

For example

```
---
configs:
- config_name: default
  data_files:
  - split: train
    path: "train/*"
  - split: test
    path: "test/*"
---
```

See the docs at https://huggingface.co/docs/hub/en/datasets-manual-configuration

@lhoestq How can we choose in this case ? https://huggingface.co/datasets/datasets-examples/doc-image-5


choose what ? sorry I didn't get it ^^'
function `load_dataset`  can't solve folder path with regex characters like "[]"
State: open

### Describe the bug

When using the `load_dataset` function with a folder path containing regex special characters (such as "[]"), the issue occurs due to how the path is handled in the `resolve_pattern` function. This function passes the unprocessed path directly to `AbstractFileSystem.glob`, which supports regular expressions. As a result, the globbing mechanism interprets these characters as regex patterns, leading to a traversal of the entire disk partition instead of confining the search to the intended directory.

### Steps to reproduce the bug

just create a folder like `E:\[D_DATA]\koch_test`, then `load_dataset("parquet", data_dir="E:\[D_DATA]\\test", split="train")`
it will keep searching the whole disk.

I add two `print` in `glob` and `resolve_pattern` to see the path

### Expected behavior

it should load the dataset as in normal folders

### Environment info

- `datasets` version: 3.3.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Hi ! Have you tried escaping the glob special characters `[` and `]` ?

btw note that`AbstractFileSystem.glob` doesn't support regex, instead it supports glob patterns as in the python library [glob](https://docs.python.org/3/library/glob.html)
load_dataset with streaming hangs on parquet datasets
State: open

### Describe the bug

When I try to load a dataset with parquet files (e.g. "bigcode/the-stack") the dataset loads, but python interpreter can't exit and hangs

### Steps to reproduce the bug

```python3
import datasets

print('Start')
dataset = datasets.load_dataset("bigcode/the-stack", data_dir="data/yaml", streaming=True, split="train")
it = iter(dataset)
next(it)
print('Finish')
```

The program prints finish but doesn't exit and hangs indefinitely.
I tried this on two different machines and several datasets.

### Expected behavior

The program exits successfully

### Environment info

datasets==3.4.1
Python 3.12.9.
MacOS and Ubuntu Linux

Hi ! The issue comes from `pyarrow`, I reported it here: https://github.com/apache/arrow/issues/45214 (feel free to comment / thumb up).

Alternatively we can try to find something else than `ParquetFileFragment.to_batches()` to iterate on Parquet data and keep the option the pass `filters=`...
Fix local pdf loading
State: closed

fir this error when accessing a local pdf

```
File ~/.pyenv/versions/3.12.2/envs/hf-datasets/lib/python3.12/site-packages/pdfminer/psparser.py:220, in PSBaseParser.seek(self, pos)
    218 """Seeks the parser to the given position."""
    219 log.debug("seek: %r", pos)
--> 220 self.fp.seek(pos)
    221 # reset the status for nextline()
    222 self.bufpos = pos

ValueError: seek of closed file
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7466). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Adds EXR format to store depth images in float32
State: open

This PR adds the EXR feature to store depth images (or can be normals, etc) in float32.
It relies on [openexr_numpy](https://github.com/martinResearch/openexr_numpy/tree/main) to manipulate EXR images. 



Hi ! I'mn wondering if this shouldn't this be an `Image()` type and decoded as a `PIL.Image` ?

This would make it easier to integrate with the rest of the HF ecosystem, and you could still get a numpy array using `ds = ds.with_format("numpy")` which sets all the images to be formatted as numpy arrays

@lhoestq do you mean to add the decoder, and exr extension to the image format? Yes, that probably would be better 

yes exactly
Document the HF_DATASETS_CACHE env variable
State: closed

### Feature request

Hello,

I have a use case where my team is sharing models and dataset in shared directory to avoid duplication.
I noticed that the [cache documentation for datasets](https://huggingface.co/docs/datasets/main/en/cache) only mention the `HF_HOME` environment variable but never the `HF_DATASETS_CACHE`.

It should be nice to add `HF_DATASETS_CACHE` to datasets documentation if it's an intended feature.
If it's not, I think a depreciation warning would be appreciated.

### Motivation

This variable is fully working and similar to what `HF_HUB_CACHE` does for models, so it's nice to know that this exists. This seems to be a quick change to implement.

### Your contribution

I could contribute since this is only affecting a small portion of the documentation

Strongly agree to this, in addition, I am also suffering to change the cache location similar to other issues (since I changed the environmental variables).
https://github.com/huggingface/datasets/issues/6886

`HF_DATASETS_CACHE` should be documented there indeed, feel free to open a PR :) 

Hey, I’d love to work on this issue! Could you assign it to me?

sure ! you can also comment #self-assign in an issue and a bot assigns you automatically :)
Problems with local dataset after upgrade from 3.3.2 to 3.4.0
State: open

### Describe the bug

I was not able to open a local saved dataset anymore that was created using an older datasets version after the upgrade yesterday from datasets 3.3.2 to 3.4.0

The traceback is 

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/arrow/arrow.py", line 67, in _generate_tables
    batches = pa.ipc.open_stream(f)
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 190, in open_stream
    return RecordBatchStreamReader(source, options=options,
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 52, in __init__
    self._open(source, options=options, memory_pool=memory_pool)
  File "pyarrow/ipc.pxi", line 1006, in pyarrow.lib._RecordBatchStreamReader._open
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Expected to read 538970747 metadata bytes, but only read 2126

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1855, in _prepare_split_single
    for _, table in generator:
  File "/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/arrow/arrow.py", line 69, in _generate_tables
    reader = pa.ipc.open_file(f)
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 234, in open_file
    return RecordBatchFileReader(
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 110, in __init__
    self._open(source, footer_offset=footer_offset,
  File "pyarrow/ipc.pxi", line 1090, in pyarrow.lib._RecordBatchFileReader._open
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not an Arrow file
```

### Steps to reproduce the bug

Load a dataset from a local folder with 

```
dataset = load_dataset(
                args.train_data_dir,
                cache_dir=args.cache_dir,
            )
```
as it is done for example in the training script for SD3 controlnet.

This is the minimal script to test it: 

```
from datasets import load_dataset

def main():
    dataset = load_dataset(
        "local_dataset",  
    )
    print(dataset)
    print("Sample data:", dataset["train"][0])

if __name__ == "__main__":
    main()
````

### Expected behavior

Work in 3.4.0 like in 3.3.2

### Environment info

- `datasets` version: 3.4.0
- Platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0

Hi ! I just released 3.4.1 with a fix, let me know if it's working now !
Add IterableDataset.decode with multithreading
State: closed

Useful for dataset streaming for multimodal datasets, and especially for lerobot.
It speeds up streaming up to 20 times.

When decoding is enabled (default), media types are decoded:

* audio -> dict of "array" and "sampling_rate" and "path"
* image -> PIL.Image
* video -> torchvision.io.VideoReader

You can enable multithreading using `num_threads`. This is especially useful to speed up remote
data streaming. However it can be slower than `num_threads=0` for local data on fast disks.

PS: Disabling decoding is useful if you want to iterate on the paths or bytes of the media files
without actually decoding their content.

Example: Speed up streaming with multithreading:

```py
>>> import os
>>> from datasets import load_dataset
>>> from tqdm import tqdm
>>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True)
>>> num_threads = min(32, (os.cpu_count() or 1) + 4)
>>> ds = ds.decode(num_threads=num_threads)
>>> for _ in tqdm(ds):  # 20 times faster !
...     ...
```

why not multiprocessing ? decoding is done with the GIL released in soundfile/PIL/torchvision so multiprocessing would just use more memory

TODO
- [x] test
- [x] add to docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7450). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`datasets.disable_caching` doesn't work
State: open

When I use `Dataset.from_generator(my_gen)` to load my dataset, it simply skips my changes to the generator function.
I tried `datasets.disable_caching`, but it doesn't work!

cc

Yes I have the same issue.  It's a confusingly named function.  See [here](https://github.com/huggingface/datasets/blob/main/src/datasets/fingerprint.py#L115-L130)

```
...
If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
    More precisely, if the caching is disabled:
    - cache files are always recreated
    - cache files are written to a temporary directory that is deleted when session closes
    - cache files are named using a random hash instead of the dataset fingerprint
```

Also, unfortunately the member variable `ds.cache_files` is not populated either.

I'll let you know if I find a solution.
pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'
State: closed

### Describe the bug

A dict with its keys are all str but get following error
```python
test_data=[{'input_ids':[1,2,3],'labels':[[Counter({2:1})]]}]
dataset = datasets.Dataset.from_list(test_data)
```
```bash
pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'
```

### Steps to reproduce the bug

.

### Expected behavior

.

### Environment info

datasets 3.3.2

I think the Counter object you used in 'labels' may be the issue, since the {2:1} inside is the dict and 2 is the int

> I think the Counter object you used in 'labels' may be the issue, since the {2:1} inside is the dict and 2 is the int我认为您在 'labels' 中使用的 Counter 对象可能是问题所在，因为里面的 {2：1} 是 dict，而 2 是 int

Yes, that's the point.
Fix small bugs with async map
State: closed

helpful for the next PR to enable parallel image/audio/video decoding and make multimodal datasets go brr (e.g. for lerobot)

- fix with_indices
- fix resuming with save_state_dict() / load_state_dict() - omg that wasn't easy
- remove unnecessary decoding in map() to enable parallelism in FormattedExampleIterable later

small bonus: keeping features in batch()

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7445). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
index error when num_shards > len(dataset)
State: open

In `ds.push_to_hub()` and `ds.save_to_disk()`, `num_shards` must be smaller than or equal to the number of rows in the dataset, but currently this is not checked anywhere inside these functions. Attempting to invoke these functions with `num_shards > len(dataset)` should raise an informative `ValueError`.

I frequently work with datasets with a small number of rows where each row is pretty large, so I often encounter this issue, where the function runs until the shard index in `ds.shard(num_shards, indx)` goes out of bounds. Ideally, a `ValueError` should be raised before reaching this point (i.e. as soon as `ds.push_to_hub()` or `ds.save_to_disk()` is invoked with `num_shards > len(dataset)`).

It seems that adding something like:
```python
if len(self) < num_shards:
   raise ValueError(f"num_shards ({num_shards}) must be smaller than or equal to the number of rows in the dataset ({len(self)}). Please either reduce num_shards or increase max_shard_size to make sure num_shards <= len(dataset).")
```
to the beginning of the definition of the `ds.shard()` function [here](https://github.com/huggingface/datasets/blob/f693f4e93aabafa878470c80fd42ddb10ec550d6/src/datasets/arrow_dataset.py#L4728) would deal with this issue for both `ds.push_to_hub()` and `ds.save_to_disk()`, but I'm not exactly sure if this is the best place to raise the `ValueError` (it seems that a more correct way to do it would be to write separate checks for `ds.push_to_hub()` and `ds.save_to_disk()`). I'd be happy to submit a PR if you think something along these lines would be acceptable.

Actually, looking at the code a bit more carefully, maybe an even better solution is to explicitly set `num_shards=len(self)` somewhere inside both `push_to_hub()` and `save_to_disk()` when these functions are invoked with `num_shards > len(dataset)`.
Flexible Loader
State: open

### Feature request

Can we have a utility function that will use `load_from_disk` when given the local path and `load_dataset` if given an HF dataset?

It can be something as simple as this one:

```
def load_hf_dataset(path_or_name):
    if os.path.exists(path_or_name):
        return load_from_disk(path_or_name)
    else:
        return load_dataset(path_or_name)
```

### Motivation

This can be done inside the user codebase, too, but in my experience, it becomes repetitive code.

### Your contribution

I can open a pull request.

Ideally `save_to_disk` should save in a format compatible with load_dataset, wdyt ?

> Ideally `save_to_disk` should save in a format compatible with load_dataset, wdyt ?

That would be perfect if not at least a flexible loader.

@lhoestq For now, you can use this small utility library: [nanoml](https://pypi.org/project/nanoml/)
```python
from nanoml.data import load_dataset_flexible
```

I actively develop and maintain this utility library. Open to contributors. Please open issues, PR, or feature requests.
Fix multi gpu process example
State: closed

to is not an inplace function.

But i am not sure about this code anyway, i think this is modifying the global variable `model` everytime the function is called? Which is on every batch? So it is juggling the same model on every gpu right? Isnt that very inefficient? 

Okay nevermind looks like to works both ways for models. but my doubt still remains, isnt this changing the device of the model every batch?
Allow dataset row indexing with np.int types (#7423)
State: closed

@lhoestq 
Proposed fix for #7423. Added a couple simple tests as requested. I had some test failures related to Java and pyspark even when installing with dev but these don't seem to be related to the changes here and fail for me even on clean main.

The typeerror raised when using the wrong type is: "Wrong key type: '{key}' of type '{type(key)}'. Expected one of int, slice, range, str or Iterable." I think that is fine. But I could modify the int part to something more generic (although I'm not sure what) if wanted.

+1

@lhoestq can you take a look at this?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7438). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you
Refactor `string_to_dict` to return `None` if there is no match instead of raising `ValueError`
State: closed

Making this change, as encouraged here:

* https://github.com/huggingface/datasets/pull/7434#discussion_r1979933054

instead of having the pattern of using `try`-`except` to handle when there is no match, we can instead check if the return value is `None`; we can also assert that the return value should not be `None` if we know that should be true

cc: @lhoestq 

I am going to rebase #7434 onto this branch. Then we can merge this one first if you approve, and then #7434.

@lhoestq any thoughts here?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7435). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

It looks like I was unsafely asserting that `source_url_fields is not None` in `image.py`, `video.py` and `audio.py` (which did not correspond to the `except ValueError` like was there previously). I've changed it to handle `source_url_fields is None`.

Can we re-run CI on this one?

Sweet! These failures are looking spurious due to connectivity issues. Can the failing run be retried?

@lhoestq Sorry to double ping, but can this PR be reviewed? I think it is ready!
Refactor `Dataset.map` to reuse cache files mapped with different `num_proc`
State: closed

Fixes #7433

This refactor unifies `num_proc is None or num_proc == 1` and `num_proc > 1`; instead of handling them completely separately where one uses a list of kwargs and shards and the other just uses a single set of kwargs and `self`, by wrapping the `num_proc == 1` case in a list and making the difference just whether or not you use a pool, you set up either case to be able to load each other's cache files just by changing `num_shards`; `num_proc == 1` can sequentially load the shards of a dataset mapped `num_shards > 1` and map any missing shards

Other than the structural refactor, the main contribution of this PR is `existing_cache_file_map`, which uses a regex of `cache_file_name` and `suffix_template` to find existing cache files, grouped by their `num_shards`; using this data structure, we can reset `num_shards` to an existing set of cache files, and load them accordingly

@lhoestq please let me know what you think about this.

It looks like I can't change the merge target to #7435, so it will look like there is a bunch of extra stuff until #7435 is in main.

@lhoestq Thanks so much for reviewing #7435! Now that that's merged, I think this PR is ready!! Can you kick off CI when you get the chance?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7434). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Do you mind kicking off CI again?

The change I made to support windows paths in 637c1600fe7dd601eff571fda446937bd96c5c84 ended up breaking causing these tests in [tests/test_data_files.py](https://github.com/huggingface/datasets/actions/runs/13858546629/job/38781008643#step:10:6991). When I removed `glob_pattern_to_regex` in 583c28e7560b9d6db2e13048731f41ec8fa11361, none of the tests failed. So I'm thinking the `unicode_escape` may be handling the what `glob_pattern_to_regex` was doing.


@lhoestq will you have a chance to review this today?

Any update?

> LGTM and sorry for the delay !
> 
> note that the CI failures are unrelated to this PR :)

Great job!

great job to @ringohoffman ! ;)
`Dataset.map` ignores existing caches and remaps when ran with different `num_proc`
State: closed

### Describe the bug

If you `map` a dataset and save it to a specific `cache_file_name` with a specific `num_proc`, and then call map again with that same existing `cache_file_name` but a different `num_proc`, the dataset will be re-mapped.

### Steps to reproduce the bug

1. Download a dataset
```python
import datasets

dataset = datasets.load_dataset("ylecun/mnist")
```

```
Generating train split: 100%|██████████| 60000/60000 [00:00<00:00, 116429.85 examples/s]
Generating test split: 100%|██████████| 10000/10000 [00:00<00:00, 103310.27 examples/s]
```

2. `map` and cache it with a specific `num_proc`

```python
cache_file_name="./cache/train.map"
dataset["train"].map(lambda x: x, cache_file_name=cache_file_name, num_proc=2)
```

```
Map (num_proc=2): 100%|██████████| 60000/60000 [00:01<00:00, 53764.03 examples/s]
```

3. `map` it with a different `num_proc` and the same `cache_file_name` as before

```python
dataset["train"].map(lambda x: x, cache_file_name=cache_file_name, num_proc=3)
```

```
Map (num_proc=3): 100%|██████████| 60000/60000 [00:00<00:00, 65377.12 examples/s]
```

### Expected behavior

If I specify an existing `cache_file_name`, I don't expect using a different `num_proc` than the one that was used to generate it to cause the dataset to have be be re-mapped.

### Environment info

```console
$ datasets-cli env

- `datasets` version: 3.3.2
- Platform: Linux-5.15.0-131-generic-x86_64-with-glibc2.35
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
```

This feels related: https://github.com/huggingface/datasets/issues/3044

@lhoestq This comment specifically, I agree:

* https://github.com/huggingface/datasets/issues/3044#issuecomment-1239877570

> Almost a year later and I'm in a similar boat. Using custom fingerprints and when using multiprocessing the cached datasets are saved with a template at the end of the filename (something like "000001_of_000008" for every process of num_proc). So if in the next time you run the script you set num_proc to a different number, the cache cannot be used.
> 
> Is there any way to get around this? I am processing a huge dataset so I do the processing on one machine and then transfer the processed data to another in its cache dir but currently that's not possible due to num_proc mismatch.
Error in code "Time to slice and dice" from course "NLP Course"
State: closed

### Describe the bug

When we execute code 
```
frequencies = (
    train_df["condition"]
    .value_counts()
    .to_frame()
    .reset_index()
    .rename(columns={"index": "condition", "condition": "frequency"})
)
frequencies.head()
```
answer should be like this

condition      | frequency
birth control | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

but he is different

frequency      | count
birth control  | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

this is not correct, correct code
```
frequencies = (
    train_df["condition"]
    .value_counts()
    .to_frame()
    .reset_index()
    .rename(columns={"index": "condition", "count": "frequency"})
)
````

### Steps to reproduce the bug

```
frequencies = (
train_df["condition"]
.value_counts()
.to_frame()
.reset_index()
.rename(columns={"index": "condition", "condition": "frequency"})
)
frequencies.head()
```

### Expected behavior


condition      | frequency
birth control | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

### Environment info

Google Colab

You should open an issue in the NLP course website / github page. I'm closing this issue if you don't mind

ok, i don't mind, i'll mark the error there
Improved type annotation
State: open

I've refined several type annotations throughout the codebase to align with current best practices and enhance overall clarity. Given the complexity of the code, there may still be areas that need further attention. I welcome any feedback or suggestions to make these improvements even better.

 - Fixes #7202 

@lhoestq  Could someone please take a quick look or let me know if there’s anything I should change? Thanks!

could you fix the conflicts ? I think some type annotations have been improved since your first commit

It should be good now.
I'm happy to add more annotations or refine further if needed—just let me know!
Row indexing a dataset with numpy integers
State: closed

### Feature request

Allow indexing datasets with a scalar numpy integer type.

### Motivation

Indexing a dataset with a scalar numpy.int* object raises a TypeError. This is due to the test in `datasets/formatting/formatting.py:key_to_query_type`

``` python
def key_to_query_type(key: Union[int, slice, range, str, Iterable]) -> str:
    if isinstance(key, int):
        return "row"
    elif isinstance(key, str):
        return "column"
    elif isinstance(key, (slice, range, Iterable)):
        return "batch"
    _raise_bad_key_type(key)
```

In the row case, it checks if key is an int, which returns false when key is integer like but not a builtin python integer type. This is counterintuitive because a numpy array of np.int64s can be used for the batch case.

For example:

``` python
import numpy as np

import datasets

dataset = datasets.Dataset.from_dict({"a": [1, 2, 3, 4], "b": [5, 6, 7, 8]})

# Regular indexing
dataset[0]
dataset[:2]

# Indexing with numpy data types (expect same results)
idx = np.asarray([0, 1])
dataset[idx]  # Succeeds when using an array of np.int64 values
dataset[idx[0]]  # Fails with TypeError when using scalar np.int64
```

For the user, this can be solved by wrapping `idx[0]` in `int` but the test could also be changed in `key_to_query_type` to accept a less strict definition of int.

``` diff
+import numbers
+
def key_to_query_type(key: Union[int, slice, range, str, Iterable]) -> str:
+   if isinstance(key, numbers.Integral):
-   if isinstance(key, int):
        return "row"
    elif isinstance(key, str):
        return "column"
    elif isinstance(key, (slice, range, Iterable)):
        return "batch"
    _raise_bad_key_type(key)
```

Looking at how others do it, pandas has an `is_integer` definition that it checks which uses `is_integer_object` defined in `pandas/_libs/utils.pxd`:

``` cython
cdef inline bint is_integer_object(object obj) noexcept:
    """
    Cython equivalent of

    `isinstance(val, (int, np.integer)) and not isinstance(val, (bool, np.timedelta64))`

    Parameters
    ----------
    val : object

    Returns
    -------
    is_integer : bool

    Notes
    -----
    This counts np.timedelta64 objects as integers.
    """
    return (not PyBool_Check(obj) and isinstance(obj, (int, cnp.integer))
            and not is_timedelta64_object(obj))
```

This would be less flexible as it explicitly checks for numpy integer, but worth noting that they had the need to ensure the key is not a bool.

### Your contribution

I can submit a pull request with the above changes after checking that indexing succeeds with the numpy integer type. Or if there is a different integer check that would be preferred I could add that.

If there is a reason not to want this behavior that is fine too.


Would be cool to be consistent when it comes to indexing with numpy objects, if we do accept numpy arrays we should indeed accept numpy integers. Your idea sounds reasonable, I'd also be in favor of adding a simple test as well
pyarrow.lib.arrowinvalid: cannot mix list and non-list, non-null values with map function
State: open

### Describe the bug

Encounter pyarrow.lib.arrowinvalid error with map function in some example when loading the dataset

### Steps to reproduce the bug

```
from datasets import load_dataset
from PIL import Image, PngImagePlugin

dataset = load_dataset("leonardPKU/GEOQA_R1V_Train_8K")
system_prompt="You are a helpful AI Assistant"
def make_conversation(example):
    prompt = []

    prompt.append({"role": "system", "content": system_prompt})
    prompt.append(
            {
                "role": "user", 
                "content": [
                        {"type": "image"},
                        {"type": "text", "text": example["problem"]},
                        ]
            }
        )
    return {"prompt": prompt}

def check_data_types(example):
    for key, value in example.items():
        if key == 'image':
            if not isinstance(value, PngImagePlugin.PngImageFile):
                print(value)
        if key == "problem" or key == "solution":
            if not isinstance(value, str):
                print(value)

    return example

dataset = dataset.map(check_data_types)
dataset = dataset.map(make_conversation)
```

### Expected behavior

Successfully process the dataset with map

### Environment info

datasets==3.3.1

@lhoestq 

Can you try passing text: None for the image object ? Pyarrow expects all the objects to have the exact same type, in particular the dicttionaries in "content" should all have the keys "type" and "text"

The following modification on system prompt works, but it is different from the usual way to use it.
```
def make_conversation(example):
    prompt = []

    prompt.append({"role": "system", "content": [{"type": "text", "text": system_prompt}]})
    prompt.append(
            {
                "role": "user", 
                "content": [
                        {"type": "image"},
                        {"type": "text", "text": example["problem"]},
                        ]
            }
        )
    return {"prompt": prompt}
```

Good to know ! But yes Arrow / Parquet have this typing limitation (which is great to ensure data integrity, but constraining at the same time). It's is really blocking you, feel free to ping the arrow team / community if they plan to have a Union type or a JSON type

I encounter exactly the similar problem when using pyarrow. This issue truly helps a lot.
Shard Dataset at specific indices
State: open

I have a dataset of sequences, where each example in the sequence is a separate row in the dataset (similar to LeRobotDataset). When running `Dataset.save_to_disk` how can I provide indices where it's possible to shard the dataset such that no episode spans more than 1 shard. Consequently, when I run `Dataset.load_from_disk`, how can I load just a subset of the shards to save memory and time on different ranks?

I guess an alternative to this would be, given a loaded `Dataset`, how can I run `Dataset.shard` such that sharding doesn't split any episode across shards?

Hi ! if it's an option I'd suggest to have one sequence per row instead.

Otherwise you'd have to make your own save/load mechanism

Saving one sequence per row is very difficult and heavy and makes all the optimizations pointless. How would a custom save/load mechanism look like?

You can use `pyarrow` for example to save/load individual arrow or parquet files and control what they contain
Documentation on multiple media files of the same type with WebDataset
State: open

The [current documentation](https://huggingface.co/docs/datasets/en/video_dataset) on a creating a video dataset includes only examples with one media file and one json. It would be useful to have examples where multiple files of the same type are included. For example, in a sign language dataset, you may have a base video and a video annotation of the extracted pose. According to the WebDataset documentation, this should be able to be done with period separated filenames. For example:

```e39871fd9fd74f55.base.mp4
e39871fd9fd74f55.pose.mp4
e39871fd9fd74f55.json
f18b91585c4d3f3e.base.mp4
f18b91585c4d3f3e.pose.mp4
f18b91585c4d3f3e.json
... 
```

If you can confirm that this method of including multiple media files works with huggingface datasets and include an example in the documentation, I'd appreciate it.

Yes this is correct and it works with huggingface datasets as well ! Feel free to include an example here: https://github.com/huggingface/datasets/blob/main/docs/source/video_dataset.mdx
Attempt to fix multiprocessing hang by closing and joining the pool before termination
State: closed

https://github.com/huggingface/datasets/issues/6393 has plagued me on and off for a very long time. I have had various workarounds (one time combining two filter calls into one filter call removed the issue, another time making rank 0 go first resolved a cache race condition, one time i think upgrading the version of something resolved it). I don't know hf datasets well enough to fully understand the root cause, but I _think_ this PR fixes it.

Evidence: I have an LLM Foundry training yaml/script (datasets version 3.2.0) that results in a hang ~1/10 times (for a baseline for this testing, it was 2/36 runs that hung). I also reran with the latest datasets version (3.3.1) and got 4/36 hung. Installing datasets from this PR, I was able to successful run the script 144 times without a hang occurring. Assuming the base probability is 1/10, this should be more than enough times to have confidence it works.

After adding some logging, I could see that the code hung during the __exit__ of the mp pool context manager, after all shards had been processed, and the tqdm context manager had exited.

My best explanation: When multiprocessing pool __exit__ is called, it calls pool.terminate, which forcefully exits all the processes (and calls code related to this that I haven't looked at closely). I'm guessing this forceful termination has a bad interaction with some multithreading/multiprocessing that hf datasets does. If we instead call pool.close and pool.join before the pool.terminate happens, perhaps whatever that bad interaction is is able to complete gracefully, and then terminate call proceeds without issue.

If this PR seems good to you, I'd be very appreciative if you were able to do a patch release including it. Thank you!

@lhoestq 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7411). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the fix! We have been affected by this a lot when we try to use LLM Foundry with larger multimodal ICL datasets. 

@lorabit110 are you able to test it out for your case as well? Would be great to get a second validation that it actually fixes the issue. Thanks!
Adding Core Maintainer List to CONTRIBUTING.md
State: closed

### Feature request

I propose adding a core maintainer list to the `CONTRIBUTING.md` file.

### Motivation

The Transformers and Liger-Kernel projects maintain lists of core maintainers for each module. 
However, the Datasets project doesn't have such a list. 

### Your contribution

I have nothing to add here.

@lhoestq

there is no per-module maintainer and the list is me alone nowadays ^^'

@lhoestq 
Oh... I feel for you. 
What are your criteria for choosing a core maintainer? 
It seems like it's too much work for you to manage all this code by yourself.

Also, if you don't mind, can you check this PR for me?
#7368 I'd like this to be added as soon as possible because I need it.
Lazy loading of environment variables
State: open

### Describe the bug

Loading a `.env` file after an `import datasets` call does not correctly use the environment variables. 

This is due the fact that environment variables are read at import time:
https://github.com/huggingface/datasets/blob/de062f0552a810c52077543c1169c38c1f0c53fc/src/datasets/config.py#L155C1-L155C80

### Steps to reproduce the bug

```bash
# make tmp dir
mkdir -p /tmp/debug-env
# make .env file
echo HF_HOME=/tmp/debug-env/data > /tmp/debug-env/.env
# first load dotenv, downloads to /tmp/debug-env/data
uv run --with datasets,python-dotenv python3 -c \
'import dotenv; dotenv.load_dotenv("/tmp/debug-env/.env"); import datasets; datasets.load_dataset("Anthropic/hh-rlhf")' 
# first import datasets, downloads to `~/.cache/huggingface`
uv run --with datasets,python-dotenv python3 -c \
'import datasets; import dotenv; dotenv.load_dotenv("/tmp/debug-env/.env"); datasets.load_dataset("Anthropic/hh-rlhf")' 

```

### Expected behavior

I expect that setting environment variables with something like this:

```python3
if __name__ == "__main__":
    load_dotenv()
    main()
```
works correctly. 

### Environment info

    "datasets>=3.3.0",


Many python packages out there, including `huggingface_hub`, do load the environment variables on import.
You should `load_dotenv()` before importing the libraries.

For example you can move all you imports inside your `main()` function
Performance regression in `dataset.filter`
State: closed

### Describe the bug

We're filtering dataset of ~1M (small-ish) records. At some point in the code we do `dataset.filter`, before (including 3.2.0) it was taking couple of seconds, and now it takes 4 hours.

We use 16 threads/workers, and stack trace at them look as follows:
```
Traceback (most recent call last):
  File "/python/lib/python3.12/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/python/lib/python3.12/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/python/lib/python3.12/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3511, in _map_single
    for i, batch in iter_outputs(shard_iterable):
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3461, in iter_outputs
    yield i, apply_function(example, i, offset=offset)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3390, in apply_function
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 6416, in get_indices_from_mask_function
    indices_array = indices_mapping.column(0).take(indices_array)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 1079, in pyarrow.lib.ChunkedArray.take
  File "/python/lib/python3.12/site-packages/pyarrow/compute.py", line 458, in take
    def take(data, indices, *, boundscheck=True, memory_pool=None):
```

### Steps to reproduce the bug

1. Save dataset of 1M records in arrow
2. Filter it with 16 threads
3. Watch it take too long

### Expected behavior

Filtering done fast

### Environment info

datasets 3.3.0, python 3.12

Thanks for reporting, I'll fix the regression today

I just released `datasets` 3.3.1 with a fix, let me know if it's good now :)

@lhoestq it fixed the issue.

This was (very) fast, thank you very much!