KRadim/github-issues-minimal huggingface dataset:
(refactor) remove redundant logic in _check_valid_index_key
State: open

This PR contributes a minor refactor, in a small function in `src/datasets/formatting/formatting.py`. No change in logic. 

In the original code, there are separate if-conditionals for `isinstance(key, range)` and `isinstance(key, Iterable)`, with essentially the same logic. 

This PR combines these two using a single if statement.

**Considerations**

1. Although range in python is guaranteed to have integers, internally calling `int()` on an object that is already an int is negligible. (In python it returns the original object. It doesn't create a new integer object or perform any actual conversion)

2. Technically a range is already an Iterable, and we could just do `isinstance(key, Iterable)` but I explicitly did `isinstance(key, (range, Iterable))` just to be super obvious and consistent that both cases are handled because I see `slice, range, Iterable` everywhere in this `formatting.py`

3. This PR removes the `if len(key)>0` conditional. I think it is cleaner to have it this way for three reasons.
- There was originally no else statement and the code would have failed silently anyway.  
- The if len(key)>0 should be caught much earlier, rather than in `formatting.py`. 
- There are actually multiple cases where this would fail, if len(key)>0, if key is non numeric or float, or if key is a list of lists. It's clunky to state all this and the error be thrown during max or indexing. 


**Previous PR and Issues Checks**

1. No known PR or Issues (both closed or open) in hf datasets repository 

**Tests**

1. Tested using Dataset (load_dataset("wikitext", "wikitext-103-raw-v1")), Pytorch DataLoader, with a Pytorch BatchSampler (list of indexes returned instead of single index).
Implement capability to restore non-nullability in Features
State: closed

This PR attempts to keep track of non_nullable pyarrow fields when converting a `pa.Schema` to `Features`. At the same time, when outputting the `arrow_schema`, the original non-nullable fields are restored. This allows for more consistent behavior and avoids breaking behavior as illustrated in #7479.

I am by no means a pyarrow expert so some logic in `find_non_nullable_fields` may not perfect. Not sure if more logic (type checks) are needed for deep-checking a given schema. Maybe there are other pyarrow structures that need to be covered?

Tests are added, but again, these may not have sufficient coverage in terms of pyarrow structure types.

closes #7479
deal with python `10_000` legal number in slice syntax
State: closed
Labels: name: enhancement

### Feature request

```
In [6]: ds = datasets.load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1000]")

In [7]: ds = datasets.load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1_000]")
[dozens of frames skipped]
File /usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py:444, in _str_to_read_instruction(spec)
    442 res = _SUB_SPEC_RE.match(spec)
    443 if not res:
--> 444     raise ValueError(f"Unrecognized instruction format: {spec}")
ValueError: Unrecognized instruction format: train_sft[:1_000]
```

It took me a while to understand what the problem was. But apparently `pyarrow` doesn't allow python numbers that may include `_` as in `1_000`. The `_` aids readability since `10_000_000` vs `10000000` is obviously easier to grasp of what the actual number is.

Feature request:

ideally `datasets` being a python module will do the right thing and convert python numbers into whatever pyarrow supports - in this case stripping `_`s.

Second best it'd err and tell the user that using numbers with `_` in split slices is not acceptible, so that the user won't have to deal with a huge pyarrow assert they know nothing about.

Thank you!
HF_DATASETS_CACHE ignored?
State: open

### Describe the bug

I'm struggling to get things to respect HF_DATASETS_CACHE.

Rationale: I'm on a system that uses NFS for homedir, so downloading to NFS is expensive, slow, and wastes valuable quota compared to local disk. Instead, it seems to rely mostly on HF_HUB_CACHE.

Current version: 3.2.1dev. In the process of testing 3.4.0

### Steps to reproduce the bug

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]

dump.py:
```python
from datasets import load_dataset
dataset = load_dataset("HuggingFaceFW/fineweb", name="sample-100BT", split="train")
```

Repro steps
```bash
# ensure no cache
$ mv ~/.cache/huggingface ~/.cache/huggingface.bak

$ export HF_DATASETS_CACHE=/tmp/roller/datasets
$ rm -rf ${HF_DATASETS_CACHE}
$ env | grep HF | grep -v TOKEN
HF_DATASETS_CACHE=/tmp/roller/datasets

$ python dump.py
# (omitted for brevity)

# (while downloading) 
$ du -hcs ~/.cache/huggingface/hub
18G     hub
18G     total

# (after downloading)
$ du -hcs ~/.cache/huggingface/hub
```

It's a shame because datasets supports s3 (which I could really use right now) but hub does not.

### Expected behavior

* ~/.cache/huggingface/hub stays empty
* /tmp/roller/datasets becomes full of stuff

### Environment info

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]
Features.from_arrow_schema is destructive
State: open

### Describe the bug

I came across this, perhaps niche, bug where `Features` does not/cannot account for pyarrow's `nullable=False` option in Fields. Interestingly, I found that in regular "flat" fields this does not necessarily lead to conflicts, but when a non-nullable field is in a struct, an incompatibility arises.

It's not easy to explain in words, so the minimal example below should help I hope.

Note that I suggest a solution in the comments in the code, simply allowing `Dataset.to_parquet` to allow for a `schema` argument which, when provided, will override the default ds.features.arrow_schema.

### Steps to reproduce the bug

```python
import os
from datasets import Dataset, Features

import pyarrow as pa
import pyarrow.parquet as pq

# HF datasets is destructive when you call Features.from_arrow_schema(schema) on a schema 
# because it will not account for nullable and non-nullable fields in structs (it will always allow nullable)
# Reloading the same dataset with the original schema will raise an error because the schema is not the same anymore
non_nullable_schema = pa.schema(
    [
        pa.field("text", pa.string(), nullable=False),
        pa.field("meta",
            pa.struct(
                [
                    pa.field("date", pa.list_(pa.string()), nullable=False),
                ],
            ),
        ),

    ]
)
print("ORIGINAL SCHEMA")
print(non_nullable_schema)
print()

feats = Features.from_arrow_schema(non_nullable_schema)

print("FEATUR-IZED SCHEMA (nullable-restrictions are gone)")
print(feats.arrow_schema)
print()

ds = Dataset.from_dict(
    {
        "text": ["a", "b", "c"],
        "meta": [{"date": ["2021-01-01"]}, {"date": ["2021-01-02"]}, {"date": ["2021-01-03"]}],
    },
    features=feats,
)

fname = "tmp.parquet"

# This is not possible: TypeError: pyarrow.parquet.core.ParquetWriter() got multiple values for keyword argument 'schema'
# Though I believe this would be the easiest fix: allow schema to be passed to to_parquet and overwrite the schema in the dataset
# ds.to_parquet(fname, schema=non_nullable_schema)

ds.to_parquet(fname)

try:
    _ = pq.read_table(fname, schema=non_nullable_schema)
finally:
    os.unlink(fname)
```


### Expected behavior

- Non-destructive behavior when converting an arrow schema to Features; or
- the ability to override the default arrow schema with a custom one

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-427.20.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.10
- `huggingface_hub` version: 0.27.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
What is the canonical way to compress a Dataset?
State: open

Given that Arrow is the preferred backend for a Dataset, what is a user supposed to do if they want concurrent reads, concurrent writes AND on-disk compression for a larger dataset?

Parquet would be the obvious answer except that there is no native support for writing sharded, parquet datasets concurrently [[1](https://github.com/huggingface/datasets/issues/7047)].

Am I missing something?  

And if so, why is this not the standard/default way that `Dataset`'s work as they do in Xarray, Ray Data, Composer, etc.?
IterableDataset's state_dict shard_example_idx is always equal to the number of samples in a shard
State: closed

### Describe the bug

I've noticed a strange behaviour with Iterable state_dict: the value of shard_example_idx is always equal to the amount of samples in a shard.


### Steps to reproduce the bug

I am reusing the example from the doc
```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=1)
state_dict = None
# Iterate through the dataset and print examples
for idx, example in enumerate(ds):
    print(example)
    if idx == 2:
        state_dict = ds.state_dict()
        print("checkpoint")
        break
print(state_dict)
```
Returns:
```
{'a': 0}
{'a': 1}
checkpoint
{'examples_iterable': {'shard_idx': 0, 'shard_example_idx': 6, 'type': 'ArrowExamplesIterable'}, 'epoch': 0}
```


### Expected behavior

shard_example_idx should be 2 instead of 6
If we run with num_shards=2, then shard_example_idx is 3 instead of 2 and so on.

### Environment info

- `datasets` version: 3.4.1
- Platform: macOS-14.6.1-arm64-arm-64bit
- Python version: 3.12.9
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
Webdataset data format problem
State: closed

### Describe the bug

Please see https://huggingface.co/datasets/ejschwartz/idioms/discussions/1

Error code: FileFormatMismatchBetweenSplitsError

All three splits, train, test, and validation, use webdataset. But only the train split has more than one file. How can I force the other two splits to also be interpreted as being the webdataset format?  (I don't think there is currently a way, but happy to be told that I am wrong.)

### Steps to reproduce the bug

```
import datasets
datasets.load_dataset("ejschwartz/idioms")

### Expected behavior

The dataset loads.  Alternatively, there is a YAML syntax for manually specifying the format.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.28.1
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
Adding argument to `_get_data_files_patterns`
State: closed
Labels: name: enhancement

### Feature request

How about adding if the user already know about the pattern?

https://github.com/huggingface/datasets/blob/a256b85cbc67aa3f0e75d32d6586afc507cf535b/src/datasets/data_files.py#L252

### Motivation

While using this load_dataset people might use 10M of images for the local files.
However, due to searching all the appropriate file pattern in fsspec, purely searching this pattern takes more than 10 hours (real use-case).

### Your contribution

Yeah I can make this happen if this seems valid. @lhoestq WDYT?
such like
```
def _get_data_files_patterns(pattern_resolver: Callable[[str], list[str]], patterns: PATTERNS) -> dict[str, list[str]]:
```
Is it possible to shard a single-sharded IterableDataset?
State: closed

I thought https://github.com/huggingface/datasets/pull/7252 might be applicable but looking at it maybe not.

Say we have a process, eg. a database query, that can return data in slightly different order each time. So, the initial query needs to be run by a single thread (not to mention running multiple times incurs more cost too). But the results are also big enough that we don't want to materialize it entirely and instead stream it with an IterableDataset.

But after we have the results we want to split it up across workers to parallelize processing.

Is something like this possible to do?

Here's a failed attempt. The end result should be that each of the shards has unique data, but unfortunately with this attempt the generator gets run once in each shard and the results end up with duplicates...

```
import random
import datasets


def gen():
  print('RUNNING GENERATOR!')
  items = list(range(10))
  random.shuffle(items)
  yield from items


ds = datasets.IterableDataset.from_generator(gen)

print('dataset contents:')
for item in ds:
  print(item)
print()

print('dataset contents (2):')
for item in ds:
  print(item)
print()


num_shards = 3


def sharded(shard_id):
  for i, example in enumerate(ds):
    if i % num_shards in shard_id:
      yield example


ds1 = datasets.IterableDataset.from_generator(
  sharded, gen_kwargs={'shard_id': list(range(num_shards))}
)

for shard in range(num_shards):
  print('shard', shard)
  for item in ds1.shard(num_shards, shard):
    print(item)
```
Custom split name with the web interface
State: closed

### Describe the bug

According the doc here: https://huggingface.co/docs/hub/datasets-file-names-and-splits#custom-split-name
it should infer the split name from the subdir of data or the beg of the name of the files in data.
When doing this manually through web upload it does not work. it uses "train" as a unique split.
example: https://huggingface.co/datasets/eole-nlp/estimator_chatml

### Steps to reproduce the bug

follow the link above

### Expected behavior

there should be two splits "mlqe" and "1720_da"

### Environment info

website
function `load_dataset`  can't solve folder path with regex characters like "[]"
State: open

### Describe the bug

When using the `load_dataset` function with a folder path containing regex special characters (such as "[]"), the issue occurs due to how the path is handled in the `resolve_pattern` function. This function passes the unprocessed path directly to `AbstractFileSystem.glob`, which supports regular expressions. As a result, the globbing mechanism interprets these characters as regex patterns, leading to a traversal of the entire disk partition instead of confining the search to the intended directory.

### Steps to reproduce the bug

just create a folder like `E:\[D_DATA]\koch_test`, then `load_dataset("parquet", data_dir="E:\[D_DATA]\\test", split="train")`
it will keep searching the whole disk.

I add two `print` in `glob` and `resolve_pattern` to see the path

### Expected behavior

it should load the dataset as in normal folders

### Environment info

- `datasets` version: 3.3.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
load_dataset with streaming hangs on parquet datasets
State: open

### Describe the bug

When I try to load a dataset with parquet files (e.g. "bigcode/the-stack") the dataset loads, but python interpreter can't exit and hangs

### Steps to reproduce the bug

```python3
import datasets

print('Start')
dataset = datasets.load_dataset("bigcode/the-stack", data_dir="data/yaml", streaming=True, split="train")
it = iter(dataset)
next(it)
print('Finish')
```

The program prints finish but doesn't exit and hangs indefinitely.
I tried this on two different machines and several datasets.

### Expected behavior

The program exits successfully

### Environment info

datasets==3.4.1
Python 3.12.9.
MacOS and Ubuntu Linux
Fix local pdf loading
State: closed

fir this error when accessing a local pdf

```
File ~/.pyenv/versions/3.12.2/envs/hf-datasets/lib/python3.12/site-packages/pdfminer/psparser.py:220, in PSBaseParser.seek(self, pos)
    218 """Seeks the parser to the given position."""
    219 log.debug("seek: %r", pos)
--> 220 self.fp.seek(pos)
    221 # reset the status for nextline()
    222 self.bufpos = pos

ValueError: seek of closed file
```
List of images behave differently on IterableDataset and Dataset
State: closed

### Describe the bug

This code:

```python
def train_iterable_gen():
        images = np.array(load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg").resize((128, 128)))
        yield {
            "images": np.expand_dims(images, axis=0),
            "messages": [
                {
                    "role": "user",
                    "content": [{"type": "image", "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" }]
                },
                {
                    "role": "assistant",
                    "content": [{"type": "text", "text": "duck" }]
                }
            ]
        }
    train_ds = Dataset.from_generator(train_iterable_gen,
                                      features=Features({
                                           'images': [datasets.Image(mode=None, decode=True, id=None)],
                                           'messages': [{'content': [{'text': datasets.Value(dtype='string', id=None), 'type': datasets.Value(dtype='string', id=None) }], 'role': datasets.Value(dtype='string', id=None)}]
                                           } )
                                           )
```

works as I'd expect; if I iterate the dataset then the `images` column returns a `List[PIL.Image.Image]`, i.e. `'images': [<PIL.PngImagePlugin.PngImageFile image mode=RGB size=128x128 at 0x77EFB7EF4680>]`.

But if I change `Dataset` to `IterableDataset`, the `images` column changes into `'images': [{'path': None, 'bytes': ..]`

### Steps to reproduce the bug

The code above +

```python
def load_image(url):
        response = requests.get(url)
        image = Image.open(io.BytesIO(response.content))
        return image
```

I'm feeding it to SFTTrainer

### Expected behavior

Dataset and IterableDataset would behave the same

### Environment info

```yaml
requires-python = ">=3.12"
dependencies = [
    "av>=14.1.0",
    "boto3>=1.36.7",
    "datasets>=3.3.2",
    "docker>=7.1.0",
    "google-cloud-storage>=2.19.0",
    "grpcio>=1.70.0",
    "grpcio-tools>=1.70.0",
    "moviepy>=2.1.2",
    "open-clip-torch>=2.31.0",
    "opencv-python>=4.11.0.86; sys_platform == 'darwin'",
    "opencv-python-headless>=4.11.0.86; sys_platform == 'linux'",
    "pandas>=2.2.3",
    "pillow>=10.4.0",
    "plotly>=6.0.0",
    "py-spy>=0.4.0",
    "pydantic>=2.10.6",
    "pydantic-settings>=2.7.1",
    "pymysql>=1.1.1",
    "ray[data,default,serve,train,tune]>=2.43.0",
    "torch>=2.6.0",
    "torchmetrics>=1.6.1",
    "torchvision>=0.21.0",
    "transformers[torch]@git+https://github.com/huggingface/transformers",
    "wandb>=0.19.4",
    # https://github.com/Dao-AILab/flash-attention/issues/833
    "flash-attn @ https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.3/flash_attn-2.7.3+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whl; sys_platform == 'linux'",
    "trl@https://github.com/huggingface/trl.git",
    "peft>=0.14.0",
]
```
Document the HF_DATASETS_CACHE env variable
State: closed
Labels: name: enhancement

### Feature request

Hello,

I have a use case where my team is sharing models and dataset in shared directory to avoid duplication.
I noticed that the [cache documentation for datasets](https://huggingface.co/docs/datasets/main/en/cache) only mention the `HF_HOME` environment variable but never the `HF_DATASETS_CACHE`.

It should be nice to add `HF_DATASETS_CACHE` to datasets documentation if it's an intended feature.
If it's not, I think a depreciation warning would be appreciated.

### Motivation

This variable is fully working and similar to what `HF_HUB_CACHE` does for models, so it's nice to know that this exists. This seems to be a quick change to implement.

### Your contribution

I could contribute since this is only affecting a small portion of the documentation
.add_faiss_index and .add_elasticsearch_index returns ImportError at Google Colab
State: open

### Describe the bug

At Google Colab
```!pip install faiss-cpu``` works
```import faiss``` no error
but
```embeddings_dataset.add_faiss_index(column='embeddings')```
returns
```
[/usr/local/lib/python3.11/dist-packages/datasets/search.py](https://localhost:8080/#) in init(self, device, string_factory, metric_type, custom_index)
247 self.faiss_index = custom_index
248 if not _has_faiss:
--> 249 raise ImportError(
250 "You must install Faiss to use FaissIndex. To do so you can run conda install -c pytorch faiss-cpu or conda install -c pytorch faiss-gpu. "
251 "A community supported package is also available on pypi: pip install faiss-cpu or pip install faiss-gpu. "
```

because
```_has_faiss = importlib.util.find_spec("faiss") is not None```    at the beginning of ```datasets/search.py``` returns ```False```
when
the same code at colab notebook returns
```ModuleSpec(name='faiss', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7b7851449f50>, origin='/usr/local/lib/python3.11/dist-packages/faiss/init.py', submodule_search_locations=['/usr/local/lib/python3.11/dist-packages/faiss'])```

But
```
import datasets
datasets.search._has_faiss
```
at ```colab notebook``` also returns ```False```

The same story with   ```_has_elasticsearch```

### Steps to reproduce the bug

1. Follow https://huggingface.co/learn/nlp-course/chapter5/6?fw=pt at Google Colab
2. till ```embeddings_dataset.add_faiss_index(column='embeddings')```
3. ```embeddings_dataset.add_elasticsearch_index(column='embeddings')```
4. https://colab.research.google.com/drive/1h2cjuiClblqzbNQgrcoLYOC8zBqTLLcv#scrollTo=3ddzRp72auOF

### Expected behavior

I've only started Tutorial and don't know exactly. But something tells me that ```embeddings_dataset.add_faiss_index(column='embeddings')```
should work without ```Import Error```

### Environment info

Google Colab notebook with default config
Problems with local dataset after upgrade from 3.3.2 to 3.4.0
State: open

### Describe the bug

I was not able to open a local saved dataset anymore that was created using an older datasets version after the upgrade yesterday from datasets 3.3.2 to 3.4.0

The traceback is 

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/arrow/arrow.py", line 67, in _generate_tables
    batches = pa.ipc.open_stream(f)
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 190, in open_stream
    return RecordBatchStreamReader(source, options=options,
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 52, in __init__
    self._open(source, options=options, memory_pool=memory_pool)
  File "pyarrow/ipc.pxi", line 1006, in pyarrow.lib._RecordBatchStreamReader._open
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Expected to read 538970747 metadata bytes, but only read 2126

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1855, in _prepare_split_single
    for _, table in generator:
  File "/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/arrow/arrow.py", line 69, in _generate_tables
    reader = pa.ipc.open_file(f)
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 234, in open_file
    return RecordBatchFileReader(
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 110, in __init__
    self._open(source, footer_offset=footer_offset,
  File "pyarrow/ipc.pxi", line 1090, in pyarrow.lib._RecordBatchFileReader._open
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not an Arrow file
```

### Steps to reproduce the bug

Load a dataset from a local folder with 

```
dataset = load_dataset(
                args.train_data_dir,
                cache_dir=args.cache_dir,
            )
```
as it is done for example in the training script for SD3 controlnet.

This is the minimal script to test it: 

```
from datasets import load_dataset

def main():
    dataset = load_dataset(
        "local_dataset",  
    )
    print(dataset)
    print("Sample data:", dataset["train"][0])

if __name__ == "__main__":
    main()
````

### Expected behavior

Work in 3.4.0 like in 3.3.2

### Environment info

- `datasets` version: 3.4.0
- Platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
Add IterableDataset.decode with multithreading
State: closed

Useful for dataset streaming for multimodal datasets, and especially for lerobot.
It speeds up streaming up to 20 times.

When decoding is enabled (default), media types are decoded:

* audio -> dict of "array" and "sampling_rate" and "path"
* image -> PIL.Image
* video -> torchvision.io.VideoReader

You can enable multithreading using `num_threads`. This is especially useful to speed up remote
data streaming. However it can be slower than `num_threads=0` for local data on fast disks.

PS: Disabling decoding is useful if you want to iterate on the paths or bytes of the media files
without actually decoding their content.

Example: Speed up streaming with multithreading:

```py
>>> import os
>>> from datasets import load_dataset
>>> from tqdm import tqdm
>>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True)
>>> num_threads = min(32, (os.cpu_count() or 1) + 4)
>>> ds = ds.decode(num_threads=num_threads)
>>> for _ in tqdm(ds):  # 20 times faster !
...     ...
```

why not multiprocessing ? decoding is done with the GIL released in soundfile/PIL/torchvision so multiprocessing would just use more memory

TODO
- [x] test
- [x] add to docs
Epochs shortened after resuming mid-epoch with Iterable dataset+StatefulDataloader(persistent_workers=True)
State: closed

### Describe the bug

When `torchdata.stateful_dataloader.StatefulDataloader(persistent_workers=True)` the epochs after resuming only iterate through the examples that were left in the epoch when the training was interrupted. For example, in the script below training is interrupted on step 124 (epoch 1) when 3 batches are left. Then after resuming, the rest of epochs (2 and 3) only iterate through these 3 batches.

### Steps to reproduce the bug

Run the following script with and with PERSISTENT_WORKERS=true.

```python
# !/usr/bin/env python3
# torch==2.5.1
# datasets==3.3.2
# torchdata>=0.9.0
import datasets
import pprint
from torchdata.stateful_dataloader import StatefulDataLoader

import os

PERSISTENT_WORKERS = (
    os.environ.get("PERSISTENT_WORKERS", "False").lower() == "true"
)

# PERSISTENT_WORKERS = True  # Incorrect resume


# ds = datasets.load_from_disk("dataset").to_iterable_dataset(num_shards=4)
def generator():
    for i in range(128):
        yield {"x": i}


ds = datasets.Dataset.from_generator(
    generator, features=datasets.Features({"x": datasets.Value("int32")})
).to_iterable_dataset(num_shards=4)

dl = StatefulDataLoader(
    ds, batch_size=2, num_workers=2, persistent_workers=PERSISTENT_WORKERS
)
global_step = 0
epoch = 0
ds_state_dict = None
state_dict = None
resumed = False
while True:
    if epoch >= 3:
        break
    if state_dict is not None:
        dl.load_state_dict(state_dict)
        state_dict = None
        ds_state_dict = None
        resumed = True
        print("resumed")
    for i, batch in enumerate(dl):
        print(f"epoch: {epoch}, global_step: {global_step}, batch: {batch}")
        global_step += 1  # consume datapoint
        # simulate error
        if global_step == 124 and not resumed:
            ds_state_dict = ds.state_dict()
            state_dict = dl.state_dict()
            print("checkpoint")
            print("ds_state_dict")
            pprint.pprint(ds_state_dict)
            print("dl_state_dict")
            pprint.pprint(state_dict)
            break

    if state_dict is None:
        ds.set_epoch(epoch)
        epoch += 1
```

The script checkpoints when there are three batches left in the second epoch. After resuming, only the last three batches are repeated in the rest of the epochs.

If it helps, following are the two state_dicts for the dataloader save at the same step with the two settings. The left one is for `PERSISTENT_WORKERS=False`

![Image](https://github.com/user-attachments/assets/c97d6502-d7bd-4ef4-ae2d-66fe1a9732b1)

### Expected behavior

All the elements in the dataset should be iterated through in the epochs following the one where we resumed. The expected behavior can be seen by setting `PERSISTENT_WORKERS=False`. 

### Environment info

torch==2.5.1
datasets==3.3.2
torchdata>=0.9.0
pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'
State: closed

### Describe the bug

A dict with its keys are all str but get following error
```python
test_data=[{'input_ids':[1,2,3],'labels':[[Counter({2:1})]]}]
dataset = datasets.Dataset.from_list(test_data)
```
```bash
pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'
```

### Steps to reproduce the bug

.

### Expected behavior

.

### Environment info

datasets 3.3.2
Fix small bugs with async map
State: closed

helpful for the next PR to enable parallel image/audio/video decoding and make multimodal datasets go brr (e.g. for lerobot)

- fix with_indices
- fix resuming with save_state_dict() / load_state_dict() - omg that wasn't easy
- remove unnecessary decoding in map() to enable parallelism in FormattedExampleIterable later

small bonus: keeping features in batch()
index error when num_shards > len(dataset)
State: open

In `ds.push_to_hub()` and `ds.save_to_disk()`, `num_shards` must be smaller than or equal to the number of rows in the dataset, but currently this is not checked anywhere inside these functions. Attempting to invoke these functions with `num_shards > len(dataset)` should raise an informative `ValueError`.

I frequently work with datasets with a small number of rows where each row is pretty large, so I often encounter this issue, where the function runs until the shard index in `ds.shard(num_shards, indx)` goes out of bounds. Ideally, a `ValueError` should be raised before reaching this point (i.e. as soon as `ds.push_to_hub()` or `ds.save_to_disk()` is invoked with `num_shards > len(dataset)`).

It seems that adding something like:
```python
if len(self) < num_shards:
   raise ValueError(f"num_shards ({num_shards}) must be smaller than or equal to the number of rows in the dataset ({len(self)}). Please either reduce num_shards or increase max_shard_size to make sure num_shards <= len(dataset).")
```
to the beginning of the definition of the `ds.shard()` function [here](https://github.com/huggingface/datasets/blob/f693f4e93aabafa878470c80fd42ddb10ec550d6/src/datasets/arrow_dataset.py#L4728) would deal with this issue for both `ds.push_to_hub()` and `ds.save_to_disk()`, but I'm not exactly sure if this is the best place to raise the `ValueError` (it seems that a more correct way to do it would be to write separate checks for `ds.push_to_hub()` and `ds.save_to_disk()`). I'd be happy to submit a PR if you think something along these lines would be acceptable.
Flexible Loader
State: open
Labels: name: enhancement

### Feature request

Can we have a utility function that will use `load_from_disk` when given the local path and `load_dataset` if given an HF dataset?

It can be something as simple as this one:

```
def load_hf_dataset(path_or_name):
    if os.path.exists(path_or_name):
        return load_from_disk(path_or_name)
    else:
        return load_dataset(path_or_name)
```

### Motivation

This can be done inside the user codebase, too, but in my experience, it becomes repetitive code.

### Your contribution

I can open a pull request.
Allow dataset row indexing with np.int types (#7423)
State: closed

@lhoestq 
Proposed fix for #7423. Added a couple simple tests as requested. I had some test failures related to Java and pyspark even when installing with dev but these don't seem to be related to the changes here and fail for me even on clean main.

The typeerror raised when using the wrong type is: "Wrong key type: '{key}' of type '{type(key)}'. Expected one of int, slice, range, str or Iterable." I think that is fine. But I could modify the int part to something more generic (although I'm not sure what) if wanted.
Refactor `string_to_dict` to return `None` if there is no match instead of raising `ValueError`
State: closed

Making this change, as encouraged here:

* https://github.com/huggingface/datasets/pull/7434#discussion_r1979933054

instead of having the pattern of using `try`-`except` to handle when there is no match, we can instead check if the return value is `None`; we can also assert that the return value should not be `None` if we know that should be true
Refactor `Dataset.map` to reuse cache files mapped with different `num_proc`
State: closed

Fixes #7433

This refactor unifies `num_proc is None or num_proc == 1` and `num_proc > 1`; instead of handling them completely separately where one uses a list of kwargs and shards and the other just uses a single set of kwargs and `self`, by wrapping the `num_proc == 1` case in a list and making the difference just whether or not you use a pool, you set up either case to be able to load each other's cache files just by changing `num_shards`; `num_proc == 1` can sequentially load the shards of a dataset mapped `num_shards > 1` and map any missing shards

Other than the structural refactor, the main contribution of this PR is `existing_cache_file_map`, which uses a regex of `cache_file_name` and `suffix_template` to find existing cache files, grouped by their `num_shards`; using this data structure, we can reset `num_shards` to an existing set of cache files, and load them accordingly
`Dataset.map` ignores existing caches and remaps when ran with different `num_proc`
State: closed

### Describe the bug

If you `map` a dataset and save it to a specific `cache_file_name` with a specific `num_proc`, and then call map again with that same existing `cache_file_name` but a different `num_proc`, the dataset will be re-mapped.

### Steps to reproduce the bug

1. Download a dataset
```python
import datasets

dataset = datasets.load_dataset("ylecun/mnist")
```

```
Generating train split: 100%|██████████| 60000/60000 [00:00<00:00, 116429.85 examples/s]
Generating test split: 100%|██████████| 10000/10000 [00:00<00:00, 103310.27 examples/s]
```

2. `map` and cache it with a specific `num_proc`

```python
cache_file_name="./cache/train.map"
dataset["train"].map(lambda x: x, cache_file_name=cache_file_name, num_proc=2)
```

```
Map (num_proc=2): 100%|██████████| 60000/60000 [00:01<00:00, 53764.03 examples/s]
```

3. `map` it with a different `num_proc` and the same `cache_file_name` as before

```python
dataset["train"].map(lambda x: x, cache_file_name=cache_file_name, num_proc=3)
```

```
Map (num_proc=3): 100%|██████████| 60000/60000 [00:00<00:00, 65377.12 examples/s]
```

### Expected behavior

If I specify an existing `cache_file_name`, I don't expect using a different `num_proc` than the one that was used to generate it to cause the dataset to have be be re-mapped.

### Environment info

```console
$ datasets-cli env

- `datasets` version: 3.3.2
- Platform: Linux-5.15.0-131-generic-x86_64-with-glibc2.35
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
```
Issues with large Datasets
State: open

### Describe the bug

If the coco annotation file is too large the dataset will not be able to load it, not entirely sure were the issue is but I am guessing it is due to the code trying to load it all as one line into a dataframe. This was for object detections.

My current work around is the following code but would be nice to be able to do it without worrying about it also probably there is a better way of doing it:
`
            dataset_dict = json.load(open("./local_data/annotations/train.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'], 'categories': dataset_dict['categories']}, ignore_index=True)
            train=Dataset.from_pandas(df)

            dataset_dict = json.load(open("./local_data/annotations/validation.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'],
                             'categories': dataset_dict['categories']}, ignore_index=True)
            val = Dataset.from_pandas(df)
            dataset_dict = json.load(open("./local_data/annotations/test.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'],
                             'categories': dataset_dict['categories']}, ignore_index=True)
            test = Dataset.from_pandas(df)
            dataset = DatasetDict({'train': train, 'validation': val, 'test': test})
`

### Steps to reproduce the bug

1) step up directory in and have the json files in coco format

-local_data
|-images
|---1.jpg
|---2.jpg
|---....
|---n.jpg
|-annotations
|---test.json
|---train.json
|---validation.json

2) try to load local_data into a dataset if the file is larger than about 300kb it will cause an error.

### Expected behavior

That it loads the jsons preferably in the same format as it has done with a smaller size.

### Environment info

- `datasets` version: 3.3.3.dev0
- Platform: Linux-6.11.0-17-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.29.0
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
Error in code "Time to slice and dice" from course "NLP Course"
State: closed

### Describe the bug

When we execute code 
```
frequencies = (
    train_df["condition"]
    .value_counts()
    .to_frame()
    .reset_index()
    .rename(columns={"index": "condition", "condition": "frequency"})
)
frequencies.head()
```
answer should be like this

condition      | frequency
birth control | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

but he is different

frequency      | count
birth control  | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

this is not correct, correct code
```
frequencies = (
    train_df["condition"]
    .value_counts()
    .to_frame()
    .reset_index()
    .rename(columns={"index": "condition", "count": "frequency"})
)
````

### Steps to reproduce the bug

```
frequencies = (
train_df["condition"]
.value_counts()
.to_frame()
.reset_index()
.rename(columns={"index": "condition", "condition": "frequency"})
)
frequencies.head()
```

### Expected behavior


condition      | frequency
birth control | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

### Environment info

Google Colab
load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")  TypeError: 'NoneType' object is not callable
State: open

### Describe the bug

from datasets import load_dataset
lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")

or
configs = get_dataset_config_names("livecodebench/code_generation_lite", trust_remote_code=True)

both error:
Traceback (most recent call last):
File "", line 1, in
File "/workspace/miniconda/envs/grpo/lib/python3.10/site-packages/datasets/load.py", line 2131, in load_dataset
builder_instance = load_dataset_builder(
File "/workspace/miniconda/envs/grpo/lib/python3.10/site-packages/datasets/load.py", line 1888, in load_dataset_builder
builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable

### Steps to reproduce the bug

from datasets import get_dataset_config_names

configs = get_dataset_config_names("livecodebench/code_generation_lite", trust_remote_code=True)
OR 
lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")

### Expected behavior

load  datasets  livecodebench/code_generation_lite


### Environment info

 import  datasets
version '3.3.2'
Row indexing a dataset with numpy integers
State: open
Labels: name: enhancement

### Feature request

Allow indexing datasets with a scalar numpy integer type.

### Motivation

Indexing a dataset with a scalar numpy.int* object raises a TypeError. This is due to the test in `datasets/formatting/formatting.py:key_to_query_type`

``` python
def key_to_query_type(key: Union[int, slice, range, str, Iterable]) -> str:
    if isinstance(key, int):
        return "row"
    elif isinstance(key, str):
        return "column"
    elif isinstance(key, (slice, range, Iterable)):
        return "batch"
    _raise_bad_key_type(key)
```

In the row case, it checks if key is an int, which returns false when key is integer like but not a builtin python integer type. This is counterintuitive because a numpy array of np.int64s can be used for the batch case.

For example:

``` python
import numpy as np

import datasets

dataset = datasets.Dataset.from_dict({"a": [1, 2, 3, 4], "b": [5, 6, 7, 8]})

# Regular indexing
dataset[0]
dataset[:2]

# Indexing with numpy data types (expect same results)
idx = np.asarray([0, 1])
dataset[idx]  # Succeeds when using an array of np.int64 values
dataset[idx[0]]  # Fails with TypeError when using scalar np.int64
```

For the user, this can be solved by wrapping `idx[0]` in `int` but the test could also be changed in `key_to_query_type` to accept a less strict definition of int.

``` diff
+import numbers
+
def key_to_query_type(key: Union[int, slice, range, str, Iterable]) -> str:
+   if isinstance(key, numbers.Integral):
-   if isinstance(key, int):
        return "row"
    elif isinstance(key, str):
        return "column"
    elif isinstance(key, (slice, range, Iterable)):
        return "batch"
    _raise_bad_key_type(key)
```

Looking at how others do it, pandas has an `is_integer` definition that it checks which uses `is_integer_object` defined in `pandas/_libs/utils.pxd`:

``` cython
cdef inline bint is_integer_object(object obj) noexcept:
    """
    Cython equivalent of

    `isinstance(val, (int, np.integer)) and not isinstance(val, (bool, np.timedelta64))`

    Parameters
    ----------
    val : object

    Returns
    -------
    is_integer : bool

    Notes
    -----
    This counts np.timedelta64 objects as integers.
    """
    return (not PyBool_Check(obj) and isinstance(obj, (int, cnp.integer))
            and not is_timedelta64_object(obj))
```

This would be less flexible as it explicitly checks for numpy integer, but worth noting that they had the need to ensure the key is not a bool.

### Your contribution

I can submit a pull request with the above changes after checking that indexing succeeds with the numpy integer type. Or if there is a different integer check that would be preferred I could add that.

If there is a reason not to want this behavior that is fine too.
Import order crashes script execution
State: open

### Describe the bug

Hello,

I'm trying to convert an HF dataset into a TFRecord so I'm importing `tensorflow` and `datasets` to do so.
Depending in what order I'm importing those librairies, my code hangs forever and is unkillable (CTRL+C doesn't work, I need to kill my shell entirely).

Thank you for your help
🙏 

### Steps to reproduce the bug

If you run the following script, this will hang forever : 
```python
import tensorflow as tf
import datasets

dataset = datasets.load_dataset("imagenet-1k", split="validation", streaming=True)
print(next(iter(dataset)))
```

however running the following will work fine (I just changed the order of the imports) :
```python
import datasets
import tensorflow as tf

dataset = datasets.load_dataset("imagenet-1k", split="validation", streaming=True)
print(next(iter(dataset)))
```

### Expected behavior

I'm expecting the script to reach the end and my case print the content of the first item in the dataset 
```
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=408x500 at 0x70C646A03110>, 'label': 91}
```

### Environment info

```
$ datasets-cli env
- `datasets` version: 3.3.2
- Platform: Linux-6.8.0-1017-aws-x86_64-with-glibc2.35
- Python version: 3.11.7
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
```

I'm also using `tensorflow==2.18.0`.
pyarrow.lib.arrowinvalid: cannot mix list and non-list, non-null values with map function
State: open

### Describe the bug

Encounter pyarrow.lib.arrowinvalid error with map function in some example when loading the dataset

### Steps to reproduce the bug

```
from datasets import load_dataset
from PIL import Image, PngImagePlugin

dataset = load_dataset("leonardPKU/GEOQA_R1V_Train_8K")
system_prompt="You are a helpful AI Assistant"
def make_conversation(example):
    prompt = []

    prompt.append({"role": "system", "content": system_prompt})
    prompt.append(
            {
                "role": "user", 
                "content": [
                        {"type": "image"},
                        {"type": "text", "text": example["problem"]},
                        ]
            }
        )
    return {"prompt": prompt}

def check_data_types(example):
    for key, value in example.items():
        if key == 'image':
            if not isinstance(value, PngImagePlugin.PngImageFile):
                print(value)
        if key == "problem" or key == "solution":
            if not isinstance(value, str):
                print(value)

    return example

dataset = dataset.map(check_data_types)
dataset = dataset.map(make_conversation)
```

### Expected behavior

Successfully process the dataset with map

### Environment info

datasets==3.3.1
Shard Dataset at specific indices
State: open

I have a dataset of sequences, where each example in the sequence is a separate row in the dataset (similar to LeRobotDataset). When running `Dataset.save_to_disk` how can I provide indices where it's possible to shard the dataset such that no episode spans more than 1 shard. Consequently, when I run `Dataset.load_from_disk`, how can I load just a subset of the shards to save memory and time on different ranks?

I guess an alternative to this would be, given a loaded `Dataset`, how can I run `Dataset.shard` such that sharding doesn't split any episode across shards?
Documentation on multiple media files of the same type with WebDataset
State: open

The [current documentation](https://huggingface.co/docs/datasets/en/video_dataset) on a creating a video dataset includes only examples with one media file and one json. It would be useful to have examples where multiple files of the same type are included. For example, in a sign language dataset, you may have a base video and a video annotation of the extracted pose. According to the WebDataset documentation, this should be able to be done with period separated filenames. For example:

```e39871fd9fd74f55.base.mp4
e39871fd9fd74f55.pose.mp4
e39871fd9fd74f55.json
f18b91585c4d3f3e.base.mp4
f18b91585c4d3f3e.pose.mp4
f18b91585c4d3f3e.json
... 
```

If you can confirm that this method of including multiple media files works with huggingface datasets and include an example in the documentation, I'd appreciate it.
Index Error Invalid Ket is out of bounds for size 0 for code-search-net/code_search_net dataset
State: open

### Describe the bug

I am trying to do model pruning on sentence-transformers/all-mini-L6-v2 for the code-search-net/code_search_net dataset using INCTrainer class 
However I am getting below error

```
raise IndexError(f"Invalid Key: {key is our of bounds for size {size}")
IndexError: Invalid key: 1840208 is out of bounds for size 0
```

### Steps to reproduce the bug

Model pruning on the above dataset using the below guide 

https://huggingface.co/docs/optimum/en/intel/neural_compressor/optimization#pruning
### Expected behavior

The modsl should be successfully pruned 

### Environment info

Torch version: 2.4.1
Python version: 3.8.10