KRadim/github-issues-minimal huggingface dataset:
Adding column with dict struction when mapping lead to wrong order
State: open

### Describe the bug

in `map()` function, I want to add a new column with a dict structure.
```
def map_fn(example):
  example['text'] = {'user': ..., 'assistant': ...}
  return example
```
However this leads to a wrong order `{'assistant':..., 'user':...}` in the dataset.
Thus I can't concatenate two datasets due to the different feature structures.
[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example
This seems an issue in low level pyarrow library instead of datasets, however, I think datasets should allow concatenate two datasets actually in the same structure.

### Steps to reproduce the bug

[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example

### Expected behavior

two datasets could be concatenated.

### Environment info

N/A
ArrayXD with None as leading dim incompatible with DatasetCardData
State: open

### Describe the bug

Creating a dataset with ArrayXD features leads to errors when downloading from hub due to DatasetCardData removing the Nones

@lhoestq 

### Steps to reproduce the bug

```python
import numpy as np
from datasets import Array2D, Dataset, Features, load_dataset

def examples_generator():
    for i in range(4):
        yield {
            "array_1d": np.zeros((10,1), dtype="uint16"),
            "array_2d": np.zeros((10, 1), dtype="uint16"),
        }

features = Features(array_1d=Array2D((None,1), "uint16"), array_2d=Array2D((None, 1), "uint16"))
dataset = Dataset.from_generator(examples_generator, features=features)
dataset.push_to_hub("alex-hh/test_array_1d2d")
ds = load_dataset("alex-hh/test_array_1d2d")
```

Source of error appears to be DatasetCardData.to_dict invoking DatasetCardData._remove_none
```python
from huggingface_hub import DatasetCardData
from datasets.info import DatasetInfosDict

dataset_card_data = DatasetCardData()
DatasetInfosDict({"default": dataset.info.copy()}).to_dataset_card_data(dataset_card_data)
print(dataset_card_data.to_dict())  # removes Nones in shape
```

### Expected behavior

Should be possible to load datasets saved with shape None in leading dimension

### Environment info

3.0.2 and latest huggingface_hub
Feature Request: Add functionality to pass split types like train, test in DatasetDict.map
State: closed

Hello datasets!

We often encounter situations where we need to preprocess data differently depending on split types such as train, valid, and test. 

However, while DatasetDict.map has features to pass rank or index, there's no functionality to pass split types. 
Therefore, I propose adding a 'with_splits' parameter to DatasetDict, which would allow passing the split type through fn_kwargs.
incompatibily issue when using load_dataset with datasets==3.0.1
State: open

### Describe the bug

There is a bug when using load_dataset with dataset version at 3.0.1 .
Please see below in the "steps to reproduce the bug".
To resolve the bug, I had to downgrade to version 2.21.0
OS: Ubuntu 24 (AWS instance)
Python: same bug under 3.12 and 3.10


The error I had was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/load.py", line 2096, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 1647, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 977, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_6_0/cb17afd34f5799f97e8f48398748f83006335b702bd785f9880797838d541b81/common_voice_6_0.py", line 159, in _split_generators
    archive_path = dl_manager.download(self._get_bundle_url(self.config.name, bundle_url_template))
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_manager.py", line 150, in download
    download_config = self.download_config.copy()
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_config.py", line 73, in copy
    return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
TypeError: DownloadConfig.__init__() got an unexpected keyword argument 'ignore_url_params'

### Steps to reproduce the bug

1. install dataset with ```pip install datasets --upgrade```
2. launch python; from datasets import loaad_dataset
3. run load_dataset("mozilla-foundation/common_voice_6_0")
4. exit python
5. uninstall datasets; then ```pip install datasets==2.21.0```
6. launch python; from datasets import loaad_dataset
7. run load_dataset("mozilla-foundation/common_voice_6_0")
8. Everything runs great now

### Expected behavior

Be able to download a dataset without error

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 3.0.1
- Platform: Linux-6.8.0-1017-aws-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1
Video support
State: closed

(wip and experimental)

adding the `Video` type based on `VideoReader` from `decord`

```python
>>>from datasets import load_dataset
>>> ds = load_dataset("path/to/videos", split="train").with_format("torch")
>>> print(ds[0]["video"])
<decord.video_reader.VideoReader object at 0x337a47910>
>>> print(ds[0]["video"][0])
tensor([[[73, 73, 73],
         [73, 73, 73],
         [73, 73, 73],
         ...,
         [23, 23, 23],
         [23, 23, 23],
         [23, 23, 23]]], dtype=torch.uint8)
```

the storage is the same as for audio and images: `{"path": pa.string(), "bytes": pa.binary()}` and I did a small to keep the hf:// URL in the "path" field if possible, this way the viewer can link to fiels on the hub if possible
Composite (multi-column) features
State: open
Labels: name: enhancement

### Feature request

Structured data types (graphs etc.) might often be most efficiently stored as multiple columns, which then need to be combined during feature decoding

Although it is currently possible to nest features as structs, my impression is that in particular when dealing with e.g. a feature composed of multiple numpy array / ArrayXD's, it would be more efficient to store each ArrayXD as a separate column (though I'm not sure by how much)

Perhaps specification / implementation could be supported by something like:

```
features=Features(**{("feature0", "feature1")=Features(feature0=Array2D((None,10), dtype="float32"), feature1=Array2D((None,10), dtype="float32"))
```

### Motivation

Defining efficient composite feature types based on numpy arrays for representing data such as graphs with multiple node and edge attributes is currently challenging.

### Your contribution

Possibly able to contribute
fast array extraction
State: open

Implements #7210 using method suggested in https://github.com/huggingface/datasets/pull/7207#issuecomment-2411789307


```python
import numpy as np
from datasets import Dataset, Features, Array3D
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```
~0.02 s vs 0.9s on main

```python
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
< 0.01 s vs 1.3 s on main

@lhoestq I can see this breaks a bunch of array-related tests but can update the test cases if you would support making this change?

I also added an Array1D feature which will always be decoded into a numpy array and likewise improves extraction performance:

```python
from datasets import Dataset, Features, Array1D, Sequence, Value
array_features=Features(**{"array0": Array1D((None,), dtype="float32"), "array1": Array1D((None,), dtype="float32")})
sequence_features=Features(**{"array0": Sequence(feature=Value("float32"), length=-1), "array1": Sequence(feature=Value("float32"), length=-1)})
array_dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [20000,10000]*25] for i in range(2)}, features=array_features)
sequence_dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [20000,10000]*25] for i in range(2)}, features=sequence_features)


```python
t0 = time.time()
for ex in array_dataset.to_iterable_dataset():
    pass
t1 = time.time()
```
< 0.01 s

```python
t0 = time.time()
for ex in sequence_dataset.to_iterable_dataset():
    pass
t1 = time.time()
```
~1.1s

And also added support for extracting structs of arrays as dicts of numpy arrays:

```python
import numpy as np
from datasets import Dataset, Features, Array3D, Sequence
features=Features(struct={"array0": Array3D((None,10,10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")}, _list=Sequence(feature=Array3D((None,10,10), dtype="float32")))
dataset = Dataset.from_dict({"struct": [{f"array{i}": np.zeros((x,10,10), dtype=np.float32) for i in range(2)} for x in [2000,1000]*25], "_list": [[np.zeros((x,10,10), dtype=np.float32) for i in range(2)] for x in [2000,1000]*25]}, features=features)
```

```python
t0 = time.time()
for ex in dataset.to_iterable_dataset():
    pass
t1 = time.time()
assert isinstance(ex["struct"]["array0"], np.ndarray) and ex["struct"]["array0"].ndim == 3
```
~0.02 s and no exception vs ~7s with an exception on main
Add R as a How to use from the Polars (R) Library as an option
State: open
Labels: name: enhancement

### Feature request

The boiler plate code to access a dataset via the hugging face file system is very useful. Please addd 


## Add Polars (R) option
The equivailent code works, because the [Polars-R](https://github.com/pola-rs/r-polars) wrapper has hugging faces funcitonaliy as well.

```r
library(polars)

df <- pl$read_parquet("hf://datasets/SALURBAL/core__admin_cube_public/core__admin_cube_public.parquet")
```

## Polars (python) option
![image](https://github.com/user-attachments/assets/8f1bcd19-e578-4b18-b324-7cc00b80ac0a)


## Libraries Currently

![image](https://github.com/user-attachments/assets/0cf50063-f9db-443c-97b4-3ef0664b6e6e)




### Motivation

There are many data/analysis/research/statistics teams (particularly in academia and pharma) that use R as the default language. R has great integration with most of the newer data techs (arrow, parquet, polars) and having this included could really help in bringing this community into the hugging faces ecosystem.

**This is a small/low-hanging-fruit front end change but would make a big impact expanding the community**

### Your contribution

I am not sure which repositroy this should be in, but I have experience in R, Python and JS and happy to submit a PR in the appropriate repository.
Huggingface GIT returns null as Content-Type instead of application/x-git-receive-pack-result
State: open

### Describe the bug

We push changes to our datasets programmatically. Our git client jGit reports that the hf git server returns null as Content-Type after a push.

### Steps to reproduce the bug

A basic kotlin application:
```
   val person = PersonIdent(
        "padmalcom",
        "padmalcom@sth.com"
    )

    val cp = UsernamePasswordCredentialsProvider(
        "padmalcom",
        "mysecrettoken"
    )

    val git =
        KGit.cloneRepository {
            setURI("https://huggingface.co/datasets/sth/images")
            setTimeout(60)
            setProgressMonitor(TextProgressMonitor())
            setCredentialsProvider(cp)
        }

    FileOutputStream("./images/images.csv").apply { writeCsv(images) }
    git.add {
        addFilepattern("images.csv")
    }

    for (i in images) {
        FileUtils.copyFile(
            File("./files/${i.id}"),
            File("./images/${i.id + File(i.fileName).extension }")
        )
        git.add {
            addFilepattern("${i.id + File(i.fileName).extension }")
        }
    }

    val revCommit = git.commit {
        author = person
        message = "Uploading images at " + LocalDateTime.now()
            .format(DateTimeFormatter.ISO_DATE_TIME)
        setCredentialsProvider(cp)
    }

    val push = git.push {
        setCredentialsProvider(cp)
    }
```

### Expected behavior

The git server is expected to return the Content-Type _application/x-git-receive-pack-result_.

### Environment info

It is independent from the datasets library.
Fallback to arrow defaults when loading dataset with custom features that aren't registered locally
State: open

### Describe the bug

Datasets allows users to create and register custom features.

However if datasets are then pushed to the hub, this means that anyone calling load_dataset without registering the custom Features in the same way as the dataset creator will get an error message.

It would be nice to offer a fallback in this case.

### Steps to reproduce the bug

```python
load_dataset("alex-hh/custom-features-example")
```
(Dataset creation process - must be run in separate session so that NewFeature isn't registered in session in which download is attempted:)

```python
from dataclasses import dataclass, field
import pyarrow as pa
from datasets.features.features import register_feature

from datasets import Dataset, Features, Value, load_dataset
from datasets import Feature

@dataclass
class NewFeature(Feature):
    _type: str = field(default="NewFeature", init=False, repr=False)
    def __call__(self):
        return pa.int32()

def examples_generator():
    for i in range(5):
        yield {"feature": i}

ds = Dataset.from_generator(examples_generator, features=Features(feature=NewFeature()))
ds.push_to_hub("alex-hh/custom-features-example")
register_feature(NewFeature, "NewFeature")
```



### Expected behavior

It would be nice, and offer greater extensibility, if there was some kind of graceful fallback mechanism in place for cases where user-defined features are stored in the dataset but not available locally.

### Environment info

3.0.2
add CustomFeature base class to support user-defined features with encoding/decoding logic
State: closed

intended as fix for #7220 if this kind of extensibility is something that datasets is willing to support!


```python
from datasets.features.features import CustomFeature


class ListOfStrs(CustomFeature):
    requires_encoding = True
    def _encode_example(self, value):
        if isinstance(value, str):
            return [str]
        else:
            return value
feats = Features(strlist=ListOfStrs())
feats.encode_example({"strlist": "a"})["strlist"] == feats["strlist"].encode_example("a")
```
Custom features not compatible with special encoding/decoding logic
State: open

### Describe the bug

It is possible to register custom features using datasets.features.features.register_feature (https://github.com/huggingface/datasets/pull/6727)

However such features are not compatible with Features.encode_example/decode_example if they require special encoding / decoding logic because encode_nested_example / decode_nested_example checks whether the feature is in a fixed list of encodable types:

https://github.com/huggingface/datasets/blob/16a121d7821a7691815a966270f577e2c503473f/src/datasets/features/features.py#L1349

This prevents the extensibility of features to complex cases

### Steps to reproduce the bug

```python
class ListOfStrs:
    def encode_example(self, value):
        if isinstance(value, str):
            return [str]
        else:
            return value
feats = Features(strlist=ListOfStrs())
assert feats.encode_example({"strlist": "a"})["strlist"] = feats["strlist"].encode_example("a")}
```

### Expected behavior

Registered feature types should be encoded based on some property of the feature (e.g. requires_encoding)?

### Environment info

3.0.2
ds.map(f, num_proc=10) is slower than df.apply
State: open

### Describe the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

time cost: 
1. df.apply: 100%|██████████| 12500592/12500592 [00:13<00:00, 959825.47it/s]
2. ds.map: Map (num_proc=10):  31%
 3899028/12500592 [00:28<00:38, 222532.89 examples/s]

### Steps to reproduce the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

### Expected behavior

ds.map is ~num_proc faster than df.apply

### Environment info

pandas: 2.2.2
datasets: 2.19.1
Iterable dataset map with explicit features causes slowdown for Sequence features
State: open

### Describe the bug

When performing map, it's nice to be able to pass the new feature type, and indeed required by interleave and concatenate datasets.

However, this can cause a major slowdown for certain types of array features due to the features being re-encoded.

This is separate to the slowdown reported in #7206 

### Steps to reproduce the bug

```
from datasets import Dataset, Features, Array3D, Sequence, Value
import numpy as np
import time
features=Features(**{"array0": Sequence(feature=Value("float32"), length=-1), "array1": Sequence(feature=Value("float32"), length=-1)})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [5000,10000]*25] for i in range(2)}, features=features)
```

```
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~1.5 s on main


```
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x, features=features)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~ 3 s on main

### Expected behavior

I'm not 100% sure whether passing new feature types to formatted outputs of map should be supported or not, but assuming it should, then there should be a cost-free way to specify the new feature type - knowing feature type is required by interleave_datasets and concatenate_datasets for example

### Environment info

3.0.2
Formatted map + with_format(None) changes array dtype for iterable datasets
State: open

### Describe the bug

When applying with_format -> map -> with_format(None), array dtypes seem to change, even if features are passed 

### Steps to reproduce the bug

```python
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32")})
dataset = Dataset.from_dict({f"array0": [np.zeros((100,10,10), dtype=np.float32)]*25}, features=features)
ds = dataset.to_iterable_dataset().with_format("numpy").map(lambda x: x, features=features)
ex_0 = next(iter(ds))
ds = dataset.to_iterable_dataset().with_format("numpy").map(lambda x: x, features=features).with_format(None)
ex_1 = next(iter(ds))
assert ex_1["array0"].dtype == ex_0["array0"].dtype, f"{ex_1['array0'].dtype} {ex_0['array0'].dtype}"
```


### Expected behavior

Dtypes should be preserved.

### Environment info

3.0.2
Add with_rank to Dataset.from_generator
State: open
Labels: name: enhancement

### Feature request

Add `with_rank` to `Dataset.from_generator` similar to `Dataset.map` and `Dataset.filter`. 

### Motivation

As for `Dataset.map` and `Dataset.filter`, this is useful when creating cache files using multi-GPU, where the rank can be used to select GPU IDs. For now, rank can be added in the `gen_kwars` argument; however, this, in turn, includes the rank when computing the fingerprint.

### Your contribution

Added #7199 which passes rank based on the `job_id` set by `num_proc`.
Windows do not supprot signal.alarm and singal.signal
State: open

### Describe the bug

signal.alarm and signal.signal are used in the load.py module, but these are not supported by Windows.

### Steps to reproduce the bug

 lighteval accelerate --model_args "pretrained=gpt2,trust_remote_code=True" --tasks "community|kinit_sts" --custom_tasks "community_tasks/kinit_evals.py" --output_dir "./evals"

### Expected behavior

proceed with input(..) method

### Environment info

Windows 11
Describe only selected fields in README
State: open
Labels: name: enhancement

### Feature request

Hi Datasets team! 

Is it possible to add the ability to describe only selected fields of the dataset files in `README.md`? For example, I have this open dataset ([open-llm-leaderboard/results](https://huggingface.co/datasets/open-llm-leaderboard/results?row=0)) and I want to describe only some fields in order not to overcomplicate the Dataset Preview and filter out some fields 

### Motivation

The `Results` dataset for the Open LLM Leaderboard contains json files with a complex nested structure. I would like to add `README.md` there to use the SQL console, for example. But if I describe the structure of this dataset completely, it will overcomplicate the use of Dataset Preview and the total number of columns will exceed 50 

### Your contribution

I'm afraid I'm not familiar with the project structure, so I won't be able to open a PR, but I'll try to help with something else if possible
Convert Array features to numpy arrays rather than lists by default
State: open
Labels: name: enhancement

### Feature request

It is currently quite easy to cause massive slowdowns when using datasets and not familiar with the underlying data conversions by e.g. making bad choices of formatting.

Would it be more user-friendly to set defaults that avoid this as much as possible? e.g. format Array features as numpy arrays rather than python lists

### Motivation

Default array formatting leads to slow performance: e.g.

```python
import numpy as np
from datasets import Dataset, Features, Array3D
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```

```python
t0 = time.time()
for ex in ds:
   pass
t1 = time.time()
```
~1.4 s

```python
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~10s

```python
ds = dataset.with_format("numpy")
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~0.04s

```python
ds = dataset.to_iterable_dataset().with_format("numpy")
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~0.04s

### Your contribution

May be able to contribute
Iterable dataset.filter should not override features
State: closed

### Describe the bug

When calling filter on an iterable dataset, the features get set to None

### Steps to reproduce the bug

import numpy as np
import time
from datasets import Dataset, Features, Array3D

```python
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
ds = dataset.to_iterable_dataset()
orig_column_names = ds.column_names
ds = ds.filter(lambda x: True)
assert ds.column_names == orig_column_names
```

### Expected behavior

Filter should preserve features information

### Environment info

3.0.2
Slow iteration for iterable dataset with numpy formatting for array data
State: open

### Describe the bug

When working with large arrays, setting with_format to e.g. numpy then applying map causes a significant slowdown for iterable datasets.

### Steps to reproduce the bug

```python
import numpy as np
import time
from datasets import Dataset, Features, Array3D

features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```

Then
```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
print(t1-t0)
```

takes 27 s, whereas

```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy")
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
print(t1 - t0)
```

takes ~1s

### Expected behavior

Map should not introduce a slowdown when formatting is enabled.

### Environment info

3.0.2
fix unbatched arrow map for iterable datasets
State: closed

Fixes the bug when applying map to an arrow-formatted iterable dataset described here:

https://github.com/huggingface/datasets/issues/6833#issuecomment-2399903885

```python

from datasets import load_dataset
ds = load_dataset("rotten_tomatoes", split="train", streaming=True)
ds = ds.with_format("arrow").map(lambda x: x)
for ex in ds:
    pass
```

@lhoestq
`from_parquet` return type annotation
State: open

### Describe the bug

As already posted in https://github.com/microsoft/pylance-release/issues/6534, the correct type hinting fails when building a dataset using the `from_parquet` constructor.
Their suggestion is to comprehensively annotate the method's return type to better align with the docstring information.

### Steps to reproduce the bug

```python
from datasets import Dataset

dataset = Dataset.from_parquet(path_or_paths="file")
dataset.map(lambda x: {"new": x["old"]}, batched=True)
```

### Expected behavior

map is a [valid](https://huggingface.co/docs/datasets/v3.0.1/en/package_reference/main_classes#datasets.Dataset.map), no error should be thrown.

### Environment info

- `datasets` version: 3.0.1
- Platform: macOS-15.0.1-arm64-arm-64bit
- Python version: 3.12.6
- `huggingface_hub` version: 0.25.1
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1
Fix the environment variable for huggingface cache
State: closed

Resolve  #6256. As far as I tested, `HF_DATASETS_CACHE` was ignored and I could not specify the cache directory at all except for the default one by this environment variable. `HF_HOME` has worked. Perhaps the recent change on file downloading by `huggingface_hub` could affect this bug.

In my testing, I could not specify the cache directory even by `load_dataset("dataset_name" cache_dir="...")`. It might be another issue. I also welcome any advice to solve this issue.
Add repeat method to datasets
State: closed

Following up on discussion in #6623 and #7198 I thought this would be pretty useful for my case so had a go at implementing.

My main motivation is to be able to call iterable_dataset.repeat(None).take(samples_per_epoch) to safely avoid timeout issues in a distributed training setting. This would provide a straightforward workaround for several open issues related to this situation:  https://github.com/huggingface/datasets/issues/6437, https://github.com/huggingface/datasets/issues/6594, https://github.com/huggingface/datasets/issues/6623, https://github.com/huggingface/datasets/issues/6719.

@lhoestq let me know if this looks on the right track!
ConnectionError: Couldn't reach 'allenai/c4' on the Hub (ConnectionError)数据集下不下来，怎么回事
State: open

### Describe the bug

from datasets import load_dataset
print("11")
traindata = load_dataset('ptb_text_only', 'penn_treebank', split='train')
print("22")
valdata = load_dataset('ptb_text_only',
                           'penn_treebank',
                           split='validation')

### Steps to reproduce the bug

1

### Expected behavior

1

### Environment info

1
concatenate_datasets does not preserve shuffling state
State: open

### Describe the bug

After concatenate datasets on an iterable dataset, the shuffling state is destroyed, similar to #7156 

This means concatenation cant be used for resolving uneven numbers of samples across devices when using iterable datasets in a distributed setting as discussed in #6623 

I also noticed that the number of shards is the same after concatenation, which I found surprising, but I don't understand the internals well enough to know whether this is actually surprising or not

### Steps to reproduce the bug

```python
import datasets
import torch.utils.data


def gen(shards):
    yield {"shards": shards}


def main():
    dataset1 = datasets.IterableDataset.from_generator(
        gen, gen_kwargs={"shards": list(range(25))}  # TODO: how to understand this?
    )
    dataset2 = datasets.IterableDataset.from_generator(
        gen, gen_kwargs={"shards": list(range(25, 50))}  # TODO: how to understand this?
    )
    dataset1 = dataset1.shuffle(buffer_size=1)
    dataset2 = dataset2.shuffle(buffer_size=1)
    print(dataset1.n_shards)
    print(dataset2.n_shards)

    dataset = datasets.concatenate_datasets(
        [dataset1, dataset2]
    )
    print(dataset.n_shards)
    # dataset = dataset1

    dataloader = torch.utils.data.DataLoader(
        dataset,
        batch_size=8,
        num_workers=0,
    )

    for i, batch in enumerate(dataloader):
        print(batch)
    print("\nNew epoch")

    dataset = dataset.set_epoch(1)

    for i, batch in enumerate(dataloader):
        print(batch)


if __name__ == "__main__":
    main()
```

### Expected behavior

Shuffling state should be preserved

### Environment info

Latest datasets
Support of num_workers (multiprocessing) in map for IterableDataset
State: open
Labels: name: enhancement

### Feature request

Currently, IterableDataset doesn't support setting num_worker in .map(), which results in slow processing here. Could we add support for it? As .map() can be run in the batch fashion (e.g., batch_size is default to 1000 in datasets), it seems to be doable for IterableDataset as the regular Dataset.

### Motivation

Improving data processing efficiency

### Your contribution

Testing
Add repeat() for iterable datasets
State: closed
Labels: name: enhancement

### Feature request

It would be useful to be able to straightforwardly repeat iterable datasets indefinitely, to provide complete control over starting and ending of iteration to the user.

An IterableDataset.repeat(n) function could do this automatically

### Motivation

This feature was discussed in this issue https://github.com/huggingface/datasets/issues/7147, and would resolve the need to use the hack of interleave datasets with probability 0 as a simple way to achieve this functionality.

An additional benefit might be the simplification of the use of iterable datasets in a distributed setting:
If the user can assume that datasets will repeat indefinitely, then issues around different numbers of samples appearing on different devices (e.g. https://github.com/huggingface/datasets/issues/6437, https://github.com/huggingface/datasets/issues/6594, https://github.com/huggingface/datasets/issues/6623, https://github.com/huggingface/datasets/issues/6719) can potentially be straightforwardly resolved by simply doing:

ids.repeat(None).take(n_samples_per_epoch)


### Your contribution

I'm not familiar enough with the codebase to assess how straightforward this would be to implement.

If it might be very straightforward, I could possibly have a go.
Solution to issue: #7080 Modified load_dataset function, so that it prompts the user to select a dataset when subdatasets or splits (train, test) are available
State: closed

# Feel free to give suggestions please..

### This PR is raised because of issue: https://github.com/huggingface/datasets/issues/7080

![image](https://github.com/user-attachments/assets/8fbc604f-f0a5-4a59-a63e-aa4c26442c83)

### This PR gives solution to https://github.com/huggingface/datasets/issues/7080

1. Checking whether the dataset has splits or subdatasets.
2. Printing the available splits/subdatasets.
3. Asking the user to choose which one to load.
4. Loading only the selected dataset based on the user's input.

### Key Changes:

1. Available Splits/Subdatasets: The code checks for available splits/subdatasets using builder_instance.info.splits.keys().
2. User Prompt: If splits are found, it prints them out and prompts the user to select one.
3. Loading Based on User Input: The dataset is loaded based on the user's choice.

This way, the dataset loading function will interactively prompt the user to select which subdataset or split they want to load instead of automatically loading all of them.
Datasets conflicts with fsspec 2024.9
State: open

### Describe the bug

Installing both in latest versions are not possible

`pip install "datasets==3.0.1" "fsspec==2024.9.0"`

But using older version of datasets is ok

`pip install "datasets==1.24.4" "fsspec==2024.9.0"`



### Steps to reproduce the bug

`pip install "datasets==3.0.1" "fsspec==2024.9.0"`


### Expected behavior

install both versions.

### Environment info

debian 11.
python 3.10.15
Audio preview in dataset viewer for audio array data without a path/filename
State: open
Labels: name: enhancement

### Feature request

Huggingface has quite a comprehensive set of guides for [audio datasets](https://huggingface.co/docs/datasets/en/audio_dataset). It seems, however, all these guides assume the audio array data to be decoded/inserted into a HF dataset always originates from individual files. The [Audio-dataclass](https://github.com/huggingface/datasets/blob/3.0.1/src/datasets/features/audio.py#L20) appears designed with this assumption in mind. Looking at its source code it returns a dictionary with the keys `path`, `array` and `sampling_rate`. 

However, sometimes users may have different pipelines where they themselves decode the audio array. This feature request has to do with wishing some clarification in guides on whether it is possible, and in such case how users can insert already decoded audio array data into datasets (pandas DataFrame, HF dataset or whatever) that are later saved as parquet, and still get a functioning audio preview in the dataset viewer. 

Do I perhaps need to write a tempfile of my audio array slice to wav and capture the bytes object with `io.BytesIO` and pass that to `Audio()`? 

### Motivation

I'm working with large audio datasets, and my pipeline reads (decodes) audio from larger files, and slices the relevant portions of audio from that larger file based on metadata I have available. 

The pipeline is designed this way to avoid having to store multiple copies of data, and to avoid having to store tens of millions of small files. 

I tried [test-uploading parquet files](https://huggingface.co/datasets/Lauler/riksdagen_test) where I store the audio array data of decoded slices of audio in an `audio` column with a dictionary with the keys `path`, `array` and `sampling_rate`. But I don't know the secret sauce of what the Huggingface Hub expects and requires to be able to display audio previews correctly. 

### Your contribution

I could contribute a tool agnostic guide of creating HF audio datasets directly as parquet to the HF documentation if there is an interest. Provided you help me figure out the secret sauce of what the dataset viewer expects to display the preview correctly.
shard_data_sources() got an unexpected keyword argument 'worker_id'
State: open

### Describe the bug

```
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 238, in __iter__
[rank0]:     for key_example in islice(self.generate_examples_fn(**gen_kwags), shard_example_idx_start, None):
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/packaged_modules/generator/generator.py", line 32, in _generate_examples
[rank0]:     for idx, ex in enumerate(self.config.generator(**gen_kwargs)):
[rank0]:   File "/home/qinghao/workdir/doremi/doremi/dataloader.py", line 337, in take_data_generator
[rank0]:     for ex in ds:
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1791, in __iter__
[rank0]:     yield from self._iter_pytorch()
[rank0]:   File "/home/qinghao/miniconda3/envs/doremi/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 1704, in _iter_pytorch
[rank0]:     ex_iterable = ex_iterable.shard_data_sources(worker_id=worker_info.id, num_workers=worker_info.num_workers)
[rank0]: TypeError: UpdatableRandomlyCyclingMultiSourcesExamplesIterable.shard_data_sources() got an unexpected keyword argument 'worker_id'
```

### Steps to reproduce the bug

IterableDataset cannot use

### Expected behavior

can work on datasets==2.10,  but will raise error for later versions.

### Environment info

datasets==3.0.1
pinning `dill<0.3.9` without pinning `multiprocess` 
State: closed

### Describe the bug

The [latest `multiprocess` release](https://github.com/uqfoundation/multiprocess/releases/tag/0.70.17) requires `dill>=0.3.9` which causes issues when installing `datasets` without backtracking during package version resolution. Is it possible to add a pin for multiprocess so something like `multiprocess<=0.70.16` so that the `dill` version is compatible?

### Steps to reproduce the bug

NA

### Expected behavior

NA

### Environment info

NA
CI benchmarks are broken
State: closed
Labels: name: maintenance

Since Aug 30, 2024, CI benchmarks are broken: https://github.com/huggingface/datasets/actions/runs/11108421214/job/30861323975
```
{"level":"error","message":"Resource not accessible by integration","name":"HttpError","request":{"body":"{\"body\":\"<details>\\n<summary>Show benchmarks</summary>\\n\\nPyArrow==8.0.0\\n\\n<details>\\n<summary>Show updated benchmarks!</summary>\\n\\n### Benchmark: benchmark_array_xd.json\\n\\n| metric | read_batch_formatted_as_numpy after write_array2d |
...
"headers":{"accept":"application/vnd.github.v3+json","authorization":"token [REDACTED]","content-type":"application/json; charset=utf-8","user-agent":"octokit-rest.js/18.0.0 octokit-core.js/3.6.0 Node.js/16.20.2 (linux; x64)"},"method":"POST","request":{"agent":{"_events":{},"_eventsCount":2,"cache":
...
"response":{"data":{"documentation_url":"https://docs.github.com/rest/issues/comments#create-an-issue-comment","message":"Resource not accessible by integration","status":"403"},
...
"stack":"HttpError: Resource not accessible by integration\n    at /usr/lib/node_modules/@dvcorg/cml/node_modules/@octokit/request/dist-node/index.js:86:21\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Job.doExecute (/usr/lib/node_modules/@dvcorg/cml/node_modules/bottleneck/light.js:405:18)","status":403}
```
Pin dill<0.3.9 to fix CI
State: closed

Pin dill<0.3.9 to fix CI for deps-latest.

Note that dill-0.3.9 was released yesterday Sep 29, 2024:
- https://pypi.org/project/dill/0.3.9/
- https://github.com/uqfoundation/dill/releases/tag/0.3.9

Fix #7183.
CI is broken for deps-latest
State: closed

See: https://github.com/huggingface/datasets/actions/runs/11106149906/job/30853879890

```
=========================== short test summary info ============================
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_filter_caching_on_disk - AssertionError: Lists differ: [{'fi[44 chars] {'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}] != [{'fi[44 chars] {'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}]

First differing element 1:
{'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}
{'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}

  [{'filename': '/tmp/tmp6xcyyjs4/dataset0.arrow'},
-  {'filename': '/tmp/tmp6xcyyjs4/cache-9533fe2601cd3e48.arrow'}]
?                                        ^^^^^ --------

+  {'filename': '/tmp/tmp6xcyyjs4/cache-e6e0a8b830976289.arrow'}]
?                                       ++++++++++ ^^  +
FAILED tests/test_arrow_dataset.py::BaseDatasetTest::test_map_caching_on_disk - AssertionError: Lists differ: [{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}] != [{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}]

First differing element 0:
{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}
{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}

- [{'filename': '/tmp/tmp5gxrti_n/cache-e58d327daec8626f.arrow'}]
?                                         ^^ -----------

+ [{'filename': '/tmp/tmp5gxrti_n/cache-d87234c5763e54a3.arrow'}]
?                                       +++++++++++  ^^
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_regex - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ignores_line_definition_of_function - AssertionError: '52e56ee04ad92499' != '0a4f75cec280f634'
- 52e56ee04ad92499
+ 0a4f75cec280f634
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ipython_function - AssertionError: 'a6bd2041ca63d6c0' != '517bf36b7eecdef5'
- a6bd2041ca63d6c0
+ 517bf36b7eecdef5
FAILED tests/test_fingerprint.py::HashingTest::test_hash_tiktoken_encoding - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_compiled_module - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_generator - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_tensor - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_doesnt_depend_on_order - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_stable - NameError: name 'log' is not defined
ERROR tests/test_iterable_dataset.py::test_iterable_dataset_from_file - NameError: name 'log' is not defined
= 11 failed, 2850 passed, 3 skipped, 23 warnings, 1 error in 191.06s (0:03:11) =
```
Support features in metadata configs
State: closed

Support features in metadata configs, like:
```
configs:
  - config_name: default
    features:
      - name: id
        dtype: int64
      - name:  name
        dtype: string
      - name: score
        dtype: float64
```

This will allow to avoid inference of data types.

Currently, we allow passing this information in the `dataset_info` (instead of `configs`) field, but this is not intuitive and it is not properly documented.

TODO:
- [ ] Document usage
Memory leak when wrapping datasets into PyTorch Dataset without explicit deletion
State: closed

### Describe the bug

I've encountered a memory leak when wrapping the HuggingFace dataset into a PyTorch Dataset. The RAM usage constantly increases during iteration if items are not explicitly deleted after use.








### Steps to reproduce the bug


Steps to reproduce:

Create a PyTorch Dataset wrapper for 'nebula/cc12m':
````
from torch.utils.data import Dataset
from tqdm import tqdm
from datasets import load_dataset
from torchvision import transforms

Image.MAX_IMAGE_PIXELS = None

class CC12M(Dataset):
    def __init__(self, path_or_name='nebula/cc12m', split='train', transform=None, single_caption=True):
        self.raw_dataset = load_dataset(path_or_name)[split]

        if transform is None:
            self.transform = transforms.Compose([
                transforms.Resize((224, 224)),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                transforms.Normalize(
                    mean=[0.48145466, 0.4578275, 0.40821073],
                    std=[0.26862954, 0.26130258, 0.27577711]
                )
            ])
        else:
            self.transform = transforms.Compose(transform)

        self.single_caption = single_caption
        self.length = len(self.raw_dataset)

    def __len__(self):
        return self.length

    def __getitem__(self, index):
        item = self.raw_dataset[index]
        caption = item['txt']
        with io.BytesIO(item['webp']) as buffer:
            image = Image.open(buffer).convert('RGB')
            if self.transform:
                image = self.transform(image)
        # del item  # Uncomment this line to prevent the memory leak
        return image, caption
````

Iterate through the dataset without the del item line in __getitem__.

Observe RAM usage increasing constantly.

Add del item at the end of __getitem__:

```
def __getitem__(self, index):
    item = self.raw_dataset[index]
    caption = item['txt']
    with io.BytesIO(item['webp']) as buffer:
        image = Image.open(buffer).convert('RGB')
        if self.transform:
            image = self.transform(image)
    del item  # This line prevents the memory leak
    return image, caption
```


Iterate through the dataset again and observe that RAM usage remains stable.



### Expected behavior


Expected behavior:
RAM usage should remain stable during iteration without needing to explicitly delete items.

Actual behavior:
RAM usage constantly increases unless items are explicitly deleted after use

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-4.18.0-513.5.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.12.4
- `huggingface_hub` version: 0.24.6
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1
[FSTimeoutError] load_dataset
State: closed

### Describe the bug

When using `load_dataset`to load [HuggingFaceM4/VQAv2](https://huggingface.co/datasets/HuggingFaceM4/VQAv2), I am getting `FSTimeoutError`. 

### Error

```
TimeoutError: 

The above exception was the direct cause of the following exception:

FSTimeoutError                            Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/fsspec/asyn.py](https://klh9mr78js-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240924-060116_RC00_678132060#) in sync(loop, func, timeout, *args, **kwargs)
     99     if isinstance(return_result, asyncio.TimeoutError):
    100         # suppress asyncio.TimeoutError, raise FSTimeoutError
--> 101         raise FSTimeoutError from return_result
    102     elif isinstance(return_result, BaseException):
    103         raise return_result

FSTimeoutError:
```

It usually fails around 5-6 GB.

<img width="847" alt="Screenshot 2024-09-26 at 9 10 19 PM" src="https://github.com/user-attachments/assets/ff91995a-fb55-4de6-8214-94025d6c8470">


### Steps to reproduce the bug

To reproduce it, run this in colab notebook:

```
!pip install -q -U datasets

from datasets import load_dataset
ds = load_dataset('HuggingFaceM4/VQAv2', split="train[:10%]")
```



### Expected behavior

It should download properly.

### Environment info

Using Colab Notebook.
CI is broken: No solution found when resolving dependencies
State: closed
Labels: name: bug

See: https://github.com/huggingface/datasets/actions/runs/11046967444/job/30687294297
```
Run uv pip install --system -r additional-tests-requirements.txt --no-deps
  × No solution found when resolving dependencies:
  ╰─▶ Because the current Python version (3.8.18) does not satisfy Python>=3.9
      and torchdata==0.10.0a0+1a98f21 depends on Python>=3.9, we can conclude
      that torchdata==0.10.0a0+1a98f21 cannot be used.
      And because only torchdata==0.10.0a0+1a98f21 is available and
      you require torchdata, we can conclude that your requirements are
      unsatisfiable.
Error: Process completed with exit code 1.
```