kardwalker/my_dataset huggingface dataset:
GitHub issue #7282: Faulty datasets.exceptions.ExpectedMoreSplitsError
State: open | Created: 2024-11-07T20:15:01Z

### Describe the bug

Trying to download only the 'validation' split of my dataset; instead hit the error `datasets.exceptions.ExpectedMoreSplitsError`.
Appears to be the same undesired behavior as reported in [#6939](https://github.com/huggingface/datasets/issues/6939), but with `data_files`, not `data_dir`.

Here is the Traceback:
```
Traceback (most recent call last):
  File "/home/user/app/app.py", line 12, in <module>
    ds = load_dataset('datacomp/imagenet-1k-random0.0', token=GATED_IMAGENET, data_files={'validation': 'data/val*'}, split='validation', trust_remote_code=True)
  File "/usr/local/lib/python3.10/site-packages/datasets/load.py", line 2154, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 1018, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/site-packages/datasets/utils/info_utils.py", line 68, in verify_splits
    raise ExpectedMoreSplitsError(str(set(expected_splits) - set(recorded_splits)))
datasets.exceptions.ExpectedMoreSplitsError: {'train', 'test'}
```

Note: I am using the `data_files` argument only because I am trying to specify that I only want the 'validation' split, and the whole dataset will be downloaded even when the `split='validation'` argument is specified, unless you also specify `data_files`, as described here: https://discuss.huggingface.co/t/how-can-i-download-a-specific-split-of-a-dataset/79027


### Steps to reproduce the bug

1. Create a Space with the default blank 'gradio' SDK https://huggingface.co/new-space
2. Create a file 'app.py' that loads a dataset to only extract a 'validation' split:

`ds = load_dataset('datacomp/imagenet-1k-random0.0', token=GATED_IMAGENET, data_files={'validation': 'data/val*'}, split='validation', trust_remote_code=True)`

### Expected behavior

Downloading validation split.

### Environment info

Default environment for creating a new Space. Relevant to this bug, that is:

```
FROM docker.io/library/python:3.10@sha256:fd0fa50d997eb56ce560c6e5ca6a1f5cf8fdff87572a16ac07fb1f5ca01eb608

--> RUN pip install --no-cache-dir pip==22.3.1 && 	pip install --no-cache-dir 	datasets 	"huggingface-hub>=0.19" "hf-transfer>=0.1.4" "protobuf<4" "click<8.1" 
```
GitHub issue #7281: File not found error
State: open | Created: 2024-11-07T09:04:49Z

### Describe the bug

I get a FileNotFoundError:
<img width="944" alt="image" src="https://github.com/user-attachments/assets/1336bc08-06f6-4682-a3c0-071ff65efa87">


### Steps to reproduce the bug

See screenshot.



### Expected behavior

I want to load one audiofile from the dataset. 

### Environment info

MacOs Intel 14.6.1 (23G93)
Python 3.10.9
Numpy 1.23
Datasets latest version
GitHub issue #7280: Add filename in error message when ReadError or similar occur
State: open | Created: 2024-11-07T06:00:53Z

Please update error messages to include relevant information for debugging when loading datasets with `load_dataset()` that may have a few corrupted files. 

Whenever downloading a full dataset, some files might be corrupted (either at the source or from downloading corruption). 

However the errors often only let me know it was a tar file if `tarfile.ReadError` appears on the traceback, and I imagine similarly for other file types. 

This makes it really hard to debug which file is corrupted, and when dealing with very large datasets, it shouldn't be necessary to force download everything again.
GitHub issue #7275: load_dataset
State: open | Created: 2024-11-04T03:01:44Z

### Describe the bug

I am performing two operations I see on a hugging face tutorial (Fine-tune a language model), and I am defining every aspect inside the mapped functions, also some imports of the library because it doesnt identify anything not defined outside that function where the dataset elements are being mapped:

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

`- lm_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    batch_size=batch_size,
    num_proc=4,
)

- tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])
def tokenize_function(examples):
    model_checkpoint = 'gpt2'
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    return tokenizer(examples["text"])`


### Steps to reproduce the bug

Currently handle all the imports inside the function

### Expected behavior

The code must work es expected in the notebook, but currently this is not happening.

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

### Environment info

print(transformers.__version__)

4.46.1
GitHub issue #7269: Memory leak when streaming
State: open | Created: 2024-10-31T13:33:52Z

### Describe the bug

I try to use a dataset with streaming=True, the issue I have is that the RAM usage becomes higher and higher until it is no longer sustainable.

I understand that huggingface store data in ram during the streaming, and more worker in dataloader there are, more a lot of shard will be stored in ram, but the issue I have is that the ram usage is not constant. So after each new shard loaded, the ram usage will be higher and higher.

### Steps to reproduce the bug

You can run this code and see you ram usage, after each shard of 255 examples, your ram usage will be extended.
```py
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("WaveGenAI/dataset", streaming=True)

dataloader = DataLoader(dataset["train"], num_workers=3)

for i, data in enumerate(dataloader):
    print(i, end="\r")
```

### Expected behavior

The Ram usage should be always the same (just 3 shards loaded in the ram).

### Environment info

- `datasets` version: 3.0.1
- Platform: Linux-6.10.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1
GitHub issue #7268: load_from_disk
State: open | Created: 2024-10-31T11:51:56Z

### Describe the bug

I have data saved with save_to_disk. The data is big (700Gb). When I try loading it, the only option is load_from_disk, and this function copies the data to a tmp directory, causing me to run out of disk space. Is there an alternative solution to that?

### Steps to reproduce the bug

when trying to load data using load_From_disk after being saved using save_to_disk 

### Expected behavior

run out of disk space

### Environment info

lateest version
GitHub issue #7267: Source installation fails on Macintosh with python 3.10
State: open | Created: 2024-10-31T10:18:45Z

### Describe the bug

Hi, 

Decord is a dev dependency not maintained since couple years. 
It does not have an ARM package available rendering it uninstallable on non-intel based macs 

Suggestion is to move to eva-decord (https://github.com/georgia-tech-db/eva-decord)  which doesnt have this problem. 

Happy to raise a PR


### Steps to reproduce the bug

Source installation as mentioned in contributinog.md

### Expected behavior

Installation without decord failing to be installed.

### Environment info

python=3.10, M3 Mac
GitHub issue #7266: The dataset viewer should be available soon. Please retry later.
State: closed | Created: 2024-10-30T16:32:00Z

### Describe the bug

After waiting for 2 hours, it still presents ``The dataset viewer should be available soon. Please retry later.''

### Steps to reproduce the bug

dataset link: https://huggingface.co/datasets/BryanW/HI_EDIT

### Expected behavior

Present the dataset viewer.

### Environment info

NA
GitHub issue #7261: Cannot load the cache when mapping the dataset
State: open | Created: 2024-10-29T08:29:40Z

### Describe the bug

I'm training the flux controlnet. The train_dataset.map() takes long time to finish. However, when I killed one training process and want to restart a new training with the same dataset. I can't reuse the mapped result even I defined the cache dir for the dataset.

with accelerator.main_process_first():
        from datasets.fingerprint import Hasher

        # fingerprint used by the cache for the other processes to load the result
        # details: https://github.com/huggingface/diffusers/pull/4038#discussion_r1266078401
        new_fingerprint = Hasher.hash(args)
        train_dataset = train_dataset.map(
            compute_embeddings_fn, batched=True, new_fingerprint=new_fingerprint, batch_size=10,
        )

### Steps to reproduce the bug

train flux controlnet and start again



### Expected behavior

will not map again

### Environment info

latest diffusers
GitHub issue #7260: cache can't cleaned or disabled
State: open | Created: 2024-10-29T03:15:28Z

### Describe the bug

I tried following ways, the cache can't be disabled. 

I got 2T data, but I also got more than 2T cache file. I got pressure on storage. I need to diable cache or cleaned immediately after processed. Following ways are all not working, please give some help!

```python
from datasets import disable_caching
from transformers import AutoTokenizer
disable_caching()

tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_path)
def tokenization_fn(examples):
    column_name = 'text' if 'text' in examples else 'data'
    tokenized_inputs = tokenizer(
        examples[column_name], return_special_tokens_mask=True, truncation=False,
        max_length=tokenizer.model_max_length
    )
    return tokenized_inputs

data = load_dataset('json', data_files=save_local_path, split='train', cache_dir=None)
data.cleanup_cache_files()
updated_dataset = data.map(tokenization_fn, load_from_cache_file=False)
updated_dataset .cleanup_cache_files()
```

### Expected behavior

no cache file generated

### Environment info

Ubuntu 20.04.6 LTS
datasets  3.0.2
GitHub issue #7254: mismatch for datatypes when providing `Features` with `Array2D` and user specified `dtype` and using with_format("numpy")
State: open | Created: 2024-10-26T22:06:27Z

### Describe the bug

If the user provides a `Features` type value to `datasets.Dataset` with members having `Array2D` with a value for `dtype`, it is not respected during `with_format("numpy")` which should return a `np.array` with `dtype` that the user provided for `Array2D`. It seems for floats, it will be set to `float32` and for ints it will be set to `int64`

### Steps to reproduce the bug

```python
import numpy as np
import datasets
from datasets import Dataset, Features, Array2D

print(f"datasets version: {datasets.__version__}")

data_info = {
    "arr_float" : "float64",
    "arr_int" : "int32"
}

sample = {key : [np.zeros([4, 5], dtype=dtype)] for key, dtype in data_info.items()}

features = {key : Array2D(shape=(None, 5), dtype=dtype) for key, dtype in data_info.items()}
features = Features(features)

dataset = Dataset.from_dict(sample, features=features)

ds = dataset.with_format("numpy")
for key in features:
    print(f"{key} feature dtype: ", ds.features[key].dtype)
    print(f"{key} dtype:", ds[key].dtype)
```

Output:
```bash
datasets version: 3.0.2
arr_float feature dtype:  float64
arr_float dtype: float32
arr_int feature dtype:  int32
arr_int dtype: int64
```

### Expected behavior

It should return a `np.array` with `dtype` that the user provided for the corresponding member in the `Features` type value

### Environment info

- `datasets` version: 3.0.2
- Platform: Linux-6.11.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.1
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0
GitHub issue #7253: Unable to upload a large dataset zip either from command line or UI
State: open | Created: 2024-10-26T13:17:06Z

### Describe the bug

Unable to upload a large dataset zip from command line or UI. UI simply says error. I am trying to a upload a tar.gz file of 17GB.

<img width="550" alt="image" src="https://github.com/user-attachments/assets/f9d29024-06c8-49c4-a109-0492cff79d34">
<img width="755" alt="image" src="https://github.com/user-attachments/assets/a8d4acda-7f02-4279-9c2d-b2e0282b4faa">


### Steps to reproduce the bug

Upload a large file 

### Expected behavior

The file should upload without any issue.

### Environment info

None
GitHub issue #7252: Add IterableDataset.shard()
State: closed | Created: 2024-10-25T11:07:12Z

Will be useful to distribute a dataset across workers (other than pytorch) like spark

I also renamed `.n_shards` -> `.num_shards` for consistency and kept the old name for backward compatibility. And a few changes in internal functions for consistency as well (rank, world_size -> num_shards, index)

Breaking change: the new default for `contiguous` in `Dataset.shard()` is `True`, but imo not a big deal since I couldn't find any usage of `contiguous=False` internally (we always do contiguous=True for map-style datasets since its more optimized) or in the wild
GitHub issue #7249: How to debugging
State: open | Created: 2024-10-24T01:03:51Z

### Describe the bug

I wanted to use my own script to handle the processing, and followed the tutorial documentation by rewriting the MyDatasetConfig and MyDatasetBuilder (which contains the _info,_split_generators and _generate_examples methods) classes. Testing with simple data was able to output the results of the processing, but when I wished to do more complex processing, I found that I was unable to debug (even the simple samples were inaccessible). There are no errors reported, and I am able to print the _info,_split_generators and _generate_examples messages, but I am unable to access the breakpoints.

### Steps to reproduce the bug

# my_dataset.py
import json
import datasets


class MyDatasetConfig(datasets.BuilderConfig):
    def __init__(self, **kwargs):
        super(MyDatasetConfig, self).__init__(**kwargs)


class MyDataset(datasets.GeneratorBasedBuilder):
    VERSION = datasets.Version("1.0.0")

    BUILDER_CONFIGS = [
        MyDatasetConfig(
            name="default",
            version=VERSION,
            description="myDATASET"
        ),
    ]

    def _info(self):
        print("info")  # breakpoints
        return datasets.DatasetInfo(
            description="myDATASET",
            features=datasets.Features(
                {
                    "id": datasets.Value("int32"),
                    "text": datasets.Value("string"),
                    "label": datasets.ClassLabel(names=["negative", "positive"]),
                }
            ),
            supervised_keys=("text", "label"),
        )

    def _split_generators(self, dl_manager):
 
        print("generate")  # breakpoints
        data_file = "data.json"  

        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN, gen_kwargs={"filepath": data_file}
            ),
        ]

    def _generate_examples(self, filepath):
        print("example")  # breakpoints
        with open(filepath, encoding="utf-8") as f:
            data = json.load(f)
            for idx, sample in enumerate(data):
                yield idx, {
                    "id": sample["id"],
                    "text": sample["text"],
                    "label": sample["label"],
                }

#main.py
import os
os.environ["TRANSFORMERS_NO_MULTIPROCESSING"] = "1" 

from datasets import load_dataset

dataset = load_dataset("my_dataset.py", split="train", cache_dir=None)

print(dataset[:5])

### Expected behavior

Pause at breakpoints while running debugging

### Environment info

pycharm
GitHub issue #7247: Adding column with dict struction when mapping lead to wrong order
State: open | Created: 2024-10-22T18:55:11Z

### Describe the bug

in `map()` function, I want to add a new column with a dict structure.
```
def map_fn(example):
  example['text'] = {'user': ..., 'assistant': ...}
  return example
```
However this leads to a wrong order `{'assistant':..., 'user':...}` in the dataset.
Thus I can't concatenate two datasets due to the different feature structures.
[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example
This seems an issue in low level pyarrow library instead of datasets, however, I think datasets should allow concatenate two datasets actually in the same structure.

### Steps to reproduce the bug

[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example

### Expected behavior

two datasets could be concatenated.

### Environment info

N/A
GitHub issue #7243: ArrayXD with None as leading dim incompatible with DatasetCardData
State: open | Created: 2024-10-21T15:08:13Z

### Describe the bug

Creating a dataset with ArrayXD features leads to errors when downloading from hub due to DatasetCardData removing the Nones

@lhoestq 

### Steps to reproduce the bug

```python
import numpy as np
from datasets import Array2D, Dataset, Features, load_dataset

def examples_generator():
    for i in range(4):
        yield {
            "array_1d": np.zeros((10,1), dtype="uint16"),
            "array_2d": np.zeros((10, 1), dtype="uint16"),
        }

features = Features(array_1d=Array2D((None,1), "uint16"), array_2d=Array2D((None, 1), "uint16"))
dataset = Dataset.from_generator(examples_generator, features=features)
dataset.push_to_hub("alex-hh/test_array_1d2d")
ds = load_dataset("alex-hh/test_array_1d2d")
```

Source of error appears to be DatasetCardData.to_dict invoking DatasetCardData._remove_none
```python
from huggingface_hub import DatasetCardData
from datasets.info import DatasetInfosDict

dataset_card_data = DatasetCardData()
DatasetInfosDict({"default": dataset.info.copy()}).to_dataset_card_data(dataset_card_data)
print(dataset_card_data.to_dict())  # removes Nones in shape
```

### Expected behavior

Should be possible to load datasets saved with shape None in leading dimension

### Environment info

3.0.2 and latest huggingface_hub
GitHub issue #7241: `push_to_hub` overwrite argument
State: closed | Created: 2024-10-20T03:23:26Z

### Feature request

Add an `overwrite` argument to the `push_to_hub` method.

### Motivation

I want to overwrite a repo without deleting it on Hugging Face. Is this possible? I couldn't find anything in the documentation or tutorials.

### Your contribution

I can create a PR.
GitHub issue #7240: Feature Request: Add functionality to pass split types like train, test in DatasetDict.map
State: closed | Created: 2024-10-19T09:59:12Z

Hello datasets!

We often encounter situations where we need to preprocess data differently depending on split types such as train, valid, and test. 

However, while DatasetDict.map has features to pass rank or index, there's no functionality to pass split types. 
Therefore, I propose adding a 'with_splits' parameter to DatasetDict, which would allow passing the split type through fn_kwargs.
GitHub issue #7238: incompatibily issue when using load_dataset with datasets==3.0.1
State: open | Created: 2024-10-18T21:25:23Z

### Describe the bug

There is a bug when using load_dataset with dataset version at 3.0.1 .
Please see below in the "steps to reproduce the bug".
To resolve the bug, I had to downgrade to version 2.21.0
OS: Ubuntu 24 (AWS instance)
Python: same bug under 3.12 and 3.10


The error I had was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/load.py", line 2096, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 1647, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 977, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_6_0/cb17afd34f5799f97e8f48398748f83006335b702bd785f9880797838d541b81/common_voice_6_0.py", line 159, in _split_generators
    archive_path = dl_manager.download(self._get_bundle_url(self.config.name, bundle_url_template))
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_manager.py", line 150, in download
    download_config = self.download_config.copy()
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_config.py", line 73, in copy
    return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
TypeError: DownloadConfig.__init__() got an unexpected keyword argument 'ignore_url_params'

### Steps to reproduce the bug

1. install dataset with ```pip install datasets --upgrade```
2. launch python; from datasets import loaad_dataset
3. run load_dataset("mozilla-foundation/common_voice_6_0")
4. exit python
5. uninstall datasets; then ```pip install datasets==2.21.0```
6. launch python; from datasets import loaad_dataset
7. run load_dataset("mozilla-foundation/common_voice_6_0")
8. Everything runs great now

### Expected behavior

Be able to download a dataset without error

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 3.0.1
- Platform: Linux-6.8.0-1017-aws-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1
GitHub issue #7230: Video support
State: closed | Created: 2024-10-15T18:17:29Z

(wip and experimental)

adding the `Video` type based on `VideoReader` from `decord`

```python
>>>from datasets import load_dataset
>>> ds = load_dataset("path/to/videos", split="train").with_format("torch")
>>> print(ds[0]["video"])
<decord.video_reader.VideoReader object at 0x337a47910>
>>> print(ds[0]["video"][0])
tensor([[[73, 73, 73],
         [73, 73, 73],
         [73, 73, 73],
         ...,
         [23, 23, 23],
         [23, 23, 23],
         [23, 23, 23]]], dtype=torch.uint8)
```

the storage is the same as for audio and images: `{"path": pa.string(), "bytes": pa.binary()}` and I did a small to keep the hf:// URL in the "path" field if possible, this way the viewer can link to fiels on the hub if possible
GitHub issue #7228: Composite (multi-column) features
State: open | Created: 2024-10-14T23:59:19Z

### Feature request

Structured data types (graphs etc.) might often be most efficiently stored as multiple columns, which then need to be combined during feature decoding

Although it is currently possible to nest features as structs, my impression is that in particular when dealing with e.g. a feature composed of multiple numpy array / ArrayXD's, it would be more efficient to store each ArrayXD as a separate column (though I'm not sure by how much)

Perhaps specification / implementation could be supported by something like:

```
features=Features(**{("feature0", "feature1")=Features(feature0=Array2D((None,10), dtype="float32"), feature1=Array2D((None,10), dtype="float32"))
```

### Motivation

Defining efficient composite feature types based on numpy arrays for representing data such as graphs with multiple node and edge attributes is currently challenging.

### Your contribution

Possibly able to contribute
GitHub issue #7227: fast array extraction
State: open | Created: 2024-10-14T20:51:32Z

Implements #7210 using method suggested in https://github.com/huggingface/datasets/pull/7207#issuecomment-2411789307


```python
import numpy as np
from datasets import Dataset, Features, Array3D
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```
~0.02 s vs 0.9s on main

```python
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
< 0.01 s vs 1.3 s on main

@lhoestq I can see this breaks a bunch of array-related tests but can update the test cases if you would support making this change?

I also added an Array1D feature which will always be decoded into a numpy array and likewise improves extraction performance:

```python
from datasets import Dataset, Features, Array1D, Sequence, Value
array_features=Features(**{"array0": Array1D((None,), dtype="float32"), "array1": Array1D((None,), dtype="float32")})
sequence_features=Features(**{"array0": Sequence(feature=Value("float32"), length=-1), "array1": Sequence(feature=Value("float32"), length=-1)})
array_dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [20000,10000]*25] for i in range(2)}, features=array_features)
sequence_dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [20000,10000]*25] for i in range(2)}, features=sequence_features)


```python
t0 = time.time()
for ex in array_dataset.to_iterable_dataset():
    pass
t1 = time.time()
```
< 0.01 s

```python
t0 = time.time()
for ex in sequence_dataset.to_iterable_dataset():
    pass
t1 = time.time()
```
~1.1s

And also added support for extracting structs of arrays as dicts of numpy arrays:

```python
import numpy as np
from datasets import Dataset, Features, Array3D, Sequence
features=Features(struct={"array0": Array3D((None,10,10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")}, _list=Sequence(feature=Array3D((None,10,10), dtype="float32")))
dataset = Dataset.from_dict({"struct": [{f"array{i}": np.zeros((x,10,10), dtype=np.float32) for i in range(2)} for x in [2000,1000]*25], "_list": [[np.zeros((x,10,10), dtype=np.float32) for i in range(2)] for x in [2000,1000]*25]}, features=features)
```

```python
t0 = time.time()
for ex in dataset.to_iterable_dataset():
    pass
t1 = time.time()
assert isinstance(ex["struct"]["array0"], np.ndarray) and ex["struct"]["array0"].ndim == 3
```
~0.02 s and no exception vs ~7s with an exception on main
GitHub issue #7226: Add R as a How to use from the Polars (R) Library as an option
State: open | Created: 2024-10-14T19:56:07Z

### Feature request

The boiler plate code to access a dataset via the hugging face file system is very useful. Please addd 


## Add Polars (R) option
The equivailent code works, because the [Polars-R](https://github.com/pola-rs/r-polars) wrapper has hugging faces funcitonaliy as well.

```r
library(polars)

df <- pl$read_parquet("hf://datasets/SALURBAL/core__admin_cube_public/core__admin_cube_public.parquet")
```

## Polars (python) option
![image](https://github.com/user-attachments/assets/8f1bcd19-e578-4b18-b324-7cc00b80ac0a)


## Libraries Currently

![image](https://github.com/user-attachments/assets/0cf50063-f9db-443c-97b4-3ef0664b6e6e)




### Motivation

There are many data/analysis/research/statistics teams (particularly in academia and pharma) that use R as the default language. R has great integration with most of the newer data techs (arrow, parquet, polars) and having this included could really help in bringing this community into the hugging faces ecosystem.

**This is a small/low-hanging-fruit front end change but would make a big impact expanding the community**

### Your contribution

I am not sure which repositroy this should be in, but I have experience in R, Python and JS and happy to submit a PR in the appropriate repository.
GitHub issue #7225: Huggingface GIT returns null as Content-Type instead of application/x-git-receive-pack-result
State: open | Created: 2024-10-14T14:33:06Z

### Describe the bug

We push changes to our datasets programmatically. Our git client jGit reports that the hf git server returns null as Content-Type after a push.

### Steps to reproduce the bug

A basic kotlin application:
```
   val person = PersonIdent(
        "padmalcom",
        "padmalcom@sth.com"
    )

    val cp = UsernamePasswordCredentialsProvider(
        "padmalcom",
        "mysecrettoken"
    )

    val git =
        KGit.cloneRepository {
            setURI("https://huggingface.co/datasets/sth/images")
            setTimeout(60)
            setProgressMonitor(TextProgressMonitor())
            setCredentialsProvider(cp)
        }

    FileOutputStream("./images/images.csv").apply { writeCsv(images) }
    git.add {
        addFilepattern("images.csv")
    }

    for (i in images) {
        FileUtils.copyFile(
            File("./files/${i.id}"),
            File("./images/${i.id + File(i.fileName).extension }")
        )
        git.add {
            addFilepattern("${i.id + File(i.fileName).extension }")
        }
    }

    val revCommit = git.commit {
        author = person
        message = "Uploading images at " + LocalDateTime.now()
            .format(DateTimeFormatter.ISO_DATE_TIME)
        setCredentialsProvider(cp)
    }

    val push = git.push {
        setCredentialsProvider(cp)
    }
```

### Expected behavior

The git server is expected to return the Content-Type _application/x-git-receive-pack-result_.

### Environment info

It is independent from the datasets library.
GitHub issue #7223: Fallback to arrow defaults when loading dataset with custom features that aren't registered locally
State: open | Created: 2024-10-12T16:08:20Z

### Describe the bug

Datasets allows users to create and register custom features.

However if datasets are then pushed to the hub, this means that anyone calling load_dataset without registering the custom Features in the same way as the dataset creator will get an error message.

It would be nice to offer a fallback in this case.

### Steps to reproduce the bug

```python
load_dataset("alex-hh/custom-features-example")
```
(Dataset creation process - must be run in separate session so that NewFeature isn't registered in session in which download is attempted:)

```python
from dataclasses import dataclass, field
import pyarrow as pa
from datasets.features.features import register_feature

from datasets import Dataset, Features, Value, load_dataset
from datasets import Feature

@dataclass
class NewFeature(Feature):
    _type: str = field(default="NewFeature", init=False, repr=False)
    def __call__(self):
        return pa.int32()

def examples_generator():
    for i in range(5):
        yield {"feature": i}

ds = Dataset.from_generator(examples_generator, features=Features(feature=NewFeature()))
ds.push_to_hub("alex-hh/custom-features-example")
register_feature(NewFeature, "NewFeature")
```



### Expected behavior

It would be nice, and offer greater extensibility, if there was some kind of graceful fallback mechanism in place for cases where user-defined features are stored in the dataset but not available locally.

### Environment info

3.0.2
GitHub issue #7221: add CustomFeature base class to support user-defined features with encoding/decoding logic
State: closed | Created: 2024-10-11T20:10:27Z

intended as fix for #7220 if this kind of extensibility is something that datasets is willing to support!


```python
from datasets.features.features import CustomFeature


class ListOfStrs(CustomFeature):
    requires_encoding = True
    def _encode_example(self, value):
        if isinstance(value, str):
            return [str]
        else:
            return value
feats = Features(strlist=ListOfStrs())
feats.encode_example({"strlist": "a"})["strlist"] == feats["strlist"].encode_example("a")
```
GitHub issue #7220: Custom features not compatible with special encoding/decoding logic
State: open | Created: 2024-10-11T19:20:11Z

### Describe the bug

It is possible to register custom features using datasets.features.features.register_feature (https://github.com/huggingface/datasets/pull/6727)

However such features are not compatible with Features.encode_example/decode_example if they require special encoding / decoding logic because encode_nested_example / decode_nested_example checks whether the feature is in a fixed list of encodable types:

https://github.com/huggingface/datasets/blob/16a121d7821a7691815a966270f577e2c503473f/src/datasets/features/features.py#L1349

This prevents the extensibility of features to complex cases

### Steps to reproduce the bug

```python
class ListOfStrs:
    def encode_example(self, value):
        if isinstance(value, str):
            return [str]
        else:
            return value
feats = Features(strlist=ListOfStrs())
assert feats.encode_example({"strlist": "a"})["strlist"] = feats["strlist"].encode_example("a")}
```

### Expected behavior

Registered feature types should be encoded based on some property of the feature (e.g. requires_encoding)?

### Environment info

3.0.2
GitHub issue #7217: ds.map(f, num_proc=10) is slower than df.apply
State: open | Created: 2024-10-11T11:04:05Z

### Describe the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

time cost: 
1. df.apply: 100%|██████████| 12500592/12500592 [00:13<00:00, 959825.47it/s]
2. ds.map: Map (num_proc=10):  31%
 3899028/12500592 [00:28<00:38, 222532.89 examples/s]

### Steps to reproduce the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

### Expected behavior

ds.map is ~num_proc faster than df.apply

### Environment info

pandas: 2.2.2
datasets: 2.19.1
GitHub issue #7215: Iterable dataset map with explicit features causes slowdown for Sequence features
State: open | Created: 2024-10-10T22:08:20Z

### Describe the bug

When performing map, it's nice to be able to pass the new feature type, and indeed required by interleave and concatenate datasets.

However, this can cause a major slowdown for certain types of array features due to the features being re-encoded.

This is separate to the slowdown reported in #7206 

### Steps to reproduce the bug

```
from datasets import Dataset, Features, Array3D, Sequence, Value
import numpy as np
import time
features=Features(**{"array0": Sequence(feature=Value("float32"), length=-1), "array1": Sequence(feature=Value("float32"), length=-1)})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [5000,10000]*25] for i in range(2)}, features=features)
```

```
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~1.5 s on main


```
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x, features=features)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~ 3 s on main

### Expected behavior

I'm not 100% sure whether passing new feature types to formatted outputs of map should be supported or not, but assuming it should, then there should be a cost-free way to specify the new feature type - knowing feature type is required by interleave_datasets and concatenate_datasets for example

### Environment info

3.0.2
GitHub issue #7214: Formatted map + with_format(None) changes array dtype for iterable datasets
State: open | Created: 2024-10-10T12:45:16Z

### Describe the bug

When applying with_format -> map -> with_format(None), array dtypes seem to change, even if features are passed 

### Steps to reproduce the bug

```python
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32")})
dataset = Dataset.from_dict({f"array0": [np.zeros((100,10,10), dtype=np.float32)]*25}, features=features)
ds = dataset.to_iterable_dataset().with_format("numpy").map(lambda x: x, features=features)
ex_0 = next(iter(ds))
ds = dataset.to_iterable_dataset().with_format("numpy").map(lambda x: x, features=features).with_format(None)
ex_1 = next(iter(ds))
assert ex_1["array0"].dtype == ex_0["array0"].dtype, f"{ex_1['array0'].dtype} {ex_0['array0'].dtype}"
```


### Expected behavior

Dtypes should be preserved.

### Environment info

3.0.2
GitHub issue #7213: Add with_rank to Dataset.from_generator
State: open | Created: 2024-10-10T12:15:29Z

### Feature request

Add `with_rank` to `Dataset.from_generator` similar to `Dataset.map` and `Dataset.filter`. 

### Motivation

As for `Dataset.map` and `Dataset.filter`, this is useful when creating cache files using multi-GPU, where the rank can be used to select GPU IDs. For now, rank can be added in the `gen_kwars` argument; however, this, in turn, includes the rank when computing the fingerprint.

### Your contribution

Added #7199 which passes rank based on the `job_id` set by `num_proc`.
GitHub issue #7212: Windows do not supprot signal.alarm and singal.signal
State: open | Created: 2024-10-10T12:00:19Z

### Describe the bug

signal.alarm and signal.signal are used in the load.py module, but these are not supported by Windows.

### Steps to reproduce the bug

 lighteval accelerate --model_args "pretrained=gpt2,trust_remote_code=True" --tasks "community|kinit_sts" --custom_tasks "community_tasks/kinit_evals.py" --output_dir "./evals"

### Expected behavior

proceed with input(..) method

### Environment info

Windows 11
GitHub issue #7211: Describe only selected fields in README
State: open | Created: 2024-10-09T16:25:47Z

### Feature request

Hi Datasets team! 

Is it possible to add the ability to describe only selected fields of the dataset files in `README.md`? For example, I have this open dataset ([open-llm-leaderboard/results](https://huggingface.co/datasets/open-llm-leaderboard/results?row=0)) and I want to describe only some fields in order not to overcomplicate the Dataset Preview and filter out some fields 

### Motivation

The `Results` dataset for the Open LLM Leaderboard contains json files with a complex nested structure. I would like to add `README.md` there to use the SQL console, for example. But if I describe the structure of this dataset completely, it will overcomplicate the use of Dataset Preview and the total number of columns will exceed 50 

### Your contribution

I'm afraid I'm not familiar with the project structure, so I won't be able to open a PR, but I'll try to help with something else if possible
GitHub issue #7210: Convert Array features to numpy arrays rather than lists by default
State: open | Created: 2024-10-09T13:05:21Z

### Feature request

It is currently quite easy to cause massive slowdowns when using datasets and not familiar with the underlying data conversions by e.g. making bad choices of formatting.

Would it be more user-friendly to set defaults that avoid this as much as possible? e.g. format Array features as numpy arrays rather than python lists

### Motivation

Default array formatting leads to slow performance: e.g.

```python
import numpy as np
from datasets import Dataset, Features, Array3D
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```

```python
t0 = time.time()
for ex in ds:
   pass
t1 = time.time()
```
~1.4 s

```python
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~10s

```python
ds = dataset.with_format("numpy")
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~0.04s

```python
ds = dataset.to_iterable_dataset().with_format("numpy")
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~0.04s

### Your contribution

May be able to contribute
GitHub issue #7208: Iterable dataset.filter should not override features
State: closed | Created: 2024-10-09T10:23:45Z

### Describe the bug

When calling filter on an iterable dataset, the features get set to None

### Steps to reproduce the bug

import numpy as np
import time
from datasets import Dataset, Features, Array3D

```python
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
ds = dataset.to_iterable_dataset()
orig_column_names = ds.column_names
ds = ds.filter(lambda x: True)
assert ds.column_names == orig_column_names
```

### Expected behavior

Filter should preserve features information

### Environment info

3.0.2
GitHub issue #7207: apply formatting after iter_arrow to speed up format -> map, filter for iterable datasets
State: closed | Created: 2024-10-08T15:44:53Z

I got to this by hacking around a bit but it seems to solve #7206 

I have no idea if this approach makes sense or would break something else? 

Could maybe work on a full pr if this looks reasonable @lhoestq ? I imagine the same issue might affect other iterable dataset methods?
GitHub issue #7206: Slow iteration for iterable dataset with numpy formatting for array data
State: open | Created: 2024-10-08T15:38:11Z

### Describe the bug

When working with large arrays, setting with_format to e.g. numpy then applying map causes a significant slowdown for iterable datasets.

### Steps to reproduce the bug

```python
import numpy as np
import time
from datasets import Dataset, Features, Array3D

features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```

Then
```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
print(t1-t0)
```

takes 27 s, whereas

```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy")
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
print(t1 - t0)
```

takes ~1s

### Expected behavior

Map should not introduce a slowdown when formatting is enabled.

### Environment info

3.0.2
GitHub issue #7204: fix unbatched arrow map for iterable datasets
State: closed | Created: 2024-10-08T13:54:09Z

Fixes the bug when applying map to an arrow-formatted iterable dataset described here:

https://github.com/huggingface/datasets/issues/6833#issuecomment-2399903885

```python

from datasets import load_dataset
ds = load_dataset("rotten_tomatoes", split="train", streaming=True)
ds = ds.with_format("arrow").map(lambda x: x)
for ex in ds:
    pass
```

@lhoestq
GitHub issue #7202: `from_parquet` return type annotation
State: open | Created: 2024-10-08T09:08:10Z

### Describe the bug

As already posted in https://github.com/microsoft/pylance-release/issues/6534, the correct type hinting fails when building a dataset using the `from_parquet` constructor.
Their suggestion is to comprehensively annotate the method's return type to better align with the docstring information.

### Steps to reproduce the bug

```python
from datasets import Dataset

dataset = Dataset.from_parquet(path_or_paths="file")
dataset.map(lambda x: {"new": x["old"]}, batched=True)
```

### Expected behavior

map is a [valid](https://huggingface.co/docs/datasets/v3.0.1/en/package_reference/main_classes#datasets.Dataset.map), no error should be thrown.

### Environment info

- `datasets` version: 3.0.1
- Platform: macOS-15.0.1-arm64-arm-64bit
- Python version: 3.12.6
- `huggingface_hub` version: 0.25.1
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1
GitHub issue #7200: Fix the environment variable for huggingface cache
State: closed | Created: 2024-10-05T11:54:35Z

Resolve  #6256. As far as I tested, `HF_DATASETS_CACHE` was ignored and I could not specify the cache directory at all except for the default one by this environment variable. `HF_HOME` has worked. Perhaps the recent change on file downloading by `huggingface_hub` could affect this bug.

In my testing, I could not specify the cache directory even by `load_dataset("dataset_name" cache_dir="...")`. It might be another issue. I also welcome any advice to solve this issue.
GitHub issue #7198: Add repeat method to datasets
State: closed | Created: 2024-10-04T10:45:16Z

Following up on discussion in #6623 and #7198 I thought this would be pretty useful for my case so had a go at implementing.

My main motivation is to be able to call iterable_dataset.repeat(None).take(samples_per_epoch) to safely avoid timeout issues in a distributed training setting. This would provide a straightforward workaround for several open issues related to this situation:  https://github.com/huggingface/datasets/issues/6437, https://github.com/huggingface/datasets/issues/6594, https://github.com/huggingface/datasets/issues/6623, https://github.com/huggingface/datasets/issues/6719.

@lhoestq let me know if this looks on the right track!
GitHub issue #7197: ConnectionError: Couldn't reach 'allenai/c4' on the Hub (ConnectionError)数据集下不下来，怎么回事
State: open | Created: 2024-10-04T09:33:25Z

### Describe the bug

from datasets import load_dataset
print("11")
traindata = load_dataset('ptb_text_only', 'penn_treebank', split='train')
print("22")
valdata = load_dataset('ptb_text_only',
                           'penn_treebank',
                           split='validation')

### Steps to reproduce the bug

1

### Expected behavior

1

### Environment info

1