dalssy/data_del huggingface dataset:
GitHub issue: Efficient Image Augmentation in Hugging Face Datasets
Author: fabiozappo (NONE)
State: open | Created: 2024-11-26 16:50:32

### Describe the bug

 I'm using the Hugging Face datasets library to load images in batch and would like to apply a torchvision transform to solve the inconsistent image sizes in the dataset and apply some on the fly image augmentation. I can just think about using the collate_fn, but seems quite inefficient. 
 
 I'm new to the Hugging Face datasets library, I didn't find nothing in the documentation or the issues here on github.

Is there an existing way to add image transformations directly to the dataset loading pipeline? 

### Steps to reproduce the bug

from datasets import load_dataset
from torch.utils.data import DataLoader

```python
def collate_fn(batch):
    images = [item['image'] for item in batch]
    texts = [item['text'] for item in batch]
    return {
        'images': images,
        'texts': texts
    }

dataset = load_dataset("Yuki20/pokemon_caption", split="train")
dataloader = DataLoader(dataset, batch_size=4, collate_fn=collate_fn)

# Output shows varying image sizes:
# [(1280, 1280), (431, 431), (789, 789), (769, 769)]
```

### Expected behavior

I'm looking for a way to resize images on-the-fly when loading the dataset, similar to PyTorch's Dataset.__getitem__ functionality. This would be more efficient than handling resizing in the collate_fn.


### Environment info

- `datasets` version: 3.1.0
- Platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.11.10
- `huggingface_hub` version: 0.26.2
- PyArrow version: 18.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
GitHub issue: wrong return type for `IterableDataset.shard()`
Author: ysngshn (NONE)
State: open | Created: 2024-11-22 17:25:46

### Describe the bug

`IterableDataset.shard()` has the wrong typing for its return as `"Dataset"`. It should be `"IterableDataset"`. Makes my IDE unhappy.

### Steps to reproduce the bug

look at [the source code](https://github.com/huggingface/datasets/blob/main/src/datasets/iterable_dataset.py#L2668)?

### Expected behavior

Correct return type as `"IterableDataset"`

### Environment info

datasets==3.1.0
GitHub issue: Why return_tensors='pt' doesn't work？
Author: bw-wang19 (NONE)
State: open | Created: 2024-11-15 15:01:23

### Describe the bug

I tried to add input_ids to dataset with map(), and I used the return_tensors='pt', but why I got the callback with the type of List？
![image](https://github.com/user-attachments/assets/ab046e20-2174-4e91-9cd6-4a296a43e83c)


### Steps to reproduce the bug

![image](https://github.com/user-attachments/assets/5d504d4c-22c7-4742-99a1-9cab78739b17)

### Expected behavior

Sorry for this silly question, I'm noob on using this tool. But I think it should return a tensor value as I have used the protocol？
When I tokenize only one sentence using tokenized_input=tokenizer(input, return_tensors='pt' )，it does return in tensor type. Why doesn't it work in map()？

### Environment info

transformers>=4.41.2,<=4.45.0
datasets>=2.16.0,<=2.21.0
accelerate>=0.30.1,<=0.34.2
peft>=0.11.1,<=0.12.0
trl>=0.8.6,<=0.9.6
gradio>=4.0.0
pandas>=2.0.0
scipy
einops
sentencepiece
tiktoken
protobuf
uvicorn
pydantic
fastapi
sse-starlette
matplotlib>=3.7.0
fire
packaging
pyyaml
numpy<2.0.0
GitHub issue: `Dataset.save_to_disk` hangs when using num_proc > 1
Author: JohannesAck (NONE)
State: open | Created: 2024-11-14 05:25:13

### Describe the bug

Hi, I'm encountered a small issue when saving datasets that led to the saving taking up to multiple hours.
Specifically, [`Dataset.save_to_disk`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.save_to_disk) is a lot slower when using `num_proc>1` than when using `num_proc=1`

The documentation mentions that "Multiprocessing is disabled by default.", but there is no explanation on how to enable it.

### Steps to reproduce the bug

```
import numpy as np
from datasets import Dataset

n_samples = int(4e6)
n_tokens_sample = 100
data_dict = {
    'tokens' : np.random.randint(0, 100, (n_samples, n_tokens_sample)),
}

dataset = Dataset.from_dict(data_dict)
dataset.save_to_disk('test_dataset', num_proc=1)
dataset.save_to_disk('test_dataset', num_proc=4)
dataset.save_to_disk('test_dataset', num_proc=8)
```

This results in:
```
>>> dataset.save_to_disk('test_dataset', num_proc=1)
Saving the dataset (7/7 shards): 100%|██████████████| 4000000/4000000 [00:17<00:00, 228075.15 examples/s]
>>> dataset.save_to_disk('test_dataset', num_proc=4)
Saving the dataset (7/7 shards): 100%|██████████████| 4000000/4000000 [01:49<00:00, 36583.75 examples/s]
>>> dataset.save_to_disk('test_dataset', num_proc=8)
Saving the dataset (8/8 shards): 100%|██████████████| 4000000/4000000 [02:11<00:00, 30518.43 examples/s]
```

With larger datasets it can take hours, but I didn't benchmark that for this bug report.

### Expected behavior

I would expect using `num_proc>1` to be faster instead of slower than `num_proc=1`. 

### Environment info

- `datasets` version: 3.1.0
- Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.26.2
- PyArrow version: 18.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1
GitHub issue: Support for identifier-based automated split construction
Author: alex-hh (CONTRIBUTOR)
State: open | Created: 2024-11-10 07:45:19
Labels: name: enhancement

### Feature request

As far as I understand, automated construction of splits for hub datasets is currently based on either file names or directory structure ([as described here](https://huggingface.co/docs/datasets/en/repository_structure))

It would seem to be pretty useful to also allow splits to be based on identifiers of individual examples

This could be configured like 
{"split_name": {"column_name": [column values in split]}}

(This in turn requires unique 'index' columns, which could be explicitly supported or just assumed to be defined appropriately by the user).

I guess a potential downside would be that shards would end up spanning different splits - is this something that can be handled somehow? Would this only affect streaming from hub?

### Motivation

The main motivation would be that all data files could be stored in a single directory, and multiple sets of splits could be generated from the same data. This is often useful for large datasets with multiple distinct sets of splits.

This could all be configured via the README.md yaml configs

### Your contribution

May be able to contribute if it seems like a good idea
GitHub issue: Concurrent loading in `load_from_disk` - `num_proc` as a param 
Author: unography (NONE)
State: closed | Created: 2024-11-08 23:21:40
Labels: name: enhancement

### Feature request

https://github.com/huggingface/datasets/pull/6464 mentions a `num_proc` param while loading dataset from disk, but can't find that in the documentation and code anywhere

### Motivation

Make loading large datasets from disk faster

### Your contribution

Happy to contribute if given pointers
GitHub issue: Faulty datasets.exceptions.ExpectedMoreSplitsError
Author: meg-huggingface (CONTRIBUTOR)
State: open | Created: 2024-11-07 20:15:01

### Describe the bug

Trying to download only the 'validation' split of my dataset; instead hit the error `datasets.exceptions.ExpectedMoreSplitsError`.
Appears to be the same undesired behavior as reported in [#6939](https://github.com/huggingface/datasets/issues/6939), but with `data_files`, not `data_dir`.

Here is the Traceback:
```
Traceback (most recent call last):
  File "/home/user/app/app.py", line 12, in <module>
    ds = load_dataset('datacomp/imagenet-1k-random0.0', token=GATED_IMAGENET, data_files={'validation': 'data/val*'}, split='validation', trust_remote_code=True)
  File "/usr/local/lib/python3.10/site-packages/datasets/load.py", line 2154, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/site-packages/datasets/builder.py", line 1018, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/site-packages/datasets/utils/info_utils.py", line 68, in verify_splits
    raise ExpectedMoreSplitsError(str(set(expected_splits) - set(recorded_splits)))
datasets.exceptions.ExpectedMoreSplitsError: {'train', 'test'}
```

Note: I am using the `data_files` argument only because I am trying to specify that I only want the 'validation' split, and the whole dataset will be downloaded even when the `split='validation'` argument is specified, unless you also specify `data_files`, as described here: https://discuss.huggingface.co/t/how-can-i-download-a-specific-split-of-a-dataset/79027


### Steps to reproduce the bug

1. Create a Space with the default blank 'gradio' SDK https://huggingface.co/new-space
2. Create a file 'app.py' that loads a dataset to only extract a 'validation' split:

`ds = load_dataset('datacomp/imagenet-1k-random0.0', token=GATED_IMAGENET, data_files={'validation': 'data/val*'}, split='validation', trust_remote_code=True)`

### Expected behavior

Downloading validation split.

### Environment info

Default environment for creating a new Space. Relevant to this bug, that is:

```
FROM docker.io/library/python:3.10@sha256:fd0fa50d997eb56ce560c6e5ca6a1f5cf8fdff87572a16ac07fb1f5ca01eb608

--> RUN pip install --no-cache-dir pip==22.3.1 && 	pip install --no-cache-dir 	datasets 	"huggingface-hub>=0.19" "hf-transfer>=0.1.4" "protobuf<4" "click<8.1" 
```
GitHub issue: File not found error
Author: MichielBontenbal (NONE)
State: open | Created: 2024-11-07 09:04:49

### Describe the bug

I get a FileNotFoundError:
<img width="944" alt="image" src="https://github.com/user-attachments/assets/1336bc08-06f6-4682-a3c0-071ff65efa87">


### Steps to reproduce the bug

See screenshot.



### Expected behavior

I want to load one audiofile from the dataset. 

### Environment info

MacOs Intel 14.6.1 (23G93)
Python 3.10.9
Numpy 1.23
Datasets latest version
GitHub issue: Add filename in error message when ReadError or similar occur
Author: elisa-aleman (NONE)
State: open | Created: 2024-11-07 06:00:53

Please update error messages to include relevant information for debugging when loading datasets with `load_dataset()` that may have a few corrupted files. 

Whenever downloading a full dataset, some files might be corrupted (either at the source or from downloading corruption). 

However the errors often only let me know it was a tar file if `tarfile.ReadError` appears on the traceback, and I imagine similarly for other file types. 

This makes it really hard to debug which file is corrupted, and when dealing with very large datasets, it shouldn't be necessary to force download everything again.
GitHub issue: load_dataset
Author: santiagobp99 (NONE)
State: open | Created: 2024-11-04 03:01:44

### Describe the bug

I am performing two operations I see on a hugging face tutorial (Fine-tune a language model), and I am defining every aspect inside the mapped functions, also some imports of the library because it doesnt identify anything not defined outside that function where the dataset elements are being mapped:

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

`- lm_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    batch_size=batch_size,
    num_proc=4,
)

- tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])
def tokenize_function(examples):
    model_checkpoint = 'gpt2'
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    return tokenizer(examples["text"])`


### Steps to reproduce the bug

Currently handle all the imports inside the function

### Expected behavior

The code must work es expected in the notebook, but currently this is not happening.

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

### Environment info

print(transformers.__version__)

4.46.1
GitHub issue: Memory leak when streaming
Author: Jourdelune (NONE)
State: open | Created: 2024-10-31 13:33:52

### Describe the bug

I try to use a dataset with streaming=True, the issue I have is that the RAM usage becomes higher and higher until it is no longer sustainable.

I understand that huggingface store data in ram during the streaming, and more worker in dataloader there are, more a lot of shard will be stored in ram, but the issue I have is that the ram usage is not constant. So after each new shard loaded, the ram usage will be higher and higher.

### Steps to reproduce the bug

You can run this code and see you ram usage, after each shard of 255 examples, your ram usage will be extended.
```py
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("WaveGenAI/dataset", streaming=True)

dataloader = DataLoader(dataset["train"], num_workers=3)

for i, data in enumerate(dataloader):
    print(i, end="\r")
```

### Expected behavior

The Ram usage should be always the same (just 3 shards loaded in the ram).

### Environment info

- `datasets` version: 3.0.1
- Platform: Linux-6.10.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1
GitHub issue: load_from_disk
Author: ghaith-mq (NONE)
State: open | Created: 2024-10-31 11:51:56

### Describe the bug

I have data saved with save_to_disk. The data is big (700Gb). When I try loading it, the only option is load_from_disk, and this function copies the data to a tmp directory, causing me to run out of disk space. Is there an alternative solution to that?

### Steps to reproduce the bug

when trying to load data using load_From_disk after being saved using save_to_disk 

### Expected behavior

run out of disk space

### Environment info

lateest version
GitHub issue: Source installation fails on Macintosh with python 3.10
Author: mayankagarwals (NONE)
State: open | Created: 2024-10-31 10:18:45

### Describe the bug

Hi, 

Decord is a dev dependency not maintained since couple years. 
It does not have an ARM package available rendering it uninstallable on non-intel based macs 

Suggestion is to move to eva-decord (https://github.com/georgia-tech-db/eva-decord)  which doesnt have this problem. 

Happy to raise a PR


### Steps to reproduce the bug

Source installation as mentioned in contributinog.md

### Expected behavior

Installation without decord failing to be installed.

### Environment info

python=3.10, M3 Mac
GitHub issue: The dataset viewer should be available soon. Please retry later.
Author: viiika (NONE)
State: closed | Created: 2024-10-30 16:32:00

### Describe the bug

After waiting for 2 hours, it still presents ``The dataset viewer should be available soon. Please retry later.''

### Steps to reproduce the bug

dataset link: https://huggingface.co/datasets/BryanW/HI_EDIT

### Expected behavior

Present the dataset viewer.

### Environment info

NA
GitHub issue: Cannot load the cache when mapping the dataset
Author: zhangn77 (NONE)
State: open | Created: 2024-10-29 08:29:40

### Describe the bug

I'm training the flux controlnet. The train_dataset.map() takes long time to finish. However, when I killed one training process and want to restart a new training with the same dataset. I can't reuse the mapped result even I defined the cache dir for the dataset.

with accelerator.main_process_first():
        from datasets.fingerprint import Hasher

        # fingerprint used by the cache for the other processes to load the result
        # details: https://github.com/huggingface/diffusers/pull/4038#discussion_r1266078401
        new_fingerprint = Hasher.hash(args)
        train_dataset = train_dataset.map(
            compute_embeddings_fn, batched=True, new_fingerprint=new_fingerprint, batch_size=10,
        )

### Steps to reproduce the bug

train flux controlnet and start again



### Expected behavior

will not map again

### Environment info

latest diffusers
GitHub issue: cache can't cleaned or disabled
Author: charliedream1 (NONE)
State: open | Created: 2024-10-29 03:15:28

### Describe the bug

I tried following ways, the cache can't be disabled. 

I got 2T data, but I also got more than 2T cache file. I got pressure on storage. I need to diable cache or cleaned immediately after processed. Following ways are all not working, please give some help!

```python
from datasets import disable_caching
from transformers import AutoTokenizer
disable_caching()

tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_path)
def tokenization_fn(examples):
    column_name = 'text' if 'text' in examples else 'data'
    tokenized_inputs = tokenizer(
        examples[column_name], return_special_tokens_mask=True, truncation=False,
        max_length=tokenizer.model_max_length
    )
    return tokenized_inputs

data = load_dataset('json', data_files=save_local_path, split='train', cache_dir=None)
data.cleanup_cache_files()
updated_dataset = data.map(tokenization_fn, load_from_cache_file=False)
updated_dataset .cleanup_cache_files()
```

### Expected behavior

no cache file generated

### Environment info

Ubuntu 20.04.6 LTS
datasets  3.0.2
GitHub issue: mismatch for datatypes when providing `Features` with `Array2D` and user specified `dtype` and using with_format("numpy")
Author: Akhil-CM (NONE)
State: open | Created: 2024-10-26 22:06:27

### Describe the bug

If the user provides a `Features` type value to `datasets.Dataset` with members having `Array2D` with a value for `dtype`, it is not respected during `with_format("numpy")` which should return a `np.array` with `dtype` that the user provided for `Array2D`. It seems for floats, it will be set to `float32` and for ints it will be set to `int64`

### Steps to reproduce the bug

```python
import numpy as np
import datasets
from datasets import Dataset, Features, Array2D

print(f"datasets version: {datasets.__version__}")

data_info = {
    "arr_float" : "float64",
    "arr_int" : "int32"
}

sample = {key : [np.zeros([4, 5], dtype=dtype)] for key, dtype in data_info.items()}

features = {key : Array2D(shape=(None, 5), dtype=dtype) for key, dtype in data_info.items()}
features = Features(features)

dataset = Dataset.from_dict(sample, features=features)

ds = dataset.with_format("numpy")
for key in features:
    print(f"{key} feature dtype: ", ds.features[key].dtype)
    print(f"{key} dtype:", ds[key].dtype)
```

Output:
```bash
datasets version: 3.0.2
arr_float feature dtype:  float64
arr_float dtype: float32
arr_int feature dtype:  int32
arr_int dtype: int64
```

### Expected behavior

It should return a `np.array` with `dtype` that the user provided for the corresponding member in the `Features` type value

### Environment info

- `datasets` version: 3.0.2
- Platform: Linux-6.11.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.1
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0
GitHub issue: Unable to upload a large dataset zip either from command line or UI
Author: vakyansh (NONE)
State: open | Created: 2024-10-26 13:17:06

### Describe the bug

Unable to upload a large dataset zip from command line or UI. UI simply says error. I am trying to a upload a tar.gz file of 17GB.

<img width="550" alt="image" src="https://github.com/user-attachments/assets/f9d29024-06c8-49c4-a109-0492cff79d34">
<img width="755" alt="image" src="https://github.com/user-attachments/assets/a8d4acda-7f02-4279-9c2d-b2e0282b4faa">


### Steps to reproduce the bug

Upload a large file 

### Expected behavior

The file should upload without any issue.

### Environment info

None
GitHub issue: Add IterableDataset.shard()
Author: lhoestq (MEMBER)
State: closed | Created: 2024-10-25 11:07:12

Will be useful to distribute a dataset across workers (other than pytorch) like spark

I also renamed `.n_shards` -> `.num_shards` for consistency and kept the old name for backward compatibility. And a few changes in internal functions for consistency as well (rank, world_size -> num_shards, index)

Breaking change: the new default for `contiguous` in `Dataset.shard()` is `True`, but imo not a big deal since I couldn't find any usage of `contiguous=False` internally (we always do contiguous=True for map-style datasets since its more optimized) or in the wild
GitHub issue: How to debugging
Author: ShDdu (NONE)
State: open | Created: 2024-10-24 01:03:51

### Describe the bug

I wanted to use my own script to handle the processing, and followed the tutorial documentation by rewriting the MyDatasetConfig and MyDatasetBuilder (which contains the _info,_split_generators and _generate_examples methods) classes. Testing with simple data was able to output the results of the processing, but when I wished to do more complex processing, I found that I was unable to debug (even the simple samples were inaccessible). There are no errors reported, and I am able to print the _info,_split_generators and _generate_examples messages, but I am unable to access the breakpoints.

### Steps to reproduce the bug

# my_dataset.py
import json
import datasets


class MyDatasetConfig(datasets.BuilderConfig):
    def __init__(self, **kwargs):
        super(MyDatasetConfig, self).__init__(**kwargs)


class MyDataset(datasets.GeneratorBasedBuilder):
    VERSION = datasets.Version("1.0.0")

    BUILDER_CONFIGS = [
        MyDatasetConfig(
            name="default",
            version=VERSION,
            description="myDATASET"
        ),
    ]

    def _info(self):
        print("info")  # breakpoints
        return datasets.DatasetInfo(
            description="myDATASET",
            features=datasets.Features(
                {
                    "id": datasets.Value("int32"),
                    "text": datasets.Value("string"),
                    "label": datasets.ClassLabel(names=["negative", "positive"]),
                }
            ),
            supervised_keys=("text", "label"),
        )

    def _split_generators(self, dl_manager):
 
        print("generate")  # breakpoints
        data_file = "data.json"  

        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN, gen_kwargs={"filepath": data_file}
            ),
        ]

    def _generate_examples(self, filepath):
        print("example")  # breakpoints
        with open(filepath, encoding="utf-8") as f:
            data = json.load(f)
            for idx, sample in enumerate(data):
                yield idx, {
                    "id": sample["id"],
                    "text": sample["text"],
                    "label": sample["label"],
                }

#main.py
import os
os.environ["TRANSFORMERS_NO_MULTIPROCESSING"] = "1" 

from datasets import load_dataset

dataset = load_dataset("my_dataset.py", split="train", cache_dir=None)

print(dataset[:5])

### Expected behavior

Pause at breakpoints while running debugging

### Environment info

pycharm
GitHub issue: Adding column with dict struction when mapping lead to wrong order
Author: chchch0109 (NONE)
State: open | Created: 2024-10-22 18:55:11

### Describe the bug

in `map()` function, I want to add a new column with a dict structure.
```
def map_fn(example):
  example['text'] = {'user': ..., 'assistant': ...}
  return example
```
However this leads to a wrong order `{'assistant':..., 'user':...}` in the dataset.
Thus I can't concatenate two datasets due to the different feature structures.
[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example
This seems an issue in low level pyarrow library instead of datasets, however, I think datasets should allow concatenate two datasets actually in the same structure.

### Steps to reproduce the bug

[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example

### Expected behavior

two datasets could be concatenated.

### Environment info

N/A
GitHub issue: ArrayXD with None as leading dim incompatible with DatasetCardData
Author: alex-hh (CONTRIBUTOR)
State: open | Created: 2024-10-21 15:08:13

### Describe the bug

Creating a dataset with ArrayXD features leads to errors when downloading from hub due to DatasetCardData removing the Nones

@lhoestq 

### Steps to reproduce the bug

```python
import numpy as np
from datasets import Array2D, Dataset, Features, load_dataset

def examples_generator():
    for i in range(4):
        yield {
            "array_1d": np.zeros((10,1), dtype="uint16"),
            "array_2d": np.zeros((10, 1), dtype="uint16"),
        }

features = Features(array_1d=Array2D((None,1), "uint16"), array_2d=Array2D((None, 1), "uint16"))
dataset = Dataset.from_generator(examples_generator, features=features)
dataset.push_to_hub("alex-hh/test_array_1d2d")
ds = load_dataset("alex-hh/test_array_1d2d")
```

Source of error appears to be DatasetCardData.to_dict invoking DatasetCardData._remove_none
```python
from huggingface_hub import DatasetCardData
from datasets.info import DatasetInfosDict

dataset_card_data = DatasetCardData()
DatasetInfosDict({"default": dataset.info.copy()}).to_dataset_card_data(dataset_card_data)
print(dataset_card_data.to_dict())  # removes Nones in shape
```

### Expected behavior

Should be possible to load datasets saved with shape None in leading dimension

### Environment info

3.0.2 and latest huggingface_hub
GitHub issue: `push_to_hub` overwrite argument
Author: ceferisbarov (NONE)
State: closed | Created: 2024-10-20 03:23:26
Labels: name: enhancement

### Feature request

Add an `overwrite` argument to the `push_to_hub` method.

### Motivation

I want to overwrite a repo without deleting it on Hugging Face. Is this possible? I couldn't find anything in the documentation or tutorials.

### Your contribution

I can create a PR.
GitHub issue: Feature Request: Add functionality to pass split types like train, test in DatasetDict.map
Author: jp1924 (NONE)
State: open | Created: 2024-10-19 09:59:12

Hello datasets!

We often encounter situations where we need to preprocess data differently depending on split types such as train, valid, and test. 

However, while DatasetDict.map has features to pass rank or index, there's no functionality to pass split types. 
Therefore, I propose adding a 'with_splits' parameter to DatasetDict, which would allow passing the split type through fn_kwargs.
GitHub issue: incompatibily issue when using load_dataset with datasets==3.0.1
Author: jupiterMJM (NONE)
State: open | Created: 2024-10-18 21:25:23

### Describe the bug

There is a bug when using load_dataset with dataset version at 3.0.1 .
Please see below in the "steps to reproduce the bug".
To resolve the bug, I had to downgrade to version 2.21.0
OS: Ubuntu 24 (AWS instance)
Python: same bug under 3.12 and 3.10


The error I had was:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/load.py", line 2096, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 1647, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/builder.py", line 977, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/ubuntu/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_6_0/cb17afd34f5799f97e8f48398748f83006335b702bd785f9880797838d541b81/common_voice_6_0.py", line 159, in _split_generators
    archive_path = dl_manager.download(self._get_bundle_url(self.config.name, bundle_url_template))
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_manager.py", line 150, in download
    download_config = self.download_config.copy()
  File "/home/ubuntu/miniconda3/envs/maxence_env/lib/python3.10/site-packages/datasets/download/download_config.py", line 73, in copy
    return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
TypeError: DownloadConfig.__init__() got an unexpected keyword argument 'ignore_url_params'

### Steps to reproduce the bug

1. install dataset with ```pip install datasets --upgrade```
2. launch python; from datasets import loaad_dataset
3. run load_dataset("mozilla-foundation/common_voice_6_0")
4. exit python
5. uninstall datasets; then ```pip install datasets==2.21.0```
6. launch python; from datasets import loaad_dataset
7. run load_dataset("mozilla-foundation/common_voice_6_0")
8. Everything runs great now

### Expected behavior

Be able to download a dataset without error

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 3.0.1
- Platform: Linux-6.8.0-1017-aws-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.26.0
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1
GitHub issue: Video support
Author: lhoestq (MEMBER)
State: closed | Created: 2024-10-15 18:17:29

(wip and experimental)

adding the `Video` type based on `VideoReader` from `decord`

```python
>>>from datasets import load_dataset
>>> ds = load_dataset("path/to/videos", split="train").with_format("torch")
>>> print(ds[0]["video"])
<decord.video_reader.VideoReader object at 0x337a47910>
>>> print(ds[0]["video"][0])
tensor([[[73, 73, 73],
         [73, 73, 73],
         [73, 73, 73],
         ...,
         [23, 23, 23],
         [23, 23, 23],
         [23, 23, 23]]], dtype=torch.uint8)
```

the storage is the same as for audio and images: `{"path": pa.string(), "bytes": pa.binary()}` and I did a small to keep the hf:// URL in the "path" field if possible, this way the viewer can link to fiels on the hub if possible
GitHub issue: Composite (multi-column) features
Author: alex-hh (CONTRIBUTOR)
State: open | Created: 2024-10-14 23:59:19
Labels: name: enhancement

### Feature request

Structured data types (graphs etc.) might often be most efficiently stored as multiple columns, which then need to be combined during feature decoding

Although it is currently possible to nest features as structs, my impression is that in particular when dealing with e.g. a feature composed of multiple numpy array / ArrayXD's, it would be more efficient to store each ArrayXD as a separate column (though I'm not sure by how much)

Perhaps specification / implementation could be supported by something like:

```
features=Features(**{("feature0", "feature1")=Features(feature0=Array2D((None,10), dtype="float32"), feature1=Array2D((None,10), dtype="float32"))
```

### Motivation

Defining efficient composite feature types based on numpy arrays for representing data such as graphs with multiple node and edge attributes is currently challenging.

### Your contribution

Possibly able to contribute
GitHub issue: fast array extraction
Author: alex-hh (CONTRIBUTOR)
State: open | Created: 2024-10-14 20:51:32

Implements #7210 using method suggested in https://github.com/huggingface/datasets/pull/7207#issuecomment-2411789307


```python
import numpy as np
from datasets import Dataset, Features, Array3D
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```
~0.02 s vs 0.9s on main

```python
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
< 0.01 s vs 1.3 s on main

@lhoestq I can see this breaks a bunch of array-related tests but can update the test cases if you would support making this change?

I also added an Array1D feature which will always be decoded into a numpy array and likewise improves extraction performance:

```python
from datasets import Dataset, Features, Array1D, Sequence, Value
array_features=Features(**{"array0": Array1D((None,), dtype="float32"), "array1": Array1D((None,), dtype="float32")})
sequence_features=Features(**{"array0": Sequence(feature=Value("float32"), length=-1), "array1": Sequence(feature=Value("float32"), length=-1)})
array_dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [20000,10000]*25] for i in range(2)}, features=array_features)
sequence_dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [20000,10000]*25] for i in range(2)}, features=sequence_features)


```python
t0 = time.time()
for ex in array_dataset.to_iterable_dataset():
    pass
t1 = time.time()
```
< 0.01 s

```python
t0 = time.time()
for ex in sequence_dataset.to_iterable_dataset():
    pass
t1 = time.time()
```
~1.1s

And also added support for extracting structs of arrays as dicts of numpy arrays:

```python
import numpy as np
from datasets import Dataset, Features, Array3D, Sequence
features=Features(struct={"array0": Array3D((None,10,10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")}, _list=Sequence(feature=Array3D((None,10,10), dtype="float32")))
dataset = Dataset.from_dict({"struct": [{f"array{i}": np.zeros((x,10,10), dtype=np.float32) for i in range(2)} for x in [2000,1000]*25], "_list": [[np.zeros((x,10,10), dtype=np.float32) for i in range(2)] for x in [2000,1000]*25]}, features=features)
```

```python
t0 = time.time()
for ex in dataset.to_iterable_dataset():
    pass
t1 = time.time()
assert isinstance(ex["struct"]["array0"], np.ndarray) and ex["struct"]["array0"].ndim == 3
```
~0.02 s and no exception vs ~7s with an exception on main
GitHub issue: Add R as a How to use from the Polars (R) Library as an option
Author: ran-codes (NONE)
State: open | Created: 2024-10-14 19:56:07
Labels: name: enhancement

### Feature request

The boiler plate code to access a dataset via the hugging face file system is very useful. Please addd 


## Add Polars (R) option
The equivailent code works, because the [Polars-R](https://github.com/pola-rs/r-polars) wrapper has hugging faces funcitonaliy as well.

```r
library(polars)

df <- pl$read_parquet("hf://datasets/SALURBAL/core__admin_cube_public/core__admin_cube_public.parquet")
```

## Polars (python) option
![image](https://github.com/user-attachments/assets/8f1bcd19-e578-4b18-b324-7cc00b80ac0a)


## Libraries Currently

![image](https://github.com/user-attachments/assets/0cf50063-f9db-443c-97b4-3ef0664b6e6e)




### Motivation

There are many data/analysis/research/statistics teams (particularly in academia and pharma) that use R as the default language. R has great integration with most of the newer data techs (arrow, parquet, polars) and having this included could really help in bringing this community into the hugging faces ecosystem.

**This is a small/low-hanging-fruit front end change but would make a big impact expanding the community**

### Your contribution

I am not sure which repositroy this should be in, but I have experience in R, Python and JS and happy to submit a PR in the appropriate repository.
GitHub issue: Huggingface GIT returns null as Content-Type instead of application/x-git-receive-pack-result
Author: padmalcom (NONE)
State: open | Created: 2024-10-14 14:33:06

### Describe the bug

We push changes to our datasets programmatically. Our git client jGit reports that the hf git server returns null as Content-Type after a push.

### Steps to reproduce the bug

A basic kotlin application:
```
   val person = PersonIdent(
        "padmalcom",
        "padmalcom@sth.com"
    )

    val cp = UsernamePasswordCredentialsProvider(
        "padmalcom",
        "mysecrettoken"
    )

    val git =
        KGit.cloneRepository {
            setURI("https://huggingface.co/datasets/sth/images")
            setTimeout(60)
            setProgressMonitor(TextProgressMonitor())
            setCredentialsProvider(cp)
        }

    FileOutputStream("./images/images.csv").apply { writeCsv(images) }
    git.add {
        addFilepattern("images.csv")
    }

    for (i in images) {
        FileUtils.copyFile(
            File("./files/${i.id}"),
            File("./images/${i.id + File(i.fileName).extension }")
        )
        git.add {
            addFilepattern("${i.id + File(i.fileName).extension }")
        }
    }

    val revCommit = git.commit {
        author = person
        message = "Uploading images at " + LocalDateTime.now()
            .format(DateTimeFormatter.ISO_DATE_TIME)
        setCredentialsProvider(cp)
    }

    val push = git.push {
        setCredentialsProvider(cp)
    }
```

### Expected behavior

The git server is expected to return the Content-Type _application/x-git-receive-pack-result_.

### Environment info

It is independent from the datasets library.
GitHub issue: Fallback to arrow defaults when loading dataset with custom features that aren't registered locally
Author: alex-hh (CONTRIBUTOR)
State: open | Created: 2024-10-12 16:08:20

### Describe the bug

Datasets allows users to create and register custom features.

However if datasets are then pushed to the hub, this means that anyone calling load_dataset without registering the custom Features in the same way as the dataset creator will get an error message.

It would be nice to offer a fallback in this case.

### Steps to reproduce the bug

```python
load_dataset("alex-hh/custom-features-example")
```
(Dataset creation process - must be run in separate session so that NewFeature isn't registered in session in which download is attempted:)

```python
from dataclasses import dataclass, field
import pyarrow as pa
from datasets.features.features import register_feature

from datasets import Dataset, Features, Value, load_dataset
from datasets import Feature

@dataclass
class NewFeature(Feature):
    _type: str = field(default="NewFeature", init=False, repr=False)
    def __call__(self):
        return pa.int32()

def examples_generator():
    for i in range(5):
        yield {"feature": i}

ds = Dataset.from_generator(examples_generator, features=Features(feature=NewFeature()))
ds.push_to_hub("alex-hh/custom-features-example")
register_feature(NewFeature, "NewFeature")
```



### Expected behavior

It would be nice, and offer greater extensibility, if there was some kind of graceful fallback mechanism in place for cases where user-defined features are stored in the dataset but not available locally.

### Environment info

3.0.2
GitHub issue: add CustomFeature base class to support user-defined features with encoding/decoding logic
Author: alex-hh (CONTRIBUTOR)
State: open | Created: 2024-10-11 20:10:27

intended as fix for #7220 if this kind of extensibility is something that datasets is willing to support!


```python
from datasets.features.features import CustomFeature


class ListOfStrs(CustomFeature):
    requires_encoding = True
    def _encode_example(self, value):
        if isinstance(value, str):
            return [str]
        else:
            return value
feats = Features(strlist=ListOfStrs())
feats.encode_example({"strlist": "a"})["strlist"] == feats["strlist"].encode_example("a")
```
GitHub issue: Custom features not compatible with special encoding/decoding logic
Author: alex-hh (CONTRIBUTOR)
State: open | Created: 2024-10-11 19:20:11

### Describe the bug

It is possible to register custom features using datasets.features.features.register_feature (https://github.com/huggingface/datasets/pull/6727)

However such features are not compatible with Features.encode_example/decode_example if they require special encoding / decoding logic because encode_nested_example / decode_nested_example checks whether the feature is in a fixed list of encodable types:

https://github.com/huggingface/datasets/blob/16a121d7821a7691815a966270f577e2c503473f/src/datasets/features/features.py#L1349

This prevents the extensibility of features to complex cases

### Steps to reproduce the bug

```python
class ListOfStrs:
    def encode_example(self, value):
        if isinstance(value, str):
            return [str]
        else:
            return value
feats = Features(strlist=ListOfStrs())
assert feats.encode_example({"strlist": "a"})["strlist"] = feats["strlist"].encode_example("a")}
```

### Expected behavior

Registered feature types should be encoded based on some property of the feature (e.g. requires_encoding)?

### Environment info

3.0.2
GitHub issue: ds.map(f, num_proc=10) is slower than df.apply
Author: lanlanlanlanlanlan365 (NONE)
State: open | Created: 2024-10-11 11:04:05

### Describe the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

time cost: 
1. df.apply: 100%|██████████| 12500592/12500592 [00:13<00:00, 959825.47it/s]
2. ds.map: Map (num_proc=10):  31%
 3899028/12500592 [00:28<00:38, 222532.89 examples/s]

### Steps to reproduce the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

### Expected behavior

ds.map is ~num_proc faster than df.apply

### Environment info

pandas: 2.2.2
datasets: 2.19.1
GitHub issue: Iterable dataset map with explicit features causes slowdown for Sequence features
Author: alex-hh (CONTRIBUTOR)
State: open | Created: 2024-10-10 22:08:20

### Describe the bug

When performing map, it's nice to be able to pass the new feature type, and indeed required by interleave and concatenate datasets.

However, this can cause a major slowdown for certain types of array features due to the features being re-encoded.

This is separate to the slowdown reported in #7206 

### Steps to reproduce the bug

```
from datasets import Dataset, Features, Array3D, Sequence, Value
import numpy as np
import time
features=Features(**{"array0": Sequence(feature=Value("float32"), length=-1), "array1": Sequence(feature=Value("float32"), length=-1)})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,), dtype=np.float32) for x in [5000,10000]*25] for i in range(2)}, features=features)
```

```
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~1.5 s on main


```
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x, features=features)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
```
~ 3 s on main

### Expected behavior

I'm not 100% sure whether passing new feature types to formatted outputs of map should be supported or not, but assuming it should, then there should be a cost-free way to specify the new feature type - knowing feature type is required by interleave_datasets and concatenate_datasets for example

### Environment info

3.0.2
GitHub issue: Formatted map + with_format(None) changes array dtype for iterable datasets
Author: alex-hh (CONTRIBUTOR)
State: open | Created: 2024-10-10 12:45:16

### Describe the bug

When applying with_format -> map -> with_format(None), array dtypes seem to change, even if features are passed 

### Steps to reproduce the bug

```python
features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32")})
dataset = Dataset.from_dict({f"array0": [np.zeros((100,10,10), dtype=np.float32)]*25}, features=features)
ds = dataset.to_iterable_dataset().with_format("numpy").map(lambda x: x, features=features)
ex_0 = next(iter(ds))
ds = dataset.to_iterable_dataset().with_format("numpy").map(lambda x: x, features=features).with_format(None)
ex_1 = next(iter(ds))
assert ex_1["array0"].dtype == ex_0["array0"].dtype, f"{ex_1['array0'].dtype} {ex_0['array0'].dtype}"
```


### Expected behavior

Dtypes should be preserved.

### Environment info

3.0.2
GitHub issue: Add with_rank to Dataset.from_generator
Author: muthissar (NONE)
State: open | Created: 2024-10-10 12:15:29
Labels: name: enhancement

### Feature request

Add `with_rank` to `Dataset.from_generator` similar to `Dataset.map` and `Dataset.filter`. 

### Motivation

As for `Dataset.map` and `Dataset.filter`, this is useful when creating cache files using multi-GPU, where the rank can be used to select GPU IDs. For now, rank can be added in the `gen_kwars` argument; however, this, in turn, includes the rank when computing the fingerprint.

### Your contribution

Added #7199 which passes rank based on the `job_id` set by `num_proc`.
GitHub issue: Windows do not supprot signal.alarm and singal.signal
Author: TomasJavurek (NONE)
State: open | Created: 2024-10-10 12:00:19

### Describe the bug

signal.alarm and signal.signal are used in the load.py module, but these are not supported by Windows.

### Steps to reproduce the bug

 lighteval accelerate --model_args "pretrained=gpt2,trust_remote_code=True" --tasks "community|kinit_sts" --custom_tasks "community_tasks/kinit_evals.py" --output_dir "./evals"

### Expected behavior

proceed with input(..) method

### Environment info

Windows 11
GitHub issue: Describe only selected fields in README
Author: alozowski (NONE)
State: open | Created: 2024-10-09 16:25:47
Labels: name: enhancement

### Feature request

Hi Datasets team! 

Is it possible to add the ability to describe only selected fields of the dataset files in `README.md`? For example, I have this open dataset ([open-llm-leaderboard/results](https://huggingface.co/datasets/open-llm-leaderboard/results?row=0)) and I want to describe only some fields in order not to overcomplicate the Dataset Preview and filter out some fields 

### Motivation

The `Results` dataset for the Open LLM Leaderboard contains json files with a complex nested structure. I would like to add `README.md` there to use the SQL console, for example. But if I describe the structure of this dataset completely, it will overcomplicate the use of Dataset Preview and the total number of columns will exceed 50 

### Your contribution

I'm afraid I'm not familiar with the project structure, so I won't be able to open a PR, but I'll try to help with something else if possible