dalssy/data_del huggingface dataset:
GitHub issue: Invalid YAML in README.md: unknown tag !<tag:yaml.org,2002:python/tuple>
Author: juanqui (NONE)
State: open | Created: 2024-05-24 14:59:45

### Describe the bug

I wrote a notebook to load an existing dataset, process it, and upload as a private dataset using `dataset.push_to_hub(...)` at the end. The push to hub is failing with:

```
ValueError: Invalid metadata in README.md.
- Invalid YAML in README.md: unknown tag !<tag:yaml.org,2002:python[/tuple](http://192.168.1.128:8888/tuple)> (50:11)

 47 |             - 4
 48 |             - 4
 49 |             - 8
 50 |           - !!binary |
----------------^
 51 |             TwAAAA==
 52 |           '1': !!python[/object/apply](http://192.168.1.128:8888/object/apply):nump ...
```

My dataset has a `train` and `validation` dataset.  These are the features:

```
{'c1': Value(dtype='string', id=None),
 'c2': Value(dtype='string', id=None),
 'c3': [{'value': Value(dtype='string', id=None),
   'start': Value(dtype='int64', id=None),
   'end': Value(dtype='int64', id=None),
   'label': Value(dtype='string', id=None)}],
 'c4': Value(dtype='string', id=None),
 'c5': Value(dtype='string', id=None),
 'c6': Value(dtype='string', id=None),
 'c7': Value(dtype='string', id=None),
 'c8': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None),
 'c9': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'c10': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
 'labels': Sequence(feature=ClassLabel(names=['O', 'B-ABC', 'I-ABC', ...], id=None), length=-1, id=None),
 'c12': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
```

This used to work until I decided to cast the `labels` feature to a `Sequence(ClassLabel(...))` type with:

```
ds['train'] = ds['train'].cast_column("labels", Sequence(ClassLabel(names=list(labels))))
ds['validation'] = ds['validation'].cast_column("labels", Sequence(ClassLabel(names=list(labels))))
```

### Steps to reproduce the bug

1. Start with any token classification dataset.
2. Add a `labels` column with data such as `[0,0,0,12,13,13,13,0,0]`.
3. Cast the label column from `Sequence` to `Sequence(ClassLabel))` with:

```
labels = ['O', 'B-TEST', 'I-TEST']
ds = ds.cast_column("labels", Sequence(ClassLabel(names=labels)))
```

4. Push to hub with `ds.push_to_hub("me/awesome-stuff-dataset")`

### Expected behavior

I expected `push_to_hub` to successfully push my dataset to the hub without error.

### Environment info

Python 3.11.9

datasets==2.19.1
transformers==4.41.1
PyYAML==6.0.1
GitHub issue: ```push_to_hub()``` - Prevent Automatic Generation of Splits 
Author: jetlime (NONE)
State: closed | Created: 2024-05-22 23:52:15

### Describe the bug

I currently have a dataset which has not been splited. When pushing the dataset to my hugging face dataset repository, it is split into a testing and training set. How can I prevent the split from happening?

### Steps to reproduce the bug

1. Have a unsplit dataset 

```python
Dataset({ features: ['input', 'output', 'Attack', '__index_level_0__'], num_rows: 944685 })
```

2. Push it to huggingface

```python
dataset.push_to_hub(dataset_name)
```

3. On the hugging face dataset repo, the dataset then appears to be splited:

![image](https://github.com/huggingface/datasets/assets/29337128/b4fbc141-42b0-4f49-98df-dd479648fe09)

4. Indeed, when loading the dataset from this repo, the dataset is split in two testing and training set.

```python
from datasets import load_dataset, Dataset

dataset = load_dataset("Jetlime/NF-CSE-CIC-IDS2018-v2", streaming=True)
dataset
```
output: 

```
IterableDatasetDict({
    train: IterableDataset({
        features: ['input', 'output', 'Attack', '__index_level_0__'],
        n_shards: 2
    })
    test: IterableDataset({
        features: ['input', 'output', 'Attack', '__index_level_0__'],
        n_shards: 1
    })
```

### Expected behavior

The dataset shall not be splited, as not requested.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-6.2.0-35-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.23.0
- PyArrow version: 15.0.2
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1
GitHub issue: Validate config name and data_files in packaged modules
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-22 13:36:33

Validate the config attributes `name` and `data_files` in packaged modules by making the derived classes call their parent `__post_init__` method.

Note that their parent `BuilderConfig` validates its attributes `name` and `data_files` in its `__post_init__` method: https://github.com/huggingface/datasets/blob/60d21efbc01e15d0b596ac1072750cbecd91548a/src/datasets/builder.py#L128-L137

This PR makes the derived config classes call their parent `__post_init__` method to validate their `name` and `data_files` attributes.
GitHub issue: Column order is nondeterministic when loading from JSON
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-22 05:30:14
Labels: name: bug

As reported by @meg-huggingface, the order of the JSON object keys is not preserved while loading a dataset from a JSON file with a list of objects.

For example, when loading a JSON files with a list of objects, each with the following ordered keys:
- [ID, Language, Topic], 

the resulting dataset may have columns:
- [ID, Topic, Language], or
- [Topic, Language, ID], or
- [Topic, ID, Language],...

This issue is caused by the use of a Python set (which does not preserve the order):
https://github.com/huggingface/datasets/blob/60d21efbc01e15d0b596ac1072750cbecd91548a/src/datasets/packaged_modules/json/json.py#L168
introduced in
- #5772
GitHub issue: Add MedImg for streaming
Author: lhallee (NONE)
State: open | Created: 2024-05-22 00:55:30
Labels: name: dataset request

### Feature request

Host the MedImg dataset (similar to Imagenet but for biomedical images). 

### Motivation

There is a clear need for biomedical image foundation models and large scale biomedical datasets that are easily streamable. This would be an excellent tool for the biomedical community.

### Your contribution

MedImg can be found [here](https://www.cuilab.cn/medimg/#).
GitHub issue: Remove dead code for non-dict data_files from packaged modules
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-21 12:10:24

Remove dead code for non-dict data_files from packaged modules.

Since the merge of this PR:
- #2986

the builders' variable self.config.data_files is always a dict, which makes the condition on (str, list, tuple) dead code.
GitHub issue: Fail to load "stas/c4-en-10k" dataset since 2.16 version
Author: guch8017 (NONE)
State: closed | Created: 2024-05-20 02:43:59

### Describe the bug

When update datasets library to version 2.16+ ( I test it on 2.16, 2.19.0 and 2.19.1), using the following code to load stas/c4-en-10k dataset 

```python
from datasets import load_dataset, Dataset
dataset = load_dataset('stas/c4-en-10k')
```

and then it raise UnicodeDecodeError like

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 2523, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 2195, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 1846, in dataset_module_factory
    raise e1 from None
  File "/home/*/conda3/envs/watermark/lib/python3.10/site-packages/datasets/load.py", line 1798, in dataset_module_factory
    can_load_config_from_parquet_export = "DEFAULT_CONFIG_NAME" not in f.read()
  File "/home/*/conda3/envs/watermark/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
```

I found that fs.open loads a gzip file and parses it like plain text using utf-8 encoder.

```python
fs = HfFileSystem('https://huggingface.co')
fs.open("datasets/stas/c4-en-10k/c4-en-10k.py", "rb")
data = fs.read()    # data is gzip bytes begin with b'\x1f\x8b\x08\x00\x00\tn\x88\x00...' 
data2 = unzip_gzip_bytes(data)    #  data2 is what we want: '# coding=utf-8\n# Copyright 2020 The HuggingFace Datasets...'
```

### Steps to reproduce the bug

1. Install datasets between version 2.16 and 2.19
2. Use `datasets.load_dataset` method to load `stas/c4-en-10k` dataset.

### Expected behavior

Load dataset normally.

### Environment info

Platform = Linux-5.4.0-159-generic-x86_64-with-glibc2.35
Python = 3.10.14
Datasets = 2.19
GitHub issue: Support the deserialization of json lines files comprised of lists
Author: umarbutler (NONE)
State: open | Created: 2024-05-18 05:07:23
Labels: name: enhancement

### Feature request

I manage a somewhat large and popular Hugging Face dataset known as the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus). I recently updated my corpus to be stored in a json lines file where each line is an array and each element represents a value at a particular column. Previously, my corpus was stored as a json lines file where each line was a dictionary and the keys were the fields.

Essentially, a line in my json lines file used to look like this:
```json
{"version_id":"","type":"","jurisdiction":"","source":"","citation":"","url":"","when_scraped":"","text":""}
```

And now it looks like this:
```json
["","","","","","","",""]
```

This saves 65 bytes per document and allows me very quickly serialise and deserialise documents via `msgspec`.

After making this change, I found that `datasets` was incapable of deserialising my Corpus without a custom loading script, even if I ensured that the `dataset_info` field in my dataset card contained the desired names of my features.

I would like to request that functionality be added to support this format which is more memory-efficent and faster than using dictionaries.

### Motivation

The [documentation](https://huggingface.co/docs/datasets/en/dataset_script) for creating dataset loading scripts asserts that:
> In the next major release, the new safety features of 🤗 Datasets will disable running dataset loading scripts by default, and you will have to pass trust_remote_code=True to load datasets that require running a dataset script.

I would rather not require my users to pass `trust_remote_code=True` which means that I will need built-in support for this format.

### Your contribution

I would be happy to submit a PR for this if this is something you would incorporate into `datasets` and if I can be pointed to where the code would need to go.
GitHub issue: irc_disentangle - Issue with splitting data
Author: eor51355 (NONE)
State: closed | Created: 2024-05-17 23:19:37

### Describe the bug

I am trying to access your database through python using "datasets.load_dataset("irc_disentangle")" and I am getting this error message:

ValueError: Instruction "train" corresponds to no data!

### Steps to reproduce the bug

import datasets
ds = datasets.load_dataset('irc_disentangle')
ds

### Expected behavior

The data is supposed to load into ds and be accessable as such:
ds['train'][1050], ds['train'][1055]

### Environment info

I tired Python 3.12 and 3.10
GitHub issue: Extraction protocol for arrow files is not defined
Author: radulescupetru (NONE)
State: open | Created: 2024-05-17 16:01:41

### Describe the bug

Passing files with `.arrow` extension into data_files argument, at least when `streaming=True` is very slow.

### Steps to reproduce the bug

 Basically it goes through the `_get_extraction_protocol` method located [here](https://github.com/huggingface/datasets/blob/main/src/datasets/utils/file_utils.py#L820)
The method then looks at some base known extensions where `arrow` is not defined so it proceeds to determine the compression with the magic number method which is slow when dealing with a lot of files which are stored in s3 and by looking at this predefined list, I don't see `arrow` in there either so in the end it return None:

```
MAGIC_NUMBER_TO_COMPRESSION_PROTOCOL = {
    bytes.fromhex("504B0304"): "zip",
    bytes.fromhex("504B0506"): "zip",  # empty archive
    bytes.fromhex("504B0708"): "zip",  # spanned archive
    bytes.fromhex("425A68"): "bz2",
    bytes.fromhex("1F8B"): "gzip",
    bytes.fromhex("FD377A585A00"): "xz",
    bytes.fromhex("04224D18"): "lz4",
    bytes.fromhex("28B52FFD"): "zstd",
}
```

### Expected behavior

My expectation is that `arrow` would be in the known lists so it would return None without going through the magic number method.

### Environment info

datasets 2.19.0
GitHub issue: Add the option of saving in parquet instead of arrow 
Author: arita37 (NONE)
State: open | Created: 2024-05-16 13:35:51
Labels: name: enhancement

### Feature request

In dataset.save_to_disk('/path/to/save/dataset'),

add the option to save in parquet format

dataset.save_to_disk('/path/to/save/dataset', format="parquet"),

because arrow is not used for Production Big data.... (only parquet)



### Motivation

because arrow is not used for Production Big data.... (only parquet)

### Your contribution

I can do the testing !
GitHub issue: Make CLI convert_to_parquet not raise error if no rights to create script branch
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-16 12:21:27

Make CLI convert_to_parquet not raise error if no rights to create "script" branch.

Not that before this PR, the error was not critical because it was raised at the end of the script, once all the rest of the steps were already performed.

Fix #6901.

Bug introduced in datasets-2.19.0 by:
- #6809
GitHub issue: HTTPError 403 raised by CLI convert_to_parquet when creating script branch on 3rd party repos
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-16 11:40:22
Labels: name: bug

CLI convert_to_parquet cannot create "script" branch on 3rd party repos.

It can only create it on repos where the user executing the script has write access.

Otherwise, a 403 Forbidden HTTPError is raised:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/ORG/DATASET/branch/script

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/datasets_cli.py", line 41, in main
    service.run()
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/convert_to_parquet.py", line 92, in run
    create_branch(dataset_id, branch="script", repo_type="dataset", token=token, exist_ok=True)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/hf_api.py", line 5503, in create_branch
    hf_raise_for_status(response)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 367, in hf_raise_for_status
    raise HfHubHTTPError(message, response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError:  (Request ID: Root=1-6645ee0d-4db1ed8a1fbe04956be15897;139a6e23-df7d-4f62-b5ba-adb6d8e6e696)

403 Forbidden: Forbidden: cannot write to script.
Cannot access content at: https://huggingface.co/api/datasets/ORG/DATASET/branch/script.
If you are trying to create or update content,make sure you have a token with the `write` role.
```
GitHub issue: [WebDataset] KeyError with user-defined `Features` when a field is missing in an example
Author: lhoestq (MEMBER)
State: closed | Created: 2024-05-15 17:48:34

reported at https://huggingface.co/datasets/ProGamerGov/synthetic-dataset-1m-dalle3-high-quality-captions/discussions/1

```
File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 109, in _generate_examples
    example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
```
GitHub issue: List of dictionary features get standardized
Author: sohamparikh (NONE)
State: open | Created: 2024-05-15 14:11:35

### Describe the bug

Hi, i’m trying to create a HF dataset from a list using Dataset.from_list.

Each sample in the list is a dict with the same keys (which will be my features). The values for each feature are a list of dictionaries, and each such dictionary has a different set of keys. However, the datasets library standardizes all dictionaries under a feature and adds all possible keys (with None value) from all the dictionaries under that feature.

How can I keep the same set of keys as in the original list for each dictionary under a feature?

### Steps to reproduce the bug

```
from datasets import Dataset

# Define a function to generate a sample with "tools" feature
def generate_sample():
    # Generate random sample data
    sample_data = {
        "text": "Sample text",
        "feature_1": []
    }
    
    # Add feature_1 with random keys for this sample
    feature_1 = [{"key1": "value1"}, {"key2": "value2"}]  # Example feature_1 with random keys
    sample_data["feature_1"].extend(feature_1)
    
    return sample_data

# Generate multiple samples
num_samples = 10
samples = [generate_sample() for _ in range(num_samples)]

# Create a Hugging Face Dataset
dataset = Dataset.from_list(samples)
dataset[0]
```

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1', 'key2': None}, {'key1': None, 'key2': 'value2'}]}```

### Expected behavior

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1'}, {'key2': 'value2'}]}```

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-5.15.0-1040-nvidia-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.0
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
GitHub issue: Fix YAML error in README files appearing on GitHub
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-14 05:21:57

Fix YAML error in README files appearing on GitHub.

See error message:
![Screenshot from 2024-05-14 06-58-02](https://github.com/huggingface/datasets/assets/8515462/7984cc4e-96ee-4e83-99a4-4c0c5791fa05)


Fix #6897.
GitHub issue: datasets template guide :: issue in documentation YAML
Author: bghira (NONE)
State: closed | Created: 2024-05-13 17:33:59

### Describe the bug

There is a YAML error at the top of the page, and I don't think it's supposed to be there 

### Steps to reproduce the bug

1. Browse to [this tutorial document](https://github.com/huggingface/datasets/blob/main/templates/README_guide.md)
2. Observe a big red error at the top
3. The rest of the document remains functional

### Expected behavior

I think the YAML block should be displayed or ignored.

### Environment info

N/A
GitHub issue: Add support for categorical/dictionary types
Author: EthanSteinberg (CONTRIBUTOR)
State: closed | Created: 2024-05-12 07:15:08

Arrow has a very useful dictionary/categorical type (https://arrow.apache.org/docs/python/generated/pyarrow.dictionary.html). This data type has significant speed, memory and disk benefits over pa.string() when there are only a few unique text strings in a column.

Unfortunately, huggingface datasets currently does not support this type. So huggingface datasets cannot natively read many parquet files that use this datatype .This PR adds support for Huggingface Datasets to read categorical/dictionary data.

Note: This PR functions by simply converting those dictionary/categorical types to strings. This means that huggingface datasets cannot take advantage of the compute benefits of categoricals, but it significantly simplifies logic. At this time, I do not think it makes sense to optimize categorical support within huggingface datasets and that we should only try to optimize later, if necessary.

Closes #5706
GitHub issue: Unable to load JSON saved using `to_json`
Author: DarshanDeshpande (NONE)
State: closed | Created: 2024-05-12 01:02:51

### Describe the bug

Datasets stored in the JSON format cannot be loaded using `json.load()`

### Steps to reproduce the bug

```
import json
from datasets import load_dataset

dataset = load_dataset("squad")
train_dataset, test_dataset = dataset["train"], dataset["validation"]
test_dataset.to_json("full_dataset.json")

# This works
loaded_test = load_dataset("json", data_files="full_dataset.json")

# This fails
loaded_test = json.load(open("full_dataset.json", "r"))
```

### Expected behavior

The JSON should be correctly formatted when writing so that it can be loaded using `json.load()`.

### Environment info

Colab: https://colab.research.google.com/drive/1st1iStFUVgu9ZPvnzSzL4vDeYWDwYpUm?usp=sharing
GitHub issue: add `with_transform` and/or `set_transform` to IterableDataset
Author: not-lain (NONE)
State: open | Created: 2024-05-10 01:00:12
Labels: name: enhancement

### Feature request

when working with a really large dataset it would save us a lot of time (and compute resources) to use either with_transform or the set_transform from the Dataset class instead of waiting for the entire dataset to map

### Motivation

don't want to wait for a really long dataset to map, this would give IterableDataset an extra advantage over the Dataset class.
reducing time and resources

### Your contribution

I am a little busy with my job search lately, but would post about this feature in my social media.
Apologies again (dad going to kick me out soon), if I ever have some free time I will contribute to making this a reality, but that's going to be hard
      / (┬┬﹏┬┬)\
GitHub issue: fix bug #6877
Author: arthasking123 (NONE)
State: closed | Created: 2024-05-09 13:38:40

fix bug #6877 due to maybe f becomes invaild after yield process
the results are below:


Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:01<00:00, 420.41it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 26148.48it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 409731.44it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 289720.84it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 26663.42it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 434056.21it/s]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 13222.33files/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:04<00:00, 180.67files/s]
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [01:35<00:00,  8.70files/s]
Generating train split: 1571592 examples [00:08, 176736.09 examples/s]
Generating test split: 85533 examples [00:01, 48224.56 examples/s]
Generating validation split: 86246 examples [00:01, 50164.16 examples/s]

Fix https://github.com/huggingface/datasets/issues/6877.

CC: @natolambert
GitHub issue: FAISS load to None
Author: brainer3220 (NONE)
State: open | Created: 2024-05-09 02:43:50

### Describe the bug

I've use FAISS with Datasets and save to FAISS.

Then load to save FAISS then no error, then ds to None

```python
ds.load_faiss_index('embeddings', 'my_index.faiss')
```

### Steps to reproduce the bug

# 1.
```python
ds_with_embeddings = ds.map(lambda example: {'embeddings': model(transforms(example['image']).unsqueeze(0)).squeeze()}, batch_size=64)

ds_with_embeddings.add_faiss_index(column='embeddings')

ds_with_embeddings.save_faiss_index('embeddings', 'index.faiss')
```

# 2.
```python
ds.load_faiss_index('embeddings', 'my_index.faiss')
```

### Expected behavior

Add column in Datasets.

### Environment info

Google Colab, SageMaker Notebook
GitHub issue: load_dataset with data_dir and cache_dir set fail with not supported
Author: fah (NONE)
State: open | Created: 2024-05-08 19:52:35

### Describe the bug

with python 3.11 I execute:
```py
from transformers import Wav2Vec2Processor, Data2VecAudioModel
import torch
from torch import nn
from datasets import load_dataset, concatenate_datasets

# load demo audio and set processor
dataset_clean = load_dataset("librispeech_asr", "clean", split="validation", data_dir="data", cache_dir="cache")
```
This fails in the last line with 
```log
Found cached dataset librispeech_asr (file:///Users/as/Documents/Project/git/audio2vec/cache/librispeech_asr/clean-data_dir=data/2.1.0/cff5df6e7955c80a67f80e27e7e655de71c689e2d2364bece785b972acb37fe7)
Traceback (most recent call last):
  File "/Users/as/Documents/Project/git/audio2vec/src/music2vec-v1.py", line 7, in <module>
    dataset_clean = load_dataset("librispeech_asr", "clean", split="validation", data_dir="data", cache_dir="cache")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/as/anaconda3/lib/python3.11/site-packages/datasets/load.py", line 1810, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/as/anaconda3/lib/python3.11/site-packages/datasets/builder.py", line 1113, in as_dataset
    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
```

### Steps to reproduce the bug

I setup an venv with requirements.txt
```txt
transformers==4.40.2
torch==2.2.2
datasets==2.16.0
fsspec==2023.9.2
```

pip freeze is:
```
aiohttp==3.9.5
aiosignal==1.3.1
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.16.0
dill==0.3.7
filelock==3.14.0
frozenlist==1.4.1
fsspec==2023.9.2
huggingface-hub==0.23.0
idna==3.7
Jinja2==3.1.4
MarkupSafe==2.1.5
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.15
networkx==3.3
numpy==1.26.4
packaging==24.0
pandas==2.2.2
pyarrow==16.0.0
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
regex==2024.4.28
requests==2.31.0
safetensors==0.4.3
six==1.16.0
sympy==1.12
tokenizers==0.19.1
torch==2.2.2
tqdm==4.66.4
transformers==4.40.2
typing_extensions==4.11.0
tzdata==2024.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4

```

I execute this on a M1 Mac.


### Expected behavior

I don't understand the error message. Why is "local" caching not supported. Would it possible to give some additional hint with the error message how to solve this issue?


### Environment info

source ....
python -u example.py
GitHub issue: CI is broken after jax-0.4.27 release: AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-08 07:01:47
Labels: name: bug

After jax-0.4.27 release (https://github.com/google/jax/releases/tag/jax-v0.4.27), our CI is broken with the error:
```Python traceback
AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'. Did you mean: 'devices'?
```

See: https://github.com/huggingface/datasets/actions/runs/8997488610/job/24715736153
```Python traceback
___________________ FormatterTest.test_jax_formatter_device ____________________
[gw1] linux -- Python 3.10.14 /opt/hostedtoolcache/Python/3.10.14/x64/bin/python

self = <tests.test_formatting.FormatterTest testMethod=test_jax_formatter_device>

    @require_jax
    def test_jax_formatter_device(self):
        import jax
    
        from datasets.formatting import JaxFormatter
    
        pa_table = self._create_dummy_table()
        device = jax.devices()[0]
        formatter = JaxFormatter(device=str(device))
        row = formatter.format_row(pa_table)
>       assert row["a"].device() == device
E       AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'. Did you mean: 'devices'?

tests/test_formatting.py:630: AttributeError
```
GitHub issue: Require Pillow >= 9.4.0 to avoid AttributeError when loading image dataset
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-08 06:43:29

Require Pillow >= 9.4.0 to avoid AttributeError when loading image dataset.

The `PIL.Image.ExifTags` that we use in our code was implemented in Pillow-9.4.0: https://github.com/python-pillow/Pillow/commit/24a5405a9f7ea22f28f9c98b3e407292ea5ee1d3

The bug #6881 was introduced in datasets-2.19.0 by this PR:
- #6739

Fix #6881.
GitHub issue: AttributeError: module 'PIL.Image' has no attribute 'ExifTags'
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-08 06:33:57
Labels: name: bug

When trying to load an image dataset in an old Python environment (with Pillow-8.4.0), an error is raised:
```Python traceback
AttributeError: module 'PIL.Image' has no attribute 'ExifTags'
```
The error traceback:
```Python traceback
~/huggingface/datasets/src/datasets/iterable_dataset.py in __iter__(self)
   1391                 # `IterableDataset` automatically fills missing columns with None.
   1392                 # This is done with `_apply_feature_types_on_example`.
-> 1393                 example = _apply_feature_types_on_example(
   1394                     example, self.features, token_per_repo_id=self._token_per_repo_id
   1395                 )

~/huggingface/datasets/src/datasets/iterable_dataset.py in _apply_feature_types_on_example(example, features, token_per_repo_id)
   1080     encoded_example = features.encode_example(example)
   1081     # Decode example for Audio feature, e.g.
-> 1082     decoded_example = features.decode_example(encoded_example, token_per_repo_id=token_per_repo_id)
   1083     return decoded_example
   1084 

~/huggingface/datasets/src/datasets/features/features.py in decode_example(self, example, token_per_repo_id)
   1974 
-> 1975         return {
   1976             column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
   1977             if self._column_requires_decoding[column_name]

~/huggingface/datasets/src/datasets/features/features.py in <dictcomp>(.0)
   1974 
   1975         return {
-> 1976             column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
   1977             if self._column_requires_decoding[column_name]
   1978             else value

~/huggingface/datasets/src/datasets/features/features.py in decode_nested_example(schema, obj, token_per_repo_id)
   1339         # we pass the token to read and decode files from private repositories in streaming mode
   1340         if obj is not None and schema.decode:
-> 1341             return schema.decode_example(obj, token_per_repo_id=token_per_repo_id)
   1342     return obj
   1343 

~/huggingface/datasets/src/datasets/features/image.py in decode_example(self, value, token_per_repo_id)
    187             image = PIL.Image.open(BytesIO(bytes_))
    188         image.load()  # to avoid "Too many open files" errors
--> 189         if image.getexif().get(PIL.Image.ExifTags.Base.Orientation) is not None:
    190             image = PIL.ImageOps.exif_transpose(image)
    191         if self.mode and self.mode != image.mode:

~/huggingface/datasets/venv/lib/python3.9/site-packages/PIL/Image.py in __getattr__(name)
     75                 )
     76                 return categories[name]
---> 77         raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
     78 
     79 

AttributeError: module 'PIL.Image' has no attribute 'ExifTags'
```

### Environment info

Since datasets 2.19.0
GitHub issue: Batched mapping does not raise an error if values for an existing column are empty
Author: felix-schneider (NONE)
State: open | Created: 2024-05-07 11:02:40

### Describe the bug

Using `Dataset.map(fn, batched=True)` allows resizing the dataset by returning a dict of lists, all of which must be the same size. If they are not the same size, an error like `pyarrow.lib.ArrowInvalid: Column 1 named x expected length 1 but got length 0` is raised.

This is not the case if the function returns an empty list for an existing column in the dataset. In that case, the dataset is silently resized to 0 rows.

### Steps to reproduce the bug

MWE:

```
import datasets
data = datasets.Dataset.from_dict({"test": [1]})

def mapping_fn(examples):
    return {"test": [], "y": [1]}

data = data.map(mapping_fn, batched=True)
print(len(data))
```

Note  that when returning `"x": []`, the error is raised correctly, also when returning `"test": [1,2]`.

### Expected behavior

Expected an exception: `pyarrow.lib.ArrowInvalid: Column 1 named test expected length 1 but got length 0` or `pyarrow.lib.ArrowInvalid: Column 2 named y expected length 0 but got length 1`.

Any exception would be acceptable.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-5.4.0-153-generic-x86_64-with-glibc2.31
- Python version: 3.11.8
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
GitHub issue: Create function to convert to parquet
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-07 10:27:07

Analogously with `delete_from_hub`, this PR:
- creates the Python function `convert_to_parquet`
- makes the corresponding CLI command use that function.

This way, the functionality can be used both from a terminal and from a Python console.

This PR also implements a test for convert_to_parquet function.
GitHub issue: Unpin hfh
Author: lhoestq (MEMBER)
State: closed | Created: 2024-05-06 18:10:49

Needed to use those in dataset-viewer:

- dev version of hfh https://github.com/huggingface/dataset-viewer/pull/2781: don't span the hub with /paths-info requests
- dev version of datasets at https://github.com/huggingface/datasets/pull/6875: don't write too big logs in the viewer

close https://github.com/huggingface/datasets/issues/6863
GitHub issue: Use pandas ujson in JSON loader to improve performance
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-06 12:01:27

Use pandas ujson in JSON loader to improve performance.

Note that `datasets` has `pandas` as required dependency. And `pandas` includes `ujson` in `pd.io.json.ujson_loads`.

Fix #6867.

CC: @natolambert
GitHub issue: datasets.BuilderConfig does not work.
Author: jdm4pku (NONE)
State: closed | Created: 2024-05-05 08:08:55

### Describe the bug

I custom a BuilderConfig and GeneratorBasedBuilder.
Here is the code for BuilderConfig
```
class UIEConfig(datasets.BuilderConfig):

    def __init__(
            self,
            *args,
            data_dir=None,
            instruction_file=None,
            instruction_strategy=None,
            task_config_dir=None,
            num_examples=None,
            max_num_instances_per_task=None,
            max_num_instances_per_eval_task=None,
            over_sampling=None,
            **kwargs
    ):
        super().__init__(*args, **kwargs)
        self.data_dir = data_dir
        self.num_examples = num_examples
        self.over_sampling = over_sampling
        self.instructions = self._parse_instruction(instruction_file)
        self.task_configs = self._parse_task_config(task_config_dir)
        self.instruction_strategy = instruction_strategy
        self.max_num_instances_per_task = max_num_instances_per_task
        self.max_num_instances_per_eval_task = max_num_instances_per_eval_task
```
Besides, here is the code for GeneratorBasedBuilder.
```
class UIEInstructions(datasets.GeneratorBasedBuilder):
    VERSION = datasets.Version("2.0.0")
    BUILDER_CONFIG_CLASS = UIEConfig
    BUILDER_CONFIGS = [
        UIEConfig(name="default", description="Default config for NaturalInstructions")
    ]
    DEFAULT_CONFIG_NAME = "default"
```

Here is the load_dataset
```
raw_datasets = load_dataset(
        os.path.join(CURRENT_DIR, "uie_dataset.py"),
        data_dir=data_args.data_dir,
        task_config_dir=data_args.task_config_dir,
        instruction_file=data_args.instruction_file,
        instruction_strategy=data_args.instruction_strategy,
        cache_dir=data_cache_dir,  # for debug, change dataset size, otherwise open it
        max_num_instances_per_task=data_args.max_num_instances_per_task,
        max_num_instances_per_eval_task=data_args.max_num_instances_per_eval_task,
        num_examples=data_args.num_examples,
        over_sampling=data_args.over_sampling
    )
```
Finally, I met the error.
```
BuilderConfig UIEConfig(name='default', version=0.0.0, data_dir=None, data_files=None, description='Default config for NaturalInstructions') doesn't have a 'task_config_dir' key.
```

I debugged the code, but I find the parameters added by me may not work.

### Steps to reproduce the bug

https://github.com/BeyonderXX/InstructUIE/blob/master/src/uie_dataset.py

### Expected behavior

```
BuilderConfig UIEConfig(name='default', version=0.0.0, data_dir=None, data_files=None, description='Default config for NaturalInstructions') doesn't have a 'task_config_dir' key.
```

### Environment info

torch                     2.3.0+cu118
transformers              4.40.1
python 3.8
GitHub issue: Improve performance of JSON loader
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-04 15:04:16
Labels: name: enhancement

As reported by @natolambert, loading regular JSON files with `datasets` shows poor performance.

The cause is that we use the `json` Python standard library instead of other faster libraries. See my old comment: https://github.com/huggingface/datasets/pull/2638#pullrequestreview-706983714
> There are benchmarks that compare different JSON packages, with the Standard Library one among the worst performant: 
> - https://github.com/ultrajson/ultrajson#benchmarks
> - https://github.com/ijl/orjson#performance

I remember having a discussion about this and it was decided that it was better not to include an additional dependency on a 3rd-party library.

However:
- We already depend on `pandas` and `pandas` depends on `ujson`: so we have an indirect dependency on `ujson`
- Even if the above were not the case, we always could include `ujson` as an optional extra dependency, and check at runtime if it is installed to decide which library to use, either json or ujson
GitHub issue: Example on Semantic segmentation contains bug
Author: ducha-aiki (NONE)
State: open | Created: 2024-05-03 09:40:12

### Describe the bug

https://huggingface.co/docs/datasets/en/semantic_segmentation shows wrong example with torchvision transforms.
Specifically, as one can see in screenshot below, the object boundaries have weird colors. 

<img width="689" alt="image" src="https://github.com/huggingface/datasets/assets/4803565/59aa0e2c-2e3e-415b-9d42-2314044c5aee">


Original example with `albumentations` is correct 
<img width="705" alt="image" src="https://github.com/huggingface/datasets/assets/4803565/27dbd725-cea5-4e48-ba59-7050c3ce17b3">


That is because `torch vision.transforms.Resize` interpolates with bilinear everything which is wrong when used for segmentation labels - you just cannot mix them. Overall, `torchvision.transforms` is designed for classification only and cannot be used to images and masks together, unless you write two separate branches of augmentations.


The correct way would be to use `v2` version of transforms and convert the segmentation labels to https://pytorch.org/vision/main/generated/torchvision.tv_tensors.Mask.html#torchvision.tv_tensors.Mask object




### Steps to reproduce the bug

Go to the website.
<img width="689" alt="image" src="https://github.com/huggingface/datasets/assets/4803565/ea1276d0-d69a-48cf-b9c2-cd61217815ef">

https://huggingface.co/docs/datasets/en/semantic_segmentation

### Expected behavior

Results, similar to `albumentation`. Or remove the torch vision part altogether. Or use `kornia` instead.

### Environment info

Irrelevant
GitHub issue: Dataset 'rewardsignal/reddit_writing_prompts' doesn't exist on the Hub
Author: vinodrajendran001 (NONE)
State: closed | Created: 2024-05-03 06:03:30

### Describe the bug

The dataset `rewardsignal/reddit_writing_prompts` is missing in Huggingface Hub.

### Steps to reproduce the bug

```
from datasets import load_dataset

prompt_response_dataset = load_dataset("rewardsignal/reddit_writing_prompts", data_files="prompt_responses_full.csv", split='train[:80%]')
```

### Expected behavior

DatasetNotFoundError: Dataset 'rewardsignal/reddit_writing_prompts' doesn't exist on the Hub or cannot be accessed

### Environment info

Nothing to do with versions
GitHub issue: Fix load_dataset for data_files with protocols other than HF
Author: matstrand (CONTRIBUTOR)
State: closed | Created: 2024-05-03 01:43:47

Fixes huggingface/datasets/issues/6598

I've added a new test case and a solution. Before applying the solution the test case was failing with the same error described in the linked issue.

MRE:
```
pip install "datasets[s3]"
python -c "from datasets import load_dataset; load_dataset('csv', data_files={'train': 's3://noaa-gsod-pds/2024/A5125600451.csv'})"
```
GitHub issue: Fix CI by temporarily pinning huggingface-hub < 0.23.0
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-02 16:40:04

As a hotfix for CI, temporarily pin `huggingface-hub` upper version

Fix #6860.

Revert once root cause is fixed, see:
- https://github.com/huggingface/transformers/issues/30618
GitHub issue: CI fails after huggingface_hub-0.23.0 release: FutureWarning: "resume_download"
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-02 13:24:17
Labels: name: bug

CI fails after latest huggingface_hub-0.23.0 release: https://github.com/huggingface/huggingface_hub/releases/tag/v0.23.0

```
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_bertscore - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_frugalscore - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_perplexity - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer_with_cache - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_arrow_dataset.py::MiscellaneousDatasetTest::test_set_format_encode - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
```
GitHub issue: Support folder-based datasets with large metadata.jsonl
Author: gbenson (NONE)
State: open | Created: 2024-05-02 09:07:26

I tried creating an `imagefolder` dataset with a 714MB `metadata.jsonl` but got the error below.  This pull request fixes the problem by increasing the block size like the message suggests.
```
>>> from datasets import load_dataset
>>> dataset = load_dataset("imagefolder", data_dir="data-for-upload")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
      ...
  File "/path/to/datasets/packaged_modules/folder_based_builder/folder_based_builder.py", line 245, in _read_metadata
    return paj.read_json(f)
  File "pyarrow/_json.pyx", line 308, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```
GitHub issue: Segmentation fault
Author: scampion (NONE)
State: closed | Created: 2024-05-02 08:28:49

### Describe the bug

Using various version for datasets, I'm no more longer able to load that dataset without a segmentation fault. 
Several others files are also concerned. 

### Steps to reproduce the bug

# Create a new venv 
python3 -m venv venv_test 
source venv_test/bin/activate

# Install the latest version 
pip install datasets

# Load that dataset
python3 -q -X faulthandler -c "from datasets import load_dataset;  load_dataset('EuropeanParliament/Eurovoc', '1998-09')"

### Expected behavior

Data must be loaded 

### Environment info

datasets==2.19.0
Python 3.11.7
Darwin  22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-8796.121.2~5/RELEASE_X86_64 x86_64
GitHub issue: Fix line-endings in tests on Windows
Author: albertvillanova (MEMBER)
State: closed | Created: 2024-05-02 07:49:15

EDIT:
~~Fix test_delete_from_hub on Windows by passing explicit encoding.~~
Fix test_delete_from_hub and test_xgetsize_private by uploading the README file content directly (encoding the string), instead of writing a local file and uploading it.

Note that local files created on Windows will have "\r\n" line endings, instead of "\n".
These are no longer transformed to "\n" by the Hub.

Fix #6856.