KRadim/github-issues-minimal huggingface dataset:
ImageFolder dataset builder does not read the validation data set if it is named as "val"
State: closed
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**

Currently, the `'imagefolder'` data set builder in [`load_dataset()`](https://github.com/huggingface/datasets/blob/2.4.0/src/datasets/load.py#L1541] ) only [supports](https://github.com/huggingface/datasets/blob/6c609a322da994de149b2c938f19439bca99408e/src/datasets/data_files.py#L31) the following names as the validation data set directory name: `["validation", "valid", "dev"]`. When the validation directory is named as `'val'`, the Data set will not have a validation split. I expected this to be a trivial task but ended up spending a lot of time before knowing that only the above names are supported.

Here's a minimal example of `val` not being recognized:

```python
import os
import numpy as np
import cv2   
from datasets import load_dataset

# creating a dummy data set with the following structure:

# ROOT
# | -- train
# | ---- class_1
# | ---- class_2
# | -- val
# | ---- class_1
# | ---- class_2


ROOT = "data"


for which in ["train", "val"]:
  for class_name in ["class_1", "class_2"]:
    dir_name = os.path.join(ROOT, which, class_name)
    if not os.path.exists(dir_name):
      os.makedirs(dir_name)
    for i in range(10):
       cv2.imwrite(
           os.path.join(dir_name, f"{i}.png"),
           np.random.random((224, 224))
           )

# trying to create a data set
dataset = load_dataset(
    "imagefolder", 
    data_dir=ROOT
)

>> dataset
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 20
    })
})

# ^ note how the dataset only has a 'train' subset

```

**Describe the solution you'd like**

The suggestion is to include `"val"` to  [that list ](https://github.com/huggingface/datasets/blob/6c609a322da994de149b2c938f19439bca99408e/src/datasets/data_files.py#L31) as that's a commonly used phrase to name the validation directory. 

Also, In the documentation, explicitly mention that only such directory names are supported as train/val/test directories to avoid confusion.

**Describe alternatives you've considered**

In the documentation, explicitly mention that only such directory names are supported as train/val/test directories without adding `val` to the above list.


**Additional context**

A question asked in the forum: [
Loading an imagenet-style image dataset with train/val directories](https://discuss.huggingface.co/t/loading-an-imagenet-style-image-dataset-with-train-val-directories/21554)
Is it possible to pass multiple links to a split in load script?
State: open
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
I wanted to use a python loading script in hugging face datasets that use different sources of text (it's somehow a compilation of multiple datasets + my own dataset) based on how `load_dataset` [works](https://huggingface.co/docs/datasets/loading) I assumed I could do something like bellow in my loading script:

```python
...
_URL = "MY_DATASET_URL/resolve/main/data/"
_URLS = {
    "train": [
        "FIRST_URL_TO.txt",
        _URL + "train-00000-of-00001-676bfebbc8742592.parquet"
    ]
}
...
```
but when loading the dataset it raises the following error:
```python
File ~/.local/lib/python3.8/site-packages/datasets/builder.py:704, in DatasetBuilder.download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702         logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703 if not downloaded_from_gcs:
--> 704     self._download_and_prepare(
    705         dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
...
    668     if isinstance(a, str):
    669         # Force-cast str subclasses to str (issue #21127)
    670         parts.append(str(a))

TypeError: expected str, bytes or os.PathLike object, not list
```

**Describe the solution you'd like**
I believe since it's possible for `load_dataset` to get list of URLs instead of just a URL for `train` split it can be possible here too.

**Describe alternatives you've considered**
An alternative solution would be to download all needed datasets locally and `push_to_hub` them all, but since the datasets I'm talking about are huge it's not among my options. 

**Additional context**
I think loading `text` beside the `parquet` is completely a different issue but I believe I can figure it out by proposing a config for my dataset to load each entry of `_URLS['train']` separately either by `load_dataset("text", ...` or `load_dataset("parquet", ...`.
Fix missing tags in dataset cards
State: closed

Fix missing tags in dataset cards:
- boolq
- break_data
- definite_pronoun_resolution
- emo
- kor_nli
- pg19
- quartz
- sciq
- squad_es
- wmt14
- wmt15
- wmt16
- wmt17
- wmt18
- wmt19
- wmt_t2t

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.
Misalignment between card tag validation and docs
State: open
Labels: name: bug

## Describe the bug
As pointed out in other issue: https://github.com/huggingface/datasets/pull/4827#discussion_r943536284
the validation of the dataset card tags is not aligned with its documentation: e.g.
- implementation: `license: List[str]`
- docs: `license: Union[str, List[str]]`

They should be aligned.

CC: @julien-c
Support PIL Image objects in `add_item`/`add_column`
State: open

Fix #4796 

PS: We should also improve the type inference in `OptimizedTypeSequence` to make it possible to also infer the complex types (only `Image` currently) in nested arrays (e.g.  `[[pil_image], [pil_image, pil_image]]` or `[{"img": pil_image}`]), but I plan to address this in a separate PR.
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
State: closed
Labels: name: bug

Hi, when i try to run prepare_dataset function in [fine tuning ASR tutorial 4](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tuning_Wav2Vec2_for_English_ASR.ipynb) , i got this error.
I got this error
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
There is no other logs available, so i have no clue what is the cause of it.
```

def prepare_dataset(batch):
    audio = batch["path"]
    # batched output is "un-batched"
    batch["input_values"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
    batch["input_length"] = len(batch["input_values"])
    with processor.as_target_processor():
        batch["labels"] = processor(batch["text"]).input_ids
    return batch

data = data.map(prepare_dataset, remove_columns=data.column_names["train"],
                      num_proc=4)
```


Specify the actual results or traceback.
There is no traceback except
`Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.`
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-5.15.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.4.3
Fix loading example in opus dataset cards
State: closed

This PR:
- fixes the examples to load the datasets, with the corrected dataset name, in their dataset cards for:
  - opus_dgt
  - opus_paracrawl
  - opus_wikipedia
- fixes their dataset cards with the missing required information: title, data instances/fields/splits
- enumerates the supported languages
- adds a missing citation reference for opus_wikipedia

Related to:
- #4806
Fix bug in function validate_type for Python >= 3.9
State: closed

Fix `validate_type` function, so that it uses `get_origin` instead. This makes the function forward compatible.

This fixes #4811 because:
```python
In [4]: typing.Optional[str]
Out[4]: typing.Optional[str]

In [5]: get_origin(typing.Optional[str])
Out[5]: typing.Union
```

Fix #4811.
Bug in function validate_type for Python >= 3.9
State: closed
Labels: name: bug

## Describe the bug
The function `validate_type` assumes that the type `typing.Optional[str]` is automatically transformed to `typing.Union[str, NoneType]`.
```python
In [4]: typing.Optional[str]
Out[4]: typing.Union[str, NoneType]
```

However, this is not the case for Python 3.9:
```python
In [3]: typing.Optional[str]
Out[3]: typing.Optional[str]
```
Wrong example in opus_gnome dataset card
State: closed
Labels: name: bug

## Describe the bug
I found that [the example on opus_gone dataset ](https://github.com/huggingface/datasets/tree/main/datasets/opus_gnome#dataset-summary) doesn't work.

## Steps to reproduce the bug
```python
 load_dataset("gnome", lang1="it", lang2="pl")
```
`"gnome"` should be `"opus_gnome"`

## Expected results
```bash
100%
1/1 [00:00<00:00, 42.09it/s]
DatasetDict({
    train: Dataset({
        features: ['id', 'translation'],
        num_rows: 8368
    })
})
```

## Actual results
```bash
 Couldn't find 'gnome' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/main/datasets/gnome/gnome.py
```

## Environment info
- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-120-generic-x86_64-with-glibc2.27
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3
streaming dataset with concatenating splits raises an error
State: open
Labels: name: bug

## Describe the bug
streaming dataset with concatenating splits raises an error

## Steps to reproduce the bug
```python
from datasets import load_dataset

# no error
repo = "nateraw/ade20k-tiny"
dataset = load_dataset(repo, split="train+validation")
```

```python
from datasets import load_dataset

# error
repo = "nateraw/ade20k-tiny"
dataset = load_dataset(repo, split="train+validation", streaming=True)
```

```sh
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-4-a6ae02d63899>](https://localhost:8080/#) in <module>()
      3 # error
      4 repo = "nateraw/ade20k-tiny"
----> 5 dataset = load_dataset(repo, split="train+validation", streaming=True)

1 frames
[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in as_streaming_dataset(self, split, base_path)
   1030             splits_generator = splits_generators[split]
   1031         else:
-> 1032             raise ValueError(f"Bad split: {split}. Available splits: {list(splits_generators)}")
   1033 
   1034         # Create a dataset for each of the given splits

ValueError: Bad split: train+validation. Available splits: ['validation', 'train']
```

[Colab](https://colab.research.google.com/drive/1wMj08_0bym9jnGgByib4lsBPu8NCZBG9?usp=sharing)

## Expected results
load successfully or throws an error saying it is not supported.

## Actual results
above

## Environment info
- `datasets` version: 2.4.0
- Platform: Windows-10-10.0.22000-SP0 (windows11 x64)
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
Support `pipeline` argument in inspect.py functions
State: open
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**

The `wikipedia` dataset requires a `pipeline` argument to build the list of splits:

https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py#L937

But this is currently not supported in `get_dataset_config_info`:

https://github.com/huggingface/datasets/blob/main/src/datasets/inspect.py#L373-L375

which is called by other functions, e.g. `get_dataset_split_names`.

**Additional context**

The dataset viewer is not working out-of-the-box on `wikipedia` for this reason:

https://huggingface.co/datasets/wikipedia/viewer

<img width="637" alt="Capture d’écran 2022-08-08 à 12 01 16" src="https://user-images.githubusercontent.com/1676121/183461838-5330783b-0269-4ba7-a999-314cde2023d8.png">
`with_format` behavior is inconsistent on different datasets
State: open
Labels: name: bug

## Describe the bug
I found a case where `with_format` does not transform the dataset to the requested format.

## Steps to reproduce the bug
Run:

```python
from transformers import AutoTokenizer, AutoFeatureExtractor
from datasets import load_dataset

raw = load_dataset("glue", "sst2", split="train")
raw = raw.select(range(100))

tokenizer = AutoTokenizer.from_pretrained("philschmid/tiny-bert-sst2-distilled")

def preprocess_func(examples):
    return tokenizer(examples["sentence"], padding=True, max_length=256, truncation=True)

data = raw.map(preprocess_func, batched=True)

print(type(data[0]["input_ids"]))

data = data.with_format("torch", columns=["input_ids"])

print(type(data[0]["input_ids"]))
```

printing as expected:

```python
<class 'list'>
<class 'torch.Tensor'>
```

Then run:

```python
raw = load_dataset("beans", split="train")
raw = raw.select(range(100))

preprocessor = AutoFeatureExtractor.from_pretrained("nateraw/vit-base-beans")

def preprocess_func(examples):
    imgs = [img.convert("RGB") for img in examples["image"]]
    return preprocessor(imgs)

data = raw.map(preprocess_func, batched=True)

print(type(data[0]["pixel_values"]))

data = data.with_format("torch", columns=["pixel_values"])

print(type(data[0]["pixel_values"]))
```

Printing, unexpectedly

```python
<class 'list'>
<class 'list'>
```

## Expected results
`with_format` should transform into the requested format; it's not the case.

## Actual results
`type(data[0]["pixel_values"])` should be `torch.Tensor` in the example above

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: dev version, commit 44af3fafb527302282f6b6507b952de7435f0979
- Platform: Linux
- Python version: 3.9.12
- PyArrow version: 7.0.0
Fix fine classes in trec dataset
State: closed

This PR:
- replaces the fine labels, so that there are 50 instead of 47
- once more labels are added, all they (fine and coarse) have been re-ordered, so that they align with the order in: https://cogcomp.seas.upenn.edu/Data/QA/QC/definition.html
- the feature names have been fixed: `fine_label` instead of `label-fine`
  - to sneak-case (underscores instead of hyphens)
  - words have been reordered

Fix #4790.
support LargeListArray in pyarrow
State: closed

```python
import numpy as np
import datasets
a = np.zeros((5000000, 768))
res = datasets.Dataset.from_dict({'embedding': a})

'''
  File '/home/wenjiaxin/anaconda3/envs/data/lib/python3.8/site-packages/datasets/arrow_writer.py', line 178, in __arrow_array__
    out = numpy_to_pyarrow_listarray(data)
  File "/home/wenjiaxin/anaconda3/envs/data/lib/python3.8/site-packages/datasets/features/features.py", line 1173, in numpy_to_pyarrow_listarray
    offsets = pa.array(np.arange(n_offsets + 1) * step_offsets, type=pa.int32())
  File "pyarrow/array.pxi", line 312, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 2147483904 not in range: -2147483648 to 2147483647
'''
```


Loading a large numpy array currently raises the error above as the type of offsets is `int32`. 
And pyarrow has supported [LargeListArray](https://arrow.apache.org/docs/python/generated/pyarrow.LargeListArray.html) for this case.
video dataset loader/parser
State: closed
Labels: name: enhancement

you know how you can [use `load_dataset` with any arbitrary csv file](https://huggingface.co/docs/datasets/loading#csv)? and you can also [use it to load a local image dataset](https://huggingface.co/docs/datasets/image_load#local-files)?

could you please add functionality to load a video dataset? it would be really cool if i could point it to a bunch of video files and use pytorch to start looping through batches of videos. like if my batch size is 16, each sample in the batch is a frame from a video. i'm competing in the [minerl challenge](https://www.aicrowd.com/challenges/neurips-2022-minerl-basalt-competition), and it would be awesome to use the HF ecosystem.
Shard generator
State: closed

Hi everyone! I was using Hugging Face datasets to process some very large datasets and found that it would be quite handy to have a feature that will allow to "split" these large datasets into chunks with equal size. Even better - be able to run through these chunks one by one in simple and convenient way. So I decided to add the method called shard_generator() to the main Dataset class. It works similar to shard method but it returns a generator of datasets with equal size (defined by shard_size attribute).
Example:
```python
>>> from datasets import load_dataset
>>> ds = load_dataset("rotten_tomatoes", split="validation")
>>> ds
Dataset({
      features: ['text', 'label'],
      num_rows: 1066
})
>>> next(ds.shard_generator(300))
Dataset({
      features: ['text', 'label'],
      num_rows: 300
})
```
I hope it can be helpful to someone. Thanks!
ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset
State: open
Labels: name: bug

## Describe the bug

When adding a Pillow image to an existing Dataset on the hub, `add_item` fails due to the Pillow image not being automatically converted into the Image feature.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from PIL import Image

dataset = load_dataset("hf-internal-testing/example-documents")

# load any random Pillow image
image = Image.open("/content/cord_example.png").convert("RGB")

new_image = {'image': image}
dataset['test'] = dataset['test'].add_item(new_image)
```

## Expected results
The image should be automatically casted to the Image feature when using `add_item`. For now, this can be fixed by using `encode_example`:

```
import datasets

feature = datasets.Image(decode=False)
new_image = {'image': feature.encode_example(image)}
dataset['test'] = dataset['test'].add_item(new_image)
```

## Actual results

```
ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB size=576x864 at 0x7F7CCC4589D0> with type Image: did not recognize Python value type when inferring an Arrow data type
```
Missing MBPP splits
State: closed
Labels: name: bug

(@albertvillanova)
The [MBPP dataset on the Hub](https://huggingface.co/datasets/mbpp) has only a test split for both its "full" and its "sanitized" subset, while the [paper](https://arxiv.org/abs/2108.07732) states in subsection 2.1 regarding the full split:
> In the experiments described later in the paper, we hold out 10 problems for **few-shot prompting**, another 500 as our **test** dataset (which is used to evaluate both few-shot inference and fine-tuned models), 374 problems for **fine-tuning**, and the rest for **validation**.

If the dataset on the Hub should reproduce most closely what the original authors use, I guess this four-way split should be reflected. 

The paper doesn't explicitly state the task_id ranges of the splits, but the [GitHub readme](https://github.com/google-research/google-research/tree/master/mbpp) referenced in the paper specifies exact task_id ranges, although it misstates the total number of samples:
> We specify a train and test split to use for evaluation. Specifically:
> 
> * Task IDs 11-510 are used for evaluation.
> * Task IDs 1-10 and 511-1000 are used for training and/or prompting. We typically used 1-10 for few-shot prompting, although you can feel free to use any of the training examples.

I.e. the few-shot, train and validation splits are combined into one split, with a soft suggestion of using the first ten for few-shot prompting. It is not explicitly stated whether the 374 fine-tuning samples mentioned in the paper have task_id 511 to 784 or 601 to 974 or are randomly sampled from task_id 511 to 974.

Regarding the "sanitized" split the paper states the following:
> For evaluations involving the edited dataset, we perform comparisons with 100 problems that appear in both the original and edited dataset, using the same held out 10 problems for few-shot prompting and 374 problems for fine-tuning. 

The statement doesn't appear to be very precise, as among the 10 few-shot problems, those with task_id 1, 5 and 10 are not even part of the sanitized variant, and many from the task_id range from 511 to 974 are missing (e.g. task_id 511 to 553). I suppose the idea the task_id ranges for each split remain the same, even if some of the task_ids are not present. That would result in 7 few-shot, 257 test, 141 train and 22 validation examples in the sanitized split.
Add DocVQA
State: open
Labels: name: dataset request

## Adding a Dataset
- **Name:** DocVQA
- **Description:** Document Visual Question Answering (DocVQA) seeks to inspire a “purpose-driven” point of view in Document Analysis and Recognition research, where the document content is extracted and used to respond to high-level tasks defined by the human consumers of this information. 
- **Paper:** https://arxiv.org/abs/2007.00398
- **Data:** https://www.docvqa.org/datasets/docvqa
- **Motivation:** Models like LayoutLM and Donut in the Transformers library are fine-tuned on DocVQA. Would be very handy to directly load this dataset from the hub.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).
Dataset Viewer issue for Team-PIXEL/rendered-wikipedia-english
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/Team-PIXEL/rendered-wikipedia-english/viewer/rendered-wikipedia-en/train

### Description

The dataset can be loaded fine but the viewer shows this error:

```
Server Error
Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.
```

I'm guessing this is because I recently renamed the dataset. Based on related issues (e.g. https://github.com/huggingface/datasets/issues/4759) , is there something server-side that needs to be refreshed?

### Owner

Yes
Issue with fine classes in trec dataset
State: closed
Labels: name: bug

## Describe the bug
According to their paper, the TREC dataset contains 2 kinds of classes:
- 6 coarse classes: TREC-6
- 50 fine classes: TREC-50

However, our implementation only has 47 (instead of 50) fine classes. The reason for this is that we only considered the last segment of the label, which is repeated for several coarse classes:
- We have one `desc` fine label instead of 2:
  - `DESC:desc`
  - `HUM:desc`
- We have one `other` fine label instead of 3:
  - `ENTY:other`
  - `LOC:other`
  - `NUM:other`

From their paper:
> We define a two-layered taxonomy, which represents a natural semantic classification for typical answers in the TREC task. The hierarchy contains 6 coarse classes and 50 fine classes,

> Each coarse class contains a non-overlapping set of fine classes.
NonMatchingChecksumError in mbpp dataset
State: closed
Labels: name: bug

## Describe the bug
As reported on the Hub [Fix Checksum Mismatch](https://huggingface.co/datasets/mbpp/discussions/1), there is a `NonMatchingChecksumError` when loading mbpp dataset

## Steps to reproduce the bug
```python
ds = load_dataset("mbpp", "full")
```

## Expected results
Loading of the dataset without any exception raised.

## Actual results
```
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-1-a3fbdd3ed82e> in <module>
----> 1 ds = load_dataset("mbpp", "full")

.../huggingface/datasets/src/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1791 
   1792     # Download and prepare data
-> 1793     builder_instance.download_and_prepare(
   1794         download_config=download_config,
   1795         download_mode=download_mode,
.../huggingface/datasets/src/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703                     if not downloaded_from_gcs:
--> 704                         self._download_and_prepare(
    705                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706                         )

.../huggingface/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)
   1225 
   1226     def _download_and_prepare(self, dl_manager, verify_infos):
-> 1227         super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
   1228 
   1229     def _get_examples_iterable_for_split(self, split_generator: SplitGenerator) -> ExamplesIterable:

.../huggingface/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    773         # Checksums verification
    774         if verify_infos and dl_manager.record_checksums:
--> 775             verify_checksums(
    776                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    777             )

.../huggingface/datasets/src/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://raw.githubusercontent.com/google-research/google-research/master/mbpp/mbpp.jsonl']
```
.save_to_disk('path', fs=s3) TypeError 
State: closed
Labels: name: bug

The following code:
```python
import datasets

train_dataset, test_dataset = load_dataset("imdb", split=["train", "test"])
s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)
train_dataset.save_to_disk("s3://datasets/", fs=s3)

```
produces following traceback:

```shell
File "C:\Users\Hong Knop\AppData\Local\Programs\Python\Python310\lib\site-packages\botocore\auth.py", line 374, in scope
    return '/'.join(scope)

```
I invoke print(scope) in <auth.py> (line 373) and find this:

```python
[('4VA08VLL3VTKQJKCAI8M',), '20220803', 'us-east-1', 's3', 'aws4_request']

```
Add Multiface dataset
State: open
Labels: name: dataset request, name: vision

## Adding a Dataset
- **Name:** Multiface dataset
- **Description:** f high quality recordings of the faces of 13 identities, each captured in a multi-view capture stage performing various facial expressions. An average of 12,200 (v1 scripts) to 23,000 (v2 scripts) frames per subject with capture rate at 30 fps
- **Data:** https://github.com/facebookresearch/multiface

The whole dataset is 65TB though, so I'm not sure

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
State: closed
Labels: name: bug

## Describe the bug
Following the example in CodeParrot, I receive an array size limitation error when deduplicating larger datasets.

## Steps to reproduce the bug
```python
dataset_name = "the_pile"
ds = load_dataset(dataset_name, split="train")
ds = ds.map(preprocess, num_proc=num_workers)
uniques = set(ds.unique("hash"))
```
Gists for minimum reproducible example:
https://gist.github.com/conceptofmind/c5804428ea1bd89767815f9cd5f02d9a
https://gist.github.com/conceptofmind/feafb07e236f28d79c2d4b28ffbdb6e2

## Expected results
Chunking and writing out a deduplicated dataset. 

## Actual results
```
return dataset._data.column(column).unique().to_pylist()
File "pyarrow/table.pxi", line 394, in pyarrow.lib.ChunkedArray.unique
File "pyarrow/_compute.pyx", line 531, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 330, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 124, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
```
Fix label renaming and add a battery of tests
State: closed

This PR makes some changes to label renaming in `to_tf_dataset()`, both to fix some issues when users input something we weren't expecting, and also to make it easier to deprecate label renaming in future, if/when we want to move this special-casing logic to a function in `transformers`.

The main changes are:
- Label renaming now only happens when the `auto_rename_labels` argument is set. For backward compatibility, this defaults to `True` for now.
- If the user requests "label" but the data collator renames that column to "labels", the label renaming logic will now handle that case correctly.
- Added a battery of tests to make this more reliable in future.
- Adds an optimization to loading in `to_tf_dataset()` for unshuffled datasets (uses slicing instead of a list of indices)

Fixes #4772
RuntimeError when using torchaudio 0.12.0 to load MP3 audio file
State: closed

Current version of `torchaudio` (0.12.0) raises a RuntimeError when trying to use `sox_io` backend but non-Python dependency `sox` is not installed:
https://github.com/pytorch/audio/blob/2e1388401c434011e9f044b40bc8374f2ddfc414/torchaudio/backend/sox_io_backend.py#L21-L29
```python
def _fail_load(
    filepath: str,
    frame_offset: int = 0,
    num_frames: int = -1,
    normalize: bool = True,
    channels_first: bool = True,
    format: Optional[str] = None,
) -> Tuple[torch.Tensor, int]:
    raise RuntimeError("Failed to load audio from {}".format(filepath))
```

Maybe we should raise a more actionable error message so that the user knows how to fix it.

UPDATE:
- this is an incompatibility of latest torchaudio (0.12.0) and the sox backend

TODO:
- [x] as a temporary solution, we should recommend installing torchaudio<0.12.0
  - #4777 
  - #4785
- [ ] however, a stable solution must be found for torchaudio>=0.12.0

Related to: 
- https://github.com/huggingface/transformers/issues/18379
Training hangs at the end of epoch, with set_transform/with_transform+multiple workers
State: open
Labels: name: bug

## Describe the bug
I use load_dataset() (I tried with [wiki](https://huggingface.co/datasets/wikipedia) and my own json data) and use set_transform/with_transform for preprocessing. But it hangs at the end of the 1st epoch if dataloader_num_workers>=1. No problem with single worker. 


## Steps to reproduce the bug
```python
train_dataset = datasets.load_dataset("wikipedia", "20220301.en",
                                      split='train', 
                                      cache_dir=model_args.cache_dir,
                                      streaming=False)
train_dataset.set_transform(psg_parse_fn)
train_dataloader = DataLoader(
            train_dataset,
            batch_size=args.train_batch_size,
            sampler=DistributedSampler(train_dataset),
            collate_fn=data_collator,
            drop_last=args.dataloader_drop_last,
            num_workers=args.dataloader_num_workers,
        )
```

## Expected results


## Actual results
It simply hangs. The ending step is num_example/batch_size (one epoch).

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Linux-5.4.170+-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.1
AssertionError when using label_cols in to_tf_dataset 
State: closed
Labels: name: bug

## Describe the bug
An incorrect `AssertionError` is raised when using `label_cols` in `to_tf_dataset` and the label's key name is `label`.

The assertion is in this line:
https://github.com/huggingface/datasets/blob/2.4.0/src/datasets/arrow_dataset.py#L475

## Steps to reproduce the bug
```python
from datasets import load_dataset
from transformers import DefaultDataCollator

dataset = load_dataset('glue', 'mrpc', split='train')

tf_dataset = dataset.to_tf_dataset(
    columns=["sentence1", "sentence2", "idx"],
    label_cols=["label"],
    batch_size=16,
    collate_fn=DefaultDataCollator(return_tensors="tf"),
)
```

## Expected results
No assertion error.

## Actual results
```
AssertionError: in user code:

    File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 475, in split_features_and_labels  *
        assert set(features.keys()).union(labels.keys()) == set(input_batch.keys())
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.18.0-305.45.1.el8_4.ppc64le-ppc64le-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.3
More rigorous shape inference in to_tf_dataset
State: closed

`tf.data` needs to know the shape of tensors emitted from a `tf.data.Dataset`. Although `None` dimensions are possible, overusing them can cause problems - Keras uses the dataset tensor spec at compile-time, and so saying that a dimension is `None` when it's actually constant can hurt performance, or even cause training to fail for dimensions that are needed to determine the shape of weight tensors!

The compromise I used here was to sample several batches from the underlying dataset and apply the `collate_fn` to them, and then to see which dimensions were "empirically variable". There's an obvious problem here, though - if you sample 10 batches and they all have the same shape on a certain dimension, there's still a small chance that the 11th batch will be different, and Keras will throw an error  if a dataset tries to emit a tensor whose shape doesn't match the spec.

I encountered this bug in practice once or twice for datasets that were mostly-but-not-totally constant on a given dimension, and I still don't have a perfect solution, but this PR should greatly reduce the risk. It samples many more batches, and also samples very small batches (size 2) - this increases the variability, making it more likely that a few outlier samples will be detected.

Ideally, of course, we'd determine the full output shape analytically, but that's surprisingly tricky when the `collate_fn` can be any arbitrary Python code!
Improve features resolution in streaming
State: closed

`IterableDataset._resolve_features` was returning the features sorted alphabetically by column name, which is not consistent with non-streaming. I changed this and used the order of columns from the data themselves. It was causing some inconsistencies in the dataset viewer as well.

I also fixed `interleave_datasets` that was not filling missing columns with None, because it was not using the columns from `IterableDataset._resolve_features`

cc @severo
parallel searching in multi-gpu setting using faiss
State: open

While I notice that  `add_faiss_index` has supported assigning multiple GPUs, I am still confused about how it works. 

Does the `search-batch` function automatically parallelizes the input queries to different gpus?https://github.com/huggingface/datasets/blob/d76599bdd4d186b2e7c4f468b05766016055a0a5/src/datasets/search.py#L360
Issue with offline mode
State: closed
Labels: name: bug

## Describe the bug
I can't retrieve a cached dataset with offline mode enabled

## Steps to reproduce the bug

To reproduce my issue, first, you'll need to run a script that will cache the dataset
```python
import os
os.environ["HF_DATASETS_OFFLINE"] = "0"

import datasets

datasets.logging.set_verbosity_info()
ds_name = "SaulLu/toy_struc_dataset"
ds = datasets.load_dataset(ds_name)
print(ds)
```
then, you can try to reload it in offline mode:
```python
import os
os.environ["HF_DATASETS_OFFLINE"] = "1"

import datasets

datasets.logging.set_verbosity_info()
ds_name = "SaulLu/toy_struc_dataset"
ds = datasets.load_dataset(ds_name)
print(ds)
```

## Expected results
I would have expected the 2nd snippet not to return any errors

## Actual results
The 2nd snippet returns:
```
Traceback (most recent call last):
  File "/home/lucile_huggingface_co/sandbox/evaluate/test_cache_datasets.py", line 8, in <module>
    ds = datasets.load_dataset(ds_name)
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1723, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1500, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1241, in dataset_module_factory
    raise ConnectionError(f"Couln't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couln't reach the Hugging Face Hub for dataset 'SaulLu/toy_struc_dataset': Offline mode is enabled.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

Maybe I'm misunderstanding something in the use of the offline mode (see [doc](https://huggingface.co/docs/datasets/v2.4.0/en/loading#offline)), is that the case?
Dataset Viewer issue for Toygar/turkish-offensive-language-detection
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/Toygar/turkish-offensive-language-detection

### Description

Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.

Hi, I provided train.csv, test.csv and valid.csv files. However, viewer says dataset does not exist. 
Should I need to do anything else?

### Owner

Yes
Document better when relative paths are transformed to URLs
State: closed
Labels: name: documentation

As discussed with @ydshieh, when passing a relative path as `data_dir` to `load_dataset` of a dataset hosted on the Hub, the relative path is transformed to the corresponding URL of the Hub dataset.

Currently, we mention this in our docs here: [Create a dataset loading script > Download data files and organize splits](https://huggingface.co/docs/datasets/v2.4.0/en/dataset_script#download-data-files-and-organize-splits)
> If the data files live in the same folder or repository of the dataset script, you can just pass the relative paths to the files instead of URLs.

Maybe we should document better how relative paths are handled, not only when creating a dataset loading script, but also when passing to `load_dataset`:
- `data_dir`
- `data_files`

CC: @stevhliu
Datasets.map causes incorrect overflow_to_sample_mapping when used with tokenizers and small batch size
State: open
Labels: name: bug

## Describe the bug
When using `tokenizer`, we can retrieve the field `overflow_to_sample_mapping`, since long samples will be overflown into multiple token sequences.

However, when tokenizing is done via `Dataset.map`, with `n_proc > 1`, the `overflow_to_sample_mapping` field is wrong.  This seems to be because each tokenizer only looks at its share of the samples, and maps to the index _within its share_, but then `Dataset.map` collates them together.

## Steps to reproduce the bug

1. Make a dataset of 3 strings.
2. Tokenize via Dataset.map with n_proc = 8
3. Inspect the `overflow_to_sample_mapping` field


## Expected results
`[0, 1, 2]`

## Actual results
`[0, 0, 0]`

Notes:

1. I have not yet extracted a minimal example, but the above works reliably
2. If the dataset is large, I've yet to determine if this bug still happens a. not at all b. always c. on the small, leftover batch at the end.
Add `language_bcp47` tag
State: closed

Following (internal) https://github.com/huggingface/moon-landing/pull/3509, we need to move the bcp47 tags to `language_bcp47` and keep the `language` tag for iso 639 1-2-3 codes. In particular I made sure that all the tags in `languages` are not longer than 3 characters. I moved the rest to `language_bcp47` and fixed some of them.

After this PR is merged I think we can simplify the language validation from the DatasetMetadata class (and keep it bare-bone just for the tagging app)

PS: the CI is failing because of missing content in dataset cards that are unrelated to this PR
Easily create loading script for benchmark comprising multiple huggingface datasets
State: closed

Hi,

I would like to create a loading script for a benchmark comprising multiple huggingface datasets.
The function _split_generators needs to return the files for the respective dataset. However, the files are not always in the same location for each dataset. I want to just make a wrapper dataset that provides a single interface to all the underlying datasets. 
I thought about downloading the files with the load_dataset function and then providing the link to the cached file. But this seems a bit inelegant to me. What approach would you propose to do this?

Please let me know if you have any questions.

Cheers,
Joel
Shard parquet in `download_and_prepare`
State: closed

Following https://github.com/huggingface/datasets/pull/4724 (needs to be merged first)

It's good practice to shard parquet files to enable parallelism with spark/dask/etc.

I added the `max_shard_size` parameter to `download_and_prepare` (default to 500MB for parquet, and None for arrow).

```python
from datasets import *

output_dir = "./output_dir"  # also supports "s3://..."
builder = load_dataset_builder("squad")
builder.download_and_prepare(output_dir, file_format="parquet", max_shard_size="5MB")
```

### Implementation details

The examples are written to a parquet file until `ParquetWriter._num_bytes > max_shard_size`. When this happens, a new writer is instantiated to start writing the next shard. At the end, all the shards are renamed to include the total number of shards in their names: `{builder.name}-{split}-{shard_id:05d}-of-{num_shards:05d}.parquet`

I also added the `MAX_SHARD_SIZE` config variable (default to 500MB)

TODO:
- [x] docstrings
- [x] docs
- [x] tests

cc @severo
Allow `list_datasets` to include private datasets
State: closed
Labels: name: enhancement

I am working with a large collection of private datasets, it would be convenient for me to be able to list them.

I would envision extending the convention of using `use_auth_token` keyword argument to `list_datasets` function, then calling:

```
list_datasets(use_auth_token="my_token")
```

would return the list of all datasets I have permissions to view, including private ones. The only current alternative I see is to use the hub website to manually obtain the list of dataset names - this is in the context of BigScience where respective private spaces contain hundreds of datasets, so not very convenient to list manually.
Remove instructions to generate dummy data from our docs
State: closed
Labels: name: documentation

In our docs, we indicate to generate the dummy data: https://huggingface.co/docs/datasets/dataset_script#testing-data-and-checksum-metadata

However:
- dummy data makes sense only for datasets in our GitHub repo: so that we can test their loading with our CI
- for datasets on the Hub:
  - they do not pass any CI test requiring dummy data
  - there are no instructions on how they can test their dataset locally using the dummy data
  - the generation of the dummy data assumes our GitHub directory structure:
    - the dummy data will be generated under `./datasets/<dataset_name>/dummy` even if locally there is no `./datasets` directory (which is the usual case). See issue:
      - #4742 

CC: @stevhliu
Dummy data nowhere to be found
State: closed
Labels: name: bug

## Describe the bug
To finalize my dataset, I wanted to create dummy data as per the guide and I ran 

```shell
 datasets-cli dummy_data datasets/hebban-reviews --auto_generate
```

where hebban-reviews is [this repo](https://huggingface.co/datasets/BramVanroy/hebban-reviews). And even though the scripts runs and shows a message at the end that it succeeded, I cannot find the dummy data anywhere. Where is it?

## Expected results

To see the dummy data in the datasets' folder or in the folder where I ran the command.

## Actual results

I see the following message but I cannot find the dummy data anywhere.

```
Dummy data generation done and dummy data test succeeded for config 'filtered''.
Automatic dummy data generation succeeded for all configs of '.\datasets\hebban-reviews\'
```

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
Fix multiprocessing in map_nested 
State: closed

As previously discussed:

Before, multiprocessing was not used in `map_nested` if `num_proc` was greater than or equal to `len(iterable)`.
- Multiprocessing was not used e.g. when passing `num_proc=20` but having 19 files to download
- As by default, `DownloadManager` sets `num_proc=16`, before multiprocessing was only used when `len(iterable)>16` by default

Now, if `num_proc` is greater than or equal to ``len(iterable)``, `num_proc` is set to ``len(iterable)`` and multiprocessing is used.
- We pass the variable `parallel_min_length=16`, so that multiprocessing is only used if at least 16 files to be downloaded
- ~As by default, `DownloadManager` sets `num_proc=16`, now multiprocessing is used when `len(iterable)>1` by default~

See discussion below.

~After having had to fix some tests (87602ac), I am wondering:~
- ~do we want to have multiprocessing by default?~
  - ~please note that `DownloadManager.download` sets `num_proc=16` by default~
- ~or would it be better to ask the user to set it explicitly if they want multiprocessing (and default to `num_proc=1`)?~

Fix #4636.

CC: @nateraw
Dataset Viewer issue for deepklarity/huggingface-spaces-dataset
State: closed
Labels: name: dataset-viewer

### Link

https://huggingface.co/datasets/deepklarity/huggingface-spaces-dataset/viewer/deepklarity--huggingface-spaces-dataset/train

### Description

Hi Team, 
I'm getting the following error on a uploaded dataset. I'm getting the same status for a couple of hours now. The dataset size is `<1MB` and the format is csv, so I'm not sure if it's supposed to take this much time or not. 
```
Status code:   400
Exception:     Status400Error
Message:       The split is being processed. Retry later.
```

Is there any explicit step to be taken to get the viewer to work? 

### Owner

Yes
Package rouge-score cannot be imported
State: closed
Labels: name: bug

## Describe the bug
After the today release of `rouge_score-0.0.7` it seems no longer importable. Our CI fails: https://github.com/huggingface/datasets/runs/7463218591?check_suite_focus=true
```
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_builder_class_bigbench
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_builder_configs_bigbench
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_bigbench
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_rouge
```
with errors:
```
>   from rouge_score import rouge_scorer
E   ModuleNotFoundError: No module named 'rouge_score'
```
```
E           ImportError: To be able to use rouge, you need to install the following dependency: rouge_score.
E           Please install it using 'pip install rouge_score' for instance'
```