NyssaW/github-issues huggingface dataset:
Support PIL Image objects in `add_item`/`add_column`

Fix #4796 

PS: We should also improve the type inference in `OptimizedTypeSequence` to make it possible to also infer the complex types (only `Image` currently) in nested arrays (e.g.  `[[pil_image], [pil_image, pil_image]]` or `[{"img": pil_image}`]), but I plan to address this in a separate PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4828). All of your documentation changes will be reflected on that endpoint.

Hey @mariosasko could we please merge this? I'm still getting the original error at #4796 .

Are you planning to continue working on this?
[Windows] Fix Access Denied when using os.rename()

In this PR, we are including an additional step when `os.rename()` raises a PermissionError.

Basically, we will use `shutil.move()` on the temp files.

Fix #2937 

Comments:
Cool thank you ! Maybe we can just replace `os.rename` by `shutil.move` instead ?

> Cool thank you ! Maybe we can just replace `os.rename` by `shutil.move` instead ?

Yes, I think that could be a better solution, but I didn't test it in Linux (e.g. Ubuntu) to guarantee that `os.rename()` could be completely replaced by `shutil.move()`.

AFAIK `shutil.move` does call `os.rename` first before doing extra work to make it work on windows, so this is should be a safe safe change for linux ;)

> AFAIK `shutil.move` does call `os.rename` first before doing extra work to make it work on windows, so this is should be a safe safe change for linux ;)

alright, let me change the PR then.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4825). All of your documentation changes will be reflected on that endpoint.

Hi @lhoestq looks like one of the tests failed, but is not related to this change, do I need to do something from my side?
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.

Hi, when i try to run prepare_dataset function in [fine tuning ASR tutorial 4](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tuning_Wav2Vec2_for_English_ASR.ipynb) , i got this error.
I got this error
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
There is no other logs available, so i have no clue what is the cause of it.
```

def prepare_dataset(batch):
    audio = batch["path"]
    # batched output is "un-batched"
    batch["input_values"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
    batch["input_length"] = len(batch["input_values"])
    with processor.as_target_processor():
        batch["labels"] = processor(batch["text"]).input_ids
    return batch

data = data.map(prepare_dataset, remove_columns=data.column_names["train"],
                      num_proc=4)
```


Specify the actual results or traceback.
There is no traceback except
`Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.`
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-5.15.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.4.3


Comments:
Fixed by installing either resampy<3 or resampy>=4
Add add cc-by-sa-2.5 license tag

- [ ] add it to moon-landing
- [ ] add it to hub-docs 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4818). All of your documentation changes will be reflected on that endpoint.

I think we can close this PR because the `standard_licenses.tsv` file was removed from this repo and we no longer perform any dataset card validation.
Fix loading example in opus dataset cards

This PR:
- fixes the examples to load the datasets, with the corrected dataset name, in their dataset cards for:
  - opus_dgt
  - opus_paracrawl
  - opus_wikipedia
- fixes their dataset cards with the missing required information: title, data instances/fields/splits
- enumerates the supported languages
- adds a missing citation reference for opus_wikipedia

Related to:
- #4806

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix bug in function validate_type for Python >= 3.9

Fix `validate_type` function, so that it uses `get_origin` instead. This makes the function forward compatible.

This fixes #4811 because:
```python
In [4]: typing.Optional[str]
Out[4]: typing.Optional[str]

In [5]: get_origin(typing.Optional[str])
Out[5]: typing.Union
```

Fix #4811.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Bug in function validate_type for Python >= 3.9

## Describe the bug
The function `validate_type` assumes that the type `typing.Optional[str]` is automatically transformed to `typing.Union[str, NoneType]`.
```python
In [4]: typing.Optional[str]
Out[4]: typing.Union[str, NoneType]
```

However, this is not the case for Python 3.9:
```python
In [3]: typing.Optional[str]
Out[3]: typing.Optional[str]
```


Comments:
Complete the mlqa dataset card

I fixed the issue #4808 

Details of PR:
- Added languages included in the dataset. 
- Added task id and task category. 
- Updated the citation information. 

Fix #4808.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for your contribution, @eldhoittangeorge.
> 
> The CI error message: https://github.com/huggingface/datasets/runs/7743526624?check_suite_focus=true
> 
> ```
> E           ValueError: The following issues have been found in the dataset cards:
> E           YAML tags:
> E           __init__() missing 5 required positional arguments: 'annotations_creators', 'language_creators', 'license', 'size_categories', and 'source_datasets'
> ```

I will fix the CI error.

@eldhoittangeorge, thanks again for all the fixes. Just a minor one before we can merge this PR: https://github.com/huggingface/datasets/runs/7744885754?check_suite_focus=true
```
E           YAML tags:
E           Could not validate the metadata, found the following errors:
E           * field 'language_creators':
E           	['unknown'] are not registered tags for 'language_creators', reference at https://github.com/huggingface/datasets/tree/main/src/datasets/utils/resources/creators.json
```

> 

Thanks, I updated the file. 
A small suggestion can you mention this link https://github.com/huggingface/datasets/tree/main/src/datasets/utils/resources/  in the contribution page. So that others will know the acceptable values for the tags.
Wrong example in opus_gnome dataset card

## Describe the bug
I found that [the example on opus_gone dataset ](https://github.com/huggingface/datasets/tree/main/datasets/opus_gnome#dataset-summary) doesn't work.

## Steps to reproduce the bug
```python
 load_dataset("gnome", lang1="it", lang2="pl")
```
`"gnome"` should be `"opus_gnome"`

## Expected results
```bash
100%
1/1 [00:00<00:00, 42.09it/s]
DatasetDict({
    train: Dataset({
        features: ['id', 'translation'],
        num_rows: 8368
    })
})
```

## Actual results
```bash
 Couldn't find 'gnome' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/main/datasets/gnome/gnome.py
```

## Environment info
- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-120-generic-x86_64-with-glibc2.27
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3


Comments:
Support `pipeline` argument in inspect.py functions

**Is your feature request related to a problem? Please describe.**

The `wikipedia` dataset requires a `pipeline` argument to build the list of splits:

https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py#L937

But this is currently not supported in `get_dataset_config_info`:

https://github.com/huggingface/datasets/blob/main/src/datasets/inspect.py#L373-L375

which is called by other functions, e.g. `get_dataset_split_names`.

**Additional context**

The dataset viewer is not working out-of-the-box on `wikipedia` for this reason:

https://huggingface.co/datasets/wikipedia/viewer

<img width="637" alt="Capture d’écran 2022-08-08 à 12 01 16" src="https://user-images.githubusercontent.com/1676121/183461838-5330783b-0269-4ba7-a999-314cde2023d8.png">



Comments:
Now: the preview (first-rows) works, but not the conversion to parquet. See https://huggingface.co/datasets/wikipedia/viewer/20220301.de/train

```
_split_generators() missing 1 required positional argument: 'pipeline'

Error code:   UnexpectedError
```
Fix fine classes in trec dataset

This PR:
- replaces the fine labels, so that there are 50 instead of 47
- once more labels are added, all they (fine and coarse) have been re-ordered, so that they align with the order in: https://cogcomp.seas.upenn.edu/Data/QA/QC/definition.html
- the feature names have been fixed: `fine_label` instead of `label-fine`
  - to sneak-case (underscores instead of hyphens)
  - words have been reordered

Fix #4790.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
video dataset loader/parser

you know how you can [use `load_dataset` with any arbitrary csv file](https://huggingface.co/docs/datasets/loading#csv)? and you can also [use it to load a local image dataset](https://huggingface.co/docs/datasets/image_load#local-files)?

could you please add functionality to load a video dataset? it would be really cool if i could point it to a bunch of video files and use pytorch to start looping through batches of videos. like if my batch size is 16, each sample in the batch is a frame from a video. i'm competing in the [minerl challenge](https://www.aicrowd.com/challenges/neurips-2022-minerl-basalt-competition), and it would be awesome to use the HF ecosystem.

Comments:
Hi! We've just started discussing the video support in `datasets` (decoding backends, video feature type, etc.), so I believe we should have something tangible by the end of this year.

Also, if you have additional video features in mind that you would like to see, feel free to let us know

Coool thanks @mariosasko 

Hey @mariosasko, I was wondering if there's a way to load video data currently in the library? 
Alternatively is there a way I could hack it through the dataset.from_dict() method? I tried to hack it, but the issue I run into is that earlier I was doing a `cast_column()` call for the `Image` feature, but now I'm not sure about to do if I want the dataset to have the following keys when I call from_dict on it:
`{"caption":[list of text captions], "video_frames": [list of image lists with one image list corresponding to one video]}`

Maybe something like `cast_column("video_frames", List(Image))` ..
(This is assuming I have already extracted frames from video)
Torgo dataset creation


Comments:
Hi @YingLi001, thanks for your proposal to add this dataset.

However, now we add datasets directly to the Hub (instead of our GitHub repository). You have the instructions in our docs: 
- [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)
- [Create a dataset card](https://huggingface.co/docs/datasets/dataset_card)
- [Share](https://huggingface.co/docs/datasets/share)

Feel free to ask if you need any additional support/help.
Missing MBPP splits

(@albertvillanova)
The [MBPP dataset on the Hub](https://huggingface.co/datasets/mbpp) has only a test split for both its "full" and its "sanitized" subset, while the [paper](https://arxiv.org/abs/2108.07732) states in subsection 2.1 regarding the full split:
> In the experiments described later in the paper, we hold out 10 problems for **few-shot prompting**, another 500 as our **test** dataset (which is used to evaluate both few-shot inference and fine-tuned models), 374 problems for **fine-tuning**, and the rest for **validation**.

If the dataset on the Hub should reproduce most closely what the original authors use, I guess this four-way split should be reflected. 

The paper doesn't explicitly state the task_id ranges of the splits, but the [GitHub readme](https://github.com/google-research/google-research/tree/master/mbpp) referenced in the paper specifies exact task_id ranges, although it misstates the total number of samples:
> We specify a train and test split to use for evaluation. Specifically:
> 
> * Task IDs 11-510 are used for evaluation.
> * Task IDs 1-10 and 511-1000 are used for training and/or prompting. We typically used 1-10 for few-shot prompting, although you can feel free to use any of the training examples.

I.e. the few-shot, train and validation splits are combined into one split, with a soft suggestion of using the first ten for few-shot prompting. It is not explicitly stated whether the 374 fine-tuning samples mentioned in the paper have task_id 511 to 784 or 601 to 974 or are randomly sampled from task_id 511 to 974.

Regarding the "sanitized" split the paper states the following:
> For evaluations involving the edited dataset, we perform comparisons with 100 problems that appear in both the original and edited dataset, using the same held out 10 problems for few-shot prompting and 374 problems for fine-tuning. 

The statement doesn't appear to be very precise, as among the 10 few-shot problems, those with task_id 1, 5 and 10 are not even part of the sanitized variant, and many from the task_id range from 511 to 974 are missing (e.g. task_id 511 to 553). I suppose the idea the task_id ranges for each split remain the same, even if some of the task_ids are not present. That would result in 7 few-shot, 257 test, 141 train and 22 validation examples in the sanitized split.

Comments:
Thanks for reporting this as well, @stadlerb.

I suggest waiting for the answer of the data owners... 

@albertvillanova The first author of the paper responded to the upstream issue:
> Task IDs 11-510 are the 500 test problems. We use 90 problems (511-600) for validation and then remaining 374 for fine-tuning (601-974). The other problems can be used as desired, either for training or few-shot prompting (although this should be specified).

Thanks for the follow-up, @stadlerb.

Would you be willing to open a Pull Request to address this issue? :wink: 

Opened a [PR](https://github.com/huggingface/datasets/pull/4943) to implement this--lmk if you have any feedback
Add DocVQA

## Adding a Dataset
- **Name:** DocVQA
- **Description:** Document Visual Question Answering (DocVQA) seeks to inspire a “purpose-driven” point of view in Document Analysis and Recognition research, where the document content is extracted and used to respond to high-level tasks defined by the human consumers of this information. 
- **Paper:** https://arxiv.org/abs/2007.00398
- **Data:** https://www.docvqa.org/datasets/docvqa
- **Motivation:** Models like LayoutLM and Donut in the Transformers library are fine-tuned on DocVQA. Would be very handy to directly load this dataset from the hub.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).


Comments:
Thanks for proposing, @NielsRogge.

Please, note this dataset requires registering in their website and their Terms and Conditions state we cannot distribute their URL:
```
1. You will NOT distribute the download URLs
...
```
Dataset Viewer issue for Team-PIXEL/rendered-wikipedia-english

### Link

https://huggingface.co/datasets/Team-PIXEL/rendered-wikipedia-english/viewer/rendered-wikipedia-en/train

### Description

The dataset can be loaded fine but the viewer shows this error:

```
Server Error
Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.
```

I'm guessing this is because I recently renamed the dataset. Based on related issues (e.g. https://github.com/huggingface/datasets/issues/4759) , is there something server-side that needs to be refreshed?

### Owner

Yes

Comments:
Thanks for reporting. It's a known issue that should be fixed soon. Meanwhile, I had to manually trigger the dataset viewer. It's OK now.
Note that the extreme aspect ratio of the images generates another issue, that we're inspecting.
Issue with fine classes in trec dataset

## Describe the bug
According to their paper, the TREC dataset contains 2 kinds of classes:
- 6 coarse classes: TREC-6
- 50 fine classes: TREC-50

However, our implementation only has 47 (instead of 50) fine classes. The reason for this is that we only considered the last segment of the label, which is repeated for several coarse classes:
- We have one `desc` fine label instead of 2:
  - `DESC:desc`
  - `HUM:desc`
- We have one `other` fine label instead of 3:
  - `ENTY:other`
  - `LOC:other`
  - `NUM:other`

From their paper:
> We define a two-layered taxonomy, which represents a natural semantic classification for typical answers in the TREC task. The hierarchy contains 6 coarse classes and 50 fine classes,

> Each coarse class contains a non-overlapping set of fine classes.



Comments:
Fix NonMatchingChecksumError in mbpp dataset

Fix issue reported on the Hub: https://huggingface.co/datasets/mbpp/discussions/1

Fix #4787. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for the quick response! Before noticing that you already had implemented the fix, I already had implemened my own version. I'd also suggest bumping the major version because the contents of the dataset changed, even if only slightly.
I'll attach my version of the affected files: [mbpp-checksum-changes.zip](https://github.com/huggingface/datasets/files/9258161/mbpp-checksum-changes.zip).

Hi @stadlerb, thanks for your feedback.

We normally update the major version whenever there is a new dataset release, usually with a breaking change in schema. The patch version is updated whenever there is a small correction in the dataset that does not change its schema.

As a side note for future contributions, please note that this dataset is hosted in our library GitHub repository. Therefore, the PRs to GitHub-hosted datasets needs being done through GitHub.

Currently added datasets are hosted on the Hub and for them, PRs can be done through the Hub.

I just noticed another problem with the dataset: The [GitHub page](https://github.com/google-research/google-research/tree/master/mbpp) and the [paper](http://arxiv.org/abs/2108.07732) mention a train-test split, which is not reflected in the dataloader. I'll open a new issue regarding this later.
NonMatchingChecksumError in mbpp dataset

## Describe the bug
As reported on the Hub [Fix Checksum Mismatch](https://huggingface.co/datasets/mbpp/discussions/1), there is a `NonMatchingChecksumError` when loading mbpp dataset

## Steps to reproduce the bug
```python
ds = load_dataset("mbpp", "full")
```

## Expected results
Loading of the dataset without any exception raised.

## Actual results
```
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-1-a3fbdd3ed82e> in <module>
----> 1 ds = load_dataset("mbpp", "full")

.../huggingface/datasets/src/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1791 
   1792     # Download and prepare data
-> 1793     builder_instance.download_and_prepare(
   1794         download_config=download_config,
   1795         download_mode=download_mode,
.../huggingface/datasets/src/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703                     if not downloaded_from_gcs:
--> 704                         self._download_and_prepare(
    705                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706                         )

.../huggingface/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)
   1225 
   1226     def _download_and_prepare(self, dl_manager, verify_infos):
-> 1227         super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
   1228 
   1229     def _get_examples_iterable_for_split(self, split_generator: SplitGenerator) -> ExamplesIterable:

.../huggingface/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    773         # Checksums verification
    774         if verify_infos and dl_manager.record_checksums:
--> 775             verify_checksums(
    776                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    777             )

.../huggingface/datasets/src/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://raw.githubusercontent.com/google-research/google-research/master/mbpp/mbpp.jsonl']
```



Comments:
.save_to_disk('path', fs=s3) TypeError 

The following code:
```python
import datasets

train_dataset, test_dataset = load_dataset("imdb", split=["train", "test"])
s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)
train_dataset.save_to_disk("s3://datasets/", fs=s3)

```
produces following traceback:

```shell
File "C:\Users\Hong Knop\AppData\Local\Programs\Python\Python310\lib\site-packages\botocore\auth.py", line 374, in scope
    return '/'.join(scope)

```
I invoke print(scope) in <auth.py> (line 373) and find this:

```python
[('4VA08VLL3VTKQJKCAI8M',), '20220803', 'us-east-1', 's3', 'aws4_request']

```

Comments:
Add Multiface dataset

## Adding a Dataset
- **Name:** Multiface dataset
- **Description:** f high quality recordings of the faces of 13 identities, each captured in a multi-view capture stage performing various facial expressions. An average of 12,200 (v1 scripts) to 23,000 (v2 scripts) frames per subject with capture rate at 30 fps
- **Data:** https://github.com/facebookresearch/multiface

The whole dataset is 65TB though, so I'm not sure

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).


Comments:
Hi @osanseviero I would like to add this dataset.

Hey @nandwalritik! Thanks for offering to help!

This dataset might be somewhat complex and I'm concerned about it being 65 TB, which would be quite expensive to host. @lhoestq @mariosasko I would love your input if you think it's worth adding this dataset.

Thanks for proposing this interesting dataset, @osanseviero.

Please note that the data files are already hosted in a third-party server: e.g. the index of data files for entity "6795937" is at https://fb-baas-f32eacb9-8abb-11eb-b2b8-4857dd089e15.s3.amazonaws.com/MugsyDataRelease/v0.0/identities/6795937/index.html 
- audio.tar: https://fb-baas-f32eacb9-8abb-11eb-b2b8-4857dd089e15.s3.amazonaws.com/MugsyDataRelease/v0.0/identities/6795937/audio.tar
- ...

Therefore, in principle, we don't need to host them on our Hub: it would be enough to just implement a loading script in the corresponding Hub dataset repo, e.g. "facebook/multiface"...
Docs for creating a loading script for image datasets

This PR is a first draft of creating a loading script for image datasets. Feel free to let me know if there are any specificities I'm missing for this. 🙂 

To do:
- [x] Document how to create different configurations.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

IMO it would make more sense to add a "Create image dataset" page with two main sections - a no-code approach with `imagefolder` + metadata (preferred way), and with a loading script (advanced). It should be clear when to choose which. If we leave this as-is, the user who jumps straight to the Vision section could be under the impression that writing a loading script is the preferred way to share a vision dataset due to how this subsection starts:
```
Write a dataset loading script to share a dataset.
```
  
Also, I think a note explaining how to make a dataset gated/disable the viewer to hide the data would be beneficial (it's pretty common to require submitting a form to access a CV dataset).

Great suggestion @mariosasko! I added your suggestions, let me know what you think. For gated dataset access, I just added a tip referring users to the relevant docs since it's more of a Hub feature than `datasets` feature.

Thanks, looks much better now :). I would also move the sections explaining how to create an `imagefolder` for the specific task from the [loading page](https://raw.githubusercontent.com/huggingface/datasets/main/docs/source/image_load.mdx) to this one. IMO it makes more sense to have the basic info (imagefolder structure + `load_dataset` call) there + a link to this page for info on how to create an image folder dataset.

Good idea! Moved everything about `imagefolder` + metadata to the create an image dataset section since the `load_dataset` call is the same for different computer vision tasks. 

Thanks for all the feedbacks! 🥰

What do you think about creating how to share an `ImageFolder` dataset in a separate PR? I think we should create a new section under `Vision` for how to share an image dataset.

I love it thanks ! I think moving forward we can use CSV instead of JSON Lines in the docs ;)
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648

## Describe the bug
Following the example in CodeParrot, I receive an array size limitation error when deduplicating larger datasets.

## Steps to reproduce the bug
```python
dataset_name = "the_pile"
ds = load_dataset(dataset_name, split="train")
ds = ds.map(preprocess, num_proc=num_workers)
uniques = set(ds.unique("hash"))
```
Gists for minimum reproducible example:
https://gist.github.com/conceptofmind/c5804428ea1bd89767815f9cd5f02d9a
https://gist.github.com/conceptofmind/feafb07e236f28d79c2d4b28ffbdb6e2

## Expected results
Chunking and writing out a deduplicated dataset. 

## Actual results
```
return dataset._data.column(column).unique().to_pylist()
File "pyarrow/table.pxi", line 394, in pyarrow.lib.ChunkedArray.unique
File "pyarrow/_compute.pyx", line 531, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 330, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 124, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
```

Comments:
Thanks for reporting @conceptofmind.

Could you please give details about your environment? 
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
```

Hi @albertvillanova ,

Here is the environment information:
```
- `datasets` version: 2.3.2
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.27
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.2
```
Thanks,

Enrico

I think this issue is solved here https://discuss.huggingface.co/t/minhash-deduplication/19992/12?u=loubnabnl, this only happens for very large datasets we will update it in CodeParrot code

Hi @loubnabnl,

Yes, the issue is solved in the discussion thread.

I will close this issue.

Thank you again for all of your help.

Enrico

Thanks @loubnabnl for pointing out the solution to this issue.
Update local loading script docs

This PR clarifies the local loading script section to include how to load a dataset after you've modified the local loading script (closes #4732).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4778). All of your documentation changes will be reflected on that endpoint.

I would rather have a section in the docs that explains how to modify the script of an existing dataset (`inspect_dataset` + modification + `load_dataset`) instead of focusing on the GH datasets bundled with the source (only applicable for devs).

Good idea! I went with @mariosasko's suggestion to use `inspect_dataset` instead of cloning a dataset repository since it's a good opportunity to show off more of the library's lesser-known functions if that's ok with everyone :)

One advantage of cloning the repo is that it fetches potential data files referenced inside a script using relative paths, so if we decide to use `inspect_dataset`, we should at least add a tip to explain this limitation and how to circumvent it.

Oh you're right. Calling `load_dataset` on the modified script without having the files that come with it is not ideal. I agree it should be `git clone` instead - and inspect is for inspection only ^^'
RuntimeError when using torchaudio 0.12.0 to load MP3 audio file

Current version of `torchaudio` (0.12.0) raises a RuntimeError when trying to use `sox_io` backend but non-Python dependency `sox` is not installed:
https://github.com/pytorch/audio/blob/2e1388401c434011e9f044b40bc8374f2ddfc414/torchaudio/backend/sox_io_backend.py#L21-L29
```python
def _fail_load(
    filepath: str,
    frame_offset: int = 0,
    num_frames: int = -1,
    normalize: bool = True,
    channels_first: bool = True,
    format: Optional[str] = None,
) -> Tuple[torch.Tensor, int]:
    raise RuntimeError("Failed to load audio from {}".format(filepath))
```

Maybe we should raise a more actionable error message so that the user knows how to fix it.

UPDATE:
- this is an incompatibility of latest torchaudio (0.12.0) and the sox backend

TODO:
- [x] as a temporary solution, we should recommend installing torchaudio<0.12.0
  - #4777 
  - #4785
- [ ] however, a stable solution must be found for torchaudio>=0.12.0

Related to: 
- https://github.com/huggingface/transformers/issues/18379

Comments:
Requiring torchaudio<0.12.0 isn't really a viable solution because that implies torch<0.12.0 which means no sm_86 CUDA support which means no RTX 3090 support in PyTorch.

But in my case, the error only occurs if `_fallback_load` resolves to `_fail_load` inside torchaudio 0.12.0 which is only the case if FFMPEG initialization failed:  https://github.com/pytorch/audio/blob/b1f510fa5681e92ee82bdc6b2d1ed896799fc32c/torchaudio/backend/sox_io_backend.py#L36-L47

That means the proper solution for torchaudio>=0.12.0 is to check `torchaudio._extension._FFMPEG_INITIALIZED` and if it is False, then we need to remind the user to install a dynamically linked ffmpeg 4.1.8 and then maybe call `torchaudio._extension._init_ffmpeg()` to force a user-visible exception showing the missing ffmpeg dynamic library name.

On my system, installing 

- libavcodec.so.58 
- libavdevice.so.58  
- libavfilter.so.7  
- libavformat.so.58  
- libavutil.so.56  
- libswresample.so.3  
- libswscale.so.5

from ffmpeg 4.1.8 made HF datasets 2.3.2 work just fine with  torchaudio 0.12.1+cu116:

```python3
import sox, torchaudio, datasets
print('torchaudio', torchaudio.__version__)
print('datasets', datasets.__version__)
torchaudio._extension._init_ffmpeg()
print(torchaudio._extension._FFMPEG_INITIALIZED)
waveform, sample_rate = torchaudio.load('/workspace/.cache/huggingface/datasets/downloads/extracted/8e5aa88585efa2a4c74c6664b576550d32b7ff9c3d1d17cc04f44f11338c3dc6/cv-corpus-8.0-2022-01-19/en/clips/common_voice_en_100038.mp3', format='mp3')
print(waveform.shape)
```

```
torchaudio 0.12.1+cu116
datasets 2.3.2
True
torch.Size([1, 369792])
```

Related: https://github.com/huggingface/datasets/issues/4889

Closing as we no longer use `torchaudio` for decoding MP3 files.
Streaming not supported in Theivaprakasham/wildreceipt

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments:
Thanks for reporting @NitishkKarra.

The root source of the issue is that streaming mode is not supported out-of-the-box for that dataset, because it contains a TAR file.

We have opened a discussion in the corresponding Hub dataset page, pointing out this issue: https://huggingface.co/datasets/Theivaprakasham/wildreceipt/discussions/1

I'm closing this issue here, so this discussion is transferred there instead.
Training hangs at the end of epoch, with set_transform/with_transform+multiple workers

## Describe the bug
I use load_dataset() (I tried with [wiki](https://huggingface.co/datasets/wikipedia) and my own json data) and use set_transform/with_transform for preprocessing. But it hangs at the end of the 1st epoch if dataloader_num_workers>=1. No problem with single worker. 


## Steps to reproduce the bug
```python
train_dataset = datasets.load_dataset("wikipedia", "20220301.en",
                                      split='train', 
                                      cache_dir=model_args.cache_dir,
                                      streaming=False)
train_dataset.set_transform(psg_parse_fn)
train_dataloader = DataLoader(
            train_dataset,
            batch_size=args.train_batch_size,
            sampler=DistributedSampler(train_dataset),
            collate_fn=data_collator,
            drop_last=args.dataloader_drop_last,
            num_workers=args.dataloader_num_workers,
        )
```

## Expected results


## Actual results
It simply hangs. The ending step is num_example/batch_size (one epoch).

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Linux-5.4.170+-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.1

Comments:
Document loading from relative path

This PR describes loading a dataset from the Hub by specifying a relative path in `data_dir` or `data_files` in `load_dataset` (see #4757).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the feedback!

I agree that adding it to `load_hub.mdx` is probably a bit too specific, especially for beginners reading the tutorials. Since this clarification is closely related to loading from the Hub (the only difference being the presence/absence of a loading script), I think it makes the most sense to keep it somewhere in `loading.mdx`. What do you think about adding a Warning in Loading >>> Hugging Face Hub that explains the difference between relative/absolute paths when there is a script?

What about updating the section about "manual download" ? I think it goes there no ?

https://huggingface.co/docs/datasets/v2.4.0/en/loading#manual-download

Updated the manual download section :)

Thanks ! Pinging @albertvillanova to review this change, and then I think we're good to merge
More rigorous shape inference in to_tf_dataset

`tf.data` needs to know the shape of tensors emitted from a `tf.data.Dataset`. Although `None` dimensions are possible, overusing them can cause problems - Keras uses the dataset tensor spec at compile-time, and so saying that a dimension is `None` when it's actually constant can hurt performance, or even cause training to fail for dimensions that are needed to determine the shape of weight tensors!

The compromise I used here was to sample several batches from the underlying dataset and apply the `collate_fn` to them, and then to see which dimensions were "empirically variable". There's an obvious problem here, though - if you sample 10 batches and they all have the same shape on a certain dimension, there's still a small chance that the 11th batch will be different, and Keras will throw an error  if a dataset tries to emit a tensor whose shape doesn't match the spec.

I encountered this bug in practice once or twice for datasets that were mostly-but-not-totally constant on a given dimension, and I still don't have a perfect solution, but this PR should greatly reduce the risk. It samples many more batches, and also samples very small batches (size 2) - this increases the variability, making it more likely that a few outlier samples will be detected.

Ideally, of course, we'd determine the full output shape analytically, but that's surprisingly tricky when the `collate_fn` can be any arbitrary Python code!

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Improve features resolution in streaming

`IterableDataset._resolve_features` was returning the features sorted alphabetically by column name, which is not consistent with non-streaming. I changed this and used the order of columns from the data themselves. It was causing some inconsistencies in the dataset viewer as well.

I also fixed `interleave_datasets` that was not filling missing columns with None, because it was not using the columns from `IterableDataset._resolve_features`

cc @severo 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Just took your comment into account @mariosasko , let me know if it's good for you now :)
Dataset Viewer issue for Toygar/turkish-offensive-language-detection

### Link

https://huggingface.co/datasets/Toygar/turkish-offensive-language-detection

### Description

Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.

Hi, I provided train.csv, test.csv and valid.csv files. However, viewer says dataset does not exist. 
Should I need to do anything else?

### Owner

Yes

Comments:
I refreshed the dataset viewer manually, it's fixed now. Sorry for the inconvenience.
<img width="1557" alt="Capture d’écran 2022-07-28 à 09 17 39" src="https://user-images.githubusercontent.com/1676121/181514666-92d7f8e1-ddc1-4769-84f3-f1edfdb902e8.png">
Document better when relative paths are transformed to URLs

As discussed with @ydshieh, when passing a relative path as `data_dir` to `load_dataset` of a dataset hosted on the Hub, the relative path is transformed to the corresponding URL of the Hub dataset.

Currently, we mention this in our docs here: [Create a dataset loading script > Download data files and organize splits](https://huggingface.co/docs/datasets/v2.4.0/en/dataset_script#download-data-files-and-organize-splits)
> If the data files live in the same folder or repository of the dataset script, you can just pass the relative paths to the files instead of URLs.

Maybe we should document better how relative paths are handled, not only when creating a dataset loading script, but also when passing to `load_dataset`:
- `data_dir`
- `data_files`

CC: @stevhliu 

Comments:
Add `language_bcp47` tag

Following (internal) https://github.com/huggingface/moon-landing/pull/3509, we need to move the bcp47 tags to `language_bcp47` and keep the `language` tag for iso 639 1-2-3 codes. In particular I made sure that all the tags in `languages` are not longer than 3 characters. I moved the rest to `language_bcp47` and fixed some of them.

After this PR is merged I think we can simplify the language validation from the DatasetMetadata class (and keep it bare-bone just for the tagging app)

PS: the CI is failing because of missing content in dataset cards that are unrelated to this PR

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Easily create loading script for benchmark comprising multiple huggingface datasets

Hi,

I would like to create a loading script for a benchmark comprising multiple huggingface datasets.
The function _split_generators needs to return the files for the respective dataset. However, the files are not always in the same location for each dataset. I want to just make a wrapper dataset that provides a single interface to all the underlying datasets. 
I thought about downloading the files with the load_dataset function and then providing the link to the cached file. But this seems a bit inelegant to me. What approach would you propose to do this?

Please let me know if you have any questions.

Cheers,
Joel

Comments:
Hi ! I think the simplest is to copy paste the `_split_generators` code from the other datasets and do a bunch of if-else, as in the glue dataset: https://huggingface.co/datasets/glue/blob/main/glue.py#L467

Ok, I see. Thank you
Shard parquet in `download_and_prepare`

Following https://github.com/huggingface/datasets/pull/4724 (needs to be merged first)

It's good practice to shard parquet files to enable parallelism with spark/dask/etc.

I added the `max_shard_size` parameter to `download_and_prepare` (default to 500MB for parquet, and None for arrow).

```python
from datasets import *

output_dir = "./output_dir"  # also supports "s3://..."
builder = load_dataset_builder("squad")
builder.download_and_prepare(output_dir, file_format="parquet", max_shard_size="5MB")
```

### Implementation details

The examples are written to a parquet file until `ParquetWriter._num_bytes > max_shard_size`. When this happens, a new writer is instantiated to start writing the next shard. At the end, all the shards are renamed to include the total number of shards in their names: `{builder.name}-{split}-{shard_id:05d}-of-{num_shards:05d}.parquet`

I also added the `MAX_SHARD_SIZE` config variable (default to 500MB)

TODO:
- [x] docstrings
- [x] docs
- [x] tests

cc @severo 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This is ready for review cc @mariosasko :) please let me know what you think !
Dataset Viewer issue for yanekyuk/wikikey

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments:
The dataset is empty, as far as I can tell: there are no files in the repository at https://huggingface.co/datasets/yanekyuk/wikikey/tree/main

Maybe the viewer can display a better message for empty datasets

OK. Closing as it's not an error. We will work on making the error message a lot clearer.
Allow `list_datasets` to include private datasets

I am working with a large collection of private datasets, it would be convenient for me to be able to list them.

I would envision extending the convention of using `use_auth_token` keyword argument to `list_datasets` function, then calling:

```
list_datasets(use_auth_token="my_token")
```

would return the list of all datasets I have permissions to view, including private ones. The only current alternative I see is to use the hub website to manually obtain the list of dataset names - this is in the context of BigScience where respective private spaces contain hundreds of datasets, so not very convenient to list manually.

Comments:
Thanks for opening this issue :)

If it can help, I think you can already use `huggingface_hub` to achieve this:
```python
>>> from huggingface_hub import HfApi
>>> [ds_info.id for ds_info in HfApi().list_datasets(use_auth_token=token) if ds_info.private]
['bigscience/xxxx', 'bigscience-catalogue-data/xxxxxxx', ... ]
```

---------

Though the latest versions of `huggingface_hub` that contain this feature are not available on python 3.6, so maybe we should first drop support for python 3.6 (see #4460) to update `list_datasets` in `datasets` as well (or we would have to copy/paste some `huggingface_hub` code)

Great, thanks @lhoestq the workaround works! I think it would be intuitive to have the support directly in `datasets` but it makes sense to wait given that the workaround exists :)

i also think that going forward we should replace more and more implementations inside datasets with the corresponding ones from `huggingface_hub` (same as we're doing in `transformers`)

`datasets.list_datasets` is now deprecated in favor of `huggingface_hub.list_datasets` (returns private datasets when `token` is present), so I'm closing this issue.
Remove instructions to generate dummy data from our docs

In our docs, we indicate to generate the dummy data: https://huggingface.co/docs/datasets/dataset_script#testing-data-and-checksum-metadata

However:
- dummy data makes sense only for datasets in our GitHub repo: so that we can test their loading with our CI
- for datasets on the Hub:
  - they do not pass any CI test requiring dummy data
  - there are no instructions on how they can test their dataset locally using the dummy data
  - the generation of the dummy data assumes our GitHub directory structure:
    - the dummy data will be generated under `./datasets/<dataset_name>/dummy` even if locally there is no `./datasets` directory (which is the usual case). See issue:
      - #4742 

CC: @stevhliu 

Comments:
Note that for me personally, conceptually all the dummy data (even for "canonical" datasets) should be superseded by `datasets-server`, which performs some kind of CI/CD of datasets (including the canonical ones)

I totally agree: next step should be rethinking if dummy data makes sense for canonical datasets (once we have datasets-server) and eventually remove it.

But for now, we could at least start by removing the indication to generate dummy data from our docs.
Dummy data nowhere to be found

## Describe the bug
To finalize my dataset, I wanted to create dummy data as per the guide and I ran 

```shell
 datasets-cli dummy_data datasets/hebban-reviews --auto_generate
```

where hebban-reviews is [this repo](https://huggingface.co/datasets/BramVanroy/hebban-reviews). And even though the scripts runs and shows a message at the end that it succeeded, I cannot find the dummy data anywhere. Where is it?

## Expected results

To see the dummy data in the datasets' folder or in the folder where I ran the command.

## Actual results

I see the following message but I cannot find the dummy data anywhere.

```
Dummy data generation done and dummy data test succeeded for config 'filtered''.
Automatic dummy data generation succeeded for all configs of '.\datasets\hebban-reviews\'
```

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 8.0.0
- Pandas version: 1.4.3



Comments:
Hi @BramVanroy, thanks for reporting.

First of all, please note that you do not need the dummy data: this was the case when we were adding datasets to the `datasets` library (on this GitHub repo), so that we could test the correct loading of all datasets with our CI. However, this is no longer the case for datasets on the Hub.
- We should definitely update our docs.

Second, the dummy data is generated locally:
- in your case, the dummy data will be generated inside the directory:  `./datasets/hebban-reviews/dummy`
- please note the preceding `./datasets` directory: the reason for this is that the command to generate the dummy data was specifically created for our `datasets` library, and therefore assumes our directory structure: commands are run from the root directory of our GitHub repo, and datasets scripts are under `./datasets` 


 

I have opened an Issue to update the instructions on dummy data generation:
- #4744

Dummy data generation is deprecated now, so I think we can close this issue.