konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
adding mafand to datasets

I'm addding the MAFAND dataset by Masakhane based on the paper/repository below:

Paper: https://aclanthology.org/2022.naacl-main.223/
Code: https://github.com/masakhane-io/lafand-mt

Please, help merge this

Everything works except for creating dummy data file

_The documentation is not available anymore as the PR was closed or merged._

Hi @dadelani, thanks for your awesome contribution!!! :heart: 

However, now we are using the Hub to add new datasets, instead of this GitHub repo. 

You could share this dataset under your Hub organization namespace: [Masakhane NLP](https://huggingface.co/masakhane). This way the dataset will be accessible using:
```python
ds = load_dataset("masakhane/mafand")
```

You have the procedure documented in our online docs: 
- [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)
- [Share](https://huggingface.co/docs/datasets/share)

Moreover, datasets shared on the Hub no longer need the dummy data files.

Please, feel free to ping me if you need any further guidance/support.

thank you for the comment. I have moved it to the Hub https://huggingface.co/datasets/masakhane/mafand

Great job, @dadelani!!

Please, note that in the README.md file, the YAML tags should be preceded and followed by three dashes `---`, so that they are properly parsed. See, e.g.: https://raw.githubusercontent.com/huggingface/datasets/main/templates/README.md

Also you could replace the line:
```
# Dataset Card for [Needs More Information]
```
with
```
# Dataset Card for MAFAND-MT
```

Great, thank you for the feedback. I have fixed both issues.
Dataset Viewer issue for MoritzLaurer/multilingual_nli

### Link

_No response_

### Description

I've just uploaded a new dataset to the hub and the viewer does not work for some reason, see here: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli

It displays the error: 
```
Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.
```

Weirdly enough the dataviewer works for an earlier version of the same dataset. The only difference is that it is smaller, but I'm not aware of other changes I have made: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli_test

Do you know why the dataviewer is not working?

### Owner

_No response_

Thanks for reporting  @MoritzLaurer.

Currently, the dataset preview is working properly: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli

Please note that when a dataset is modified, it might take some time until the preview is completely updated.

@severo might it be worth adding a clearer error message, something like "The preview is updating, please retry later"?

Thanks for your response. You are right, its now working well. I had waited for 30 min or so and refreshed several times and thought there was some other error. Yeah, a different error message sounds like a good idea to avoid confusion. 

I'm closing this issue then.

> @severo might it be worth adding a clearer error message, something like "The preview is updating, please retry later"?

Yes, it's a known issue, and we're about to ship a better version
Allow pathlib PoxisPath in Dataset.read_json

**Is your feature request related to a problem? Please describe.**
```
from pathlib import Path
from datasets import Dataset
ds = Dataset.read_json(Path('data.json'))
```
causes an error
```
AttributeError: 'PosixPath' object has no attribute 'decode'
```

**Describe the solution you'd like**

It should be able to accept PosixPath and read the json from inside.

This same error will occur using `ds = datasets.load_dataset('json', data_files=['test.jsonl'])`

@cccntu I want to make a quick fix for this, but I am struggling to find where the json dataset builder is. Do you know?

@vvvm23 I think you mean think:
```python
ds = datasets.load_dataset('json', data_files=[Path('test.jsonl')])
```
And the place you want to modify is here:
```
utils/file_utils.py:64, in is_remote_url(url_or_filename)
     63 def is_remote_url(url_or_filename: str) -> bool:
---> 64     parsed = urlparse(url_or_filename)
     65     return parsed.scheme in ("http", "https", "s3", "gs", "hdfs", "ftp")
```

Probably just need to check first if `url_or_filename` is [PathLike](https://docs.python.org/3/library/os.html#os.PathLike) and return False early.

Btw, I tried installing from main, and ran my code above and got a different error. Probably because the API have changed.
`AttributeError: module 'datasets' has no attribute 'read_json'`


> @vvvm23 I think you mean think:

You are correct, thanks!

> Probably just need to check first if url_or_filename is [PathLike](https://docs.python.org/3/library/os.html#os.PathLike) and return False early.

Is PathLike sufficient, or should I check the file exists here? Or is that handled later?

I think here we just want to avoid passing Path to urlparse. A simpler solution is to add a str() call and convert the input to string before passing to the next step. No need to check anything.

Above PR should do your first suggestion. Hope that works for you, as I am going on holiday and won't be able to change much :wink: 

I just came across this problem with `DatasetDict.from_parquet()`. The error still seems to originate from the same place:
```
  File "/usr/local/lib/python3.12/dist-packages/datasets/utils/file_utils.py", line 88, in is_relative_path
    return urlparse(url_or_filename).scheme == "" and not os.path.isabs(url_or_filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
```
The PR looks like it would still solve everything. Is there a reason this got delayed?
TFDS wiki_dialog dataset to Huggingface dataset

## Adding a Dataset
- **Name:** *Wiki_dialog*
- **Description: https://github.com/google-research/dialog-inpainting#:~:text=JSON%20object%2C%20for-,example,-%3A
- **Paper: https://arxiv.org/abs/2205.09073
- **Data: https://github.com/google-research/dialog-inpainting
- **Motivation:** *Research and Development on biggest corpus of dialog data*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).


@albertvillanova any help ? The linked dataset is in beam format which is similar to wikipedia dataset in huggingface that you  scripted..

Nvm, I was able to port it to huggingface datasets, will upload to the hub soon

https://huggingface.co/datasets/djaym7/wiki_dialog

Thanks for the addition, @djaym7.
Using disk for memory with the method `from_dict`

**Is your feature request related to a problem? Please describe.**
I start with an empty dataset. In a loop, at each iteration, I create a new dataset with the method `from_dict` (based on some data I load) and I concatenate this new dataset with the one at the previous iteration. After some iterations, I have an OOM error.

**Describe the solution you'd like**
The method `from_dict` loads the data in RAM. It could be good to add an option to use the disk instead.

**Describe alternatives you've considered**
To solve the problem, I have to do an intermediate step where I save the new datasets at each iteration with `save_to_disk`. Once it's done, I open them all and concatenate them.


This issue was also causing an OOM in @nateraw 's workflow and shows again that behavior is confusing - we should definitely switch to using the disk IMO
Add collection3 dataset


Hi @pefimov. Thanks for you awesome work on this dataset contribution.

However, now we are using the Hub to add new datasets, instead of this GitHub repo. 

You could share this dataset under the appropriate Hub organization namespace. This way the dataset will be accessible using:
```python
ds = load_dataset("<org_namespace>/collection3")
```

You have the procedure documented in our online docs: 
- [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)
- [Share](https://huggingface.co/docs/datasets/share)

Moreover, datasets shared on the Hub no longer need the dummy data files.

Please, feel free to ping me if you need any further guidance/support. 

> However, now we are using the Hub to add new datasets, instead of this GitHub repo.
> 
> You could share this dataset under the appropriate Hub organization namespace. This way the dataset will be accessible using:
> 
> ```python
> ds = load_dataset("<org_namespace>/collection3")
> ```
> 
Hi @albertvillanova . Thank you for your response.

I thought that Collection3 is large and important dataset in Russian presented in 2016 but not represented in huggingface.

Also I am not related to authors or organisation of dataset

The current policy of sharing datasets on the Hub instead of in this GitHub repo has no relation with the importance of the dataset: https://huggingface.co/docs/datasets/share#datasets-on-github-legacy 
> The distinction between a Hub dataset and a dataset from GitHub only comes from the legacy sharing workflow. It does not involve any ranking, decisioning, or opinion regarding the contents of the dataset itself.

It is not required to be an author/owner (or belong to the organization that is owner) of the dataset in order to share it on the Hub (as it was not the case when sharing them on this GitHub repo). 

It is recommended to share it under an organization namespace that makes sense though. For this specific dataset, do you know of a clear organization under which it could be shared on the Hub? Maybe "labinform", or "Information Research Laboratory" or "Lomonosov Moscow State University"?

In cases like this, where the org is not evident, one possibility could be to contact the dataset owners/creators and ask them. According the publication paper, the authors are:
- V.A. Mozharova
- N.V. Loukachevitch

I think maybe it would be worth contacting them.

@pefimov I have contacted the authors (and put you in CC).

Reply from the authors:
> It is better to use name: Research Computing Center of Lomonosov Moscow State University (short name RCC-MSU)
> https://rcc.msu.ru/en

I have created the corresponding org namespace and dataset empty repository: https://huggingface.co/datasets/RCC-MSU/collection3

@pefimov feel free to open a PR on the Hub if you are willing to do so: 
- Go to the *Community* tab on the repo: https://huggingface.co/datasets/RCC-MSU/collection3/discussions
- And click: *New pull request* button

Docs: [Pull requests and Discussions](https://huggingface.co/docs/hub/repositories-pull-requests-discussions) on the Hub

Thanks
can't install using conda on Windows 10

## Describe the bug
I wanted to install using conda or Anaconda navigator. That didn't work, so I had to install using pip.

## Steps to reproduce the bug
conda install -c huggingface -c conda-forge datasets

## Expected results
Should have indicated successful installation.

## Actual results
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
... took forever, so I cancelled it with ctrl-c

## Environment info
- `datasets` version: 2.4.0  # after installing with pip
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.12
- PyArrow version: 9.0.0
- Pandas version: 1.4.2
- conda version: 4.13.0

conda info

     active environment : base
    active env location : G:\anaconda2022
            shell level : 1
       user config file : C:\Users\michael\.condarc
 populated config files : C:\Users\michael\.condarc
          conda version : 4.13.0
    conda-build version : 3.21.8
         python version : 3.9.12.final.0
       virtual packages : __cuda=11.1=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : G:\anaconda2022  (writable)
      conda av data dir : G:\anaconda2022\etc\conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/pytorch/win-64
                          https://conda.anaconda.org/pytorch/noarch
                          https://conda.anaconda.org/huggingface/win-64
                          https://conda.anaconda.org/huggingface/noarch
                          https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/anaconda-fusion/win-64
                          https://conda.anaconda.org/anaconda-fusion/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : G:\anaconda2022\pkgs
                          C:\Users\michael\.conda\pkgs
                          C:\Users\michael\AppData\Local\conda\conda\pkgs
       envs directories : G:\anaconda2022\envs
                          C:\Users\michael\.conda\envs
                          C:\Users\michael\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.13.0 requests/2.27.1 CPython/3.9.12 Windows/10 Windows/10.0.19044
          administrator : False
             netrc file : None
           offline mode : False
map() function removes columns when input_columns is not None

## Describe the bug
The map function, removes features from the dataset that are not present in the _input_columns_ list of columns, despite the columns being removed not mentioned in the _remove_columns_ argument.
## Steps to reproduce the bug
```python
from datasets import Dataset
ds = Dataset.from_dict({"a" : [1,2,3],"b" : [0,1,0], "c" : [2,4,5]})

def double(x,y):
  x = x*2
  y = y*2
  return {"d" : x, "e" : y}

ds.map(double, input_columns=["a","c"])
```

## Expected results
```
Dataset({
    features: ['a', 'b', 'c', 'd', 'e'],
    num_rows: 3
})
```
## Actual results
```
Dataset({
    features: ['a', 'c', 'd', 'e'],
    num_rows: 3
})
```

In this specific example feature **b** should not be removed.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: linux (colab)
- Python version: 3.7.13
- PyArrow version: 6.0.1


Hi! Thanks for reporting! This looks like a bug. I've just opened a PR with the fix.

Awesome! Thank you. I'll close the issue once the PR gets merged. :-)

I guess we should reopen after the revert by:
- #5006
No preprocessed wikipedia is working on huggingface/datasets

## Describe the bug
20220301 wikipedia dump has been deprecated, so now there is no working wikipedia dump on huggingface

https://huggingface.co/datasets/wikipedia

https://dumps.wikimedia.org/enwiki/




Thanks for reporting @aninrusimha.

Please, note that the preprocessed datasets are still available, as described in the dataset card, e.g.: https://huggingface.co/datasets/wikipedia
```python
ds = load_dataset("wikipedia", "20220301.en")
``` 

This is working now, but I was getting an error a few days ago when running an existing script. Unfortunately I did not do a proper bug report, but for some reason I was unable to load the dataset due to a request being made to the wikimedia website. However, its working now. Thanks for the reply!
file missing when load_dataset with openwebtext on windows

## Describe the bug
0015896-b1054262f7da52a0518521e29c8e352c.txt is missing when I run run_mlm.py with openwebtext. I check the cache_path and can not find 0015896-b1054262f7da52a0518521e29c8e352c.txt. but I can find this file in the 17ecf461bfccd469a1fbc264ccb03731f8606eea7b3e2e8b86e13d18040bf5b3/urlsf_subset00-16_data.xz  with 7-zip.

## Steps to reproduce the bug
```sh
python run_mlm.py --model_type roberta --tokenizer_name roberta-base --dataset_name openwebtext --per_device_train_batch_size 8 --per_device_eval_batch_size 8 --do_train --do_eval --output_dir F:/model/roberta-base
```
or 
```python
from datasets import load_dataset
load_dataset("openwebtext", None, cache_dir=None, use_auth_token=None)
```

## Expected results
Loading is successful
## Actual results
Traceback (most recent call last):
  File "D:\Python\v3.8.5\lib\site-packages\datasets\builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "D:\Python\v3.8.5\lib\site-packages\datasets\builder.py", line 1227, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "D:\Python\v3.8.5\lib\site-packages\datasets\builder.py", line 795, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file. 
Original error:
[Errno 22] Invalid argument: 'F://huggingface/datasets/downloads/extracted/0901d27f43b7e9ac0577da0d0061c8c632ba0b70ecd1b4bfb21562d9b7486faa/0015896-b1054262f7da52a0518521e29c8e352c.txt'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: windows
- Python version: 3.8.5 
- PyArrow version: 9.0.0


I have tried to extract ```0015896-b1054262f7da52a0518521e29c8e352c.txt``` from ```17ecf461bfccd469a1fbc264ccb03731f8606eea7b3e2e8b86e13d18040bf5b3/urlsf_subset00-16_data.xz``` with 7-zip
and put the file into cache_path ```F://huggingface/datasets/downloads/extracted/0901d27f43b7e9ac0577da0d0061c8c632ba0b70ecd1b4bfb21562d9b7486faa```
there is still raise the same error and I  find the file  was removed from cache_path after I run the run_mlm.py with ```python run_mlm.py --model_type roberta --tokenizer_name roberta-base --dataset_name openwebtext --per_device_train_batch_size 8 --per_device_eval_batch_size 8 --do_train --do_eval --output_dir F:/model/roberta-base```.
Fix test of _get_extraction_protocol for TAR files

While working in another PR, I discovered an xpass test (a test that is supposed to xfail but nevertheless passes) when testing `_get_extraction_protocol`: https://github.com/huggingface/datasets/runs/7818845285?check_suite_focus=true
```
XPASS tests/test_streaming_download_manager.py::test_streaming_dl_manager_get_extraction_protocol_throws[https://foo.bar/train.tar] 
```

This PR:
- refactors the test so that it tests the raise of the exceptions instead of xfailing
- fixes the test for TAR files: it does not raise an exception, but returns "tar"
- fixes some tests wrongly named: exchange `test_streaming_dl_manager_get_extraction_protocol` with `test_streaming_dl_manager_get_extraction_protocol_gg_drive`

_The documentation is not available anymore as the PR was closed or merged._
Update documentation card of miam dataset

Hi !
Paper has been published at EMNLP.

_The documentation is not available anymore as the PR was closed or merged._

Ahahah :D not sur how i broke something by updating the README :D 

Thanks for the fix @PierreColombo. 

Once a README is modified, our CI runs tests on it, requiring additional quality fixes, so that all READMEs are progressively improved and have some minimal tags/sections/information.

For this specific README file, the additional quality requirements of the CI are: https://github.com/huggingface/datasets/runs/7819924428?check_suite_focus=true
```
E           The following issues were found for the README at `/home/runner/work/datasets/datasets/datasets/miam/README.md`:
E           -	Section `Additional Information` is missing subsection: `Dataset Curators`.
E           -	Section `Additional Information` is missing subsection: `Contributions`.
E           -	`Additional Information` has an extra subsection: `Benchmark Curators`. Skipping further validation checks for this subsection as expected structure is unknown.
```

Thanks a lot Albert :)))
Mark CI tests as xfail if Hub HTTP error

In order to make testing more robust (and avoid merges to master with red tests), we could mark tests as xfailed (instead of failed) when the Hub raises some temporary HTTP errors.

This PR:
- marks tests as xfailed only if the Hub raises a 500 error for:
  - test_upstream_hub
- makes pytest report the xfailed/xpassed tests.

More tests could also be marked if needed.

Examples of CI failures due to temporary Hub HTTP errors:
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files
  - https://github.com/huggingface/datasets/runs/7806855399?check_suite_focus=true
    `requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://hub-ci.huggingface.co/api/datasets/__DUMMY_TRANSFORMERS_USER__/test-16603108028233/commit/main (Request ID: aZeAQ5yLktoGHQYBcJ3zo)`
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_no_token
  - https://github.com/huggingface/datasets/runs/7840022996?check_suite_focus=true
    `requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://s3.us-east-1.amazonaws.com/lfs-staging.huggingface.co/repos/81/e3/81e3b831fa9bf23190ec041f26ef7ff6d6b71c1a937b8ec1ef1f1f05b508c089/caae596caa179cf45e7c9ac0c6d9a9cb0fe2d305291bfbb2d8b648ae26ed38b6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4N7VTDGOZQA2IKWK%2F20220815%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220815T144713Z&X-Amz-Expires=900&X-Amz-Signature=5ddddfe8ef2b0601e80ab41c78a4d77d921942b0d8160bcab40ff894095e6823&X-Amz-SignedHeaders=host&x-id=PutObject`
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_private
  - https://github.com/huggingface/datasets/runs/7835921082?check_suite_focus=true
    `requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://hub-ci.huggingface.co/api/repos/create (Request ID: gL_1I7i2dii9leBhlZen-) - Internal Error - We're working hard to fix that as soon as possible!`
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_to_hub_custom_features_image_list
  - https://github.com/huggingface/datasets/runs/7835920900?check_suite_focus=true
  - This is not 500, but 404:
    `requests.exceptions.HTTPError: 404 Client Error: Not Found for url: [https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test-16605586458339.git/info/lfs/objects](https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test-16605586458339.git/info/lfs/objects/batch)`


_The documentation is not available anymore as the PR was closed or merged._
Add 'val' to  VALIDATION_KEYWORDS. 

This PR fixes #4839 by adding the word `"val"` to the `VALIDATION_KEYWORDS` so that the `load_dataset()` method with `imagefolder` (and probably, some other directives as well) reads folders named `"val"` as well.

I think the supported keywords have to be mentioned in the documentation as well, but I couldn't think of a proper place to add that.

@mariosasko not sure about how the reviewing process works. Maybe you can have a look because we discussed this elsewhere?

Hi, thanks! 

Let's add one pattern with `val` to this test before merging: 
https://github.com/huggingface/datasets/blob/b88a656cf94c4ad972154371c83c1af759fde522/tests/test_data_files.py#L598

_The documentation is not available anymore as the PR was closed or merged._

@akt42 note that there is some info about splits keywords in the docs: https://huggingface.co/docs/datasets/main/en/repository_structure#split-names-keywords. I agree it's not clear that it applies not only to filenames, but to directories as well.

I think "val" should be now added to the documentation source file here: https://github.com/huggingface/datasets/blob/main/docs/source/repository_structure.mdx?plain=1#L98

@polinaeterna Thanks for notifying us that there is a list of supported keywords

I've added "val" to that list and a test.
Dataset Viewer issue for darragh/demo_data_raw3

### Link

https://huggingface.co/datasets/darragh/demo_data_raw3

### Description

```
Exception:     ValueError
Message:       Arrow type extension<arrow.py_extension_type<pyarrow.lib.UnknownExtensionType>> does not have a datasets dtype equivalent.
```
reported by @NielsRogge 

### Owner

No

do you have an idea of why it can occur @huggingface/datasets? The dataset consists of a single parquet file.

Thanks for reporting @severo.

I'm not able to reproduce that error. I get instead:
```
FileNotFoundError: [Errno 2] No such file or directory: 'orix/data/ChiSig/唐合乐-9-3.jpg'
```

Which pyarrow version are you using? Mine is 6.0.1. 

OK, I get now your error when not streaming.

OK!

If it's useful, the pyarrow version is 7.0.0:

https://github.com/huggingface/datasets-server/blob/487c39d87998f8d5a35972f1027d6c8e588e622d/services/worker/poetry.lock#L1537-L1543

Apparently, there is something weird with that Parquet file: its schema is:
```
images: extension<arrow.py_extension_type<pyarrow.lib.UnknownExtensionType>>
```

I have forced a right schema:
```python
from datasets import Features, Image, load_dataset

features = Features({"images": Image()})
ds = datasets.load_dataset("parquet", split="train", data_files="train-00000-of-00001.parquet", features=features)
```
and then recreated a new Parquet file:
```python
ds.to_parquet("train.parquet")
```

Now this Parquet file has the right schema:
```
images: struct<bytes: binary, path: string>
  child 0, bytes: binary
  child 1, path: string
```
and can be loaded normally:
```python
In [26]: ds = load_dataset("parquet", split="train", data_files="dataset.parquet")
n [27]: ds
Out[27]: 
Dataset({
    features: ['images'],
    num_rows: 20
})
```
ImageFolder dataset builder does not read the validation data set if it is named as "val"

**Is your feature request related to a problem? Please describe.**

Currently, the `'imagefolder'` data set builder in [`load_dataset()`](https://github.com/huggingface/datasets/blob/2.4.0/src/datasets/load.py#L1541] ) only [supports](https://github.com/huggingface/datasets/blob/6c609a322da994de149b2c938f19439bca99408e/src/datasets/data_files.py#L31) the following names as the validation data set directory name: `["validation", "valid", "dev"]`. When the validation directory is named as `'val'`, the Data set will not have a validation split. I expected this to be a trivial task but ended up spending a lot of time before knowing that only the above names are supported.

Here's a minimal example of `val` not being recognized:

```python
import os
import numpy as np
import cv2   
from datasets import load_dataset

# creating a dummy data set with the following structure:

# ROOT
# | -- train
# | ---- class_1
# | ---- class_2
# | -- val
# | ---- class_1
# | ---- class_2


ROOT = "data"


for which in ["train", "val"]:
  for class_name in ["class_1", "class_2"]:
    dir_name = os.path.join(ROOT, which, class_name)
    if not os.path.exists(dir_name):
      os.makedirs(dir_name)
    for i in range(10):
       cv2.imwrite(
           os.path.join(dir_name, f"{i}.png"),
           np.random.random((224, 224))
           )

# trying to create a data set
dataset = load_dataset(
    "imagefolder", 
    data_dir=ROOT
)

>> dataset
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 20
    })
})

# ^ note how the dataset only has a 'train' subset

```

**Describe the solution you'd like**

The suggestion is to include `"val"` to  [that list ](https://github.com/huggingface/datasets/blob/6c609a322da994de149b2c938f19439bca99408e/src/datasets/data_files.py#L31) as that's a commonly used phrase to name the validation directory. 

Also, In the documentation, explicitly mention that only such directory names are supported as train/val/test directories to avoid confusion.

**Describe alternatives you've considered**

In the documentation, explicitly mention that only such directory names are supported as train/val/test directories without adding `val` to the above list.


**Additional context**

A question asked in the forum: [
Loading an imagenet-style image dataset with train/val directories](https://discuss.huggingface.co/t/loading-an-imagenet-style-image-dataset-with-train-val-directories/21554)

#take
Add support for CSV metadata files to ImageFolder

Fix #4814

_The documentation is not available anymore as the PR was closed or merged._

Cool thanks ! Maybe let's include this change after the refactoring from FolderBasedBuilder in #3963 to avoid dealing with too many unpleasant conflicts ?

@lhoestq I resolved the conflicts (AudioFolder also supports CSV metadata now). Let me know what you think.


@lhoestq Thanks for the suggestion! Indeed it makes more sense to use CSV as the default format in the folder-based builders.
Is it possible to pass multiple links to a split in load script?

**Is your feature request related to a problem? Please describe.**
I wanted to use a python loading script in hugging face datasets that use different sources of text (it's somehow a compilation of multiple datasets + my own dataset) based on how `load_dataset` [works](https://huggingface.co/docs/datasets/loading) I assumed I could do something like bellow in my loading script:

```python
...
_URL = "MY_DATASET_URL/resolve/main/data/"
_URLS = {
    "train": [
        "FIRST_URL_TO.txt",
        _URL + "train-00000-of-00001-676bfebbc8742592.parquet"
    ]
}
...
```
but when loading the dataset it raises the following error:
```python
File ~/.local/lib/python3.8/site-packages/datasets/builder.py:704, in DatasetBuilder.download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702         logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703 if not downloaded_from_gcs:
--> 704     self._download_and_prepare(
    705         dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
...
    668     if isinstance(a, str):
    669         # Force-cast str subclasses to str (issue #21127)
    670         parts.append(str(a))

TypeError: expected str, bytes or os.PathLike object, not list
```

**Describe the solution you'd like**
I believe since it's possible for `load_dataset` to get list of URLs instead of just a URL for `train` split it can be possible here too.

**Describe alternatives you've considered**
An alternative solution would be to download all needed datasets locally and `push_to_hub` them all, but since the datasets I'm talking about are huge it's not among my options. 

**Additional context**
I think loading `text` beside the `parquet` is completely a different issue but I believe I can figure it out by proposing a config for my dataset to load each entry of `_URLS['train']` separately either by `load_dataset("text", ...` or `load_dataset("parquet", ...`.
Fix missing tags in dataset cards

Fix missing tags in dataset cards:
- boolq
- break_data
- definite_pronoun_resolution
- emo
- kor_nli
- pg19
- quartz
- sciq
- squad_es
- wmt14
- wmt15
- wmt16
- wmt17
- wmt18
- wmt19
- wmt_t2t

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

_The documentation is not available anymore as the PR was closed or merged._
Misalignment between card tag validation and docs

## Describe the bug
As pointed out in other issue: https://github.com/huggingface/datasets/pull/4827#discussion_r943536284
the validation of the dataset card tags is not aligned with its documentation: e.g.
- implementation: `license: List[str]`
- docs: `license: Union[str, List[str]]`

They should be aligned.

CC: @julien-c 


(Note that the doc is aligned with the hub validation rules, and the "ground truth" is the hub validation rules given that they apply to all datasets, not just the canonical ones)

Instead of our own implementation, we now use `huggingface_hub`'s `DatasetCardData`, which has the correct type hint, so I think we can close this issue.
Support PIL Image objects in `add_item`/`add_column`

Fix #4796 

PS: We should also improve the type inference in `OptimizedTypeSequence` to make it possible to also infer the complex types (only `Image` currently) in nested arrays (e.g.  `[[pil_image], [pil_image, pil_image]]` or `[{"img": pil_image}`]), but I plan to address this in a separate PR.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4828). All of your documentation changes will be reflected on that endpoint.

Hey @mariosasko could we please merge this? I'm still getting the original error at #4796 .

Are you planning to continue working on this?
[Windows] Fix Access Denied when using os.rename()

In this PR, we are including an additional step when `os.rename()` raises a PermissionError.

Basically, we will use `shutil.move()` on the temp files.

Fix #2937 

Cool thank you ! Maybe we can just replace `os.rename` by `shutil.move` instead ?

> Cool thank you ! Maybe we can just replace `os.rename` by `shutil.move` instead ?

Yes, I think that could be a better solution, but I didn't test it in Linux (e.g. Ubuntu) to guarantee that `os.rename()` could be completely replaced by `shutil.move()`.

AFAIK `shutil.move` does call `os.rename` first before doing extra work to make it work on windows, so this is should be a safe safe change for linux ;)

> AFAIK `shutil.move` does call `os.rename` first before doing extra work to make it work on windows, so this is should be a safe safe change for linux ;)

alright, let me change the PR then.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4825). All of your documentation changes will be reflected on that endpoint.

Hi @lhoestq looks like one of the tests failed, but is not related to this change, do I need to do something from my side?
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.

Hi, when i try to run prepare_dataset function in [fine tuning ASR tutorial 4](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tuning_Wav2Vec2_for_English_ASR.ipynb) , i got this error.
I got this error
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
There is no other logs available, so i have no clue what is the cause of it.
```

def prepare_dataset(batch):
    audio = batch["path"]
    # batched output is "un-batched"
    batch["input_values"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
    batch["input_length"] = len(batch["input_values"])
    with processor.as_target_processor():
        batch["labels"] = processor(batch["text"]).input_ids
    return batch

data = data.map(prepare_dataset, remove_columns=data.column_names["train"],
                      num_proc=4)
```


Specify the actual results or traceback.
There is no traceback except
`Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.`
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-5.15.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.4.3


Fixed by installing either resampy<3 or resampy>=4
Add add cc-by-sa-2.5 license tag

- [ ] add it to moon-landing
- [ ] add it to hub-docs 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4818). All of your documentation changes will be reflected on that endpoint.

I think we can close this PR because the `standard_licenses.tsv` file was removed from this repo and we no longer perform any dataset card validation.
Fix loading example in opus dataset cards

This PR:
- fixes the examples to load the datasets, with the corrected dataset name, in their dataset cards for:
  - opus_dgt
  - opus_paracrawl
  - opus_wikipedia
- fixes their dataset cards with the missing required information: title, data instances/fields/splits
- enumerates the supported languages
- adds a missing citation reference for opus_wikipedia

Related to:
- #4806

_The documentation is not available anymore as the PR was closed or merged._
Fix bug in function validate_type for Python >= 3.9

Fix `validate_type` function, so that it uses `get_origin` instead. This makes the function forward compatible.

This fixes #4811 because:
```python
In [4]: typing.Optional[str]
Out[4]: typing.Optional[str]

In [5]: get_origin(typing.Optional[str])
Out[5]: typing.Union
```

Fix #4811.

_The documentation is not available anymore as the PR was closed or merged._
Bug in function validate_type for Python >= 3.9

## Describe the bug
The function `validate_type` assumes that the type `typing.Optional[str]` is automatically transformed to `typing.Union[str, NoneType]`.
```python
In [4]: typing.Optional[str]
Out[4]: typing.Union[str, NoneType]
```

However, this is not the case for Python 3.9:
```python
In [3]: typing.Optional[str]
Out[3]: typing.Optional[str]
```
Complete the mlqa dataset card

I fixed the issue #4808 

Details of PR:
- Added languages included in the dataset. 
- Added task id and task category. 
- Updated the citation information. 

Fix #4808.

_The documentation is not available anymore as the PR was closed or merged._

> Thanks for your contribution, @eldhoittangeorge.
> 
> The CI error message: https://github.com/huggingface/datasets/runs/7743526624?check_suite_focus=true
> 
> ```
> E           ValueError: The following issues have been found in the dataset cards:
> E           YAML tags:
> E           __init__() missing 5 required positional arguments: 'annotations_creators', 'language_creators', 'license', 'size_categories', and 'source_datasets'
> ```

I will fix the CI error.

@eldhoittangeorge, thanks again for all the fixes. Just a minor one before we can merge this PR: https://github.com/huggingface/datasets/runs/7744885754?check_suite_focus=true
```
E           YAML tags:
E           Could not validate the metadata, found the following errors:
E           * field 'language_creators':
E           	['unknown'] are not registered tags for 'language_creators', reference at https://github.com/huggingface/datasets/tree/main/src/datasets/utils/resources/creators.json
```

> 

Thanks, I updated the file. 
A small suggestion can you mention this link https://github.com/huggingface/datasets/tree/main/src/datasets/utils/resources/  in the contribution page. So that others will know the acceptable values for the tags.
Wrong example in opus_gnome dataset card

## Describe the bug
I found that [the example on opus_gone dataset ](https://github.com/huggingface/datasets/tree/main/datasets/opus_gnome#dataset-summary) doesn't work.

## Steps to reproduce the bug
```python
 load_dataset("gnome", lang1="it", lang2="pl")
```
`"gnome"` should be `"opus_gnome"`

## Expected results
```bash
100%
1/1 [00:00<00:00, 42.09it/s]
DatasetDict({
    train: Dataset({
        features: ['id', 'translation'],
        num_rows: 8368
    })
})
```

## Actual results
```bash
 Couldn't find 'gnome' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/main/datasets/gnome/gnome.py
```

## Environment info
- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-120-generic-x86_64-with-glibc2.27
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3
Support `pipeline` argument in inspect.py functions

**Is your feature request related to a problem? Please describe.**

The `wikipedia` dataset requires a `pipeline` argument to build the list of splits:

https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py#L937

But this is currently not supported in `get_dataset_config_info`:

https://github.com/huggingface/datasets/blob/main/src/datasets/inspect.py#L373-L375

which is called by other functions, e.g. `get_dataset_split_names`.

**Additional context**

The dataset viewer is not working out-of-the-box on `wikipedia` for this reason:

https://huggingface.co/datasets/wikipedia/viewer

<img width="637" alt="Capture d’écran 2022-08-08 à 12 01 16" src="https://user-images.githubusercontent.com/1676121/183461838-5330783b-0269-4ba7-a999-314cde2023d8.png">



Now: the preview (first-rows) works, but not the conversion to parquet. See https://huggingface.co/datasets/wikipedia/viewer/20220301.de/train

```
_split_generators() missing 1 required positional argument: 'pipeline'

Error code:   UnexpectedError
```
Fix fine classes in trec dataset

This PR:
- replaces the fine labels, so that there are 50 instead of 47
- once more labels are added, all they (fine and coarse) have been re-ordered, so that they align with the order in: https://cogcomp.seas.upenn.edu/Data/QA/QC/definition.html
- the feature names have been fixed: `fine_label` instead of `label-fine`
  - to sneak-case (underscores instead of hyphens)
  - words have been reordered

Fix #4790.

_The documentation is not available anymore as the PR was closed or merged._
video dataset loader/parser

you know how you can [use `load_dataset` with any arbitrary csv file](https://huggingface.co/docs/datasets/loading#csv)? and you can also [use it to load a local image dataset](https://huggingface.co/docs/datasets/image_load#local-files)?

could you please add functionality to load a video dataset? it would be really cool if i could point it to a bunch of video files and use pytorch to start looping through batches of videos. like if my batch size is 16, each sample in the batch is a frame from a video. i'm competing in the [minerl challenge](https://www.aicrowd.com/challenges/neurips-2022-minerl-basalt-competition), and it would be awesome to use the HF ecosystem.

Hi! We've just started discussing the video support in `datasets` (decoding backends, video feature type, etc.), so I believe we should have something tangible by the end of this year.

Also, if you have additional video features in mind that you would like to see, feel free to let us know

Coool thanks @mariosasko 

Hey @mariosasko, I was wondering if there's a way to load video data currently in the library? 
Alternatively is there a way I could hack it through the dataset.from_dict() method? I tried to hack it, but the issue I run into is that earlier I was doing a `cast_column()` call for the `Image` feature, but now I'm not sure about to do if I want the dataset to have the following keys when I call from_dict on it:
`{"caption":[list of text captions], "video_frames": [list of image lists with one image list corresponding to one video]}`

Maybe something like `cast_column("video_frames", List(Image))` ..
(This is assuming I have already extracted frames from video)
Torgo dataset creation


Hi @YingLi001, thanks for your proposal to add this dataset.

However, now we add datasets directly to the Hub (instead of our GitHub repository). You have the instructions in our docs: 
- [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)
- [Create a dataset card](https://huggingface.co/docs/datasets/dataset_card)
- [Share](https://huggingface.co/docs/datasets/share)

Feel free to ask if you need any additional support/help.
Missing MBPP splits

(@albertvillanova)
The [MBPP dataset on the Hub](https://huggingface.co/datasets/mbpp) has only a test split for both its "full" and its "sanitized" subset, while the [paper](https://arxiv.org/abs/2108.07732) states in subsection 2.1 regarding the full split:
> In the experiments described later in the paper, we hold out 10 problems for **few-shot prompting**, another 500 as our **test** dataset (which is used to evaluate both few-shot inference and fine-tuned models), 374 problems for **fine-tuning**, and the rest for **validation**.

If the dataset on the Hub should reproduce most closely what the original authors use, I guess this four-way split should be reflected. 

The paper doesn't explicitly state the task_id ranges of the splits, but the [GitHub readme](https://github.com/google-research/google-research/tree/master/mbpp) referenced in the paper specifies exact task_id ranges, although it misstates the total number of samples:
> We specify a train and test split to use for evaluation. Specifically:
> 
> * Task IDs 11-510 are used for evaluation.
> * Task IDs 1-10 and 511-1000 are used for training and/or prompting. We typically used 1-10 for few-shot prompting, although you can feel free to use any of the training examples.

I.e. the few-shot, train and validation splits are combined into one split, with a soft suggestion of using the first ten for few-shot prompting. It is not explicitly stated whether the 374 fine-tuning samples mentioned in the paper have task_id 511 to 784 or 601 to 974 or are randomly sampled from task_id 511 to 974.

Regarding the "sanitized" split the paper states the following:
> For evaluations involving the edited dataset, we perform comparisons with 100 problems that appear in both the original and edited dataset, using the same held out 10 problems for few-shot prompting and 374 problems for fine-tuning. 

The statement doesn't appear to be very precise, as among the 10 few-shot problems, those with task_id 1, 5 and 10 are not even part of the sanitized variant, and many from the task_id range from 511 to 974 are missing (e.g. task_id 511 to 553). I suppose the idea the task_id ranges for each split remain the same, even if some of the task_ids are not present. That would result in 7 few-shot, 257 test, 141 train and 22 validation examples in the sanitized split.

Thanks for reporting this as well, @stadlerb.

I suggest waiting for the answer of the data owners... 

@albertvillanova The first author of the paper responded to the upstream issue:
> Task IDs 11-510 are the 500 test problems. We use 90 problems (511-600) for validation and then remaining 374 for fine-tuning (601-974). The other problems can be used as desired, either for training or few-shot prompting (although this should be specified).

Thanks for the follow-up, @stadlerb.

Would you be willing to open a Pull Request to address this issue? :wink: 

Opened a [PR](https://github.com/huggingface/datasets/pull/4943) to implement this--lmk if you have any feedback
Add DocVQA

## Adding a Dataset
- **Name:** DocVQA
- **Description:** Document Visual Question Answering (DocVQA) seeks to inspire a “purpose-driven” point of view in Document Analysis and Recognition research, where the document content is extracted and used to respond to high-level tasks defined by the human consumers of this information. 
- **Paper:** https://arxiv.org/abs/2007.00398
- **Data:** https://www.docvqa.org/datasets/docvqa
- **Motivation:** Models like LayoutLM and Donut in the Transformers library are fine-tuned on DocVQA. Would be very handy to directly load this dataset from the hub.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).


Thanks for proposing, @NielsRogge.

Please, note this dataset requires registering in their website and their Terms and Conditions state we cannot distribute their URL:
```
1. You will NOT distribute the download URLs
...
```
Dataset Viewer issue for Team-PIXEL/rendered-wikipedia-english

### Link

https://huggingface.co/datasets/Team-PIXEL/rendered-wikipedia-english/viewer/rendered-wikipedia-en/train

### Description

The dataset can be loaded fine but the viewer shows this error:

```
Server Error
Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.
```

I'm guessing this is because I recently renamed the dataset. Based on related issues (e.g. https://github.com/huggingface/datasets/issues/4759) , is there something server-side that needs to be refreshed?

### Owner

Yes

Thanks for reporting. It's a known issue that should be fixed soon. Meanwhile, I had to manually trigger the dataset viewer. It's OK now.
Note that the extreme aspect ratio of the images generates another issue, that we're inspecting.