noamaanMulla-03/datasets-issues huggingface dataset:
Mark CI tests as xfail if Hub HTTP error

In order to make testing more robust (and avoid merges to master with red tests), we could mark tests as xfailed (instead of failed) when the Hub raises some temporary HTTP errors.

This PR:
- marks tests as xfailed only if the Hub raises a 500 error for:
  - test_upstream_hub
- makes pytest report the xfailed/xpassed tests.

More tests could also be marked if needed.

Examples of CI failures due to temporary Hub HTTP errors:
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files
  - https://github.com/huggingface/datasets/runs/7806855399?check_suite_focus=true
    `requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://hub-ci.huggingface.co/api/datasets/__DUMMY_TRANSFORMERS_USER__/test-16603108028233/commit/main (Request ID: aZeAQ5yLktoGHQYBcJ3zo)`
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_no_token
  - https://github.com/huggingface/datasets/runs/7840022996?check_suite_focus=true
    `requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://s3.us-east-1.amazonaws.com/lfs-staging.huggingface.co/repos/81/e3/81e3b831fa9bf23190ec041f26ef7ff6d6b71c1a937b8ec1ef1f1f05b508c089/caae596caa179cf45e7c9ac0c6d9a9cb0fe2d305291bfbb2d8b648ae26ed38b6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4N7VTDGOZQA2IKWK%2F20220815%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220815T144713Z&X-Amz-Expires=900&X-Amz-Signature=5ddddfe8ef2b0601e80ab41c78a4d77d921942b0d8160bcab40ff894095e6823&X-Amz-SignedHeaders=host&x-id=PutObject`
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_private
  - https://github.com/huggingface/datasets/runs/7835921082?check_suite_focus=true
    `requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://hub-ci.huggingface.co/api/repos/create (Request ID: gL_1I7i2dii9leBhlZen-) - Internal Error - We're working hard to fix that as soon as possible!`
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_to_hub_custom_features_image_list
  - https://github.com/huggingface/datasets/runs/7835920900?check_suite_focus=true
  - This is not 500, but 404:
    `requests.exceptions.HTTPError: 404 Client Error: Not Found for url: [https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test-16605586458339.git/info/lfs/objects](https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test-16605586458339.git/info/lfs/objects/batch)`



_The documentation is not available anymore as the PR was closed or merged._
Fix: wmt datasets - fix CWMT zh subsets

Fix https://github.com/huggingface/datasets/issues/4575

TODO: run `datasets-cli test`:
- [x] wmt17
- [x] wmt18
- [x] wmt19


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4871). All of your documentation changes will be reflected on that endpoint.
Document better that loading a dataset passing its name does not use the local script

As reported by @TrentBrick here https://github.com/huggingface/datasets/issues/4725#issuecomment-1191858596, it could be more clear that loading a dataset by passing its name does not use the (modified) local script of it.

What he did:
- he installed `datasets` from source
- he modified locally `datasets/the_pile/the_pile.py` loading script
- he tried to load it but using `load_dataset("the_pile")` instead of `load_dataset("datasets/the_pile")`
  - as explained here https://github.com/huggingface/datasets/issues/4725#issuecomment-1191040245:
    - the former does not use the local script, but instead it downloads a copy of `the_pile.py` from our GitHub, caches it locally (inside `~/.cache/huggingface/modules`) and uses that.

He suggests adding a more clear explanation about this. He suggests adding it maybe in [Installation > source](https://huggingface.co/docs/datasets/installation))

CC: @stevhliu 

Labels: name: documentation

Thanks for the feedback!

I think since this issue is closely related to loading, I can add a clearer explanation under [Load > local loading script](https://huggingface.co/docs/datasets/main/en/loading#local-loading-script).

That makes sense but I think having a line about it under https://huggingface.co/docs/datasets/installation#source the "source" header here would be useful. My mental model of `pip install -e .` does not include the fact that the source files aren't actually being used. 

Thanks for sharing your perspective. I think the `load_dataset` function is the only one that pulls from GitHub, and since this use-case is very specific, I don't think we need to include such a broad clarification in the Installation section.

Feel free to check out the linked PR and let me know if it needs any additional explanation 😊
Update local loading script docs

This PR clarifies the local loading script section to include how to load a dataset after you've modified the local loading script (closes #4732).

Labels: name: documentation

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4778). All of your documentation changes will be reflected on that endpoint.

I would rather have a section in the docs that explains how to modify the script of an existing dataset (`inspect_dataset` + modification + `load_dataset`) instead of focusing on the GH datasets bundled with the source (only applicable for devs).

Good idea! I went with @mariosasko's suggestion to use `inspect_dataset` instead of cloning a dataset repository since it's a good opportunity to show off more of the library's lesser-known functions if that's ok with everyone :)

One advantage of cloning the repo is that it fetches potential data files referenced inside a script using relative paths, so if we decide to use `inspect_dataset`, we should at least add a tip to explain this limitation and how to circumvent it.

Oh you're right. Calling `load_dataset` on the modified script without having the files that come with it is not ideal. I agree it should be `git clone` instead - and inspect is for inspection only ^^'
Add collection3 dataset


Labels: name: wontfix

Hi @pefimov. Thanks for you awesome work on this dataset contribution.

However, now we are using the Hub to add new datasets, instead of this GitHub repo. 

You could share this dataset under the appropriate Hub organization namespace. This way the dataset will be accessible using:
```python
ds = load_dataset("<org_namespace>/collection3")
```

You have the procedure documented in our online docs: 
- [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)
- [Share](https://huggingface.co/docs/datasets/share)

Moreover, datasets shared on the Hub no longer need the dummy data files.

Please, feel free to ping me if you need any further guidance/support. 

> However, now we are using the Hub to add new datasets, instead of this GitHub repo.
> 
> You could share this dataset under the appropriate Hub organization namespace. This way the dataset will be accessible using:
> 
> ```python
> ds = load_dataset("<org_namespace>/collection3")
> ```
> 
Hi @albertvillanova . Thank you for your response.

I thought that Collection3 is large and important dataset in Russian presented in 2016 but not represented in huggingface.

Also I am not related to authors or organisation of dataset

The current policy of sharing datasets on the Hub instead of in this GitHub repo has no relation with the importance of the dataset: https://huggingface.co/docs/datasets/share#datasets-on-github-legacy 
> The distinction between a Hub dataset and a dataset from GitHub only comes from the legacy sharing workflow. It does not involve any ranking, decisioning, or opinion regarding the contents of the dataset itself.

It is not required to be an author/owner (or belong to the organization that is owner) of the dataset in order to share it on the Hub (as it was not the case when sharing them on this GitHub repo). 

It is recommended to share it under an organization namespace that makes sense though. For this specific dataset, do you know of a clear organization under which it could be shared on the Hub? Maybe "labinform", or "Information Research Laboratory" or "Lomonosov Moscow State University"?

In cases like this, where the org is not evident, one possibility could be to contact the dataset owners/creators and ask them. According the publication paper, the authors are:
- V.A. Mozharova
- N.V. Loukachevitch

I think maybe it would be worth contacting them.

@pefimov I have contacted the authors (and put you in CC).

Reply from the authors:
> It is better to use name: Research Computing Center of Lomonosov Moscow State University (short name RCC-MSU)
> https://rcc.msu.ru/en

I have created the corresponding org namespace and dataset empty repository: https://huggingface.co/datasets/RCC-MSU/collection3

@pefimov feel free to open a PR on the Hub if you are willing to do so: 
- Go to the *Community* tab on the repo: https://huggingface.co/datasets/RCC-MSU/collection3/discussions
- And click: *New pull request* button

Docs: [Pull requests and Discussions](https://huggingface.co/docs/hub/repositories-pull-requests-discussions) on the Hub

Thanks
Added names of less-studied languages

Added names of less-studied languages (nru – Narua and jya – Japhug) for existing datasets.


OK, I removed Glottolog codes and only added ISO 639-3 ones. The former are for the moment in corpus card description, language details, and in subcorpora names.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4880). All of your documentation changes will be reflected on that endpoint.
[Windows] Fix Access Denied when using os.rename()

In this PR, we are including an additional step when `os.rename()` raises a PermissionError.

Basically, we will use `shutil.move()` on the temp files.

Fix #2937 


Cool thank you ! Maybe we can just replace `os.rename` by `shutil.move` instead ?

> Cool thank you ! Maybe we can just replace `os.rename` by `shutil.move` instead ?

Yes, I think that could be a better solution, but I didn't test it in Linux (e.g. Ubuntu) to guarantee that `os.rename()` could be completely replaced by `shutil.move()`.

AFAIK `shutil.move` does call `os.rename` first before doing extra work to make it work on windows, so this is should be a safe safe change for linux ;)

> AFAIK `shutil.move` does call `os.rename` first before doing extra work to make it work on windows, so this is should be a safe safe change for linux ;)

alright, let me change the PR then.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4825). All of your documentation changes will be reflected on that endpoint.

Hi @lhoestq looks like one of the tests failed, but is not related to this change, do I need to do something from my side?
Fix language tags resource file

This PR fixes/updates/adds ALL language tags from IANA (as of 2022-08-08).

This PR also removes all BCP47 suffixes (the languages file only contains language subtags, i.e. ISO 639 1 or 2 codes; no script/region/variant suffixes). See:
- #4753


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4882). All of your documentation changes will be reflected on that endpoint.
Windows CI is unable to test streaming properly because of SSL issues

In https://github.com/huggingface/datasets/pull/3041 the windows tests were skipped because of SSL issues with moon-staging.huggingface.co:443

The issue appears only on windows with asyncio. On Linux it works. With requests it works as well. And with the production environment huggingface.co it also works.

to reproduce on windows:
```python
import fsspec

# use any URL to a file in a dataset repo
url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"

fsspec.open(url).open()
```
raises
```python
FileNotFoundError: https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes
```
because of
```python
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host moon-staging.huggingface.co:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')]
```

Labels: name: streaming

I think this problem is already fixed:
```python
In [4]: import fsspec
   ...:
   ...: url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"
   ...:
   ...: fsspec.open(url).open()
Out[4]: <File-like object HTTPFileSystem, https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattribu
```

No I'm still having this issue on my windows, and so does the CI
Add citation to ro_sts and ro_sts_parallel datasets

This PR adds the citation information to `ro_sts_parallel` and `ro_sts_parallel` datasets, once they have replied our request for that information:
- https://github.com/dumitrescustefan/RO-STS/issues/4


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4892). All of your documentation changes will be reflected on that endpoint.
Document better when relative paths are transformed to URLs

As discussed with @ydshieh, when passing a relative path as `data_dir` to `load_dataset` of a dataset hosted on the Hub, the relative path is transformed to the corresponding URL of the Hub dataset.

Currently, we mention this in our docs here: [Create a dataset loading script > Download data files and organize splits](https://huggingface.co/docs/datasets/v2.4.0/en/dataset_script#download-data-files-and-organize-splits)
> If the data files live in the same folder or repository of the dataset script, you can just pass the relative paths to the files instead of URLs.

Maybe we should document better how relative paths are handled, not only when creating a dataset loading script, but also when passing to `load_dataset`:
- `data_dir`
- `data_files`

CC: @stevhliu 

Labels: name: documentation
Document loading from relative path

This PR describes loading a dataset from the Hub by specifying a relative path in `data_dir` or `data_files` in `load_dataset` (see #4757).

Labels: name: documentation

_The documentation is not available anymore as the PR was closed or merged._

Thanks for the feedback!

I agree that adding it to `load_hub.mdx` is probably a bit too specific, especially for beginners reading the tutorials. Since this clarification is closely related to loading from the Hub (the only difference being the presence/absence of a loading script), I think it makes the most sense to keep it somewhere in `loading.mdx`. What do you think about adding a Warning in Loading >>> Hugging Face Hub that explains the difference between relative/absolute paths when there is a script?

What about updating the section about "manual download" ? I think it goes there no ?

https://huggingface.co/docs/datasets/v2.4.0/en/loading#manual-download

Updated the manual download section :)

Thanks ! Pinging @albertvillanova to review this change, and then I think we're good to merge
Re-add code and und language tags

This PR fixes the removal of 2 language tags done by:
- #4882

The tags are:
- "code": this is not a IANA tag but needed
- "und": this is one of the special scoped tags removed by 0d53202b9abce6fd0358cb00d06fcfd904b875af
  - used in "mc4" and "udhr" datasets


_The documentation is not available anymore as the PR was closed or merged._
Update GLUE evaluation metadata

This PR updates the evaluation metadata for GLUE to:

* Include defaults for all configs except `ax` (which only has a `test` split with no known labels)
* Fix the default split from `test` to `validation` since `test` splits in GLUE have no labels (they're private)
* Fix the `task_id` for some existing defaults

cc @sashavor @douwekiela 


_The documentation is not available anymore as the PR was closed or merged._
[LibriSpeech] Fix dev split local_extracted_archive for 'all' config

We define the keys for the `_DL_URLS` of the dev split as `dev.clean` and `dev.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L60-L61

These keys get forwarded to the `dl_manager` and thus the `local_extracted_archive`.

However, when calling `SplitGenerator` for the dev sets, we query the `local_extracted_archive` keys `validation.clean` and `validation.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L212
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L219

The consequence of this is that the `local_extracted_archive` arg passed to `_generate_examples` is always `None`, as the keys `validation.clean` and `validation.other` do not exists in the `local_extracted_archive`.

When defining the `audio_file` in `_generate_examples`, since `local_extracted_archive` is always `None`, we always omit the `local_extracted_archive` path from the `audio_file` path, **even** if in non-streaming mode:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L259-L263

Thus, `audio_file` will only ever be the streaming path (`audio_file`, not `os.path.join(local_extracted_archive, audio_file)`).

This PR fixes the `.get()` keys for the `local_extracted_archive` for the dev splits.



_The documentation is not available anymore as the PR was closed or merged._

This PR fixes a bug introduced in:
- #4184
ImageFolder dataset builder does not read the validation data set if it is named as "val"

**Is your feature request related to a problem? Please describe.**

Currently, the `'imagefolder'` data set builder in [`load_dataset()`](https://github.com/huggingface/datasets/blob/2.4.0/src/datasets/load.py#L1541] ) only [supports](https://github.com/huggingface/datasets/blob/6c609a322da994de149b2c938f19439bca99408e/src/datasets/data_files.py#L31) the following names as the validation data set directory name: `["validation", "valid", "dev"]`. When the validation directory is named as `'val'`, the Data set will not have a validation split. I expected this to be a trivial task but ended up spending a lot of time before knowing that only the above names are supported.

Here's a minimal example of `val` not being recognized:

```python
import os
import numpy as np
import cv2   
from datasets import load_dataset

# creating a dummy data set with the following structure:

# ROOT
# | -- train
# | ---- class_1
# | ---- class_2
# | -- val
# | ---- class_1
# | ---- class_2


ROOT = "data"


for which in ["train", "val"]:
  for class_name in ["class_1", "class_2"]:
    dir_name = os.path.join(ROOT, which, class_name)
    if not os.path.exists(dir_name):
      os.makedirs(dir_name)
    for i in range(10):
       cv2.imwrite(
           os.path.join(dir_name, f"{i}.png"),
           np.random.random((224, 224))
           )

# trying to create a data set
dataset = load_dataset(
    "imagefolder", 
    data_dir=ROOT
)

>> dataset
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 20
    })
})

# ^ note how the dataset only has a 'train' subset

```

**Describe the solution you'd like**

The suggestion is to include `"val"` to  [that list ](https://github.com/huggingface/datasets/blob/6c609a322da994de149b2c938f19439bca99408e/src/datasets/data_files.py#L31) as that's a commonly used phrase to name the validation directory. 

Also, In the documentation, explicitly mention that only such directory names are supported as train/val/test directories to avoid confusion.

**Describe alternatives you've considered**

In the documentation, explicitly mention that only such directory names are supported as train/val/test directories without adding `val` to the above list.


**Additional context**

A question asked in the forum: [
Loading an imagenet-style image dataset with train/val directories](https://discuss.huggingface.co/t/loading-an-imagenet-style-image-dataset-with-train-val-directories/21554)

Labels: name: enhancement

#take
Add 'val' to  VALIDATION_KEYWORDS. 

This PR fixes #4839 by adding the word `"val"` to the `VALIDATION_KEYWORDS` so that the `load_dataset()` method with `imagefolder` (and probably, some other directives as well) reads folders named `"val"` as well.

I think the supported keywords have to be mentioned in the documentation as well, but I couldn't think of a proper place to add that.


@mariosasko not sure about how the reviewing process works. Maybe you can have a look because we discussed this elsewhere?

Hi, thanks! 

Let's add one pattern with `val` to this test before merging: 
https://github.com/huggingface/datasets/blob/b88a656cf94c4ad972154371c83c1af759fde522/tests/test_data_files.py#L598

_The documentation is not available anymore as the PR was closed or merged._

@akt42 note that there is some info about splits keywords in the docs: https://huggingface.co/docs/datasets/main/en/repository_structure#split-names-keywords. I agree it's not clear that it applies not only to filenames, but to directories as well.

I think "val" should be now added to the documentation source file here: https://github.com/huggingface/datasets/blob/main/docs/source/repository_structure.mdx?plain=1#L98

@polinaeterna Thanks for notifying us that there is a list of supported keywords

I've added "val" to that list and a test.
None Type error for swda datasets

## Describe the bug
I got `'NoneType' object is not callable` error while calling the swda datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("swda")
```

## Expected results
Run without error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Python version: 3.8.10


Labels: name: bug

Thanks for reporting @hannan72 ! I couldn't reproduce the error on my side, can you share the full stack trace please ?

Thanks a lot for your response @lhoestq 
The problem is solved accidentally today and I don't know exactly why it was happened yesterday.
The issue can be closed.

Ok, let us know if you encounter the issue again ;)
Invalid download URL of WMT17 `zh-en` data 

## Describe the bug
Partial data (wmt17 zh-en) cannot be downloaded due to an invalid URL.

## Steps to reproduce the bug
```python
from datasets import load_dataset
 dataset = load_dataset('wmt17','zh-en')
```

## Expected results
ConnectionError: Couldn't reach ftp://cwmt-wmt:cwmt-wmt@datasets.nju.edu.cn/parallel/casia2015.zip

Labels: name: bug

name: dataset bug

Fixed in the current release. Close this issue.