helmo/github-issues huggingface dataset:
Update `xor_tydi_qa` url to v1.1
State: closed | Author: changjonathanc (CONTRIBUTOR)

The dataset is updated and the old url no longer works. So I updated it.

I faced a bug while trying to fix this. Documenting the solution here. Maybe we can add it to the doc (`CONTRIBUTING.md` and `ADD_NEW_DATASET.md`).
> And to make the command work without the ExpectedMoreDownloadedFiles error, you just need to use the --ignore_verifications flag.
https://github.com/huggingface/datasets/issues/2076#issuecomment-803904366

Comments: Just noticed while 
```load_dataset('local_path/datastes/xor_tydi_qa')``` works,
```load_dataset('xor_tydi_qa')``` 
outputs an error: 
`
FileNotFoundError: Couldn't find file at https://nlp.cs.washington.edu/xorqa/XORQA_site/data/xor_dev_retrieve_eng_span.jsonl
`
(the old url)

I tired clearing the cache `.cache/huggingface/modules` and `.cache/huggingface/datasets`, didn't work.

Anyone know how to fix this? Thanks.

It seems like the error is not on your end. By default, the lib tries to download the version of the dataset script that matches the version of the lib, and that version of the script is, in your case, broken because the old URL no longer works. Once this PR gets merged, you can wait for the new release or set `script_version` to `"master"` in `load_dataset` to get the fixed version of the script.

@mariosasko Thanks! It works now.

Pasting the docstring here for reference.
```
    script_version (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset script to load:

        - For canonical datasets in the `huggingface/datasets` library like "squad", the default version of the module is the local version fo the lib.
          You can specify a different version from your local version of the lib (e.g. "master" or "1.2.0") but it might cause compatibility issues.
        - For community provided datasets like "lhoestq/squad" that have their own git repository on the Datasets Hub, the default version "main" corresponds to the "main" branch.
          You can specify a different version that the default "main" by using a commit sha or a git tag of the dataset repository.
```
Branch name didn't work, but commit sha works.

Regarding the issue you mentioned about the `--ignore_verifications` flag, I think we should actually change the current behavior of the `--save_infos` flag to make it ignore the verifications as well, so that you don't need to specific `--ignore_verifications` in this case.

@lhoestq I realized I forgot to change this:

https://github.com/huggingface/datasets/blob/fdbf5a97d3393f4a91e4cddcabe364029508f7ce/datasets/xor_tydi_qa/xor_tydi_qa.py#L72-L73

What should I do?

Oh indeed. Please open a PR to change this. This should be 1.1.0
dataset adversarial_qa has no answers in the "test" set
State: closed | Author: bjascob (NONE)
Labels: name: bug

## Describe the bug
When loading the adversarial_qa dataset the 'test' portion has no answers.  Only the 'train' and 'validation' portions do.  This occurs with all four of the configs ('adversarialQA', 'dbidaf', 'dbert', 'droberta')

## Steps to reproduce the bug
```
from   datasets import load_dataset
examples = load_dataset('adversarial_qa', 'adversarialQA', script_version="master")['test']
print('Loaded {:,} examples'.format(len(examples)))
has_answers = 0
for e in examples:
    if e['answers']['text']:
        has_answers += 1
print('{:,} have answers'.format(has_answers))
>>> Loaded 3,000 examples
>>> 0 have answers

examples = load_dataset('adversarial_qa', 'adversarialQA', script_version="master")['validation']
<...code above...>
>>> Loaded 3,000 examples
>>> 3,000 have answers
```

## Expected results
If 'test' is a valid dataset, it should have answers. Also note that all of the 'train' and 'validation' sets have answers, there are no "no answer" questions with this set (not sure if this is correct or not).

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.8.0-53-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyArrow version: 1.0.0



Comments: Hi ! I'm pretty sure that the answers are not made available for the test set on purpose because it is part of the DynaBench benchmark, for which you can submit your predictions on the website.
In any case we should mention this in the dataset card of this dataset.

Makes sense, but not intuitive for someone searching through the datasets.  Thanks for adding the note to clarify.
`yelp_polarity` is broken
State: closed | Author: JetRunner (CONTRIBUTOR)

![image](https://user-images.githubusercontent.com/22514219/120828150-c4a35b00-c58e-11eb-8083-a537cee4dbb3.png)


Comments: ```
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/script_runner.py", line 332, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 233, in <module>
    configs = get_confs(option)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 604, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 588, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 148, in get_confs
    builder_cls = nlp.load.import_main_class(module_path[0], dataset=True)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/datasets/load.py", line 85, in import_main_class
    module = importlib.import_module(module_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/sasha/.cache/huggingface/modules/datasets_modules/datasets/yelp_polarity/a770787b2526bdcbfc29ac2d9beb8e820fbc15a03afd3ebc4fb9d8529de57544/yelp_polarity.py", line 36, in <module>
    from datasets.tasks import TextClassification
```

Solved by updating the `nlpviewer`
Sentence Boundaries missing in Dataset: xtreme / udpos
State: closed | Author: cosmeowpawlitan (CONTRIBUTOR)
Labels: name: bug

I was browsing through annotation guidelines, as suggested by the datasets introduction.

The guidlines saids "There must be exactly one blank line after every sentence, including the last sentence in the file. Empty sentences are not allowed." in the [Sentence Boundaries and Comments section](https://universaldependencies.org/format.html#sentence-boundaries-and-comments)

But the sentence boundaries seems not to be represented by huggingface datasets features well. I found out that multiple sentence are concatenated together as a 1D array, without any delimiter.

PAN-x, which is another token classification subset from xtreme do represent the sentence boundary using a 2D array.

You may compare in PAN-x.en and udpos.English in the explorer:
 https://huggingface.co/datasets/viewer/?dataset=xtreme

Comments: Hi,

This is a known issue. More info on this issue can be found in #2061. If you are looking for an open-source contribution, there are step-by-step instructions in the linked issue that you can follow to fix it.

Closed by #2466.
Some tests hang on Windows
State: closed | Author: mariosasko (COLLABORATOR)
Labels: name: bug

Currently, several tests hang on Windows if the max path limit of 260 characters is not disabled. This happens due to the changes introduced by #2223 that cause an infinite loop in `WindowsFileLock` described in #2220.  This can be very tricky to debug, so I think now is a good time to address these issues/PRs. IMO throwing an error is too harsh, but maybe we can emit a warning in the top-level `__init__.py ` on startup if long paths are not enabled.


Comments: Hi ! That would be nice indeed to at least have a warning, since we don't handle the max path length limit.
Also if we could have an error instead of an infinite loop I'm sure windows users will appreciate that

Unfortunately, I know this problem very well... 😅 

I remember having proposed to throw an error instead of hanging in an infinite loop #2220: 60c7d1b6b71469599a27147a08100f594e7a3f84, 8c8ab60018b00463edf1eca500e434ff061546fc 
but @lhoestq told me:
> Note that the filelock module comes from this project that hasn't changed in years - while still being used by ten of thousands of projects:
https://github.com/benediktschmitt/py-filelock
> 
> Unless we have proper tests for this, I wouldn't recommend to change it

I opened an Issue requesting a warning/error at startup for that case: #2224

@albertvillanova Thanks for additional info on this issue.

Yes, I think the best option is to throw an error instead of suppressing it in a loop. I've considered 2 more options, but I don't really like them:
1. create a temporary file with a filename longer than 255 characters on import; if this fails, long paths are not enabled and raise a warning. I'm not sure about this approach because I don't like the idea of creating a temporary file on import for this purpose.
2. check if long paths are enabled with [this code](https://stackoverflow.com/a/46546731/14095927). As mentioned in the comment, this code relies on an undocumented function and Win10-specific.
add english language tags for ~100 datasets
State: closed | Author: VictorSanh (CONTRIBUTOR)

As discussed on Slack, I have manually checked for ~100 datasets that they have at least one subset in English. This information was missing so adding into the READMEs.

Note that I didn't check all the subsets so it's possible that some of the datasets have subsets in other languages than English...

Comments: Fixing the tags of all the datasets is out of scope for this PR so I'm merging even though the CI fails because of the missing tags
Remove `extended` field from dataset tagger
State: closed | Author: lewtun (MEMBER)
Labels: name: bug

## Describe the bug
While working on #2435 I used the [dataset tagger](https://huggingface.co/datasets/tagging/) to generate the missing tags for the YAML metadata of each README.md file. However, it seems that our CI raises an error when the `extended` field is included:

```
dataset_name = 'arcd'

    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
    def test_changed_dataset_card(dataset_name):
        card_path = repo_path / "datasets" / dataset_name / "README.md"
        assert card_path.exists()
        error_messages = []
        try:
            ReadMe.from_readme(card_path)
        except Exception as readme_error:
            error_messages.append(f"The following issues have been found in the dataset cards:\nREADME:\n{readme_error}")
        try:
            DatasetMetadata.from_readme(card_path)
        except Exception as metadata_error:
            error_messages.append(
                f"The following issues have been found in the dataset cards:\nYAML tags:\n{metadata_error}"
            )
    
        if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           __init__() got an unexpected keyword argument 'extended'

tests/test_dataset_cards.py:70: ValueError
```

Consider either removing this tag from the tagger or including it as part of the validation step in the CI.

cc @yjernite 

Comments: The tagger also doesn't insert the value for the `size_categories` field automatically, so this should be fixed too

Thanks for reporting. Indeed the `extended` tag doesn't exist. Not sure why we had that in the tagger.
The repo of the tagger is here if someone wants to give this a try: https://github.com/huggingface/datasets-tagging
Otherwise I can probably fix it next week

I've opened a PR on `datasets-tagging` to fix the issue 🚀 

thanks ! this is fixed now
Fix NQ features loading: reorder fields of features to match nested fields order in arrow data
State: closed | Author: lhoestq (MEMBER)

As mentioned in #2401, there is an issue when loading the features of `natural_questions` since the order of the nested fields in the features don't match. The order is important since it matters for the underlying arrow schema.

To fix that I re-order the features based on the arrow schema:

```python
inferred_features = Features.from_arrow_schema(arrow_table.schema)
self.info.features = self.info.features.reorder_fields_as(inferred_features)
assert self.info.features.type == inferred_features.type
```

The re-ordering is a recursive function. It takes into account that the `Sequence` feature type is a struct of list and not a list of struct.

Now it's possible to load `natural_questions` again :)
Update DatasetMetadata and ReadMe
State: closed | Author: gchhablani (CONTRIBUTOR)

This PR contains the changes discussed in #2395.

**Edit**:
In addition to those changes, I'll be updating the `ReadMe` as follows:

Currently, `Section` has separate parsing and validation error lists. In `.validate()`, we add these lists to the final lists and throw errors.

One way to make `ReadMe` consistent with `DatasetMetadata` and add a separate `.validate()` method is to throw separate parsing and validation errors. 

This way, we don't have to throw validation errors, but only parsing errors in `__init__ ()`. We can have an option in `__init__()` to suppress parsing errors so that an object is created for validation. Doing this will allow the user to get all the errors in one go.

In `test_dataset_cards` , we are already catching error messages and appending to a list. This can be done for `ReadMe()`  for parsing errors, and `ReadMe(...,suppress_errors=True); readme.validate()`  for validation, separately.

**Edit 2**:
The only parsing issue we have as of now is multiple headings at the same level with the same name. I assume this will happen very rarely, but it is still better to throw an error than silently pick one of them. It should be okay to separate it this way. 

Wdyt @lhoestq ?
Insert Extractive QA templates for SQuAD-like datasets
State: closed | Author: lewtun (MEMBER)

This PR adds task templates for 9 SQuAD-like templates with the following properties:

* 1 config
* A schema that matches the `squad` one (i.e. same column names, especially for the nested `answers` column because the current implementation does not support casting with mismatched columns. see #2434)
* Less than 20GB (my laptop can't handle more right now)

The aim of this PR is to provide a few datasets to experiment with the task template integration in other libraries / services. 

PR #2429 should be merged before this one.

cc @abhi1thakur 

Comments: hi @lhoestq @SBrandeis i've now added the missing YAML tags, so this PR should be good to go :)

urgh, the windows tests are failing because of encoding issues 😢 

```
dataset_name = 'squad_kor_v1'

    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
    def test_changed_dataset_card(dataset_name):
        card_path = repo_path / "datasets" / dataset_name / "README.md"
        assert card_path.exists()
        error_messages = []
        try:
            ReadMe.from_readme(card_path)
        except Exception as readme_error:
            error_messages.append(f"The following issues have been found in the dataset cards:\nREADME:\n{readme_error}")
        try:
            DatasetMetadata.from_readme(card_path)
        except Exception as metadata_error:
            error_messages.append(
                f"The following issues have been found in the dataset cards:\nYAML tags:\n{metadata_error}"
            )
    
        if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           README:
E           'charmap' codec can't decode byte 0x90 in position 2283: character maps to <undefined>
E           The following issues have been found in the dataset cards:
E           YAML tags:
E           'charmap' codec can't decode byte 0x90 in position 2283: character maps to <undefined>
```

Seems like the encoding issues on windows is also being tackled in #2418 - will see if this solves the problem in the current PR
Fix CI six installation on linux
State: closed | Author: lhoestq (MEMBER)

For some reason we end up with this error in the linux CI when running pip install .[tests]
```
pip._vendor.resolvelib.resolvers.InconsistentCandidate: Provided candidate AlreadyInstalledCandidate(six 1.16.0 (/usr/local/lib/python3.6/site-packages)) does not satisfy SpecifierRequirement('six>1.9'), SpecifierRequirement('six>1.9'), SpecifierRequirement('six>=1.11'), SpecifierRequirement('six~=1.15'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.5.2'), SpecifierRequirement('six>=1.9.0'), SpecifierRequirement('six>=1.11.0'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.6.1'), SpecifierRequirement('six>=1.9'), SpecifierRequirement('six>=1.5'), SpecifierRequirement('six<2.0'), SpecifierRequirement('six<2.0'), SpecifierRequirement('six'), SpecifierRequirement('six'), SpecifierRequirement('six~=1.15.0'), SpecifierRequirement('six'), SpecifierRequirement('six<2.0,>=1.6.1'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.5.2'), SpecifierRequirement('six>=1.9.0')
```
example CI failure here:
https://app.circleci.com/pipelines/github/huggingface/datasets/6200/workflows/b64fdec9-f9e6-431c-acd7-e9f2c440c568/jobs/38247

The main version requirement comes from tensorflow: `six~=1.15.0`
So I pinned the six version to this.
DuplicatedKeysError when trying to load adversarial_qa
State: closed | Author: hanss0n (NONE)
Labels: name: bug

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
dataset = load_dataset('adversarial_qa', 'adversarialQA')
```

## Expected results
The dataset should be loaded into memory

## Actual results

>DuplicatedKeysError: FAILURE TO GENERATE DATASET !
>Found duplicate Key: 4d3cb5677211ee32895ca9c66dad04d7152254d4
>Keys should be unique and deterministic in nature
>
>
>During handling of the above exception, another exception occurred:
>
>DuplicatedKeysError                       Traceback (most recent call last)
>
>/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
>    347         for hash, key in self.hkey_record:
>    348             if hash in tmp_record:
>--> 349                 raise DuplicatedKeysError(key)
>    350             else:
>    351                 tmp_record.add(hash)
>
>DuplicatedKeysError: FAILURE TO GENERATE DATASET !
>Found duplicate Key: 4d3cb5677211ee32895ca9c66dad04d7152254d4
>Keys should be unique and deterministic in nature

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.109+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyArrow version: 3.0.0


Comments: Thanks for reporting !
#2433 fixed the issue, thanks @mariosasko :)

We'll do a patch release soon of the library.
In the meantime, you can use the fixed version of adversarial_qa by adding `script_version="master"` in `load_dataset`
Add version-specific BibTeX
State: closed | Author: albertvillanova (MEMBER)

As pointed out by @lhoestq in #2411, after the creation of the Zenodo DOI for Datasets, a new BibTeX entry is created with each release.

This PR adds a version-specific BibTeX entry, besides the existing one which is generic for the project.

See version-specific BibTeX entry here: https://zenodo.org/record/4817769/export/hx#.YLSyd6j7RPY

Comments: Maybe we should only keep one citation ?
cc @thomwolf @yjernite 

For info:
- The one automatically generated by Zenodo is version-specific, and a new one will be generated after each release.
- Zenodo has also generated a project-specific DOI (they call it *Concept DOI* as opposed to *Version DOI*), but currently this only redirects to the DOI page of the latest version.
- All the information automatically generated by Zenodo can be corrected/customized if necessary.
  - If we decide to correct/update metadata, take into account that there are the following fields (among others): Authors, Contributors, Title, Description, Keywords, Additional Notes, License,...

According to Zenodo: https://help.zenodo.org/#versioning
> **Which DOI should I use in citations?**
> 
> You should normally always use the DOI for the specific version of your record in citations. This is to ensure that other researchers can access the exact research artefact you used for reproducibility. By default, Zenodo uses the specific version to generate citations.
> 
> You can use the Concept DOI representing all versions in citations when it is desirable to cite an evolving research artifact, without being specific about the version.

Thanks for the details ! As zenodo says we should probably just show the versioned DOI. And we can remove the old citation.

I have removed the old citation.

What about the new one? Should we customize it? I have fixed some author names (replaced nickname with first and family names). Note that the list of authors is created automatically by Zenodo from this list: https://github.com/huggingface/datasets/graphs/contributors
I do not know if this default automatic list of authors is what we want to show in the citation...
Rename QuestionAnswering template to QuestionAnsweringExtractive
State: closed | Author: lewtun (MEMBER)

Following the discussion with @thomwolf  in #2255, this PR renames the QA template to distinguish extractive vs abstractive QA. The abstractive template will be added in a future PR.

Comments: > I like having "extractive" in the name to make things explicit. However this creates an inconsistency with transformers.
> 
> See
> https://huggingface.co/transformers/task_summary.html#extractive-question-answering
> 
> But this is minor IMO and I'm ok with this renaming

yes i chose this convention because it allows us to match the `QuestionAnsweringXxx` naming and i think it's better to have `task_name-subtask_name` should auto-complete ever become part of the Hub :)
Fix Docstring Mistake: dataset vs. metric
State: closed | Author: PhilipMay (CONTRIBUTOR)

PR to fix #2412

Comments: IMO this PR is ready for review. I do not know why tests fail...

The CI fail is unrelated to this PR, and it has been fixed on master, merging :)

> I just have one comment: we use rouge, not rogue :p

Oops!

rebased on master