jamesLeeeeeee/datasets-github-issues huggingface dataset:
`Proto_qa` hosting seems to be broken
Author: VictorSanh | Pull request: no | State: closed

## Describe the bug
The hosting (on Github) of the `proto_qa` dataset seems broken. I haven't investigated more yet, just flagging it for now. 

@zaidalyafeai if you want to dive into it, I think it's just a matter of changing the links in `proto_qa.py`

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("proto_qa")
```

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/load.py", line 751, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 630, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/hf/.cache/huggingface/modules/datasets_modules/datasets/proto_qa/445346efaad5c5f200ecda4aa7f0fb50ff1b55edde3003be424a2112c3e8102e/proto_qa.py", line 131, in _split_generators
    train_fpath = dl_manager.download(_URLs[self.config.name]["train"])
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 199, in download
    num_proc=download_config.num_proc,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 195, in map_nested
    return function(data_struct)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 218, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/iesl/protoqa-data/master/data/train/protoqa_train.jsonl
```
Revert default in-memory for small datasets
Author: albertvillanova | Pull request: no | State: closed

Users are reporting issues and confusion about setting default in-memory to True for small datasets.

We see 2 clear use cases of Datasets:
- the "canonical" way, where you can work with very large datasets, as they are memory-mapped and cached (after every transformation)
- some edge cases (speed benchmarks, interactive/exploratory analysis,...), where default in-memory can explicitly be enabled, and no caching will be done

After discussing with @lhoestq we have agreed to:
- revert this feature (implemented in #2182)
- explain in the docs how to optimize speed/performance by setting default in-memory

cc: @stas00 https://github.com/huggingface/datasets/pull/2409#issuecomment-856210552
Add align_labels_with_mapping function
Author: lewtun | Pull request: yes | State: closed

This PR adds a helper function to align the `label2id` mapping between a `datasets.Dataset` and a classifier (e.g. a transformer with a `PretrainedConfig.label2id` dict), with the alignment performed on the dataset itself.

This will help us with the Hub evaluation, where we won't know in advance whether a model that is fine-tuned on say MNLI has the same mappings as the MNLI dataset we load from `datasets`.

An example where this is needed is if we naively try to evaluate `microsoft/deberta-base-mnli` on `mnli` because the model config has the following mappings:

```python
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  },
  "label2id": {
    "CONTRADICTION": 0,
    "ENTAILMENT": 2,
    "NEUTRAL": 1
  }
```

while the `mnli` dataset has the `contradiction` and `neutral` labels swapped:

```python
id2label = {0: 'entailment', 1: 'neutral', 2: 'contradiction'}
label2id = {'contradiction': 2, 'entailment': 0, 'neutral': 1}
```

As a result, we get a much lower accuracy during evaluation:

```python
from datasets import load_dataset
from transformers.trainer_utils import EvalPrediction
from transformers import AutoModelForSequenceClassification, Trainer

# load dataset for evaluation
mnli = load_dataset("glue", "mnli", split="test")
# load model
model_ckpt = "microsoft/deberta-base-mnli"
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
# preprocess, create trainer ...
mnli_enc = ...
trainer = Trainer(model, args=args, tokenizer=tokenizer)
# generate preds
preds = trainer.predict(mnli_enc)
# preds.label_ids misalinged with model.config => returns wrong accuracy (too low)!
compute_metrics(EvalPrediction(preds.predictions, preds.label_ids))
```

The fix is to use the helper function before running the evaluation to make sure the label IDs are aligned:

```python
mnli_enc_aligned = mnli_enc.align_labels_with_mapping(label2id=config.label2id, label_column="label")
# preds now aligned and everyone is happy :)
preds = trainer.predict(mnli_enc_aligned)
```

cc @thomwolf @lhoestq
MRPC test set differences between torch and tensorflow datasets
Author: FredericOdermatt | Pull request: no | State: closed

## Describe the bug
When using `load_dataset("glue", "mrpc")` to load the MRPC dataset, the test set includes the labels. When using `tensorflow_datasets.load('glue/{}'.format('mrpc'))` to load the dataset the test set does not contain the labels. There should be consistency between torch and tensorflow ways of importing the GLUE datasets.

## Steps to reproduce the bug

Minimal working code  
```python
from datasets import load_dataset
import tensorflow as tf
import tensorflow_datasets

# torch
dataset = load_dataset("glue", "mrpc")
# tf
data = tensorflow_datasets.load('glue/{}'.format('mrpc'))
data = list(data['test'].as_numpy_iterator())
for i in range(40,50):
  tf_sentence1 = data[i]['sentence1'].decode("utf-8") 
  tf_sentence2 = data[i]['sentence2'].decode("utf-8") 

  tf_label = data[i]['label']
  
  index = data[i]['idx']
  print('Index {}'.format(index))
  torch_sentence1 = dataset['test']['sentence1'][index]
  torch_sentence2 = dataset['test']['sentence2'][index]

  torch_label = dataset['test']['label'][index]
  print('Tensorflow: \n\tSentence1 {}\n\tSentence2 {}\n\tLabel {}'.format(tf_sentence1, tf_sentence2, tf_label))
  print('Torch: \n\tSentence1 {}\n\tSentence2 {}\n\tLabel {}'.format(torch_sentence1, torch_sentence2, torch_label))
```

Sample output  
```
Index 954
Tensorflow: 
	Sentence1 Sabri Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate yesterday on charges of violating U.S. arms-control laws .
	Sentence2 The elder Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate Wednesday on charges of violating U.S. arms control laws .
	Label -1
Torch: 
	Sentence1 Sabri Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate yesterday on charges of violating U.S. arms-control laws .
	Sentence2 The elder Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate Wednesday on charges of violating U.S. arms control laws .
	Label 1
Index 711
Tensorflow: 
	Sentence1 Others keep records sealed for as little as five years or as much as 30 .
	Sentence2 Some states make them available immediately ; others keep them sealed for as much as 30 years .
	Label -1
Torch: 
	Sentence1 Others keep records sealed for as little as five years or as much as 30 .
	Sentence2 Some states make them available immediately ; others keep them sealed for as much as 30 years .
	Label 0
```

## Expected results
I would expect the datasets to be independent of whether I am working with torch or tensorflow.

## Actual results
Test set labels are provided in the `datasets.load_datasets()` for MRPC. However MRPC is the only task where the test set labels are not -1.

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.109+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyArrow version: 3.0.0
BLUE file not found
Author: mirfan899 | Pull request: no | State: closed

Hi, I'm having the following issue when I try to load the `blue` metric.

```shell
import datasets
metric = datasets.load_metric('blue')
Traceback (most recent call last):
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 320, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.7.0/metrics/blue/blue.py
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 332, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/metrics/blue/blue.py
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 605, in load_metric
    dataset=False,
  File "/home/irfan/environments/Perplexity_Transformers/lib/python3.6/site-packages/datasets/load.py", line 343, in prepare_module
    combined_path, github_file_path
FileNotFoundError: Couldn't find file locally at blue/blue.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.7.0/metrics/blue/blue.py.
The file is also not present on the master branch on github.
```
Here is dataset installed version info
```shell
pip freeze | grep datasets
datasets==1.7.0
```
Update `xor_tydi_qa` url to v1.1
Author: cccntu | Pull request: yes | State: closed

The dataset is updated and the old url no longer works. So I updated it.

I faced a bug while trying to fix this. Documenting the solution here. Maybe we can add it to the doc (`CONTRIBUTING.md` and `ADD_NEW_DATASET.md`).
> And to make the command work without the ExpectedMoreDownloadedFiles error, you just need to use the --ignore_verifications flag.
https://github.com/huggingface/datasets/issues/2076#issuecomment-803904366
dataset adversarial_qa has no answers in the "test" set
Author: bjascob | Pull request: no | State: closed

## Describe the bug
When loading the adversarial_qa dataset the 'test' portion has no answers.  Only the 'train' and 'validation' portions do.  This occurs with all four of the configs ('adversarialQA', 'dbidaf', 'dbert', 'droberta')

## Steps to reproduce the bug
```
from   datasets import load_dataset
examples = load_dataset('adversarial_qa', 'adversarialQA', script_version="master")['test']
print('Loaded {:,} examples'.format(len(examples)))
has_answers = 0
for e in examples:
    if e['answers']['text']:
        has_answers += 1
print('{:,} have answers'.format(has_answers))
>>> Loaded 3,000 examples
>>> 0 have answers

examples = load_dataset('adversarial_qa', 'adversarialQA', script_version="master")['validation']
<...code above...>
>>> Loaded 3,000 examples
>>> 3,000 have answers
```

## Expected results
If 'test' is a valid dataset, it should have answers. Also note that all of the 'train' and 'validation' sets have answers, there are no "no answer" questions with this set (not sure if this is correct or not).

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.8.0-53-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyArrow version: 1.0.0
Sentence Boundaries missing in Dataset: xtreme / udpos
Author: jerryIsHere | Pull request: no | State: closed

I was browsing through annotation guidelines, as suggested by the datasets introduction.

The guidlines saids "There must be exactly one blank line after every sentence, including the last sentence in the file. Empty sentences are not allowed." in the [Sentence Boundaries and Comments section](https://universaldependencies.org/format.html#sentence-boundaries-and-comments)

But the sentence boundaries seems not to be represented by huggingface datasets features well. I found out that multiple sentence are concatenated together as a 1D array, without any delimiter.

PAN-x, which is another token classification subset from xtreme do represent the sentence boundary using a 2D array.

You may compare in PAN-x.en and udpos.English in the explorer:
 https://huggingface.co/datasets/viewer/?dataset=xtreme
Some tests hang on Windows
Author: mariosasko | Pull request: no | State: closed

Currently, several tests hang on Windows if the max path limit of 260 characters is not disabled. This happens due to the changes introduced by #2223 that cause an infinite loop in `WindowsFileLock` described in #2220.  This can be very tricky to debug, so I think now is a good time to address these issues/PRs. IMO throwing an error is too harsh, but maybe we can emit a warning in the top-level `__init__.py ` on startup if long paths are not enabled.
Remove `extended` field from dataset tagger
Author: lewtun | Pull request: no | State: closed

## Describe the bug
While working on #2435 I used the [dataset tagger](https://huggingface.co/datasets/tagging/) to generate the missing tags for the YAML metadata of each README.md file. However, it seems that our CI raises an error when the `extended` field is included:

```
dataset_name = 'arcd'

    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
    def test_changed_dataset_card(dataset_name):
        card_path = repo_path / "datasets" / dataset_name / "README.md"
        assert card_path.exists()
        error_messages = []
        try:
            ReadMe.from_readme(card_path)
        except Exception as readme_error:
            error_messages.append(f"The following issues have been found in the dataset cards:\nREADME:\n{readme_error}")
        try:
            DatasetMetadata.from_readme(card_path)
        except Exception as metadata_error:
            error_messages.append(
                f"The following issues have been found in the dataset cards:\nYAML tags:\n{metadata_error}"
            )
    
        if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           __init__() got an unexpected keyword argument 'extended'

tests/test_dataset_cards.py:70: ValueError
```

Consider either removing this tag from the tagger or including it as part of the validation step in the CI.

cc @yjernite
Fix NQ features loading: reorder fields of features to match nested fields order in arrow data
Author: lhoestq | Pull request: yes | State: closed

As mentioned in #2401, there is an issue when loading the features of `natural_questions` since the order of the nested fields in the features don't match. The order is important since it matters for the underlying arrow schema.

To fix that I re-order the features based on the arrow schema:

```python
inferred_features = Features.from_arrow_schema(arrow_table.schema)
self.info.features = self.info.features.reorder_fields_as(inferred_features)
assert self.info.features.type == inferred_features.type
```

The re-ordering is a recursive function. It takes into account that the `Sequence` feature type is a struct of list and not a list of struct.

Now it's possible to load `natural_questions` again :)
Update DatasetMetadata and ReadMe
Author: gchhablani | Pull request: yes | State: closed

This PR contains the changes discussed in #2395.

**Edit**:
In addition to those changes, I'll be updating the `ReadMe` as follows:

Currently, `Section` has separate parsing and validation error lists. In `.validate()`, we add these lists to the final lists and throw errors.

One way to make `ReadMe` consistent with `DatasetMetadata` and add a separate `.validate()` method is to throw separate parsing and validation errors. 

This way, we don't have to throw validation errors, but only parsing errors in `__init__ ()`. We can have an option in `__init__()` to suppress parsing errors so that an object is created for validation. Doing this will allow the user to get all the errors in one go.

In `test_dataset_cards` , we are already catching error messages and appending to a list. This can be done for `ReadMe()`  for parsing errors, and `ReadMe(...,suppress_errors=True); readme.validate()`  for validation, separately.

**Edit 2**:
The only parsing issue we have as of now is multiple headings at the same level with the same name. I assume this will happen very rarely, but it is still better to throw an error than silently pick one of them. It should be okay to separate it this way. 

Wdyt @lhoestq ?
Insert Extractive QA templates for SQuAD-like datasets
Author: lewtun | Pull request: yes | State: closed

This PR adds task templates for 9 SQuAD-like templates with the following properties:

* 1 config
* A schema that matches the `squad` one (i.e. same column names, especially for the nested `answers` column because the current implementation does not support casting with mismatched columns. see #2434)
* Less than 20GB (my laptop can't handle more right now)

The aim of this PR is to provide a few datasets to experiment with the task template integration in other libraries / services. 

PR #2429 should be merged before this one.

cc @abhi1thakur
Fix CI six installation on linux
Author: lhoestq | Pull request: yes | State: closed

For some reason we end up with this error in the linux CI when running pip install .[tests]
```
pip._vendor.resolvelib.resolvers.InconsistentCandidate: Provided candidate AlreadyInstalledCandidate(six 1.16.0 (/usr/local/lib/python3.6/site-packages)) does not satisfy SpecifierRequirement('six>1.9'), SpecifierRequirement('six>1.9'), SpecifierRequirement('six>=1.11'), SpecifierRequirement('six~=1.15'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.5.2'), SpecifierRequirement('six>=1.9.0'), SpecifierRequirement('six>=1.11.0'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.6.1'), SpecifierRequirement('six>=1.9'), SpecifierRequirement('six>=1.5'), SpecifierRequirement('six<2.0'), SpecifierRequirement('six<2.0'), SpecifierRequirement('six'), SpecifierRequirement('six'), SpecifierRequirement('six~=1.15.0'), SpecifierRequirement('six'), SpecifierRequirement('six<2.0,>=1.6.1'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.5.2'), SpecifierRequirement('six>=1.9.0')
```
example CI failure here:
https://app.circleci.com/pipelines/github/huggingface/datasets/6200/workflows/b64fdec9-f9e6-431c-acd7-e9f2c440c568/jobs/38247

The main version requirement comes from tensorflow: `six~=1.15.0`
So I pinned the six version to this.
DuplicatedKeysError when trying to load adversarial_qa
Author: hanss0n | Pull request: no | State: closed

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
dataset = load_dataset('adversarial_qa', 'adversarialQA')
```

## Expected results
The dataset should be loaded into memory

## Actual results

>DuplicatedKeysError: FAILURE TO GENERATE DATASET !
>Found duplicate Key: 4d3cb5677211ee32895ca9c66dad04d7152254d4
>Keys should be unique and deterministic in nature
>
>
>During handling of the above exception, another exception occurred:
>
>DuplicatedKeysError                       Traceback (most recent call last)
>
>/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
>    347         for hash, key in self.hkey_record:
>    348             if hash in tmp_record:
>--> 349                 raise DuplicatedKeysError(key)
>    350             else:
>    351                 tmp_record.add(hash)
>
>DuplicatedKeysError: FAILURE TO GENERATE DATASET !
>Found duplicate Key: 4d3cb5677211ee32895ca9c66dad04d7152254d4
>Keys should be unique and deterministic in nature

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.109+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyArrow version: 3.0.0
Add version-specific BibTeX
Author: albertvillanova | Pull request: yes | State: closed

As pointed out by @lhoestq in #2411, after the creation of the Zenodo DOI for Datasets, a new BibTeX entry is created with each release.

This PR adds a version-specific BibTeX entry, besides the existing one which is generic for the project.

See version-specific BibTeX entry here: https://zenodo.org/record/4817769/export/hx#.YLSyd6j7RPY
Saving Graph/Structured Data in Datasets
Author: gsh199449 | Pull request: no | State: closed

Thanks for this amazing library! And my question is I have structured data that is organized with a graph. For example, a dataset with users' friendship relations and user's articles. When I try to save a python dict in the dataset, an error occurred ``did not recognize Python value type when inferring an Arrow data type''.
Although I also know that storing a python dict in pyarrow datasets is not the best practice, but I have no idea about how to save structured data in the Datasets. 

Thank you very much for your help.
load_from_disk and save_to_disk are not compatible with each other
Author: roholazandie | Pull request: no | State: closed

## Describe the bug
load_from_disk and save_to_disk are not compatible. When I use save_to_disk to save a dataset to disk it works perfectly but given the same directory load_from_disk throws an error that it can't find state.json. looks like the load_from_disk only works on one split

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("art")
dataset.save_to_disk("mydir")
d = Dataset.load_from_disk("mydir")
```

## Expected results
It is expected that these two functions be the reverse of each other without more manipulation

## Actual results
FileNotFoundError: [Errno 2] No such file or directory: 'mydir/art/state.json'

## Environment info
- `datasets` version: 1.6.2
- Platform: Linux-5.4.0-73-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
Make datasets PEP-561 compliant
Author: SBrandeis | Pull request: yes | State: closed

Allows to type-check datasets with `mypy` when imported as a third-party library

PEP-561: https://www.python.org/dev/peps/pep-0561
MyPy doc on the subject: https://mypy.readthedocs.io/en/stable/installed_packages.html