noamaanMulla-03/datasets-issues huggingface dataset:
dataset adversarial_qa has no answers in the "test" set

## Describe the bug
When loading the adversarial_qa dataset the 'test' portion has no answers.  Only the 'train' and 'validation' portions do.  This occurs with all four of the configs ('adversarialQA', 'dbidaf', 'dbert', 'droberta')

## Steps to reproduce the bug
```
from   datasets import load_dataset
examples = load_dataset('adversarial_qa', 'adversarialQA', script_version="master")['test']
print('Loaded {:,} examples'.format(len(examples)))
has_answers = 0
for e in examples:
    if e['answers']['text']:
        has_answers += 1
print('{:,} have answers'.format(has_answers))
>>> Loaded 3,000 examples
>>> 0 have answers

examples = load_dataset('adversarial_qa', 'adversarialQA', script_version="master")['validation']
<...code above...>
>>> Loaded 3,000 examples
>>> 3,000 have answers
```

## Expected results
If 'test' is a valid dataset, it should have answers. Also note that all of the 'train' and 'validation' sets have answers, there are no "no answer" questions with this set (not sure if this is correct or not).

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.8.0-53-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyArrow version: 1.0.0



Labels: name: bug

Hi ! I'm pretty sure that the answers are not made available for the test set on purpose because it is part of the DynaBench benchmark, for which you can submit your predictions on the website.
In any case we should mention this in the dataset card of this dataset.

Makes sense, but not intuitive for someone searching through the datasets.  Thanks for adding the note to clarify.
MRPC test set differences between torch and tensorflow datasets

## Describe the bug
When using `load_dataset("glue", "mrpc")` to load the MRPC dataset, the test set includes the labels. When using `tensorflow_datasets.load('glue/{}'.format('mrpc'))` to load the dataset the test set does not contain the labels. There should be consistency between torch and tensorflow ways of importing the GLUE datasets.

## Steps to reproduce the bug

Minimal working code  
```python
from datasets import load_dataset
import tensorflow as tf
import tensorflow_datasets

# torch
dataset = load_dataset("glue", "mrpc")
# tf
data = tensorflow_datasets.load('glue/{}'.format('mrpc'))
data = list(data['test'].as_numpy_iterator())
for i in range(40,50):
  tf_sentence1 = data[i]['sentence1'].decode("utf-8") 
  tf_sentence2 = data[i]['sentence2'].decode("utf-8") 

  tf_label = data[i]['label']
  
  index = data[i]['idx']
  print('Index {}'.format(index))
  torch_sentence1 = dataset['test']['sentence1'][index]
  torch_sentence2 = dataset['test']['sentence2'][index]

  torch_label = dataset['test']['label'][index]
  print('Tensorflow: \n\tSentence1 {}\n\tSentence2 {}\n\tLabel {}'.format(tf_sentence1, tf_sentence2, tf_label))
  print('Torch: \n\tSentence1 {}\n\tSentence2 {}\n\tLabel {}'.format(torch_sentence1, torch_sentence2, torch_label))
```

Sample output  
```
Index 954
Tensorflow: 
	Sentence1 Sabri Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate yesterday on charges of violating U.S. arms-control laws .
	Sentence2 The elder Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate Wednesday on charges of violating U.S. arms control laws .
	Label -1
Torch: 
	Sentence1 Sabri Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate yesterday on charges of violating U.S. arms-control laws .
	Sentence2 The elder Yakou , an Iraqi native who is a legal U.S. resident , appeared before a federal magistrate Wednesday on charges of violating U.S. arms control laws .
	Label 1
Index 711
Tensorflow: 
	Sentence1 Others keep records sealed for as little as five years or as much as 30 .
	Sentence2 Some states make them available immediately ; others keep them sealed for as much as 30 years .
	Label -1
Torch: 
	Sentence1 Others keep records sealed for as little as five years or as much as 30 .
	Sentence2 Some states make them available immediately ; others keep them sealed for as much as 30 years .
	Label 0
```

## Expected results
I would expect the datasets to be independent of whether I am working with torch or tensorflow.

## Actual results
Test set labels are provided in the `datasets.load_datasets()` for MRPC. However MRPC is the only task where the test set labels are not -1.

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.109+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyArrow version: 3.0.0


Labels: name: bug

Realized that `tensorflow_datasets` is not provided by Huggingface and should therefore raise the issue there.
Update `xor_tydi_qa` url to v1.1

The dataset is updated and the old url no longer works. So I updated it.

I faced a bug while trying to fix this. Documenting the solution here. Maybe we can add it to the doc (`CONTRIBUTING.md` and `ADD_NEW_DATASET.md`).
> And to make the command work without the ExpectedMoreDownloadedFiles error, you just need to use the --ignore_verifications flag.
https://github.com/huggingface/datasets/issues/2076#issuecomment-803904366


Just noticed while 
```load_dataset('local_path/datastes/xor_tydi_qa')``` works,
```load_dataset('xor_tydi_qa')``` 
outputs an error: 
`
FileNotFoundError: Couldn't find file at https://nlp.cs.washington.edu/xorqa/XORQA_site/data/xor_dev_retrieve_eng_span.jsonl
`
(the old url)

I tired clearing the cache `.cache/huggingface/modules` and `.cache/huggingface/datasets`, didn't work.

Anyone know how to fix this? Thanks.

It seems like the error is not on your end. By default, the lib tries to download the version of the dataset script that matches the version of the lib, and that version of the script is, in your case, broken because the old URL no longer works. Once this PR gets merged, you can wait for the new release or set `script_version` to `"master"` in `load_dataset` to get the fixed version of the script.

@mariosasko Thanks! It works now.

Pasting the docstring here for reference.
```
    script_version (:class:`~utils.Version` or :obj:`str`, optional): Version of the dataset script to load:

        - For canonical datasets in the `huggingface/datasets` library like "squad", the default version of the module is the local version fo the lib.
          You can specify a different version from your local version of the lib (e.g. "master" or "1.2.0") but it might cause compatibility issues.
        - For community provided datasets like "lhoestq/squad" that have their own git repository on the Datasets Hub, the default version "main" corresponds to the "main" branch.
          You can specify a different version that the default "main" by using a commit sha or a git tag of the dataset repository.
```
Branch name didn't work, but commit sha works.

Regarding the issue you mentioned about the `--ignore_verifications` flag, I think we should actually change the current behavior of the `--save_infos` flag to make it ignore the verifications as well, so that you don't need to specific `--ignore_verifications` in this case.

@lhoestq I realized I forgot to change this:

https://github.com/huggingface/datasets/blob/fdbf5a97d3393f4a91e4cddcabe364029508f7ce/datasets/xor_tydi_qa/xor_tydi_qa.py#L72-L73

What should I do?

Oh indeed. Please open a PR to change this. This should be 1.1.0
Add version-specific BibTeX

As pointed out by @lhoestq in #2411, after the creation of the Zenodo DOI for Datasets, a new BibTeX entry is created with each release.

This PR adds a version-specific BibTeX entry, besides the existing one which is generic for the project.

See version-specific BibTeX entry here: https://zenodo.org/record/4817769/export/hx#.YLSyd6j7RPY


Maybe we should only keep one citation ?
cc @thomwolf @yjernite 

For info:
- The one automatically generated by Zenodo is version-specific, and a new one will be generated after each release.
- Zenodo has also generated a project-specific DOI (they call it *Concept DOI* as opposed to *Version DOI*), but currently this only redirects to the DOI page of the latest version.
- All the information automatically generated by Zenodo can be corrected/customized if necessary.
  - If we decide to correct/update metadata, take into account that there are the following fields (among others): Authors, Contributors, Title, Description, Keywords, Additional Notes, License,...

According to Zenodo: https://help.zenodo.org/#versioning
> **Which DOI should I use in citations?**
> 
> You should normally always use the DOI for the specific version of your record in citations. This is to ensure that other researchers can access the exact research artefact you used for reproducibility. By default, Zenodo uses the specific version to generate citations.
> 
> You can use the Concept DOI representing all versions in citations when it is desirable to cite an evolving research artifact, without being specific about the version.

Thanks for the details ! As zenodo says we should probably just show the versioned DOI. And we can remove the old citation.

I have removed the old citation.

What about the new one? Should we customize it? I have fixed some author names (replaced nickname with first and family names). Note that the list of authors is created automatically by Zenodo from this list: https://github.com/huggingface/datasets/graphs/contributors
I do not know if this default automatic list of authors is what we want to show in the citation...
Microsoft CodeXGlue

Datasets from https://github.com/microsoft/CodeXGLUE

This contains 13 datasets:

code_x_glue_cc_clone_detection_big_clone_bench
code_x_glue_cc_clone_detection_poj_104
code_x_glue_cc_cloze_testing_all
code_x_glue_cc_cloze_testing_maxmin
code_x_glue_cc_code_completion_line
code_x_glue_cc_code_completion_token
code_x_glue_cc_code_refinement
code_x_glue_cc_code_to_code_trans
code_x_glue_cc_defect_detection
code_x_glue_ct_code_to_text
code_x_glue_tc_nl_code_search_adv
code_x_glue_tc_text_to_code
code_x_glue_tt_text_to_text



#978 is working on adding code refinement

maybe we should keep the CodeXGlue benchmark (as glue) and don't merge the code_refinement dataset proposed in #978 ?

cc @reshinthadithyan

Hi @madlag and @lhoestq , I am extremely interested in getting this dataset into HF's library as I research in this area a lot. I see that it hasn't been updated in a while, but it is very close to being finished. If no one is currently working on this, I'd be happy to do any final touches that might be needed to get this merged.

Hi @ncoop57 ! Thanks for your interest and sorry for the inactivity on this PR.
Sure feel free to create another PR to continue this one ! This one was really close to being merged so I think it won't require that much changes. In addition to my previous comments, there should also be a "Contributions" subsection (see the template of the README [here](https://github.com/huggingface/datasets/blob/master/templates/README.md))

Superseded by https://github.com/huggingface/datasets/pull/2357 .
Support sliced list arrays in cast

There is this issue in pyarrow:
```python
import pyarrow as pa

arr = pa.array([[i * 10] for i in range(4)])
arr.cast(pa.list_(pa.int32()))  # works

arr = arr.slice(1)
arr.cast(pa.list_(pa.int32()))  # fails
# ArrowNotImplementedError("Casting sliced lists (non-zero offset) not yet implemented")
```

However in `Dataset.cast` we slice tables to cast their types (it's memory intensive), so we have the same issue.
Because of this it is currently not possible to cast a Dataset with a Sequence feature type (unless the table is small enough to not be sliced).

In this PR I fixed this by resetting the offset of `pyarrow.ListArray` arrays to zero in the table before casting.
I used `pyarrow.compute.subtract` function to update the offsets of the ListArray.

cc @abhi1thakur @SBrandeis
Revert default in-memory for small datasets

Users are reporting issues and confusion about setting default in-memory to True for small datasets.

We see 2 clear use cases of Datasets:
- the "canonical" way, where you can work with very large datasets, as they are memory-mapped and cached (after every transformation)
- some edge cases (speed benchmarks, interactive/exploratory analysis,...), where default in-memory can explicitly be enabled, and no caching will be done

After discussing with @lhoestq we have agreed to:
- revert this feature (implemented in #2182)
- explain in the docs how to optimize speed/performance by setting default in-memory

cc: @stas00 https://github.com/huggingface/datasets/pull/2409#issuecomment-856210552

Labels: name: enhancement

cc: @krandiash (pinged in reverted PR).
load_from_disk and save_to_disk are not compatible with each other

## Describe the bug
load_from_disk and save_to_disk are not compatible. When I use save_to_disk to save a dataset to disk it works perfectly but given the same directory load_from_disk throws an error that it can't find state.json. looks like the load_from_disk only works on one split

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("art")
dataset.save_to_disk("mydir")
d = Dataset.load_from_disk("mydir")
```

## Expected results
It is expected that these two functions be the reverse of each other without more manipulation

## Actual results
FileNotFoundError: [Errno 2] No such file or directory: 'mydir/art/state.json'

## Environment info
- `datasets` version: 1.6.2
- Platform: Linux-5.4.0-73-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>




Hi,

`load_dataset` returns an instance of `DatasetDict` if `split` is not specified, so instead of `Dataset.load_from_disk`, use `DatasetDict.load_from_disk` to load the dataset from disk.

Thanks it worked!

Though I see a stream of issues open by people lost between datasets and datasets dicts so maybe there is here something that could be better in terms of UX. Could be better error handling or something else smarter to even avoid said errors but maybe we should think about this. Reopening to use this issue as a discussion place but feel free to open a new open if you prefer @lhoestq @albertvillanova 

We should probably improve the error message indeed.

Also note that there exists a function `load_from_disk` that can load a Dataset or a DatasetDict. Under the hood it calls either `Dataset.load_from_disk` or `DatasetDict.load_from_disk`:


```python
from datasets import load_from_disk

dataset_dict = load_from_disk("path/to/dataset/dict")
single_dataset = load_from_disk("path/to/single/dataset")
```

I just opened #2437 to improve the error message

Superseded by #2462
Add cache dir for in-memory datasets

Adds the cache dir attribute to DatasetInfo as suggested by @lhoestq.

Should fix #2322 


Yes, having `cache_dir` as an attribute looks cleaner.





Good job! Looking forward to this new feature! 🥂

@lhoestq Sorry for the late reply. Yes, I'll start working on tests. Thanks for the detailed explanation of the current issues with caching (like the idea of adding the `use_caching` parameter to `load_dataset`) 

@lhoestq Sure. I'm aware this is a high-priority issue to some extent, so feel free to take over.

Few suggestions I have:
* there is a slight difference between setting `use_caching` to `False` in `load_dataset` and disabling caching globally with `set_caching_enabled(False)` because the former will never execute the following code (`self._cache_dir` is always `False`): 
https://github.com/huggingface/datasets/blob/c231abdb174987419bbde3360b5b9d6a4672c736/src/datasets/arrow_dataset.py#L1807-L1824
, so I'm just checking whether this is intended (if yes, maybe the docs should mention this) or not?
* think we should add the `use_caching` parameter to every method that has the `keep_in_memory` (and `in_memory` 😃) parameter in its signature for better consistency, but I say let's address this in a separate PR. IMO we need one more PR that will deal exclusively with consistency in the caching logic.

Hi @mariosasko 
We discussed internally and we think that this feature might not be the direction we're doing to take for these reasons:

- it goes against our simple definition of caching: on-disk == uses file cache, and in-memory == nothing is written to disk. I think it adds too much complexity just for a minimal flexibility addition
- there are a few edge cases which are really confusing:
    - map on an in memory dataset with a cache_file_name specified by the user -> should the result be in memory or from disk ?
    - it would require a special cache directory just for in memory datasets, since they don’t have a preferred directory for caching
- it would break a lot of stuff and would require to rewrite significant parts of the core code and the tests


So in the end we're probably going to close this PR.
Let me know what you think, and thank you anyway for your help on this !

Hi,

I'm fine with that. I agree this adds too much complexity. Btw, I like the idea of reverting default in-memory for small datasets that led to this PR.

Superseded by #2460 (to close issue #2458).
Remove `extended` field from dataset tagger

## Describe the bug
While working on #2435 I used the [dataset tagger](https://huggingface.co/datasets/tagging/) to generate the missing tags for the YAML metadata of each README.md file. However, it seems that our CI raises an error when the `extended` field is included:

```
dataset_name = 'arcd'

    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
    def test_changed_dataset_card(dataset_name):
        card_path = repo_path / "datasets" / dataset_name / "README.md"
        assert card_path.exists()
        error_messages = []
        try:
            ReadMe.from_readme(card_path)
        except Exception as readme_error:
            error_messages.append(f"The following issues have been found in the dataset cards:\nREADME:\n{readme_error}")
        try:
            DatasetMetadata.from_readme(card_path)
        except Exception as metadata_error:
            error_messages.append(
                f"The following issues have been found in the dataset cards:\nYAML tags:\n{metadata_error}"
            )
    
        if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           __init__() got an unexpected keyword argument 'extended'

tests/test_dataset_cards.py:70: ValueError
```

Consider either removing this tag from the tagger or including it as part of the validation step in the CI.

cc @yjernite 

Labels: name: bug

The tagger also doesn't insert the value for the `size_categories` field automatically, so this should be fixed too

Thanks for reporting. Indeed the `extended` tag doesn't exist. Not sure why we had that in the tagger.
The repo of the tagger is here if someone wants to give this a try: https://github.com/huggingface/datasets-tagging
Otherwise I can probably fix it next week

I've opened a PR on `datasets-tagging` to fix the issue 🚀 

thanks ! this is fixed now
fix: adjusting indexing for the labels.

The labels index were mismatching the actual ones used in the dataset. Specifically `0` is used  for `SUPPORTS` and `1` is used for `REFUTES`
After this change, the `README.md` now reflects the content of `dataset_infos.json`.

Signed-off-by: Matteo Manica <drugilsberg@gmail.com>


> Good catch ! Thanks for fixing it

My pleasure🙏
Add KLUE dataset

Add `KLUE (Korean Language Understanding Evaluation)` dataset released recently from [paper](https://arxiv.org/abs/2105.09680), [github](https://github.com/KLUE-benchmark/KLUE) and [webpage](https://klue-benchmark.com/tasks).
Please let me know if there's anything missing in the code or README.
Thanks!





I'm not sure why I got error like below when I auto-generate dummy data "mrc" 
```
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature
```

> I'm not sure why I got error like below when I auto-generate dummy data "mrc"
> 
> ```
> datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
> Found duplicate Key: 0
> Keys should be unique and deterministic in nature
> ```

Please check out the suggestion below. I think it might be a cause.

> > I'm not sure why I got error like below when I auto-generate dummy data "mrc"
> > ```
> > datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
> > Found duplicate Key: 0
> > Keys should be unique and deterministic in nature
> > ```
> 
> Please check out the suggestion below. I think it might be a cause.

The problem was `id_` in mrc when yield was not unique. (I used index in `enumerate(paragraphs)` by mistake)
I fixed it and update all the things

To fix the CI you can just merge master into your branch and it should be all green hopefully :)

@lhoestq
Thanks for reviewing!

It's harder than I thought to add dataset card. 😅 
I checked and updated your suggestion (script, readme details, dummy data). 

dummy data is little bit larger than expected because `ner` dataset is about 80 lines and `dp` dataset is about 25 lines to avoid 0 examples.

I'm not sure why some CI keep fails, can u check for this?

Thanks ! That makes sense for ner and dp

For mrc on the other hand there are still too many examples, maybe you can generate the dummy data for 5 examples for all tasks except ner and dp ?

> Thanks ! That makes sense for ner and dp
> 
> For mrc on the other hand there are still too many examples, maybe you can generate the dummy data for 5 examples for all tasks except ner and dp ?

Yes, I generate default lines in dataset-cli for other dataset except "dp" and "ner"
I fixed mrc dataset, hope it's fine now :)

the reason CI failed was I forgot to merge master into my branch 😅