kardwalker/my_dataset huggingface dataset:
GitHub issue #2444: Sentence Boundaries missing in Dataset: xtreme / udpos
State: closed | Created: 2021-06-04T09:10:26Z

I was browsing through annotation guidelines, as suggested by the datasets introduction.

The guidlines saids "There must be exactly one blank line after every sentence, including the last sentence in the file. Empty sentences are not allowed." in the [Sentence Boundaries and Comments section](https://universaldependencies.org/format.html#sentence-boundaries-and-comments)

But the sentence boundaries seems not to be represented by huggingface datasets features well. I found out that multiple sentence are concatenated together as a 1D array, without any delimiter.

PAN-x, which is another token classification subset from xtreme do represent the sentence boundary using a 2D array.

You may compare in PAN-x.en and udpos.English in the explorer:
 https://huggingface.co/datasets/viewer/?dataset=xtreme
GitHub issue #2443: Some tests hang on Windows
State: closed | Created: 2021-06-03T00:27:30Z

Currently, several tests hang on Windows if the max path limit of 260 characters is not disabled. This happens due to the changes introduced by #2223 that cause an infinite loop in `WindowsFileLock` described in #2220.  This can be very tricky to debug, so I think now is a good time to address these issues/PRs. IMO throwing an error is too harsh, but maybe we can emit a warning in the top-level `__init__.py ` on startup if long paths are not enabled.
GitHub issue #2442: add english language tags for ~100 datasets
State: closed | Created: 2021-06-02T16:24:56Z

As discussed on Slack, I have manually checked for ~100 datasets that they have at least one subset in English. This information was missing so adding into the READMEs.

Note that I didn't check all the subsets so it's possible that some of the datasets have subsets in other languages than English...
GitHub issue #2440: Remove `extended` field from dataset tagger
State: closed | Created: 2021-06-01T17:18:42Z

## Describe the bug
While working on #2435 I used the [dataset tagger](https://huggingface.co/datasets/tagging/) to generate the missing tags for the YAML metadata of each README.md file. However, it seems that our CI raises an error when the `extended` field is included:

```
dataset_name = 'arcd'

    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
    def test_changed_dataset_card(dataset_name):
        card_path = repo_path / "datasets" / dataset_name / "README.md"
        assert card_path.exists()
        error_messages = []
        try:
            ReadMe.from_readme(card_path)
        except Exception as readme_error:
            error_messages.append(f"The following issues have been found in the dataset cards:\nREADME:\n{readme_error}")
        try:
            DatasetMetadata.from_readme(card_path)
        except Exception as metadata_error:
            error_messages.append(
                f"The following issues have been found in the dataset cards:\nYAML tags:\n{metadata_error}"
            )
    
        if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           __init__() got an unexpected keyword argument 'extended'

tests/test_dataset_cards.py:70: ValueError
```

Consider either removing this tag from the tagger or including it as part of the validation step in the CI.

cc @yjernite
GitHub issue #2438: Fix NQ features loading: reorder fields of features to match nested fields order in arrow data
State: closed | Created: 2021-06-01T16:09:30Z

As mentioned in #2401, there is an issue when loading the features of `natural_questions` since the order of the nested fields in the features don't match. The order is important since it matters for the underlying arrow schema.

To fix that I re-order the features based on the arrow schema:

```python
inferred_features = Features.from_arrow_schema(arrow_table.schema)
self.info.features = self.info.features.reorder_fields_as(inferred_features)
assert self.info.features.type == inferred_features.type
```

The re-ordering is a recursive function. It takes into account that the `Sequence` feature type is a struct of list and not a list of struct.

Now it's possible to load `natural_questions` again :)
GitHub issue #2437: Better error message when using the wrong load_from_disk
State: closed | Created: 2021-06-01T09:43:22Z

As mentioned in #2424, the error message when one tries to use `Dataset.load_from_disk` to load a DatasetDict object (or _vice versa_) can be improved. I added a suggestion in the error message to let users know that they should use the other one.
GitHub issue #2436: Update DatasetMetadata and ReadMe
State: closed | Created: 2021-06-01T09:32:37Z

This PR contains the changes discussed in #2395.

**Edit**:
In addition to those changes, I'll be updating the `ReadMe` as follows:

Currently, `Section` has separate parsing and validation error lists. In `.validate()`, we add these lists to the final lists and throw errors.

One way to make `ReadMe` consistent with `DatasetMetadata` and add a separate `.validate()` method is to throw separate parsing and validation errors. 

This way, we don't have to throw validation errors, but only parsing errors in `__init__ ()`. We can have an option in `__init__()` to suppress parsing errors so that an object is created for validation. Doing this will allow the user to get all the errors in one go.

In `test_dataset_cards` , we are already catching error messages and appending to a list. This can be done for `ReadMe()`  for parsing errors, and `ReadMe(...,suppress_errors=True); readme.validate()`  for validation, separately.

**Edit 2**:
The only parsing issue we have as of now is multiple headings at the same level with the same name. I assume this will happen very rarely, but it is still better to throw an error than silently pick one of them. It should be okay to separate it this way. 

Wdyt @lhoestq ?
GitHub issue #2435: Insert Extractive QA templates for SQuAD-like datasets
State: closed | Created: 2021-05-31T14:09:11Z

This PR adds task templates for 9 SQuAD-like templates with the following properties:

* 1 config
* A schema that matches the `squad` one (i.e. same column names, especially for the nested `answers` column because the current implementation does not support casting with mismatched columns. see #2434)
* Less than 20GB (my laptop can't handle more right now)

The aim of this PR is to provide a few datasets to experiment with the task template integration in other libraries / services. 

PR #2429 should be merged before this one.

cc @abhi1thakur
GitHub issue #2432: Fix CI six installation on linux
State: closed | Created: 2021-05-31T13:15:36Z

For some reason we end up with this error in the linux CI when running pip install .[tests]
```
pip._vendor.resolvelib.resolvers.InconsistentCandidate: Provided candidate AlreadyInstalledCandidate(six 1.16.0 (/usr/local/lib/python3.6/site-packages)) does not satisfy SpecifierRequirement('six>1.9'), SpecifierRequirement('six>1.9'), SpecifierRequirement('six>=1.11'), SpecifierRequirement('six~=1.15'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.5.2'), SpecifierRequirement('six>=1.9.0'), SpecifierRequirement('six>=1.11.0'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.6.1'), SpecifierRequirement('six>=1.9'), SpecifierRequirement('six>=1.5'), SpecifierRequirement('six<2.0'), SpecifierRequirement('six<2.0'), SpecifierRequirement('six'), SpecifierRequirement('six'), SpecifierRequirement('six~=1.15.0'), SpecifierRequirement('six'), SpecifierRequirement('six<2.0,>=1.6.1'), SpecifierRequirement('six'), SpecifierRequirement('six>=1.5.2'), SpecifierRequirement('six>=1.9.0')
```
example CI failure here:
https://app.circleci.com/pipelines/github/huggingface/datasets/6200/workflows/b64fdec9-f9e6-431c-acd7-e9f2c440c568/jobs/38247

The main version requirement comes from tensorflow: `six~=1.15.0`
So I pinned the six version to this.
GitHub issue #2431: DuplicatedKeysError when trying to load adversarial_qa
State: closed | Created: 2021-05-31T12:11:19Z

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
dataset = load_dataset('adversarial_qa', 'adversarialQA')
```

## Expected results
The dataset should be loaded into memory

## Actual results

>DuplicatedKeysError: FAILURE TO GENERATE DATASET !
>Found duplicate Key: 4d3cb5677211ee32895ca9c66dad04d7152254d4
>Keys should be unique and deterministic in nature
>
>
>During handling of the above exception, another exception occurred:
>
>DuplicatedKeysError                       Traceback (most recent call last)
>
>/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
>    347         for hash, key in self.hkey_record:
>    348             if hash in tmp_record:
>--> 349                 raise DuplicatedKeysError(key)
>    350             else:
>    351                 tmp_record.add(hash)
>
>DuplicatedKeysError: FAILURE TO GENERATE DATASET !
>Found duplicate Key: 4d3cb5677211ee32895ca9c66dad04d7152254d4
>Keys should be unique and deterministic in nature

## Environment info
- `datasets` version: 1.7.0
- Platform: Linux-5.4.109+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyArrow version: 3.0.0
GitHub issue #2430: Add version-specific BibTeX
State: closed | Created: 2021-05-31T10:05:42Z

As pointed out by @lhoestq in #2411, after the creation of the Zenodo DOI for Datasets, a new BibTeX entry is created with each release.

This PR adds a version-specific BibTeX entry, besides the existing one which is generic for the project.

See version-specific BibTeX entry here: https://zenodo.org/record/4817769/export/hx#.YLSyd6j7RPY
GitHub issue #2426: Saving Graph/Structured Data in Datasets
State: closed | Created: 2021-05-29T13:35:21Z

Thanks for this amazing library! And my question is I have structured data that is organized with a graph. For example, a dataset with users' friendship relations and user's articles. When I try to save a python dict in the dataset, an error occurred ``did not recognize Python value type when inferring an Arrow data type''.
Although I also know that storing a python dict in pyarrow datasets is not the best practice, but I have no idea about how to save structured data in the Datasets. 

Thank you very much for your help.
GitHub issue #2424: load_from_disk and save_to_disk are not compatible with each other
State: closed | Created: 2021-05-28T23:07:10Z

## Describe the bug
load_from_disk and save_to_disk are not compatible. When I use save_to_disk to save a dataset to disk it works perfectly but given the same directory load_from_disk throws an error that it can't find state.json. looks like the load_from_disk only works on one split

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("art")
dataset.save_to_disk("mydir")
d = Dataset.load_from_disk("mydir")
```

## Expected results
It is expected that these two functions be the reverse of each other without more manipulation

## Actual results
FileNotFoundError: [Errno 2] No such file or directory: 'mydir/art/state.json'

## Environment info
- `datasets` version: 1.6.2
- Platform: Linux-5.4.0-73-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
GitHub issue #2422: Fix save_to_disk nested features order in dataset_info.json
State: closed | Created: 2021-05-28T15:03:28Z

Fix issue https://github.com/huggingface/datasets/issues/2267

The order of the nested features matters (pyarrow limitation), but the save_to_disk method was saving the features types as JSON with `sort_keys=True`, which was breaking the order of the nested features.
GitHub issue #2417: Make datasets PEP-561 compliant
State: closed | Created: 2021-05-27T16:16:17Z

Allows to type-check datasets with `mypy` when imported as a third-party library

PEP-561: https://www.python.org/dev/peps/pep-0561
MyPy doc on the subject: https://mypy.readthedocs.io/en/stable/installed_packages.html
GitHub issue #2416: Add KLUE dataset
State: closed | Created: 2021-05-27T15:49:51Z

Add `KLUE (Korean Language Understanding Evaluation)` dataset released recently from [paper](https://arxiv.org/abs/2105.09680), [github](https://github.com/KLUE-benchmark/KLUE) and [webpage](https://klue-benchmark.com/tasks).
Please let me know if there's anything missing in the code or README.
Thanks!
GitHub issue #2415: Cached dataset not loaded
State: closed | Created: 2021-05-27T15:40:06Z

## Describe the bug
I have a large dataset (common_voice, english) where I use several map and filter functions.
Sometimes my cached datasets after specific functions are not loaded.
I always use the same arguments, same functions, no seed…

## Steps to reproduce the bug
```python
def filter_by_duration(batch):
    return (
        batch["duration"] <= 10
        and batch["duration"] >= 1
        and len(batch["target_text"]) > 5
    )

def prepare_dataset(batch):
    batch["input_values"] = processor(
        batch["speech"], sampling_rate=batch["sampling_rate"][0]
    ).input_values
    with processor.as_target_processor():
        batch["labels"] = processor(batch["target_text"]).input_ids
    return batch

train_dataset = train_dataset.filter(
    filter_by_duration,
    remove_columns=["duration"],
    num_proc=data_args.preprocessing_num_workers,
)

# PROBLEM HERE -> below function is reexecuted and cache is not loaded
train_dataset = train_dataset.map(
    prepare_dataset,
    remove_columns=train_dataset.column_names,
    batch_size=training_args.per_device_train_batch_size,
    batched=True,
    num_proc=data_args.preprocessing_num_workers,
)

# Later in script
set_caching_enabled(False)
# apply map on trained model to eval/test sets

```

## Expected results
The cached dataset should always be reloaded.

## Actual results
The function is reexecuted.

I have access to cached files `cache-xxxxx.arrow`.
Is there a way I can somehow load manually 2 versions and see how the hash was created for debug purposes (to know if it's an issue with dataset or function)?

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.6.2
- Platform: Linux-5.8.0-45-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
GitHub issue #2413: AttributeError: 'DatasetInfo' object has no attribute 'task_templates'
State: closed | Created: 2021-05-27T13:44:28Z

## Describe the bug
Hello, 
I'm trying to add dataset and contribute, but test keep fail with below cli.
` RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<my_dataset>`

## Steps to reproduce the bug
It seems like a bug when I see an error with the existing dataset, not the dataset I'm trying to add.

` RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<any_dataset>`


## Expected results
All test passed

## Actual results
```
                # check that dataset is not empty
                self.parent.assertListEqual(sorted(dataset_builder.info.splits.keys()), sorted(dataset))
                for split in dataset_builder.info.splits.keys():
                    # check that loaded datset is not empty
                    self.parent.assertTrue(len(dataset[split]) > 0)
    
                # check that we can cast features for each task template
>               task_templates = dataset_builder.info.task_templates
E               AttributeError: 'DatasetInfo' object has no attribute 'task_templates'

tests/test_dataset_common.py:175: AttributeError
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.6.2
- Platform: Darwin-20.4.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
GitHub issue #2412: Docstring mistake: dataset vs. metric
State: closed | Created: 2021-05-27T13:39:11Z

This:

https://github.com/huggingface/datasets/blob/d95b95f8cf3cb0cff5f77a675139b584dcfcf719/src/datasets/load.py#L582

Should better be something like:

`a metric identifier on HuggingFace AWS bucket (list all available metrics and ids with ``datasets.list_metrics()``)`

I can provide a PR l8er...
GitHub issue #2407: .map() function got an unexpected keyword argument 'cache_file_name'
State: closed | Created: 2021-05-27T01:54:26Z

## Describe the bug

I'm trying to save the result of datasets.map() to a specific file, so that I can easily share it among multiple computers without reprocessing the dataset. However, when I try to pass an argument 'cache_file_name' to the .map() function, it throws an error that ".map() function got an unexpected keyword argument 'cache_file_name'". 

I believe I'm using the latest dataset 1.6.2. Also seems like the document and the actual code indicates there is an argument 'cache_file_name' for the .map() function.

Here is the code I use
## Steps to reproduce the bug
```datasets = load_from_disk(dataset_path=my_path)

[...]

def tokenize_function(examples):
    return tokenizer(examples[text_column_name])

logger.info("Mapping dataset to tokenized dataset.")
tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
    num_proc=preprocessing_num_workers,
    remove_columns=column_names,
    load_from_cache_file=True,
   cache_file_name="my_tokenized_file"
)
```

## Actual results
    tokenized_datasets = datasets.map(
TypeError: map() got an unexpected keyword argument 'cache_file_name'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:1.6.2
- Platform:Linux-4.18.0-193.28.1.el8_2.x86_64-x86_64-with-glibc2.10
- Python version:3.8.5
- PyArrow version:3.0.0
GitHub issue #2404: Paperswithcode dataset mapping
State: closed | Created: 2021-05-25T18:14:26Z

This is a continuation of https://github.com/huggingface/huggingface_hub/pull/43, encoded directly inside dataset cards.

As discussed:
- `paperswithcode_id: null` when the dataset doesn't exist on paperswithcode's side.
- I've added this new key at the end of the yaml instead of ordering all keys alphabetically as pyyaml's default. No strong opinion on that one though
GitHub issue #2403: Free datasets with cache file in temp dir on exit
State: closed | Created: 2021-05-24T22:15:11Z

This PR properly cleans up the memory-mapped tables that reference the cache files inside the temp dir.
Since the built-in `_finalizer` of `TemporaryDirectory` can't be modified, this PR defines its own `TemporaryDirectory` class that accepts a custom clean-up function.

Fixes #2402
GitHub issue #2402: PermissionError on Windows when using temp dir for caching
State: closed | Created: 2021-05-24T21:22:59Z

Currently, the following code raises a PermissionError on master if working on Windows:

```python
# run as a script or call exit() in REPL to initiate the temp dir cleanup
from datasets import *
d = load_dataset("sst", split="train", keep_in_memory=False)
set_caching_enabled(False)
d.map(lambda ex: ex)
```

Error stack trace:
```
Traceback (most recent call last):
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\weakref.py", line 624, in _exitfunc
    f()
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\weakref.py", line 548, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\tempfile.py", line 799, in _cleanup
    _shutil.rmtree(name)
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\shutil.py", line 500, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\shutil.py", line 395, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\shutil.py", line 393, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Mario\\AppData\\Local\\Temp\\tmp20epyhmq\\cache-87a87ffb5a956e68.arrow'
```
GitHub issue #2400: Concatenate several datasets with removed columns is not working.
State: closed | Created: 2021-05-24T17:40:15Z

## Describe the bug

You can't concatenate datasets when you removed columns before.

## Steps to reproduce the bug
```python
from datasets import load_dataset, concatenate_datasets

wikiann= load_dataset("wikiann","en")

wikiann["train"] = wikiann["train"].remove_columns(["langs","spans"])
wikiann["test"] = wikiann["test"].remove_columns(["langs","spans"])

assert wikiann["train"].features.type == wikiann["test"].features.type

concate = concatenate_datasets([wikiann["train"],wikiann["test"]])
```

## Expected results
Merged dataset 


## Actual results
```python
ValueError: External features info don't match the dataset:
Got
{'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'ner_tags': Sequence(feature=ClassLabel(num_classes=7, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC'], names_file=None, id=None), length=-1, id=None), 'langs': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'spans': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
with type
struct<langs: list<item: string>, ner_tags: list<item: int64>, spans: list<item: string>, tokens: list<item: string>>

but expected something like
{'ner_tags': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
with type
struct<ner_tags: list<item: int64>, tokens: list<item: string>>
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: ~1.6.2~ 1.5.0
- Platform: macos
- Python version: 3.8.5
- PyArrow version: 3.0.0
GitHub issue #2398: News_commentary Dataset Translation Pairs are of Incorrect Language Specified Pairs
State: closed | Created: 2021-05-24T10:03:34Z

I used load_dataset to load the news_commentary dataset for "ar-en" translation pairs but found translations from Arabic to Hindi.  

```
train_ds = load_dataset("news_commentary", "ar-en", split='train[:98%]')
val_ds = load_dataset("news_commentary", "ar-en", split='train[98%:]')

# filtering out examples that are not ar-en translations but ar-hi
val_ds = val_ds.filter(lambda example, indice: indice not in chain(range(1312,1327) ,range(1384,1399), range(1030,1042)), with_indices=True)
```

* I'm fairly new to using datasets so I might be doing something wrong
GitHub issue #2396: strange datasets from OSCAR corpus
State: open | Created: 2021-05-23T13:06:02Z

![image](https://user-images.githubusercontent.com/50871412/119260850-4f876b80-bc07-11eb-8894-124302600643.png)
![image](https://user-images.githubusercontent.com/50871412/119260875-675eef80-bc07-11eb-9da4-ee27567054ac.png)
From the [official site ](https://oscar-corpus.com/), the Yue Chinese dataset should have 2.2KB data.
7 training instances is obviously not a right number.
As I can read Yue Chinese, I call tell the last instance is definitely not something that would appear on Common Crawl.
And even if you don't read Yue Chinese, you can tell the first six instance are problematic.
(It is embarrassing, as the 7 training instances look exactly like something from a pornographic novel or flitting messages in a chat of a dating app)
It might not be the problem of the huggingface/datasets implementation, because when I tried to download the dataset from the official site, I found out that the zip file is corrupted.
I will try to inform the host of OSCAR corpus later.
Awy a remake about this dataset in huggingface/datasets is needed, perhaps after the host of the dataset fixes the issue.

> Hi @jerryIsHere , sorry for the late response! Sadly this is normal, the problem comes form fasttext's classifier which we used to create the original corpus. In general the classifier is not really capable of properly recognizing Yue Chineese so the file ends un being just noise from Common Crawl. Some of these problems with OSCAR were already discussed [here](https://arxiv.org/pdf/2103.12028.pdf) but we are working on explicitly documenting the problems by language on our website. In fact, could please you open an issue on [our repo](https://github.com/oscar-corpus/oscar-website/issues) as well so that we can track it?

Thanks a lot, the new post is here:
https://github.com/oscar-corpus/oscar-website/issues/11
GitHub issue #2395: `pretty_name` for dataset in YAML tags
State: closed | Created: 2021-05-22T09:24:45Z

I'm updating `pretty_name` for datasets in YAML tags as discussed with @lhoestq. Here are the first 10, please let me know if they're looking good.

If dataset has 1 config, I've added `pretty_name` as `config_name: full_name_of_dataset` as config names were `plain_text`, `default`, `squad` etc (not so important in this case) whereas when dataset has >1 configs, I've added `config_name: full_name_of_dataset+config_name` so as to let user know about the `config` here.
GitHub issue #2391: Missing original answers in kilt-TriviaQA
State: closed | Created: 2021-05-21T14:57:07Z

I previously opened an issue at https://github.com/facebookresearch/KILT/issues/42 but from the answer of @fabiopetroni it seems that the problem comes from HF-datasets

## Describe the bug
The `answer` field in kilt-TriviaQA, e.g. `kilt_tasks['train_triviaqa'][0]['output']['answer']` contains a list of alternative answer which are accepted for the question.  
However it'd be nice to know the original answer to the question (the only fields in `output` are `'answer', 'meta', 'provenance'`)

## How to fix
It can be fixed by retrieving the original answer from the original TriviaQA (e.g. `trivia_qa['train'][0]['answer']['value']`), perhaps at the same place as here where one retrieves the questions https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md#loading-the-kilt-knowledge-source-and-task-data

cc @yjernite who previously answered to an issue about KILT and TriviaQA :)
GitHub issue #2387: datasets 1.6 ignores cache
State: closed | Created: 2021-05-21T00:12:58Z

Moving from https://github.com/huggingface/transformers/issues/11801#issuecomment-845546612 

Quoting @VictorSanh:

> 
> I downgraded datasets to `1.5.0` and printed `tokenized_datasets.cache_files` (L335):
> 
> > `{'train': [{'filename': '/home/victor/.cache/huggingface/datasets/openwebtext10k/plain_text/1.0.0/3a8df094c671b4cb63ed0b41f40fb3bd855e9ce2e3765e5df50abcdfb5ec144b/cache-c6aefe81ca4e5152.arrow'}], 'validation': [{'filename': '/home/victor/.cache/huggingface/datasets/openwebtext10k/plain_text/1.0.0/3a8df094c671b4cb63ed0b41f40fb3bd855e9ce2e3765e5df50abcdfb5ec144b/cache-97cf4c813e6469c6.arrow'}]}`
> 
> while the same command with the latest version of datasets (actually starting at `1.6.0`) gives:
> > `{'train': [], 'validation': []}`
> 

I also confirm that downgrading to `datasets==1.5.0` makes things fast again - i.e. cache is used.

to reproduce:
```
USE_TF=0 python  examples/pytorch/language-modeling/run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name "stas/openwebtext-10k" \
    --output_dir output_dir \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --max_train_samples 1000 \
    --max_eval_samples 200 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --num_train_epochs 1 \
    --warmup_steps 8 \
    --block_size 64 \
    --fp16 \
    --report_to none
```

the first time the startup is slow and some 5 tqdm bars. It shouldn't do it on consequent runs. but with `datasets>1.5.0` it rebuilds on every run.

@lhoestq
GitHub issue #2386: Accessing Arrow dataset cache_files
State: closed | Created: 2021-05-20T23:57:43Z

## Describe the bug
In datasets 1.5.0 the following code snippet would have printed the cache_files:

```
train_data = load_dataset('conll2003', split='train', cache_dir='data')
print(train_data.cache_files[0]['filename'])

```

However, in the newest release (1.6.1), it prints an empty list.

I also tried loading the dataset with `keep_in_memory=True` argument but still `cache_files` is empty.

Was wondering if this is a bug or I need to pass additional arguments so I can access the cache_files.
GitHub issue #2384: Add args description to DatasetInfo
State: closed | Created: 2021-05-20T13:53:10Z

Closes #2354 

I am not sure what `post_processed` and `post_processing_size` correspond to, so have left them empty for now. I also took a guess at some of the other fields like `dataset_size` vs `size_in_bytes`, so might have misunderstood their meaning.
GitHub issue #2382: DuplicatedKeysError: FAILURE TO GENERATE DATASET ! load_dataset('head_qa', 'en')
State: closed | Created: 2021-05-19T15:49:48Z

Hello everyone,

I try to use head_qa dataset in [https://huggingface.co/datasets/viewer/?dataset=head_qa&config=en](url)

```
!pip install datasets
from datasets import load_dataset
dataset = load_dataset(
   'head_qa', 'en')
```
When I write above load_dataset(.), it throws the following:

```
DuplicatedKeysError                       Traceback (most recent call last)

<ipython-input-6-ea87002d32f0> in <module>()
      2 from datasets import load_dataset
      3 dataset = load_dataset(
----> 4    'head_qa', 'en')

5 frames

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
    347         for hash, key in self.hkey_record:
    348             if hash in tmp_record:
--> 349                 raise DuplicatedKeysError(key)
    350             else:
    351                 tmp_record.add(hash)

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 1
Keys should be unique and deterministic in nature
```
How can I fix the error? Thanks
GitHub issue #2380: maintain YAML structure reading from README
State: closed | Created: 2021-05-19T12:12:07Z

How YAML used be loaded earlier in the string (structure of YAML was affected because of this and YAML for datasets with multiple configs was not being loaded correctly):
```
annotations_creators:
labeled_final:
- expert-generated
labeled_swap:
- expert-generated
unlabeled_final:
- machine-generated
language_creators:
- machine-generated
languages:
- en
licenses:
- other
multilinguality:
- monolingual
size_categories:
labeled_final:
- 10K<n<100K
labeled_swap:
- 10K<n<100K
unlabeled_final:
- 100K<n<1M
source_datasets:
- original
task_categories:
- text-classification
- text-scoring
task_ids:
- semantic-similarity-classification
- semantic-similarity-scoring
- text-scoring-other-paraphrase-identification
```
How YAML is loaded in string now:
```
annotations_creators:
  labeled_final:
  - expert-generated
  labeled_swap:
  - expert-generated
  unlabeled_final:
  - machine-generated
language_creators:
- machine-generated
languages:
- en
licenses:
- other
multilinguality:
- monolingual
size_categories:
  labeled_final:
  - 10K<n<100K
  labeled_swap:
  - 10K<n<100K
  unlabeled_final:
  - 100K<n<1M
source_datasets:
- original
task_categories:
- text-classification
- text-scoring
task_ids:
- semantic-similarity-classification
- semantic-similarity-scoring
- text-scoring-other-paraphrase-identification
```
GitHub issue #2378: Add missing dataset_infos.json files
State: open | Created: 2021-05-19T08:11:12Z

Some of the datasets in `datasets` are missing a `dataset_infos.json` file, e.g.

```
[PosixPath('datasets/chr_en/chr_en.py'), PosixPath('datasets/chr_en/README.md')]
[PosixPath('datasets/telugu_books/README.md'), PosixPath('datasets/telugu_books/telugu_books.py')]
[PosixPath('datasets/reclor/README.md'), PosixPath('datasets/reclor/reclor.py')]
[PosixPath('datasets/json/README.md')]
[PosixPath('datasets/csv/README.md')]
[PosixPath('datasets/wikihow/wikihow.py'), PosixPath('datasets/wikihow/README.md')]
[PosixPath('datasets/c4/c4.py'), PosixPath('datasets/c4/README.md')]
[PosixPath('datasets/text/README.md')]
[PosixPath('datasets/lm1b/README.md'), PosixPath('datasets/lm1b/lm1b.py')]
[PosixPath('datasets/pandas/README.md')]
```

For `json`, `text`, csv`, and `pandas` this is expected, but not for the others which should be fixed
GitHub issue #2377: ArrowDataset.save_to_disk produces files that cannot be read using pyarrow.feather
State: open | Created: 2021-05-19T02:04:37Z

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from pyarrow import feather

dataset = load_dataset('imdb', split='train')
dataset.save_to_disk('dataset_dir')
table = feather.read_table('dataset_dir/dataset.arrow')
```

## Expected results
I expect that the saved dataset can be read by the official Apache Arrow methods.

## Actual results
```
  File "/usr/local/lib/python3.7/site-packages/pyarrow/feather.py", line 236, in read_table
    reader.open(source, use_memory_map=memory_map)
  File "pyarrow/feather.pxi", line 67, in pyarrow.lib.FeatherReader.open
  File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not a Feather V1 or Arrow IPC file
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets-1.6.2
- Platform: Linux
- Python version: 3.7
- PyArrow version: 0.17.1, also 2.0.0
GitHub issue #2376: Improve task api code quality
State: closed | Created: 2021-05-18T23:13:40Z

Improves the code quality of the `TaskTemplate` dataclasses.

Changes:
* replaces `return NotImplemented` with raise `NotImplementedError` 
* replaces `sorted` with `len` in the uniqueness check 
* defines `label2id` and `id2label` in the `TextClassification` template as properties
* replaces the `object.__setattr__(self, attr, value)` syntax with (IMO nicer) `self.__dict__[attr] = value`
GitHub issue #2373: Loading dataset from local path
State: closed | Created: 2021-05-18T15:20:50Z

I'm trying to load a local dataset with the code below

```
ds = datasets.load_dataset('my_script.py', 
                           data_files='corpus.txt', 
                           data_dir='/data/dir', 
                           cache_dir='.')
```
But internally a BuilderConfig is created, which tries to use getmtime on the data_files string, without using data_dir. Is this a bug or am I not using the load_dataset correctly?

https://github.com/huggingface/datasets/blob/bc61954083f74e6460688202e9f77dde2475319c/src/datasets/builder.py#L153
GitHub issue #2372: ConvQuestions benchmark added
State: closed | Created: 2021-05-18T15:16:50Z

Hello,
I would like to integrate our dataset on conversational QA. The answers are grounded in the KG.
The work was published in CIKM 2019 (https://dl.acm.org/doi/10.1145/3357384.3358016).
We hope for further research on how to deal with the challenges of factoid conversational QA.
Thanks! :)
GitHub issue #2371: Align question answering tasks with sub-domains
State: closed | Created: 2021-05-18T09:47:59Z

As pointed out by @thomwolf in #2255 we should consider breaking with the pipeline taxonomy of `transformers` to account for the various types of question-answering domains:

> `question-answering` exists in two forms: abstractive and extractive question answering.
> 
> we can keep a generic `question-answering` but then it will probably mean diferrent schema of input/output for both (abstractive will have text for both while extractive can use spans indication as well as text).
> 
> Or we can also propose to use `abstractive-question-answering` and `extractive-question-answering` for instance.
> Maybe we could have `question-answering-abstractive` and `question-answering-extractive` if somehow we can use a for a completion or search in the future (detail).
> Actually I see that people are more organizing in terms of general and sub-tasks, for instance on paperwithcode: https://paperswithcode.com/area/natural-language-processing and on nlpprogress: https://github.com/sebastianruder/NLP-progress/blob/master/english/question_answering.md#squad
> 
> Probably the best is to align with one of these in terms of denomination, PaperWithCode is probably the most active and maintained and we work with them as well.
> Maybe you want to check with a few QA datasets that this schema make sense. Typically NaturalQuestions, TriviaQA and can be good second datasets to compare to and be sure of the generality of the schema.
> 
> A good recent list of QA datasets to compare the schemas among, is for instance in the UnitedQA paper: https://arxiv.org/abs/2101.00178

Investigate which grouping of QA is best suited for `datasets` and adapt / extend the QA task template accordingly.
GitHub issue #2370: Adding HendrycksTest dataset
State: closed | Created: 2021-05-17T18:53:05Z

Adding Hendrycks test from https://arxiv.org/abs/2009.03300.
I'm having a bit of trouble with dummy data creation because some lines in the csv files aren't being loaded properly (only the first entry loaded in a row of length 6). The dataset is loading just fine. Hope you can kindly help!
Thank you!
GitHub issue #2369: correct labels of conll2003
State: closed | Created: 2021-05-17T17:37:54Z

# What does this PR

It fixes/extends the `ner_tags` for conll2003 to include all. 
Paper reference https://arxiv.org/pdf/cs/0306050v1.pdf
Model reference https://huggingface.co/elastic/distilbert-base-cased-finetuned-conll03-english/blob/main/config.json
GitHub issue #2367: Remove getchildren from hyperpartisan news detection
State: closed | Created: 2021-05-17T13:10:37Z

`Element.getchildren()` is now deprecated in the ElementTree library (I think in python 3.9, so it still passes the automated tests which are using 3.6. But for those of us on bleeding-edge distros it now fails).

https://bugs.python.org/issue29209
GitHub issue #2366: Json loader fails if user-specified features don't match the json data fields order
State: closed | Created: 2021-05-17T10:26:08Z

If you do
```python
dataset = load_dataset("json", data_files=data_files, features=features)
```
Then depending on the order of the features in the json data field it fails:
```python
[...]
~/Desktop/hf/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files)
     94             if self.config.schema:
     95                 # Cast allows str <-> int/float, while parse_option explicit_schema does NOT
---> 96                 pa_table = pa_table.cast(self.config.schema)
     97             yield i, pa_table
[...]
ValueError: Target schema's field names are not matching the table's field names: ['tokens', 'ner_tags'], ['ner_tags', 'tokens']
```

This is because one must first re-order the columns of the table to match the `self.config.schema` before calling cast.

One way to fix the `cast` would be to replace it with:
```python
# reorder the arrays if necessary + cast to schema
# we can't simply use .cast here because we may need to change the order of the columns
pa_table = pa.Table.from_arrays([pa_table[name] for name in schema.names], schema=schema)
```
GitHub issue #2365: Missing ClassLabel encoding in Json loader
State: closed | Created: 2021-05-17T10:19:10Z

Currently if you want to load a json dataset this way
```python
dataset = load_dataset("json", data_files=data_files, features=features)
```
Then if your features has ClassLabel types and if your json data needs class label encoding (i.e. if the labels in the json files are strings and not integers), then it would fail:
```python
[...]
~/Desktop/hf/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files)
     94             if self.config.schema:
     95                 # Cast allows str <-> int/float, while parse_option explicit_schema does NOT
---> 96                 pa_table = pa_table.cast(self.config.schema)
     97             yield i, pa_table
[...]
ArrowInvalid: Failed to parse string: 'O' as a scalar of type int64
```

This is because it just tries to cast the string data to integers, without applying the mapping str->int first

The current workaround is to do instead
```python
dataset = load_dataset("json", data_files=data_files)
dataset = dataset.map(features.encode_example, features=features)
```
GitHub issue #2364: README updated for SNLI, MNLI
State: closed | Created: 2021-05-15T11:37:59Z

Closes #2275. Mentioned about -1 labels in MNLI, SNLI and how they should be removed before training. @lhoestq `check_code_quality` test might fail for MNLI as the license name `other-Open Portion of the American National Corpus`  is not a registered tag for 'licenses'
GitHub issue #2359: Allow model labels to be passed during task preparation
State: closed | Created: 2021-05-14T13:58:28Z

Models have a config with label2id. And we have the same for datasets with the ClassLabel feature type. At one point either the model or the dataset must sync with the other. It would be great to do that on the dataset side.

For example for sentiment classification on amazon reviews with you could have these labels:
- "1 star", "2 stars", "3 stars", "4 stars", "5 stars"
- "1", "2", "3", "4", "5"

Some models may use the first set, while other models use the second set.

Here in the `TextClassification` class, the user can only specify one set of labels, while many models could actually be compatible but have different sets of labels. Should we allow users to pass a list of compatible labels sets ?

Then in terms of API, users could use `dataset.prepare_for_task("text-classification", labels=model.labels)` or something like that.

The label set could also be the same but not in the same order. For NLI for example, some models use `["neutral", "entailment", "contradiction"]` and some others use `["neutral", "contradiction", "entailment"]`, so we should take care of updating the order of the labels in the dataset to match the labels order of the model.

Let me know what you think ! This can be done in a future PR

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/pull/2255#discussion_r632412792_
GitHub issue #2358: Roman Urdu Stopwords List
State: closed | Created: 2021-05-13T18:29:27Z

A list of most frequently used Roman Urdu words with different spellings and usages.
This is a very basic effort to collect some basic stopwords for Roman Urdu to help efforts of analyzing text data in roman Urdu which makes up a huge part of daily internet interaction of Roman-Urdu users.
GitHub issue #2357: Adding Microsoft CodeXGlue Datasets
State: closed | Created: 2021-05-13T00:43:01Z

Hi there, this is a new pull request to get the CodeXGlue datasets into the awesome HF datasets lib. Most of the work has been done in this PR #997 by the awesome @madlag. However, that PR has been stale for a while now and so I spoke with @lhoestq about finishing up the final mile and so he told me to open a new PR with the final changes :smile:. 

I believe I've met all of the changes still left in the old PR to do, except for the change to the languages. I believe the READMEs should include the different programming languages used rather than just using the tag "code" as when searching for datasets, SE researchers may specifically be looking only for what type of programming language and so being able to quickly filter will be very valuable. Let me know what you think of that or if you still believe it should be the "code" tag @lhoestq.
GitHub issue #2355: normalized TOCs and titles in data cards
State: closed | Created: 2021-05-12T20:59:59Z

I started fixing some of the READMEs that were failing the tests introduced by @gchhablani but then realized that there were some consistent differences between earlier and newer versions of some of the titles (e.g. Data Splits vs Data Splits Sample Size, Supported Tasks vs Supported Tasks and Leaderboards). We also had different versions of the Table of Content

This PR normalizes all of them to the newer version
GitHub issue #2354: Document DatasetInfo attributes
State: closed | Created: 2021-05-12T20:01:29Z

**Is your feature request related to a problem? Please describe.**
As noted in PR #2255, the attributes of `DatasetInfo` are not documented in the [docs](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=datasetinfo#datasetinfo). It would be nice to do so :)
GitHub issue #2351: simpllify faiss index save
State: closed | Created: 2021-05-12T03:54:10Z

Fixes #2350

In some cases, Faiss GPU index objects do not have neither "device" nor "getDevice". Possibly this happens when some part of the index is computed on CPU.

In particular, this would happen with the index `OPQ16_128,IVF512,PQ32` (issue #2350). I did check it, but it is likely that `OPQ` or `PQ` transforms cause it.

I propose, instead of using the index object to get the device, to infer it form the `FaissIndex.device` field as it is done in `.add_vectors`. Here we assume that `.device` always corresponds to the index placement and it seems reasonable.
GitHub issue #2350: `FaissIndex.save` throws error on GPU
State: closed | Created: 2021-05-12T03:41:56Z

## Describe the bug

After training an index with a factory string `OPQ16_128,IVF512,PQ32` on GPU, `.save_faiss_index` throws this error.

```
  File "index_wikipedia.py", line 119, in <module>
    data["train"].save_faiss_index("text_emb", index_save_path)
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/datasets/search.py", line 470, in save_faiss_index
    index.save(file)
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/datasets/search.py", line 334, in save
    faiss.write_index(index, str(file))
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/faiss/swigfaiss_avx2.py", line 5654, in write_index
    return _swigfaiss.write_index(*args)
RuntimeError: Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /root/miniconda3/conda-bld/faiss-pkg_1613235005464/work/faiss/impl/index_write.cpp:453: don't know how to serialize this type of index
```

## Steps to reproduce the bug

Any dataset will do, I just selected a familiar one.

```python
import numpy as np
import datasets
INDEX_STR = "OPQ16_128,IVF512,PQ32"
INDEX_SAVE_PATH = "will_not_save.faiss"

data = datasets.load_dataset("Fraser/news-category-dataset", split=f"train[:10000]")

def encode(item):
    return {"text_emb": np.random.randn(768).astype(np.float32)}

data = data.map(encode)

data.add_faiss_index(column="text_emb", string_factory=INDEX_STR, train_size=10_000, device=0)
data.save_faiss_index("text_emb", INDEX_SAVE_PATH)
```

## Expected results
Saving the index

## Actual results
Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) ... don't know how to serialize this type of index

## Environment info
- `datasets` version: 1.6.2
- Platform: Linux-4.15.0-142-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


I will be proposing a fix in a couple of minutes