kardwalker/my_dataset huggingface dataset:
GitHub issue #1286: [libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0):  terminate called after throwing an instance of 'google::protobuf::FatalException'   what():  CHECK failed: (index) >= (0):  Aborted
State: closed | Created: 2020-12-08T09:44:15Z

Hi
I am getting this error when evaluating on wmt16-ro-en using finetune_trainer.py of huggingface repo. thank for your help

{'epoch': 20.0}                                                                                                                                             
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:16<00:00,  1.22it/s]
12/08/2020 10:41:19 - INFO - seq2seq.trainers.trainer -   Saving model checkpoint to outputs/experiment/joint/finetune/lr-2e-5
12/08/2020 10:41:24 - INFO - __main__ -   {'wmt16-en-ro': Dataset(features: {'src_texts': Value(dtype='string', id=None), 'task': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 1998), 'qnli': Dataset(features: {'src_texts': Value(dtype='string', id=None), 'task': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 5462), 'scitail': Dataset(features: {'src_texts': Value(dtype='string', id=None), 'task': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 1303)}
12/08/2020 10:41:24 - INFO - __main__ -   *** Evaluate ***
12/08/2020 10:41:24 - INFO - seq2seq.utils.utils -   using task specific params for wmt16-en-ro: {'max_length': 300, 'num_beams': 4}
12/08/2020 10:41:24 - INFO - seq2seq.trainers.trainer -   ***** Running Evaluation *****
12/08/2020 10:41:24 - INFO - seq2seq.trainers.trainer -     Num examples = 1998
12/08/2020 10:41:24 - INFO - seq2seq.trainers.trainer -     Batch size = 64
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:37<00:00,  1.19s/it][libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (index) >= (0): 
Aborted
GitHub issue #1285: boolq does not work 
State: closed | Created: 2020-12-08T09:28:47Z

Hi
I am getting this error when trying to load boolq, thanks for your help

ts_boolq_default_0.1.0_2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11.lock
Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 274, in <module>
    main()
  File "finetune_t5_trainer.py", line 147, in main
    for task in data_args.tasks]
  File "finetune_t5_trainer.py", line 147, in <listcomp>
    for task in data_args.tasks]
  File "/remote/idiap.svm/user.active/rkarimi/dev/ruse/seq2seq/tasks/tasks.py", line 58, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/remote/idiap.svm/user.active/rkarimi/dev/ruse/seq2seq/tasks/tasks.py", line 54, in load_dataset
    return datasets.load_dataset(self.task.name, split=split)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 149, in download_custom
    custom_download(url, path)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/tensorflow/python/lib/io/file_io.py", line 516, in copy_v2
    compat.path_to_bytes(src), compat.path_to_bytes(dst), overwrite)
tensorflow.python.framework.errors_impl.AlreadyExistsError: file already exists
GitHub issue #1283: Add dutch book review dataset
State: closed | Created: 2020-12-08T08:50:48Z

- Name: Dutch Book Review Dataset (DBRD)
- Description: The DBRD (pronounced dee-bird) dataset contains over 110k book reviews along with associated binary sentiment polarity labels and is intended as a benchmark for sentiment classification in Dutch.
- Paper: https://arxiv.org/abs/1910.00896
- Data: https://github.com/benjaminvdb/DBRD
- Motivation: A large (real-life) dataset of Dutch book reviews and sentiment polarity (positive/negative), based on the associated rating.

Checks
- [x] Create the dataset script /datasets/dbrd/dbrd.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _info(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.
GitHub issue #1282: add thaiqa_squad
State: closed | Created: 2020-12-08T08:14:38Z

Example format is a little different from SQuAD since `thaiqa` always have one answer per question so I added a check to convert answers to lists if they are not already one to future-proof additional questions that might have multiple answers.

`thaiqa_squad` is an open-domain, extractive question answering dataset (4,000 questions in `train` and 74 questions in `dev`) in [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format, originally created by [NECTEC](https://www.nectec.or.th/en/) from Wikipedia articles and adapted to [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format by [PyThaiNLP](https://github.com/PyThaiNLP/).
GitHub issue #1276: add One Million Posts Corpus
State: closed | Created: 2020-12-08T00:50:08Z

- **Name:** One Million Posts Corpus
- **Description:** The “One Million Posts” corpus is an annotated data set consisting of user comments posted to an Austrian newspaper website (in German language).
- **Paper:** https://dl.acm.org/doi/10.1145/3077136.3080711
- **Data:** https://github.com/OFAI/million-post-corpus
- **Motivation:** Big German (real-life) dataset containing different annotations around forum moderation with expert annotations.

### Checkbox

- [X] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [X] Fill the `_DESCRIPTION` and `_CITATION` variables
- [X] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [X] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [X] Generate the metadata file `dataset_infos.json` for all configurations
- [X] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [X] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [X] Both tests for the real data and the dummy data pass.
GitHub issue #1274: oclar-dataset
State: closed | Created: 2020-12-07T23:56:45Z

Opinion Corpus for Lebanese Arabic Reviews (OCLAR) corpus is utilizable for Arabic sentiment classification on reviews, including hotels, restaurants, shops, and others. : [homepage](http://archive.ics.uci.edu/ml/datasets/Opinion+Corpus+for+Lebanese+Arabic+Reviews+%28OCLAR%29#)
GitHub issue #1270: add DFKI SmartData Corpus
State: closed | Created: 2020-12-07T23:03:48Z

- **Name:** DFKI SmartData Corpus
- **Description:** DFKI SmartData Corpus is a dataset of 2598 German-language documents which has been annotated with fine-grained geo-entities, such as streets, stops and routes, as well as standard named entity types.
- **Paper:** https://www.dfki.de/fileadmin/user_upload/import/9427_lrec_smartdata_corpus.pdf
- **Data:** https://github.com/DFKI-NLP/smartdata-corpus
- **Motivation:** Contains fine-grained NER labels for German.

### Checkbox

- [X] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [X] Fill the `_DESCRIPTION` and `_CITATION` variables
- [X] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [X] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [X] Generate the metadata file `dataset_infos.json` for all configurations
- [X] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [X] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [X] Both tests for the real data and the dummy data pass.
GitHub issue #1265: Add CovidQA dataset
State: closed | Created: 2020-12-07T17:06:51Z

This PR adds CovidQA, a question answering dataset specifically designed for COVID-19, built by hand from knowledge gathered from Kaggle’s COVID-19 Open Research Dataset Challenge.

Link to the paper: https://arxiv.org/pdf/2004.11339.pdf
Link to the homepage: https://covidqa.ai
GitHub issue #1260: Added NewsPH Raw Dataset
State: closed | Created: 2020-12-07T15:17:53Z

Added the raw version of the NewsPH dataset, which was used to automatically generate the NewsPH-NLI corpus. Dataset of news articles in Filipino from mainstream Philippine news sites on the internet. Can be used as a language modeling dataset or to reproduce the NewsPH-NLI dataset.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks
GitHub issue #1259: Add KorQPair dataset
State: closed | Created: 2020-12-07T14:33:57Z

This PR adds a [Korean paired question dataset](https://github.com/songys/Question_pair) containing labels indicating whether two questions in a given pair are semantically identical. This dataset was used to evaluate the performance of [KoGPT2](https://github.com/SKT-AI/KoGPT2#subtask-evaluations) on a phrase detection downstream task.
GitHub issue #1253: add thainer
State: closed | Created: 2020-12-07T13:41:54Z

ThaiNER (v1.3) is a 6,456-sentence named entity recognition dataset created from expanding the 2,258-sentence
[unnamed dataset](http://pioneer.chula.ac.th/~awirote/Data-Nutcha.zip) by
[Tirasaroj and Aroonmanakun (2012)](http://pioneer.chula.ac.th/~awirote/publications/).
It is used to train NER taggers in [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp).
The NER tags are annotated by [Tirasaroj and Aroonmanakun (2012)]((http://pioneer.chula.ac.th/~awirote/publications/))
for 2,258 sentences and the rest by [@wannaphong](https://github.com/wannaphong/).
The POS tags are done by [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp)'s `perceptron` engine trained on `orchid_ud`.
[@wannaphong](https://github.com/wannaphong/) is now the only maintainer of this dataset.
GitHub issue #1252: Add Naver sentiment movie corpus
State: closed | Created: 2020-12-07T13:33:45Z

Supersedes #1168 

> This PR adds the [Naver sentiment movie corpus](https://github.com/e9t/nsmc), a dataset containing Korean movie reviews from Naver, the most commonly used search engine in Korea. This dataset is often used to benchmark models on Korean NLP tasks, as seen in [this paper](https://www.aclweb.org/anthology/2020.lrec-1.199.pdf).
GitHub issue #1237: Add AmbigQA dataset
State: closed | Created: 2020-12-07T09:07:19Z

# AmbigQA: Answering Ambiguous Open-domain Questions Dataset
Adding the [AmbigQA](https://nlp.cs.washington.edu/ambigqa/) dataset as part of the sprint 🎉 (from Open dataset list for Dataset sprint)

Added both the light and full versions (as seen on the dataset homepage)
The json format changes based on the value of one 'type' field, so I set the unavailable field to an empty list. This is explained in the README -> Data Fields

```py
train_light_dataset = load_dataset('./datasets/ambig_qa',"light",split="train")
val_light_dataset = load_dataset('./datasets/ambig_qa',"light",split="validation")
train_full_dataset = load_dataset('./datasets/ambig_qa',"full",split="train")
val_full_dataset = load_dataset('./datasets/ambig_qa',"full",split="validation")


for example in train_light_dataset:
    for i,t in enumerate(example['annotations']['type']):
        if t =='singleAnswer':
            # use the example['annotations']['answer'][i]
            # example['annotations']['qaPairs'][i] - > is []
            print(example['annotations']['answer'][i])
        else:
            # use the example['annotations']['qaPairs'][i]
            # example['annotations']['answer'][i] - > is []
            print(example['annotations']['qaPairs'][i])

```

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)
GitHub issue #1222: Add numeric fused head dataset
State: closed | Created: 2020-12-06T20:46:53Z

Adding the [NFH: Numeric Fused Head](https://nlp.biu.ac.il/~lazary/fh/) dataset.

Everything looks sensible and I've included both the identification and resolution tasks. I haven't personally used this dataset in my research so am unable to specify what the default configuration / supervised keys should be.

I've filled out the basic info on the model card to the best of my knowledge but it's a little tricky to understand exactly what the fields represent.

Dataset author: @yanaiela
GitHub issue #1221: Add HKCanCor
State: closed | Created: 2020-12-06T20:32:07Z

This PR adds the [Hong Kong Cantonese Corpus](http://compling.hss.ntu.edu.sg/hkcancor/), by [Luke and Wong 2015](http://compling.hss.ntu.edu.sg/hkcancor/data/LukeWong_Hong-Kong-Cantonese-Corpus.pdf). 

The dummy data included here was manually created, as the original dataset uses a xml-like format (see a copy hosted [here](https://github.com/fcbond/hkcancor/blob/master/sample/d1_v.txt) for example) that requires a few processing steps.
GitHub issue #1219: Add Korean NER dataset
State: closed | Created: 2020-12-06T20:19:06Z

Supersedes #1177 

> This PR adds the [Korean named entity recognition dataset](https://github.com/kmounlp/NER). This dataset has been used in many downstream tasks, such as training [KoBERT](https://github.com/SKTBrain/KoBERT) for NER, as seen in this [KoBERT-CRF implementation](https://github.com/eagle705/pytorch-bert-crf-ner).
GitHub issue #1218: Add WMT20 MLQE 3 shared tasks
State: closed | Created: 2020-12-06T19:59:12Z

3 tasks for the WMT 20 MLQE shared tasks -> 3 different datasets

(I re-created #1137 because it was too messy).

Note that in L199 `task3.py`, I used `logging.warning` to print some missing data in the train set.
GitHub issue #1216: Add limit
State: closed | Created: 2020-12-06T19:46:18Z

This PR adds [LiMiT](https://github.com/ilmgut/limit_dataset), a dataset for literal motion classification/extraction by [Manotas et al., 2020](https://www.aclweb.org/anthology/2020.findings-emnlp.88.pdf).
GitHub issue #1214: adding medical-questions-pairs dataset
State: closed | Created: 2020-12-06T19:30:12Z

This dataset consists of 3048 similar and dissimilar medical question pairs hand-generated and labeled by Curai's doctors.
Dataset : https://github.com/curai/medical-question-pair-dataset
Paper : https://drive.google.com/file/d/1CHPGBXkvZuZc8hpr46HeHU6U6jnVze-s/view
GitHub issue #1213: add taskmaster3
State: closed | Created: 2020-12-06T17:56:03Z

Adding Taskmaster-3 dataset
https://github.com/google-research-datasets/Taskmaster/tree/master/TM-3-2020.

The dataset structure almost same as original dataset with these two changes

1. In original dataset, each `apis` has a `args` filed which is a `dict`  with variable keys, which represent the name and value of the args. Here converted that to a `list` of `dict` with keys `arg_name` and `arg_value`. For ex.

```python
args = {"name.movie": "Mulan", "name.theater": ": "Mountain AMC 16"}
```
becomes 
```python
[
  {
    "arg_name": "name.movie",
    "arg_value": "Mulan"
  },
  {
    "arg_name": "name.theater",
    "arg_value": "Mountain AMC 16"
  }
]
```

2. Each `apis` has a `response` which is also a `dict` with variable keys representing response name/type and it's value. As above converted it to `list` of `dict` with keys `response_name` and `response_value`.
GitHub issue #1211: Add large spanish corpus
State: closed | Created: 2020-12-06T17:06:50Z

Adds a collection of Spanish corpora that can be useful for pretraining language models. 

Following a nice suggestion from @yjernite we provide the user with three main ways to preprocess / load either 

* the whole corpus (17GB!)
* one specific sub-corpus
* the whole corpus, but return a single split. this is useful if you want to cache the whole preprocessing step once and interact with individual sub-corpora

See the dataset card for more details.

Ready for review!
GitHub issue #1209: [AfriBooms] Dataset exists already
State: closed | Created: 2020-12-06T16:35:13Z

When trying to add "AfriBooms": https://docs.google.com/spreadsheets/d/12ShVow0M6RavnzbBEabm5j5dv12zBaf0y-niwEPPlo4/edit#gid=1386399609 I noticed that the dataset exists already as a config of Universal Dependencies (universal_dependencies.py). I checked and the data exactly matches so that the new data link does not give any new data.

This PR improves the config's description a bit by linking to the paper.
GitHub issue #1205: add lst20 with manual download
State: closed | Created: 2020-12-06T14:49:10Z

passed on local:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_lst20
```
Not sure how to test:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_lst20
```

```
LST20 Corpus is a dataset for Thai language processing developed by National Electronics and Computer Technology Center (NECTEC), Thailand.
It offers five layers of linguistic annotation: word boundaries, POS tagging, named entities, clause boundaries, and sentence boundaries.
At a large scale, it consists of 3,164,002 words, 288,020 named entities, 248,181 clauses, and 74,180 sentences, while it is annotated with
16 distinct POS tags. All 3,745 documents are also annotated with one of 15 news genres. Regarding its sheer size, this dataset is
considered large enough for developing joint neural models for NLP.
Manually download at https://aiforthai.in.th/corpus.php
```
GitHub issue #1202: Medical question pairs
State: closed | Created: 2020-12-06T14:09:07Z

This dataset consists of 3048 similar and dissimilar medical question pairs hand-generated and labeled by Curai's doctors.
Dataset : https://github.com/curai/medical-question-pair-dataset
Paper : https://drive.google.com/file/d/1CHPGBXkvZuZc8hpr46HeHU6U6jnVze-s/view
**No splits added**
GitHub issue #1200: Update ADD_NEW_DATASET.md
State: closed | Created: 2020-12-06T13:31:32Z

Windows needs special treatment again: unfortunately adding `torch` to the requirements does not work well (crashing the installation). Users should first install torch manually and then continue with the other commands.

This issue arises all the time when adding torch as a dependency, but because so many novice users seem to participate in adding datasets, it may be useful to add an explicit note for Windows users to ensure that they do not run into issues.
GitHub issue #1199: Turkish NER dataset, script works fine, couldn't generate dummy data
State: closed | Created: 2020-12-06T12:00:03Z

I've written the script (Turkish_NER.py) that includes dataset. The dataset is a zip inside another zip, and it's extracted as .DUMP file. However, after preprocessing I only get .arrow file. After I ran the script with no error messages, I get .arrow file of dataset, LICENSE and dataset_info.json.
GitHub issue #1195: addition of py_ast
State: closed | Created: 2020-12-06T10:00:52Z

The dataset consists of parsed Parsed ASTs that were used to train and evaluate the DeepSyn tool. 
The Python programs are collected from GitHub repositories
by removing duplicate files, removing project forks (copy of another existing repository)
,keeping only programs that parse and have at most 30'000 nodes in the AST and 
we aim to remove obfuscated files
GitHub issue #1192: Add NewsPH_NLI dataset
State: closed | Created: 2020-12-06T04:00:31Z

This PR adds the NewsPH-NLI Dataset, the first benchmark dataset for sentence entailment in the low-resource Filipino language. Constructed through exploting the structure of news articles. Contains 600,000 premise-hypothesis pairs, in 70-15-15 split for training, validation, and testing.

Link to the paper: https://arxiv.org/pdf/2010.11574.pdf

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks
GitHub issue #1190: Add Fake News Detection in Filipino dataset
State: closed | Created: 2020-12-06T03:12:15Z

This PR adds the Fake News Filipino Dataset, a low-resource fake news detection corpora in Filipino. Contains 3,206 expertly-labeled news samples, half of which are real and half of which are fake.

Link to the paper: http://www.lrec-conf.org/proceedings/lrec2020/index.html

Link to the dataset/repo: https://github.com/jcblaisecruz02/Tagalog-fake-news
GitHub issue #1189: Add Dengue dataset in Filipino
State: closed | Created: 2020-12-06T02:50:47Z

This PR adds the Dengue Dataset, a benchmark dataset for low-resource multiclass classification, with 4,015 training, 500 testing, and 500 validation examples, each labeled as part of five classes. Each sample can be a part of multiple classes. Collected as tweets.

Link to the paper: https://ieeexplore.ieee.org/document/8459963

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks
GitHub issue #1185: Add Hate Speech Dataset in Filipino
State: closed | Created: 2020-12-06T02:01:56Z

This PR adds the Hate Speech Dataset, a text classification dataset in Filipino, consisting 10k tweets (training set) that are labeled as hate speech or non-hate speech. Released with 4,232 validation and 4,232 testing samples. Collected during the 2016 Philippine Presidential Elections.

Link to the paper: https://pcj.csp.org.ph/index.php/pcj/issue/download/29/PCJ%20V14%20N1%20pp1-14%202019

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks
GitHub issue #1184: Add Adversarial SQuAD dataset
State: closed | Created: 2020-12-05T23:51:57Z

# Adversarial SQuAD

Adding the Adversarial [SQuAD](https://github.com/robinjia/adversarial-squad) dataset as part of the sprint 🎉 
This dataset adds adversarial sentences to a subset of the SQuAD dataset's dev examples. How to get the original squad example id is explained in readme->Data Instances. The whole data is intended for use in evaluation. (Which could of course be also used for training if one wants). So there is no classical train/val/test split, but a split based on the number of adversaries added.

There are 2 splits of this dataset:

- AddSent: Has up to five candidate adversarial sentences that don't answer the question, but have a lot of words in common with the question. This adversary is does not query the model in any way.
- AddOneSent: Similar to AddSent, but just one candidate sentences was picked at random. This adversary is does not query the model in any way.

(The AddAny and AddCommon datasets mentioned in the paper are dynamically generated based on model's output distribution thus are not included here)

The failing test look like some unrelated timeout thing, will probably clear if rerun.
- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)
GitHub issue #1182: ADD COVID-QA dataset
State: closed | Created: 2020-12-05T23:31:56Z

This PR adds the COVID-QA dataset, a question answering dataset consisting of 2,019 question/answer pairs annotated by volunteer biomedical experts on scientific articles related to COVID-19

Link to the paper: https://openreview.net/forum?id=JENSKEEzsoU
Link to the dataset/repo: https://github.com/deepset-ai/COVID-QA
GitHub issue #1180: Add  KorQuAD v2 Dataset
State: closed | Created: 2020-12-05T21:33:34Z

# The Korean Question Answering Dataset v2
Adding the [KorQuAD](https://korquad.github.io/) v2 dataset as part of the sprint 🎉 
This dataset is very similar to SQuAD and is an extension of [squad_kor_v1](https://github.com/huggingface/datasets/pull/1178) which is why I added it as `squad_kor_v2`. 

- Crowd generated questions and answer (1-answer per question) for Wikipedia articles. Differently from V1 it includes the html structure and markup, which makes it a different enough dataset. (doesn't share ids between v1 and v2 either)

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

Edit: 🤦 looks like squad_kor_v1 commit sneaked in here too
GitHub issue #1178: Add  KorQuAD v1 Dataset
State: closed | Created: 2020-12-05T21:25:46Z

# The Korean Question Answering Dataset
Adding the [KorQuAD](https://korquad.github.io/KorQuad%201.0/) v1 dataset as part of the sprint 🎉 
This dataset is very similar to SQuAD which is why I added it as `squad_kor_v1`. There is also a v2 which I added [here](https://github.com/huggingface/datasets/pull/1180).

- Crowd generated questions and answer (1-answer per question) for Wikipedia articles.

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)
GitHub issue #1177: Add Korean NER dataset
State: closed | Created: 2020-12-05T20:56:00Z

This PR adds the [Korean named entity recognition dataset](https://github.com/kmounlp/NER). This dataset has been used in many downstream tasks, such as training [KoBERT](https://github.com/SKTBrain/KoBERT) for NER, as seen in this [KoBERT-CRF implementation](https://github.com/eagle705/pytorch-bert-crf-ner).
GitHub issue #1168: Add Naver sentiment movie corpus
State: closed | Created: 2020-12-05T17:25:23Z

This PR adds the [Naver sentiment movie corpus](https://github.com/e9t/nsmc), a dataset containing Korean movie reviews from Naver, the most commonly used search engine in Korea. This dataset is often used to benchmark models on Korean NLP tasks, as seen in [this paper](https://www.aclweb.org/anthology/2020.lrec-1.199.pdf).
GitHub issue #1167: ❓ On-the-fly tokenization with datasets, tokenizers, and torch Datasets and Dataloaders
State: closed | Created: 2020-12-05T17:02:56Z

Hi there,

I have a question regarding "on-the-fly" tokenization. This question was elicited by reading the "How to train a new language model from scratch using Transformers and Tokenizers" [here](https://huggingface.co/blog/how-to-train). Towards the end there is this sentence: "If your dataset is very large, you can opt to load and tokenize examples on the fly, rather than as a preprocessing step". I've tried coming up with a solution that would combine both `datasets` and `tokenizers`, but did not manage to find a good pattern.

I guess the solution would entail wrapping a dataset into a Pytorch dataset.

As a concrete example from the [docs](https://huggingface.co/transformers/custom_datasets.html)

```python
import torch

class SquadDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        # instead of doing this beforehand, I'd like to do tokenization on the fly
        self.encodings = encodings 

    def __getitem__(self, idx):
        return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}

    def __len__(self):
        return len(self.encodings.input_ids)

train_dataset = SquadDataset(train_encodings)
```

How would one implement this with "on-the-fly" tokenization exploiting the vectorized capabilities of tokenizers?


----

Edit: I have come up with this solution. It does what I want, but I feel it's not very elegant

```python
class CustomPytorchDataset(Dataset):
    def __init__(self):
        self.dataset = some_hf_dataset(...)
        self.tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")

    def __getitem__(self, batch_idx):
        instance = self.dataset[text_col][batch_idx]
        tokenized_text = self.tokenizer(instance, truncation=True, padding=True)
        return tokenized_text

    def __len__(self):
        return len(self.dataset)

    @staticmethod
    def collate_fn(batch):
        # batch is a list, however it will always contain 1 item because we should not use the
        # batch_size argument as batch_size is controlled by the sampler
        return {k: torch.tensor(v) for k, v in batch[0].items()}

torch_ds = CustomPytorchDataset()

# NOTE: batch_sampler returns list of integers and since here we have SequentialSampler
# it returns: [1, 2, 3], [4, 5, 6], etc. - check calling `list(batch_sampler)`
batch_sampler = BatchSampler(SequentialSampler(torch_ds), batch_size=3, drop_last=True)

# NOTE: no `batch_size` as now the it is controlled by the sampler!
dl = DataLoader(dataset=torch_ds, sampler=batch_sampler, collate_fn=torch_ds.collate_fn)
```
GitHub issue #1160: adding TabFact dataset
State: closed | Created: 2020-12-05T13:05:52Z

Adding TabFact: A Large-scale Dataset for Table-based Fact Verification.

https://github.com/wenhuchen/Table-Fact-Checking

- The tables are stored as individual csv files, so need to download 16,573 🤯  csv files. As a result the `datasets_infos.json` file is huge (6.62 MB).
- Original dataset has nested structure where, where table is one example and each table has multiple statements,
flattening the structure here so that each statement is one example.
GitHub issue #1144: Add JFLEG
State: closed | Created: 2020-12-04T22:36:38Z

This PR adds [JFLEG ](https://www.aclweb.org/anthology/E17-2037/), an English grammatical error correction benchmark. 

The tests were successful on real data, although it would be great if I can get some guidance on the **dummy data**. Basically, **for each source sentence there are 4 possible gold standard target sentences**. The original dataset comprise files in a flat structure, labelled by split then by source/target (e.g., dev.src, dev.ref0, ..., dev.ref3). Not sure what is the best way of adding this.

I imagine I can treat each distinct source-target pair as its own split? But having so many copies of the source sentence feels redundant, and it would make it less convenient to end-users who might want to access multiple gold standard targets simultaneously.
GitHub issue #1143: Add the Winograd Schema Challenge
State: closed | Created: 2020-12-04T22:26:59Z

Adds the Winograd Schema Challenge, including configs for the more canonical wsc273 as well as wsc285 with 12 new examples.

- https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html

The data format was a bit of a nightmare but I think I got it to a workable format.
GitHub issue #1139: Add ReFreSD dataset
State: closed | Created: 2020-12-04T20:45:11Z

This PR adds the **ReFreSD dataset**. 
The original data is hosted [on this github repo](https://github.com/Elbria/xling-SemDiv) and we use the `REFreSD_rationale` to expose all the data. 


Need feedback on:
- I couldn't generate the dummy data. The file we download is a tsv file, but without extension, I suppose this is the problem. I'm sure there is a simple trick to make this work. 
- The feature names. 
  - I don't know if it's better to stick to the classic `sentence1`, `sentence2` or to `sentence_en`, `sentence_fr` to be more explicit. 
  - There is a binary label (called `label`, no problem here), and a 3-class label called `#3_labels` in the original tsv. I changed it to `all_labels` but I'm sure there is better. 
- The rationales are lists of integers, extracted as a string at first. I wonder what's the best way to treat them, any idea? Also, I couldn't manage to make a `Sequence` of `int8` but I'm sure I've missed something simple. 

Thanks in advance
GitHub issue #1137: add wmt mlqe 2020 shared task
State: closed | Created: 2020-12-04T19:45:34Z

First commit for Shared task 1 (wmt_mlqw_task1) of WMT20 MLQE (quality estimation of machine translation)
Note that I copied the tags in the README for only one (of the 7 configurations): `en-de`.
There is one configuration for each pair of languages.
GitHub issue #1129: Adding initial version of cord-19 dataset
State: closed | Created: 2020-12-04T17:03:17Z

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.

### TODO:
- [x] add more metadata
- [x] add full text
- [x] add pre-computed document embedding
GitHub issue #1115: Incorrect URL for MRQA SQuAD train subset
State: closed | Created: 2020-12-04T14:05:24Z

https://github.com/huggingface/datasets/blob/4ef4c8f8b7a60e35c6fa21115fca9faae91c9f74/datasets/mrqa/mrqa.py#L53

The URL for `train+SQuAD` subset of MRQA points to the dev set instead of train set. It should be `https://s3.us-east-2.amazonaws.com/mrqa/release/v2/train/SQuAD.jsonl.gz`.
GitHub issue #1112: Initial version of cord-19 dataset from AllenAI with only the abstract
State: closed | Created: 2020-12-04T13:36:39Z

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [ ] Both tests for the real data and the dummy data pass.

### TODO:
- [ ] add more metadata
- [ ] add full text
- [ ] add pre-computed document embedding
GitHub issue #1110: Using a feature named "_type" fails with certain operations
State: closed | Created: 2020-12-04T12:56:33Z

A column named `_type` leads to a `TypeError: unhashable type: 'dict'` for certain operations:
```python
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({"_type": ["whatever"]}).map()
concatenate_datasets([ds])
# or simply
Dataset(ds._data)
```
Context: We are using datasets to persist data coming from elasticsearch to feed to our pipeline, and elasticsearch has a `_type` field, hence the strange name of the column.

Not sure if you wish to support this specific column name, but if you do i would be happy to try a fix and provide a PR. I already had a look into it and i think the culprit is the `datasets.features.generate_from_dict` function. It uses the hard coded `_type` string to figure out if it reached the end of the nested feature object from a serialized dict.

Best wishes and keep up the awesome work!
GitHub issue #1107: Add arsentd_lev dataset
State: closed | Created: 2020-12-04T11:31:04Z

Add The Arabic Sentiment Twitter Dataset for Levantine dialect (ArSenTD-LEV)

Paper: [ArSentD-LEV: A Multi-Topic Corpus for Target-based Sentiment Analysis in Arabic Levantine Tweets](https://arxiv.org/abs/1906.01830)
Homepage: http://oma-project.com/
GitHub issue #1090: add thaisum
State: closed | Created: 2020-12-04T05:54:48Z

ThaiSum, a large-scale corpus for Thai text summarization obtained from several online news websites namely Thairath, ThaiPBS, Prachathai, and The Standard. This dataset consists of over 350,000 article and summary pairs written by journalists. We evaluate the performance of various existing summarization models on ThaiSum dataset and analyse the characteristic of the dataset to present its difficulties.
GitHub issue #1087: Add Big Patent dataset
State: closed | Created: 2020-12-04T04:37:30Z

* More info on the dataset: https://evasharma.github.io/bigpatent/
* There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.
GitHub issue #1086: adding cdt dataset
State: closed | Created: 2020-12-04T01:28:11Z

- **Name:** *Cyberbullying Detection Task*
- **Description:** *The Cyberbullying Detection task was part of 2019 edition of PolEval competition. The goal is to predict if a given Twitter message contains a cyberbullying (harmful) content.*
- **Data:** *https://github.com/ptaszynski/cyberbullying-Polish*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*
GitHub issue #1084: adding cdsc dataset
State: closed | Created: 2020-12-04T00:10:05Z

- **Name**: *cdsc (domains: cdsc-e & cdsc-r)*
- **Description**: *Polish CDSCorpus consists of 10K Polish sentence pairs which are human-annotated for semantic relatedness and entailment. The dataset may be used for the evaluation of compositional distributional semantics models of Polish. The dataset was presented at ACL 2017. Please refer to the Wróblewska and Krasnowska-Kieraś (2017) for a detailed description of the resource.*
- **Data**: *http://2019.poleval.pl/index.php/tasks/*
- **Motivation**: *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*
GitHub issue #1083: Add the multilingual Exams dataset
State: closed | Created: 2020-12-04T00:06:04Z

https://github.com/mhardalov/exams-qa

`multilingual` configs have all languages mixed together

`crosslingual` mixes the languages for test but separates them for train and dec, so I've made one config per language for train/dev data and one config with the joint test set
GitHub issue #1080: Add WikiANN NER dataset
State: closed | Created: 2020-12-03T23:09:24Z

This PR adds the full set of 176 languages from the balanced train/dev/test splits of WikiANN / PAN-X from: https://github.com/afshinrahimi/mmner

Until now, only 40 of these languages were available in `datasets` as part of the XTREME benchmark

Courtesy of the dataset author, we can now download this dataset from a Dropbox URL without needing a manual download anymore 🥳, so at some point it would be worth updating the PAN-X subset of XTREME as well 😄 

Link to gist with some snippets for producing dummy data: https://gist.github.com/lewtun/5b93294ab6dbcf59d1493dbe2cfd6bb9

P.S. @yjernite I think I was confused about needing to generate a set of YAML tags per config, so ended up just adding a single one in the README.
GitHub issue #1079: nkjp-ner
State: closed | Created: 2020-12-03T22:47:26Z

- **Name:** *nkjp-ner*
- **Description:** *The NKJP-NER is based on a human-annotated part of NKJP. We extracted sentences with named entities of exactly one type. The task is to predict the type of the named entity.*
- **Data:** *https://klejbenchmark.com/tasks/*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*
GitHub issue #1064: Not support links with 302 redirect 
State: closed | Created: 2020-12-03T17:04:43Z

I have an issue adding this download link https://github.com/jitkapat/thailitcorpus/releases/download/v.2.0/tlc_v.2.0.tar.gz

it might be because it is not a direct link (it returns 302 and redirects to aws that returns 403 for head requests). 

```
r.head("https://github.com/jitkapat/thailitcorpus/releases/download/v.2.0/tlc_v.2.0.tar.gz", allow_redirects=True)                                      
# <Response [403]>
```
GitHub issue #1062: Add KorNLU dataset
State: closed | Created: 2020-12-03T16:52:39Z

Added Korean NLU datasets. The link to the dataset can be found [here](https://github.com/kakaobrain/KorNLUDatasets) and the paper can be found [here](https://arxiv.org/abs/2004.03289)

**Note**: The MNLI tsv file is broken, so this code currently excludes the file. Please suggest other alternative if any @lhoestq 

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data
GitHub issue #1060: Fix squad V2 metric script
State: closed | Created: 2020-12-03T16:23:32Z

The current squad v2 metric doesn't work with the squad (v1 or v2) datasets. The script is copied from `squad_evaluate` in transformers that requires the labels (with multiple answers) to be like this:
```
references = [{'id': 'a', 'answers': [
    {'text': 'Denver Broncos', 'answer_start': 177},
    {'text': 'Denver Broncos', 'answer_start': 177}
]}]
```
while the dataset had references like this:
```
references = [{'id': 'a', 'answers': 
    {'text': ['Denver Broncos' 'Denver Broncos'], 'answer_start': [177, 177]}
}]
```

Using one or the other format fails with the current squad v2 metric:
```
from datasets import load_metric
metric = load_metric("squad_v2")
predictions = [{'id': 'a', 'prediction_text': 'Denver Broncos', 'no_answer_probability': 0.0}]
references = [{'id': 'a', 'answers': [
    {'text': 'Denver Broncos', 'answer_start': 177},
    {'text': 'Denver Broncos', 'answer_start': 177}
]}]
metric.compute(predictions=predictions, references=references)
```
fails as well as
```
from datasets import load_metric
metric = load_metric("squad_v2")
predictions = [{'id': 'a', 'prediction_text': 'Denver Broncos', 'no_answer_probability': 0.0}]
references = [{'id': 'a', 'answers': 
    {'text': ['Denver Broncos' 'Denver Broncos'], 'answer_start': [177, 177]}
}]
metric.compute(predictions=predictions, references=references)
```

This is because arrow reformats the references behind the scene.

With this PR (tested locally), both the snippets up there work and return proper results.
GitHub issue #1056: Add deal_or_no_dialog
State: closed | Created: 2020-12-03T15:38:07Z

Add deal_or_no_dialog Dataset

github: https://github.com/facebookresearch/end-to-end-negotiator
Paper: [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://arxiv.org/abs/1706.05125)
GitHub issue #1053: Fix dataset URL and file names, and add column name in "Social Bias Frames" dataset
State: closed | Created: 2020-12-03T13:03:05Z

# Why I did
When I use "social_bias_frames" datasets in this library, I got 404 Errors.
So, I fixed this error and another some problems that I faced to use the dataset.

# What I did
* Modify this dataset URL
* Modify this dataset file names
* Add a "dataSource" column

Thank you!
GitHub issue #1050: Add GoEmotions
State: closed | Created: 2020-12-03T12:49:53Z

Adds the GoEmotions dataset, a nice emotion classification dataset with 27 (multi-)label annotations on reddit comments. Includes both a large raw version and a narrowed version with predefined train/test/val splits, which I've included as separate configs with the latter as a default.

- Webpage/repo: https://github.com/google-research/google-research/tree/master/goemotions
- Paper: https://arxiv.org/abs/2005.00547
GitHub issue #1048: Adding NCHLT dataset
State: closed | Created: 2020-12-03T11:59:25Z

https://repo.sadilar.org/handle/20.500.12185/7/discover?filtertype_0=database&filtertype_1=title&filter_relational_operator_1=contains&filter_relational_operator_0=equals&filter_1=&filter_0=Monolingual+Text+Corpora%3A+Annotated&filtertype=project&filter_relational_operator=equals&filter=NCHLT+Text+II
GitHub issue #1047: Add KorNLU
State: closed | Created: 2020-12-03T11:50:54Z

Added Korean NLU datasets. The link to the dataset can be found [here](https://github.com/kakaobrain/KorNLUDatasets) and the paper can be found [here](https://arxiv.org/abs/2004.03289)

**Note**: The MNLI tsv file is broken, so this code currently excludes the file. Please suggest other alternative if any @lhoestq 

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data
GitHub issue #1046: Dataset.map() turns tensors into lists?
State: closed | Created: 2020-12-03T11:43:46Z

I apply `Dataset.map()` to a function that returns a dict of torch tensors (like a tokenizer from the repo transformers). However, in the mapped dataset, these tensors have turned to lists!

```import datasets
import torch  
from datasets import load_dataset                                                                                                                 
print("version datasets", datasets.__version__)

dataset = load_dataset("snli", split='train[0:50]')  

def tokenizer_fn(example):
    # actually uses a tokenizer which does something like:
    return {'input_ids': torch.tensor([[0, 1, 2]])}

print("First item in dataset:\n", dataset[0])
tokenized = tokenizer_fn(dataset[0])
print("Tokenized hyp:\n", tokenized)
dataset_tok = dataset.map(tokenizer_fn, batched=False,
        remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
dataset_tok = dataset.map(tokenizer_fn, batched=False,
                          remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
```

The output is:

```
version datasets 1.1.3
Reusing dataset snli (/home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c)
First item in dataset:
 {'premise': 'A person on a horse jumps over a broken down airplane.', 'hypothesis': 'A person is training his horse for a competition.', 'label': 1}
Tokenized hyp:
 {'input_ids': tensor([[0, 1, 2]])}
Loading cached processed dataset at /home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c/cache-fe38f449fe9ac46f.arrow
Tokenized using map:
 {'input_ids': [[0, 1, 2]]}
<class 'torch.Tensor'> <class 'list'>
```

Or am I doing something wrong?
GitHub issue #1042: Add Big Patent dataset
State: closed | Created: 2020-12-03T11:07:59Z

- More info on the dataset: https://evasharma.github.io/bigpatent/
- There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.
- ~Currently, there are no dummy data for this dataset yet as I'm facing some problems with generating them. I'm trying to add them later.~