init commit for MultiReQA for third PR with all issues fixed

3rd PR w.r.t. PR #1349 with all the issues fixed. As #1349 had uploaded other files along with the multi_re_qa dataset


Add german common crawl dataset

Adding a subpart of the Common Crawl which was extracted with this repo https://github.com/facebookresearch/cc_net and additionally filtered for duplicates


Add yoruba wordsim353

Added WordSim-353 evaluation dataset for Yoruba


Imppres

2nd PR ever! Hopefully I'm starting to get the hang of this. This is for the IMPPRES dataset. Please let me know of any corrections or changes that need to be made.


Can't map dataset (loaded from csv)

Hello! I am trying to load single csv file with two columns: ('label': str, 'text' str), where is label is str of two possible classes.

Below steps are similar with [this notebook](https://colab.research.google.com/drive/1-JIJlao4dI-Ilww_NnTc0rxtp-ymgDgM?usp=sharing), where bert model and tokenizer are used to classify lmdb loaded dataset. Only one difference it is the dataset loaded from .csv file.
Here is how I load it:

```python
data_path = 'data.csv'
data = pd.read_csv(data_path)

# process class name to indices
classes = ['neg', 'pos']
class_to_idx = { cl: i for i, cl in enumerate(classes) }

# now data is like {'label': int, 'text' str}
data['label'] = data['label'].apply(lambda x: class_to_idx[x])

# load dataset and map it with defined `tokenize` function
features = Features({
  target: ClassLabel(num_classes=2, names=['neg', 'pos'], names_file=None, id=None),
  feature: Value(dtype='string', id=None),
})
dataset = Dataset.from_pandas(data, features=features)
dataset.map(tokenize, batched=True, batch_size=len(dataset))
```

It ruins on the last line with following error:
```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-112-32b6275ce418> in <module>()
      9 })
     10 dataset = Dataset.from_pandas(data, features=features)
---> 11 dataset.map(tokenizer, batched=True, batch_size=len(dataset))

2 frames
/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1237         test_inputs = self[:2] if batched else self[0]
   1238         test_indices = [0, 1] if batched else 0
-> 1239         update_data = does_function_return_dict(test_inputs, test_indices)
   1240         logger.info("Testing finished, running the mapping function on the dataset")
   1241

/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py in does_function_return_dict(inputs, indices)
   1208             fn_args = [inputs] if input_columns is None else [inputs[col] for col in input_columns]
   1209             processed_inputs = (
-> 1210                 function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1211             )
   1212             does_return_dict = isinstance(processed_inputs, Mapping)

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py in __call__(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   2281             )
   2282         ), (
-> 2283             "text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) "
   2284             "or `List[List[str]]` (batch of pretokenized examples)."
   2285         )

AssertionError: text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).
```

which I think is not expected. I also tried the same steps using `Dataset.from_csv` which resulted in the same error.

For reproducing this, I used [this dataset from kaggle](https://www.kaggle.com/team-ai/spam-text-message-classification)


adding fake-news-english-2


Add dataset yoruba_wordsim353

Contains loading script as well as dataset card including YAML tags.


Add Turkish News Category Dataset (270K)

This PR adds the Turkish News Categories Dataset (270K) dataset which is a text classification dataset by me and @yavuzKomecoglu.  Turkish news dataset consisting of **273601 news** in **17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.


Add arabic dialects

Data loading script and dataset card for Dialectal Arabic Resources dataset.
Fixed git issues from PR  #976


WIP: Vinay/add peer read dataset


Add Shrinked Turkish NER from Kaggle.

Add Shrinked Turkish NER from [Kaggle](https://www.kaggle.com/behcetsenturk/shrinked-twnertc-turkish-ner-data-by-kuzgunlar).


Add Hate Speech and Offensive Language Detection dataset

Add [Hate Speech and Offensive Language Detection dataset](https://github.com/t-davidson/hate-speech-and-offensive-language) from [this paper](https://arxiv.org/abs/1703.04009).


Adding BioCreative II Gene Mention corpus


Add OffComBR

Add [OffComBR](https://github.com/rogersdepelle/OffComBR) from [Offensive Comments in the Brazilian Web: a dataset and baseline results](https://sol.sbc.org.br/index.php/brasnam/article/view/3260/3222) paper.

But I'm having a hard time generating dummy data since the original dataset extion is `.arff` and the [_create_dummy_data function](https://github.com/huggingface/datasets/blob/a4aeaf911240057286a01bff1b1d75a89aedd57b/src/datasets/commands/dummy_data.py#L185) doesn't allow it.


Adding the ASSIN dataset

Adding the ASSIN dataset, a Portuguese language dataset for Natural Language Inference and Semantic Similarity Scoring


2 typos

Corrected 2 typos


Add penn treebank dataset


Adding the ASSIN dataset


adding fake-news-english


Add Tweet Eval Dataset


Add Portuguese Hate Speech dataset

Binary Portuguese Hate Speech dataset from [this paper](https://www.aclweb.org/anthology/W19-3510/).


Adding TaPaCo Dataset with README.md


Add Acronym Identification Dataset


Add dataset clickbait_news_bg

Adding a new dataset - clickbait_news_bg


adding covid-tweets-japanese (again)

I had mistaken use git rebase, I was so hurried to fix it. However, I didn't fully consider the use of git reset , so I unintendedly stopped PR (#1367) altogether. Sorry about that.
I'll make a new PR.


Add reasoning_bg

Adding reading comprehension dataset for Bulgarian language


Add European Union Education and Culture Translation Memory (EAC-TM) dataset

Adding the EAC Translation Memory dataset : https://ec.europa.eu/jrc/en/language-technologies/eac-translation-memory


Add HoVer Dataset

HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification
https://arxiv.org/abs/2011.03088


Add Neural Code Search Dataset


datasets card-creator link added

dataset card creator link has been added
link: https://huggingface.co/datasets/card-creator/


initial commit for MultiReQA for second PR

Since last PR #1349 had some issues passing the tests. So, a new PR is generated.


Add WikiSource Dataset


Add OfisPublik Dataset


Add script_version suggestion when dataset/metric not found

Adds a helpful prompt to the error message when a dataset/metric is not found, suggesting the user might need to pass `script_version="master"` if the dataset was added recently. The whole error looks like:

> Couldn't find file locally at blah/blah.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1/metrics/blah/blah.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/met
rics/blah/blah.py.
If the dataset was added recently, you may need to to pass script_version="master" to find the loading script on the master branch.


Add KDE4 Dataset


Add MultiParaCrawl Dataset


Add SPC Dataset


add amazon polarity dataset

This corresponds to the amazon (binary dataset) requested in https://github.com/huggingface/datasets/issues/353


hind_encorp

resubmit of hind_encorp file changes


Add LIAR dataset

Add LIAR dataset from [“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection](https://www.aclweb.org/anthology/P17-2067/).


Add RecipeNLG Dataset (manual download)


add best2009

`best2009` is a Thai word-tokenization dataset from encyclopedia, novels, news and articles by [NECTEC](https://www.nectec.or.th/) (148,995/2,252 lines of train/test). It was created for [BEST 2010: Word Tokenization Competition](https://thailang.nectec.or.th/archive/indexa290.html?q=node/10). The test set answers are not provided publicly.


Add News Commentary Dataset


added conv ai 2

Dataset : https://github.com/DeepPavlov/convai/tree/master/2018


adding UNPC

Adding United Nations Parallel Corpus
http://opus.nlpl.eu/UNPC.php


Add twi text c3

Added Twi texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/


Add Tatoeba Dataset


Add yoruba text c3

Added Yoruba texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/


Add FACTCK.BR dataset

This PR adds [FACTCK.BR](https://github.com/jghm-f/FACTCK.BR) dataset from [FACTCK.BR: a new dataset to study fake news](https://dl.acm.org/doi/10.1145/3323503.3361698).


adding marathi-wiki dataset

Adding marathi-wiki-articles dataset.


Add SETimes Dataset


Add OPUS EMEA Dataset


Add OPUS Tilde Model Dataset


Add OPUS ECB Dataset


Add OPUS Books Dataset


Adding Scielo

Adding Scielo: Parallel corpus of full-text articles in Portuguese, English and Spanish from SciELO
https://sites.google.com/view/felipe-soares/datasets#h.p_92uSCyAjWSRB


Add OPUS PHP Dataset


Use passed --cache_dir for modules cache

When passed `--cache_dir` arg:
```shell
python datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>
```
it is not used for caching the modules, which are cached in the default location at `.cache/huggingface/modules`.

With this fix, the modules will be cached at `<my-cache-dir>/modules`.


Re-adding narrativeqa dataset

An update of #309.


adding covid-tweets-japanese

Adding COVID-19 Japanese Tweets Dataset as part of the sprint.

Testing with dummy data is not working (the file is said to not exist). Sorry for the incomplete PR.


Adding Hope EDI dataset


Add Mkqa dataset

# MKQA: Multilingual Knowledge Questions & Answers Dataset
Adding the [MKQA](https://github.com/apple/ml-mkqa) dataset as part of the sprint 🎉

There is no official data splits so I added just a `train` split.

differently from the original:
- answer:type field is a ClassLabel (I thought it might be possible to train on this as a label for categorizing questions)
- answer:entity field has a default value of empty string '' (since this key is not available for all in original)
- answer:alias has default value of []

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


Narrative QA (Manual Download Stories) Dataset

Narrative QA with manual download for stories.


Adding OPUS MultiUN

Adding UnMulti
http://www.euromatrixplus.net/multi-un/


adding opus_infopankki

Adding opus_infopankki
http://opus.nlpl.eu/infopankki-v1.php


adding bprec

Brand-Product Relation Extraction Corpora in Polish


add wisesight1000

`wisesight1000` contains Thai social media texts randomly drawn from the full `wisesight-sentiment`, tokenized by human annotators. Out of the labels `neg` (negative), `neu` (neutral), `pos` (positive), `q` (question), 250 samples each. Some texts are removed because they look like spam.Because these samples are representative of real world content, we believe having these annotaed samples will allow the community to robustly evaluate tokenization algorithms.


Add JNLPBA


Add spider dataset

This PR adds the Spider dataset, a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases.

Dataset website: https://yale-lily.github.io/spider
Paper link: https://www.aclweb.org/anthology/D18-1425/


Youtube caption corrections

This PR adds a new dataset of YouTube captions, error and corrections. This dataset was created in just the last week, as inspired by this sprint!


Add StackOverflow StackSample dataset

This PR adds the StackOverflow StackSample dataset from Kaggle: https://www.kaggle.com/stackoverflow/stacksample

Ran through all of the steps. However, since my dataset requires manually downloading the data, I was unable to run the pytest on the real dataset (the dummy data pytest passed).


Addition of py_ast dataset

@lhoestq as discussed in PR #1195


Add TweetQA dataset

This PR adds the  TweetQA dataset, the first dataset for QA on social media data by leveraging news media and crowdsourcing.

Paper: https://arxiv.org/abs/1907.06292
Repository: https://tweetqa.github.io/


New instruction for how to generate dataset_infos.json

Add additional instructions for how to generate dataset_infos.json for manual download datasets. Information courtesy of `Taimur Ibrahim` from the slack channel


change url for prachathai67k to internet archive

`prachathai67k` is currently downloaded from git-lfs of PyThaiNLP github. Since the size is quite large (~250MB), I moved the URL to archive.org in order to prevent rate limit issues.


added craigslist_bargians

`craigslist_bargains` data set from [here](https://worksheets.codalab.org/worksheets/0x453913e76b65495d8b9730d41c7e0a0c/)

(Cleaned up version of #1278)


add LeNER-Br dataset

Adding the LeNER-Br dataset, a Portuguese language dataset for named entity recognition


initial commit for MultiReQA

Added MultiReQA, which is a dataset containing the sentence boundary annotation from eight publicly available QA datasets including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, BioASQ, RelationExtraction, and TextbookQA.


add Yoruba NER dataset

Added Yoruba GV dataset based on this paper


Add spanish billion words corpus

Add an unannotated Spanish corpus of nearly 1.5 billion words, compiled from different resources from the web.


Add MultiBooked dataset

Add dataset.


First commit of NarrativeQA Dataset

Added NarrativeQA dataset and included a manual downloading option to download scripts from the original scripts provided by the authors.


Add hausa ner corpus

Added Hausa VOA NER data


Add LiveQA

This PR adds LiveQA, the Chinese real-time/timeline-based QA task by [Liu et al., 2020](https://arxiv.org/pdf/2010.00526.pdf).


[yaml] Fix metadata according to pre-specified scheme

@lhoestq @yjernite


added references to only data card creator to all guides

We can now use the wonderful online form for dataset cards created by @evrardts


:fist: ¡Viva la Independencia!

Adds the Catalonia Independence Corpus for stance-detection of Tweets.

Ready for review!


hate_speech_18 initial commit


Add GigaFren Dataset


Add spanish billion words

Add an unannotated corpus of the Spanish language of nearly 1.5 billion words, compiled from different resources from the web.

The dataset needs 10 GB (download: 1.89 GiB, generated: 8.34 GiB, post-processed: Unknown size, total: 10.22 GiB), the test using dummy data pass but my laptop isn't able to run it on the real data (I left it running for over 8 hours and it didn't finish).


Add dataset Yoruba BBC Topic Classification

Added new dataset Yoruba BBC Topic Classification

Contains loading script as well as dataset card including YAML tags.


Added Bianet dataset

Hi :hugs:, This is a PR for [Bianet: A parallel news corpus in Turkish, Kurdish and English; Source](http://opus.nlpl.eu/Bianet.php) dataset


Add QED Amara Dataset


Add Tanzil Dataset


Add Open Subtitles Dataset


First version of the new dataset hausa_voa_topics

Contains loading script as well as dataset card including YAML tags.


added un_ga dataset

Hi :hugs:, This is a PR for [United nations general assembly resolutions: A six-language parallel corpus](http://opus.nlpl.eu/UN.php) dataset


Add yoruba ner corpus


Added the NewsPH Raw dataset and corresponding dataset card

This PR adds the original NewsPH dataset which is used to autogenerate the NewsPH-NLI dataset. Reopened a new PR as the previous one had problems.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


Add msr_genomics_kbcomp dataset


TEP: Tehran English-Persian parallel corpus
more info : http://opus.nlpl.eu/TEP.php


Add humicroedit dataset

Pull request for adding humicroedit dataset


❓ Sharing ElasticSearch indexed dataset

dataset request

Requesting to add a new dataset

Hi there,

First of all, thank you very much for this amazing library. Datasets have become my preferred data structure for basically everything I am currently doing.

**Question:** I'm working with a dataset and I have an elasticsearch container running at localhost:9200. I added an elasticsearch index and I was wondering

- how can I know where it has been saved?

- how can I share the indexed dataset with others?

I tried to dig into the docs, but could not find anything about that.

Thank you very much for your help.

Best,
Pietro

Edit: apologies for the wrong label


Add CC-News dataset of English language articles

Adds [CC-News](https://commoncrawl.org/2016/10/news-dataset-available/) dataset. It contains 708241 English language news articles. Although each article has a language field these tags are not reliable. I've used Spacy language detection [pipeline](https://spacy.io/universe/project/spacy-langdetect) to confirm that the article language is indeed English.

The prepared dataset is temporarily hosted on my private Google Storage [bucket](https://storage.googleapis.com/hf_datasets/cc_news.tar.gz). We can move it to HF storage and update this PR before merging.


add indonlu benchmark datasets

The IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for the Indonesian language. There are 12 datasets in IndoNLU.


added dutch_social

The Dutch social media tweets dataset. Which has a total of more than 210k tweets in dutch language. These tweets have been machine annotated with sentiment scores (`label` feature) and `industry` and `hisco_codes`

It can be used for sentiment analysis, multi-label classification and entity tagging


Added the WikiText-TL39 dataset and corresponding card

This PR adds the WikiText-TL-39 Filipino Language Modeling dataset. Restarted a new pull request since there were problems with the earlier one.

Paper: https://arxiv.org/abs/1907.00409
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


adding wili-2018 language identification dataset


ethos first commit

Ethos passed all the tests except from this one:
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>

with this error:
E               OSError: Cannot find data file.
E               Original error:
E               [Errno 2] No such file or directory:


add 10k German News Article Dataset


Allow GitHub releases as dataset source

# Summary

Providing a GitHub release URL to `DownloadManager.download()` currently throws a `ConnectionError: Couldn't reach [DOWNLOAD_URL]`. This PR fixes this problem by adding an exception for GitHub releases in `datasets.utils.file_utils.get_from_cache()`.

# Reproduce

```
import datasets
url = 'http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz'
result = datasets.utils.file_utils.get_from_cache(url)

# Returns: ConnectionError: Couldn't reach http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz
```

# Cause

GitHub releases returns a HTTP status 403 (FOUND), indicating that the request is being redirected (to AWS S3, in this case). `get_from_cache()` checks whether the status is 200 (OK) or if it is part of two exceptions (Google Drive or Firebase), otherwise the mentioned error is thrown.

# Solution

Just like the exceptions for Google Drive and Firebase, add a condition for GitHub releases URLs that return the HTTP status 403. If this is the case, continue normally.


add yelp_review_full

This corresponds to the Yelp-5 requested in https://github.com/huggingface/datasets/issues/353
I included the dataset card.


Add snips built in intents 2016 12

This PR proposes to add the Snips.ai built in intents dataset. The first configuration added is for the intent labels only, but the dataset includes entity slots that may in future be added as alternate configurations.


Add HateSpeech Corpus for Polish

This PR adds a HateSpeech Corpus for Polish, containing offensive language examples.

- **Homepage:** http://zil.ipipan.waw.pl/HateSpeech
- **Paper:** http://www.qualitativesociologyreview.org/PL/Volume38/PSJ_13_2_Troszynski_Wawer.pdf


Jigsaw toxicity pred

Requires manually downloading data from Kaggle.


Add OPUS Bible Corpus (102 Languages)


Add OffensEval-TR 2020 Dataset

This PR adds the OffensEval-TR 2020 dataset which is a Turkish offensive language corpus by me and @basakbuluz. The corpus consist of randomly sampled tweets and annotated in a similar way to [OffensEval](https://sites.google.com/site/offensevalsharedtask/) and [GermEval](https://projects.fzai.h-da.de/iggsa/).

- **Homepage:** [offensive-turkish](https://coltekin.github.io/offensive-turkish/)
- **Paper:** [A Corpus of Turkish Offensive Language on Social Media](https://coltekin.github.io/offensive-turkish/troff.pdf)
- **Point of Contact:** [Çağrı Çöltekin](ccoltekin@sfs.uni-tuebingen.de)


Add SAMSum Corpus dataset

Did not spent much time writing README, might update later.

Copied description and some stuff from tensorflow_datasets
https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/summarization/samsum.py


Add Wiki Lingua Dataset

Hello,

This is my first PR.

I have added Wiki Lingua Dataset along with dataset card to the best of my knowledge.
There was one hiccup though. I was unable to create dummy data because the data is in pkl format.
From the document, I see that:
```At the moment it supports data files in the following format: txt, csv, tsv, jsonl, json, xml```


adding capes

Adding Parallel corpus of theses and dissertation abstracts in Portuguese and English from CAPES
https://sites.google.com/view/felipe-soares/datasets#h.p_kxOR6EhHm2a6


add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC)

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


[README] Added Windows command to enable slow tests

The Windows command to run slow tests has caused issues, so this adds a functional Windows command.


adding eitb_parcc

Adding EiTB-ParCC: Parallel Corpus of Comparable News
http://opus.nlpl.eu/EiTB-ParCC.php


adding opus_openoffice

Adding Opus OpenOffice: http://opus.nlpl.eu/OpenOffice.php
8 languages, 28 bitexts


Add Danish NER dataset


arxiv dataset added

**adding arXiv dataset**: arXiv dataset and metadata of 1.7M+ scholarly papers across STEM
dataset link: https://www.kaggle.com/Cornell-University/arxiv


added dutch_social

WIP
As some tests did not clear! 👎🏼


can't load "german_legal_entity_recognition" dataset

FileNotFoundError: Couldn't find file locally at german_legal_entity_recognition/german_legal_entity_recognition.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py


Add OPUS Ted Talks 2013


OPUS Ted Talks 2013


The Snips Built In Intents 2016 dataset.


add hrenwac_para


adding opus_euconst

Adding EUconst, a parallel corpus collected from the European Constitution.
21 languages, 210 bitexts


arXiv dataset added


adding pubmed_qa dataset

Pubmed QA dataset:
PQA-L(abeled) 1k
PQA-U(labeled) 61.2k
PQA-A(rtifical labeled) 211.3k


imdb dataset cannot be downloaded

hi
please find error below getting imdb train spli:
thanks

`
datasets.load_dataset>>> datasets.load_dataset("imdb", split="train")`

errors

```
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Downloading and preparing dataset imdb/plain_text (download: 80.23 MiB, generated: 127.06 MiB, post-processed: Unknown size, total: 207.28 MiB) to /idiap/temp/rkarimi/cache_home_1/datasets/imdb/plain_text/1.0.0/90099cb476936b753383ba2ae6ab2eae419b2e87f71cd5189cb9c8e5814d12a3...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 558, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 73, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=7486451, num_examples=5628, dataset_name='imdb')}]

```


Jigsaw toxicity classification dataset added

The dataset requires manually downloading data from Kaggle.


Add CodeSearchNet corpus dataset

This PR adds the CodeSearchNet corpus proxy dataset for semantic code search: https://github.com/github/CodeSearchNet
I have had a few issues, mentioned below. Would appreciate some help on how to solve them.

## Issues generating dataset card
Is there something wrong with my declaration of the dataset features ?
```
features=datasets.Features(
    {
        "repository_name": datasets.Value("string"),
        "func_path_in_repository": datasets.Value("string"),
        "func_name": datasets.Value("string"),
        "whole_func_string": datasets.Value("string"),
        "language": datasets.Value("string"),
        "func_code_string": datasets.Value("string"),
        "func_code_tokens": datasets.Sequence(datasets.Value("string")),
        "func_documentation_string": datasets.Value("string"),
        "func_documentation_tokens": datasets.Sequence(datasets.Value("string")),
        "split_name": datasets.Value("string"),
        "func_code_url": datasets.Value("string"),
        # TODO - add licensing info in the examples
    }
),

```
When running the streamlite app for tagging the dataset on my machine, I get the following error :
![image](https://user-images.githubusercontent.com/33657802/101469132-9ed12c80-3944-11eb-94ff-2d9c1d0ea080.png)

## Issues with dummy data
Due to the unusual structure of the data, I have been unable to generate dummy data automatically.
I tried to generate it manually, but pytests fail when using the manually-generated dummy data ! Pytests work fine when using the real data.
```
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.7.9, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
plugins: xdist-2.1.0, forked-1.3.0
collected 1 item

tests/test_dataset_common.py F                                                                                [100%]

=================================================================================================== FAILURES ====================================================================================================
________________________________________________________________________ LocalDatasetTest.test_load_dataset_all_configs_code_search_net _________________________________________________________________________
self = <tests.test_dataset_common.LocalDatasetTest testMethod=test_load_dataset_all_configs_code_search_net>, dataset_name = 'code_search_net'

    @slow
    def test_load_dataset_all_configs(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True)

tests/test_dataset_common.py:237:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:198: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
--------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------
Downloading and preparing dataset code_search_net/all (download: 1.00 MiB, generated: 1.00 MiB, post-processed: Unknown size, total: 2.00 MiB) to /tmp/tmppx78sj24/code_search_net/all/1.0.0...
Dataset code_search_net downloaded and prepared to /tmp/tmppx78sj24/code_search_net/all/1.0.0. Subsequent calls will reuse this data.
--------------------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------------------
... (irrelevant info - Deprecation warnings)
============================================================================================ short test summary info ============================================================================================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_code_search_net - AssertionError: False is not true
========================================================================================= 1 failed, 4 warnings in 3.00s ========================================================================================
```

## Note : Data structure in S3
The data is stored on S3, and organized by programming languages.
It is stored in the following repository structure:
```
.
├── <language_name>  # e.g. python
│   └── final
│       └── jsonl
│           ├── test
│           │   └── <language_name>_test_0.jsonl.gz
│           ├── train
│           │   ├── <language_name>_train_0.jsonl.gz
│           │   ├── <language_name>_train_1.jsonl.gz
│           │   ├── ...
│           │   └── <language_name>_train_n.jsonl.gz
│           └── valid
│               └── <language_name>_valid_0.jsonl.gz
├── <language_name>_dedupe_definitions_v2.pkl
└── <language_name>_licenses.pkl
```


'iwslt2017-ro-nl', cannot be downloaded

dataset bug

A bug in a dataset script provided in the library

Hi
I am trying
`>>> datasets.load_dataset("iwslt2017", 'iwslt2017-ro-nl', split="train")`

getting this error thank you for your help
```
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Downloading and preparing dataset iwsl_t217/iwslt2017-ro-nl (download: 314.07 MiB, generated: 39.92 MiB, post-processed: Unknown size, total: 354.00 MiB) to /idiap/temp/rkarimi/cache_home_1/datasets/iwsl_t217/iwslt2017-ro-nl/1.0.0/cca6935a0851a8ceac1202a62c958738bdfa23c57a51bc52ac1c5ebd2aa172cd...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/iwslt2017/cca6935a0851a8ceac1202a62c958738bdfa23c57a51bc52ac1c5ebd2aa172cd/iwslt2017.py", line 118, in _split_generators
    dl_dir = dl_manager.download_and_extract(MULTI_URL)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 254, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 179, in download
    num_proc=download_config.num_proc,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 216, in map_nested
    return function(data_struct)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 477, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz

```


Hi
I am getting this error when evaluating on wmt16-ro-en using finetune_trainer.py of huggingface repo. thank for your help

{'epoch': 20.0}
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:16<00:00,  1.22it/s]
12/08/2020 10:41:19 - INFO - seq2seq.trainers.trainer -   Saving model checkpoint to outputs/experiment/joint/finetune/lr-2e-5
12/08/2020 10:41:24 - INFO - __main__ -   {'wmt16-en-ro': Dataset(features: {'src_texts': Value(dtype='string', id=None), 'task': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 1998), 'qnli': Dataset(features: {'src_texts': Value(dtype='string', id=None), 'task': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 5462), 'scitail': Dataset(features: {'src_texts': Value(dtype='string', id=None), 'task': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 1303)}
12/08/2020 10:41:24 - INFO - __main__ -   *** Evaluate ***
12/08/2020 10:41:24 - INFO - seq2seq.utils.utils -   using task specific params for wmt16-en-ro: {'max_length': 300, 'num_beams': 4}
12/08/2020 10:41:24 - INFO - seq2seq.trainers.trainer -   ***** Running Evaluation *****
12/08/2020 10:41:24 - INFO - seq2seq.trainers.trainer -     Num examples = 1998
12/08/2020 10:41:24 - INFO - seq2seq.trainers.trainer -     Batch size = 64
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:37<00:00,  1.19s/it][libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0):
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (index) >= (0):
Aborted


boolq does not work

Hi
I am getting this error when trying to load boolq, thanks for your help

ts_boolq_default_0.1.0_2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11.lock
Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 274, in <module>
    main()
  File "finetune_t5_trainer.py", line 147, in main
    for task in data_args.tasks]
  File "finetune_t5_trainer.py", line 147, in <listcomp>
    for task in data_args.tasks]
  File "/remote/idiap.svm/user.active/rkarimi/dev/ruse/seq2seq/tasks/tasks.py", line 58, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/remote/idiap.svm/user.active/rkarimi/dev/ruse/seq2seq/tasks/tasks.py", line 54, in load_dataset
    return datasets.load_dataset(self.task.name, split=split)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 149, in download_custom
    custom_download(url, path)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/tensorflow/python/lib/io/file_io.py", line 516, in copy_v2
    compat.path_to_bytes(src), compat.path_to_bytes(dst), overwrite)
tensorflow.python.framework.errors_impl.AlreadyExistsError: file already exists


Update coqa dataset url

`datasets.stanford.edu` is invalid.


Add dutch book review dataset

- Name: Dutch Book Review Dataset (DBRD)
- Description: The DBRD (pronounced dee-bird) dataset contains over 110k book reviews along with associated binary sentiment polarity labels and is intended as a benchmark for sentiment classification in Dutch.
- Paper: https://arxiv.org/abs/1910.00896
- Data: https://github.com/benjaminvdb/DBRD
- Motivation: A large (real-life) dataset of Dutch book reviews and sentiment polarity (positive/negative), based on the associated rating.

Checks
- [x] Create the dataset script /datasets/dbrd/dbrd.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _info(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


add thaiqa_squad

Example format is a little different from SQuAD since `thaiqa` always have one answer per question so I added a check to convert answers to lists if they are not already one to future-proof additional questions that might have multiple answers.

`thaiqa_squad` is an open-domain, extractive question answering dataset (4,000 questions in `train` and 74 questions in `dev`) in [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format, originally created by [NECTEC](https://www.nectec.or.th/en/) from Wikipedia articles and adapted to [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format by [PyThaiNLP](https://github.com/PyThaiNLP/).


adding hybrid_qa

Adding  HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
https://github.com/wenhuchen/HybridQA


disaster response messages dataset


added para_pat

Dataset link : https://figshare.com/articles/ParaPat_The_Multi-Million_Sentences_Parallel_Corpus_of_Patents_Abstracts/12627632
Working on README.md currently


Craigslist bargains

`craigslist_bargains` dataset from [here](https://worksheets.codalab.org/worksheets/0x453913e76b65495d8b9730d41c7e0a0c/)


add One Million Posts Corpus

- **Name:** One Million Posts Corpus
- **Description:** The “One Million Posts” corpus is an annotated data set consisting of user comments posted to an Austrian newspaper website (in German language).
- **Paper:** https://dl.acm.org/doi/10.1145/3077136.3080711
- **Data:** https://github.com/OFAI/million-post-corpus
- **Motivation:** Big German (real-life) dataset containing different annotations around forum moderation with expert annotations.

### Checkbox

- [X] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [X] Fill the `_DESCRIPTION` and `_CITATION` variables
- [X] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [X] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [X] Generate the metadata file `dataset_infos.json` for all configurations
- [X] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [X] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [X] Both tests for the real data and the dummy data pass.


Yoruba GV NER added

I just added Yoruba GV NER dataset from this paper https://www.aclweb.org/anthology/2020.lrec-1.335/


oclar-dataset

Opinion Corpus for Lebanese Arabic Reviews (OCLAR) corpus is utilizable for Arabic sentiment classification on reviews, including hotels, restaurants, shops, and others. : [homepage](http://archive.ics.uci.edu/ml/datasets/Opinion+Corpus+for+Lebanese+Arabic+Reviews+%28OCLAR%29#)


Created wiki_movies dataset.

First PR (ever). Hopefully this movies dataset is useful to others!


Psc


SMS Spam Dataset

Hi :) I added this [SMS Spam Dataset](http://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection)


add DFKI SmartData Corpus

- **Name:** DFKI SmartData Corpus
- **Description:** DFKI SmartData Corpus is a dataset of 2598 German-language documents which has been annotated with fine-grained geo-entities, such as streets, stops and routes, as well as standard named entity types.
- **Paper:** https://www.dfki.de/fileadmin/user_upload/import/9427_lrec_smartdata_corpus.pdf
- **Data:** https://github.com/DFKI-NLP/smartdata-corpus
- **Motivation:** Contains fine-grained NER labels for German.

### Checkbox

- [X] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [X] Fill the `_DESCRIPTION` and `_CITATION` variables
- [X] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [X] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [X] Generate the metadata file `dataset_infos.json` for all configurations
- [X] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [X] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [X] Both tests for the real data and the dummy data pass.


Adding OneStopEnglish corpus dataset

This PR adds OneStopEnglish Corpus containing texts classified into reading levels (elementary, intermediate, advance) for automatic readability assessment and text simplification.

Link to the paper: https://www.aclweb.org/anthology/W18-0535.pdf


new pr for Turkish NER


Has part


removing unzipped hansards dummy data

which were added by mistake


Add CovidQA dataset

This PR adds CovidQA, a question answering dataset specifically designed for COVID-19, built by hand from knowledge gathered from Kaggle’s COVID-19 Open Research Dataset Challenge.

Link to the paper: https://arxiv.org/pdf/2004.11339.pdf
Link to the homepage: https://covidqa.ai


enriched webnlg dataset rebase

Rebase of #1206 !


Added kannada news headlines classification dataset.

Manual Download of a kaggle dataset. Mostly followed process as ms_terms.


Adding msr_genomics_kbcomp dataset


Add Google Sentence Compression dataset

For more information: https://www.aclweb.org/anthology/D13-1155.pdf


Added NewsPH Raw Dataset

Added the raw version of the NewsPH dataset, which was used to automatically generate the NewsPH-NLI corpus. Dataset of news articles in Filipino from mainstream Philippine news sites on the internet. Can be used as a language modeling dataset or to reproduce the NewsPH-NLI dataset.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


Add KorQPair dataset

This PR adds a [Korean paired question dataset](https://github.com/songys/Question_pair) containing labels indicating whether two questions in a given pair are semantically identical. This dataset was used to evaluate the performance of [KoGPT2](https://github.com/SKT-AI/KoGPT2#subtask-evaluations) on a phrase detection downstream task.


arXiv dataset added


Add Swahili news classification dataset

Add Swahili news classification dataset


adding LiMiT dataset

Adding LiMiT: The Literal Motion in Text Dataset
https://github.com/ilmgut/limit_dataset


[doc] nlp/viewer ➡datasets/viewer

cc @srush


Added WikiText-TL-39

This PR adds the WikiText-TL-39 Filipino Language Modeling dataset.

Paper: https://arxiv.org/abs/1907.00409
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


add thainer

ThaiNER (v1.3) is a 6,456-sentence named entity recognition dataset created from expanding the 2,258-sentence
[unnamed dataset](http://pioneer.chula.ac.th/~awirote/Data-Nutcha.zip) by
[Tirasaroj and Aroonmanakun (2012)](http://pioneer.chula.ac.th/~awirote/publications/).
It is used to train NER taggers in [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp).
The NER tags are annotated by [Tirasaroj and Aroonmanakun (2012)]((http://pioneer.chula.ac.th/~awirote/publications/))
for 2,258 sentences and the rest by [@wannaphong](https://github.com/wannaphong/).
The POS tags are done by [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp)'s `perceptron` engine trained on `orchid_ud`.
[@wannaphong](https://github.com/wannaphong/) is now the only maintainer of this dataset.


Add Naver sentiment movie corpus

Supersedes #1168

> This PR adds the [Naver sentiment movie corpus](https://github.com/e9t/nsmc), a dataset containing Korean movie reviews from Naver, the most commonly used search engine in Korea. This dataset is often used to benchmark models on Korean NLP tasks, as seen in [this paper](https://www.aclweb.org/anthology/2020.lrec-1.199.pdf).


Add Wiki Atomic Edits Dataset (43M edits)


added Nergrit dataset

Nergrit Corpus is a dataset collection for Indonesian Named Entity Recognition, Statement Extraction, and Sentiment Analysis. This PR is only for the Named Entity Recognition.


Add doc2dial dataset

### Doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset v0.9

Once complete this will add the [Doc2dial](https://doc2dial.github.io/data.html) dataset from the generic data sets list.


Update step-by-step guide about the dataset cards

Small update in the step-by-step guide about the dataset cards to indicate it can be created and completing while exploring the dataset.


Adding indonlu dataset

IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for Bahasa Indonesia. It contains 12 datasets.


Add Google Turkish Treebank Dataset


arxiv dataset added


Add Google Noun Verb Dataset


adding bprec


Opus elhuyar dataset for MT task having languages pair in Spanish to Basque
More info : http://opus.nlpl.eu/Elhuyar.php


Multi Domain Sentiment Analysis Dataset (MDSA)


add yelp_review_full dataset

This corresponds to the Yelp-5 requested in https://github.com/huggingface/datasets/issues/353


adding poem_sentiment

Adding poem_sentiment dataset.
https://github.com/google-research-datasets/poem-sentiment


Add AmbigQA dataset

# AmbigQA: Answering Ambiguous Open-domain Questions Dataset
Adding the [AmbigQA](https://nlp.cs.washington.edu/ambigqa/) dataset as part of the sprint 🎉 (from Open dataset list for Dataset sprint)

Added both the light and full versions (as seen on the dataset homepage)
The json format changes based on the value of one 'type' field, so I set the unavailable field to an empty list. This is explained in the README -> Data Fields

```py
train_light_dataset = load_dataset('./datasets/ambig_qa',"light",split="train")
val_light_dataset = load_dataset('./datasets/ambig_qa',"light",split="validation")
train_full_dataset = load_dataset('./datasets/ambig_qa',"full",split="train")
val_full_dataset = load_dataset('./datasets/ambig_qa',"full",split="validation")

for example in train_light_dataset:
    for i,t in enumerate(example['annotations']['type']):
        if t =='singleAnswer':
            # use the example['annotations']['answer'][i]
            # example['annotations']['qaPairs'][i] - > is []
            print(example['annotations']['answer'][i])
        else:
            # use the example['annotations']['qaPairs'][i]
            # example['annotations']['answer'][i] - > is []
            print(example['annotations']['qaPairs'][i])

```

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


Opus finlex dataset of language pair  Finnish and Swedish

Added Opus_finlex dataset of language pair  Finnish and Swedish
More info : http://opus.nlpl.eu/Finlex.php


Wino bias

The PR will fail circleCi tests because of the requirement of manual loading of data. Fresh PR because of messed up history of the previous one.


Added ade_corpus_v2, with 3 configs for relation extraction and classification task

Adverse Drug Reaction Data: ADE-Corpus-V2 dataset added configs for different tasks with given data


Add Curiosity Dialogs Dataset

Add Facebook [Curiosity Dialogs](https://github.com/facebookresearch/curiosity) Dataset.


Add Grail QA dataset

For more information: https://dki-lab.github.io/GrailQA/


Add Urdu Sentiment Corpus (USC)

@lhoestq opened a clean PR containing only relevant files.

old PR #1140


Add Urdu fake news dataset

@lhoestq opened a clean PR containing only relevant files.

old PR #1125


Muchocine - Spanish movie reviews dataset


add opus_100 dataset

This PR will add [opus100 dataset](http://opus.nlpl.eu/opus-100.php).


readme: remove link to Google's responsible AI practices

...maybe we'll find a company that reallly stands behind responsible AI practices ;)


Add menyo_20k_mt dataset

Add menyo_20k_mt dataset


Add Winobias dataset

Pardon me for different commits with same message. There were conflicts after I rebased master while simultaneously pushing my changes to local repo, hence the duplicate entries.


adding conceptnet5

Adding the conceptnet5 and omcs txt files used to create the conceptnet5 dataset. Conceptne5 is a common sense dataset. More info can be found here: https://github.com/commonsense/conceptnet5/wiki


🇸🇪  Added Swedish Reviews dataset for sentiment classification in Sw…

perhaps: @lhoestq 🤗


Add numeric fused head dataset

Adding the [NFH: Numeric Fused Head](https://nlp.biu.ac.il/~lazary/fh/) dataset.

Everything looks sensible and I've included both the identification and resolution tasks. I haven't personally used this dataset in my research so am unable to specify what the default configuration / supervised keys should be.

I've filled out the basic info on the model card to the best of my knowledge but it's a little tricky to understand exactly what the fields represent.

Dataset author: @yanaiela


Add HKCanCor

This PR adds the [Hong Kong Cantonese Corpus](http://compling.hss.ntu.edu.sg/hkcancor/), by [Luke and Wong 2015](http://compling.hss.ntu.edu.sg/hkcancor/data/LukeWong_Hong-Kong-Cantonese-Corpus.pdf).

The dummy data included here was manually created, as the original dataset uses a xml-like format (see a copy hosted [here](https://github.com/fcbond/hkcancor/blob/master/sample/d1_v.txt) for example) that requires a few processing steps.


add Korean HateSpeech dataset


Add Korean NER dataset

Supersedes #1177

> This PR adds the [Korean named entity recognition dataset](https://github.com/kmounlp/NER). This dataset has been used in many downstream tasks, such as training [KoBERT](https://github.com/SKTBrain/KoBERT) for NER, as seen in this [KoBERT-CRF implementation](https://github.com/eagle705/pytorch-bert-crf-ner).


Add WMT20 MLQE 3 shared tasks

3 tasks for the WMT 20 MLQE shared tasks -> 3 different datasets

(I re-created #1137 because it was too messy).

Note that in L199 `task3.py`, I used `logging.warning` to print some missing data in the train set.


adding DataCommons fact checking

Adding the data from: https://datacommons.org/factcheck/

Had to cheat a bit with the dummy data as the test doesn't recognize `.txt.gz`: had to rename uncompressed files with the `.gz` extension manually without actually compressing


Add limit

This PR adds [LiMiT](https://github.com/ilmgut/limit_dataset), a dataset for literal motion classification/extraction by [Manotas et al., 2020](https://www.aclweb.org/anthology/2020.findings-emnlp.88.pdf).


Add irc disentanglement

added files for irc disentanglement dataset
was unable to test dummy data as a result of vpn/proxy issues


adding medical-questions-pairs dataset

This dataset consists of 3048 similar and dissimilar medical question pairs hand-generated and labeled by Curai's doctors.
Dataset : https://github.com/curai/medical-question-pair-dataset
Paper : https://drive.google.com/file/d/1CHPGBXkvZuZc8hpr46HeHU6U6jnVze-s/view


add taskmaster3

Adding Taskmaster-3 dataset
https://github.com/google-research-datasets/Taskmaster/tree/master/TM-3-2020.

The dataset structure almost same as original dataset with these two changes

1. In original dataset, each `apis` has a `args` filed which is a `dict`  with variable keys, which represent the name and value of the args. Here converted that to a `list` of `dict` with keys `arg_name` and `arg_value`. For ex.

```python
args = {"name.movie": "Mulan", "name.theater": ": "Mountain AMC 16"}
```
becomes
```python
[
  {
    "arg_name": "name.movie",
    "arg_value": "Mulan"
  },
  {
    "arg_name": "name.theater",
    "arg_value": "Mountain AMC 16"
  }
]
```

2. Each `apis` has a `response` which is also a `dict` with variable keys representing response name/type and it's value. As above converted it to `list` of `dict` with keys `response_name` and `response_value`.


Add Sanskrit Classic texts in datasets


Add large spanish corpus

Adds a collection of Spanish corpora that can be useful for pretraining language models.

Following a nice suggestion from @yjernite we provide the user with three main ways to preprocess / load either

* the whole corpus (17GB!)
* one specific sub-corpus
* the whole corpus, but return a single split. this is useful if you want to cache the whole preprocessing step once and interact with individual sub-corpora

See the dataset card for more details.

Ready for review!


Add XSUM Hallucination Annotations Dataset

Adding Google [XSum Hallucination Annotations](https://github.com/google-research-datasets/xsum_hallucination_annotations) dataset.


[AfriBooms] Dataset exists already

When trying to add "AfriBooms": https://docs.google.com/spreadsheets/d/12ShVow0M6RavnzbBEabm5j5dv12zBaf0y-niwEPPlo4/edit#gid=1386399609 I noticed that the dataset exists already as a config of Universal Dependencies (universal_dependencies.py). I checked and the data exactly matches so that the new data link does not give any new data.

This PR improves the config's description a bit by linking to the paper.


Add HKCanCor

(Apologies, didn't manage the branches properly and the PR got too messy. Going to open a new PR with everything in order)


Add msr_genomics_kbcomp Dataset


Adding Enriched WebNLG dataset

This pull requests adds the `en` and `de` versions of the [Enriched WebNLG](https://github.com/ThiagoCF05/webnlg) dataset


add lst20 with manual download

passed on local:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_lst20
```
Not sure how to test:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_lst20
```

```
LST20 Corpus is a dataset for Thai language processing developed by National Electronics and Computer Technology Center (NECTEC), Thailand.
It offers five layers of linguistic annotation: word boundaries, POS tagging, named entities, clause boundaries, and sentence boundaries.
At a large scale, it consists of 3,164,002 words, 288,020 named entities, 248,181 clauses, and 74,180 sentences, while it is annotated with
16 distinct POS tags. All 3,745 documents are also annotated with one of 15 news genres. Regarding its sheer size, this dataset is
considered large enough for developing joint neural models for NLP.
Manually download at https://aiforthai.in.th/corpus.php
```


adding meta_woz dataset


Add Neural Code Search Dataset


Medical question pairs

This dataset consists of 3048 similar and dissimilar medical question pairs hand-generated and labeled by Curai's doctors.
Dataset : https://github.com/curai/medical-question-pair-dataset
Paper : https://drive.google.com/file/d/1CHPGBXkvZuZc8hpr46HeHU6U6jnVze-s/view
**No splits added**


adding medical-questions-pairs


Update ADD_NEW_DATASET.md

Windows needs special treatment again: unfortunately adding `torch` to the requirements does not work well (crashing the installation). Users should first install torch manually and then continue with the other commands.

This issue arises all the time when adding torch as a dependency, but because so many novice users seem to participate in adding datasets, it may be useful to add an explicit note for Windows users to ensure that they do not run into issues.


Turkish NER dataset, script works fine, couldn't generate dummy data

I've written the script (Turkish_NER.py) that includes dataset. The dataset is a zip inside another zip, and it's extracted as .DUMP file. However, after preprocessing I only get .arrow file. After I ran the script with no error messages, I get .arrow file of dataset, LICENSE and dataset_info.json.


Add ALT

ALT dataset -- https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/


add taskmaster-2

Adding taskmaster-2 dataset.
https://github.com/google-research-datasets/Taskmaster/tree/master/TM-2-2020


Add IWSLT'15 English-Vietnamese machine translation Data

Preprocessed Dataset from IWSLT'15 English-Vietnamese machine translation: English-Vietnamese.

from https://nlp.stanford.edu/projects/nmt/data/iwslt15.en-vi/


addition of py_ast

The dataset consists of parsed Parsed ASTs that were used to train and evaluate the DeepSyn tool.
The Python programs are collected from GitHub repositories
by removing duplicate files, removing project forks (copy of another existing repository)
,keeping only programs that parse and have at most 30'000 nodes in the AST and
we aim to remove obfuscated files


Add msr_text_compression

Add [MSR Abstractive Text Compression Dataset](https://msropendata.com/datasets/f8ce2ec9-7fbd-48f7-a8bb-2d2279373563)


add taskmaster-1

Adding Taskmaster-1 dataset
https://github.com/google-research-datasets/Taskmaster/tree/master/TM-1-2019


Add NewsPH_NLI dataset

This PR adds the NewsPH-NLI Dataset, the first benchmark dataset for sentence entailment in the low-resource Filipino language. Constructed through exploting the structure of news articles. Contains 600,000 premise-hypothesis pairs, in 70-15-15 split for training, validation, and testing.

Link to the paper: https://arxiv.org/pdf/2010.11574.pdf

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


Added Translator Human Parity Data For a Chinese-English news transla…

…tion system from Open dataset list for Dataset sprint, Microsoft Datasets tab.


Add Fake News Detection in Filipino dataset

This PR adds the Fake News Filipino Dataset, a low-resource fake news detection corpora in Filipino. Contains 3,206 expertly-labeled news samples, half of which are real and half of which are fake.

Link to the paper: http://www.lrec-conf.org/proceedings/lrec2020/index.html

Link to the dataset/repo: https://github.com/jcblaisecruz02/Tagalog-fake-news


Add Dengue dataset in Filipino

This PR adds the Dengue Dataset, a benchmark dataset for low-resource multiclass classification, with 4,015 training, 500 testing, and 500 validation examples, each labeled as part of five classes. Each sample can be a part of multiple classes. Collected as tweets.

Link to the paper: https://ieeexplore.ieee.org/document/8459963

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


adding hind_encorp dataset

adding Hindi_Encorp05 dataset


Added AQUA-RAT (Algebra Question Answering with Rationales) Dataset


all test passed

need help creating dummy data


Add Hate Speech Dataset in Filipino

This PR adds the Hate Speech Dataset, a text classification dataset in Filipino, consisting 10k tweets (training set) that are labeled as hate speech or non-hate speech. Released with 4,232 validation and 4,232 testing samples. Collected during the 2016 Philippine Presidential Elections.

Link to the paper: https://pcj.csp.org.ph/index.php/pcj/issue/download/29/PCJ%20V14%20N1%20pp1-14%202019

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


Add Adversarial SQuAD dataset

# Adversarial SQuAD

Adding the Adversarial [SQuAD](https://github.com/robinjia/adversarial-squad) dataset as part of the sprint 🎉
This dataset adds adversarial sentences to a subset of the SQuAD dataset's dev examples. How to get the original squad example id is explained in readme->Data Instances. The whole data is intended for use in evaluation. (Which could of course be also used for training if one wants). So there is no classical train/val/test split, but a split based on the number of adversaries added.

There are 2 splits of this dataset:

- AddSent: Has up to five candidate adversarial sentences that don't answer the question, but have a lot of words in common with the question. This adversary is does not query the model in any way.
- AddOneSent: Similar to AddSent, but just one candidate sentences was picked at random. This adversary is does not query the model in any way.

(The AddAny and AddCommon datasets mentioned in the paper are dynamically generated based on model's output distribution thus are not included here)

The failing test look like some unrelated timeout thing, will probably clear if rerun.
- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


add mkb dataset

This PR will add Mann Ki Baat dataset (parallel data for Indian languages).


ADD COVID-QA dataset

This PR adds the COVID-QA dataset, a question answering dataset consisting of 2,019 question/answer pairs annotated by volunteer biomedical experts on scientific articles related to COVID-19

Link to the paper: https://openreview.net/forum?id=JENSKEEzsoU
Link to the dataset/repo: https://github.com/deepset-ai/COVID-QA


added emotions detection in arabic dataset

Dataset for Emotions detection in Arabic text

more info: https://github.com/AmrMehasseb/Emotional-Tone


Add  KorQuAD v2 Dataset

# The Korean Question Answering Dataset v2
Adding the [KorQuAD](https://korquad.github.io/) v2 dataset as part of the sprint 🎉
This dataset is very similar to SQuAD and is an extension of [squad_kor_v1](https://github.com/huggingface/datasets/pull/1178) which is why I added it as `squad_kor_v2`.

- Crowd generated questions and answer (1-answer per question) for Wikipedia articles. Differently from V1 it includes the html structure and markup, which makes it a different enough dataset. (doesn't share ids between v1 and v2 either)

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

Edit: 🤦 looks like squad_kor_v1 commit sneaked in here too


Small update to the doc: add flatten_indices in doc


Add  KorQuAD v1 Dataset

# The Korean Question Answering Dataset
Adding the [KorQuAD](https://korquad.github.io/KorQuad%201.0/) v1 dataset as part of the sprint 🎉
This dataset is very similar to SQuAD which is why I added it as `squad_kor_v1`. There is also a v2 which I added [here](https://github.com/huggingface/datasets/pull/1180).

- Crowd generated questions and answer (1-answer per question) for Wikipedia articles.

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


Add Korean NER dataset

This PR adds the [Korean named entity recognition dataset](https://github.com/kmounlp/NER). This dataset has been used in many downstream tasks, such as training [KoBERT](https://github.com/SKTBrain/KoBERT) for NER, as seen in this [KoBERT-CRF implementation](https://github.com/eagle705/pytorch-bert-crf-ner).


Add OpenPI Dataset

Add the OpenPI Dataset by AI2 (AllenAI)


added ReDial dataset

Updating README
Dataset link: https://redialdata.github.io/website/datasheet


Add Universal Morphologies

Adding unimorph universal morphology annotations for 110 languages, pfew!!!

one lemma per row with all possible forms and annotations

https://unimorph.github.io/


add wikipedia biography dataset

My first PR containing the Wikipedia biographies dataset. I have followed all the steps in the [guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md). It passes all the tests.


Add proto_qa dataset

Added dataset tags as required.


Add imdb Urdu Reviews dataset.

Added the imdb Urdu reviews dataset. More info about the dataset over here.


Fix path handling for Windows


Add Opus fiskmo dataset for Finnish and Swedish for MT task

Adding fiskmo, a massive parallel corpus for Finnish and Swedish.
for more info : http://opus.nlpl.eu/fiskmo.php


Add Naver sentiment movie corpus

This PR adds the [Naver sentiment movie corpus](https://github.com/e9t/nsmc), a dataset containing Korean movie reviews from Naver, the most commonly used search engine in Korea. This dataset is often used to benchmark models on Korean NLP tasks, as seen in [this paper](https://www.aclweb.org/anthology/2020.lrec-1.199.pdf).


❓ On-the-fly tokenization with datasets, tokenizers, and torch Datasets and Dataloaders

question

Further information is requested

generic discussion

Generic discussion on the library

Hi there,

I have a question regarding "on-the-fly" tokenization. This question was elicited by reading the "How to train a new language model from scratch using Transformers and Tokenizers" [here](https://huggingface.co/blog/how-to-train). Towards the end there is this sentence: "If your dataset is very large, you can opt to load and tokenize examples on the fly, rather than as a preprocessing step". I've tried coming up with a solution that would combine both `datasets` and `tokenizers`, but did not manage to find a good pattern.

I guess the solution would entail wrapping a dataset into a Pytorch dataset.

As a concrete example from the [docs](https://huggingface.co/transformers/custom_datasets.html)

```python
import torch

class SquadDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        # instead of doing this beforehand, I'd like to do tokenization on the fly
        self.encodings = encodings

    def __getitem__(self, idx):
        return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}

    def __len__(self):
        return len(self.encodings.input_ids)

train_dataset = SquadDataset(train_encodings)
```

How would one implement this with "on-the-fly" tokenization exploiting the vectorized capabilities of tokenizers?

----

Edit: I have come up with this solution. It does what I want, but I feel it's not very elegant

```python
class CustomPytorchDataset(Dataset):
    def __init__(self):
        self.dataset = some_hf_dataset(...)
        self.tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")

    def __getitem__(self, batch_idx):
        instance = self.dataset[text_col][batch_idx]
        tokenized_text = self.tokenizer(instance, truncation=True, padding=True)
        return tokenized_text

    def __len__(self):
        return len(self.dataset)

    @staticmethod
    def collate_fn(batch):
        # batch is a list, however it will always contain 1 item because we should not use the
        # batch_size argument as batch_size is controlled by the sampler
        return {k: torch.tensor(v) for k, v in batch[0].items()}

torch_ds = CustomPytorchDataset()

# NOTE: batch_sampler returns list of integers and since here we have SequentialSampler
# it returns: [1, 2, 3], [4, 5, 6], etc. - check calling `list(batch_sampler)`
batch_sampler = BatchSampler(SequentialSampler(torch_ds), batch_size=3, drop_last=True)

# NOTE: no `batch_size` as now the it is controlled by the sampler!
dl = DataLoader(dataset=torch_ds, sampler=batch_sampler, collate_fn=torch_ds.collate_fn)
```


Opus montenegrinsubs

Opus montenegrinsubs - language pair en-me
more info : http://opus.nlpl.eu/MontenegrinSubs.php


Add ar rest reviews

added restaurants reviews in Arabic for sentiment analysis tasks


Add DaNe dataset


Added  memat : Xhosa-English parallel corpora
for more info : http://opus.nlpl.eu/memat.php


Add Mocha dataset

More information: https://allennlp.org/mocha


Linguisticprobing

Adding Linguistic probing datasets from
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties
 https://www.aclweb.org/anthology/P18-1198/


adding TabFact dataset

Adding TabFact: A Large-scale Dataset for Table-based Fact Verification.

https://github.com/wenhuchen/Table-Fact-Checking

- The tables are stored as individual csv files, so need to download 16,573 🤯  csv files. As a result the `datasets_infos.json` file is huge (6.62 MB).
- Original dataset has nested structure where, where table is one example and each table has multiple statements,
flattening the structure here so that each statement is one example.


Add Roman Urdu dataset

This PR adds the [Roman Urdu dataset](https://archive.ics.uci.edu/ml/datasets/Roman+Urdu+Data+Set#).


Add BBC Hindi NLI Dataset

# Dataset Card for BBC Hindi NLI Dataset

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

- HomePage : https://github.com/midas-research/hindi-nli-data
- Paper : "https://www.aclweb.org/anthology/2020.aacl-main.71"
- Point of Contact : https://github.com/midas-research/hindi-nli-data

### Dataset Summary

- Dataset for Natural Language Inference in Hindi Language. BBC Hindi Dataset consists of textual-entailment pairs.
- Each row of the Datasets if made up of 4 columns - Premise, Hypothesis, Label and Topic.
- Context and Hypothesis is written in Hindi while Entailment_Label is in English.
- Entailment_label is of 2 types - entailed and not-entailed.
- Dataset can be used to train models for Natural Language Inference tasks in Hindi Language.
[More Information Needed]

### Supported Tasks and Leaderboards

- Natural Language Inference for Hindi

### Languages

Dataset is in Hindi

## Dataset Structure

- Data is structured in TSV format.
- Train and Test files are in seperate files

### Dataset Instances

An example of 'train' looks as follows.

```
{'hypothesis': 'यह खबर की सूचना है|', 'label': 'entailed', 'premise': 'गोपनीयता की नीति', 'topic': '1'}

```
### Data Fields

- Each row contatins 4 columns - Premise, Hypothesis, Label and Topic.

### Data Splits

- Train : 15553
- Valid : 2581
- Test : 2593

## Dataset Creation

- We employ a recasting technique from Poliak et al. (2018a,b) to convert publicly available BBC Hindi news text classification datasets in Hindi and pose them as TE problems
- In this recasting process, we build template hypotheses for each class in the label taxonomy
- Then, we pair the original annotated sentence with each of the template hypotheses to create TE samples.
- For more information on the recasting process, refer to paper "https://www.aclweb.org/anthology/2020.aacl-main.71"

### Source Data

Source Dataset for the recasting process is the BBC Hindi Headlines Dataset(https://github.com/NirantK/hindi2vec/releases/tag/bbc-hindi-v0.1)

#### Initial Data Collection and Normalization

-  BBC Hindi News Classification Dataset contains 4, 335 Hindi news headlines tagged across 14 categories: India, Pakistan,news, International, entertainment, sport, science, China, learning english, social, southasia, business, institutional, multimedia
-  We processed this dataset to combine two sets of relevant but low prevalence classes.
- Namely, we merged the samples from Pakistan, China, international, and southasia as one class called international.
- Likewise, we also merged samples from news, business, social, learning english, and institutional as news.
- Lastly, we also removed the class multimedia because there were very few samples.

#### Who are the source language producers?

Pls refer to this paper: "https://www.aclweb.org/anthology/2020.aacl-main.71"

### Annotations

#### Annotation process

Annotation process has been described in Dataset Creation Section.

#### Who are the annotators?

Annotation is done automatically.

### Personal and Sensitive Information

No Personal and Sensitive Information is mentioned in the Datasets.

## Considerations for Using the Data

Pls refer to this paper: https://www.aclweb.org/anthology/2020.aacl-main.71

### Discussion of Biases

Pls refer to this paper: https://www.aclweb.org/anthology/2020.aacl-main.71

### Other Known Limitations

No other known limitations

## Additional Information

Pls refer to this link: https://github.com/midas-research/hindi-nli-data

### Dataset Curators

It is written in the repo : https://github.com/avinsit123/hindi-nli-data that
- This corpus can be used freely for research purposes.
- The paper listed below provide details of the creation and use of the corpus. If you use the corpus, then please cite the paper.
- If interested in commercial use of the corpus, send email to midas@iiitd.ac.in.
- If you use the corpus in a product or application, then please credit the authors and Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi appropriately. Also, if you send us an email, we will be thrilled to know about how you have used the corpus.
- Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi, India disclaims any responsibility for the use of the corpus and does not provide technical support. However, the contact listed above will be happy to respond to queries and clarifications.
- Rather than redistributing the corpus, please direct interested parties to this page
- Please feel free to send us an email:
  - with feedback regarding the corpus.
  - with information on how you have used the corpus.
  - if interested in having us analyze your data for natural language inference.
  - if interested in a collaborative research project.

### Licensing Information

Copyright (C) 2019 Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi (MIDAS, IIIT-Delhi).
Pls contact authors for any information on the dataset.

### Citation Information

```
    @inproceedings{uppal-etal-2020-two,
    title = "Two-Step Classification using Recasted Data for Low Resource Settings",
    author = "Uppal, Shagun  and
      Gupta, Vivek  and
      Swaminathan, Avinash  and
      Zhang, Haimin  and
      Mahata, Debanjan  and
      Gosangi, Rakesh  and
      Shah, Rajiv Ratn  and
      Stent, Amanda",
    booktitle = "Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing",
    month = dec,
    year = "2020",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.aacl-main.71",
    pages = "706--719",
    abstract = "An NLP model{'}s ability to reason should be independent of language. Previous works utilize Natural Language Inference (NLI) to understand the reasoning ability of models, mostly focusing on high resource languages like English. To address scarcity of data in low-resource languages such as Hindi, we use data recasting to create NLI datasets for four existing text classification datasets. Through experiments, we show that our recasted dataset is devoid of statistical irregularities and spurious patterns. We further study the consistency in predictions of the textual entailment models and propose a consistency regulariser to remove pairwise-inconsistencies in predictions. We propose a novel two-step classification method which uses textual-entailment predictions for classification task. We further improve the performance by using a joint-objective for classification and textual entailment. We therefore highlight the benefits of data recasting and improvements on classification performance using our approach with supporting experimental results.",
}
```


Add dataset XhosaNavy English -Xhosa

Add dataset XhosaNavy English -Xhosa
More info : http://opus.nlpl.eu/XhosaNavy.php


add telugu-news corpus

Adding Telugu News Corpus to datasets.


Add BSD

This PR adds BSD, the Japanese-English business dialogue corpus by
[Rikters et al., 2020](https://www.aclweb.org/anthology/D19-5204.pdf).


Opus sardware

Added Opus sardware dataset for machine translation English to Sardinian.
for more info : http://opus.nlpl.eu/sardware.php


Adding dataset for proto_qa in huggingface datasets library

Added dataset for ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning
Followed all steps for adding a new dataset.


hindi discourse analysis dataset commit


adding psc dataset


adding dyk dataset


Fix typo in the comment in _info function


adding polemo2 dataset


Vinay/add/telugu books

Real data tests are failing as this dataset needs to be manually downloaded


Add LINNAEUS


Add Species-800


Add JFLEG

This PR adds [JFLEG ](https://www.aclweb.org/anthology/E17-2037/), an English grammatical error correction benchmark.

The tests were successful on real data, although it would be great if I can get some guidance on the **dummy data**. Basically, **for each source sentence there are 4 possible gold standard target sentences**. The original dataset comprise files in a flat structure, labelled by split then by source/target (e.g., dev.src, dev.ref0, ..., dev.ref3). Not sure what is the best way of adding this.

I imagine I can treat each distinct source-target pair as its own split? But having so many copies of the source sentence feels redundant, and it would make it less convenient to end-users who might want to access multiple gold standard targets simultaneously.


Add the Winograd Schema Challenge

Adds the Winograd Schema Challenge, including configs for the more canonical wsc273 as well as wsc285 with 12 new examples.

- https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html

The data format was a bit of a nightmare but I think I got it to a workable format.


Fix PerSenT

New PR for dataset PerSenT


Add GitHub version of ETH Py150 Corpus

Add the redistributable version of **ETH Py150 Corpus**


Add Urdu Sentiment Corpus (USC).

Added Urdu Sentiment Corpus. More details about the dataset over here.


Add ReFreSD dataset

This PR adds the **ReFreSD dataset**.
The original data is hosted [on this github repo](https://github.com/Elbria/xling-SemDiv) and we use the `REFreSD_rationale` to expose all the data.

Need feedback on:
- I couldn't generate the dummy data. The file we download is a tsv file, but without extension, I suppose this is the problem. I'm sure there is a simple trick to make this work.
- The feature names.
  - I don't know if it's better to stick to the classic `sentence1`, `sentence2` or to `sentence_en`, `sentence_fr` to be more explicit.
  - There is a binary label (called `label`, no problem here), and a 3-class label called `#3_labels` in the original tsv. I changed it to `all_labels` but I'm sure there is better.
- The rationales are lists of integers, extracted as a string at first. I wonder what's the best way to treat them, any idea? Also, I couldn't manage to make a `Sequence` of `int8` but I'm sure I've missed something simple.

Thanks in advance


updated after the class name update

@lhoestq <---


add wmt mlqe 2020 shared task

First commit for Shared task 1 (wmt_mlqw_task1) of WMT20 MLQE (quality estimation of machine translation)
Note that I copied the tags in the README for only one (of the 7 configurations): `en-de`.
There is one configuration for each pair of languages.


minor change in description in paws-x.py  and updated dataset_infos


added paws

Updating README and tags for dataset card in a while


adding xquad-r dataset


Adding XQUAD-R Dataset


Add Urdu Sentiment Corpus (USC).


adding discovery


Adding initial version of cord-19 dataset

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.

### TODO:
- [x] add more metadata
- [x] add full text
- [x] add pre-computed document embedding


Add xquad-r dataset


Add wikiqaar dataset

Arabic Wiki Question Answering Corpus.


Adding babi dataset

Adding the English version of bAbI.

Samples are taken from ParlAI for consistency with the main users at the moment.

Supersede #945 (problem with the rebase) and adresses the issues mentioned in the review (dummy data are smaller now and code comments are fixed).


Add Urdu fake news dataset.

Added Urdu fake news dataset. More information about the dataset can be found here.


Add Xitsonga Ner

Clean Xitsonga Ner PR


adding cdt dataset


Add Urdu fake news.


Add conda environment activation

Added activation of Conda environment before installing.


Add Google Great Code Dataset


Add Tashkeela dataset

Arabic Vocalized Words Dataset.


Fix incorrect MRQA train+SQuAD URL

Fix issue #1115


add dbpedia_14 dataset

This dataset corresponds to the DBpedia dataset requested in https://github.com/huggingface/datasets/issues/353.


Incorrect URL for MRQA SQuAD train subset

https://github.com/huggingface/datasets/blob/4ef4c8f8b7a60e35c6fa21115fca9faae91c9f74/datasets/mrqa/mrqa.py#L53

The URL for `train+SQuAD` subset of MRQA points to the dev set instead of train set. It should be `https://s3.us-east-2.amazonaws.com/mrqa/release/v2/train/SQuAD.jsonl.gz`.


Add sesotho ner corpus

Clean Sesotho PR


add qed

adding QED: Dataset for Explanations in Question Answering
https://github.com/google-research-datasets/QED
https://arxiv.org/abs/2009.06354


Initial version of cord-19 dataset from AllenAI with only the abstract

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [ ] Both tests for the real data and the dummy data pass.

### TODO:
- [ ] add more metadata
- [ ] add full text
- [ ] add pre-computed document embedding


Add Siswati Ner corpus

Clean Siswati PR


Using a feature named "_type" fails with certain operations

A column named `_type` leads to a `TypeError: unhashable type: 'dict'` for certain operations:
```python
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({"_type": ["whatever"]}).map()
concatenate_datasets([ds])
# or simply
Dataset(ds._data)
```
Context: We are using datasets to persist data coming from elasticsearch to feed to our pipeline, and elasticsearch has a `_type` field, hence the strange name of the column.

Not sure if you wish to support this specific column name, but if you do i would be happy to try a fix and provide a PR. I already had a look into it and i think the culprit is the `datasets.features.generate_from_dict` function. It uses the hard coded `_type` string to figure out if it reached the end of the nested feature object from a serialized dict.

Best wishes and keep up the awesome work!


add woz_dialogue

Adding Wizard-of-Oz task oriented dialogue dataset
https://github.com/nmrksic/neural-belief-tracker/tree/master/data/woz
https://arxiv.org/abs/1604.04562


Add Sepedi NER corpus

Finally a clean PR for Sepedi


Add arsentd_lev dataset

Add The Arabic Sentiment Twitter Dataset for Levantine dialect (ArSenTD-LEV)

Paper: [ArSentD-LEV: A Multi-Topic Corpus for Target-based Sentiment Analysis in Arabic Levantine Tweets](https://arxiv.org/abs/1906.01830)
Homepage: http://oma-project.com/


Add Urdu fake news


add xquad_r dataset


add TLC

Added TLC dataset


Add support to download kaggle datasets

enhancement

New feature or request

We can use API key


Add retries to download manager

New feature or request


Add Wikicorpus dataset

Add dataset.


Urdu fake news

Added Bend the Truth urdu fake news dataset. More inforation here.


Add tamilmixsentiment data


Add ToTTo Dataset

Adds a brand new table to text dataset: https://github.com/google-research-datasets/ToTTo


Add MSRA NER labels

Fixes #940


FIX matinf link in ADD_NEW_DATASET.md


Add TupleInf Open IE Dataset

For more information: https://allenai.org/data/tuple-ie


add urdu fake news dataset

Added Urdu fake news dataset. The dataset can be found here.


Add NCBI Disease Corpus dataset


Add Coached Conversation Preference Dataset

Adding [Coached Conversation Preference Dataset](https://research.google/tools/datasets/coached-conversational-preference-elicitation/)


Add Google wellformed query dataset

This pull request will add Google wellformed_query dataset. Link of dataset is https://github.com/google-research-datasets/query-wellformedness


add thaisum

ThaiSum, a large-scale corpus for Thai text summarization obtained from several online news websites namely Thairath, ThaiPBS, Prachathai, and The Standard. This dataset consists of over 350,000 article and summary pairs written by journalists. We evaluate the performance of various existing summarization models on ThaiSum dataset and analyse the characteristic of the dataset to present its difficulties.


add sharc_modified

Adding modified ShARC dataset https://github.com/nikhilweee/neural-conv-qa


add xquad_r dataset


Add Big Patent dataset

* More info on the dataset: https://evasharma.github.io/bigpatent/
* There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.


adding cdt dataset

- **Name:** *Cyberbullying Detection Task*
- **Description:** *The Cyberbullying Detection task was part of 2019 edition of PolEval competition. The goal is to predict if a given Twitter message contains a cyberbullying (harmful) content.*
- **Data:** *https://github.com/ptaszynski/cyberbullying-Polish*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*


add mutual friends conversational dataset

Mutual friends dataset
WIP

TODO:
- scenario_kbs (bug with pyarrow conversion)
- download from codalab checksums bug


adding cdsc dataset

- **Name**: *cdsc (domains: cdsc-e & cdsc-r)*
- **Description**: *Polish CDSCorpus consists of 10K Polish sentence pairs which are human-annotated for semantic relatedness and entailment. The dataset may be used for the evaluation of compositional distributional semantics models of Polish. The dataset was presented at ACL 2017. Please refer to the Wróblewska and Krasnowska-Kieraś (2017) for a detailed description of the resource.*
- **Data**: *http://2019.poleval.pl/index.php/tasks/*
- **Motivation**: *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*


Add the multilingual Exams dataset

https://github.com/mhardalov/exams-qa

`multilingual` configs have all languages mixed together

`crosslingual` mixes the languages for test but separates them for train and dec, so I've made one config per language for train/dev data and one config with the joint test set


Myanmar news dataset

Add news topic classification dataset in Myanmar / Burmese languagess

This data was collected in 2017 by Aye Hninn Khine , and published on GitHub with a GPL license
https://github.com/ayehninnkhine/MyanmarNewsClassificationSystem


Add Knowledge-Enhanced Language Model Pre-training (KELM)

Adds the KELM dataset.

- Webpage/repo: https://github.com/google-research-datasets/KELM-corpus
- Paper: https://arxiv.org/pdf/2010.12688.pdf


Add WikiANN NER dataset

This PR adds the full set of 176 languages from the balanced train/dev/test splits of WikiANN / PAN-X from: https://github.com/afshinrahimi/mmner

Until now, only 40 of these languages were available in `datasets` as part of the XTREME benchmark

Courtesy of the dataset author, we can now download this dataset from a Dropbox URL without needing a manual download anymore 🥳, so at some point it would be worth updating the PAN-X subset of XTREME as well 😄

Link to gist with some snippets for producing dummy data: https://gist.github.com/lewtun/5b93294ab6dbcf59d1493dbe2cfd6bb9

P.S. @yjernite I think I was confused about needing to generate a set of YAML tags per config, so ended up just adding a single one in the README.


nkjp-ner

- **Name:** *nkjp-ner*
- **Description:** *The NKJP-NER is based on a human-annotated part of NKJP. We extracted sentences with named entities of exactly one type. The task is to predict the type of the named entity.*
- **Data:** *https://klejbenchmark.com/tasks/*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*


add AJGT dataset

Arabic Jordanian General Tweets.


Added glucose dataset

This PR adds the [Glucose](https://github.com/ElementalCognition/glucose) dataset.


quac quac / coin coin

Add QUAC (Question Answering in Context)
I linearized most of the dictionnaries to lists.
Referenced to the authors' datasheet for the dataset card.
🦆🦆🦆
Coin coin


adding cleaned verion of E2E NLG

Found at: https://github.com/tuetschek/e2e-cleaning


Swedish MT STS-B

Added a Swedish machine translated version of the well known STS-B Corpus


Add DialogRE dataset

Adding the [DialogRE](https://github.com/nlpdata/dialogre) dataset Version 2.

- All tests passed successfully.


actually uses the previously declared VERSION on the configs in the template


add xlrd to test package requirements

Adds `xlrd` package to the test requirements to handle scripts that use `pandas` to load excel files


add conv_ai

Adding ConvAI dataset https://github.com/DeepPavlov/convai/tree/master/2017


Test


Add Pubmed (citation + abstract) dataset (2020).


add xquad-r dataset


Add ChrEn

Adding the Cherokee English machine translation dataset of https://github.com/ZhangShiyue/ChrEn


Not support links with 302 redirect

bug

Something isn't working

enhancement

New feature or request

I have an issue adding this download link https://github.com/jitkapat/thailitcorpus/releases/download/v.2.0/tlc_v.2.0.tar.gz

it might be because it is not a direct link (it returns 302 and redirects to aws that returns 403 for head requests).

```
r.head("https://github.com/jitkapat/thailitcorpus/releases/download/v.2.0/tlc_v.2.0.tar.gz", allow_redirects=True)
# <Response [403]>
```


Add the Ud treebank

This PR adds the 183 datasets in 104 languages of the UD Treebank.


Add KorNLU dataset

Added Korean NLU datasets. The link to the dataset can be found [here](https://github.com/kakaobrain/KorNLUDatasets) and the paper can be found [here](https://arxiv.org/abs/2004.03289)

**Note**: The MNLI tsv file is broken, so this code currently excludes the file. Please suggest other alternative if any @lhoestq

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


add labr dataset

Arabic Book Reviews dataset.


Fix squad V2 metric script

The current squad v2 metric doesn't work with the squad (v1 or v2) datasets. The script is copied from `squad_evaluate` in transformers that requires the labels (with multiple answers) to be like this:
```
references = [{'id': 'a', 'answers': [
    {'text': 'Denver Broncos', 'answer_start': 177},
    {'text': 'Denver Broncos', 'answer_start': 177}
]}]
```
while the dataset had references like this:
```
references = [{'id': 'a', 'answers':
    {'text': ['Denver Broncos' 'Denver Broncos'], 'answer_start': [177, 177]}
}]
```

Using one or the other format fails with the current squad v2 metric:
```
from datasets import load_metric
metric = load_metric("squad_v2")
predictions = [{'id': 'a', 'prediction_text': 'Denver Broncos', 'no_answer_probability': 0.0}]
references = [{'id': 'a', 'answers': [
    {'text': 'Denver Broncos', 'answer_start': 177},
    {'text': 'Denver Broncos', 'answer_start': 177}
]}]
metric.compute(predictions=predictions, references=references)
```
fails as well as
```
from datasets import load_metric
metric = load_metric("squad_v2")
predictions = [{'id': 'a', 'prediction_text': 'Denver Broncos', 'no_answer_probability': 0.0}]
references = [{'id': 'a', 'answers':
    {'text': ['Denver Broncos' 'Denver Broncos'], 'answer_start': [177, 177]}
}]
metric.compute(predictions=predictions, references=references)
```

This is because arrow reformats the references behind the scene.

With this PR (tested locally), both the snippets up there work and return proper results.


Add TLC

Added TLC dataset


added paws-x dataset

Added paws-x dataset. Updating README and tags in the dataset card in a while


Adding TamilMixSentiment


Add deal_or_no_dialog

Add deal_or_no_dialog Dataset

github: https://github.com/facebookresearch/end-to-end-negotiator
Paper: [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://arxiv.org/abs/1706.05125)


Add hebrew-sentiment

hebrew-sentiment dataset is ready! (including tests, tags etc)


Add dataset - SemEval 2014 - Task 1

Adding the dataset of SemEval 2014 Task 1

Found the dataset under the shared Google Sheet > Recurring Task Datasets
Task Homepage - https://alt.qcri.org/semeval2014/task1

Thank you!


Fix dataset URL and file names, and add column name in "Social Bias Frames" dataset

# Why I did
When I use "social_bias_frames" datasets in this library, I got 404 Errors.
So, I fixed this error and another some problems that I faced to use the dataset.

# What I did
* Modify this dataset URL
* Modify this dataset file names
* Add a "dataSource" column

Thank you!


add sharc dataset

This PR adds the ShARC dataset.

More info:
https://sharc-data.github.io/index.html


Add Facebook SimpleQuestionV2

Add simple questions v2: https://research.fb.com/downloads/babi/


Add GoEmotions

Adds the GoEmotions dataset, a nice emotion classification dataset with 27 (multi-)label annotations on reddit comments. Includes both a large raw version and a narrowed version with predefined train/test/val splits, which I've included as separate configs with the latter as a default.

- Webpage/repo: https://github.com/google-research/google-research/tree/master/goemotions
- Paper: https://arxiv.org/abs/2005.00547


Add siswati ner corpus


Adding NCHLT dataset


Add KorNLU