noamaanMulla-03/datasets-issues huggingface dataset:
add inquisitive

Adding inquisitive qg dataset

More info: https://github.com/wjko2/INQUISITIVE


`dummy_data` right now contains all article files, keeping only the required articles for dummy data fails the dummy data test.
Any idea ?

> `dummy_data` right now contains all article files, keeping only the required articles for dummy data fails the dummy data test.
> Any idea ?

We should definitely find a way to make it work with only a few articles.

If it doesn't work right now for dummy data, I guess it's because it tries to load every single article file ?

If so, then maybe you can use `os.listdir` method to first check all the data files available in the path where the `articles.tgz` file is extracted. Then you can simply iter through the data files and depending on their ID, include them in the train or test set. With this method you should be able to have only a few articles files per split in the dummy data. Does that make sense ?

fixed! so the issue was, `articles_ids` were prepared based on the number of files in articles dir, so for dummy data questions it was not able to load some articles due to incorrect ids and the test was failing
Wongnai - Thai reviews dataset

40,000 reviews, previously released on GitHub ( https://github.com/wongnai/wongnai-corpus ) with an LGPL license, and on a closed Kaggle competition ( https://www.kaggle.com/c/wongnai-challenge-review-rating-prediction/ )


Thank you for contributing a Thai dataset, @mapmeld ! I'm super hyped. 
One comment I may add is that wongnai-corpus has two datasets: review classification (this) and word tokenization (https://github.com/wongnai/wongnai-corpus/blob/master/search/labeled_queries_by_judges.txt).
Would it be possible for you to rename this one something along the line of `wongnai-reviews` so that when/if we include the word tokenization dataset, we will know which is which.

This helps solve my check_code_quality issue.
```
make style
black --line-length 119 --target-version py36 datasets/wongnai
flake8 datasets/wongnai
isort datasets/wongnai/wongnai.py
```

@cstorm125 thanks! following your suggestions on formatting and on naming the dataset

I am writing a blog post about Thai NLP and transformers (example: mBERT does 1-2 character tokens instead of doing word segmentation), started adding this dataset to use as an example, and then saw you were adding other datasets. Great work! And if you know any Thai BERT models beyond https://github.com/ThAIKeras/bert we should maybe talk over email!
[WIP] Add multi woz

This PR adds version 2.2 of the Multi-domain Wizard of OZ dataset: https://github.com/budzianowski/multiwoz/tree/master/data/MultiWOZ_2.2

It was a pretty big chunk of work to figure out the structure, so I stil have tol add the description to the README.md

On the plus side the structure is broadly similar to that of the Google Schema Guided dialogue [dataset](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue), so will take care of that one next.
Adding the WebNLG dataset

This PR adds data from the WebNLG challenge, with one configuration per release and challenge iteration.

More information can be found [here](https://webnlg-challenge.loria.fr/)

Unfortunately, the data itself comes from a pretty large number of small XML files, so the dummy data ends up being quite large (8.4 MB even keeping only one example per file).


This is task is part of the GEM suite so will actually need a more complete dataset card. I'm taking a break for now though and will get back to it before merging :)
Add SciTLDR Dataset

Adds the SciTLDR Dataset by AI2
Added README card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents


CI failures seem to be unrelated (related to `norwegian_ner`)
```
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
```

you can just rebase from master to fix the CI :) 

can you just rebase from master before we merge ?

Sorry, the rebase from master went horribly wrong, I guess I'll just open another PR

Closing this one due to a mistake in rebasing :(

Continued in #1014
Add People's Daily NER dataset



> LGTM thanks :)
> 
> 
> 
> Before we merge, could you add a dataset card ? see here for more info: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card
> 
> 
> 
> Note that only the tags at the top of the dataset card are mandatory, if you feel like it's going to take too much time writing the rest to fill it all you can just skip the paragraphs

Nope. I don't think there is a citation. Also, can I do the dataset card later (maybe in bulk)?

We're doing one PR = one dataset to keep track of things. Feel free to add the tags later in this PR if you want to.
Also only the tags are required now, because we don't want people spending too much time on the cards

added @lhoestq 

Merging since the CI is fixed on master
Add CLINC150 Dataset

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


Looks like your PR now shows changes in many other files than the ones for CLINC150.
Feel free to create another branch and another PR

created new [PR](https://github.com/huggingface/datasets/pull/1016)

closing this!
Add SciTLDR Dataset (Take 2)

Adds the SciTLDR Dataset by AI2
Added the `README.md` card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents

Continued from #986 


@lhoestq please review this PR when you get free

If the CI fails just because of `RemoteDatasetTest` errors it's fine, they're fixed on master

> If the CI fails just because of `RemoteDatasetTest` errors it's fine, they're fixed on master

The same 3 tests are failing again :(
```
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
```

One trick if you want to add more datasets to avoid these errors : you can just rebase the master branch of your fork from the master branch of the repo. Then each time you make a new branch from master on your fork, it will include the fix for these errors

> One trick if you want to add more datasets to avoid these errors : you can just rebase the master branch of your fork from the master branch of the repo. Then each time you make a new branch from master on your fork, it will include the fix for these errors

Yes, I almost always do that, but somehow seems even this branch got old 😓 
I also do the following if I directly create a new branch locally: `git checkout -b <branchname> upstream/master` so it stays up-to date irrespective of my fork, still don't know how this crept in again

Merging this one since the CI is fixed on master
Add CLINC150 dataset

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data
Add Gutenberg time references dataset

This PR adds the gutenberg_time dataset: https://arxiv.org/abs/2011.04124


Description: "A clean data resource containing all explicit time references in a dataset of 52,183 novels whose full text is available via Project Gutenberg and the Hathi Trust Digital Library 2." > This is just the Gutenberg part.

Also, the paragraph at the top of the file would make a good Dataset Summary in the README :)
add generated_reviews_enth

`generated_reviews_enth` is created as part of [scb-mt-en-th-2020](https://arxiv.org/pdf/2007.03541.pdf) for machine translation task. This dataset (referred to as `generated_reviews_yn` in [scb-mt-en-th-2020](https://arxiv.org/pdf/2007.03541.pdf)) are English product reviews generated by [CTRL](https://arxiv.org/abs/1909.05858), translated by Google Translate API and annotated as accepted or rejected (`correct`) based on fluency and adequacy of the translation by human annotators. This allows it to be used for English-to-Thai translation quality esitmation (binary label), machine translation, and sentiment analysis.
Add E2E NLG

Adding the E2E NLG dataset.

More info here : http://www.macs.hw.ac.uk/InteractionLab/E2E/

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template and at least fill the tags 
- [x] Both tests for the real data and the dummy data pass.
[GEM] add DART data-to-text generation dataset

## Adding a Dataset
- **Name:** DART
- **Description:** DART consists of 82,191 examples across different domains with each input being a semantic RDF triple set derived from data records in tables and the tree ontology of the schema, annotated with sentence descriptions that cover all facts in the triple set.
- **Paper:** https://arxiv.org/abs/2007.02871v1
- **Data:** https://github.com/Yale-LILY/dart
- **Motivation:** the dataset will likely be included in the GEM benchmark

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Labels: name: dataset request

Is this a duplicate of #924 ?

Yup, closing! Haven't been keeping track of the solved issues during the sprint.
[GEM] add Schema-Guided Dialogue

## Adding a Dataset
- **Name:** The Schema-Guided Dialogue Dataset
- **Description:** The Schema-Guided Dialogue (SGD) dataset consists of over 20k annotated multi-domain, task-oriented conversations between a human and a virtual assistant. These conversations involve interactions with services and APIs spanning 20 domains, ranging from banks and events to media, calendar, travel, and weather.
- **Paper:** https://arxiv.org/pdf/2002.01359.pdf https://arxiv.org/pdf/2004.15006.pdf
- **Data:** https://github.com/google-research-datasets/dstc8-schema-guided-dialogue
- **Motivation:** Included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Labels: name: dataset request
[GEM] Add E2E dataset

## Adding a Dataset
- **Name:** E2E NLG dataset (for End-to-end natural language generation)
- **Description:**a dataset for training end-to-end, datadriven natural language generation systems in the restaurant domain, the datasets consists of 5,751 dialogue-act Meaning Representations (structured data) and 8.1 reference free-text utterances per dialogue-act on average
- **Paper:** https://arxiv.org/pdf/1706.09254.pdf https://arxiv.org/abs/1901.07931
- **Data:** http://www.macs.hw.ac.uk/InteractionLab/E2E/#data
- **Motivation:** This dataset will likely be included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Labels: name: dataset request
[GEM] Add WebNLG dataset

## Adding a Dataset
- **Name:** WebNLG
- **Description:** WebNLG consists of Data/Text pairs where the data is a set of triples extracted from DBpedia and the text is a verbalisation of these triples (16,095 data inputs and 42,873 data-text pairs). The data is available in English and Russian
- **Paper:** https://www.aclweb.org/anthology/P17-1017.pdf
- **Data:** https://webnlg-challenge.loria.fr/download/
- **Motivation:** Included in the GEM shared task, multilingual

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Labels: name: dataset request
[GEM] add WikiAuto text simplification dataset

## Adding a Dataset
- **Name:** WikiAuto
- **Description:** Sentences in English Wikipedia and their corresponding sentences in Simple English Wikipedia that are written with simpler grammar and word choices. A lot of lexical and syntactic paraphrasing. 
- **Paper:** https://www.aclweb.org/anthology/2020.acl-main.709.pdf
- **Data:** https://github.com/chaojiang06/wiki-auto
- **Motivation:** Included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Labels: name: dataset request
[GEM] add ASSET text simplification dataset

## Adding a Dataset
- **Name:** ASSET
- **Description:** ASSET is a crowdsourced
multi-reference corpus for assessing sentence simplification in English where each simplification was produced by executing several rewriting transformations.
- **Paper:** https://www.aclweb.org/anthology/2020.acl-main.424.pdf
- **Data:** https://github.com/facebookresearch/asset
- **Motivation:** Included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Labels: name: dataset request
Fix dataset URL and file names, and add column name in "Social Bias Frames" dataset

# Why I did
When I use "social_bias_frames" datasets in this library, I got 404 Errors.
So, I fixed this error and another some problems that I faced to use the dataset.

# What I did
* Modify this dataset URL
* Modify this dataset file names
* Add a "dataSource" column

Thank you!


Thanks a lot, looks good!
Hi

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
D

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
add wiki_hop

This PR adds the WikiHop dataset from the QAngaroo multi hop reading comprehension datasets

More info:
http://qangaroo.cs.ucl.ac.uk/index.html



Also the dummy data files are quite big (500KB)
If you could reduce that that would be nice (just look at the files inside and remove unecessary chunks of texts)
in general dummy data are just a few KB and we suggest to not get higher than 50KB

Having light dummy data makes the repo faster to clone
add dream dataset

Adding Dream: a Dataset and for Dialogue-Based Reading Comprehension

More details:
https://dataset.org/dream/
https://github.com/nlpdata/dream


> Awesome good job !
> 
> Could you also add a dataset card using the template guide here : https://github.com/huggingface/datasets/blob/master/templates/README_guide.md
> If you can't fill some fields then just leave `[N/A]`

Quick amendment: `[N/A]` is for fields that are not relevant: if you can't find the information just leave `[More Information Needed]`

@lhoestq since datset cards are optional for this sprint I'll add those later. Good for merge.

Indeed we only require the tags to be added now (the yaml part at the top of the dataset card).
Could you add them please ?
You can find more infos here : https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

@lhoestq added tags, I'll fill rest of the info after current sprint :)

The tests are failing tests for other datasets, not this one.

@lhoestq  could you tell me why these tests are failing, they don't seem related to this PR.
add scb_mt_enth_2020

## scb-mt-en-th-2020: A Large English-Thai Parallel Corpus

The primary objective of our work is to build a large-scale English-Thai dataset for machine translation.
We construct an English-Thai machine translation dataset with over 1 million segment pairs, curated from various sources,
namely news, Wikipedia articles, SMS messages, task-based dialogs, web-crawled data and government documents.
Methodology for gathering data, building parallel texts and removing noisy sentence pairs are presented in a reproducible manner.
We train machine translation models based on this dataset. Our models' performance are comparable to that of
Google Translation API (as of May 2020) for Thai-English and outperform Google when the Open Parallel Corpus (OPUS) is
included in the training data for both Thai-English and English-Thai translation.
The dataset, pre-trained models, and source code to reproduce our work are available for public use.
Add KorNLU

Added Korean NLU datasets. The link to the dataset can be found [here](https://github.com/kakaobrain/KorNLUDatasets) and the paper can be found [here](https://arxiv.org/abs/2004.03289)

**Note**: The MNLI tsv file is broken, so this code currently excludes the file. Please suggest other alternative if any @lhoestq 

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


the CI error about `social_bias_frames` is fixed on master so it's fine

created new [PR](https://github.com/huggingface/datasets/pull/1062)

looks like this PR includes many changes to other files that the ones related to KorNLU
Could you create another branch and another PR please ?

Wow crazy timing

hahahaha
Add GoEmotions

Adds the GoEmotions dataset, a nice emotion classification dataset with 27 (multi-)label annotations on reddit comments. Includes both a large raw version and a narrowed version with predefined train/test/val splits, which I've included as separate configs with the latter as a default.

- Webpage/repo: https://github.com/google-research/google-research/tree/master/goemotions
- Paper: https://arxiv.org/abs/2005.00547


Whoops, didn't mean for that to be merged yet (my bad). I'm reaching out to the authors since we'd like their feedback on the best way to have the `author` field anonymized or removed. Will send a patch once they get back to me.
Add UN Universal Declaration of Human Rights (UDHR)

Universal declaration of human rights with translations in 464 languages and dialects.

- UN page: https://www.ohchr.org/EN/UDHR/Pages/UDHRIndex.aspx
- Raw data source: https://unicode.org/udhr/index.html

Each instance of the dataset corresponds to one translation of the document. Since there's only one instance per language (and because there are 500 languages so the dummy data would be messy), I opted to just include them all under the same single config. I wasn't able to find any kind of license so I just copied the copyright notice.

I was pretty careful careful generating the language tags so they _should_ all be correct & consistent BCP-47 codes per the docs.
Add ZEST: ZEroShot learning from Task descriptions

Adds the ZEST dataset on zero-shot learning from task descriptions from AI2.

- Webpage: https://allenai.org/data/zest
- Paper: https://arxiv.org/abs/2011.08115

The nature of this dataset made the supported task tags tricky if you wouldn't mind giving any feedback @yjernite. Also let me know if you think we should have a `other-task-generalization` or something like that...


Looks good to me, we can ping the authors for more info later. And yes apply `other-task` labels liberally, we can sort them out later :) 

Looks ready to merge when you're ready @joeddav
Add Big Patent dataset

- More info on the dataset: https://evasharma.github.io/bigpatent/
- There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.
- ~Currently, there are no dummy data for this dataset yet as I'm facing some problems with generating them. I'm trying to add them later.~


Looks like this PR include changes about many other files than the ones related to big patent.
Could you create another branch and another PR ?

@lhoestq Just created a new PR here: https://github.com/huggingface/datasets/pull/1087
Requirements should specify pyarrow<1

I was looking at the docs on [Perplexity](https://huggingface.co/transformers/perplexity.html) via GPT2. When you load datasets and try to load Wikitext, you get the error,

```
module 'pyarrow' has no attribute 'PyExtensionType'
```
I traced it back to datasets having installed PyArrow 1.0.1 but there's not pinning in the setup file. 

https://github.com/huggingface/datasets/blob/e86a2a8f869b91654e782c9133d810bb82783200/setup.py#L68

Downgrading by installing `pip install "pyarrow<1"` resolved the issue.


Hello @mathcass I would want to work on this issue. May I do the same? 

@punitaojha, certainly. Feel free to work on this. Let me know if you need any help or clarity.

Hello @mathcass 
1. I did fork the repository and clone the same on my local system. 

2. Then learnt about how we can publish our package on pypi.org. Also, found some instructions on same in setup.py documentation.

3. Then I Perplexity document link that you shared above. I created a colab link from there keep both tensorflow and pytorch means a mixed option and tried to run it in colab but I encountered no errors at a point where you mentioned. Can you help me to figure out the issue. 

4.Here is the link of the colab file with my saved responses. 
https://colab.research.google.com/drive/1hfYz8Ira39FnREbxgwa_goZWpOojp2NH?usp=sharing

Also, please share some links which made you conclude that pyarrow < 1 would help. 

Access granted for the colab link. 

Thanks for looking at this @punitaojha and thanks for sharing the notebook. 

I just tried to reproduce this on my own (based on the environment where I had this issue) and I can't reproduce it somehow. If I run into this again, I'll include some steps to reproduce it. I'll close this as invalid. 

Thanks again. 

I am sorry for hijacking this closed issue, but I believe I was able to reproduce this very issue. Strangely enough, it also turned out that running `pip install "pyarrow<1" --upgrade` did indeed fix the issue (PyArrow was installed in version `0.14.1` in my case).

Please see the Colab below:

https://colab.research.google.com/drive/15QQS3xWjlKW2aK0J74eEcRFuhXUddUST

Thanks!
nkjp-ner

- **Name:** *nkjp-ner*
- **Description:** *The NKJP-NER is based on a human-annotated part of NKJP. We extracted sentences with named entities of exactly one type. The task is to predict the type of the named entity.*
- **Data:** *https://klejbenchmark.com/tasks/*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*