helmo/github-issues huggingface dataset:
new pr for Turkish NER
State: closed | Author: merveenoyan (CONTRIBUTOR)


Comments: Can you run `make style` to fix the code format ?

Also it looks like the file `file_downloaded/TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.zip/TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.DUMP` is missing inside the dummy_data.zip


(note that `TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.zip` is a directory name, not an actual zip file)

Hi Quentin, thank you for your patience with me. I've fixed the preprocessing pipeline, got this very weird error that Yacine told me to push. I've pushed it and after I'll find out that it will work, I will have my final pr on styling.

looks like you removed the dataset script file in your latest commit, is it expected ?
Add CovidQA dataset
State: closed | Author: olinguyen (CONTRIBUTOR)

This PR adds CovidQA, a question answering dataset specifically designed for COVID-19, built by hand from knowledge gathered from Kaggle’s COVID-19 Open Research Dataset Challenge.

Link to the paper: https://arxiv.org/pdf/2004.11339.pdf
Link to the homepage: https://covidqa.ai

Comments: It seems to share the same name as this dataset: https://openreview.net/forum?id=JENSKEEzsoU

> It seems to share the same name as this dataset: https://openreview.net/forum?id=JENSKEEzsoU

you're right it can be confusing. I'll add the organization/research group for clarity: `covid_qa_castorini`. I added the dataset you shared as `covid_qa_deepset` in another PR (#1182) 

Thanks for avoiding the name collision !
Added NewsPH Raw Dataset
State: closed | Author: jcblaisecruz02 (NONE)

Added the raw version of the NewsPH dataset, which was used to automatically generate the NewsPH-NLI corpus. Dataset of news articles in Filipino from mainstream Philippine news sites on the internet. Can be used as a language modeling dataset or to reproduce the NewsPH-NLI dataset.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments: looks like this PR has changes to many files other than the ones for `NewsPH`

Can you create another branch and another PR please ?
Add KorQPair dataset
State: closed | Author: jaketae (CONTRIBUTOR)

This PR adds a [Korean paired question dataset](https://github.com/songys/Question_pair) containing labels indicating whether two questions in a given pair are semantically identical. This dataset was used to evaluate the performance of [KoGPT2](https://github.com/SKT-AI/KoGPT2#subtask-evaluations) on a phrase detection downstream task. 

Comments: dummy data is missing

Hey @cceyda, thanks for pointing that out. I thought I'd added it, but seems like that wasn't the case. Just pushed a new commit with the dummy data.
Added WikiText-TL-39
State: closed | Author: jcblaisecruz02 (NONE)

This PR adds the WikiText-TL-39 Filipino Language Modeling dataset.

Paper: https://arxiv.org/abs/1907.00409
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments: looks like this PR also includes changes about another dataset `covid_qa_deepset`

Could you create another branch and another PR that only includes the changes for the wikitext-tl-39 dataset ?
add thainer
State: closed | Author: cstorm125 (CONTRIBUTOR)

ThaiNER (v1.3) is a 6,456-sentence named entity recognition dataset created from expanding the 2,258-sentence
[unnamed dataset](http://pioneer.chula.ac.th/~awirote/Data-Nutcha.zip) by
[Tirasaroj and Aroonmanakun (2012)](http://pioneer.chula.ac.th/~awirote/publications/).
It is used to train NER taggers in [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp).
The NER tags are annotated by [Tirasaroj and Aroonmanakun (2012)]((http://pioneer.chula.ac.th/~awirote/publications/))
for 2,258 sentences and the rest by [@wannaphong](https://github.com/wannaphong/).
The POS tags are done by [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp)'s `perceptron` engine trained on `orchid_ud`.
[@wannaphong](https://github.com/wannaphong/) is now the only maintainer of this dataset.
Add Naver sentiment movie corpus
State: closed | Author: jaketae (CONTRIBUTOR)

Supersedes #1168 

> This PR adds the [Naver sentiment movie corpus](https://github.com/e9t/nsmc), a dataset containing Korean movie reviews from Naver, the most commonly used search engine in Korea. This dataset is often used to benchmark models on Korean NLP tasks, as seen in [this paper](https://www.aclweb.org/anthology/2020.lrec-1.199.pdf).
Add doc2dial dataset
State: closed | Author: KMFODA (CONTRIBUTOR)

### Doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset v0.9

Once complete this will add the [Doc2dial](https://doc2dial.github.io/data.html) dataset from the generic data sets list.

Comments: It not always practical to use nested `Sequence`. If you have troubles with sequence you can use lists instead. 

For example
```python

features=datasets.Features(
    {
        "dial_id": datasets.Value("string"),
        "doc_id": datasets.Value("string"),
        "domain": datasets.Value("string"),
        "turns": [
            {
                "turn_id": datasets.Value("int32"),
                "role": datasets.Value("string"),
                "da": datasets.Value("string"),
                "reference": [
                    {
                        "keys" : datasets.Value("string"),
                        "values": datasets.Value("string"),   
                    }

                ],
                "utterance": datasets.Value("string"),
            }
        ],
    }
),
```

this way `turns` will be a list of dict, and the "reference" key of `turns` will be a list of dict as well

No problem thanks for all your help getting this to the final stages! Added .gitignore, removed .lock and applied the changes you asked for.
Adding indonlu dataset
State: closed | Author: yasirabd (CONTRIBUTOR)

IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for Bahasa Indonesia. It contains 12 datasets.

Comments: looks like this PR includes changes about many files other than the ones for IndoNLU
Could you create another branch and another PR please ?

> looks like this PR includes changes about many files other than the ones for IndoNLU
> Could you create another branch and another PR please ?

Okay I'll make it
Add Google Turkish Treebank Dataset
State: closed | Author: abhishekkrthakur (CONTRIBUTOR)
Labels: name: dataset contribution


Comments: Thanks for your contribution, @abhishekkrthakur. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add Google Noun Verb Dataset
State: closed | Author: abhishekkrthakur (CONTRIBUTOR)
Labels: name: dataset contribution


Comments: Thanks for your contribution, @abhishekkrthakur. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
adding bprec
State: closed | Author: kldarek (CONTRIBUTOR)


Comments: looks like this PR includes changes to many files other than the ones related to bprec
Can you create another branch and another PR please ?

> looks like this PR includes changes to many files other than the ones related to bprec
> Can you create another branch and another PR please ?

Yes, I realized I messed this one up, learning my way :) I'll close this one and open another hopefully clean PR :) Thanks!
Multi Domain Sentiment Analysis Dataset (MDSA)
State: closed | Author: abhishekkrthakur (CONTRIBUTOR)
Labels: name: dataset contribution


Comments: can you also run `make style` to format the code ?

I'll come back to this one in sometime :) @lhoestq 

Also if you would use `xml.etree.ElementTree` to parse the XML it would be awesome, because right now you're using an external dependency `xmltodict `

> Also if you would use xml.etree.ElementTree to parse the XML it would be awesome, because right now you're using an external dependency xmltodict

Its pseudo xml so elementtree fails. xmltodict seems to be working quite good for this. do we have examples of pseudo xml datasets?

for the other pseudo xml the text is parsed manually

Can you add `xmltodict` to the test dependencies in setup.py please to fix the CI please ?

Also can you add the dataset card with the tags and run `make style` ?

Hi :) have you had a chance to fix the test dependency and apply `make style` ?

Feel fee to ping me when it's ready for a review

Thanks for your contribution, @abhishekkrthakur. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add AmbigQA dataset
State: closed | Author: cceyda (CONTRIBUTOR)

# AmbigQA: Answering Ambiguous Open-domain Questions Dataset
Adding the [AmbigQA](https://nlp.cs.washington.edu/ambigqa/) dataset as part of the sprint 🎉 (from Open dataset list for Dataset sprint)

Added both the light and full versions (as seen on the dataset homepage)
The json format changes based on the value of one 'type' field, so I set the unavailable field to an empty list. This is explained in the README -> Data Fields

```py
train_light_dataset = load_dataset('./datasets/ambig_qa',"light",split="train")
val_light_dataset = load_dataset('./datasets/ambig_qa',"light",split="validation")
train_full_dataset = load_dataset('./datasets/ambig_qa',"full",split="train")
val_full_dataset = load_dataset('./datasets/ambig_qa',"full",split="validation")


for example in train_light_dataset:
    for i,t in enumerate(example['annotations']['type']):
        if t =='singleAnswer':
            # use the example['annotations']['answer'][i]
            # example['annotations']['qaPairs'][i] - > is []
            print(example['annotations']['answer'][i])
        else:
            # use the example['annotations']['qaPairs'][i]
            # example['annotations']['answer'][i] - > is []
            print(example['annotations']['qaPairs'][i])

```

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)
Added ade_corpus_v2, with 3 configs for relation extraction and classification task
State: closed | Author: Nilanshrajput (CONTRIBUTOR)

Adverse Drug Reaction Data: ADE-Corpus-V2 dataset added configs for different tasks with given data

Comments: @lhoestq I have added the tags they are in separate files for 3 different configs

@lhoestq  thanks for the review I added your suggested changes.

merging since the CI is fixed on master
Add Curiosity Dialogs Dataset
State: closed | Author: vineeths96 (CONTRIBUTOR)

Add Facebook [Curiosity Dialogs](https://github.com/facebookresearch/curiosity) Dataset.

Comments: @lhoestq I tried manually creating the dummy files. But unfortunately it was raising an error during testing the dummy data (regarding JSON parsing).

The JSONs are pretty big so I cannot actually open it without crashing the text editor.

 Do you have any suggestions?

@lhoestq I have made all the changes you mentioned.
Muchocine - Spanish movie reviews dataset
State: closed | Author: mapmeld (CONTRIBUTOR)


Comments: Hi @mapmeld !
have you had a chance to take a look at my suggestions ?

Feel free to ping me if you have questions or when you're ready for a review

@lhoestq unfortunately I don't have any more information about where the dataset comes from

It's fine, you can just add the sections titles back and leave the content with `[More Information Needed]`



added missing sections, updated the Python code ✅
adding conceptnet5
State: closed | Author: huu4ontocord (CONTRIBUTOR)

Adding the conceptnet5 and omcs txt files used to create the conceptnet5 dataset. Conceptne5 is a common sense dataset. More info can be found here: https://github.com/commonsense/conceptnet5/wiki

Comments: Thank you. I'll make those changes. but I'm having problems trying to push my changes to my fork


Hi, I've removed the TODO, and added a README.md. How do I push these changes?


Also, what docstring are you recommending?


> Hi, I've removed the TODO, and added a README.md. How do I push these changes?

you can just commit and push your changes to the same branch as your first commit.

@ghomasHudson I've tried it but still getting code quality error.  I've removed all blank lines, etc. required by flake8. Don't know what else to do

> @ghomasHudson I've tried it but still getting code quality error. I've removed all blank lines, etc. required by flake8. Don't know what else to do

Did you run `make style` before committing? When I run it, it fixes some things (e.g. Splitting line 96 which is currently too long).

I think @yjernite is looking into this. I did "make style" but nothing happens

looks like your PR includes changes about many other files than the ones related to conceptnet5

could you create another branch and another PR please ?

@lhoestq I'm not sure what I did wrong. What did I push that wasn't conceptnet5? How do I see this?

 did this

make style
flake8 datasets
git add datasets/<your_dataset_name>
git commit
git fetch upstream
git rebase upstream/master
git pull
git push -u origin conceptnet5

Thanks for rebasing and force push :) 

Yeah! Thank you @lhoestq, @ghomasHudson  and @yjernite !
Add numeric fused head dataset
State: closed | Author: ghomasHudson (CONTRIBUTOR)

Adding the [NFH: Numeric Fused Head](https://nlp.biu.ac.il/~lazary/fh/) dataset.

Everything looks sensible and I've included both the identification and resolution tasks. I haven't personally used this dataset in my research so am unable to specify what the default configuration / supervised keys should be.

I've filled out the basic info on the model card to the best of my knowledge but it's a little tricky to understand exactly what the fields represent.

Dataset author: @yanaiela 

Comments: > Thanks for adding this @ghomasHudson!
> I added some comments for some of the fields.
> 
> Also, I'm not sure about this since I haven't used the library yet, but maybe it's worth adding the identification and resolution as two separate datasets?

Thanks for replying @yanaiela - I hope this will make your dataset more accessible to a wider audience - I've added the changes to the model card you suggested.

In terms of the identification and resolution tasks, I've currently added them as separate `splits` in huggingface/datasets so you can load identification like this:

```
import datasets
dataset = datasets.load_dataset("numeric_fused_head", "identification")
print(dataset["train"][0])
>> {"tokens": ["The", "quick", "brown", "fox",....], "start_index": 11, "end_index": 12, "label": 0}
```
And resolution like this:

```
import datasets
dataset = datasets.load_dataset("numeric_fused_head", "resolution")
print(dataset["train"][0])
>> {"tokens": ["The", "quick", "brown", "fox",....], "head": ["AGE"], "anchors_indices": [12], ...}
```

I hope so too, thanks!

Re the splits, that makes sense to me.
Add HKCanCor
State: closed | Author: j-chim (CONTRIBUTOR)

This PR adds the [Hong Kong Cantonese Corpus](http://compling.hss.ntu.edu.sg/hkcancor/), by [Luke and Wong 2015](http://compling.hss.ntu.edu.sg/hkcancor/data/LukeWong_Hong-Kong-Cantonese-Corpus.pdf). 

The dummy data included here was manually created, as the original dataset uses a xml-like format (see a copy hosted [here](https://github.com/fcbond/hkcancor/blob/master/sample/d1_v.txt) for example) that requires a few processing steps.
add Korean HateSpeech dataset
State: closed | Author: stevhliu (MEMBER)


Comments: It looks like you forgot to `make style` (I forget it a lot too 🤦 )
+ add dummy data

hi @cceyda 👋, thanks for the hint! it looks like i've run into some other errors though in `_split_generators` or `_generate_examples`.  do you have any idea of what's wrong here? 😅

I get the same errors on another pr too, so it probably has something to do with circleci, waiting on help.

the `RemoteDatasetTest ` error on the CI is fixed on master so it's fine

merging since the CI is fixed on master
Add Korean NER dataset
State: closed | Author: jaketae (CONTRIBUTOR)

Supersedes #1177 

> This PR adds the [Korean named entity recognition dataset](https://github.com/kmounlp/NER). This dataset has been used in many downstream tasks, such as training [KoBERT](https://github.com/SKTBrain/KoBERT) for NER, as seen in this [KoBERT-CRF implementation](https://github.com/eagle705/pytorch-bert-crf-ner).
Add WMT20 MLQE 3 shared tasks
State: closed | Author: VictorSanh (CONTRIBUTOR)

3 tasks for the WMT 20 MLQE shared tasks -> 3 different datasets

(I re-created #1137 because it was too messy).

Note that in L199 `task3.py`, I used `logging.warning` to print some missing data in the train set.

Comments: Thanks for the comments Quentin!
I integrated them

It should be ok now!
Sorry I wasn't attentive enough.
(tests are currently failing, I understand it's from other datasets)

merging since the CI is fixed on master
Add limit
State: closed | Author: j-chim (CONTRIBUTOR)

This PR adds [LiMiT](https://github.com/ilmgut/limit_dataset), a dataset for literal motion classification/extraction by [Manotas et al., 2020](https://www.aclweb.org/anthology/2020.findings-emnlp.88.pdf).

Comments: My bad, didn't see this on the open dataset list. Closing this since it overlaps with PR#1256
adding medical-questions-pairs dataset
State: closed | Author: tuner007 (CONTRIBUTOR)

This dataset consists of 3048 similar and dissimilar medical question pairs hand-generated and labeled by Curai's doctors.
Dataset : https://github.com/curai/medical-question-pair-dataset
Paper : https://drive.google.com/file/d/1CHPGBXkvZuZc8hpr46HeHU6U6jnVze-s/view
add taskmaster3
State: closed | Author: patil-suraj (CONTRIBUTOR)

Adding Taskmaster-3 dataset
https://github.com/google-research-datasets/Taskmaster/tree/master/TM-3-2020.

The dataset structure almost same as original dataset with these two changes

1. In original dataset, each `apis` has a `args` filed which is a `dict`  with variable keys, which represent the name and value of the args. Here converted that to a `list` of `dict` with keys `arg_name` and `arg_value`. For ex.

```python
args = {"name.movie": "Mulan", "name.theater": ": "Mountain AMC 16"}
```
becomes 
```python
[
  {
    "arg_name": "name.movie",
    "arg_value": "Mulan"
  },
  {
    "arg_name": "name.theater",
    "arg_value": "Mountain AMC 16"
  }
]
```

2. Each `apis` has a `response` which is also a `dict` with variable keys representing response name/type and it's value. As above converted it to `list` of `dict` with keys `response_name` and `response_value`.





Comments: (you were unlucky, my rule of thumb for reducing the dummy data is to check whether they're above 50KB and you're at 52KB ^^')

> (you were unlucky, my rule of thumb for reducing the dummy data is to check whether they're above 50KB and you're at 52KB ^^')

Oops :(

Thanks for the suggestion, will reduce the size
Add large spanish corpus
State: closed | Author: lewtun (MEMBER)

Adds a collection of Spanish corpora that can be useful for pretraining language models. 

Following a nice suggestion from @yjernite we provide the user with three main ways to preprocess / load either 

* the whole corpus (17GB!)
* one specific sub-corpus
* the whole corpus, but return a single split. this is useful if you want to cache the whole preprocessing step once and interact with individual sub-corpora

See the dataset card for more details.

Ready for review!
[AfriBooms] Dataset exists already
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

When trying to add "AfriBooms": https://docs.google.com/spreadsheets/d/12ShVow0M6RavnzbBEabm5j5dv12zBaf0y-niwEPPlo4/edit#gid=1386399609 I noticed that the dataset exists already as a config of Universal Dependencies (universal_dependencies.py). I checked and the data exactly matches so that the new data link does not give any new data.

This PR improves the config's description a bit by linking to the paper.

Comments: It's so cool seeing all these datasets fly by and see how they are still of interest. I did my internship at the research group of Liesbeth Augustinus et al. They're a very kind group of people!

merging since the CI is fixed on master
Adding Enriched WebNLG dataset
State: closed | Author: TevenLeScao (CONTRIBUTOR)

This pull requests adds the `en` and `de` versions of the [Enriched WebNLG](https://github.com/ThiagoCF05/webnlg) dataset

Comments: Nice :) 

could you add the tags and also remove all the dummy data files that are not zipped ? The diff currently shows 800 files changes xD

Aaaaand it's rebase time - the new one is at #1264 !

closing this one since a new PR was created
add lst20 with manual download
State: closed | Author: cstorm125 (CONTRIBUTOR)

passed on local:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_lst20
```
Not sure how to test:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_lst20
```

```
LST20 Corpus is a dataset for Thai language processing developed by National Electronics and Computer Technology Center (NECTEC), Thailand.
It offers five layers of linguistic annotation: word boundaries, POS tagging, named entities, clause boundaries, and sentence boundaries.
At a large scale, it consists of 3,164,002 words, 288,020 named entities, 248,181 clauses, and 74,180 sentences, while it is annotated with
16 distinct POS tags. All 3,745 documents are also annotated with one of 15 news genres. Regarding its sheer size, this dataset is
considered large enough for developing joint neural models for NLP.
Manually download at https://aiforthai.in.th/corpus.php
```

Comments: The pytest suite doesn't allow manual downloads so we just make sure that the `datasets-cli test` command to run without errors instead

@lhoestq Changes made. Thank you for the review. I've made some same mistakes for https://github.com/huggingface/datasets/pull/1253 too. Will fix them before review.
Add Neural Code Search Dataset
State: closed | Author: vinaykudari (CONTRIBUTOR)


Comments: > Really good thanks !
> 
> I left a few comments

Thanks, resolved them :) 

looks like this PR includes changes about many other files than the ones for Code Search

can you create another branch and another PR please ?

> looks like this PR includes changes about many other files than the ones for Code Search
> 
> can you create another branch and another PR please ?

Okay sure
Medical question pairs
State: closed | Author: tuner007 (CONTRIBUTOR)

This dataset consists of 3048 similar and dissimilar medical question pairs hand-generated and labeled by Curai's doctors.
Dataset : https://github.com/curai/medical-question-pair-dataset
Paper : https://drive.google.com/file/d/1CHPGBXkvZuZc8hpr46HeHU6U6jnVze-s/view
**No splits added**
Update ADD_NEW_DATASET.md
State: closed | Author: BramVanroy (CONTRIBUTOR)

Windows needs special treatment again: unfortunately adding `torch` to the requirements does not work well (crashing the installation). Users should first install torch manually and then continue with the other commands.

This issue arises all the time when adding torch as a dependency, but because so many novice users seem to participate in adding datasets, it may be useful to add an explicit note for Windows users to ensure that they do not run into issues.