konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Add OffensEval-TR 2020 Dataset

This PR adds the OffensEval-TR 2020 dataset which is a Turkish offensive language corpus by me and @basakbuluz. The corpus consist of randomly sampled tweets and annotated in a similar way to [OffensEval](https://sites.google.com/site/offensevalsharedtask/) and [GermEval](https://projects.fzai.h-da.de/iggsa/).

- **Homepage:** [offensive-turkish](https://coltekin.github.io/offensive-turkish/)
- **Paper:** [A Corpus of Turkish Offensive Language on Social Media](https://coltekin.github.io/offensive-turkish/troff.pdf)
- **Point of Contact:** [Çağrı Çöltekin](ccoltekin@sfs.uni-tuebingen.de)

@lhoestq, can you please review this PR? 

> Awesome thank you !

Thanks for the small fixes @lhoestq 

@coltekin, we have added the data set that you created an article that says "Turkish Attack Language Community in Social Media", HuggingFace dataset update sprint for you. We added Sprint quickly for a short time. I hope you welcome it too. The dataset is accessible at https://huggingface.co/datasets/offenseval2020_tr. 

Thank you for the heads up. I am not familiar with the terminology above (no idea what a sprint is), but I am happy that you found the data useful. Please feel free to distribute/use it as you see fit.

The OffensEval version you included in your data set has only binary labels. There is also a version [here](https://coltekin.github.io/offensive-turkish/troff-v1.0.tsv.gz) which also includes fine-grained labels similar to the OffensEval English data set - Just in case it would be of interest.

If you have questions about the data set, or need more information please let me know.
Add SAMSum Corpus dataset

Did not spent much time writing README, might update later.

Copied description and some stuff from tensorflow_datasets
https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/summarization/samsum.py

also to fix the check_code_quality CI you have to remove the imports of the unused `csv` and `os`

@lhoestq Thanks for the review! I have done what you asked, README is also updated. 🤗 
The CI fails because of the added dependency. I have never used circleCI before, so I am curious how will you solve that?

I just added `py7zr` to our test dependencies

merging since the CI is fixed on master

Thanks! 🤗
Add Wiki Lingua Dataset

Hello,

This is my first PR. 

I have added Wiki Lingua Dataset along with dataset card to the best of my knowledge.
There was one hiccup though. I was unable to create dummy data because the data is in pkl format.
From the document, I see that:
```At the moment it supports data files in the following format: txt, csv, tsv, jsonl, json, xml```



I am done adding the dataset. Requesting to review and advise.

looks like this PR has changes about many other files than the ones for WIki Lingua 

Can you create another branch and another PR please ?

Any reason to have english as the default config over the other languages ?

> looks like this PR has changes about many other files than the ones for WIki Lingua
> 
> Can you create another branch and another PR please ?

Ok, I will create another branch and submit a fresh PR.

> Any reason to have english as the default config over the other languages ?

The data for all other languages has a direct reference to English article. Thus, I kept English as default.

closing in favor of #1470
add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC)

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


I created a clean PR where I also incorporated the suggested changes here: https://github.com/huggingface/datasets/pull/1449
can't load "german_legal_entity_recognition" dataset

FileNotFoundError: Couldn't find file locally at german_legal_entity_recognition/german_legal_entity_recognition.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py


Please if you could tell me more about the error? 

1. Please check the directory you've been working on
2. Check for any typos

> Please if you could tell me more about the error?
> 
> 1. Please check the directory you've been working on
> 2. Check for any typos

Error happens during the execution of this line:
dataset = load_dataset("german_legal_entity_recognition")

Also, when I try to open mentioned links via Opera I have errors "404: Not Found" and "This XML file does not appear to have any style information associated with it. The document tree is shown below." respectively.

Hello @nataly-obr, the `german_legal_entity_recognition` dataset has not yet been released (it is part of the coming soon v2 release).

You can still access it now if you want, but you will need to install `datasets` via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`

Please let me know if it solves the issue :)
The Snips Built In Intents 2016 dataset.

This PR proposes to add the Snips.ai built in intents dataset. The first configuration added is for the intent labels only, but the dataset includes entity slots that may in future be added as alternate configurations.

It is not clear how to automatically add the dummy data if the source data is a more complex json format. Should I manually take a fraction of the source data and include it as dummy data?

Will tag the dataset and update the dataset card.
Add dutch book review dataset

- Name: Dutch Book Review Dataset (DBRD)
- Description: The DBRD (pronounced dee-bird) dataset contains over 110k book reviews along with associated binary sentiment polarity labels and is intended as a benchmark for sentiment classification in Dutch.
- Paper: https://arxiv.org/abs/1910.00896
- Data: https://github.com/benjaminvdb/DBRD
- Motivation: A large (real-life) dataset of Dutch book reviews and sentiment polarity (positive/negative), based on the associated rating.

Checks
- [x] Create the dataset script /datasets/dbrd/dbrd.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _info(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

> Really cool thanks !
> 
> I left some (minor) comments

Thank you for your comments! 👍 I went ahead and improved the dataset card using your suggestions and some tweaks of my own. I hope you like it! 😄
Craigslist bargains

`craigslist_bargains` dataset from [here](https://worksheets.codalab.org/worksheets/0x453913e76b65495d8b9730d41c7e0a0c/)

Seeing this in the CircleCI builds, this is what I was originally getting before I started messing around with the download URLS to try to fix this:

`FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpwvji917g/extracted/d6185140afb24ad8fee67392100a478269cba286b0d88915a137fdf88872de14/dummy_data/train__VARIABLE_MISUSE__SStuB.txt-00001-of-00300'`

Could this be because of the files in my `dummy_data.zip`? I had to manually create it, and it looked like the test was looking for the following files, so I created the `.zip` with this structure:

```
Archive:  dummy_data.zip
   creating: dummy_data/
  inflating: dummy_data/blobtest     
  inflating: dummy_data/parsed.jsontrain  
  inflating: dummy_data/parsed.jsonvalidation 
```

Going to close this out and link to a new (cleaner) PR
add One Million Posts Corpus

- **Name:** One Million Posts Corpus
- **Description:** The “One Million Posts” corpus is an annotated data set consisting of user comments posted to an Austrian newspaper website (in German language).
- **Paper:** https://dl.acm.org/doi/10.1145/3077136.3080711
- **Data:** https://github.com/OFAI/million-post-corpus
- **Motivation:** Big German (real-life) dataset containing different annotations around forum moderation with expert annotations.

### Checkbox

- [X] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [X] Fill the `_DESCRIPTION` and `_CITATION` variables
- [X] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [X] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [X] Generate the metadata file `dataset_infos.json` for all configurations
- [X] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [X] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [X] Both tests for the real data and the dummy data pass.


merging since the CI is fixed on master
Created wiki_movies dataset.

First PR (ever). Hopefully this movies dataset is useful to others!

looks like your PR includes changes about many other files than the ones for wiki_movies

Can you create another branch and another PR please ?

I'm happy to. What's the best way to do that (sorry, I'm new to PRs etc.)?

Sure !

First please save your new dataset files somewhere.
Then you can do in this order:
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
git checkout -b my-new-branch-name
```
This will create a new branch from the updated master branch.
Then you can re-add your files and commit + push them

Once it's done you should be able to create a new PR using your new branch :) 

Done!

closing in favor of #1485
new pr for Turkish NER


Can you run `make style` to fix the code format ?

Also it looks like the file `file_downloaded/TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.zip/TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.DUMP` is missing inside the dummy_data.zip


(note that `TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.zip` is a directory name, not an actual zip file)

Hi Quentin, thank you for your patience with me. I've fixed the preprocessing pipeline, got this very weird error that Yacine told me to push. I've pushed it and after I'll find out that it will work, I will have my final pr on styling.

looks like you removed the dataset script file in your latest commit, is it expected ?
Add CovidQA dataset

This PR adds CovidQA, a question answering dataset specifically designed for COVID-19, built by hand from knowledge gathered from Kaggle’s COVID-19 Open Research Dataset Challenge.

Link to the paper: https://arxiv.org/pdf/2004.11339.pdf
Link to the homepage: https://covidqa.ai

It seems to share the same name as this dataset: https://openreview.net/forum?id=JENSKEEzsoU

> It seems to share the same name as this dataset: https://openreview.net/forum?id=JENSKEEzsoU

you're right it can be confusing. I'll add the organization/research group for clarity: `covid_qa_castorini`. I added the dataset you shared as `covid_qa_deepset` in another PR (#1182) 

Thanks for avoiding the name collision !
Added NewsPH Raw Dataset

Added the raw version of the NewsPH dataset, which was used to automatically generate the NewsPH-NLI corpus. Dataset of news articles in Filipino from mainstream Philippine news sites on the internet. Can be used as a language modeling dataset or to reproduce the NewsPH-NLI dataset.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

looks like this PR has changes to many files other than the ones for `NewsPH`

Can you create another branch and another PR please ?
Add KorQPair dataset

This PR adds a [Korean paired question dataset](https://github.com/songys/Question_pair) containing labels indicating whether two questions in a given pair are semantically identical. This dataset was used to evaluate the performance of [KoGPT2](https://github.com/SKT-AI/KoGPT2#subtask-evaluations) on a phrase detection downstream task. 

dummy data is missing

Hey @cceyda, thanks for pointing that out. I thought I'd added it, but seems like that wasn't the case. Just pushed a new commit with the dummy data.
Added WikiText-TL-39

This PR adds the WikiText-TL-39 Filipino Language Modeling dataset.

Paper: https://arxiv.org/abs/1907.00409
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

looks like this PR also includes changes about another dataset `covid_qa_deepset`

Could you create another branch and another PR that only includes the changes for the wikitext-tl-39 dataset ?
Add doc2dial dataset

### Doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset v0.9

Once complete this will add the [Doc2dial](https://doc2dial.github.io/data.html) dataset from the generic data sets list.

It not always practical to use nested `Sequence`. If you have troubles with sequence you can use lists instead. 

For example
```python

features=datasets.Features(
    {
        "dial_id": datasets.Value("string"),
        "doc_id": datasets.Value("string"),
        "domain": datasets.Value("string"),
        "turns": [
            {
                "turn_id": datasets.Value("int32"),
                "role": datasets.Value("string"),
                "da": datasets.Value("string"),
                "reference": [
                    {
                        "keys" : datasets.Value("string"),
                        "values": datasets.Value("string"),   
                    }

                ],
                "utterance": datasets.Value("string"),
            }
        ],
    }
),
```

this way `turns` will be a list of dict, and the "reference" key of `turns` will be a list of dict as well

No problem thanks for all your help getting this to the final stages! Added .gitignore, removed .lock and applied the changes you asked for.
Adding indonlu dataset

IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for Bahasa Indonesia. It contains 12 datasets.

looks like this PR includes changes about many files other than the ones for IndoNLU
Could you create another branch and another PR please ?

> looks like this PR includes changes about many files other than the ones for IndoNLU
> Could you create another branch and another PR please ?

Okay I'll make it
Multi Domain Sentiment Analysis Dataset (MDSA)


can you also run `make style` to format the code ?

I'll come back to this one in sometime :) @lhoestq 

Also if you would use `xml.etree.ElementTree` to parse the XML it would be awesome, because right now you're using an external dependency `xmltodict `

> Also if you would use xml.etree.ElementTree to parse the XML it would be awesome, because right now you're using an external dependency xmltodict

Its pseudo xml so elementtree fails. xmltodict seems to be working quite good for this. do we have examples of pseudo xml datasets?

for the other pseudo xml the text is parsed manually

Can you add `xmltodict` to the test dependencies in setup.py please to fix the CI please ?

Also can you add the dataset card with the tags and run `make style` ?

Hi :) have you had a chance to fix the test dependency and apply `make style` ?

Feel fee to ping me when it's ready for a review

Thanks for your contribution, @abhishekkrthakur. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add Curiosity Dialogs Dataset

Add Facebook [Curiosity Dialogs](https://github.com/facebookresearch/curiosity) Dataset.

@lhoestq I tried manually creating the dummy files. But unfortunately it was raising an error during testing the dummy data (regarding JSON parsing).

The JSONs are pretty big so I cannot actually open it without crashing the text editor.

 Do you have any suggestions?

@lhoestq I have made all the changes you mentioned.
Muchocine - Spanish movie reviews dataset


Hi @mapmeld !
have you had a chance to take a look at my suggestions ?

Feel free to ping me if you have questions or when you're ready for a review

@lhoestq unfortunately I don't have any more information about where the dataset comes from

It's fine, you can just add the sections titles back and leave the content with `[More Information Needed]`



added missing sections, updated the Python code ✅
adding conceptnet5

Adding the conceptnet5 and omcs txt files used to create the conceptnet5 dataset. Conceptne5 is a common sense dataset. More info can be found here: https://github.com/commonsense/conceptnet5/wiki

Thank you. I'll make those changes. but I'm having problems trying to push my changes to my fork


Hi, I've removed the TODO, and added a README.md. How do I push these changes?


Also, what docstring are you recommending?


> Hi, I've removed the TODO, and added a README.md. How do I push these changes?

you can just commit and push your changes to the same branch as your first commit.

@ghomasHudson I've tried it but still getting code quality error.  I've removed all blank lines, etc. required by flake8. Don't know what else to do

> @ghomasHudson I've tried it but still getting code quality error. I've removed all blank lines, etc. required by flake8. Don't know what else to do

Did you run `make style` before committing? When I run it, it fixes some things (e.g. Splitting line 96 which is currently too long).

I think @yjernite is looking into this. I did "make style" but nothing happens

looks like your PR includes changes about many other files than the ones related to conceptnet5

could you create another branch and another PR please ?

@lhoestq I'm not sure what I did wrong. What did I push that wasn't conceptnet5? How do I see this?

 did this

make style
flake8 datasets
git add datasets/<your_dataset_name>
git commit
git fetch upstream
git rebase upstream/master
git pull
git push -u origin conceptnet5

Thanks for rebasing and force push :) 

Yeah! Thank you @lhoestq, @ghomasHudson  and @yjernite !
Add numeric fused head dataset

Adding the [NFH: Numeric Fused Head](https://nlp.biu.ac.il/~lazary/fh/) dataset.

Everything looks sensible and I've included both the identification and resolution tasks. I haven't personally used this dataset in my research so am unable to specify what the default configuration / supervised keys should be.

I've filled out the basic info on the model card to the best of my knowledge but it's a little tricky to understand exactly what the fields represent.

Dataset author: @yanaiela 

> Thanks for adding this @ghomasHudson!
> I added some comments for some of the fields.
> 
> Also, I'm not sure about this since I haven't used the library yet, but maybe it's worth adding the identification and resolution as two separate datasets?

Thanks for replying @yanaiela - I hope this will make your dataset more accessible to a wider audience - I've added the changes to the model card you suggested.

In terms of the identification and resolution tasks, I've currently added them as separate `splits` in huggingface/datasets so you can load identification like this:

```
import datasets
dataset = datasets.load_dataset("numeric_fused_head", "identification")
print(dataset["train"][0])
>> {"tokens": ["The", "quick", "brown", "fox",....], "start_index": 11, "end_index": 12, "label": 0}
```
And resolution like this:

```
import datasets
dataset = datasets.load_dataset("numeric_fused_head", "resolution")
print(dataset["train"][0])
>> {"tokens": ["The", "quick", "brown", "fox",....], "head": ["AGE"], "anchors_indices": [12], ...}
```

I hope so too, thanks!

Re the splits, that makes sense to me.
add Korean HateSpeech dataset


It looks like you forgot to `make style` (I forget it a lot too 🤦 )
+ add dummy data

hi @cceyda 👋, thanks for the hint! it looks like i've run into some other errors though in `_split_generators` or `_generate_examples`.  do you have any idea of what's wrong here? 😅

I get the same errors on another pr too, so it probably has something to do with circleci, waiting on help.

the `RemoteDatasetTest ` error on the CI is fixed on master so it's fine

merging since the CI is fixed on master
Add WMT20 MLQE 3 shared tasks

3 tasks for the WMT 20 MLQE shared tasks -> 3 different datasets

(I re-created #1137 because it was too messy).

Note that in L199 `task3.py`, I used `logging.warning` to print some missing data in the train set.

Thanks for the comments Quentin!
I integrated them

It should be ok now!
Sorry I wasn't attentive enough.
(tests are currently failing, I understand it's from other datasets)

merging since the CI is fixed on master
Add limit

This PR adds [LiMiT](https://github.com/ilmgut/limit_dataset), a dataset for literal motion classification/extraction by [Manotas et al., 2020](https://www.aclweb.org/anthology/2020.findings-emnlp.88.pdf).

My bad, didn't see this on the open dataset list. Closing this since it overlaps with PR#1256
add taskmaster3

Adding Taskmaster-3 dataset
https://github.com/google-research-datasets/Taskmaster/tree/master/TM-3-2020.

The dataset structure almost same as original dataset with these two changes

1. In original dataset, each `apis` has a `args` filed which is a `dict`  with variable keys, which represent the name and value of the args. Here converted that to a `list` of `dict` with keys `arg_name` and `arg_value`. For ex.

```python
args = {"name.movie": "Mulan", "name.theater": ": "Mountain AMC 16"}
```
becomes 
```python
[
  {
    "arg_name": "name.movie",
    "arg_value": "Mulan"
  },
  {
    "arg_name": "name.theater",
    "arg_value": "Mountain AMC 16"
  }
]
```

2. Each `apis` has a `response` which is also a `dict` with variable keys representing response name/type and it's value. As above converted it to `list` of `dict` with keys `response_name` and `response_value`.





(you were unlucky, my rule of thumb for reducing the dummy data is to check whether they're above 50KB and you're at 52KB ^^')

> (you were unlucky, my rule of thumb for reducing the dummy data is to check whether they're above 50KB and you're at 52KB ^^')

Oops :(

Thanks for the suggestion, will reduce the size
[AfriBooms] Dataset exists already

When trying to add "AfriBooms": https://docs.google.com/spreadsheets/d/12ShVow0M6RavnzbBEabm5j5dv12zBaf0y-niwEPPlo4/edit#gid=1386399609 I noticed that the dataset exists already as a config of Universal Dependencies (universal_dependencies.py). I checked and the data exactly matches so that the new data link does not give any new data.

This PR improves the config's description a bit by linking to the paper.

It's so cool seeing all these datasets fly by and see how they are still of interest. I did my internship at the research group of Liesbeth Augustinus et al. They're a very kind group of people!

merging since the CI is fixed on master
Adding Enriched WebNLG dataset

This pull requests adds the `en` and `de` versions of the [Enriched WebNLG](https://github.com/ThiagoCF05/webnlg) dataset

Nice :) 

could you add the tags and also remove all the dummy data files that are not zipped ? The diff currently shows 800 files changes xD

Aaaaand it's rebase time - the new one is at #1264 !

closing this one since a new PR was created
add lst20 with manual download

passed on local:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_lst20
```
Not sure how to test:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_lst20
```

```
LST20 Corpus is a dataset for Thai language processing developed by National Electronics and Computer Technology Center (NECTEC), Thailand.
It offers five layers of linguistic annotation: word boundaries, POS tagging, named entities, clause boundaries, and sentence boundaries.
At a large scale, it consists of 3,164,002 words, 288,020 named entities, 248,181 clauses, and 74,180 sentences, while it is annotated with
16 distinct POS tags. All 3,745 documents are also annotated with one of 15 news genres. Regarding its sheer size, this dataset is
considered large enough for developing joint neural models for NLP.
Manually download at https://aiforthai.in.th/corpus.php
```

The pytest suite doesn't allow manual downloads so we just make sure that the `datasets-cli test` command to run without errors instead

@lhoestq Changes made. Thank you for the review. I've made some same mistakes for https://github.com/huggingface/datasets/pull/1253 too. Will fix them before review.
Turkish NER dataset, script works fine, couldn't generate dummy data

I've written the script (Turkish_NER.py) that includes dataset. The dataset is a zip inside another zip, and it's extracted as .DUMP file. However, after preprocessing I only get .arrow file. After I ran the script with no error messages, I get .arrow file of dataset, LICENSE and dataset_info.json. 

the .DUMP file looks like a txt with one example per line so adding `--match_text_files *.DUMP --n_lines 50`  to the dummy generation command might work .

We can close this PR since a new PR was open at #1268
Add ALT

ALT dataset -- https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/

the `RemoteDatasetTest ` erros in the CI are fixed on master so it's fine

used `Translation ` feature type and fixed few typos as you suggested.

Sorry, I made a mistake. please see new PR here. https://github.com/huggingface/datasets/pull/1436
addition of py_ast

The dataset consists of parsed Parsed ASTs that were used to train and evaluate the DeepSyn tool. 
The Python programs are collected from GitHub repositories
by removing duplicate files, removing project forks (copy of another existing repository)
,keeping only programs that parse and have at most 30'000 nodes in the AST and 
we aim to remove obfuscated files

Hi @reshinthadithyan !

As mentioned on the Slack, it would be better in this case to parse the file lines into the following feature structure:
```python
"ast": datasets.Sequence(
    {
        "type": datasets.Value("string"),
        "value": datasets.Value("string"),
        "children": datasets.Sequence(datasets.Value("int32")),
    },
)
```

Here are a few more things to fix before we can move forward:
- the class name needs to be the CamelCase equivalent of the script name, so here it will have to be `PyAst`
- the `README.md` needs to have the tags at the top
- The homepage/info list at the top should be in the same format as the template (added a suggestion)
- You should add the dataset tags and field description to the README as described here: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

Good luck, let us know if you need any help!

Hello @yjernite, changes have been made as we talked. Hope this would suffice. Thanks. Feel free to point out any room to improvement.

Good progress! Here's what still needs to be done:
- first, you need to rebase to master for the tests to pass :)
- the information in your `Data Fields` paragraph should go into `Data Instances`. Data fields should describe the fields one by one, as in e.g. https://github.com/huggingface/datasets/tree/master/datasets/eli5#data-fields
- you still need to add the YAML tags obtained with the tagging app

Should be good to go after that!

Hello @yjernite, changes as talked are being done.

Looks like this PR includes changes about many other files than the ones for py_ast

Could you create another branch and another PR please ?
Added Translator Human Parity Data For a Chinese-English news transla…

…tion system from Open dataset list for Dataset sprint, Microsoft Datasets tab.

Can you run `make style` to format the code and fix the CI please ?

> Can you run `make style` to format the code and fix the CI please ?

I ran `make style` before this PR and just a few minutes ago. No changes to the code. Not sure why the CI is failing.

Also, I attempted to see if I can get the source Chinese sentences from `wmt17` dataset. But this call `data = load_dataset('wmt17', "zh-en")` failed with this error: `FileNotFoundError: Couldn't find file at https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz`. I think it should be possible and fairly straightforward to get the pairing source sentences from it. I just can not test it right now.

The `RemoteDatasetTest ` errors in the CI are fixed on master so it's fine

merging since the CI is fixed on master
Add Fake News Detection in Filipino dataset

This PR adds the Fake News Filipino Dataset, a low-resource fake news detection corpora in Filipino. Contains 3,206 expertly-labeled news samples, half of which are real and half of which are fake.

Link to the paper: http://www.lrec-conf.org/proceedings/lrec2020/index.html

Link to the dataset/repo: https://github.com/jcblaisecruz02/Tagalog-fake-news

Hi! I'm the author of this paper (surprised to see our datasets have been added already).

That paper link only leads to the conference index, here's a link to the actual paper: https://www.aclweb.org/anthology/2020.lrec-1.316/

Would it be fine if I also edited your gsheet entry to reflect this change?

Hi Jan, please go ahead and update. I see you are also in the sprint slack channel. Let me know if what else needs updating. Thanks.
adding hind_encorp dataset

adding Hindi_Encorp05 dataset

help needed in dummy data

extension of the file is .plaintext so dummy data generation is failing


you can add the `--match_text_file "*.plaintext"` flag when generating the dummy data

also it looks like the PR is empty, is this expected ?

yes it is expected because I  made all my changes in  PR  #1186  then I again run code and open PR #1188 to see if this time test passes or not only so there is no code change from #1186 to #1188  
i tried  --match_text_file "*.plaintext" this time it is also not generating dummy data don't know why

well this PR includes no code change at all, can you make sure you added your changes in this one ?

feel free to ping me when you have added the files so I can take a look and help you with the dummy data

how to do that i dont know did i have to open new PR


 actually all my changes are visible in  #1186 but  don't know how to show same changes here

these are a the which i did in #1186 and same in #1188 
![1](https://user-images.githubusercontent.com/56379013/101646577-b4864500-3a5d-11eb-8a5a-91b1b441040a.png)
![2](https://user-images.githubusercontent.com/56379013/101646965-32e2e700-3a5e-11eb-94d9-276e602c6ded.png)
![4](https://user-images.githubusercontent.com/56379013/101646989-38d8c800-3a5e-11eb-92bb-d9c4cb2c3595.png)
![5](https://user-images.githubusercontent.com/56379013/101647017-41c99980-3a5e-11eb-87cf-5268e79df19d.png)
![6](https://user-images.githubusercontent.com/56379013/101647038-48581100-3a5e-11eb-8d05-f67834fcaa7b.png)

![8](https://user-images.githubusercontent.com/56379013/101647080-55750000-3a5e-11eb-8455-8936a35b35c2.png)
![9](https://user-images.githubusercontent.com/56379013/101647084-55750000-3a5e-11eb-988e-ae87f0b252a0.png)
![10](https://user-images.githubusercontent.com/56379013/101647182-6f164780-3a5e-11eb-8af3-f0b0186483c9.png)
![11](https://user-images.githubusercontent.com/56379013/101647230-7c333680-3a5e-11eb-9aeb-2b4ce65965e0.png)
![13](https://user-images.githubusercontent.com/56379013/101647257-848b7180-3a5e-11eb-871c-2fd77b047320.png)
![14](https://user-images.githubusercontent.com/56379013/101647268-89502580-3a5e-11eb-9e2a-b9f7ff1fc95e.png)
these same codes are in both #1186 and #1188 so because it is already present from PR #1186 because of that it is showing zeor code change in #1188 because it is already present from #1186 how i can show or highlight those changes











well for me https://github.com/huggingface/datasets/pull/1188/files is blank

This PR tries to merge the master branch of you fork into this repo, however I can't find changes with your files inside your master branch.

Maybe you can fork again the repo and try to create another PR ?

@lhoestq  i  opened a new pr #1438 but this time it fails many circl ci tests

Closing this one since a new PR was created
Add Adversarial SQuAD dataset

# Adversarial SQuAD

Adding the Adversarial [SQuAD](https://github.com/robinjia/adversarial-squad) dataset as part of the sprint 🎉 
This dataset adds adversarial sentences to a subset of the SQuAD dataset's dev examples. How to get the original squad example id is explained in readme->Data Instances. The whole data is intended for use in evaluation. (Which could of course be also used for training if one wants). So there is no classical train/val/test split, but a split based on the number of adversaries added.

There are 2 splits of this dataset:

- AddSent: Has up to five candidate adversarial sentences that don't answer the question, but have a lot of words in common with the question. This adversary is does not query the model in any way.
- AddOneSent: Similar to AddSent, but just one candidate sentences was picked at random. This adversary is does not query the model in any way.

(The AddAny and AddCommon datasets mentioned in the paper are dynamically generated based on model's output distribution thus are not included here)

The failing test look like some unrelated timeout thing, will probably clear if rerun.
- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

the CI error was just a connection error due to all the activity on the repo this week ^^'
I re-ran it so it should be good now

I hadn't realized the problem with the dummies since it had passed without errors.
Suggestion: maybe we can show the user a warning based on the generated dummy size.

Thanks for changing to configs ! Looks all good now :) 

Before we merge, can you re-lighten the dummy data please if you don't mind ? The idea is to have them weigh only a few KB (currently it's 50KB each). Feel free to remove any unnecessary files or chunk of text

(also you can ignore the `RemoteDatasetTest ` CI errors, they're fixed on master )

merging since the CI is fixed on master
ADD COVID-QA dataset

This PR adds the COVID-QA dataset, a question answering dataset consisting of 2,019 question/answer pairs annotated by volunteer biomedical experts on scientific articles related to COVID-19

Link to the paper: https://openreview.net/forum?id=JENSKEEzsoU
Link to the dataset/repo: https://github.com/deepset-ai/COVID-QA

merging since the CI is fixed on master

Wow, thanks for including this dataset from my side as well!
Add  KorQuAD v2 Dataset

# The Korean Question Answering Dataset v2
Adding the [KorQuAD](https://korquad.github.io/) v2 dataset as part of the sprint 🎉 
This dataset is very similar to SQuAD and is an extension of [squad_kor_v1](https://github.com/huggingface/datasets/pull/1178) which is why I added it as `squad_kor_v2`. 

- Crowd generated questions and answer (1-answer per question) for Wikipedia articles. Differently from V1 it includes the html structure and markup, which makes it a different enough dataset. (doesn't share ids between v1 and v2 either)

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

Edit: 🤦 looks like squad_kor_v1 commit sneaked in here too

looks like this PR also includes the changes for the V1
Could you only include the files of the V2 ?

hmm I have made the dummy data lighter retested on local and it passed not sure why it fails here?

merging since the CI is fixed on master
Add Korean NER dataset

This PR adds the [Korean named entity recognition dataset](https://github.com/kmounlp/NER). This dataset has been used in many downstream tasks, such as training [KoBERT](https://github.com/SKTBrain/KoBERT) for NER, as seen in this [KoBERT-CRF implementation](https://github.com/eagle705/pytorch-bert-crf-ner).

Closed via #1219
Add OpenPI Dataset

Add the OpenPI Dataset by AI2 (AllenAI)

Hi @Bharat123rox ! It looks like some of the dummy data is broken or missing. Did you auto-generate it? Does the local test pass for you?

@yjernite requesting you to have a look as to why the tests are failing only on Windows, there seems to be a backslash error somewhere, could it be the result of `os.path.join` and what should be the fix for this?

This is the `black` output locally:
```
(datasets_env) datasets (openpi) > black --check --line-length 119 --target-version py36 datasets/openpi/
All done! ✨ 🍰 ✨
1 file would be left unchanged.
```

Can you check your version of black (should be `20.8b1`) and run `make style again`? (And don't forget to rebase before pushing ;) )

The other test was a time-out error so should be good on the next commit

Thanks @yjernite the CI tests finally passed!!

Hi @Bharat123rox did you manage to join the different config into one using the IDs ?

Feel free to ping me when you're ready for the next review :) 

> Hi @Bharat123rox did you manage to join the different config into one using the IDs ?
> 
> Feel free to ping me when you're ready for the next review :) 

Not yet @lhoestq still working on this! Meanwhile please review #1507 where I added the SelQA dataset :)

Ok ! Let me review SelQA then :) 
Thanks for your help !

Apologies for the very late response. Here is the openpi dataset file with a single file per partition after merging `id_answers, answers.jsonl, question.jsonl , question_metadata.jsonl`

https://github.com/allenai/openpi-dataset/blob/main/data/gold-v1.1/dev.jsonl

Nice thank you @nikett !

Hi @Bharat123rox , when you get a chance, please feel free to use the dataset from the repo ( [Link](https://github.com/allenai/openpi-dataset/blob/main/data/gold-v1.1/dev.jsonl) ) . Please let me know if any file is missing! Thank you 

Hi @Bharat123rox are you working on this? 

@nikett Sorry I'm no longer working on this as I'm out of time for it, please feel free to raise a new PR for this



We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest to create this dataset there. Please, feel free to tell us if you need some help.
Add Universal Morphologies

Adding unimorph universal morphology annotations for 110 languages, pfew!!!

one lemma per row with all possible forms and annotations

https://unimorph.github.io/

Sorry for the delay, changed the default language to "ady" (first alphabetical) and only downloading the relevant files for each config (dataset_infos is till 918KB though)

Thanks for merging it ! Looks all good

Looks like I didn't reply to your last message, sorry about that.
Feel free to ping me when this happens :)
add wikipedia biography dataset

My first PR containing the Wikipedia biographies dataset. I have followed all the steps in the [guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md). It passes all the tests.

Does anyone know why am I getting this "Some checks were not successful" message? For the _code_quality_ one, I have successfully run the flake8 command.

Ok, I need to update the README.md, but don't know if that will fix the errors

Hi @ACR0S , thanks for adding the dataset!

It looks like `black` is throwing the code quality error: you need to run `make style` with the latest version of `black` (`black --version` should return 20.8b1)

We also added a requirement to specify encodings when using the python `open` function (line 163 in the current version of your script)

Finally, you will need to add the tags and field descriptions to the README as described here https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

Let us know if you have any further questions!

Also, please leave the full template of the readme  with the `[More Information Needed]` paragraphs: you don't have to fill them out now but it will make it easier for us to go back to later :) 

Thank you for your help, @yjernite! I have updated everything (finally run the _make style_, added the tags, the ecoding to the _open_ function and put back the empty fields in the README). Hope it works now! :)

LGTM!

merging since the CI is fixed on master
Add Naver sentiment movie corpus

This PR adds the [Naver sentiment movie corpus](https://github.com/e9t/nsmc), a dataset containing Korean movie reviews from Naver, the most commonly used search engine in Korea. This dataset is often used to benchmark models on Korean NLP tasks, as seen in [this paper](https://www.aclweb.org/anthology/2020.lrec-1.199.pdf). 

Closed via #1252
Add ar rest reviews

added restaurants reviews in Arabic for sentiment analysis tasks

Copy-pasted from the Slack discussion:
the annotation and language creators should be found , not unknown
the example should go under the "Data Instances" paragraph, not "Data fields"
can you remove the abstract from the citation and add it to the dataset description? More people will see that

@yjernite done! thanks for the feedback

@lhoestq not sure why it's failing tests now, I only changed cosmetics

You can ignores these errors
```

=========================== short test summary info ===========================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_great_code
```

they're fixed on master

Feel free to ping me for the final review once you managed to change to ClassLabel :) 

Hey @lhoestq I was able to fix it !! I think the same errors appeared on circleCI and now it's hopefully ready to be merged?

@lhoestq done! thanks for your review 

merging since the CI is fixed on master
Linguisticprobing

Adding Linguistic probing datasets from
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties
 https://www.aclweb.org/anthology/P18-1198/

Thanks for your contribution, @sileod.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

As you already created this dataset under your organization namespace (https://huggingface.co/datasets/metaeval/linguisticprobing), I think we can safely close this PR.

We would suggest you add a dataset card with the YAML tags, to make it searchable and discoverable.

Please, feel free to tell us if you need some help.
adding TabFact dataset

Adding TabFact: A Large-scale Dataset for Table-based Fact Verification.

https://github.com/wenhuchen/Table-Fact-Checking

- The tables are stored as individual csv files, so need to download 16,573 🤯  csv files. As a result the `datasets_infos.json` file is huge (6.62 MB).
- Original dataset has nested structure where, where table is one example and each table has multiple statements,
flattening the structure here so that each statement is one example.

FYI you guys are on GitHub's homepage 😍

<img width="1589" alt="Screenshot 2020-12-09 at 12 34 28" src="https://user-images.githubusercontent.com/326577/101624883-a0ecc700-39e8-11eb-8a97-11af0d036536.png">


Yeayy 😍 🔥
Add BSD 

This PR adds BSD, the Japanese-English business dialogue corpus by 
[Rikters et al., 2020](https://www.aclweb.org/anthology/D19-5204.pdf). 

Glad to have more Japanese data! Couple of comments:
- the abbreviation might confuse some people as there is also an OPUS BSD corpus, would you mind renaming it as `bsd_ja_en`?
- `flake8` is throwing some errors, you can run it locally (`flake8 datasets`) and fix what it tells you until it's happy :)
- We're not using `os.path.join` for URLs as it's unstable across systems (introduces backslashes on Windows). Can you write the URLs explicitly instead?

Thanks!

Fantastic, looks great!

> Fantastic, looks great!

Thanks for your help @yjernite, really appreciate it!

The RemoteDatasetTest is fixed on master so it's fine

merging since the CI is fixed on master
Add Species-800


thanks @lhoestq ! I probably need to do the same change in the `SplitGenerator`s (lines 107, 110 and 113). I'll open a new PR for that

Yes indeed ! Good catch 👍 
Feel free to open a PR and ping me

Hi , theres a issue pulling species_800 dataset , throws  google drive  error 

error: 

```
raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://drive.google.com/u/0/uc?id=1OletxmPYNkz2ltOr9pyT0b0iBtUWxslh&export=download/ (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='drive.google.com', port=443): Read timed out. (read timeout=10)")))
```
code: 
```
from datasets import load_dataset

dataset = load_dataset("species_800")
```

Hi @obonyojimmy! I am running the same commands and they work for me. Did you check your internet connection?
Add JFLEG

This PR adds [JFLEG ](https://www.aclweb.org/anthology/E17-2037/), an English grammatical error correction benchmark. 

The tests were successful on real data, although it would be great if I can get some guidance on the **dummy data**. Basically, **for each source sentence there are 4 possible gold standard target sentences**. The original dataset comprise files in a flat structure, labelled by split then by source/target (e.g., dev.src, dev.ref0, ..., dev.ref3). Not sure what is the best way of adding this.

I imagine I can treat each distinct source-target pair as its own split? But having so many copies of the source sentence feels redundant, and it would make it less convenient to end-users who might want to access multiple gold standard targets simultaneously. 

Hi @j-chim ! You're right it does feel redundant: your option works better, but I'd even suggest having the references in a Sequence feature, which you can declare as:
```
	    features=datasets.Features(
                {
                    "sentence": datasets.Value("string"),
                    "corrections": datasets.Sequence(datasets.Value("string")),
                }
            ),
```

To create the dummy data, you just need to tell the generator which files it should use, which you can do with:
`python datasets-cli dummy_data datasets/<your-dataset-folder> --auto_generate --match_text_files "train*,dev*,test*"`


Many thanks for this @yjernite! I've incorporated your feedback and sorted out the dummy data.
Add Urdu Sentiment Corpus (USC). 

Added Urdu Sentiment Corpus. More details about the dataset over <a href="https://github.com/MuhammadYaseenKhan/Urdu-Sentiment-Corpus">here</a>. 

@lhoestq have made the suggested changes in the README file.

@lhoestq Created a new PR #1231 with only the relevant files.
closing this one :)