helmo/github-issues huggingface dataset:
Ar cov19
State: closed | Author: Fatima-Haouari (CONTRIBUTOR)

Adding ArCOV-19 dataset. ArCOV-19 is an Arabic COVID-19 Twitter dataset that covers the period from 27th of January till 30th of April 2020. ArCOV-19 is the first publicly-available Arabic Twitter dataset covering COVID-19 pandemic that includes over 1M tweets alongside the propagation networks of the most-popular subset of them (i.e., most-retweeted and-liked). The propagation networks include both retweets and conversational threads (i.e., threads of replies). ArCOV-19 is designed to enable research under several domains including natural language processing, information retrieval, and social computing, among others. 

Comments: merging since the CI is fixed on master
Add 1.5 billion words Arabic corpus 
State: closed | Author: zaidalyafeai (CONTRIBUTOR)

Needs https://github.com/huggingface/datasets/pull/1429 to work. 

Comments: Can't pass dummy data tests. For the instructions, it asks me to generate the following file `dummy_data/Youm7_XML_utf_8.rar/Youm7_utf_8.xml` which is strange, any ideas @lhoestq ?

cc: I tested the data locally and it works, maybe the dummy tests doesn't support `rar` ? 

In the dummy_data.zip files you must include the rar file as if is was already extracted.
In particular here `Youm7_XML_utf_8.rar` is a directory (not an archive).

Also I'm getting `BadRarFile: Failed the read enough data: req=16384 got=51` while trying to download and extract the `Alittihad_XML_utf_8.rar` file. Do you have this issue as well ?

I have rarfile 4.0

Sorry it was my mistake, I missed up the directories, it works now. Not sure why you got that error. I have the same version of `rarfile`. Between, there were some suggestions to change the dataset from `1bn_words_arabic` to `arabic_billion_words` like https://github.com/huggingface/datasets/tree/master/datasets/spanish_billion_words. 


I'm ok with renaming the dataset `arabic_billion_words` if you want.
Note that you will need to rename class name `ArabicBillionWords` instead of `BillionWords`
(though `BillionWords` was not matching `1bn_words_arabic` anyway)

You will need to regenerate the dataset_infos.json file after this change.
OR alternatively just replace all mentions of `billion_words` with `arabic_billion_words` in dataset_infos.json <- this trick should save you some time :)

Hmmm I'm still not able to run it on my side because of the rar error (I'm running macos)
I just tried with rarfile 3.1 and it didn't work either.
I would like to be able to run it end-to-end on my side before merging if you don't mind. Let me investigate this issue a little bit

No worries, I will investigate it as well. 

I created a minimal example in [colab ](https://colab.research.google.com/drive/11ijesuGbrQylANka0VdsZ5vXwIuxkheY?usp=sharing).

Nice thanks, maybe it's just an issue on my side then

Ok I managed to solve the BadRarFile issue on my side :) 
To fix it I had to install the `unrar` tool for macos (though it seems it's not available with `brew install` anymore, I had to install it from elsewhere).
Add german common crawl dataset
State: closed | Author: Phil1108 (NONE)
Labels: name: dataset contribution

Adding a subpart of the Common Crawl which was extracted with this repo https://github.com/facebookresearch/cc_net and additionally filtered for duplicates 

Comments: Hi @Phil1108 !
Have you had a chance to take a look at my suggestions ?
Feel free to ping me if you have questions or if you're ready for a review

Thanks again for adding this dataset, this one is very useful !

> 
> 
> Hi @Phil1108 !
> Have you had a chance to take a look at my suggestions ?
> Feel free to ping me if you have questions or if you're ready for a review
> 
> Thanks again for adding this dataset, this one is very useful !

Hello @lhoestq , 

okay so should we go on with only the 2 parts or wait until the hosting of all 200 GB is cleared at HF? 
If you want to merge it with the 2 parts only at first und upgrade later to all, sure I'll include the minor changes in the next few days and update it here

I think we can wait a bit to find a solution for the hosting and update the PR to include all the files when it's done.
In the meantime you can update the PR to take into account the suggestions regarding the dataset card, the parallel downloads and the use of eval.

A safer alternative to eval is `ast.literal_eval`, it should do the job

Thanks for your contribution, @Phil1108. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add Turkish News Category Dataset (270K)
State: closed | Author: basakbuluz (CONTRIBUTOR)

This PR adds the Turkish News Categories Dataset (270K) dataset which is a text classification dataset by me and @yavuzKomecoglu.  Turkish news dataset consisting of **273601 news** in **17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.



Comments: @lhoestq, can you please review this PR?


@SBrandeis,
Sorry. All of the latest version came to my branch. You can find final version. 
Resubmitted as a clean final version of #1466
I have completed all the review comments.

Closing this as PR is now https://github.com/huggingface/datasets/pull/1466
Add Hate Speech and Offensive Language Detection dataset
State: closed | Author: hugoabonizio (CONTRIBUTOR)

Add [Hate Speech and Offensive Language Detection dataset](https://github.com/t-davidson/hate-speech-and-offensive-language) from [this paper](https://arxiv.org/abs/1703.04009).

Comments: @lhoestq done! The failing testes don't seem to be related, it seems to be a connection issue, if I understand it correctly.

@lhoestq done!

merging since the CI is fixed on master
Add OffComBR
State: closed | Author: hugoabonizio (CONTRIBUTOR)

Add [OffComBR](https://github.com/rogersdepelle/OffComBR) from [Offensive Comments in the Brazilian Web: a dataset and baseline results](https://sol.sbc.org.br/index.php/brasnam/article/view/3260/3222) paper.

But I'm having a hard time generating dummy data since the original dataset extion is `.arff` and the [_create_dummy_data function](https://github.com/huggingface/datasets/blob/a4aeaf911240057286a01bff1b1d75a89aedd57b/src/datasets/commands/dummy_data.py#L185) doesn't allow it.

Comments: Hello @hugoabonizio, thanks for the contribution.
Regarding the fake data, you can generate it manually.
Running the `python datasets-cli dummy_data datasets/offcombr` should give you instructions on how to manually create the dummy data.
For reference, here is a spec for `.arff` files : https://www.cs.waikato.ac.nz/ml/weka/arff.html

@lhoestq again the failing tests doesn't seem to be related

merging since the CI is fixed on master
Add Tweet Eval Dataset
State: closed | Author: abhishekkrthakur (CONTRIBUTOR)


Comments: Hi @lhoestq,

Seeing that it has been almost two months to this draft, I'm willing to take this forward if you and @abhishekkrthakur don't mind. :)

Hi @gchhablani !
Sure if @abhishekkrthakur doesn't mind
Thanks for your help :)

Please feel free :) 

Hi @lhoestq, @abhishekkrthakur 

I believe this can be closed. Merged in #1829.
Adding TaPaCo Dataset with README.md
State: closed | Author: pacman100 (CONTRIBUTOR)


Comments: We want to keep the repo as light as possible so that it doesn't take ages to clone, that's why we ask for small dummy data files (especially when there are many of them). Let me know if you have questions or if we can help you on this

Hello @lhoestq , made the changes as you suggested and pushed, please review. By default, the dummy data was generated the way it was by the dummy data auto generate command. Thank you.
adding covid-tweets-japanese (again)
State: closed | Author: forest1988 (CONTRIBUTOR)

I had mistaken use git rebase, I was so hurried to fix it. However, I didn't fully consider the use of git reset , so I unintendedly stopped PR (#1367) altogether. Sorry about that.
I'll make a new PR.

Comments: README.md is not created yet. I'll add it soon.

Thank you for your detailed code review! It's so helpful.
I'll reflect them to the code in 24 hours.

You may have told me in Slack (I cannot find the conversation log though I've looked through threads), but I'm sorry it seems I'm still misunderstanding how to get YAML from the tagger.
I'm now asking on Slack if I am looking at the tagger the wrong way.

One more thing I'd like to ask.
Should I make changes by myself, or can I use the "Commit suggestion" feature?
I'm new to this feature and I don't know how the rules work in this repository, so I'd like to ask just in case.

Thank you very much for merging!
Add reasoning_bg
State: closed | Author: saradhix (CONTRIBUTOR)

Adding reading comprehension dataset for Bulgarian language

Comments: Hi @saradhix have you had the chance to reduce the size of the dummy data ?

Feel free to ping me when it's done so we can merge :) 

@lhoestq I have reduced the size of the dummy data manually and pushed the changes.

The CI errors are not related to your dataset.
They're fixed on master, you can ignore them

merging since the CI is fixed on master
initial commit for MultiReQA for second PR
State: closed | Author: Karthik-Bhaskar (CONTRIBUTOR)

Since last PR #1349 had some issues passing the tests. So, a new PR is generated.

Comments: Subsequent [PR #1426 ](https://github.com/huggingface/datasets/pull/1426) since this PR has uploaded other files along with the MultiReQA dataset.

closing this one since a new PR has been created
Add script_version suggestion when dataset/metric not found
State: closed | Author: joeddav (CONTRIBUTOR)

Adds a helpful prompt to the error message when a dataset/metric is not found, suggesting the user might need to pass `script_version="master"` if the dataset was added recently. The whole error looks like:

> Couldn't find file locally at blah/blah.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1/metrics/blah/blah.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/met
rics/blah/blah.py.
If the dataset was added recently, you may need to to pass script_version="master" to find the loading script on the master branch.
add amazon polarity dataset
State: closed | Author: hfawaz (CONTRIBUTOR)

This corresponds to the amazon (binary dataset) requested in https://github.com/huggingface/datasets/issues/353

Comments: `amazon_polarity` is probably a subset of `amazon_us_reviews` but I am not entirely sure about that.
I guess `amazon_polarity` will help in reproducing results of papers using this dataset since even if it is a subset from `amazon_us_reviews`, it is not trivial how to extract `amazon_polarity` from `amazon_us_reviews`, especially since `amazon_us_reviews` was released after `amazon_polarity`. 

do you know what the problem would be ? should I pull the master before ? @lhoestq 

The error just appeared on master. I will try to fix it today.
You can ignore them since it's not related to the dataset you added

merging since the CI is fixed on master

Great thanks for the help.
Add LIAR dataset
State: closed | Author: hugoabonizio (CONTRIBUTOR)

Add LIAR dataset from [“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection](https://www.aclweb.org/anthology/P17-2067/).

Comments: @lhoestq done! The failing testes don't seem to be related, it seems to be a connection issue, if I understand it correctly.

merging since the CI is fixed on master
add best2009
State: closed | Author: cstorm125 (CONTRIBUTOR)

`best2009` is a Thai word-tokenization dataset from encyclopedia, novels, news and articles by [NECTEC](https://www.nectec.or.th/) (148,995/2,252 lines of train/test). It was created for [BEST 2010: Word Tokenization Competition](https://thailang.nectec.or.th/archive/indexa290.html?q=node/10). The test set answers are not provided publicly.
added conv ai 2
State: closed | Author: rkc007 (CONTRIBUTOR)

Dataset : https://github.com/DeepPavlov/convai/tree/master/2018

Comments: @lhoestq Thank you for the suggestions. I added the changes to the branch and seems after rebasing it to master, all the commits previous commits got added. Should I create a new PR or should I keep this one only ? 

closing this one in favor of #1527
Add twi text c3
State: closed | Author: dadelani (CONTRIBUTOR)

Added Twi texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/

Comments: looks like this PR includes changes about other datasets

Can you only include the changes related to twi text c3 please ?

Hi @lhoestq , I have removed the unnecessary files. Can you please confirm?

You might need to either find a way to go back to the commit before it changes 389 files or create a new branch.

okay, I have created another branch, see the latest pull https://github.com/huggingface/datasets/pull/1518  @cstorm125 

Hii please follow me

Closing this one in favor of #1518
Add yoruba text c3
State: closed | Author: dadelani (CONTRIBUTOR)

Added Yoruba texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/

Comments: looks like this PR includes changes about other datasets


Thanks for the review. I'm a bit confused how to remove the files. Every time I add a new branch name using the following commands:

git fetch upstream
git rebase upstream/master
git checkout -b a-descriptive-name-for-my-changes

and push to the origin, this issue occurs

Can you try to create the branch from the master branch of your fork ?

first update your master branch:
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
```

then create a new one:
```
git checkout -b my-new-branch
```

I think you were still having the files because you were creating the new branch from a branch in which you've committed the files, instead of creating the new branch from the master branch

Got it, will correct that. Thanks

@lhoestq , I have removed the unnecessary files. Looks like I still have one error. How do I resolve this?

> @lhoestq , I have removed the unnecessary files. Looks like I still have one error. How do I resolve this?

I think it's connection error on piqa dataset. Can you try triggering the test again? I usually resolve similar issues with:
```
git fetch upstream
git rebase upstream/master
git push -u -f origin your_branch_name
```

thank you  @cstorm125 

I have created another pull request for this https://github.com/huggingface/datasets/pull/1515 @cstorm125  @lhoestq 

Hii please follow me

merging since the CI is fixed on master

Great, thanks a lot
Add FACTCK.BR dataset
State: closed | Author: hugoabonizio (CONTRIBUTOR)

This PR adds [FACTCK.BR](https://github.com/jghm-f/FACTCK.BR) dataset from [FACTCK.BR: a new dataset to study fake news](https://dl.acm.org/doi/10.1145/3323503.3361698).

Comments: @lhoestq done!

merging since the CI is fixed on master
adding marathi-wiki dataset
State: closed | Author: ekdnam (NONE)
Labels: name: dataset contribution

Adding marathi-wiki-articles dataset. 

Comments: Can you make it a draft PR until you've added the dataset please ? @ekdnam 

Done

Thanks for your contribution, @ekdnam. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Re-adding narrativeqa dataset
State: closed | Author: ghomasHudson (CONTRIBUTOR)

An update of #309. 

Comments: @lhoestq I think I've fixed the dummy data - it finally passes! I'll add the model card now.

@lhoestq - pretty happy with it now

> Awesome thank you !
> 
> Could you try to reduce the size of the dummy_data.zip file before we merge ? (it's 300KB right now)
> 
> To do so feel free to take a look inside it and remove all the unnecessary files and chunks of text, to only keep a few examples. The idea is to have a zip file that is only a few KB

Ah, it only contains 1 example for each split. I think the problem is that I include an entire story (like in the full dataset). We can probably get away with a summarised version.

> Nice thank you, can you make it even lighter if possible ? Something round 10KB would be awesone
> We try to keep the repo light so that it doesn't take ages to clone. So we have to make sure the dummy data are as small as possible for every single dataset.

Have trimmed a little more out of each example now.
adding covid-tweets-japanese
State: closed | Author: forest1988 (CONTRIBUTOR)

Adding COVID-19 Japanese Tweets Dataset as part of the sprint.

Testing with dummy data is not working (the file is said to not exist). Sorry for the incomplete PR.

Comments: I think it's because the file you download uncompresses into a file and not a folder so `--autogenerate` couldn't create dummy data for you. See in your dummy_data.zip if there is a file there. If not, manually create your dummy data and compress them to dummy_data.zip.

@cstorm125 Thank you for the comment! 
As you point out, it seems my code has something wrong about downloading and uncompressing the file.
However, my manually created dummy data seems to contain a file of the required format.

On Colaboratory,
`!unzip /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip`
returns:

```
Archive:  /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip
   creating: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/
 extracting: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/data.csv.bz2  
```

The original data is `data.csv.bz2`, and I had a very hard time dealing with uncompressing bzip2.
I think I could handle it, but there may be problems remain.
Add Mkqa dataset
State: closed | Author: cceyda (CONTRIBUTOR)

# MKQA: Multilingual Knowledge Questions & Answers Dataset
Adding the [MKQA](https://github.com/apple/ml-mkqa) dataset as part of the sprint 🎉

There is no official data splits so I added just a `train` split.
 
differently from the original:
- answer:type field is a ClassLabel (I thought it might be possible to train on this as a label for categorizing questions)
- answer:entity field has a default value of empty string '' (since this key is not available for all in original)
- answer:alias has default value of []

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


Comments: the `RemoteDatasetTest ` error pf the CI is fixed on master so it's fine

merging since the CI is fixed on master
Narrative QA (Manual Download Stories) Dataset
State: closed | Author: rsanjaykamath (CONTRIBUTOR)

Narrative QA with manual download for stories. 

Comments: Hi ! Maybe we can rename it `narrativeqa_manual` to make it explicit that this one requires manual download contrary to `narrativeqa` ?
It's important to have this one as well, in case the `narrativeqa` one suffers from download issues (checksums or dead links for example).

You can also copy the dataset card from `narrativeqa` and add the dummy data as well

Thanks @lhoestq will do all this and submit a request in the coming days. 😊 

Closing this as another pull request is already done.
add wisesight1000
State: closed | Author: cstorm125 (CONTRIBUTOR)

`wisesight1000` contains Thai social media texts randomly drawn from the full `wisesight-sentiment`, tokenized by human annotators. Out of the labels `neg` (negative), `neu` (neutral), `pos` (positive), `q` (question), 250 samples each. Some texts are removed because they look like spam.Because these samples are representative of real world content, we believe having these annotaed samples will allow the community to robustly evaluate tokenization algorithms.
Add spider dataset
State: closed | Author: olinguyen (CONTRIBUTOR)

This PR adds the Spider dataset, a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases.

Dataset website: https://yale-lily.github.io/spider
Paper link: https://www.aclweb.org/anthology/D18-1425/
add LeNER-Br dataset
State: closed | Author: jonatasgrosman (CONTRIBUTOR)

Adding the LeNER-Br dataset, a Portuguese language dataset for named entity recognition 

Comments: I don't know what happened, my first commit passed on all checks, but after just a README.md update one of the scripts failed, is it normal? 😕 

Looks like a flaky connection error, I've launched a re-run, it should be fine :)

The RemoteDatasetTest error in the CI is just a connection error, we can ignore it

merging since the CI is fixed on master
initial commit for MultiReQA 
State: closed | Author: Karthik-Bhaskar (CONTRIBUTOR)

Added MultiReQA, which is a dataset containing the sentence boundary annotation from eight publicly available QA datasets including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, BioASQ, RelationExtraction, and TextbookQA. 

Comments: looks like this dataset includes changes about many other files than the ones for multi_re_qa

Can you create another branch and another PR please ?

> looks like this dataset includes changes about many other files than the ones for multi_re_qa
> 
> Can you create another branch and another PR please ?

Sure I will do that. Thank you.
Add spanish billion words corpus
State: closed | Author: mariagrandury (CONTRIBUTOR)

Add an unannotated Spanish corpus of nearly 1.5 billion words, compiled from different resources from the web.

Comments: Thank you for your feedback! I've reduced the dummy data size to 2KB.

I had to rebase to fix `RemoteDatasetTest` fails, sorry about the 80 commits. 
I could create a new clean PR if you prefer.

I have seen that in similar cases you have suggested to other contributors to create another branch and another PR, so I will do that.

Yes thank you !

The new PR is #1476 :)
hate_speech_18 initial commit
State: closed | Author: czabo (CONTRIBUTOR)


Comments: > Nice thanks !
> 
> Can you rename the dataset folder and the dataset script name `hate_speech18` instead of `hate_speech_18` to follow the snake case convention we're using ?
> 
> Also it looks like the dummy_data.zip file is quite big (almost 4MB).
> Can you try to reduce its size ?
> 
> To do so feel free to take a look inside it and remove all the unnecessary files or chunks of texts. The idea is to only keep a few examples

Done, thanks! 

Re-opened in https://github.com/huggingface/datasets/pull/1486
Add spanish billion words
State: closed | Author: mariagrandury (CONTRIBUTOR)

Add an unannotated corpus of the Spanish language of nearly 1.5 billion words, compiled from different resources from the web.

The dataset needs 10 GB (download: 1.89 GiB, generated: 8.34 GiB, post-processed: Unknown size, total: 10.22 GiB), the test using dummy data pass but my laptop isn't able to run it on the real data (I left it running for over 8 hours and it didn't finish).

Comments: The tests failed because of ```RemoteDatasetTest``` so I tried ```git rebase``` and messed everything up. I've made a new clean PR (#1347).
added un_ga dataset
State: closed | Author: param087 (CONTRIBUTOR)

Hi :hugs:, This is a PR for [United nations general assembly resolutions: A six-language parallel corpus](http://opus.nlpl.eu/UN.php) dataset

Comments: Looks like this PR includes changes about many other files than the ones for un_ga

Can you create another branch an another PR please ?

@lhoestq, Thank you for suggestions. I have made the changes and raised the new PR https://github.com/huggingface/datasets/pull/1569.