noamaanMulla-03/datasets-issues huggingface dataset:
Add HKCanCor

This PR adds the [Hong Kong Cantonese Corpus](http://compling.hss.ntu.edu.sg/hkcancor/), by [Luke and Wong 2015](http://compling.hss.ntu.edu.sg/hkcancor/data/LukeWong_Hong-Kong-Cantonese-Corpus.pdf). 

The dummy data included here was manually created, as the original dataset uses a xml-like format (see a copy hosted [here](https://github.com/fcbond/hkcancor/blob/master/sample/d1_v.txt) for example) that requires a few processing steps.
adding conceptnet5

Adding the conceptnet5 and omcs txt files used to create the conceptnet5 dataset. Conceptne5 is a common sense dataset. More info can be found here: https://github.com/commonsense/conceptnet5/wiki


Thank you. I'll make those changes. but I'm having problems trying to push my changes to my fork


Hi, I've removed the TODO, and added a README.md. How do I push these changes?


Also, what docstring are you recommending?


> Hi, I've removed the TODO, and added a README.md. How do I push these changes?

you can just commit and push your changes to the same branch as your first commit.

@ghomasHudson I've tried it but still getting code quality error.  I've removed all blank lines, etc. required by flake8. Don't know what else to do

> @ghomasHudson I've tried it but still getting code quality error. I've removed all blank lines, etc. required by flake8. Don't know what else to do

Did you run `make style` before committing? When I run it, it fixes some things (e.g. Splitting line 96 which is currently too long).

I think @yjernite is looking into this. I did "make style" but nothing happens

looks like your PR includes changes about many other files than the ones related to conceptnet5

could you create another branch and another PR please ?

@lhoestq I'm not sure what I did wrong. What did I push that wasn't conceptnet5? How do I see this?

 did this

make style
flake8 datasets
git add datasets/<your_dataset_name>
git commit
git fetch upstream
git rebase upstream/master
git pull
git push -u origin conceptnet5

Thanks for rebasing and force push :) 

Yeah! Thank you @lhoestq, @ghomasHudson  and @yjernite !
initial commit for MultiReQA 

Added MultiReQA, which is a dataset containing the sentence boundary annotation from eight publicly available QA datasets including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, BioASQ, RelationExtraction, and TextbookQA. 


looks like this dataset includes changes about many other files than the ones for multi_re_qa

Can you create another branch and another PR please ?

> looks like this dataset includes changes about many other files than the ones for multi_re_qa
> 
> Can you create another branch and another PR please ?

Sure I will do that. Thank you.
adding covid-tweets-japanese

Adding COVID-19 Japanese Tweets Dataset as part of the sprint.

Testing with dummy data is not working (the file is said to not exist). Sorry for the incomplete PR.


I think it's because the file you download uncompresses into a file and not a folder so `--autogenerate` couldn't create dummy data for you. See in your dummy_data.zip if there is a file there. If not, manually create your dummy data and compress them to dummy_data.zip.

@cstorm125 Thank you for the comment! 
As you point out, it seems my code has something wrong about downloading and uncompressing the file.
However, my manually created dummy data seems to contain a file of the required format.

On Colaboratory,
`!unzip /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip`
returns:

```
Archive:  /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip
   creating: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/
 extracting: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/data.csv.bz2  
```

The original data is `data.csv.bz2`, and I had a very hard time dealing with uncompressing bzip2.
I think I could handle it, but there may be problems remain.
Arabic pos dialect

A README.md and loading script for the Arabic POS Dialect dataset. The README is missing the sections on personal information, biases, and limitations, as it would probably be better for those to be filled by someone who can read the contents of the dataset and is familiar with Arabic NLP. 


looks like this PR includes changes about many other files than the oens for Araboc POS Dialect

Can you create a another branch and another PR please ?

Sorry! I'm not sure how I managed to do that. I'll make a new branch.
Add dutch book review dataset

- Name: Dutch Book Review Dataset (DBRD)
- Description: The DBRD (pronounced dee-bird) dataset contains over 110k book reviews along with associated binary sentiment polarity labels and is intended as a benchmark for sentiment classification in Dutch.
- Paper: https://arxiv.org/abs/1910.00896
- Data: https://github.com/benjaminvdb/DBRD
- Motivation: A large (real-life) dataset of Dutch book reviews and sentiment polarity (positive/negative), based on the associated rating.

Checks
- [x] Create the dataset script /datasets/dbrd/dbrd.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _info(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


> Really cool thanks !
> 
> I left some (minor) comments

Thank you for your comments! 👍 I went ahead and improved the dataset card using your suggestions and some tweaks of my own. I hope you like it! 😄
Add ALT

ALT dataset -- https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/


the `RemoteDatasetTest ` erros in the CI are fixed on master so it's fine

used `Translation ` feature type and fixed few typos as you suggested.

Sorry, I made a mistake. please see new PR here. https://github.com/huggingface/datasets/pull/1436
add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC)

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.



I created a clean PR where I also incorporated the suggested changes here: https://github.com/huggingface/datasets/pull/1449
Allow GitHub releases as dataset source

# Summary

Providing a GitHub release URL to `DownloadManager.download()` currently throws a `ConnectionError: Couldn't reach [DOWNLOAD_URL]`. This PR fixes this problem by adding an exception for GitHub releases in `datasets.utils.file_utils.get_from_cache()`.

# Reproduce

```
import datasets
url = 'http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz'
result = datasets.utils.file_utils.get_from_cache(url)

# Returns: ConnectionError: Couldn't reach http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz
```

# Cause

GitHub releases returns a HTTP status 403 (FOUND), indicating that the request is being redirected (to AWS S3, in this case). `get_from_cache()` checks whether the status is 200 (OK) or if it is part of two exceptions (Google Drive or Firebase), otherwise the mentioned error is thrown.

# Solution

Just like the exceptions for Google Drive and Firebase, add a condition for GitHub releases URLs that return the HTTP status 403. If this is the case, continue normally.
Update step-by-step guide for windows

Update step-by-step guide for windows to give an alternative to `make style`.


Hi @thomwolf, for simplification purposes, I think you could remove the "`pip install ...`" steps from this commit, 'cause these deps (black, isort, flake8) are already installed on `pip install -e ".[dev]"` on the [Start by preparing your environment](https://github.com/huggingface/datasets/blob/704107f924e74445f6f0fbde69a218b72178b588/ADD_NEW_DATASET.md#start-by-preparing-your-environment)
[GEM] add ToTTo Table-to-text dataset

## Adding a Dataset
- **Name:** ToTTo
- **Description:** ToTTo is an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description.
- **Paper:** https://arxiv.org/abs/2004.14373
- **Data:** https://github.com/google-research-datasets/totto
- **Motivation:** Included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Labels: name: dataset request

closed via #1098
add LeNER-Br dataset

Adding the LeNER-Br dataset, a Portuguese language dataset for named entity recognition 


I don't know what happened, my first commit passed on all checks, but after just a README.md update one of the scripts failed, is it normal? 😕 

Looks like a flaky connection error, I've launched a re-run, it should be fine :)

The RemoteDatasetTest error in the CI is just a connection error, we can ignore it

merging since the CI is fixed on master
add wisesight1000

`wisesight1000` contains Thai social media texts randomly drawn from the full `wisesight-sentiment`, tokenized by human annotators. Out of the labels `neg` (negative), `neu` (neutral), `pos` (positive), `q` (question), 250 samples each. Some texts are removed because they look like spam.Because these samples are representative of real world content, we believe having these annotaed samples will allow the community to robustly evaluate tokenization algorithms.
ethos first commit

Ethos passed all the tests except from this one: 
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>

with this error: 
E               OSError: Cannot find data file. 
E               Original error:
E               [Errno 2] No such file or directory: 


> Nice thanks !
> 
> I left a few comments
> 
> Also it looks like this PR includes changes about other files than the ones for ethos
> 
> Can you create another branch and another PR please ?

@lhoestq Should I close this PR? The new one is the: #1453

You can create another PR and close this one if you don't mind

> You can create another PR and close this one if you don't mind

Perfect! You should see the #1453 PR for the fixed version! Thanks
Add Mkqa dataset

# MKQA: Multilingual Knowledge Questions & Answers Dataset
Adding the [MKQA](https://github.com/apple/ml-mkqa) dataset as part of the sprint 🎉

There is no official data splits so I added just a `train` split.
 
differently from the original:
- answer:type field is a ClassLabel (I thought it might be possible to train on this as a label for categorizing questions)
- answer:entity field has a default value of empty string '' (since this key is not available for all in original)
- answer:alias has default value of []

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)



the `RemoteDatasetTest ` error pf the CI is fixed on master so it's fine

merging since the CI is fixed on master
SNLI dataset contains labels with value -1

```
import datasets
nli_data = datasets.load_dataset("snli")
train_data = nli_data['train']
train_labels = train_data['label']
label_set = set(train_labels)
print(label_set)
```

**Output:**
`{0, 1, 2, -1}`


I believe the `-1` label is used for missing/NULL data as per HuggingFace Dataset conventions. If I recall correctly SNLI has some entries with no (gold) labels in the dataset.

Ah, you're right. The dataset has some pairs with missing labels. Thanks for reminding me.
Add script_version suggestion when dataset/metric not found

Adds a helpful prompt to the error message when a dataset/metric is not found, suggesting the user might need to pass `script_version="master"` if the dataset was added recently. The whole error looks like:

> Couldn't find file locally at blah/blah.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1/metrics/blah/blah.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/met
rics/blah/blah.py.
If the dataset was added recently, you may need to to pass script_version="master" to find the loading script on the master branch.
Reddit jokes

196k Reddit Jokes dataset
Dataset link- https://raw.githubusercontent.com/taivop/joke-dataset/master/reddit_jokes.json


@lhoestq would you please rerun the test, 

I re-started the test.

@lhoestq let's hold off on merging for now though, having a conversation on Slack about some of the offensive content in the dataset and how/whether we want to present it.
Add HoVer multi-hop fact verification dataset

## Adding a Dataset
- **Name:** HoVer
- **Description:** https://twitter.com/YichenJiang9/status/1326954363806429186 contains 20K claim verification examples
- **Paper:** https://arxiv.org/abs/2011.03088
- **Data:** https://hover-nlp.github.io/
- **Motivation:** There are still few multi-hop information extraction benchmarks (HotpotQA, which dataset wase based off, notwithstanding)

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Labels: name: dataset request

Hi @yjernite  I'm new but wanted to contribute. Has anyone already taken this problem and do you think it is suitable for newbies?

Hi @tenjjin! This dataset is still up for grabs! Here's the link with the guide to add it. You should play around with the library first (download and look at a few datasets), then follow the steps here:

https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md

Closed by #1399
Add spanish billion words corpus

Add an unannotated Spanish corpus of nearly 1.5 billion words, compiled from different resources from the web.


Thank you for your feedback! I've reduced the dummy data size to 2KB.

I had to rebase to fix `RemoteDatasetTest` fails, sorry about the 80 commits. 
I could create a new clean PR if you prefer.

I have seen that in similar cases you have suggested to other contributors to create another branch and another PR, so I will do that.

Yes thank you !

The new PR is #1476 :)
add amazon polarity dataset

This corresponds to the amazon (binary dataset) requested in https://github.com/huggingface/datasets/issues/353


`amazon_polarity` is probably a subset of `amazon_us_reviews` but I am not entirely sure about that.
I guess `amazon_polarity` will help in reproducing results of papers using this dataset since even if it is a subset from `amazon_us_reviews`, it is not trivial how to extract `amazon_polarity` from `amazon_us_reviews`, especially since `amazon_us_reviews` was released after `amazon_polarity`. 

do you know what the problem would be ? should I pull the master before ? @lhoestq 

The error just appeared on master. I will try to fix it today.
You can ignore them since it's not related to the dataset you added

merging since the CI is fixed on master

Great thanks for the help.
Re-adding narrativeqa dataset

An update of #309. 


@lhoestq I think I've fixed the dummy data - it finally passes! I'll add the model card now.

@lhoestq - pretty happy with it now

> Awesome thank you !
> 
> Could you try to reduce the size of the dummy_data.zip file before we merge ? (it's 300KB right now)
> 
> To do so feel free to take a look inside it and remove all the unnecessary files and chunks of text, to only keep a few examples. The idea is to have a zip file that is only a few KB

Ah, it only contains 1 example for each split. I think the problem is that I include an entire story (like in the full dataset). We can probably get away with a summarised version.

> Nice thank you, can you make it even lighter if possible ? Something round 10KB would be awesone
> We try to keep the repo light so that it doesn't take ages to clone. So we have to make sure the dummy data are as small as possible for every single dataset.

Have trimmed a little more out of each example now.
Add narrativeQA

Redo of #1368 #309 #499

In redoing the dummy data a few times, I ended up adding a load of files to git. Hopefully this should work.


@lhoestq this is now only failing some random windows test (it appears to be somewhere in wnut_17)

This is a connection error, you can ignore it :) 
The level of activity on the lib is quite overwhelming, it stresses a bit the CI ^^
Add Turkish News Category Dataset (270K)

This PR adds the Turkish News Categories Dataset (270K) dataset which is a text classification dataset by me and @yavuzKomecoglu.  Turkish news dataset consisting of **273601 news** in **17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.




@lhoestq, can you please review this PR?


@SBrandeis,
Sorry. All of the latest version came to my branch. You can find final version. 
Resubmitted as a clean final version of #1466
I have completed all the review comments.

Closing this as PR is now https://github.com/huggingface/datasets/pull/1466
Add Turkish News Category Dataset (270K).Updates were made for review…

This PR adds the **Turkish News Categories Dataset (270K)** dataset which is a text classification dataset by me and @yavuzKomecoglu. Turkish news dataset consisting of **273601 news in 17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.

**Note**: Resubmitted as a clean version of the previous Pull Request(#1419).  @SBrandeis @lhoestq 


@SBrandeis, What exactly is it that makes the tests fail? Can you help me please?

These errors
```
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
```
appeared on master 3 hours ago and are now fixed.
(it was due to today's update of `xlrd` that broke two datasets)

You can ignore them, they're not related to your dataset

> These errors
> 
> ```
> =========================== short test summary info ============================
> FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
> FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
> FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
> FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
> ```
> 
> appeared on master 3 hours ago and are now fixed.
> (it was due to today's update of `xlrd` that broke two datasets)
> 
> You can ignore them, they're not related to your dataset

**If it is not a problem caused by us, we have already completed the review notes. You can then check it out and confirm?**

merging since the CI is fixed on master
Ar cov19

Adding ArCOV-19 dataset. ArCOV-19 is an Arabic COVID-19 Twitter dataset that covers the period from 27th of January till 30th of April 2020. ArCOV-19 is the first publicly-available Arabic Twitter dataset covering COVID-19 pandemic that includes over 1M tweets alongside the propagation networks of the most-popular subset of them (i.e., most-retweeted and-liked). The propagation networks include both retweets and conversational threads (i.e., threads of replies). ArCOV-19 is designed to enable research under several domains including natural language processing, information retrieval, and social computing, among others. 


merging since the CI is fixed on master
add thai_toxicity_tweet

Thai Toxicity Tweet Corpus contains 3,300 tweets (506 tweets with texts missing) annotated by humans with guidelines including a 44-word dictionary. The author obtained 2,027 and 1,273 toxic and non-toxic tweets, respectively; these were labeled by three annotators. The result of corpus analysis indicates that tweets that include toxic words are not always toxic. Further, it is more likely that a tweet is toxic, if it contains toxic words indicating their original meaning. Moreover, disagreements in annotation are primarily because of sarcasm, unclear existing target, and word sense ambiguity.

Notes from data cleaner: The data is included into [huggingface/datasets](https://www.github.com/huggingface/datasets) in Dec 2020. By this time, 506 of the tweets are not available publicly anymore. We denote these by `TWEET_NOT_FOUND` in `tweet_text`.
Processing can be found at [this PR](https://github.com/tmu-nlp/ThaiToxicityTweetCorpus/pull/1).
add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC) [PROPER]

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.



linter your code with flake8 and also run the commands present in Makefile for proper formatting 


merging since the CI is fixed on master