konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Adding NELL

NELL is a knowledge base and knowledge graph along with sentences used to create the KB. See http://rtw.ml.cmu.edu/rtw/ for more details.

hi @lhoestq, I wanted to push another change to this branch b/c I found a bug in the parsing. I need to swap arg1 and arg2. I tried to git push -u origin nell but it didn't work. So I tried to do git push --force -u origin nell which seems to work, but nothing is happening to this branch. I think this is because it's closed. Do I need to open another PR?

The change should be below in _generate_examples:
                    best_arg1 = row[9]
                    best_arg2 = row[8]


Hi @ontocord !

Yup the easiest thing to do is to open a new PR with a title like "Fix NELL dataset argument order". If it's a simple fix we can look at it pretty fast :)
added conv_ai_3 dataset

Dataset : https://github.com/aliannejadi/ClariQ/


@lhoestq Thank you for suggesting changes. I fixed all the changes you suggested. Can you please review it again? 

@lhoestq Thank you for reviewing and suggesting changes. I made the requested changes. Can you please review it again?

Thanks @lhoestq for reviewing it again. I made the required changes. Can you please have a look ?

merging since the CI is fixed on master
Added Times of India News Headlines Dataset

Dataset name: Times of India News Headlines
link: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/DPQMQH

@lhoestq @abhishekkrthakur what happened here ?


@lhoestq everything alright here ?

@tanmoyio please have patience. @lhoestq has to look at 150+ PRs and it may take time. The PR looks good to me but we wait for his confirmation :) 🤗
Adding medical database chinese and english

Error in creating dummy dataset

Let me know it that helps !
Also feel free to ping me if you have other questions or if I can help you.

Now I am getting an Assertion Error!
![image](https://user-images.githubusercontent.com/16264631/101943915-f5bf5600-3c11-11eb-84e5-045bbc472162.png)


All tests have passed. However, PyTest is still failing with the `AssertionError` as before. Also _datasets_info.json_ actually does not seem to provide much info. Please review and let me know what has to be improved.

Thanks!

[PR-1503](https://github.com/huggingface/datasets/pull/1503) on the COVID dialog dataset from the same University has similar features. I kept it separate because it is only on COVID qa. Also consists of only single files per language. Kindly let me know if it has to be added as two more configurations to this existing dataset itself. If it has to be added, then can I still freeze the file names like in PR-1503?

It's ok to have them separate
Add narrativeQA

Redo of #1368 #309 #499

In redoing the dummy data a few times, I ended up adding a load of files to git. Hopefully this should work.

@lhoestq this is now only failing some random windows test (it appears to be somewhere in wnut_17)

This is a connection error, you can ignore it :) 
The level of activity on the lib is quite overwhelming, it stresses a bit the CI ^^
Inconsistent argument names.

Just find it a wee bit odd that in the transformers library `predictions` are those made by the model:
https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_utils.py#L51-L61

While in many datasets metrics they are the ground truth labels:
https://github.com/huggingface/datasets/blob/c3f53792a744ede18d748a1133b6597fdd2d8d18/metrics/accuracy/accuracy.py#L31-L40

Do you think predictions & references should be swapped? I'd be willing to do some refactoring here if you agree.

Also for the `Accuracy` metric the `accuracy_score` method should have its args in the opposite order so `accuracy_score(predictions, references,,,)`.

Thanks for pointing this out ! 🕵🏻 
Predictions and references should indeed be swapped in the docstring.
However, the call to `accuracy_score` should not be changed, it [signature](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score) being:
```
sklearn.metrics.accuracy_score(y_true, y_pred, *, normalize=True, sample_weight=None)
```

Feel free to open a PR if you want to fix this :)
Adding the HAREM dataset

Adding the HAREM dataset, a Portuguese language dataset for NER tasks

Thanks for the changes !

Sorry if I wasn't clear about the suggestion of adding the `raw` dataset as well.
By `raw` I meant the dataset with its original features, i.e. not tokenized to follow the conll format for NER.
The `raw` dataset has data fields `doc_text`, `doc_id` and `entities`.

Alright, @lhoestq, now I understand your suggestion, but the JSON files the script downloads aren't actually the "raw" HAREM dataset, the [real raw version](https://www.linguateca.pt/primeiroHAREM/harem/ColeccaoDouradaHAREM.zip) of it is on XML format that needs a lot of preprocessing. Those JSON are just pre-processed versions of the dataset.

I can make the config of the raw version of the pre-processed dataset, but I'll leave a comment on the dataset summary about those details.

Oh I see ! Sorry I thought there really were the raw (i.e. original/unprocessed) files

The original xml format doesn't seem very practical to use :/ 
We can ignore that and only keep the default and selective configs then.
Can you revert to the old configs ?

Thanks again for adding this dataset !

Hi @lhoestq, I've reverted the changes and put more details on README about the dataset.
Add wiki lingua dataset

Hello @lhoestq ,

I am opening a fresh pull request as advised in my original PR https://github.com/huggingface/datasets/pull/1308

Thanks

it’s failing because of `RemoteDatasetTest.test_load_dataset_orange_sum`
which i think is not the dataset you are doing a PR for. Try rebasing with:
```
git fetch upstream
git rebase upstream/master
git push -u -f origin your_branch
```

> it’s failing because of `RemoteDatasetTest.test_load_dataset_orange_sum`
> which i think is not the dataset you are doing a PR for. Try rebasing with:
> 
> ```
> git fetch upstream
> git rebase upstream/master
> git push -u -f origin your_branch
> ```

Thanks, my branch seems to be up to date. 
```Current branch add-wiki-lingua-dataset is up to date.```

Also where do the google drive urls come from ?

looks like this PR includes changes about many other files than the ones for wiki_lingua.

Can you create another branch and another PR ?
(or you can try to fix this branch with rebase and push force if you're familiar with it)

Thanks for fixing the dummy data and removing the glob call :) 

> looks like this PR includes changes about many other files than the ones for wiki_lingua.
> 
> Can you create another branch and another PR ?
> (or you can try to fix this branch with rebase and push force if you're familiar with it)

Easier to create a new branch and submit, I have submitted a new PR #1582 

Closing this one in favor of #1582
add Indonesian newspapers (id_newspapers_2018)

The dataset contains around 500K articles (136M of words) from 7 Indonesian newspapers. The size of uncompressed 500K json files (newspapers-json.tgz) is around 2.2GB.

Looks like there's a `Path` issue on windows. Could you try switching to
`glob.glob(os.path.join(article_dir, "*.json"))`

> Looks like there's a `Path` issue on windows. Could you try switching to
> `glob.glob(os.path.join(article_dir, "*.json"))`

Thanks, I replaced it with glob. Let's see if it solves the issue. Anyway, the main directory has a space, could it make the issue on windows? the test on linux don't have this problem.

It seems glob doesn't help also. Btw, one of the failing test tried to connect aws which failed:
```
C:\tools\miniconda3\lib\site-packages\urllib3\connection.py:160: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

address = ('s3.amazonaws.com', 443), timeout = 10, source_address = None
socket_options = [(6, 1, 1)]

```
Why did it try to connect to aws? I don't use it.

It seems that the circleci make a test for whole datasets repository, that means if only one of the dataset in the official repository has a download issue, this will also affect the test of a new dataset like mine, isn't it?
I changed the url to my newspaper dataset which contains only few simple json files and simple directory structure. But it still failed. And it failed not only on windows test. This is one of the error message:
```
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
===== 4 failed, 2667 passed, 2052 skipped, 4 warnings in 432.05s (0:07:12) =====

Exited with code exit status 1
CircleCI received exit code 1
```
The test failed on twitter dataset even my dataset has nothing to do with twitter? 

merging since the CI is fixed on master

Hi, thanks for merging the dataset. I create a new PR (#1499) since I need to update the link to the dataset.
adding snow_simplified_japanese_corpus

Adding simplified Japanese corpus "SNOW T15" and "SNOW T23".
They contain original Japanese, simplified Japanese, and original English (the original text is gotten from en-ja translation corpus). Hence, it can be used not only for Japanese simplification but also for en-ja translation.

- http://www.jnlp.org/SNOW/T15
- http://www.jnlp.org/SNOW/T23

merging since the CI is fixed on master

Thank you for the updates and merging!
Add Turkish News Category Dataset (270K).Updates were made for review…

This PR adds the **Turkish News Categories Dataset (270K)** dataset which is a text classification dataset by me and @yavuzKomecoglu. Turkish news dataset consisting of **273601 news in 17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.

**Note**: Resubmitted as a clean version of the previous Pull Request(#1419).  @SBrandeis @lhoestq 

@SBrandeis, What exactly is it that makes the tests fail? Can you help me please?

These errors
```
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
```
appeared on master 3 hours ago and are now fixed.
(it was due to today's update of `xlrd` that broke two datasets)

You can ignore them, they're not related to your dataset

> These errors
> 
> ```
> =========================== short test summary info ============================
> FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
> FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
> FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
> FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
> ```
> 
> appeared on master 3 hours ago and are now fixed.
> (it was due to today's update of `xlrd` that broke two datasets)
> 
> You can ignore them, they're not related to your dataset

**If it is not a problem caused by us, we have already completed the review notes. You can then check it out and confirm?**

merging since the CI is fixed on master
Reddit jokes

196k Reddit Jokes dataset
Dataset link- https://raw.githubusercontent.com/taivop/joke-dataset/master/reddit_jokes.json

@lhoestq would you please rerun the test, 

I re-started the test.

@lhoestq let's hold off on merging for now though, having a conversation on Slack about some of the offensive content in the dataset and how/whether we want to present it.
Added conv ai 2 (Again)

The original PR -> https://github.com/huggingface/datasets/pull/1383

Reason for creating again - 

The reason I had to create the PR again was due to the master rebasing issue. After rebasing the changes, all the previous commits got added to the branch. 

Looking perfect to me, need to rerun the tests


Thanks, @tanmoyio. 
How do I rerun the tests? Should I change something or push a new commit?

@rkc007 you don't need to rerun it, @lhoestq @yjernite will rerun it, as there are huge number of PRs in the queue it might take lil bit of time. 

ive just re-run the tests

Thank you @abhishekkrthakur. Can you please rerun it again? It seems something was broken in CI during the previous test.

@lhoestq Sorry for the mess. I don't know why this keeps on happening. I tried step by step process of updating the PR but seems something is wrong. This happened for 2nd time with the same PR. Apologies for that. 

New PR -> https://github.com/huggingface/datasets/pull/1527
Also, I fixed everything in the new PR.
add Bengali Hate Speech dataset


@lhoestq I think you might want to look at the dataset, and the first data instances mentioned in the README.md is very much offensive. Though this dataset is based on hate speech but I found the dataset heavily disturbing as Bengali is my native language.

Hi @tanmoyio indeed you're right.
We should *at least* add very explicit mentions in the dataset card that the content of this dataset contains very offensive language. We should also put it in perspective with the tasks it tries to solve, the annotation process and the limitations.

We have to make sure that nothing is unclear/misleading nor could lead to bad usage of the dataset.

What do you think @tanmoyio ?
Also feel free to suggest modifications in the dataset cards if you feel like some sections require corrections or more details

> Hi @tanmoyio indeed you're right.
> We should _at least_ add very explicit mentions in the dataset card that the content of this dataset contains very offensive language. We should also put it in perspective with the tasks it tries to solve, the annotation process and the limitations.
> 
> We have to make sure that nothing is unclear/misleading nor could lead to bad usage of the dataset.
> 
> What do you think @tanmoyio ?
> Also feel free to suggest modifications in the dataset cards if you feel like some sections require corrections or more details

yeah I agree with you. It would be good if "Personal and Sensitive Information" and "Considerations for Using the Data" is being explained properly in the README.md. @stevhliu 

please let me know if there is anything else you'd like to see!

This looks ok to merge for me. Let me know @stevhliu and @tanmoyio if you want to add something or if it looks good to you

looks good to me @lhoestq 👍 

merging since the CI is fixed on master
Add id_nergrit_corpus

Nergrit Corpus is a dataset collection of Indonesian Named Entity Recognition, Statement Extraction, and Sentiment Analysis. 
Recently my PR for id_nergrit_ner has been accepted and merged to the main branch. The id_nergrit_ner has only one dataset (NER), and this new PR renamed the dataset from id_nergrit_ner to id_nergrit_corpus and added 2 other remaining datasets (Statement Extraction, and Sentiment Analysis.)

merging since the CI is fixed on master
SNLI dataset contains labels with value -1

```
import datasets
nli_data = datasets.load_dataset("snli")
train_data = nli_data['train']
train_labels = train_data['label']
label_set = set(train_labels)
print(label_set)
```

**Output:**
`{0, 1, 2, -1}`

I believe the `-1` label is used for missing/NULL data as per HuggingFace Dataset conventions. If I recall correctly SNLI has some entries with no (gold) labels in the dataset.

Ah, you're right. The dataset has some pairs with missing labels. Thanks for reminding me.
add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC) [PROPER]

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


linter your code with flake8 and also run the commands present in Makefile for proper formatting 


merging since the CI is fixed on master
Update step-by-step guide for windows

Update step-by-step guide for windows to give an alternative to `make style`.

Hi @thomwolf, for simplification purposes, I think you could remove the "`pip install ...`" steps from this commit, 'cause these deps (black, isort, flake8) are already installed on `pip install -e ".[dev]"` on the [Start by preparing your environment](https://github.com/huggingface/datasets/blob/704107f924e74445f6f0fbde69a218b72178b588/ADD_NEW_DATASET.md#start-by-preparing-your-environment)
FileNotFound remotly, can't load a dataset

```py
!pip install datasets
import datasets as ds

corpus = ds.load_dataset('large_spanish_corpus')
```
gives the error

> FileNotFoundError: Couldn't find file locally at large_spanish_corpus/large_spanish_corpus.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/large_spanish_corpus/large_spanish_corpus.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/large_spanish_corpus/large_spanish_corpus.py

not just `large_spanish_corpus`,  `zest` too, but `squad` is available. 

this was using colab and localy 

This dataset will be available in version-2 of the library. If you want to use this dataset now, install datasets from `master` branch rather.

Command to install datasets from `master` branch:
`!pip install git+https://github.com/huggingface/datasets.git@master`

Closing this, thanks @VasudevGupta7
Adding english plaintext jokes dataset

This PR adds a dataset of 200k English plaintext Jokes from three sources: Reddit, Stupidstuff, and Wocka.
Link: https://github.com/taivop/joke-dataset 

This is my second PR. 
First was: [#1269 ](https://github.com/huggingface/datasets/pull/1269)

Hi @purvimisal, thanks for your contributions!

This jokes dataset has come up before, and after a conversation with the initial submitter, we decided not to add it then. Humor is important, but looking at the actual data points in this set raises several concerns :) 

The main issue is the Reddit part of the dataset which has most of the examples. A cursory look at the data shows a large number of highly offensive jokes that reproduce some pretty harmful biases (the second one from the top is a Holocaust joke). 

The other two sources have similar issues (especially the "Blond Jokes") to a slightly lesser extent.

While such datasets can be useful in the right context, there is a real concern that people using the library might miss some of this context (however much we outline it), and end up unwittingly training models that rely on some pretty racist and sexist associations.

We would recommend skipping this dataset altogether.

If you feel really strongly about having a joke dataset, then we would ask that you:
- remove the Reddit part of the dataset altogether
- write an in-depth description of the social biases present in the remaining data

Let us know which of the two you decide! And if you want recommendations on other datasets to add, hit us up on Slack 🤗 

Hi @yjernite, thanks so much. I should've totally thought about this earlier. The harmful biases make so much sense. I should've consulted before making a PR. 
I will be closing this one and skipping this dataset altogether. 
Thanks again
A descriptive name for my changes

hind encorp resubmited

I have noticed that the master branch of your fork has diverged from the one of the repo. This is probably what causes the mess in the github diff "Files changed".

I would suggest to re-fork the `datasets` repo and recreate a new branch and a new PR. 

You're pretty close to having all things ready to merge !
Feel free to ping me when you have a new PR

Closing this one in favor of #1575
Add Indosum dataset


Hi @prasastoadi have you had a chance to take a look at my suggestions ?

Feel free to ping ;e if you have questions or when you're ready for a review

Thanks for your contribution, @prasastoadi. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
add_sofc_materials_articles

adding [SOFC-Exp Corpus](https://arxiv.org/abs/2006.03039)

Hey @lhoestq , thanks for the feedback on this! I updated the `_generate_examples` with some comments on the process, and reduced the `dummy_data.zip` down quite a bit as well. 

For the dummy data, I reduced the text to only three sentences, and aligned the corresponding entity/token/sentence annotations to that (reduced accordingly). The frames file is a strange combined format for the annotations and I found if I reduced that that would break the parser no matter what I did, so I left that as is. The difference between a reduced frames and non-reduced frames file in the compressed dummy data was only about ~4kb, so hopefully leaving this as is will be ok!
Ar cov19

Adding ArCOV-19 dataset. ArCOV-19 is an Arabic COVID-19 Twitter dataset that covers the period from 27th of January till 30th of April 2020. ArCOV-19 is the first publicly-available Arabic Twitter dataset covering COVID-19 pandemic that includes over 1M tweets alongside the propagation networks of the most-popular subset of them (i.e., most-retweeted and-liked). The propagation networks include both retweets and conversational threads (i.e., threads of replies). ArCOV-19 is designed to enable research under several domains including natural language processing, information retrieval, and social computing, among others. 

merging since the CI is fixed on master
Add 1.5 billion words Arabic corpus 

Needs https://github.com/huggingface/datasets/pull/1429 to work. 

Can't pass dummy data tests. For the instructions, it asks me to generate the following file `dummy_data/Youm7_XML_utf_8.rar/Youm7_utf_8.xml` which is strange, any ideas @lhoestq ?

cc: I tested the data locally and it works, maybe the dummy tests doesn't support `rar` ? 

In the dummy_data.zip files you must include the rar file as if is was already extracted.
In particular here `Youm7_XML_utf_8.rar` is a directory (not an archive).

Also I'm getting `BadRarFile: Failed the read enough data: req=16384 got=51` while trying to download and extract the `Alittihad_XML_utf_8.rar` file. Do you have this issue as well ?

I have rarfile 4.0

Sorry it was my mistake, I missed up the directories, it works now. Not sure why you got that error. I have the same version of `rarfile`. Between, there were some suggestions to change the dataset from `1bn_words_arabic` to `arabic_billion_words` like https://github.com/huggingface/datasets/tree/master/datasets/spanish_billion_words. 


I'm ok with renaming the dataset `arabic_billion_words` if you want.
Note that you will need to rename class name `ArabicBillionWords` instead of `BillionWords`
(though `BillionWords` was not matching `1bn_words_arabic` anyway)

You will need to regenerate the dataset_infos.json file after this change.
OR alternatively just replace all mentions of `billion_words` with `arabic_billion_words` in dataset_infos.json <- this trick should save you some time :)

Hmmm I'm still not able to run it on my side because of the rar error (I'm running macos)
I just tried with rarfile 3.1 and it didn't work either.
I would like to be able to run it end-to-end on my side before merging if you don't mind. Let me investigate this issue a little bit

No worries, I will investigate it as well. 

I created a minimal example in [colab ](https://colab.research.google.com/drive/11ijesuGbrQylANka0VdsZ5vXwIuxkheY?usp=sharing).

Nice thanks, maybe it's just an issue on my side then

Ok I managed to solve the BadRarFile issue on my side :) 
To fix it I had to install the `unrar` tool for macos (though it seems it's not available with `brew install` anymore, I had to install it from elsewhere).
Add german common crawl dataset

Adding a subpart of the Common Crawl which was extracted with this repo https://github.com/facebookresearch/cc_net and additionally filtered for duplicates 

Hi @Phil1108 !
Have you had a chance to take a look at my suggestions ?
Feel free to ping me if you have questions or if you're ready for a review

Thanks again for adding this dataset, this one is very useful !

> 
> 
> Hi @Phil1108 !
> Have you had a chance to take a look at my suggestions ?
> Feel free to ping me if you have questions or if you're ready for a review
> 
> Thanks again for adding this dataset, this one is very useful !

Hello @lhoestq , 

okay so should we go on with only the 2 parts or wait until the hosting of all 200 GB is cleared at HF? 
If you want to merge it with the 2 parts only at first und upgrade later to all, sure I'll include the minor changes in the next few days and update it here

I think we can wait a bit to find a solution for the hosting and update the PR to include all the files when it's done.
In the meantime you can update the PR to take into account the suggestions regarding the dataset card, the parallel downloads and the use of eval.

A safer alternative to eval is `ast.literal_eval`, it should do the job

Thanks for your contribution, @Phil1108. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add Turkish News Category Dataset (270K)

This PR adds the Turkish News Categories Dataset (270K) dataset which is a text classification dataset by me and @yavuzKomecoglu.  Turkish news dataset consisting of **273601 news** in **17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.



@lhoestq, can you please review this PR?


@SBrandeis,
Sorry. All of the latest version came to my branch. You can find final version. 
Resubmitted as a clean final version of #1466
I have completed all the review comments.

Closing this as PR is now https://github.com/huggingface/datasets/pull/1466
Add Hate Speech and Offensive Language Detection dataset

Add [Hate Speech and Offensive Language Detection dataset](https://github.com/t-davidson/hate-speech-and-offensive-language) from [this paper](https://arxiv.org/abs/1703.04009).

@lhoestq done! The failing testes don't seem to be related, it seems to be a connection issue, if I understand it correctly.

@lhoestq done!

merging since the CI is fixed on master
Add OffComBR

Add [OffComBR](https://github.com/rogersdepelle/OffComBR) from [Offensive Comments in the Brazilian Web: a dataset and baseline results](https://sol.sbc.org.br/index.php/brasnam/article/view/3260/3222) paper.

But I'm having a hard time generating dummy data since the original dataset extion is `.arff` and the [_create_dummy_data function](https://github.com/huggingface/datasets/blob/a4aeaf911240057286a01bff1b1d75a89aedd57b/src/datasets/commands/dummy_data.py#L185) doesn't allow it.

Hello @hugoabonizio, thanks for the contribution.
Regarding the fake data, you can generate it manually.
Running the `python datasets-cli dummy_data datasets/offcombr` should give you instructions on how to manually create the dummy data.
For reference, here is a spec for `.arff` files : https://www.cs.waikato.ac.nz/ml/weka/arff.html

@lhoestq again the failing tests doesn't seem to be related

merging since the CI is fixed on master
Add Tweet Eval Dataset


Hi @lhoestq,

Seeing that it has been almost two months to this draft, I'm willing to take this forward if you and @abhishekkrthakur don't mind. :)

Hi @gchhablani !
Sure if @abhishekkrthakur doesn't mind
Thanks for your help :)

Please feel free :) 

Hi @lhoestq, @abhishekkrthakur 

I believe this can be closed. Merged in #1829.
Adding TaPaCo Dataset with README.md


We want to keep the repo as light as possible so that it doesn't take ages to clone, that's why we ask for small dummy data files (especially when there are many of them). Let me know if you have questions or if we can help you on this

Hello @lhoestq , made the changes as you suggested and pushed, please review. By default, the dummy data was generated the way it was by the dummy data auto generate command. Thank you.
adding covid-tweets-japanese (again)

I had mistaken use git rebase, I was so hurried to fix it. However, I didn't fully consider the use of git reset , so I unintendedly stopped PR (#1367) altogether. Sorry about that.
I'll make a new PR.

README.md is not created yet. I'll add it soon.

Thank you for your detailed code review! It's so helpful.
I'll reflect them to the code in 24 hours.

You may have told me in Slack (I cannot find the conversation log though I've looked through threads), but I'm sorry it seems I'm still misunderstanding how to get YAML from the tagger.
I'm now asking on Slack if I am looking at the tagger the wrong way.

One more thing I'd like to ask.
Should I make changes by myself, or can I use the "Commit suggestion" feature?
I'm new to this feature and I don't know how the rules work in this repository, so I'd like to ask just in case.

Thank you very much for merging!
Add reasoning_bg

Adding reading comprehension dataset for Bulgarian language

Hi @saradhix have you had the chance to reduce the size of the dummy data ?

Feel free to ping me when it's done so we can merge :) 

@lhoestq I have reduced the size of the dummy data manually and pushed the changes.

The CI errors are not related to your dataset.
They're fixed on master, you can ignore them

merging since the CI is fixed on master
add amazon polarity dataset

This corresponds to the amazon (binary dataset) requested in https://github.com/huggingface/datasets/issues/353

`amazon_polarity` is probably a subset of `amazon_us_reviews` but I am not entirely sure about that.
I guess `amazon_polarity` will help in reproducing results of papers using this dataset since even if it is a subset from `amazon_us_reviews`, it is not trivial how to extract `amazon_polarity` from `amazon_us_reviews`, especially since `amazon_us_reviews` was released after `amazon_polarity`. 

do you know what the problem would be ? should I pull the master before ? @lhoestq 

The error just appeared on master. I will try to fix it today.
You can ignore them since it's not related to the dataset you added

merging since the CI is fixed on master

Great thanks for the help.
Add twi text c3

Added Twi texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/

looks like this PR includes changes about other datasets

Can you only include the changes related to twi text c3 please ?

Hi @lhoestq , I have removed the unnecessary files. Can you please confirm?

You might need to either find a way to go back to the commit before it changes 389 files or create a new branch.

okay, I have created another branch, see the latest pull https://github.com/huggingface/datasets/pull/1518  @cstorm125 

Hii please follow me

Closing this one in favor of #1518
Add yoruba text c3

Added Yoruba texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/

looks like this PR includes changes about other datasets


Thanks for the review. I'm a bit confused how to remove the files. Every time I add a new branch name using the following commands:

git fetch upstream
git rebase upstream/master
git checkout -b a-descriptive-name-for-my-changes

and push to the origin, this issue occurs

Can you try to create the branch from the master branch of your fork ?

first update your master branch:
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
```

then create a new one:
```
git checkout -b my-new-branch
```

I think you were still having the files because you were creating the new branch from a branch in which you've committed the files, instead of creating the new branch from the master branch

Got it, will correct that. Thanks

@lhoestq , I have removed the unnecessary files. Looks like I still have one error. How do I resolve this?

> @lhoestq , I have removed the unnecessary files. Looks like I still have one error. How do I resolve this?

I think it's connection error on piqa dataset. Can you try triggering the test again? I usually resolve similar issues with:
```
git fetch upstream
git rebase upstream/master
git push -u -f origin your_branch_name
```

thank you  @cstorm125 

I have created another pull request for this https://github.com/huggingface/datasets/pull/1515 @cstorm125  @lhoestq 

Hii please follow me

merging since the CI is fixed on master

Great, thanks a lot
adding marathi-wiki dataset

Adding marathi-wiki-articles dataset. 

Can you make it a draft PR until you've added the dataset please ? @ekdnam 

Done

Thanks for your contribution, @ekdnam. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Re-adding narrativeqa dataset

An update of #309. 

@lhoestq I think I've fixed the dummy data - it finally passes! I'll add the model card now.

@lhoestq - pretty happy with it now

> Awesome thank you !
> 
> Could you try to reduce the size of the dummy_data.zip file before we merge ? (it's 300KB right now)
> 
> To do so feel free to take a look inside it and remove all the unnecessary files and chunks of text, to only keep a few examples. The idea is to have a zip file that is only a few KB

Ah, it only contains 1 example for each split. I think the problem is that I include an entire story (like in the full dataset). We can probably get away with a summarised version.

> Nice thank you, can you make it even lighter if possible ? Something round 10KB would be awesone
> We try to keep the repo light so that it doesn't take ages to clone. So we have to make sure the dummy data are as small as possible for every single dataset.

Have trimmed a little more out of each example now.
adding covid-tweets-japanese

Adding COVID-19 Japanese Tweets Dataset as part of the sprint.

Testing with dummy data is not working (the file is said to not exist). Sorry for the incomplete PR.

I think it's because the file you download uncompresses into a file and not a folder so `--autogenerate` couldn't create dummy data for you. See in your dummy_data.zip if there is a file there. If not, manually create your dummy data and compress them to dummy_data.zip.

@cstorm125 Thank you for the comment! 
As you point out, it seems my code has something wrong about downloading and uncompressing the file.
However, my manually created dummy data seems to contain a file of the required format.

On Colaboratory,
`!unzip /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip`
returns:

```
Archive:  /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip
   creating: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/
 extracting: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/data.csv.bz2  
```

The original data is `data.csv.bz2`, and I had a very hard time dealing with uncompressing bzip2.
I think I could handle it, but there may be problems remain.
Add Mkqa dataset

# MKQA: Multilingual Knowledge Questions & Answers Dataset
Adding the [MKQA](https://github.com/apple/ml-mkqa) dataset as part of the sprint 🎉

There is no official data splits so I added just a `train` split.
 
differently from the original:
- answer:type field is a ClassLabel (I thought it might be possible to train on this as a label for categorizing questions)
- answer:entity field has a default value of empty string '' (since this key is not available for all in original)
- answer:alias has default value of []

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


the `RemoteDatasetTest ` error pf the CI is fixed on master so it's fine

merging since the CI is fixed on master
Narrative QA (Manual Download Stories) Dataset

Narrative QA with manual download for stories. 

Hi ! Maybe we can rename it `narrativeqa_manual` to make it explicit that this one requires manual download contrary to `narrativeqa` ?
It's important to have this one as well, in case the `narrativeqa` one suffers from download issues (checksums or dead links for example).

You can also copy the dataset card from `narrativeqa` and add the dummy data as well

Thanks @lhoestq will do all this and submit a request in the coming days. 😊 

Closing this as another pull request is already done.
add LeNER-Br dataset

Adding the LeNER-Br dataset, a Portuguese language dataset for named entity recognition 

I don't know what happened, my first commit passed on all checks, but after just a README.md update one of the scripts failed, is it normal? 😕 

Looks like a flaky connection error, I've launched a re-run, it should be fine :)

The RemoteDatasetTest error in the CI is just a connection error, we can ignore it

merging since the CI is fixed on master
initial commit for MultiReQA 

Added MultiReQA, which is a dataset containing the sentence boundary annotation from eight publicly available QA datasets including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, BioASQ, RelationExtraction, and TextbookQA. 

looks like this dataset includes changes about many other files than the ones for multi_re_qa

Can you create another branch and another PR please ?

> looks like this dataset includes changes about many other files than the ones for multi_re_qa
> 
> Can you create another branch and another PR please ?

Sure I will do that. Thank you.
Add spanish billion words corpus

Add an unannotated Spanish corpus of nearly 1.5 billion words, compiled from different resources from the web.

Thank you for your feedback! I've reduced the dummy data size to 2KB.

I had to rebase to fix `RemoteDatasetTest` fails, sorry about the 80 commits. 
I could create a new clean PR if you prefer.

I have seen that in similar cases you have suggested to other contributors to create another branch and another PR, so I will do that.

Yes thank you !

The new PR is #1476 :)
hate_speech_18 initial commit


> Nice thanks !
> 
> Can you rename the dataset folder and the dataset script name `hate_speech18` instead of `hate_speech_18` to follow the snake case convention we're using ?
> 
> Also it looks like the dummy_data.zip file is quite big (almost 4MB).
> Can you try to reduce its size ?
> 
> To do so feel free to take a look inside it and remove all the unnecessary files or chunks of texts. The idea is to only keep a few examples

Done, thanks! 

Re-opened in https://github.com/huggingface/datasets/pull/1486
Add spanish billion words

Add an unannotated corpus of the Spanish language of nearly 1.5 billion words, compiled from different resources from the web.

The dataset needs 10 GB (download: 1.89 GiB, generated: 8.34 GiB, post-processed: Unknown size, total: 10.22 GiB), the test using dummy data pass but my laptop isn't able to run it on the real data (I left it running for over 8 hours and it didn't finish).

The tests failed because of ```RemoteDatasetTest``` so I tried ```git rebase``` and messed everything up. I've made a new clean PR (#1347).
added un_ga dataset

Hi :hugs:, This is a PR for [United nations general assembly resolutions: A six-language parallel corpus](http://opus.nlpl.eu/UN.php) dataset

Looks like this PR includes changes about many other files than the ones for un_ga

Can you create another branch an another PR please ?

@lhoestq, Thank you for suggestions. I have made the changes and raised the new PR https://github.com/huggingface/datasets/pull/1569.
❓ Sharing ElasticSearch indexed dataset 

Hi there,

First of all, thank you very much for this amazing library. Datasets have become my preferred data structure for basically everything I am currently doing.

**Question:** I'm working with a dataset and I have an elasticsearch container running at localhost:9200. I added an elasticsearch index and I was wondering

- how can I know where it has been saved?

- how can I share the indexed dataset with others?

I tried to dig into the docs, but could not find anything about that.

Thank you very much for your help.

Best,
Pietro

Edit: apologies for the wrong label

Hello @pietrolesci , I am not sure to understand what you are trying to do here.

If you're looking for ways to save a dataset on disk, you can you the `save_to_disk` method:
```python
>>> import datasets
>>> loaded_dataset = datasets.load("dataset_name")
>>> loaded_dataset.save_to_disk("/path/on/your/disk")
```

The saved dataset can later be retrieved using:
```python
>>> loaded_dataset = datasets.Dataset.load_from_disk("/path/on/your/disk")
```

Also, I'd recommend posting your question directly in the issue section of the [elasticsearch repo](https://github.com/elastic/elasticsearch)

Hi @SBrandeis,

Thanks a lot for picking up my request. 

Maybe I can clarify my use-case with a bit of context. Say I have the IMDb dataset. I create an ES index on it. Now I can save and reload the dataset from disk normally. Once I reload the dataset, it is easy to retrieve the ES index on my machine. I was wondering: is there a way I can share the (now) indexed version of the IMDb dataset with my colleagues without requiring them to re-index it?

Thanks a lot in advance for your consideration.

Best,

Pietro

Thanks for the clarification.

I am not familiar with ElasticSearch, but if I understand well you're trying to migrate your data along with the ES index.
My advice would be to check out ES documentation, for instance, this might help you: https://www.elastic.co/guide/en/cloud/current/ec-migrate-data.html

Let me know if it helps
Add CC-News dataset of English language articles

Adds [CC-News](https://commoncrawl.org/2016/10/news-dataset-available/) dataset. It contains 708241 English language news articles. Although each article has a language field these tags are not reliable. I've used Spacy language detection [pipeline](https://spacy.io/universe/project/spacy-langdetect) to confirm that the article language is indeed English. 

The prepared dataset is temporarily hosted on my private Google Storage [bucket](https://storage.googleapis.com/hf_datasets/cc_news.tar.gz). We can move it to HF storage and update this PR before merging. 

@vblagoje nice work, please add the README.md file and it would be ready

@lhoestq @tanmoyio @yjernite please have a look at the dataset card. Don't forget that the dataset is still hosted on my private gs bucket and should eventually be moved to the HF bucket

I will move the files soon and ping you when it's done and with the new URLs :) 

Hi !

I just moved the file to a HF bucket. It's available at https://storage.googleapis.com/huggingface-nlp/datasets/cc_news/cc_news.tar.gz

Sorry for the delay ^^'

@lhoestq no worries, updated PR with the new URL and rebased to master
added dutch_social

The Dutch social media tweets dataset. Which has a total of more than 210k tweets in dutch language. These tweets have been machine annotated with sentiment scores (`label` feature) and `industry` and `hisco_codes`

It can be used for sentiment analysis, multi-label classification and entity tagging

@lhoestq 
Updated the `dummy_data.zip `(<10kb)I had to reduce it to just a few samples. 
Train-Test-Dev  (20-5-5 samples) 

But the push also added changes from other PRs (probably because of a rebase!) So the files changed tab shows 466 files were changed!  


Thanks ! The dummy data are all good now :) 

Looks like this PR includes changes to many other files than the ones for dutch_social now.

Can you create another branch and another PR please ?

> 
> Can you create another branch and another PR please ?
@lhoestq 

I did a rebase. Now it doesn't include the other files. Does that help? 



Yes thanks !
adding wili-2018 language identification dataset


@lhoestq Not sure what happened, I just changed the py file but it is showing some TensorFlow error now.

You can ignore it.
It's caused by the Tensorflow update that happened 30min ago. They added breaking changes.
I'm working on a fix on the master branch right now


oh okay, btw I have made the required change for reading the CSV, I think it should be fine now, please take a look at it when you have some time.

merging since the CI is fixed on master
ethos first commit

Ethos passed all the tests except from this one: 
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>

with this error: 
E               OSError: Cannot find data file. 
E               Original error:
E               [Errno 2] No such file or directory: 

> Nice thanks !
> 
> I left a few comments
> 
> Also it looks like this PR includes changes about other files than the ones for ethos
> 
> Can you create another branch and another PR please ?

@lhoestq Should I close this PR? The new one is the: #1453

You can create another PR and close this one if you don't mind

> You can create another PR and close this one if you don't mind

Perfect! You should see the #1453 PR for the fixed version! Thanks
add 10k German News Article Dataset


You can just create another branch from master on your fork and create another PR:

first update your master branch
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
```

then create a new branch
```
git checkout -b my-new-branch-name
```

Then you can add, commit and push the gnad10 files and open a new PR

closing in favor of #1572
Add snips built in intents 2016 12

This PR proposes to add the Snips.ai built in intents dataset. The first configuration added is for the intent labels only, but the dataset includes entity slots that may in future be added as alternate configurations.

It is not clear how to automatically add the dummy data if the source data is a more complex json format. Should I manually take a fraction of the source data and include it as dummy data?


Added a fraction of the real data as dummy data.

merging since the CI is fixed on master
Add HateSpeech Corpus for Polish

This PR adds a HateSpeech Corpus for Polish, containing offensive language examples.

- **Homepage:** http://zil.ipipan.waw.pl/HateSpeech
- **Paper:** http://www.qualitativesociologyreview.org/PL/Volume38/PSJ_13_2_Troszynski_Wawer.pdf

@lhoestq Do you think using the ClassLabel is correct if we don't know the meaning of them?

Once we find out the meanings we can still add them to the dataset card

Feel free to ping me when the PR is ready for the final review