noamaanMulla-03/datasets-issues huggingface dataset:
Add Hate Speech and Offensive Language Detection dataset

Add [Hate Speech and Offensive Language Detection dataset](https://github.com/t-davidson/hate-speech-and-offensive-language) from [this paper](https://arxiv.org/abs/1703.04009).


@lhoestq done! The failing testes don't seem to be related, it seems to be a connection issue, if I understand it correctly.

@lhoestq done!

merging since the CI is fixed on master
Add OffComBR

Add [OffComBR](https://github.com/rogersdepelle/OffComBR) from [Offensive Comments in the Brazilian Web: a dataset and baseline results](https://sol.sbc.org.br/index.php/brasnam/article/view/3260/3222) paper.

But I'm having a hard time generating dummy data since the original dataset extion is `.arff` and the [_create_dummy_data function](https://github.com/huggingface/datasets/blob/a4aeaf911240057286a01bff1b1d75a89aedd57b/src/datasets/commands/dummy_data.py#L185) doesn't allow it.


Hello @hugoabonizio, thanks for the contribution.
Regarding the fake data, you can generate it manually.
Running the `python datasets-cli dummy_data datasets/offcombr` should give you instructions on how to manually create the dummy data.
For reference, here is a spec for `.arff` files : https://www.cs.waikato.ac.nz/ml/weka/arff.html

@lhoestq again the failing tests doesn't seem to be related

merging since the CI is fixed on master
Added Times of India News Headlines Dataset

Dataset name: Times of India News Headlines
link: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/DPQMQH


@lhoestq @abhishekkrthakur what happened here ?


@lhoestq everything alright here ?

@tanmoyio please have patience. @lhoestq has to look at 150+ PRs and it may take time. The PR looks good to me but we wait for his confirmation :) 🤗
adding wili-2018 language identification dataset



@lhoestq Not sure what happened, I just changed the py file but it is showing some TensorFlow error now.

You can ignore it.
It's caused by the Tensorflow update that happened 30min ago. They added breaking changes.
I'm working on a fix on the master branch right now


oh okay, btw I have made the required change for reading the CSV, I think it should be fine now, please take a look at it when you have some time.

merging since the CI is fixed on master
A descriptive name for my changes

hind encorp resubmited


I have noticed that the master branch of your fork has diverged from the one of the repo. This is probably what causes the mess in the github diff "Files changed".

I would suggest to re-fork the `datasets` repo and recreate a new branch and a new PR. 

You're pretty close to having all things ready to merge !
Feel free to ping me when you have a new PR

Closing this one in favor of #1575
Adding SQA dataset

As discussed in #880 

Seems like automatic dummy-data generation doesn't work if the builder is a `ArrowBasedBuilder`, do you think you could take a look @lhoestq ?


This dataset seems to have around 1000 configs. Therefore when creating the dummy data we end up with hundreds of MB of dummy data which we don't want to add in the repo.
Let's make this PR on hold for now and find a solution after the sprint of next week

Closing in favor of #1566
Add Hippocorpus Dataset



> Before we merge can you try to reduce the size of the dummy_data.zip file ?
> 
> To do so feel free to only keep a few lines of the csv files ans also remove unnecessary chunks of texts (for example keep only the first sentences of a story).

Hi @lhoestq, I have reduced the size of the dummy_data.zip file by making the necessary changes you had suggested. 

merging since the CI is fixed on master
Add Microsoft Research Sequential Question Answering (SQA) Dataset

For more information: https://msropendata.com/datasets/b25190ed-0f59-47b1-9211-5962858142c2


I proposed something a few weeks ago in #898 (un-merged) but I think that the way that @mattbui added the dataset in the present PR is smarter and simpler should replace my PR #898.

(Narrator voice: *And it was around that time that Thomas realized that the community was now a lot smarter than him and he should hand-over the library he had started with @lhoestq to the community and stop pretending he knew everything about it.*)
Add WMT20 MLQE 3 shared tasks

3 tasks for the WMT 20 MLQE shared tasks -> 3 different datasets

(I re-created #1137 because it was too messy).

Note that in L199 `task3.py`, I used `logging.warning` to print some missing data in the train set.


Thanks for the comments Quentin!
I integrated them

It should be ok now!
Sorry I wasn't attentive enough.
(tests are currently failing, I understand it's from other datasets)

merging since the CI is fixed on master
Add semeval 2020 task 11

Adding in propaganda detection task (task 11) from Sem Eval 2020


@SBrandeis : Thanks for the feedback! Just updated to use context manager for the `open`s and removed the placeholder text from the `README`!

Great, thanks @ZacharySBrown !
Failing tests seem to be unrelated to your changes, merging the current master branch into yours should fix them.
FileNotFound remotly, can't load a dataset

```py
!pip install datasets
import datasets as ds

corpus = ds.load_dataset('large_spanish_corpus')
```
gives the error

> FileNotFoundError: Couldn't find file locally at large_spanish_corpus/large_spanish_corpus.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/large_spanish_corpus/large_spanish_corpus.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/large_spanish_corpus/large_spanish_corpus.py

not just `large_spanish_corpus`,  `zest` too, but `squad` is available. 

this was using colab and localy 


This dataset will be available in version-2 of the library. If you want to use this dataset now, install datasets from `master` branch rather.

Command to install datasets from `master` branch:
`!pip install git+https://github.com/huggingface/datasets.git@master`

Closing this, thanks @VasudevGupta7
Added the dataset clickbait_news_bg

There was a problem with my [previous PR 1445](https://github.com/huggingface/datasets/pull/1445) after rebasing, so I'm copying the dataset code into a new branch and submitting a new PR.


Hi @tsvm  Great work! 
Since you have raised a clean PR could you close the earlier one -   #1445 ? 


> Hi @tsvm Great work!
> Since you have raised a clean PR could you close the earlier one - #1445 ?

Done.
can't load local dataset: pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries

Trying to load a local SQuAD-formatted dataset (from a JSON file, about 60MB):
```
dataset = nlp.load_dataset(path='json', data_files={nlp.Split.TRAIN: ["./path/to/file.json"]})
```
causes
```
Traceback (most recent call last):
  File "dataloader.py", line 9, in <module>
    ["./path/to/file.json"]})
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 483, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 719, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False):
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/datasets/json/88c1bc5c68489f7eda549ed05a5a738527c613b3e7a4ee3524d9d233353a949b/json.py", line 53, in _generate_tables
    file, read_options=self.config.pa_read_options, parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 191, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```

I haven't been able to find any reports of this specific pyarrow error here or elsewhere. 

Labels: name: dataset bug

I am able to reproduce this with the official SQuAD `train-v2.0.json` file downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/

I am facing this issue in transformers library 3.0.2 while reading a csv using datasets.
Is this fixed in latest version? 
I updated the latest version 4.0.1 but still getting this error. What could cause this error?
added dutch_social

The Dutch social media tweets dataset. Which has a total of more than 210k tweets in dutch language. These tweets have been machine annotated with sentiment scores (`label` feature) and `industry` and `hisco_codes`

It can be used for sentiment analysis, multi-label classification and entity tagging


@lhoestq 
Updated the `dummy_data.zip `(<10kb)I had to reduce it to just a few samples. 
Train-Test-Dev  (20-5-5 samples) 

But the push also added changes from other PRs (probably because of a rebase!) So the files changed tab shows 466 files were changed!  


Thanks ! The dummy data are all good now :) 

Looks like this PR includes changes to many other files than the ones for dutch_social now.

Can you create another branch and another PR please ?

> 
> Can you create another branch and another PR please ?
@lhoestq 

I did a rebase. Now it doesn't include the other files. Does that help? 



Yes thanks !
Added dataset Makhzan

Need help with the dummy data.


The only CI error comes from 
```
FAILED tests/test_file_utils.py::TempSeedTest::test_tensorflow
```

which is not related to your PR and is fixed on master.

You can ignore it

@lhoestq I've made the changes. Please review and merge. 

I have a similar PR https://github.com/huggingface/datasets/pull/1562 for another dataset. I'll incorporate your comment about sorting and reducing dummy dataset size there.

The CI raises an error `FAILED tests/test_file_utils.py::TempSeedTest::test_tensorflow` but it's not related to this dataset.
This issue is fixed on master

You did all the work ;) thanks

merging since the CI is fixed on master
Add wiki lingua dataset

Hello @lhoestq ,

I am opening a fresh pull request as advised in my original PR https://github.com/huggingface/datasets/pull/1308

Thanks


it’s failing because of `RemoteDatasetTest.test_load_dataset_orange_sum`
which i think is not the dataset you are doing a PR for. Try rebasing with:
```
git fetch upstream
git rebase upstream/master
git push -u -f origin your_branch
```

> it’s failing because of `RemoteDatasetTest.test_load_dataset_orange_sum`
> which i think is not the dataset you are doing a PR for. Try rebasing with:
> 
> ```
> git fetch upstream
> git rebase upstream/master
> git push -u -f origin your_branch
> ```

Thanks, my branch seems to be up to date. 
```Current branch add-wiki-lingua-dataset is up to date.```

Also where do the google drive urls come from ?

looks like this PR includes changes about many other files than the ones for wiki_lingua.

Can you create another branch and another PR ?
(or you can try to fix this branch with rebase and push force if you're familiar with it)

Thanks for fixing the dummy data and removing the glob call :) 

> looks like this PR includes changes about many other files than the ones for wiki_lingua.
> 
> Can you create another branch and another PR ?
> (or you can try to fix this branch with rebase and push force if you're familiar with it)

Easier to create a new branch and submit, I have submitted a new PR #1582 

Closing this one in favor of #1582
FileNotFoundError for `amazon_polarity`

Version: `datasets==v1.1.3`

### Reproduction
```python
from datasets import load_dataset
data = load_dataset("amazon_polarity")
```
crashes with
```bash
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/amazon_polarity/amazon_polarity.py
```
and 
```bash
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/amazon_polarity/amazon_polarity.py
```
and
```bash
FileNotFoundError: Couldn't find file locally at amazon_polarity/amazon_polarity.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/amazon_polarity/amazon_polarity.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/amazon_polarity/amazon_polarity.py
```


Hi @phtephanx , the `amazon_polarity` dataset has not been released yet. It will be available in the coming soon v2of `datasets` :) 

You can still access it now if you want, but you will need to install datasets via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`
can't load "german_legal_entity_recognition" dataset

FileNotFoundError: Couldn't find file locally at german_legal_entity_recognition/german_legal_entity_recognition.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py



Please if you could tell me more about the error? 

1. Please check the directory you've been working on
2. Check for any typos

> Please if you could tell me more about the error?
> 
> 1. Please check the directory you've been working on
> 2. Check for any typos

Error happens during the execution of this line:
dataset = load_dataset("german_legal_entity_recognition")

Also, when I try to open mentioned links via Opera I have errors "404: Not Found" and "This XML file does not appear to have any style information associated with it. The document tree is shown below." respectively.

Hello @nataly-obr, the `german_legal_entity_recognition` dataset has not yet been released (it is part of the coming soon v2 release).

You can still access it now if you want, but you will need to install `datasets` via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`

Please let me know if it solves the issue :)
Add  KorQuAD v2 Dataset

# The Korean Question Answering Dataset v2
Adding the [KorQuAD](https://korquad.github.io/) v2 dataset as part of the sprint 🎉 
This dataset is very similar to SQuAD and is an extension of [squad_kor_v1](https://github.com/huggingface/datasets/pull/1178) which is why I added it as `squad_kor_v2`. 

- Crowd generated questions and answer (1-answer per question) for Wikipedia articles. Differently from V1 it includes the html structure and markup, which makes it a different enough dataset. (doesn't share ids between v1 and v2 either)

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

Edit: 🤦 looks like squad_kor_v1 commit sneaked in here too


looks like this PR also includes the changes for the V1
Could you only include the files of the V2 ?

hmm I have made the dummy data lighter retested on local and it passed not sure why it fails here?

merging since the CI is fixed on master
Add Adversarial SQuAD dataset

# Adversarial SQuAD

Adding the Adversarial [SQuAD](https://github.com/robinjia/adversarial-squad) dataset as part of the sprint 🎉 
This dataset adds adversarial sentences to a subset of the SQuAD dataset's dev examples. How to get the original squad example id is explained in readme->Data Instances. The whole data is intended for use in evaluation. (Which could of course be also used for training if one wants). So there is no classical train/val/test split, but a split based on the number of adversaries added.

There are 2 splits of this dataset:

- AddSent: Has up to five candidate adversarial sentences that don't answer the question, but have a lot of words in common with the question. This adversary is does not query the model in any way.
- AddOneSent: Similar to AddSent, but just one candidate sentences was picked at random. This adversary is does not query the model in any way.

(The AddAny and AddCommon datasets mentioned in the paper are dynamically generated based on model's output distribution thus are not included here)

The failing test look like some unrelated timeout thing, will probably clear if rerun.
- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


the CI error was just a connection error due to all the activity on the repo this week ^^'
I re-ran it so it should be good now

I hadn't realized the problem with the dummies since it had passed without errors.
Suggestion: maybe we can show the user a warning based on the generated dummy size.

Thanks for changing to configs ! Looks all good now :) 

Before we merge, can you re-lighten the dummy data please if you don't mind ? The idea is to have them weigh only a few KB (currently it's 50KB each). Feel free to remove any unnecessary files or chunk of text

(also you can ignore the `RemoteDatasetTest ` CI errors, they're fixed on master )

merging since the CI is fixed on master
Turkish NER dataset, script works fine, couldn't generate dummy data

I've written the script (Turkish_NER.py) that includes dataset. The dataset is a zip inside another zip, and it's extracted as .DUMP file. However, after preprocessing I only get .arrow file. After I ran the script with no error messages, I get .arrow file of dataset, LICENSE and dataset_info.json. 


the .DUMP file looks like a txt with one example per line so adding `--match_text_files *.DUMP --n_lines 50`  to the dummy generation command might work .

We can close this PR since a new PR was open at #1268
Add HateSpeech Corpus for Polish

This PR adds a HateSpeech Corpus for Polish, containing offensive language examples.

- **Homepage:** http://zil.ipipan.waw.pl/HateSpeech
- **Paper:** http://www.qualitativesociologyreview.org/PL/Volume38/PSJ_13_2_Troszynski_Wawer.pdf


@lhoestq Do you think using the ClassLabel is correct if we don't know the meaning of them?

Once we find out the meanings we can still add them to the dataset card

Feel free to ping me when the PR is ready for the final review
Add SelQA Dataset

Add the SelQA Dataset, a new benchmark for selection-based question answering tasks
Repo: https://github.com/emorynlp/selqa/
Paper: https://arxiv.org/pdf/1606.08513.pdf


Hii please follow me

The CI error `FAILED tests/test_file_utils.py::TempSeedTest::test_tensorflow` is not related with this dataset and is fixed on master. You can ignore it

merging since the Ci is fixed on master
Create XML dummy data without loading all dataset in memory

While I was adding one XML dataset, I noticed that all the dataset was loaded in memory during the dummy data generation process (using nearly all my laptop RAM).

Looking at the code, I have found that the origin is the use of `ET.parse()`. This method loads **all the file content in memory**.

In order to fix this, I have refactorized the code and use `ET.iterparse()` instead, which **parses the file content incrementally**.

I have also implemented a test.
add_sofc_materials_articles

adding [SOFC-Exp Corpus](https://arxiv.org/abs/2006.03039)


Hey @lhoestq , thanks for the feedback on this! I updated the `_generate_examples` with some comments on the process, and reduced the `dummy_data.zip` down quite a bit as well. 

For the dummy data, I reduced the text to only three sentences, and aligned the corresponding entity/token/sentence annotations to that (reduced accordingly). The frames file is a strange combined format for the annotations and I found if I reduced that that would break the parser no matter what I did, so I left that as is. The difference between a reduced frames and non-reduced frames file in the compressed dummy data was only about ~4kb, so hopefully leaving this as is will be ok!
Add id_nergrit_corpus

Nergrit Corpus is a dataset collection of Indonesian Named Entity Recognition, Statement Extraction, and Sentiment Analysis. 
Recently my PR for id_nergrit_ner has been accepted and merged to the main branch. The id_nergrit_ner has only one dataset (NER), and this new PR renamed the dataset from id_nergrit_ner to id_nergrit_corpus and added 2 other remaining datasets (Statement Extraction, and Sentiment Analysis.)


merging since the CI is fixed on master
adding snow_simplified_japanese_corpus

Adding simplified Japanese corpus "SNOW T15" and "SNOW T23".
They contain original Japanese, simplified Japanese, and original English (the original text is gotten from en-ja translation corpus). Hence, it can be used not only for Japanese simplification but also for en-ja translation.

- http://www.jnlp.org/SNOW/T15
- http://www.jnlp.org/SNOW/T23


merging since the CI is fixed on master

Thank you for the updates and merging!
Add nq_open question answering dataset

Added nq_open Open-domain question answering dataset.

The NQ-Open task is currently being used to evaluate submissions to the EfficientQA competition, which is part of the NeurIPS 2020 competition track.


@SBrandeis  thanks for the review, I applied your suggested changes, but CI is failing now not sure about the error.

Many thanks @Nilanshrajput  !
The failing tests on CI are not related to your changes, merging master on your branch should fix them :)
If you're interested in what causes the CI to fail, checkout [this commit](https://github.com/huggingface/datasets/commit/9a0f1e20ca1e783cb14c1ab1cc2f54b0b5b201e8)

@SBrandeis done!


Hello @Nilanshrajput, your PR includes changes from other branches too now (485 files changed)
Would you mind creating another branch from master with your changes and opening a new PR?

@SBrandeis sorry i messed up the git history, #1587 I opened this new pr! 

closing in favor of #1587
Add reasoning_bg

Adding reading comprehension dataset for Bulgarian language


Hi @saradhix have you had the chance to reduce the size of the dummy data ?

Feel free to ping me when it's done so we can merge :) 

@lhoestq I have reduced the size of the dummy data manually and pushed the changes.

The CI errors are not related to your dataset.
They're fixed on master, you can ignore them

merging since the CI is fixed on master
Add eHealth Knowledge Discovery dataset

This Spanish dataset can be used to mine knowledge from unstructured health texts. 

In particular, for:
- Entity recognition
- Relation extraction



Thank you very much for your review @lewtun ! 

I've updated the script metadata, created the README and fixed the two details you commented.

Ready for another review! 🤗 

I've updated the task tag as we discussed and also added a couple of lines of code to make sure I include all the available examples.

Thank you again!
IWSLT-17 Link Broken

```
FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz
```

Labels: name: duplicate

name: dataset bug

Sorry, this is a duplicate of #1287. Not sure why it didn't come up when I searched `iwslt` in the issues list.

Closing this since its a duplicate
Opus CAPES added

Dataset : http://opus.nlpl.eu/CAPES.php


@lhoestq I saw some common changes you made on the other PR's (Similar Opus Datasets). I fixed those changes here. Can you please review it once ? 
Thanks.

Hi @rkc007 , thanks for the contribution.
Unfortunately, the CAPES dataset has already been added here: #1307
I'm closing the PR 

@lhoestq FYI