Adding NewsQA dataset

> Generate the dummy dataset then regenerate the dataset_info.json file,

The pytest scripts do not accept manual directory inputs for the data provided manually. This is why the tests fail.

don't use the --auto-generate argument and you will get a brief instructions on how to create dummy data for your dataset,
also you dont have to run the pytest for main dataset if your data is needed to be downloaded manually, just run the pytest for dummy dataset, and when you will create the json you need to provide the main data directory path by using this argument --data_dir

Thanks for your help, @tanmoyio
I tried with and without --auto_generate flag.
Here are the issues.

**With --auto_generate**
`python datasets-cli dummy_data datasets/newsqa/ --auto_generate `
`Traceback (most recent call last):
  File "datasets-cli", line 36, in <module>
    service.run()
  File "/Users/sanjaykamath/Python_Projects/HuggingFace/datasets/src/datasets/commands/dummy_data.py", line 321, in run
    keep_uncompressed=self._keep_uncompressed,
  File "/Users/sanjaykamath/Python_Projects/HuggingFace/datasets/src/datasets/commands/dummy_data.py", line 340, in _autogenerate_dummy_data
    dataset_builder._split_generators(dl_manager)
  File "/Users/sanjaykamath/.cache/huggingface/modules/datasets_modules/datasets/newsqa/7a565b204506c1fd91047290073be54d3ae05fa2b0ab17ae0bc6f709350fcbca/newsqa.py", line 180, in _split_generators
    path_to_manual_folder = os.path.abspath(os.path.expanduser(dl_manager.manual_dir))
  File "/Users/sanjaykamath/anaconda3/envs/huggingface/lib/python3.7/posixpath.py", line 235, in expanduser
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
`

**Without --auto_generate**
`python datasets-cli dummy_data datasets/newsqa/`
`Dataset  with config BuilderConfig(name='combined-csv', version=1.0.0, data_dir=None, data_files=None, description='This part of the dataset covers the whole dataset in the combined format of CSV as mentioned here: https://github.com/Maluuba/newsqa#csv') seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file None.
Traceback (most recent call last):
  File "datasets-cli", line 36, in <module>
    service.run()
  File "/Users/sanjaykamath/Python_Projects/HuggingFace/datasets/src/datasets/commands/dummy_data.py", line 326, in run
    dataset_builder=dataset_builder, mock_dl_manager=mock_dl_manager
  File "/Users/sanjaykamath/Python_Projects/HuggingFace/datasets/src/datasets/commands/dummy_data.py", line 406, in _print_dummy_data_instructions
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
`

Excellent comments. Thanks @lhoestq for your valuable comments.
I've changed everything you had mentioned and the tests pass now.
Let me know if something still needs to be changed.

Thank you very much @lhoestq @tanmoyio @yjernite @thomwolf for all your support :)

Since the dataset has legal restrictions to circulate the original data. It has to be manually downloaded by the user and loaded to the library.


add Bengali Hate Speech dataset

@lhoestq I think you might want to look at the dataset, and the first data instances mentioned in the README.md is very much offensive. Though this dataset is based on hate speech but I found the dataset heavily disturbing as Bengali is my native language.

Hi @tanmoyio indeed you're right.
We should *at least* add very explicit mentions in the dataset card that the content of this dataset contains very offensive language. We should also put it in perspective with the tasks it tries to solve, the annotation process and the limitations.

We have to make sure that nothing is unclear/misleading nor could lead to bad usage of the dataset.

What do you think @tanmoyio ?
Also feel free to suggest modifications in the dataset cards if you feel like some sections require corrections or more details

> Hi @tanmoyio indeed you're right.
> We should _at least_ add very explicit mentions in the dataset card that the content of this dataset contains very offensive language. We should also put it in perspective with the tasks it tries to solve, the annotation process and the limitations.
>
> We have to make sure that nothing is unclear/misleading nor could lead to bad usage of the dataset.
>
> What do you think @tanmoyio ?
> Also feel free to suggest modifications in the dataset cards if you feel like some sections require corrections or more details

yeah I agree with you. It would be good if "Personal and Sensitive Information" and "Considerations for Using the Data" is being explained properly in the README.md. @stevhliu

please let me know if there is anything else you'd like to see!

This looks ok to merge for me. Let me know @stevhliu and @tanmoyio if you want to add something or if it looks good to you


Add Google Conceptual Captions Dataset (manual download)


Add id_nergrit_corpus

Nergrit Corpus is a dataset collection of Indonesian Named Entity Recognition, Statement Extraction, and Sentiment Analysis.
Recently my PR for id_nergrit_ner has been accepted and merged to the main branch. The id_nergrit_ner has only one dataset (NER), and this new PR renamed the dataset from id_nergrit_ner to id_nergrit_corpus and added 2 other remaining datasets (Statement Extraction, and Sentiment Analysis.)


add hrenwac_para


Add CC100 Dataset

Closes #773


Add HEAD-QA: A Healthcare Dataset for Complex Reasoning

Thank you for your review @lhoestq, I've changed the types of `qid` and `ra` and now they are integers as `aid`.

Ready for another review!

HEAD-QA is a multi-choice HEAlthcare Dataset, the questions come from exams to access a specialized position in the
Spanish healthcare system.


Add kinnews_kirnews

Add kinnews and kirnews


Adding ethos dataset clean

> Thanks !

Thanks as well for your hard work 😊!!

I addressed the comments on the PR1318


SNLI dataset contains labels with value -1

I believe the `-1` label is used for missing/NULL data as per HuggingFace Dataset conventions. If I recall correctly SNLI has some entries with no (gold) labels in the dataset.

Ah, you're right. The dataset has some pairs with missing labels. Thanks for reminding me.

```
import datasets
nli_data = datasets.load_dataset("snli")
train_data = nli_data['train']
train_labels = train_data['label']
label_set = set(train_labels)
print(label_set)
```

**Output:**
`{0, 1, 2, -1}`


Add European Center for Disease Control and Preventions's (ECDC) Translation Memory dataset

ECDC-TM homepage: https://ec.europa.eu/jrc/en/language-technologies/ecdc-translation-memory


Fix version in bible_para


add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC) [PROPER]

linter your code with flake8 and also run the commands present in Makefile for proper formatting

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


add thai_toxicity_tweet

Thai Toxicity Tweet Corpus contains 3,300 tweets (506 tweets with texts missing) annotated by humans with guidelines including a 44-word dictionary. The author obtained 2,027 and 1,273 toxic and non-toxic tweets, respectively; these were labeled by three annotators. The result of corpus analysis indicates that tweets that include toxic words are not always toxic. Further, it is more likely that a tweet is toxic, if it contains toxic words indicating their original meaning. Moreover, disagreements in annotation are primarily because of sarcasm, unclear existing target, and word sense ambiguity.

Notes from data cleaner: The data is included into [huggingface/datasets](https://www.github.com/huggingface/datasets) in Dec 2020. By this time, 506 of the tweets are not available publicly anymore. We denote these by `TWEET_NOT_FOUND` in `tweet_text`.
Processing can be found at [this PR](https://github.com/tmu-nlp/ThaiToxicityTweetCorpus/pull/1).


Update step-by-step guide for windows

Hi @thomwolf, for simplification purposes, I think you could remove the "`pip install ...`" steps from this commit, 'cause these deps (black, isort, flake8) are already installed on `pip install -e ".[dev]"` on the [Start by preparing your environment](https://github.com/huggingface/datasets/blob/704107f924e74445f6f0fbde69a218b72178b588/ADD_NEW_DATASET.md#start-by-preparing-your-environment)

Update step-by-step guide for windows to give an alternative to `make style`.


Add Bing Coronavirus Query Set


Added dataset clickbait_news_bg

Looks like this PR includes changes about many other files than the ones for clickbait_news_bg

Can you create another branch and another PR please ?

I created a new branch with the dataset code and submitted a new PR for it: https://github.com/huggingface/datasets/pull/1568


FileNotFound remotly, can't load a dataset

This dataset will be available in version-2 of the library. If you want to use this dataset now, install datasets from `master` branch rather.

Command to install datasets from `master` branch:
`!pip install git+https://github.com/huggingface/datasets.git@master`

```py
!pip install datasets
import datasets as ds

corpus = ds.load_dataset('large_spanish_corpus')
```
gives the error

> FileNotFoundError: Couldn't find file locally at large_spanish_corpus/large_spanish_corpus.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/large_spanish_corpus/large_spanish_corpus.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/large_spanish_corpus/large_spanish_corpus.py

not just `large_spanish_corpus`,  `zest` too, but `squad` is available.

this was using colab and localy


Add OPUS Wikimedia Translations Dataset


Create XML dummy data without loading all dataset in memory

While I was adding one XML dataset, I noticed that all the dataset was loaded in memory during the dummy data generation process (using nearly all my laptop RAM).

Looking at the code, I have found that the origin is the use of `ET.parse()`. This method loads **all the file content in memory**.

In order to fix this, I have refactorized the code and use `ET.iterparse()` instead, which **parses the file content incrementally**.

I have also implemented a test.


Add Igbo-English Machine Translation Dataset


Adding english plaintext jokes dataset

Hi @purvimisal, thanks for your contributions!

This jokes dataset has come up before, and after a conversation with the initial submitter, we decided not to add it then. Humor is important, but looking at the actual data points in this set raises several concerns :)

The main issue is the Reddit part of the dataset which has most of the examples. A cursory look at the data shows a large number of highly offensive jokes that reproduce some pretty harmful biases (the second one from the top is a Holocaust joke).

The other two sources have similar issues (especially the "Blond Jokes") to a slightly lesser extent.

While such datasets can be useful in the right context, there is a real concern that people using the library might miss some of this context (however much we outline it), and end up unwittingly training models that rely on some pretty racist and sexist associations.

We would recommend skipping this dataset altogether.

If you feel really strongly about having a joke dataset, then we would ask that you:
- remove the Reddit part of the dataset altogether
- write an in-depth description of the social biases present in the remaining data

Let us know which of the two you decide! And if you want recommendations on other datasets to add, hit us up on Slack 🤗

Hi @yjernite, thanks so much. I should've totally thought about this earlier. The harmful biases make so much sense. I should've consulted before making a PR.
I will be closing this one and skipping this dataset altogether.
Thanks again

This PR adds a dataset of 200k English plaintext Jokes from three sources: Reddit, Stupidstuff, and Wocka.
Link: https://github.com/taivop/joke-dataset

This is my second PR.
First was: [#1269 ](https://github.com/huggingface/datasets/pull/1269)


Update README.md

1k-10k -> 1k-1M

3 separate configs are available with min. 1K and max. 211.3k examples


A descriptive name for my changes

I have noticed that the master branch of your fork has diverged from the one of the repo. This is probably what causes the mess in the github diff "Files changed".

I would suggest to re-fork the `datasets` repo and recreate a new branch and a new PR.

You're pretty close to having all things ready to merge !
Feel free to ping me when you have a new PR

hind encorp resubmited


Add Indosum dataset

Hi @prasastoadi have you had a chance to take a look at my suggestions ?

Feel free to ping ;e if you have questions or when you're ready for a review


add ALT

The errors in de CI are fixed on master so it's fine

ALT dataset -- https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/


Add FreebaseQA dataset

@yjernite @lhoestq Any suggestions on how to get the dummy data generator to recognize the columns? The structure of the json is:
```
{
  "Dataset": "FreebaseQA-eval",
  "Version": "1.0",
  "Questions": [
    {
      "Question-ID": "FreebaseQA-eval-0",
      "RawQuestion": "Who is the female presenter of the Channel 4 quiz show '1001 things you should know'?",
      "ProcessedQuestion": "who is the female presenter of the channel 4 quiz show '1001 things you should know'",
      "Parses": [
        {
          "Parse-Id": "FreebaseQA-eval-0.P0",
          "PotentialTopicEntityMention": "1001 things you should know",
          "TopicEntityName": "1001 things you should know",
          "TopicEntityMid": "m.0nd3t34",
          "InferentialChain": "tv.tv_program.regular_personal_appearances..tv.tv_regular_personal_appearance.person",
          "Answers": [
            {
              "AnswersMid": "m.0216y_",
              "AnswersName": [
                "sandi toksvig"
              ]
            }
          ]
        }
      ]
    },
   ...
  ]
}
```

Thanks!

Unfortunately this json structure is not recognized by the auto-generation yet, so you'd have to create the dummy data manually.
You can get some instructions on how to do that with: `python datasets-cli dummy_data datasets/freebase_qa`
We can definitely help you with that if there are too many files!

@yjernite Thanks for the instructions. I manually added dummy data and created the zip file but one of the splits seem to return an empty list.

```
tests/test_dataset_common.py F                        [100%]

========================= FAILURES ==========================
_ LocalDatasetTest.test_load_dataset_all_configs_freebase_qa _

self = <tests.test_dataset_common.LocalDatasetTest testMethod=test_load_dataset_all_configs_freebase_qa>
dataset_name = 'freebase_qa'

    @slow
    def test_load_dataset_all_configs(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True)

tests/test_dataset_common.py:237:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:198: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
```

Note that the dataset has `train`, `eval`, and `dev` (no test split). I am not sure if I am mapping them correctly when I called the Split Generator.

The dummy json files must follow the exact same structure as the original json files.

However it looks like the dummy json files you have in your dummy_data.zip file are not structured the same way.
For example the original json is a dict with a field "Questions" that is a list of items.
However your dummy json is simply a list of items.

Can you update your dummy json files to follow the same structure ?

And I'm pretty sure that this structure is supported by the dummy data auto-generation tool
```
python datasets-cli dummy_data ./datasets/freebase_qa --json_field "Questions"
```

Hi @anaerobeth did you manage to get the dummy data right ?

Feel free to ping me if you have questions or when you're ready for a review

Thanks for your help! I am able to create the dummy data with the dict structure as suggested. I'll add the tags and update this PR shortly.

Also don't forget to run `make style` to fix the code formatting check in the CI :)

Hi @anaerobeth ! Have you had a chance to consider updating the dataset script to yield one example per question ?

Feel free to ping me if you have questions or if I can help :)

Hi @lhoestq,

I am willing to take this forward if you and @anaerobeth don't mind.

Hi @gchhablani thanks for proposing your help :)
Sure if you want to take this forward feel free to do so.
Also pinging @anaerobeth to make sure that you both don't work on the same thing at the same time

Hi ! Closing this one since the dataset was added in #1814

Thanks you two @anaerobeth and @gchhablani for adding this dataset !

This PR adds the FreebaseQA dataset: A Trivia-type QA Data Set over the Freebase Knowledge Graph

Repo: https://github.com/kelvin-jiang/FreebaseQA

Paper: https://www.aclweb.org/anthology/N19-1028.pdf

## TODO: create dummy data

Error encountered when running `python datasets-cli dummy_data datasets/freebase_qa --auto_generate`
```
    f"Couldn't parse columns {list(json_data.keys())}. "
ValueError: Couldn't parse columns ['Dataset', 'Version', 'Questions']. Maybe specify which json field must be used to read the data with --json_field <my_field>.
```


add_sofc_materials_articles

Hey @lhoestq , thanks for the feedback on this! I updated the `_generate_examples` with some comments on the process, and reduced the `dummy_data.zip` down quite a bit as well.

For the dummy data, I reduced the text to only three sentences, and aligned the corresponding entity/token/sentence annotations to that (reduced accordingly). The frames file is a strange combined format for the annotations and I found if I reduced that that would break the parser no matter what I did, so I left that as is. The difference between a reduced frames and non-reduced frames file in the compressed dummy data was only about ~4kb, so hopefully leaving this as is will be ok!

adding [SOFC-Exp Corpus](https://arxiv.org/abs/2006.03039)


Adding the ASSIN 2 dataset

Adding the ASSIN 2 dataset, a Portuguese language dataset for Natural Language Inference and Semantic Similarity Scoring


Adding journalists questions dataset

@lhoestq Thanks a lot for checking! I hope I addressed all your comments.

This is my first dataset to be added to HF.


Ar cov19

Adding ArCOV-19 dataset. ArCOV-19 is an Arabic COVID-19 Twitter dataset that covers the period from 27th of January till 30th of April 2020. ArCOV-19 is the first publicly-available Arabic Twitter dataset covering COVID-19 pandemic that includes over 1M tweets alongside the propagation networks of the most-popular subset of them (i.e., most-retweeted and-liked). The propagation networks include both retweets and conversational threads (i.e., threads of replies). ArCOV-19 is designed to enable research under several domains including natural language processing, information retrieval, and social computing, among others.


Add 1.5 billion words Arabic corpus

Can't pass dummy data tests. For the instructions, it asks me to generate the following file `dummy_data/Youm7_XML_utf_8.rar/Youm7_utf_8.xml` which is strange, any ideas @lhoestq ?

cc: I tested the data locally and it works, maybe the dummy tests doesn't support `rar` ?

In the dummy_data.zip files you must include the rar file as if is was already extracted.
In particular here `Youm7_XML_utf_8.rar` is a directory (not an archive).

Also I'm getting `BadRarFile: Failed the read enough data: req=16384 got=51` while trying to download and extract the `Alittihad_XML_utf_8.rar` file. Do you have this issue as well ?

I have rarfile 4.0

Sorry it was my mistake, I missed up the directories, it works now. Not sure why you got that error. I have the same version of `rarfile`. Between, there were some suggestions to change the dataset from `1bn_words_arabic` to `arabic_billion_words` like https://github.com/huggingface/datasets/tree/master/datasets/spanish_billion_words.

I'm ok with renaming the dataset `arabic_billion_words` if you want.
Note that you will need to rename class name `ArabicBillionWords` instead of `BillionWords`
(though `BillionWords` was not matching `1bn_words_arabic` anyway)

You will need to regenerate the dataset_infos.json file after this change.
OR alternatively just replace all mentions of `billion_words` with `arabic_billion_words` in dataset_infos.json <- this trick should save you some time :)

Hmmm I'm still not able to run it on my side because of the rar error (I'm running macos)
I just tried with rarfile 3.1 and it didn't work either.
I would like to be able to run it end-to-end on my side before merging if you don't mind. Let me investigate this issue a little bit

No worries, I will investigate it as well.

I created a minimal example in [colab ](https://colab.research.google.com/drive/11ijesuGbrQylANka0VdsZ5vXwIuxkheY?usp=sharing).

Nice thanks, maybe it's just an issue on my side then

Ok I managed to solve the BadRarFile issue on my side :)
To fix it I had to install the `unrar` tool for macos (though it seems it's not available with `brew install` anymore, I had to install it from elsewhere).

Needs https://github.com/huggingface/datasets/pull/1429 to work.


extract rar files

Unfortunately, I didn't find any native python libraries for extracting rar files. The user has to manually install `sudo apt-get install unrar`. Discussion with @yjernite is in the slack channel.


Add twi wordsim353

Add twi WordSim 353


Hebrew project BenYehuda

Added Hebrew corpus from https://github.com/projectbenyehuda/public_domain_dump


init commit for MultiReQA for third PR with all issues fixed

good dataset card as well :)
@lhoestq Thank you :)

3rd PR w.r.t. PR #1349 with all the issues fixed. As #1349 had uploaded other files along with the multi_re_qa dataset


Add german common crawl dataset

Hi @Phil1108 !
Have you had a chance to take a look at my suggestions ?
Feel free to ping me if you have questions or if you're ready for a review

Thanks again for adding this dataset, this one is very useful !

>
>
> Hi @Phil1108 !
> Have you had a chance to take a look at my suggestions ?
> Feel free to ping me if you have questions or if you're ready for a review
>
> Thanks again for adding this dataset, this one is very useful !

Hello @lhoestq ,

okay so should we go on with only the 2 parts or wait until the hosting of all 200 GB is cleared at HF?
If you want to merge it with the 2 parts only at first und upgrade later to all, sure I'll include the minor changes in the next few days and update it here

I think we can wait a bit to find a solution for the hosting and update the PR to include all the files when it's done.
In the meantime you can update the PR to take into account the suggestions regarding the dataset card, the parallel downloads and the use of eval.

A safer alternative to eval is `ast.literal_eval`, it should do the job

Adding a subpart of the Common Crawl which was extracted with this repo https://github.com/facebookresearch/cc_net and additionally filtered for duplicates


Add yoruba wordsim353

Added WordSim-353 evaluation dataset for Yoruba


Imppres

Feel free to ping me once you're ready for another review :)

I incorporated all the changes but when I go to rebase I get the following error:
```python
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git rebase upstream/master
error: cannot rebase: You have unstaged changes.
error: Please commit or stash them.
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git stash
Saved working directory and index state WIP on imppres: 51736236 Incorporated secondary sets as configurations instead of splits.
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git rebase upstream/master
CONFLICT (add/add): Merge conflict in datasets/wiki_movies/wiki_movies.py
Auto-merging datasets/wiki_movies/wiki_movies.py
CONFLICT (add/add): Merge conflict in datasets/wiki_movies/dataset_infos.json
Auto-merging datasets/wiki_movies/dataset_infos.json
CONFLICT (add/add): Merge conflict in datasets/wiki_movies/README.md
Auto-merging datasets/wiki_movies/README.md
error: could not apply 04d08587... Created wiki_movies dataset.
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 04d08587... Created wiki_movies dataset.
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git branch
* (no branch, rebasing imppres)
  imppres
  logiqa_en
  master
  wiki_movies
  wiki_movies_htl
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git checkout imppres
datasets/wiki_movies/README.md: needs merge
datasets/wiki_movies/dataset_infos.json: needs merge
datasets/wiki_movies/wiki_movies.py: needs merge
error: you need to resolve your current index first
aclifton@pop-os:~/hf_datasets_sprint/datasets$
```

I think it's because the current branch includes changes about wiki_movies.

Can you create a new branch from `master` and create another PR please ?

I get this response when I try to switch to master:
```
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git checkout master
datasets/wiki_movies/README.md: needs merge
datasets/wiki_movies/dataset_infos.json: needs merge
datasets/wiki_movies/wiki_movies.py: needs merge
error: you need to resolve your current index first
```

Maybe you have to remove the changes in wiki_movies before checkout to master
```
git stash
```

should do the job

Here is what I get:
```
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git stash
datasets/wiki_movies/README.md: needs merge
datasets/wiki_movies/dataset_infos.json: needs merge
datasets/wiki_movies/wiki_movies.py: needs merge
```

Ok I see
Looks like you're in a `merge` process.
You can abort it with `git reset --merge`

Then `git checkout master` should work

So close! I got the new branch made and went through all the tests. When I went to push, I got the following:
```
aclifton@pop-os:~/hf_datasets_sprint/datasets$ git push -u origin imppres
Username for 'https://github.com': aclifton314
Password for 'https://aclifton314@github.com':
To https://github.com/aclifton314/datasets
 ! [rejected]          imppres -> imppres (non-fast-forward)
error: failed to push some refs to 'https://github.com/aclifton314/datasets'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
```

after a rebase you need to `git push --force`

2nd PR ever! Hopefully I'm starting to get the hang of this. This is for the IMPPRES dataset. Please let me know of any corrections or changes that need to be made.


Can't map dataset (loaded from csv)

Please could you post the whole script? I can't reproduce your issue. After updating the feature names/labels to match with the data, everything works fine for me. Try to update datasets/transformers to the newest version.

Actually, the problem was how `tokenize` function was defined. This was completely my side mistake, so there are really no needs in this issue anymore

Hello! I am trying to load single csv file with two columns: ('label': str, 'text' str), where is label is str of two possible classes.

Below steps are similar with [this notebook](https://colab.research.google.com/drive/1-JIJlao4dI-Ilww_NnTc0rxtp-ymgDgM?usp=sharing), where bert model and tokenizer are used to classify lmdb loaded dataset. Only one difference it is the dataset loaded from .csv file.
Here is how I load it:

```python
data_path = 'data.csv'
data = pd.read_csv(data_path)

# process class name to indices
classes = ['neg', 'pos']
class_to_idx = { cl: i for i, cl in enumerate(classes) }

# now data is like {'label': int, 'text' str}
data['label'] = data['label'].apply(lambda x: class_to_idx[x])

# load dataset and map it with defined `tokenize` function
features = Features({
  target: ClassLabel(num_classes=2, names=['neg', 'pos'], names_file=None, id=None),
  feature: Value(dtype='string', id=None),
})
dataset = Dataset.from_pandas(data, features=features)
dataset.map(tokenize, batched=True, batch_size=len(dataset))
```

It ruins on the last line with following error:
```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-112-32b6275ce418> in <module>()
      9 })
     10 dataset = Dataset.from_pandas(data, features=features)
---> 11 dataset.map(tokenizer, batched=True, batch_size=len(dataset))

2 frames
/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1237         test_inputs = self[:2] if batched else self[0]
   1238         test_indices = [0, 1] if batched else 0
-> 1239         update_data = does_function_return_dict(test_inputs, test_indices)
   1240         logger.info("Testing finished, running the mapping function on the dataset")
   1241

/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py in does_function_return_dict(inputs, indices)
   1208             fn_args = [inputs] if input_columns is None else [inputs[col] for col in input_columns]
   1209             processed_inputs = (
-> 1210                 function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   1211             )
   1212             does_return_dict = isinstance(processed_inputs, Mapping)

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py in __call__(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   2281             )
   2282         ), (
-> 2283             "text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) "
   2284             "or `List[List[str]]` (batch of pretokenized examples)."
   2285         )

AssertionError: text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).
```

which I think is not expected. I also tried the same steps using `Dataset.from_csv` which resulted in the same error.

For reproducing this, I used [this dataset from kaggle](https://www.kaggle.com/team-ai/spam-text-message-classification)


adding fake-news-english-2


Add dataset yoruba_wordsim353

Contains loading script as well as dataset card including YAML tags.


Add Turkish News Category Dataset (270K)

@lhoestq, can you please review this PR?

@SBrandeis,
Sorry. All of the latest version came to my branch. You can find final version.
Resubmitted as a clean final version of #1466
I have completed all the review comments.

Closing this as PR is now https://github.com/huggingface/datasets/pull/1466

This PR adds the Turkish News Categories Dataset (270K) dataset which is a text classification dataset by me and @yavuzKomecoglu.  Turkish news dataset consisting of **273601 news** in **17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.


Add arabic dialects

Data loading script and dataset card for Dialectal Arabic Resources dataset.
Fixed git issues from PR  #976


WIP: Vinay/add peer read dataset


Add Shrinked Turkish NER from Kaggle.

Add Shrinked Turkish NER from [Kaggle](https://www.kaggle.com/behcetsenturk/shrinked-twnertc-turkish-ner-data-by-kuzgunlar).


Add Hate Speech and Offensive Language Detection dataset

@lhoestq done! The failing testes don't seem to be related, it seems to be a connection issue, if I understand it correctly.

Add [Hate Speech and Offensive Language Detection dataset](https://github.com/t-davidson/hate-speech-and-offensive-language) from [this paper](https://arxiv.org/abs/1703.04009).


Adding BioCreative II Gene Mention corpus


Add OffComBR

Hello @hugoabonizio, thanks for the contribution.
Regarding the fake data, you can generate it manually.
Running the `python datasets-cli dummy_data datasets/offcombr` should give you instructions on how to manually create the dummy data.
For reference, here is a spec for `.arff` files : https://www.cs.waikato.ac.nz/ml/weka/arff.html

@lhoestq again the failing tests doesn't seem to be related

Add [OffComBR](https://github.com/rogersdepelle/OffComBR) from [Offensive Comments in the Brazilian Web: a dataset and baseline results](https://sol.sbc.org.br/index.php/brasnam/article/view/3260/3222) paper.

But I'm having a hard time generating dummy data since the original dataset extion is `.arff` and the [_create_dummy_data function](https://github.com/huggingface/datasets/blob/a4aeaf911240057286a01bff1b1d75a89aedd57b/src/datasets/commands/dummy_data.py#L185) doesn't allow it.


Adding the ASSIN dataset

Adding the ASSIN dataset, a Portuguese language dataset for Natural Language Inference and Semantic Similarity Scoring


2 typos

Corrected 2 typos


Add penn treebank dataset

@yjernite I have updated the PR to be language modeling task specific. Please review!

Yes a line corresponds to a sentence in this data.


Adding the ASSIN dataset

I wrongly commited data from another branch in this PR, I'll close this a reopen another PR with the fixed branch


adding fake-news-english

also don't forget to format your code using `make style` to fix the CI


Add Tweet Eval Dataset

Hi @lhoestq,

Seeing that it has been almost two months to this draft, I'm willing to take this forward if you and @abhishekkrthakur don't mind. :)

Hi @gchhablani !
Sure if @abhishekkrthakur doesn't mind
Thanks for your help :)

Hi @lhoestq, @abhishekkrthakur

I believe this can be closed. Merged in #1829.


Add Portuguese Hate Speech dataset

@lhoestq done! (The failing tests don't seem to be related)

Binary Portuguese Hate Speech dataset from [this paper](https://www.aclweb.org/anthology/W19-3510/).


Adding TaPaCo Dataset with README.md

We want to keep the repo as light as possible so that it doesn't take ages to clone, that's why we ask for small dummy data files (especially when there are many of them). Let me know if you have questions or if we can help you on this

Hello @lhoestq , made the changes as you suggested and pushed, please review. By default, the dummy data was generated the way it was by the dummy data auto generate command. Thank you.


Add Acronym Identification Dataset


Add dataset clickbait_news_bg

Closing this pull request, will submit a new one for this dataset.

Adding a new dataset - clickbait_news_bg


adding covid-tweets-japanese (again)

README.md is not created yet. I'll add it soon.

Thank you for your detailed code review! It's so helpful.
I'll reflect them to the code in 24 hours.

You may have told me in Slack (I cannot find the conversation log though I've looked through threads), but I'm sorry it seems I'm still misunderstanding how to get YAML from the tagger.
I'm now asking on Slack if I am looking at the tagger the wrong way.

One more thing I'd like to ask.
Should I make changes by myself, or can I use the "Commit suggestion" feature?
I'm new to this feature and I don't know how the rules work in this repository, so I'd like to ask just in case.

I had mistaken use git rebase, I was so hurried to fix it. However, I didn't fully consider the use of git reset , so I unintendedly stopped PR (#1367) altogether. Sorry about that.
I'll make a new PR.


Add reasoning_bg

Hi @saradhix have you had the chance to reduce the size of the dummy data ?

Feel free to ping me when it's done so we can merge :)

@lhoestq I have reduced the size of the dummy data manually and pushed the changes.

The CI errors are not related to your dataset.
They're fixed on master, you can ignore them

Adding reading comprehension dataset for Bulgarian language


Add European Union Education and Culture Translation Memory (EAC-TM) dataset

Adding the EAC Translation Memory dataset : https://ec.europa.eu/jrc/en/language-technologies/eac-translation-memory


Add HoVer Dataset

@lhoestq all comments addressed :)
merging since the CI is fixed on master

HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification
https://arxiv.org/abs/2011.03088


Add Neural Code Search Dataset

@lhoestq Refactored into new branch, please review :)

The `RemoteDatasetTest ` errors in the CI are fixed on master so it's fine


datasets card-creator link added

dataset card creator link has been added
link: https://huggingface.co/datasets/card-creator/


initial commit for MultiReQA for second PR

Subsequent [PR #1426 ](https://github.com/huggingface/datasets/pull/1426) since this PR has uploaded other files along with the MultiReQA dataset.

closing this one since a new PR has been created

Since last PR #1349 had some issues passing the tests. So, a new PR is generated.


Add WikiSource Dataset


Add OfisPublik Dataset


Add script_version suggestion when dataset/metric not found

Adds a helpful prompt to the error message when a dataset/metric is not found, suggesting the user might need to pass `script_version="master"` if the dataset was added recently. The whole error looks like:

> Couldn't find file locally at blah/blah.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1/metrics/blah/blah.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/met
rics/blah/blah.py.
If the dataset was added recently, you may need to to pass script_version="master" to find the loading script on the master branch.


Add KDE4 Dataset


Add MultiParaCrawl Dataset


Add SPC Dataset


add amazon polarity dataset

`amazon_polarity` is probably a subset of `amazon_us_reviews` but I am not entirely sure about that.
I guess `amazon_polarity` will help in reproducing results of papers using this dataset since even if it is a subset from `amazon_us_reviews`, it is not trivial how to extract `amazon_polarity` from `amazon_us_reviews`, especially since `amazon_us_reviews` was released after `amazon_polarity`.

do you know what the problem would be ? should I pull the master before ? @lhoestq

The error just appeared on master. I will try to fix it today.
You can ignore them since it's not related to the dataset you added

This corresponds to the amazon (binary dataset) requested in https://github.com/huggingface/datasets/issues/353


hind_encorp

resubmit of hind_encorp file changes


Add LIAR dataset

@lhoestq done! The failing testes don't seem to be related, it seems to be a connection issue, if I understand it correctly.

Add LIAR dataset from [“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection](https://www.aclweb.org/anthology/P17-2067/).


Add RecipeNLG Dataset (manual download)

@lhoestq yes. I asked the authors for direct link but unfortunately we need to fill a form (captcha)


add best2009

`best2009` is a Thai word-tokenization dataset from encyclopedia, novels, news and articles by [NECTEC](https://www.nectec.or.th/) (148,995/2,252 lines of train/test). It was created for [BEST 2010: Word Tokenization Competition](https://thailang.nectec.or.th/archive/indexa290.html?q=node/10). The test set answers are not provided publicly.


Add News Commentary Dataset


added conv ai 2

@lhoestq Thank you for the suggestions. I added the changes to the branch and seems after rebasing it to master, all the commits previous commits got added. Should I create a new PR or should I keep this one only ?

Dataset : https://github.com/DeepPavlov/convai/tree/master/2018


adding UNPC

merging since the CI just had a connection error

Adding United Nations Parallel Corpus
http://opus.nlpl.eu/UNPC.php


Add twi text c3

looks like this PR includes changes about other datasets

Can you only include the changes related to twi text c3 please ?

Hi @lhoestq , I have removed the unnecessary files. Can you please confirm?

You might need to either find a way to go back to the commit before it changes 389 files or create a new branch.

okay, I have created another branch, see the latest pull https://github.com/huggingface/datasets/pull/1518  @cstorm125

Added Twi texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/


Add Tatoeba Dataset


Add yoruba text c3

looks like this PR includes changes about other datasets

Thanks for the review. I'm a bit confused how to remove the files. Every time I add a new branch name using the following commands:

git fetch upstream
git rebase upstream/master
git checkout -b a-descriptive-name-for-my-changes

and push to the origin, this issue occurs

Can you try to create the branch from the master branch of your fork ?

first update your master branch:
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
```

then create a new one:
```
git checkout -b my-new-branch
```

I think you were still having the files because you were creating the new branch from a branch in which you've committed the files, instead of creating the new branch from the master branch

> @lhoestq , I have removed the unnecessary files. Looks like I still have one error. How do I resolve this?

I think it's connection error on piqa dataset. Can you try triggering the test again? I usually resolve similar issues with:
```
git fetch upstream
git rebase upstream/master
git push -u -f origin your_branch_name
```

I have created another pull request for this https://github.com/huggingface/datasets/pull/1515 @cstorm125  @lhoestq

Added Yoruba texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/


Add FACTCK.BR dataset

@lhoestq done!
merging since the CI is fixed on master

This PR adds [FACTCK.BR](https://github.com/jghm-f/FACTCK.BR) dataset from [FACTCK.BR: a new dataset to study fake news](https://dl.acm.org/doi/10.1145/3323503.3361698).


adding marathi-wiki dataset

Can you make it a draft PR until you've added the dataset please ? @ekdnam

Adding marathi-wiki-articles dataset.


Add SETimes Dataset


Add OPUS EMEA Dataset


Add OPUS Tilde Model Dataset


Add OPUS ECB Dataset


Add OPUS Books Dataset


Adding Scielo

Adding Scielo: Parallel corpus of full-text articles in Portuguese, English and Spanish from SciELO
https://sites.google.com/view/felipe-soares/datasets#h.p_92uSCyAjWSRB


Add OPUS PHP Dataset


Use passed --cache_dir for modules cache

I have a question: why not using a tmp dir instead, like the DummyDataGeneratorDownloadManager does?

Hi @lhoestq, I am trying to understand better the logic...

Why do we have a `dynamic_module_path` besides the modules cache path?
```python
DYNAMIC_MODULES_PATH = os.path.join(HF_MODULES_CACHE, "datasets_modules")
```
Moreover, 2 subdirectories (for datasets and for metrics) were created inside it:
```python
DATASETS_PATH = os.path.join(DYNAMIC_MODULES_PATH, "datasets")
METRICS_PATH = os.path.join(DYNAMIC_MODULES_PATH, "metrics")
```

Hi :)
The modules cache path is the path added to `sys.path`.
Therefore inside we need to have a folder that is going to be a package: `datasets_modules`.
This package will contain dynamic modules, i.e. datasets and metrics modules added on-the-fly.
Then we have two sub-modules `datasets_modules.datasets` and `datasets_modules.metrics`.

Maybe we can make things more explicit in the code with some comments explaining the structure, and maybe better variable naming as well..

Also I wanted to say that I started to work on offline loading of modules in #1726 and actually it lead to do similar changes to what you did to control the path where modules are stored.

Hi @lhoestq, I see...

Indeed I was also creating a draft for test_load, to clarify the expected behavior... ;)

So, for the command line:
```sh
python datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>
```
the `cache_dir` argument refers to dataset cache dir. We do not have control over the modules cache dir, but we would like to have. And if I understand well, you suggest adding another argument `dynamic_module_path`. Am I right?

> So, for the command line:
>
> ```shell
> python datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>
> ```
>
> the `cache_dir` argument refers to dataset cache dir. We do not have control over the modules cache dir, but we would like to have. And if I understand well, you suggest adding another argument `dynamic_module_path`. Am I right?

Yes the cache_dir is used to download files and also so save the dataset arrow files.
This is indeed different from the path for dynamic modules.

I suggested to have `dynamic_module_path` as a parameter but actually this is the parent directory `hf_modules_cache` that we would need (it's the one that is passed to `init_dynamic_modules ` that we need to add to `sys.path`).

Currently it's already possible to override it using the env variable `HF_MODULES_CACHE` but we can imagine having it as a parameter as well.

This way the user controls both the `cache_dir` and the `hf_modules_cache` which are the two places used by the library to read/write stuff.

I think #1726 is going to be merged pretty soon. Maybe can work on this as soon as it's merged to avoid doing the same things twice and to avoid conflicts ?

I agree. Indeed I took some of your code in one of my last commit, to try to implement the logic you described.

When passed `--cache_dir` arg:
```shell
python datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>
```
it is not used for caching the modules, which are cached in the default location at `.cache/huggingface/modules`.

With this fix, the modules will be cached at `<my-cache-dir>/modules`.


Re-adding narrativeqa dataset

@lhoestq I think I've fixed the dummy data - it finally passes! I'll add the model card now.

> Awesome thank you !
>
> Could you try to reduce the size of the dummy_data.zip file before we merge ? (it's 300KB right now)
>
> To do so feel free to take a look inside it and remove all the unnecessary files and chunks of text, to only keep a few examples. The idea is to have a zip file that is only a few KB

Ah, it only contains 1 example for each split. I think the problem is that I include an entire story (like in the full dataset). We can probably get away with a summarised version.

> Nice thank you, can you make it even lighter if possible ? Something round 10KB would be awesone
> We try to keep the repo light so that it doesn't take ages to clone. So we have to make sure the dummy data are as small as possible for every single dataset.

Have trimmed a little more out of each example now.

An update of #309.


adding covid-tweets-japanese

I think it's because the file you download uncompresses into a file and not a folder so `--autogenerate` couldn't create dummy data for you. See in your dummy_data.zip if there is a file there. If not, manually create your dummy data and compress them to dummy_data.zip.

@cstorm125 Thank you for the comment!
As you point out, it seems my code has something wrong about downloading and uncompressing the file.
However, my manually created dummy data seems to contain a file of the required format.

On Colaboratory,
`!unzip /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip`
returns:

```
Archive:  /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip
   creating: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/
 extracting: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/data.csv.bz2
```

The original data is `data.csv.bz2`, and I had a very hard time dealing with uncompressing bzip2.
I think I could handle it, but there may be problems remain.

Adding COVID-19 Japanese Tweets Dataset as part of the sprint.

Testing with dummy data is not working (the file is said to not exist). Sorry for the incomplete PR.


Adding Hope EDI dataset

@lhoestq Have addressed your comments. Please review. Thanks.


Add Mkqa dataset

the `RemoteDatasetTest ` error pf the CI is fixed on master so it's fine

# MKQA: Multilingual Knowledge Questions & Answers Dataset
Adding the [MKQA](https://github.com/apple/ml-mkqa) dataset as part of the sprint 🎉

There is no official data splits so I added just a `train` split.

differently from the original:
- answer:type field is a ClassLabel (I thought it might be possible to train on this as a label for categorizing questions)
- answer:entity field has a default value of empty string '' (since this key is not available for all in original)
- answer:alias has default value of []

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


Narrative QA (Manual Download Stories) Dataset

Hi ! Maybe we can rename it `narrativeqa_manual` to make it explicit that this one requires manual download contrary to `narrativeqa` ?
It's important to have this one as well, in case the `narrativeqa` one suffers from download issues (checksums or dead links for example).

You can also copy the dataset card from `narrativeqa` and add the dummy data as well

Thanks @lhoestq will do all this and submit a request in the coming days. 😊

Closing this as another pull request is already done.

Narrative QA with manual download for stories.


Adding OPUS MultiUN

Adding UnMulti
http://www.euromatrixplus.net/multi-un/


adding opus_infopankki

Adding opus_infopankki
http://opus.nlpl.eu/infopankki-v1.php


adding bprec

@lhoestq I think this is ready for review, I assume the errors (connection) are unrelated to the PR :)

Brand-Product Relation Extraction Corpora in Polish


add wisesight1000

`wisesight1000` contains Thai social media texts randomly drawn from the full `wisesight-sentiment`, tokenized by human annotators. Out of the labels `neg` (negative), `neu` (neutral), `pos` (positive), `q` (question), 250 samples each. Some texts are removed because they look like spam.Because these samples are representative of real world content, we believe having these annotaed samples will allow the community to robustly evaluate tokenization algorithms.


Add JNLPBA


Add spider dataset

This PR adds the Spider dataset, a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases.

Dataset website: https://yale-lily.github.io/spider
Paper link: https://www.aclweb.org/anthology/D18-1425/


Youtube caption corrections

Sorry about forgetting flake8.
Rather than use up the circleci resources on a new push with only formatting changes, I will wait to push until the results from all tests finish and/or any feedback comes in... probably tomorrow for me.

So... my normal work is with mercurial and seem to have clearly forked this up using git... :(

What I did is after calling:
```
git fetch upstream
git rebase upstream/master
```

I then I attempt to pull in my most recent changes UI commit changes based on @lhoestq's feedback with:
```
git pull
```
... which I now suspect undid the above fetch and rebase. Will look into fixing later today when I have more time. Sorry!

My dummy data seems quite large as a single row is composed of tokens/labels for an entire youtube video, with at least one row required for each file, which in this case 1 file per 13 youtube channels.

To make it smaller I passed `--n_lines 1` to reduce about 5x.

I then manually reduced size of the particularly long youtube lectures to get the size to about 30KB. However, after recompressing into a zip, and running dummy data test I got the following error:
`FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_youtube_caption_corrections - OSError: Cannot find data file. `, despite file being there, which I haven't had a chance yet to debug.

I wrote a small script to generate a smaller json file for the dummy_data, with the hope that I could resolve the pytest error noted above (in case related to a manual typo I could have introduce), however the test contains to fail locally... here's to hoping it can pass on remote!

Sorry for delayed comments here. Last commit made two changes:
- Increased the valency of the labels from just True/False to more categories to describe the various types of diffs encountered. This required some rewrite of the README
- Reduced the number of remote files to be downloaded from 13 to 4, by combining all 13 of the channel-specific files together, and the splitting them up in a way to meet Github file size requirements. This also reduces size of the dummy-data.

@lhoestq, thank you for the great feedback, especially given how busy you guys are now!

I checked out GitHub release tags and looks cool. I have added the version tag to the url, instead of the commit sha as originally suggested, with the hope that it serves the same purpose of pinning the content to this url. Please let me know if I have misunderstood.

In regard to dynamically changing the number of files downloaded by first downloading a JSON listing the files, I love that idea. But I am a little confused, as I was thinking that any changes to the dataset itself would require a new PR with an updated `dataset_infos.json`, e.g. `num_examples` would increase.

If the purpose of this is not to permit dynamic (without a PR needed) growth of the number of files, but instead to provide stability to the consumers of the dataset, maybe I continued use the release tags, maintaining access to old releases could serve this purpose? I am still learning about these release tags...

For dynamic datasets, i.e. datasets that evolve over time, we support custom configurations: they are configurations that are not part of the BUILDER_CONFIGS or in the dataset_infos.json

For example for wikipedia, you can use the latest wiki dump by specifying `date=` inside `load_dataset()`. A configuration is created on the fly for this date and is used to build the dataset using the latest data.

Therefore we don't need to have PRs to update the script for each wikipedia release.

One downside though is that we don't have metadata in advance such as the size of the dataset.

I think this could be a nice addition for the youtube caption dataset in the future to be have a system of releases and be able to load the version we want easily. What do you think ?

> For dynamic datasets, i.e. datasets that evolve over time, we support custom configurations: they are configurations that are not part of the BUILDER_CONFIGS or in the dataset_infos.json
>

> I think this could be a nice addition for the youtube caption dataset in the future to be have a system of releases and be able to load the version we want easily. What do you think ?

Thank you for the suggestion! This sounds great! I will take a look at the some datasets that do this, and would love to give it a try in the future, if I continue to grow the captions dataset in a meaningful way.

Appreciate all the help on this. It has been a really great experience for me. :)

Excited to merge! And sorry to be such a github n00b, but from what I've quickly read, I don't 'Close pull request', but rather the next steps are action taken on your end... Please let me know if there is some action to be taken at my end first. :/

This PR adds a new dataset of YouTube captions, error and corrections. This dataset was created in just the last week, as inspired by this sprint!


Add StackOverflow StackSample dataset

@lhoestq Thanks for the review and suggestions! I've added your comments and pushed the changes. I'm having issues with the dummy data still. When I run the dummy data test

```bash
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_so_stacksample
```
I get this error:

```
___________________________________________ LocalDatasetTest.test_load_dataset_all_configs_so_stacksample ____________________________________________

self = <tests.test_dataset_common.LocalDatasetTest testMethod=test_load_dataset_all_configs_so_stacksample>, dataset_name = 'so_stacksample'

    @slow
    def test_load_dataset_all_configs(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True)

tests/test_dataset_common.py:237:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:198: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true

FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_so_stacksample - AssertionError: False is not true
```

I tried formatting the data similar to other datasets, but I think I don't have my csv's in the zip folder with the proper name. I also ran the command that's supposed to outline the exact steps I need to perform to get them into the correct format, but I followed them and they don't seem to be working still :/. Any help would be greatly appreciated!

Ok I found the issue with the dummy data.
It's currently failing because it's not generating a single example using the dummy csv file.
That's because there's only only line in the dummy csv file, and this line is skipped using the `next()` call used to ignore the headers of the csv.

To fix the dummy data you must add headers to the dummy csv files.

Also can you make sure that all the original CSV files have headers ? i.e. check that their first line is just the column names

> Ok I found the issue with the dummy data.
> It's currently failing because it's not generating a single example using the dummy csv file.
> That's because there's only only line in the dummy csv file, and this line is skipped using the `next()` call used to ignore the headers of the csv.
>
> To fix the dummy data you must add headers to the dummy csv files.

Oh man, I bamboozled myself! Thank you @lhoestq for catching that! I've updated the dummy csv's to include headers and also confirmed that they all have headers, so I am not throwing away any information with that `next()` call. When I run the test locally for the dummy data it passes, so hopefully it is good to go :D

This PR adds the StackOverflow StackSample dataset from Kaggle: https://www.kaggle.com/stackoverflow/stacksample

Ran through all of the steps. However, since my dataset requires manually downloading the data, I was unable to run the pytest on the real dataset (the dummy data pytest passed).


Addition of py_ast dataset

@lhoestq as discussed in PR #1195


Add TweetQA dataset

This PR adds the  TweetQA dataset, the first dataset for QA on social media data by leveraging news media and crowdsourcing.

Paper: https://arxiv.org/abs/1907.06292
Repository: https://tweetqa.github.io/


New instruction for how to generate dataset_infos.json

Add additional instructions for how to generate dataset_infos.json for manual download datasets. Information courtesy of `Taimur Ibrahim` from the slack channel


change url for prachathai67k to internet archive

`prachathai67k` is currently downloaded from git-lfs of PyThaiNLP github. Since the size is quite large (~250MB), I moved the URL to archive.org in order to prevent rate limit issues.


added craigslist_bargians

`craigslist_bargains` data set from [here](https://worksheets.codalab.org/worksheets/0x453913e76b65495d8b9730d41c7e0a0c/)

(Cleaned up version of #1278)


add LeNER-Br dataset

I don't know what happened, my first commit passed on all checks, but after just a README.md update one of the scripts failed, is it normal? 😕

Looks like a flaky connection error, I've launched a re-run, it should be fine :)

The RemoteDatasetTest error in the CI is just a connection error, we can ignore it

Adding the LeNER-Br dataset, a Portuguese language dataset for named entity recognition


initial commit for MultiReQA

looks like this dataset includes changes about many other files than the ones for multi_re_qa

Can you create another branch and another PR please ?

> looks like this dataset includes changes about many other files than the ones for multi_re_qa
>
> Can you create another branch and another PR please ?

Sure I will do that. Thank you.

Added MultiReQA, which is a dataset containing the sentence boundary annotation from eight publicly available QA datasets including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, BioASQ, RelationExtraction, and TextbookQA.


add Yoruba NER dataset

Thank you. Okay, other pull requests only have one dataset

The `RemoteDatasetTest` error in the CI is just a connection error, we can ignore it

Added Yoruba GV dataset based on this paper


Add spanish billion words corpus

Thank you for your feedback! I've reduced the dummy data size to 2KB.

I had to rebase to fix `RemoteDatasetTest` fails, sorry about the 80 commits.
I could create a new clean PR if you prefer.

I have seen that in similar cases you have suggested to other contributors to create another branch and another PR, so I will do that.

Add an unannotated Spanish corpus of nearly 1.5 billion words, compiled from different resources from the web.


Add MultiBooked dataset

There' still an issue with the dummy data, let me take a look

Add dataset.


First commit of NarrativeQA Dataset

Added NarrativeQA dataset and included a manual downloading option to download scripts from the original scripts provided by the authors.


Add hausa ner corpus

Added Hausa VOA NER data


Add LiveQA

This PR adds LiveQA, the Chinese real-time/timeline-based QA task by [Liu et al., 2020](https://arxiv.org/pdf/2010.00526.pdf).


[yaml] Fix metadata according to pre-specified scheme

@lhoestq @yjernite


added references to only data card creator to all guides

We can now use the wonderful online form for dataset cards created by @evrardts


:fist: ¡Viva la Independencia!

I've added the changes / fixes - ready for a second pass :)

Adds the Catalonia Independence Corpus for stance-detection of Tweets.

Ready for review!


hate_speech_18 initial commit

> Nice thanks !
>
> Can you rename the dataset folder and the dataset script name `hate_speech18` instead of `hate_speech_18` to follow the snake case convention we're using ?
>
> Also it looks like the dummy_data.zip file is quite big (almost 4MB).
> Can you try to reduce its size ?
>
> To do so feel free to take a look inside it and remove all the unnecessary files or chunks of texts. The idea is to only keep a few examples

Done, thanks!


Add GigaFren Dataset


Add spanish billion words

The tests failed because of ```RemoteDatasetTest``` so I tried ```git rebase``` and messed everything up. I've made a new clean PR (#1347).

Add an unannotated corpus of the Spanish language of nearly 1.5 billion words, compiled from different resources from the web.

The dataset needs 10 GB (download: 1.89 GiB, generated: 8.34 GiB, post-processed: Unknown size, total: 10.22 GiB), the test using dummy data pass but my laptop isn't able to run it on the real data (I left it running for over 8 hours and it didn't finish).


Add dataset Yoruba BBC Topic Classification

Added new dataset Yoruba BBC Topic Classification

Contains loading script as well as dataset card including YAML tags.


Added Bianet dataset

Hi :hugs:, This is a PR for [Bianet: A parallel news corpus in Turkish, Kurdish and English; Source](http://opus.nlpl.eu/Bianet.php) dataset


Add QED Amara Dataset


Add Tanzil Dataset


Add Open Subtitles Dataset


First version of the new dataset hausa_voa_topics

Contains loading script as well as dataset card including YAML tags.


added un_ga dataset

Looks like this PR includes changes about many other files than the ones for un_ga

Can you create another branch an another PR please ?

@lhoestq, Thank you for suggestions. I have made the changes and raised the new PR https://github.com/huggingface/datasets/pull/1569.

Hi :hugs:, This is a PR for [United nations general assembly resolutions: A six-language parallel corpus](http://opus.nlpl.eu/UN.php) dataset


Add yoruba ner corpus


Added the NewsPH Raw dataset and corresponding dataset card

This PR adds the original NewsPH dataset which is used to autogenerate the NewsPH-NLI dataset. Reopened a new PR as the previous one had problems.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


Add msr_genomics_kbcomp dataset


TEP: Tehran English-Persian parallel corpus
more info : http://opus.nlpl.eu/TEP.php


Add humicroedit dataset

Updated the commit with the generated yaml tags

Pull request for adding humicroedit dataset


❓ Sharing ElasticSearch indexed dataset

dataset request

Requesting to add a new dataset

Hello @pietrolesci , I am not sure to understand what you are trying to do here.

If you're looking for ways to save a dataset on disk, you can you the `save_to_disk` method:
```python
>>> import datasets
>>> loaded_dataset = datasets.load("dataset_name")
>>> loaded_dataset.save_to_disk("/path/on/your/disk")
```

The saved dataset can later be retrieved using:
```python
>>> loaded_dataset = datasets.Dataset.load_from_disk("/path/on/your/disk")
```

Also, I'd recommend posting your question directly in the issue section of the [elasticsearch repo](https://github.com/elastic/elasticsearch)

Hi @SBrandeis,

Thanks a lot for picking up my request.

Maybe I can clarify my use-case with a bit of context. Say I have the IMDb dataset. I create an ES index on it. Now I can save and reload the dataset from disk normally. Once I reload the dataset, it is easy to retrieve the ES index on my machine. I was wondering: is there a way I can share the (now) indexed version of the IMDb dataset with my colleagues without requiring them to re-index it?

Thanks a lot in advance for your consideration.

Best,

Pietro

Thanks for the clarification.

I am not familiar with ElasticSearch, but if I understand well you're trying to migrate your data along with the ES index.
My advice would be to check out ES documentation, for instance, this might help you: https://www.elastic.co/guide/en/cloud/current/ec-migrate-data.html

Let me know if it helps

Hi there,

First of all, thank you very much for this amazing library. Datasets have become my preferred data structure for basically everything I am currently doing.

**Question:** I'm working with a dataset and I have an elasticsearch container running at localhost:9200. I added an elasticsearch index and I was wondering

- how can I know where it has been saved?

- how can I share the indexed dataset with others?

I tried to dig into the docs, but could not find anything about that.

Thank you very much for your help.

Best,
Pietro

Edit: apologies for the wrong label


Add CC-News dataset of English language articles

@vblagoje nice work, please add the README.md file and it would be ready

@lhoestq @tanmoyio @yjernite please have a look at the dataset card. Don't forget that the dataset is still hosted on my private gs bucket and should eventually be moved to the HF bucket

I will move the files soon and ping you when it's done and with the new URLs :)

Hi !

I just moved the file to a HF bucket. It's available at https://storage.googleapis.com/huggingface-nlp/datasets/cc_news/cc_news.tar.gz

Sorry for the delay ^^'

@lhoestq no worries, updated PR with the new URL and rebased to master

Adds [CC-News](https://commoncrawl.org/2016/10/news-dataset-available/) dataset. It contains 708241 English language news articles. Although each article has a language field these tags are not reliable. I've used Spacy language detection [pipeline](https://spacy.io/universe/project/spacy-langdetect) to confirm that the article language is indeed English.

The prepared dataset is temporarily hosted on my private Google Storage [bucket](https://storage.googleapis.com/hf_datasets/cc_news.tar.gz). We can move it to HF storage and update this PR before merging.


add indonlu benchmark datasets

The IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for the Indonesian language. There are 12 datasets in IndoNLU.


added dutch_social

@lhoestq
Updated the `dummy_data.zip `(<10kb)I had to reduce it to just a few samples.
Train-Test-Dev  (20-5-5 samples)

But the push also added changes from other PRs (probably because of a rebase!) So the files changed tab shows 466 files were changed!

Thanks ! The dummy data are all good now :)

Looks like this PR includes changes to many other files than the ones for dutch_social now.

Can you create another branch and another PR please ?

>
> Can you create another branch and another PR please ?
@lhoestq

I did a rebase. Now it doesn't include the other files. Does that help?

The Dutch social media tweets dataset. Which has a total of more than 210k tweets in dutch language. These tweets have been machine annotated with sentiment scores (`label` feature) and `industry` and `hisco_codes`

It can be used for sentiment analysis, multi-label classification and entity tagging


Added the WikiText-TL39 dataset and corresponding card

This PR adds the WikiText-TL-39 Filipino Language Modeling dataset. Restarted a new pull request since there were problems with the earlier one.

Paper: https://arxiv.org/abs/1907.00409
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks


adding wili-2018 language identification dataset

@lhoestq Not sure what happened, I just changed the py file but it is showing some TensorFlow error now.

You can ignore it.
It's caused by the Tensorflow update that happened 30min ago. They added breaking changes.
I'm working on a fix on the master branch right now

oh okay, btw I have made the required change for reading the CSV, I think it should be fine now, please take a look at it when you have some time.


ethos first commit

> Nice thanks !
>
> I left a few comments
>
> Also it looks like this PR includes changes about other files than the ones for ethos
>
> Can you create another branch and another PR please ?

@lhoestq Should I close this PR? The new one is the: #1453

> You can create another PR and close this one if you don't mind

Perfect! You should see the #1453 PR for the fixed version! Thanks

Ethos passed all the tests except from this one:
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>

with this error:
E               OSError: Cannot find data file.
E               Original error:
E               [Errno 2] No such file or directory:


add 10k German News Article Dataset

You can just create another branch from master on your fork and create another PR:

first update your master branch
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
```

then create a new branch
```
git checkout -b my-new-branch-name
```

Then you can add, commit and push the gnad10 files and open a new PR


Allow GitHub releases as dataset source

# Summary

Providing a GitHub release URL to `DownloadManager.download()` currently throws a `ConnectionError: Couldn't reach [DOWNLOAD_URL]`. This PR fixes this problem by adding an exception for GitHub releases in `datasets.utils.file_utils.get_from_cache()`.

# Reproduce

```
import datasets
url = 'http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz'
result = datasets.utils.file_utils.get_from_cache(url)

# Returns: ConnectionError: Couldn't reach http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz
```

# Cause

GitHub releases returns a HTTP status 403 (FOUND), indicating that the request is being redirected (to AWS S3, in this case). `get_from_cache()` checks whether the status is 200 (OK) or if it is part of two exceptions (Google Drive or Firebase), otherwise the mentioned error is thrown.

# Solution

Just like the exceptions for Google Drive and Firebase, add a condition for GitHub releases URLs that return the HTTP status 403. If this is the case, continue normally.


add yelp_review_full

This corresponds to the Yelp-5 requested in https://github.com/huggingface/datasets/issues/353
I included the dataset card.


Add snips built in intents 2016 12

It is not clear how to automatically add the dummy data if the source data is a more complex json format. Should I manually take a fraction of the source data and include it as dummy data?

Added a fraction of the real data as dummy data.

This PR proposes to add the Snips.ai built in intents dataset. The first configuration added is for the intent labels only, but the dataset includes entity slots that may in future be added as alternate configurations.


Add HateSpeech Corpus for Polish

@lhoestq Do you think using the ClassLabel is correct if we don't know the meaning of them?

Once we find out the meanings we can still add them to the dataset card

Feel free to ping me when the PR is ready for the final review

This PR adds a HateSpeech Corpus for Polish, containing offensive language examples.

- **Homepage:** http://zil.ipipan.waw.pl/HateSpeech
- **Paper:** http://www.qualitativesociologyreview.org/PL/Volume38/PSJ_13_2_Troszynski_Wawer.pdf


Jigsaw toxicity pred

Requires manually downloading data from Kaggle.


Add OPUS Bible Corpus (102 Languages)


Add OffensEval-TR 2020 Dataset

@lhoestq, can you please review this PR?

> Awesome thank you !

Thanks for the small fixes @lhoestq

@coltekin, we have added the data set that you created an article that says "Turkish Attack Language Community in Social Media", HuggingFace dataset update sprint for you. We added Sprint quickly for a short time. I hope you welcome it too. The dataset is accessible at https://huggingface.co/datasets/offenseval2020_tr.

Thank you for the heads up. I am not familiar with the terminology above (no idea what a sprint is), but I am happy that you found the data useful. Please feel free to distribute/use it as you see fit.

The OffensEval version you included in your data set has only binary labels. There is also a version [here](https://coltekin.github.io/offensive-turkish/troff-v1.0.tsv.gz) which also includes fine-grained labels similar to the OffensEval English data set - Just in case it would be of interest.

If you have questions about the data set, or need more information please let me know.

This PR adds the OffensEval-TR 2020 dataset which is a Turkish offensive language corpus by me and @basakbuluz. The corpus consist of randomly sampled tweets and annotated in a similar way to [OffensEval](https://sites.google.com/site/offensevalsharedtask/) and [GermEval](https://projects.fzai.h-da.de/iggsa/).

- **Homepage:** [offensive-turkish](https://coltekin.github.io/offensive-turkish/)
- **Paper:** [A Corpus of Turkish Offensive Language on Social Media](https://coltekin.github.io/offensive-turkish/troff.pdf)
- **Point of Contact:** [Çağrı Çöltekin](ccoltekin@sfs.uni-tuebingen.de)


Add SAMSum Corpus dataset

also to fix the check_code_quality CI you have to remove the imports of the unused `csv` and `os`

@lhoestq Thanks for the review! I have done what you asked, README is also updated. 🤗
The CI fails because of the added dependency. I have never used circleCI before, so I am curious how will you solve that?

I just added `py7zr` to our test dependencies

Did not spent much time writing README, might update later.

Copied description and some stuff from tensorflow_datasets
https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/summarization/samsum.py


Add Wiki Lingua Dataset

I am done adding the dataset. Requesting to review and advise.

looks like this PR has changes about many other files than the ones for WIki Lingua

Can you create another branch and another PR please ?

> looks like this PR has changes about many other files than the ones for WIki Lingua
>
> Can you create another branch and another PR please ?

Ok, I will create another branch and submit a fresh PR.

> Any reason to have english as the default config over the other languages ?

The data for all other languages has a direct reference to English article. Thus, I kept English as default.

Hello,

This is my first PR.

I have added Wiki Lingua Dataset along with dataset card to the best of my knowledge.
There was one hiccup though. I was unable to create dummy data because the data is in pkl format.
From the document, I see that:
```At the moment it supports data files in the following format: txt, csv, tsv, jsonl, json, xml```


adding capes

Adding Parallel corpus of theses and dissertation abstracts in Portuguese and English from CAPES
https://sites.google.com/view/felipe-soares/datasets#h.p_kxOR6EhHm2a6


add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC)

I created a clean PR where I also incorporated the suggested changes here: https://github.com/huggingface/datasets/pull/1449

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


[README] Added Windows command to enable slow tests

The Windows command to run slow tests has caused issues, so this adds a functional Windows command.


adding eitb_parcc

Adding EiTB-ParCC: Parallel Corpus of Comparable News
http://opus.nlpl.eu/EiTB-ParCC.php


adding opus_openoffice

Adding Opus OpenOffice: http://opus.nlpl.eu/OpenOffice.php
8 languages, 28 bitexts


Add Danish NER dataset


arxiv dataset added

Readme added
@lhoestq is it looking alright ?

**adding arXiv dataset**: arXiv dataset and metadata of 1.7M+ scholarly papers across STEM
dataset link: https://www.kaggle.com/Cornell-University/arxiv


added dutch_social

Closing this since a new pull request has been made.

WIP
As some tests did not clear! 👎🏼


can't load "german_legal_entity_recognition" dataset

Please if you could tell me more about the error?

1. Please check the directory you've been working on
2. Check for any typos

> Please if you could tell me more about the error?
>
> 1. Please check the directory you've been working on
> 2. Check for any typos

Error happens during the execution of this line:
dataset = load_dataset("german_legal_entity_recognition")

Also, when I try to open mentioned links via Opera I have errors "404: Not Found" and "This XML file does not appear to have any style information associated with it. The document tree is shown below." respectively.

Hello @nataly-obr, the `german_legal_entity_recognition` dataset has not yet been released (it is part of the coming soon v2 release).

You can still access it now if you want, but you will need to install `datasets` via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`

Please let me know if it solves the issue :)

FileNotFoundError: Couldn't find file locally at german_legal_entity_recognition/german_legal_entity_recognition.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py


Add OPUS Ted Talks 2013


OPUS Ted Talks 2013


The Snips Built In Intents 2016 dataset.

It is not clear how to automatically add the dummy data if the source data is a more complex json format. Should I manually take a fraction of the source data and include it as dummy data?

Will tag the dataset and update the dataset card.


add hrenwac_para


adding opus_euconst

Adding EUconst, a parallel corpus collected from the European Constitution.
21 languages, 210 bitexts


arXiv dataset added


adding pubmed_qa dataset

Pubmed QA dataset:
PQA-L(abeled) 1k
PQA-U(labeled) 61.2k
PQA-A(rtifical labeled) 211.3k


imdb dataset cannot be downloaded

Hi @rabeehk , I am unable to reproduce your problem locally.
Can you try emptying the cache (removing the content of `/idiap/temp/rkarimi/cache_home_1/datasets`) and retry ?

Hi,
thanks, I did remove the cache and still the same error here

```
>>> a = datasets.load_dataset("imdb", split="train")
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Downloading and preparing dataset imdb/plain_text (download: 80.23 MiB, generated: 127.06 MiB, post-processed: Unknown size, total: 207.28 MiB) to /idiap/temp/rkarimi/cache_home_1/datasets/imdb/plain_text/1.0.0/90099cb476936b753383ba2ae6ab2eae419b2e87f71cd5189cb9c8e5814d12a3...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 558, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 73, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=4902716, num_examples=3680, dataset_name='imdb')}]
```

datasets version
```
datasets                  1.1.2                     <pip>
tensorflow-datasets       4.1.0                     <pip>

```

hi
please find error below getting imdb train spli:
thanks

`
datasets.load_dataset>>> datasets.load_dataset("imdb", split="train")`

errors

```
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Downloading and preparing dataset imdb/plain_text (download: 80.23 MiB, generated: 127.06 MiB, post-processed: Unknown size, total: 207.28 MiB) to /idiap/temp/rkarimi/cache_home_1/datasets/imdb/plain_text/1.0.0/90099cb476936b753383ba2ae6ab2eae419b2e87f71cd5189cb9c8e5814d12a3...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 558, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 73, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=7486451, num_examples=5628, dataset_name='imdb')}]

```


Jigsaw toxicity classification dataset added

The dataset requires manually downloading data from Kaggle.


Add CodeSearchNet corpus dataset

This PR adds the CodeSearchNet corpus proxy dataset for semantic code search: https://github.com/github/CodeSearchNet
I have had a few issues, mentioned below. Would appreciate some help on how to solve them.

## Issues generating dataset card
Is there something wrong with my declaration of the dataset features ?
```
features=datasets.Features(
    {
        "repository_name": datasets.Value("string"),
        "func_path_in_repository": datasets.Value("string"),
        "func_name": datasets.Value("string"),
        "whole_func_string": datasets.Value("string"),
        "language": datasets.Value("string"),
        "func_code_string": datasets.Value("string"),
        "func_code_tokens": datasets.Sequence(datasets.Value("string")),
        "func_documentation_string": datasets.Value("string"),
        "func_documentation_tokens": datasets.Sequence(datasets.Value("string")),
        "split_name": datasets.Value("string"),
        "func_code_url": datasets.Value("string"),
        # TODO - add licensing info in the examples
    }
),

```
When running the streamlite app for tagging the dataset on my machine, I get the following error :
![image](https://user-images.githubusercontent.com/33657802/101469132-9ed12c80-3944-11eb-94ff-2d9c1d0ea080.png)

## Issues with dummy data
Due to the unusual structure of the data, I have been unable to generate dummy data automatically.
I tried to generate it manually, but pytests fail when using the manually-generated dummy data ! Pytests work fine when using the real data.
```
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.7.9, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
plugins: xdist-2.1.0, forked-1.3.0
collected 1 item

tests/test_dataset_common.py F                                                                                [100%]

=================================================================================================== FAILURES ====================================================================================================
________________________________________________________________________ LocalDatasetTest.test_load_dataset_all_configs_code_search_net _________________________________________________________________________
self = <tests.test_dataset_common.LocalDatasetTest testMethod=test_load_dataset_all_configs_code_search_net>, dataset_name = 'code_search_net'

    @slow
    def test_load_dataset_all_configs(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True)

tests/test_dataset_common.py:237:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:198: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
--------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------
Downloading and preparing dataset code_search_net/all (download: 1.00 MiB, generated: 1.00 MiB, post-processed: Unknown size, total: 2.00 MiB) to /tmp/tmppx78sj24/code_search_net/all/1.0.0...
Dataset code_search_net downloaded and prepared to /tmp/tmppx78sj24/code_search_net/all/1.0.0. Subsequent calls will reuse this data.
--------------------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------------------
... (irrelevant info - Deprecation warnings)
============================================================================================ short test summary info ============================================================================================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_code_search_net - AssertionError: False is not true
========================================================================================= 1 failed, 4 warnings in 3.00s ========================================================================================
```

## Note : Data structure in S3
The data is stored on S3, and organized by programming languages.
It is stored in the following repository structure:
```
.
├── <language_name>  # e.g. python
│   └── final
│       └── jsonl
│           ├── test
│           │   └── <language_name>_test_0.jsonl.gz
│           ├── train
│           │   ├── <language_name>_train_0.jsonl.gz
│           │   ├── <language_name>_train_1.jsonl.gz
│           │   ├── ...
│           │   └── <language_name>_train_n.jsonl.gz
│           └── valid
│               └── <language_name>_valid_0.jsonl.gz
├── <language_name>_dedupe_definitions_v2.pkl
└── <language_name>_licenses.pkl
```


'iwslt2017-ro-nl', cannot be downloaded

dataset bug

A bug in a dataset script provided in the library

the same issue with datasets.load_dataset("iwslt2017", 'iwslt2017-en-nl', split=split), .....

even with setting master like the following command, still remains

datasets.load_dataset("iwslt2017", 'iwslt2017-en-nl', split="train", script_version="master")

Looks like the data has been moved from its original location to google drive

New url: https://drive.google.com/u/0/uc?id=12ycYSzLIG253AFN35Y6qoyf9wtkOjakp&export=download

Hi
I am trying
`>>> datasets.load_dataset("iwslt2017", 'iwslt2017-ro-nl', split="train")`

getting this error thank you for your help
```
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Downloading and preparing dataset iwsl_t217/iwslt2017-ro-nl (download: 314.07 MiB, generated: 39.92 MiB, post-processed: Unknown size, total: 354.00 MiB) to /idiap/temp/rkarimi/cache_home_1/datasets/iwsl_t217/iwslt2017-ro-nl/1.0.0/cca6935a0851a8ceac1202a62c958738bdfa23c57a51bc52ac1c5ebd2aa172cd...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/iwslt2017/cca6935a0851a8ceac1202a62c958738bdfa23c57a51bc52ac1c5ebd2aa172cd/iwslt2017.py", line 118, in _split_generators
    dl_dir = dl_manager.download_and_extract(MULTI_URL)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 254, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 179, in download
    num_proc=download_config.num_proc,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 216, in map_nested
    return function(data_struct)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 477, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz

```