helmo/github-issues huggingface dataset:
Add GoEmotions
State: closed | Author: joeddav (CONTRIBUTOR)

Adds the GoEmotions dataset, a nice emotion classification dataset with 27 (multi-)label annotations on reddit comments. Includes both a large raw version and a narrowed version with predefined train/test/val splits, which I've included as separate configs with the latter as a default.

- Webpage/repo: https://github.com/google-research/google-research/tree/master/goemotions
- Paper: https://arxiv.org/abs/2005.00547

Comments: Whoops, didn't mean for that to be merged yet (my bad). I'm reaching out to the authors since we'd like their feedback on the best way to have the `author` field anonymized or removed. Will send a patch once they get back to me.
Adding NCHLT dataset
State: closed | Author: Narsil (CONTRIBUTOR)

https://repo.sadilar.org/handle/20.500.12185/7/discover?filtertype_0=database&filtertype_1=title&filter_relational_operator_1=contains&filter_relational_operator_0=equals&filter_1=&filter_0=Monolingual+Text+Corpora%3A+Annotated&filtertype=project&filter_relational_operator=equals&filter=NCHLT+Text+II

Comments: merging since the CI is fixed on master
Add KorNLU
State: closed | Author: sumanthd17 (CONTRIBUTOR)

Added Korean NLU datasets. The link to the dataset can be found [here](https://github.com/kakaobrain/KorNLUDatasets) and the paper can be found [here](https://arxiv.org/abs/2004.03289)

**Note**: The MNLI tsv file is broken, so this code currently excludes the file. Please suggest other alternative if any @lhoestq 

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

Comments: the CI error about `social_bias_frames` is fixed on master so it's fine

created new [PR](https://github.com/huggingface/datasets/pull/1062)

looks like this PR includes many changes to other files that the ones related to KorNLU
Could you create another branch and another PR please ?

Wow crazy timing

hahahaha
Dataset.map() turns tensors into lists?
State: closed | Author: tombosc (NONE)

I apply `Dataset.map()` to a function that returns a dict of torch tensors (like a tokenizer from the repo transformers). However, in the mapped dataset, these tensors have turned to lists!

```import datasets
import torch  
from datasets import load_dataset                                                                                                                 
print("version datasets", datasets.__version__)

dataset = load_dataset("snli", split='train[0:50]')  

def tokenizer_fn(example):
    # actually uses a tokenizer which does something like:
    return {'input_ids': torch.tensor([[0, 1, 2]])}

print("First item in dataset:\n", dataset[0])
tokenized = tokenizer_fn(dataset[0])
print("Tokenized hyp:\n", tokenized)
dataset_tok = dataset.map(tokenizer_fn, batched=False,
        remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
dataset_tok = dataset.map(tokenizer_fn, batched=False,
                          remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
```

The output is:

```
version datasets 1.1.3
Reusing dataset snli (/home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c)
First item in dataset:
 {'premise': 'A person on a horse jumps over a broken down airplane.', 'hypothesis': 'A person is training his horse for a competition.', 'label': 1}
Tokenized hyp:
 {'input_ids': tensor([[0, 1, 2]])}
Loading cached processed dataset at /home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c/cache-fe38f449fe9ac46f.arrow
Tokenized using map:
 {'input_ids': [[0, 1, 2]]}
<class 'torch.Tensor'> <class 'list'>
```

Or am I doing something wrong?


Comments: A solution is to have the tokenizer return a list instead of a tensor, and then use `dataset_tok.set_format(type = 'torch')` to convert that list into a tensor. Still not sure if bug.

It is expected behavior, you should set the format to `"torch"` as you mentioned to get pytorch tensors back.
By default datasets returns pure python objects.
Add Big Patent dataset
State: closed | Author: mattbui (CONTRIBUTOR)

- More info on the dataset: https://evasharma.github.io/bigpatent/
- There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.
- ~Currently, there are no dummy data for this dataset yet as I'm facing some problems with generating them. I'm trying to add them later.~

Comments: Looks like this PR include changes about many other files than the ones related to big patent.
Could you create another branch and another PR ?

@lhoestq Just created a new PR here: https://github.com/huggingface/datasets/pull/1087
Add UN Universal Declaration of Human Rights (UDHR)
State: closed | Author: joeddav (CONTRIBUTOR)

Universal declaration of human rights with translations in 464 languages and dialects.

- UN page: https://www.ohchr.org/EN/UDHR/Pages/UDHRIndex.aspx
- Raw data source: https://unicode.org/udhr/index.html

Each instance of the dataset corresponds to one translation of the document. Since there's only one instance per language (and because there are 500 languages so the dummy data would be messy), I opted to just include them all under the same single config. I wasn't able to find any kind of license so I just copied the copyright notice.

I was pretty careful careful generating the language tags so they _should_ all be correct & consistent BCP-47 codes per the docs.
Fix docs indentation issues
State: closed | Author: albertvillanova (MEMBER)

Replace tabs with spaces.

Comments: is this an issue ?

Yes @lhoestq, look at the docs site. For example, in https://huggingface.co/docs/datasets/add_dataset.html, look at the indentation in the code block under the sentence:
> Here are the features of the SQuAD dataset for instance, which is taken from the squad dataset loading script:
add wiki_hop
State: closed | Author: patil-suraj (CONTRIBUTOR)

This PR adds the WikiHop dataset from the QAngaroo multi hop reading comprehension datasets

More info:
http://qangaroo.cs.ucl.ac.uk/index.html


Comments: Also the dummy data files are quite big (500KB)
If you could reduce that that would be nice (just look at the files inside and remove unecessary chunks of texts)
in general dummy data are just a few KB and we suggest to not get higher than 50KB

Having light dummy data makes the repo faster to clone
add scb_mt_enth_2020
State: closed | Author: cstorm125 (CONTRIBUTOR)

## scb-mt-en-th-2020: A Large English-Thai Parallel Corpus

The primary objective of our work is to build a large-scale English-Thai dataset for machine translation.
We construct an English-Thai machine translation dataset with over 1 million segment pairs, curated from various sources,
namely news, Wikipedia articles, SMS messages, task-based dialogs, web-crawled data and government documents.
Methodology for gathering data, building parallel texts and removing noisy sentence pairs are presented in a reproducible manner.
We train machine translation models based on this dataset. Our models' performance are comparable to that of
Google Translation API (as of May 2020) for Thai-English and outperform Google when the Open Parallel Corpus (OPUS) is
included in the training data for both Thai-English and English-Thai translation.
The dataset, pre-trained models, and source code to reproduce our work are available for public use.
Add support for ".txm" format
State: closed | Author: albertvillanova (MEMBER)

In dummy data generation, add support for XML-like ".txm" file format.

Also support filenames with additional compression extension: ".txm.gz".

Comments: Neat! Looks like you need a rebase and then should be good to go :) 

Done, @yjernite, @lhoestq.

If you agree, we could merge this.

Hi ! yes sure :) can you just merge master into this branch before we merge ?

Done @lhoestq
IIT B English to Hindi machine translation dataset
State: closed | Author: spatil6 (CONTRIBUTOR)

Adding IIT Bombay English-Hindi Corpus dataset
more info : http://www.cfilt.iitb.ac.in/iitb_parallel/

Comments: Please note that this dataset is actually behind a form that one needs to fill. However, the link is direct. I'm not sure what should the approach be in this case.

also pinging @thomwolf 
The dataset webpage returns a form when trying to download the dataset (form here : http://www.cfilt.iitb.ac.in/iitb_parallel/dataset.html).
However the url we get with the form can be used for the dataset script.
Should we ask the authors or use the urls this way ?

> also pinging @thomwolf
> The dataset webpage returns a form when trying to download the dataset (form here : http://www.cfilt.iitb.ac.in/iitb_parallel/dataset.html).
> However the url we get with the form can be used for the dataset script.
> Should we ask the authors or use the urls this way ?

I had discussion on this with @thomwolf . We have already sent email to author of this dataset.

Hi @spatil6 !
Any news from the authors ?

IIT B folks will add this dataset to repo.
add crows_pairs
State: closed | Author: patil-suraj (CONTRIBUTOR)

This PR adds CrowS-Pairs datasets.

More info:
https://github.com/nyu-mll/crows-pairs/
https://arxiv.org/pdf/2010.00133.pdf

Comments: looks good now :) wdyt @yjernite ?

Looks good to merge for me, can edit the dataset card later if required. Merging
allegro_reviews dataset 
State: closed | Author: abecadel (CONTRIBUTOR)

- **Name:** *allegro_reviews*
- **Description:** *Allegro Reviews is a sentiment analysis dataset, consisting of 11,588 product reviews written in Polish and extracted from Allegro.pl - a popular e-commerce marketplace. Each review contains at least 50 words and has a rating on a scale from one (negative review) to five (positive review).*
- **Data:** *https://github.com/allegro/klejbenchmark-allegroreviews*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*
Add PEC
State: closed | Author: zhongpeixiang (CONTRIBUTOR)

A persona-based empathetic conversation dataset.

Comments: I'm a bit frustrated now to get this right.

Hey @zhongpeixiang!
Really nice addition here!

Did you officially joined the sprint by posting [on the forum thread](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176) and joining our slack?
I can't seem to find you there! Should I add you directly with your gmail address?

> Hey @zhongpeixiang!
> Really nice addition here!
> 
> Did you officially joined the sprint by posting [on the forum thread](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176) and joining our slack?
> I can't seem to find you there! Should I add you directly with your gmail address?

Thank you for the invitation. This initiative is awesome. Sadly I’m occupied by my thesis writing this month. Good luck 🤗

As you want @zhongpeixiang (I was maybe not clear but that just mean that by posting on the forum thread that you participated in the current event you will get a special gift (a tee-shirt) for the contribution that you have already done here :-) Nothing more to do)

> As you want @zhongpeixiang (I was maybe not clear but that just mean that by posting on the forum thread that you participated in the current event you will get a special gift (a tee-shirt) for the contribution that you have already done here :-) Nothing more to do)

Oh, I misunderstood the post. I'm glad to join.
Hi
State: closed | Author: suemori87 (NONE)

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Add Sesotho Ner
State: closed | Author: yvonnegitau (CONTRIBUTOR)


Comments: looks like this PR include changes to other files (sepedi)
could you try to only include the files related to the addition of sesotho ner ?

I think i need to clean up my local repo. I am committing everything a fresh after sepedi

Feel free to ping me when yuo have a clean PR and it's ready to review :)

closing in favor of #1114
Add ZEST: ZEroShot learning from Task descriptions
State: closed | Author: joeddav (CONTRIBUTOR)

Adds the ZEST dataset on zero-shot learning from task descriptions from AI2.

- Webpage: https://allenai.org/data/zest
- Paper: https://arxiv.org/abs/2011.08115

The nature of this dataset made the supported task tags tricky if you wouldn't mind giving any feedback @yjernite. Also let me know if you think we should have a `other-task-generalization` or something like that...

Comments: Looks good to me, we can ping the authors for more info later. And yes apply `other-task` labels liberally, we can sort them out later :) 

Looks ready to merge when you're ready @joeddav
Add Schema Guided Dialogue dataset
State: closed | Author: yjernite (MEMBER)

This PR adds the Schema Guided Dialogue dataset created for the DSTC8 challenge
- https://github.com/google-research-datasets/dstc8-schema-guided-dialogue

A bit simpler than MultiWOZ, the only tricky thing is the sequence of dictionaries that had to be linearized. There is a config for the data proper, and a config for the schemas.
Add Gutenberg time references dataset
State: closed | Author: TevenLeScao (CONTRIBUTOR)

This PR adds the gutenberg_time dataset: https://arxiv.org/abs/2011.04124

Comments: Description: "A clean data resource containing all explicit time references in a dataset of 52,183 novels whose full text is available via Project Gutenberg and the Hathi Trust Digital Library 2." > This is just the Gutenberg part.

Also, the paragraph at the top of the file would make a good Dataset Summary in the README :)
Add CLINC150 dataset
State: closed | Author: sumanthd17 (CONTRIBUTOR)

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data
Add SciTLDR Dataset (Take 2)
State: closed | Author: bharatr21 (CONTRIBUTOR)

Adds the SciTLDR Dataset by AI2
Added the `README.md` card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents

Continued from #986 

Comments: @lhoestq please review this PR when you get free

If the CI fails just because of `RemoteDatasetTest` errors it's fine, they're fixed on master

> If the CI fails just because of `RemoteDatasetTest` errors it's fine, they're fixed on master

The same 3 tests are failing again :(
```
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
```

One trick if you want to add more datasets to avoid these errors : you can just rebase the master branch of your fork from the master branch of the repo. Then each time you make a new branch from master on your fork, it will include the fix for these errors

> One trick if you want to add more datasets to avoid these errors : you can just rebase the master branch of your fork from the master branch of the repo. Then each time you make a new branch from master on your fork, it will include the fix for these errors

Yes, I almost always do that, but somehow seems even this branch got old 😓 
I also do the following if I directly create a new branch locally: `git checkout -b <branchname> upstream/master` so it stays up-to date irrespective of my fork, still don't know how this crept in again

Merging this one since the CI is fixed on master
Add Bilingual Corpus of Arabic-English Parallel Tweets
State: closed | Author: sumanthd17 (CONTRIBUTOR)

Added Bilingual Corpus of Arabic-English Parallel Tweets. The link to the dataset can be found [here](https://alt.qcri.org/wp-content/uploads/2020/08/Bilingual-Corpus-of-Arabic-English-Parallel-Tweets.zip) and the paper can be found [here](https://www.aclweb.org/anthology/2020.bucc-1.3.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

Comments: IMO, the problem with this dataset is that it is not really a text/nlp dataset. These are just collections of tweet ids. So, ultimately, one needs to crawl twitter to get the actual text.

That's true.



at least it's clear in the description that one needs to collect the tweets : 
```
This resource is a result of a generic method for collecting parallel tweets.
```

Looks like this is failing for other datasets. Should I rebase it and push again?
Also rebasing and pushing is reflecting changes in many other files (ultimately forcing me to open a new branch and a new PR) any way to avoid this?

No let me merge this one directly, it's fine

merging since the CI is fixed on master
how large datasets are handled under the hood 
State: closed | Author: rabeehkarimimahabadi (NONE)

Hi
I want to use multiple large datasets with a mapping style dataloader, where they cannot fit into memory, could you tell me how you handled the datasets under the hood? is this you bring all in memory in case of mapping style ones? or is this some sharding under the hood and you bring in memory when necessary, thanks 

Comments: This library uses Apache Arrow under the hood to store datasets on disk.
The advantage of Apache Arrow is that it allows to memory map the dataset. This allows to load datasets bigger than memory and with almost no RAM usage. It also offers excellent I/O speed.

For example when you access one element or one batch
```python
from datasets import load_dataset

squad = load_dataset("squad", split="train")
first_element = squad[0]
one_batch = squad[:8]
```

then only this element/batch is loaded in memory, while the rest of the dataset is memory mapped.

How can we change how much data is loaded to memory with Arrow? I think that I am having some performance issue with it. When Arrow loads the data from disk it does it in multiprocess? It's almost twice slower training with arrow than in memory.

EDIT:
My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.

> How can we change how much data is loaded to memory with Arrow? I think that I am having some performance issue with it. When Arrow loads the data from disk it does it in multiprocess? It's almost twice slower training with arrow than in memory.

Loading arrow data from disk is done with memory-mapping. This allows to load huge datasets without filling your RAM.
Memory mapping is almost instantaneous and is done within one process.

Then, the speed of querying examples from the dataset is I/O bounded depending on your disk. If it's an SSD then fetching examples from the dataset will be very fast.
But since the I/O speed of an SSD is lower than the one of RAM it's expected to be slower to fetch data from disk than from memory.
Still, if you load the dataset in different processes then it can be faster but there will still be the I/O bottleneck of the disk.

> EDIT:
> My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.

Ok let me know if that helps !
add generated_reviews_enth
State: closed | Author: cstorm125 (CONTRIBUTOR)

`generated_reviews_enth` is created as part of [scb-mt-en-th-2020](https://arxiv.org/pdf/2007.03541.pdf) for machine translation task. This dataset (referred to as `generated_reviews_yn` in [scb-mt-en-th-2020](https://arxiv.org/pdf/2007.03541.pdf)) are English product reviews generated by [CTRL](https://arxiv.org/abs/1909.05858), translated by Google Translate API and annotated as accepted or rejected (`correct`) based on fluency and adequacy of the translation by human annotators. This allows it to be used for English-to-Thai translation quality esitmation (binary label), machine translation, and sentiment analysis.
Microsoft CodeXGlue
State: closed | Author: madlag (CONTRIBUTOR)

Datasets from https://github.com/microsoft/CodeXGLUE

This contains 13 datasets:

code_x_glue_cc_clone_detection_big_clone_bench
code_x_glue_cc_clone_detection_poj_104
code_x_glue_cc_cloze_testing_all
code_x_glue_cc_cloze_testing_maxmin
code_x_glue_cc_code_completion_line
code_x_glue_cc_code_completion_token
code_x_glue_cc_code_refinement
code_x_glue_cc_code_to_code_trans
code_x_glue_cc_defect_detection
code_x_glue_ct_code_to_text
code_x_glue_tc_nl_code_search_adv
code_x_glue_tc_text_to_code
code_x_glue_tt_text_to_text


Comments: #978 is working on adding code refinement

maybe we should keep the CodeXGlue benchmark (as glue) and don't merge the code_refinement dataset proposed in #978 ?

cc @reshinthadithyan

Hi @madlag and @lhoestq , I am extremely interested in getting this dataset into HF's library as I research in this area a lot. I see that it hasn't been updated in a while, but it is very close to being finished. If no one is currently working on this, I'd be happy to do any final touches that might be needed to get this merged.

Hi @ncoop57 ! Thanks for your interest and sorry for the inactivity on this PR.
Sure feel free to create another PR to continue this one ! This one was really close to being merged so I think it won't require that much changes. In addition to my previous comments, there should also be a "Contributions" subsection (see the template of the README [here](https://github.com/huggingface/datasets/blob/master/templates/README.md))

Superseded by https://github.com/huggingface/datasets/pull/2357 .
Problem downloading amazon_reviews_multi
State: closed | Author: hfawaz (CONTRIBUTOR)

Thanks for adding the dataset. 
After trying to load the dataset, I am getting the following error: 
`ConnectionError: Couldn't reach https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_fr_train.json
`
I used the following code to load the dataset: 
`load_dataset(
            dataset_name,
            "all_languages",
            cache_dir=".data"
        )`

I am using version 1.1.3 of `datasets`

Note that I can perform a successfull `wget https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_fr_train.json`

Comments: Hi @hfawaz ! This is working fine for me. Is it a repeated occurence? Have you tried from the latest verion?

Hi, it seems a connection problem. 
Now it says: 
`ConnectionError: Couldn't reach https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_ja_train.json`