konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
add wmt mlqe 2020 shared task

First commit for Shared task 1 (wmt_mlqw_task1) of WMT20 MLQE (quality estimation of machine translation)
Note that I copied the tags in the README for only one (of the 7 configurations): `en-de`.
There is one configuration for each pair of languages.

re-created in #1218 because this was too messy
Adding initial version of cord-19 dataset

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.

### TODO:
- [x] add more metadata
- [x] add full text
- [x] add pre-computed document embedding

Hi @ggdupont !
Have you had a chance to take a look at my suggestions ?
Feel free to ping me if you have questions or when you're ready for a review

> Hi @ggdupont !
> Have you had a chance to take a look at my suggestions ?
> Feel free to ping me if you have questions or when you're ready for a review

Yes I did, just busy period (and no time on weekend right now ;-) )

With some delay, reduced the dummy data and had t rebase

Thanks !

It looks like the rebase messed up the github diff for this PR (2.000+ files changed)
Could you create another branch and another PR please ?

Cleaned PR: https://github.com/huggingface/datasets/pull/1850
Adding babi dataset

Adding the English version of bAbI.

Samples are taken from ParlAI for consistency with the main users at the moment.

Supersede #945 (problem with the rebase) and adresses the issues mentioned in the review (dummy data are smaller now and code comments are fixed).

This is ok now @lhoestq

I've included the tweak to `dummy_data` to only use the data transmitted to `_generate_examples` by default (it only do that if it can find at least one path to an existing file in the `gen_kwargs` and this can be unactivated with a flag).

Should I extract it in another PR or is it ok like this?

Nice !
Could you add the dummy data generation trick in another PR ?
I think we can also extend it to make it work not only with data files paths but also with data directories (sometimes it's one of the parent directory that is passed to gen_kwargs, not the actual path to the file).
This will help a lot to make the dummy data lighter !

This PR can be closed due to #2053 @lhoestq
Add Urdu fake news dataset.

Added Urdu fake news dataset. More information about the dataset can be found <a href="https://github.com/MaazAmjad/Datasets-for-Urdu-news">here</a>.

@lhoestq looks like a lot of files were updated... shall I create a new PR?

Hi @chaitnayabasava ! you can try rebasing and see if that fixes the number of files changed, otherwise please do open a new PR with only the relevant files and close this one :) 

Created a new PR #1230.
closing this one :)
Fix incorrect MRQA train+SQuAD URL

Fix issue #1115 


Thanks ! could you regenerate the dataset_infos.json file ?

```
datasets-cli test ./datasets/mrqa --save_infos --all_configs --ignore_verifications
```

also cc @VictorSanh 

Oooops, good catch @jimmycode 

> Thanks ! could you regenerate the dataset_infos.json file ?
> 
> ```
> datasets-cli test ./datasets/mrqa --save_infos --all_configs --ignore_verifications
> ```
> 
> also cc @VictorSanh

Updated the `dataset_infos.json` file.
add dbpedia_14 dataset

This dataset corresponds to the DBpedia dataset requested in https://github.com/huggingface/datasets/issues/353.

Thanks for the review. 
Cheers!

Hi @hfawaz, this week we are doing the 🤗 `datasets` sprint (see some details [here](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176)).

Nothing more to do on your side but it means that if you register on the thread I linked above, you can have some goodies for the present dataset that you have already added (and a special goodie if you want to spend more time and add 2 other datasets as well).

If you want to join, just tell me (or post on the thread on the HuggingFace forum: https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176)

Hello @thomwolf 
Thanks for the feedback and for this invitation, indeed I would be glad to join you guys (you can add me). 
I will see if I have the time to implement a couple of datasets. 
Cheers! 

@hfawaz invited you to the slack with your uha email.

Check your spam folder if you can't find the invitation :)

Oh thanks, but can you invite me on my gmail: hassanismailfawaz@gmail.com 
UHA is my old organization, I haven't had the time to update my online profiles yet.
Thank you
Incorrect URL for MRQA SQuAD train subset

https://github.com/huggingface/datasets/blob/4ef4c8f8b7a60e35c6fa21115fca9faae91c9f74/datasets/mrqa/mrqa.py#L53

The URL for `train+SQuAD` subset of MRQA points to the dev set instead of train set. It should be `https://s3.us-east-2.amazonaws.com/mrqa/release/v2/train/SQuAD.jsonl.gz`.

good catch !
Initial version of cord-19 dataset from AllenAI with only the abstract

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [ ] Both tests for the real data and the dummy data pass.

### TODO:
- [ ] add more metadata
- [ ] add full text
- [ ] add pre-computed document embedding

too ugly, I'll make a clean one
Using a feature named "_type" fails with certain operations

A column named `_type` leads to a `TypeError: unhashable type: 'dict'` for certain operations:
```python
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({"_type": ["whatever"]}).map()
concatenate_datasets([ds])
# or simply
Dataset(ds._data)
```
Context: We are using datasets to persist data coming from elasticsearch to feed to our pipeline, and elasticsearch has a `_type` field, hence the strange name of the column.

Not sure if you wish to support this specific column name, but if you do i would be happy to try a fix and provide a PR. I already had a look into it and i think the culprit is the `datasets.features.generate_from_dict` function. It uses the hard coded `_type` string to figure out if it reached the end of the nested feature object from a serialized dict.

Best wishes and keep up the awesome work!

Thanks for reporting !

Indeed this is a keyword in the library that is used to encode/decode features to a python dictionary that we can save/load to json.
We can probably change `_type` to something that is less likely to collide with user feature names.
In this case we would want something backward compatible though.

Feel free to try a fix and open a PR, and to ping me if I can help :)
Add arsentd_lev dataset

Add The Arabic Sentiment Twitter Dataset for Levantine dialect (ArSenTD-LEV)

Paper: [ArSentD-LEV: A Multi-Topic Corpus for Target-based Sentiment Analysis in Arabic Levantine Tweets](https://arxiv.org/abs/1906.01830)
Homepage: http://oma-project.com/

thanks ! can you also regenerate the dataset_infos.json file please ?
Add support to download kaggle datasets

We can use API key

Hey, I think this is great idea. Any plan to integrate kaggle private datasets loading to `datasets`?

The workflow for downloading a Kaggle dataset and turning it into an HF dataset is pretty simple:
```python
!kaggle datasets download -p path
ds = load_dataset(path)
```

Native support would make our download logic even more complex, and I don't think this is a good idea considering this particular feature is not requested often. 

PS: Kaggle should integrate their API with `fsspec` to allow us to use a common interface if they are interested in tighter integrations
Add Big Patent dataset

* More info on the dataset: https://evasharma.github.io/bigpatent/
* There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.

@lhoestq reduced the dummy data size to around 19MB in total and added the dataset card.

@lhoestq so I ended up removing all the nested JSON objects in the gz datafile and keep only one object with minimal content: `{"publication_number": "US-8230922-B2", "abstract": "dummy abstract", "application_number": "US-201113163519-A", "description": "dummy description"}`. 

They're reduced to 35KB in total (2.5KB per domain and 17.5KB for all domains), hopefully, they're small enough.
adding cdt dataset

- **Name:** *Cyberbullying Detection Task*
- **Description:** *The Cyberbullying Detection task was part of 2019 edition of PolEval competition. The goal is to predict if a given Twitter message contains a cyberbullying (harmful) content.*
- **Data:** *https://github.com/ptaszynski/cyberbullying-Polish*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*

> Thanks for adding this one !
> 
> I left a few comments
> 
> after the change you'll need to regenerate the dataset_infos.json file as well

dataset_infos.json regenerated

looks like this PR includes changes to many files other that the ones for CDT
could you create another branch and another PR please ?
Add WikiANN NER dataset

This PR adds the full set of 176 languages from the balanced train/dev/test splits of WikiANN / PAN-X from: https://github.com/afshinrahimi/mmner

Until now, only 40 of these languages were available in `datasets` as part of the XTREME benchmark

Courtesy of the dataset author, we can now download this dataset from a Dropbox URL without needing a manual download anymore 🥳, so at some point it would be worth updating the PAN-X subset of XTREME as well 😄 

Link to gist with some snippets for producing dummy data: https://gist.github.com/lewtun/5b93294ab6dbcf59d1493dbe2cfd6bb9

P.S. @yjernite I think I was confused about needing to generate a set of YAML tags per config, so ended up just adding a single one in the README.

Dataset card added, so ready for review!
Add Pubmed (citation + abstract) dataset (2020).


LGTM! ftp addition looks fine but maybe have a look @thomwolf ?

It's not finished yet, I need to run the tests on the full dataset (it was running this weekend, there is an error somewhere deep)


@yjernite Ready for review !
@thomwolf 

So I tried to follow closely the original format that means I still had to drop information (namely tags on elements are simply discarded for now but they don't seem to carry critical information).
Some elements are also discarded they tend to not come up often:
 - The most notable is Author affiliation, which seems to be all over the place in terms of what it look meaning it's hard to actually get a consistent format.
 - Journal is the same, all the elements in there can be wildly different so I decided to drop it for now instead of trying to figure out a way to have a common presentation. (the DOI and medline ID are present so it can be recovered).

I think this PR could go as it but we probably should add a way to get easier information to use with a config.
For instance `{"title": "string", "abstract": "string", "authors": List[str], "substances": List[str]}` maybe ? (substances for instance is a tricky one, some substances only have an international identifier, some have simply a common name, some both)

It's relatively easy to do I think it's mostly discarding other fields and renaming some deep structure into a flat one.

Look ok to me but @lhoestq is the master on the Download Manager side
Not support links with 302 redirect 

I have an issue adding this download link https://github.com/jitkapat/thailitcorpus/releases/download/v.2.0/tlc_v.2.0.tar.gz

it might be because it is not a direct link (it returns 302 and redirects to aws that returns 403 for head requests). 

```
r.head("https://github.com/jitkapat/thailitcorpus/releases/download/v.2.0/tlc_v.2.0.tar.gz", allow_redirects=True)                                      
# <Response [403]>
```

Hi !
This kind of links is now supported by the library since #1316

> Hi !
> This kind of links is now supported by the library since #1316

I updated links in TLC datasets to be the github links in this pull request 
 https://github.com/huggingface/datasets/pull/1737

Everything works now. Thank you.
Add KorNLU dataset

Added Korean NLU datasets. The link to the dataset can be found [here](https://github.com/kakaobrain/KorNLUDatasets) and the paper can be found [here](https://arxiv.org/abs/2004.03289)

**Note**: The MNLI tsv file is broken, so this code currently excludes the file. Please suggest other alternative if any @lhoestq 

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

Nice thank you !
Can you regenerate the dataset_infos.json file ? Since we changed the features we must update it

Then I think we'll be good to merge :)
Fix squad V2 metric script

The current squad v2 metric doesn't work with the squad (v1 or v2) datasets. The script is copied from `squad_evaluate` in transformers that requires the labels (with multiple answers) to be like this:
```
references = [{'id': 'a', 'answers': [
    {'text': 'Denver Broncos', 'answer_start': 177},
    {'text': 'Denver Broncos', 'answer_start': 177}
]}]
```
while the dataset had references like this:
```
references = [{'id': 'a', 'answers': 
    {'text': ['Denver Broncos' 'Denver Broncos'], 'answer_start': [177, 177]}
}]
```

Using one or the other format fails with the current squad v2 metric:
```
from datasets import load_metric
metric = load_metric("squad_v2")
predictions = [{'id': 'a', 'prediction_text': 'Denver Broncos', 'no_answer_probability': 0.0}]
references = [{'id': 'a', 'answers': [
    {'text': 'Denver Broncos', 'answer_start': 177},
    {'text': 'Denver Broncos', 'answer_start': 177}
]}]
metric.compute(predictions=predictions, references=references)
```
fails as well as
```
from datasets import load_metric
metric = load_metric("squad_v2")
predictions = [{'id': 'a', 'prediction_text': 'Denver Broncos', 'no_answer_probability': 0.0}]
references = [{'id': 'a', 'answers': 
    {'text': ['Denver Broncos' 'Denver Broncos'], 'answer_start': [177, 177]}
}]
metric.compute(predictions=predictions, references=references)
```

This is because arrow reformats the references behind the scene.

With this PR (tested locally), both the snippets up there work and return proper results.

The script with changes is used and tested in [#8924](https://github.com/huggingface/transformers/pull/8924). It gives the same results as the old `evaluate_squad` function when used on the same predictions.

merging since the CI is fixed on master
Add hebrew-sentiment

hebrew-sentiment dataset is ready! (including tests, tags etc)

@elronbandel it looks like something went wrong with the renaming, as the old files are still in the PR. Can you `git rm datasets/hebrew-sentiment` ?

merging since the CI is fixed on master

This is the old version of the data.
Here is the fixed version.
https://github.com/OnlpLab/Hebrew-Sentiment-Data

I hope I would find time to open a PR. I think it supposed to be only to change the data path 

Cool ! Sure feel free to open a PR if you have some time :) and feel free to ping me for review or if you have questions
Fix dataset URL and file names, and add column name in "Social Bias Frames" dataset

# Why I did
When I use "social_bias_frames" datasets in this library, I got 404 Errors.
So, I fixed this error and another some problems that I faced to use the dataset.

# What I did
* Modify this dataset URL
* Modify this dataset file names
* Add a "dataSource" column

Thank you!

Thanks a lot, looks good!
Add GoEmotions

Adds the GoEmotions dataset, a nice emotion classification dataset with 27 (multi-)label annotations on reddit comments. Includes both a large raw version and a narrowed version with predefined train/test/val splits, which I've included as separate configs with the latter as a default.

- Webpage/repo: https://github.com/google-research/google-research/tree/master/goemotions
- Paper: https://arxiv.org/abs/2005.00547

Whoops, didn't mean for that to be merged yet (my bad). I'm reaching out to the authors since we'd like their feedback on the best way to have the `author` field anonymized or removed. Will send a patch once they get back to me.
Adding NCHLT dataset

https://repo.sadilar.org/handle/20.500.12185/7/discover?filtertype_0=database&filtertype_1=title&filter_relational_operator_1=contains&filter_relational_operator_0=equals&filter_1=&filter_0=Monolingual+Text+Corpora%3A+Annotated&filtertype=project&filter_relational_operator=equals&filter=NCHLT+Text+II

merging since the CI is fixed on master
Add KorNLU

Added Korean NLU datasets. The link to the dataset can be found [here](https://github.com/kakaobrain/KorNLUDatasets) and the paper can be found [here](https://arxiv.org/abs/2004.03289)

**Note**: The MNLI tsv file is broken, so this code currently excludes the file. Please suggest other alternative if any @lhoestq 

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

the CI error about `social_bias_frames` is fixed on master so it's fine

created new [PR](https://github.com/huggingface/datasets/pull/1062)

looks like this PR includes many changes to other files that the ones related to KorNLU
Could you create another branch and another PR please ?

Wow crazy timing

hahahaha
Dataset.map() turns tensors into lists?

I apply `Dataset.map()` to a function that returns a dict of torch tensors (like a tokenizer from the repo transformers). However, in the mapped dataset, these tensors have turned to lists!

```import datasets
import torch  
from datasets import load_dataset                                                                                                                 
print("version datasets", datasets.__version__)

dataset = load_dataset("snli", split='train[0:50]')  

def tokenizer_fn(example):
    # actually uses a tokenizer which does something like:
    return {'input_ids': torch.tensor([[0, 1, 2]])}

print("First item in dataset:\n", dataset[0])
tokenized = tokenizer_fn(dataset[0])
print("Tokenized hyp:\n", tokenized)
dataset_tok = dataset.map(tokenizer_fn, batched=False,
        remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
dataset_tok = dataset.map(tokenizer_fn, batched=False,
                          remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
```

The output is:

```
version datasets 1.1.3
Reusing dataset snli (/home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c)
First item in dataset:
 {'premise': 'A person on a horse jumps over a broken down airplane.', 'hypothesis': 'A person is training his horse for a competition.', 'label': 1}
Tokenized hyp:
 {'input_ids': tensor([[0, 1, 2]])}
Loading cached processed dataset at /home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c/cache-fe38f449fe9ac46f.arrow
Tokenized using map:
 {'input_ids': [[0, 1, 2]]}
<class 'torch.Tensor'> <class 'list'>
```

Or am I doing something wrong?


A solution is to have the tokenizer return a list instead of a tensor, and then use `dataset_tok.set_format(type = 'torch')` to convert that list into a tensor. Still not sure if bug.

It is expected behavior, you should set the format to `"torch"` as you mentioned to get pytorch tensors back.
By default datasets returns pure python objects.
Add Big Patent dataset

- More info on the dataset: https://evasharma.github.io/bigpatent/
- There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.
- ~Currently, there are no dummy data for this dataset yet as I'm facing some problems with generating them. I'm trying to add them later.~

Looks like this PR include changes about many other files than the ones related to big patent.
Could you create another branch and another PR ?

@lhoestq Just created a new PR here: https://github.com/huggingface/datasets/pull/1087
add wiki_hop

This PR adds the WikiHop dataset from the QAngaroo multi hop reading comprehension datasets

More info:
http://qangaroo.cs.ucl.ac.uk/index.html


Also the dummy data files are quite big (500KB)
If you could reduce that that would be nice (just look at the files inside and remove unecessary chunks of texts)
in general dummy data are just a few KB and we suggest to not get higher than 50KB

Having light dummy data makes the repo faster to clone
Add support for ".txm" format

In dummy data generation, add support for XML-like ".txm" file format.

Also support filenames with additional compression extension: ".txm.gz".

Neat! Looks like you need a rebase and then should be good to go :) 

Done, @yjernite, @lhoestq.

If you agree, we could merge this.

Hi ! yes sure :) can you just merge master into this branch before we merge ?

Done @lhoestq
IIT B English to Hindi machine translation dataset

Adding IIT Bombay English-Hindi Corpus dataset
more info : http://www.cfilt.iitb.ac.in/iitb_parallel/

Please note that this dataset is actually behind a form that one needs to fill. However, the link is direct. I'm not sure what should the approach be in this case.

also pinging @thomwolf 
The dataset webpage returns a form when trying to download the dataset (form here : http://www.cfilt.iitb.ac.in/iitb_parallel/dataset.html).
However the url we get with the form can be used for the dataset script.
Should we ask the authors or use the urls this way ?

> also pinging @thomwolf
> The dataset webpage returns a form when trying to download the dataset (form here : http://www.cfilt.iitb.ac.in/iitb_parallel/dataset.html).
> However the url we get with the form can be used for the dataset script.
> Should we ask the authors or use the urls this way ?

I had discussion on this with @thomwolf . We have already sent email to author of this dataset.

Hi @spatil6 !
Any news from the authors ?

IIT B folks will add this dataset to repo.
Add PEC

A persona-based empathetic conversation dataset.

I'm a bit frustrated now to get this right.

Hey @zhongpeixiang!
Really nice addition here!

Did you officially joined the sprint by posting [on the forum thread](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176) and joining our slack?
I can't seem to find you there! Should I add you directly with your gmail address?

> Hey @zhongpeixiang!
> Really nice addition here!
> 
> Did you officially joined the sprint by posting [on the forum thread](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176) and joining our slack?
> I can't seem to find you there! Should I add you directly with your gmail address?

Thank you for the invitation. This initiative is awesome. Sadly I’m occupied by my thesis writing this month. Good luck 🤗

As you want @zhongpeixiang (I was maybe not clear but that just mean that by posting on the forum thread that you participated in the current event you will get a special gift (a tee-shirt) for the contribution that you have already done here :-) Nothing more to do)

> As you want @zhongpeixiang (I was maybe not clear but that just mean that by posting on the forum thread that you participated in the current event you will get a special gift (a tee-shirt) for the contribution that you have already done here :-) Nothing more to do)

Oh, I misunderstood the post. I'm glad to join.
Add ZEST: ZEroShot learning from Task descriptions

Adds the ZEST dataset on zero-shot learning from task descriptions from AI2.

- Webpage: https://allenai.org/data/zest
- Paper: https://arxiv.org/abs/2011.08115

The nature of this dataset made the supported task tags tricky if you wouldn't mind giving any feedback @yjernite. Also let me know if you think we should have a `other-task-generalization` or something like that...

Looks good to me, we can ping the authors for more info later. And yes apply `other-task` labels liberally, we can sort them out later :) 

Looks ready to merge when you're ready @joeddav
Add Gutenberg time references dataset

This PR adds the gutenberg_time dataset: https://arxiv.org/abs/2011.04124

Description: "A clean data resource containing all explicit time references in a dataset of 52,183 novels whose full text is available via Project Gutenberg and the Hathi Trust Digital Library 2." > This is just the Gutenberg part.

Also, the paragraph at the top of the file would make a good Dataset Summary in the README :)
Add SciTLDR Dataset (Take 2)

Adds the SciTLDR Dataset by AI2
Added the `README.md` card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents

Continued from #986 

@lhoestq please review this PR when you get free

If the CI fails just because of `RemoteDatasetTest` errors it's fine, they're fixed on master

> If the CI fails just because of `RemoteDatasetTest` errors it's fine, they're fixed on master

The same 3 tests are failing again :(
```
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
```

One trick if you want to add more datasets to avoid these errors : you can just rebase the master branch of your fork from the master branch of the repo. Then each time you make a new branch from master on your fork, it will include the fix for these errors

> One trick if you want to add more datasets to avoid these errors : you can just rebase the master branch of your fork from the master branch of the repo. Then each time you make a new branch from master on your fork, it will include the fix for these errors

Yes, I almost always do that, but somehow seems even this branch got old 😓 
I also do the following if I directly create a new branch locally: `git checkout -b <branchname> upstream/master` so it stays up-to date irrespective of my fork, still don't know how this crept in again

Merging this one since the CI is fixed on master
Add Bilingual Corpus of Arabic-English Parallel Tweets

Added Bilingual Corpus of Arabic-English Parallel Tweets. The link to the dataset can be found [here](https://alt.qcri.org/wp-content/uploads/2020/08/Bilingual-Corpus-of-Arabic-English-Parallel-Tweets.zip) and the paper can be found [here](https://www.aclweb.org/anthology/2020.bucc-1.3.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

IMO, the problem with this dataset is that it is not really a text/nlp dataset. These are just collections of tweet ids. So, ultimately, one needs to crawl twitter to get the actual text.

That's true.



at least it's clear in the description that one needs to collect the tweets : 
```
This resource is a result of a generic method for collecting parallel tweets.
```

Looks like this is failing for other datasets. Should I rebase it and push again?
Also rebasing and pushing is reflecting changes in many other files (ultimately forcing me to open a new branch and a new PR) any way to avoid this?

No let me merge this one directly, it's fine

merging since the CI is fixed on master
how large datasets are handled under the hood 

Hi
I want to use multiple large datasets with a mapping style dataloader, where they cannot fit into memory, could you tell me how you handled the datasets under the hood? is this you bring all in memory in case of mapping style ones? or is this some sharding under the hood and you bring in memory when necessary, thanks 

This library uses Apache Arrow under the hood to store datasets on disk.
The advantage of Apache Arrow is that it allows to memory map the dataset. This allows to load datasets bigger than memory and with almost no RAM usage. It also offers excellent I/O speed.

For example when you access one element or one batch
```python
from datasets import load_dataset

squad = load_dataset("squad", split="train")
first_element = squad[0]
one_batch = squad[:8]
```

then only this element/batch is loaded in memory, while the rest of the dataset is memory mapped.

How can we change how much data is loaded to memory with Arrow? I think that I am having some performance issue with it. When Arrow loads the data from disk it does it in multiprocess? It's almost twice slower training with arrow than in memory.

EDIT:
My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.

> How can we change how much data is loaded to memory with Arrow? I think that I am having some performance issue with it. When Arrow loads the data from disk it does it in multiprocess? It's almost twice slower training with arrow than in memory.

Loading arrow data from disk is done with memory-mapping. This allows to load huge datasets without filling your RAM.
Memory mapping is almost instantaneous and is done within one process.

Then, the speed of querying examples from the dataset is I/O bounded depending on your disk. If it's an SSD then fetching examples from the dataset will be very fast.
But since the I/O speed of an SSD is lower than the one of RAM it's expected to be slower to fetch data from disk than from memory.
Still, if you load the dataset in different processes then it can be faster but there will still be the I/O bottleneck of the disk.

> EDIT:
> My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.

Ok let me know if that helps !
Microsoft CodeXGlue

Datasets from https://github.com/microsoft/CodeXGLUE

This contains 13 datasets:

code_x_glue_cc_clone_detection_big_clone_bench
code_x_glue_cc_clone_detection_poj_104
code_x_glue_cc_cloze_testing_all
code_x_glue_cc_cloze_testing_maxmin
code_x_glue_cc_code_completion_line
code_x_glue_cc_code_completion_token
code_x_glue_cc_code_refinement
code_x_glue_cc_code_to_code_trans
code_x_glue_cc_defect_detection
code_x_glue_ct_code_to_text
code_x_glue_tc_nl_code_search_adv
code_x_glue_tc_text_to_code
code_x_glue_tt_text_to_text


#978 is working on adding code refinement

maybe we should keep the CodeXGlue benchmark (as glue) and don't merge the code_refinement dataset proposed in #978 ?

cc @reshinthadithyan

Hi @madlag and @lhoestq , I am extremely interested in getting this dataset into HF's library as I research in this area a lot. I see that it hasn't been updated in a while, but it is very close to being finished. If no one is currently working on this, I'd be happy to do any final touches that might be needed to get this merged.

Hi @ncoop57 ! Thanks for your interest and sorry for the inactivity on this PR.
Sure feel free to create another PR to continue this one ! This one was really close to being merged so I think it won't require that much changes. In addition to my previous comments, there should also be a "Contributions" subsection (see the template of the README [here](https://github.com/huggingface/datasets/blob/master/templates/README.md))

Superseded by https://github.com/huggingface/datasets/pull/2357 .
Problem downloading amazon_reviews_multi

Thanks for adding the dataset. 
After trying to load the dataset, I am getting the following error: 
`ConnectionError: Couldn't reach https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_fr_train.json
`
I used the following code to load the dataset: 
`load_dataset(
            dataset_name,
            "all_languages",
            cache_dir=".data"
        )`

I am using version 1.1.3 of `datasets`

Note that I can perform a successfull `wget https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_fr_train.json`

Hi @hfawaz ! This is working fine for me. Is it a repeated occurence? Have you tried from the latest verion?

Hi, it seems a connection problem. 
Now it says: 
`ConnectionError: Couldn't reach https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_ja_train.json`
making sure datasets are not loaded in memory and distributed training of them

Hi
I am dealing with large-scale datasets which I need to train distributedly, I used the shard function to divide the dataset across the cores, without any sampler, this does not work for distributed training and does not become any faster than 1 TPU core. 1) how I can make sure data is not loaded in memory 2) in case of distributed training with iterative datasets which measures needs to be taken? Is this all sharding the data only. I was wondering if there can be possibility for me to discuss this with someone with distributed training with iterative datasets using dataset library. thanks 

my implementation of sharding per TPU core: https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/trainers/t5_trainer.py#L316 
my implementation of dataloader for this case https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/tasks/tasks.py#L496 

Hi!  You can use the `assert not bool(dataset.cache_files)` assertion to ensure your data is in memory. And I suggest using `accelerate` for distributed training.
Add SciTLDR Dataset

Adds the SciTLDR Dataset by AI2
Added README card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents

CI failures seem to be unrelated (related to `norwegian_ner`)
```
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
```

you can just rebase from master to fix the CI :) 

can you just rebase from master before we merge ?

Sorry, the rebase from master went horribly wrong, I guess I'll just open another PR

Closing this one due to a mistake in rebasing :(

Continued in #1014
Wongnai - Thai reviews dataset

40,000 reviews, previously released on GitHub ( https://github.com/wongnai/wongnai-corpus ) with an LGPL license, and on a closed Kaggle competition ( https://www.kaggle.com/c/wongnai-challenge-review-rating-prediction/ )

Thank you for contributing a Thai dataset, @mapmeld ! I'm super hyped. 
One comment I may add is that wongnai-corpus has two datasets: review classification (this) and word tokenization (https://github.com/wongnai/wongnai-corpus/blob/master/search/labeled_queries_by_judges.txt).
Would it be possible for you to rename this one something along the line of `wongnai-reviews` so that when/if we include the word tokenization dataset, we will know which is which.

This helps solve my check_code_quality issue.
```
make style
black --line-length 119 --target-version py36 datasets/wongnai
flake8 datasets/wongnai
isort datasets/wongnai/wongnai.py
```

@cstorm125 thanks! following your suggestions on formatting and on naming the dataset

I am writing a blog post about Thai NLP and transformers (example: mBERT does 1-2 character tokens instead of doing word segmentation), started adding this dataset to use as an example, and then saw you were adding other datasets. Great work! And if you know any Thai BERT models beyond https://github.com/ThAIKeras/bert we should maybe talk over email!
Add code refinement

### OVERVIEW
Millions of open-source projects with numerous bug fixes
are available in code repositories. This proliferation
of software development histories can be leveraged to
learn how to fix common programming bugs
Code refinement aims to automatically fix bugs in the code,
which can contribute to reducing the cost of bug-fixes for developers.
Given a piece of Java code with bugs,
the task is to remove the bugs to output the refined code.

Also cc @madlag since I recall you wanted to work on CodeXGlue as well ?

Yes, sorry I did not see earlier your message. I added 34 on the 35 datasets in CodeXGlue, tomorrow I will wrap it up, and so I will remove my version for code_refinement. Maybe we can just have a renaming after the merge, to have a consistent naming with all the other codexglue datasets ? What do you think @reshinthadithyan ?

> Yes, sorry I did not see earlier your message. I added 34 on the 35 datasets in CodeXGlue, tomorrow I will wrap it up, and so I will remove my version for code_refinement. Maybe we can just have a renaming after the merge, to have a consistent naming with all the other codexglue datasets ? What do you think @reshinthadithyan ?

Hello @madlag, I think you can retain that in your script. Let's stick onto the same file like how Glue is maintained.

Hi @reshinthadithyan ! Are you still working on this version of the dataset or are we going with @madlag 's only ?

> Hi @reshinthadithyan ! Are you still working on this version of the dataset or are we going with @madlag 's only ?

Hello, yes. We are going with Madlag's
Arabic pos dialect

A README.md and loading script for the Arabic POS Dialect dataset. The README is missing the sections on personal information, biases, and limitations, as it would probably be better for those to be filled by someone who can read the contents of the dataset and is familiar with Arabic NLP. 

looks like this PR includes changes about many other files than the oens for Araboc POS Dialect

Can you create a another branch and another PR please ?

Sorry! I'm not sure how I managed to do that. I'll make a new branch.
Adding The Microsoft Terminology Collection dataset.


I have to manually copy a dataset_infos.json file from other dataset and modify it since the `datasets-cli` isn't able to handle manually downloaded datasets yet (as far as I know).

you can generate the dataset_infos.json file even for dataset with manual data
To do so just specify `--data_dir <path/to/the/folder/containing/the/manual/data>`

Also, dummy_data seems having difficulty to handle manually downloaded datasets. `python datasets-cli dummy_data datasets/ms_terms  --data_dir ...` reported `error: unrecognized arguments: --data_dir` error. Without `--data_dir`, it reported this error:
```
Dataset ms_terms with config BuilderConfig(name='ms_terms-full', version=1.0.0, data_dir=None, data_files=None, description='...\n') seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file None.
Traceback (most recent call last):
  File "datasets-cli", line 36, in <module>
    service.run()
  File "/Users/lzhao/Downloads/huggingface/datasets/src/datasets/commands/dummy_data.py", line 326, in run
    dataset_builder=dataset_builder, mock_dl_manager=mock_dl_manager
  File "/Users/lzhao/Downloads/huggingface/datasets/src/datasets/commands/dummy_data.py", line 406, in _print_dummy_data_instructions
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

Oh yes `--data_dir` seems to only be supported for the `datasets_cli test` command. Sorry about that.

Can you try to build the dummy_data.zip file manually ?

It has to be inside `./datasets/ms_terms/dummy/ms_terms-full/1.0.0`.
Inside this folder, please create a folder `dummy_data` that contains a dummy file `MicrosoftTermCollection.tbx` (with just a few examples in it). Then you can zip the `dummy_data` folder to `dummy_data.zip`

Then you can check if it worked using the command
```
pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ms_terms
```

Feel free to use some debugging print statements in your script if it doesn't work first try to see what `dl_manager.manual_dir` ends up being and also `path_to_manual_file`.

Feel free to ping me if you have other questions

`pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ms_terms` gave `1 passed, 4 warnings in 8.13s`. Existing datasets, like `wikihow`, and `newsroom`, also report 4 warnings. So, I guess that is not related to this dataset.

Could you run `make style` before we merge @leoxzhao ?

the other errors are fixed on master so it's fine

> Could you run `make style` before we merge @leoxzhao ?

Sure thing. Done. Thanks Quentin. I have other datasets in mind. All of which requires manual download. This process is very helpful

Thank you :)
Add Children's Book Test (CBT) dataset

Add the Children's Book Test (CBT) from Facebook (Hill et al. 2016).

Sentence completion given a few sentences as context from a children's book.

Hi @lhoestq,

I guess this PR can be closed since we merged #2044?

I have used the same link for the homepage, as it is where the dataset is provided, hope that is okay?

Closing in favor of #2044, thanks again :)

> I have used the same link for the homepage, as it is where the dataset is provided, hope that is okay?

Yea it's ok actually, at that time I thought there was another homepage for this dataset
Add CLINC150 Dataset

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

Looks like your PR now shows changes in many other files than the ones for CLINC150.
Feel free to create another branch and another PR

created new [PR](https://github.com/huggingface/datasets/pull/1016)

closing this!
Adding the WebNLG dataset

This PR adds data from the WebNLG challenge, with one configuration per release and challenge iteration.

More information can be found [here](https://webnlg-challenge.loria.fr/)

Unfortunately, the data itself comes from a pretty large number of small XML files, so the dummy data ends up being quite large (8.4 MB even keeping only one example per file).

This is task is part of the GEM suite so will actually need a more complete dataset card. I'm taking a break for now though and will get back to it before merging :)
sample multiple datasets 

Hi
I am dealing with multiple datasets, I need to have a dataloader over them with a condition that in each batch data samples are coming from one of the datasets. My main question is: 
-  I need to have a way to sample the datasets first with some weights, lets say 2x dataset1 1x dataset2, could you point me how I can do it

sub-questions:
- I want to concat sampled datasets and define one dataloader on it, then I need a way to make sure batches come from 1 dataset in each iteration, could you assist me how I can do?
- I use iterative-type of datasets, but I need a method of shuffling still since it brings accuracy performance issues if not doing it, thanks for the help. 

here I share my dataloader currently for multiple tasks: https://gist.github.com/rabeehkarimimahabadi/39f9444a4fb6f53dcc4fca5d73bf8195 

I need to train my model distributedly with this dataloader, "MultiTasksataloader", currently this does not work in distributed fasion,
to save on memory I tried to use iterative datasets, could you have a look in this dataloader and tell me if this is indeed the case? not sure how to make datasets being iterative to not load them in memory, then I remove the sampler for dataloader, and shard the data per core, could you tell me please how I should implement this case in datasets library? and how do you find my implementation in terms of correctness? thanks 


Hi @rabeehkarimimahabadi any luck with updating the multi-task data loader to work with distributed training?

Hi @pushkalkatara yes I solved it back then, here please find my implementation https://github.com/rabeehk/hyperformer/blob/main/hyperformer/data/multitask_sampler.py 

Thanks @rabeehk for sharing. 

The sampler basically returns a list of integers to sample from each task's dataset. I was wondering how to use it with two `torch.Dataset` of different tasks. Also, do I need to shard across processes while creating an Iterable Dataset?


We now have `interleave_datasets` in the API that allows you to cycle/sample with probabilities (with various stopping strategies) through a list of datasets. However, more specific behavior should be implemented manually.

Hi @mariosasko, @pushkalkatara 

I have multi dataset for ASR task. I have multilingual such as: English, China, Japan, Vietnamese. Each dataset is loaded as dataset and they are imbalance dataset. Now I training, I want to equal sampler, it means each batch loader in training has equal number sampler in each language.

For example: Batch_size = 16:
English sample:4
China sample:4
Japan sample:4
Vietnamese sample:4 

How can I do it? I saw that  interleave_datasets only splits data with probability an concatenates all of them after that, it not affect in training data loader. Forgiving me if I wrong.

Thank you for your help.