NyssaW/github-issues huggingface dataset:
making sure datasets are not loaded in memory and distributed training of them

Hi
I am dealing with large-scale datasets which I need to train distributedly, I used the shard function to divide the dataset across the cores, without any sampler, this does not work for distributed training and does not become any faster than 1 TPU core. 1) how I can make sure data is not loaded in memory 2) in case of distributed training with iterative datasets which measures needs to be taken? Is this all sharding the data only. I was wondering if there can be possibility for me to discuss this with someone with distributed training with iterative datasets using dataset library. thanks 

Comments:
my implementation of sharding per TPU core: https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/trainers/t5_trainer.py#L316 
my implementation of dataloader for this case https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/tasks/tasks.py#L496 

Hi!  You can use the `assert not bool(dataset.cache_files)` assertion to ensure your data is in memory. And I suggest using `accelerate` for distributed training.
Add SciTLDR Dataset

Adds the SciTLDR Dataset by AI2
Added README card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents

Comments:
CI failures seem to be unrelated (related to `norwegian_ner`)
```
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
```

you can just rebase from master to fix the CI :) 

can you just rebase from master before we merge ?

Sorry, the rebase from master went horribly wrong, I guess I'll just open another PR

Closing this one due to a mistake in rebasing :(

Continued in #1014
Add GAP dataset

GAP dataset
Gender bias coreference resolution

Comments:
This dataset already exists apparently, sorry :/ 
see
https://github.com/huggingface/datasets/blob/master/datasets/gap/gap.py

Feel free to re-use the dataset card you did for `/datasets/gap`


oh heck, my bad 🤦‍♂️ sorry

I think you should also delete this branch.
Wongnai - Thai reviews dataset

40,000 reviews, previously released on GitHub ( https://github.com/wongnai/wongnai-corpus ) with an LGPL license, and on a closed Kaggle competition ( https://www.kaggle.com/c/wongnai-challenge-review-rating-prediction/ )

Comments:
Thank you for contributing a Thai dataset, @mapmeld ! I'm super hyped. 
One comment I may add is that wongnai-corpus has two datasets: review classification (this) and word tokenization (https://github.com/wongnai/wongnai-corpus/blob/master/search/labeled_queries_by_judges.txt).
Would it be possible for you to rename this one something along the line of `wongnai-reviews` so that when/if we include the word tokenization dataset, we will know which is which.

This helps solve my check_code_quality issue.
```
make style
black --line-length 119 --target-version py36 datasets/wongnai
flake8 datasets/wongnai
isort datasets/wongnai/wongnai.py
```

@cstorm125 thanks! following your suggestions on formatting and on naming the dataset

I am writing a blog post about Thai NLP and transformers (example: mBERT does 1-2 character tokens instead of doing word segmentation), started adding this dataset to use as an example, and then saw you were adding other datasets. Great work! And if you know any Thai BERT models beyond https://github.com/ThAIKeras/bert we should maybe talk over email!
[WIP] Add multi woz

This PR adds version 2.2 of the Multi-domain Wizard of OZ dataset: https://github.com/budzianowski/multiwoz/tree/master/data/MultiWOZ_2.2

It was a pretty big chunk of work to figure out the structure, so I stil have tol add the description to the README.md

On the plus side the structure is broadly similar to that of the Google Schema Guided dialogue [dataset](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue), so will take care of that one next.

Comments:
Add code refinement

### OVERVIEW
Millions of open-source projects with numerous bug fixes
are available in code repositories. This proliferation
of software development histories can be leveraged to
learn how to fix common programming bugs
Code refinement aims to automatically fix bugs in the code,
which can contribute to reducing the cost of bug-fixes for developers.
Given a piece of Java code with bugs,
the task is to remove the bugs to output the refined code.

Comments:
Also cc @madlag since I recall you wanted to work on CodeXGlue as well ?

Yes, sorry I did not see earlier your message. I added 34 on the 35 datasets in CodeXGlue, tomorrow I will wrap it up, and so I will remove my version for code_refinement. Maybe we can just have a renaming after the merge, to have a consistent naming with all the other codexglue datasets ? What do you think @reshinthadithyan ?

> Yes, sorry I did not see earlier your message. I added 34 on the 35 datasets in CodeXGlue, tomorrow I will wrap it up, and so I will remove my version for code_refinement. Maybe we can just have a renaming after the merge, to have a consistent naming with all the other codexglue datasets ? What do you think @reshinthadithyan ?

Hello @madlag, I think you can retain that in your script. Let's stick onto the same file like how Glue is maintained.

Hi @reshinthadithyan ! Are you still working on this version of the dataset or are we going with @madlag 's only ?

> Hi @reshinthadithyan ! Are you still working on this version of the dataset or are we going with @madlag 's only ?

Hello, yes. We are going with Madlag's
Arabic pos dialect

A README.md and loading script for the Arabic POS Dialect dataset. The README is missing the sections on personal information, biases, and limitations, as it would probably be better for those to be filled by someone who can read the contents of the dataset and is familiar with Arabic NLP. 

Comments:
looks like this PR includes changes about many other files than the oens for Araboc POS Dialect

Can you create a another branch and another PR please ?

Sorry! I'm not sure how I managed to do that. I'll make a new branch.
Adding The Microsoft Terminology Collection dataset.


Comments:
I have to manually copy a dataset_infos.json file from other dataset and modify it since the `datasets-cli` isn't able to handle manually downloaded datasets yet (as far as I know).

you can generate the dataset_infos.json file even for dataset with manual data
To do so just specify `--data_dir <path/to/the/folder/containing/the/manual/data>`

Also, dummy_data seems having difficulty to handle manually downloaded datasets. `python datasets-cli dummy_data datasets/ms_terms  --data_dir ...` reported `error: unrecognized arguments: --data_dir` error. Without `--data_dir`, it reported this error:
```
Dataset ms_terms with config BuilderConfig(name='ms_terms-full', version=1.0.0, data_dir=None, data_files=None, description='...\n') seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file None.
Traceback (most recent call last):
  File "datasets-cli", line 36, in <module>
    service.run()
  File "/Users/lzhao/Downloads/huggingface/datasets/src/datasets/commands/dummy_data.py", line 326, in run
    dataset_builder=dataset_builder, mock_dl_manager=mock_dl_manager
  File "/Users/lzhao/Downloads/huggingface/datasets/src/datasets/commands/dummy_data.py", line 406, in _print_dummy_data_instructions
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

Oh yes `--data_dir` seems to only be supported for the `datasets_cli test` command. Sorry about that.

Can you try to build the dummy_data.zip file manually ?

It has to be inside `./datasets/ms_terms/dummy/ms_terms-full/1.0.0`.
Inside this folder, please create a folder `dummy_data` that contains a dummy file `MicrosoftTermCollection.tbx` (with just a few examples in it). Then you can zip the `dummy_data` folder to `dummy_data.zip`

Then you can check if it worked using the command
```
pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ms_terms
```

Feel free to use some debugging print statements in your script if it doesn't work first try to see what `dl_manager.manual_dir` ends up being and also `path_to_manual_file`.

Feel free to ping me if you have other questions

`pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ms_terms` gave `1 passed, 4 warnings in 8.13s`. Existing datasets, like `wikihow`, and `newsroom`, also report 4 warnings. So, I guess that is not related to this dataset.

Could you run `make style` before we merge @leoxzhao ?

the other errors are fixed on master so it's fine

> Could you run `make style` before we merge @leoxzhao ?

Sure thing. Done. Thanks Quentin. I have other datasets in mind. All of which requires manual download. This process is very helpful

Thank you :)
Add Children's Book Test (CBT) dataset

Add the Children's Book Test (CBT) from Facebook (Hill et al. 2016).

Sentence completion given a few sentences as context from a children's book.

Comments:
Hi @lhoestq,

I guess this PR can be closed since we merged #2044?

I have used the same link for the homepage, as it is where the dataset is provided, hope that is okay?

Closing in favor of #2044, thanks again :)

> I have used the same link for the homepage, as it is where the dataset is provided, hope that is okay?

Yea it's ok actually, at that time I thought there was another homepage for this dataset
Add CLINC150 Dataset

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

Comments:
Looks like your PR now shows changes in many other files than the ones for CLINC150.
Feel free to create another branch and another PR

created new [PR](https://github.com/huggingface/datasets/pull/1016)

closing this!
Add CLINC150 Dataset

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

Comments:
Adding the WebNLG dataset

This PR adds data from the WebNLG challenge, with one configuration per release and challenge iteration.

More information can be found [here](https://webnlg-challenge.loria.fr/)

Unfortunately, the data itself comes from a pretty large number of small XML files, so the dummy data ends up being quite large (8.4 MB even keeping only one example per file).

Comments:
This is task is part of the GEM suite so will actually need a more complete dataset card. I'm taking a break for now though and will get back to it before merging :)
sample multiple datasets 

Hi
I am dealing with multiple datasets, I need to have a dataloader over them with a condition that in each batch data samples are coming from one of the datasets. My main question is: 
-  I need to have a way to sample the datasets first with some weights, lets say 2x dataset1 1x dataset2, could you point me how I can do it

sub-questions:
- I want to concat sampled datasets and define one dataloader on it, then I need a way to make sure batches come from 1 dataset in each iteration, could you assist me how I can do?
- I use iterative-type of datasets, but I need a method of shuffling still since it brings accuracy performance issues if not doing it, thanks for the help. 

Comments:
here I share my dataloader currently for multiple tasks: https://gist.github.com/rabeehkarimimahabadi/39f9444a4fb6f53dcc4fca5d73bf8195 

I need to train my model distributedly with this dataloader, "MultiTasksataloader", currently this does not work in distributed fasion,
to save on memory I tried to use iterative datasets, could you have a look in this dataloader and tell me if this is indeed the case? not sure how to make datasets being iterative to not load them in memory, then I remove the sampler for dataloader, and shard the data per core, could you tell me please how I should implement this case in datasets library? and how do you find my implementation in terms of correctness? thanks 


Hi @rabeehkarimimahabadi any luck with updating the multi-task data loader to work with distributed training?

Hi @pushkalkatara yes I solved it back then, here please find my implementation https://github.com/rabeehk/hyperformer/blob/main/hyperformer/data/multitask_sampler.py 

Thanks @rabeehk for sharing. 

The sampler basically returns a list of integers to sample from each task's dataset. I was wondering how to use it with two `torch.Dataset` of different tasks. Also, do I need to shard across processes while creating an Iterable Dataset?


We now have `interleave_datasets` in the API that allows you to cycle/sample with probabilities (with various stopping strategies) through a list of datasets. However, more specific behavior should be implemented manually.

Hi @mariosasko, @pushkalkatara 

I have multi dataset for ASR task. I have multilingual such as: English, China, Japan, Vietnamese. Each dataset is loaded as dataset and they are imbalance dataset. Now I training, I want to equal sampler, it means each batch loader in training has equal number sampler in each language.

For example: Batch_size = 16:
English sample:4
China sample:4
Japan sample:4
Vietnamese sample:4 

How can I do it? I saw that  interleave_datasets only splits data with probability an concatenates all of them after that, it not affect in training data loader. Forgiving me if I wrong.

Thank you for your help.
Added PragmEval benchmark


Comments:
> Really cool ! Thanks for adding this one :)
> Good job at adding all those citations for each task
> 
> Looks like the dummy data test doesn't pass. Maybe some files are missing in the dummy_data.zip files ?
> The error reports `pragmeval/verifiability/train.tsv` to be missing
> 
> Also could you add the tags part of the dataset card (the rest is optional) ?
> See more info here : https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

In the prior commits I generated dataset_infos and the dummy files myself
Now they are generated with the cli, and the tests now seem to be passing better
I will look into the tag


Looks like you did a good job with dummy data in the first place !
The downside of automatically generated dummy data is that the files are heavier (here 40KB per file).
If you could replace the generated dummy files with the one you created yourself it would be awesome, since the one you did yourself are way lighter (around 1KB per file). Using small files make `git clone` run faster so we encourage to use small dummy_data files.

could you rebase from master ? it should fix the CI

> could you rebase from master ? it should fix the CI

I think it is due to the file structure of the dummy data that causes test failure. The automatically generated dummy data pass the tests

Indeed the error reports that `pragmeval/verifiability/train.tsv` is missing for the verifiability dummy_data.zip file.
To fix that you should add the missing data files in each dummy_data.zip file.
To test that your dummy data work you can run
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_
```
if some file is missing it should tell you which one

Also it looks like you haven't rebased from master yet, even though you did a `rebase` commit. 

rebasing should fix the other CI fails

It's ok if we have `RemoteDatasetTest ` errors, they're fixed on master

merging since the CI is fixed on master

Hey @sileod! Super nice to see you participating ;)

Did you officially joined the sprint by posting on [the forum thread](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176) and joining our slack?

I can't seem to find you there! Should I add you directly with your gmail address?

Hi @sileod 👋
add prachathai67k

`prachathai-67k`: News Article Corpus and Multi-label Text Classificdation from Prachathai.com
The prachathai-67k dataset was scraped from the news site Prachathai.
We filtered out those articles with less than 500 characters of body text, mostly images and cartoons.
It contains 67,889 articles wtih 12 curated tags from August 24, 2004 to November 15, 2018.
The dataset was originally scraped by @lukkiddd and cleaned by @cstorm125.
You can also see preliminary exploration at https://github.com/PyThaiNLP/prachathai-67k/blob/master/exploration.ipynb

Comments:
Test failing for same issues as https://github.com/huggingface/datasets/pull/939
Please advise.

```
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_flue
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_flue
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_flue
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_xglue
===== 7 failed, 1309 passed, 932 skipped, 11 warnings in 166.71s (0:02:46) =====
```

Closing and opening a new pull request to solve rebase issues

To be continued on https://github.com/huggingface/datasets/pull/982
added health_fact dataset 

Added dataset Explainable Fact-Checking for Public Health Claims (dataset_id: health_fact)

Comments:
Hi @lhoestq,
Initially I tried int(-1) only in place of nan labels and missing values but I kept on getting this error ```pyarrow.lib.ArrowTypeError: Expected bytes, got a 'int' object``` maybe because I'm sending int values (-1) to objects which are string type
Prachathai67k

Add `prachathai-67k`: News Article Corpus and Multi-label Text Classificdation from Prachathai.com

The `prachathai-67k` dataset was scraped from the news site [Prachathai](prachathai.com). We filtered out those articles with less than 500 characters of body text, mostly images and cartoons. It contains 67,889 articles wtih 12 curated tags from August 24, 2004 to November 15, 2018. The dataset was originally scraped by [@lukkiddd](https://github.com/lukkiddd) and cleaned by [@cstorm125](https://github.com/cstorm125). Download the dataset [here](https://www.dropbox.com/s/fsxepdka4l2pr45/prachathai-67k.zip?dl=1). You can also see preliminary exploration in [exploration.ipynb](https://github.com/PyThaiNLP/prachathai-67k/blob/master/exploration.ipynb).

This dataset is a part of [pyThaiNLP](https://github.com/PyThaiNLP/) Thai text [classification-benchmarks](https://github.com/PyThaiNLP/classification-benchmarks). For the benchmark, we selected the following tags with substantial volume that resemble **classifying types of articles**:

* `การเมือง` - politics
* `สิทธิมนุษยชน` - human_rights
* `คุณภาพชีวิต` - quality_of_life
* `ต่างประเทศ` - international
* `สังคม` - social
* `สิ่งแวดล้อม` - environment
* `เศรษฐกิจ` - economics
* `วัฒนธรรม` - culture
* `แรงงาน` - labor
* `ความมั่นคง` - national_security
* `ไอซีที` - ict
* `การศึกษา` - education

Comments:
Wrongly branching from existing branch of wisesight_sentiment. Closing and opening another one specifically for prachathai67k
D

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Add People's Daily NER dataset


Comments:
> LGTM thanks :)
> 
> 
> 
> Before we merge, could you add a dataset card ? see here for more info: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card
> 
> 
> 
> Note that only the tags at the top of the dataset card are mandatory, if you feel like it's going to take too much time writing the rest to fill it all you can just skip the paragraphs

Nope. I don't think there is a citation. Also, can I do the dataset card later (maybe in bulk)?

We're doing one PR = one dataset to keep track of things. Feel free to add the tags later in this PR if you want to.
Also only the tags are required now, because we don't want people spending too much time on the cards

added @lhoestq 

Merging since the CI is fixed on master
Local machine/cluster Beam Datasets example/tutorial

Hi,

I'm wondering if https://huggingface.co/docs/datasets/beam_dataset.html has an non-GCP or non-Dataflow version example/tutorial? I tried to migrate it to run on DirectRunner and SparkRunner, however, there were way too many runtime errors that I had to fix during the process, and even so I wasn't able to get either runner correctly producing the desired output.

Thanks!
Shang

Comments:
I tried to make it run once on the SparkRunner but it seems that this runner has some issues when it is run locally.
From my experience the DirectRunner is fine though, even if it's clearly not memory efficient.

It would be awesome though to make it work locally on a SparkRunner !
Did you manage to make your processing work ?

We've deprecated the Beam API in `datasets`. As part of it, the Beam datasets have also been converted to non-Beam-based to make using them straightforward.
add PIB dataset

This pull request will add PIB dataset.

Comments:
Hi, 

I am unable to get success in these tests. Can someone help me by pointing out possible errors?

Thanks

Hi ! you can read the tests by logging in to circleci.

Anyway for information here are the errors : 
```
datasets/pib/pib.py:19:1: F401 'csv' imported but unused
datasets/pib/pib.py:20:1: F401 'json' imported but unused
datasets/pib/pib.py:36:84: W291 trailing whitespace
```
and 
```
FAILED tests/test_file_encoding.py::TestFileEncoding::test_no_encoding_on_file_open
```

To fix the `test_no_encoding_on_file_open` you just have to specify an encoding while opening a text file. For example `encoding="utf-8"`


All suggested changes are done.

Nice ! can you re-generate the dataset_infos.json file to take into account the feature type change ?
```
datasets-cli test ./datasets/pib --save_infos --all_configs --ignore_verifications
```
And also format your code ?
```
make style
```
Add NumerSense

Adds the NumerSense dataset
- Webpage/leaderboard: https://inklab.usc.edu/NumerSense/
- Paper: https://arxiv.org/abs/2005.00683
- Description: NumerSense is a new numerical commonsense reasoning probing task, with a diagnostic dataset consisting of 3,145 masked-word-prediction probes. Basically, it's a benchmark to see whether your MLM can figure out the right number in a fill-in-the-blank task based on commonsense knowledge (a bird has **two** legs)

Comments:
[WIP] complex_webqa - Error zipfile.BadZipFile: Bad CRC-32

Have a string `zipfile.BadZipFile: Bad CRC-32 for file 'web_snippets_train.json'` error when downloading the largest file from dropbox: `https://www.dropbox.com/sh/7pkwkrfnwqhsnpo/AABVENv_Q9rFtnM61liyzO0La/web_snippets_train.json.zip?dl=1`

Didn't managed to see how to solve that.

Putting aside for now.


Comments:
Thanks for your contribution, @thomwolf. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest that you create this dataset there. Please, feel free to tell us if you need some help.
Lambada

Added LAMBADA dataset.

A couple of points of attention (mostly because I am not sure)
- The training data are compressed in a .tar file inside the main tar.gz file. I had to manually un-tar the training file to access the examples.
- The dev and test splits don't have the `category` field so I put `None` by default.

Happy to make changes if it doesn't respect the guidelines!

Victor

Comments:
Add the Multilingual Amazon Reviews Corpus

- **Name:** Multilingual Amazon Reviews Corpus* (`amazon_reviews_multi`)
- **Description:** A collection of Amazon reviews in English, Japanese, German, French, Spanish and Chinese.
- **Paper:** https://arxiv.org/abs/2010.02573

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

Comments:
add inquisitive

Adding inquisitive qg dataset

More info: https://github.com/wjko2/INQUISITIVE

Comments:
`dummy_data` right now contains all article files, keeping only the required articles for dummy data fails the dummy data test.
Any idea ?

> `dummy_data` right now contains all article files, keeping only the required articles for dummy data fails the dummy data test.
> Any idea ?

We should definitely find a way to make it work with only a few articles.

If it doesn't work right now for dummy data, I guess it's because it tries to load every single article file ?

If so, then maybe you can use `os.listdir` method to first check all the data files available in the path where the `articles.tgz` file is extracted. Then you can simply iter through the data files and depending on their ID, include them in the train or test set. With this method you should be able to have only a few articles files per split in the dummy data. Does that make sense ?

fixed! so the issue was, `articles_ids` were prepared based on the number of files in articles dir, so for dummy data questions it was not able to load some articles due to incorrect ids and the test was failing
Add DART

- **Name:** *DART*
- **Description:** *DART is a large dataset for open-domain structured data record to text generation.*
- **Paper:** *https://arxiv.org/abs/2007.02871*
- **Data:** *https://github.com/Yale-LILY/dart#leaderboard*

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

Comments:
LGTM!
Add CC-100 dataset

Add CC-100.

Close #773 

Comments:
Hello @lhoestq, I would like just to ask you if it is OK that I include this feature 9f32ba1 in this PR or you would prefer to have it in a separate one.

I was wondering whether include also a test, but I did not find any test for the other file formats...

Hi ! Sure that would be valuable to support .xz files. Feel free to open a separate PR for this.
And feel free to create the first test case for extracting compressed files if you have some inspiration (maybe create test_file_utils.py ?). We can still spend more time on tests next week when the sprint is over though so don't spend too much time on it.

@lhoestq, DONE! ;) See PR #950.

Thanks for adding support for `.xz` files :)

Feel free to rebase from master to include it in your PR

@lhoestq DONE; I have merged instead, to avoid changing the history of my public PR ;)

Hi @lhoestq, I would need that you generate the dataset_infos.json and the dummy data for this dataset with a bigger computer. Sorry, but my laptop did not succeed...

Thanks for your work @albertvillanova 
We'll definitely look into it after this sprint :)

Looks like #1456 added CC100 already.
The difference with your approach is that this implementation uses the `BuilderConfig` parameters to allow the creation of custom configs for all the languages, without having to specify them in the `BUILDER_CONFIGS` class attribute.
For example even if the dataset doesn't have a config for english already, you can still load the english CC100 with
```python
from datasets import load_dataset

load_dataset("cc100", lang="en")
```

@lhoestq, oops!! I remember having assigned this dataset to me in the Google sheet, besides having mentioned the corresponding issue in the Pull Request... Nevermind! :)

Yes indeed I can see that...
Sorry for noticing that only now 

The code of the other PR ended up being pretty close to yours though
If you want to add more details to the cc100 dataset card or in the script feel to do so, any addition is welcome
Add XOR QA Dataset

Added XOR Question Answering Dataset. The link to the dataset can be found [here](https://nlp.cs.washington.edu/xorqa/)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

Comments:
Hi @sumanthd17 

Looks like a good start! You will also need to add a Dataset card, following the instructions given [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#manually-tag-the-dataset-and-write-the-dataset-card)

I followed the instructions mentioned there but my dataset isn't showing up in the dropdown list. Am I missing something here? @yjernite 

> I followed the instructions mentioned there but my dataset isn't showing up in the dropdown list. Am I missing something here? @yjernite

The best way is to run the tagging app locally and provide it the location to the `dataset_infos.json` after you've run the CLI:
https://github.com/huggingface/datasets-tagging


This is a really good data card!!

Small changes to make it even better:
- Tags: the dataset has both "original" data and data that is "extended" from a source dataset: TydiQA - you should choose both options in the tagging apps
- The language and annotation creator tags are off: the language here is the questions: I understand it's a mix of crowd-sourced and expert-generated? Is there any machine translation involved? The annotations are the span selections: is that crowd-sourced?
- Personal and sensitive information: there should be a statement there, even if only to say that none could be found or that it only mentions public figures
add dream dataset

Adding Dream: a Dataset and for Dialogue-Based Reading Comprehension

More details:
https://dataset.org/dream/
https://github.com/nlpdata/dream

Comments:
> Awesome good job !
> 
> Could you also add a dataset card using the template guide here : https://github.com/huggingface/datasets/blob/master/templates/README_guide.md
> If you can't fill some fields then just leave `[N/A]`

Quick amendment: `[N/A]` is for fields that are not relevant: if you can't find the information just leave `[More Information Needed]`

@lhoestq since datset cards are optional for this sprint I'll add those later. Good for merge.

Indeed we only require the tags to be added now (the yaml part at the top of the dataset card).
Could you add them please ?
You can find more infos here : https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

@lhoestq added tags, I'll fill rest of the info after current sprint :)

The tests are failing tests for other datasets, not this one.

@lhoestq  could you tell me why these tests are failing, they don't seem related to this PR.
wrong length with datasets 

Hi
I have a MRPC dataset which I convert it to seq2seq format, then this is of this format:

`Dataset(features: {'src_texts': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 10)
`

I feed it to a dataloader:
```
dataloader = DataLoader(
            train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler,
            collate_fn=self.data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.dataloader_num_workers,
        )
```

now if I type len(dataloader) this is 1, which is wrong, and this needs to be 10. could you assist me please? thanks 


Comments:
Also, I cannot first convert it to torch format, since huggingface seq2seq_trainer codes process the datasets afterwards during datacollector function to make it optimize for TPUs. 

sorry I misunderstood length of dataset with dataloader, closed. thanks
Add conll2002

Adding the Conll2002 dataset for NER.

More info here : https://www.clips.uantwerpen.be/conll2002/ner/

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


Comments:
Shall we change the hashing to encoding to reduce potential replicated cache files?

Hi there. For now, we are using `xxhash` to hash the transformations to fingerprint and we will save a copy of the processed dataset to disk if there is a new hash value. However, there are some transformations that are idempotent or commutative to each other. I think that encoding the transformation chain as the fingerprint may help in those cases, for example, use `base64.urlsafe_b64encode`. In this way, before we want to save a new copy, we can decode the transformation chain and normalize it to prevent omit potential reuse. As the main targets of this project are the really large datasets that cannot be loaded entirely in memory, I believe it would save a lot of time if we can avoid some write.

If you have interest in this, I'd love to help :).

Comments:
This is an interesting idea !
Do you have ideas about how to approach the decoding and the normalization ?

@lhoestq
I think we first need to save the transformation chain to a list in `self._fingerprint`. Then we can
- decode all the current saved datasets to see if there is already one that is equivalent to the transformation we need now.
- or, calculate all the possible hash value of the current chain for comparison so that we could continue to use hashing.
If we find one, we can adjust the list in `self._fingerprint` to it.

As for the transformation reordering rules, we can just start with some manual rules, like two sort on the same column should merge to one, filter and select can change orders.

And for encoding and decoding, we can just manually specify `sort` is 0, `shuffling` is 2 and create a base-n number or use some general algorithm like `base64.urlsafe_b64encode`.

Because we are not doing lazy evaluation now, we may not be able to normalize the transformation to its minimal form. If we want to support that, we can provde a `Sequential` api and let user input a list or transformation, so that user would not use the intermediate datasets. This would look like tf.data.Dataset.
My new dataset PEC

A new dataset PEC published in EMNLP 2020.

Comments:
How to resolve these failed checks?

Thanks for adding this one :) 

To fix the check_code_quality, please run `make style` with the latest version of black, isort, flake8
To fix the test_no_encoding_on_file_open, make sure to specify the encoding each time you call `open()` on a text file.
For example : `encoding="utf-8"`
To fix the test_load_dataset_pec , you must add the dummy_data.zip file. It is used to test the dataset script and make sure it runs fine. To add it, please refer to the steps in https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-add-a-dataset



Could you also add a dataset card ? you can find a template here : https://github.com/huggingface/datasets/blob/master/templates/README.md

That would be awesome

> Thanks for adding this one :)
> 
> To fix the check_code_quality, please run `make style` with the latest version of black, isort, flake8
> To fix the test_no_encoding_on_file_open, make sure to specify the encoding each time you call `open()` on a text file.
> For example : `encoding="utf-8"`
> To fix the test_load_dataset_pec , you must add the dummy_data.zip file. It is used to test the dataset script and make sure it runs fine. To add it, please refer to the steps in https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-add-a-dataset

Thank you for the detailed suggestion.

I have added dummy_data but it still failed the DistributedDatasetTest check. My dataset has a central file (containing a python dict) that needs to be accessed by each data example. Is it because the central file cannot be distributed (which would lead to a partial dictionary)?

Specifically, the central file contains a dictionary of speakers with their attributes. Each data example is also associated with a speaker. As of now, I keep the central file and data files separately. If I remove the central file by appending the speaker attributes to each data example, then there would be lots of redundancy because there are lots of duplicate speakers in the data files.

The `DistributedDatasetTest` fail and the changes of this PR are not related, there was just a bug in the CI. You can ignore it

> Really cool thanks !
> 
> Could you make the dummy files smaller ? For example by reducing the size of persona.txt ?
> I also left a comment about the files concatenation. It would be cool to replace that with simple iterations through the different files.
> 
> Then once this is done, you can add a dataset card using the template guide here : https://github.com/huggingface/datasets/blob/master/templates/README_guide.md
> If some fields can't be filled, just leave `[N/A]`

Small change: if you don't have the information for a field, please leave `[More Information Needed]` rather than `[N/A]`

The full information can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#manually-tag-the-dataset-and-write-the-dataset-card)
Grindr meeting app web.Grindr

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).

Comments:
Add FiNER dataset

Hi,

this PR adds "A Finnish News Corpus for Named Entity Recognition" as new `finer` dataset.

The dataset is described in [this paper](https://arxiv.org/abs/1908.04212). The data is publicly available in [this GitHub](https://github.com/mpsilfve/finer-data).

Notice: they provide two testsets. The additional test dataset taken from Wikipedia is named as "test_wikipedia" split.

Comments:
> That's really cool thank you !
> 
> Could you also add a dataset card ?
> You can find a template here : https://github.com/huggingface/datasets/blob/master/templates/README.md

The full information for adding a dataset card can be found here :) 
https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#manually-tag-the-dataset-and-write-the-dataset-card


Thanks your suggestions! I've fixed them, and currently working on the dataset card!

@yjernite and @lhoestq I will add the dataset card a bit later in a separate PR if that's ok for you!

Yes I want to re-emphasize if it was not clear that dataset cards are optional for the sprint. 

Only the tags are required for merging a datasets.

Please try to enforce this rule as well @lhoestq and @yjernite 

Yes @stefan-it if you could just add the tags (the yaml part at the top of the dataset card) that'd be perfect :) 

Oh, sorry, will add them now!


Initial README file is now added :) 

the `RemoteDatasetTest ` errors in the CI are fixed on master so it's fine

merging since the CI is fixed on master
Disallow backslash in urls

Following #903 @albertvillanova noticed that there are sometimes bad usage of `os.path.join` in datasets scripts to create URLS. However this should be avoided since it doesn't work on windows.

I'm suggesting a test to make sure we that all the urls don't have backslashes in them in the datasets scripts.
The tests works by adding a callback feature to the MockDownloadManager used to test the dataset scripts. In a download callback I just make sure that the url is valid.

Comments:
Looks like the test doesn't detect all the problems fixed by #907 , I'll fix that

Ok found why it doesn't detect the problems fixed by #907 . That's because for all those datasets the urls are actually fine (no backslash) on windows, even if it uses `os.path.join`.

This is because of the behavior of `os.path.join` on windows when the first path ends with a slash : 

```python
import os
os.path.join("https://test.com/foo", "bar.txt")
# 'https://test.com/foo\\bar.txt'
os.path.join("https://test.com/foo/", "bar.txt")
# 'https://test.com/foo/bar.txt'
```

However even though the urls are correct, this is definitely bad practice and we should never use `os.path.join` for urls
Fix URL with backslash in Windows

In Windows, `os.path.join` generates URLs containing backslashes, when the first "path" does not end with a slash. 

In general, `os.path.join` should be avoided to generate URLs.

Comments:
@lhoestq I was indeed working on that... to make another commit on this feature branch...

But as you prefer... nevermind! :)

Ah what do you have in mind for the tests ? I was thinking of adding a check in the MockDownloadManager used for tests based on dummy data. I'm creating a PR right now, I'd be happy to have your opinion

Indeed I was thinking of something similar: monckeypatching the HTTP request...

Therefore, if you agree, I am removing all the rest of `os.path.join`, both from the code and the docs...

If you spot other `os.path.join` for urls in dataset scripts or metrics scripts feel free to fix them.
In the library itself (/src/datasets) it should be fine since there are tests and a windows CI, but if you have doubts of some usage of `os.path.join` somewhere, let me know.

Alright create the test in #905 .
The windows CI is failing for all the datasets that have bad usage of `os.path.join` for urls.
There are of course the ones you fixed in this PR (thanks again !) but I found others as well such as pg19 and blimp.
You can check the full list by looking at the CI failures of the commit 1ce3354

I am merging this one as well as #906 that should fix all of the datasets.
Then I'll rebase #905 which adds the test that checks for bad urls and make sure it' all green now
Addition of Nl2Bash Dataset

## Overview
The NL2Bash data contains over 10,000 instances of linux shell commands and their corresponding natural language descriptions provided by experts, from the Tellina system. The dataset features 100+ commonly used shell utilities.
## Footnotes
The following dataset marks the first ML on source code related Dataset in datasets module. It'll be really useful as a lot of the research direction involves Transformer Based Model.
Thanks.
### Reference Links

> Paper Link = https://arxiv.org/pdf/1802.08979.pdf 
> Github Link = https://github.com/TellinaTool/nl2bash







Comments:
Hello, thanks. I had a talk with the dataset authors, found out that the data now is obsolete and they'll get a stable version soon. So temporality closing the PR.
 Although I have a question, What should _id_ be in the return statement? Should that be something like a start index (or) the type of split will do? Thanks. 

@reshinthadithyan we should hold off on this for a couple of weeks till NeurIPS concludes. The [NLC2CMD](http://nlc2cmd.us-east.mybluemix.net/) data will be out then; which includes a cleaner version of this NL2Bash data. The older data is sort of obsolete now. 

Ah nvm you already commented 😆
datasets.load_dataset() custom chaching directory bug

Hello,
I'm having issue with loading a dataset with a custom `cache_dir`. Despite specifying the output dir, it is still downloaded to 
 `~/.cache`.

## Environment info
- `datasets` version: 1.1.3
- Platform: Linux-4.19.129-aufs-1-x86_64-with-debian-10.1
- Python version: 3.7.3

## The code I'm running:
```python
import datasets
from pathlib import Path

validation_dataset = datasets.load_dataset("natural_questions", split="validation[:5%]", cache_dir=Path("./data"))  
```

## The output:

* The dataset is downloaded to my home directory's `.cache` 
* A new empty directory named "`natural_questions` is created in the specified directory `.data`
* `tree data` in the shell outputs:
```
data
└── natural_questions
    └── default
        └── 0.0.2
3 directories, 0 files
```

The output:
```
Downloading: 8.61kB [00:00, 5.11MB/s]                                                                                                                                                                              
Downloading: 13.6kB [00:00, 7.89MB/s]                                                                                                                                                                              
Using custom data configuration default                                                                                                                                                                            
Downloading and preparing dataset natural_questions/default (download: 41.97 GiB, generated: 92.95 GiB, post-processed: Unknown size, total: 134.92 GiB) to ./data/natural_questions/default/0.0.2/867dbbaf9137c1b8
3ecb19f5eb80559e1002ea26e702c6b919cfa81a17a8c531...                                                                                                                                                                
Downloading: 100%|██████████████████████████████████████████████████| 13.6k/13.6k [00:00<00:00, 1.51MB/s]                                                                                                          
Downloading:   7%|███▎                                            | 6.70G/97.4G [03:46<1:37:05, 15.6MB/s]
```

## Expected behaviour:
The dataset "Natural Questions" should be downloaded to the directory "./data"


Comments:
Thanks for reporting ! I'm looking into it.