helmo/github-issues huggingface dataset:
Add E2E NLG
State: closed | Author: lhoestq (MEMBER)

Adding the E2E NLG dataset.

More info here : http://www.macs.hw.ac.uk/InteractionLab/E2E/

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template and at least fill the tags 
- [x] Both tests for the real data and the dummy data pass.
making sure datasets are not loaded in memory and distributed training of them
State: closed | Author: rabeehk (CONTRIBUTOR)

Hi
I am dealing with large-scale datasets which I need to train distributedly, I used the shard function to divide the dataset across the cores, without any sampler, this does not work for distributed training and does not become any faster than 1 TPU core. 1) how I can make sure data is not loaded in memory 2) in case of distributed training with iterative datasets which measures needs to be taken? Is this all sharding the data only. I was wondering if there can be possibility for me to discuss this with someone with distributed training with iterative datasets using dataset library. thanks 

Comments: my implementation of sharding per TPU core: https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/trainers/t5_trainer.py#L316 
my implementation of dataloader for this case https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/tasks/tasks.py#L496 

Hi!  You can use the `assert not bool(dataset.cache_files)` assertion to ensure your data is in memory. And I suggest using `accelerate` for distributed training.
Add SciTLDR Dataset
State: closed | Author: bharatr21 (CONTRIBUTOR)

Adds the SciTLDR Dataset by AI2
Added README card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents

Comments: CI failures seem to be unrelated (related to `norwegian_ner`)
```
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
```

you can just rebase from master to fix the CI :) 

can you just rebase from master before we merge ?

Sorry, the rebase from master went horribly wrong, I guess I'll just open another PR

Closing this one due to a mistake in rebasing :(

Continued in #1014
Add GAP dataset
State: closed | Author: VictorSanh (CONTRIBUTOR)

GAP dataset
Gender bias coreference resolution

Comments: This dataset already exists apparently, sorry :/ 
see
https://github.com/huggingface/datasets/blob/master/datasets/gap/gap.py

Feel free to re-use the dataset card you did for `/datasets/gap`


oh heck, my bad 🤦‍♂️ sorry

I think you should also delete this branch.
Wongnai - Thai reviews dataset
State: closed | Author: mapmeld (CONTRIBUTOR)

40,000 reviews, previously released on GitHub ( https://github.com/wongnai/wongnai-corpus ) with an LGPL license, and on a closed Kaggle competition ( https://www.kaggle.com/c/wongnai-challenge-review-rating-prediction/ )

Comments: Thank you for contributing a Thai dataset, @mapmeld ! I'm super hyped. 
One comment I may add is that wongnai-corpus has two datasets: review classification (this) and word tokenization (https://github.com/wongnai/wongnai-corpus/blob/master/search/labeled_queries_by_judges.txt).
Would it be possible for you to rename this one something along the line of `wongnai-reviews` so that when/if we include the word tokenization dataset, we will know which is which.

This helps solve my check_code_quality issue.
```
make style
black --line-length 119 --target-version py36 datasets/wongnai
flake8 datasets/wongnai
isort datasets/wongnai/wongnai.py
```

@cstorm125 thanks! following your suggestions on formatting and on naming the dataset

I am writing a blog post about Thai NLP and transformers (example: mBERT does 1-2 character tokens instead of doing word segmentation), started adding this dataset to use as an example, and then saw you were adding other datasets. Great work! And if you know any Thai BERT models beyond https://github.com/ThAIKeras/bert we should maybe talk over email!
[WIP] Add multi woz
State: closed | Author: yjernite (MEMBER)

This PR adds version 2.2 of the Multi-domain Wizard of OZ dataset: https://github.com/budzianowski/multiwoz/tree/master/data/MultiWOZ_2.2

It was a pretty big chunk of work to figure out the structure, so I stil have tol add the description to the README.md

On the plus side the structure is broadly similar to that of the Google Schema Guided dialogue [dataset](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue), so will take care of that one next.
Add code refinement
State: closed | Author: reshinthadithyan (CONTRIBUTOR)

### OVERVIEW
Millions of open-source projects with numerous bug fixes
are available in code repositories. This proliferation
of software development histories can be leveraged to
learn how to fix common programming bugs
Code refinement aims to automatically fix bugs in the code,
which can contribute to reducing the cost of bug-fixes for developers.
Given a piece of Java code with bugs,
the task is to remove the bugs to output the refined code.

Comments: Also cc @madlag since I recall you wanted to work on CodeXGlue as well ?

Yes, sorry I did not see earlier your message. I added 34 on the 35 datasets in CodeXGlue, tomorrow I will wrap it up, and so I will remove my version for code_refinement. Maybe we can just have a renaming after the merge, to have a consistent naming with all the other codexglue datasets ? What do you think @reshinthadithyan ?

> Yes, sorry I did not see earlier your message. I added 34 on the 35 datasets in CodeXGlue, tomorrow I will wrap it up, and so I will remove my version for code_refinement. Maybe we can just have a renaming after the merge, to have a consistent naming with all the other codexglue datasets ? What do you think @reshinthadithyan ?

Hello @madlag, I think you can retain that in your script. Let's stick onto the same file like how Glue is maintained.

Hi @reshinthadithyan ! Are you still working on this version of the dataset or are we going with @madlag 's only ?

> Hi @reshinthadithyan ! Are you still working on this version of the dataset or are we going with @madlag 's only ?

Hello, yes. We are going with Madlag's
Add ROPES dataset
State: closed | Author: VictorSanh (CONTRIBUTOR)

ROPES dataset 
Reasoning over paragraph effects in situations - testing a system's ability to apply knowledge from a passage of text to a new situation. The task is framed into a reading comprehension task following squad-style extractive qa.

One thing to note: labels of the test set are hidden (leaderboard submission) so I encoded that as an empty list (ropes.py:L125)
Arabic pos dialect
State: closed | Author: mcmillanmajora (CONTRIBUTOR)

A README.md and loading script for the Arabic POS Dialect dataset. The README is missing the sections on personal information, biases, and limitations, as it would probably be better for those to be filled by someone who can read the contents of the dataset and is familiar with Arabic NLP. 

Comments: looks like this PR includes changes about many other files than the oens for Araboc POS Dialect

Can you create a another branch and another PR please ?

Sorry! I'm not sure how I managed to do that. I'll make a new branch.
Adding The Microsoft Terminology Collection dataset.
State: closed | Author: leoxzhao (CONTRIBUTOR)


Comments: I have to manually copy a dataset_infos.json file from other dataset and modify it since the `datasets-cli` isn't able to handle manually downloaded datasets yet (as far as I know).

you can generate the dataset_infos.json file even for dataset with manual data
To do so just specify `--data_dir <path/to/the/folder/containing/the/manual/data>`

Also, dummy_data seems having difficulty to handle manually downloaded datasets. `python datasets-cli dummy_data datasets/ms_terms  --data_dir ...` reported `error: unrecognized arguments: --data_dir` error. Without `--data_dir`, it reported this error:
```
Dataset ms_terms with config BuilderConfig(name='ms_terms-full', version=1.0.0, data_dir=None, data_files=None, description='...\n') seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file None.
Traceback (most recent call last):
  File "datasets-cli", line 36, in <module>
    service.run()
  File "/Users/lzhao/Downloads/huggingface/datasets/src/datasets/commands/dummy_data.py", line 326, in run
    dataset_builder=dataset_builder, mock_dl_manager=mock_dl_manager
  File "/Users/lzhao/Downloads/huggingface/datasets/src/datasets/commands/dummy_data.py", line 406, in _print_dummy_data_instructions
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

Oh yes `--data_dir` seems to only be supported for the `datasets_cli test` command. Sorry about that.

Can you try to build the dummy_data.zip file manually ?

It has to be inside `./datasets/ms_terms/dummy/ms_terms-full/1.0.0`.
Inside this folder, please create a folder `dummy_data` that contains a dummy file `MicrosoftTermCollection.tbx` (with just a few examples in it). Then you can zip the `dummy_data` folder to `dummy_data.zip`

Then you can check if it worked using the command
```
pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ms_terms
```

Feel free to use some debugging print statements in your script if it doesn't work first try to see what `dl_manager.manual_dir` ends up being and also `path_to_manual_file`.

Feel free to ping me if you have other questions

`pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ms_terms` gave `1 passed, 4 warnings in 8.13s`. Existing datasets, like `wikihow`, and `newsroom`, also report 4 warnings. So, I guess that is not related to this dataset.

Could you run `make style` before we merge @leoxzhao ?

the other errors are fixed on master so it's fine

> Could you run `make style` before we merge @leoxzhao ?

Sure thing. Done. Thanks Quentin. I have other datasets in mind. All of which requires manual download. This process is very helpful

Thank you :)
Add Children's Book Test (CBT) dataset
State: closed | Author: thomwolf (MEMBER)

Add the Children's Book Test (CBT) from Facebook (Hill et al. 2016).

Sentence completion given a few sentences as context from a children's book.

Comments: Hi @lhoestq,

I guess this PR can be closed since we merged #2044?

I have used the same link for the homepage, as it is where the dataset is provided, hope that is okay?

Closing in favor of #2044, thanks again :)

> I have used the same link for the homepage, as it is where the dataset is provided, hope that is okay?

Yea it's ok actually, at that time I thought there was another homepage for this dataset
Add CS Restaurants dataset
State: closed | Author: TevenLeScao (CONTRIBUTOR)

This PR adds the Czech restaurants dataset for Czech NLG.

Comments: Oh yeah, for some reason I thought you had to do it after the merge, I'll get on it

Weird, now the CI seems to fail because of other datasets (XGLUE, Norwegian_NER)

Yea you just need to rebase from master

Re-opening a PR without the messed-up rebase
Add CLINC150 Dataset
State: closed | Author: sumanthd17 (CONTRIBUTOR)

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

Comments: Looks like your PR now shows changes in many other files than the ones for CLINC150.
Feel free to create another branch and another PR

created new [PR](https://github.com/huggingface/datasets/pull/1016)

closing this!
Add CLINC150 Dataset
State: closed | Author: sumanthd17 (CONTRIBUTOR)

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data
Adding the WebNLG dataset
State: closed | Author: yjernite (MEMBER)

This PR adds data from the WebNLG challenge, with one configuration per release and challenge iteration.

More information can be found [here](https://webnlg-challenge.loria.fr/)

Unfortunately, the data itself comes from a pretty large number of small XML files, so the dummy data ends up being quite large (8.4 MB even keeping only one example per file).

Comments: This is task is part of the GEM suite so will actually need a more complete dataset card. I'm taking a break for now though and will get back to it before merging :)
sample multiple datasets 
State: closed | Author: rabeehk (CONTRIBUTOR)

Hi
I am dealing with multiple datasets, I need to have a dataloader over them with a condition that in each batch data samples are coming from one of the datasets. My main question is: 
-  I need to have a way to sample the datasets first with some weights, lets say 2x dataset1 1x dataset2, could you point me how I can do it

sub-questions:
- I want to concat sampled datasets and define one dataloader on it, then I need a way to make sure batches come from 1 dataset in each iteration, could you assist me how I can do?
- I use iterative-type of datasets, but I need a method of shuffling still since it brings accuracy performance issues if not doing it, thanks for the help. 

Comments: here I share my dataloader currently for multiple tasks: https://gist.github.com/rabeehkarimimahabadi/39f9444a4fb6f53dcc4fca5d73bf8195 

I need to train my model distributedly with this dataloader, "MultiTasksataloader", currently this does not work in distributed fasion,
to save on memory I tried to use iterative datasets, could you have a look in this dataloader and tell me if this is indeed the case? not sure how to make datasets being iterative to not load them in memory, then I remove the sampler for dataloader, and shard the data per core, could you tell me please how I should implement this case in datasets library? and how do you find my implementation in terms of correctness? thanks 


Hi @rabeehkarimimahabadi any luck with updating the multi-task data loader to work with distributed training?

Hi @pushkalkatara yes I solved it back then, here please find my implementation https://github.com/rabeehk/hyperformer/blob/main/hyperformer/data/multitask_sampler.py 

Thanks @rabeehk for sharing. 

The sampler basically returns a list of integers to sample from each task's dataset. I was wondering how to use it with two `torch.Dataset` of different tasks. Also, do I need to shard across processes while creating an Iterable Dataset?


We now have `interleave_datasets` in the API that allows you to cycle/sample with probabilities (with various stopping strategies) through a list of datasets. However, more specific behavior should be implemented manually.

Hi @mariosasko, @pushkalkatara 

I have multi dataset for ASR task. I have multilingual such as: English, China, Japan, Vietnamese. Each dataset is loaded as dataset and they are imbalance dataset. Now I training, I want to equal sampler, it means each batch loader in training has equal number sampler in each language.

For example: Batch_size = 16:
English sample:4
China sample:4
Japan sample:4
Vietnamese sample:4 

How can I do it? I saw that  interleave_datasets only splits data with probability an concatenates all of them after that, it not affect in training data loader. Forgiving me if I wrong.

Thank you for your help.
Added PragmEval benchmark
State: closed | Author: sileod (CONTRIBUTOR)


Comments: > Really cool ! Thanks for adding this one :)
> Good job at adding all those citations for each task
> 
> Looks like the dummy data test doesn't pass. Maybe some files are missing in the dummy_data.zip files ?
> The error reports `pragmeval/verifiability/train.tsv` to be missing
> 
> Also could you add the tags part of the dataset card (the rest is optional) ?
> See more info here : https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

In the prior commits I generated dataset_infos and the dummy files myself
Now they are generated with the cli, and the tests now seem to be passing better
I will look into the tag


Looks like you did a good job with dummy data in the first place !
The downside of automatically generated dummy data is that the files are heavier (here 40KB per file).
If you could replace the generated dummy files with the one you created yourself it would be awesome, since the one you did yourself are way lighter (around 1KB per file). Using small files make `git clone` run faster so we encourage to use small dummy_data files.

could you rebase from master ? it should fix the CI

> could you rebase from master ? it should fix the CI

I think it is due to the file structure of the dummy data that causes test failure. The automatically generated dummy data pass the tests

Indeed the error reports that `pragmeval/verifiability/train.tsv` is missing for the verifiability dummy_data.zip file.
To fix that you should add the missing data files in each dummy_data.zip file.
To test that your dummy data work you can run
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_
```
if some file is missing it should tell you which one

Also it looks like you haven't rebased from master yet, even though you did a `rebase` commit. 

rebasing should fix the other CI fails

It's ok if we have `RemoteDatasetTest ` errors, they're fixed on master

merging since the CI is fixed on master

Hey @sileod! Super nice to see you participating ;)

Did you officially joined the sprint by posting on [the forum thread](https://discuss.huggingface.co/t/open-to-the-community-one-week-team-effort-to-reach-v2-0-of-hf-datasets-library/2176) and joining our slack?

I can't seem to find you there! Should I add you directly with your gmail address?

Hi @sileod 👋
add prachathai67k
State: closed | Author: cstorm125 (CONTRIBUTOR)

`prachathai-67k`: News Article Corpus and Multi-label Text Classificdation from Prachathai.com
The prachathai-67k dataset was scraped from the news site Prachathai.
We filtered out those articles with less than 500 characters of body text, mostly images and cartoons.
It contains 67,889 articles wtih 12 curated tags from August 24, 2004 to November 15, 2018.
The dataset was originally scraped by @lukkiddd and cleaned by @cstorm125.
You can also see preliminary exploration at https://github.com/PyThaiNLP/prachathai-67k/blob/master/exploration.ipynb

Comments: Test failing for same issues as https://github.com/huggingface/datasets/pull/939
Please advise.

```
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_flue
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_class_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_flue
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_builder_configs_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_flue
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_norwegian_ner
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_xglue
===== 7 failed, 1309 passed, 932 skipped, 11 warnings in 166.71s (0:02:46) =====
```

Closing and opening a new pull request to solve rebase issues

To be continued on https://github.com/huggingface/datasets/pull/982
added health_fact dataset 
State: closed | Author: bhavitvyamalik (CONTRIBUTOR)

Added dataset Explainable Fact-Checking for Public Health Claims (dataset_id: health_fact)

Comments: Hi @lhoestq,
Initially I tried int(-1) only in place of nan labels and missing values but I kept on getting this error ```pyarrow.lib.ArrowTypeError: Expected bytes, got a 'int' object``` maybe because I'm sending int values (-1) to objects which are string type
Prachathai67k
State: closed | Author: cstorm125 (CONTRIBUTOR)

Add `prachathai-67k`: News Article Corpus and Multi-label Text Classificdation from Prachathai.com

The `prachathai-67k` dataset was scraped from the news site [Prachathai](prachathai.com). We filtered out those articles with less than 500 characters of body text, mostly images and cartoons. It contains 67,889 articles wtih 12 curated tags from August 24, 2004 to November 15, 2018. The dataset was originally scraped by [@lukkiddd](https://github.com/lukkiddd) and cleaned by [@cstorm125](https://github.com/cstorm125). Download the dataset [here](https://www.dropbox.com/s/fsxepdka4l2pr45/prachathai-67k.zip?dl=1). You can also see preliminary exploration in [exploration.ipynb](https://github.com/PyThaiNLP/prachathai-67k/blob/master/exploration.ipynb).

This dataset is a part of [pyThaiNLP](https://github.com/PyThaiNLP/) Thai text [classification-benchmarks](https://github.com/PyThaiNLP/classification-benchmarks). For the benchmark, we selected the following tags with substantial volume that resemble **classifying types of articles**:

* `การเมือง` - politics
* `สิทธิมนุษยชน` - human_rights
* `คุณภาพชีวิต` - quality_of_life
* `ต่างประเทศ` - international
* `สังคม` - social
* `สิ่งแวดล้อม` - environment
* `เศรษฐกิจ` - economics
* `วัฒนธรรม` - culture
* `แรงงาน` - labor
* `ความมั่นคง` - national_security
* `ไอซีที` - ict
* `การศึกษา` - education

Comments: Wrongly branching from existing branch of wisesight_sentiment. Closing and opening another one specifically for prachathai67k
The FLUE Benchmark
State: closed | Author: jplu (CONTRIBUTOR)

This PR adds the [FLUE](https://github.com/getalp/Flaubert/tree/master/flue) benchmark which is a set of different datasets to evaluate models for French content.

Two datasets are missing, the French Treebank that we can use only for research purpose and we are not allowed to distribute, and the Word Sense disambiguation for Nouns that will be added later.
D
State: closed | Author: CryptoMiKKi (NONE)

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).