helmo/github-issues huggingface dataset:
Add People's Daily NER dataset
State: closed | Author: JetRunner (CONTRIBUTOR)


Comments: > LGTM thanks :)
> 
> 
> 
> Before we merge, could you add a dataset card ? see here for more info: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card
> 
> 
> 
> Note that only the tags at the top of the dataset card are mandatory, if you feel like it's going to take too much time writing the rest to fill it all you can just skip the paragraphs

Nope. I don't think there is a citation. Also, can I do the dataset card later (maybe in bulk)?

We're doing one PR = one dataset to keep track of things. Feel free to add the tags later in this PR if you want to.
Also only the tags are required now, because we don't want people spending too much time on the cards

added @lhoestq 

Merging since the CI is fixed on master
Local machine/cluster Beam Datasets example/tutorial
State: closed | Author: shangw-nvidia (NONE)

Hi,

I'm wondering if https://huggingface.co/docs/datasets/beam_dataset.html has an non-GCP or non-Dataflow version example/tutorial? I tried to migrate it to run on DirectRunner and SparkRunner, however, there were way too many runtime errors that I had to fix during the process, and even so I wasn't able to get either runner correctly producing the desired output.

Thanks!
Shang

Comments: I tried to make it run once on the SparkRunner but it seems that this runner has some issues when it is run locally.
From my experience the DirectRunner is fine though, even if it's clearly not memory efficient.

It would be awesome though to make it work locally on a SparkRunner !
Did you manage to make your processing work ?

We've deprecated the Beam API in `datasets`. As part of it, the Beam datasets have also been converted to non-Beam-based to make using them straightforward.
add PIB dataset
State: closed | Author: thevasudevgupta (CONTRIBUTOR)

This pull request will add PIB dataset.

Comments: Hi, 

I am unable to get success in these tests. Can someone help me by pointing out possible errors?

Thanks

Hi ! you can read the tests by logging in to circleci.

Anyway for information here are the errors : 
```
datasets/pib/pib.py:19:1: F401 'csv' imported but unused
datasets/pib/pib.py:20:1: F401 'json' imported but unused
datasets/pib/pib.py:36:84: W291 trailing whitespace
```
and 
```
FAILED tests/test_file_encoding.py::TestFileEncoding::test_no_encoding_on_file_open
```

To fix the `test_no_encoding_on_file_open` you just have to specify an encoding while opening a text file. For example `encoding="utf-8"`


All suggested changes are done.

Nice ! can you re-generate the dataset_infos.json file to take into account the feature type change ?
```
datasets-cli test ./datasets/pib --save_infos --all_configs --ignore_verifications
```
And also format your code ?
```
make style
```
Add NumerSense
State: closed | Author: joeddav (CONTRIBUTOR)

Adds the NumerSense dataset
- Webpage/leaderboard: https://inklab.usc.edu/NumerSense/
- Paper: https://arxiv.org/abs/2005.00683
- Description: NumerSense is a new numerical commonsense reasoning probing task, with a diagnostic dataset consisting of 3,145 masked-word-prediction probes. Basically, it's a benchmark to see whether your MLM can figure out the right number in a fill-in-the-blank task based on commonsense knowledge (a bird has **two** legs)
[WIP] complex_webqa - Error zipfile.BadZipFile: Bad CRC-32
State: closed | Author: thomwolf (MEMBER)
Labels: name: dataset contribution

Have a string `zipfile.BadZipFile: Bad CRC-32 for file 'web_snippets_train.json'` error when downloading the largest file from dropbox: `https://www.dropbox.com/sh/7pkwkrfnwqhsnpo/AABVENv_Q9rFtnM61liyzO0La/web_snippets_train.json.zip?dl=1`

Didn't managed to see how to solve that.

Putting aside for now.


Comments: Thanks for your contribution, @thomwolf. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest that you create this dataset there. Please, feel free to tell us if you need some help.
Lambada
State: closed | Author: VictorSanh (CONTRIBUTOR)

Added LAMBADA dataset.

A couple of points of attention (mostly because I am not sure)
- The training data are compressed in a .tar file inside the main tar.gz file. I had to manually un-tar the training file to access the examples.
- The dev and test splits don't have the `category` field so I put `None` by default.

Happy to make changes if it doesn't respect the guidelines!

Victor
Add the Multilingual Amazon Reviews Corpus
State: closed | Author: joeddav (CONTRIBUTOR)

- **Name:** Multilingual Amazon Reviews Corpus* (`amazon_reviews_multi`)
- **Description:** A collection of Amazon reviews in English, Japanese, German, French, Spanish and Chinese.
- **Paper:** https://arxiv.org/abs/2010.02573

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.
add inquisitive
State: closed | Author: patil-suraj (CONTRIBUTOR)

Adding inquisitive qg dataset

More info: https://github.com/wjko2/INQUISITIVE

Comments: `dummy_data` right now contains all article files, keeping only the required articles for dummy data fails the dummy data test.
Any idea ?

> `dummy_data` right now contains all article files, keeping only the required articles for dummy data fails the dummy data test.
> Any idea ?

We should definitely find a way to make it work with only a few articles.

If it doesn't work right now for dummy data, I guess it's because it tries to load every single article file ?

If so, then maybe you can use `os.listdir` method to first check all the data files available in the path where the `articles.tgz` file is extracted. Then you can simply iter through the data files and depending on their ID, include them in the train or test set. With this method you should be able to have only a few articles files per split in the dummy data. Does that make sense ?

fixed! so the issue was, `articles_ids` were prepared based on the number of files in articles dir, so for dummy data questions it was not able to load some articles due to incorrect ids and the test was failing
Add DART
State: closed | Author: lhoestq (MEMBER)

- **Name:** *DART*
- **Description:** *DART is a large dataset for open-domain structured data record to text generation.*
- **Paper:** *https://arxiv.org/abs/2007.02871*
- **Data:** *https://github.com/Yale-LILY/dart#leaderboard*

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

Comments: LGTM!
Add CC-100 dataset
State: closed | Author: albertvillanova (MEMBER)
Labels: name: wontfix

Add CC-100.

Close #773 

Comments: Hello @lhoestq, I would like just to ask you if it is OK that I include this feature 9f32ba1 in this PR or you would prefer to have it in a separate one.

I was wondering whether include also a test, but I did not find any test for the other file formats...

Hi ! Sure that would be valuable to support .xz files. Feel free to open a separate PR for this.
And feel free to create the first test case for extracting compressed files if you have some inspiration (maybe create test_file_utils.py ?). We can still spend more time on tests next week when the sprint is over though so don't spend too much time on it.

@lhoestq, DONE! ;) See PR #950.

Thanks for adding support for `.xz` files :)

Feel free to rebase from master to include it in your PR

@lhoestq DONE; I have merged instead, to avoid changing the history of my public PR ;)

Hi @lhoestq, I would need that you generate the dataset_infos.json and the dummy data for this dataset with a bigger computer. Sorry, but my laptop did not succeed...

Thanks for your work @albertvillanova 
We'll definitely look into it after this sprint :)

Looks like #1456 added CC100 already.
The difference with your approach is that this implementation uses the `BuilderConfig` parameters to allow the creation of custom configs for all the languages, without having to specify them in the `BUILDER_CONFIGS` class attribute.
For example even if the dataset doesn't have a config for english already, you can still load the english CC100 with
```python
from datasets import load_dataset

load_dataset("cc100", lang="en")
```

@lhoestq, oops!! I remember having assigned this dataset to me in the Google sheet, besides having mentioned the corresponding issue in the Pull Request... Nevermind! :)

Yes indeed I can see that...
Sorry for noticing that only now 

The code of the other PR ended up being pretty close to yours though
If you want to add more details to the cc100 dataset card or in the script feel to do so, any addition is welcome
Add XOR QA Dataset
State: closed | Author: sumanthd17 (CONTRIBUTOR)

Added XOR Question Answering Dataset. The link to the dataset can be found [here](https://nlp.cs.washington.edu/xorqa/)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data

Comments: Hi @sumanthd17 

Looks like a good start! You will also need to add a Dataset card, following the instructions given [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#manually-tag-the-dataset-and-write-the-dataset-card)

I followed the instructions mentioned there but my dataset isn't showing up in the dropdown list. Am I missing something here? @yjernite 

> I followed the instructions mentioned there but my dataset isn't showing up in the dropdown list. Am I missing something here? @yjernite

The best way is to run the tagging app locally and provide it the location to the `dataset_infos.json` after you've run the CLI:
https://github.com/huggingface/datasets-tagging


This is a really good data card!!

Small changes to make it even better:
- Tags: the dataset has both "original" data and data that is "extended" from a source dataset: TydiQA - you should choose both options in the tagging apps
- The language and annotation creator tags are off: the language here is the questions: I understand it's a mix of crowd-sourced and expert-generated? Is there any machine translation involved? The annotations are the span selections: is that crowd-sourced?
- Personal and sensitive information: there should be a statement there, even if only to say that none could be found or that it only mentions public figures
add dream dataset
State: closed | Author: patil-suraj (CONTRIBUTOR)

Adding Dream: a Dataset and for Dialogue-Based Reading Comprehension

More details:
https://dataset.org/dream/
https://github.com/nlpdata/dream

Comments: > Awesome good job !
> 
> Could you also add a dataset card using the template guide here : https://github.com/huggingface/datasets/blob/master/templates/README_guide.md
> If you can't fill some fields then just leave `[N/A]`

Quick amendment: `[N/A]` is for fields that are not relevant: if you can't find the information just leave `[More Information Needed]`

@lhoestq since datset cards are optional for this sprint I'll add those later. Good for merge.

Indeed we only require the tags to be added now (the yaml part at the top of the dataset card).
Could you add them please ?
You can find more infos here : https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

@lhoestq added tags, I'll fill rest of the info after current sprint :)

The tests are failing tests for other datasets, not this one.

@lhoestq  could you tell me why these tests are failing, they don't seem related to this PR.
wrong length with datasets 
State: closed | Author: rabeehk (CONTRIBUTOR)

Hi
I have a MRPC dataset which I convert it to seq2seq format, then this is of this format:

`Dataset(features: {'src_texts': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 10)
`

I feed it to a dataloader:
```
dataloader = DataLoader(
            train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler,
            collate_fn=self.data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.dataloader_num_workers,
        )
```

now if I type len(dataloader) this is 1, which is wrong, and this needs to be 10. could you assist me please? thanks 


Comments: Also, I cannot first convert it to torch format, since huggingface seq2seq_trainer codes process the datasets afterwards during datacollector function to make it optimize for TPUs. 

sorry I misunderstood length of dataset with dataloader, closed. thanks
Add conll2002
State: closed | Author: lhoestq (MEMBER)

Adding the Conll2002 dataset for NER.

More info here : https://www.clips.uantwerpen.be/conll2002/ner/

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.
Add Swedish NER Corpus
State: closed | Author: abhishekkrthakur (CONTRIBUTOR)


Comments: Yes the use of configs is optional

@abhishekkrthakur we want to keep track of the information that is and isn't in the dataset cards so we're asking everyone to use the full template :) If there is some information in there that you really can't find or don't feel qualified to add, you can just leave the `[More Information Needed]` text
Shall we change the hashing to encoding to reduce potential replicated cache files?
State: open | Author: zhuzilin (NONE)
Labels: name: enhancement

name: generic discussion

Hi there. For now, we are using `xxhash` to hash the transformations to fingerprint and we will save a copy of the processed dataset to disk if there is a new hash value. However, there are some transformations that are idempotent or commutative to each other. I think that encoding the transformation chain as the fingerprint may help in those cases, for example, use `base64.urlsafe_b64encode`. In this way, before we want to save a new copy, we can decode the transformation chain and normalize it to prevent omit potential reuse. As the main targets of this project are the really large datasets that cannot be loaded entirely in memory, I believe it would save a lot of time if we can avoid some write.

If you have interest in this, I'd love to help :).

Comments: This is an interesting idea !
Do you have ideas about how to approach the decoding and the normalization ?

@lhoestq
I think we first need to save the transformation chain to a list in `self._fingerprint`. Then we can
- decode all the current saved datasets to see if there is already one that is equivalent to the transformation we need now.
- or, calculate all the possible hash value of the current chain for comparison so that we could continue to use hashing.
If we find one, we can adjust the list in `self._fingerprint` to it.

As for the transformation reordering rules, we can just start with some manual rules, like two sort on the same column should merge to one, filter and select can change orders.

And for encoding and decoding, we can just manually specify `sort` is 0, `shuffling` is 2 and create a base-n number or use some general algorithm like `base64.urlsafe_b64encode`.

Because we are not doing lazy evaluation now, we may not be able to normalize the transformation to its minimal form. If we want to support that, we can provde a `Sequential` api and let user input a list or transformation, so that user would not use the intermediate datasets. This would look like tf.data.Dataset.
My new dataset PEC
State: closed | Author: zhongpeixiang (CONTRIBUTOR)

A new dataset PEC published in EMNLP 2020.

Comments: How to resolve these failed checks?

Thanks for adding this one :) 

To fix the check_code_quality, please run `make style` with the latest version of black, isort, flake8
To fix the test_no_encoding_on_file_open, make sure to specify the encoding each time you call `open()` on a text file.
For example : `encoding="utf-8"`
To fix the test_load_dataset_pec , you must add the dummy_data.zip file. It is used to test the dataset script and make sure it runs fine. To add it, please refer to the steps in https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-add-a-dataset



Could you also add a dataset card ? you can find a template here : https://github.com/huggingface/datasets/blob/master/templates/README.md

That would be awesome

> Thanks for adding this one :)
> 
> To fix the check_code_quality, please run `make style` with the latest version of black, isort, flake8
> To fix the test_no_encoding_on_file_open, make sure to specify the encoding each time you call `open()` on a text file.
> For example : `encoding="utf-8"`
> To fix the test_load_dataset_pec , you must add the dummy_data.zip file. It is used to test the dataset script and make sure it runs fine. To add it, please refer to the steps in https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-add-a-dataset

Thank you for the detailed suggestion.

I have added dummy_data but it still failed the DistributedDatasetTest check. My dataset has a central file (containing a python dict) that needs to be accessed by each data example. Is it because the central file cannot be distributed (which would lead to a partial dictionary)?

Specifically, the central file contains a dictionary of speakers with their attributes. Each data example is also associated with a speaker. As of now, I keep the central file and data files separately. If I remove the central file by appending the speaker attributes to each data example, then there would be lots of redundancy because there are lots of duplicate speakers in the data files.

The `DistributedDatasetTest` fail and the changes of this PR are not related, there was just a bug in the CI. You can ignore it

> Really cool thanks !
> 
> Could you make the dummy files smaller ? For example by reducing the size of persona.txt ?
> I also left a comment about the files concatenation. It would be cool to replace that with simple iterations through the different files.
> 
> Then once this is done, you can add a dataset card using the template guide here : https://github.com/huggingface/datasets/blob/master/templates/README_guide.md
> If some fields can't be filled, just leave `[N/A]`

Small change: if you don't have the information for a field, please leave `[More Information Needed]` rather than `[N/A]`

The full information can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#manually-tag-the-dataset-and-write-the-dataset-card)
Grindr meeting app web.Grindr
State: closed | Author: jackin34 (NONE)

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).
Add FiNER dataset
State: closed | Author: stefan-it (CONTRIBUTOR)

Hi,

this PR adds "A Finnish News Corpus for Named Entity Recognition" as new `finer` dataset.

The dataset is described in [this paper](https://arxiv.org/abs/1908.04212). The data is publicly available in [this GitHub](https://github.com/mpsilfve/finer-data).

Notice: they provide two testsets. The additional test dataset taken from Wikipedia is named as "test_wikipedia" split.

Comments: > That's really cool thank you !
> 
> Could you also add a dataset card ?
> You can find a template here : https://github.com/huggingface/datasets/blob/master/templates/README.md

The full information for adding a dataset card can be found here :) 
https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#manually-tag-the-dataset-and-write-the-dataset-card


Thanks your suggestions! I've fixed them, and currently working on the dataset card!

@yjernite and @lhoestq I will add the dataset card a bit later in a separate PR if that's ok for you!

Yes I want to re-emphasize if it was not clear that dataset cards are optional for the sprint. 

Only the tags are required for merging a datasets.

Please try to enforce this rule as well @lhoestq and @yjernite 

Yes @stefan-it if you could just add the tags (the yaml part at the top of the dataset card) that'd be perfect :) 

Oh, sorry, will add them now!


Initial README file is now added :) 

the `RemoteDatasetTest ` errors in the CI are fixed on master so it's fine

merging since the CI is fixed on master
Disallow backslash in urls
State: closed | Author: lhoestq (MEMBER)

Following #903 @albertvillanova noticed that there are sometimes bad usage of `os.path.join` in datasets scripts to create URLS. However this should be avoided since it doesn't work on windows.

I'm suggesting a test to make sure we that all the urls don't have backslashes in them in the datasets scripts.
The tests works by adding a callback feature to the MockDownloadManager used to test the dataset scripts. In a download callback I just make sure that the url is valid.

Comments: Looks like the test doesn't detect all the problems fixed by #907 , I'll fix that

Ok found why it doesn't detect the problems fixed by #907 . That's because for all those datasets the urls are actually fine (no backslash) on windows, even if it uses `os.path.join`.

This is because of the behavior of `os.path.join` on windows when the first path ends with a slash : 

```python
import os
os.path.join("https://test.com/foo", "bar.txt")
# 'https://test.com/foo\\bar.txt'
os.path.join("https://test.com/foo/", "bar.txt")
# 'https://test.com/foo/bar.txt'
```

However even though the urls are correct, this is definitely bad practice and we should never use `os.path.join` for urls