NyssaW/github-issues huggingface dataset:
Wikipedia postprocessing

Hi, thanks for this library!

Running this code:

```py
import datasets
wikipedia = datasets.load_dataset("wikipedia", "20200501.de")
print(wikipedia['train']['text'][0])
```

I get:

```
mini|Ricardo Flores Magón
mini|Mexikanische Revolutionäre, Magón in der Mitte anführend, gegen die Diktatur von Porfirio Diaz, Ausschnitt des Gemälde „Tierra y Libertad“ von Idelfonso Carrara (?) von 1930.

Ricardo Flores Magón (* 16. September 1874 in San Antonio Eloxochitlán im mexikanischen Bundesstaat Oaxaca; † 22. November 1922 im Bundesgefängnis Leavenworth im US-amerikanischen Bundesstaat Kansas) war als Journalist, Gewerkschafter und Literat ein führender anarchistischer Theoretiker und Aktivist, der die revolutionäre mexikanische Bewegung radikal beeinflusste. Magón war Gründer der Partido Liberal Mexicano und Mitglied der Industrial Workers of the World.

Politische Biografie 
Journalistisch und politisch kämpfte er und sein Bruder sehr kompromisslos gegen die Diktatur Porfirio Diaz. Philosophisch und politisch orientiert an radikal anarchistischen Idealen und den Erfahrungen seiner indigenen Vorfahren bei der gemeinschaftlichen Bewirtschaftung des Gemeindelandes, machte er die Forderung „Land und Freiheit“ (Tierra y Libertad) populär. Besonders Francisco Villa und Emiliano Zapata griffen die Forderung Land und Freiheit auf. Seine Philosophie hatte großen Einfluss auf die Landarbeiter. 1904 floh er in die USA und gründete 1906 die Partido Liberal Mexicano. Im Exil lernte er u. a. Emma Goldman kennen. Er verbrachte die meiste Zeit seines Lebens in Gefängnissen und im Exil und wurde 1918 in den USA wegen „Behinderung der Kriegsanstrengungen“ zu zwanzig Jahren Gefängnis verurteilt. Zu seinem Tod gibt es drei verschiedene Theorien. Offiziell starb er an Herzversagen. Librado Rivera, der die Leiche mit eigenen Augen gesehen hat, geht davon aus, dass Magón von einem Mitgefangenen erdrosselt wurde. Die staatstreue Gewerkschaftszeitung CROM veröffentlichte 1923 einen Beitrag, nachdem Magón von einem Gefängniswärter erschlagen wurde.
mini|Die Brüder Ricardo (links) und Enrique Flores Magón (rechts) vor dem Los Angeles County Jail, 1917

[...]
```

so some Markup like `mini|` is still left. Should I run another parser on this text before feeding it to an ML model or is this a known imperfection of parsing Wiki markup?

Apologies if this has been asked before.

Comments:
Hi @bminixhofer ! Parsing WikiMedia is notoriously difficult: this processing used [mwparserfromhell](https://github.com/earwig/mwparserfromhell) which is pretty good but not perfect.

As an alternative, you can also use the Wiki40b dataset which was pre-processed using an un-released Google internal tool

Ok, thanks! I'll try the Wiki40b dataset.

If anyone else is concerned about this, `wiki40b` does indeed seem very well cleaned.
[GEM] add WikiLingua cross-lingual abstractive summarization dataset

## Adding a Dataset
- **Name:** WikiLingua
- **Description:** The dataset includes ~770k article and summary pairs in 18 languages from WikiHow. The gold-standard article-summary alignments across languages were extracted by aligning the images that are used to describe each how-to step in an article.
- **Paper:** https://arxiv.org/pdf/2010.03093.pdf
- **Data:** https://github.com/esdurmus/Wikilingua
- **Motivation:** Included in the GEM shared task. Multilingual.

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
Hey @yjernite. This is a very interesting dataset. Would love to work on adding it but I see that the link to the data is to a gdrive folder. Can I just confirm wether dlmanager can handle gdrive urls or would this have to be a manual dl?

Hi @KMFODA ! A version of WikiLingua is actually already accessible in the [GEM dataset](https://huggingface.co/datasets/gem)

You can use it for example to load the French to English translation with:
```python
from datasets import load_dataset
wikilingua = load_dataset("gem", "wiki_lingua_french_fr")
```

Closed by https://github.com/huggingface/datasets/pull/1807
[GEM] add ASSET text simplification dataset

## Adding a Dataset
- **Name:** ASSET
- **Description:** ASSET is a crowdsourced
multi-reference corpus for assessing sentence simplification in English where each simplification was produced by executing several rewriting transformations.
- **Paper:** https://www.aclweb.org/anthology/2020.acl-main.424.pdf
- **Data:** https://github.com/facebookresearch/asset
- **Motivation:** Included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
[GEM] add WikiAuto text simplification dataset

## Adding a Dataset
- **Name:** WikiAuto
- **Description:** Sentences in English Wikipedia and their corresponding sentences in Simple English Wikipedia that are written with simpler grammar and word choices. A lot of lexical and syntactic paraphrasing. 
- **Paper:** https://www.aclweb.org/anthology/2020.acl-main.709.pdf
- **Data:** https://github.com/chaojiang06/wiki-auto
- **Motivation:** Included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
[GEM] Add WebNLG dataset

## Adding a Dataset
- **Name:** WebNLG
- **Description:** WebNLG consists of Data/Text pairs where the data is a set of triples extracted from DBpedia and the text is a verbalisation of these triples (16,095 data inputs and 42,873 data-text pairs). The data is available in English and Russian
- **Paper:** https://www.aclweb.org/anthology/P17-1017.pdf
- **Data:** https://webnlg-challenge.loria.fr/download/
- **Motivation:** Included in the GEM shared task, multilingual

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
[GEM] add ToTTo Table-to-text dataset

## Adding a Dataset
- **Name:** ToTTo
- **Description:** ToTTo is an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description.
- **Paper:** https://arxiv.org/abs/2004.14373
- **Data:** https://github.com/google-research-datasets/totto
- **Motivation:** Included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
closed via #1098
[GEM] add Schema-Guided Dialogue

## Adding a Dataset
- **Name:** The Schema-Guided Dialogue Dataset
- **Description:** The Schema-Guided Dialogue (SGD) dataset consists of over 20k annotated multi-domain, task-oriented conversations between a human and a virtual assistant. These conversations involve interactions with services and APIs spanning 20 domains, ranging from banks and events to media, calendar, travel, and weather.
- **Paper:** https://arxiv.org/pdf/2002.01359.pdf https://arxiv.org/pdf/2004.15006.pdf
- **Data:** https://github.com/google-research-datasets/dstc8-schema-guided-dialogue
- **Motivation:** Included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
[GEM] MultiWOZ dialogue dataset

## Adding a Dataset
- **Name:** MultiWOZ (Multi-Domain Wizard-of-Oz)
- **Description:** 10k annotated human-human dialogues. Each dialogue consists of a goal, multiple user and system utterances as well as a belief state. Only system utterances are annotated with dialogue acts – there are no annotations from the user side.
- **Paper:** https://arxiv.org/pdf/2007.12720.pdf
- **Data:** https://github.com/budzianowski/multiwoz
- **Motivation:** Will likely be part of the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
Hi @yjernite can I help in adding this dataset? 

I am excited about this because this will be my first contribution to the datasets library as well as to hugginface.

Resolved via https://github.com/huggingface/datasets/pull/979
[GEM] Add E2E dataset

## Adding a Dataset
- **Name:** E2E NLG dataset (for End-to-end natural language generation)
- **Description:**a dataset for training end-to-end, datadriven natural language generation systems in the restaurant domain, the datasets consists of 5,751 dialogue-act Meaning Representations (structured data) and 8.1 reference free-text utterances per dialogue-act on average
- **Paper:** https://arxiv.org/pdf/1706.09254.pdf https://arxiv.org/abs/1901.07931
- **Data:** http://www.macs.hw.ac.uk/InteractionLab/E2E/#data
- **Motivation:** This dataset will likely be included in the GEM shared task

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
Add accuracy, precision, recall and F1 metrics

This PR adds several single metrics, namely:

- Accuracy
- Precision
- Recall
- F1

They all uses under the hood the sklearn metrics of the same name. They allow different useful features when training a multilabel/multiclass model:
- have a macro/micro/per label/weighted/binary/per sample score
- score only the selected labels (usually what we call the positive labels) and ignore the negative ones. For example in case of a Named Entity Recognition task, positive labels are (`PERSON`, `LOCATION` or `ORGANIZATION`) and the negative one is `O`.

Comments:
datasets freezes 

Hi, I want to load these two datasets and convert them to Dataset format in torch and the code freezes for me, could you have a look please? thanks 

dataset1 = load_dataset("squad", split="train[:10]")
dataset1 = dataset1.set_format(type='torch', columns=['context', 'answers', 'question'])

dataset2 = load_dataset("imdb", split="train[:10]")
dataset2 = dataset2.set_format(type="torch", columns=["text", "label"])
print(len(dataset1))


Comments:
Pytorch is unable to convert strings to tensors unfortunately.
You can use `set_format(type="torch")` on columns that can be converted to tensors, such as token ids.

This makes me think that we should probably raise an error or at least a warning when one tries to create pytorch tensors out of text columns

Ultimately, we decided to return a list instead of an error when formatting a string column with the format type `"torch"`.

If you think an error would be more appropriate, please open a new issue.
Make save function use deterministic global vars order

The `dumps` function need to be deterministic for the caching mechanism.
However in #816 I noticed that one of dill's method to recursively check the globals of a function may return the globals in different orders each time it's used. To fix that I sort the globals by key in the `globs` dictionary.
I had to add a rectified `save_function` to the saving functions registry of the Pickler to make it work.

This should fix #816 

Comments:
Sorry, asking for help here, but the dill thread stop around 2013. Is it possible to use dill deterministically? I tried to monkeypatch the solution presented here into dill, but I suppose it requires forking their project.

Hi ! What we did was to subclass `dill`'s Pickler to fix the non-deterministic behaviors, and it's been working fine. A fork should also do the job
Fix type hints pickling in python 3.6

Type hints can't be properly pickled in python 3.6. This was causing errors the `run_mlm.py` script from `transformers` with python 3.6

However Cloupickle proposed a [fix](https://github.com/cloudpipe/cloudpickle/pull/318/files) to make it work anyway.
The idea is just to implement the pickling/unpickling of parameterized type hints. There is one detail though: since in python 3.6 we can't use `isinstance` on type hints, then we can't use pickle saving functions registry directly. Therefore we just wrap the `save_global` method of the Pickler.

This should fix https://github.com/huggingface/transformers/issues/8212 for python 3.6 and make `run_mlm.py` support python 3.6

cc @sgugger 

Comments:
Add MRQA dataset

## Adding a Dataset
- **Name:** MRQA
- **Description:** Collection of different (subsets of) QA datasets all converted to the same format to evaluate out-of-domain generalization (the datasets come from different domains, distributions, etc.). Some datasets are used for training and others are used for evaluation. This dataset was collected as part of MRQA 2019's shared task 
- **Paper:** https://arxiv.org/abs/1910.09753
- **Data:** https://github.com/mrqa/MRQA-Shared-Task-2019
- **Motivation:** Out-of-domain generalization is becoming (has become) a de-factor evaluation for NLU systems

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).

Comments:
Done! cf #1117 and #1022
[Caching] Dill globalvars() output order is not deterministic and can cause cache issues.

Dill uses `dill.detect.globalvars` to get the globals used by a function in a recursive dump. `globalvars` returns a dictionary of all the globals that a dumped function needs. However the order of the keys in this dict is not deterministic and can cause caching issues.

To fix that one could register an implementation of dill's `save_function` in the `datasets` pickler that sorts the globals keys before dumping a function.

Comments:
To show the issue:
```
python -c "from datasets.fingerprint import Hasher; a=[]; func = lambda : len(a); print(Hasher.hash(func))"
```
doesn't always return the same ouput since `globs` is a dictionary with "a" and "len" as keys but sometimes not in the same order
How to implement DistributedSampler with datasets 

Hi,
I am using your datasets to define my dataloaders, and I am training finetune_trainer.py in huggingface repo on them.
I need a distributedSampler to be able to train the models on TPUs being able to distribute the load across the TPU cores. Could you tell me how I can implement the distribued sampler when using datasets in which datasets are iterative? To give you more context, I have multiple of datasets and I need to write sampler for this case. thanks. 

Comments:
Hi Apparently I need to shard the data and give one host a chunk, could you provide me please with examples on how to do it? I want to use it jointly with finetune_trainer.py in huggingface repo seq2seq examples. thanks. 

Hey @rabeehkarimimahabadi I'm actually looking for the same feature. Did you manage to get somewhere?

@rabeehkarimimahabadi need the same feature

Hi! I think you can use the `accelerate` library for that, which implements distributed sampling.
nlp viewer error

Hello, 
when I select amazon_us_reviews in nlp viewer, it shows error.
https://huggingface.co/nlp/viewer/?dataset=amazon_us_reviews
![image](https://user-images.githubusercontent.com/30210529/98447334-4aa81200-2124-11eb-9dca-82c3ab34ccc2.png)


Comments:
and also for 'blog_authorship_corpus'
https://huggingface.co/nlp/viewer/?dataset=blog_authorship_corpus
![image](https://user-images.githubusercontent.com/30210529/98557329-5c182800-22a4-11eb-9b01-5b910fb8fcd4.png)


Is this the problem of my local computer or ??

Related to:
- #673
Fix seqeval metric

The current seqeval metric returns the following error when computed:
```
~/.cache/huggingface/modules/datasets_modules/metrics/seqeval/78a944d83252b5a16c9a2e49f057f4c6e02f18cc03349257025a8c9aea6524d8/seqeval.py in _compute(self, predictions, references, suffix)
    102         scores = {}
    103         for type_name, score in report.items():
--> 104             scores[type_name]["precision"] = score["precision"]
    105             scores[type_name]["recall"] = score["recall"]
    106             scores[type_name]["f1"] = score["f1-score"]

KeyError: 'LOC'
```
This is because the current code basically tries to do:
```
scores = {}
scores["LOC"]["precision"] = some_value
```
which does not work in python. This PR fixes that while keeping the previous nested structure of results, with the same keys.

Comments:
Add Google Taskmaster dataset

## Adding a Dataset
- **Name:** Taskmaster
- **Description:** A large dataset of task-oriented dialogue with annotated goals (55K dialogues covering entertainment and travel reservations)
- **Paper:** https://arxiv.org/abs/1909.05358
- **Data:** https://github.com/google-research-datasets/Taskmaster
- **Motivation:** One of few annotated datasets of this size for goal-oriented dialogue

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
Hey @yjernite. Was going to start working on this but found taskmaster 1,2 & 3 in the datasets library already so think this can be closed now?

You are absolutely right :) 

Closed by https://github.com/huggingface/datasets/pull/1193 https://github.com/huggingface/datasets/pull/1197 https://github.com/huggingface/datasets/pull/1213
dataset(dgs): initial dataset loading script

When trying to create dummy data I get:

> Dataset datasets with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has t o be created with less guidance. Make sure you create the file dummy_data.

I am not sure how to manually create the dummy_data (what exactly it should contain)

Also note, this library says:
> ImportError: To be able to use this dataset, you need to install the following dependencies['pympi'] using 'pip install pympi' for instance'

When you actually need to `pip install pympi-ling`


Comments:
Hi @AmitMY, 

Were you able to figure this out?

I did not.
With all the limitations this repo currently has, I had to create a repo of my own using tfds to mitigate them. 
https://github.com/sign-language-processing/datasets/tree/master/sign_language_datasets/datasets/dgs_corpus

Closing as I don't know how to support this PR further
Quail dataset urls are out of date

<h3>Code</h3>

```
from datasets import load_dataset
quail = load_dataset('quail')
```

<h3>Error</h3>

```
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/text-machine-lab/quail/master/quail_v1.2/xml/ordered/quail_1.2_train.xml
```


As per [quail v1.3 commit](https://github.com/text-machine-lab/quail/commit/506501cfa34d9ec6c042d31026ba6fea6bcec8ff) it looks like the location and suggested ordering has changed. In [https://github.com/huggingface/datasets/blob/master/datasets/quail/quail.py#L52-L58](https://github.com/huggingface/datasets/blob/master/datasets/quail/quail.py#L52-L58) the quail v1.2 datasets are being pointed to, which don't exist anymore.

Comments:
Hi ! Thanks for reporting.
We should fix the urls and use quail 1.3.
If you want to contribute feel free to fix the urls and open a PR :) 

Done! PR [https://github.com/huggingface/datasets/pull/820](https://github.com/huggingface/datasets/pull/820)

Updated links and also regenerated the metadata and dummy data for v1.3 in order to pass verifications as described here: [https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset](https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset). 

Closing since #820 is merged.
Thanks again for fixing the urls :)
On loading a metric from datasets, I get the following error

`from datasets import load_metric`

`metric = load_metric('bleurt')`

Traceback:
210 class _ArrayXDExtensionType(pa.PyExtensionType):
    211 
    212     ndims: int = None

AttributeError: module 'pyarrow' has no attribute 'PyExtensionType'

Any help will be appreciated. Thank you. 

Comments:
Hi ! We support only pyarrow > 0.17.1 so that we have access to the `PyExtensionType` object.
Could you update pyarrow and try again ?
```
pip install --upgrade pyarrow
```
Empty output/answer in TriviaQA test set (both in 'kilt_tasks' and 'trivia_qa')

# The issue

It's all in the title, it appears to be fine on the train and validation sets.

Is there some kind of mapping to do like for the questions (see https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md) ? 

# How to reproduce
```py
from datasets import load_dataset
kilt_tasks = load_dataset("kilt_tasks")
trivia_qa = load_dataset('trivia_qa', 'unfiltered.nocontext')
# both in "kilt_tasks"
In [18]: any([output['answer'] for output in kilt_tasks['test_triviaqa']['output']])                                                                                                                        
Out[18]: False
# and "trivia_qa"
In [13]: all([answer['value'] == '<unk>' for answer in trivia_qa['test']['answer']])                                                                                                                        
Out[13]: True
# appears to be fine on the train and validation sets.
In [14]: all([answer['value'] == '<unk>' for answer in trivia_qa['train']['answer']])                                                                                                                       
Out[14]: False

In [15]: all([answer['value'] == '<unk>' for answer in trivia_qa['validation']['answer']])                                                                                                                  
Out[15]: False

In [16]: any([output['answer'] for output in kilt_tasks['train_triviaqa']['output']])                                                                                                                       
Out[16]: True

In [17]: any([output['answer'] for output in kilt_tasks['validation_triviaqa']['output']])                                                                                                                  
Out[17]: True

```

Comments:
cc @yjernite is this expected ?

Yes: TriviaQA has a private test set for the leaderboard [here](https://competitions.codalab.org/competitions/17208)

For the KILT training and validation portions, you need to link the examples from the TriviaQA dataset as detailed here:
https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md

Oh ok, I guess I read the paper too fast 😅, thank you for your answer!
Add XGlue

Dataset is ready to merge. An important feature of this dataset is that for each config the train data is in English, while dev and test data are in multiple languages. Therefore, @lhoestq and I decided offline that we will give the dataset the following API, *e.g.* for 

```python
load_dataset("xglue", "ner")  # would give the splits 'train', 'validation.en', 'test.en', 'validation.es', 'test.es', ... 
```
=> therefore one can load a single language test via

```python
load_dataset("xglue", "ner", split="test.es")
```

Close #749.

Comments:
Really cool to add XGlue, this will be a nice addition !

Splits shouldn't depend on the language. There must be configurations for each language, as we're doing for xnli, xtreme, etc.
For example for XGlue we'll have these configurations: NER.de, NER.en etc.
How to join two datasets?

Hi,

I'm wondering if it's possible to join two (preprocessed) datasets with the same number of rows but different labels? 

I'm currently trying to create paired sentences for BERT from `wikipedia/'20200501.en`, and I couldn't figure out a way to create a paired sentence using `.map()` where the second sentence is **not** the next sentence (i.e., from a different article) of the first sentence.

Thanks!

Comments:
Hi this is also my question. thanks 

Hi ! Currently the only way to add new fields to a dataset is by using `.map` and picking items from the other dataset


Closing this one. Feel free to re-open if you have other questions about this issue.

Also linking another discussion about joining datasets: #853
Token classification labels are strings and we don't have the list of labels

Not sure if this is an issue we want to fix or not, putting it here so it's not forgotten. Right now, in token classification datasets, the labels for NER, POS and the likes are typed as `Sequence` of `strings`, which is wrong in my opinion. These should be `Sequence` of `ClassLabel` or some types that gives easy access to the underlying labels.

The main problem for preprocessing those datasets is that the list of possible labels is not stored inside the `Dataset` object which makes converting the labels to IDs quite difficult (you either have to know the list of labels in advance or run a full pass through the dataset to get the list of labels, the `unique` method being useless with the type `Sequence[str]`).

Comments:
Indeed. Pinging @stefan-it here if he want to give an expert opinion :)

Related is https://github.com/huggingface/datasets/pull/636

Should definitely be a ClassLabel 👍 

Already done.
Descriptions of raw and processed versions of wikitext are inverted

Nothing of importance, but it looks like the descriptions of wikitext-n-v1 and wikitext-n-raw-v1 are inverted for both n=2 and n=103. I just verified by loading them and the `<unk>` tokens are present in the non-raw versions, which confirms that it's a mere inversion of the descriptions and not of the datasets themselves.

Also it would be nice if those descriptions appeared in the dataset explorer.

https://github.com/huggingface/datasets/blob/87bd0864845ea0a1dd7167918dc5f341bf807bd3/datasets/wikitext/wikitext.py#L52

Comments:
Yes indeed ! Thanks for reporting

Fixed by:
- #3241
[Datasets] fix discofuse links

The discofuse links were changed: https://github.com/google-research-datasets/discofuse/commit/d27641016eb5b3eb2af03c7415cfbb2cbebe8558. 
The old links are broken

I changed the links and created the new dataset_infos.json.

Pinging @thomwolf @lhoestq for notification.

Comments:
add amazon reviews

Adds the Amazon US Reviews dataset as requested in #353. Converted from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/amazon_us_reviews). cc @clmnt @sshleifer 

Comments:
@patrickvonplaten Yeah this is adapted from tfds so a lot is just how they wrote the code. Addressed your comments and also simplified the weird `AmazonUSReviewsConfig` definition. Will merge once tests pass.

Thanks for checking this one :) 
Looks good to me 

Just one question : is there a particular reason to use `names=["Y", "N"]` in this order ? Usually the positive label is at index 1 and the negative one at index 0 for binary classification

> is there a particular reason to use `names=["Y", "N"]` in this order ? Usually the positive label is at index 1 and the negative one at index 0 for binary classification

Hmm that's a good point. I'll submit a quick fix.
Error running pip install -e ".[dev]" on MacOS 10.13.6: faiss/python does not exist

I was following along with https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset when I ran into this error.

```sh
git clone https://github.com/huggingface/datasets
cd datasets
virtualenv venv -p python3 --system-site-packages
source venv/bin/activate
pip install -e ".[dev]"
```


![image](https://user-images.githubusercontent.com/59632/97868518-72871800-1cd5-11eb-9cd2-37d4e9d20b39.png)

![image](https://user-images.githubusercontent.com/59632/97868592-977b8b00-1cd5-11eb-8f3c-0c409616149c.png)

Python 3.7.7


Comments:
I saw that `faiss-cpu` 1.6.4.post2 was released recently to fix the installation on macos. It should work now

Closing this one.
Feel free to re-open if you still have issues
dataset(ncslgr): add initial loading script

Its a small dataset, but its heavily annotated
https://www.bu.edu/asllrp/ncslgr.html

![image](https://user-images.githubusercontent.com/5757359/97838609-3c539380-1ce9-11eb-885b-a15d4c91ea49.png)


Comments:
Hi @AmitMY, sorry for leaving you hanging for a minute :) 

We've developed a new pipeline for adding datasets with a few extra steps, including adding a dataset card. You can find the full process [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md)

Would you be up for adding the tags and description in the README.md so we can merge this cool dataset?

@lhoestq should be ready for another review :) 

Awesome thank you !

It looks like the PR now includes changes from other PR that were previously merged. 
Feel free to create another branch and another PR so that we can have a clean diff.


Closing for #958
failed to reuse cache

I packed the `load_dataset ` in a function of class, and cached data in a directory. But when I import the class and use the function, the data still have to be downloaded again. The information (Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.28 GiB, post-processed: Unknown size, total: 1.82 GiB) to ******) which logged to terminal shows the path is right to the cache directory, but the files still have to be downloaded again.

Comments:
Adding nli_tr dataset

Hello,

In this pull request, we have implemented the necessary interface to add our recent dataset [NLI-TR](https://github.com/boun-tabi/NLI-TR).  The datasets will be presented on a full paper at EMNLP 2020 this month. [[arXiv link] ](https://arxiv.org/pdf/2004.14963.pdf)

The dataset is the neural machine translation of SNLI and MultiNLI datasets into Turkish.  So, we followed a similar format with the original datasets hosted in the HuggingFace datasets hub.  

Our dataset is designed to be accessed as follows by following the interface of the GLUE dataset that provides multiple datasets in a single interface over the HuggingFace datasets hub.  

```
from datasets import load_dataset
multinli_tr = load_dataset("nli_tr", "multinli_tr")
snli_tr = load_dataset("nli_tr", "snli_tr")
```

Thanks for your help in reviewing our pull request.

Comments:
Thank you @lhoestq for the time you take to review our pull request. We appreciate your help.

We've made the changes you described. Hope that it is ready for being merged. Please let me know if you have any additional requests for revisions.
feat(dataset): multiprocessing _generate_examples

forking this out of #741, this issue is only regarding multiprocessing

I'd love if there was a dataset configuration parameter `workers`, where when it is `1` it behaves as it does right now, and when its `>1` maybe `_generate_examples` can also get the `pool` and return an iterable using the pool.

In my use case, I would instead of:
```python
for datum in data:
     yield self.load_datum(datum)
```
do:
```python
return pool.map(self.load_datum, data)
```

As the dataset in question, as an example, has **only** 7000 rows, and takes 10 seconds to load each row on average, it takes almost 20 hours to load the entire dataset.
If this was a larger dataset (and many such datasets exist), it would take multiple days to complete.

Using multiprocessing, for example, 40 cores, could speed it up dramatically. For this dataset, hopefully to fully load in under an hour.

Comments:
I agree that would be cool :)
Right now the only distributed dataset builder is based on Apache Beam so you can use distributed processing frameworks like Dataflow, Spark, Flink etc. to build your dataset but it's not really well suited for single-worker parallel processing afaik

`_generate_examples` can now be run in parallel thanks to https://github.com/huggingface/datasets/pull/5107. You can find more info [here](https://huggingface.co/docs/datasets/dataset_script#sharding).
feat(aslg_pc12): add dev and test data splits

For reproducibility sake, it's best if there are defined dev and test splits.

The original paper author did not define splits for the entire dataset, not for the sample loaded via this library, so I decided to define:
- 5/7th for train
- 1/7th for dev
- 1/7th for test


Comments:
Hi ! I'm not sure we should make this split decision arbitrarily on our side. Users can split it afterwards to whatever they want using `dataset.train_test_split` for example.
Moreover it looks like there's already papers that use this dataset and propose their own splits ([here](http://xanthippi.ceid.upatras.gr/HealthSign/resources/Publications/sitis_paper_25_10.pdf) 80-20) 
What do you think ?

I was not aware of the `train_test_split` method, thanks!
Soe ven though it contributes to reproducibility, no need to do this split then.
Add XNLI train set

I added the train set that was built using the translated MNLI.
Now you can load the dataset specifying one language:
```python
from datasets import load_dataset

xnli_en = load_dataset("xnli", "en")
print(xnli_en["train"][0])
# {'hypothesis': 'Product and geography are what make cream skimming work .', 'label': 1, 'premise': 'Conceptually cream skimming has two basic dimensions - product and geography .'}
print(xnli_en["test"][0])                                                                                                                   
# {'hypothesis': 'I havent spoken to him again.', 'label': 2, 'premise': "Well, I wasn't even thinking about that, but I was so frustrated, and, I ended up talking to him again."}
```

Cc @sgugger 

Comments:
Hi! Thanks for adding the translated MNLI! Do you know what translations system / model you used when you created the datasets in the other languages?

According to the [paper](https://arxiv.org/pdf/1809.05053.pdf) it's the result of the work of professional translators ;)

Thanks for getting back to me.

The training data is not from translators. And it appears to be machine
translation for all languages. If we can know what system was used to
create the training data that would be great!

Yifan.


On Thu, Jun 9, 2022, 05:51 Quentin Lhoest ***@***.***> wrote:

> According to the paper <https://arxiv.org/pdf/1809.05053.pdf> it's the
> result of the work of professional translators ;)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/781#issuecomment-1150914429>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKLKWDAPTMGB6BE5GJ4GULVOG5BLANCNFSM4TE67NMQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


> The training data is not from translators.

What makes you think that ? The paper litteraly says

> we hire translators to translate the resulting sentences into 15 languages using the One Hour Translation platform.

However the annotators only did test and validation sets, as this was what
in the paper: “we construct an evaluation set for XLU by extending the
development and test sets of the Multi-Genre Natural Language Inference
Corpus (MultiNLI) to 15 languages".

On Thu, Jun 9, 2022 at 10:35 AM Quentin Lhoest ***@***.***>
wrote:

> The training data is not from translators.
>
> What makes you think that ? The paper litteraly says
>
> we hire translators to translate the resulting sentences into 15 languages
> using the One Hour Translation platform.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/781#issuecomment-1151202195>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKLKWFZOQPLK4WSKFRLW6DVOH6LLANCNFSM4TE67NMQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
Feature/fidelity metrics from emnlp2020 evaluating and characterizing human rationales

This metric computes fidelity (Yu et al. 2019, DeYoung et al. 2019) and normalized fidelity (Carton et al. 2020).

Comments:
Hi ! This looks interesting, thanks for adding it :) 

For metrics there should only be two features fields: references and predictions.
Both of them can be defined as you want using nested structures if you need to.
Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for

> Hi ! This looks interesting, thanks for adding it :)
> 
> For metrics there should only be two features fields: references and predictions.
> Both of them can be defined as you want using nested structures if you need to.
> Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
> All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for

The `predictions` are the predicted labels by a model for a particular input. Do you mean making `prob_y_hat` - the probability of the prediction being the predicted label, `prob_y_hat_alpha` - the probability of the prediction being the predicted label when the input is reduced subject to alpha and the `null_difference` is the difference between the probability of the prediction being the predicted label in full information minus the probability in zero information a part of references? Also, I have added the description for other parameters in kwargs_description. I can expand it if that makes sense?

I think every value that is generated by the model (so label, prob_y_hat, prob_y_hat_alpha etc.) should be in `predictions`.
Feel free to add more details in the kwargs_description, this is very useful for the end user.

Hi @lhoestq , I have updated the code according to your feedback. Please, let me know if it looks good and can be merged now.

Metrics are deprecated in `datasets` and `evaluate` should be used instead: https://github.com/huggingface/evaluate
Unexpected behavior when loading cached csv file?

I read a csv file from disk and forgot so specify the right delimiter. When i read the csv file again specifying the right delimiter it had no effect since it was using the cached dataset. I am not sure if this is unwanted behavior since i can always specify `download_mode="force_redownload"`. But i think it would be nice if the information what `delimiter` or what `column_names` were used would influence the identifier of the cached dataset.

Small snippet to reproduce the behavior:
```python
import datasets

with open("dummy_data.csv", "w") as file:
    file.write("test,this;text\n")

print(datasets.load_dataset("csv", data_files="dummy_data.csv", split="train").column_names)
# ["test", "this;text"]

print(datasets.load_dataset("csv", data_files="dummy_data.csv", split="train", delimiter=";").column_names)
# still ["test", "this;text"]
```

By the way, thanks a lot for this amazing library! :)

Comments:
Hi ! Thanks for reporting.
The same issue was reported in #730 (but with the encodings instead of the delimiter). It was fixed by #770 .
The fix will be available in the next release :)

Thanks for the prompt reply and terribly sorry for the spam! 
Looking forward to the new release!
Properly delete metrics when a process is killed

Tests are flaky when using metrics in distributed setup.
There is because of one test that make sure that using two possibly incompatible metric computation (same exp id) either works or raises the right error.
However if the error is raised, all the processes of the metric are killed, and the open files (arrow + lock files) are not closed correctly. This causes PermissionError on Windows when deleting the temporary directory.

To fix that I added a `finally` clause in the function passed to multiprocess to properly close the files when the process exits.

Comments:
Adding CC-100: Monolingual Datasets from Web Crawl Data

## Adding a Dataset
- **Name:** CC-100: Monolingual Datasets from Web Crawl Data
- **Description:** https://twitter.com/alex_conneau/status/1321507120848625665
- **Paper:** https://arxiv.org/abs/1911.02116
- **Data:** http://data.statmt.org/cc-100/
- **Motivation:** A large scale multi-lingual language modeling dataset. Text is de-duplicated and filtered by how "Wikipedia-like" it is, hopefully helping avoid some of the worst parts of the common crawl.

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
cc @aconneau ;) 

These dataset files are no longer available.  https://data.statmt.org/cc-100/ files provided in this link are no longer available. Can anybody fix that issue?
@abhishekkrthakur @yjernite 

Hi ! Can you open an issue to report this problem ? This will help keep track of the fix :)

Ok
Using `Dataset.map` with `n_proc>1` print multiple progress bars

When using `Dataset.map` with `n_proc > 1`, only one of the processes should print a progress bar (to make the output readable). Right now, `n_proc` progress bars are printed.

Comments:
Yes it allows to monitor the speed of each process. Currently each process takes care of one shard of the dataset.

At one point we can consider using streaming batches to a pool of processes instead of sharding the dataset in `num_proc` parts. At that point it will be easy to use only one progress bar

Hi @lhoestq, I am facing a similar issue, it is annoying when lots of progress bars are printed. Is there a way to turn off this behavior? 

You can disable the progress bars with
```python
import datasets

datasets.disable_progress_bar()
```
How to choose proper download_mode in function load_dataset?

Hi, I am a beginner to datasets and I try to use datasets to load my csv file.
my csv file looks like this

``` 
text,label
"Effective but too-tepid biopic",3
"If you sometimes like to go to the movies to have fun , Wasabi is a good place to start .",4
"Emerges as something rare , an issue movie that 's so honest and keenly observed that it does n't feel like one .",5
```

First I try to use this command to load my csv file .  

``` python
dataset=load_dataset('csv', data_files=['sst_test.csv'])
```

It seems good, but when i try to overwrite the convert_options to convert  'label' columns from int64 to float32 like this.

``` python
import pyarrow as pa
from pyarrow import csv
read_options = csv.ReadOptions(block_size=1024*1024)
parse_options = csv.ParseOptions()
convert_options = csv.ConvertOptions(column_types={'text': pa.string(), 'label': pa.float32()})
dataset = load_dataset('csv', data_files=['sst_test.csv'], read_options=read_options,
                       parse_options=parse_options, convert_options=convert_options)
```

It keeps the same:

```shell
Dataset(features: {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}, num_rows: 2210)
```

I think this issue is caused by the parameter "download_mode" Default to REUSE_DATASET_IF_EXISTS because after I delete the cache_dir, it seems right.

Is it a bug? How to choose proper download_mode to avoid this issue?


Comments:
`download_mode=datasets.GenerateMode.FORCE_REDOWNLOAD` should work.
This makes me think we we should rename this to DownloadMode.FORCE_REDOWNLOAD. Currently that's confusing

Can we just use `features=...` in `load_dataset` for this @lhoestq?

Indeed you should use `features` in this case. 
```python
features = Features({'text': Value('string'), 'label': Value('float32')})
dataset = load_dataset('csv', data_files=['sst_test.csv'], features=features)
```
Note that because of an issue with the caching when you change the features (see #750 ) you still need to specify the `FORCE_REDOWNLOAD ` flag. I'm working on a fix for this one

https://github.com/huggingface/datasets/issues/769#issuecomment-717837832
> This makes me think we we should rename this to DownloadMode.FORCE_REDOWNLOAD. Currently that's confusing

@lhoestq do you still think we should rename it?


It's no big deal, but since it can be confusing to users I think it's worth renaming it, and deprecate `GenerateMode` until `datasets` 2.0 at least. IMO it's confusing to have `download_mode=GenerateMode.something`
Add a `lazy_map` method to `Dataset` and `DatasetDict`

The library is great, but it would be even more awesome with a `lazy_map` method implemented on `Dataset` and `DatasetDict`. This would apply a function on a give item but when the item is requested. Two use cases:

1. load image on the fly
2. apply a random function and get different outputs at each epoch (like data augmentation or randomly masking a part of a sentence for BERT-like objectives).

Comments:
This is cool! I think some aspects to think about and decide in terms of API are:
- do we allow several methods (chained i guess)
- how do we inspect the currently set method(s)
- how do we control/reset them
Add option for named splits when using ds.train_test_split

### Feature Request 🚀 

Can we add a way to name your splits when using the `.train_test_split` function?

In almost every use case I've come across, I have a `train` and a `test` split in my `DatasetDict`, and I want to create a `validation` split. Therefore, its kinda useless to get a `test` split back from `train_test_split`, as it'll just overwrite my real `test` split that I intended to keep.

### Workaround

this is my hack for dealin with this, for now :slightly_smiling_face:

```python
from datasets import load_dataset
​
​
ds = load_dataset('imdb')
ds['train'], ds['validation'] = ds['train'].train_test_split(.1).values()
```


Comments:
Yes definitely we should give more flexibility to control the name of the splits outputted by `train_test_split`.

Related is the very interesting feedback from @bramvanroy on how we should improve this method: https://discuss.huggingface.co/t/how-to-split-main-dataset-into-train-dev-test-as-datasetdict/1090/5

And in particular that it should advantageously be able to split in 3 splits as well instead of just 2 like we copied from sklearn.
[GEM] add DART data-to-text generation dataset

## Adding a Dataset
- **Name:** DART
- **Description:** DART consists of 82,191 examples across different domains with each input being a semantic RDF triple set derived from data records in tables and the tree ontology of the schema, annotated with sentence descriptions that cover all facts in the triple set.
- **Paper:** https://arxiv.org/abs/2007.02871v1
- **Data:** https://github.com/Yale-LILY/dart
- **Motivation:** the dataset will likely be included in the GEM benchmark

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
Is this a duplicate of #924 ?

Yup, closing! Haven't been keeping track of the solved issues during the sprint.
[GEM] Add DART data-to-text generation dataset

## Adding a Dataset
- **Name:** DART
- **Description:** DART consists of 82,191 examples across different domains with each input being a semantic RDF triple set derived from data records in tables and the tree ontology of the schema, annotated with sentence descriptions that cover all facts in the triple set.
- **Paper:** https://arxiv.org/abs/2007.02871v1
- **Data:** https://github.com/Yale-LILY/dart
- **Motivation:** It will likely be included in the GEM generation evaluation benchmark

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Comments:
Fixed errors in bertscore related to custom baseline

[bertscore version 0.3.6 ](https://github.com/Tiiiger/bert_score) added support for custom baseline files. This update added extra argument `baseline_path` to BERTScorer class as well as some extra boolean parameters `use_custom_baseline` in functions like `get_hash(model, num_layers, idf, rescale_with_baseline, use_custom_baseline)`.

This PR fix those matching errors in bertscore metric implementation.

Comments:
Downloaded datasets are not usable offline

I've been trying to use the IMDB dataset offline, but after downloading it and turning off the internet it still raises  an error from the ```requests``` library trying to reach for the online dataset.
Is this the intended behavior ?
(Sorry, I wrote the the first version of this issue while still on nlp 0.3.0).

Comments:
Yes currently you need an internet connection because the lib tries to check for the etag of the dataset script online to see if you don't have it locally already.

If we add a way to store the etag/hash locally after the first download, it would allow users to first download the dataset with an internet connection, and still have it working without an internet connection.

I'll let you know when we add this feature.

Already fixed by:
- #1726