helmo/github-issues huggingface dataset:
dataset(dgs): initial dataset loading script
State: closed | Author: AmitMY (CONTRIBUTOR)

When trying to create dummy data I get:

> Dataset datasets with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has t o be created with less guidance. Make sure you create the file dummy_data.

I am not sure how to manually create the dummy_data (what exactly it should contain)

Also note, this library says:
> ImportError: To be able to use this dataset, you need to install the following dependencies['pympi'] using 'pip install pympi' for instance'

When you actually need to `pip install pympi-ling`


Comments: Hi @AmitMY, 

Were you able to figure this out?

I did not.
With all the limitations this repo currently has, I had to create a repo of my own using tfds to mitigate them. 
https://github.com/sign-language-processing/datasets/tree/master/sign_language_datasets/datasets/dgs_corpus

Closing as I don't know how to support this PR further
Quail dataset urls are out of date
State: closed | Author: ngdodd (CONTRIBUTOR)

<h3>Code</h3>

```
from datasets import load_dataset
quail = load_dataset('quail')
```

<h3>Error</h3>

```
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/text-machine-lab/quail/master/quail_v1.2/xml/ordered/quail_1.2_train.xml
```


As per [quail v1.3 commit](https://github.com/text-machine-lab/quail/commit/506501cfa34d9ec6c042d31026ba6fea6bcec8ff) it looks like the location and suggested ordering has changed. In [https://github.com/huggingface/datasets/blob/master/datasets/quail/quail.py#L52-L58](https://github.com/huggingface/datasets/blob/master/datasets/quail/quail.py#L52-L58) the quail v1.2 datasets are being pointed to, which don't exist anymore.

Comments: Hi ! Thanks for reporting.
We should fix the urls and use quail 1.3.
If you want to contribute feel free to fix the urls and open a PR :) 

Done! PR [https://github.com/huggingface/datasets/pull/820](https://github.com/huggingface/datasets/pull/820)

Updated links and also regenerated the metadata and dummy data for v1.3 in order to pass verifications as described here: [https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset](https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset). 

Closing since #820 is merged.
Thanks again for fixing the urls :)
On loading a metric from datasets, I get the following error
State: closed | Author: laibamehnaz (NONE)

`from datasets import load_metric`

`metric = load_metric('bleurt')`

Traceback:
210 class _ArrayXDExtensionType(pa.PyExtensionType):
    211 
    212     ndims: int = None

AttributeError: module 'pyarrow' has no attribute 'PyExtensionType'

Any help will be appreciated. Thank you. 

Comments: Hi ! We support only pyarrow > 0.17.1 so that we have access to the `PyExtensionType` object.
Could you update pyarrow and try again ?
```
pip install --upgrade pyarrow
```
Empty output/answer in TriviaQA test set (both in 'kilt_tasks' and 'trivia_qa')
State: closed | Author: PaulLerner (CONTRIBUTOR)

# The issue

It's all in the title, it appears to be fine on the train and validation sets.

Is there some kind of mapping to do like for the questions (see https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md) ? 

# How to reproduce
```py
from datasets import load_dataset
kilt_tasks = load_dataset("kilt_tasks")
trivia_qa = load_dataset('trivia_qa', 'unfiltered.nocontext')
# both in "kilt_tasks"
In [18]: any([output['answer'] for output in kilt_tasks['test_triviaqa']['output']])                                                                                                                        
Out[18]: False
# and "trivia_qa"
In [13]: all([answer['value'] == '<unk>' for answer in trivia_qa['test']['answer']])                                                                                                                        
Out[13]: True
# appears to be fine on the train and validation sets.
In [14]: all([answer['value'] == '<unk>' for answer in trivia_qa['train']['answer']])                                                                                                                       
Out[14]: False

In [15]: all([answer['value'] == '<unk>' for answer in trivia_qa['validation']['answer']])                                                                                                                  
Out[15]: False

In [16]: any([output['answer'] for output in kilt_tasks['train_triviaqa']['output']])                                                                                                                       
Out[16]: True

In [17]: any([output['answer'] for output in kilt_tasks['validation_triviaqa']['output']])                                                                                                                  
Out[17]: True

```

Comments: cc @yjernite is this expected ?

Yes: TriviaQA has a private test set for the leaderboard [here](https://competitions.codalab.org/competitions/17208)

For the KILT training and validation portions, you need to link the examples from the TriviaQA dataset as detailed here:
https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md

Oh ok, I guess I read the paper too fast 😅, thank you for your answer!
Add XGlue
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

Dataset is ready to merge. An important feature of this dataset is that for each config the train data is in English, while dev and test data are in multiple languages. Therefore, @lhoestq and I decided offline that we will give the dataset the following API, *e.g.* for 

```python
load_dataset("xglue", "ner")  # would give the splits 'train', 'validation.en', 'test.en', 'validation.es', 'test.es', ... 
```
=> therefore one can load a single language test via

```python
load_dataset("xglue", "ner", split="test.es")
```

Close #749.

Comments: Really cool to add XGlue, this will be a nice addition !

Splits shouldn't depend on the language. There must be configurations for each language, as we're doing for xnli, xtreme, etc.
For example for XGlue we'll have these configurations: NER.de, NER.en etc.
How to join two datasets?
State: closed | Author: shangw-nvidia (NONE)

Hi,

I'm wondering if it's possible to join two (preprocessed) datasets with the same number of rows but different labels? 

I'm currently trying to create paired sentences for BERT from `wikipedia/'20200501.en`, and I couldn't figure out a way to create a paired sentence using `.map()` where the second sentence is **not** the next sentence (i.e., from a different article) of the first sentence.

Thanks!

Comments: Hi this is also my question. thanks 

Hi ! Currently the only way to add new fields to a dataset is by using `.map` and picking items from the other dataset


Closing this one. Feel free to re-open if you have other questions about this issue.

Also linking another discussion about joining datasets: #853
Token classification labels are strings and we don't have the list of labels
State: closed | Author: sgugger (CONTRIBUTOR)
Labels: name: enhancement

name: Dataset discussion

Not sure if this is an issue we want to fix or not, putting it here so it's not forgotten. Right now, in token classification datasets, the labels for NER, POS and the likes are typed as `Sequence` of `strings`, which is wrong in my opinion. These should be `Sequence` of `ClassLabel` or some types that gives easy access to the underlying labels.

The main problem for preprocessing those datasets is that the list of possible labels is not stored inside the `Dataset` object which makes converting the labels to IDs quite difficult (you either have to know the list of labels in advance or run a full pass through the dataset to get the list of labels, the `unique` method being useless with the type `Sequence[str]`).

Comments: Indeed. Pinging @stefan-it here if he want to give an expert opinion :)

Related is https://github.com/huggingface/datasets/pull/636

Should definitely be a ClassLabel 👍 

Already done.
Descriptions of raw and processed versions of wikitext are inverted
State: closed | Author: fraboniface (NONE)
Labels: name: dataset bug

Nothing of importance, but it looks like the descriptions of wikitext-n-v1 and wikitext-n-raw-v1 are inverted for both n=2 and n=103. I just verified by loading them and the `<unk>` tokens are present in the non-raw versions, which confirms that it's a mere inversion of the descriptions and not of the datasets themselves.

Also it would be nice if those descriptions appeared in the dataset explorer.

https://github.com/huggingface/datasets/blob/87bd0864845ea0a1dd7167918dc5f341bf807bd3/datasets/wikitext/wikitext.py#L52

Comments: Yes indeed ! Thanks for reporting

Fixed by:
- #3241
[Datasets] fix discofuse links
State: closed | Author: patrickvonplaten (CONTRIBUTOR)

The discofuse links were changed: https://github.com/google-research-datasets/discofuse/commit/d27641016eb5b3eb2af03c7415cfbb2cbebe8558. 
The old links are broken

I changed the links and created the new dataset_infos.json.

Pinging @thomwolf @lhoestq for notification.
add amazon reviews
State: closed | Author: joeddav (CONTRIBUTOR)

Adds the Amazon US Reviews dataset as requested in #353. Converted from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/amazon_us_reviews). cc @clmnt @sshleifer 

Comments: @patrickvonplaten Yeah this is adapted from tfds so a lot is just how they wrote the code. Addressed your comments and also simplified the weird `AmazonUSReviewsConfig` definition. Will merge once tests pass.

Thanks for checking this one :) 
Looks good to me 

Just one question : is there a particular reason to use `names=["Y", "N"]` in this order ? Usually the positive label is at index 1 and the negative one at index 0 for binary classification

> is there a particular reason to use `names=["Y", "N"]` in this order ? Usually the positive label is at index 1 and the negative one at index 0 for binary classification

Hmm that's a good point. I'll submit a quick fix.
Error running pip install -e ".[dev]" on MacOS 10.13.6: faiss/python does not exist
State: closed | Author: shawwn (NONE)

I was following along with https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset when I ran into this error.

```sh
git clone https://github.com/huggingface/datasets
cd datasets
virtualenv venv -p python3 --system-site-packages
source venv/bin/activate
pip install -e ".[dev]"
```


![image](https://user-images.githubusercontent.com/59632/97868518-72871800-1cd5-11eb-9cd2-37d4e9d20b39.png)

![image](https://user-images.githubusercontent.com/59632/97868592-977b8b00-1cd5-11eb-8f3c-0c409616149c.png)

Python 3.7.7


Comments: I saw that `faiss-cpu` 1.6.4.post2 was released recently to fix the installation on macos. It should work now

Closing this one.
Feel free to re-open if you still have issues
dataset(ncslgr): add initial loading script
State: closed | Author: AmitMY (CONTRIBUTOR)

Its a small dataset, but its heavily annotated
https://www.bu.edu/asllrp/ncslgr.html

![image](https://user-images.githubusercontent.com/5757359/97838609-3c539380-1ce9-11eb-885b-a15d4c91ea49.png)


Comments: Hi @AmitMY, sorry for leaving you hanging for a minute :) 

We've developed a new pipeline for adding datasets with a few extra steps, including adding a dataset card. You can find the full process [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md)

Would you be up for adding the tags and description in the README.md so we can merge this cool dataset?

@lhoestq should be ready for another review :) 

Awesome thank you !

It looks like the PR now includes changes from other PR that were previously merged. 
Feel free to create another branch and another PR so that we can have a clean diff.


Closing for #958
failed to reuse cache
State: closed | Author: WangHexie (NONE)

I packed the `load_dataset ` in a function of class, and cached data in a directory. But when I import the class and use the function, the data still have to be downloaded again. The information (Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.28 GiB, post-processed: Unknown size, total: 1.82 GiB) to ******) which logged to terminal shows the path is right to the cache directory, but the files still have to be downloaded again.
Adding nli_tr dataset
State: closed | Author: e-budur (CONTRIBUTOR)

Hello,

In this pull request, we have implemented the necessary interface to add our recent dataset [NLI-TR](https://github.com/boun-tabi/NLI-TR).  The datasets will be presented on a full paper at EMNLP 2020 this month. [[arXiv link] ](https://arxiv.org/pdf/2004.14963.pdf)

The dataset is the neural machine translation of SNLI and MultiNLI datasets into Turkish.  So, we followed a similar format with the original datasets hosted in the HuggingFace datasets hub.  

Our dataset is designed to be accessed as follows by following the interface of the GLUE dataset that provides multiple datasets in a single interface over the HuggingFace datasets hub.  

```
from datasets import load_dataset
multinli_tr = load_dataset("nli_tr", "multinli_tr")
snli_tr = load_dataset("nli_tr", "snli_tr")
```

Thanks for your help in reviewing our pull request.

Comments: Thank you @lhoestq for the time you take to review our pull request. We appreciate your help.

We've made the changes you described. Hope that it is ready for being merged. Please let me know if you have any additional requests for revisions.
feat(dataset): multiprocessing _generate_examples
State: closed | Author: AmitMY (CONTRIBUTOR)

forking this out of #741, this issue is only regarding multiprocessing

I'd love if there was a dataset configuration parameter `workers`, where when it is `1` it behaves as it does right now, and when its `>1` maybe `_generate_examples` can also get the `pool` and return an iterable using the pool.

In my use case, I would instead of:
```python
for datum in data:
     yield self.load_datum(datum)
```
do:
```python
return pool.map(self.load_datum, data)
```

As the dataset in question, as an example, has **only** 7000 rows, and takes 10 seconds to load each row on average, it takes almost 20 hours to load the entire dataset.
If this was a larger dataset (and many such datasets exist), it would take multiple days to complete.

Using multiprocessing, for example, 40 cores, could speed it up dramatically. For this dataset, hopefully to fully load in under an hour.

Comments: I agree that would be cool :)
Right now the only distributed dataset builder is based on Apache Beam so you can use distributed processing frameworks like Dataflow, Spark, Flink etc. to build your dataset but it's not really well suited for single-worker parallel processing afaik

`_generate_examples` can now be run in parallel thanks to https://github.com/huggingface/datasets/pull/5107. You can find more info [here](https://huggingface.co/docs/datasets/dataset_script#sharding).
feat(aslg_pc12): add dev and test data splits
State: closed | Author: AmitMY (CONTRIBUTOR)

For reproducibility sake, it's best if there are defined dev and test splits.

The original paper author did not define splits for the entire dataset, not for the sample loaded via this library, so I decided to define:
- 5/7th for train
- 1/7th for dev
- 1/7th for test


Comments: Hi ! I'm not sure we should make this split decision arbitrarily on our side. Users can split it afterwards to whatever they want using `dataset.train_test_split` for example.
Moreover it looks like there's already papers that use this dataset and propose their own splits ([here](http://xanthippi.ceid.upatras.gr/HealthSign/resources/Publications/sitis_paper_25_10.pdf) 80-20) 
What do you think ?

I was not aware of the `train_test_split` method, thanks!
Soe ven though it contributes to reproducibility, no need to do this split then.
Add XNLI train set
State: closed | Author: lhoestq (MEMBER)

I added the train set that was built using the translated MNLI.
Now you can load the dataset specifying one language:
```python
from datasets import load_dataset

xnli_en = load_dataset("xnli", "en")
print(xnli_en["train"][0])
# {'hypothesis': 'Product and geography are what make cream skimming work .', 'label': 1, 'premise': 'Conceptually cream skimming has two basic dimensions - product and geography .'}
print(xnli_en["test"][0])                                                                                                                   
# {'hypothesis': 'I havent spoken to him again.', 'label': 2, 'premise': "Well, I wasn't even thinking about that, but I was so frustrated, and, I ended up talking to him again."}
```

Cc @sgugger 

Comments: Hi! Thanks for adding the translated MNLI! Do you know what translations system / model you used when you created the datasets in the other languages?

According to the [paper](https://arxiv.org/pdf/1809.05053.pdf) it's the result of the work of professional translators ;)

Thanks for getting back to me.

The training data is not from translators. And it appears to be machine
translation for all languages. If we can know what system was used to
create the training data that would be great!

Yifan.


On Thu, Jun 9, 2022, 05:51 Quentin Lhoest ***@***.***> wrote:

> According to the paper <https://arxiv.org/pdf/1809.05053.pdf> it's the
> result of the work of professional translators ;)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/781#issuecomment-1150914429>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKLKWDAPTMGB6BE5GJ4GULVOG5BLANCNFSM4TE67NMQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


> The training data is not from translators.

What makes you think that ? The paper litteraly says

> we hire translators to translate the resulting sentences into 15 languages using the One Hour Translation platform.

However the annotators only did test and validation sets, as this was what
in the paper: “we construct an evaluation set for XLU by extending the
development and test sets of the Multi-Genre Natural Language Inference
Corpus (MultiNLI) to 15 languages".

On Thu, Jun 9, 2022 at 10:35 AM Quentin Lhoest ***@***.***>
wrote:

> The training data is not from translators.
>
> What makes you think that ? The paper litteraly says
>
> we hire translators to translate the resulting sentences into 15 languages
> using the One Hour Translation platform.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/781#issuecomment-1151202195>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKLKWFZOQPLK4WSKFRLW6DVOH6LLANCNFSM4TE67NMQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
Feature/fidelity metrics from emnlp2020 evaluating and characterizing human rationales
State: closed | Author: rathoreanirudh (NONE)
Labels: name: transfer-to-evaluate

This metric computes fidelity (Yu et al. 2019, DeYoung et al. 2019) and normalized fidelity (Carton et al. 2020).

Comments: Hi ! This looks interesting, thanks for adding it :) 

For metrics there should only be two features fields: references and predictions.
Both of them can be defined as you want using nested structures if you need to.
Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for

> Hi ! This looks interesting, thanks for adding it :)
> 
> For metrics there should only be two features fields: references and predictions.
> Both of them can be defined as you want using nested structures if you need to.
> Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
> All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for

The `predictions` are the predicted labels by a model for a particular input. Do you mean making `prob_y_hat` - the probability of the prediction being the predicted label, `prob_y_hat_alpha` - the probability of the prediction being the predicted label when the input is reduced subject to alpha and the `null_difference` is the difference between the probability of the prediction being the predicted label in full information minus the probability in zero information a part of references? Also, I have added the description for other parameters in kwargs_description. I can expand it if that makes sense?

I think every value that is generated by the model (so label, prob_y_hat, prob_y_hat_alpha etc.) should be in `predictions`.
Feel free to add more details in the kwargs_description, this is very useful for the end user.

Hi @lhoestq , I have updated the code according to your feedback. Please, let me know if it looks good and can be merged now.

Metrics are deprecated in `datasets` and `evaluate` should be used instead: https://github.com/huggingface/evaluate
Unexpected behavior when loading cached csv file?
State: closed | Author: dcfidalgo (CONTRIBUTOR)

I read a csv file from disk and forgot so specify the right delimiter. When i read the csv file again specifying the right delimiter it had no effect since it was using the cached dataset. I am not sure if this is unwanted behavior since i can always specify `download_mode="force_redownload"`. But i think it would be nice if the information what `delimiter` or what `column_names` were used would influence the identifier of the cached dataset.

Small snippet to reproduce the behavior:
```python
import datasets

with open("dummy_data.csv", "w") as file:
    file.write("test,this;text\n")

print(datasets.load_dataset("csv", data_files="dummy_data.csv", split="train").column_names)
# ["test", "this;text"]

print(datasets.load_dataset("csv", data_files="dummy_data.csv", split="train", delimiter=";").column_names)
# still ["test", "this;text"]
```

By the way, thanks a lot for this amazing library! :)

Comments: Hi ! Thanks for reporting.
The same issue was reported in #730 (but with the encodings instead of the delimiter). It was fixed by #770 .
The fix will be available in the next release :)

Thanks for the prompt reply and terribly sorry for the spam! 
Looking forward to the new release!
Properly delete metrics when a process is killed
State: closed | Author: lhoestq (MEMBER)

Tests are flaky when using metrics in distributed setup.
There is because of one test that make sure that using two possibly incompatible metric computation (same exp id) either works or raises the right error.
However if the error is raised, all the processes of the metric are killed, and the open files (arrow + lock files) are not closed correctly. This causes PermissionError on Windows when deleting the temporary directory.

To fix that I added a `finally` clause in the function passed to multiprocess to properly close the files when the process exits.