AniketTec/github-issues huggingface dataset:
Title: Better DuplicateKeysError error to help the user debug the issue
State: closed
Labels: description: New feature or request
name: enhancement

description: Good for newcomers
name: good first issue

As mentioned in https://github.com/huggingface/datasets/issues/2552 it would be nice to improve the error message when a dataset fails to build because there are duplicate example keys.

The current one is
```python
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 48
Keys should be unique and deterministic in nature
```

and we could have something that guides the user to debugging the issue:
```python
DuplicateKeysError: both 42th and 1337th examples have the same keys `48`.
Please fix the dataset script at <path/to/the/dataset/script>
```

Comments:
Title: Multilabel metrics not supported
State: closed
Labels: description: Something isn't working
name: bug

When I try to use a metric like F1 macro I get the following error:

```
TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'
```
There is an explicit casting here:

https://github.com/huggingface/datasets/blob/fc79f61cbbcfa0e8c68b28c0a8257f17e768a075/src/datasets/features.py#L274

And looks like this is because here

https://github.com/huggingface/datasets/blob/fc79f61cbbcfa0e8c68b28c0a8257f17e768a075/metrics/f1/f1.py#L88

the features can only be integers, so we cannot use that F1 for multilabel. Instead, if I create the following F1 (ints replaced with sequence of ints), it will work:

```python
class F1(datasets.Metric):
    def _info(self):
        return datasets.MetricInfo(
            description=_DESCRIPTION,
            citation=_CITATION,
            inputs_description=_KWARGS_DESCRIPTION,
            features=datasets.Features(
                {
                    "predictions": datasets.Sequence(datasets.Value("int32")),
                    "references": datasets.Sequence(datasets.Value("int32")),
                }
            ),
            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html"],
        )

    def _compute(self, predictions, references, labels=None, pos_label=1, average="binary", sample_weight=None):
        return {
            "f1": f1_score(
                references,
                predictions,
                labels=labels,
                pos_label=pos_label,
                average=average,
                sample_weight=sample_weight,
            ),
        }
```


Comments:
Title: load_dataset("web_nlg") NonMatchingChecksumError
State: closed
Labels: description: Something isn't working
name: bug

Hi! It seems the WebNLG dataset gives a NonMatchingChecksumError.

## Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset('web_nlg', name="release_v3.0_en", split="dev")
```

Gives

```
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://gitlab.com/shimorina/webnlg-dataset/-/archive/master/webnlg-dataset-master.zip']
```

## Environment info
- `datasets` version: 1.8.0
- Platform: macOS-11.3.1-x86_64-i386-64bit
- Python version: 3.9.4
- PyArrow version: 3.0.0

Also tested on Linux, with python 3.6.8

Comments:
Title: Keys should be unique error on code_search_net
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
Loading `code_search_net` seems not possible at the moment.

## Steps to reproduce the bug
```python
>>> load_dataset('code_search_net')
Downloading: 8.50kB [00:00, 3.09MB/s]                                                                                                                                           
Downloading: 19.1kB [00:00, 10.1MB/s]                                                                                                                                           
No config specified, defaulting to: code_search_net/all
Downloading and preparing dataset code_search_net/all (download: 4.77 GiB, generated: 5.99 GiB, post-processed: Unknown size, total: 10.76 GiB) to /Users/thomwolf/.cache/huggingface/datasets/code_search_net/all/1.0.0/b3e8278faf5d67da1d06981efbeac3b76a2900693bd2239bbca7a4a3b0d6e52a...
Traceback (most recent call last):         
  File "/Users/thomwolf/Documents/GitHub/datasets/src/datasets/builder.py", line 1067, in _prepare_split
    writer.write(example, key)
  File "/Users/thomwolf/Documents/GitHub/datasets/src/datasets/arrow_writer.py", line 343, in write
    self.check_duplicate_keys()
  File "/Users/thomwolf/Documents/GitHub/datasets/src/datasets/arrow_writer.py", line 354, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 48
Keys should be unique and deterministic in nature
```

## Environment info
- `datasets` version: 1.8.1.dev0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: 2.0.0


Comments:
Title: Fix FileSystems documentation
State: closed

### What this fixes:
This PR resolves several issues I discovered in the documentation on the `datasets.filesystems` module ([this page](https://huggingface.co/docs/datasets/filesystems.html)).

### What were the issues?
When I originally tried implementing the code examples I faced several bugs attributed to:

- out of date [botocore](https://github.com/boto/botocore) call signatures
- capitalization errors in the `S3FileSystem` class name (written as `S3Filesystem` in one place)
- call signature errors for the `S3FileSystem` class constructor (uses parameter `sessions` instead of `session` in some places) (see [`s3fs`](https://s3fs.readthedocs.io/en/latest/api.html#s3fs.core.S3FileSystem) for where this constructor signature is defined)

### Testing/reviewing notes
Instructions for generating the documentation locally: [here](https://github.com/huggingface/datasets/tree/master/docs#generating-the-documentation).

Comments:
Title: Allow for incremental cumulative metric updates in a distributed setup
State: closed
Labels: description: New feature or request
name: enhancement

Currently, using a metric allows for one of the following:
- Per example/batch metrics
- Cumulative metrics over the whole data

What I'd like is to have an efficient way to get cumulative metrics over the examples/batches added so far, in order to display it as part of the progress bar during training/evaluation.

Since most metrics are just an average of per-example metrics (which aren't?), an efficient calculation can be done as follows:
`((score_cumulative * n_cumulative) + (score_new * n_new)) / (n_cumulative+ n_new)`
where `n` and `score`  refer to number of examples and metric score, `cumulative` refers to the cumulative metric and `new` refers to the addition of new examples.

If you don't want to add this capability in the library, a simple solution exists so users can do it themselves:
It is easy to implement for a single process setup, but in a distributed one there is no way to get the correct `n_new`.
The solution for this is to return the number of examples that was used to compute the metrics in `.compute()` by adding the following line here:
https://github.com/huggingface/datasets/blob/5a3221785311d0ce86c2785b765e86bd6997d516/src/datasets/metric.py#L402-L403
```
output["number_of_examples"] = len(predictions)
```
and also remove the log message here so it won't spam:
https://github.com/huggingface/datasets/blob/3db67f5ff6cbf807b129d2b4d1107af27623b608/src/datasets/metric.py#L411

If this change is ok with you, I'll open a pull request.


Comments:
Title: Handling unlabeled datasets
State: closed
Labels: description: New feature or request
name: enhancement

Hi!

Is there a way for datasets to produce unlabeled instances (e.g., the `ClassLabel` can be nullable).

For example, I want to use the MNLI dataset reader ( https://github.com/huggingface/datasets/blob/master/datasets/multi_nli/multi_nli.py ) on a file that doesn't have the `gold_label` field. I tried setting `"label": data.get("gold_label")`, but got the following error:

```
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/builder.py", line 989, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 953, in encode_example
    return encode_nested_example(self, example)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 848, in encode_nested_example
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 848, in <dictcomp>
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 875, in encode_nested_example
    return schema.encode_example(obj)
  File "/home/nfliu/miniconda3/envs/debias/lib/python3.7/site-packages/datasets/features.py", line 653, in encode_example
    if not -1 <= example_data < self.num_classes:
TypeError: '<=' not supported between instances of 'int' and 'NoneType'
```

What's the proper way to handle reading unlabeled datasets, especially for downstream usage with Transformers?

Comments:
Title: Field order issue in loading json
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
The `load_dataset` function expects columns in alphabetical order when loading json files.

Similar bug was previously reported for csv in #623 and fixed in #684.
## Steps to reproduce the bug

For a json file `j.json`,
```
{"c":321, "a": 1, "b": 2}
```
Running the following,
```
f= datasets.Features({'a': Value('int32'), 'b': Value('int32'), 'c': Value('int32')})
json_data = datasets.load_dataset('json', data_files='j.json', features=f)
```


## Expected results
A successful load.
## Actual results
```
File "pyarrow/table.pxi", line 1409, in pyarrow.lib.Table.cast
ValueError: Target schema's field names are not matching the table's field names: ['c', 'a', 'b'], ['a', 'b', 'c']
```

## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-3.10.0-957.1.3.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyArrow version: 3.0.0



Comments:
Title: Dataset load_from_disk is too slow
State: open
Labels: description: Something isn't working
name: bug

@lhoestq 
## Describe the bug
It's not normal that I have to wait 7-8 hours for a dataset to be loaded from disk, as there are no preprocessing steps, it's only loading it with load_from_disk. I have 96 cpus, however only 1 is used for this, which is inefficient. Moreover, its usage is at 1%... This is happening in the context of a language model training, therefore I'm wasting 100$ each time I have to load the dataset from disk again (because the spot instance was stopped by aws and I need to relaunch it for example). 

## Steps to reproduce the bug
Just get the oscar in  spanish (around 150GGB) and try to first save in disk and then load the processed dataset. It's not dependent on the task you're doing, it just depends on the size of the text dataset.

## Expected results
I expect the dataset to be loaded in a normal time, by using the whole machine for loading it, I mean if you store the dataset in multiple files (.arrow) and then load it from multiple files, you can use multiprocessing for that and therefore don't waste so much time. 


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Ubuntu 18
- Python version: 3.8


I've seen you're planning to include a streaming mode for load_dataset, but that only saves the downloading and processing time, that's not being a problem for me, you cannot save the pure loading from disk time, therefore that's not a solution for my use case or for anyone who wants to use your library for training a language model. 

Comments:
Title: `datasets.keyhash.DuplicatedKeysError` for `drop` and `adversarial_qa/adversarialQA`
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
Failure to generate the datasets (`drop` and subset `adversarialQA` from `adversarial_qa`) because of duplicate keys.

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("drop")
load_dataset("adversarial_qa", "adversarialQA")
```

## Expected results
The examples keys should be unique.

## Actual results
```bash
>>> load_dataset("drop")
Using custom data configuration default
Downloading and preparing dataset drop/default (download: 7.92 MiB, generated: 111.88 MiB, post-processed: Unknown size, total: 119.80 MiB) to /home/hf/.cache/huggingface/datasets/drop/default/0.1.0/7a94f1e2bb26c4b5c75f89857c06982967d7416e5af935a9374b9bccf5068026...
Traceback (most recent call last):         
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/load.py", line 751, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 992, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/arrow_writer.py", line 409, in finalize
    self.check_duplicate_keys()
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/arrow_writer.py", line 349, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 28553293-d719-441b-8f00-ce3dc6df5398
Keys should be unique and deterministic in nature
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.7.0
- Platform: Linux-5.4.0-1044-gcp-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyArrow version: 3.0.0


Comments:
Title: Remove task templates if required features are removed during `Dataset.map`
State: closed

This PR fixes a bug reported by @craffel where removing a dataset's columns during `Dataset.map` triggered a `KeyError` because the `TextClassification` template tried to access the removed columns during `DatasetInfo.__post_init__`:

```python
from datasets import load_dataset

# `yelp_polarity` comes with a `TextClassification` template
ds = load_dataset("yelp_polarity", split="test")
ds
# Dataset({
#     features: ['text', 'label'],
#     num_rows: 38000
# })

# Triggers KeyError: 'label' - oh noes!
ds.map(lambda x: {"inputs": 0}, remove_columns=ds.column_names)
```

I wrote a unit test to make sure I could reproduce the error and then patched a fix.

Comments:
Title: Loading partial dataset when debugging
State: open

I am using PyTorch Lightning along with datasets (thanks for so many datasets already prepared and the great splits). 

Every time I execute load_dataset  for the imdb dataset it takes some time even if I specify a split involving very few samples. I guess this due to hashing as per the other issues.

Is there a way to only load part of the dataset on load_dataset? This would really speed up my workflow.
Something like a debug mode would really help. Thanks!

Comments:
Title: Use `Audio` features for `AutomaticSpeechRecognition` task template
State: closed
Labels: description: New feature or request
name: enhancement

In #2533 we added a task template for speech recognition that relies on the file paths to the audio files. As pointed out by @SBrandeis this is brittle as it doesn't port easily across different OS'. 

The solution is to use dedicated `Audio` features when casting the dataset. These features are not yet available in `datasets`, but should be included in the `AutomaticSpeechRecognition` template once they are.

Comments:
Title: Add task template for automatic speech recognition
State: closed

This PR adds a task template for automatic speech recognition. In this task, the input is a path to an audio file which the model consumes to produce a transcription.

Usage:

```python
from datasets import load_dataset
from datasets.tasks import AutomaticSpeechRecognition

ds = load_dataset("timit_asr", split="train[:10]")
# Dataset({
#     features: ['file', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
#     num_rows: 10
# })

task = AutomaticSpeechRecognition(audio_file_column="file", transcription_column="text")
ds.prepare_for_task(task)
# Dataset({
#     features: ['audio_file', 'transcription'],
#     num_rows: 10
# })
```

Comments:
Title: Tokenizer's normalization preprocessor cause misalignment in return_offsets_mapping for tokenizer classification task
State: closed
Labels: description: Something isn't working
name: bug

[This colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) implements a token classification input pipeline extending the logic from [this hugging example](https://huggingface.co/transformers/custom_datasets.html#tok-ner).

The pipeline works fine with most instance in different languages, but unfortunately, [the Japanese Kana ligature (a form of abbreviation? I don't know Japanese well)](https://en.wikipedia.org/wiki/Kana_ligature) break the alignment of `return_offsets_mapping`:
![image](https://user-images.githubusercontent.com/50871412/122904371-db192700-d382-11eb-8917-1775db76db69.png)

Without the try catch block, it riase `ValueError: NumPy boolean array indexing assignment cannot assign 88 input values to the 87 output values where the mask is true`, example shown here [(another colab notebook)](https://colab.research.google.com/drive/1MmOqf3ppzzdKKyMWkn0bJy6DqzOO0SSm?usp=sharing)

It is clear that the normalizer is the process that break the alignment, as it is observed that `tokenizer._tokenizer.normalizer.normalize_str('ヿ')` return 'コト'.

One workaround is to include `tokenizer._tokenizer.normalizer.normalize_str` before the tokenizer preprocessing pipeline, which is also provided in the [first colab notebook](https://colab.research.google.com/drive/151gKyo0YIwnlznrOHst23oYH_a3mAe3Z?usp=sharing) with the name `udposTestDatasetWorkaround`.

I guess similar logics should be included inside the tokenizer and the offsets_mapping generation process such that user don't need to include them in their code. But I don't understand the code of tokenizer well that I think I am not able to do this.

p.s.
**I am using my own dataset building script in the provided example, but the script should be equivalent to the changes made by this [update](https://github.com/huggingface/datasets/pull/2466)**
`get_dataset `is just a simple wrapping for `load_dataset`
and the `tokenizer` is just `XLMRobertaTokenizerFast.from_pretrained("xlm-roberta-large")`

Comments:
Title: Fix dev version
State: closed

The dev version that ends in `.dev0` should be greater than the current version.
However it happens that `1.8.0 > 1.8.0.dev0` for example.
Therefore we need to use `1.8.1.dev0` for example in this case.

I updated the dev version to use `1.8.1.dev0`, and I also added a comment in the setup.py in the release steps about this.

Comments:
Title: Add summarization template
State: closed

This PR adds a task template for text summarization. As far as I can tell, we do not need to distinguish between "extractive" or "abstractive" summarization - both can be handled with this template.

Usage:

```python
from datasets import load_dataset
from datasets.tasks import Summarization

ds = load_dataset("xsum", split="train")
# Dataset({
#     features: ['document', 'summary', 'id'],
#     num_rows: 204045
# })

summarization = Summarization(text_column="document", summary_column="summary")
ds.prepare_for_task(summarization)
# Dataset({
#     features: ['text', 'summary'],
#     num_rows: 204045
# })
```


Comments:
Title: Add COCO datasets
State: open
Labels: description: Requesting to add a new dataset
name: dataset request

description: Vision datasets
name: vision

## Adding a Dataset
- **Name:** COCO
- **Description:** COCO is a large-scale object detection, segmentation, and captioning dataset.
- **Paper + website:** https://cocodataset.org/#home
- **Data:** https://cocodataset.org/#download
- **Motivation:** It would be great to have COCO available in HuggingFace datasets, as we are moving beyond just text. COCO includes multi-modalities (images + text), as well as a huge amount of images annotated with objects, segmentation masks, keypoints etc., on which models like DETR (which I recently added to HuggingFace Transformers) are trained. Currently, one needs to download everything from the website and place it in a local folder, but it would be much easier if we can directly access it through the datasets API.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Title: Documentation Mistakes in Dataset: emotion
State: closed
Labels: description: Something isn't working
name: bug

As per documentation,
Dataset: emotion
Homepage: https://github.com/dair-ai/emotion_dataset

Dataset: https://github.com/huggingface/datasets/blob/master/datasets/emotion/emotion.py

Permalink: https://huggingface.co/datasets/viewer/?dataset=emotion

Emotion is a dataset of English Twitter messages with eight basic emotions: anger, anticipation, disgust, fear, joy, sadness, surprise, and trust. For more detailed information please refer to the paper.

But when we view the data, there are only 6 emotions, anger, fear, joy, sadness, surprise, and trust.

Comments:
Title: Datasets with tricky task templates
State: closed
Labels: description: Discussions on the datasets
name: Dataset discussion

I'm collecting a list of datasets here that don't follow the "standard" taxonomy and require further investigation to implement task templates for.

## Text classification

* [hatexplain](https://huggingface.co/datasets/hatexplain): ostensibly a form of text classification, but not in the standard `(text, target)` format and each sample appears to be tokenized.
* [muchocine](https://huggingface.co/datasets/muchocine): contains two candidate text columns (long-form and summary) which in principle requires two `TextClassification` templates which is not currently supported 

Comments:
Title: Add task templates for tydiqa and xquad
State: closed

This PR adds question-answering templates to the remaining datasets that are linked to a model on the Hub.

Notes: 

* I could not test the tydiqa implementation since I don't have enough disk space 😢 . But I am confident the template works :)
* there exist other datasets like `fquad` and `mlqa` which are candidates for question-answering templates, but some work is needed to handle the ordering of nested column described in #2434 



Comments: