jamesLeeeeeee/datasets-github-issues huggingface dataset:
Add COCO dataset
Author: mariosasko | Pull request: yes | State: closed

This PR adds the MS COCO dataset. Compared to the [TFDS](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/object_detection/coco.py) script, this implementation adds 8 additional configs to cover the tasks other than object detection.

Some notes:
* the data exposed by TFDS is contained in the `2014`, `2015`, `2017` and `2017_panoptic_segmentation` configs here
* I've updated `encode_nested_example` for easier handling of missing values (cc @lhoestq @albertvillanova; will add tests if you are OK with the changes in `features.py`)
* this implementation should fix https://github.com/huggingface/datasets/pull/3377#issuecomment-985559427

TODOs:
- [x] dataset card
- [ ] dummy data

cc @merveenoyan 

Closes #2526
Push dataset infos.json to Hub
Author: lhoestq | Pull request: yes | State: closed

When doing `push_to_hub`, the feature types are lost (see issue https://github.com/huggingface/datasets/issues/3394).
This PR fixes this by also pushing a `dataset_infos.json` file to the Hub, that stores the feature types.

Other minor changes:
- renamed the `___` separator to `--`, since `--` is now disallowed in a name in the back-end.

I tested this feature with datasets like conll2003 that has feature types like `ClassLabel` that were previously lost.

Close https://github.com/huggingface/datasets/issues/3394

I would like to include this in today's release (though not mandatory), so feel free to comment/suggest changes
Unable to load 'cnn_dailymail' dataset
Author: talha1503 | Pull request: no | State: closed

## Describe the bug
I wanted to load cnn_dailymail dataset from huggingface datasets on Google Colab, but I am getting an error while loading it.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('cnn_dailymail', '3.0.0', ignore_verifications = True)
```

## Expected results
Expecting to load 'cnn_dailymail' dataset.

## Actual results
`NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'`


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
Author: koukoulala | Pull request: no | State: open

## Describe the bug
A clear and concise description of what the bug is.

using latest datasets=datasets-1.16.1-py3-none-any.whl
process my own multilingual dataset by following codes, and the number of rows in all dataset is 306000, the max_length of each sentence is 256:
![image](https://user-images.githubusercontent.com/30341159/146865779-3d25d011-1f42-4026-9e1b-76f6e1d172e9.png)

then I get this error:
![image](https://user-images.githubusercontent.com/30341159/146865844-e60a404c-5f3a-403c-b2f1-acd943b5cdb8.png)

I have seen the issue in #2134 and #2150, so I don't understand why latest repo still can't deal with big dataset.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: linux docker
- Python version: 3.6
Update swahili_news dataset
Author: albertvillanova | Pull request: no | State: closed

Please note also: the HuggingFace version at https://huggingface.co/datasets/swahili_news is outdated. An updated version, with deduplicated text and official splits, can be found at https://zenodo.org/record/5514203.

## Adding a Dataset
- **Name:** swahili_news

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Related to:
- bigscience-workshop/data_tooling#107
Don't encode lists as strings when using `Value("string")`
Author: lhoestq | Pull request: yes | State: closed

Following https://github.com/huggingface/datasets/pull/3456#event-5792250497 it looks like `datasets` can silently convert lists to strings using `str()`, instead of raising an error.

This PR fixes this and should fix the issue with WER showing low values if the input format is not right.
Add CMU Graphics Lab Motion Capture dataset
Author: osanseviero | Pull request: no | State: open

## Adding a Dataset
- **Name:** CMU Graphics Lab Motion Capture database
- **Description:** The database contains free motions which you can download and use. 
- **Data:** http://mocap.cs.cmu.edu/
- **Motivation:** Nice motion capture dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
[WER] Better error message for wer
Author: patrickvonplaten | Pull request: yes | State: closed

Currently we have the following problem when using the WER. When the input format to the WER metric is wrong, instead of throwing an error message a word-error-rate is computed which is incorrect. E.g. when doing the following: 


```python
from datasets import load_metric

wer = load_metric("wer")

target_str = ["hello this is nice", "hello the weather is bloomy"]
pred_str = [["hello it's nice"], ["hello it's the weather"]]

print("Wrong:", wer.compute(predictions=pred_str, references=target_str))
print("Correct", wer.compute(predictions=[x[0] for x in pred_str], references=target_str))
```
We get:
```
Wrong: 1.0
Correct 0.5555555555555556
```

meaning that we get a word-error rate for incorrectly passed input formats. We should raise an error here instead so that people don't spent hours fixing a model while it's their incorrect evaluation metric is the problem for a low WER.
Easier information editing
Author: borgr | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
It requires a lot of effort to improve a datasheet. 

**Describe the solution you'd like**
UI or at least a link to the place where the code that needs to be edited is (and an easy way to edit this code directly from the site, without cloning, branching, makefile etc.)

**Describe alternatives you've considered**
The current Ux is to have the 8 steps for contribution while One just wishes to change a line a type etc.
[Staging] Update dataset repos automatically on the Hub
Author: lhoestq | Pull request: yes | State: closed

Let's have a script that updates the dataset repositories on staging for now. This way we can make sure it works fine before going in prod.

Related to https://github.com/huggingface/datasets/issues/3341

The script runs on each commit on `master`. It checks the datasets that were changed, and it pushes the changes to the corresponding repositories on the Hub.

If there's a new dataset, then a new repository is created.

If the commit is a new release of `datasets`, it also pushes the tag to all the repositories.
Unexpected behavior doing Split + Filter
Author: jbrachat | Pull request: no | State: closed

## Describe the bug
I observed unexpected behavior when applying 'train_test_split' followed by 'filter' on dataset. Elements of the training dataset eventually end up in the test dataset (after applying the 'filter') 

## Steps to reproduce the bug
```
from datasets import Dataset
import pandas as pd
dic = {'x': [1,2,3,4,5,6,7,8,9], 'y':['q','w','e','r','t','y','u','i','o']}
df = pd.DataFrame.from_dict(dic)
dataset = Dataset.from_pandas(df)
split_dataset = dataset.train_test_split(test_size=0.5, shuffle=False, seed=42)
train_dataset = split_dataset["train"]
eval_dataset = split_dataset["test"]
eval_dataset_2 = eval_dataset.filter(lambda example: example['x'] % 2 == 0)
print( eval_dataset['x'])
print(eval_dataset_2['x'])
```
One observes that elements in eval_dataset2 are actually coming from the training dataset...

## Expected results
The expected results would be that the filtered eval dataset would only contain elements from the original eval dataset.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Windows 10
- Python version: 3.7
- PyArrow version: 5.0.0
Add `__add__()`, `__iadd__()` and similar to `Dataset` class
Author: sgraaf | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
No.

**Describe the solution you'd like**
I would like to be able to concatenate datasets as follows:
```python
>>> dataset["train"] += dataset["validation"]
```

... instead of using `concatenate_datasets()`:
```python
>>> raw_datasets["train"] = concatenate_datasets([raw_datasets["train"], raw_datasets["validation"]])
>>> del raw_datasets["validation"]
```

**Describe alternatives you've considered**
Well, I have considered `concatenate_datasets()` 😀

**Additional context**
N.a.
JSONDecodeError with HuggingFace dataset viewer
Author: kathrynchapman | Pull request: no | State: closed

## Dataset viewer issue for 'pubmed_neg'

**Link:** https://huggingface.co/datasets/IGESML/pubmed_neg

I am getting the error: 
Status code:   400
Exception:     JSONDecodeError
Message:       Expecting property name enclosed in double quotes: line 61 column 2 (char 1202)

I have checked all files - I am not using single quotes anywhere. Not sure what is causing this issue.

Am I the one who added this dataset ? Yes
Remove redundant local path information in audio/image datasets
Author: mariosasko | Pull request: yes | State: closed

Remove the redundant path information in the audio/image dataset as discussed in https://github.com/huggingface/datasets/pull/3430#issuecomment-994734828

TODOs:
* [ ] merge https://github.com/huggingface/datasets/pull/3430
* [ ] merge https://github.com/huggingface/datasets/pull/3364
* [ ] re-generate the info files of the updated audio datasets

cc:  @patrickvonplaten @anton-l @nateraw (I expect this to break the audio/vision examples in Transformers; after this change you'll be able to access underlying paths as follows `dset = dset.cast_column("audio", Audio(..., decode=False)); path = dset[0]["audio"]`)
Extend text to support yielding lines, paragraphs or documents
Author: albertvillanova | Pull request: yes | State: closed

Add `config.row` option to `text` module to allow yielding lines (default, current case), paragraphs or documents.

Feel free to comment on the name of the config parameter `row`:
- Currently, the docs state datasets are made of rows and columns
- Other names I considered: `example`, `item`
Add QuALITY dataset
Author: lewtun | Pull request: no | State: open

## Adding a Dataset
- **Name:** QuALITY
- **Description:** A challenging question answering with very long contexts (Twitter [thread](https://twitter.com/sleepinyourhat/status/1471225421794529281?s=20))
- **Paper:** No ArXiv link yet, but draft is [here](https://github.com/nyu-mll/quality/blob/main/quality_preprint.pdf)
- **Data:** GitHub repo [here](https://github.com/nyu-mll/quality)
- **Motivation:** This dataset would serve as a nice way to benchmark long-range Transformer models like BigBird, Longformer and their descendants. In particular, it would be very interesting to see how the S4 model fares on this given it's impressive performance on the Long Range Arena

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Update BLEURT hyperlink
Author: lewtun | Pull request: yes | State: closed

The description of BLEURT on the hf.co website has a strange use of URL hyperlinking. This PR attempts to fix this, although I am not 100% sure Markdown syntax is allowed on the frontend or not.

![Screen Shot 2021-12-15 at 17 31 27](https://user-images.githubusercontent.com/26859204/146226432-c83cbdaf-f57d-4999-b53c-85da718ff7fb.png)
Improve Wikipedia Loading Script
Author: geohci | Pull request: yes | State: closed

* More structured approach to detecting redirects
* Remove redundant template filter code (covered by strip_code)
* Add language-specific lists of additional media namespace aliases for filtering
* Add language-specific lists of category namespace aliases for new link text cleaning step
* Remove magic words (parser directions like __TOC__ that occasionally occur in text)

Fix #3400

With support from @albertvillanova 
CC @yjernite
Add The People's Speech
Author: mariosasko | Pull request: no | State: closed

## Adding a Dataset
- **Name:** The People's Speech
- **Description:** a massive English-language dataset of audio transcriptions of full sentences. 
- **Paper:** https://openreview.net/pdf?id=R8CwidgJ0yT
- **Data:** https://mlcommons.org/en/peoples-speech/
- **Motivation:** With over 30,000 hours of speech, this dataset is the largest and most diverse freely available English speech recognition corpus today.


[The article](https://thegradient.pub/new-datasets-to-democratize-speech-recognition-technology-2/) which may be useful when working on the dataset.

cc: @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Add Multilingual Spoken Words dataset
Author: albertvillanova | Pull request: no | State: closed

## Adding a Dataset
- **Name:** Multilingual Spoken Words
- **Description:** Multilingual Spoken Words Corpus is a large and growing audio dataset of spoken words in 50 languages for academic research and commercial applications in keyword spotting and spoken term search, licensed under CC-BY 4.0. The dataset contains more than 340,000 keywords, totaling 23.4 million 1-second spoken examples (over 6,000 hours). 

  Read more: https://mlcommons.org/en/news/spoken-words-blog/
- **Paper:** https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/fe131d7f5a6b38b23cc967316c13dae2-Paper-round2.pdf
- **Data:** https://mlcommons.org/en/multilingual-spoken-words/
- **Motivation:** 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Unable to resolve any data file after loading once
Author: LzyFischer | Pull request: no | State: closed

when I rerun my program, it occurs this error
" Unable to resolve any data file that matches '['**train*']' at /data2/whr/lzy/open_domain_data/retrieval/wiki_dpr with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'zip']", so how could i deal with this problem?
thx.
And below is my code .
![image](https://user-images.githubusercontent.com/84694183/146023446-d75fdec8-65c1-484f-80d8-6c20ff5e994b.png)
Make decoding of Audio and Image feature optional
Author: mariosasko | Pull request: yes | State: closed

Add the `decode` argument (`True` by default) to the `Audio` and the `Image` feature to make it possible to toggle on/off decoding of these features.

Even though we've discussed that on Slack, I'm not removing the `_storage_dtype` argument of the Audio feature in this PR to avoid breaking the Audio feature tests.
Make cast cacheable (again) on Windows
Author: mariosasko | Pull request: yes | State: closed

`cast` currently emits the following warning when called on Windows:
```
Parameter 'function'=<function Dataset.cast.<locals>.<lambda> at 0x000001C930571EA0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting 
and caching to work. If you reuse this transform, the caching mechanism will consider it to be different 
from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
```

It seems like the issue stems from the `config.PYARROW_VERSION` object not being serializable on Windows (tested with `dumps(lambda: config.PYARROW_VERSION)`), so I'm fixing this by capturing `config.PYARROW_VERSION.major` before the lambda definition.
Getting configs names takes too long
Author: severo | Pull request: no | State: open


## Steps to reproduce the bug
```python
from datasets import get_dataset_config_names
get_dataset_config_names("allenai/c4")
```

## Expected results

I would expect to get the answer quickly, at least in less than 10s

## Actual results

It takes about 45s on my environment

## Environment info

- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-1022-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1
data duplicate when setting num_works > 1 with streaming data
Author: cloudyuyuyu | Pull request: no | State: closed

## Describe the bug
The data is repeated num_works times when we load_dataset with streaming and set num_works > 1 when construct dataloader

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import pandas as pd
import numpy as np
import os

from datasets import load_dataset
from torch.utils.data import DataLoader
from tqdm import tqdm
import shutil

NUM_OF_USER = 1000000
NUM_OF_ACTION = 50000
NUM_OF_SEQUENCE = 10000
NUM_OF_FILES = 32
NUM_OF_WORKERS = 16

if __name__ == "__main__":
    shutil.rmtree("./dataset")
    for i in range(NUM_OF_FILES):
        sequence_data = pd.DataFrame(
            {
                "imei": np.random.randint(1, NUM_OF_USER, size=NUM_OF_SEQUENCE),
                "sequence": np.random.randint(1, NUM_OF_ACTION, size=NUM_OF_SEQUENCE)
            }
        )

        if not os.path.exists("./dataset"):
            os.makedirs("./dataset")

        sequence_data.to_csv(f"./dataset/sequence_data_{i}.csv",
                            
 index=False)

    dataset = load_dataset("csv",
                           data_files=[os.path.join("./dataset",file) for file in os.listdir("./dataset") if file.endswith(".csv")],
                           split="train",
                           streaming=True).with_format("torch")
    data_loader = DataLoader(dataset,
                             batch_size=1024,
                             num_workers=NUM_OF_WORKERS)
    
    result = pd.DataFrame()
    for i, batch in tqdm(enumerate(data_loader)):
        result = pd.concat([result, 
                           pd.DataFrame(batch)],
                           axis=0)
    result.to_csv(f"num_work_{NUM_OF_WORKERS}.csv", index=False)

```

## Expected results
data do not duplicate 

## Actual results
data duplicate NUM_OF_WORKERS = 16 
![image](https://user-images.githubusercontent.com/16486492/145748707-9d2df25b-2f4f-4d7b-a83e-242be4fc8934.png)


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:datasets==1.14.0
- Platform:transformers==4.11.3
- Python version:3.8
- PyArrow version:
Error about load_metric
Author: jiacheng-ye | Pull request: no | State: closed

## Describe the bug
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1371, in load_metric
    metric = metric_cls(
TypeError: 'NoneType' object is not callable

## Steps to reproduce the bug
```python
metric = load_metric("glue", "sst2")
```


## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-4.15.0-161-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyArrow version: 6.0.1
Add eli5_category dataset
Author: jingshenSN2 | Pull request: yes | State: closed

This pull request adds a categorized Long-form question answering dataset `ELI5_Category`. It's a new variant of the [ELI5](https://huggingface.co/datasets/eli5) dataset that uses the Reddit tags to alleviate the training/validation overlapping in the origin ELI5 dataset.

A [report](https://celeritasml.netlify.app/posts/2021-12-01-eli5c/)(Section 2) on this dataset.
`.to_json` is extremely slow after `.select`
Author: eladsegal | Pull request: no | State: open

## Describe the bug
Saving a dataset to JSON with `to_json` is extremely slow after using `.select` on the original dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset

original = load_dataset("squad", split="train")
original.to_json("from_original.json")  # Takes 0 seconds

selected_subset1 = original.select([i for i in range(len(original))])
selected_subset1.to_json("from_select1.json")  # Takes 212 seconds

selected_subset2 = original.select([i for i in range(int(len(original) / 2))])
selected_subset2.to_json("from_select2.json")  # Takes 90 seconds
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: master (https://github.com/huggingface/datasets/commit/6090f3cfb5c819f441dd4a4bb635e037c875b044)
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 6.0.0
Fix type of bridge field in QED
Author: mariosasko | Pull request: yes | State: closed

Use `Value("string")` instead of `Value("bool")` for the feature type of the `"bridge"` field in the QED dataset. If the value is `False`, set to `None`.

The following paragraph in the QED repo explains the purpose of this field:
>Each annotation in referential_equalities is a pair of spans, the question_reference and the sentence_reference, corresponding to an entity mention in the question and the selected_sentence respectively. As described in the paper, sentence_references can be "bridged in", in which case they do not correspond with any actual span in the selected_sentence. Hence, sentence_reference spans contain an additional field, bridge, which is a prepositional phrase when a reference is bridged, and is False otherwise. Prepositional phrases serve to link bridged references to an anchoring phrase in the selected_sentence. In the case a sentence_reference is bridged, the start and end, as well as the span string, map to such an anchoring phrase in the selected_sentence.

Fix #3346

cc @VictorSanh
disaster_response_messages unavailable
Author: sacdallago | Pull request: no | State: closed

## Dataset viewer issue for '* disaster_response_messages*'

**Link:** https://huggingface.co/datasets/disaster_response_messages

Dataset unavailable. Link dead: https://datasets.appen.com/appen_datasets/disaster_response_data/disaster_response_messages_training.csv

Am I the one who added this dataset ?No