jamesLeeeeeee/datasets-github-issues huggingface dataset:
Add "Adversarial GLUE" dataset to datasets library
Author: jxmorris12 | Pull request: yes | State: closed

Adds the Adversarial GLUE dataset: https://adversarialglue.github.io/

```python
>>> import datasets
>>> >>> datasets.load_dataset('adv_glue')
Using the latest cached version of the module from /home/jxm3/.cache/huggingface/modules/datasets_modules/datasets/adv_glue/26709a83facad2830d72d4419dd179c0be092f4ad3303ad0ebe815d0cdba5cb4 (last modified on Mon Mar  7 19:19:48 2022) since it couldn't be found locally at adv_glue., or remotely on the Hugging Face Hub.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jxm3/random/datasets/src/datasets/load.py", line 1657, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/jxm3/random/datasets/src/datasets/load.py", line 1510, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 1021, in __init__
    super().__init__(*args, **kwargs)
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 258, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 337, in _create_builder_config
    raise ValueError(
ValueError: Config name is missing.
Please pick one among the available configs: ['adv_sst2', 'adv_qqp', 'adv_mnli', 'adv_mnli_mismatched', 'adv_qnli', 'adv_rte']
Example of usage:
	`load_dataset('adv_glue', 'adv_sst2')`
>>> datasets.load_dataset('adv_glue', 'adv_sst2')['validation'][0]
Reusing dataset adv_glue (/home/jxm3/.cache/huggingface/datasets/adv_glue/adv_sst2/1.0.0/3719a903f606f2c96654d87b421bc01114c37084057cdccae65cd7bc24b10933)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 604.11it/s]
{'sentence': "it 's an uneven treat that bores fun at the democratic exercise while also examining its significance for those who take part .", 'label': 1, 'idx': 0}
```
NonMatchingChecksumError when checksum is None
Author: jxmorris12 | Pull request: no | State: closed

I ran into the following error when adding a new dataset:

```bash
expected_checksums = {'https://adversarialglue.github.io/dataset/dev.zip': {'checksum': None, 'num_bytes': 40662}}
recorded_checksums = {'https://adversarialglue.github.io/dataset/dev.zip': {'checksum': 'efb4cbd3aa4a87bfaffc310ae951981cc0a36c6c71c6425dd74e5b55f2f325c9', 'num_bytes': 40662}}
verification_name = 'dataset source files'

    def verify_checksums(expected_checksums: Optional[dict], recorded_checksums: dict, verification_name=None):
        if expected_checksums is None:
            logger.info("Unable to verify checksums.")
            return
        if len(set(expected_checksums) - set(recorded_checksums)) > 0:
            raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
        if len(set(recorded_checksums) - set(expected_checksums)) > 0:
            raise UnexpectedDownloadedFile(str(set(recorded_checksums) - set(expected_checksums)))
        bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
        for_verification_name = " for " + verification_name if verification_name is not None else ""
        if len(bad_urls) > 0:
            error_msg = "Checksums didn't match" + for_verification_name + ":\n"
>           raise NonMatchingChecksumError(error_msg + str(bad_urls))
E           datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
E           ['https://adversarialglue.github.io/dataset/dev.zip']

src/datasets/utils/info_utils.py:40: NonMatchingChecksumError
```

## Expected results
The dataset downloads correctly, and there is no error.

## Actual results
Datasets library is looking for a checksum of None, and it gets a non-None checksum, and throws an error. This is clearly a bug.
Datasets' cache not re-used
Author: gejinchen | Pull request: no | State: open

## Describe the bug
For most tokenizers I have tested (e.g. the RoBERTa tokenizer), the data preprocessing cache are not fully reused in the first few runs, although their `.arrow` cache files are in the cache directory.

## Steps to reproduce the bug
Here is a reproducer. The GPT2 tokenizer works perfectly with caching, but not the RoBERTa tokenizer in this example.
```python
from datasets import load_dataset
from transformers import AutoTokenizer

raw_datasets = load_dataset("wikitext", "wikitext-2-raw-v1")
# tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
text_column_name = "text"
column_names = raw_datasets["train"].column_names

def tokenize_function(examples):
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)

tokenized_datasets = raw_datasets.map(
    tokenize_function,
    batched=True,
    remove_columns=column_names,
    load_from_cache_file=True,
    desc="Running tokenizer on every text in dataset",
)
```

## Expected results
No tokenization would be required after the 1st run. Everything should be loaded from the cache.

## Actual results
Tokenization for some subsets are repeated at the 2nd and 3rd run. Starting from the 4th run, everything are loaded from cache.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Ubuntu 18.04.6 LTS
- Python version: 3.6.9
- PyArrow version: 6.0.1
add RMSE and MAE metrics.
Author: dnaveenr | Pull request: yes | State: closed

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Please suggest any changes if required. Thank you.
Add rmse and mae metrics.
Author: dnaveenr | Pull request: yes | State: closed

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Any suggestions and changes required will be helpful.
Align IterableDataset.shuffle with Dataset.shuffle
Author: lhoestq | Pull request: yes | State: closed

From #3444 , Dataset.shuffle can have the same API than IterableDataset.shuffle (i.e. in streaming mode).

Currently you can pass an optional seed to both if you want, BUT currently IterableDataset.shuffle always requires a buffer_size, used for approximate shuffling. I propose using a reasonable default value (maybe 1000) instead.

In this PR, I set the default `buffer_size` value to 1,000, and I reorder the `IterableDataset.shuffle` arguments to match `Dataset.shuffle`, i.e. making `seed` the first argument.
Pyright reportPrivateImportUsage when `from datasets import load_dataset` 
Author: lkhphuc | Pull request: no | State: closed

## Describe the bug
Pyright complains about module not exported. 

## Steps to reproduce the bug
Use an editor/IDE with Pyright Language server with default configuration:
```python
from datasets import load_dataset
```

## Expected results
No complain from Pyright

## Actual results
Pyright complain below:
```
`load_dataset` is not exported from module "datasets"
 Import from "datasets.load" instead [reportPrivateImportUsage] 
```
Importing from `datasets.load` does indeed solves the problem but I believe importing directly from top level `datasets` is the intended usage per the documentation.

## Environment info
- `datasets` version: 1.18.3
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.10
- PyArrow version: 7.0.0
Pin responses to fix CI for Windows
Author: albertvillanova | Pull request: yes | State: closed

Temporarily fix CI for Windows by pinning `responses`. 

See: https://app.circleci.com/pipelines/github/huggingface/datasets/10292/workflows/83de4a55-bff7-43ec-96f7-0c335af5c050/jobs/63355

Fix: #3839
Add a data type for labeled images (image segmentation)
Author: severo | Pull request: no | State: open

It might be a mix of Image and ClassLabel, and the color palette might be generated automatically.

---

### Example

every pixel in the images of the annotation column (in https://huggingface.co/datasets/scene_parse_150) has a value that gives its class, and the dataset itself is associated with a color palette (eg https://github.com/open-mmlab/mmsegmentation/blob/98a353b674c6052d319e7de4e5bcd65d670fcf84/mmseg/datasets/ade.py#L47) that maps every class with a color.

So we might want to render the image as a colored image instead of a black and white one.

<img width="785" alt="156741519-fbae6844-2606-4c28-837e-279d83d00865" src="https://user-images.githubusercontent.com/1676121/157005263-7058c584-2b70-465a-ad94-8a982f726cf4.png">

See https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/features/labeled_image.py for reference in Tensorflow
Making Hugging Face the place to go for Graph NNs datasets
Author: omarespejel | Pull request: no | State: open

Let's make Hugging Face Datasets the central hub for GNN datasets :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the GNN field.

What are some datasets worth integrating into the Hugging Face hub?

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Special thanks to @napoles-uach for his collaboration on identifying the first ones:

- [ ] [SNAP-Stanford OGB Datasets](https://github.com/snap-stanford/ogb).
- [ ] [SNAP-Stanford Pretrained GNNs Chemistry and Biology Datasets](https://github.com/snap-stanford/pretrain-gnns).
- [ ] [TUDatasets](https://chrsmrrs.github.io/datasets/) (A collection of benchmark datasets for graph classification and regression)

cc @osanseviero
when using to_tf_dataset with shuffle is true, not all completed batches are made
Author: greenned | Pull request: no | State: closed

## Describe the bug
when converting a dataset to tf_dataset by using to_tf_dataset with shuffle true, the remainder is not converted to one batch

## Steps to reproduce the bug
this is the sample code below
https://colab.research.google.com/drive/1_oRXWsR38ElO1EYF9ayFoCU7Ou1AAej4?usp=sharing


## Expected results
regardless of shuffle is true or not, 67 rows dataset should be 5 batches when batch size is 16.

## Actual results
4 batches

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1
Got error when load cnn_dailymail dataset
Author: wgong0510 | Pull request: no | State: closed

When using datasets.load_dataset method to load cnn_dailymail dataset,  got error as below:

- windows os: FileNotFoundError: [WinError 3] 系统找不到指定的路径。: 'D:\\SourceCode\\DataScience\\HuggingFace\\Data\\downloads\\1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b\\cnn\\stories'
- google colab: NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'

The code is to load dataset:
windows os:
```
from datasets import load_dataset

dataset = load_dataset("cnn_dailymail", "3.0.0", cache_dir="D:\\SourceCode\\DataScience\\HuggingFace\\Data")

```
google colab:
```
import datasets

train_data = datasets.load_dataset("cnn_dailymail", "3.0.0", split="train")
```
[📄 Docs] Create a `datasets` performance guide.
Author: dynamicwebpaige | Pull request: no | State: open

## Brief Overview
Downloading, saving, and preprocessing large datasets from the `datasets` library can often result in [performance bottlenecks](https://github.com/huggingface/datasets/issues/3735). These performance snags can be challenging to identify and to debug, especially for users who are less experienced with building deep learning experiments.

## Feature Request
Could we create a performance guide for using `datasets`, similar to:
* [Better performance with the `tf.data` API](https://github.com/huggingface/datasets/issues/3735)
* [Analyze `tf.data` performance with the TF Profiler](https://www.tensorflow.org/guide/data_performance_analysis)

This performance guide should detail practical options for improving performance with `datasets`, and enumerate any common best practices. It should also show how to use tools like the PyTorch Profiler or the TF Profiler to identify any performance bottlenecks (example below).

![image](https://user-images.githubusercontent.com/3712347/156859152-a3cb9565-3ec6-4d39-8e77-56d0a75a4954.png)

## Related Issues
* [wiki_dpr pre-processing performance #1670](https://github.com/huggingface/datasets/issues/1670)
* [Adjusting chunk size for streaming datasets #3499](https://github.com/huggingface/datasets/issues/3499)
* [how large datasets are handled under the hood #1004](https://github.com/huggingface/datasets/issues/1004)
* [using map on loaded Tokenizer 10x - 100x slower than default Tokenizer? #1830](https://github.com/huggingface/datasets/issues/1830)
* [Best way to batch a large dataset? #315](https://github.com/huggingface/datasets/issues/315)
* [Saving processed dataset running infinitely #1911](https://github.com/huggingface/datasets/issues/1911)
The Pile's _FEATURE spec seems to be incorrect
Author: dlwh | Pull request: no | State: closed

## Describe the bug
If you look at https://huggingface.co/datasets/the_pile/blob/main/the_pile.py:

For "all"
*  the pile_set_name is never set for data
* there's actually an id field inside of "meta" 

For subcorpora pubmed_central and hacker_news:
* the meta is specified to be a string, but it's actually a dict with an id field inside.

## Steps to reproduce the bug


## Expected results
Feature spec should match the data I'd think?

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
Add IterableDataset.filter
Author: lhoestq | Pull request: yes | State: closed

_Needs https://github.com/huggingface/datasets/pull/3801 to be merged first_

I added `IterableDataset.filter` with an API that is a subset of `Dataset.filter`:
```python
def filter(self, function, batched=False, batch_size=1000, with_indices=false, input_columns=None):
```

TODO:
- [x] tests
- [x] docs

related to https://github.com/huggingface/datasets/issues/3444 and https://github.com/huggingface/datasets/issues/3753
500 internal server error when trying to open a dataset composed of Zarr stores
Author: jacobbieker | Pull request: no | State: closed

## Describe the bug
The dataset [openclimatefix/mrms](https://huggingface.co/datasets/openclimatefix/mrms) gives a 500 server error when trying to open it on the website, or through code.

The dataset doesn't have a loading script yet, and I did push two [xarray](https://docs.xarray.dev/en/stable/) Zarr stores of data there recentlyish. The Zarr stores are composed of lots of small files, which I am guessing is probably the problem, as we have another [OCF dataset](https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv) using xarray and Zarr, but with the Zarr stored on GCP public datasets instead of directly in HF datasets, and that one opens fine. 

In general, we were hoping to use HF datasets to release some more public geospatial datasets as benchmarks, which are commonly stored as Zarr stores as they can be compressed well and deal with the multi-dimensional data and coordinates fairly easily compared to other formats, but with this error, I'm assuming we should try a different format? 

For context, we are trying to have complete public model+data reimplementations of some SOTA weather and solar nowcasting models, like [MetNet, MetNet-2,](https://github.com/openclimatefix/metnet) [DGMR](https://github.com/openclimatefix/skillful_nowcasting), and [others](https://github.com/openclimatefix/graph_weather), which all have large, complex datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("openclimatefix/mrms")
```

## Expected results
The dataset should be downloaded or open up

## Actual results
A 500 internal server error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.15.25-1-MANJARO-x86_64-with-glibc2.35
- Python version: 3.9.10
- PyArrow version: 7.0.0
Add Biwi Kinect Head Pose Database
Author: osanseviero | Pull request: no | State: closed

## Adding a Dataset
- **Name:** Biwi Kinect Head Pose Database
- **Description:** Over 15K images of 20 people recorded with a Kinect while turning their heads around freely. For each frame, depth and rgb images are provided, together with ground in the form of the 3D location of the head and its rotation angles.
- **Data:** [*link to the Github repository or current dataset location*](https://icu.ee.ethz.ch/research/datsets.html)
- **Motivation:** Useful pose estimation dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Update Wikipedia dataset
Author: albertvillanova | Pull request: yes | State: closed

This PR combines all updates to Wikipedia dataset.

Once approved, this will be used to generate the pre-processed Wikipedia datasets.

Finally, this PR will be able to be merged into master:
- NOT using squash
- BUT a regular MERGE (or REBASE+MERGE), so that all commits are preserved

TODO:
- [x] #3435
- [x] #3789
- [x] #3825
- [x] Run to get the pre-processed data for big languages (backward compatibility)
- [x] #3958

CC: @geohci
`pubmed_qa` checksum mismatch
Author: jon-tow | Pull request: no | State: closed

## Describe the bug
Loading [`pubmed_qa`](https://huggingface.co/datasets/pubmed_qa) results in a mismatched checksum error.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import datasets
try:
    datasets.load_dataset("pubmed_qa", "pqa_labeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_unlabeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_artificial")
except Exception as e:
    print(e)
```

## Expected results
Successful download.

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: macOS
- Python version: 3.8.1
- PyArrow version: 3.0.0
Support for "sources" parameter in the add() and add_batch() methods in datasets.metric - SARI
Author: lmvasque | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
The methods `add_batch` and `add` from the `Metric` [class](https://github.com/huggingface/datasets/blob/1675ad6a958435b675a849eafa8a7f10fe0f43bc/src/datasets/metric.py) does not work with [SARI](https://github.com/huggingface/datasets/blob/master/metrics/sari/sari.py) metric. This metric not only relies on the predictions and references, but also in the input. 

For example, when the `add_batch` method is used, then the `compute()` method fails:
```
metric = load_metric("sari")
metric.add_batch(
    predictions=["About 95 you now get in ."], 
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
metric.compute()
> TypeError: _compute() missing 1 required positional argument: 'sources'
```

Therefore, the `compute() `method can only be used standalone:
```
metric = load_metric("sari")
result = metric.compute(
    sources=["About 95 species are currently accepted ."],
    predictions=["About 95 you now get in ."],
    references=[["About 95 species are currently known .","About 95 species are now accepted .","95 species are now accepted ."]])
> {'sari': 26.953601953601954}
```
  
**Describe the solution you'd like**
Support for an additional parameter `sources` in the `add_batch` and `add` of the `Metric` class.
```
add_batch(*, sources=None, predictions=None, references=None, **kwargs)
add(*, sources=None, predictions=None, references=None, **kwargs)
compute()
```

**Describe alternatives you've considered**
I've tried to override the `add_batch` and `add`, however, these are highly dependent to the `Metric` class. We could also write a simple function that compute the scores of a sentences list, but then we lose the functionality from the original [add](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add) and [add_batch method](https://huggingface.co/docs/datasets/_modules/datasets/metric.html#Metric.add_batch). 

**Additional context**
These methods are used in the transformers [pytorch examples](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py).
Simplify Common Voice code
Author: lhoestq | Pull request: yes | State: closed

In #3736 we introduced one method to generate examples when streaming, that is different from the one when not streaming.

In this PR I propose a new implementation which is simpler: it only has one function, based on `iter_archive`. And you still have access to local audio files when loading the dataset in non-streaming mode.

cc @patrickvonplaten @polinaeterna @anton-l @albertvillanova since this will become the template for many audio datasets to come.

This change can also trivially be applied to the other audio datasets that already exist. Using this line, you can get access to local files in non-streaming mode:
```python
local_extracted_archive = dl_manager.extract(archive_path) if not dl_manager.is_streaming else None
```
Fix iter_archive getting reset
Author: lhoestq | Pull request: yes | State: closed

The `DownloadManager.iter_archive` method currently returns an iterator - which is  **empty** once you iter over it once. This means you can't pass the same archive iterator to several splits.

To fix that, I changed the ouput of `DownloadManager.iter_archive` to be an iterable that you can iterate over several times, instead of a one-time-use iterator.

The `StreamingDownloadManager.iter_archive` already returns an appropriate iterable, and the code added in this PR is inspired from the one in `streaming_download_manager.py`