[DO NOT MERGE] Test doc-builder

_The documentation is not available anymore as the PR was closed or merged._

Docs built successfully, so closing this.

This is a test PR to ensure the changes in https://github.com/huggingface/doc-builder/pull/164 don't break anything in `datasets`


Support float data types in pearsonr/spearmanr metrics

Fix #4053.


Modify datatype from `int32` to `float` for pearsonr, spearmanr.

enhancement

New feature or request

@Woodywarhol9 good catch, thanks for reporting.

We are fixing this.

**Is your feature request related to a problem? Please describe.**
- Now [Pearsonr](https://github.com/huggingface/datasets/blob/master/metrics/pearsonr/pearsonr.py) and [Spearmanr](https://github.com/huggingface/datasets/blob/master/metrics/spearmanr/spearmanr.py) both get input data as 'int32'.

**Describe the solution you'd like**
- Considering that those metrics are widely used for the STS task(labels are in 'float' data type),
it would be better to modify datatype from 'int32' to 'float' for getting exact values of similarity.


Hi @klyuhang9,

I'm sorry but I can't reproduce your problem:
```python
In [2]: metric = load_metric('glue', 'rte')
Downloading builder script: 5.76kB [00:00, 2.40MB/s]
```

Could you please, retry to load the metric? Sometimes there are temporary connectivity issues.

Feel free to re-open this issue of the problem persists.

Hi, friend. I meet a problem.
When I run the code:
`metric = load_metric('glue', 'rte')`

There is a problem raising:
`metric = metric_cls(
TypeError: 'NoneType' object is not callable `

I don't know why. Thanks for your help!


Hi @klyuhang9,

I'm sorry but I can't reproduce your problem:
```python
In [4]: ds = load_dataset("glue", "sst2", download_mode="force_redownload")
Downloading builder script: 28.8kB [00:00, 9.15MB/s]
Downloading metadata: 28.7kB [00:00, 10.7MB/s]
Downloading and preparing dataset glue/sst2 (download: 7.09 MiB, generated: 4.78 MiB, post-processed: Unknown size, total: 11.88 MiB) to .../.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.44M/7.44M [00:01<00:00, 4.12MB/s]
Dataset glue downloaded and prepared to .../.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96da6053ad. Subsequent calls will reuse this data.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1047.96it/s]

In [5]: ds
Out[5]:
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 872
    })
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1821
    })
})
```

Please, note that sometimes GitHub has some temporary connectivity issues. Feel free to retry and re-open this issue if the problem persists.

Hi, I meet a problem.
When I run the code:
`dataset = load_dataset('glue','sst2')`

There is a issue raising:
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.0.0/datasets/glue/glue.py

I don't know why; it is ok when I use Google Chrome to view this url.
Thanks for your help!


Add RVL-CDIP dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4050). All of your documentation changes will be reflected on that endpoint.

Thanks a lot for inputs. I'll use the URL suggested and check.

>  we need to implement the streamable (can't use os.path.join) and the non-streamable versions of _generate_examples.

Sure. I will check the reference and try this out, will get back to you if I face any issues.

> The labels-only data file URL doesn't work for me, so feel free to ask the authors whether they are OK with us hosting the file on the Hub/S3 (to speed up the streamable version)

Just checked. The author (Adam Harley) has responded positively and allowed us to host the file. Do I share the file with you for hosting it on Hub/S3 ?

> Just checked. The author (Adam Harley) has responded positively and allowed us to host the file. Do I share the file with you for hosting it on Hub/S3 ?

Yes, feel free to e-mail me the file. Then I'll create a repo under my namespace and push the file there. We run a GH action on a GH dataset after merging to create its repo on the Hub, so after this PR is merged, I'll push the file to the "official" namespace and update the download link.

> You can use this URL to avoid manual download: https://drive.google.com/uc?export=download&id=0Bz1dfcnrpXM-MUt4cHNzUEFXcmc

For some reason, the direct download doesn't seem to work for me even with this URL.
```
Downloading and preparing dataset rvl_cdip/default to ~/.cache/huggingface/datasets/rvl_cdip/default/1.0.0/ea152149e06310d60a9ef3c3020199dd4780bb952a773ba5aac6b57d59f12628...
Downloading data files: 100%|█████████████████████████████████████████████████████| 1/1 [00:00<00:00, 6307.22it/s]
{'rvl-cdip': '~/.cache/huggingface/datasets/downloads/07ef956a33750078d570d76fefe9fed49f7dc32ecf6e872d690de11e66bbe869'}
```
And this directory does not exist. Am I doing something wrong ?
To verify, I tried using [gdown](https://github.com/wkentaro/gdown) for the above URL, we get the following :
```
Access denied with the following error:

        Cannot retrieve the public link of the file. You may need to change
        the permission to 'Anyone with the link', or have had many accesses.

You may still be able to access the file from the browser:
```
----

> Yes, feel free to e-mail me the file. Then I'll create a repo under my namespace and push the file there. We run a GH action on a GH dataset after merging to create its repo on the Hub, so after this PR is merged, I'll push the file to the "official" namespace and update the download link.

Got it. I've sent you an email with the file. Thank you.

Actually this URL works for direct download :
`https://drive.google.com/uc?export=download&confirm=pbef&id=0Bz1dfcnrpXM-MUt4cHNzUEFXcmc`
Ref : https://github.com/wkentaro/gdown/issues/146#issuecomment-1042382215

I'm working on the streamable versions of _generate_examples as well, will update you regarding this.

Google Drive is a tricky host, and it's easy to exceed daily download quota limits, so if we are allowed to host the `rvl-cdip.tar.gz` file, I can push it to the Hub.

Just checked, the authors have agreed. He mentioned that he had complaints about the GDrive link.
You can push it to the Hub and share the link. :)

I have added :
- streaming support for rvl-cdip.tar.gz file. [ Need to test this ]

Is it possible for you to upload the train.txt, test.txt, val.txt files separately to the Hub instead of labels_only.tar.gz file.
Currently during the tests in stream mode, we get :
`NotImplementedError: Extraction protocol for TAR archives like 'https://huggingface.co/datasets/mariosasko/rvl_cdip/resolve/main/labels_only.tar.gz' is not implemented in streaming mode. Please use dl_manager.iter_archive instead.`
If the label files are present as .txt files then we can directly use dl_manager.download.

Resolves #2762

Dataset Request : Add RVL-CDIP dataset [#2762](https://github.com/huggingface/datasets/issues/2762)
This PR adds the RVL-CDIP dataset.

The dataset contains Google Drive link for download and wasn't getting downloaded automatically, so I have provided manual_download_instructions.
- I have added the dummy_data.zip as well.

Needed inputs on how I can run the real data and the dummy data tests for datasets with manual download ?

Inputs and suggestions for improvement are welcome. Thank you.


Create metric card for the Code Eval metric

if possible, give relevant names to your Pull requests @sashavor (make it easier to scan the repo activity) Thanks!

updating them now! thanks for the feedback @julien-c

Creating initial Code Eval metric card


Split size error on `amazon_us_reviews` / `PC_v1_00` dataset

bug

Something isn't working

good first issue

Good for newcomers

Follow-up: I have confirmed there are no duplicate lines via `sort amazon_reviews_us_PC_v1_00.tsv | uniq -cd` after extracting the raw file.

Hi @trentonstrong, thanks for reporting!

I confirm that loading this dataset configuration throws a `NonMatchingSplitsSizesError`:
```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=350242049, num_examples=785730, dataset_name='amazon_us_reviews'), 'recorded': SplitInfo(name='train', num_bytes=3982712078, num_examples=6908554, dataset_name='amazon_us_reviews')}]
```

Also thank you for your offer to fix this. You can find information about how to update the metadata JSON file here: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#automatically-add-code-metadata
```shell
datasets-cli test datasets/amazon_us_reviews --save_infos --all_configs
```
Please, feel free to open a PR with this fix. And do not hesitate to ping me if you need any help.

## Describe the bug
When downloading this subset as of 3-28-2022 you will encounter a split size error after the dataset is extracted. The extracted dataset has roughly ~6m rows while the split expects <1m.

Upon digging a little deeper, I downloaded the raw files from `https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_PC_v1_00.tsv.gz` and extracted them. A line count via `wc -l` confirms the ~6m number that we see and the data looks valid at a glance (I did not check for duplicate rows). My guess is this file has either been updated in place or there is a bug in the dataset metadata.

Happy to submit a PR and fix this up if turns out to be a metadata issue but wanted to get some other :eyes: on it first.

## Steps to reproduce the bug
```python
load_dataset('amazon_us_reviews', 'PC_v1_00')
```

## Expected results
Dataset is downloaded and extracted successfully.

## Actual results
An split size exception is thrown.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Dataset.unique(column: str) -> ArrowNotImplementedError

bug

Something isn't working

Hi @orkenstein, thanks for reporting.

Please note that for this case, our `datasets` library uses under the hood the Apache Arrow `unique` function: https://arrow.apache.org/docs/python/generated/pyarrow.compute.unique.html#pyarrow.compute.unique

And currently the Apache Arrow `unique` function is only implemented for these input types (see info in their [docs](https://arrow.apache.org/docs/cpp/compute.html#array-wise-vector-functions)): Boolean, Null, Numeric, Temporal, Binary- and String-like.

However, the data types of the `wikiann` dataset are all `list<item: string>` (see its [dataset card](https://huggingface.co/datasets/wikiann#data-fields)), and thus, not yet supported by the Apache Arrow `unique` function.

As a workaround solution you can use pandas:
```python
from datasets import load_dataset

dataset = load_dataset('wikiann', 'en', split='train')
df = dataset.to_pandas()
unique_df = df[~df.tokens.apply(tuple).duplicated()]  # from https://stackoverflow.com/a/46958336/17517845
```

Note that pandas loads the dataset in memory (this one is small so it's fine).

@lhoestq thank you! I will fall back to this method for now

## Describe the bug
I'm trying to use `unique()` function, but it fails
## Steps to reproduce the bug
1. Get dataset
2. Call `unique`
3. Error

# Sample code to reproduce the bug
```python
!pip show datasets

from datasets import load_dataset

dataset = load_dataset('wikiann', 'en')

dataset['train'].column_names
dataset['train'].unique(dataset['train'].column_names[0])
```

## Expected results
It would be nice to actually see unique items

## Actual results
Error:
```python
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
[<ipython-input-10-5e0de07ed42c>](https://s0qyv2vjaji-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220324-060046-RC00_436956229#) in <module>()
      6
      7 dataset['train'].column_names
----> 8 dataset['train'].unique(dataset['train'].column_names[0])

5 frames
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Function unique has no kernel matching input types (array[list<item: string>])
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Google Collab
- Python version: 3.7.13
- PyArrow version: 6.0.1


Create metric card for XNLI

_The documentation is not available anymore as the PR was closed or merged._

Proposing a metric card for XNLI


Fix CLI dummy data generation

PR:
- #3868

broke the CLI dummy data generation.

Fix #4044.


CLI dummy data generation is broken

Something isn't working

## Describe the bug
We get a TypeError when running CLI dummy data generation:
```shell
datasets-cli dummy_data datasets/<your-dataset-folder> --auto_generate
```
gives:
```
  File ".../huggingface/datasets/src/datasets/commands/dummy_data.py", line 361, in _autogenerate_dummy_data
    dataset_builder._prepare_split(split_generator)
TypeError: _prepare_split() missing 1 required positional argument: 'check_duplicate_keys'
```


Create metric card for CUAD

Proposing a CUAD metric card


Standardize metric ranges

enhancement

New feature or request

Several common metrics, like `exact_match` and `f1`, sometimes range from 0-1, and sometimes from 0-100.

As discussed with @lhoestq , we think it makes more sense to report them from 0-1, which would entail changing the code of metrics such as [CUAD](https://github.com/huggingface/datasets/blob/master/metrics/cuad/cuad.py).

@emibaylor and I will add other metrics here that we see reporting scores from 0-100 instead of 0-1.


Add support for IIIF in datasets

New feature or request

Hi! Thanks for the detailed analysis of adding IIIF support. I like the idea of "using IIIF through datasets scripts" due to its ease of use. Another approach that I like is yielding image ids and using the `piffle` library (which offers a bit more flexibility) + `map` to download + cache images. We can handle bad URLs in `map`  by returning `None`. Plus, we can add a `Dataset Preprocessing` section with the code that explains this approach to the card of such datasets. WDYT?

> currently, IIIF is mainly used by cultural heritage organizations (museums, archives etc.) The adoption of IIIF in this sector has been growing but it's possible that adoption won't be extended to other industries which may also be a source of image data for training ML models.

This is why (currently) adding a new feature type would be overkill, IMO.

This is a feature request for support for IIIF in `datasets`. Apologies for the long issue. I have also used a different format to the usual feature request since I think that makes more sense but happy to use the standard template if preferred.

## What is [IIIF](https://iiif.io/)?

IIIF (International Image Interoperability Framework)
> is a set of open standards for delivering high-quality, attributed digital objects online at scale. It’s also an international community developing and implementing the IIIF APIs. IIIF is backed by a consortium of leading cultural institutions.

The tl;dr is that IIIF provides various specifications for implementing useful functionality for:
- Institutions to make available images for various use cases
- Users to have a consistent way of interacting/requesting these images
- For developers to have a common standard for developing tools for working with IIIF images that will work across all institutions that implement a particular IIIF standard (for example the image viewer for the BNF can also work for the Library of Congress if they both use IIIF).

Some institutions that various levels of support IIF include: The British Library, Internet Archive, Library of Congress, Wikidata. There are also many smaller institutions that have IIIF support. An incomplete list can be found here: https://iiif.io/guides/finding_resources/

## IIIF APIs

IIIF consists of a number of APIs which could be integrated with datasets. I think the most obvious candidate for inclusion would be the [Image API](https://iiif.io/api/image/3.0/)

### IIIF Image API
The Image API https://iiif.io/api/image/3.0/ is likely the most suitable first candidate for integration with datasets. The Image API offers a consistent protocol for requesting images via a URL:

```{scheme}://{server}{/prefix}/{identifier}/{region}/{size}/{rotation}/{quality}.{format}```

A concrete example of this:

```https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/full/full/0/default.jpg```

As you can see the scheme offers a number of options that can be specified in the URL, for example, size. Using the example URL we return:

![](https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/full/full/0/default.jpg)

We can change the size to request a size of 250 by 250, this is done by changing the size from `full` to `250,250` i.e. switching the URL to `https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/full/250,250/0/default.jpg`

![](https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/full/250,250/0/default.jpg)

We can also request the image with max width 250, max height 250 whilst maintaining the aspect ratio using `!w,h`. i.e. change the url to `https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/full/!250,250/0/default.jpg`

![](https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/full/!250,250/0/default.jpg)

A full overview of the options for size can be found here: https://iiif.io/api/image/3.0/#42-size

## Why would/could this be useful for datasets?

There are a few reasons why support for the IIIF Image API could be useful. Broadly the ability to have more control over how an image is returned from a server is useful for many ML workflows:
- images can be requested in the right size, this prevents having to download/stream large images when the actual desired size is much smaller
- can select a subset of an image: it is possible to select a sub-region of an image, this could be useful for example when you already have a bounding box for a subset of an image and then want to use this subset of an image for another task. For example, https://github.com/Living-with-machines/nnanno uses IIIF to request parts of a newspaper image that have been detected as 'photograph', 'illustration' etc for downstream use.
- options for quality, rotation, the format can all be encoded in the URL request.

These may become particularly useful when pre-training models on large image datasets where the cost of downloading images with 1600 pixel width when you actually want 240 has a larger impact.

## What could this look like in datasets?

I think there are various ways in which support for IIIF could potentially be included in `datasets`. These suggestions aren't fully fleshed out but hopefully, give a sense of possible approaches that match existing `datasets` methods in their approach.

### Use through datasets scripts

Loading images via URL is already supported. There are a few possible 'extras' that could be included when using IIIF. One option is to leverage the IIIF protocol in datasets scripts, i.e. the dataset script can expose the IIIF options via the dataset script:

```python
ds = load_dataset("iiif_dataset", image_size="250,250", fmt="jpg")
```

This is already possible. The approach to parsing the IIIF URLs would be left to the person creating the dataset script.

### Support through dataset scripts (with some datasets support)

This is similar to the above but `datasets` would offer some way of saying this is a iiif URL and then expose the options associated with IIIF images automatically. i.e. if you did something like:

```python
features = {"label": ClassLabel(names=['dog','cat']),
					"url": datasets.IIIFURL()}
```
inside your loading script, you would automatically have exposed `size`, `fmt` etc. options when loading the dataset.

### Other possible integrations

Some other possible pseudocode ways that a user could interact with IIIF URLs:

The ability to cast to an `IIIFImage` feature type:

```
ds.cast_column('url', IIIFImage, download=False)
```

The ability to specify some options associated with IIIF urls.
```
ds = ds.set_iiif_options(column='url', size="250,250")
```

I think all of these would rely on having an `IIIFImage` feature type - this would be a little bit of a Frankenstein between a `string` and `datasets.Image`. I think most of the actual image behaviour would be exactly the same as `datasets.Image`, the difference would be that the underlying URL could be modified in various ways.

## prerequisite  requirements

There are a few pre-requisites that I can anticipate. This doesn't cover a full implementation of IIIF support which would have different requirements depending on the approach taken to implementing IIIF. Some of these features would be useful independently of adding IIIF support:

### support for handling failed images loaded via a URL (or a specific IIIFImage feature).

Working with images via web requests will inevitably return the odd failed request. If these images are then requests and don't return it would be useful to have a `None` returned instead of an error. For example, when using `push_to_hub` `datasets` will try and include the image but currently fails with bad URLs.

```python
from datasets import Dataset
import datasets
urls = ['https://stacks.stanford.edu/image/iiif/hg676jb4964%2F0380_796-44/full/!250,250/0/default.jpg']*3
urls.append("badurl.com/image.jpg")
data = {"url":urls}
ds = Dataset.from_dict(data)
ds = ds.cast_column('url', datasets.Image())
ds[3]['url']
```
returns a `FileNotFoundError`, for streaming large datasets of images using their URLs it could be useful to have `None` returned instead. This has implications for the actual training loop i.e. you now need to somehow skip those examples because of this it might not be desirable to support this.

### Caching support
Since IIIF requests images via a URL it would be great to have a way of not requesting the images multiple times. This is tracked in https://github.com/huggingface/datasets/issues/3142 and I think this would also be very desirable to have here particularly as one of the primary use cases of IIIF may be to do unsupervised pre-training on large datasets of IIIF URLs.

### Support for Parsing IIIF URLs
This gets closer to the actual implementation. Here the requirement would be some way for `datasets` to parse a URL that the users specify is an IIIF URL. An example of a Python library that does this: https://github.com/Princeton-CDH/piffle. I also have a rough version that uses `dataclasses` which I can share.

## Why it might not be worthwhile/suitable for datasets

There are some reasons that this might not be worth implementing:
- currently, IIIF is mainly used by cultural heritage organizations (museums, archives etc.) The adoption of IIIF in this sector has been growing but it's possible that adoption won't be extended to other industries which may also be a source of image data for training ML models.
- It may end up being better to leave this to the user. It would for example be possible for someone to write map functions to change an IIIF URL to the correct size etc. Adding direct support for IIIF in datasets may potentially not be worth the trouble.
- The impact of different approaches to doing image scaling can impact the downstream model's performance, see: https://twitter.com/wightmanr/status/1479528581466243073?s=20. Since different IIIF image servers may implement different approaches to resizing images this could have a downstream impact on model performance. think this is something that could be flagged to the end-user in the documentation. This probably also falls into general "gotchas" that probably aren't the `datasets` libraries' role to protect users from.

Some of the requirements outlined above would be useful for images anyway. These could be implemented prior to a final decision about whether IIIF support could/should be added to datasets.

## Suggested next steps:

I realise this is a long and slightly open-ended issue. I am happy to clarify/answer questions on IIIF and possible integrations. If the prerequisite requirements seem worth exploring/are better explored in their own issues let me know and I can open new issues for those.


Calling existing metrics from other metrics

New feature or request

That's definitely the way to go to avoid implementation bugs and making sure we can fix issues globally when we detect them in a metric. Thanks for reporting!

There are several cases of metrics calling other metrics, e.g. [Wiki Split](https://huggingface.co/metrics/wiki_split) which calls [BLEU](https://huggingface.co/metrics/bleu) and [SARI](https://huggingface.co/metrics/sari). These are all currently re-implemented each time (often with external code).

A potentially more efficient and centralized way of doing things would maybe to have a single implementation and calling that implementation.

E.g. @lhoestq 's proposal:
```
def _compute(...):
    bleu = load_metric("bleu", cache_dir=self.cache_dir, seed=self.seed)
    output = bleu._compute(...)
```

Something to keep in mind for the big metric reorg!


Support streaming xcopa dataset


[DO NOT MERGE] Test doc-builder with skipped installation feature

Fix in https://github.com/huggingface/doc-builder/pull/162 works as expected (docs build), closing this

This PR is just for testing that we can build PR docs with changes made on the [`skip-install-for-real`](https://github.com/huggingface/doc-builder/tree/skip-install-for-real) branch of `doc-builder`.


Error while building documentation

bug

Something isn't working

After some investigation, maybe the bug is in `doc-builder`.

I've opened an issue there:
- huggingface/doc-builder#160

Fixed by @lewtun (thank you):
- huggingface/doc-builder@31fe6c8bc7225810e281c2f6c6cd32f38828c504

## Describe the bug
Documentation building is failing:
- https://github.com/huggingface/datasets/runs/5716300989?check_suite_focus=true

```
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Unable to find datasets.filesystems.S3FileSystem in datasets. Make sure the path to that object is correct.
```


Fix building of documentation

Documentation building is failing:
- https://github.com/huggingface/datasets/runs/5716300989?check_suite_focus=true

```
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Unable to find datasets.filesystems.S3FileSystem in datasets. Make sure the path to that object is correct.
```

Fix #4037.


Add zero_division argument to precision and recall metrics

Fix #4025.


Fix null checksum in xcopa dataset


Fix checksum error in cats_vs_dogs dataset

Recent PR updated the metadata JSON file of cats_vs_dogs dataset:
- #3878

However, that new JSON file contains a None checksum.

This PR fixes it.

Fix #4032.


can't download cats_vs_dogs dataset

Something isn't working

Thnaks for reporting @RRaphaell.

We are fixing it.

## Describe the bug
can't download cats_vs_dogs dataset. error: Checksums didn't match for dataset source files

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("cats_vs_dogs")
```

## Expected results
loaded successfully.

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip']

## Environment info
fresh google colab notebook


Something isn't working

Hi @cathyxl, thanks for reporting.

Indeed, we have recently updated the loading script of that dataset (and fixed that bug as well):
- #4002

That fix will be available in our next `datasets` library release. In the meantime, you can incorporate that fix by:
- installing `datasets` from our GitHub repo:
```bash
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
- forcing the data files to be redownloaded
```python
ds = load_dataset('conll2012_ontonotesv5', 'english_v4', split="test", download_mode="force_redownload")
```

Feel free to re-open this issue if the problem persists.

## Describe the bug
Cannot load the dataset conll2012_ontonotesv5

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset
dataset = load_dataset('conll2012_ontonotesv5', 'english_v4', split="test")
print(dataset)

```

## Expected results
The datasets should be downloaded successfully

## Actual results
raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/zmycy7t9h9-1.zip']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux-5.4.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 7.0.0


Use a constant for the articles regex in SQuAD v2

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4030). All of your documentation changes will be reflected on that endpoint.

The main reason for doing this is to be able to change the articles list if using another language, for example. It's not the most elegant solution but at least it makes the metric more extensible with no drawbacks.

BTW, what could be the best way to make this more generic (i.e., SQuAD in other languages)? Maybe receive a regex as an optional param, with the current value as the default? Similarly for SQuAD v1 (can't they re-use code?).


Add FAISS .range_search() method for retrieving all texts from dataset above similarity threshold

enhancement

New feature or request

Hi ! You can access the faiss index with
```python
faiss_index = my_dataset.get_index("my_index_name").faiss_index
```
and then do whatever you want with it, e.g. query it using range_search:
```python
threshold = 0.95
limits, distances, indices = faiss_index.range_search(x=xq, thresh=threshold)

texts = dataset[indices]
```

wow, that's great, thank you for the explanation. (if that's not already in the documentation, could be worth adding it)

which type of faiss index is Datasets using? I looked into faiss recently and I understand that there are several different types of indexes and the choice is important, e.g. regarding which distance metric you use (euclidian vs. cosine/dot product), the size of my dataset etc. can I chose the type of index somehow as well?

`Dataset.add_faiss_index` has a `string_factory` parameter, used to set the type of index (see the faiss documentation about [index factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)). Alternatively, you can pass an index you've defined yourself using faiss with the `custom_index` parameter of `Dataset.add_faiss_index`

Here is the full documentation of `Dataset.add_faiss_index`: https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes#datasets.Dataset.add_faiss_index

**Is your feature request related to a problem? Please describe.**
I would like to retrieve all texts from a dataset, which are semantically similar to a specific input text (query), above a certain (cosine) similarity threshold. My dataset is very large (Wikipedia), so I need to use Datasets and FAISS for this. I would like to be able to repeat many different queries on the dataset quickly.

**Describe the solution you'd like**
dataset objects currently have the .get_nearest_examples() method for text retrieval via FAISS. But this only allows retrieving a specific number of K texts instead of everything above a specified similarity threshold.
It would be great if HF Datasets would also support the FAISS method .range_search() for retrieving texts above a certain similarity threshold.
see details here: https://github.com/facebookresearch/faiss/issues/1273

**Describe alternatives you've considered**
I've considered using native FAISS, but doing this via HF datasets would be better. My assumption is that Dataset features like dataset streaming make it easier to work with large datasets

**Additional context**
The concrete use-case is: I have a large dataset (wikipedia) and I would like to retrieve all paragraphs which are similar to a query. I will use sentence-transformers for encoding the texts.


Fix docs on audio feature installation

This PR:
- Removes the explicit installation of `librosa` (this is installed with `pip install datasets[audio]`
- Adds the warning for Linux users to install manually the non-Python package `libsndfile`
- Explains that the installation of `torchaudio` is only necessary to support loading audio datasets containing MP3 audio files

Related to #4000.


ElasticSearch Indexing example: TypeError: __init__() missing 1 required positional argument: 'scheme'

bug

Something isn't working

duplicate

This issue or pull request already exists

Hi, @MoritzLaurer, thanks for reporting.

Normally this is due to a mismatch between the versions of your Elasticsearch client and server:
- your ES client is passing only keyword arguments to your ES server
- whereas your ES server expects a positional argument called 'scheme'

In order to fix this, you should align the major versions of both Elasticsearch client and server.

You can have more info:
- on this other issue page: https://github.com/huggingface/datasets/issues/3956#issuecomment-1072115173
- Elasticsearch client docs: https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/overview.html#_compatibility

Feel free to re-open this issue if the problem persists.

Duplicate of:
- #3956

1. Check elasticsearch version
```
import elasticsearch
print(elasticsearch.__version__)
```
Ex: 7.9.1
2. Uninstall current elasticsearch package
`pip uninstall elasticsearch`
3. Install elasticsearch 7.9.1 package
`pip install elasticsearch==7.9.1`

## Describe the bug
I am following the example in the documentation for elastic search step by step (on google colab): https://huggingface.co/docs/datasets/faiss_es#elasticsearch

```
from datasets import load_dataset
squad = load_dataset('crime_and_punish', split='train[:1000]')
```

When I run the line:
`squad.add_elasticsearch_index("context", host="localhost", port="9200")`

I get the error:
`TypeError: __init__() missing 1 required positional argument: 'scheme'`

## Expected results
No error message

## Actual results

```
TypeError                                 Traceback (most recent call last)

[<ipython-input-23-9205593edef3>](https://localhost:8080/#) in <module>()
      1 import elasticsearch
----> 2 squad.add_elasticsearch_index("text", host="localhost", port="9200")

6 frames

[/usr/local/lib/python3.7/dist-packages/elasticsearch/_sync/client/utils.py](https://localhost:8080/#) in host_mapping_to_node_config(host)
    209         options["path_prefix"] = options.pop("url_prefix")
    210
--> 211     return NodeConfig(**options)  # type: ignore
    212
    213

TypeError: __init__() missing 1 required positional argument: 'scheme'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.0
- Platform: Linux, Google Colab
- Python version: Google Colab (probably 3.7)
- PyArrow version: ?


Support streaming xtreme dataset for bucc18 config.


Missing argument in precision/recall

enhancement

New feature or request

Thanks for the suggestion, @Dref360.

We are adding that argument.

**Is your feature request related to a problem? Please describe.**
[`sklearn.metrics.precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html) accepts an argument `zero_division`, but it is not available in [precision Metric](https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py#L117)

Same issue is present for Recall.

**Describe the solution you'd like**
Support for **kwargs or adding a new field for `zero_division`.

**Describe alternatives you've considered**
I could filter the warnings myself, but that is not ideal.

**Additional context**
I can make the requested changes if this is approved.


Doc: image_process small tip

This tip is unnecessary, i.e., Pillow will already be installed since the `Image` feature requires it for encoding and decoding. Thanks anyway.

cc @stevhliu I've noticed we are missing the installation section in the doc (`pip install datasets[vision]`). I can add it myself.

I've added a small tip in the `image_process` doc


Replace yahoo_answers_topics data url

The CI is failing because of issues in the dataset cards that are unrelated to this PR - merging

I replaced the Google Drive URL of the dataset by the FastAI one, since we've had some issues with Google Drive.


medical_dialog zh has very slow _generate_examples

bug

Something isn't working

## Describe the bug
After downloading the files from Google Drive, `load_dataset("medical_dialog", "zh", data_dir="./")` takes an unreasonable amount of time. Generating the train/test split for 33% of the dataset takes over 4.5 hours.

## Steps to reproduce the bug
The easiest way I've found to download files from Google Drive is to use `gdown` and use Google Colab because the download speeds will be very high due to the fact that they are both in Google Cloud.

```python
file_ids = [
        "1AnKxGEuzjeQsDHHqL3NqI_aplq2hVL_E",
        "1tt7weAT1SZknzRFyLXOT2fizceUUVRXX",
        "1A64VBbsQ_z8wZ2LDox586JIyyO6mIwWc",
        "1AKntx-ECnrxjB07B6BlVZcFRS4YPTB-J",
        "1xUk8AAua_x27bHUr-vNoAuhEAjTxOvsu",
        "1ezKTfe7BgqVN5o-8Vdtr9iAF0IueCSjP",
        "1tA7bSOxR1RRNqZst8cShzhuNHnayUf7c",
        "1pA3bCFA5nZDhsQutqsJcH3d712giFb0S",
        "1pTLFMdN1A3ro-KYghk4w4sMz6aGaMOdU",
        "1dUSnG0nUPq9TEQyHd6ZWvaxO0OpxVjXD",
        "1UfCH05nuWiIPbDZxQzHHGAHyMh8dmPQH",
]
for i in file_ids:
    url = f"https://drive.google.com/uc?id={i}"
    !gdown $url

from datasets import load_dataset

ds = load_dataset("medical_dialog", "zh", data_dir="./")
```

## Expected results
Faster load time

## Actual results
`Generating train split: 33%: 625519/1921127 [4:31:03<31:39:20, 11.37 examples/s]`

## Environment info
- `datasets` version: 2.0.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

@vrindaprabhu , could you take a look at this since you implemented it? I think the `_generate_examples` function might need to be rewritten


datasets.load_metric can not load a local metirc

Something isn't working

## Describe the bug
No matter how I hard try to tell load_metric that I want to load a local metric file, it still continues to fetch things on the Internet. And unfortunately it says 'ConnectionError: Couldn't reach'. However I can download this file without connectionerror and tell load_metric its local directory. And it comes back where it begins...

## Steps to reproduce the bug
```python
metric = load_metric(path=r'C:\Users\Gare\PycharmProjects\Gare\blue\bleu.py')
    ConnectionError: Couldn't reach https://github.com/tensorflow/nmt/raw/master/nmt/scripts/bleu.py

metric = load_metric(path='bleu')
    ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.12.1/metrics/bleu/bleu.py

metric = load_metric(path='./blue/bleu.py')
    ConnectionError: Couldn't reach https://github.com/tensorflow/nmt/raw/master/nmt/scripts/bleu.py
```

## Expected results
I do read the docs [here](https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.load_metric). There are no other parameters that help function to distinguish from local and online file but path. As what I code above, it should load from local.

## Actual results

> metric = load_metric(path=r'C:\Users\Gare\PycharmProjects\Gare\blue\bleu.py')

> ~\AppData\Local\Temp\ipykernel_19636\1855752034.py in <module>
----> 1 metric = load_metric(path=r'C:\Users\Gare\PycharmProjects\Gare\blue\bleu.py')
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, script_version, **metric_init_kwargs)
    817         if data_files is None and data_dir is not None:
    818             data_files = os.path.join(data_dir, "**")
--> 819
    820         self.name = name
    821         self.revision = revision
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, return_associated_base_path, data_files, **download_kwargs)
    639         self,
    640         path: str,
--> 641         download_config: Optional[DownloadConfig] = None,
    642         download_mode: Optional[DownloadMode] = None,
    643         dynamic_modules_path: Optional[str] = None,
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\utils\file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    297             token = hf_api.HfFolder.get_token()
    298         if token:
--> 299             headers["authorization"] = f"Bearer {token}"
    300     return headers
    301
D:\Program Files\Anaconda\envs\Gare\lib\site-packages\datasets\utils\file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    604             def _resumable_file_manager():
    605                 with open(incomplete_path, "a+b") as f:
--> 606                     yield f
    607
    608             temp_file_manager = _resumable_file_manager
ConnectionError: Couldn't reach https://github.com/tensorflow/nmt/raw/master/nmt/scripts/bleu.py

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Windows-10-10.0.22000-SP0
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.3.4

Any advice would be appreciated.


Representing dictionaries (json) objects as features

enhancement

New feature or request

In the process of adding a new dataset to the hub, I stumbled upon the inability to represent dictionaries that contain different key names, unknown in advance (and may differ between samples), original asked in the [forum](https://discuss.huggingface.co/t/representing-nested-dictionary-with-different-keys/16442).

For instance:

```
sample1 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
}}
sample2 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
}}
sample3 = {"nps": {
  "a": {"id": 0, "text": "text1"},
  "b": {"id": 1, "text": "text2"},
  "c": {"id": 2, "text": "text3"},
  "d": {"id": 3, "text": "text4"},
}}
```

the `nps` field cannot be represented as a Feature while maintaining its original structure.
@lhoestq suggested to add JSON as a new feature type, which will solve this problem.

It seems like an alternative solution would be to change the original data format, which isn't an optimal solution in my case. Moreover, JSON is a common structure, that will likely to be useful in future datasets as well.


Hotfix failing CI tests on Windows

_The documentation is not available anymore as the PR was closed or merged._

This PR makes a hotfix for our CI Windows tests: https://app.circleci.com/pipelines/github/huggingface/datasets/11092/workflows/9cfdb1dd-0fec-4fe0-8122-5f533192ebdc/jobs/67414

Fix #4118

I guess this issue is related to this PR:
- huggingface/huggingface_hub#815


Failing CI tests on Windows

bug

Something isn't working

## Describe the bug
Our CI Windows tests are failing from yesterday: https://app.circleci.com/pipelines/github/huggingface/datasets/11092/workflows/9cfdb1dd-0fec-4fe0-8122-5f533192ebdc/jobs/67414


Something isn't working

Hi @arymbe, thanks for reporting.

Unfortunately, I'm not able to reproduce your problem.

Could you please write the complete stack trace? That way we will be able to see which package originates the exception.

Hello, thank you for your fast replied. this is the complete error that I got

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

Input In [27], in <module>
----> 1 from datasets import load_dataset

venv/lib/python3.8/site-packages/datasets/__init__.py:39, in <module>
     37 from .arrow_dataset import Dataset, concatenate_datasets
     38 from .arrow_reader import ReadInstruction
---> 39 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     40 from .combine import interleave_datasets
     41 from .dataset_dict import DatasetDict, IterableDatasetDict

venv/lib/python3.8/site-packages/datasets/builder.py:40, in <module>
     32 from .arrow_reader import (
     33     HF_GCP_BASE_URL,
     34     ArrowReader,
   (...)
     37     ReadInstruction,
     38 )
     39 from .arrow_writer import ArrowWriter, BeamWriter
---> 40 from .data_files import DataFilesDict, sanitize_patterns
     41 from .dataset_dict import DatasetDict, IterableDatasetDict
     42 from .features import Features

venv/lib/python3.8/site-packages/datasets/data_files.py:297, in <module>
    292     except FileNotFoundError:
    293         raise FileNotFoundError(f"The directory at {base_path} doesn't contain any data file") from None
    296 def _resolve_single_pattern_in_dataset_repository(
--> 297     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    298     pattern: str,
    299     allowed_extensions: Optional[list] = None,
    300 ) -> List[PurePath]:
    301     data_files_ignore = FILES_TO_IGNORE
    302     fs = HfFileSystem(repo_info=dataset_info)

AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

This is weird... It is long ago that the package `huggingface_hub` has a submodule called `hf_api`.

Maybe you have a problem with your installed `huggingface_hub`...

Could you please try to update it?
```shell
pip install -U huggingface_hub
```

Yap, I've updated several times. Then, I've tried numeral combination of datasets and huggingface_hub versions. However, I think your point is right that there is a problem with my huggingface_hub installation. I'll try another way to find the solution. I'll update it later when I get the solution. Thank you :)

I'm sorry I can't reproduce your problem.

Maybe you could try to create a new Python virtual environment and install all dependencies there from scratch. You can use either:
- Python venv: https://docs.python.org/3/library/venv.html
- or conda venv (if you are using conda): https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

## Describe the bug
Could you help me please. I got this following error.

AttributeError: module 'huggingface_hub' has no attribute 'hf_api'

## Steps to reproduce the bug
when I imported the datasets

# Sample code to reproduce the bug
from datasets import list_datasets, load_dataset, list_metrics, load_metric

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: macOS-12.3-x86_64-i386-64bit
- Python version: 3.8.9
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
- Huggingface-hub: 0.5.0
- Transformers: 4.18.0

Thank you in advance.


Pretty print dataset info files

maybe just do it from now on no? (i.e. not for existing `dataset_infos.json` files)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4116). All of your documentation changes will be reflected on that endpoint.

> maybe just do it from now on no? (i.e. not for existing dataset_infos.json files)

Yes, or do this only for datasets created with `push_to_hub` to (always) keep the GH datasets small?

I reverted the change to avoid the size increase and added the `pretty_print` flag, which pretty-prints the JSON, and that flag is only True for datasets created with `push_to_hub`.

Adds indentation to the `dataset_infos.json` file when saving for nicer diffs.

(suggested by @julien-c)

This PR also updates the info files of the GH datasets. Note that this change adds more than **10 MB** to the repo size (the total file size before the change: 29.672298 MB, after: 41.666475 MB), so I'm not sure this change is a good idea.

`src/datasets/info.py` is the only relevant file for reviewers.


ImageFolder add option to ignore some folders like '.ipynb_checkpoints'

enhancement

New feature or request

Maybe it would be nice to ignore private dirs like this one (ones starting with `.`) by default.

CC @mariosasko

**Is your feature request related to a problem? Please describe.**
I sometimes like to peek at the dataset images from jupyterlab. thus '.ipynb_checkpoints' folder appears where my dataset is and (just realized) leads to accidental duplicate image additions. I think this is an easy enough thing to miss especially if the dataset is very large.

**Describe the solution you'd like**
maybe have an option `ignore` or something .gitignore style
`dataset = load_dataset("imagefolder", data_dir="./data/original", ignore="regex?")`

**Describe alternatives you've considered**
Could filter out manually


Allow downloading just some columns of a dataset

New feature or request

In the general case you can’t always reduce the quantity of data to download, since you can’t parse CSV or JSON data without downloading the whole files right ? ^^ However we could explore this case-by-case I guess

Actually for csv pandas has `usecols` which allows loading a subset of columns in a more efficient way afaik, but yes, you're right this might be more complex than I thought.

**Is your feature request related to a problem? Please describe.**
Some people are interested in doing label analysis of a CV dataset without downloading all the images. Downloading the whole dataset does not always makes sense for this kind of use case

**Describe the solution you'd like**
Be able to just download some columns of a dataset, such as doing
```python
load_dataset("huggan/wikiart",columns=["artist", "genre"])
```

Although this might make things a bit complicated in terms of local caching of datasets.


Multiprocessing with FileLock fails in python 3.9

bug

Something isn't working

On python 3.9, this code hangs:
```python
from multiprocessing import Pool
from filelock import FileLock

def run(i):
    print(f"got the lock in multi process [{i}]")

with FileLock("tmp.lock"):
    with Pool(2) as pool:
        pool.map(run, range(2))

```

This is because the subprocesses try to acquire the lock from the main process for some reason. This is not the case in older versions of python.

This can cause many issues in python 3.9. In particular, we use multiprocessing to fetch data files when you load a dataset (as long as there are >16 data files). Therefore `imagefolder` hangs, and I expect any dataset that needs to download >16 files to hang as well.

Let's see if we can fix this and have a CI that runs on 3.9.

cc @mariosasko @julien-c


ImageFolder with Grayscale images dataset

Hi! Replacing:
```python
transformed_dataset = dataset.with_transform(transforms)
transformed_dataset.set_format(type="torch", device="cuda")
```

with:
```python
def transform_func(examples):
    examples["image"] = [transforms(img).to("cuda") for img in examples["image"]]
    return examples

transformed_dataset = dataset.with_transform(transform_func)
```
should fix the issue. `datasets` doesn't support chaining of transforms (you can think of `set_format`/`with_format` as a predefined transform func for `set_transform`/`with_transforms`), so the last transform (in your case, `set_format`) takes precedence over the previous ones (in your case `with_format`). And the PyTorch formatter is not supported by the Image feature, hence the error (adding support for that is on our short-term roadmap).

Ok thanks a lot for the code snippet!

I love the way `datasets` is easy to use but it made it really long to pre-process all the images (400.000 in my case) before training anything. `ImageFolder` from pytorch is faster in my case but force to have the images on local.

I don't know how to speed up the process without switching to `ImageFolder` :smile:

You can pass `ignore_verifications=True` in `load_dataset` to skip checksum verification, which takes a lot of time if the number of files is large. We will consider making this the default behavior.

Hi, I'm facing a problem with a grayscale images dataset I have uploaded [here](https://huggingface.co/datasets/ChainYo/rvl-cdip) (RVL-CDIP)

I'm getting an error while I want to use images for training a model with PyTorch DataLoader. Here is the full traceback:

```bash
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1765, in __getitem__
    return self._getitem(
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1750, in _getitem
    formatted_output = format_table(
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 532, in format_table
    return formatter(pa_table, query_type=query_type)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/formatting.py", line 281, in __call__
    return self.format_row(pa_table)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 58, in format_row
    return self.recursive_tensorize(row)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 54, in recursive_tensorize
    return map_nested(self._recursive_tensorize, data_struct, map_list=False)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 314, in map_nested
    mapped = [
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 315, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 267, in _single_map_nested
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 267, in <dictcomp>
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 251, in _single_map_nested
    return function(data_struct)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 51, in _recursive_tensorize
    return self._tensorize(data_struct)
  File "/home/chainyo/miniconda3/envs/gan-bird/lib/python3.8/site-packages/datasets/formatting/torch_formatter.py", line 38, in _tensorize
    if np.issubdtype(value.dtype, np.integer):
AttributeError: 'bytes' object has no attribute 'dtype'
```

I don't really understand why the image is still a bytes object while I used transformations on it. Here the code I used to upload the dataset (and it worked well):

```python
train_dataset = load_dataset("imagefolder", data_dir="data/train")
train_dataset = train_dataset["train"]
test_dataset = load_dataset("imagefolder", data_dir="data/test")
test_dataset = test_dataset["train"]
val_dataset = load_dataset("imagefolder", data_dir="data/val")
val_dataset = val_dataset["train"]

dataset = DatasetDict({
    "train": train_dataset,
    "val": val_dataset,
    "test": test_dataset
})
dataset.push_to_hub("ChainYo/rvl-cdip")
```

Now here is the code I am using to get the dataset and prepare it for training:

```python
img_size = 512
batch_size = 128
normalize = [(0.5), (0.5)]
data_dir = "ChainYo/rvl-cdip"

dataset = load_dataset(data_dir, split="train")

transforms = transforms.Compose([
        transforms.Resize(img_size),
        transforms.CenterCrop(img_size),
        transforms.ToTensor(),
        transforms.Normalize(*normalize)
])

transformed_dataset = dataset.with_transform(transforms)
transformed_dataset.set_format(type="torch", device="cuda")

train_dataloader = torch.utils.data.DataLoader(
    transformed_dataset, batch_size=batch_size, shuffle=True, num_workers=4, pin_memory=True
)
```

But this get me the error above. I don't understand why it's doing this kind of weird thing?
Do I need to map something on the dataset? Something like this:

```python
labels = dataset.features["label"].names
num_labels = dataset.features["label"].num_classes

def preprocess_data(examples):
    images = [ex.convert("RGB") for ex in examples["image"]]
    labels = [ex for ex in examples["label"]]
    return {"images": images, "labels": labels}

features = Features({
    "images": Image(decode=True, id=None),
    "labels": ClassLabel(num_classes=num_labels, names=labels)
})

decoded_dataset = dataset.map(preprocess_data, remove_columns=dataset.column_names, features=features, batched=True, batch_size=100)
```


Update security policy


Matthews Correlation Metric Card

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4110). All of your documentation changes will be reflected on that endpoint.


Add Spearmanr Metric Card

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4109). All of your documentation changes will be reflected on that endpoint.


Perplexity Speedup

WRT the high values, can you add some unit tests with some [string, model] pairs and their resulting perplexity code, and @TristanThrush can run the same pairs through his version of the code?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4108). All of your documentation changes will be reflected on that endpoint.

> I thought that the perplexity metric should output the average perplexity value of all the strings that it gets as input (not a perplexity value per string, as the new version does). @lhoestq , @TristanThrush thoughts?

I support this change from Emi. If we have a perplexity function that loads GPT2 and then returns an average over all of the strings, then it is impossible to get multiple perplexities of a batch of strings efficiently. If we have this new perplexity function that is built for batching, then it is possible to get a batch of perplexities efficiently and you can still compute the average efficiently afterwards.

Thanks a lot for working on this @emibaylor @TristanThrush :)

For consistency with the other metrics, I think it's nice if we return the mean perplexity. Though I agree that having the separate perplexities per sample can also be useful. What do you think about returning both ?
```python
return {"perplexities": ppls, "mean_perplexity": np.mean(ppls)}
```
we're also doing this for the COMET metric.

This PR makes necessary changes to perplexity such that:
- it runs much faster (via batching)
- it throws an error when input is empty, or when input is one word without <BOS> token
- it adds the option to add a <BOS> token

Issues:
- The values returned are extremely high, and I'm worried they aren't correct. Even if they are correct, they are sometimes returned as `inf`, which is not very useful (see [comment below](https://github.com/huggingface/datasets/pull/4108#discussion_r843931094) for some of the output values).
  - If the values are not correct, can you help me find the error?
  - If the values are correct, it might be worth it to measure something like perplexity per word, which would allow us to get actual values for the larger perplexities, instead of just `inf`

Future:
- `stride` is not currently implemented here. I have some thoughts on how to make it happen with batching, but I think it would be better to get another set of eyes to look at any possible errors causing such large values now rather than later.


Unable to view the dataset and loading the same dataset throws the error - ArrowInvalid: Exceeded maximum rows

Something isn't working

  It's not related to the dataset viewer in itself. I can replicate the error with:

```
>>> import datasets as ds
>>> d = ds.load_dataset('Pavithree/explainLikeImFive')
Using custom data configuration Pavithree--explainLikeImFive-b68b6d8112cd8a51
Downloading and preparing dataset json/Pavithree--explainLikeImFive to /home/slesage/.cache/huggingface/datasets/json/Pavithree--explainLikeImFive-b68b6d8112cd8a51/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b...
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 305M/305M [00:03<00:00, 98.6MB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17.9M/17.9M [00:00<00:00, 75.7MB/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11.9M/11.9M [00:00<00:00, 70.6MB/s]
Downloading data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:05<00:00,  1.92s/it]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1948.42it/s]
Failed to read file '/home/slesage/.cache/huggingface/datasets/downloads/5fee9c8819754df277aee6f252e4db6897d785231c21938407b8862ca871d246' with error <class 'pyarrow.lib.ArrowInvalid'>: Exceeded maximum rows
Traceback (most recent call last):
  File "/home/slesage/hf/datasets/src/datasets/packaged_modules/json/json.py", line 144, in _generate_tables
    dataset = json.load(f)
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 916 (char 915)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets/src/datasets/load.py", line 1691, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 694, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 1151, in _prepare_split
    for key, table in logging.tqdm(
  File "/home/slesage/.pyenv/versions/datasets/lib/python3.8/site-packages/tqdm/std.py", line 1168, in __iter__
    for obj in iterable:
  File "/home/slesage/hf/datasets/src/datasets/packaged_modules/json/json.py", line 146, in _generate_tables
    raise e
  File "/home/slesage/hf/datasets/src/datasets/packaged_modules/json/json.py", line 122, in _generate_tables
    pa_table = paj.read_json(
  File "pyarrow/_json.pyx", line 246, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Exceeded maximum rows
```

cc @lhoestq @albertvillanova @mariosasko

It seems that train.json is not a valid JSON Lines file: it has several JSON objects in the first line (the 915th character in the first line starts a new object, and there's no "\n")

You need to have one JSON object per line

I'm closing this issue.

@Pavithree, please, feel free to re-open it if fixing the JSON file does not solve it.

## Dataset viewer issue - -ArrowInvalid: Exceeded maximum rows

**Link:** *https://huggingface.co/datasets/Pavithree/explainLikeImFive*

*This is the subset of original eli5 dataset  https://huggingface.co/datasets/vblagoje/lfqa. I just filtered the data samples which belongs to one particular subreddit thread. However, the dataset preview for train split  returns  the below mentioned error:
Status code:   400
Exception:     ArrowInvalid
Message:       Exceeded maximum rows
When I try to load the same dataset it returns ArrowInvalid: Exceeded maximum rows error*

Am I the one who added this dataset ? Yes


Support huggingface_hub 0.5

Looks like GH actions is not able to resolve `huggingface_hub` 0.5.0, I'm investivating

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4106). All of your documentation changes will be reflected on that endpoint.

I'm glad to see changes in `huggingface_hub` are simplifying code here.

seems to supersede #4102, feel free to close mine :)

maybe just cherry-pick the docstring fix

I think I've found the issue:
- https://github.com/huggingface/huggingface_hub/pull/790

Good catch, `huggingface_hub` doesn't support python 3.6 anymore indeed, therefore we should keep support for 0.4.0. I'm reverting the requirement version bump for now.

We can update the requirement once we drop support for python 3.6 in `datasets`

@lhoestq, I've opened this PR on `huggingface_hub`:
- https://github.com/huggingface/huggingface_hub/pull/823

Is there any strong reason why `huggingface_hub` no longer supports Python 3.6?

I think `datasets` can drop support for 3.6 soon. But for now maybe let's keep support for 0.4.0, python 3.6 users are not affected by https://github.com/huggingface/datasets/issues/4105 anyway.

`huggingface_hub` doesn't not have to support 3.6 again just for the CI IMO

@lhoestq I commented on the PR, that IMO it is not a good practice to drop support for Python 3.6 without a previous deprecation cycle.

Re-added support for older versions. I ended up checking `huggingface_hub` version to use the old, deprecated API for <0.5.0

I find it good practice to have all dependency version related code in a single file so that when you decide to remove support for an old version of a dependency it's easy to find and remove them, hence suggesting `utils/_fixes.py` in https://github.com/huggingface/datasets/issues/4105#issuecomment-1090041204

I used your suggestion @adrinjalali , I just replace the try/except with a check on the version of `huggingface_hub`

Following https://github.com/huggingface/datasets/issues/4105

`huggingface_hub` deprecated some parameters in `HfApi` in 0.5. This PR updates all the calls to HfApi to remove all the deprecations, <s>and I set the `hugginface_hub` requirement to `>=0.5.0`</s>

cc @adrinjalali @LysandreJik


push to hub fails with huggingface-hub 0.5.0

Something isn't working

Hi ! Indeed there was a breaking change in `huggingface_hub` 0.5.0 in `HfApi.create_repo`, which is called here in `datasets` by passing the org name in both the `repo_id` and the `organization` arguments:

https://github.com/huggingface/datasets/blob/2230f7f7d7fbaf102cff356f5a8f3bd1561bea43/src/datasets/arrow_dataset.py#L3363-L3369

I think we should fix that in `huggingface_hub`, will keep you posted. In the meantime please use `huggingface_hub` 0.4.0

I'll be sending a fix for this later today on the `huggingface_hub` side.

The error would be converted to a `FutureWarning` if `datasets` uses kwargs instead of positional, for example here:

https://github.com/huggingface/datasets/blob/2230f7f7d7fbaf102cff356f5a8f3bd1561bea43/src/datasets/arrow_dataset.py#L3363-L3369

to be:

``` python
            api.create_repo(
                name=dataset_name,
                token=token,
                repo_type="dataset",
                organization=organization,
                private=private,
            )
```

But `name` and `organization` are deprecated in `huggingface_hub=0.5`, and people should pass `repo_id='org/name` instead. Note that `repo_id` was introduced in 0.5 and if `datasets` wants to support older `huggingface_hub` versions (which I encourage it to do), there needs to be a helper function to do that. It can be something like:

```python
def create_repo(
    client,
    name: str,
    token: Optional[str] = None,
    organization: Optional[str] = None,
    private: Optional[bool] = None,
    repo_type: Optional[str] = None,
    exist_ok: Optional[bool] = False,
    space_sdk: Optional[str] = None,
) -> str:
    try:
        return client.create_repo(
            repo_id=f"{organization}/{name}",
            token=token,
            private=private,
            repo_type=repo_type,
            exist_ok=exist_ok,
            space_sdk=space_sdk,
        )
    except TypeError:
        return client.create_repo(
            name=name,
            organization=organization,
            token=token,
            private=private,
            repo_type=repo_type,
            exist_ok=exist_ok,
            space_sdk=space_sdk,
        )
```

in a `utils/_fixes.py` kinda file and and be used internally.

I'll be sending a patch to `huggingface_hub` to convert the error reported in this issue to a `FutureWarning`.

PR with the hotfix on the `huggingface_hub` side: https://github.com/huggingface/huggingface_hub/pull/822

We can definitely change `push_to_hub` to use `repo_id` in `datasets` and require `huggingface_hub>=0.5.0`.

Let me open a PR :)

`huggingface_hub` 0.5.1 just got released with a fix, feel free to update `huggingface_hub` ;)

## Describe the bug
`ds.push_to_hub` is failing when updating a dataset in the form "org_id/repo_id"

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("rubrix/news_test")
ds.push_to_hub("<your-user>/news_test", token="<your-token>")
```

## Expected results
The dataset is successfully uploaded

## Actual results
An error validation is raised:

```bash
if repo_id and (name or organization):
>           raise ValueError(
                "Only pass `repo_id` and leave deprecated `name` and "
                "`organization` to be None."
E               ValueError: Only pass `repo_id` and leave deprecated `name` and `organization` to be None.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.1
- `huggingface-hub`: 0.5
- Platform: macOS
- Python version: 3.8.12
- PyArrow version: 6.0.0

cc @adrinjalali


Add time series data - stock market

dataset request

Requesting to add a new dataset

Can I use instructions present in below link for time series dataset as well?
https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md

cc'ing @kashif and @NielsRogge for visibility!

@INF800 happy to add this dataset! I will try to set a PR by the end of the day... if you can kindly point me to the dataset? Also, note we have a bunch of time series datasets checked in e.g. `electricity_load_diagrams` or `monash_tsf`, and ideally this dataset could also be in a similar format.

Thankyou. This is how raw data looks like before cleaning for an individual stocks:

1. https://github.com/INF800/marktech/tree/raw-data/f/data/raw
2. https://github.com/INF800/marktech/tree/raw-data/t/data/raw
3. https://github.com/INF800/marktech/tree/raw-data/rdfn/data/raw
4. https://github.com/INF800/marktech/tree/raw-data/irbt/data/raw
5. https://github.com/INF800/marktech/tree/raw-data/hll/data/raw
6. https://github.com/INF800/marktech/tree/raw-data/infy/data/raw
7. https://github.com/INF800/marktech/tree/raw-data/reli/data/raw
8. https://github.com/INF800/marktech/tree/raw-data/hdbk/data/raw

> Scraping is automated using GitHub Actions. So, everyday we will see a new file added in the above links.

I can rewrite the cleaning scripts to make sure it fits HF dataset standards. (P.S I am very much new to HF dataset)

The data set above can be converted into univariate regression / multivariate regression / sequence to sequence generation dataset etc. So, do we have some kind of transformation modules that will read the dataset as some type of dataset (`GenericTimeData`) and convert it to other possible dataset relating to a specific ML task. **By having this kind of transformation module, I only have to add data once** and use transformation module whenever necessary

Additionally, having some kind of versioning for the dataset will be really helpful because it will keep on updating - especially time series datasets

thanks @INF800 I'll have a look. I believe it should be possible to incorporate this into the time-series format.

## Adding a Time Series Dataset
- **Name:** 2min ticker data for stock market
- **Description:** 8 stocks' data collected for 1month post ukraine-russia war. 4 NSE stocks and 4 NASDAQ stocks. Along with technical indicators (additional features) as shown in below image
- **Data:** Collected by myself from investing.com
- **Motivation:** Test applicability of transformer based model on stock market / time series problem

![image](https://user-images.githubusercontent.com/45640029/161904077-52fe97cb-3720-4e3f-98ee-7f6720a056e2.png)


Add the `GSM8K` dataset


[hub] Fix `api.create_repo` call?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4102). All of your documentation changes will be reflected on that endpoint.


enhancement

New feature or request

Hi! Can you please specify the full name of the dataset? IIRC `full_numbers` is one of the configs of the `svhn` dataset, and its generation is slow due to data being stored in binary Matlab files. Even if you specify a specific split, `datasets` downloads all of them, but we plan to fix that soon and only download the requested split.

If you are in a hurry, download the `svhn` script [here](`https://huggingface.co/datasets/svhn/blob/main/svhn.py`), remove [this code](https://huggingface.co/datasets/svhn/blob/main/svhn.py#L155-L162), and run:
```python
from datasets import load_dataset
dset = load_dataset("path/to/your/local/script.py", "full_numbers")
```

And to make loading easier in Colab, you can create a dataset repo on the Hub and upload the script there. Or push the script to Google Drive and mount the drive in Colab.

How can I download only the train and test split for full numbers using load_dataset()?

I do not need the extra split and it will take 40 mins just to download in Colab. I have very short time in hand. Please help.


Improve RedCaps dataset card

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4100). All of your documentation changes will be reflected on that endpoint.

This PR modifies the RedCaps card to:
* fix the formatting of the Point of Contact fields on the Hub
* speed up the image fetching logic (aligns it with the [img2dataset](https://github.com/rom1504/img2dataset) tool) and make it more robust (return None if **any** exception is thrown)


bug

Something isn't working

Hi @andreybond, thanks for reporting.

Unfortunately, I'm not able to able to reproduce your issue:
```python
In [4]: from datasets import load_dataset
   ...: datasets = load_dataset("nielsr/XFUN", "xfun.ja")

In [5]: datasets
Out[5]:
DatasetDict({
    train: Dataset({
        features: ['id', 'input_ids', 'bbox', 'labels', 'image', 'entities', 'relations'],
        num_rows: 194
    })
    validation: Dataset({
        features: ['id', 'input_ids', 'bbox', 'labels', 'image', 'entities', 'relations'],
        num_rows: 71
    })
})
```

The only reason I can imagine this issue may arise is if your default encoding is not "UTF-8" (and it is ASCII instead). This is usually the case on Windows machines; but you say your environment is a Linux machine. Maybe you change your machine default encoding?

Could you please check this?
```python
In [6]: import sys

In [7]: sys.getdefaultencoding()
Out[7]: 'utf-8'
```

I opened a PR in the original dataset loading script:
-  microsoft/unilm#677

and fixed the corresponding dataset script on the Hub:
- https://huggingface.co/datasets/nielsr/XFUN/commit/73ba5e026621e05fb756ae0f267eb49971f70ebd

import sys
sys.getdefaultencoding()

returned: 'utf-8'

---------------------

I've just cloned master branch - your fix works! Thank you!

## Describe the bug
Error "UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)" is thrown when downloading dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset
datasets = load_dataset("nielsr/XFUN", "xfun.ja")
```

## Expected results
Dataset should be downloaded without exceptions

## Actual results
Stack trace (for the second-time execution):
Downloading and preparing dataset xfun/xfun.ja to /root/.cache/huggingface/datasets/nielsr___xfun/xfun.ja/0.0.0/e06e948b673d1be9a390a83c05c10e49438bf03dd85ae9a4fe06f8747a724477...
Downloading data files: 100%
2/2 [00:00<00:00, 88.48it/s]
Extracting data files: 100%
2/2 [00:00<00:00, 79.60it/s]

UnicodeDecodeErrorTraceback (most recent call last)
<ipython-input-31-79c26bd1109c> in <module>
      1 from datasets import load_dataset
      2
----> 3 datasets = load_dataset("nielsr/XFUN", "xfun.ja")

/usr/local/lib/python3.6/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    604         )
    605
--> 606         # By default, return all splits
    607         if split is None:
    608             split = {s: s for s in self.info.splits}

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    692         Args:
    693             split: `datasets.Split` which subset of the data to read.
--> 694
    695         Returns:
    696             `Dataset`

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _prepare_split(self, split_generator, check_duplicate_keys)

/usr/local/lib/python3.6/dist-packages/tqdm/notebook.py in __iter__(self)
    252         if not self.disable:
    253             self.display(check_delay=False)
--> 254
    255     def __iter__(self):
    256         try:

/usr/local/lib/python3.6/dist-packages/tqdm/std.py in __iter__(self)
   1183             for obj in iterable:
   1184                 yield obj
-> 1185             return
   1186
   1187         mininterval = self.mininterval

~/.cache/huggingface/modules/datasets_modules/datasets/nielsr--XFUN/e06e948b673d1be9a390a83c05c10e49438bf03dd85ae9a4fe06f8747a724477/XFUN.py in _generate_examples(self, filepaths)
    140             logger.info("Generating examples from = %s", filepath)
    141             with open(filepath[0], "r") as f:
--> 142                 data = json.load(f)
    143
    144             for doc in data["documents"]:

/usr/lib/python3.6/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    294
    295     """
--> 296     return loads(fp.read(),
    297         cls=cls, object_hook=object_hook,
    298         parse_float=parse_float, parse_int=parse_int,

/usr/lib/python3.6/encodings/ascii.py in decode(self, input, final)
     24 class IncrementalDecoder(codecs.IncrementalDecoder):
     25     def decode(self, input, final=False):
---> 26         return codecs.ascii_decode(input, self.errors)[0]
     27
     28 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 213: ordinal not in range(128)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0 (but reproduced with many previous versions)
- Platform: Docker: Linux da5b74136d6b 5.3.0-1031-azure #32~18.04.1-Ubuntu SMP Mon Jun 22 15:27:23 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ; Base docker image is : huggingface/transformers-pytorch-cpu
- Python version: 3.6.9
- PyArrow version:   6.0.1


Proposing WikiSplit metric card

_The documentation is not available anymore as the PR was closed or merged._

A quick Github tip ;) To avoid running N times the CI, you can push all the changes at once: go to Files Changed tab, and on each suggestion there's a "add to commit batch" and then you can do one commit for all the suggestions you want to approve ;)

Pinging @lhoestq to ensure that my distinction between the dataset and the metric are clear :sweat_smile:


Updating FrugalScore metric card

removing duplicate paragraph


Add support for streaming Zarr stores for hosted datasets

enhancement

New feature or request

Hi @jacobbieker, thanks for your request and study of possible alternatives.

We are very interested in finding a way to make `datasets` useful to you.

Looking at the Zarr docs, I saw that among its storage alternatives, there is the ZIP file format: https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore

This might be convenient for many reasons:
- On the one hand, we avoid the Git issue with huge number of small files: chunks files are compressed into a single ZIP file
- On the other hand, the ZIP file format is specially suited for streaming data because it allows random access to its component files (i.e. it supports random access to its chunks)

Anyway, I think that a Python loading script will be necessary: you need to implement additional logic to select certain chunks (based on date or other criteria).

Please, let me know if this makes sense to you.

**Is your feature request related to a problem? Please describe.**
Lots of geospatial data is stored in the Zarr format. This format works well for n-dimensional data and coordinates, and can have good compression. Unfortunately, HF datasets doesn't support streaming in data in Zarr format as far as I can tell. Zarr stores are designed to be easily streamed in from cloud storage, especially with xarray and fsspec. Since geospatial data tends to be very large, and on the order of TBs of data or 10's of TBs of data for a single dataset, it can be difficult to store the dataset locally for users. Just adding Zarr stores with HF git doesn't work well (see https://github.com/huggingface/datasets/issues/3823) as Zarr splits the data into lots of small chunks for fast loading, and that doesn't work well with git. I've somewhat gotten around that issue by tarring each Zarr store and uploading them as a single file, which seems to be working (see https://huggingface.co/datasets/openclimatefix/gfs-reforecast for example data files, although the script isn't written yet). This does mean that streaming doesn't quite work though. On the other hand, in https://huggingface.co/datasets/openclimatefix/eumetsat_uk_hrv we stream in a Zarr store from a public GCP bucket quite easily.

**Describe the solution you'd like**
A way to upload Zarr stores for hosted datasets so that we can stream it with xarray and fsspec.

**Describe alternatives you've considered**
Tarring each Zarr store individually and just extracting them in the dataset script -> Downside this is a lot of data that probably doesn't fit locally for a lot of potential users.
Pre-prepare examples in a format like Parquet -> Would use a lot more storage, and a lot less flexibility, in the eumetsat_uk_hrv, we use the one Zarr store for multiple different configurations.


fix typo in rename_column error message

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4095). All of your documentation changes will be reflected on that endpoint.

I feel bad submitting such a tiny change as a PR but it confused me today 😄


Helo Mayfrends

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


elena-soare/crawled-ecommerce: missing dataset

dataset-viewer

Related to the dataset viewer on huggingface.co

It's a bug! Thanks for reporting, I'm looking at it.

By the way, the error on our part is due to the huge size of every row (~90MB). The dataset viewer does not support such big dataset rows for the moment.
Anyway, we're working to give a hint about this in the dataset viewer.

elena-soare/crawled-ecommerce

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Fix dataset `amazon_us_reviews` metadata - 4/4/2022

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4092). All of your documentation changes will be reflected on that endpoint.

Fixes #4048 by running `dataset-cli test` to reprocess data and regenerate metadata. Additionally I've updated the README to include up-to-date counts for the subsets.


Build a Dataset One Example at a Time Without Loading All Data Into Memory

enhancement

New feature or request

Hi! Yes, the problem with `add_item` is that it keeps examples in memory, so you are left with these options:
* writing a dataset loading script in which you iterate over `custom_example_dict_streamer` and yield the examples (in `_generate examples`)
* storing the data in a JSON/CSV/Parquet/TXT file and using `Dataset.from_{format}`
* using `add_item` + `save_to_disk` on smaller chunks:
    ```python
    from datasets import Dataset, concatenate_datasets
    MAX_SAMPLES_IN_MEMORY = 1000
    samples_in_dset = 0
    dset = Dataset.from_dict({"col1": [], "col2": []})  # empty dataset
    path_to_save_dir = "path/to/save/dir"
    num_chunks = 0
    for example_dict in custom_example_dict_streamer("/path/to/raw/data"):
        dset = dset.add_item(example_dict)
        samples_in_dset += 1
        if samples_in_dset == MAX_SAMPLES_IN_MEMORY:
            samples_in_dset = 0
            dset.save_to_disk(f"{path_to_save_dir}{num_chunks}")
            num_chunks =+ 1
            dset = Dataset.from_dict({"col1": [], "col2": []})  # empty dataset
    if samples_in_dset > 0:
        dset.save_to_disk(f"{path_to_save_dir}{num_chunks}")
        num_chunks =+ 1
    loaded_dsets = []  # memory-mapped
    for chunk_num in range(num_chunks):
        dset = Dataset.load_from_disk(f"{path_to_save_dir}{chunk_num}")
        loaded_dsets.append(dset)
    final_dset = concatenate_datasets(dset)
    ```
    If you still have issues with this approach, you can try to delete unused datasets with `gc.collect()` to free some memory.

This is really elegant, thank you @mariosasko!  I will try this.

**Is your feature request related to a problem? Please describe.**
I have a very large dataset stored on disk in a custom format.  I have some custom code that reads one data example at a time and yields it in the form of a dictionary.  I want to construct a `Dataset` with all examples, and then save it to disk.  I later want to load the saved `Dataset` and use it like any other HuggingFace dataset, get splits, wrap it in a PyTorch `DataLoader`, etc.  **Crucially, I do not ever want to materialize all the data in memory while building the dataset.**

**Describe the solution you'd like**
I would like to be able to do something like the following.  Notice how each example is read and then immediately added to the dataset.  We do not store all the data in memory when constructing the `Dataset`.  If it helps, I will know the schema of my dataset before hand.
```

# Initialize an empty Dataset, possibly from a known schema.
dataset = Dataset()

# Read in examples one by one using a custom data streamer.
for example_dict in custom_example_dict_streamer("/path/to/raw/data"):

    # Add this example to the dict but do not store it in memory.
    dataset.add_item(example_dict)

# Save the final dataset to disk as an Arrow-backed dataset.
dataset.save_to_disk("/path/to/dataset")

...

# I'd like to be able to later `load_from_disk` and use the loaded Dataset
# just like any other memory-mapped pyarrow-backed HuggingFace dataset...
loaded_dataset = Dataset.load_from_disk("/path/to/dataset")
loaded_dataset.set_format(type="torch", columnns=["foo", "bar", "baz"])
dataloader = torch.utils.data.DataLoader(loaded_dataset, batch_size=16)
...

```

**Describe alternatives you've considered**
I initially tried to read all the data into memory, construct a Pandas DataFrame and then call `Dataset.from_pandas`.  This would not work as it requires storing all the data in memory.  It seems that there is an `add_item` method already -- I tried to implement something like the desired API written above, but I've not been able to initialize an empty `Dataset` (this seems to require several layers of constructing `datasets.table.Table` which requires constructing a `pyarrow.lib.Table`, etc).  I also considered writing my data to multiple sharded CSV files or JSON files and then using `from_csv` or `from_json`.  I'd prefer not to do this because (1) I'd prefer to avoid the intermediate step of creating these temp CSV/JSON files and (2) I'm not sure if `from_csv` and `from_json` use memory-mapping.

Do you have any suggestions on how I'd be able to achieve this use case?  Does something already exist to support this?  Thank you very much in advance!


Avoid writing empty license files

This PR avoids the creation of empty `LICENSE` files.


Create metric card for Frugal Score

Proposing metric card for Frugal Score.

@albertvillanova or @lhoestq  -- there are certain aspects that I'm not 100% sure on (such as how exactly the distillation between BertScore and FrugalScore is done) -- so if you find that something isn't clear, please let me know!


Remove unused legacy Beam utils

This PR removes unused legacy custom `WriteToParquet`, once official Apache Beam includes the patch since version 2.22.0:
- Patch PR: https://github.com/apache/beam/pull/11699
- Issue: https://issues.apache.org/jira/browse/BEAM-10022

In relation with:
- #204


Fix BeamWriter output Parquet file

Since now, the `BeamWriter` saved a Parquet file with a simplified schema, where each field value was serialized to JSON. That resulted in Parquet files larger than Arrow files.

This PR:
- writes Parquet file preserving original schema and without serialization, thus avoiding serialization overhead and resulting in a smaller output file size.
- fixes `parquet_to_arrow` function


Dataset viewer issue for McGill-NLP/feedbackQA

Hi @cslizc, thanks for reporting.

I have just forced the refresh of the corresponding cache and the preview is working now.

## Dataset viewer issue for '*McGill-NLP/feedbackQA*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/McGill-NLP/feedbackQA)*

*short description of the issue*
The dataset can be loaded correctly with `load_dataset` but the preview doesn't work. Error message:

```
Status code:   400
Exception:     Status400Error
Message:       Not found. Maybe the cache is missing, or maybe the dataset does not exist.
```

Am I the one who added this dataset ? Yes


bug

Something isn't working

Now, I can't find any reference to set_progress_bar_enabled in the code.

I think it have been deleted

Hi @virilo,

Please note that since `datasets` version 2.0.0, we have aligned with `transformers` the management of the progress bar (among other things):
- #3897

Now, you should update your code to use `datasets.logging.disable_progress_bar`.

You have more info in our docs: [Logging methods](https://huggingface.co/docs/datasets/package_reference/logging_methods)

## Describe the bug

datasets.set_progress_bar_enabled(False) not working in datasets v2

## Steps to reproduce the bug
```python
datasets.set_progress_bar_enabled(False)
```

## Expected results
datasets not using any progress bar

## Actual results

AttributeError: module 'datasets' has no attribute 'set_progress_bar_enabled

## Environment info

datasets version 2


Errors in `Train with Datasets` Tensorflow code section on Huggingface.co

Something isn't working

Hi @blackhat-coder, thanks for reporting.

Please note that the `transformers` library updated their data collators API last year (version 4.10.0):
- huggingface/transformers#13105

now requiring to pass `return_tensors` argument at Data Collator instantiation.

And therefore, we also updated in the `datasets` library documentation all the examples using `transformers` data collators.

If you would like to follow our examples, please update your installed `transformers` version:
```
pip install -U transformers
```

## Describe the bug
Hi
### Error 1
Running the Tensforlow code on [Huggingface](https://huggingface.co/docs/datasets/use_dataset) gives a TypeError: __init__() got an unexpected keyword argument 'return_tensors'
### Error 2
`DataCollatorWithPadding` isn't imported

## Steps to reproduce the bug
```python
import tensorflow as tf
from datasets import load_dataset
from transformers import AutoTokenizer
dataset = load_dataset('glue', 'mrpc', split='train')
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
dataset = dataset.map(lambda e: tokenizer(e['sentence1'], truncation=True, padding='max_length'), batched=True)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
train_dataset = dataset["train"].to_tf_dataset(
  columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'],
  shuffle=True,
  batch_size=16,
  collate_fn=data_collator,
)
```
This is the same code on Huggingface.co

## Actual results
TypeError: __init__() got an unexpected keyword argument 'return_tensors'

## Environment info
- `datasets` version: 2.0.0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.7
- PyArrow version: 6.0.0
- Pandas version: 1.4.1
>


Add SacreBLEU Metric Card

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4083). All of your documentation changes will be reflected on that endpoint.


Add chrF(++) Metric Card

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4082). All of your documentation changes will be reflected on that endpoint.


Close parquet writer properly in `push_to_hub`

We don’t call writer.close(), which causes https://github.com/huggingface/datasets/issues/4077. It can happen that we upload the file before the writer is garbage collected and writes the footer.

I fixed this by explicitly closing the parquet writer.

Close https://github.com/huggingface/datasets/issues/4077.