Add MetaShift dataset

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq Please could you review this when you get time. Thank you.

Thanks a lot for your inputs @mariosasko .
> Maybe we can add the generated meta-graphs to the card as images (with attributions)?

Yes. We can do this for the default set of classes. Will add this.

> Would be cool if we could have them as additional configs. Also, maybe we could have configs that expose [image metadata](https://github.com/Weixin-Liang/MetaShift/tree/main/dataset/meta_data) from the https://nlp.stanford.edu/data/gqa/sceneGraphs.zip file (this file is downloaded in the script but not used).

I'll try adding the bonus section as additional config.
Regarding exposing the image metadata with a config parameter, how will we showcase/display this information ?

> Regarding exposing the image metadata with a config parameter, how will we showcase/display this information ?

Oh, I forgot to mention that. Let's add a `Dataset Usage` section to the card to document the params (similar to this: https://huggingface.co/datasets/electricity_load_diagrams#dataset-usage). Also, feel free to add the constants that can be tuned as config params (e.g. `IMAGE_SUBSET_SIZE_THRESHOLD` or the `5` in `len(subject_data) <= 5`).

Okay. Got it. Will add these and constants as config parameters.

The image metadata from scene graphs looks like this :
```json
{
    "2407890": {
        "width": 640,
        "height": 480,
        "location": "living room",
        "weather": none,
        "objects": {
            "271881": {
                "name": "chair",
                "x": 220,
                "y": 310,
                "w": 50,
                "h": 80,
                "attributes": ["brown", "wooden", "small"],
                "relations": {
                    "32452": {
                        "name": "on",
                        "object": "275312"
                    },
                    "32452": {
                        "name": "near",
                        "object": "279472"
                    }
                }
            }
        }
    }
}
```
``load_dataset("metashift", selected_classes=["cat", "dog", ...], image_metadata=True)``
How do we showcase/display the image metadata(json) information ?

> How do we showcase/display the image metadata(json) information ?

We can add the JSON fields as keys to the features dict:
```python
  if self.config.image_metadata:
      features.update({"width": Value("int"), "height": Value("int"), "location": Value("string"), ...})
```

P.S. Would rename `image_metadata`  to `with_image_metadata`

I have added the following :
- Added the meta-graphs to the card as images under the Section "Dataset Meta-Graphs".
- Generate the Attributes-Dataset using config parameter. [ [Link](https://github.com/Weixin-Liang/MetaShift#bonus-generate-the-metashift-attributes-dataset-subsets-defined-by-subject-attributes) ]
- Expose image metadata using config parameter.
Format of the image metadata is as follows : [Link](https://cs.stanford.edu/people/dorarad/gqa/download.html)
I have modified the "Objects" which is dict to a list of dicts with an additional parameter named object_id.
I have defined the structure as follows :
```
{
    "width": datasets.Value("int64"),
    "height": datasets.Value("int64"),
    "location": datasets.Value("string"),
    "weather": datasets.Value("string"),
    "objects": datasets.Sequence(
        {
            "object_id": datasets.Value("string"),
            "name": datasets.Value("string"),
            "x": datasets.Value("int64"),
            "y": datasets.Value("int64"),
            "w": datasets.Value("int64"),
            "h": datasets.Value("int64"),
            "attributes": datasets.Sequence(datasets.Value("string")),
            "relations": datasets.Sequence(
                {
                    "name": datasets.Value("string"),
                    "object": datasets.Value("string"),
                }
            ),
        }
    ),
}
```
Problem is that objects is not being shown as list of dicts. The output looks as follows :

> metashift_dataset['train'][0]

```json
{'image_id': '2338755', 'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x281 at 0x7F066C5A49D0>, 'label': 0, 'context': 'ground', 'width': 500, 'height': 281, 'location': None, 'weather': None, 'objects': {'object_id': ['3070704', '3070705', '3070706', '2416713', '3070702', '2790660', '3063157', '2354960', '2037127', '2392939', '2912743', '2125407', '2735257', '3260906', '2351018', '3288269', '3699852', '2734378', '3421201', '2863115'], 'name': ['bicycle', 'bicycle', 'bicycle', 'boot', 'bicycle', 'motorcycle', 'pepperoni', 'head', 'building', 'wall', 'shorts', 'people', 'wheel', 'bricks', 'man', 'cat', 'boot', 'door', 'ground', 'building'], 'x': [137, 371, 458, 215, 468, 399, 368, 245, 0, 140, 260, 284, 138, 451, 339, 187, 210, 26, 0, 313], 'y': [116, 86, 94, 150, 91, 80, 107, 22, 0, 44, 109, 69, 145, 226, 69, 22, 230, 0, 119, 0], 'w': [197, 27, 15, 73, 24, 53, 9, 37, 289, 46, 43, 30, 74, 28, 35, 116, 53, 107, 500, 55], 'h': [126, 25, 38, 128, 43, 50, 16, 44, 158, 73, 51, 52, 97, 15, 73, 252, 46, 147, 162, 77], 'attributes': [[], [], [], ['white'], [], [], [], [], [], [], [], [], [], [], [], ['white'], ['white'], ['large', 'black'], ['brick'], []], 'relations': [{'name': ['to the left of'], 'object': ['3260906']}, {'name': ['to the left of', 'to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the right of'], 'object': ['3070706', '2351018', '2125407', '2790660', '2037127', '3070702', '3288269']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the right of'], 'object': ['2351018', '3070705', '3070702', '2790660', '3063157']}, {'name': ['to the right of'], 'object': ['2735257']}, {'name': ['to the right of', 'to the right of', 'to the right of', 'to the right of', 'to the right of'], 'object': ['2351018', '2790660', '3070706', '3070705', '3063157']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the right of', 'to the right of', 'to the right of'], 'object': ['3070705', '2351018', '3070702', '3070706', '3063157', '2125407', '2037127', '3288269']}, {'name': ['to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the right of', 'to the left of', 'to the right of'], 'object': ['2037127', '3070706', '3070702', '2912743', '3288269', '2790660', '2125407']}, {'name': ['to the left of', 'to the right of'], 'object': ['2863115', '2734378']}, {'name': ['to the left of', 'to the left of', 'to the left of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['3070705', '2351018', '3063157', '2125407', '2790660', '2863115']}, {'name': ['to the left of', 'to the right of', 'to the left of'], 'object': ['2125407', '2734378', '3288269']}, {'name': ['to the left of', 'on', 'to the left of'], 'object': ['2351018', '3288269', '3063157']}, {'name': ['to the left of', 'to the left of', 'to the right of', 'to the left of', 'to the right of', 'to the left of'], 'object': ['3063157', '2351018', '2037127', '3070705', '2392939', '2790660']}, {'name': ['to the left of', 'to the left of'], 'object': ['2416713', '3288269']}, {'name': ['to the right of'], 'object': ['3070704']}, {'name': ['to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the left of', 'to the right of', 'walking down'], 'object': ['2037127', '2790660', '2125407', '3070705', '3070706', '2912743', '3070702', '3288269', '3421201']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['2392939', '2734378', '2790660', '2735257', '3063157', '3070705', '2351018', '2863115']}, {'name': [], 'object': []}, {'name': ['of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['2037127', '2354960', '3288269', '2392939']}, {'name': [], 'object': []}, {'name': ['to the right of', 'to the right of', 'to the right of'], 'object': ['2037127', '3288269', '2354960']}]}}
```
Expected output of image_metadata would be :
```
{'height': 281,
 'location': None,
 'objects': [{'attributes': [],
              'h': 126,
              'name': 'bicycle',
              'object_id': '3070704',
              'relations': [{'name': 'to the left of', 'object': '3260906'}],
              'w': 197,
              'x': 137,
              'y': 116},
             {'attributes': [],
              'h': 25,
              'name': 'bicycle',
              'object_id': '3070705',
              'relations': [{'name': 'to the left of', 'object': '3070706'},
                            {'name': 'to the right of', 'object': '2351018'},
                            {'name': 'to the right of', 'object': '2125407'},
                            {'name': 'to the left of', 'object': '2790660'},
                            {'name': 'to the right of', 'object': '2037127'},
                            {'name': 'to the left of', 'object': '3070702'},
                            {'name': 'to the right of', 'object': '3288269'}],
              'w': 27,
              'x': 371,
              'y': 86},
             {'attributes': ['white'],
              'h': 252,
              'name': 'cat',
              'object_id': '3288269',
              'relations': [{'name': 'to the right of', 'object': '2392939'},
                            {'name': 'to the right of', 'object': '2734378'},
                            {'name': 'to the left of', 'object': '2790660'},
                            {'name': 'to the right of', 'object': '2735257'},
                            {'name': 'to the left of', 'object': '3063157'},
                            {'name': 'to the left of', 'object': '3070705'},
                            {'name': 'to the left of', 'object': '2351018'},
                            {'name': 'to the left of', 'object': '2863115'}],
              'w': 116,
              'x': 187,
              'y': 22},
             {'attributes': ['white'],
              'h': 46,
              'name': 'boot',
              'object_id': '3699852',
              'relations': [],
              'w': 53,
              'x': 210,
              'y': 230},
             .
             .
             .
             {'attributes': ['large', 'black'],
              'h': 147,
              'name': 'door',
              'object_id': '2734378',
              'relations': [{'name': 'of', 'object': '2037127'},
                            {'name': 'to the left of', 'object': '2354960'},
                            {'name': 'to the left of', 'object': '3288269'},
                            {'name': 'to the left of', 'object': '2392939'}],
              'w': 107,
              'x': 26,
              'y': 0},
             {'attributes': ['brick'],
              'h': 162,
              'name': 'ground',
              'object_id': '3421201',
              'relations': [],
              'w': 500,
              'x': 0,
              'y': 119},
             {'attributes': [],
              'h': 77,
              'name': 'building',
              'object_id': '2863115',
              'relations': [{'name': 'to the right of', 'object': '2037127'},
                            {'name': 'to the right of', 'object': '3288269'},
                            {'name': 'to the right of', 'object': '2354960'}],
              'w': 55,
              'x': 313,
              'y': 0}],
 'weather': None,
 'width': 500}

```

May I know how to get the list of dicts representation correctly ?

---
To-Do :

- [x] Generate dataset_infos.json file.
- [x] Add “Dataset Usage” section in the cards and write about the config parameters.
- [x] Add the constants that can be tuned as config params.

> Problem is that objects is not being shown as list of dicts. The output looks as follows :

That's expected. We convert a sequence of dictionaries to a dictionary of sequences to keep the formatting aligned with Tensorflow Datasets. You could disable this behavior by replacing `"objects": datasets.Sequence(object_fields_dict)` with `"objects": [object_fields_dict]`, but that's not what we usually do, so let's keep it like that.

Also, to limit the size of the dataset repo, please remove the pushed images and pass URLs to the images instead under the `src` attribute (and specify `alt` in case the URLs go down).

I'll do a proper review again after you are finished with the dummy data.

> That's expected.

Okay. Got it. Thanks. I thought I was doing something wrong.

> Also, to limit the size of the dataset repo, please remove the pushed images and pass URLs to the images instead under the src attribute (and specify alt in case the URLs go down).

Sure. Where do we host these images ? Can I upload them to any free image hosting platform or is there any particular website you use ?

> I'll do a proper review again after you are finished with the dummy data.

Sure. Thanks. I'm working on this part. Will update you.

Update :
- I have generated the dataset_infos.json file.

> I suggest you try to generate the dataset_infos.json file first, and then I can help with the dummy data.

I am having issues creating the dummy data. I get the following which I use the command :

`datasets-cli dummy_data datasets/metashift`

```
Dataset metashift with config MetashiftConfig(name='metashift', version=1.0.0, data_dir=None, data_files=None, description=None) seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data/full-candidate-subsets.pkl.
Traceback (most recent call last):
  File "datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/datasets/commands/dummy_data.py", line 324, in run
    dataset_builder=dataset_builder, mock_dl_manager=mock_dl_manager
  File "/datasets/commands/dummy_data.py", line 407, in _print_dummy_data_instructions
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

> Feel free to host the images online (on imgur for example) :)

Sure. Will do that.

Thanks for the explanation regarding the dummy data zip files. I will try it out and let you know.

Instead of uploading the images to a hosting service, you can directly reference their GitHub URLs (open the image in the MetaShift repo -> click Download -> copy the image URL). For instance, this is the URL of one of the images:`https://raw.githubusercontent.com/Weixin-Liang/MetaShift/main/docs/figures/Cat-MetaGraph.jpg`. Also, feel free to replace `main` with the most recent commit hash in the copied URLs to make them more robust.

@mariosasko I've actually created metagraphs for all the default classes other than those present in the GitHub Repo and included all of them. :)  The Repo has them only for two classes.

In case we want to limit the no.of meta graphs included, we can stick to the github URLs from the repo itself.

Update :
- I could add the dummy data and get the dummy data test to work. Since we have a preprocessing step on the dataset, one of the .pkl file size is on the higher side. This was done for the tests to pass. I hope that is okay. The dummy.zip file size is about 273K.

To-Do :
- [x] Update Dataset Structure in the data cards to include Data Instances when config parameters are used.

Please could you review when you get time. Thank you.

Thanks a lot for your suggestions, Mario. The thing I learnt from the review is that I need to make better sentence formations. I will keep this in mind. :)

Thanks a lot for your support. @mariosasko  and @lhoestq .

> Super impressed by your work on this, congrats :)

Its my first dataset contribution to the 🤗 Datasets library, I'm super excited. Thank you. :)

Also, I think we can close this request issue now, [#3813](https://github.com/huggingface/datasets/issues/3813)

This PR adds the MetaShift dataset.

Dataset Request  : Add MetaShift dataset [#3813](https://github.com/huggingface/datasets/issues/3813)

@lhoestq As discussed,
- I have copied the preprocessing script and modified it as required to not create new directories and folders and instead yield the images.
-  I do the preprocessing in _split_generators to get the required data which is then passed to _generate_examples.
-  Beyond the generated MetaShift dataset, the original preprocess script also generates the meta-graphs for each class, I have currently not included this part. [ Ref : [Link](https://github.com/Weixin-Liang/MetaShift#generate-full-metashift) ]
-  There is a Bonus section, the authors share.  I have currently not included this part. [ Ref : [Link](https://github.com/Weixin-Liang/MetaShift#bonus-generate-the-metashift-attributes-dataset-subsets-defined-by-subject-attributes) ]
- I had a basic test script which downloaded the dataset and tested the basic functionality. Things seems fine.
For real data, I performed the following test :
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_metashift
============================================== test session starts ===============================================
platform linux -- Python 3.7.11, pytest-7.0.1, pluggy-1.0.0
rootdir: ./datasets
plugins: hydra-core-1.1.1, datadir-1.3.1, forked-1.4.0, xdist-2.5.0
collected 1 item

tests/test_dataset_common.py .                                                                             [100%]

========================================= 1 passed in 4821.25s (1:20:21) =========================================
```
- I couldn't get the dummy dataset. Need some inputs here.
Error as follows :
```
Using custom data configuration default
Dataset metashift with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data/full-candidate-subsets.pkl.
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

To-Do :

- [x] Currently I am using the default _SELECTED_CLASSES. I need to use config option here as suggested
- [x] Complete fields in the Dataset Card.
- [x] Tagging the dataset using the Datasets Tagging app.

Need your help and suggestions for improvement. Thank you


Add exact match metric

Adding the exact match metric and its metric card.

Note: Some of the tests have failed, but I wanted to make a PR anyway so that the rest of the code can be reviewed if anyone has time. I'll look into + work on fixing the failed tests when I'm back online after the weekend


Create README.md for WER metric

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3898). All of your documentation changes will be reflected on that endpoint.

For ASR you can probably ping @patrickvonplaten

Ah only noticed now that ` # Values from popular papers` is from a template. @lhoestq @sashavor - not really sure if this section is useful in general really.

IMO, it's more confusing/misleading than it helps. E.g. a value of 0.03 WER on a fake read-out audio dataset is not better than a WER of 0.3 on a real-world noisy, conversational audio dataset. I think the same holds true for other metrics no? I can think of very little metrics where a metric value is not dataset dependent. E.g. perplexity is super dataset dependent, summarization metrics like ROUGE as well, ...

Also, I don't really see what this section tries to achieve - is the idea here to give the reader some papers that use this metric to better understand in which context it is used? Should we maybe rename the section to `Popular papers making use of this metric` or something?

I put "Values from popular papers" as a subsection of "Output values" -- I hope that's a compromise that works for everyone :hugs:

Proposing a draft WER metric card, @lhoestq I'm not very certain about "Values from popular papers" -- I don't know ASR very well, what do you think of the examples I found?


Align tqdm control/cache control with Transformers

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3897). All of your documentation changes will be reflected on that endpoint.

This PR:
* aligns the `tqdm` logic with Transformers (follows https://github.com/huggingface/transformers/pull/15167) by moving the code to `utils/logging.py`, adding `enable_progres_bar`/`disable_progres_bar` and removing `set_progress_bar_enabled` (a note for @lhoestq: I'm not adding `logging.tqdm` to the public namespace in this PR to avoid the situation where `from datasets import *; tqdm` would overshadow the standard `tqdm`
* aligns the cache control with the new `tqdm` logic by adding `enable_caching`/`disable_caching` to the  public namespace and deprecating `set_caching_enabled` (not fully removing it because it's used more often than `set_progress_bar_enabled` and has a dedicated example in the old docs)

Fix #3586


Missing google file for `multi_news` dataset

dataset-viewer

Related to the dataset viewer on huggingface.co

`datasets` 1.18.4 fixes the issue when you load the dataset with `load_dataset`.

When loading in streaming mode, the fix is indeed on https://github.com/huggingface/datasets/pull/3843 which will be merged soon :)

That is. The PR #3843 was just opened a bit later we had made our 1.18.4 patch release...
Once merged, that will fix this issue.

OK. Should fix the viewer for 50 datasets

<img width="148" alt="Capture d’écran 2022-03-14 à 11 51 02" src="https://user-images.githubusercontent.com/1676121/158157853-6c544a47-2d6d-4ac4-964a-6f10951ec36b.png">

## Dataset viewer issue for '*multi_news*'

**Link:** https://huggingface.co/datasets/multi_news

```
Server error

Status code:   400
Exception:     FileNotFoundError
Message:       https://drive.google.com/uc?export=download&id=1vRY2wM6rlOZrf9exGTm5pXj5ExlVwJ0C/multi-news-original/train.src
```

Am I the one who added this dataset ? No


Fix code examples indentation

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3895). All of your documentation changes will be reflected on that endpoint.

My last commit should have fixed it, I don't know why the dev doc build is not showing my last changes

Let me merge this and we can see on `master` how it renders, until the dev doc build is fixed

Some code examples are currently not rendered correctly. I think this is because they are over-indented

cc @mariosasko


[docs] make dummy data creation optional

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3894). All of your documentation changes will be reflected on that endpoint.

The dev doc build rendering doesn't seem to be updated with my last commit for some reason

Merging it anyway since I'd like to share this page with users 🙃

Related to #3507 : dummy data for datasets created on the Hugging Face Hub are optional.

We can discuss later to make them optional for datasets in this repository as well


Add default branch for doc building

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3893). All of your documentation changes will be reflected on that endpoint.

Yes! And when we discovered on the Transformers side that this check fails on the GitHub actions, we added a config attribute to have a default. Setting in Transformers fixed the issue of the doc being deployed to main, so porting the fix here too :-)

Since other libraries use `main` as their default branch and it's now the standard default, you have to specify a different name in the doc config if you're using `master` like datasets (`doc-builder` tries to guess it, but in the job, we have weird checkout of merge commits so it doesn't always manage to get it right).

This PR makes sure it will always use master for the dev doc (until you decide to switchto main)


Fix CLI test checksums

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3892). All of your documentation changes will be reflected on that endpoint.

Feel free to merge if it's good for you :)

I've added a test @lhoestq. Once all green, I'll merge.

Last failing tests do not have nothing to do with this PR.

Previous PR:
- #3796

introduced a side effect: `datasets-cli test` generates `dataset_infos.json` with `None` checksum values.

See:
- #3805

This PR introduces a way for `datasets-cli test` to force to record infos, even if `verify_infos=False`

Close #3848.

CC: @craffel


Fix race condition in doc build

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3891). All of your documentation changes will be reflected on that endpoint.

Following https://github.com/huggingface/datasets/runs/5499386744 it seems that race conditions that create issues when updating the doc. I took the same approach as in `transformers` to fix race conditions


Update beans download urls

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3890). All of your documentation changes will be reflected on that endpoint.

@albertvillanova Thanks for investigating and fixing that issue. I regenerated the `dataset_infos.json` file.

Replace the old URLs with the Hub [URLs](https://huggingface.co/datasets/beans/tree/main/data).

Also reported by @stevhliu.

Fix #3889


Cannot load beans dataset (Couldn't reach the dataset)

dataset bug

A bug in a dataset script provided in the library

Hi ! A pull request is open to fix the dataset, we'll release a patch soon with a new release of `datasets` :)

## Describe the bug
The beans dataset is unavailable to download.

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset('beans')
```

## Expected results
The dataset would be downloaded with no issue.

## Actual results

```
ConnectionError: Couldn't reach https://storage.googleapis.com/ibeans/train.zip (error 403)
```

[It looks like the billing of this project has been disabled because it is associated with a delinquent account.](https://storage.googleapis.com/ibeans/train.zip )

## Environment info
Google Colab


IterableDataset columns and feature types

generic discussion

Generic discussion on the library

Right now, an IterableDataset (e.g. when streaming a dataset) doesn't require to know the list of columns it contains, nor their types: `my_iterable_dataset.features` may be `None`

However it's often interesting to know the column types and types. This helps knowing what's inside your dataset without having to manually check a few examples, and this is useful to prepare a processing pipeline or to train models.

Here are a few cases that lead to `features` being `None`:

1. when loading a dataset with `load_dataset` on CSV, JSON Lines, etc. files: type inference is only done when iterating over the dataset
2. when calling `map`, because we don't know in advance what's the output of the user's function passed to `map`
3. when calling `rename_columns`, `remove_columns`, etc. because they rely on `map`

Things we can consider, for each point above:

1.a infer the type automatically from the first samples on the dataset using prefetching, when the dataset builder doesn't provide the `features`
2.a allow the user to specify the `features` as an argument to `map` (this would be consistent with the non-streaming API)
2.b prefetch the first output value to infer the type
3.a don't rely on `map` directly and reuse the previous `features` and rename/remove the corresponding ones

The thing is that prefetching can take a few seconds, while the operations above are instantaneous since no data are downloaded. Therefore I'm not sure whether this solution may be worth it. Maybe prefetching could also be done when explicitly asked by the user

cc @mariosasko @albertvillanova


ImageFolder improvements

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3887). All of your documentation changes will be reflected on that endpoint.

This PR adds the following improvements to the `imagefolder` dataset:
* skip the extract step for image files (as discussed in https://github.com/huggingface/datasets/pull/2830#discussion_r816817919)
* option to drop labels by setting `drop_labels=True` (useful for image pretraining cc @NielsRogge). This is faster than loading a dataset and removing the `label` column because we don't need to iterate over the files to infer class labels.


Retry HfApi call inside push_to_hub when 504 error

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3886). All of your documentation changes will be reflected on that endpoint.

I made it more robust by increasing the wait time, and I also added some logs when a request is retried. Let me know if it's ok for you

At the end you did not set the agreed max value of 60s.

Moreover, with the new numbers, there is a slight contradiction: although you set max_retries=5, we will only make 4 retries at most because of the combined values of `base_wait_time` and `max_wait_time`.

Yea I thought that in total we could wait 1min, but if we have a max_wait_time of 20sec between each request it's fine IMO

> Moreover, with the new numbers, there is a slight contradiction: although you set max_retries=5, we will only make 4 retries at most because of the combined values of base_wait_time and max_wait_time.

What makes you think this ? If the exponential wait time becomes bigger than `max_wait_time` then it still does the retry, but after a wait time of `max_wait_time`

Sorry, I meant 4 retries **with exponential backoff**; the fifth one is with constant backoff.

OK, and one question: do you think that the retries do not affect the time the server needs to be operational again and able to process the request? I guess that if does not affect, then the cause are other users' requests, or others; not our specific request.

Just to be sure:
- Then 20s at most between consecutive requests do not impact the server.
- And we expect after a total of 5 retries (within a total 50s of wait time + request processing/uploading time), the server should be able to come back to normality.

> do you think that the retries do not affect the time the server needs for being able to process the request (I guess in this case the cause are other users' requests, or other causes; not our specific request).

Yes I don't think the retries would affect the server, I think the cause of the 504 errors is elsewhere

> Just to be sure:
>
> Then 20s at most between consecutive requests do not impact the server.
> And we expect after a total of 5 retries (within a total 50s of wait time + request processing/uploading time), the server should be able to come back to normality.

Yes I think it's fine for now, we can still adapt this later if needed

Will be curious to see the impact of this in terms of upload reliability! Don't forget to let us know when you have more data. cc @huggingface/moon-landing-back

Ass suggested by @lhoestq in #3872, this PR:
- Implements a retry function
- Retries HfApi call inside `push_to_hub` when 504 error. To be agreed:
  - max_retries = 2 (at 0.5 and 1 seconds)

Fix #3872.


Fix some shuffle docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3885). All of your documentation changes will be reflected on that endpoint.

Following #3842 some docs were still outdated (with `buffer_size` as the first argument)


Fix bug in METEOR metric due to nltk version

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3884). All of your documentation changes will be reflected on that endpoint.

Fix #3883.


The metric Meteor doesn't work for nltk ==3.6.4

bug

Something isn't working

Hi @zhaowei-wang98, thanks for reporting.

We are fixing it...

## Describe the bug
Using the metric Meteor with nltk == 3.6.4 gives a TypeError:
TypeError: descriptor 'lower' for 'str' objects doesn't apply to a 'list' object

## Steps to reproduce the bug
```python
import datasets
metric = datasets.load_metric("meteor")
predictions = ["hello world"]
references = ["hello world"]
metric.compute(predictions=predictions, references=references)
```

## Expected results
TypeError: descriptor 'lower' for 'str' objects doesn't apply to a 'list' object

I think this TypeError exists because input sentences are tokenized into lists of tokens and the str.lower() is applied to this list of tokens.

## Actual results
No error but a meteor score

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: linux
- Python version: 3.8.12
- PyArrow version: 7.0.0


Image process doc

documentation

Improvements or additions to documentation

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3882). All of your documentation changes will be reflected on that endpoint.

This PR is a first draft of how to process image data. It adds:

- Load an image dataset with `image` and `path` (adds tip about `decode=False` param to access the path and bytes, thanks to @mariosasko).
- Load an image using the `ImageFolder` builder. I know there is an [example](https://huggingface.co/docs/datasets/master/en/loading#image-folders) of this already, but I also wanted to add it here so users don't miss it. This doc seems important for centralizing all of the image-related things so far. Datasets has grown so quickly 🚀  now that I think maybe splitting up the How-to guides by modality may be better since working with vision/audio data is slightly different from what users have seen up until now. This way we can continue to scale the docs to better accommodate vision/audio things.
- Add a data augmentation with `set_transform`. There is only 1 example here so far, but we can certainly add more.

Todo:
- [x] Couldn't figure out why my augmentation function works with `set_transform` but not `map` 🥲. Working with @mariosasko on this!


How to use Image folder

question

Further information is requested

Even this from docs throw same error
```
dataset = load_dataset("imagefolder", data_files="https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip", split="train")

```

Hi @INF800,

Please note that the `imagefolder` feature enhancement was just recently merged to our master branch (https://github.com/huggingface/datasets/commit/207be676bffe9d164740a41a883af6125edef135), but has not yet been released.

We are planning to make the 2.0 release of our library in the coming days and then that feature will be available by updating your `datasets` library from PyPI.

In the meantime, you can incorporate that feature if you install our library from our GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```

Then:
```python
In [1]: from datasets import load_dataset
        ds = load_dataset("imagefolder", data_files="https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip", split="train")
Using custom data configuration default-7eb4e80d960deb18
Downloading and preparing dataset image_folder/default to .../.cache/huggingface/datasets/image_folder/default-7eb4e80d960deb18/0.0.0/8de8dc6d68ce3c81cc102b93cc82ede27162b5d30cd003094f935942c8294f60...
Downloading data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 690.19it/s]
Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 852.85it/s]
Dataset image_folder downloaded and prepared to .../.cache/huggingface/datasets/image_folder/default-7eb4e80d960deb18/0.0.0/8de8dc6d68ce3c81cc102b93cc82ede27162b5d30cd003094f935942c8294f60. Subsequent calls will reuse this data.

In [2]: ds
Out[2]:
Dataset({
    features: ['image', 'label'],
    num_rows: 25000
})
```

Hey @albertvillanova. Does this load entire dataset in memory? Because I am facing huge trouble with loading very big datasets (OOM errors)

Can you provide the error stack trace? The loader only stores the `data_files` dict, which can get big after globbing. Then, the OOM error would mean you don't have enough memory to keep all the paths to the image files. You can circumvent this by generating an archive and loading the dataset from there. Maybe we can optimize the globbing part in our data files resolution at some point, cc @lhoestq for visibility.

Hey, memory error is resolved. It was fluke.

But there is another issue. Currently `load_dataset("imagefolder", data_dir="./path/to/train",)` takes only `train` as arg to `split` parameter.

I am creating vaildation dataset using

```
ds_valid = datasets.DatasetDict(valid=load_dataset("imagefolder", data_dir="./path/to/valid",)['train'])
```

`data_dir="path/to/folder"` is a shorthand syntax fox `data_files={"train": "path/to/folder/**"}`, so use `data_files` in that case instead:
```python
ds = load_dataset("imagefolder", data_files={"train": "path/to/train/**", "test": "path/to/test/**", "valid": "path/to/valid/**"})
```

And there was another issue. I loaded black and white images (jpeg file). Using load dataset. It reads it as PIL jpeg data format. But instead of converting it into 3 channel tensor, input to collator function is coming as a single channel tensor.

We don't apply any additional preprocessing on top of `PIL.Image.open(image_file)`, so you need to do the conversion yourself:

```python
def to_rgb(batch):
    batch["image"] = [img.convert("RGB") for img in batch["image"]]
    return batch

ds_rgb = ds.map(to_rgb, batched=True)
```

Please use our Forum for questions of this kind in the future.

Ran this code
```
 load_dataset("imagefolder", data_dir="./my-dataset")
```

`https://raw.githubusercontent.com/huggingface/datasets/master/datasets/imagefolder/imagefolder.py` missing
```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/tmp/ipykernel_33/1648737256.py in <module>
----> 1 load_dataset("imagefolder", data_dir="./my-dataset")

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1684         revision=revision,
   1685         use_auth_token=use_auth_token,
-> 1686         **config_kwargs,
   1687     )
   1688

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, script_version, **config_kwargs)
   1511         download_config.use_auth_token = use_auth_token
   1512     dataset_module = dataset_module_factory(
-> 1513         path, revision=revision, download_config=download_config, download_mode=download_mode, data_files=data_files
   1514     )
   1515

/opt/conda/lib/python3.7/site-packages/datasets/load.py in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_files, **download_kwargs)
   1200                         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1201                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
-> 1202                     ) from None
   1203                 raise e1 from None
   1204     else:

FileNotFoundError: Couldn't find a dataset script at /kaggle/working/imagefolder/imagefolder.py or any data file in the same directory. Couldn't find 'imagefolder' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/imagefolder/imagefolder.py
```


Change the framework switches to the new syntax

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3880). All of your documentation changes will be reflected on that endpoint.

This PR updates the syntax of the framework-specific code samples. With this new syntax, you'll be able to:

- have paragraphs of text be framework-specific instead of just code samples
- have support for Flax code samples if you want.

This should be merged after https://github.com/huggingface/doc-builder/pull/63 and https://github.com/huggingface/doc-builder/pull/130


SQuAD v2 metric: create README.md

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3879). All of your documentation changes will be reflected on that endpoint.

Proposing SQuAD v2 metric card


Update cats_vs_dogs size

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3878). All of your documentation changes will be reflected on that endpoint.

Maybe `NonMatchingSplitsSizesError` errors should also tell the user to try using a more recent version of the dataset to get the fixes ?

@lhoestq Good idea. Will open a new PR to improve the error messages of NonMatchingSplitsSizesError, NonMatchingChecksumsError, ...

It seems like 12 new examples have been added to the `cats_vs_dogs`. This PR updates the size in the card and the info file to avoid a verification error (reported by @stevhliu).


Align metadata to DCAT/DCAT-AP

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Align to DCAT metadata to describe datasets

**Describe the solution you'd like**
Reuse terms and structure from DCAT in the metadata file, ideally generate a json-ld file dcat compliant

**Describe alternatives you've considered**

**Additional context**
DCAT is a W3C standard extended in Europe with DCAT-AP, an example is data.europa.eu publishing datasets metadata in DCAT-AP


Fix download_mode in dataset_module_factory

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3876). All of your documentation changes will be reflected on that endpoint.

Fix `download_mode` value set in `dataset_module_factory`.

Before the fix, it was set to `bool` (default to `False`).

Also set properly its default value in all public functions.


Module namespace cleanup for v2.0

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3875). All of your documentation changes will be reflected on that endpoint.

@severo No, this PR doesn't fix that issue in the current state. We can fix it by adding `__all__` to `datasets/__init__.py` and `datasets/formatting/__init__.py`. However, this would require updating `__all__` for each new function/class definition, which could become cumbersome, and we can't do this dynamically because `mypy` is a static type checker.

@lhoestq @albertvillanova WDYT?

Feel free to merge this one if it's good for you :)

This is an attempt to make the user-facing `datasets`' submodule namespace cleaner:

In particular, this PR does the following:
* removes the unused `zip_nested` and `flatten_nest_dict` and their accompanying tests
* removes `pyarrow` from the top-level namespace
* properly uses `__all__` and the `from  <module> import *` syntax to avoid importing the `<module>`'s submodules
* cleans up the `utils` namespace
* moves the `temp_seed` context manage from `datasets/utils/file_utils.py` to `datasets/utils/py_utils.py`


add MSE and MAE metrics - V2

@mariosasko New PR here. I'm not sure how to add you as a co-author here.  Also I see flake8 tests are failing, any inputs on how to resolve this ?
Also, let me know if any other changes are required. Thank you.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3874). All of your documentation changes will be reflected on that endpoint.

Created a new pull request to resolve unrelated changes in PR caused due to rebasing.
Ref Older PR :  [#3845](https://github.com/huggingface/datasets/pull/3845)

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)


Create SQuAD metric README.md

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3873). All of your documentation changes will be reflected on that endpoint.

Oh one last thing I almost forgot, I think I would add a section "Examples" with examples of inputs and outputs and in particular: an example giving maximal values, an examples giving minimal values and maybe a standard examples from SQuAD. What do you think?

Proposal for a metrics card structure (with an example based on the SQuAD metric).
@thomwolf @lhoestq @douwekiela @lewtun -- feel free to comment on structure or content (it's an initial draft, so I realize there's stuff missing!).


HTTP error 504 Server Error: Gateway Time-out

is pushing directly with git (and git-lfs) an option for you?

I have installed git-lfs and doing this push with that

yes but is there any way you could try pushing with `git` command line directly instead of `push_to_hub`?

Okay. I didnt saved the dataset to my local machine. So, I processed the dataset and pushed it directly to the hub. I think I should try saving those dataset to my local machine by `save_to_disk` and then push it with git command line

cc @lhoestq @albertvillanova @LysandreJik because maybe I'm giving dumb advice here 😅

`push_to_hub` is the preferred way of uploading a dataset to the Hub, which can then be reloaded with `load_dataset`. Feel free to try again and see if the server is working as expected now. Maybe we can add a retry mechanism in the meantime to workaround 504 errors.

Regarding `save_to_disk`, this must only be used for local serialization (because it's uncompressed and compatible with memory-mapping). If you upload a dataset saved with `save_to_disk` to the Hub, then to reload it you will have to download/clone the repository locally by yourself and use `load_from_disk`.

I am trying to push a large dataset(450000+) records with the help of `push_to_hub()`
While pushing, it gives some error like this.

```
Traceback (most recent call last):
  File "data_split_speech.py", line 159, in <module>
    data_new_2.push_to_hub("user-name/dataset-name",private=True)
  File "/opt/conda/lib/python3.8/site-packages/datasets/dataset_dict.py", line 951, in push_to_hub
    repo_id, split, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub(
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3556, in _push_parquet_shards_to_hub
    api.upload_file(
  File "/opt/conda/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1017, in upload_file
    raise err
  File "/opt/conda/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1008, in upload_file
    r.raise_for_status()
  File "/opt/conda/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/user-name/dataset-name/upload/main/data/train2-00041-of-00064.parquet

```
Can anyone help me to resolve this issue.


add pandas to env command

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3871). All of your documentation changes will be reflected on that endpoint.

Think failures are unrelated - feel free to merge whenever you want :-)

Pandas is a required packages and used quite a bit. I don't see any downside with adding its version to the `datasets-cli env` command.


Add wikitablequestions dataset

@lhoestq Would you mind reviewing it when you're available? Thanks!

> Awesome thanks for adding this dataset ! :) The dataset script and dataset cards look pretty good
>
> It looks like your `dummy_data.zip` files are quite big though (>1MB each), do you think we can reduce their sizes ? This way this git repository doesn't become too big

I have manually reduced the `dummy_data.zip` and its current size is about 54KB. Hope it is fine for you!

@lhoestq I think the dataset is ready to merge now. Any follow-up question is welcome :-D

> Thanks ! It looks all good now :)

Awesome! Thanks for your quick response!


Making the Hub the place for datasets in Portuguese

dataset request

Requesting to add a new dataset

Hi @omarespejel! I think the philosophy for `datasets` issues is to create concrete issues with proposals to add a specific, individual dataset rather than umbrella issues for things such as datasets for a language, since we could end up with hundreds of issues (one per language). I see NILC - USP has many datasets, I would suggest to either create an issue for their datasets, or even better, we are trying to push to upload datasets as community datasets instead of adding them to the core library as guided in https://huggingface.co/docs/datasets/share. That would have the additional benefit that the dataset would live under the NILC organization.

@lhoestq correct me if I'm wrong please 😄

Let's make Hugging Face Datasets the central hub for datasets in Portuguese :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the Portuguese speaking community.

What are some datasets in Portuguese worth integrating into the Hugging Face hub?

Special thanks to @augusnunes for his collaboration on identifying the first ones:

-  [NILC - USP](http://www.nilc.icmc.usp.br/nilc/index.php/tools-and-resources).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

cc @osanseviero


Ignore duplicate keys if `ignore_verifications=True`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3868). All of your documentation changes will be reflected on that endpoint.

Cool thanks ! Could you add a test please ?

Currently, it's impossible to generate a dataset if some keys from `_generate_examples` are duplicated. This PR allows skipping the check for duplicate keys if `ignore_verifications` is set to `True`.


Update for the rename doc-builder -> hf-doc-utils

~~@julien-c there was a vote 🙂 https://huggingface.slack.com/archives/C021H1P1HKR/p1646405136644739~~

oh I see you already commeented in the thread as well

Thanks ! It looks all good to me (provided `hf-doc-utils` is the name we keep in the end). I'm fine with this name, and `hf-doc-builder` is also fine IMHO

ok, this is definitely not a hill I'll die on =) @mishig25 @sgugger

This PR adapts the job to the upcoming change of name of `doc-builder`.


Bring back imgs so that forsk dont get broken

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3866). All of your documentation changes will be reflected on that endpoint.

I think we just need to keep `datasets_logo_name.jpg` and `course_banner.png` because they appear in the README.md of the forks of `datasets`. The other images can be removed


Add logo img

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3865). All of your documentation changes will be reflected on that endpoint.


Update image dataset tags

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3864). All of your documentation changes will be reflected on that endpoint.

Align the existing image datasets' tags with new tags introduced in #3800.


Update code blocks

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3863). All of your documentation changes will be reflected on that endpoint.

Following https://github.com/huggingface/datasets/pull/3860#issuecomment-1061756712 and https://github.com/huggingface/datasets/pull/3690 we need to update the code blocks to use markdown instead of sphinx


Manipulate columns on IterableDataset (rename columns, cast, etc.)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3862). All of your documentation changes will be reflected on that endpoint.

> IIUC we check if columns are present/not present directly in the yielded examples and not in info.features because info.features can be None (after map, for instance)?

Yes exactly

> We should develop a solution that ensures info.features is never None. For example, one approach would be to infer them from examples in map and make them promotable from Value("null") to a specific type, in case of None values.

I agree this would be useful. Though inferring the type requires to start streaming some data, which takes a few seconds (compared to being instantaneous right now).

Let's discuss this in a new issue maybe ?

I added:
- add_column
- cast
- rename_column
- rename_columns

related to https://github.com/huggingface/datasets/issues/3444

TODO:
- [x] docs
- [x] tests


big_patent cased version

Requesting to add a new dataset

Hi! I am interested in working with the big_patent dataset.

In Tensorflow, there are a number of versions of the dataset:

- 1.0.0 : lower cased tokenized words
- 2.0.0 : Update to use cased raw strings
- 2.1.2 (default): Fix update to cased raw strings.

The version in the huggingface `datasets` library is the 1.0.0. I would be very interested in using the 2.1.2 cased version (used more, recently, for example in the Pegasus paper), but it does not seem to be supported (I tried using the `revision` parameter in `load_datasets`). Is there a way to already load it, or would it be possible to add that version?


Small doc fixes

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3860). All of your documentation changes will be reflected on that endpoint.

There are still some `.. code-block:: python` (e.g. see [this](https://huggingface.co/docs/datasets/master/en/package_reference/main_classes#datasets.Dataset.align_labels_with_mapping)) directives in our codebase, so maybe we can remove those as well as part of this PR.


Unable to dowload big_patent (FileNotFoundError)

bug

Something isn't working

duplicate

This issue or pull request already exists

Hi @slvcsl, thanks for reporting.

Yesterday we just made a patch release of our `datasets` library that fixes this issue: version 1.18.4.
https://pypi.org/project/datasets/#history

Please, feel free to update `datasets` library to the latest version:
```shell
pip install -U datasets
```
And then you should force redownload of the data file to update your local cache:
```python
ds = load_dataset("big_patent", "g", split="validation", download_mode="force_redownload")
```
- Note that before the fix, you just downloaded and cached the Google Drive virus scan warning page, instead of the data file

This issue was already reported
- #3784

and its root cause is a change in the Google Drive service. See:
- #3786

We already fixed it. See:
- #3787

## Describe the bug
I am trying to download some splits of the big_patent dataset, using the following code:

`ds = load_dataset("big_patent", "g", split="validation", download_mode="force_redownload")
`
However, this leads to a FileNotFoundError.

FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-3-8d8a745706a9>](https://localhost:8080/#) in <module>()
      1 from datasets import load_dataset
----> 2 ds = load_dataset("big_patent", "g", split="validation", download_mode="force_redownload")

8 frames
[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1705         ignore_verifications=ignore_verifications,
   1706         try_from_hf_gcs=try_from_hf_gcs,
-> 1707         use_auth_token=use_auth_token,
   1708     )
   1709

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    593                     if not downloaded_from_gcs:
    594                         self._download_and_prepare(
--> 595                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    596                         )
    597                     # Sync info

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    659         split_dict = SplitDict(dataset_name=self.name)
    660         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 661         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    662
    663         # Checksums verification

[/root/.cache/huggingface/modules/datasets_modules/datasets/big_patent/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c/big_patent.py](https://localhost:8080/#) in _split_generators(self, dl_manager)
    123         split_types = ["train", "val", "test"]
    124         extract_paths = dl_manager.extract(
--> 125             {k: os.path.join(dl_path, "bigPatentData", k + ".tar.gz") for k in split_types}
    126         )
    127         extract_paths = {k: os.path.join(extract_paths[k], k) for k in split_types}

[/usr/local/lib/python3.7/dist-packages/datasets/utils/download_manager.py](https://localhost:8080/#) in extract(self, path_or_paths, num_proc)
    282         download_config.extract_compressed_file = True
    283         extracted_paths = map_nested(
--> 284             partial(cached_path, download_config=download_config), path_or_paths, num_proc=num_proc, disable_tqdm=False
    285         )
    286         path_or_paths = NestedDataStructure(path_or_paths)

[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types, disable_tqdm)
    260         mapped = [
    261             _single_map_nested((function, obj, types, None, True))
--> 262             for obj in utils.tqdm(iterable, disable=disable_tqdm)
    263         ]
    264     else:

[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in <listcomp>(.0)
    260         mapped = [
    261             _single_map_nested((function, obj, types, None, True))
--> 262             for obj in utils.tqdm(iterable, disable=disable_tqdm)
    263         ]
    264     else:

[/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py](https://localhost:8080/#) in _single_map_nested(args)
    194     # Singleton first to spare some computation
    195     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 196         return function(data_struct)
    197
    198     # Reduce logging to keep things readable in multiprocessing with tqdm

[/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py](https://localhost:8080/#) in cached_path(url_or_filename, download_config, **download_kwargs)
    314     elif is_local_path(url_or_filename):
    315         # File, but it doesn't exist.
--> 316         raise FileNotFoundError(f"Local file {url_or_filename} doesn't exist")
    317     else:
    318         # Something unknown

FileNotFoundError: Local file /root/.cache/huggingface/datasets/downloads/extracted/ad068abb3e11f9f2f5440b62e37eb2b03ee515df9de1637c55cd1793b68668b2/bigPatentData/train.tar.gz doesn't exist

I have tried this in a number of machines, including on Colab, so I think this is not environment dependent.

How do I load the bigPatent dataset?


Udpate index.mdx margins

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3858). All of your documentation changes will be reflected on that endpoint.


Order of dataset changes due to glob.glob.

generic discussion

Generic discussion on the library

I agree using `glob.glob` alone is bad practice because it's not deterministic. Using `sorted` is a nice solution.

Note that the `xglob` function you are referring to in the `streaming_download_manager.py` code just extends `glob.glob` for URLs - we don't change its behavior. That's why it has no `sorted()`

## Describe the bug

After discussion with @lhoestq, just want to mention here that `glob.glob(...)` should always be used in combination with `sorted(...)` to make sure the list of files returned by `glob.glob(...)` doesn't change depending on the OS system.

There are currently multiple datasets that use `glob.glob()` without making use of `sorted(...)` even the streaming download manager (if I'm not mistaken):
https://github.com/huggingface/datasets/blob/c14bfeb4af89da14f870de5ddaa584b08aa08eeb/src/datasets/utils/streaming_download_manager.py#L483


Fix push_to_hub with null images

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3856). All of your documentation changes will be reflected on that endpoint.

This code currently raises an error because of the null image:
```python
import datasets

dataset_dict = { 'name': ['image001.jpg', 'image002.jpg'], 'image': ['cat.jpg', None] }
features = datasets.Features({
    'name': datasets.Value('string'),
    'image': datasets.Image(),
})
dataset = datasets.Dataset.from_dict(dataset_dict, features)
dataset.push_to_hub("username/dataset") # this line produces an error: 'NoneType' object is not subscriptable
```
I fixed this in this PR

TODO:
- [x] add a test


Bad error message when loading private dataset

bug

Something isn't working

We raise the error “ FileNotFoundError: can’t find the dataset” mainly to follow best practice in security (otherwise users could be able to guess what private repositories users/orgs may have)

We can indeed reformulate this and add the "If this is a private repository,..." part !

## Describe the bug

A pretty common behavior of an interaction between the Hub and datasets is the following.
An organization adds a dataset in private mode and wants to load it afterward.

```python
from transformers import load_dataset

ds = load_dataset("NewT5/dummy_data", "dummy")
```

This command then fails with:

```bash
FileNotFoundError: Couldn't find a dataset script at /home/patrick/NewT5/dummy_data/dummy_data.py or any data file in the same directory. Couldn't find 'NewT5/dummy_data' on the Hugging Face Hub either: FileNotFoundError: Dataset 'NewT5/dummy_data' doesn't exist on the Hub
```

**even though** the user has access to the website `NewT5/dummy_data` since she/he is part of the org.

We need to improve the error message here similar to how @sgugger, @LysandreJik and @julien-c have done it for transformers IMO.

## Steps to reproduce the bug

E.g. execute the following code to see the different error messages between `transformes` and `datasets`.

1. Transformers
```python
from transformers import BertModel

BertModel.from_pretrained("NewT5/dummy_model")
```

The error message is clearer here - it gives:

```
OSError: patrickvonplaten/gpt2-xl is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.
```

Let's maybe do the same for datasets? The PR was introduced to `transformers` here:
https://github.com/huggingface/transformers/pull/15261

## Expected results

Better error message

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4.dev0
- Platform: Linux-5.15.15-76051515-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyArrow version: 6.0.1


load only England English dataset from common voice english dataset

question

Further information is requested

Hi @amanjaiswal777,

First note that the dataset you are trying to load is deprecated: it was the Common Voice dataset release as of Dec 2020.

Currently, Common Voice dataset releases are directly hosted on the Hub, under the Mozilla Foundation organization: https://huggingface.co/mozilla-foundation

For example, to get their latest Common Voice relase (8.0):
- Go to the dataset page and request access permission (Mozilla Foundation requires this for people willing to use their datasets): https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
- Looking at the dataset card, you can check that data instances have, among other fields, the ones you are interested in: "accent", "age",...
- Then you can load their "en" language dataset as usual, besides passing your authentication token (more info on auth token here: https://huggingface.co/docs/hub/security)
  ```python
  from datasets import load_dataset
  ds_en = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token=True)
  ```
- Finally, you can filter only the data instances you are interested in (more info on `filter` here: https://huggingface.co/docs/datasets/process#select-and-filter):
  ```python
  ds_england_en = ds_en.filter(lambda item: item["accent"] == "England English")
  ```

Feel free to reopen this issue if you need further assistance.

training_data = load_dataset("common_voice", "en",split='train[:250]+validation[:250]')
testing_data = load_dataset("common_voice", "en", split="test[:200]")

I'm trying to load only 8% of the English common voice data with accent == "England English." Can somebody assist me with this?

**Typical Voice Accent Proportions:**

- 24% United States English
- 8% England English
- 5% India and South Asia (India, Pakistan, Sri Lanka)
- 3% Australian English
- 3% Canadian English
- 2% Scottish English
- 1% Irish English
- 1% Southern African (South Africa, Zimbabwe, Namibia)
- 1% New Zealand English

Can we replicate this for Age as well?

**Age proportions of the common voice:-**

- 24% 19 - 29
- 14% 30 - 39
- 10% 40 - 49
- 6% < 19
- 4% 50 - 59
- 4% 60 - 69
- 1% 70 – 79


add ontonotes_conll dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3853). All of your documentation changes will be reflected on that endpoint.

The CI fail is unrelated to this dataset, merging :)

# Introduction of the dataset
OntoNotes v5.0 is the final version of OntoNotes corpus, and is a large-scale, multi-genre,
multilingual corpus manually annotated with syntactic, semantic and discourse information.

This dataset is the version of OntoNotes v5.0 extended and used in the CoNLL-2012 shared task
, includes v4 train/dev and v9 test data for English/Chinese/Arabic and corrected version v12 train/dev/test data (English only).

This dataset is widely used in name entity recognition, coreference resolution, and semantic role labeling.

In dataset loading script, I modify and use the code of [AllenNLP/Ontonotes](https://docs.allennlp.org/models/main/models/common/ontonotes/#ontonotes) to read the special conll files while don't get extra package dependency.

# Some workarounds I did

1. task ids
I add tasks that I can't find anywhere `semantic-role-labeling`, `lemmatization`, and `word-sense-disambiguation` to the task category `structure-prediction`, because they are related to "syntax". I feel there is another good name for the task category since some tasks mentioned aren't related to structure, but I have no good idea.

2. `dl_manage.extract`
Since we'll get another zip after unzip the downloaded zip data, I have to use `dl_manager.extract` directly inside `_generate_examples`. But when testing dummy data, `dl_manager.extract` do nothing. So I make a conditional such that it  manually  extract data when testing dummy data.

# Help
Don't know how to fix the building doc error.


Redundant add dataset information and dead link.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3852). All of your documentation changes will be reflected on that endpoint.

> Alternatively, you can follow the steps to [add a dataset](https://huggingface.co/docs/datasets/add_dataset.html) and [share a dataset](https://huggingface.co/docs/datasets/share_dataset.html) in the documentation.

The "add a dataset link" gives 404 Error, and the share_dataset link has changed. I feel this information is redundant/deprecated now since we have a more detailed guide for "How to add a dataset?".


Load audio dataset error

bug

Something isn't working

Hi @lemoner20, thanks for reporting.

I'm sorry but I cannot reproduce your problem:
```python
In [1]: from datasets import load_dataset, load_metric, Audio
   ...: raw_datasets = load_dataset("superb", "ks", split="train")
   ...: print(raw_datasets[0]["audio"])
Downloading builder script: 30.2kB [00:00, 13.0MB/s]
Downloading metadata: 38.0kB [00:00, 16.6MB/s]
Downloading and preparing dataset superb/ks (download: 1.45 GiB, generated: 9.64 MiB, post-processed: Unknown size, total: 1.46 GiB) to .../.cache/huggingface/datasets/superb/ks/1.9.0/fc1f59e1fa54262dfb42de99c326a806ef7de1263ece177b59359a1a3354a9c9...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.49G/1.49G [00:37<00:00, 39.3MB/s]
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 71.3M/71.3M [00:01<00:00, 36.1MB/s]
Downloading data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:41<00:00, 20.67s/it]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:28<00:00, 14.24s/it]
Dataset superb downloaded and prepared to .../.cache/huggingface/datasets/superb/ks/1.9.0/fc1f59e1fa54262dfb42de99c326a806ef7de1263ece177b59359a1a3354a9c9. Subsequent calls will reuse this data.
{'path': '.../.cache/huggingface/datasets/downloads/extracted/8571921d3088b48f58f75b2e514815033e1ffbd06aa63fd4603691ac9f1c119f/_background_noise_/doing_the_dishes.wav', 'array': array([ 0.        ,  0.        ,  0.        , ..., -0.00592041,
       -0.00405884, -0.00253296], dtype=float32), 'sampling_rate': 16000}
```

Which version of `datasets` are you using? Could you please fill in the environment info requested in the bug report template? You can run the command `datasets-cli env` and copy-and-paste its output below
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:

@albertvillanova Thanks for your reply. The environment info below

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-4.19.91-007.ali4000.alios7.x86_64-x86_64-with-debian-buster-sid
- Python version: 3.6.12
- PyArrow version: 6.0.1

Thanks @lemoner20,

I cannot reproduce your issue in datasets version 1.18.3 either.

Maybe redownloading the data file may work if you had already cached this dataset previously. Could you please try passing "force_redownload"?
```python
raw_datasets = load_dataset("superb", "ks", split="train", download_mode="force_redownload")

Thanks, @albertvillanova,

I install the python package of **librosa=0.9.1** again, it works now!

## Load audio dataset error

Hi, when I load audio dataset following https://huggingface.co/docs/datasets/audio_process and https://github.com/huggingface/datasets/tree/master/datasets/superb,
```
from datasets import load_dataset, load_metric, Audio
raw_datasets = load_dataset("superb", "ks", split="train")
print(raw_datasets[0]["audio"])
```
following errors occur
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-169-3f8253239fa0> in <module>
----> 1 raw_datasets[0]["audio"]

/usr/lib/python3.6/site-packages/datasets/arrow_dataset.py in __getitem__(self, key)
   1924         """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
   1925         return self._getitem(
-> 1926             key,
   1927         )
   1928

/usr/lib/python3.6/site-packages/datasets/arrow_dataset.py in _getitem(self, key, decoded, **kwargs)
   1909         pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
   1910         formatted_output = format_table(
-> 1911             pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
   1912         )
   1913         return formatted_output

/usr/lib/python3.6/site-packages/datasets/formatting/formatting.py in format_table(table, key, formatter, format_columns, output_all_columns)
    530     python_formatter = PythonFormatter(features=None)
    531     if format_columns is None:
--> 532         return formatter(pa_table, query_type=query_type)
    533     elif query_type == "column":
    534         if key in format_columns:

/usr/lib/python3.6/site-packages/datasets/formatting/formatting.py in __call__(self, pa_table, query_type)
    279     def __call__(self, pa_table: pa.Table, query_type: str) -> Union[RowFormat, ColumnFormat, BatchFormat]:
    280         if query_type == "row":
--> 281             return self.format_row(pa_table)
    282         elif query_type == "column":
    283             return self.format_column(pa_table)

/usr/lib/python3.6/site-packages/datasets/formatting/formatting.py in format_row(self, pa_table)
    310         row = self.python_arrow_extractor().extract_row(pa_table)
    311         if self.decoded:
--> 312             row = self.python_features_decoder.decode_row(row)
    313         return row
    314

/usr/lib/python3.6/site-packages/datasets/formatting/formatting.py in decode_row(self, row)
    219
    220     def decode_row(self, row: dict) -> dict:
--> 221         return self.features.decode_example(row) if self.features else row
    222
    223     def decode_column(self, column: list, column_name: str) -> list:

/usr/lib/python3.6/site-packages/datasets/features/features.py in decode_example(self, example)
   1320             else value
   1321             for column_name, (feature, value) in utils.zip_dict(
-> 1322                 {key: value for key, value in self.items() if key in example}, example
   1323             )
   1324         }

/usr/lib/python3.6/site-packages/datasets/features/features.py in <dictcomp>(.0)
   1319             if self._column_requires_decoding[column_name]
   1320             else value
-> 1321             for column_name, (feature, value) in utils.zip_dict(
   1322                 {key: value for key, value in self.items() if key in example}, example
   1323             )

/usr/lib/python3.6/site-packages/datasets/features/features.py in decode_nested_example(schema, obj)
   1053     # Object with special decoding:
   1054     elif isinstance(schema, (Audio, Image)):
-> 1055         return schema.decode_example(obj) if obj is not None else None
   1056     return obj
   1057

/usr/lib/python3.6/site-packages/datasets/features/audio.py in decode_example(self, value)
    100                 array, sampling_rate = self._decode_non_mp3_file_like(file)
    101             else:
--> 102                 array, sampling_rate = self._decode_non_mp3_path_like(path)
    103         return {"path": path, "array": array, "sampling_rate": sampling_rate}
    104

/usr/lib/python3.6/site-packages/datasets/features/audio.py in _decode_non_mp3_path_like(self, path)
    143
    144         with xopen(path, "rb") as f:
--> 145             array, sampling_rate = librosa.load(f, sr=self.sampling_rate, mono=self.mono)
    146         return array, sampling_rate
    147

/usr/lib/python3.6/site-packages/librosa/core/audio.py in load(path, sr, mono, offset, duration, dtype, res_type)
    110
    111     y = []
--> 112     with audioread.audio_open(os.path.realpath(path)) as input_file:
    113         sr_native = input_file.samplerate
    114         n_channels = input_file.channels

/usr/lib/python3.6/posixpath.py in realpath(filename)
    392     """Return the canonical path of the specified filename, eliminating any
    393 symbolic links encountered in the path."""
--> 394     filename = os.fspath(filename)
    395     path, ok = _joinrealpath(filename[:0], filename, {})
    396     return abspath(path)

TypeError: expected str, bytes or os.PathLike object, not _io.BufferedReader
```

## Expected results
```
>>> raw_datasets[0]["audio"]
{'array': array([-0.0005188 , -0.00109863,  0.00030518, ...,  0.01730347,
        0.01623535,  0.01724243]),
'path': '/root/.cache/huggingface/datasets/downloads/extracted/bb3a06b491a64aff422f307cd8116820b4f61d6f32fcadcfc554617e84383cb7/bed/026290a7_nohash_0.wav',
'sampling_rate': 16000}
```


[feat] Add tqdm arguments

In this PR, tqdm arguments can be passed to the map() function and such, in order to be more flexible.


Add "Adversarial GLUE" dataset to datasets library

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq can you review when you have some time?

Hi @lhoestq -- thanks so much for your review! I just added the stuff you requested to the README.md, including an example from the dataset, the table of contents, and lots of section headers with "More Information Needed" below. Let me know if there's anything else I need to do!

Feel free to also merge `master` into your branch to get the latest updates for the tests ;)

thanks @lhoestq - just made all the updates you requested!

Adds the Adversarial GLUE dataset: https://adversarialglue.github.io/

```python
>>> import datasets
>>> >>> datasets.load_dataset('adv_glue')
Using the latest cached version of the module from /home/jxm3/.cache/huggingface/modules/datasets_modules/datasets/adv_glue/26709a83facad2830d72d4419dd179c0be092f4ad3303ad0ebe815d0cdba5cb4 (last modified on Mon Mar  7 19:19:48 2022) since it couldn't be found locally at adv_glue., or remotely on the Hugging Face Hub.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jxm3/random/datasets/src/datasets/load.py", line 1657, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/jxm3/random/datasets/src/datasets/load.py", line 1510, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 1021, in __init__
    super().__init__(*args, **kwargs)
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 258, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 337, in _create_builder_config
    raise ValueError(
ValueError: Config name is missing.
Please pick one among the available configs: ['adv_sst2', 'adv_qqp', 'adv_mnli', 'adv_mnli_mismatched', 'adv_qnli', 'adv_rte']
Example of usage:
	`load_dataset('adv_glue', 'adv_sst2')`
>>> datasets.load_dataset('adv_glue', 'adv_sst2')['validation'][0]
Reusing dataset adv_glue (/home/jxm3/.cache/huggingface/datasets/adv_glue/adv_sst2/1.0.0/3719a903f606f2c96654d87b421bc01114c37084057cdccae65cd7bc24b10933)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 604.11it/s]
{'sentence': "it 's an uneven treat that bores fun at the democratic exercise while also examining its significance for those who take part .", 'label': 1, 'idx': 0}
```


NonMatchingChecksumError when checksum is None

Something isn't working

Hi @jxmorris12, thanks for reporting.

The objective of `verify_checksums` is to check that both checksums are equal. Therefore if one is None and the other is non-None, they are not equal, and the function accordingly raises a NonMatchingChecksumError. That behavior is expected.

The question is: how did you generate the expected checksum? Normally, it should not be None. To properly generate it (it is contained in the `dataset_infos.json` file), you should have runned: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md
```shell
datasets-cli test <your-dataset-folder> --save_infos --all_configs
```

On the other hand, you should take into account that the generation of this file is NOT mandatory for personal/community datasets (we only require it for "canonical" datasets, i.e., datasets added to our library GitHub repository: https://github.com/huggingface/datasets/tree/master/datasets). Therefore, other option would be just to delete the `dataset_infos.json` file. If that file is not present, the function `verify_checksums` is not executed.

Finally, you can circumvent the `verify_checksums` function by passing `ignore_verifications=True` to `load_dataset`:
```python
load_dataset(..., ignore_verifications=True)
```

Thanks @albertvillanova!

That's fine. I did run that command when I was adding a new dataset. Maybe because the command crashed in the middle, the checksum wasn't stored properly. I don't know where the bug is happening. But either (i) `verify_checksums` should properly handle this edge case, where the passed checksum is None or (ii) the `datasets-cli test` shouldn't generate a corrupted dataset_infos.json file.

Just a more high-level thing, I was trying to follow the instructions for adding a dataset in the CONTRIBUTING.md, so if running that command isn't even necessary, that should probably be mentioned in the document, right? But that's somewhat of a moot point, since something isn't working quite right internally if I was able to get into this corrupted state in the first place, just by following those instructions.

Hi @jxmorris12,

Definitely, your `dataset_infos.json` was corrupted (and wrongly contains expected None checksum).

While we further investigate how this can happen and fix it, feel free to delete your `dataset_infos.json` file and recreate it with:
```shell
datasets-cli test <your-dataset-folder> --save_infos --all_configs
```

Also note that `verify_checksum` is working as expected: if it receives a None and and a non-None checksums as input pair, it must raise an exception: they are not equal. That is not a bug.

At a higher level, also note that we are preparing the release of `datasets` version 2.0, and some docs are being updated...

In order to add a dataset, I think the most updated instructions are in our official documentation pages: https://huggingface.co/docs/datasets/share

Thanks for the info. Maybe you can update the contributing.md if it's not up-to-date.

Hi @jxmorris12, we have discovered the bug why `None` checksums wrongly appeared when generating the `dataset_infos.json` file:
- #3892

The fix will be accessible once this PR merged. And we are planning to do our 2.0 release today.

We are also working on updating all our docs for our release today.

Thanks @albertvillanova - congrats on the release!

I ran into the following error when adding a new dataset:

```bash
expected_checksums = {'https://adversarialglue.github.io/dataset/dev.zip': {'checksum': None, 'num_bytes': 40662}}
recorded_checksums = {'https://adversarialglue.github.io/dataset/dev.zip': {'checksum': 'efb4cbd3aa4a87bfaffc310ae951981cc0a36c6c71c6425dd74e5b55f2f325c9', 'num_bytes': 40662}}
verification_name = 'dataset source files'

    def verify_checksums(expected_checksums: Optional[dict], recorded_checksums: dict, verification_name=None):
        if expected_checksums is None:
            logger.info("Unable to verify checksums.")
            return
        if len(set(expected_checksums) - set(recorded_checksums)) > 0:
            raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
        if len(set(recorded_checksums) - set(expected_checksums)) > 0:
            raise UnexpectedDownloadedFile(str(set(recorded_checksums) - set(expected_checksums)))
        bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
        for_verification_name = " for " + verification_name if verification_name is not None else ""
        if len(bad_urls) > 0:
            error_msg = "Checksums didn't match" + for_verification_name + ":\n"
>           raise NonMatchingChecksumError(error_msg + str(bad_urls))
E           datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
E           ['https://adversarialglue.github.io/dataset/dev.zip']

src/datasets/utils/info_utils.py:40: NonMatchingChecksumError
```

## Expected results
The dataset downloads correctly, and there is no error.

## Actual results
Datasets library is looking for a checksum of None, and it gets a non-None checksum, and throws an error. This is clearly a bug.


Datasets' cache not re-used

Something isn't working

<s>I think this is because the tokenizer is stateful and because the order in which the splits are processed is not deterministic. Because of that, the hash of the tokenizer may change for certain splits, which causes issues with caching.

To fix this we can try making the order of the splits deterministic for map.</s>

Actually this is not because of the order of the splits, but most likely because the tokenizer used to process the second split is in a state that has been modified by the first split.

Therefore after reloading the first split from the cache, then the second split can't be reloaded since the tokenizer hasn't seen the first split (and therefore is considered a different tokenizer).

This is a bit trickier to fix, we can explore fixing this next week maybe

Sorry didn't have the bandwidth to take care of this yet - will re-assign when I'm diving into it again !

## Describe the bug
For most tokenizers I have tested (e.g. the RoBERTa tokenizer), the data preprocessing cache are not fully reused in the first few runs, although their `.arrow` cache files are in the cache directory.

## Steps to reproduce the bug
Here is a reproducer. The GPT2 tokenizer works perfectly with caching, but not the RoBERTa tokenizer in this example.
```python
from datasets import load_dataset
from transformers import AutoTokenizer

raw_datasets = load_dataset("wikitext", "wikitext-2-raw-v1")
# tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
text_column_name = "text"
column_names = raw_datasets["train"].column_names

def tokenize_function(examples):
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)

tokenized_datasets = raw_datasets.map(
    tokenize_function,
    batched=True,
    remove_columns=column_names,
    load_from_cache_file=True,
    desc="Running tokenizer on every text in dataset",
)
```

## Expected results
No tokenization would be required after the 1st run. Everything should be loaded from the cache.

## Actual results
Tokenization for some subsets are repeated at the 2nd and 3rd run. Starting from the 4th run, everything are loaded from cache.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Ubuntu 18.04.6 LTS
- Python version: 3.6.9
- PyArrow version: 6.0.1


Update faiss device docstring

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3846). All of your documentation changes will be reflected on that endpoint.

Following https://github.com/huggingface/datasets/pull/3721 I updated the docstring of the `device` argument of the FAISS related methods of `Dataset`


add RMSE and MAE metrics.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3845). All of your documentation changes will be reflected on that endpoint.

@mariosasko I've reopened it here. Please suggest any changes if required. Thank you.

Thanks for suggestions. :) I have added update the KWARGS_DESCRIPTION for the missing params and also changed RMSE to MSE.
While testing, I noticed that when the input is a list of lists, we get an error :
`TypeError: float() argument must be a string or a number, not 'list'`
Could you suggest the datasets.Value() attribute to support both list of floats and list of lists containing floats ?

Just add a new config to cover that case. You can do this by replacing the current `features` dict with:
```python
features=datasets.Features(
    {
        "predictions": datasets.Sequence(datasets.Value("float")),
        "references": datasets.Sequence(datasets.Value("float")),
    }
    if self.config_name == "multioutput"
    else {
        "predictions": datasets.Value("float"),
        "references": datasets.Value("float"),
     }
),
```
Feel free to suggest a better name for the config than `multioutput`

Also, could you please move the changes to a new branch and open a PR from there (for the 3rd time 😄) because the diff shows changes from unrelated PRs (maybe due to rebasing?).

Thanks for the input, I have added new config to support multi-dimensional lists and updated the examples as well.

Sure. Will do that and open a new PR for these changes.

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Please suggest any changes if required. Thank you.


Add rmse and mae metrics.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3844). All of your documentation changes will be reflected on that endpoint.

@dnaveenr This PR is in pretty good shape, so feel free to reopen it.

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Any suggestions and changes required will be helpful.


Fix Google Drive URL to avoid Virus scan warning in streaming mode

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3843). All of your documentation changes will be reflected on that endpoint.

Cool ! Looks like it breaks `test_streaming_gg_drive_gzipped` for some reason...

The streaming version of https://github.com/huggingface/datasets/pull/3787.

Fix #3835

CC: @albertvillanova


Align IterableDataset.shuffle with Dataset.shuffle

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3842). All of your documentation changes will be reflected on that endpoint.

We should also add `generator` as a param to `shuffle` to fully align the APIs, no?

I added the `generator` argument.

I had to make a few other adjustments to make it work. In particular when you call `set_epoch()` on a streaming dataset, it updates the underlying random generator by using a new effective seed. The effective seed is generated using the previous generator and the epoch number.

From #3444 , Dataset.shuffle can have the same API than IterableDataset.shuffle (i.e. in streaming mode).

Currently you can pass an optional seed to both if you want, BUT currently IterableDataset.shuffle always requires a buffer_size, used for approximate shuffling. I propose using a reasonable default value (maybe 1000) instead.

In this PR, I set the default `buffer_size` value to 1,000, and I reorder the `IterableDataset.shuffle` arguments to match `Dataset.shuffle`, i.e. making `seed` the first argument.


Pyright reportPrivateImportUsage when `from datasets import load_dataset`

Something isn't working

## Describe the bug
Pyright complains about module not exported.

## Steps to reproduce the bug
Use an editor/IDE with Pyright Language server with default configuration:
```python
from datasets import load_dataset
```

## Expected results
No complain from Pyright

## Actual results
Pyright complain below:
```
`load_dataset` is not exported from module "datasets"
 Import from "datasets.load" instead [reportPrivateImportUsage]
```
Importing from `datasets.load` does indeed solves the problem but I believe importing directly from top level `datasets` is the intended usage per the documentation.

## Environment info
- `datasets` version: 1.18.3
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.10
- PyArrow version: 7.0.0


Pin responses to fix CI for Windows

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3840). All of your documentation changes will be reflected on that endpoint.

Temporarily fix CI for Windows by pinning `responses`.

See: https://app.circleci.com/pipelines/github/huggingface/datasets/10292/workflows/83de4a55-bff7-43ec-96f7-0c335af5c050/jobs/63355

Fix: #3839


CI is broken for Windows

Something isn't working

## Describe the bug
See: https://app.circleci.com/pipelines/github/huggingface/datasets/10292/workflows/83de4a55-bff7-43ec-96f7-0c335af5c050/jobs/63355


Add a data type for labeled images (image segmentation)

enhancement

New feature or request

It might be a mix of Image and ClassLabel, and the color palette might be generated automatically.

---

### Example

every pixel in the images of the annotation column (in https://huggingface.co/datasets/scene_parse_150) has a value that gives its class, and the dataset itself is associated with a color palette (eg https://github.com/open-mmlab/mmsegmentation/blob/98a353b674c6052d319e7de4e5bcd65d670fcf84/mmseg/datasets/ade.py#L47) that maps every class with a color.

So we might want to render the image as a colored image instead of a black and white one.

<img width="785" alt="156741519-fbae6844-2606-4c28-837e-279d83d00865" src="https://user-images.githubusercontent.com/1676121/157005263-7058c584-2b70-465a-ad94-8a982f726cf4.png">

See https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/features/labeled_image.py for reference in Tensorflow


Release: 1.18.4


Logo float left

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3836). All of your documentation changes will be reflected on that endpoint.

Weird, the logo doesn't seem to be floating on my side (using Chrome) at https://huggingface.co/docs/datasets/master/en/index

https://huggingface.co/docs/datasets/index

The needed css change from moon-landing just got deployed

<img width="1000" alt="Screenshot 2022-03-07 at 09 35 29" src="https://user-images.githubusercontent.com/11827707/156996422-339ba43e-932b-4849-babf-9321cb99c922.png">


The link given on the gigaword does not work

bug

Something isn't working

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Fix dead dataset scripts creation link.

Previous link gives 404 error. Updated with a new dataset scripts creation link.


Small typos in How-to-train tutorial.


Making Hugging Face the place to go for Graph NNs datasets

dataset request

Requesting to add a new dataset

graph

Datasets for Graph Neural Networks

It will be indeed really great to add support to GNN datasets. Big :+1: for this initiative.

@napoles-uach identifies the [TUDatasets](https://chrsmrrs.github.io/datasets/) (A collection of benchmark datasets for graph classification and regression).

Added to the Tasks in the initial issue.

Thanks Omar, that is a great collection!

Great initiative! Let's keep this issue for these 3 datasets, but moving forward maybe let's create a new issue per dataset :rocket: great work @napoles-uach and @omarespejel!

Let's make Hugging Face Datasets the central hub for GNN datasets :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the GNN field.

What are some datasets worth integrating into the Hugging Face hub?

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Special thanks to @napoles-uach for his collaboration on identifying the first ones:

- [ ] [SNAP-Stanford OGB Datasets](https://github.com/snap-stanford/ogb).
- [ ] [SNAP-Stanford Pretrained GNNs Chemistry and Biology Datasets](https://github.com/snap-stanford/pretrain-gnns).
- [ ] [TUDatasets](https://chrsmrrs.github.io/datasets/) (A collection of benchmark datasets for graph classification and regression)

cc @osanseviero


when using to_tf_dataset with shuffle is true, not all completed batches are made

bug

Something isn't working

Hi @greenned, this is expected behaviour for `to_tf_dataset`. By default, we drop the smaller 'remainder' batch during training (i.e. when `shuffle=True`). If you really want to keep that batch, you can set `drop_remainder=False` when calling `to_tf_dataset()`.

@Rocketknight1 Oh, thank you. I didn't get **drop_remainder** Have a nice day!

## Describe the bug
when converting a dataset to tf_dataset by using to_tf_dataset with shuffle true, the remainder is not converted to one batch

## Steps to reproduce the bug
this is the sample code below
https://colab.research.google.com/drive/1_oRXWsR38ElO1EYF9ayFoCU7Ou1AAej4?usp=sharing

## Expected results
regardless of shuffle is true or not, 67 rows dataset should be 5 batches when batch size is 16.

## Actual results
4 batches

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1