Make decoding of Audio and Image feature optional

Closing this PR for now due to https://github.com/huggingface/datasets/issues/3145#issuecomment-993664104.

Okay, after some more thinking, I'm re-opening this PR for three reasons:
* This feature will allow us to remove the `image_file_path`/`audio_file_path` columns in our vision/audio datasets. Currently, it makes sense to keep those columns because it's not obvious how to access the underlying path information of the Image/Audio feature. However, if the user is not aware and does `dataset[0]["image_file_path"]` on our vision/audio datasets, this will be costly because the image/audio file data has to be decoded first (stored in `dataset[0]["image"]`/`dataset[0]["audio"]`)
* In CV, we often work with the so-called "half life" datasets (RedCaps, WIT, ...) that only provide image URLs and not actual image data, and some of these image URLs even go down. Our solution to this problem is to give a note on how to efficiently download the image data using `map` in the datasets cards of such datasets. This feature will remove the need for a separate `image_url` column of type `Value("string")` in such datasets. Instead, we will be able to use the `image` column of type `Image()`  (the image feature knows how to decode image URLs using `xopen`), disable decoding and use `requests.get` for download, which I expect to be faster than `xopen`.
* This feature should help us in implementing `push_to_hub` for the Image/Audio where we transfer actual image/audio data and not paths

> This feature will allow us to remove the image_file_path/audio_file_path columns in our vision/audio datasets. Currently, it makes sense to keep those columns because it's not obvious how to access the underlying path information of the Image/Audio feature. However, if the user is not aware and does dataset[0]["image_file_path"] on our vision/audio datasets, this will be costly because the image/audio file data has to be decoded first (stored in dataset[0]["image"]/dataset[0]["audio"])

That makes sense !

> Instead, we will be able to use the image column of type Image() (the image feature knows how to decode image URLs using xopen), disable decoding and use requests.get for download, which I expect to be faster than xopen.

I feel like it's a bit convoluted compared to having the `image_url` column as string, and say to users to `map` using `requests.get` with `image_url`.

Moreover I'm not 100% sure that we should have `Image` features with both local paths and URLs, since this behavior is a bit hidden the users and they don't give the same performance at all.

> This feature should help us in implementing push_to_hub for the Image/Audio where we transfer actual image/audio data and not paths

Cool !

Thanks, @lhoestq.

> I feel like it's a bit convoluted compared to having the image_url column as string, and say to users to map using requests.get with image_url.

Yes, that makes sense.

>Moreover I'm not 100% sure that we should have Image features with both local paths and URLs, since this behavior is a bit hidden the users and they don't give the same performance at all.

Do you mean we should remove support for URLs in the Image feature? Because this is what we get for free by adding support for streaming (by using `xopen` instead of `open`) and this is also what the Audio feature does.

> Do you mean we should remove support for URLs in the Image feature? Because this is what we get for free by adding support for streaming (by using xopen instead of open) and this is also what the Audio feature does.

I think it might not be ideal to have URLs in an `Image` type column for a dataset in **non-streaming** mode, since you'd expect to have everything locally. But for a streaming dataset it must use `xopen` indeed

Yes, I agree. Let's have the `image_url` columns as `Value("string")` + a note with the map function to download images for local datasets and a note with `cast_column` (which is requested in https://github.com/huggingface/datasets/issues/3369) for streamed datasets (`ds.cast_column("image_url", Image())`).

Add the `decode` argument (`True` by default) to the `Audio` and the `Image` feature to make it possible to toggle on/off decoding of these features.

Even though we've discussed that on Slack, I'm not removing the `_storage_dtype` argument of the Audio feature in this PR to avoid breaking the Audio feature tests.


Make cast cacheable (again) on Windows

`cast` currently emits the following warning when called on Windows:
```
Parameter 'function'=<function Dataset.cast.<locals>.<lambda> at 0x000001C930571EA0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting
and caching to work. If you reuse this transform, the caching mechanism will consider it to be different
from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
```

It seems like the issue stems from the `config.PYARROW_VERSION` object not being serializable on Windows (tested with `dumps(lambda: config.PYARROW_VERSION)`), so I'm fixing this by capturing `config.PYARROW_VERSION.major` before the lambda definition.


Clean squad dummy data

Some unused files were remaining, this PR removes them. We just need to keep the dummy_data.zip file


Add The Pile Enron Emails subset

Add:
- Enron Emails subset of The Pile: "enron_emails" config

Close bigscience-workshop/data_tooling#310.

CC: @StellaAthena


Update disaster_response_messages download urls (+ add validation split)

Fixes #3240, fixes #3416


Getting configs names takes too long

bug

Something isn't working

maybe related to https://github.com/huggingface/datasets/issues/2859

It looks like it's currently calling `HfFileSystem.ls()` ~8 times at the root and for each subdirectory:
- ""
- "en.noblocklist"
- "en.noclean"
- "en"
- "multilingual"
- "realnewslike"

Currently `ls` is slow because it iterates on all the files inside the repository.

An easy optimization would be to cache the result of each call to `ls`.
We can also optimize `ls` by using a tree structure per directory instead of a list of all the files.

## Steps to reproduce the bug
```python
from datasets import get_dataset_config_names
get_dataset_config_names("allenai/c4")
```

## Expected results

I would expect to get the answer quickly, at least in less than 10s

## Actual results

It takes about 45s on my environment

## Environment info

- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-1022-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1


Add RedCaps dataset

Cool ! If you want you can include `dataset_infos.json` but only for the main configurations. That's what we do for example for translation datasets when there are too many configs

Add the RedCaps dataset. I'm not adding the generated `dataset_infos.json` file for now due to its size (11 MB).

TODOs:
- [ ] dummy data
- [ ] dataset card

Close #3316


data duplicate when setting num_works > 1 with streaming data

Something isn't working

Hi ! Thanks for reporting :)

When using a PyTorch's data loader with `num_workers>1` and an iterable dataset, each worker streams the exact same data by default, resulting in duplicate data when iterating using the data loader.

We can probably fix this in `datasets` by checking `torch.utils.data.get_worker_info()` which gives the worker id if it happens.

> Hi ! Thanks for reporting :)
>
> When using a PyTorch's data loader with `num_workers>1` and an iterable dataset, each worker streams the exact same data by default, resulting in duplicate data when iterating using the data loader.
>
> We can probably fix this in `datasets` by checking `torch.utils.data.get_worker_info()` which gives the worker id if it happens.
Hi ! Thanks for reply

Do u have some plans to fix the problem?

Isn’t that somehow a bug on PyTorch side? (Just asking because this behavior seems quite general and maybe not what would be intended)

From PyTorch's documentation [here](https://pytorch.org/docs/stable/data.html#dataset-types):

> When using an IterableDataset with multi-process data loading. The same dataset object is replicated on each worker process, and thus the replicas must be configured differently to avoid duplicated data. See [IterableDataset](https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset) documentations for how to achieve this.

It looks like an intended behavior from PyTorch

As suggested in the [docstring of the IterableDataset class](https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset), we could pass a `worker_init_fn` to the DataLoader to fix this. It could be called `streaming_worker_init_fn` for example.

However, while this solution works, I'm worried that many users simply don't know about this parameter and just start their training with duplicate data without knowing it. That's why I'm more in favor of integrating the check on the worker id directly in `datasets` in our implementation of `IterableDataset.__iter__`.

## Describe the bug
The data is repeated num_works times when we load_dataset with streaming and set num_works > 1 when construct dataloader

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import pandas as pd
import numpy as np
import os

from datasets import load_dataset
from torch.utils.data import DataLoader
from tqdm import tqdm
import shutil

NUM_OF_USER = 1000000
NUM_OF_ACTION = 50000
NUM_OF_SEQUENCE = 10000
NUM_OF_FILES = 32
NUM_OF_WORKERS = 16

if __name__ == "__main__":
    shutil.rmtree("./dataset")
    for i in range(NUM_OF_FILES):
        sequence_data = pd.DataFrame(
            {
                "imei": np.random.randint(1, NUM_OF_USER, size=NUM_OF_SEQUENCE),
                "sequence": np.random.randint(1, NUM_OF_ACTION, size=NUM_OF_SEQUENCE)
            }
        )

        if not os.path.exists("./dataset"):
            os.makedirs("./dataset")

        sequence_data.to_csv(f"./dataset/sequence_data_{i}.csv",

 index=False)

    dataset = load_dataset("csv",
                           data_files=[os.path.join("./dataset",file) for file in os.listdir("./dataset") if file.endswith(".csv")],
                           split="train",
                           streaming=True).with_format("torch")
    data_loader = DataLoader(dataset,
                             batch_size=1024,
                             num_workers=NUM_OF_WORKERS)

    result = pd.DataFrame()
    for i, batch in tqdm(enumerate(data_loader)):
        result = pd.concat([result,
                           pd.DataFrame(batch)],
                           axis=0)
    result.to_csv(f"num_work_{NUM_OF_WORKERS}.csv", index=False)

```

## Expected results
data do not duplicate

## Actual results
data duplicate NUM_OF_WORKERS = 16
![image](https://user-images.githubusercontent.com/16486492/145748707-9d2df25b-2f4f-4d7b-a83e-242be4fc8934.png)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:datasets==1.14.0
- Platform:transformers==4.11.3
- Python version:3.8
- PyArrow version:


Error about load_metric

Something isn't working

Hi ! I wasn't able to reproduce your error.

Can you try to clear your cache at `~/.cache/huggingface/modules` and try again ?

## Describe the bug
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1371, in load_metric
    metric = metric_cls(
TypeError: 'NoneType' object is not callable

## Steps to reproduce the bug
```python
metric = load_metric("glue", "sst2")
```

## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-4.15.0-161-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyArrow version: 6.0.1


Adding mMARCO dataset

Hi @albertvillanova we've made a major overhaul of the loading script including all configurations we're making available. Could you please review it again?

Adding mMARCO (v1.1) to HF datasets.


Add eli5_category dataset

> Thanks a lot for adding this dataset ! Good job with the dataset card and the dataset scripts - they're really good :)
>
> I just added minor changes

Thanks for fixing typos!

This pull request adds a categorized Long-form question answering dataset `ELI5_Category`. It's a new variant of the [ELI5](https://huggingface.co/datasets/eli5) dataset that uses the Reddit tags to alleviate the training/validation overlapping in the origin ELI5 dataset.

A [report](https://celeritasml.netlify.app/posts/2021-12-01-eli5c/)(Section 2) on this dataset.


`.to_json` is extremely slow after `.select`

Something isn't working

Hi ! It's slower indeed because a datasets on which `select`/`shard`/`train_test_split`/`shuffle` has been called has to do additional steps to retrieve the data of the dataset table in the right order.

Indeed, if you call `dataset.select([0, 5, 10])`, the underlying table of the dataset is not altered to keep the examples at index 0, 5, and 10. Instead, an indices mapping is added on top of the table, that says that the first example is at index 0, the second at index 5 and the last one at index 10.

Therefore accessing the examples of the dataset is slower because of the additional step that uses the indices mapping.

The step that takes the most time is to query the dataset table from a list of indices here:

https://github.com/huggingface/datasets/blob/047dc756ed20fbf06e6bcaf910464aba0e20610a/src/datasets/formatting/formatting.py#L61-L63

In your case it can be made significantly faster by checking if the indices are contiguous. If they're contiguous, we could pass a python `slice` or `range` instead of a list of integers to `_query_table`. This way `_query_table` will do only one lookup to get the queried batch instead of `batch_size` lookups.

Given that calling `select` with contiguous indices is a common use case I'm in favor of implementing such an optimization :)
Let me know what you think

Hi, thanks for the response!
I still don't understand why it is so much slower than iterating and saving:
```python
from datasets import load_dataset

original = load_dataset("squad", split="train")
original.to_json("from_original.json")  # Takes 0 seconds

selected_subset1 = original.select([i for i in range(len(original))])
selected_subset1.to_json("from_select1.json")  # Takes 99 seconds

selected_subset2 = original.select([i for i in range(int(len(original) / 2))])
selected_subset2.to_json("from_select2.json")  # Takes 47 seconds

selected_subset3 = original.select([i for i in range(len(original)) if i % 2 == 0])
selected_subset3.to_json("from_select3.json")  # Takes 49 seconds

import json
import time
def fast_to_json(dataset, path):
    start = time.time()
    with open(path, mode="w") as f:
        for example in dataset:
            f.write(json.dumps(example, separators=(',', ':')) + "\n")
    end = time.time()
    print(f"Saved {len(dataset)} examples to {path} in {end - start} seconds.")

fast_to_json(original, "from_original_fast.json")
fast_to_json(selected_subset1, "from_select1_fast.json")
fast_to_json(selected_subset2, "from_select2_fast.json")
fast_to_json(selected_subset3, "from_select3_fast.json")
```
```
Saved 87599 examples to from_original_fast.json in 8 seconds.
Saved 87599 examples to from_select1_fast.json in 10 seconds.
Saved 43799 examples to from_select2_fast.json in 6 seconds.
Saved 43800 examples to from_select3_fast.json in 5 seconds.
```

There are slight differences between what you're doing and what `to_json` is actually doing.
In particular `to_json` currently converts batches of rows (as an arrow table) to a pandas dataframe, and then to JSON Lines. From your benchmark it looks like it's faster if we don't use pandas.

Thanks for investigating, I think we can optimize `to_json` significantly thanks to your test.

Thanks for your observations, @eladsegal! I spent some time with this and tried different approaches. Turns out that https://github.com/huggingface/datasets/blob/bb13373637b1acc55f8a468a8927a56cf4732230/src/datasets/io/json.py#L100 is giving the problem when we use `to_json` after `select`. This is when `indices` parameter in `query_table` is not `None` (if it is `None` then `to_json` should work as expected)

In order to circumvent this problem, I found out instead of doing Arrow Table -> Pandas-> JSON we can directly go to JSON by using `to_pydict()` which is a little slower than the current approach but at least `select` works properly now. Lmk what you guys think of it @lhoestq, @eladsegal?

Sounds good to me ! Feel free to also share your benchmarks for reference @bhavitvyamalik

Posting it in @eladsegal's format:

For `squad`:
Saving examples using current `to_json` in 3.63 secs
Saving examples to `from_select1_fast.json` in 5.00 secs
Saving examples to `from_select2_fast.json` in 2.45 secs
Saving examples to `from_select3_fast.json` in 2.50 secs

For `squad_v2`:
Saving examples using current `to_json` in 5.26 secs
Saving examples to `from_select1_fast.json` in 7.54 secs
Saving examples to `from_select2_fast.json` in 3.80 secs
Saving examples to `from_select3_fast.json` in 3.67 secs

## Describe the bug
Saving a dataset to JSON with `to_json` is extremely slow after using `.select` on the original dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset

original = load_dataset("squad", split="train")
original.to_json("from_original.json")  # Takes 0 seconds

selected_subset1 = original.select([i for i in range(len(original))])
selected_subset1.to_json("from_select1.json")  # Takes 212 seconds

selected_subset2 = original.select([i for i in range(int(len(original) / 2))])
selected_subset2.to_json("from_select2.json")  # Takes 90 seconds
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: master (https://github.com/huggingface/datasets/commit/6090f3cfb5c819f441dd4a4bb635e037c875b044)
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 6.0.0


Add Wikisource dataset

Add loading script for Wikisource dataset.

Fix #3399.

CC: @geohci, @yjernite


Fix type of bridge field in QED

Use `Value("string")` instead of `Value("bool")` for the feature type of the `"bridge"` field in the QED dataset. If the value is `False`, set to `None`.

The following paragraph in the QED repo explains the purpose of this field:
>Each annotation in referential_equalities is a pair of spans, the question_reference and the sentence_reference, corresponding to an entity mention in the question and the selected_sentence respectively. As described in the paper, sentence_references can be "bridged in", in which case they do not correspond with any actual span in the selected_sentence. Hence, sentence_reference spans contain an additional field, bridge, which is a prepositional phrase when a reference is bridged, and is False otherwise. Prepositional phrases serve to link bridged references to an anchoring phrase in the selected_sentence. In the case a sentence_reference is bridged, the start and end, as well as the span string, map to such an anchoring phrase in the selected_sentence.

Fix #3346

cc @VictorSanh


disaster_response_messages unavailable

dataset-viewer

Related to the dataset viewer on huggingface.co

Hi, thanks for reporting! This is a duplicate of https://github.com/huggingface/datasets/issues/3240. We are working on a fix.

## Dataset viewer issue for '* disaster_response_messages*'

**Link:** https://huggingface.co/datasets/disaster_response_messages

Dataset unavailable. Link dead: https://datasets.appen.com/appen_datasets/disaster_response_data/disaster_response_messages_training.csv

Am I the one who added this dataset ?No


Non-deterministic tests: CI tests randomly fail

bug

Something isn't working

I think it might come from two different issues:
1. Google Drive is an unreliable host, mainly because of quota limitations
2. the staging environment can sometimes raise some errors

For Google Drive tests we could set up some retries with backup URLs if necessary I guess.
For staging on the other hand, I guess we can investigate what causes this and discuss with the back-end team

## Describe the bug
Some CI tests fail randomly.

1. In https://github.com/huggingface/datasets/pull/3375/commits/c10275fe36085601cb7bdb9daee9a8f1fc734f48, there were 3 failing tests, only on Linux:
   ```
   =========================== short test summary info ============================
   FAILED tests/test_streaming_download_manager.py::test_streaming_dl_manager_get_extraction_protocol[https://drive.google.com/uc?export=download&id=1k92sUfpHxKq8PXWRr7Y5aNHXwOCNUmqh-zip]
   FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive - Fi...
   FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped
   = 3 failed, 3553 passed, 2950 skipped, 2 xfailed, 1 xpassed, 125 warnings in 192.79s (0:03:12) =
   ```

2. After re-running the CI (without any change in the code) in https://github.com/huggingface/datasets/pull/3375/commits/57bfe1f342cd3c59d2510b992d5f06a0761eb147, there was only 1 failing test (one on Linux and a different one on Windows):
   - On Linux:
      ```
      =========================== short test summary info ============================
      FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped
      = 1 failed, 3555 passed, 2950 skipped, 2 xfailed, 1 xpassed, 125 warnings in 199.76s (0:03:19) =
      ```
   - On Windows:
      ```
      =========================== short test summary info ===========================
      FAILED tests/test_load.py::test_load_dataset_builder_for_community_dataset_without_script
      = 1 failed, 3551 passed, 2954 skipped, 2 xfailed, 1 xpassed, 121 warnings in 478.58s (0:07:58) =
      ```

   The test `tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped` passes locally.

3. After re-running again the CI (without any change in the code) in https://github.com/huggingface/datasets/pull/3375/commits/39f32f2119cf91b86867216bb5c356c586503c6a, ALL the tests passed.


Skip None encoding (line deleted by accident in #3195)

Return the line deleted by accident in #3195 while [resolving merge conflicts](https://github.com/huggingface/datasets/pull/3195/commits/8b0ed15be08559056b817836a07d47acda0c4510).

Fix #3181 (finally :))


Add WIDER FACE dataset

Adds the WIDER FACE face detection benchmark.

TODOs:
* [x] dataset card
* [x] dummy data


Fix flaky test again for s3 serialization

Following https://github.com/huggingface/datasets/pull/3388 that wasn't enough (see CI error [here](https://app.circleci.com/pipelines/github/huggingface/datasets/9080/workflows/b971fb27-ff20-4220-9416-c19acdfdf6f4/jobs/55985))


[chinese wwm] load_datasets behavior not as expected when using run_mlm_wwm.py script

Something isn't working

@LysandreJik not so sure who to @
Could you help?

Hi @hyusterr, I believe it is @wlhgtc from https://github.com/huggingface/transformers/pull/9887

## Describe the bug

Model I am using (Bert, XLNet ...): bert-base-chinese

The problem arises when using:
* [https://github.com/huggingface/transformers/blob/master/examples/research_projects/mlm_wwm/run_mlm_wwm.py] the official example scripts: `rum_mlm_wwm.py`

The tasks I am working on is: pretraining whole word masking with my own dataset and ref.json file
I tried follow the run_mlm_wwm.py procedure to do whole word masking on pretraining task. my file is in .txt form, where one line represents one sample, with `9,264,784` chinese lines in total. the ref.json file is also contains 9,264,784 lines of whole word masking reference data for my chinese corpus. but when I try to adapt the run_mlm_wwm.py script, it shows that somehow after
`datasets["train"] = load_dataset(...`
`len(datasets["train"])` returns `9,265,365`
then, after `tokenized_datasets = datasets.map(...`
`len(tokenized_datasets["train"])` returns `9,265,279`
I'm really confused and tried to trace code by myself but can't know what happened after a week trial.

I want to know what happened in the `load_dataset()` function and `datasets.map` here and how did I get more lines of data than I input. so I'm here to ask.
## To reproduce
Sorry that I can't provide my data here since it did not belong to me. but I'm sure I remove the blank lines.

## Expected behavior
I expect the code run as it should. but the AssertionError in line 167 keeps raise as the line of reference json and datasets['train'] differs.

Thanks for your patient reading!

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 3.0.0


Fix dependencies conflicts in Windows CI after conda update to 4.11

For some reason the CI wasn't using python 3.6 but python 3.7 after the update to conda 4.11


Pass new_fingerprint in multiprocessing

Following https://github.com/huggingface/datasets/pull/3045

Currently one can pass `new_fingerprint` to `.map()` to use a custom fingerprint instead of the one computed by hashing the map transform. However it's ignored if `num_proc>1`.

In this PR I fixed that by passing `new_fingerprint` to `._map_single()` when `num_proc>1`.
More specifically, `new_fingerprint` with a suffix based on the process `rank` is passed, so that each process has a different `new_fingerprint`

cc @TevenLeScao @vlievin


Typo in Dataset viewer error message

Fixed, thanks
<img width="661" alt="Capture d’écran 2021-12-22 à 12 02 30" src="https://user-images.githubusercontent.com/1676121/147082881-cf700e8d-0511-4431-b214-d6cf8137db10.png">

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

When creating an empty dataset repo, the Dataset Preview provides a helpful message that no files were found. There is a tiny typo in that message: "ressource" should be "resource"

![Screen Shot 2021-12-09 at 15 31 31](https://user-images.githubusercontent.com/26859204/145415725-9cd728f0-c2c8-4b4e-a8e1-4f4d7841c94a.png)

Am I the one who added this dataset ?

N/A


Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data

enhancement

New feature or request

IMO, the data streaming test is good enough of a test that the dataset works correctly (assuming that we can more or less ensure that if streaming works then the non-streaming case will also work), so that for datasets that have a working dataset preview, we can remove the dummy data IMO. On the other hand, it seems like not all datasets have streaming enabled yet and for those datasets (if they are used a lot), I think it would be nice to continue testing some dummy data.

I don't really have an opinion regarding the JSON metadata as I don't know enough about it.

I open this PR to have a public discussion about this topic and make a decision.

As previously discussed, once we have the metadata in the dataset card (README file, containing both Markdown info and YAML tags), what is the point of having also the JSON metadata (dataset_infos.json file)?

On the other hand, the dummy data is necessary for testing (in our CI suite) that the canonical dataset loads correctly. However:
- the dataset preview feature is already an indirect test that the dataset loads correctly (it also tests it is streamable though)
- we are migrating canonical datasets to the Hub
Do we really need to continue testing them in out CI?

Also note that for generating both (dataset_infos.json file and dummy data), the entire dataset needs being downloaded. This can be an issue for huge datasets (like WIT, with 400 GB of data).

Feel free to ping other people for the discussion.

CC: @lhoestq @mariosasko @thomwolf @julien-c @patrickvonplaten @anton-l @LysandreJik @yjernite @nateraw


Allows DatasetDict.filter to have batching option

- Related to: #3244
- Fixes: #3503

We extends `.filter( ... batched: bool)` support to DatasetDict.


cast_column function not working with map function in streaming mode for Audio features

bug

Something isn't working

## Describe the bug
I am trying to use Audio class for loading audio features using custom dataset. I am able to cast 'audio' feature into 'Audio' format with cast_column function. On using map function, I am not getting 'Audio' casted feature but getting path of audio file only.
I am getting features of 'audio' of string type with load_dataset call. After using cast_column 'audio' feature is converted into 'Audio' type. But in map function I am not able to get Audio type for audio feature & getting string type data containing path of file only. So I am not able to use processor in encode function.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset, Audio
from transformers import Wav2Vec2Processor

def encode(batch, processor):
    print("Audio: ",batch['audio'])
    batch["input_values"] = processor(batch["audio"]['array'], sampling_rate=16000).input_values
    return batch

def print_ds(ds):
    iterator = iter(ds)
    for d in iterator:
        print("Data: ",d)
        break

processor = Wav2Vec2Processor.from_pretrained(pretrained_model_path)

dataset = load_dataset("custom_dataset.py","train",data_files={'train':'train_path.txt'},
                                data_dir="data", streaming=True, split="train")

print("Features: ",dataset.features)

print_ds(dataset)

dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

print("Features: ",dataset.features)

print_ds(dataset)

dataset = dataset.map(lambda x: encode(x,processor))

print("Features: ",dataset.features)

print_ds(dataset)

```

## Expected results

map function not printing Audio type features be used with processor function and getting error in processor call due to this.

## Actual results

# after load_dataset call
Features:  {'sentence': Value(dtype='string', id=None), 'audio': Value(dtype='string', id=None)}
Data:  {'sentence': 'और अपने पेट को माँ की स्वादिष्ट गरमगरम जलेबियाँ हड़पते\n', 'audio': 'data/0116_003.wav'}

# after cast_column call
Features:  {'sentence': Value(dtype='string', id=None), 'audio': Audio(sampling_rate=16000, mono=True, _storage_dtype='string', id=None)}
Data:  {'sentence': 'और अपने पेट को माँ की स्वादिष्ट गरमगरम जलेबियाँ हड़पते\n', 'audio': {'path': 'data/0116_003.wav', 'array': array([ 1.2662281e-06,  1.0264218e-06, -1.3615092e-06, ...,
        1.3017889e-02,  1.0085563e-02,  4.8155054e-03], dtype=float32), 'sampling_rate': 16000}}

# after map call
Features:  None
Audio:  data/0116_003.wav

Traceback (most recent call last):
  File "demo2.py", line 36, in <module>
    print_ds(dataset)
  File "demo2.py", line 11, in print_ds
    for d in iterator:
  File "/opt/conda/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/opt/conda/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/opt/conda/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 192, in __iter__
    yield key, self.function(example)
  File "demo2.py", line 32, in <lambda>
    dataset = dataset.map(lambda x: batch_encode(x,processor))
  File "demo2.py", line 6, in batch_encode
    batch["input_values"] = processor(batch["audio"]['array'], sampling_rate=16000).input_values
TypeError: string indices must be integers
## Environment info

- `datasets` version: 1.17.0
- Platform: Linux-4.14.243 with-debian-bullseye-sid
- Python version: 3.7.9
- PyArrow version: 6.0.1


Unable to download PUBMED_title_abstracts_2019_baseline.jsonl.zst

Something isn't working

## Describe the bug
I am unable to download the PubMed dataset from the link provided in the [Hugging Face Course (Chapter 5 Section 4)](https://huggingface.co/course/chapter5/4?fw=pt).

https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset

# This takes a few minutes to run, so go grab a tea or coffee while you wait :)
data_files = "https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst"
pubmed_dataset = load_dataset("json", data_files=data_files, split="train")
pubmed_dataset
```

I also tried with `wget` as follows.
```
wget https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst
```

## Expected results
I expect to be able to download this file.

## Actual results
Traceback
```
---------------------------------------------------------------------------
timeout                                   Traceback (most recent call last)
/usr/lib/python3/dist-packages/urllib3/connection.py in _new_conn(self)
    158         try:
--> 159             conn = connection.create_connection(
    160                 (self._dns_host, self.port), self.timeout, **extra_kw

/usr/lib/python3/dist-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     83     if err is not None:
---> 84         raise err
     85

/usr/lib/python3/dist-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     73                 sock.bind(source_address)
---> 74             sock.connect(sa)
     75             return sock

timeout: timed out

During handling of the above exception, another exception occurred:

ConnectTimeoutError                       Traceback (most recent call last)
/usr/lib/python3/dist-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    664             # Make the request on the httplib connection object.
--> 665             httplib_response = self._make_request(
    666                 conn,

/usr/lib/python3/dist-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    375         try:
--> 376             self._validate_conn(conn)
    377         except (SocketTimeout, BaseSSLError) as e:

/usr/lib/python3/dist-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
    995         if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
--> 996             conn.connect()
    997

/usr/lib/python3/dist-packages/urllib3/connection.py in connect(self)
    313         # Add certificate verification
--> 314         conn = self._new_conn()
    315         hostname = self.host

/usr/lib/python3/dist-packages/urllib3/connection.py in _new_conn(self)
    163         except SocketTimeout:
--> 164             raise ConnectTimeoutError(
    165                 self,

ConnectTimeoutError: (<urllib3.connection.VerifiedHTTPSConnection object at 0x7f06dd698850>, 'Connection to the-eye.eu timed out. (connect timeout=10.0)')

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
/usr/lib/python3/dist-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    438             if not chunked:
--> 439                 resp = conn.urlopen(
    440                     method=request.method,

/usr/lib/python3/dist-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    718
--> 719             retries = retries.increment(
    720                 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]

/usr/lib/python3/dist-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    435         if new_retry.is_exhausted():
--> 436             raise MaxRetryError(_pool, url, error or ResponseError(cause))
    437

MaxRetryError: HTTPSConnectionPool(host='the-eye.eu', port=443): Max retries exceeded with url: /public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f06dd698850>, 'Connection to the-eye.eu timed out. (connect timeout=10.0)'))

During handling of the above exception, another exception occurred:

ConnectTimeout                            Traceback (most recent call last)
/tmp/ipykernel_15104/606583593.py in <module>
      3 # This takes a few minutes to run, so go grab a tea or coffee while you wait :)
      4 data_files = "https://the-eye.eu/public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst"
----> 5 pubmed_dataset = load_dataset("json", data_files=data_files, split="train")
      6 pubmed_dataset

~/.local/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1655
   1656     # Create a dataset builder
-> 1657     builder_instance = load_dataset_builder(
   1658         path=path,
   1659         name=name,

~/.local/lib/python3.8/site-packages/datasets/load.py in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, script_version, **config_kwargs)
   1492         download_config = download_config.copy() if download_config else DownloadConfig()
   1493         download_config.use_auth_token = use_auth_token
-> 1494     dataset_module = dataset_module_factory(
   1495         path, revision=revision, download_config=download_config, download_mode=download_mode, data_files=data_files
   1496     )

~/.local/lib/python3.8/site-packages/datasets/load.py in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_files, **download_kwargs)
   1116     # Try packaged
   1117     if path in _PACKAGED_DATASETS_MODULES:
-> 1118         return PackagedDatasetModuleFactory(
   1119             path, data_files=data_files, download_config=download_config, download_mode=download_mode
   1120         ).get_module()

~/.local/lib/python3.8/site-packages/datasets/load.py in get_module(self)
    773             else get_patterns_locally(str(Path().resolve()))
    774         )
--> 775         data_files = DataFilesDict.from_local_or_remote(patterns, use_auth_token=self.downnload_config.use_auth_token)
    776         module_path, hash = _PACKAGED_DATASETS_MODULES[self.name]
    777         builder_kwargs = {"hash": hash, "data_files": data_files}

~/.local/lib/python3.8/site-packages/datasets/data_files.py in from_local_or_remote(cls, patterns, base_path, allowed_extensions, use_auth_token)
    576         for key, patterns_for_key in patterns.items():
    577             out[key] = (
--> 578                 DataFilesList.from_local_or_remote(
    579                     patterns_for_key,
    580                     base_path=base_path,

~/.local/lib/python3.8/site-packages/datasets/data_files.py in from_local_or_remote(cls, patterns, base_path, allowed_extensions, use_auth_token)
    545         base_path = base_path if base_path is not None else str(Path().resolve())
    546         data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
--> 547         origin_metadata = _get_origin_metadata_locally_or_by_urls(data_files, use_auth_token=use_auth_token)
    548         return cls(data_files, origin_metadata)
    549

~/.local/lib/python3.8/site-packages/datasets/data_files.py in _get_origin_metadata_locally_or_by_urls(data_files, max_workers, use_auth_token)
    492     data_files: List[Union[Path, Url]], max_workers=64, use_auth_token: Optional[Union[bool, str]] = None
    493 ) -> Tuple[str]:
--> 494     return thread_map(
    495         partial(_get_single_origin_metadata_locally_or_by_urls, use_auth_token=use_auth_token),
    496         data_files,

~/.local/lib/python3.8/site-packages/tqdm/contrib/concurrent.py in thread_map(fn, *iterables, **tqdm_kwargs)
     92     """
     93     from concurrent.futures import ThreadPoolExecutor
---> 94     return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
     95
     96

~/.local/lib/python3.8/site-packages/tqdm/contrib/concurrent.py in _executor_map(PoolExecutor, fn, *iterables, **tqdm_kwargs)
     74             map_args.update(chunksize=chunksize)
     75         with PoolExecutor(**pool_kwargs) as ex:
---> 76             return list(tqdm_class(ex.map(fn, *iterables, **map_args), **kwargs))
     77
     78

~/.local/lib/python3.8/site-packages/tqdm/notebook.py in __iter__(self)
    252     def __iter__(self):
    253         try:
--> 254             for obj in super(tqdm_notebook, self).__iter__():
    255                 # return super(tqdm...) will not catch exception
    256                 yield obj

~/.local/lib/python3.8/site-packages/tqdm/std.py in __iter__(self)
   1171         # (note: keep this check outside the loop for performance)
   1172         if self.disable:
-> 1173             for obj in iterable:
   1174                 yield obj
   1175             return

/usr/lib/python3.8/concurrent/futures/_base.py in result_iterator()
    617                     # Careful not to keep a reference to the popped future
    618                     if timeout is None:
--> 619                         yield fs.pop().result()
    620                     else:
    621                         yield fs.pop().result(end_time - time.monotonic())

/usr/lib/python3.8/concurrent/futures/_base.py in result(self, timeout)
    442                     raise CancelledError()
    443                 elif self._state == FINISHED:
--> 444                     return self.__get_result()
    445                 else:
    446                     raise TimeoutError()

/usr/lib/python3.8/concurrent/futures/_base.py in __get_result(self)
    387         if self._exception:
    388             try:
--> 389                 raise self._exception
    390             finally:
    391                 # Break a reference cycle with the exception in self._exception

/usr/lib/python3.8/concurrent/futures/thread.py in run(self)
     55
     56         try:
---> 57             result = self.fn(*self.args, **self.kwargs)
     58         except BaseException as exc:
     59             self.future.set_exception(exc)

~/.local/lib/python3.8/site-packages/datasets/data_files.py in _get_single_origin_metadata_locally_or_by_urls(data_file, use_auth_token)
    483     if isinstance(data_file, Url):
    484         data_file = str(data_file)
--> 485         return (request_etag(data_file, use_auth_token=use_auth_token),)
    486     else:
    487         data_file = str(data_file.resolve())

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in request_etag(url, use_auth_token)
    489 def request_etag(url: str, use_auth_token: Optional[Union[str, bool]] = None) -> Optional[str]:
    490     headers = get_authentication_headers_for_url(url, use_auth_token=use_auth_token)
--> 491     response = http_head(url, headers=headers, max_retries=3)
    492     response.raise_for_status()
    493     etag = response.headers.get("ETag") if response.ok else None

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in http_head(url, proxies, headers, cookies, allow_redirects, timeout, max_retries)
    474     headers = copy.deepcopy(headers) or {}
    475     headers["user-agent"] = get_datasets_user_agent(user_agent=headers.get("user-agent"))
--> 476     response = _request_with_retry(
    477         method="HEAD",
    478         url=url,

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in _request_with_retry(method, url, max_retries, base_wait_time, max_wait_time, timeout, **params)
    407         except (requests.exceptions.ConnectTimeout, requests.exceptions.ConnectionError) as err:
    408             if tries > max_retries:
--> 409                 raise err
    410             else:
    411                 logger.info(f"{method} request to {url} timed out, retrying... [{tries/max_retries}]")

~/.local/lib/python3.8/site-packages/datasets/utils/file_utils.py in _request_with_retry(method, url, max_retries, base_wait_time, max_wait_time, timeout, **params)
    403         tries += 1
    404         try:
--> 405             response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
    406             success = True
    407         except (requests.exceptions.ConnectTimeout, requests.exceptions.ConnectionError) as err:

/usr/lib/python3/dist-packages/requests/api.py in request(method, url, **kwargs)
     58     # cases, and look like a memory leak in others.
     59     with sessions.Session() as session:
---> 60         return session.request(method=method, url=url, **kwargs)
     61
     62

/usr/lib/python3/dist-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    531         }
    532         send_kwargs.update(settings)
--> 533         resp = self.send(prep, **send_kwargs)
    534
    535         return resp

/usr/lib/python3/dist-packages/requests/sessions.py in send(self, request, **kwargs)
    644
    645         # Send the request
--> 646         r = adapter.send(request, **kwargs)
    647
    648         # Total elapsed time of the request (approximately)

/usr/lib/python3/dist-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    502                 # TODO: Remove this in 3.0.0: see #2811
    503                 if not isinstance(e.reason, NewConnectionError):
--> 504                     raise ConnectTimeout(e, request=request)
    505
    506             if isinstance(e.reason, ResponseError):

ConnectTimeout: HTTPSConnectionPool(host='the-eye.eu', port=443): Max retries exceeded with url: /public/AI/pile_preliminary_components/PUBMED_title_abstracts_2019_baseline.jsonl.zst (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f06dd698850>, 'Connection to the-eye.eu timed out. (connect timeout=10.0)'))
```

## Environment info
- `datasets` version: 1.17.0
- Platform: Linux-5.11.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 6.0.1


Batched in filter throws error

Something isn't working

I hope this is really a bug, I could not find it among the open issues

## Describe the bug
using `batched=False` in DataSet.filter throws error
```python
TypeError: filter() got an unexpected keyword argument 'batched'
```
but in the docs it is lister as an argument.

## Steps to reproduce the bug
```python
task = "mnli"
max_length = 128
tokenizer = AutoTokenizer.from_pretrained("./pretrained_models/pretrained_models_drozd/sl250.m.gsic.titech.ac.jp:8000/21.11.17_06.30.32_roberta-base_a0057/checkpoints/smpl_400M/hf/")

dataset = load_dataset("glue", task)

task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mnli-mm": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

##### tokenization_parameters
sentence1_key, sentence2_key = task_to_keys[task]
def preprocess_function(examples, max_length):
    if sentence2_key is None:
        return tokenizer(
            examples[sentence1_key], truncation=True, max_length=max_length
        )
    return tokenizer(
        examples[sentence1_key],
        examples[sentence2_key],
        truncation=False,
        padding="max_length",
        max_length=max_length,
    )

encoded_dataset = dataset.map(
    lambda x: preprocess_function(x, max_length=max_length), batched=False
)

encoded_dataset.filter(lambda x: len(x['input_ids']) <= max_length, batched=False)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1, 1.17.0
- Platform: ubuntu
- Python version: 3.8.12


Add QuALITY

Fixes #3441.


Update pib dataset card

Related to #3496


Docs: Add VCTK dataset description

This PR is a very minor followup to #1837, with only docs changes (single comment string).


Adjusting chunk size for streaming datasets

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
I want to use mc4 which I cannot save locally, so I stream it. However, I want to process the entire dataset and filter some documents from it. With the current chunk size of around 1000 documents (right?) I hit a performance bottleneck because of the frequent decompressing.

**Describe the solution you'd like**
I would appreciate a parameter in the load_dataset function, that allows me to set the chunksize myself (to a value like 100'000 in my case). Like that, I hope to improve the processing time.


update `pretty_name` for all datasets

I made a script some time back to fetch `pretty_names` from `papers_with_code` dataset along with some other rules. Updating them in the `README` of `datasets`. Took only first 200 datasets in consideration and after some eye balling most of them were looking good to me!


Changing sampling rate in audio dataset and subsequently mapping with `num_proc > 1` leads to weird bug

bug

Something isn't working

Same error occures when using max samples with https://github.com/huggingface/transformers/blob/master/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py

Running:

```python
from datasets import load_dataset, DatasetDict
import datasets
from transformers import AutoFeatureExtractor

raw_datasets = DatasetDict()

raw_datasets["train"] = load_dataset("common_voice", "ab", split="train")

feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base")

raw_datasets = raw_datasets.cast_column(
    "audio", datasets.features.Audio(sampling_rate=feature_extractor.sampling_rate)
)

num_workers = 16

def prepare_dataset(batch):
    sample = batch["audio"]

    inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
    batch["input_values"] = inputs.input_values[0]
    batch["input_length"] = len(batch["input_values"])
    return batch

raw_datasets.map(
    prepare_dataset,
    remove_columns=next(iter(raw_datasets.values())).column_names,
    num_proc=16,
    desc="preprocess datasets",
)
```

gives

```bash
  File "/home/patrick/experiments/run_bug.py", line 25, in <module>
    raw_datasets.map(
  File "/home/patrick/python_bin/datasets/dataset_dict.py", line 492, in map
    {
  File "/home/patrick/python_bin/datasets/dataset_dict.py", line 493, in <dictcomp>
    k: dataset.map(
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2139, in map
    shards = [
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2140, in <listcomp>
    self.shard(num_shards=num_proc, index=rank, contiguous=True, keep_in_memory=keep_in_memory)
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 3164, in shard
    return self.select(
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 485, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/patrick/python_bin/datasets/fingerprint.py", line 411, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2756, in select
    return self._new_dataset_with_indices(indices_buffer=buf_writer.getvalue(), fingerprint=new_fingerprint)
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 2667, in _new_dataset_with_indices
    return Dataset(
  File "/home/patrick/python_bin/datasets/arrow_dataset.py", line 659, in __init__
    raise ValueError(
ValueError: External features info don't match the dataset:
Got
{'client_id': Value(dtype='string', id=None), 'path': Value(dtype='string', id=None), 'audio': Audio(sampling_rate=16000, mono=True, _storage_dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'up_votes': Value(dtype='int64', id=None), 'down_votes': Value(dtype='int64', id=None), 'age': Value(dtype='string', id=None), 'gender': Value(dtype='string', id=None), 'accent': Value(dtype='string', id=None), 'locale': Value(dtype='string', id=None), 'segment': Value(dtype='string', id=None)}
with type
struct<client_id: string, path: string, audio: string, sentence: string, up_votes: int64, down_votes: int64, age: string, gender: string, accent: string, locale: string, segment: string>

but expected something like
{'client_id': Value(dtype='string', id=None), 'path': Value(dtype='string', id=None), 'audio': {'path': Value(dtype='string', id=None), 'bytes': Value(dtype='binary', id=None)}, 'sentence': Value(dtype='string', id=None), 'up_votes': Value(dtype='int64', id=None), 'down_votes': Value(dtype='int64', id=None), 'age': Value(dtype='string', id=None), 'gender': Value(dtype='string', id=None), 'accent': Value(dtype='string', id=None), 'locale': Value(dtype='string', id=None), 'segment': Value(dtype='string', id=None)}
with type
struct<client_id: string, path: string, audio: struct<path: string, bytes: binary>, sentence: string, up_votes: int64, down_votes: int64, age: string, gender: string, accent: string, locale: string, segment: string>
```

Versions:

```python
- `datasets` version: 1.16.2.dev0
- Platform: Linux-5.15.8-76051508-generic-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 6.0.1
```

and `transformers`:

```
- `transformers` version: 4.16.0.dev0
- Platform: Linux-5.15.8-76051508-generic-x86_64-with-glibc2.33
- Python version: 3.9.7
```


Update version of pib dataset and make it streamable

It seems like there is still an error: `Message:       'TarContainedFile' object has no attribute 'readable'`

https://huggingface.co/datasets/pib/viewer

@severo I was wondering about that...

It works fine when I run it in streaming mode in my terminal:
```python
In [3]: from datasets import load_dataset; ds = load_dataset("pib", "gu-pa", split="train", streaming=True); item = next(iter(ds))

In [4]: item
Out[4]:
{'translation': {'gu': 'એવો નિર્ણય લેવાયો હતો કે ખંતપૂર્વકની કામગીરી હાથ ધરવા, કાયદેસર અને ટેકનિકલ મૂલ્યાંકન કરવા, વેન્ચર કેપિટલ ઇન્વેસ્ટમેન્ટ સમિતિની બેઠક યોજવા વગેરે એઆઇએફને કરવામાં આવેલ પ્રતિબદ્ધતાના 0.50 ટકા સુધી અને બાકીની રકમ એફએફએસને પૂર્ણ કરવામાં આવશે.',
  'pa': 'ਇਹ ਵੀ ਫੈਸਲਾ ਕੀਤਾ ਗਿਆ ਕਿ ਐੱਫਆਈਆਈ ਅਤੇ ਬਕਾਏ ਲਈ ਕੀਤੀਆਂ ਗਈਆਂ ਵਚਨਬੱਧਤਾਵਾਂ ਦੇ 0.50 % ਦੀ ਸੀਮਾ ਤੱਕ ਐੱਫਈਐੱਸ ਨੂੰ ਮਿਲਿਆ ਜਾਏਗਾ, ਇਸ ਨਾਲ ਉੱਦਮ ਪੂੰਜੀ ਨਿਵੇਸ਼ ਕਮੇਟੀ ਦੀ ਬੈਠਕ ਦਾ ਆਯੋਜਨ ਉਚਿਤ ਸਾਵਧਾਨੀ, ਕਾਨੂੰਨੀ ਅਤੇ ਤਕਨੀਕੀ ਮੁੱਲਾਂਕਣ ਲਈ ਸੰਚਾਲਨ ਖਰਚ ਆਦਿ ਦੀ ਪੂਰਤੀ ਹੋਵੇਗੀ।'}}
```

This PR:
- Updates version of pib dataset: from 0.0.0 to 1.3.0
- Makes the dataset streamable

Fix #3491.

CC: @severo


Add VoxLingua107

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** VoxLingua107
- **Description:** VoxLingua107 is a speech dataset for training spoken language identification models.
- **Paper:** https://arxiv.org/abs/2011.12998
- **Data:** http://bark.phon.ioc.ee/voxlingua107/
- **Motivation:** 107 languages, totaling 6628 hours for the train split.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Clone full repo to detect new tags when mirroring datasets on the Hub

The CI fail is unrelated to this PR and fixed on master, merging :)

The new releases of `datasets` were not detected because the shallow clone in the CI wasn't getting the git tags.

By cloning the full repository we can properly detect a new release, and tag all the dataset repositories accordingly
cc @SBrandeis


Fix VCTK encoding

utf-8 encoding was missing in the VCTK dataset builder added in #3351


Add `gzip` for `to_json`

(Partially) closes #3480. I have added `gzip` compression for `to_json`. I realised we can run into this compression problem with `to_csv` as well. `IOHandler` can be used for `to_csv` too. Please let me know if any changes are required.


Update version of pib dataset

Requesting to add a new dataset

On the Hub we have v0, while there exists v1.3.

Related to bigscience-workshop/data_tooling#130


Does datasets support load text from HDFS?

The raw text data is stored on HDFS due to the dataset's size is too large to store on my develop machine,
so I wander does datasets support read data from hdfs?


Avoid unnecessary list creations

Like in `join([... for s in ...])`. Also changed other things that I saw:
* Use a `with` statement for many `open` that missed them, so the files don't remain open.
* Remove unused variables.
* Many HTTP links converted into HTTPS (verified).
* Remove unnecessary "r" mode arg in `open` (double-checked it was actually the default in each case).
* Remove Python 2 style of using `super`.
* Run `pyupgrade $(find . -name "*.py" -type f) --py36-plus` (which already does some of the previous points).
* Run `dos2unix $(find . -name "*.py" -type f)` (CRLF to LF line endings).
* Fix typos.


URL query parameters are set as path in the compression hop for fsspec

bug

Something isn't working

## Describe the bug
There is an ssue with `StreamingDownloadManager._extract`.

I don't know how the test `test_streaming_gg_drive_gzipped` passes:

For
```python
TEST_GG_DRIVE_GZIPPED_URL = "https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz"
urlpath = StreamingDownloadManager().download_and_extract(TEST_GG_DRIVE_GZIPPED_URL)
```
gives `urlpath`:
```python
'gzip://uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz::https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz'
```

The gzip path makes no sense: `gzip://uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz`

## Steps to reproduce the bug
```python
from datasets.utils.streaming_download_manager import StreamingDownloadManager

dl_manager = StreamingDownloadManager()
urlpath = dl_manager.extract("https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz")
print(urlpath)
```

## Expected results
The query parameters should not be set as part of the path.


Update ADD_NEW_DATASET.md

fixed make style prompt for Windows Terminal


Fix weird spacing in ManualDownloadError message

`textwrap.dedent` works based on the spaces at the beginning. Before this change, there wasn't any space.


enhancement

New feature or request

You can add columns that you wish to set into `torch` format using `dataset.set_format("torch", ['id', 'abc'])` so that input batch of the transform only contains those columns

Sorry, I miss `output_all_columns` args and thought after `dataset.set_format("torch", columns=columns)` I can only get specific columns I assigned.

**Is your feature request related to a problem? Please describe.**
When using `dataset.set_format("torch")`, I must make sure every columns in datasets can convert to `torch`, however, sometimes I want to keep some string columns.

**Describe the solution you'd like**
skip columns which cannot set to specific format when set_format instead of raise an error.


make shape verification to use ArrayXD instead of nested lists for map

New feature or request

As describe in https://github.com/huggingface/datasets/issues/2005#issuecomment-793716753 and mentioned by @mariosasko in [image feature example](https://colab.research.google.com/drive/1mIrTnqTVkWLJWoBzT1ABSe-LFelIep1c#scrollTo=ow3XHDvf2I0B&line=1&uniqifier=1), IMO make shape verifcaiton to use ArrayXD instead of nested lists for map can help user reduce unnecessary cast. I notice datasets have done something special for `input_ids` and `attention_mask` which is also unnecessary after this feature added.


Remove unused phony rule from Makefile


Fix duplicate keys in NewsQA

* Fix duplicate keys in NewsQA when loading from CSV files.
* Fix s/narqa/newsqa/ in the download manually error message.
* Make the download manually error message show nicely when printed. Otherwise, is hard to read due to spacing issues.
* Fix the format of the license text.
* Reformat the code to make it simpler.


Fix overriding of filesystem info

Previously, `BaseCompressedFileFileSystem.info` was overridden and transformed from function to dict.

This generated a bug for filesystem methods that use `self.info()`, like e.g. `fs.isfile()`.

This PR:
- Adds tests for `fs.isfile` (that use `fs.info`).
- Fixes custom `BaseCompressedFileFileSystem.info` by removing its overriding.


the compression format requested when saving a dataset in json format is not respected

bug

Something isn't working

Thanks for reporting @SaulLu.

At first sight I think the problem is caused because `pandas` only takes into account the `compression` parameter if called with a non-null file path or buffer. And in our implementation, we call pandas `to_json` with `None` `path_or_buf`.

We should fix this:
- either handling directly the `compression` parameter ourselves
- or refactoring to pass non-null path or buffer to pandas

CC: @lhoestq

I was thinking if we can handle the `compression` parameter by ourselves? Compression types will be similar to what `pandas` offer. Initially, we can try this with 2-3 compression types and see how good/bad it is? Let me know if it sounds good, I can raise a PR for this next week

Hi ! Thanks for your help @bhavitvyamalik :)
Maybe let's start with `gzip` ? I think it's the most common use case, then if we're fine with it we can add other compression methods

## Describe the bug
In the documentation of the `to_json` method, it is stated in the parameters that
> **to_json_kwargs – Parameters to pass to pandas’s pandas.DataFrame.to_json.

however when we pass for example `compression="gzip"`, the saved file is not compressed.

Would you also have expected compression to be applied? :relaxed:

## Steps to reproduce the bug
```python
my_dict = {"a": [1, 2, 3], "b": [1, 2, 3]}
```
### Result with datasets
```python
from datasets import Dataset

dataset = Dataset.from_dict(my_dict)
dataset.to_json("dic_with_datasets.jsonl.gz", compression="gzip")
!cat dic_with_datasets.jsonl.gz
```
output
```
{"a":1,"b":1}
{"a":2,"b":2}
{"a":3,"b":3}
```
Note: I would expected to see binary data here

### Result with pandas
```python
import pandas as pd

df = pd.DataFrame(my_dict)
df.to_json("dic_with_pandas.jsonl.gz", lines=True, orient="records", compression="gzip")
!cat dic_with_pandas.jsonl.gz
```
output
```
4��a�dic_with_pandas.jsonl��VJT�2�QJ��\� ��g��yƵ���������)���
```
Note: It looks like binary data

## Expected results

I would have expected that the saved result with datasets would also be a binary file

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-4.18.0-193.70.1.el8_2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.11
- PyArrow version: 5.0.0


Dataset preview is not available (I think for all Hugging Face datasets)

Something isn't working

You're right, we have an issue today with the datasets preview. We're investigating.

It should be fixed now. Thanks for reporting.

## Dataset viewer issue for '*french_book_reviews*'

**Link:** https://huggingface.co/datasets/Abirate/french_book_reviews

**short description of the issue**
For my dataset, the dataset preview is no longer functional (it used to work: The dataset had been added the day before and it was fine...)
And, after looking over the datasets, I discovered that this issue affects all Hugging Face datasets (as of yesterday, December 23, 2021, around 10 p.m. (CET)).
**Am I the one who added this dataset** :  Yes

**Note**: here a screenshot showing the issue
![Dataset preview is not available for my dataset](https://user-images.githubusercontent.com/66887439/147333078-60734578-420d-4e91-8691-a90afeaa8948.jpg)

**And here for glue dataset :**

![Dataset preview is not available for other Hugging Face datasets(glue)](https://user-images.githubusercontent.com/66887439/147333492-26fa530c-befd-4992-8361-70c51397a25a.jpg)


Extend support for streaming datasets that use os.walk

Nice. I'll update the dataset viewer once merged, and test on these four datasets

This PR extends the support in streaming mode for datasets that use `os.walk`, by patching that function.

This PR adds support for streaming mode to datasets:
1. autshumato
1. code_x_glue_cd_code_to_text
1. code_x_glue_tc_nl_code_search_adv
1. nchlt

CC: @severo


Use `iter_files` instead of `str(Path(...)` in image dataset

`iter_archive` is about to support ZIP archives. I think we should use this no ?

see #3347 https://github.com/huggingface/datasets/pull/3379

I was interested in the support for isfile/dir in remote.

Anyway, `iter_files` will be available for community users.

I'm not a big fan of having two functions that do the same thing. What do you think ?

They do not do the same thing:
- One iterates over files in a directory
- The other I guess will iterate over the members of an archive

Makes sense ! Sounds good then - sorry for my misunderstanding

Note that `iter_archive` will be more performant for data streaming that `iter_files` thanks to the buffering so maybe in the future we can `iter_archive` for some of these datasets

Yes, @lhoestq I agree with you: once `iter_archive` supports zip files, it will be more suitable than `iter_files` for these 2 datasets.

Anyway, this PR also implements `isfile`/`isdir` in streaming mode, besides fixing `iter_files`. And I'm interested in having those in master.

Maybe, could we merge this PR into master and take note to refactor the datasets to use `iter_archive` once zip is supported?
Other option could be to split this PR into 2...

Use `iter_files` in the `beans` and the `cats_vs_dogs` dataset scripts as suggested by @albertvillanova.

Additional changes:
* Fix `iter_files` in `MockDownloadManager` (see this [CI error](https://app.circleci.com/pipelines/github/huggingface/datasets/9247/workflows/2657ff8a-b531-4fd9-a9fc-6541a72e8d83/jobs/57028))
* Add support for `os.path.isdir` and `os.path.isfile` in streaming (`os.path.isfile` is needed in `StreamingDownloadManager`'s `iter_files` to make `cats_vs_dogs` streamable)

TODO:
- [ ] add tests for `xisdir` and `xisfile`


Extend support for streaming datasets that use ET.parse

This PR extends the support in streaming mode for datasets that use `ET.parse`, by patching the function.

This PR adds support for streaming mode to datasets:
1. ami
1. assin
1. assin2
1. counter
1. enriched_web_nlg
1. europarl_bilingual
1. hyperpartisan_news_detection
1. polsum
1. qa4mre
1. quail
1. ted_talks_iwslt
1. udhr
1. web_nlg
1. winograd_wsc

CC: @severo


The rotten_tomatoes dataset of movie reviews contains some reviews in Spanish

Something isn't working

Hi @puzzler10, thanks for reporting.

Please note this dataset is not hosted on Hugging Face Hub. See:
https://github.com/huggingface/datasets/blob/c8f914473b041833fd47178fa4373cdcb56ac522/datasets/rotten_tomatoes/rotten_tomatoes.py#L42

If there are issues with the source data of a dataset, you should contact the data owners/creators instead. In the homepage associated with this dataset (http://www.cs.cornell.edu/people/pabo/movie-review-data/), you can find the authors of the dataset and how to contact them:
> If you have any questions or comments regarding this site, please send email to Bo Pang or Lillian Lee.

P.S.: Please also note that the example you gave of non-English review is in Portuguese (not Spanish). ;)

Maybe best to just put a quick sentence in the dataset description that highlights this?

## Describe the bug
See title. I don't think this is intentional and they probably should be removed. If they stay the dataset description should be at least updated to make it clear to the user.

## Steps to reproduce the bug
Go to the [dataset viewer](https://huggingface.co/datasets/viewer/?dataset=rotten_tomatoes) for the dataset, set the offset to 4160 for the train dataset, and scroll through the results. I found ones at index 4166 and 4173. There's others too (e.g. index 2888) but those two are easy to find like that.

## Expected results
English movie reviews only.

## Actual results
Example of a Spanish movie review (4173):
> "É uma pena que , mais tarde , o próprio filme abandone o tom de paródia e passe a utilizar os mesmos clichês que havia satirizado "


Decode images when iterating

If I iterate over a vision dataset, the images are not decoded, and the dictionary with the bytes is returned.

This PR enables image decoding in `Dataset.__iter__`

Close https://github.com/huggingface/datasets/issues/3473


Iterating over a vision dataset doesn't decode the images

Something isn't working

vision

Vision datasets

As discussed, I remember I set `decoded=False` here to avoid decoding just by iterating over examples of dataset. We wanted to decode only if the "audio" field (for Audio feature) was accessed.

> I set decoded=False here to avoid decoding just by iterating over examples of dataset. We wanted to decode only if the "audio" field (for Audio feature) was accessed

https://github.com/huggingface/datasets/pull/3430 will add more control to decoding, so I think it's OK to enable decoding in `__iter__` for now. After we merge the linked PR, the user can easily disable it again.

@mariosasko I wonder why there is no issue in `Audio` feature with decoding disabled in `__iter__`, whereas there is in `Image` feature.

Enabling decoding in `__iter__` will make fail Audio regressions tests: https://github.com/huggingface/datasets/runs/4608657230?check_suite_focus=true
```
=========================== short test summary info ============================
FAILED tests/features/test_audio.py::test_dataset_with_audio_feature_map_is_not_decoded
FAILED tests/features/test_audio.py::test_dataset_with_audio_feature_map_is_decoded
========================= 2 failed, 15 passed in 8.37s =========================

Please also note that the regression tests were implemented in accordance with the specifications:
- when doing a `map` (wich calls `__iter__`) of a function that doesn't access the audio field, the decoding should be disabled; this is why the decoding is disabled in `__iter__` (and only enabled in `__getitem__`).

>  I wonder why there is no issue in Audio feature with decoding disabled in __iter__, whereas there is in Image feature.

@albertvillanova Not sure if I understand this part. Currently, both the Image and the Audio feature don't decode data in `__iter__`, so their behavior is aligned there.

Therefore, this is not an issue, neither for Audio nor Image feature.

Could you please elaborate more on the expected use case? @lhoestq @NielsRogge

The expected use cases (in accordance with the specs: see #2324):
- decoding should be enabled when accessing a specific item (`__getitem__`)
- decoding should be disabled while iterating (`__iter__`) to allow preprocessing of non-audio/image features (like label or text, for example) using `.map`
-  decoding should be enabled in a `.map` only if the `.map` function accesses the audio/image feature (implemented using `LazyDict`)

For me it's not an issue, actually. I just (mistakenly) tried to iterate over a PyTorch Dataset instead of a PyTorch DataLoader,

i.e. I did this:

`batch = next(iter(train_ds)) `

whereas I actually wanted to do

`batch = next(iter(train_dataloader))`

and then it turned out that in the first case, the image was a string of bytes rather than a Pillow image, hence Quentin opened an issue.

Thanks @NielsRogge for the context.

So IMO everything is working as expected.

I'm closing this issue. Feel free to reopen it again if further changes of the specs should be addressed.

Thanks for the details :)

I still think that it's unexpected to get different results when doing
```python
for i in range(len(dataset)):
    sample = dataset[i]
```
and
```python
for sample in dataset:
    pass
```
even though I understand that if you don't need to decode the data, then decoding image or audio data when iterating is a waste of time and resources.

But in this case users can still drop the column that need decoding to get the full speed back no ?

## Describe the bug

If I load `mnist` and I iterate over the dataset, the images are not decoded, and the dictionary with the bytes is returned.

## Steps to reproduce the bug
```python
from datasets import load_dataset
import PIL

mnist = load_dataset("mnist", split="train")

first_image = mnist[0]["image"]
assert isinstance(first_image, PIL.PngImagePlugin.PngImageFile)  # passes
first_image = next(iter(mnist))["image"]
assert isinstance(first_image, PIL.PngImagePlugin.PngImageFile)  # fails
```

## Expected results

The image should be decoded, as a PIL Image

## Actual results

We get a dictionary
```
{'bytes': b'\x89PNG\r\n\x1a\n\x00..., 'path': None}
```

## Environment info

- `datasets` version: 1.17.1.dev0
- Platform: Darwin-20.6.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyArrow version: 6.0.0

The bug also exists in 1.17.0

## Investigation

I think the issue is that decoding is disabled in `__iter__`:

https://github.com/huggingface/datasets/blob/dfe5b73387c5e27de6a16b0caeb39d3b9ded66d6/src/datasets/arrow_dataset.py#L1651-L1661

Do you remember why it was disabled in the first place @albertvillanova ?
Also cc @mariosasko @NielsRogge


Fix `str(Path(...))` conversion in streaming on Linux. This should fix the streaming of the `beans` and `cats_vs_dogs` datasets.


Fix Tashkeela dataset to yield stripped text

This PR:
- Yields stripped text
- Fix path for Windows
- Adds license
- Adds more info in dataset card

Close bigscience-workshop/data_tooling#279


Fix rendering of docs

Minor fix in docs.

Currently, `ClassLabel` docstring rendering is not right.


Fix METEOR missing NLTK's omw-1.4

I also modified the doctest call to raise the exception that doctest may catch, instead of `doctest.UnexpectedException`.
This will make debugging easier if it happens again

NLTK 3.6.6 now requires `omw-1.4` to be downloaded for METEOR to work.
This should fix the CI on master


Add COCO dataset

The CI failures other than a missing dummy data file and missing fields in the card are unrelated to this PR.

Thanks a lot for this great work and fixing TFDS based script @mariosasko 🤗 will generate the dummy dataset and write the model card tomorrow!

@mariosasko I added the dataset card, I'm on the dummy data rn.

This PR adds the MS COCO dataset. Compared to the [TFDS](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/object_detection/coco.py) script, this implementation adds 8 additional configs to cover the tasks other than object detection.

Some notes:
* the data exposed by TFDS is contained in the `2014`, `2015`, `2017` and `2017_panoptic_segmentation` configs here
* I've updated `encode_nested_example` for easier handling of missing values (cc @lhoestq @albertvillanova; will add tests if you are OK with the changes in `features.py`)
* this implementation should fix https://github.com/huggingface/datasets/pull/3377#issuecomment-985559427

TODOs:
- [x] dataset card
- [ ] dummy data

cc @merveenoyan

Closes #2526


Push dataset infos.json to Hub

The change from `___` to `--` was allowed by https://github.com/huggingface/moon-landing/pull/1657

When doing `push_to_hub`, the feature types are lost (see issue https://github.com/huggingface/datasets/issues/3394).
This PR fixes this by also pushing a `dataset_infos.json` file to the Hub, that stores the feature types.

Other minor changes:
- renamed the `___` separator to `--`, since `--` is now disallowed in a name in the back-end.

I tested this feature with datasets like conll2003 that has feature types like `ClassLabel` that were previously lost.

Close https://github.com/huggingface/datasets/issues/3394

I would like to include this in today's release (though not mandatory), so feel free to comment/suggest changes


Add CRASS dataset

Hi Albert,
Thank you for your comments.
I hope I have uploaded my local git repo to include the dummy files and style reworkings.
Added YAML in Readme as well.

Please check again.

Hope it works now :)

Added crass dataset


Unable to load 'cnn_dailymail' dataset

bug

Something isn't working

duplicate

This issue or pull request already exists

Hi @talha1503, thanks for reporting.

It seems there is an issue with one of the data files hosted at Google Drive:
```
Google Drive - Quota exceeded

Sorry, you can't view or download this file at this time.

Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.
```

As you probably know, Hugging Face does not host the data, and in this case the data owner decided to host their data at Google Drive, which has quota limits.

Is there anything we could do, @lhoestq @mariosasko?

This looks related to https://github.com/huggingface/datasets/issues/996

## Describe the bug
I wanted to load cnn_dailymail dataset from huggingface datasets on Google Colab, but I am getting an error while loading it.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('cnn_dailymail', '3.0.0', ignore_verifications = True)
```

## Expected results
Expecting to load 'cnn_dailymail' dataset.

## Actual results
`NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0


struct.error: 'i' format requires -2147483648 <= number <= 2147483647

bug

Something isn't working

## Describe the bug
A clear and concise description of what the bug is.

using latest datasets=datasets-1.16.1-py3-none-any.whl
process my own multilingual dataset by following codes, and the number of rows in all dataset is 306000, the max_length of each sentence is 256:
![image](https://user-images.githubusercontent.com/30341159/146865779-3d25d011-1f42-4026-9e1b-76f6e1d172e9.png)

then I get this error:
![image](https://user-images.githubusercontent.com/30341159/146865844-e60a404c-5f3a-403c-b2f1-acd943b5cdb8.png)

I have seen the issue in #2134 and #2150, so I don't understand why latest repo still can't deal with big dataset.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: linux docker
- Python version: 3.6


Update swahili_news dataset

Update dataset with latest verion data files.

Fix #3462.
Close bigscience-workshop/data_tooling#107


Update swahili_news dataset

dataset request

Requesting to add a new dataset

Please note also: the HuggingFace version at https://huggingface.co/datasets/swahili_news is outdated. An updated version, with deduplicated text and official splits, can be found at https://zenodo.org/record/5514203.

## Adding a Dataset
- **Name:** swahili_news

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Related to:
- bigscience-workshop/data_tooling#107


Fix links in metrics description

Remove Markdown syntax for links in metrics description, as it is not properly rendered.

Related to #3437.


Don't encode lists as strings when using `Value("string")`

Following https://github.com/huggingface/datasets/pull/3456#event-5792250497 it looks like `datasets` can silently convert lists to strings using `str()`, instead of raising an error.

This PR fixes this and should fix the issue with WER showing low values if the input format is not right.


dataset.filter overwriting previously set dataset._indices values, resulting in the wrong elements being selected.

bug

Something isn't working

I think this is a duplicate of [#3190](https://github.com/huggingface/datasets/issues/3190)?

Upgrading the datasets version as per #3190 fixes this bug.
I'm Marking as closed.

## Describe the bug
When using dataset.select to select a subset of a dataset, dataset._indices are set to indicate which elements are now considered in the dataset.
The same thing happens when you shuffle the dataset; dataset._indices are set to indicate what the new order of the data is.
However, if you then use a dataset.filter, that filter interacts with those dataset._indices values in a non-intuitive manner.
https://huggingface.co/docs/datasets/_modules/datasets/arrow_dataset.html#Dataset.filter

Effectively, it looks like the original set of _indices were discared and overwritten by the set created during the filter operation.

I think this is actually an issue with how the map function handles dataset._indices. Ideally it should use the _indices it gets passed, and then return an updated _indices which reflect the map transformation applied to the starting _indices.

## Steps to reproduce the bug
```python
dataset = load_dataset('imdb', split='train', keep_in_memory=True)

dataset = dataset.shuffle(keep_in_memory=True)

dataset = dataset.select(range(0, 10), keep_in_memory=True)
print("initial 10 elements")
print(dataset['label'])   # -> [1, 1, 0, 1, 0, 0, 0, 1, 0, 0]

dataset = dataset.filter(lambda x: x['label'] == 0, keep_in_memory=True)
print("filtered 10 elements looking for label 0")
print(dataset['label'])  # -> [1, 1, 1, 1, 1, 1]
```

## Actual results
```
$ python indices_bug.py
initial 10 elements
[1, 1, 0, 1, 0, 0, 0, 1, 0, 0]
filtered 10 elements looking for label 0
[1, 1, 1, 1, 1, 1]
```

This code block first shuffles the dataset (to get a mix of label 0 and label 1).
Then it selects just the first 10 elements (the number of elements does not matter, 10 is just easy to visualize). The important part is that you select some subset of the dataset.
Finally, a filter is applied to pull out just the elements with `label == 0`.

The bug is that you cannot combine any dataset operation which sets the dataset._indices with filter.
In this case I have 2, shuffle and subset.

If you just use a single dataset._indices operation (in this case shuffle) the bug still shows up.

The shuffle sets the dataset._indices and then filter uses those indices in the map, then overwrites dataset._indices with the filter results.
```python
dataset = load_dataset('imdb', split='train', keep_in_memory=True)

dataset = dataset.shuffle(keep_in_memory=True)

dataset = dataset.filter(lambda x: x['label'] == 0, keep_in_memory=True)

dataset = dataset.select(range(0, 10), keep_in_memory=True)
print(dataset['label']) # -> [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
```

## Expected results
In an ideal world, the dataset filter would respect any dataset._indices values which had previously been set.

If you use dataset.filter with the base dataset (where dataset._indices has not been set) then the filter command works as expected.

## Environment info
Here are the commands required to rebuild the conda environment from scratch.
```
# create a virtual environment
conda create -n dataset_indices python=3.8 -y

# activate the virtual environment
conda activate dataset_indices

# install huggingface datasets
conda install datasets
```

<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Linux-5.11.0-41-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 3.0.0

### Full Conda Environment
```
$ conda env export
name: dasaset_indices
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=4.5=1_gnu
  - abseil-cpp=20210324.2=h2531618_0
  - aiohttp=3.8.1=py38h7f8727e_0
  - aiosignal=1.2.0=pyhd3eb1b0_0
  - arrow-cpp=3.0.0=py38h6b21186_4
  - attrs=21.2.0=pyhd3eb1b0_0
  - aws-c-common=0.4.57=he6710b0_1
  - aws-c-event-stream=0.1.6=h2531618_5
  - aws-checksums=0.1.9=he6710b0_0
  - aws-sdk-cpp=1.8.185=hce553d0_0
  - bcj-cffi=0.5.1=py38h295c915_0
  - blas=1.0=mkl
  - boost-cpp=1.73.0=h27cfd23_11
  - bottleneck=1.3.2=py38heb32a55_1
  - brotli=1.0.9=he6710b0_2
  - brotli-python=1.0.9=py38heb0550a_2
  - brotlicffi=1.0.9.2=py38h295c915_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.10.26=h06a4308_2
  - certifi=2021.10.8=py38h06a4308_0
  - cffi=1.14.6=py38h400218f_0
  - conllu=4.4.1=pyhd3eb1b0_0
  - cryptography=36.0.0=py38h9ce1e76_0
  - dataclasses=0.8=pyh6d0b6a4_7
  - dill=0.3.4=pyhd3eb1b0_0
  - double-conversion=3.1.5=he6710b0_1
  - et_xmlfile=1.1.0=py38h06a4308_0
  - filelock=3.4.0=pyhd3eb1b0_0
  - frozenlist=1.2.0=py38h7f8727e_0
  - gflags=2.2.2=he6710b0_0
  - glog=0.5.0=h2531618_0
  - gmp=6.2.1=h2531618_2
  - grpc-cpp=1.39.0=hae934f6_5
  - huggingface_hub=0.0.17=pyhd3eb1b0_0
  - icu=58.2=he6710b0_3
  - idna=3.3=pyhd3eb1b0_0
  - importlib-metadata=4.8.2=py38h06a4308_0
  - importlib_metadata=4.8.2=hd3eb1b0_0
  - intel-openmp=2021.4.0=h06a4308_3561
  - krb5=1.19.2=hac12032_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libboost=1.73.0=h3ff78a5_11
  - libcurl=7.80.0=h0b77cf5_0
  - libedit=3.1.20210910=h7f8727e_0
  - libev=4.33=h7f8727e_1
  - libevent=2.1.8=h1ba5d50_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.3.0=h5101ec6_17
  - libgomp=9.3.0=h5101ec6_17
  - libnghttp2=1.46.0=hce63b2e_0
  - libprotobuf=3.17.2=h4ff587b_1
  - libssh2=1.9.0=h1ba5d50_1
  - libstdcxx-ng=9.3.0=hd4cf53a_17
  - libthrift=0.14.2=hcc01f38_0
  - libxml2=2.9.12=h03d6c58_0
  - libxslt=1.1.34=hc22bd24_0
  - lxml=4.6.3=py38h9120a33_0
  - lz4-c=1.9.3=h295c915_1
  - mkl=2021.4.0=h06a4308_640
  - mkl-service=2.4.0=py38h7f8727e_0
  - mkl_fft=1.3.1=py38hd3c417c_0
  - mkl_random=1.2.2=py38h51133e4_0
  - multiprocess=0.70.12.2=py38h7f8727e_0
  - multivolumefile=0.2.3=pyhd3eb1b0_0
  - ncurses=6.3=h7f8727e_2
  - numexpr=2.7.3=py38h22e1b3c_1
  - numpy=1.21.2=py38h20f2e39_0
  - numpy-base=1.21.2=py38h79a1101_0
  - openpyxl=3.0.9=pyhd3eb1b0_0
  - openssl=1.1.1l=h7f8727e_0
  - orc=1.6.9=ha97a36c_3
  - packaging=21.3=pyhd3eb1b0_0
  - pip=21.2.4=py38h06a4308_0
  - py7zr=0.16.1=pyhd3eb1b0_1
  - pycparser=2.21=pyhd3eb1b0_0
  - pycryptodomex=3.10.1=py38h27cfd23_1
  - pyopenssl=21.0.0=pyhd3eb1b0_1
  - pyparsing=3.0.4=pyhd3eb1b0_0
  - pyppmd=0.16.1=py38h295c915_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.12=h12debd9_0
  - python-dateutil=2.8.2=pyhd3eb1b0_0
  - python-xxhash=2.0.2=py38h7f8727e_0
  - pyzstd=0.14.4=py38h7f8727e_3
  - re2=2020.11.01=h2531618_1
  - readline=8.1=h27cfd23_0
  - requests=2.26.0=pyhd3eb1b0_0
  - setuptools=58.0.4=py38h06a4308_0
  - six=1.16.0=pyhd3eb1b0_0
  - snappy=1.1.8=he6710b0_0
  - sqlite=3.36.0=hc218d9a_0
  - texttable=1.6.4=pyhd3eb1b0_0
  - tk=8.6.11=h1ccaba5_0
  - typing_extensions=3.10.0.2=pyh06a4308_0
  - uriparser=0.9.3=he6710b0_1
  - utf8proc=2.6.1=h27cfd23_0
  - wheel=0.37.0=pyhd3eb1b0_1
  - xxhash=0.8.0=h7f8727e_3
  - xz=5.2.5=h7b6447c_0
  - zipp=3.6.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7f8727e_4
  - zstd=1.4.9=haebb681_0
  - pip:
    - async-timeout==4.0.2
    - charset-normalizer==2.0.9
    - datasets==1.16.1
    - fsspec==2021.11.1
    - huggingface-hub==0.2.1
    - multidict==5.2.0
    - pandas==1.3.5
    - pyarrow==6.0.1
    - pytz==2021.3
    - pyyaml==6.0
    - tqdm==4.62.3
    - typing-extensions==4.0.1
    - urllib3==1.26.7
    - yarl==1.7.2

```


Fix duplicated tag in wikicorpus dataset card

CI is failing just because of empty sections - merging


Add CMU Graphics Lab Motion Capture dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

## Adding a Dataset
- **Name:** CMU Graphics Lab Motion Capture database
- **Description:** The database contains free motions which you can download and use.
- **Data:** http://mocap.cs.cmu.edu/
- **Motivation:** Nice motion capture dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


[WER] Better error message for wer

Hi ! I don't think this would solve this issue.
Currently it looks like there's a bug that converts the list `["hello it's nice"]` to a string `'["hello it's nice"]'` since this is what the metric expects as input. The conversion is done before the data are passed to `_compute()`.

This is `Value("string").encode_example` that is called to do the conversion. Since `str()` encoding is too permissive we should consider raising an error if the example is not a string (even though it can be converted to string).

I just opened https://github.com/huggingface/datasets/pull/3460 to fix that. It now raises an error instead of computing the wrong WER

Currently we have the following problem when using the WER. When the input format to the WER metric is wrong, instead of throwing an error message a word-error-rate is computed which is incorrect. E.g. when doing the following:

```python
from datasets import load_metric

wer = load_metric("wer")

target_str = ["hello this is nice", "hello the weather is bloomy"]
pred_str = [["hello it's nice"], ["hello it's the weather"]]

print("Wrong:", wer.compute(predictions=pred_str, references=target_str))
print("Correct", wer.compute(predictions=[x[0] for x in pred_str], references=target_str))
```
We get:
```
Wrong: 1.0
Correct 0.5555555555555556
```

meaning that we get a word-error rate for incorrectly passed input formats. We should raise an error here instead so that people don't spent hours fixing a model while it's their incorrect evaluation metric is the problem for a low WER.


Easier information editing

enhancement

New feature or request

generic discussion

Generic discussion on the library

Hi ! I guess you are talking about the dataset cards that are in this repository on github ?

I think github allows to submit a PR even for 1 line though the `Edit file` button on the page of the dataset card.

Maybe let's mention this in `CONTRIBUTING.md` ?

**Is your feature request related to a problem? Please describe.**
It requires a lot of effort to improve a datasheet.

**Describe the solution you'd like**
UI or at least a link to the place where the code that needs to be edited is (and an easy way to edit this code directly from the site, without cloning, branching, makefile etc.)

**Describe alternatives you've considered**
The current Ux is to have the 8 steps for contribution while One just wishes to change a line a type etc.


Fix iter_archive generator

This PR:
- Adds tests to DownloadManager and StreamingDownloadManager `iter_archive` for both path and file inputs
- Fixes bugs in `iter_archive` introduced in:
  - #3443

Fix #3453.


ValueError while iter_archive

bug

Something isn't working

## Describe the bug
After the merge of:
- #3443

the method `iter_archive` throws a ValueError:
```
ValueError: read of closed file
```

## Steps to reproduce the bug
```python
for path, file in dl_manager.iter_archive(archive_path):
    pass
```


enhancement

New feature or request

why the stratify option is omitted from test_train_split function?

is there any other way implement the stratify option while splitting the dataset? as it is important point to be considered while splitting the dataset.


[Staging] Update dataset repos automatically on the Hub

do keep us updated on how it's going in staging! cc @SBrandeis

Sure ! For now it works smoothly. We'll also do a new release today.

I can send you some repos to explore on staging, in case you want to see how they look like after being updated.
For example [swahili_news](https://moon-staging.huggingface.co/datasets/swahili_news/tree/main)

Let's have a script that updates the dataset repositories on staging for now. This way we can make sure it works fine before going in prod.

Related to https://github.com/huggingface/datasets/issues/3341

The script runs on each commit on `master`. It checks the datasets that were changed, and it pushes the changes to the corresponding repositories on the Hub.

If there's a new dataset, then a new repository is created.

If the commit is a new release of `datasets`, it also pushes the tag to all the repositories.


Unexpected behavior doing Split + Filter

bug

Something isn't working

Hi ! This is an issue with `datasets` 1.12. Sorry for the inconvenience. Can you update to `>=1.13` ?
see https://github.com/huggingface/datasets/issues/3190

Maybe we should also backport the bug fix to `1.12` (in a new version `1.12.2`)

## Describe the bug
I observed unexpected behavior when applying 'train_test_split' followed by 'filter' on dataset. Elements of the training dataset eventually end up in the test dataset (after applying the 'filter')

## Steps to reproduce the bug
```
from datasets import Dataset
import pandas as pd
dic = {'x': [1,2,3,4,5,6,7,8,9], 'y':['q','w','e','r','t','y','u','i','o']}
df = pd.DataFrame.from_dict(dic)
dataset = Dataset.from_pandas(df)
split_dataset = dataset.train_test_split(test_size=0.5, shuffle=False, seed=42)
train_dataset = split_dataset["train"]
eval_dataset = split_dataset["test"]
eval_dataset_2 = eval_dataset.filter(lambda example: example['x'] % 2 == 0)
print( eval_dataset['x'])
print(eval_dataset_2['x'])
```
One observes that elements in eval_dataset2 are actually coming from the training dataset...

## Expected results
The expected results would be that the filtered eval dataset would only contain elements from the original eval dataset.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Windows 10
- Python version: 3.7
- PyArrow version: 5.0.0


Add `__add__()`, `__iadd__()` and similar to `Dataset` class

enhancement

New feature or request

generic discussion

Generic discussion on the library

I was going through the codebase, and I believe the implementation of __add__() and __iadd__() will be similar to concatenate_datasets() after the elimination of code for arguments other than the list of datasets (info, split, axis).
(Assuming elimination of axis means concatenating over axis 1.)

**Is your feature request related to a problem? Please describe.**
No.

**Describe the solution you'd like**
I would like to be able to concatenate datasets as follows:
```python
>>> dataset["train"] += dataset["validation"]
```

... instead of using `concatenate_datasets()`:
```python
>>> raw_datasets["train"] = concatenate_datasets([raw_datasets["train"], raw_datasets["validation"]])
>>> del raw_datasets["validation"]
```

**Describe alternatives you've considered**
Well, I have considered `concatenate_datasets()` 😀

**Additional context**
N.a.


JSONDecodeError with HuggingFace dataset viewer

Hi ! I think the issue comes from the dataset_infos.json file: it has the "flat" field twice.

Can you try deleting this file and regenerating it please ?

Thanks! That fixed that, but now I am getting:
Server Error
Status code:   400
Exception:     KeyError
Message:       'feature'

I checked the dataset_infos.json and pubmed_neg.py script, I don't use 'feature' anywhere as a key. Is the dataset viewer expecting that I do?

It seems that the `feature` key is missing from some feature type definition in your dataset_infos.json:
```json
			"tokens": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence"
			},
			"tags": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence"
			}
```
They should be
```json
			"tokens": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence"
                                 "feature": {"dtype": "string", "id": null, "_type": "Value"}
			},
			"tags": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence",
                                 "feature": {"num_classes": 5, "names": ["-", "S", "H", "N", "C"], "names_file": null, "id": null, "_type": "ClassLabel"}
			}
```

Note that you can generate the dataset_infos.json automatically to avoid mistakes:
```bash
datasets-cli test ./path/to/dataset --save_infos
```

## Dataset viewer issue for 'pubmed_neg'

**Link:** https://huggingface.co/datasets/IGESML/pubmed_neg

I am getting the error:
Status code:   400
Exception:     JSONDecodeError
Message:       Expecting property name enclosed in double quotes: line 61 column 2 (char 1202)

I have checked all files - I am not using single quotes anywhere. Not sure what is causing this issue.

Am I the one who added this dataset ? Yes