fix-3264-change-download-urls

There seems to be problems with datasets metadata, of which I dont have access to. I think one of the datasets is from reddit. Can anyone help?

Hello @LashaO , I think the errors were caused by `_DATA_FILES` in `definite_pronoun_resolution.py`. Here are details of the test error.
```
self = BuilderConfig(name='plain_text', version=1.0.0, data_dir=None, data_files={'train': 'train.c.txt', 'test': 'test.c.txt'}, description='Plain text import of the Definite Pronoun Resolution Dataset.')

    def __post_init__(self):
        # The config name is used to name the cache directory.
        invalid_windows_characters = r"<>:/\|?*"
        for invalid_char in invalid_windows_characters:
            if invalid_char in self.name:
                raise InvalidConfigName(
                    f"Bad characters from black list '{invalid_windows_characters}' found in '{self.name}'. "
                    f"They could create issues when creating a directory for this config on Windows filesystem."
                )
        if self.data_files is not None and not isinstance(self.data_files, DataFilesDict):
>           raise ValueError(f"Expected a DataFilesDict in data_files but got {self.data_files}")
E           ValueError: Expected a DataFilesDict in data_files but got {'train': 'train.c.txt', 'test': 'test.c.txt'}
```

Hi ! Thanks for the fixes :)

Instead of uploading the `definite_pronoun_resolution` data files in this PR, maybe we can just update the URL ?
The old url was http://www.hlt.utdallas.edu/~vince/data/emnlp12/train.c.txt, but now it's https://www.hlt.utdallas.edu/~vince/data/emnlp12/train.c.txt (https instead of http)

Actually the bad certificate creates an issue with the download
```python
import datasets
datasets.DownloadManager().download("https://www.hlt.utdallas.edu/~vince/data/emnlp12/train.c.txt")
# raises: ConnectionError: Couldn't reach https://www.hlt.utdallas.edu/~vince/data/emnlp12/train.c.txt
```

Let me see if I can fix that

I uploaded them to these URLs, feel free to use them instead of having the text files here in the PR :)
https://s3.amazonaws.com/datasets.huggingface.co/definite_pronoun_resolution/train.c.txt
https://s3.amazonaws.com/datasets.huggingface.co/definite_pronoun_resolution/test.c.txt

Thank you for the tips! Having a busy week so anyone willing to commit the suggestions is welcome. Else, I will try to get back to this in a while.

@LashaO Thanks for working on this. Yes, I'll take over as we already have a request to fix the URL of the Jeopardy! dataset in a separate issue.

~~Still have to fix the error in the dummy data test of the WikiAuto dataset (so please don't merge).~~ Done! Ready for merging.


Checksum error for kilt_task_wow

bug

Something isn't working

Using `dataset = load_dataset("kilt_tasks", "wow", ignore_verifications=True)` may fix it, but I do not think it is a elegant solution.

Hi @slyviacassell, thanks for reporting.

Yes, there is an issue with the checksum verification. I'm fixing it.

And as you pointed out, in the meantime, you can circumvent the problem by passing `ignore_verifications=True`.

## Describe the bug
Checksum failed when downloads kilt_tasks_wow. See error output for details.

## Steps to reproduce the bug
```python
import datasets
datasets.load_datasets('kilt_tasks','wow')
```

## Expected results
Download successful

## Actual results
```
Downloading and preparing dataset kilt_tasks/wow (download: 72.07 MiB, generated: 61.82 MiB, post-processed: Unknown size, total: 133.89 MiB) to /root/.cache/huggingface/datasets/kilt_tasks/wow/1.0.0/57dc8b2431e76637e0c6ef79689ca4af61ed3a330e2e0cd62c8971465a35db3a...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 5121.25it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1527.42it/s]
Traceback (most recent call last):
  File "kilt_wow.py", line 30, in <module>
    main()
  File "kilt_wow.py", line 27, in main
    train, dev, test = dataset.generate_k_shot_data(k=32, seed=seed, path="../data/")
  File "/workspace/projects/CrossFit/tasks/fewshot_gym_dataset.py", line 79, in generate_k_shot_data
    dataset = self.load_dataset()
  File "kilt_wow.py", line 21, in load_dataset
    return datasets.load_dataset('kilt_tasks','wow')
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 679, in _download_and_prepare
    verify_checksums(
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['http://dl.fbaipublicfiles.com/KILT/wow-train-kilt.jsonl', 'http://dl.fbaipublicfiles.com/KILT/wow-dev-kilt.jsonl']
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-4.15.0-161-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyArrow version: 4.0.1


Downloading URL change for WikiAuto Manual, jeopardy and definite_pronoun_resolution

Something isn't working

#take
I am willing to fix this. Links can be replaced for WikiAuto Manual and jeopardy with new ones provided by authors.

As for the definite_pronoun_resolution URL, a certificate error seems to be preventing a download. I have the files on my local machine. I can include them in the dataset folder as the files are <1MB in size total.

> #take I am willing to fix this. Links can be replaced for WikiAuto Manual and jeopardy.
>
> As for the definite_pronoun_resolution URL, a certificate error seems to be preventing a download. I have the files on my local machine. Anyone has opinions on whether it is preferable for me to host them somewhere (e.g. personal GDrive account) or upload them to the dataset folder directly and use github raw URLs? The files are <1MB in size.

I am planning to fix it next few days. But my to-do list is full and I do not have the cache of definite_pronoun_resolution. I am glad that you can take this. Thanks a lot!

No problem, buddy! Will submit a PR over this weekend.

## Describe the bug
- WikiAuto Manual
The original manual datasets with the following downloading URL in this [repository](https://github.com/chaojiang06/wiki-auto) was [deleted](https://github.com/chaojiang06/wiki-auto/commit/0af9b066f2b4e02726fb8a9be49283c0ad25367f) by the author.
```
https://github.com/chaojiang06/wiki-auto/raw/master/wiki-manual/train.tsv
```

- jeopardy
The downloading URL for jeopardy may move from
```
http://skeeto.s3.amazonaws.com/share/JEOPARDY_QUESTIONS1.json.gz
```
 to
```
https://drive.google.com/file/d/0BwT5wj_P7BKXb2hfM3d2RHU1ckE/view?resourcekey=0-1abK4cJq-mqxFoSg86ieIg
```

- definite_pronoun_resolution
The following downloading URL for definite_pronoun_resolution cannot be reached for some reasons.
```
http://www.hlt.utdallas.edu/~vince/data/emnlp12/train.c.txt
```

## Steps to reproduce the bug
```python
import datasets
datasets.load_datasets('wiki_auto','manual')
datasets.load_datasets('jeopardy')
datasets.load_datasets('definite_pronoun_resolution')
```

## Expected results
Download successfully

## Actual results
- WikiAuto Manual
```
Downloading and preparing dataset wiki_auto/manual (download: 151.65 MiB, generated: 155.97 MiB, post-processed: Unknown size, total: 307.61 MiB) to /root/.cache/huggingface/datasets/wiki_auto/manual/1.0.0/5ffdd9fc62422d29bd02675fb9606f77c1251ee17169ac10b143ce07ef2f4db8...
  0%|                                                                                | 0/3 [00:00<?, ?it/s]Traceback (most recent call last):
  File "wiki_auto.py", line 43, in <module>
    main()
  File "wiki_auto.py", line 40, in main
    train, dev, test = dataset.generate_k_shot_data(k=16, seed=seed, path="../data/")
  File "/workspace/projects/CrossFit/tasks/fewshot_gym_dataset.py", line 24, in generate_k_shot_data
    dataset = self.load_dataset()
  File "wiki_auto.py", line 34, in load_dataset
    return datasets.load_dataset('wiki_auto', 'manual')
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 675, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/root/.cache/huggingface/modules/datasets_modules/datasets/wiki_auto/5ffdd9fc62422d29bd02675fb9606f77c1251ee17169ac10b143ce07ef2f4db8/wiki_auto.py", line 193, in _split_generators
    data_dir = dl_manager.download_and_extract(my_urls)
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 196, in download
    downloaded_path_or_paths = map_nested(
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 216, in map_nested
    mapped = [
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 217, in <listcomp>
    _single_map_nested((function, obj, types, None, True))
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 152, in _single_map_nested
    return function(data_struct)
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 295, in cached_path
    output_path = get_from_cache(
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 592, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://github.com/chaojiang06/wiki-auto/raw/master/wiki-manual/train.tsv
```
- jeopardy
```
Using custom data configuration default
Downloading and preparing dataset jeopardy/default (download: 12.13 MiB, generated: 34.46 MiB, post-processed: Unknown size, total: 46.59 MiB) to /root/.cache/huggingface/datasets/jeopardy/default/0.1.0/25ee3e4a73755e637b8810f6493fd36e4523dea3ca8a540529d0a6e24c7f9810...
Traceback (most recent call last):
  File "jeopardy.py", line 45, in <module>
    main()
  File "jeopardy.py", line 42, in main
    train, dev, test = dataset.generate_k_shot_data(k=32, seed=seed, path="../data/")
  File "/workspace/projects/CrossFit/tasks/fewshot_gym_dataset.py", line 79, in generate_k_shot_data
    dataset = self.load_dataset()
  File "jeopardy.py", line 36, in load_dataset
    return datasets.load_dataset("jeopardy")
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 675, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/root/.cache/huggingface/modules/datasets_modules/datasets/jeopardy/25ee3e4a73755e637b8810f6493fd36e4523dea3ca8a540529d0a6e24c7f9810/jeopardy.py", line 72, in _split_generators
    filepath = dl_manager.download_and_extract(_DATA_URL)
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 196, in download
    downloaded_path_or_paths = map_nested(
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 206, in map_nested
    return function(data_struct)
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 295, in cached_path
    output_path = get_from_cache(
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 594, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach http://skeeto.s3.amazonaws.com/share/JEOPARDY_QUESTIONS1.json.gz
```
- definite_pronoun_resolution
```
Downloading and preparing dataset definite_pronoun_resolution/plain_text (download: 222.12 KiB, generated: 239.12 KiB, post-processed: Unknown size, total: 461.24 KiB) to /root/.cache/huggingface/datasets/definite_pronoun_resolution/plain_text/1.0.0/35a1dfd4fba4afb8ba226cbbb65ac7cef0dd3cf9302d8f803740f05d2f16ceff...
  0%|                                                                                | 0/2 [00:00<?, ?it/s]Traceback (most recent call last):
  File "definite_pronoun_resolution.py", line 37, in <module>
    main()
  File "definite_pronoun_resolution.py", line 34, in main
    train, dev, test = dataset.generate_k_shot_data(k=32, seed=seed, path="../data/")
  File "/workspace/projects/CrossFit/tasks/fewshot_gym_dataset.py", line 79, in generate_k_shot_data
    dataset = self.load_dataset()
  File "definite_pronoun_resolution.py", line 28, in load_dataset
    return datasets.load_dataset('definite_pronoun_resolution')
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 675, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/root/.cache/huggingface/modules/datasets_modules/datasets/definite_pronoun_resolution/35a1dfd4fba4afb8ba226cbbb65ac7cef0dd3cf9302d8f803740f05d2f16ceff/definite_pronoun_resolution.py", line 76, in _split_generators
    files = dl_manager.download_and_extract(
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 196, in download
    downloaded_path_or_paths = map_nested(
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 216, in map_nested
    mapped = [
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 217, in <listcomp>
    _single_map_nested((function, obj, types, None, True))
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 152, in _single_map_nested
    return function(data_struct)
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 295, in cached_path
    output_path = get_from_cache(
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 594, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach http://www.hlt.utdallas.edu/~vince/data/emnlp12/train.c.txt
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-4.15.0-161-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyArrow version: 4.0.1


FET DATA

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


asserts replaced with exception for image classification task, csv, json

Fixes for csv, json in io module and image_classification task with tests referenced in https://github.com/huggingface/datasets/issues/3171


Scifi_TV_Shows: Having trouble getting viewer to find appropriate files

dataset-viewer

Related to the dataset viewer on huggingface.co

Hi ! I think this is because `iter_archive` doesn't support ZIP files yet. See https://github.com/huggingface/datasets/issues/3272

You can navigate into the archive this way instead:
```python
# in split_generators
data_dir = dl_manager.download_and_extract(url)
train_filepath = os.path.join(data_dir, "all-sci-fi-data-train.txt")
return [
        datasets.SplitGenerator(
            name=datasets.Split.TRAIN,
            gen_kwargs={
                "filepath": train_filepath,
            },
        ),
...
])

# in generate_examples
with open(filepath, encoding="utf-8") as f:
    ...
```

## Dataset viewer issue for '*Science Fiction TV Show Plots Corpus  (Scifi_TV_Shows)*'

**Link:** [link](https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows)

I tried adding both a script (https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows/blob/main/Scifi_TV_Shows.py) and some dummy examples (https://huggingface.co/datasets/lara-martin/Scifi_TV_Shows/tree/main/dummy), but the viewer still has a 404 error ("Not found. Maybe the cache is missing, or maybe the ressource does not exist."). I'm not sure what to try next. Thanks in advance!

Am I the one who added this dataset? Yes


Fix ConnectionError in Scielo dataset

The CI error is unrelated to the change.

This PR:
* allows 403 status code in HEAD requests to S3 buckets to fix the connection error in the Scielo dataset (instead of `url`, uses `response.url` to check the URL of the final endpoint)
* makes the Scielo dataset streamable

Fixes #3255.


Updating details of IRC disentanglement data

I was pleasantly surprised to find that someone had already added my dataset to the huggingface library, but some details were missing or incorrect. This PR fixes the documentation.


Reload dataset that was already downloaded with `load_from_disk` from cloud storage

enhancement

New feature or request

`load_from_disk` downloads the dataset to a temporary directory without checking if the dataset has already been downloaded once.
It would be nice to have some sort of caching for datasets downloaded this way. This could leverage the fingerprint of the dataset that was saved in the `state.json` file.


Use f-strings for string formatting

good first issue

Good for newcomers

Hi, I would be glad to help with this. Is there anyone else working on it?

Hi, I would be glad to work on this too.

Hi @Carlosbogo,

would you be interested in replacing the `.format` and `%` syntax with f-strings in the modules in the `datasets` directory since @Mehdi2402 has opened a PR that does that for all the other directories?

Oh I see. I will be glad to help with the `datasets` directory then.

f-strings offer better readability/performance than `str.format` and `%`, so we should use them in all places in our codebase unless there is good reason to keep the older syntax.

> **NOTE FOR CONTRIBUTORS**: To avoid large PRs and possible merge conflicts, do 1-3 modules per PR. Also, feel free to ignore the files located under `datasets/*`.


asserts replaced by exception for text classification task with test.

Haha it looks like you got the chance of being reviewed twice at the same time and got the same suggestion twice x)
Anyway it's all good now so we can merge !

I have replaced only a single assert in text_classification.py along with a unit test to verify an exception is raised based on https://github.com/huggingface/datasets/issues/3171 .

I would like to first understand the code contribution workflow. So keeping the change to a single file rather than making too many changes. Once this gets approved, I will look into the rest.

Thanks.


SciELO dataset ConnectionError

bug

Something isn't working

## Describe the bug
I get `ConnectionError` when I am trying to load the SciELO dataset.

When I try the URL with `requests` I get:
```
>>> requests.head("https://ndownloader.figstatic.com/files/14019287")
<Response [302]>
```
And as far as I understand redirections in `datasets` are not supported for downloads.
https://github.com/huggingface/datasets/blob/807341d0db0728073ab605c812c67f927d148f38/datasets/scielo/scielo.py#L45

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("scielo", "en-es")
```

## Expected results
Download SciELO dataset and load Dataset object

## Actual results

```
Downloading and preparing dataset scielo/en-es (download: 21.90 MiB, generated: 68.45 MiB, post-processed: Unknown size, total: 90.35 MiB) to /Users/test/.cache/huggingface/datasets/scielo/en-es/1.0.0/7e05d55a20257efeb9925ff5de65bd4884fc6ddb6d765f1ea3e8860449d90e0e...
Traceback (most recent call last):
  File "scielo.py", line 3, in <module>
    dataset = load_dataset("scielo", "en-es")
  File "../lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "../lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "../lib/python3.8/site-packages/datasets/builder.py", line 675, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/test/.cache/huggingface/modules/datasets_modules/datasets/scielo/7e05d55a20257efeb9925ff5de65bd4884fc6ddb6d765f1ea3e8860449d90e0e/scielo.py", line 77, in _split_generators
    data_dir = dl_manager.download_and_extract(_URLS[self.config.name])
  File "../lib/python3.8/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "../lib/python3.8/site-packages/datasets/utils/download_manager.py", line 196, in download
    downloaded_path_or_paths = map_nested(
  File "../lib/python3.8/site-packages/datasets/utils/py_utils.py", line 206, in map_nested
    return function(data_struct)
  File "../lib/python3.8/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "../lib/python3.8/site-packages/datasets/utils/file_utils.py", line 295, in cached_path
    output_path = get_from_cache(
  File "../lib/python3.8/site-packages/datasets/utils/file_utils.py", line 594, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://ndownloader.figstatic.com/files/14019287

```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.12
- PyArrow version: 6.0.0


Update xcopa dataset (fix checksum issues + add translated data)

The CI failures are unrelated to the changes (missing fields in the readme and the CER metric error fixed in #3252).

This PR updates the checksums (as reported [here](https://discuss.huggingface.co/t/how-to-load-dataset-locally/11601/2)) of the `xcopa` dataset. Additionally, it adds new configs that hold the translated data of the original set of configs. This data was not available at the time of adding this dataset to the lib.


Something isn't working

Hi,

thanks for reporting and providing a minimal reproducible example.

This line of the PR I've linked in our discussion on the Forum will add support for `None` values:
https://github.com/huggingface/datasets/blob/a53de01842aac65c66a49b2439e18fa93ff73ceb/src/datasets/features/features.py#L835

I expect that PR to be merged soon.

## Describe the bug
`GeneratorBasedBuilder` does not support `None` values.

## Steps to reproduce the bug
See [this repository](https://github.com/pavel-lexyr/huggingface-datasets-bug-reproduction) for minimal reproduction.

## Expected results
Dataset is initialized with a `None` value in the `value` column.

## Actual results
```
Traceback (most recent call last):
  File "main.py", line 3, in <module>
    datasets.load_dataset("./bad-data")
  File ".../datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File ".../datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File ".../datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File ".../datasets/builder.py", line 1103, in _prepare_split
    example = self.info.features.encode_example(record)
  File ".../datasets/features/features.py", line 1033, in encode_example
    return encode_nested_example(self, example)
  File ".../datasets/features/features.py", line 808, in encode_nested_example
    return {
  File ".../datasets/features/features.py", line 809, in <dictcomp>
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File ".../datasets/features/features.py", line 855, in encode_nested_example
    return schema.encode_example(obj)
  File ".../datasets/features/features.py", line 299, in encode_example
    return float(value)
TypeError: float() argument must be a string or a number, not 'NoneType'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-5.4.0-81-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 6.0.0


Fix failing CER metric test in CI after update

Fixes the [failing CER metric test](https://app.circleci.com/pipelines/github/huggingface/datasets/8644/workflows/79816553-fa2f-4756-b022-d5937f00bf7b/jobs/53298) in CI by adding support for `jiwer==2.3.0`, which was released yesterday. Also, I verified that all the tests in `metrics/cer/test_cer.py` pass after the change, so the results should be the same irrespective of the `jiwer` version.


Add ETHICS dataset

This PR adds the ETHICS dataset, including all 5 sub-datasets.
From https://arxiv.org/abs/2008.02275


Fix streaming for id_newspapers_2018

To be compatible with streaming, this dataset must use `dl_manager.iter_archive` since the data are in a .tgz file


Stream from Google Drive and other hosts

I just tried some datasets and noticed that `spider` is not working for some reason (the compression type is not recognized), resulting in FileNotFoundError. I can take a look tomorrow

I'm fixing the remaining files based on TAR archives

Streaming from Google Drive is a bit more challenging than the other host we've been supporting:
- the download URL must be updated to add the confirm token obtained by HEAD request
- it requires to use cookies to keep the connection alive
- the URL doesn't tell any information about whether the file is compressed or not

Therefore I did two things:
- I added a step for URL and headers/cookies preparation in the StreamingDownloadManager
- I added automatic compression type inference by reading the [magic number](https://en.wikipedia.org/wiki/List_of_file_signatures)

This allows to do do fancy things like
```python
from datasets.utils.streaming_download_manager import StreamingDownloadManager, xopen, xjoin, xglob

# zip file containing a train.tsv file
url = "https://drive.google.com/uc?export=download&id=1k92sUfpHxKq8PXWRr7Y5aNHXwOCNUmqh"

extracted = StreamingDownloadManager().download_and_extract(url)
for inner_file in xglob(xjoin(extracted, "*.tsv")):
    with xopen(inner_file) as f:
        # streaming starts here
        for line in f:
            print(line)
```

This should make around 80 datasets streamable. It concerns those hosted on Google Drive but also any dataset for which the URL doesn't give any information about compression. Here is the full list:

```
amazon_polarity, ami, arabic_billion_words, ascent_kb, asset, big_patent, billsum, capes, cmrc2018, cnn_dailymail,
code_x_glue_cc_code_completion_token, code_x_glue_cc_code_refinement, code_x_glue_cc_code_to_code_trans,
code_x_glue_tt_text_to_text, conll2002, craigslist_bargains, dbpedia_14, docred, ehealth_kd, emo, euronews, germeval_14,
gigaword, grail_qa, great_code, has_part, head_qa, health_fact, hope_edi, id_newspapers_2018,
igbo_english_machine_translation, irc_disentangle, jfleg, jnlpba, journalists_questions, kor_ner, linnaeus, med_hop, mrqa,
mt_eng_vietnamese, multi_news, norwegian_ner, offcombr, offenseval_dravidian, para_pat, peoples_daily_ner, pn_summary,
poleval2019_mt, pubmed_qa, qangaroo, reddit_tifu, refresd, ro_sts_parallel, russian_super_glue, samsum, sberquad, scielo,
search_qa, species_800, spider, squad_adversarial, tamilmixsentiment, tashkeela, ted_talks_iwslt, trec, turk, turkish_ner,
twi_text_c3, universal_morphologies, web_of_science, weibo_ner, wiki_bio, wiki_hop, wiki_lingua, wiki_summary, wili_2018,
wisesight1000, wnut_17, yahoo_answers_topics, yelp_review_full, yoruba_text_c3
```

Some of them may not work if the host doesn't support HTTP range requests for example

Fix https://github.com/huggingface/datasets/issues/2742
Fix https://github.com/huggingface/datasets/issues/3188


Loading big json dataset raises pyarrow.lib.ArrowNotImplementedError

Something isn't working

Hi,

this issue is similar to https://github.com/huggingface/datasets/issues/3093, so you can either use the solution provided there or try to load the data in one chunk (you can control the chunk size by specifying the `chunksize` parameter (`int`) in `load_dataset`).

@lhoestq Is this worth opening an issue on Jira? Basically, PyArrow doesn't allow casts that change the order of the struct fields because they treat `pa.struct` as an ordered sequence. Reordering fields manually in Python is probably too slow, so I think this needs to be fixed by them to be usable on our side.

I agree I would expect PyArrow to be able to handle this, do you want to open the issue @mariosasko ?
Although maybe it's possible to fix struct casting on our side without hurting performance too much, if it's simply a matter of reordering the arrays in the StructArray

## Describe the bug
When trying to create a dataset from a json file with around 25MB, the following error is raised `pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<b: int64, c: int64> to struct using function cast_struct`

Splitting the big file into smaller ones and then loading it with the `load_dataset` method did also not work.

Creating a pandas dataframe from it and then loading it with `Dataset.from_pandas` works

## Steps to reproduce the bug
```python
load_dataset("json", data_files="test.json")
```

test.json ~25MB
```json
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
{"a": {"c": 8, "b": 5}}
...
```

working.json ~160bytes
```json
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
{"a": {"c": 8, "b": 5}}
```

## Expected results
It should load the dataset from the json file without error.

## Actual results
It raises Exception `pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<b: int64, c: int64> to struct using function cast_struct`

```
Traceback (most recent call last):
  File "/Users/m/workspace/xxx/project/main.py", line 60, in <module>
    dataset = load_dataset("json", data_files="result.json")
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/datasets/load.py", line 1627, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/datasets/builder.py", line 1159, in _prepare_split
    writer.write_table(table)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/datasets/arrow_writer.py", line 428, in write_table
    pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
  File "pyarrow/table.pxi", line 1685, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 630, in pyarrow.lib._sanitize_arrays
  File "pyarrow/array.pxi", line 338, in pyarrow.lib.asarray
  File "pyarrow/table.pxi", line 304, in pyarrow.lib.ChunkedArray.cast
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/pyarrow/compute.py", line 309, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 528, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 327, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 120, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<b: int64, c: int64> to struct using function cast_struct
```

## Environment info
- `datasets` version: 1.14.0
- Platform: macOS-12.0.1-arm64-arm-64bit
- Python version: 3.9.7
- PyArrow version: 6.0.0


[tiny] fix typo in stream docs


Fix load_from_disk temporary directory

`load_from_disk` uses `tempfile.TemporaryDirectory()` instead of our `get_temporary_cache_files_directory()` function. This can cause the temporary directory to be deleted before the dataset object is garbage collected.

In practice, it prevents anyone from using methods like `shuffle` on a dataset loaded this way, because it can't write the shuffled indices in a directory that doesn't exist anymore.

In this PR I switch to using `get_temporary_cache_files_directory()` and I update the tests.

cc @mariosasko since you worked on `get_temporary_cache_files_directory()`


Fix filter method for batched=True


Remove redundant isort module placement

`isort` can place modules by itself from [version 5.0.0](https://pycqa.github.io/isort/docs/upgrade_guides/5.0.0.html#module-placement-changes-known_third_party-known_first_party-default_section-etc) onwards, making the `known_first_party` and `known_third_party` fields in `setup.cfg` redundant (this is why our CI works, even though we haven't touched these options in a while).


Adding ANERcorp-CAMeLLab dataset

dataset request

Requesting to add a new dataset

Adding ANERcorp dataset

## Adding a Dataset
- **Name:** *ANERcorp-CAMeLLab*
- **Description:** *Since its creation in 2008, the ANERcorp dataset (Benajiba & Rosso, 2008) has been a standard reference used by Arabic named entity recognition researchers around the world. However, over time, this dataset was copied over from user to user, modified slightly here and there, and split in many different configurations that made it hard to compare fairly across papers and systems.

In 2020, a group of researchers from CAMeL Lab (Habash, Alhafni and Oudah), and Mind Lab (Antoun and Baly) met with the creator of the corpus, Yassine Benajiba, to consult with him and collectively agree on an exact split, and accepted minor corrections from the original dataset. Bashar Alhafni from CAMeL Lab working with Nizar Habash implemented the decisions provided in this release.*

- **Paper:** *(a) Benajiba, Yassine, Paolo Rosso, and José Miguel Benedí Ruiz. "Anersys: An Arabic named entity recognition system based on maximum entropy." In International Conference on Intelligent Text Processing and Computational Linguistics, pp. 143-153. Springer, Berlin, Heidelberg, 2007.

(b)Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. "CAMeL Tools: An Open Source Python Toolkit, for Arabic Natural Language Processing." In Proceedings of the Conference on Language Resources and Evaluation (LREC 2020), Marseille, 2020.*
- **Data:** *https://camel.abudhabi.nyu.edu/anercorp/*
- **Motivation:** This is the standard dataset for evaluating NER performance in Arabic*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Swap descriptions of v1 and raw-v1 configs of WikiText dataset and fix metadata

Fix #3237.


Couldn't reach data file for disaster_response_messages

dataset bug

A bug in a dataset script provided in the library

It looks like the dataset isn't available anymore on appen.com

The CSV files appear to still be available at https://www.kaggle.com/landlord/multilingual-disaster-response-messages though. It says that the data are under the CC0 license so I guess we can host the dataset elsewhere instead ?

## Describe the bug
Following command gives an ConnectionError.

## Steps to reproduce the bug
```python
disaster = load_dataset('disaster_response_messages')
```

## Error
```
ConnectionError: Couldn't reach https://datasets.appen.com/appen_datasets/disaster_response_data/disaster_response_messages_training.csv
```
## Expected results
It should load dataset without an error

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Google Colab
- Python version: 3.7
- PyArrow version:


Inconsistent performance of the "arabic_billion_words" dataset

bug

Something isn't working

## Describe the bug
When downloaded from macine 1 the dataset is downloaded and parsed correctly.
When downloaded from machine two (which has a different cache directory),
the following script:

import datasets
from datasets import load_dataset
raw_dataset_elkhair_1 = load_dataset('arabic_billion_words', 'Alittihad', split="train",download_mode='force_redownload')

gives the following error:

**Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: 1.49 GiB, post-processed: Unknown size, total: 1.82 GiB) to /root/.cache/huggingface/datasets/arabic_billion_words/Alittihad/1.1.0/687a1f963284c8a766558661375ea8f7ab3fa3633f8cd9c9f42a53ebe83bfe17...
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 348M/348M [00:24<00:00, 14.0MB/s]
Traceback (most recent call last):
  File ".../why_mismatch.py", line 3, in <module>
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 709, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=1601790302, num_examples=349342, dataset_name='arabic_billion_words'), 'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='arabic_billion_words')}]**

Note that the package versions of datasets (1.15.1) and rarfile (4.0) are identical.

## Steps to reproduce the bug
import datasets
from datasets import load_dataset
raw_dataset_elkhair_1 = load_dataset('arabic_billion_words', 'Alittihad', split="train",download_mode='force_redownload')

# Sample code to reproduce the bug

## Expected results
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: 1.49 GiB, post-processed: Unknown size, total: 1.82 GiB) to .../.cache/huggingface/datasets/arabic_billion_words/Alittihad/1.1.0/687a1f963284c8a766558661375ea8f7ab3fa3633f8cd9c9f42a53ebe83bfe17...
Downloading: 100%|███████████████████████████| 348M/348M [00:22<00:00, 15.8MB/s]
Dataset arabic_billion_words downloaded and prepared to .../.cache/huggingface/datasets/arabic_billion_words/Alittihad/1.1.0/687a1f963284c8a766558661375ea8f7ab3fa3633f8cd9c9f42a53ebe83bfe17. Subsequent calls will reuse this data.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
Machine 1:
- `datasets` version: 1.15.1
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1

Machine 2 (the bugged one)
- `datasets` version: 1.15.1
- Platform: Linux-4.4.0-210-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyArrow version: 6.0.0


Reuters21578 Couldn't reach

Hi ! The URL works fine on my side today, could you try again ?

``## Adding a Dataset
- **Name:** *Reuters21578*
- **Description:** *ConnectionError: Couldn't reach https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz*
- **Data:** *https://huggingface.co/datasets/reuters21578*

`from datasets import load_dataset`
`dataset = load_dataset("reuters21578", 'ModLewis')`

ConnectionError: Couldn't reach https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz

And I try to request the link as follow:
`import requests`
`requests.head('https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz')`

SSLError: HTTPSConnectionPool(host='kdd.ics.uci.edu', port=443): Max retries exceeded with url: /databases/reuters21578/reuters21578.tar.gz (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

This problem likes #575
What should I do ?


wikitext description wrong

Something isn't working

Hi @hongyuanmei, thanks for reporting.

I'm fixing it.

## Describe the bug
Descriptions of the wikitext datasests are wrong.

## Steps to reproduce the bug
Please see: https://github.com/huggingface/datasets/blob/f6dcafce996f39b6a4bbe3a9833287346f4a4b68/datasets/wikitext/wikitext.py#L50

## Expected results
The descriptions for raw-v1 and v1 should be switched.


Loading of datasets changed in #3110 returns no examples

Something isn't working

Hi @eladsegal, thanks for reporting.

I am sorry, but I can't reproduce the bug:
```
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("qasper")
Downloading: 5.11kB [00:00, ?B/s]
Downloading and preparing dataset qasper/qasper (download: 9.88 MiB, generated: 35.11 MiB, post-processed: Unknown size, total: 44.99 MiB) to .cache\qasper\qasper\0.1.0\b99154d2a15aa54bfc669f82b2eda715a2e342e81023d39613b0e2920fdb3ad8...
Dataset qasper downloaded and prepared to .cache\qasper\qasper\0.1.0\b99154d2a15aa54bfc669f82b2eda715a2e342e81023d39613b0e2920fdb3ad8. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<?, ?it/s]

In [3]: ds
Out[3]:
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 888
    })
    validation: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 281
    })
})
```

This makes me suspect that the origin of the problem might be the cache: I didn't have this dataset in my cache, although I guess you already had it, before the code change introduced by #3110.

@lhoestq might it be possible that the code change introduced by #3110 makes "inaccessible" all previously cached TAR-based datasets?
- Before the caching system downloaded and extracted the tar dataset
- Now it only downloads the tar dataset (no extraction is done)

I can't reproduce either in my environment (macos, python 3.7).

In your case it generates zero examples. This can only happen if the extraction of the TAR archive doesn't output the right filenames. Indeed if the `qasper` script can't find the right file to load, it's currently ignored and it returns zero examples. This case was not even considered when #3110 was developed since we considered the file names to be deterministic - and not depend on your environment.

Therefore here is my hypothesis:
- either the cache is corrupted somehow with an empty TAR archive
- OR I suspect that the issue comes from python 3.8

I just tried again on python 3.8 and I was able to reproduce the issue. Let me work on a fix

Ok I found the issue. It's not related to python 3.8 in itself though. This issue happens because your local installation of `datasets` is outdated compared to the changes to datasets in #3110

To fix this you just have to pull the latest changes from `master` :)

Let me know if that helps !

--------------

Here are more details about my investigation:

It's possible to reproduce this issue if you use `datasets<=1.15.1` or before b6469baa22c174b3906c631802a7016fedea6780 and if you load the dataset after revision b6469baa22c174b3906c631802a7016fedea6780. This is because `dl_manager.iter_archive` had issues at that time (and it was not used anywhere anyway).

In particular it was returning the absolute path to extracted files instead of the relative path of the file inside the archive. This was an issue because `dl_manager.iter_archive` isn't supposed to extract the TAR archive. Instead, it iterates over all the files inside the archive, without creating a directory with the extracted content.

Therefore if you want to use the datasets on `master`, make sure that you have an up-to-date local installation of `datasets` as well, or you may face incompatibilities like this.

Thanks!
But what about code that is already using older version of datasets?
The reason I encountered this issue was that suddenly one of my repos with version 1.12.1 started getting 0 examples.
I handled it by adding `revision` to `load_dataset`, but I guess it would still be an issue for other users who doesn't know this.

Hi, in 1.12.1 it uses the dataset scripts from that time, not the one on master.

It only uses the datasets from master if you installed `datasets` from source, or if the dataset isn't available in your local version (in this case it shows a warning and it loads from master).

OK, I understand the issue a bit better now.
I see I wasn't on 1.12.1, but on 1.12.1.dev0 and since it is a dev version it uses master.
So users that use an old dev version must specify revision or else they'll encounter this problem.

BTW, when I opened the issue I installed the latest master version with
```
pip install git+git://github.com/huggingface/datasets@master#egg=datasets
```
and also used `download_mode="force_redownload"`, and it still returned 0 examples.
Now I deleted all of the cache and ran the code again, and it worked.
I'm not sure what exactly happened here, but looks like it was due to a mix of an unofficial version and its cache.

Thanks again!

## Describe the bug
Loading of datasets changed in https://github.com/huggingface/datasets/pull/3110 returns no examples:
```python
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 0
    })
})
```

## Steps to reproduce the bug
Load any of the datasets that were changed in https://github.com/huggingface/datasets/pull/3110:
```python
from datasets import load_dataset
load_dataset("qasper")

# The problem only started with the commit of #3110
load_dataset("qasper", revision="b6469baa22c174b3906c631802a7016fedea6780")
```

## Expected results
```python
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 888
    })
    validation: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 281
    })
})
```
Which can be received when specifying revision of the commit before https://github.com/huggingface/datasets/pull/3110:
```python
from datasets import load_dataset
load_dataset("qasper", revision="acfe2abda1ca79f0ce5c1896aa83b4b78af76b7d")
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.2.dev0 (master)
- Python version: 3.8.10
- PyArrow version: 3.0.0


Addd options to use updated bleurt checkpoints

Adds options to use newer recommended checkpoint (as of 2021/10/8) bleurt-20 and its distilled versions.

Updated checkpoints are described in https://github.com/google-research/bleurt/blob/master/checkpoints.md#the-recommended-checkpoint-bleurt-20

This change won't affect the default behavior of metrics/bleurt. It only adds option to load newer checkpoints as

`datasets.load_metric('bleurt', 'bleurt-20')`

`bluert-20` generates scores roughly between 0 and 1, which wasn't the case for the previous checkpoints.


Avoid PyArrow type optimization if it fails

That's good to have a way to disable this easily :)
I just find it a bit unfortunate that users would have to experience the error once and then do `DISABLE_PYARROW_TYPES_OPTIMIZATION=1`. Do you know if there's a way to simply fallback on disabling it automatically when it fails ?

@lhoestq Actually, I agree a fallback makes more sense. The current approach is not very practical indeed and would require a mention in the docs.

Replaced the env variable with a fallback!

Hmm if the fallback automatically happens without the user knowing it, then I don't think we really need to mention it. But if you really wanted to, I think the [Improve performance](https://huggingface.co/docs/datasets/cache.html#improve-performance) section would be a great place for it!

Yea I think this could just end up in a note that says that `datasets` automatically picks the most optimized integer precision for your tokenized text data to save you disk space. Maybe later if we have a page on text processing we can add this note, but for now I agree it doesn't fit well into the doc.

In particular in the "Improve performance" section we mention what users can do to speed up their computations, while this behavior is just some internal feature that users don't have control over anyway.

Adds a new variable, `DISABLE_PYARROW_TYPES_OPTIMIZATION`, to `config.py` for easier control of the Arrow type optimization.

Fix #2206


Improve repository structure docs

Continuation of the documentation started in https://github.com/huggingface/datasets/pull/3221, taking into account @stevhliu 's comments


The Xsum datasets seems not able to download.

Something isn't working

> Hi ! On my side the URL is working fine, could you try again ?

I try it again and cannot download the file (might because of my location). Could you please provide another download link(such as google drive)?   :>

> I don't know other download links - this is the one provided by the authors of the dataset. Maybe you can try downloading from another location ? There are several solutions: a VPN, a remote VM or Google Colab for example.

:> ok. Thanks for your reply.

## Describe the bug
The download Link of the Xsum dataset provided in the repository is [Link](http://bollin.inf.ed.ac.uk/public/direct/XSUM-EMNLP18-Summary-Data-Original.tar.gz). It seems not able to download.

## Steps to reproduce the bug
```python
load_dataset('xsum')
```

## Actual results
``` python
raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach http://bollin.inf.ed.ac.uk/public/direct/XSUM-EMNLP18-Summary-Data-Original.tar.gz
```


Group tests in multiprocessing workers by test file

By grouping tests by test file, we make sure that all the tests in `test_load.py` are sent to the same worker.

Therefore, the fixture `hf_token` will be called only once (and from the same worker).

Related to: #3200.
Fix #3219.


Add full tagset to conll2003 README

I also added the missing `pretty_name` tag in the dataset card to fix the CI

Even though it is possible to manually get the tagset list with

```python
dset.features[field_name].feature.names
```

I think it is useful to have an overview of the used tagset on the dataset card. This is particularly useful in light of the **dataset viewer**: the tags are encoded, so it is not immediately obvious what they are for a given sample. Adding a label-int mapping should make it easier for visitors to get a grasp of what they mean.

From user-experience perspective, I would urge the full tagsets to always be available in the README's but I understand that that would take a lot of work, probably. Perhaps it can be automated?

closes #3189


Fix URL in CITATION file

Currently the BibTeX citation parsed from the CITATION file has wrong URL (it shows the repo URL instead of the proceedings paper URL):
```
@inproceedings{Lhoest_Datasets_A_Community_2021,
author = {Lhoest, Quentin and Villanova del Moral, Albert and von Platen, Patrick and Wolf, Thomas and Šaško, Mario and Jernite, Yacine and Thakur, Abhishek and Tunstall, Lewis and Patil, Suraj and Drame, Mariama and Chaumond, Julien and Plu, Julien and Davison, Joe and Brandeis, Simon and Sanh, Victor and Le Scao, Teven and Canwen Xu, Kevin and Patry, Nicolas and Liu, Steven and McMillan-Major, Angelina and Schmid, Philipp and Gugger, Sylvain and Raw, Nathan and Lesage, Sylvain and Lozhkov, Anton and Carrigan, Matthew and Matussière, Théo and von Werra, Leandro and Debut, Lysandre and Bekman, Stas and Delangue, Clément},
booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
month = {11},
pages = {175--184},
publisher = {Association for Computational Linguistics},
title = {{Datasets: A Community Library for Natural Language Processing}},
url = {https://github.com/huggingface/datasets},
year = {2021}
}
```


Add CITATION file

Add CITATION file.


Error in `Json(datasets.ArrowBasedBuilder)` class

Something isn't working

I have additionally identified the source of the error, being that [this condition](https://github.com/huggingface/datasets/blob/fc46bba66ba4f432cc10501c16a677112e13984c/src/datasets/packaged_modules/json/json.py#L124-L126) in the file
`python3.8/site-packages/datasets/packaged_modules/json/json.py` is not being entered correctly:
```python
                                    if (
                                        isinstance(e, pa.ArrowInvalid)
                                        and "straddling" not in str(e)
                                        or block_size > len(batch)
                                    ):
```

From what I can tell, in my case the block_size simply needs to be increased, but the error message does not contain "straddling" so the condition does trigger correctly and we fail to reach [the line to increase block_size](https://github.com/huggingface/datasets/blob/fc46bba66ba4f432cc10501c16a677112e13984c/src/datasets/packaged_modules/json/json.py#L135).

Changing the condition above to simply
```python
                                    if (
                                        block_size > len(batch)
                                    ):
```

Fixes the error for me. I'm happy to create a PR containing this fix if the developers deem the other conditions unnecessary.

Hi ! I think the issue comes from the fact that your JSON file is not a valid JSON Lines file.
Each example should be on one single line.

Can you try fixing the format to have one line per example and try again ?

:open_mouth: you're right, that did it! I just put everything on a single line (my file only has a single example) and that fixed the error. Thank you so much!

## Describe the bug
When a json file contains a `text` field that is larger than the block_size, the JSON dataset builder fails.

## Steps to reproduce the bug
Create a folder that contains the following:
```
.
├── testdata
│   └── mydata.json
└── test.py
```

Please download [this file](https://github.com/huggingface/datasets/files/7491797/mydata.txt) as `mydata.json`. (The error does not occur in JSON files with shorter text, but it is reproducible when the text is long as in the file I provide)
:exclamation: :exclamation: GitHub doesn't allow me to upload JSON so this file is a TXT, and you should rename it to `.json`!

`test.py` simply contains:
```python
from datasets import load_dataset
my_dataset = load_dataset("testdata")
```

To reproduce the error, simply run
```
python test.py
```

## Expected results
The data should load correctly without error.

## Actual results
The dataset builder fails with:
```
Using custom data configuration testdata-d490389b8ab4fd82
Downloading and preparing dataset json/testdata to /home/junshern.chan/.cache/huggingface/datasets/json/testdata-d490389b8ab4fd82/0.0.0/3333a8af0db9764dfcff43a42ff26228f0f2e267f0d8a0a294452d188beadb34...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2264.74it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 447.01it/s]
Failed to read file '/home/junshern.chan/hf-json-bug/testdata/mydata.json' with error <class 'pyarrow.lib.ArrowInvalid'>: JSON parse error: Missing a name for object member. in row 0
Traceback (most recent call last):
  File "test.py", line 28, in <module>
    my_dataset = load_dataset("testdata")
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/builder.py", line 1156, in _prepare_split
    for key, table in utils.tqdm(
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/tqdm/std.py", line 1168, in __iter__
    for obj in iterable:
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/packaged_modules/json/json.py", line 146, in _generate_tables
    raise ValueError(
ValueError: Not able to read records in the JSON file at /home/junshern.chan/hf-json-bug/testdata/mydata.json. You should probably indicate the field of the JSON file containing your records. This JSON file contain the following fields: ['text']. Select the correct one and provide it as `field='XXX'` to the dataset loading method.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.0


Fix paper BibTeX citation with proceedings reference.


Update tatoeba to v2021-07-22

How about this? @lhoestq @abhishekkrthakur

Hi ! I think it would be nice if people could still be able to load the old version.
Maybe this can be a parameter ? For example to load the old version they could do
```python
load_dataset("tatoeba", lang1="en", lang2="mr", date="v2020-11-09")
```

If it sounds good to you, we can add this parameter to the TatoebaConfig:
```python
class TatoebaConfig(datasets.BuilderConfig):
    def __init__(self, *args, lang1=None, lang2=None, date="v2021-07-22", **kwargs):
    self.date = date
```
and then pass the date to the URL
```python
_BASE_URL = "https://object.pouta.csc.fi/OPUS-Tatoeba/{}/moses/{}-{}.txt.zip"
```
```python
        def _base_url(lang1, lang2, date):
            return _BASE_URL.format(date, lang1, lang2)
```

What do you think ?

`_DATE = "v" + "-".join(s.zfill(2) for s in _VERSION.split("."))` seems rather tricky but works well. How about this? @lhoestq

The CI is only failing because of the missing sections in the dataset card, and because of an issue with the CER metric that is unrelated to this PR

Tatoeba's latest version is v2021-07-22


User-pickling with dynamic sub-classing

@lhoestq Feel free to have a look. The implementation is slightly different from what you suggested. I have opted to overwrite `save` instead of meddling with `save_global`. `save_global` is called very late down in dill/pickle so it is hard to control for what is happening there. I might be wrong. Pickling is more complex than I thought!

The linked issue (`map` with spaCy) also works now!

```python
import pickle
import spacy
from spacy import Language
from datasets import load_dataset
from datasets.utils.py_utils import dumps, pklregister

@pklregister(Language, allow_subclasses=True)
def hash_spacy_language(pickler, nlp: Language):
    pickler.save(nlp.to_bytes())

def main():
    fin = r"large/file.txt"
    nlp = spacy.load("en_core_web_sm")

    def tokenize(l):
        return {"tok": [t.text for t in nlp(l["text"])]}

    ds =  load_dataset("text", data_files=fin)
    ds = ds["train"].map(tokenize)

    # Sanity check: load NLP from pickle created with our own `dumps`
    config = nlp.config
    lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
    nlp2 = lang_cls.from_config(config)
    nlp2.from_bytes(pickle.loads(dumps(nlp)))

    assert isinstance(nlp2, type(nlp))
    assert dumps(nlp) == dumps(nlp2)

if __name__ == '__main__':
    main()
```

If this all looks good to you, I'll start writing on some documentation and examples.

One more thing. This is a reduction function for SpaCy Language that should work with the new API:
```python
@pklregister(Language, allow_subclasses=True)
def hash_spacy_language(pickler, obj):
    def create_language(config, bytes_data):
        lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
        nlp = lang_cls.from_config(config)
        return nlp.from_bytes(bytes_data)

    args = (obj.config, obj.to_bytes())
    pickler.save_reduce(create_language, args, obj=obj)
```
so IMO we are missing a test with `pickler.save_reduce`.

> One more thing. This is a reduction function for SpaCy Language that should work with the new API:
>
> ```python
> @pklregister(Language, allow_subclasses=True)
> def hash_spacy_language(pickler, obj):
>     def create_language(config, bytes_data):
>         lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
>         nlp = lang_cls.from_config(config)
>         return nlp.from_bytes(bytes_data)
>
>     args = (obj.config, obj.to_bytes())
>     pickler.save_reduce(create_language, args, obj=obj)
> ```
>
> so IMO we are missing a test with `pickler.save_reduce`.

Sure that seems a good idea, but I do not quite understand what `save_reduce` does. Could you give some more info about what reduce functions do and how they differ from regular `save` and `save_global`? I've read about it but the docs nor the built-in `pickle` code seem really helpful.

I'm no pickle expect, but here is my understanding. I believe the pickler uses the reduce function when you do `loads` to reconstructs the original object from the parameters/arguments that were saved with `dumps`.

For example your sanity check could be simplified from
```python
    config = nlp.config
    lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
    nlp2 = lang_cls.from_config(config)
    nlp2 = nlp2.from_bytes(pickle.loads(dumps(nlp)))
```
to
EDIT: <s>pickle.loads(pickle.dumps(nlp))</s>
```python
    nlp2 = loads(dumps(nlp))  # using our custom pickler
```

Though note that while it can be convenient for tests, we actually don't care about the reconstruction of the object since we're only using the pickler for `dumps` to compute hashes.

> I'm no pickle expect, but here is my understanding. I believe the pickler uses the reduce function when you do `loads` to reconstructs the original object from the parameters/arguments that were saved with `dumps`.
>
> For example your sanity check could be simplified from
>
> ```python
>     config = nlp.config
>     lang_cls = spacy.util.get_lang_class(config["nlp"]["lang"])
>     nlp2 = lang_cls.from_config(config)
>     nlp2 = nlp2.from_bytes(pickle.loads(dumps(nlp)))
> ```
>
> to
>
> ```python
>     nlp2 = pickle.loads(pickle.dumps(nlp))
> ```
>
> Though note that while it can be convenient for tests, we actually don't care about the reconstruction of the object since we're only using the pickler for `dumps` to compute hashes.

Yes, the sanity check can be simplified like that _if_ we use `pickle.dumps` - but that would not test our own `dumps` functionality and would do a naive dump instead of using `to_bytes`. It won't work if we use our own `dumps`, exactly because of the reason that we want custom pickling and being able to call `to_bytes`. To reconstruct the object from the pickled bytes from `to_bytes` we need `from_bytes`. The result of pickle/dill loads will therefore always be a `bytes` object and not a `Language` object.

But `save_reduce` is called when saving, right? Not when loading, AFAICT. I am just not sure what exactly it is saving. It is _potentially_ called [at the end of `save`](https://github.com/python/cpython/blob/24af9a40a8f85af813ea89998aa4e931fcc78cd9/Lib/pickle.py#L603) but only if we haven't returned by then. I just can't figure out what that base case is.

I don't think we expect users to write the reduce function that isn't going to be used anyway. So maybe let's stick with `save` ?

@BramVanroy
As I understand `save_reduce` is very similar to `copyreg.pickle`, so I'd suggest you to check the following links:
* https://docs.python.org/3/library/copyreg.html#copyreg.pickle
* https://docs.python.org/3/library/pickle.html#object.__reduce__

@lhoestq
> I don't think we expect users to write the reduce function that isn't going to be used anyway. So maybe let's stick with save ?

I agree.

`save_reduce` is very similar to `copyreg.pickle` and `object.__reduce__`, which are part of public API (and `save` isn't), so I expect more advanced users to know how to write their own reduction functions. But, as you say, `pklregister` should also work with `save` (even though I think `save` is a bit lower-level, and harder to understand than `save_reduce`).

All our examples in `py_utils` that use `pklregister`  also use `save_reduce` in the last step, so my reduction for SpaCy is meant to be added there, and not to be written by users (because SpaCy is very popular, so the official support by us makes sense :)).

And in the tests, let's ignore the reconstruction part of pickle/dill, because it's not important for us, and focus on the generated dumps. What do you think?

@mariosasko What exactly do you mean with "isn't part of the public API"? It is [a public method](https://github.com/python/cpython/blob/24af9a40a8f85af813ea89998aa4e931fcc78cd9/Lib/pickle.py#L535) in base pickle, just like `dump` is but maybe you mean something else.

@BramVanroy Oh sorry, it's public (not prefixed with `"_"`)  but it's not documented in the docs. `save_reduce` is also not in the docs, but its signature/functionality is similar to `copyreg.pickle` and I see it more often being used in the projects on GH, so it's seems "more public" to me.

Unfortunately I feel that pickle in general is under-documented. 😄

For the documentation, I can add a brief example, maybe under "How-to Guides"? The only thing that isn't immediately obvious to me is how I can add that doc page to the TOC?

Yes great idea ! To add that doc page to the TOC, you just have to add it to the index.rst file in the "How-to guides" TOC section

@mariosasko @lhoestq Feel free to make any edits or suggestions in the text!

Hi @mariosasko. I wish you'd told me sooner, as I spent quite some time writing on this.

I'm also not sure whether it is too advanced to have in the documentation. The spaCy use-case seems potentially frequent. Or do you wish to add that case to the defaults, and whenever new issues come up that seem like frequent/obvious cases, add those internally as well?

Documenting the internal `pklregister` is overkill IMO (and it can be kept in docstrings), but we can document something higher level like `register_hash_func` once it's implemented.

So we keep the nice documentation you've written (thank you!), except we can rename it to "Advanced caching" and show an API that is similar to
```python
>>> @register_hash_func(Language, allow_subclasses=True)
>>> def hash_spacy_language(nlp: Language):
>>>     return (nlp.to_bytes(),)
```
This way we keep the documentation centered around the public API rather than the internals that may evolve/be too complicated to fit only one section.

> Or do you wish to add that case to the defaults, and whenever new issues come up that seem like frequent/obvious cases, add those internally as well?

Let's add it to the defaults since it's a frequent use-case. And also allow users to control the hashing using the API mentioned above if they face other non-trivially-hashable objects

Sure, I can have a go at implementing spaCy as a built-in. Should it be included in the tests? (Therefore adding spaCy to the tests requirements.)

Next, from your example, it seems that the return value of `register_hash_func` will be used in pickler.save automatically (calling pklregister a bit deeper). Any reason why it returns a tuple? I can work on this as well, if needed.

> Sure, I can have a go at implementing spaCy as a built-in. Should it be included in the tests? (Therefore adding spaCy to the tests requirements.)

That would be perfect !

> Next, from your example, it seems that the return value of register_hash_func will be used in pickler.save automatically (calling pklregister a bit deeper).

Yes I think so. For example register_hash_func can call pklregister with the user's function, but wrapped to use pickler.save.

> Any reason why it returns a tuple? I can work on this as well, if needed.

It can either return an arbitrary object or a tuple. I like it a bit better if it's a tuple, so users understand more easily how to make the function take into account more than one item for the hash. It's also consistent with the streamlit caching functions, that also require a tuple. No strong opinion on this though

Let me know if I can help with anything

This is a continuation of the now closed PR in https://github.com/huggingface/datasets/pull/3206. The discussion there has shaped a new approach to do this.

In this PR, behavior of `pklregister` and `Pickler` is extended. Earlier, users were already able to register custom pickle functions. That is useful if they have objects that are not easily picklable with default methods. When one registers a custom function to a type, an object of that type will be pickled with the given function by `Pickler` which looks up the type in its `dispatch` table. The downside of this method, and of `pickle` in general, is that it is limited to direct type-matching and does not allow sub-classes. In many, default, cases that is not an issue. But when you are using external libraries where classes (e.g. parsers, models) are sub-classed this is not ideal.

```python
from datasets.fingerprint import Hasher
from datasets.utils.py_utils import pklregister

class BaseParser:
    pass

class EnglishParser(BaseParser):
    pass

@pklregister(BaseParser)
def custom_pkl_func(pickler, obj):
    print(f"Called the custom pickle function for type {type(obj)}!")
    # do something with the obj and ultimately save with the pickler

base = BaseParser()
en = EnglishParser()

# Hasher.hash uses the Pickler behind the scenes
# `custom_pkl_func` called for base
Hasher.hash(base)
# `custom_pkl_func` not called for en :-(
Hasher.hash(en)
```

In the example above we'd want to sub-class `EnglishParser` to be handled in the same way as its super-class `BaseParser`. This PR solves that by allowing for a keyword-argument `allow_subclasses` in `pklregister` (default: `False`).

```python
@pklregister(BaseParser, allow_subclasses=True)
```

When this option is enabled, we not only save the function in `Pickler.dispatch` but also save it in a custom table `Pickler.subclass_dispatch` **which allows us to dynamically add sub-classes of that class to the real dispatch table**. Then, if we want to pickle an  object `obj` with `Pickler.dump()` (which ultimately will call `Pickler.save()`) we _first_  check whether any of the object's super-classes exist in `Pickler.sublcass_dispatch` and get the related custom pickle function. If we find one, we add the type of `obj` alongside the function to `Pickler.dispatch`. All of this happens at the start of the call to `Pickler.save()`. _Only then_ dill.Pickler's `save` will be called, which in turn will call `pickle._Pickler.save` which handles everything. Here, the `Pickler.dispatch` table will be used to look up custom pickler functions - and it now also includes the function for `obj`, which was copied from its super-class, which we added at the very start of our custom `Pickler.save()`.

For edge cases and, especially, for testing, a contextmanager class `TempPickleRegistry` is included that resets the pickle registry on exit to its previous state.

```python
with TempPickleRegistry():
    @pklregister(MyObjClass)
    def pickle_registry_test_false(pickler, obj):
        pickler.save(obj.fancy_method())

    some_obj = MyObjClass()
    dumps(some_obj)
    # `MyObjClass` is in Pickler.dispatch

# ... `MyObjClass` is _not_ in Pickler.dispatch anymore
```

closes https://github.com/huggingface/datasets/issues/3178

To Do
====
- [x] Write tests
- [ ] Write documentation/examples?


Update BibTeX entry

Update BibTeX entry.


Add docs for audio processing

documentation

Improvements or additions to documentation

Nice ! love it this way. I guess you can set this PR to "ready for review" ?

This PR adds documentation for the `Audio` feature. It describes:

- The difference between loading `path` and `audio`, as well as use-cases/best practices for each of them.
- Resampling audio files with `cast_column`, and then calling `ds[0]["audio"]` to automatically decode and resample to the desired sampling rate.
- Resampling with `map`.

Preview [here](https://52969-250213286-gh.circle-artifacts.com/0/docs/_build/html/audio_process.html), let me know if I'm missing anything!


Resolve data_files by split name

Really cool!
When splitting by folder, what do we use for validation set ("valid", "validation" or both)?

> When splitting by folder, what do we use for validation set ("valid", "validation" or both)?

Both are fine :) As soon as it has "valid" in it

Merging for now, if you have comments about the documentation we can address them in subsequent PRs :)

Thanks for the comments @stevhliu :) I just opened https://github.com/huggingface/datasets/pull/3233 to take them into account

As discussed in https://github.com/huggingface/datasets/issues/3027 we should automatically infer what file is supposed to go to what split automatically, based on filenames.

I added the support for different kinds of patterns, for both dataset repositories and local directories:

```
    Input structure:

        my_dataset_repository/
        ├── README.md
        └── dataset.csv

    Output patterns:

        {"train": ["*"]}
```
```
    Input structure:

        my_dataset_repository/
        ├── README.md
        ├── train.csv
        └── test.csv

        my_dataset_repository/
        ├── README.md
        └── data/
            ├── train.csv
            └── test.csv

        my_dataset_repository/
        ├── README.md
        ├── train_0.csv
        ├── train_1.csv
        ├── train_2.csv
        ├── train_3.csv
        ├── test_0.csv
        └── test_1.csv

    Output patterns:

        {"train": ["*train*"], "test": ["*test*"]}
```
```
    Input structure:

        my_dataset_repository/
        ├── README.md
        └── data/
            ├── train/
            │   ├── shard_0.csv
            │   ├── shard_1.csv
            │   ├── shard_2.csv
            │   └── shard_3.csv
            └── test/
                ├── shard_0.csv
                └── shard_1.csv

    Output patterns:

        {"train": ["*train*/*", "*train*/**/*"], "test": ["*test*/*", "*test*/**/*"]}
```

and also this pattern that allows to have custom split names, and that is the structure used by #3098 for `push_to_hub` (cc @LysandreJik ):

```
    Input structure:

        my_dataset_repository/
        ├── README.md
        └── data/
            ├── train-00000-of-00003.csv
            ├── train-00001-of-00003.csv
            ├── train-00002-of-00003.csv
            ├── test-00000-of-00001.csv
            ├── random-00000-of-00003.csv
            ├── random-00001-of-00003.csv
            └── random-00002-of-00003.csv

    Output patterns:

        {
            "train": ["data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"],
            "test": ["data/test-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"],
            "random": ["data/random-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"],
        }
```

You can check the documentation about structuring your repository [here](https://52640-250213286-gh.circle-artifacts.com/0/docs/_build/html/repository_structure.html). cc @stevhliu

Fix https://github.com/huggingface/datasets/issues/3027
Fix https://github.com/huggingface/datasets/issues/3212

In the future we can also add support for dataset configurations.


Add documentation about dataset viewer feature

enhancement

New feature or request

Add to the docs more details about the dataset viewer feature in the Hub.

CC: @julien-c


Eventual Invalid Token Error at setup of private datasets

bug

Something isn't working

## Describe the bug
From time to time, there appear Invalid Token errors with private datasets:

- https://app.circleci.com/pipelines/github/huggingface/datasets/8520/workflows/d44629f2-4749-40f8-a657-50931d0b3434/jobs/52534
  ```
  ____________ ERROR at setup of test_load_streaming_private_dataset _____________
  ValueError: Invalid token passed!

  ____ ERROR at setup of test_load_streaming_private_dataset_with_zipped_data ____
  ValueError: Invalid token passed!

  =========================== short test summary info ============================
  ERROR tests/test_load.py::test_load_streaming_private_dataset - ValueError: I...
  ERROR tests/test_load.py::test_load_streaming_private_dataset_with_zipped_data
  ```

- https://app.circleci.com/pipelines/github/huggingface/datasets/8557/workflows/a8383181-ba6d-4487-9d0a-f750b6dcb936/jobs/52763
  ```
  ____ ERROR at setup of test_load_streaming_private_dataset_with_zipped_data ____
  [gw1] linux -- Python 3.6.15 /home/circleci/.pyenv/versions/3.6.15/bin/python3.6

  hf_api = <huggingface_hub.hf_api.HfApi object at 0x7f4899bab908>
  hf_token = 'vgNbyuaLNEBuGbgCEtSBCOcPjZnngJufHkTaZvHwkXKGkHpjBPwmLQuJVXRxBuaRzNlGjlMpYRPbthfHPFWXaaEDTLiqTTecYENxukRYVAAdpeApIUPxcgsowadkTkPj'
  zip_csv_path = PosixPath('/tmp/pytest-of-circleci/pytest-0/popen-gw1/data16/dataset.csv.zip')

      @pytest.fixture(scope="session")
      def hf_private_dataset_repo_zipped_txt_data_(hf_api: HfApi, hf_token, zip_csv_path):
          repo_name = "repo_zipped_txt_data-{}".format(int(time.time() * 10e3))
          hf_api.create_repo(token=hf_token, name=repo_name, repo_type="dataset", private=True)
          repo_id = f"{USER}/{repo_name}"
          hf_api.upload_file(
              token=hf_token,
              path_or_fileobj=str(zip_csv_path),
              path_in_repo="data.zip",
              repo_id=repo_id,
  >           repo_type="dataset",
          )

  tests/hub_fixtures.py:68:

  ...

  ValueError: Invalid token passed!
  =========================== short test summary info ============================
  ERROR tests/test_load.py::test_load_streaming_private_dataset_with_zipped_data
  ```


Fix code quality in riddle_sense dataset

Fix trailing whitespace.

Fix #3217.


Fix code quality bug in riddle_sense dataset

Something isn't working

To give more context: https://github.com/psf/black/issues/318. `black` doesn't treat this as a bug, but `flake8` does.

## Describe the bug
```
datasets/riddle_sense/riddle_sense.py:36:21: W291 trailing whitespace
```


Pin version exclusion for tensorflow incompatible with keras

Once `tensorflow` version 2.6.2 is released:
- https://github.com/tensorflow/tensorflow/commit/c1867f3bfdd1042f694df7a9870be51ba80543cb
- https://pypi.org/project/tensorflow/2.6.2/

with the patch:
- tensorflow/tensorflow#52927

we can remove the temporary fix we introduced in:
- #3208

Fix #3209.


Small updates to to_tf_dataset documentation

@stevhliu Accepted both suggestions, thanks for the review!

I added a little more description about `to_tf_dataset` compared to just setting the format


Add ACAV100M Dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *ACAV100M*
- **Description:** *contains 100 million videos with high audio-visual correspondence, ideal for self-supervised video representation learning.*
- **Paper:** *https://arxiv.org/abs/2101.10803*
- **Data:** *https://github.com/sangho-vision/acav100m*
- **Motivation:** *The largest dataset (to date) for audio-visual learning.*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Fix tuple_ie download url

Fix #3204


Sort files before loading

enhancement

New feature or request

This will be fixed by https://github.com/huggingface/datasets/pull/3221

When loading a dataset that consists of several files (e.g. `my_data/data_001.json`, `my_data/data_002.json` etc.) they are not loaded in order when using `load_dataset("my_data")`.

This could lead to counter-intuitive results if, for example, the data files are sorted by date or similar since they would appear in different order in the `Dataset`.

The straightforward solution is to sort the list of files alphabetically before loading them.

cc @lhoestq


Fix disable_nullable default value to False

Currently the `disable_nullable` parameter is not consistent across all dataset transforms. For example it is `False` in `map` but `True` in `flatten_indices`.

This creates unexpected behaviors like this
```python
from datasets import Dataset, concatenate_datasets

d1 = Dataset.from_dict({"a": [0, 1, 2, 3]})
d2 = d1.filter(lambda x: x["a"] < 2).flatten_indices()
d1.data.schema == d2.data.schema  # False
```
This can cause issues when concatenating datasets for example.

For consistency I set `disable_nullable` to `False` in `flatten_indices` and I fixed some docstrings

cc @SBrandeis


Hi ! Do you have some kind of proxy in your browser that gives you access to internet ?

Maybe you're having this error because you don't have access to this URL from python ?

when I use python examples/pytorch/translation/run_translation.py     --model_name_or_path examples/pytorch/translation/opus-mt-en-ro     --do_train     --do_eval     --source_lang en     --target_lang ro     --dataset_name wmt16     --dataset_config_name ro-en     --output_dir /tmp/tst-translation     --per_device_train_batch_size=4     --per_device_eval_batch_size=4     --overwrite_output_dir     --predict_with_generate to finetune translation model on huggingface, I get the issue"ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.15.1/datasets/wmt16/wmt16.py".But I can open the https://raw.githubusercontent.com/huggingface/datasets/1.15.1/datasets/wmt16/wmt16.py by using website. What should I do to solve the issue?


Unpin keras once TF fixes its release

Related to:
- #3208


Pin keras version until TF fixes its release

Fix #3207.


CI error: Another metric with the same name already exists in Keras 2.7.0

bug

Something isn't working

## Describe the bug
Release of TensorFlow 2.7.0 contains an incompatibility with Keras. See:
- keras-team/keras#15579

This breaks our CI test suite: https://app.circleci.com/pipelines/github/huggingface/datasets/8493/workflows/055c7ae2-43bc-49b4-9f11-8fc71f35a25c/jobs/52363


[WIP] Allow user-defined hash functions via a registry

Hi @BramVanroy, thanks for your PR.

There was a bug in TensorFlow/Keras. We have made a temporary fix in master branch. Please, merge master into your PR branch, so that the CI tests pass.

```
git checkout registry
git fetch upstream master
git merge upstream/master
```

@albertvillanova Done. Although new tests will need to be added. I am looking for some feedback on my initial proposal in this PR. Reviews and ideas welcome!

Hi ! Thanks for diving into this :)

With this approach you get the right hash when doing `Hasher.hash(nlp)` but if you try to hash an object that has `nlp` as one of its attributes for example you will get different hashes every time.

This is because `Hasher.hash` is not recursive itself. Indeed what happens when you try to hash an object is that:
1. it is dumped with our custom `dill` pickler (which is recursive)
2. the bytes of the dump are hashed

To fix this we must integrate the custom hashing as a custom pickler dumping instead.

Note that we're only using the `pickler.dumps` method and not `pickler.loads` since we only use it to get hashes, so it doesn't matter if `loads` doesn't reconstruct the object exactly. What's important it only to capture all the necessary information that defines how the object transforms the data (here `nlp.to_bytes()` determines how the spacy pipeline transforms the text).

Our pickler already has a registry and you can register new dump functions with:
```python
import dill
import spacy
from datasets.utils.py_utils import pklregister

@pklregister(spacy.Language)
def _save_spacy_language(pickler, nlp):
    pickler.save_reduce(...)  # I think we can use nlp.to_bytes() here
    dill._dill.log.info(...)
```

You can find some examples of custom dump functions in `py_utils.py`

Ah, darn it. Completely missed that register. Time wasted, unfortunately.

To better understand what you mean, I figured I'd try the basis of your snippet and I've noticed quite an annoying side-effect of how the pickle dispatch table seems to work. It explicitly uses an object's [`type()`](https://github.com/python/cpython/blob/87032cfa3dc975d7442fd57dea2c6a56d31c911a/Lib/pickle.py#L557-L558), which makes sense for pickling some (primitive) types it is not ideal for more complex ones, I think. `Hasher.hash` has the same issue as far as I can tell.

https://github.com/huggingface/datasets/blob/d21ce54f2c2782f854f975eb1dc2be6f923b4314/src/datasets/fingerprint.py#L187-L191

This is very restrictive, and won't work for subclasses. In the case of spaCy, for instance, we register `Language`, but `nlp` is an instance of `English`, which is a _subclass_ of `Language`. These are different types, and so they will not match in the dispatch table. Maybe this is more general approach to cover such cases? Something like this is a start but too broad, but ideally a hierarchy is constructed and traversed of all classes in the table and the lowest class is selected to ensure that the most specific class function is dispatched.

```python
    def hash(cls, value: Any) -> str:
        # Try to match the exact type
        if type(value) in cls.dispatch:
            return cls.dispatch[type(value)](cls, value)

        # Try to match instance (superclass)
        for type_cls, func in cls.dispatch.items():
            if isinstance(value, type_cls):
                return cls.dispatch[type_cls](cls, value)

        return cls.hash_default(value)
```

This does not solve the problem for pickling, though. That is quite unfortunate IMO because that implies that users always have to specify the most specific class, which is not always obvious. (For instance, `spacy.load`'s signature returns `Language`, but as said before a subclass might be returned.)

Second, I am trying to understand `save_reduce` but I can find very little documentation about it, only the source code which is quite cryptic. Can you explain it a bit? The required arguments are not very clear to me and there is no docstring.

```python
    def save_reduce(self, func, args, state=None, listitems=None, dictitems=None, obj=None):
```

Here is an example illustrating the problem with sub-classes.

```python
import spacy

from spacy import Language
from spacy.lang.en import English

from datasets.utils.py_utils import Pickler, pklregister

# Only useful in the registry (matching with `nlp`)
# if you swap it out for very specific `English`
@pklregister(English)
def hash_spacy_language(pickler, nlp):
    pass

def main():
    print(Pickler.dispatch)
    nlp = spacy.load("en_core_web_sm")
    print(f"NLP type {type(nlp)} in dispatch table? ", type(nlp) in Pickler.dispatch)

if __name__ == '__main__':
    main()
```

Indeed that's not ideal.
Maybe we could integrate all the subclasses directly in `datasets`. That's simple to do but the catch is that if users have new subclasses of `Language` it won't work.

Otherwise we can see how to make the API simpler for users by allowing subclasses
```python
# if you swap it out for very specific `English`
@pklregister(Language, allow_subclasses=True)
def hash_spacy_language(pickler, nlp):
    pass
```

Here is an idea how to make this work, let me know what you think:

When `Pickler.dumps` is called, it uses `Pickler.save_global` which is a method that is going to be called recursively on all the objects. We can customize this part, and make it work as we want when it encounters a subclass of `Language`.

For example when it encounters a subclass of `Language`, we can dynamically register the hashing function for the subclass (`English` for example) in `Pickler.save_global`, right before calling the actual `dill.Pickler.save_global(self, obj, name=name)`:
```python
pklregister(type(obj))(hash_function_registered_for_parent_class)
dill.Pickler.save_global(self, obj, name=name)
```

In practice that means we can have an additional dispatch dictionary (similar to `Pickler.dispatch`) to store the hashing functions when `allow_subclasses=True`, and use this dictionary in `Pickler.save_global` to check if we need to use a hashing function registered with `allow_subclasses=True` and get `hash_function_registered_for_parent_class`.

If I understood you correctly, I do not think that that is enough because you are only doing this for a type and its direct parent class. You could do this for all superclasses (so traverse all ancestors and find the registered function for the first that is encountered). I can work on that, if you agree. The one thing that I am not sure about is how you want to create the secondary dispatch table. An empty dict as class variable in Pickler? (It doesn't have to be a true dispatcher, I think.)

I do not think that dynamic registration is the ideal situation (it feels a bit hacky). An alternative would be to subclass Pickle and Dill to make sure that instead of just type() checking in the dispatch table also superclasses are considered. But that is probably overkill.

> You could do this for all superclasses (so traverse all ancestors and find the registered function for the first that is encountered)

That makes sense indeed !

> The one thing that I am not sure about is how you want to create the secondary dispatch table. An empty dict as class variable in Pickler? (It doesn't have to be a true dispatcher, I think.)

Sure, let's try to not use too complicated stuff

> I do not think that dynamic registration is the ideal situation (it feels a bit hacky). An alternative would be to subclass Pickle and Dill to make sure that instead of just type() checking in the dispatch table also superclasses are considered. But that is probably overkill.

Indeed that would feel less hacky, but maybe it's too complex just for this. I feel like this part of the library is already hard to understand when you're not familiar with pickle. IMO having only a few changes that are simpler to understand is better than having a rewrite of `dill`'s core code.

Thanks a lot for your insights, it looks like we're going to have something that works well and that unlocks some nice flexibility for users :) Feel free to ping me anytime if I can help on this

Sure, thanks for brainstorming! I'll try to work on it this weekend. Will also revert the current changes in this PR and rename it.

It seems like this is going in the right direction :).

@BramVanroy Just one small suggestion for future contributions: instead of using `WIP` in the PR title, you can create a draft PR if you're still working on it.

Maybe I should just create a new (draft) PR then, seeing that I'll have to rename and revert the changes anyway? I'll link to this PR so that the discussion is at least referenced.

I can convert this PR to a draft PR. Let me know what would you prefer.

I think reverting my previous commits would make for a dirty (or confusing) commit history, so I'll just create a new one. Thanks.

Inspired by the discussion on hashing in https://github.com/huggingface/datasets/issues/3178#issuecomment-959016329, @lhoestq suggested that it would be neat to allow users more control over the hashing process. Specifically, it would be great if users can specify specific hashing functions depending on the **class** of the object.

As an example, we found in the linked topic that loaded spaCy models (`Language` objects) have different hashes when `dump`'d, but their byte representation with `Language.to_bytes()` _is_ deterministic. It would therefore be great if we could specify that for `Language` objects, the hasher should hash the objects `to_bytes()` return value instead of the object itself.

This PR adds a new, but tiny, dependency to manage the registry, namely [`catalogue`](https://github.com/explosion/catalogue).

Two files have been changed (apart from the added dependency in `setup.py`) and one file has been added.

**utils.registry** (added)

This file defines our custom Registry and builds a registry called "hashers". A Registry is basically dictionary from names (str) to functions. A function can be added to the registry by a decorator, e.g.

```python
@hashers.register(spacy.Language)
def hash_spacy_language(nlp):
    return Hasher.hash(nlp.to_bytes())
```

You'll notice that `spacy.Language` is not a string, even though the registry holds a str->func mapping. To accomplish this with classes in a dynamic way, catalogue.Registry needed to be subclassed and modified as `DatasetsRegistry`. All methods that use a name as an input are now modified so that classes are deterministically converted in strings in such a way that we can later retrieve the actual class from the string (below).

**utils.py_utils** (modified)

Added two functions to deal with classes and their qualified names, that is, their full descriptive name including the module. On the one hand it allows us to retrieve a string from a given class, e.g. given `Module` class, return `torch.nn.Module` str. Conversly, a function is added to convert such a full qualified name into a class. For instance, given the string `torch.nn.Module`, return the `Module` class. These straightforward methods allow us to interchangeably use classes and strings without any needed user interaction - they can just register a class, and behind the scenes `DatasetsRegistry` converts these to deterministic strings.

**fingerprint** (modified)

Updated Hasher.hash so that if the object to hash is an instance of a class in the registry, the registered function is used to hash the object instead of the default behavior. To do so we iterate over the registry `hashers` and convert its keys (strings) into classes, and then we can use `isinstance`.

```python
# Check if the current object is an instance that is
# applicable to the user-defined hashers. If so, hash
# with the user-defined function
for full_module_name, func in hashers.get_all().items():
    registered_cls = get_cls_from_qualname(full_module_name)
    if isinstance(value, registered_cls):
        return func(value)
```

**Putting it all together**

To test this, you can try the following example with spaCy. First install spaCy from source and checkout a specific commit.

```shell
git clone https://github.com/explosion/spaCy.git
cd spaCy/
git checkout cab9209c3dfcd1b75dfe5657f10e52c4d847a3cf
cd ..

git clone https://github.com/BramVanroy/datasets.git
cd datasets
git checkout registry
pip install -e .
pip install ../spaCy
spacy download en_core_web_sm
```

Now you can run the following script. By default it will use the custom hasher function for the Language object. You can enable the default behavior by commenting out `@hashers.register...`.

```python
import spacy

from datasets.fingerprint import Hasher
from datasets.utils.registry import hashers

# Register a function so that when the Hasher encounters a spacy.Language object
# it uses this custom function to hash instead of the default
@hashers.register(spacy.Language)
def hash_spacy_language(nlp):
    return Hasher.hash(nlp.to_bytes())

def main():
    print(hashers.get_all())
    nlp = spacy.load("en_core_web_sm")
    dump1 = Hasher.hash(nlp)
    nlp = spacy.load("en_core_web_sm")
    dump2 = Hasher.hash(nlp)
    print(dump1)
    # succeeds when using the registered custom function
    # fails if using the default
    assert dump1 == dump2

if __name__ == '__main__':
    main()
```

To do
====
- The above is just a proof-of-concept. I am open to changes/suggestions
- Tests still need to be written
- We should consider whether we can make `DatasetsRegistry` very restrictive and ONLY allowing classes. That would make testing easier - otherwise we also need to test for other sorts of objects.
- Maybe the `hashers` definition is better suited in `fingerprint`?
- Documentation/examples need to be updated
- Not sure why the logger is not working in `hash()`
- `get_cls_from_qualname` might need a fail-safe: is it possible for a full_qualname to not have a module, and if so how do we deal with that?