Something isn't working

Hi,

thanks for reporting and providing a minimal reproducible example.

This line of the PR I've linked in our discussion on the Forum will add support for `None` values:
https://github.com/huggingface/datasets/blob/a53de01842aac65c66a49b2439e18fa93ff73ceb/src/datasets/features/features.py#L835

I expect that PR to be merged soon.

## Describe the bug
`GeneratorBasedBuilder` does not support `None` values.

## Steps to reproduce the bug
See [this repository](https://github.com/pavel-lexyr/huggingface-datasets-bug-reproduction) for minimal reproduction.

## Expected results
Dataset is initialized with a `None` value in the `value` column.

## Actual results
```
Traceback (most recent call last):
  File "main.py", line 3, in <module>
    datasets.load_dataset("./bad-data")
  File ".../datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File ".../datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File ".../datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File ".../datasets/builder.py", line 1103, in _prepare_split
    example = self.info.features.encode_example(record)
  File ".../datasets/features/features.py", line 1033, in encode_example
    return encode_nested_example(self, example)
  File ".../datasets/features/features.py", line 808, in encode_nested_example
    return {
  File ".../datasets/features/features.py", line 809, in <dictcomp>
    k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File ".../datasets/features/features.py", line 855, in encode_nested_example
    return schema.encode_example(obj)
  File ".../datasets/features/features.py", line 299, in encode_example
    return float(value)
TypeError: float() argument must be a string or a number, not 'NoneType'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-5.4.0-81-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 6.0.0


Loading big json dataset raises pyarrow.lib.ArrowNotImplementedError

bug

Something isn't working

Hi,

this issue is similar to https://github.com/huggingface/datasets/issues/3093, so you can either use the solution provided there or try to load the data in one chunk (you can control the chunk size by specifying the `chunksize` parameter (`int`) in `load_dataset`).

@lhoestq Is this worth opening an issue on Jira? Basically, PyArrow doesn't allow casts that change the order of the struct fields because they treat `pa.struct` as an ordered sequence. Reordering fields manually in Python is probably too slow, so I think this needs to be fixed by them to be usable on our side.

I agree I would expect PyArrow to be able to handle this, do you want to open the issue @mariosasko ?
Although maybe it's possible to fix struct casting on our side without hurting performance too much, if it's simply a matter of reordering the arrays in the StructArray

## Describe the bug
When trying to create a dataset from a json file with around 25MB, the following error is raised `pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<b: int64, c: int64> to struct using function cast_struct`

Splitting the big file into smaller ones and then loading it with the `load_dataset` method did also not work.

Creating a pandas dataframe from it and then loading it with `Dataset.from_pandas` works

## Steps to reproduce the bug
```python
load_dataset("json", data_files="test.json")
```

test.json ~25MB
```json
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
{"a": {"c": 8, "b": 5}}
...
```

working.json ~160bytes
```json
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
{"a": {"c": 8, "b": 5}}
```

## Expected results
It should load the dataset from the json file without error.

## Actual results
It raises Exception `pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<b: int64, c: int64> to struct using function cast_struct`

```
Traceback (most recent call last):
  File "/Users/m/workspace/xxx/project/main.py", line 60, in <module>
    dataset = load_dataset("json", data_files="result.json")
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/datasets/load.py", line 1627, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/datasets/builder.py", line 1159, in _prepare_split
    writer.write_table(table)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/datasets/arrow_writer.py", line 428, in write_table
    pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
  File "pyarrow/table.pxi", line 1685, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 630, in pyarrow.lib._sanitize_arrays
  File "pyarrow/array.pxi", line 338, in pyarrow.lib.asarray
  File "pyarrow/table.pxi", line 304, in pyarrow.lib.ChunkedArray.cast
  File "/opt/homebrew/Caskroom/miniforge/base/envs/xxx/lib/python3.9/site-packages/pyarrow/compute.py", line 309, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 528, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 327, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 120, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<b: int64, c: int64> to struct using function cast_struct
```

## Environment info
- `datasets` version: 1.14.0
- Platform: macOS-12.0.1-arm64-arm-64bit
- Python version: 3.9.7
- PyArrow version: 6.0.0


Adding ANERcorp-CAMeLLab dataset

dataset request

Requesting to add a new dataset

Adding ANERcorp dataset

## Adding a Dataset
- **Name:** *ANERcorp-CAMeLLab*
- **Description:** *Since its creation in 2008, the ANERcorp dataset (Benajiba & Rosso, 2008) has been a standard reference used by Arabic named entity recognition researchers around the world. However, over time, this dataset was copied over from user to user, modified slightly here and there, and split in many different configurations that made it hard to compare fairly across papers and systems.

In 2020, a group of researchers from CAMeL Lab (Habash, Alhafni and Oudah), and Mind Lab (Antoun and Baly) met with the creator of the corpus, Yassine Benajiba, to consult with him and collectively agree on an exact split, and accepted minor corrections from the original dataset. Bashar Alhafni from CAMeL Lab working with Nizar Habash implemented the decisions provided in this release.*

- **Paper:** *(a) Benajiba, Yassine, Paolo Rosso, and José Miguel Benedí Ruiz. "Anersys: An Arabic named entity recognition system based on maximum entropy." In International Conference on Intelligent Text Processing and Computational Linguistics, pp. 143-153. Springer, Berlin, Heidelberg, 2007.

(b)Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. "CAMeL Tools: An Open Source Python Toolkit, for Arabic Natural Language Processing." In Proceedings of the Conference on Language Resources and Evaluation (LREC 2020), Marseille, 2020.*
- **Data:** *https://camel.abudhabi.nyu.edu/anercorp/*
- **Motivation:** This is the standard dataset for evaluating NER performance in Arabic*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Couldn't reach data file for disaster_response_messages

dataset bug

A bug in a dataset script provided in the library

It looks like the dataset isn't available anymore on appen.com

The CSV files appear to still be available at https://www.kaggle.com/landlord/multilingual-disaster-response-messages though. It says that the data are under the CC0 license so I guess we can host the dataset elsewhere instead ?

## Describe the bug
Following command gives an ConnectionError.

## Steps to reproduce the bug
```python
disaster = load_dataset('disaster_response_messages')
```

## Error
```
ConnectionError: Couldn't reach https://datasets.appen.com/appen_datasets/disaster_response_data/disaster_response_messages_training.csv
```
## Expected results
It should load dataset without an error

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Google Colab
- Python version: 3.7
- PyArrow version:


Inconsistent performance of the "arabic_billion_words" dataset

bug

Something isn't working

## Describe the bug
When downloaded from macine 1 the dataset is downloaded and parsed correctly.
When downloaded from machine two (which has a different cache directory),
the following script:

import datasets
from datasets import load_dataset
raw_dataset_elkhair_1 = load_dataset('arabic_billion_words', 'Alittihad', split="train",download_mode='force_redownload')

gives the following error:

**Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: 1.49 GiB, post-processed: Unknown size, total: 1.82 GiB) to /root/.cache/huggingface/datasets/arabic_billion_words/Alittihad/1.1.0/687a1f963284c8a766558661375ea8f7ab3fa3633f8cd9c9f42a53ebe83bfe17...
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 348M/348M [00:24<00:00, 14.0MB/s]
Traceback (most recent call last):
  File ".../why_mismatch.py", line 3, in <module>
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/lib/python3.8/site-packages/datasets/builder.py", line 709, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/opt/conda/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=1601790302, num_examples=349342, dataset_name='arabic_billion_words'), 'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='arabic_billion_words')}]**

Note that the package versions of datasets (1.15.1) and rarfile (4.0) are identical.

## Steps to reproduce the bug
import datasets
from datasets import load_dataset
raw_dataset_elkhair_1 = load_dataset('arabic_billion_words', 'Alittihad', split="train",download_mode='force_redownload')

# Sample code to reproduce the bug

## Expected results
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: 1.49 GiB, post-processed: Unknown size, total: 1.82 GiB) to .../.cache/huggingface/datasets/arabic_billion_words/Alittihad/1.1.0/687a1f963284c8a766558661375ea8f7ab3fa3633f8cd9c9f42a53ebe83bfe17...
Downloading: 100%|███████████████████████████| 348M/348M [00:22<00:00, 15.8MB/s]
Dataset arabic_billion_words downloaded and prepared to .../.cache/huggingface/datasets/arabic_billion_words/Alittihad/1.1.0/687a1f963284c8a766558661375ea8f7ab3fa3633f8cd9c9f42a53ebe83bfe17. Subsequent calls will reuse this data.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
Machine 1:
- `datasets` version: 1.15.1
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1

Machine 2 (the bugged one)
- `datasets` version: 1.15.1
- Platform: Linux-4.4.0-210-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyArrow version: 6.0.0


Reuters21578 Couldn't reach

Hi ! The URL works fine on my side today, could you try again ?

``## Adding a Dataset
- **Name:** *Reuters21578*
- **Description:** *ConnectionError: Couldn't reach https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz*
- **Data:** *https://huggingface.co/datasets/reuters21578*

`from datasets import load_dataset`
`dataset = load_dataset("reuters21578", 'ModLewis')`

ConnectionError: Couldn't reach https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz

And I try to request the link as follow:
`import requests`
`requests.head('https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz')`

SSLError: HTTPSConnectionPool(host='kdd.ics.uci.edu', port=443): Max retries exceeded with url: /databases/reuters21578/reuters21578.tar.gz (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

This problem likes #575
What should I do ?


wikitext description wrong

Something isn't working

Hi @hongyuanmei, thanks for reporting.

I'm fixing it.

## Describe the bug
Descriptions of the wikitext datasests are wrong.

## Steps to reproduce the bug
Please see: https://github.com/huggingface/datasets/blob/f6dcafce996f39b6a4bbe3a9833287346f4a4b68/datasets/wikitext/wikitext.py#L50

## Expected results
The descriptions for raw-v1 and v1 should be switched.


Loading of datasets changed in #3110 returns no examples

Something isn't working

Hi @eladsegal, thanks for reporting.

I am sorry, but I can't reproduce the bug:
```
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("qasper")
Downloading: 5.11kB [00:00, ?B/s]
Downloading and preparing dataset qasper/qasper (download: 9.88 MiB, generated: 35.11 MiB, post-processed: Unknown size, total: 44.99 MiB) to .cache\qasper\qasper\0.1.0\b99154d2a15aa54bfc669f82b2eda715a2e342e81023d39613b0e2920fdb3ad8...
Dataset qasper downloaded and prepared to .cache\qasper\qasper\0.1.0\b99154d2a15aa54bfc669f82b2eda715a2e342e81023d39613b0e2920fdb3ad8. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<?, ?it/s]

In [3]: ds
Out[3]:
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 888
    })
    validation: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 281
    })
})
```

This makes me suspect that the origin of the problem might be the cache: I didn't have this dataset in my cache, although I guess you already had it, before the code change introduced by #3110.

@lhoestq might it be possible that the code change introduced by #3110 makes "inaccessible" all previously cached TAR-based datasets?
- Before the caching system downloaded and extracted the tar dataset
- Now it only downloads the tar dataset (no extraction is done)

I can't reproduce either in my environment (macos, python 3.7).

In your case it generates zero examples. This can only happen if the extraction of the TAR archive doesn't output the right filenames. Indeed if the `qasper` script can't find the right file to load, it's currently ignored and it returns zero examples. This case was not even considered when #3110 was developed since we considered the file names to be deterministic - and not depend on your environment.

Therefore here is my hypothesis:
- either the cache is corrupted somehow with an empty TAR archive
- OR I suspect that the issue comes from python 3.8

I just tried again on python 3.8 and I was able to reproduce the issue. Let me work on a fix

Ok I found the issue. It's not related to python 3.8 in itself though. This issue happens because your local installation of `datasets` is outdated compared to the changes to datasets in #3110

To fix this you just have to pull the latest changes from `master` :)

Let me know if that helps !

--------------

Here are more details about my investigation:

It's possible to reproduce this issue if you use `datasets<=1.15.1` or before b6469baa22c174b3906c631802a7016fedea6780 and if you load the dataset after revision b6469baa22c174b3906c631802a7016fedea6780. This is because `dl_manager.iter_archive` had issues at that time (and it was not used anywhere anyway).

In particular it was returning the absolute path to extracted files instead of the relative path of the file inside the archive. This was an issue because `dl_manager.iter_archive` isn't supposed to extract the TAR archive. Instead, it iterates over all the files inside the archive, without creating a directory with the extracted content.

Therefore if you want to use the datasets on `master`, make sure that you have an up-to-date local installation of `datasets` as well, or you may face incompatibilities like this.

Thanks!
But what about code that is already using older version of datasets?
The reason I encountered this issue was that suddenly one of my repos with version 1.12.1 started getting 0 examples.
I handled it by adding `revision` to `load_dataset`, but I guess it would still be an issue for other users who doesn't know this.

Hi, in 1.12.1 it uses the dataset scripts from that time, not the one on master.

It only uses the datasets from master if you installed `datasets` from source, or if the dataset isn't available in your local version (in this case it shows a warning and it loads from master).

OK, I understand the issue a bit better now.
I see I wasn't on 1.12.1, but on 1.12.1.dev0 and since it is a dev version it uses master.
So users that use an old dev version must specify revision or else they'll encounter this problem.

BTW, when I opened the issue I installed the latest master version with
```
pip install git+git://github.com/huggingface/datasets@master#egg=datasets
```
and also used `download_mode="force_redownload"`, and it still returned 0 examples.
Now I deleted all of the cache and ran the code again, and it worked.
I'm not sure what exactly happened here, but looks like it was due to a mix of an unofficial version and its cache.

Thanks again!

## Describe the bug
Loading of datasets changed in https://github.com/huggingface/datasets/pull/3110 returns no examples:
```python
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 0
    })
})
```

## Steps to reproduce the bug
Load any of the datasets that were changed in https://github.com/huggingface/datasets/pull/3110:
```python
from datasets import load_dataset
load_dataset("qasper")

# The problem only started with the commit of #3110
load_dataset("qasper", revision="b6469baa22c174b3906c631802a7016fedea6780")
```

## Expected results
```python
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 888
    })
    validation: Dataset({
        features: ['id', 'title', 'abstract', 'full_text', 'qas'],
        num_rows: 281
    })
})
```
Which can be received when specifying revision of the commit before https://github.com/huggingface/datasets/pull/3110:
```python
from datasets import load_dataset
load_dataset("qasper", revision="acfe2abda1ca79f0ce5c1896aa83b4b78af76b7d")
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.2.dev0 (master)
- Python version: 3.8.10
- PyArrow version: 3.0.0


The Xsum datasets seems not able to download.

Something isn't working

> Hi ! On my side the URL is working fine, could you try again ?

I try it again and cannot download the file (might because of my location). Could you please provide another download link(such as google drive)?   :>

> I don't know other download links - this is the one provided by the authors of the dataset. Maybe you can try downloading from another location ? There are several solutions: a VPN, a remote VM or Google Colab for example.

:> ok. Thanks for your reply.

## Describe the bug
The download Link of the Xsum dataset provided in the repository is [Link](http://bollin.inf.ed.ac.uk/public/direct/XSUM-EMNLP18-Summary-Data-Original.tar.gz). It seems not able to download.

## Steps to reproduce the bug
```python
load_dataset('xsum')
```

## Actual results
``` python
raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach http://bollin.inf.ed.ac.uk/public/direct/XSUM-EMNLP18-Summary-Data-Original.tar.gz
```


Error in `Json(datasets.ArrowBasedBuilder)` class

Something isn't working

I have additionally identified the source of the error, being that [this condition](https://github.com/huggingface/datasets/blob/fc46bba66ba4f432cc10501c16a677112e13984c/src/datasets/packaged_modules/json/json.py#L124-L126) in the file
`python3.8/site-packages/datasets/packaged_modules/json/json.py` is not being entered correctly:
```python
                                    if (
                                        isinstance(e, pa.ArrowInvalid)
                                        and "straddling" not in str(e)
                                        or block_size > len(batch)
                                    ):
```

From what I can tell, in my case the block_size simply needs to be increased, but the error message does not contain "straddling" so the condition does trigger correctly and we fail to reach [the line to increase block_size](https://github.com/huggingface/datasets/blob/fc46bba66ba4f432cc10501c16a677112e13984c/src/datasets/packaged_modules/json/json.py#L135).

Changing the condition above to simply
```python
                                    if (
                                        block_size > len(batch)
                                    ):
```

Fixes the error for me. I'm happy to create a PR containing this fix if the developers deem the other conditions unnecessary.

Hi ! I think the issue comes from the fact that your JSON file is not a valid JSON Lines file.
Each example should be on one single line.

Can you try fixing the format to have one line per example and try again ?

:open_mouth: you're right, that did it! I just put everything on a single line (my file only has a single example) and that fixed the error. Thank you so much!

## Describe the bug
When a json file contains a `text` field that is larger than the block_size, the JSON dataset builder fails.

## Steps to reproduce the bug
Create a folder that contains the following:
```
.
├── testdata
│   └── mydata.json
└── test.py
```

Please download [this file](https://github.com/huggingface/datasets/files/7491797/mydata.txt) as `mydata.json`. (The error does not occur in JSON files with shorter text, but it is reproducible when the text is long as in the file I provide)
:exclamation: :exclamation: GitHub doesn't allow me to upload JSON so this file is a TXT, and you should rename it to `.json`!

`test.py` simply contains:
```python
from datasets import load_dataset
my_dataset = load_dataset("testdata")
```

To reproduce the error, simply run
```
python test.py
```

## Expected results
The data should load correctly without error.

## Actual results
The dataset builder fails with:
```
Using custom data configuration testdata-d490389b8ab4fd82
Downloading and preparing dataset json/testdata to /home/junshern.chan/.cache/huggingface/datasets/json/testdata-d490389b8ab4fd82/0.0.0/3333a8af0db9764dfcff43a42ff26228f0f2e267f0d8a0a294452d188beadb34...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2264.74it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 447.01it/s]
Failed to read file '/home/junshern.chan/hf-json-bug/testdata/mydata.json' with error <class 'pyarrow.lib.ArrowInvalid'>: JSON parse error: Missing a name for object member. in row 0
Traceback (most recent call last):
  File "test.py", line 28, in <module>
    my_dataset = load_dataset("testdata")
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/builder.py", line 1156, in _prepare_split
    for key, table in utils.tqdm(
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/tqdm/std.py", line 1168, in __iter__
    for obj in iterable:
  File "/home/junshern.chan/.casio/miniconda/envs/hf-json-bug/lib/python3.8/site-packages/datasets/packaged_modules/json/json.py", line 146, in _generate_tables
    raise ValueError(
ValueError: Not able to read records in the JSON file at /home/junshern.chan/hf-json-bug/testdata/mydata.json. You should probably indicate the field of the JSON file containing your records. This JSON file contain the following fields: ['text']. Select the correct one and provide it as `field='XXX'` to the dataset loading method.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.0


Add documentation about dataset viewer feature

enhancement

New feature or request

Add to the docs more details about the dataset viewer feature in the Hub.

CC: @julien-c


Eventual Invalid Token Error at setup of private datasets

bug

Something isn't working

## Describe the bug
From time to time, there appear Invalid Token errors with private datasets:

- https://app.circleci.com/pipelines/github/huggingface/datasets/8520/workflows/d44629f2-4749-40f8-a657-50931d0b3434/jobs/52534
  ```
  ____________ ERROR at setup of test_load_streaming_private_dataset _____________
  ValueError: Invalid token passed!

  ____ ERROR at setup of test_load_streaming_private_dataset_with_zipped_data ____
  ValueError: Invalid token passed!

  =========================== short test summary info ============================
  ERROR tests/test_load.py::test_load_streaming_private_dataset - ValueError: I...
  ERROR tests/test_load.py::test_load_streaming_private_dataset_with_zipped_data
  ```

- https://app.circleci.com/pipelines/github/huggingface/datasets/8557/workflows/a8383181-ba6d-4487-9d0a-f750b6dcb936/jobs/52763
  ```
  ____ ERROR at setup of test_load_streaming_private_dataset_with_zipped_data ____
  [gw1] linux -- Python 3.6.15 /home/circleci/.pyenv/versions/3.6.15/bin/python3.6

  hf_api = <huggingface_hub.hf_api.HfApi object at 0x7f4899bab908>
  hf_token = 'vgNbyuaLNEBuGbgCEtSBCOcPjZnngJufHkTaZvHwkXKGkHpjBPwmLQuJVXRxBuaRzNlGjlMpYRPbthfHPFWXaaEDTLiqTTecYENxukRYVAAdpeApIUPxcgsowadkTkPj'
  zip_csv_path = PosixPath('/tmp/pytest-of-circleci/pytest-0/popen-gw1/data16/dataset.csv.zip')

      @pytest.fixture(scope="session")
      def hf_private_dataset_repo_zipped_txt_data_(hf_api: HfApi, hf_token, zip_csv_path):
          repo_name = "repo_zipped_txt_data-{}".format(int(time.time() * 10e3))
          hf_api.create_repo(token=hf_token, name=repo_name, repo_type="dataset", private=True)
          repo_id = f"{USER}/{repo_name}"
          hf_api.upload_file(
              token=hf_token,
              path_or_fileobj=str(zip_csv_path),
              path_in_repo="data.zip",
              repo_id=repo_id,
  >           repo_type="dataset",
          )

  tests/hub_fixtures.py:68:

  ...

  ValueError: Invalid token passed!
  =========================== short test summary info ============================
  ERROR tests/test_load.py::test_load_streaming_private_dataset_with_zipped_data
  ```


Fix code quality bug in riddle_sense dataset

Something isn't working

To give more context: https://github.com/psf/black/issues/318. `black` doesn't treat this as a bug, but `flake8` does.

## Describe the bug
```
datasets/riddle_sense/riddle_sense.py:36:21: W291 trailing whitespace
```


Add ACAV100M Dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

## Adding a Dataset
- **Name:** *ACAV100M*
- **Description:** *contains 100 million videos with high audio-visual correspondence, ideal for self-supervised video representation learning.*
- **Paper:** *https://arxiv.org/abs/2101.10803*
- **Data:** *https://github.com/sangho-vision/acav100m*
- **Motivation:** *The largest dataset (to date) for audio-visual learning.*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Sort files before loading

enhancement

New feature or request

This will be fixed by https://github.com/huggingface/datasets/pull/3221

When loading a dataset that consists of several files (e.g. `my_data/data_001.json`, `my_data/data_002.json` etc.) they are not loaded in order when using `load_dataset("my_data")`.

This could lead to counter-intuitive results if, for example, the data files are sorted by date or similar since they would appear in different order in the `Dataset`.

The straightforward solution is to sort the list of files alphabetically before loading them.

cc @lhoestq


dataset-viewer

Related to the dataset viewer on huggingface.co

Hi ! Do you have some kind of proxy in your browser that gives you access to internet ?

Maybe you're having this error because you don't have access to this URL from python ?

when I use python examples/pytorch/translation/run_translation.py     --model_name_or_path examples/pytorch/translation/opus-mt-en-ro     --do_train     --do_eval     --source_lang en     --target_lang ro     --dataset_name wmt16     --dataset_config_name ro-en     --output_dir /tmp/tst-translation     --per_device_train_batch_size=4     --per_device_eval_batch_size=4     --overwrite_output_dir     --predict_with_generate to finetune translation model on huggingface, I get the issue"ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.15.1/datasets/wmt16/wmt16.py".But I can open the https://raw.githubusercontent.com/huggingface/datasets/1.15.1/datasets/wmt16/wmt16.py by using website. What should I do to solve the issue?


Unpin keras once TF fixes its release

Related to:
- #3208


CI error: Another metric with the same name already exists in Keras 2.7.0

bug

Something isn't working

## Describe the bug
Release of TensorFlow 2.7.0 contains an incompatibility with Keras. See:
- keras-team/keras#15579

This breaks our CI test suite: https://app.circleci.com/pipelines/github/huggingface/datasets/8493/workflows/055c7ae2-43bc-49b4-9f11-8fc71f35a25c/jobs/52363


FileNotFoundError for TupleIE dataste

Something isn't working

@mariosasko @lhoestq Could you give me an update on how to load the dataset after the fix?
Thanks.

Hi @arda-vianai,

first, you can try:
```python
import datasets
dataset = datasets.load_dataset('tuple_ie', 'all', revision="master")
```
If this doesn't work, your version of `datasets` is missing some features that are required to run the dataset script, so install the master version with the following command:
```
pip install git+https://github.com/huggingface/datasets.git
```
and then:
```python
import datasets
dataset = datasets.load_dataset('tuple_ie', 'all')
```
should work (even without `revision`).

@mariosasko
Thanks, it is working now. I actually did that before but I didn't restart the kernel. I restarted it and it works now. My bad!!!
Many thanks and great job!
-arda

Hi,
`dataset = datasets.load_dataset('tuple_ie', 'all')`

returns a FileNotFound error. Is the data not available?

Many thanks.


Add mIoU metric

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Recently, some semantic segmentation models were added to HuggingFace Transformers, including [SegFormer](https://huggingface.co/transformers/model_doc/segformer.html) and [BEiT](https://huggingface.co/transformers/model_doc/beit.html).

Semantic segmentation (which is the task of labeling every pixel of an image with a corresponding class) is typically evaluated using the Mean Intersection and Union (mIoU). Together with the upcoming Image Feature, adding this metric could be very handy when creating example scripts to fine-tune any Transformer-based model on a semantic segmentation dataset.

An implementation can be found [here](https://github.com/open-mmlab/mmsegmentation/blob/504965184c3e6bc9ec43af54237129ef21981a5f/mmseg/core/evaluation/metrics.py#L132) for instance.


Add GSM8K dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** GSM8K (short for Grade School Math 8k)
- **Description:** GSM8K is a dataset of 8.5K high quality linguistically diverse grade school math word problems created by human problem writers.
- **Paper:** https://openai.com/blog/grade-school-math/
- **Data:** https://github.com/openai/grade-school-math
- **Motivation:** The dataset is useful to investigate the reasoning abilities of large Transformer models, such as GPT-3.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Update link to datasets-tagging app

Once datasets-tagging has been transferred to Spaces:
- huggingface/datasets-tagging#22

We should update the link in Datasets.


Multiprocessing filter/map (tests) not working on Windows

bug

Something isn't working

While running the tests, I found that the multiprocessing examples fail on Windows, or rather they do not complete: they cause a deadlock. I haven't dug deep into it, but they do not seem to work as-is. I currently have no time to tests this in detail but at least the tests seem not to run correctly (deadlocking).

## Steps to reproduce the bug
```shell
pytest tests/test_arrow_dataset.py -k "test_filter_multiprocessing"
pytest tests/test_arrow_dataset.py -k "test_map_multiprocessing"
```

## Expected results
The functionality to work on all platforms.

## Actual results
Deadlock.

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.2, also tested with 3.7.9
- PyArrow version: 4.0.1


## Dataset viewer issue for '*compguesswhat*'

**Link:** https://huggingface.co/datasets/compguesswhat

File not found

Am I the one who added this dataset ? No


combination of shuffle and filter results in a bug

Something isn't working

I cannot reproduce this on master and pyarrow==4.0.1.

Hi ! There was a regression in `datasets` 1.12 that introduced this bug. It has been fixed in #3019 in 1.13

Can you try to update `datasets` and try again ?

## Describe the bug
Hi,
I would like to shuffle a dataset, then filter it based on each existing label. however, the combination of `filter`, `shuffle` seems to results in a bug. In the minimal example below, as you see in the filtered results, the filtered labels are not unique, meaning filter has not worked. Any suggestions as a temporary fix is appreciated @lhoestq.

Thanks.
 Best regards
Rabeeh

## Steps to reproduce the bug
```python
import numpy as np
import datasets

datasets = datasets.load_dataset('super_glue', 'rte', script_version="master")
shuffled_data = datasets["train"].shuffle(seed=42)
for label in range(2):
    print("label ", label)
    data = shuffled_data.filter(lambda example: int(example['label']) == label)
    print("length ", len(data), np.unique(data['label']))
```

## Expected results
Filtering per label, should only return the data with that specific label.

## Actual results
As you can see, filtered data per label, has still two labels of [0, 1]
```
label  0
length  1249 [0 1]
label  1
length  1241 [0 1]
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: linux
- Python version: 3.7.11
- PyArrow version: 5.0.0


conll2003 incorrect label explanation

Something isn't working

Hi @BramVanroy,

since these fields are of type `ClassLabel` (you can check this with `dset.features`), you can inspect the possible values with:
```python
dset.features[field_name].feature.names  # .feature because it's a sequence of labels
```

and to find the mapping between names and integers, use:
```python
dset.features[field_name].feature.int2str(value_or_values_list)  # map integer value to string value
# or
dset.features[field_name].feature.str2int(value_or_values_list)  # map string value to integer value
```

In the [conll2003](https://huggingface.co/datasets/conll2003#data-fields) README, the labels are described as follows

> - `id`: a `string` feature.
> - `tokens`: a `list` of `string` features.
> - `pos_tags`: a `list` of classification labels, with possible values including `"` (0), `''` (1), `#` (2), `$` (3), `(` (4).
> - `chunk_tags`: a `list` of classification labels, with possible values including `O` (0), `B-ADJP` (1), `I-ADJP` (2), `B-ADVP` (3), `I-ADVP` (4).
> - `ner_tags`: a `list` of classification labels, with possible values including `O` (0), `B-PER` (1), `I-PER` (2), `B-ORG` (3), `I-ORG` (4) `B-LOC` (5), `I-LOC` (6) `B-MISC` (7), `I-MISC` (8).

First of all, it would be great if we can get a list of ALL possible pos_tags.

Second, the chunk tags labels cannot be correct. The description says the values go from 0 to 4 whereas the data shows values from at least 11 to 21 and 0.

EDIT: not really a bug, sorry for mistagging.


conll2002 issues

Hi ! Thanks for reporting :)

This is related to https://github.com/huggingface/datasets/issues/2742, I'm working on it. It should fix the viewer for around 80 datasets.

Ah, hadn't seen that sorry.

The scrambled "point of contact" is a separate issue though, I think.

@lhoestq The "point of contact" is still an issue.

It will be fixed in https://github.com/huggingface/datasets/pull/3274, thanks

**Link:** https://huggingface.co/datasets/conll2002

The dataset viewer throws a server error when trying to preview the dataset.

```
Message:       Extraction protocol 'train' for file at 'https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/esp.train' is not implemented yet
```

In addition, the "point of contact" has encoding issues and does not work when clicked.

Am I the one who added this dataset ? No, @lhoestq  did


Dataset viewer for nli_tr

## Dataset viewer issue for '*nli_tr*'

**Link:** https://huggingface.co/datasets/nli_tr

Hello,

Thank you for the new dataset preview feature that will help the users to view the datasets online.

We just noticed that the dataset viewer widget in the `nli_tr` dataset shows the error below. The error must be due to a temporary problem that may have blocked access to the dataset through the dataset viewer. But the dataset is currently accessible through the link in the error message.   May we kindly ask if it would be possible to rerun the job so that it can access the dataset for the dataset viewer function?

Thank you.
Emrah

------------------------------------------
Server Error
Status code:   404
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: 'zip://snli_tr_1.0_train.jsonl::https://tabilab.cmpe.boun.edu.tr/datasets/nli_datasets/snli_tr_1.0.zip
------------------------------------------

Am I the one who added this dataset ? Yes


7z dataset preview not implemented?

## Dataset viewer issue for dataset 'samsum'

**Link:** https://huggingface.co/datasets/samsum

Server Error
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol '7z' for file at 'https://arxiv.org/src/1911.12237v2/anc/corpus.7z' is not implemented yet


`None` converted to `"None"` when loading a dataset

Something isn't working

Hi @eladsegal, thanks for reporting.

@mariosasko I saw you are already working on this, but maybe my comment will be useful to you.

All values are casted to their corresponding feature type (including `None` values). For example if the feature type is `Value("bool")`, `None` is casted to `False`.

It is true that strings were an exception, but this was recently fixed by @lhoestq (see #3158).

Thanks for reporting.

This is actually a breaking change that I think can cause issues when users preprocess their data. String columns used to be nullable. Maybe we can correct https://github.com/huggingface/datasets/pull/3158 to keep the None values and avoid this breaking change ?

EDIT: the other types (bool, int, etc) can also become nullable IMO

So what would be the best way to handle a feature that can have a null value in some of the instances? So far I used `None`.
Using the empty string won't be a good option, as it can be an actual value in the data and is not the same as not having a value at all.

Hi @eladsegal,

Use `None`. As @albertvillanova correctly pointed out, this change in conversion was introduced (by mistake) in #3158. To avoid it, install the earlier revision with:
```
pip install git+https://github.com/huggingface/datasets.git@8107844ec0e7add005db0585c772ee20adc01a5e
```

I'm making all the feature types nullable as we speak, and the fix will be merged probably early next week.

Hi @mariosasko, is there an estimation as to when this issue will be fixed?

https://github.com/huggingface/datasets/pull/3195 fixed it, we'll do a new release soon :)

For now feel free to install `datasets` from the master branch

Thanks, but unfortunately looks like it isn't fixed yet 😢
[notebook for 1.14.0](https://colab.research.google.com/drive/1SV3sFXPJMWSQgbm4pr9Y1Q8OJ4JYKcDo?usp=sharing)
[notebook for master](https://colab.research.google.com/drive/145wDpuO74MmsuI0SVLcI1IswG6aHpyhi?usp=sharing)

Oh, sorry. I deleted the fix by accident when I was resolving a merge conflict. Let me fix this real quick.

## Describe the bug
When loading a dataset `None` values of the type `NoneType` are converted to `'None'` of the type `str`.

## Steps to reproduce the bug
```python
from datasets import load_dataset
qasper = load_dataset("qasper", split="train", download_mode="reuse_cache_if_exists")
print(qasper[60]["full_text"]["section_name"])
```

When installing version 1.1.40, the output is
`[None, 'Introduction', 'Benchmark Datasets', ...]`

When installing from the master branch, the output is
`['None', 'Introduction', 'Benchmark Datasets', ...]`

Notice how the first element was changed from `NoneType` to `str`.

## Expected results
`None` should stay as is.

## Actual results
`None` is converted to a string.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: master
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 4.0.1


Cannot load dataset when the config name is "special"

Something isn't working

The issue is that the datasets are malformed. Not a bug with the datasets library

## Describe the bug

After https://github.com/huggingface/datasets/pull/3159, we can get the config name of "Check/region_1", which is "Check___region_1".

But now we cannot load the dataset (not sure it's related to the above PR though). It's the case for all the similar datasets, listed in https://github.com/huggingface/datasets-preview-backend/issues/78

## Steps to reproduce the bug
```python
>>> from datasets import get_dataset_config_names
>>> get_dataset_config_names("Check/region_1")
['Check___region_1']
>>> load_dataset("Check/region_1")
Using custom data configuration Check___region_1-d2b3bc48f11c9be2
Downloading and preparing dataset json/Check___region_1 to /home/slesage/.cache/huggingface/datasets/json/Check___region_1-d2b3bc48f11c9be2/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 4443.12it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1277.19it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/load.py", line 1632, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/builder.py", line 1159, in _prepare_split
    writer.write_table(table)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/arrow_writer.py", line 442, in write_table
    pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/arrow_writer.py", line 442, in <listcomp>
    pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
  File "pyarrow/table.pxi", line 1249, in pyarrow.lib.Table.__getitem__
  File "pyarrow/table.pxi", line 1825, in pyarrow.lib.Table.column
  File "pyarrow/table.pxi", line 1800, in pyarrow.lib.Table._ensure_integer_index
KeyError: 'Field "builder_name" does not exist in table schema'
```

Loading in streaming mode also returns something strange:

```python
>>> list(load_dataset("Check/region_1", streaming=True, split="train"))
Using custom data configuration Check___region_1-d2b3bc48f11c9be2
[{'builder_name': None, 'citation': '', 'config_name': None, 'dataset_size': None, 'description': '', 'download_checksums': None, 'download_size': None, 'features': {'speech': {'feature': {'dtype': 'float64', 'id': None, '_type': 'Value'}, 'length': -1, 'id': None, '_type': 'Sequence'}, 'sampling_rate': {'dtype': 'int64', 'id': None, '_type': 'Value'}, 'label': {'dtype': 'string', 'id': None, '_type': 'Value'}}, 'homepage': '', 'license': '', 'post_processed': None, 'post_processing_size': None, 'size_in_bytes': None, 'splits': None, 'supervised_keys': None, 'task_templates': None, 'version': None}, {'_data_files': [{'filename': 'dataset.arrow'}], '_fingerprint': 'f1702bb5533c549c', '_format_columns': ['speech', 'sampling_rate', 'label'], '_format_kwargs': {}, '_format_type': None, '_indexes': {}, '_indices_data_files': None, '_output_all_columns': False, '_split': None}]
```

## Expected results

The dataset should be loaded

## Actual results

An error occurs

## Environment info

- `datasets` version: 1.14.1.dev0
- Platform: Linux-5.11.0-1020-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1


"Property couldn't be hashed properly" even though fully picklable

Something isn't working

After some digging, I found that this is caused by `dill` and using `recurse=True)` when trying to dump the object. The problem also occurs without multiprocessing. I can only find [the following information](https://dill.readthedocs.io/en/latest/dill.html#dill._dill.dumps) about this:

> If recurse=True, then objects referred to in the global dictionary are recursively traced and pickled, instead of the default behavior of attempting to store the entire global dictionary. This is needed for functions defined via exec().

In the utils, this is explicitly enabled

https://github.com/huggingface/datasets/blob/df63614223bf1dd1feb267d39d741bada613352c/src/datasets/utils/py_utils.py#L327-L330

Is this really necessary? Is there a way around it? Also pinging the spaCy team in case this is easy to solve on their end. (I hope so.)

Hi ! Thanks for reporting

Yes `recurse=True` is necessary to be able to hash all the objects that are passed to the `map` function

EDIT: hopefully this object can be serializable soon, but otherwise we can consider adding more control to the user on how to hash objects that are not serializable (as mentioned in https://github.com/huggingface/datasets/issues/3044#issuecomment-948818210)

I submitted a PR to spacy that should fix this issue (linked above). I'll leave this open until that PR is merged.

@lhoestq After some testing I find that even with the updated spaCy, no cache files are used. I do not get any warnings though, but I can see that map is run every time I run the code. Do you have thoughts about why? If you want to try the tests below, make sure to install spaCy from [here](https://github.com/BramVanroy/spaCy) and installing the base model with `python -m spacy download en_core_web_sm`.

```python
from functools import partial
from pathlib import Path

import spacy
from datasets import Dataset
import datasets
datasets.logging.set_verbosity_debug()

def tokenize(nlp, l):
    return {"tok": [t.text for t in nlp(l["text"])]}

def main():
    fin = r"some/file/with/many/lines"
    lines = Path(fin).read_text(encoding="utf-8").splitlines()
    nlp = spacy.load("en_core_web_sm")
    ds = Dataset.from_dict({"text": lines, "text_id": list(range(len(lines)))})
    tok = partial(tokenize, nlp)
    ds = ds.map(tok, load_from_cache_file=True)
    print(ds[0:2])

if __name__ == '__main__':
    main()
```

... or with load_dataset (here I get the message that `load_dataset` can reuse the dataset, but still I see all samples being processed via the tqdm progressbar):

```python
from functools import partial

import spacy
from datasets import load_dataset
import datasets
datasets.logging.set_verbosity_debug()

def tokenize(nlp, sample):
    return {"tok": [t.text for t in nlp(sample["text"])]}

def main():
    fin = r"some/file/with/many/lines"
    nlp = spacy.load("en_core_web_sm")
    tok_func = partial(tokenize, nlp)
    ds = load_dataset('text', data_files=fin)
    ds = ds["train"].map(tok_func)
    print(ds[0:2])

if __name__ == '__main__':
    main()
```

It looks like every time you load `en_core_web_sm` you get a different python object:
```python
import spacy
from datasets.fingerprint import Hasher

nlp1 = spacy.load("en_core_web_sm")
nlp2 = spacy.load("en_core_web_sm")
Hasher.hash(nlp1), Hasher.hash(nlp2)
# ('f6196a33882fea3b', 'a4c676a071f266ff')
```
Here is a list of attributes that have different hashes for `nlp1` and `nlp2`:
- tagger
- parser
- entity
- pipeline (it's the list of the three attributes above)

I just took a look at the tagger for example and I found subtle differences (there may be other differences though):
```python
nlp1.tagger.model.tok2vec.embed.id, nlp2.tagger.model.tok2vec.embed.id
# (1721, 2243)
```

We can try to find all the differences and find the best way to hash those objects properly

Thanks for searching! I went looking, and found that this is an implementation detail of thinc

https://github.com/explosion/thinc/blob/68691e303ae68cae4bc803299016f1fc064328bf/thinc/model.py#L96-L98

Presumably (?) exactly to distinguish between different parts in memory when multiple models are loaded. Do not think that this can be changed on their end - but I will ask what exactly it is for (I'm curious).

Do you think it is overkill to write something into the hasher explicitly to deal with spaCy models? It seems like something that is beneficial to many, but I do not know if you are open to adding third-party-specific ways to deal with this. If you are, I can have a look for this specific case how we can ignore `thinc.Model.id` from the hasher.

It can be even simpler to hash the bytes of the pipeline instead
```python
nlp1.to_bytes() == nlp2.to_bytes()  # True
```

IMO we should integrate the custom hashing for spacy models into `datasets` (we use a custom Pickler for that).
What could be done on Spacy's side instead (if they think it's nice to have) is to implement a custom pickling for these classes using `to_bytes`/`from_bytes` to have deterministic pickle dumps.

Finally I think it would be nice in the future to add an API to let `datasets` users control this kind of things. Something like being able to define your own hashing if you use complex objects.
```python
@datasets.register_hash(spacy.language.Language)
def hash_spacy_language(nlp):
    return Hasher.hash(nlp.to_bytes())
```

I do not quite understand what you mean. as far as I can tell, using `to_bytes` does a pickle dump behind the scene (with `srsly`), recursively using `to_bytes` on the required objects. Therefore, the result of `to_bytes` is a deterministic pickle dump AFAICT. Or do you mean that you wish that using your own pickler and running `dumps(nlp)` should also be deterministic? I guess that would require `__setstate__` and `__getstate__` methods on all the objects that have to/from_bytes. I'll have a listen over at spaCy what they think, and if that would solve the issue. I'll try this locally first, if I find the time.

I agree that having the option to use a custom hasher would be useful. I like your suggestion!

EDIT: after trying some things and reading through their API, it seems that they explicitly do not want this. https://spacy.io/usage/saving-loading#pipeline

> When serializing the pipeline, keep in mind that this will only save out the binary data for the individual components to allow spaCy to restore them – not the entire objects. This is a good thing, because it makes serialization safe. But it also means that you have to take care of storing the config, which contains the pipeline configuration and all the relevant settings.

Best way forward therefore seems to implement the ability to specify a hasher depending on the objects that are pickled, as you suggested. I can work on this if that is useful. I could use some pointers as to how you would like to implement the `register_hash` functionality though. I assume using `catalogue` over at Explosion might be a good starting point.

Interestingly, my PR does not solve the issue discussed above. The `tokenize` function hash is different on every run, because for some reason `nlp.__call__` has a different hash every time. The issue therefore seems to run much deeper than I thought. If you have any ideas, I'm all ears.

```shell
git clone https://github.com/explosion/spaCy.git
cd spaCy/
git checkout cab9209c3dfcd1b75dfe5657f10e52c4d847a3cf
cd ..

git clone https://github.com/BramVanroy/datasets.git
cd datasets
git checkout registry
pip install -e .
pip install ../spaCy
spacy download en_core_web_sm
```

```python
import spacy

from datasets import load_dataset
from datasets.fingerprint import Hasher
from datasets.utils.registry import hashers

@hashers.register(spacy.Language)
def hash_spacy_language(nlp):
    return Hasher.hash(nlp.to_bytes())

def main():
    fin = r"your/large/file"
    nlp = spacy.load("en_core_web_sm")
    # This is now always the same yay!
    print(Hasher.hash(nlp))

    def tokenize(l):
        return {"tok": [t.text for t in nlp(l["text"])]}

    ds =  load_dataset("text", data_files=fin)
    # But this is not...
    print(Hasher.hash(tokenize))
    # ... because of this
    print(Hasher.hash(nlp.__call__))
    ds = ds["train"].map(tokenize)
    print(ds[0:2])

if __name__ == '__main__':
    main()
```

Hi ! I just answered in your PR :) In order for your custom hashing to be used for nested objects, you must integrate it into our recursive pickler that we use for hashing.

I don't quite understand the design constraints of `datasets` or the script that you're running, but my usual advice is to avoid using pickle unless you _absolutely_ have to. So for instance instead of doing your `partial` over the `nlp` object itself, can you just pass the string `en_core_web_sm` in? This will mean calling `spacy.load()` inside the work function, but this is no worse than having to call `pickle.load()` on the contents of the NLP object anyway -- in fact you'll generally find `spacy.load()` faster, apart from the disk read.

If you need to pass in the bytes data and don't want to read from disk, you could do something like this:

```
msg = (nlp.lang, nlp.to_bytes())

def unpack(lang, bytes_data):
    return spacy.blank(lang).from_bytes(bytes_data)
```

I think that should probably work: the Thinc `model.to_dict()` method (which is used by the `model.to_bytes()` method) doesn't pack the model's ID into the message, so the `nlp.to_bytes()` that you get shouldn't be affected by the global IDs. So you should get a clean message from `nlp.to_bytes()` that doesn't depend on the global state.

Hi Matthew, thanks for chiming in! We are currently implementing exactly what you suggest: `to_bytes()` as a default before pickling - but we may prefer `to_dict` to avoid double dumping.

`datasets` uses pickle dumps (actually dill) to get unique representations of processing steps (a "fingerprint" or hash). So it never needs to re-load that dump - it just needs its value to create a hash. If a fingerprint is identical to a cached fingerprint, then the result can be retrieved from the on-disk cache. (@lhoestq or @mariosasko can correct me if I'm wrong.)

I was experiencing the issue that parsing with spaCy gave me a different fingerprint on every run of the script and thus it could never load the processed dataset from cache. At first I thought the reason was that spaCy Language objects were not picklable with recursive dill, but even after [adjusting for that](https://github.com/explosion/spaCy/pull/9593) the issue persisted. @lhoestq found that this is due to the changing `id`, which you discussed [here](https://github.com/explosion/spaCy/discussions/9609#discussioncomment-1661081). So yes, you are right. On the surface there simply seems to be an incompatibility between `datasets` default caching functionality as it is currently implemented and `spacy.Language`.

The [linked PR](https://github.com/huggingface/datasets/pull/3224) aims to remedy that, though. Up to now I have put some effort into making it easier to define your own "pickling" function for a given type (and optionally any of its subclasses). That allows us to tell `datasets` that instead of doing `dill.save(nlp)` (non-deterministic), to use `dill.save(nlp.to_bytes())` (deterministic). When I find some more time, the PR [will be expanded](https://github.com/huggingface/datasets/pull/3224#issuecomment-968958528) to improve the user-experience a bit and add a built-in function to pickle `spacy.Language` as one of the defaults (using `to_bytes()`).

## Describe the bug
I am trying to tokenize a dataset with spaCy. I found that no matter what I do, the spaCy language object (`nlp`) prevents `datasets` from pickling correctly - or so the warning says - even though manually pickling is no issue. It should not be an issue either, since spaCy objects are picklable.

## Steps to reproduce the bug

Here is a [colab](https://colab.research.google.com/drive/1gt75LCBIzsmBMvvipEOvWulvyZseBiA7?usp=sharing) but for some reason I cannot reproduce it there. That may have to do with logging/tqdm on Colab, or with running things in notebooks. I tried below code on Windows and Ubuntu as a Python script and getting the same issue (warning below).

```python
import pickle

from datasets import load_dataset
import spacy

class Processor:
    def __init__(self):
        self.nlp = spacy.load("en_core_web_sm", disable=["tagger", "parser", "ner", "lemmatizer"])

    @staticmethod
    def collate(batch):
        return [d["en"] for d in batch]

    def parse(self, batch):
        batch = batch["translation"]
        return {"translation_tok": [{"en_tok": " ".join([t.text for t in doc])} for doc in self.nlp.pipe(self.collate(batch))]}

    def process(self):
        ds = load_dataset("wmt16", "de-en", split="train[:10%]")
        ds = ds.map(self.parse, batched=True, num_proc=6)

if __name__ == '__main__':
    pr = Processor()

    # succeeds
    with open("temp.pkl", "wb") as f:
        pickle.dump(pr, f)
    print("Successfully pickled!")

    pr.process()

```

---

Here is a small change that includes `Hasher.hash` that shows that the hasher cannot seem to successfully pickle parts form the NLP object.

```python

from datasets.fingerprint import Hasher
import pickle

from datasets import load_dataset
import spacy

class Processor:
    def __init__(self):
        self.nlp = spacy.load("en_core_web_sm", disable=["tagger", "parser", "ner", "lemmatizer"])

    @staticmethod
    def collate(batch):
        return [d["en"] for d in batch]

    def parse(self, batch):
        batch = batch["translation"]
        return {"translation_tok": [{"en_tok": " ".join([t.text for t in doc])} for doc in self.nlp.pipe(self.collate(batch))]}

    def process(self):
        ds = load_dataset("wmt16", "de-en", split="train[:10]")
        return ds.map(self.parse, batched=True)

if __name__ == '__main__':
    pr = Processor()

    # succeeds
    with open("temp.pkl", "wb") as f:
        pickle.dump(pr, f)
    print("Successfully pickled class instance!")

    # succeeds
    with open("temp.pkl", "wb") as f:
        pickle.dump(pr.nlp, f)
    print("Successfully pickled nlp!")

    # fails
    print(Hasher.hash(pr.nlp))
    pr.process()
```

## Expected results
This to be picklable, working (fingerprinted), and no warning.

## Actual results
In the first snippet, I get this warning
Parameter 'function'=<function Processor.parse at 0x7f44982247a0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.

In the second, I get this traceback which directs to the `Hasher.hash` line.

```
Traceback (most recent call last):
  File " \Python\Python36\lib\pickle.py", line 918, in save_global
    obj2, parent = _getattribute(module, name)
  File " \Python\Python36\lib\pickle.py", line 266, in _getattribute
    .format(name, obj))
AttributeError: Can't get local attribute 'add_codes.<locals>.ErrorsWithCodes' on <function add_codes at 0x00000296FF606EA0>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File " scratch_4.py", line 40, in <module>
    print(Hasher.hash(pr.nlp))
  File " \lib\site-packages\datasets\fingerprint.py", line 191, in hash
    return cls.hash_default(value)
  File " \lib\site-packages\datasets\fingerprint.py", line 184, in hash_default
    return cls.hash_bytes(dumps(value))
  File " \lib\site-packages\datasets\utils\py_utils.py", line 345, in dumps
    dump(obj, file)
  File " \lib\site-packages\datasets\utils\py_utils.py", line 320, in dump
    Pickler(file, recurse=True).dump(obj)
  File " \lib\site-packages\dill\_dill.py", line 498, in dump
    StockPickler.dump(self, obj)
  File " \Python\Python36\lib\pickle.py", line 409, in dump
    self.save(obj)
  File " \Python\Python36\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File " \Python\Python36\lib\pickle.py", line 634, in save_reduce
    save(state)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \lib\site-packages\dill\_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File " \Python\Python36\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File " \Python\Python36\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \Python\Python36\lib\pickle.py", line 781, in save_list
    self._batch_appends(obj)
  File " \Python\Python36\lib\pickle.py", line 805, in _batch_appends
    save(x)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \Python\Python36\lib\pickle.py", line 736, in save_tuple
    save(element)
  File " \Python\Python36\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File " \Python\Python36\lib\pickle.py", line 634, in save_reduce
    save(state)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \Python\Python36\lib\pickle.py", line 736, in save_tuple
    save(element)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \lib\site-packages\dill\_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File " \Python\Python36\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File " \Python\Python36\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \lib\site-packages\dill\_dill.py", line 1176, in save_instancemethod0
    pickler.save_reduce(MethodType, (obj.__func__, obj.__self__), obj=obj)
  File " \Python\Python36\lib\pickle.py", line 610, in save_reduce
    save(args)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \Python\Python36\lib\pickle.py", line 736, in save_tuple
    save(element)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \lib\site-packages\datasets\utils\py_utils.py", line 523, in save_function
    obj=obj,
  File " \Python\Python36\lib\pickle.py", line 610, in save_reduce
    save(args)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \Python\Python36\lib\pickle.py", line 751, in save_tuple
    save(element)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \lib\site-packages\dill\_dill.py", line 990, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File " \Python\Python36\lib\pickle.py", line 821, in save_dict
    self._batch_setitems(obj.items())
  File " \Python\Python36\lib\pickle.py", line 847, in _batch_setitems
    save(v)
  File " \Python\Python36\lib\pickle.py", line 521, in save
    self.save_reduce(obj=obj, *rv)
  File " \Python\Python36\lib\pickle.py", line 605, in save_reduce
    save(cls)
  File " \Python\Python36\lib\pickle.py", line 476, in save
    f(self, obj) # Call unbound method with explicit self
  File " \lib\site-packages\dill\_dill.py", line 1439, in save_type
    StockPickler.save_global(pickler, obj, name=name)
  File " \Python\Python36\lib\pickle.py", line 922, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class 'spacy.errors.add_codes.<locals>.ErrorsWithCodes'>: it's not found as spacy.errors.add_codes.<locals>.ErrorsWithCodes
```

## Environment info
Tried on both Linux and Windows

- `datasets` version: 1.14.0
- Platform: Windows-10-10.0.19041-SP0 + Python 3.7.9; Linux-5.11.0-38-generic-x86_64-with-Ubuntu-20.04-focal + Python 3.7.12
- PyArrow version: 6.0.0


More control over TQDM when using map/filter with multiple processes

enhancement

New feature or request

Hi,

It's hard to provide an API that would cover all use-cases with tqdm in this project.

However, you can make it work by defining a custom decorator (a bit hacky tho) as follows:
```python
import datasets

def progress_only_on_rank_0(func):
    def wrapper(*args, **kwargs):
        rank = kwargs.get("rank")
        disable_tqdm = kwargs.get("disable_tqdm", False)
        disable_tqdm = True if rank is not None and rank > 0 else disable_tqdm
        kwargs["disable_tqdm"] = disable_tqdm
        return func(*args, **kwargs)
    return wrapper

datasets.Dataset._map_single = progress_only_on_rank_0(datasets.Dataset._map_single)
```

EDIT: Ups, closed by accident.

Thanks for the provided links. `Trainer` requires this for training in multi-node distributed setting. However, `Dataset.map` doesn't support that yet.

Do you have an API for this in mind? `Dataset.map` is already bloated with the arguments, so IMO it's not a good idea to add a new arg there.

Inspiration may be found at `transformers`.

https://github.com/huggingface/transformers/blob/4a394cf53f05e73ab9bbb4b179a40236a5ffe45a/src/transformers/trainer.py#L1231-L1233

To get unique IDs for each worker, see https://stackoverflow.com/a/10192611/1150683

It would help with the clutter in my terminal if tqdm is only shown for rank 0 when using `num_proces>0` in the map and filter methods of datasets.

```python
dataset.map(lambda examples: tokenize(examples["text"]), batched=True, num_proc=6)
```

The above snippet leads to a lot of TQDM bars and depending on your terminal, these will not overwrite but keep pushing each other down.

```
 #0:   0%|          | 0/13 [00:00<?, ?ba/s]
 #1:   0%|          | 0/13 [00:00<?, ?ba/s]
 #2:   0%|          | 0/13 [00:00<?, ?ba/s]
 #3:   0%|          | 0/13 [00:00<?, ?ba/s]
 #4:   0%|          | 0/13 [00:00<?, ?ba/s]
 #5:   0%|          | 0/13 [00:00<?, ?ba/s]
 #0:   8%|          | 1/13 [00:00<?, ?ba/s]
 #1:   8%|          | 1/13 [00:00<?, ?ba/s]
...
```

Instead, it would be welcome if we had the option to only show the progress of rank 0.


`SystemError 15` thrown in `Dataset.__del__` when using `Dataset.map()` with `num_proc>1`

bug

Something isn't working

NB: even if the error is raised, the dataset is successfully cached. So restarting the script after every `map()` allows to ultimately run the whole preprocessing. But this prevents to realistically run the code over multiple nodes.

Hi,

It's not easy to debug the problem without the script. I may be wrong since I'm not very familiar with PyTorch Lightning, but shouldn't you preprocess the data in the `prepare_data` function of `LightningDataModule` and not in the `setup` function.
As you can't modify the module state in `prepare_data` (according to the docs), use the `cache_file_name` argument in `Dataset.map` there, and reload the processed data in `setup` with `Dataset.from_file(cache_file_name)`. If `num_proc>1`, check the docs on the `suffix_template` argument of `Dataset.map` to get an idea what the final `cache_file_names` are going to be.

Let me know if this helps.

Hi @mariosasko, thank you for the hint, that helped me to move forward with that issue.

I did a major refactoring of my project to disentangle my `LightningDataModule` and `Dataset`.  Just FYI, it looks like:

```python
class Builder():
   def __call__() -> DatasetDict:
      # load and preprocess the data
      return dataset

class DataModule(LightningDataModule):
    def prepare_data():
       self.builder()
    def setup():
       self.dataset = self.builder()
```

Unfortunately, the entanglement between `LightningDataModule` and `Dataset` was not the issue.

The culprit was `hydra` and a slight adjustment of the structure of my project solved this issue. The problematic project  structure was:

```
src/
 | - cli.py
 | - training/
      | -experiment.py

# code in experiment.py
def run_experiment(config):
     # preprocess data and run

# code in cli.py
@hydra.main(...)
def run(config):
    return run_experiment(config)
```

Moving `run()` from `clip.py` to `training.experiment.py` solved the issue with `SystemError 15`. No idea why.

Even if the traceback was referring to `Dataset.__del__`, the problem does not seem to be primarily related to `datasets`, so I will close this issue. Thank you for your help!

## Describe the bug
I use `datasets.map` to preprocess some data in my application. The error `SystemError 15` is thrown at the end of the execution of `Dataset.map()` (only with `num_proc>1`. Traceback included bellow.

The exception is raised only when the code runs within a specific context. Despite ~10h spent investigating this issue, I have failed to isolate the bug, so let me describe my setup.

In my project, `Dataset` is wrapped into a `LightningDataModule` and the data is preprocessed when calling `LightningDataModule.setup()`. Calling `.setup()` in an isolated script works fine (even when wrapped with `hydra.main()`). However, when calling `.setup()` within the experiment script (depends on `pytorch_lightning`), the script crashes and `SystemError 15`.

I could avoid throwing this error by modifying ` Dataset.__del__()` (see bellow), but I believe this only moves the problem somewhere else. I am completely stuck with this issue, any hint would be welcome.

```python
class Dataset()
    ...
    def __del__(self):
        if hasattr(self, "_data"):
            _ = self._data # <- ugly trick that allows avoiding the issue.
            del self._data
        if hasattr(self, "_indices"):
            del self._indices
```

## Steps to reproduce the bug
```python
# Unfortunately I couldn't isolate the bug.
```

## Expected results
Calling `Dataset.map()` without throwing an exception. Or at least raising a more detailed exception/traceback.

## Actual results
```
Exception ignored in: <function Dataset.__del__ at 0x7f7cec179160>███████████████████████████████████████████████████| 5/5 [00:05<00:00,  1.17ba/s]
Traceback (most recent call last):
  File ".../python3.8/site-packages/datasets/arrow_dataset.py", line 906, in __del__
    del self._data
  File ".../python3.8/site-packages/ray/worker.py", line 1033, in sigterm_handler
    sys.exit(signum)
SystemExit: 15

```

## Environment info

Tested on 2 environments:

**Environment 1.**
- `datasets` version: 1.14.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 6.0.0

**Environment 2.**
- `datasets` version: 1.14.0
- Platform: Linux-4.18.0-305.19.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.7
- PyArrow version: 6.0.0


Raise exceptions instead of using assertions for control flow

good first issue

Good for newcomers

Adding the remaining tasks for this issue to help new code contributors.
$ cd src/datasets && ack assert -lc
- [x] commands/convert.py:1
- [x] arrow_reader.py:3
- [x] load.py:7
- [x] utils/py_utils.py:2
- [x] features/features.py:9
- [x] arrow_writer.py:7
- [x] search.py:6
- [x] table.py:1
- [x] metric.py:3
- [x] tasks/image_classification.py:1
- [x] arrow_dataset.py:17
- [x] fingerprint.py:6
- [x] io/json.py:1
- [x] io/csv.py:1

Hi all,
I am interested in taking up `fingerprint.py`, `search.py`, `arrow_writer.py` and `metric.py`. Will raise a PR soon!

Let me look into `arrow_dataset.py`, `table.py`, `data_files.py` & `features.py`

Motivated by https://github.com/huggingface/transformers/issues/12789 in Transformers, one welcoming change would be replacing assertions with proper exceptions. The only type of assertions we should keep are those used as sanity checks.

Currently, there is a total of 87 files with the `assert` statements (located under `datasets` and `src/datasets`), so when working on this, to manage the PR size, only modify 4-5 files at most before submitting a PR.


OpenSLR/83 is empty

bug

Something isn't working

Hi @tyrius02, thanks for reporting. I see you self-assigned this issue: are you working on this?

@albertvillanova Yes. Figured I introduced the broken config, I should fix it too.

I've got it working, but I'm struggling with one of the tests. I've started a PR so I/we can work through it.

Looks like the tests all passed on the PR.

## Describe the bug
As the summary says, openslr / SLR83 / train is empty.

The dataset returned after loading indicates there are **zero** rows. The correct number should be **17877**.
## Steps to reproduce the bug
```python
import datasets

datasets.load_dataset('openslr', 'SLR83')
```

## Expected results
```
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'sentence'],
        num_rows: 17877
    })
})
```
## Actual results
```
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'sentence'],
        num_rows: 0
    })
})
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.1.dev0 (master HEAD)
- Platform: Ubuntu 20.04
- Python version: 3.7.10
- PyArrow version: 3.0.0


bookcorpusopen no longer works

Something isn't working

Hi ! Thanks for reporting :) I think #3280 should fix this

I tried with the latest changes from #3280 on google colab and it worked fine :)
We'll do a new release soon, in the meantime you can use the updated version with:
```python
load_dataset("bookcorpusopen", revision="master")
```

## Describe the bug

When using the latest version of datasets (1.14.0), I cannot use the `bookcorpusopen` dataset. The process blocks always around `9924 examples [00:06, 1439.61 examples/s]` when preparing the dataset. I also noticed that after half an hour the process is automatically killed because of the RAM usage (the machine has 1TB of RAM...).

This did not happen with 1.4.1.
I tried also `rm -rf ~/.cache/huggingface` but did not help.
Changing python version between 3.7, 3.8 and 3.9 did not help too.

## Steps to reproduce the bug
```python
import datasets
d = datasets.load_dataset('bookcorpusopen')
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Linux-5.4.0-1054-aws-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 4.0.1


Deprecate prepare_module

In version 1.13, `prepare_module` was deprecated.

Add deprecation warning and remove its usage from all the library.


Add raw data files to the Hub with GitHub LFS for canonical dataset

Hi @zlucia, I would actually suggest hosting the dataset as a huggingface.co-hosted dataset.

The only difference with a "canonical"/legacy dataset is that it's nested under an organization (here `stanford` or `stanfordnlp` for instance – completely up to you) but then you can upload your data using git-lfs (unlike "canonical" datasets where we don't host the data)

Let me know if this fits your use case!

cc'ing @osanseviero @lhoestq and rest of the team 🤗

Hi @zlucia,

As @julien-c pointed out, the way to store/host raw data files in our Hub is by using what we call "community" datasets:
- either at your personal namespace: `load_dataset("zlucia/casehold")`
- or at an organization namespace: for example, if you create the organization `reglab`, then `load_dataset("reglab/casehold")`

Please note that "canonical" datasets do not normally store/host their raw data at our Hub, but in a third-party server. For "canonical" datasets, we just host the "loading script", that is, a Python script that downloads the raw data from a third-party server, creates the HuggingFace dataset from it and caches it locally.

In order to create an organization namespace in our Hub, please follow this link: https://huggingface.co/organizations/new

There are already many organizations at our Hub (complete list here: https://huggingface.co/organizations), such as:
- Stanford CRFM: https://huggingface.co/stanford-crfm
- Stanford NLP: https://huggingface.co/stanfordnlp
- Stanford CS329S: Machine Learning Systems Design: https://huggingface.co/stanford-cs329s

Also note that you in your organization namespace:
- you can add any number of members
- you can store both raw datasets and models, and those can be immediately accessed using `datasets` and `transformers`

Once you have created an organization, these are the steps to upload/host a raw dataset:
- The no-code procedure: https://huggingface.co/docs/datasets/upload_dataset.html
- Using the command line (terminal): https://huggingface.co/docs/datasets/share.html#add-a-community-dataset

Please, feel free to ping me if you have any further questions or need help.

Ah I see, I think I was unclear whether there were benefits to uploading a canonical dataset vs. a community provided dataset. Thanks for clarifying. I'll see if we want to create an organization namespace and otherwise, will upload the dataset under my personal namespace.

I'm interested in sharing the CaseHOLD dataset (https://arxiv.org/abs/2104.08671) as a canonical dataset on the HuggingFace Hub and would like to add the raw data files to the Hub with GitHub LFS, since it seems like a more sustainable long term storage solution, compared to other storage solutions available to my team. From what I can tell, this option is not immediately supported if one follows the sharing steps detailed here: [https://huggingface.co/docs/datasets/share_dataset.html#sharing-a-canonical-dataset](https://huggingface.co/docs/datasets/share_dataset.html#sharing-a-canonical-dataset), since GitHub LFS is not supported for public forks. Is there a way to request this? Thanks!


`datasets-cli test` should work with datasets without scripts

enhancement

New feature or request

> It would be really useful to be able to run `datasets-cli test`for datasets that don't have scripts attached to them (whether the datasets are private or not).
>
> I wasn't able to run the script for a private test dataset that I had created on the hub (https://huggingface.co/datasets/huggingface/DataMeasurementsTest/tree/main) -- although @lhoestq came to save the day!

why don't you try to share that info with people, so you can also save some days.

Hi ! You can run the command if you download the repository
```
git clone https://huggingface.co/datasets/huggingface/DataMeasurementsTest
```
and run the command
```
datasets-cli test DataMeasurementsTest/DataMeasurementsTest.py
```

(though on my side it doesn't manage to download the data since the dataset is private ^^)

> Hi ! You can run the command if you download the repository
>
> ```
> git clone https://huggingface.co/datasets/huggingface/DataMeasurementsTest
> ```
>
> and run the command
>
> ```
> datasets-cli test DataMeasurementsTest/DataMeasurementsTest.py
> ```
>
> (though on my side it doesn't manage to download the data since the dataset is private ^^)

Hi! Thanks for the info.
git cannot find the repository. Do you know if they have depreciated these tests and created a new one?

> I think it's become private, but feel free to try with any other dataset like `lhoestq/test` for example at `https://huggingface.co/datasets/lhoestq/test`

your example repo and this page `https://huggingface.co/docs/datasets/add_dataset.html` helped me to solve.. thanks a lot

It would be really useful to be able to run `datasets-cli test`for datasets that don't have scripts attached  to them (whether the datasets are private or not).

I wasn't able to run the script for a private test dataset that I had created on the hub (https://huggingface.co/datasets/huggingface/DataMeasurementsTest/tree/main) -- although @lhoestq came to save the day!


Rouge and Meteor for multiple references

bug

Something isn't working

Hi @avinashsai ,

currently, multiple references are not supported. However, we could add a `multiref` config to fix that. When working with multiple references, we can accumulate them by either taking an average or the best score. Would you like to work on that?

Hi,

Currently rogue and meteor supports only single references. Can we use these metrics to calculate for multiple references?


Illegal instruction (core dumped) at datasets import

Something isn't working

It seems to be an issue with how conda-forge is building the binaries. It works on some machines, but not a machine with AMD Opteron 8384 processors.

## Describe the bug
I install datasets using conda and when I import datasets I get: "Illegal instruction (core dumped)"

## Steps to reproduce the bug

```
conda create --prefix path/to/env
conda activate path/to/env
conda install -c huggingface -c conda-forge datasets
# exits with output "Illegal instruction (core dumped)"
python -m datasets
```

## Environment info

When I run "datasets-cli env", I also get "Illegal instruction (core dumped)"

If I run the following commands:

```
conda create --prefix path/to/another/new/env
conda activate path/to/another/new/env
conda install -c huggingface transformers
transformers-cli env
```

Then I get:

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-67-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Let me know what additional information you need in order to debug this issue. Thanks in advance!


Sacrebleu unexpected behaviour/requirement for data format

Something isn't working

Hi @BramVanroy!

Good question. This project relies on PyArrow (tables) to store data too big to fit in RAM. In the case of metrics, this means that the number of predictions and references has to match to form a table.

That's why your example throws an error even though it matches the schema:
```python
refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]  # len(refs) = 2

hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.'] # len(hyps) = 3
```

Instead, it should be:
```python
refs = [
    ['The dog bit the man.', 'The dog had bit the man.'],
    ['It was not unexpected.', 'No one was surprised.'],
    ['The man bit him first.', 'The man had bitten the dog.'],
]  # len(refs) = 3

hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.'] # len(hyps) = 3
```

However, `sacreblue` works with the format that's described in your example, hence this part:
https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/metrics/sacrebleu/sacrebleu.py#L94-L99

Hope you get an idea!

Thanks, that makes sense. It is a bit unfortunate because it may be confusing to users since the input format is suddenly different than what they may expect from the underlying library/metric. But it is understandable due to how `datasets` works!

## Describe the bug
When comparing with the original `sacrebleu` implementation, the `datasets` implementation does some strange things that I do not quite understand. This issue was triggered when I was trying to implement TER and found the datasets implementation of BLEU [here](https://github.com/huggingface/datasets/pull/3153).

In the below snippet, the original sacrebleu snippet works just fine whereas the datasets implementation throws an error.

## Steps to reproduce the bug
```python
import sacrebleu
import datasets

refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]

hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

expected_bleu = 48.530827

ds_bleu = datasets.load_metric("sacrebleu")

bleu_score_sb = sacrebleu.corpus_bleu(hyps, refs).score
print(bleu_score_sb, expected_bleu)
# works: 48.5308...
bleu_score_ds = ds_bleu.compute(predictions=hyps, references=refs)["score"]
print(bleu_score_ds, expected_bleu)
# ValueError: Predictions and/or references don't match the expected format.
```
This seems to be related to how datasets forces the features format here:

https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/metrics/sacrebleu/sacrebleu.py#L94-L99

and then manipulates the references during the compute stage here

https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/metrics/sacrebleu/sacrebleu.py#L119-L122

I do not quite understand why that is required since sacrebleu handles argument parsing quite well [by itself](https://github.com/mjpost/sacrebleu/blob/2787185dd0f8d224c72ee5a831d163c2ac711a47/sacrebleu/metrics/base.py#L229).
## Actual results
Traceback (most recent call last):
  File "C:\Users\bramv\AppData\Roaming\JetBrains\PyCharm2020.3\scratches\scratch_23.py", line 23, in <module>
    bleu_score_ds = ds_bleu.compute(predictions=hyps, references=refs)["score"]
  File "C:\dev\python\datasets\src\datasets\metric.py", line 392, in compute
    self.add_batch(predictions=predictions, references=references)
  File "C:\dev\python\datasets\src\datasets\metric.py", line 439, in add_batch
    raise ValueError(
ValueError: Predictions and/or references don't match the expected format.
Expected format: {'predictions': Value(dtype='string', id='sequence'), 'references': Sequence(feature=Value(dtype='string', id='sequence'), length=-1, id='references')},
Input predictions: ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.'],
Input references: [['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'], ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.']]

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.2
- PyArrow version: 4.0.1


Faiss _is_ available on Windows

In the setup file, I find the following:

https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/setup.py#L171

However, FAISS does install perfectly fine on Windows on my system. You can also confirm this on the [PyPi page](https://pypi.org/project/faiss-cpu/#files), where Windows wheels are available. Maybe this was true for older versions? For current versions, this can be removed I think.

(This isn't really a bug but didn't know how else to tag.)

If you agree I can do a quick PR and remove that line.


Streaming with num_workers != 0

Something isn't working

## Describe the bug
When using dataset streaming with pytorch DataLoader, the setting num_workers to anything other than 0 causes the code to freeze forever before yielding the first batch.

The code owner is likely @lhoestq

## Steps to reproduce the bug

For your convenience, we've prepped a colab notebook that reproduces the bug
https://colab.research.google.com/drive/1Mgl0oTZSNIE3UeGl_oX9wPCOIxRg19h1?usp=sharing
```python
!pip install datasets==1.14.0

should_freeze_forever = True
# ^-- set this to True in order to freeze forever, set to False in order to work normally

import torch
from datasets import load_dataset

data = load_dataset("oscar", "unshuffled_deduplicated_bn", split="train", streaming=True)
data = data.map(lambda x: {"text": x["text"], "orig": f"oscar[{x['id']}]"}, batched=True)
data = data.shuffle(100, seed=1337)

data = data.with_format("torch")
loader = torch.utils.data.DataLoader(data, batch_size=2, num_workers=2 if should_freeze_forever else 0)

# v-- the code should freeze forever at this line
for i, row in enumerate(loader):
  print(row)
  if i > 10: break
print("DONE!")
```

## Expected results
The code should not freeze forever with num_workers=2

## Actual results
The code freezes forever with num_workers=2

## Environment info
- `datasets` version: 1.14.0 (also found in previous versions)
- Platform: google colab (also locally)
- Python version: 3.7, (also 3.8)
- PyArrow version: 3.0.0


CLI test command throws NonMatchingSplitsSizesError when saving infos

Something isn't working

When trying to generate a datset JSON metadata, a `NonMatchingSplitsSizesError` is thrown:
```
$ datasets-cli test datasets/arabic_billion_words --save_infos --all_configs
Testing builder 'Alittihad' (1/10)
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: Unknown size, post-processed: Unknown size, total: 332.13 MiB) to .cache\arabic_billion_words\Alittihad\1.1.0\8175ff1c9714c6d5d15b1141b6042e5edf048276bb81a9c14e35e149a7a62ae4...
Traceback (most recent call last):
  File "path\huggingface\datasets\.venv\Scripts\datasets-cli-script.py", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "path\huggingface\datasets\src\datasets\commands\datasets_cli.py", line 33, in main
    service.run()
  File "path\huggingface\datasets\src\datasets\commands\test.py", line 144, in run
    builder.download_and_prepare(
  File "path\huggingface\datasets\src\datasets\builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "path\huggingface\datasets\src\datasets\builder.py", line 709, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "path\huggingface\datasets\src\datasets\utils\info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='arabic_billion_words'), 'recorded': SplitInfo(name='train', num_bytes=1601790302, num_examples=349342, dataset_name='arabic_billion_words')}]
```

This is due because a previous run generated a wrong `dataset_info.json`.

This error can be avoided by passing `--ignore_verifications`, but I think this should be assumed when passing `--save_infos`.


[when Image type will exist] provide a way to get the data as binary + filename

enhancement

New feature or request

@severo, maybe somehow related to this PR ?
- #3129

@severo I'll keep that in mind.

You can track progress on the Image feature in #3163 (still in the early stage).

Hi ! As discussed with @severo offline it looks like the dataset viewer already supports reading PIL images, so maybe the dataset viewer doesn't need to disable decoding after all

**Is your feature request related to a problem? Please describe.**

When a dataset cell contains a value of type Image (be it from a remote URL, an Array2D/3D, or any other way to represent images), I want to be able to write the image to the disk, with the correct filename, and optionally to know its mimetype, in order to serve it on the web.

Note: this issue would apply exactly the same for the `Audio` type.

**Describe the solution you'd like**

If a "cell" has the type `Image`, provide a way to get the binary content of the file, and the filename, eg as:

```python
  filename: str
  data: bytes
```

**Describe alternatives you've considered**

A way to write the cell to the disk (passing a local directory), and then return the pathname, filename, and mimetype.


Infer the features if missing

New feature or request

**Is your feature request related to a problem? Please describe.**

Some datasets, in particular community datasets, have no info file, thus no features.

**Describe the solution you'd like**

If a dataset has no features, the first loaded data (5-10 rows) could be used to infer the type.

Related: `datasets` would provide a way to load the data, and get the rows AND the features as the result.

**Describe alternatives you've considered**

The HF hub could also provide some UI to help the dataset maintainers to explicit the types of their rows, or automatically infer them as an initial proposal.


Provide a way to check if the features (in info) match with the data of a split

New feature or request

**Is your feature request related to a problem? Please describe.**

I understand that currently the data loaded has not always the type described in the info features

**Describe the solution you'd like**

Provide a way to check if the rows have the type described by info features

**Describe alternatives you've considered**

Always check it, and raise an error when loading the data if their type doesn't match the features.


Provide a way to write a streamed dataset to the disk

New feature or request

Yes, I agree this feature is much needed. We could do something similar to what TF does (https://www.tensorflow.org/api_docs/python/tf/data/Dataset#cache).

Ideally, if the entire streamed dataset is consumed/cached, the generated cache should be reusable for the Arrow dataset.

**Is your feature request related to a problem? Please describe.**

The streaming mode allows to get the 100 first rows of a dataset very quickly. But it does not cache the answer, so a posterior call to get the same 100 rows will send a request to the server again and again.

**Describe the solution you'd like**

Provide a way to write the streamed rows of a dataset on the disk, and to load from it later.

**Describe alternatives you've considered**

Provide a third mode: `lazy`, which would use the local cache for the data that have already been fetched previously, and use streaming to get the rest of the requested data.


Add DER metric

New feature or request

Add DER metric for speaker diarization task.

This is used by SUPERB beenchmark, for example.


Fix file/directory deletion on Windows

bug

Something isn't working

Currently, on Windows, some attempts to delete a dataset file/directory will fail with the `PerimissionError`.

Examples:
- download a dataset, then force redownload it in the same session while keeping a reference to the downloaded dataset
```python
from datasets import load_dataset
dset = load_dataset("sst", split="train")
dset = load_dataset("sst", split="train", download_mode="force_redownload")
```
- try to clean up the cache files while keeping a reference to those files (via the mapped dataset):
```python
from datasets import load_dataset
dset = load_dataset("sst", split="train")
dset_mapped = dset.map(lambda _: {"dummy_col": 1})
dset.cleanup_cache_files()
```
We should fix those.