Import Error in Kaggle notebook

Something isn't working

This looks like an issue with PyArrow. Did you try reinstalling it ?

@lhoestq I did, and then let pip handle the installation in `pip import datasets`. I also tried using conda but it gives the same error.

Edit: pyarrow version on kaggle is 4.0.0, it gets replaced with 4.0.1. So, I don't think uninstalling will change anything.
```
Install Trace of datasets:

Collecting datasets
  Downloading datasets-1.9.0-py3-none-any.whl (262 kB)
     |████████████████████████████████| 262 kB 834 kB/s eta 0:00:01
Requirement already satisfied: dill in /opt/conda/lib/python3.7/site-packages (from datasets) (0.3.4)
Collecting pyarrow!=4.0.0,>=1.0.0
  Downloading pyarrow-4.0.1-cp37-cp37m-manylinux2014_x86_64.whl (21.8 MB)
     |████████████████████████████████| 21.8 MB 6.2 MB/s eta 0:00:01
Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from datasets) (3.4.0)
Requirement already satisfied: huggingface-hub<0.1.0 in /opt/conda/lib/python3.7/site-packages (from datasets) (0.0.8)
Requirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (from datasets) (1.2.4)
Requirement already satisfied: requests>=2.19.0 in /opt/conda/lib/python3.7/site-packages (from datasets) (2.25.1)
Requirement already satisfied: fsspec>=2021.05.0 in /opt/conda/lib/python3.7/site-packages (from datasets) (2021.6.1)
Requirement already satisfied: multiprocess in /opt/conda/lib/python3.7/site-packages (from datasets) (0.70.12.2)
Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from datasets) (20.9)
Collecting xxhash
  Downloading xxhash-2.0.2-cp37-cp37m-manylinux2010_x86_64.whl (243 kB)
     |████████████████████████████████| 243 kB 23.7 MB/s eta 0:00:01
Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.7/site-packages (from datasets) (1.19.5)
Requirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.7/site-packages (from datasets) (4.61.1)
Requirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from huggingface-hub<0.1.0->datasets) (3.0.12)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests>=2.19.0->datasets) (1.26.5)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests>=2.19.0->datasets) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests>=2.19.0->datasets) (2021.5.30)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests>=2.19.0->datasets) (4.0.0)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->datasets) (3.7.4.3)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->datasets) (3.4.1)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->datasets) (2.4.7)
Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from pandas->datasets) (2.8.1)
Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.7/site-packages (from pandas->datasets) (2021.1)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas->datasets) (1.15.0)
Installing collected packages: xxhash, pyarrow, datasets
  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 4.0.0
    Uninstalling pyarrow-4.0.0:
      Successfully uninstalled pyarrow-4.0.0
Successfully installed datasets-1.9.0 pyarrow-4.0.1 xxhash-2.0.2
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
```

You may need to restart your kaggle notebook after installing a newer version of `pyarrow`.

If it doesn't work we'll probably have to create an issue on [arrow's JIRA](https://issues.apache.org/jira/projects/ARROW/issues/), and maybe ask kaggle why it could fail

> You may need to restart your kaggle notebook before after installing a newer version of `pyarrow`.
>
> If it doesn't work we'll probably have to create an issue on [arrow's JIRA](https://issues.apache.org/jira/projects/ARROW/issues/), and maybe ask kaggle why it could fail

It works after restarting.
My bad, I forgot to restart the notebook. Sorry for the trouble!

## Describe the bug
Not able to import datasets library in kaggle notebooks

## Steps to reproduce the bug
```python
!pip install datasets
import datasets
```

## Expected results
No such error

## Actual results
```
ImportError                               Traceback (most recent call last)
<ipython-input-9-652e886d387f> in <module>
----> 1 import datasets

/opt/conda/lib/python3.7/site-packages/datasets/__init__.py in <module>
     31     )
     32
---> 33 from .arrow_dataset import Dataset, concatenate_datasets
     34 from .arrow_reader import ArrowReader, ReadInstruction
     35 from .arrow_writer import ArrowWriter

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in <module>
     36 import pandas as pd
     37 import pyarrow as pa
---> 38 import pyarrow.compute as pc
     39 from multiprocess import Pool, RLock
     40 from tqdm.auto import tqdm

/opt/conda/lib/python3.7/site-packages/pyarrow/compute.py in <module>
     16 # under the License.
     17
---> 18 from pyarrow._compute import (  # noqa
     19     Function,
     20     FunctionOptions,

ImportError: /opt/conda/lib/python3.7/site-packages/pyarrow/_compute.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK5arrow7compute15KernelSignature8ToStringEv
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Kaggle
- Python version: 3.7.10
- PyArrow version: 4.0.1


Error when downloading C4

bug

Something isn't working

Hi Thanks for reporting !
It looks like these files are not correctly reported in the list of expected files to download, let me fix that ;)

Alright this is fixed now. We'll do a new release soon to make the fix available.

In the meantime feel free to simply pass `ignore_verifications=True` to `load_dataset` to skip this error

@lhoestq thank you for such a quick feedback!

Hi,
I am trying to download `en` corpus from C4 dataset. However, I get an error caused by validation files download (see image). My code is very primitive:
`datasets.load_dataset('c4', 'en')`

Is this a bug or do I have some configurations missing on my server?
Thanks!

<img width="1014" alt="Снимок экрана 2021-07-20 в 11 37 17" src="https://user-images.githubusercontent.com/36672861/126289448-6e0db402-5f3f-485a-bf74-eb6e0271fc25.png">


Increase json reader block_size automatically

Currently some files can't be read with the default parameters of the JSON lines reader.
For example this one:
https://huggingface.co/datasets/thomwolf/codeparrot/resolve/main/file-000000000006.json.gz

raises a pyarrow error:
```python
ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```

The block size that is used is the default one by pyarrow (related to this [jira issue](https://issues.apache.org/jira/browse/ARROW-9612)).

To fix this issue I changed the block_size to increase automatically if there is a straddling issue when parsing a batch of json lines.

By default the value is `chunksize // 32` in order to leverage multithreading, and it doubles every time a straddling issue occurs. The block_size is then reset for each file.

cc @thomwolf @albertvillanova


Parallelize ETag requests

Since https://github.com/huggingface/datasets/pull/2628 we use the ETag or the remote data files to compute the directory in the cache where a dataset is saved. This is useful in order to reload the dataset from the cache only if the remote files haven't changed.

In this I made the ETag requests parallel using multithreading. There is also a tqdm progress bar that shows up if there are more than 16 data files.


Fix sacrebleu parameter name

DONE:
- Fix parameter name: `smooth` to `smooth_method`.
- Improve kwargs description.
- Align docs on using a metric.
- Add example of passing additional arguments in using metrics.

Related to #2669.


Fix potential DuplicatedKeysError in SQuAD

DONE:
- Fix potential DiplicatedKeysError by ensuring keys are unique.
- Align examples in the docs with SQuAD code.

We should promote as a good practice, that the keys should be programmatically generated as unique, instead of read from data (which might be not unique).


Fix potential DuplicatedKeysError in LibriSpeech

DONE:
- Fix unnecessary path join.
- Fix potential DiplicatedKeysError by ensuring keys are unique.

We should promote as a good practice, that the keys should be programmatically generated as unique, instead of read from data (which might be not unique).


Mesinesp development and training data sets have been added.

It'll be new pull request with new commits.

https://zenodo.org/search?page=1&size=20&q=mesinesp, Mesinesp has Medical Semantic Indexed records in Spanish.  Indexing is done using DeCS codes, a sort of Spanish equivalent to MeSH terms.
The Mesinesp (Spanish BioASQ track, see https://temu.bsc.es/mesinesp) development set has a total of 750 records.
The Mesinesp (Spanish BioASQ track, see https://temu.bsc.es/mesinesp) training set has a total of 369,368 records.


Using sharding to parallelize indexing

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Creating an elasticsearch index on large dataset could be quite long and cannot be parallelized on shard (the index creation is colliding)

**Describe the solution you'd like**
When working on dataset shards, if an index already exists, its mapping should be checked and if compatible, the indexing process should continue with the shard data.

Additionally, at the end of the process, the `_indexes` dict should be send back to the original dataset object (from which the shards have been created) to allow to use the index for later filtering on the whole dataset.

**Describe alternatives you've considered**
Each dataset shard could created independent partial indices. then on the whole dataset level, indices should be all referred in `_indexes` dict and be used in querying through `get_nearest_examples()`. The drawback is that the scores will be computed independently on the partial indices leading to inconsistent values for most scoring based on corpus level statistics (tf/idf, BM25).

**Additional context**
The objectives is to parallelize the index creation to speed-up the process (ie surcharging the ES server which is fine to handle large load) while later enabling search on the whole dataset.


Metric kwargs are not passed to underlying external metric f1_score

bug

Something isn't working

Hi @BramVanroy, thanks for reporting.

First, note that `"min"` is not an allowed value for `average`. According to scikit-learn [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html), `average` can only take the values: `{"micro", "macro", "samples", "weighted", "binary"} or None, default="binary"`.

Second, you should take into account that all additional metric-specific argument should be passed in the method `compute` (and not in the method `load_metric`). You can find more information in our documentation: https://huggingface.co/docs/datasets/using_metrics.html#computing-the-metric-scores

So for example, if you would like to calculate the macro-averaged F1 score, you should use:
```python
import datasets

f1 = datasets.load_metric("f1", keep_in_memory=True)
f1.add_batch(predictions=[0,2,3], references=[1, 2, 3])
f1.compute(average="macro")
```

Thanks, that was it. A bit strange though, since `load_metric` had an argument `metric_init_kwargs`. I assume that that's for specific initialisation arguments whereas `average` is for the function itself.

## Describe the bug
When I want to use F1 score with average="min", this keyword argument does not seem to be passed through to the underlying sklearn metric. This is evident because [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) throws an error telling me so.

## Steps to reproduce the bug
```python
import datasets
f1 = datasets.load_metric("f1", keep_in_memory=True, average="min")
f1.add_batch(predictions=[0,2,3], references=[1, 2, 3])
f1.compute()
```

## Expected results
No error, because `average="min"` should be passed correctly to f1_score in sklearn.

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\datasets\metric.py", line 402, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "C:\Users\bramv\.cache\huggingface\modules\datasets_modules\metrics\f1\82177930a325d4c28342bba0f116d73f6d92fb0c44cd67be32a07c1262b61cfe\f1.py", line 97, in _compute
    "f1": f1_score(
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\metrics\_classification.py", line 1071, in f1_score
    return fbeta_score(y_true, y_pred, beta=1, labels=labels,
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\metrics\_classification.py", line 1195, in fbeta_score
    _, _, f, _ = precision_recall_fscore_support(y_true, y_pred,
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\metrics\_classification.py", line 1464, in precision_recall_fscore_support
    labels = _check_set_wise_labels(y_true, y_pred, average, labels,
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\metrics\_classification.py", line 1294, in _check_set_wise_labels
    raise ValueError("Target is %s but average='binary'. Please "
ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.2
- PyArrow version: 4.0.1


Add Russian SuperGLUE

Added the missing label classes and their explanations (to the best of my understanding)

Thanks a lot ! Once the last comment about the label names is addressed we can merge :)

Hi,

This adds the [Russian SuperGLUE](https://russiansuperglue.com/) dataset. For the most part I reused the code for the original SuperGLUE, although there are some relatively minor differences in the structure that I accounted for.


Use tqdm from tqdm_utils

The current CI failure is due to modifications in the dataset script.

Merging since the CI is only failing because of dataset card issues, which is unrelated to this PR

This PR replaces `tqdm` from the `tqdm` lib with `tqdm` from `datasets.utils.tqdm_utils`. With this change, it's possible to disable progress bars just by calling `disable_progress_bar`. Note this doesn't work on Windows when using multiprocessing due to how global variables are shared between processes. Currently, there is no easy way to disable progress bars in a multiprocess setting on Windows (patching logging with `datasets.utils.logging.get_verbosity = lambda: datasets.utils.logging.NOTSET` doesn't seem to work as well), so adding support for this is a future goal. Additionally, this PR adds a unit ("ba" for batches) to the bar printed by `Dataset.to_json` (this change is motivated by https://github.com/huggingface/datasets/issues/2657).


Adds CodeClippy dataset [WIP]

CodeClippy is an opensource code dataset scrapped from github during flax-jax-community-week
https://the-eye.eu/public/AI/training_data/code_clippy_data/


Adds APPS dataset to the hub [WIP]

A loading script for [APPS dataset](https://github.com/hendrycks/apps)


[`to_json`] add multi-proc sharding support

enhancement

New feature or request

Hi @stas00,
I want to work on this issue and I was thinking why don't we use `imap` [in this loop](https://github.com/huggingface/datasets/blob/440b14d0dd428ae1b25881aa72ba7bbb8ad9ff84/src/datasets/io/json.py#L99)? This way, using offset (which is being used to slice the pyarrow table) we can convert  pyarrow table to `json` using multiprocessing. I've a small code snippet for some clarity:
```
result = list(
            pool.imap(self._apply_df, [(offset, batch_size) for offset in range(0, len(self.dataset), batch_size)])
        )
```
`_apply_df` is a function which will return `batch.to_pandas().to_json(path_or_buf=None, orient="records", lines=True)` which is basically json version of the batched pyarrow table. Later on we can concatenate it to form json file?

I think the only downside here is to write file from `imap` output (output would be a list and we'll need to iterate over it and write in a file) which might add a little overhead cost. What do you think about this?

Followed up in https://github.com/huggingface/datasets/pull/2747

As discussed on slack it appears that `to_json` is quite slow on huge datasets like OSCAR.

I implemented sharded saving, which is much much faster - but the tqdm bars all overwrite each other, so it's hard to make sense of the progress, so if possible ideally this multi-proc support could be implemented internally in `to_json` via `num_proc` argument. I guess `num_proc` will be the number of shards?

I think the user will need to use this feature wisely, since too many processes writing to say normal style HD is likely to be slower than one process.

I'm not sure whether the user should be responsible to concatenate the shards at the end  or `datasets`, either way works for my needs.

The code I was using:

```
from multiprocessing import cpu_count, Process, Queue

[...]

filtered_dataset = concat_dataset.map(filter_short_documents, batched=True, batch_size=256, num_proc=cpu_count())

DATASET_NAME = "oscar"
SHARDS = 10
def process_shard(idx):
    print(f"Sharding {idx}")
    ds_shard = filtered_dataset.shard(SHARDS, idx, contiguous=True)
    # ds_shard = ds_shard.shuffle() # remove contiguous=True above if shuffling
    print(f"Saving {DATASET_NAME}-{idx}.jsonl")
    ds_shard.to_json(f"{DATASET_NAME}-{idx}.jsonl", orient="records", lines=True, force_ascii=False)

queue = Queue()
processes = [Process(target=process_shard, args=(idx,)) for idx in range(SHARDS)]
for p in processes:
    p.start()

for p in processes:
    p.join()
```

Thank you!

@lhoestq


Load Dataset from the Hub (NO DATASET SCRIPT)

This is ready for review now :)

I would love to have some feedback on the changes in load.py @albertvillanova. There are many changes so if you have questions let me know, especially on the `resolve_data_files` functions and on the changes in `prepare_module`.

And @thomwolf if you want to take a look at the documentation, feel free to share your suggestions :)

I took your comments into account thanks !
And I made `aiohttp` a required dependency :)

Just updated the documentation :)
[share_datasets.html](https://45532-250213286-gh.circle-artifacts.com/0/docs/_build/html/share_dataset.html)

Let me know if you have some comments

Merging this one :)

We can try to integrate the changes in the docs to #2718 @stevhliu !

Baked this into the [docs](https://44335-250213286-gh.circle-artifacts.com/0/docs/_build/html/loading.html#hugging-face-hub) already, let me know if there is anything else I should add! :)

## Load the data from any Dataset repository on the Hub

This PR adds support for loading datasets from any dataset repository on the hub, without requiring any dataset script.

As a user it's now possible to create a repo and upload some csv/json/text/parquet files, and then be able to load the data in one line. Here is an example with the `allenai/c4` repository that contains a lot of compressed json lines files:
```python
from datasets import load_dataset

data_files = {"train": "en/c4-train.*.json.gz"}
c4 = load_dataset("allenai/c4", data_files=data_files, split="train", streaming=True)

print(c4.n_shards)
# 1024
print(next(iter(c4)))
# {'text': 'Beginners BBQ Class Takin...'}
```

By default it loads all the files, but as shown in the example you can choose the ones you want with unix style patterns.

Of course it's still possible to use dataset scripts since they offer the most flexibility.

## Implementation details

It uses `huggingface_hub` to list the files in a dataset repository.

If you provide a path to a local directory instead of a repository name, it works the same way but it uses `glob`.

Depending on the data files available, or passed in the `data_files` parameter, one of the available builders will be used among the csv, json, text and parquet builders.

Because of this, it's not possible to load both csv and json files at once. In this case you have to load them separately and then concatenate the two datasets for example.

## TODO

- [x] tests
- [x] docs
- [x] when huggingface_hub gets a new release, update the CI and the setup.py

Close https://github.com/huggingface/datasets/issues/2629


Add SD task for SUPERB

I make a summary about our discussion with @lewtun and @Narsil on the agreed schema for this dataset and the additional steps required to generate the 2D array labels:
- The labels for this dataset are a 2D array:
  Given an example:
  ```python
  {"record_id": record_id, "file": file, "start": start, "end": end, "speakers": [...]}
  ```
  the labels are a 2D array of shape `(num_frames, num_speakers)` where `num_frames = end - start` and `num_speakers = 2`.
- In order to avoid a too large dataset (too large disk space), `datasets` does not store the 2D array label. Instead, we store a compact form:
  ```
  "speakers": [
    {"speaker_id": speaker_0_id, "start": start_0_speaker_0, "end": end_0_speaker_0},
    {"speaker_id": speaker_0_id, "start": start_1_speaker_0, "end": end_1_speaker_0},
    {"speaker_id": speaker_1_id, "start": start_0_speaker_1, "end": end_0_speaker_1},
  ],
  ```
  - Once loaded the dataset, an additional step is required to generate the 2D array label from this compact form
  - This additional step should be a modified version of the s3prl method `_get_labeled_speech`:
    - Original s3prl `_get_labeled_speech` includes 2 functionalities: reading the audio file and transforming it into an array, and generating the label 2D array; I think we should separate these 2 functionalities
    - Original s3prl `_get_labeled_speech` performs 2 steps to generate the labels:
      - Transform start/end seconds (float) into frame numbers (int): I have already done this step to generate the dataset
      - Generate the 2D array label from the frame numbers

I also ping @osanseviero and @lhoestq to include them in the loop.

Here I would like to discuss (and agree) one of the decisions I made, as I'm not completely satisfied with it: to transform the seconds (float) into frame numbers (int) to generate this dataset.

- A priori, the most natural and general choice would be to preserve the seconds (float), because:
  - this is the way the raw data comes from
  - the transformation into frame numbers depends on the sample rate, frame_shift and subsampling

However, I finally decided to transform seconds into frame numbers because:
- for SUPERB, sampling rate, frame_shift and subsampling are fixed (`rate = 16_000`, `frame_shift = 160`, `subsampling = 1`)
- it makes easier the post-processing, as labels are generated from sample numbers:  labels are a 2D array of shape `(num_frames, num_speakers)`
- the number of examples depends on the number of frames:
  - if an example has more than 2_000 frames, then it is split into 2 examples. This is the case for `record_id = "7859-102521-0017_3983-5371-0014"`, which has 2_452 frames and it is split into 2 examples:
    ```
    {"record_id": "7859-102521-0017_3983-5371-0014", "start"= 0, "end": 2_000,...},
    {"record_id": "7859-102521-0017_3983-5371-0014", "start"= 2_000, "end": 2_452,...},
    ```

As I told you, I'm not totally convinced of this decision, and I would really appreciate your opinion.

cc: @lewtun @Narsil @osanseviero @lhoestq

It makes total sense to prepare the data to be in a format that can actually be used for model training and evaluation. That's one of the roles of this lib :)

So for me it's ok to use frames as a unit instead of seconds. Just pinging @patrickvonplaten in case he has ever played with such audio tasks and has some advice. For the context: the task is to classify which speaker is speaking,  let us know if you are aware of any convenient/standard format for this.

Also I'm not sure why you have to split an example if it's longer that 2,000 frames ?

> Also I'm not sure why you have to split an example if it's longer that 2,000 frames ?

It is a convention in SUPERB benchmark.

Note that if we agree to leave the dataset as it is now, 2 additional custom functions must be used:
- one to generate the 2D array labels
- one to load the audio file into an array, but taking into account start/end to cut the audio

Is there a way we can give these functions ready to be used? Or should we leave this entirely to the end user? This is not trivial...

You could add an example of usage in the dataset card, as it is done for other audio datasets

@albertvillanova this simple function can be edited simply to add the start/stop cuts

https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines/automatic_speech_recognition.py#L29

Windows ? What is it ? (Not sure not able to test, it's directly calling ffmpeg binary, so depending on the setup it could but can't say for sure without testing)

It's one of the OS we're supposed to support :P (for the better and for the worse)

> Note that if we agree to leave the dataset as it is now, 2 additional custom functions must be used:
>
> * one to generate the 2D array labels
> * one to load the audio file into an array, but taking into account start/end to cut the audio
>
> Is there a way we can give these functions ready to be used? Or should we leave this entirely to the end user? This is not trivial...

+1 on providing the necessary functions on the dataset card. aside from that, the current implementation looks great from my perspective!

Include the SD (Speaker Diarization) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sd-speaker-diarization).

TODO:
- [x] Generate the LibriMix corpus
- [x] Prepare the corpus for diarization
- [x] Upload these files to the superb-data repo
- [x] Transcribe the corresponding s3prl processing of these files into our superb loading script
- [x] README: tags + description sections
- ~~Add DER metric~~ (we leave the DER metric for a follow-up PR)

Related to #2619.

Close #2653.

cc: @lewtun


Move checks from _map_single to map

@lhoestq This one has been open for a while. Could you please take a look?

I forgot to update the signature of `DatasetDict.map`, so did that now.

The goal of this PR is to remove duplicated checks in the `map` logic to execute them only once whenever possible (`fn_kwargs`, `input_columns`, ...). Additionally, this PR improves the consistency (to align it with `input_columns`) of the `remove_columns` check by adding support for a single string value, which is then wrapped into a list.


Allow dataset config kwargs to be None

Close https://github.com/huggingface/datasets/issues/2658

The dataset config kwargs that were set to None we simply ignored.
This was an issue when None has some meaning for certain parameters of certain builders, like the `sep` parameter of the "csv" builder that allows to infer to separator.

cc @SBrandeis


Can't pass `sep=None` to load_dataset("csv", ...) to infer the separator via pandas.read_csv

When doing `load_dataset("csv", sep=None)`, the `sep` passed to `pd.read_csv` is still the default `sep=","` instead, which makes it impossible to make the csv loader infer the separator.

Related to https://github.com/huggingface/datasets/pull/2656

cc @SBrandeis


`to_json` reporting enhancements

New feature or request

While using `to_json` 2 things came to mind that would have made the experience easier on the user:

1. Could we have a `desc` arg for the tqdm use and a fallback to just `to_json` so that it'd be clear to the user what's happening? Surely, one can just print the description before calling json, but I thought perhaps it'd help to have it self-identify like you did for other progress bars recently.

2. It took me a while to make sense of the reported numbers:
```
 22%|██▏       | 1536/7076 [12:30:57<44:09:42, 28.70s/it]
```
So iteration here happens to be 10K samples, and the total is 70M records. But the user does't know that, so the progress bar is perfect, but the numbers it reports are meaningless until one discovers that 1it=10K samples. And one still has to convert these in the head - so it's not quick. Not exactly sure what's the best way to approach this, perhaps it can be part of `desc`? or report M or K, so it'd be built-in if it were to print, e.g.:
```
 22%|██▏       | 15360K/70760K [12:30:57<44:09:42, 28.70s/it]
```
or
```
 22%|██▏       | 15.36M/70.76M [12:30:57<44:09:42, 28.70s/it]
```
(while of course remaining friendly to small datasets)

I forget if tqdm lets you add a magnitude identifier to the running count.

Thank you!


Change `from_csv` default arguments

This is not the default in pandas right ?
We try to align our CSV loader with the pandas API.

Moreover according to their documentation, the python parser is used when sep is None, which might not be the fastest one.

Maybe users could just specify `sep=None` themselves ?
In this case we should add some documentation about this

Passing `sep=None` to pandas's `read_csv` lets pandas guess the CSV file's separator

This PR allows users to use this pandas's feature by passing `sep=None` to `Dataset.from_csv`:

```python
Dataset.from_csv(
    ...,
    sep=None
)
```


Allow the selection of multiple columns at once

New feature or request

Hi! I was looking into this and hope you can clarify a point. Your my_dataset variable would be of type DatasetDict which means the alternative you've described (dict comprehension) is what makes sense.
Is there a reason why you wouldn't want to convert my_dataset to a pandas df if you'd like to use it like one? Please let me know if I'm missing something.

Hi! Sorry for the delay.

In this case, the dataset would be a `datasets.Dataset` and we want to select multiple columns, the `idx` and `label` columns for example.

My issue is that my dataset is too big for memory if I load everything into pandas.

**Is your feature request related to a problem? Please describe.**

Similar to pandas, it would be great if we could select multiple columns at once.

**Describe the solution you'd like**
```python
my_dataset = ...  # Has columns ['idx', 'sentence', 'label']
idx, label = my_dataset[['idx', 'label']]
```

**Describe alternatives you've considered**
we can do `[dataset[col] for col in ('idx', 'label')]`

**Additional context**
This is of course very minor.


Give a user feedback if the dataset he loads is streamable or not

New feature or request

I understand it already raises a `NotImplementedError` exception, eg:

```
>>> dataset = load_dataset("journalists_questions", name="plain_text", split="train", streaming=True)

[...]
NotImplementedError: Extraction protocol for file at https://drive.google.com/uc?export=download&id=1CBrh-9OrSpKmPQBxTK_ji6mq6WTN_U9U is not implemented yet
```

**Is your feature request related to a problem? Please describe.**
I would love to know if a `dataset` is with the current implementation streamable or not.

**Describe the solution you'd like**
We could show a warning when a dataset is loaded with `load_dataset('...',streaming=True)` when its lot streamable, e.g. if it is an archive.

**Describe alternatives you've considered**
Add a new metadata tag for "streaming"


Add SD task for SUPERB

dataset request

Requesting to add a new dataset

1.11

Next minor release

Note that this subset requires us to:

* generate the LibriMix corpus from LibriSpeech
* prepare the corpus for diarization

As suggested by @lhoestq we should perform these steps locally and add the prepared data to this public repo on the Hub: https://huggingface.co/datasets/superb/superb-data

Then we can use the URLs for the files to load the data in `superb`'s dataset loading script.

For consistency, I suggest we name the folders in `superb-data` in the same way as the configs in the dataset loading script - e.g. use `sd` for speech diarization in both places :)

@lewtun @lhoestq:

I have already generated the LibriMix corpus and prepared the corpus for diarization. The output is 3 dirs (train, dev, test), each one containing 6 files: reco2dur  rttm  segments  spk2utt  utt2spk  wav.scp

Next steps:
- Upload these files to the superb-data repo
- Transcribe the corresponding s3prl processing of these files into our superb loading script

Note that processing of these files is a bit more intricate than usual datasets: https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/diarization/dataset.py#L233

Include the SD (Speaker Diarization) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sd-speaker-diarization).

Steps:
- [x] Generate the LibriMix corpus
- [x] Prepare the corpus for diarization
- [x] Upload these files to the superb-data repo
- [x] Transcribe the corresponding s3prl processing of these files into our superb loading script
- [ ] README: tags + description sections

Related to #2619.

cc: @lewtun


Fix logging docstring

Remove "no tqdm bars" from the docstring in the logging module to align it with the changes introduced in #2534.


Setting log level higher than warning does not suppress progress bar

bug

Something isn't working

Hi,

you can suppress progress bars by patching logging as follows:
```python
import datasets
import logging
datasets.logging.get_verbosity = lambda: logging.NOTSET
# map call ...
```

See https://github.com/huggingface/datasets/issues/2528 for reference

Note also that you can disable the progress bar with

```python
from datasets.utils import disable_progress_bar
disable_progress_bar()
```

See https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/src/datasets/utils/tqdm_utils.py#L84

Now the library officially recommends `set_progress_bar_enabled(False)`

```py
from datasets.utils import set_progress_bar_enabled

set_progress_bar_enabled(False)
```

source:

https://github.com/huggingface/datasets/blob/1fd47120ace13626c528367787ffa13e1a26e6c0/src/datasets/utils/tqdm_utils.py#L83-L88

From https://github.com/huggingface/datasets/pull/3897, `disable_progress_bar` is the function you should use

## Describe the bug
I would like to disable progress bars for `.map` method (and other methods like `.filter` and `load_dataset` as well).
According to #1627 one can suppress it by setting log level higher than `warning`, however doing so doesn't suppress it with version 1.9.0.

I also tried to set `DATASETS_VERBOSITY` environment variable to `error` or `critical` but it also didn't work.

## Steps to reproduce the bug
```python
import datasets

from datasets.utils.logging import set_verbosity_error

set_verbosity_error()

def dummy_map(batch):
    return batch

common_voice_train = datasets.load_dataset("common_voice", "de", split="train")
common_voice_test = datasets.load_dataset("common_voice", "de", split="test")

common_voice_train.map(dummy_map)
```

## Expected results
- The progress bar for `.map` call won't be shown

## Actual results
- The progress bar for `.map` is still shown

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.4.0-1045-aws-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyArrow version: 4.0.1


[load_dataset] shard and parallelize the process

enhancement

New feature or request

I need the same feature for distributed training

- Some huge datasets take forever to build the first time. (e.g. oscar/en) as it's done in a single cpu core.
- If the build crashes, everything done up to that point gets lost

Request: Shard the build over multiple arrow files, which would enable:
- much faster build by parallelizing the build process
- if the process crashed, the completed arrow files don't need to be re-built again

Thank you!

@lhoestq


adding progress bar / ETA for `load_dataset`

New feature or request

Please consider:
```
Downloading and preparing dataset oscar/unshuffled_deduplicated_en (download: 462.40 GiB, generated: 1.18 TiB, post-processed: Unknown size, total: 1.63 TiB) to cache/oscar/unshuffled_deduplicated_en/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2...
HF google storage unreachable. Downloading and preparing it from source
```
and no indication whatsoever of whether things work well or when it'll be done. It's important to have an estimated completion time for when doing slurm jobs since some instances have a cap on run-time.

I think for this particular job it sat for 30min in total silence and then after 30min it started generating:
```
897850 examples [07:24, 10286.71 examples/s]
```
which is already great!

Request:
1. ETA - knowing how many hours to allocate for a slurm job
2. progress bar - helps to know things are working and aren't stuck and where we are at.

Thank you!

@lhoestq


Add web_split dataset for Paraphase and Rephrase benchmark

New feature or request

## Describe:
For getting simple sentences from complex sentence there are dataset and task like wiki_split that is available in hugging face datasets. This web_split is a very similar dataset. There some research paper which states that by combining these two datasets we if we train the model it will yield better results on both tests data.

This dataset is made from web NLG data.

All the dataset related details are provided in the below repository

Github link: https://github.com/shashiongithub/Split-and-Rephrase


Fix anchor in README

1.10

Next minor release

I forgot to push this fix in #2611, so I'm sending it now.


downloading of yahoo_answers_topics dataset failed

bug

Something isn't working

Hi ! I just tested and it worked fine today for me.

I think this is because the dataset is stored on Google Drive which has a quota limit for the number of downloads per day, see this similar issue https://github.com/huggingface/datasets/issues/996

Feel free to try again today, now that the quota was reset

## Describe the bug
I get an error datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files when I try to download the yahoo_answers_topics dataset

## Steps to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')
# Sample code to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files


load_dataset processing failed with OS error after downloading a dataset

Something isn't working

Hi ! It looks like an issue with pytorch.

Could you try to run `import torch` and see if it raises an error ?

> Hi ! It looks like an issue with pytorch.
>
> Could you try to run `import torch` and see if it raises an error ?

It works. Thank you!

## Describe the bug
After downloading a dataset like opus100, there is a bug that
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS

## Steps to reproduce the bug
```python
from datasets import load_dataset
this_dataset = load_dataset('opus100', 'af-en')
```

## Expected results
there is no error when running load_dataset.

## Actual results
Specify the actual results or traceback.

Traceback (most recent call last):
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 652, in _download_and_prep
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 989, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 952, in encode_example
    example = cast_to_python_objects(example)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 219, in cast_to_python_ob
    return _cast_to_python_objects(obj)[0]
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 165, in _cast_to_python_o
    import torch
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 188, in <module>
    _load_global_deps()
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 141, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/home/anaconda3/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_hub_opus100.py", line 9, in <module>
    this_dataset = load_dataset('opus100', language_pair)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 575, in download_and_prepa
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 658, in _download_and_prep
    + str(e)
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS

## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-3.13.0-32-generic-x86_64-with-debian-jessie-sid
- Python version: 3.6.6
- PyArrow version: 3.0.0


Batched `map` not allowed to return 0 items

Something isn't working

Hi ! Thanks for reporting. Indeed it looks like type inference makes it fail. We should probably just ignore this step until a non-empty batch is passed.

Sounds good! Do you want me to propose a PR? I'm quite busy right now, but if it's not too urgent I could take a look next week.

Sure if you're interested feel free to open a PR :)

You can also ping me anytime if you have questions or if I can help !

Sorry to ping you, @lhoestq, did you have a chance to take a look at the proposed PR? Thank you!

Yes and it's all good, thank you :)

Feel free to close this issue if it's good for you

## Describe the bug
I'm trying to use `map` to filter a large dataset by selecting rows that match an expensive condition (files referenced by one of the columns need to exist in the filesystem, so we have to `stat` them). According to [the documentation](https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset), `a batch mapped function can take as input a batch of size N and return a batch of size M where M can be greater or less than N and can even be zero`.

However, when the returned batch has a size of zero (neither item in the batch fulfilled the condition), we get an `index out of bounds` error. I think that `arrow_writer.py` is [trying to infer the returned types using the first element returned](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_writer.py#L100), but no elements were returned in this case.

For this error to happen, I'm returning a dictionary that contains empty lists for the keys I want to keep, see below. If I return an empty dictionary instead (no keys), then a different error eventually occurs.

## Steps to reproduce the bug
```python
def select_rows(examples):
    # `key` is a column name that exists in the original dataset
    # The following line simulates no matches found, so we return an empty batch
    result = {'key': []}
    return result

filtered_dataset = dataset.map(
    select_rows,
    remove_columns = dataset.column_names,
    batched = True,
    num_proc = 1,
    desc = "Selecting rows with images that exist"
)
```

The code above immediately triggers the exception. If we use the following instead:

```python
def select_rows(examples):
    # `key` is a column name that exists in the original dataset
    result = {'key': []}   # or defaultdict or whatever

    # code to check for condition and append elements to result
    # some_items_found will be set to True if there were any matching elements in the batch

    return result if some_items_found else {}
```

Then it _seems_ to work, but it eventually fails with some sort of schema error. I believe it may happen when an empty batch is followed by a non-empty one, but haven't set up a test to verify it.

In my opinion, returning a dictionary with empty lists and valid column names should be accepted as a valid result with zero items.

## Expected results
The dataset would be filtered and only the matching fields would be returned.

## Actual results
An exception is encountered, as described. Using a workaround makes it fail further along the line.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.1.dev0
- Platform: Linux-5.4.0-53-generic-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 4.0.1


Enum used in map functions will raise a RecursionError with dill.

Something isn't working

I'm running into this as well. (Thank you so much for reporting @jorgeecardona — was staring at this massive stack trace and unsure what exactly was wrong!)

Hi ! Thanks for reporting :)

Until this is fixed on `dill`'s side, we could implement a custom saving in our Pickler indefined in utils.py_utils.py
There is already a suggestion in this message about how to do it:
https://github.com/uqfoundation/dill/issues/250#issuecomment-852566284

Let me know if such a workaround could help, and feel free to open a PR if you want to contribute !

I have the same bug.
the code is as follows:
![image](https://user-images.githubusercontent.com/84262181/139785849-620dd4ac-86ce-4212-8163-942bbca305aa.png)
the error is:
![image](https://user-images.githubusercontent.com/84262181/139785899-88a9bd75-c60b-45a5-b819-830c7c096f3d.png)

Look for the solution for this bug.

Hi ! I think your RecursionError comes from a different issue @BitcoinNLPer , could you open a separate issue please ?

Also which dataset are you using ? I tried loading `CodedotAI/code_clippy` but I get a different error
```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/load.py", line 1615, in load_dataset
    **config_kwargs,
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/load.py", line 1446, in load_dataset_builder
    builder_cls = import_main_class(dataset_module.module_path)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/load.py", line 101, in import_main_class
    module = importlib.import_module(module_path)
  File "/Users/quentinlhoest/.virtualenvs/hf-datasets/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/quentinlhoest/.cache/huggingface/modules/datasets_modules/datasets/CodedotAI___code_clippy/d332f69d036e8c80f47bc9a96d676c3fa30cb50af7bb81e2d4d12e80b83efc4d/code_clippy.py", line 66, in <module>
    url_elements = results.find_all("a")
AttributeError: 'NoneType' object has no attribute 'find_all'
```

## Describe the bug

Enums used in functions pass to `map` will fail at pickling with a maximum recursion exception as described here: https://github.com/uqfoundation/dill/issues/250#issuecomment-852566284

In my particular case, I use an enum to define an argument with fixed options using the `TraininigArguments` dataclass as base class and the `HfArgumentParser`. In the same file I use a `ds.map` that tries to pickle the content of the module including the definition of the enum that runs into the dill bug described above.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from enum import Enum

class A(Enum):
    a = 'a'

def main():
    a = A.a

    def f(x):
        return {} if a == a.a else x

    ds = load_dataset('cnn_dailymail', '3.0.0')['test']
    ds = ds.map(f, num_proc=15)

if __name__ == "__main__":
    main()
```

## Expected results
The known problem with dill could be prevented as explained in the link above (workaround.) Since `HFArgumentParser` nicely uses the enum class for choices it makes sense to also deal with this bug under the hood.

## Actual results

```python
  File "/home/xxxx/miniconda3/lib/python3.8/site-packages/dill/_dill.py", line 1373, in save_type
    pickler.save_reduce(_create_type, (type(obj), obj.__name__,
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 690, in save_reduce
    save(args)
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 899, in save_tuple
    save(element)
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 534, in save
    self.framer.commit_frame()
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 220, in commit_frame
    if f.tell() >= self._FRAME_SIZE_TARGET or force:
RecursionError: maximum recursion depth exceeded while calling a Python object
```

## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-5.9.0-4-amd64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 3.0.0


Support multi-worker with streaming dataset (IterableDataset).

enhancement

New feature or request

Hi ! This is a great idea :)
I think we could have something similar to what we have in `datasets.Dataset.map`, i.e. a `num_proc` parameter that tells how many processes to spawn to parallelize the data processing.

Regarding AUTOTUNE, this could be a nice feature as well, we could see how to add it in a second step

**Is your feature request related to a problem? Please describe.**
The current `.map` does not support multi-process, CPU can become bottleneck if the pre-processing is complex (e.g. t5 span masking).

**Describe the solution you'd like**
Ideally `.map` should support multi-worker like tfds, with `AUTOTUNE`.

**Describe alternatives you've considered**
A simpler solution is to shard the dataset and process it in parallel with pytorch dataloader. The shard does not need to be of equal size.
* https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

**Additional context**


load_dataset("financial_phrasebank") NonMatchingChecksumError

bug

Something isn't working

Hi! It's probably because this dataset is stored on google drive and it has a per day quota limit. It should work if you retry, I was able to initiate the download.

Similar issue [here](https://github.com/huggingface/datasets/issues/2646)

Hi ! Loading the dataset works on my side as well.
Feel free to try again and let us know if it works for you know

Thank you! I've been trying periodically for the past month, and no luck yet with this particular dataset. Just tried again and still hitting the checksum error.

Code:

`dataset = load_dataset("financial_phrasebank", "sentences_allagree") `

Traceback:

```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-2-55cc2144f31e> in <module>
----> 1 dataset = load_dataset("financial_phrasebank", "sentences_allagree")

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
    859         ignore_verifications=ignore_verifications,
    860         try_from_hf_gcs=try_from_hf_gcs,
--> 861         use_auth_token=use_auth_token,
    862     )
    863

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    582                     if not downloaded_from_gcs:
    583                         self._download_and_prepare(
--> 584                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    585                         )
    586                     # Sync info

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    642         if verify_infos:
    643             verify_checksums(
--> 644                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    645             )
    646

/opt/conda/lib/python3.7/site-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip']
```

## Describe the bug
Attempting to download the financial_phrasebank dataset results in a NonMatchingChecksumError

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("financial_phrasebank", 'sentences_allagree')
```

## Expected results
I expect to see the financial_phrasebank dataset downloaded successfully

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip']

## Environment info
- `datasets` version: 1.9.0
- Platform: Linux-4.14.232-177.418.amzn2.x86_64-x86_64-with-debian-10.6
- Python version: 3.7.10
- PyArrow version: 4.0.1


Fix docstrings

1.10

Next minor release

Fix rendering of some docstrings.


Refactor patching to specific submodule

Minor reorganization of the code, so that additional patching functions (not related to streaming) might be created.

In relation with the initial approach followed in #2631.


Streaming for the Json loader

A note is that I think we should add a few indicator of status (as mentioned by @stas00 in #2649), probably at the (1) downloading, (2) extracting and (3) reading steps. In particular when loading many very large files it's interesting to know a bit where we are in the process.

I tested locally, and the builtin `json` loader is 4x slower than `pyarrow.json`. Thanks for the comment @albertvillanova !

Therefore I switched back to using `pyarrow.json`, but only on the batch that is read. This way we don't have to deal with its `block_size`, and it only loads in memory one batch at a time.

It was not using `open` in the builder. Therefore `pyarrow.json.read_json` was downloading the full file to start yielding rows.

Moreover, it appeared that `pyarrow.json.read_json` was not really suited for streaming as it was downloading too much data and failing if `block_size` was not properly configured (related to #2573).

So I switched to using `open` which is extended to support reading from remote file progressively, and I removed the pyarrow json reader which was not practical.
Instead, I'm using the classical `json.loads` from the standard library.


Add the CIDEr metric?

enhancement

New feature or request

Hi,
I find the api in https://huggingface.co/metrics  quite useful.
I am playing around with video/image captioning task, where CIDEr is a popular metric.
Do you plan to add this into the HF ```datasets``` library?
Thanks.


Streaming for the Pandas loader

It was not using open in the builder. Therefore pd.read_pickle could fail when streaming from a private repo for example.

Indeed, when streaming, open is extended to support reading from remote files and handles authentication to the HF Hub


Streaming for the CSV loader

It was not using `open` in the builder. Therefore `pd.read_csv` was downloading the full file to start yielding rows.

Indeed, when streaming, `open` is extended to support reading from remote file progressively.


Inject ASR template for lj_speech dataset

1.10

Next minor release

Related to: #2565, #2633.

cc: @lewtun


Update ASR tags

1.10

Next minor release

This PR updates the ASR tags of the 5 datasets added in #2565 following the change of task categories in #2620


add image-classification task template

Thanks for adding a new task template - great work @nateraw  🚀 !

Snippet below is the tl;dr, but you can try it out directly here:

[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/005c025d41f0e48ae3d4ee61c0f20b70/image-classification-task-template-demo.ipynb)

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'labels'],
#         num_rows: 23410
#     })
# })

ds = ds.prepare_for_task('image-classification')
# DatasetDict({
#     train: Dataset({
#         features: ['image_file_path', 'labels'],
#         num_rows: 23410
#     })
# })
```


Delete extracted files when loading dataset

Sure @stas00, it is still a draft pull request. :)

Yes, I noticed it after reviewing - my apologies.

The problem with this approach is that it also deletes the downloaded files (if they need not be extracted). 😟

> The problem with this approach is that it also deletes the downloaded files (if they need not be extracted). worried

Right! These probably should not be deleted by default, but having an option for those users who are tight on disc space?

> Right! These probably should not be deleted by default, but having an option for those users who are tight on disc space?

I propose leaving that for another PR, and leave this one handling only with "extracted" files. Is it OK for you? :)

Awesome thanks !
I just have one question: what about image/audio datasets for which we store the path to the extracted file on the arrow data ?
In this case the default should be to keep the extracted files.

So for now I would just make `keep_extracted=True` by default until we have a way to separate extracted files that can be deleted and extracted files that are actual resources of the dataset.

@lhoestq, current implementation only deletes extracted "files", not extracted "directories", as it uses: `os.remove(path)`. I'm going to add a filter on files, so that this line does not throw an exception when passed a directory.

For audio datasets, the audio files are inside the extracted "directory", so they are not deleted.

I'm still more in favor of having `keep_extracted=True` by default:
- When working with a dataset, you call `load_dataset` many times. By default we want to keep objects extracted to not extract them over and over again (it can take a long time). Then once you know what you're doing and you want to optimize disk space, you can do `keep_extracted=False`. Deleting the extracted files by default is a regression that can lead to slow downs for people calling `load_dataset` many times, which is common when experimenting
- This behavior doesn't sound natural as a default behavior. In the rest of the library, things are cached and not removed unless you explicitly say do (`map` caching for example). Moreover the function in the download manager is called `download_and_extract`, not `download_and_extract_and_remove_extracted_files`

Let me know what you think !

I think the main issue is that after doing some work users typically move on to other datasets and the amount of disc space used keeps on growing. So your logic is very sound and perhaps what's really needed is a cleansweep function that can go through **all** datasets and clean them up to the desired degree:

- delete all extracted files
- delete all sources
- delete all caches
- delete all caches that haven't been accessed  in 6 months
- delete completely old datasets that haven't been accessed  in 6 months
- more?

So a user can launch a little application, choose what they want to clean up and voila they have just freed up a huge amount of disc space. Makes me think of Ubuntu Tweak's Janitor app - very useful.

At the moment, this process of linting is very daunting and error-prone, especially due to all those dirs/files with hash names.

@stas00 I've had the same idea. Instead of the full-fledged app, a simpler approach would be to add a new command to the CLI.

oh, CLI would be perfect. I didn't mean to request a GUI-one specifically, was just using it as an example.

One could even do a crontab to delete old datasets that haven't been accesses in X months.

@lhoestq I totally agree with you. I'm addressing that change.

@stas00, @mariosasko, that could eventually be addressed in another pull request. The objective of this PR is:
- add an option to pass to `load_dataset`, so that extracted files are deleted
- do this deletion file per file, once the file has been already used to generate the cache Arrow file

I also like the idea of having a CLI tool to help users clean their cache and save disk space, good idea !

Close #2481, close #2604, close #2591.

cc: @stas00, @thomwolf, @BirgerMoell


Progress bars are not properly rendered in Jupyter notebook

bug

Something isn't working

To add my experience when trying to debug this issue:

Seems like previously the workaround given [here](https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308) worked around this issue. But with the latest version of jupyter/tqdm I still get terminal warnings that IPython tried to send a message from a forked process.

Hi @mludv, thanks for the hint!!! :)

We will definitely take it into account to try to fix this issue... It seems somehow related to `multiprocessing` and `tqdm`...

## Describe the bug
The progress bars are not Jupyter widgets; regular progress bars appear (like in a terminal).

## Steps to reproduce the bug
```python
ds.map(tokenize, num_proc=10)
```

## Expected results
Jupyter widgets displaying the progress bars.

## Actual results
Simple plane progress bars.

cc: Reported by @thomwolf


Load datasets from the Hub without requiring a dataset script

This is so cool, let us know if we can help with anything on the hub side (@Pierrci @elishowk) 🎉

As a user I would like to be able to upload my csv/json/text/parquet/etc. files in a dataset repository on the Hugging Face Hub and be able to load this dataset with `load_dataset` without having to implement a dataset script.

Moreover I would like to be able to specify which file goes into which split using the `data_files` argument.

This feature should be compatible with private repositories and dataset streaming.

This can be implemented by checking the extension of the files in the dataset repository and then by using the right dataset builder that is already packaged in the library (csv/json/text/parquet/etc.)


Use ETag of remote data files

1.10

Next minor release

Use ETag of remote data files to create config ID.

Related to #2616.


Minor fix tests with Windows paths

1.10

Next minor release

Minor fix tests with Windows paths.


Use correct logger in metrics.py

1.10

Next minor release

Fixes #2624


can't set verbosity for `metric.py`

Something isn't working

Thanks @thomas-happify for reporting and thanks @mariosasko for the fix.

## Describe the bug
```
[2021-07-10 20:13:11,528][datasets.utils.filelock][INFO] - Lock 139705371374976 acquired on /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.lock
[2021-07-10 20:13:11,529][datasets.arrow_writer][INFO] - Done writing 32 examples in 6100 bytes /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.
[2021-07-10 20:13:11,531][datasets.arrow_dataset][INFO] - Set __getitem__(key) output type to python objects for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
[2021-07-10 20:13:11,543][/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric.py][INFO] - Removing /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow
```
As you can see, `datasets` logging come from different places.
`filelock`, `arrow_writer` & `arrow_dataset` comes from `datasets.*` which are expected
However, `metric.py` logging comes from `/conda/envs/myenv/lib/python3.8/site-packages/datasets/`

So when setting `datasets.utils.logging.set_verbosity_error()`,  it still logs the last message which is annoying during evaluation.

I had to do
```
logging.getLogger("/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric").setLevel(logging.ERROR)
```
to fully mute these messages

## Expected results
it shouldn't log these messages when setting `datasets.utils.logging.set_verbosity_error()`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: tried both 1.8.0 & 1.9.0
- Platform: Ubuntu 18.04.5 LTS
- Python version: 3.8.10
- PyArrow version: 3.0.0


[Metrics] added wiki_split metrics

Looks all good to me thanks :)
Just did some minor corrections in the docstring

Fixes: #2606

This pull request adds combine metrics for the wikisplit or English sentence split task

Reviewer: @patrickvonplaten


Integration with AugLy

enhancement

New feature or request

Hi,

you can define your own custom formatting with `Dataset.set_transform()` and then run the tokenizer with the batches of augmented data as follows:
```python
dset = load_dataset("imdb", split="train")  # Let's say we are working with the IMDB dataset
dset.set_transform(lambda ex: {"text": augly_text_augmentation(ex["text"])}, columns="text", output_all_columns=True)
dataloader = torch.utils.data.DataLoader(dset, batch_size=32)
for epoch in range(5):
    for batch in dataloader:
       tokenizer_output = tokenizer(batch.pop("text"), padding=True, truncation=True, return_tensors="pt")
       batch.update(tokenizer_output)
       output = model(**batch)
       ...
```

**Is your feature request related to a problem? Please describe.**
Facebook recently launched a library, [AugLy](https://github.com/facebookresearch/AugLy) , that has a unified API for augmentations for image, video and text.

It would be pretty exciting to have it hooked up to HF libraries so that we can make NLP models robust to misspellings or to punctuation, or emojis etc. Plus, with Transformers supporting more CV use cases, having augmentations support becomes crucial.

**Describe the solution you'd like**
The biggest difference between augmentations and preprocessing is that preprocessing happens only once, but you are running augmentations once per epoch. AugLy operates on text directly, so this breaks the typical workflow where we would run the tokenizer once, set format to pt tensors and be ready for the Dataloader.

**Describe alternatives you've considered**

One possible way of implementing these is to make a custom Dataset class where getitem(i) runs the augmentation and the tokenizer every time, though this would slow training down considerably given we wouldn't even run the tokenizer in batches.


Use prefix to allow exceed Windows MAX_PATH

Does this mean the `FileNotFoundError` that avoids infinite loop can be removed?

 > Or maybe we could leave it in case a relative path exceeds the MAX_PATH limit?

What about converting relative paths to absolute?

Nice ! Have you had a chance to test it on a windows machine with the max path limit enabled ? Afaik the CI doesn't have the path limit

Sure @lhoestq: I've tested on my machine... And this fixes most of the tests... 😅

By using this prefix, you can exceed the Windows MAX_PATH limit.

See: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#win32-file-namespaces

Related to #2524, #2220.


Add speech processing tasks

> Are there any `task_categories:automatic-speech-recognition` dataset for which we should update the tags ?

Yes there's a few - I'll fix them tomorrow :)

This PR replaces the `automatic-speech-recognition` task category with a broader `speech-processing` category.

The tasks associated with this category are derived from the [SUPERB benchmark](https://arxiv.org/abs/2105.01051), and ASR is included in this set.


Add ASR task for SUPERB

1.10

Next minor release

Wait until #2620 is merged before pushing the README tags in this PR

> Thanks!
>
> One question: aren't you adding `task_templates` to the `_info` method (and to the `dataset_infos.json`?

great catch! i've now added the asr task template (along with a mapping from superb task -> template) and updated the `dataset_infos.json` :)

> Good!
>
> I have a suggested refactoring... Tell me what you think! :)

your approach is much more elegant - i've included your suggestions 🙏

This PR starts building up the SUPERB benchmark by including the ASR task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/v0.2.0/downstream#asr-automatic-speech-recognition).

Usage:

```python
from datasets import load_dataset

asr = load_dataset("superb", "asr")
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 28539
#     })
#     validation: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 2703
#     })
#     test: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 2620
#     })
# })
```

I've used the GLUE benchmark as a guide for filling out the README.

To move fast during the evaluation PoC I propose to merge one task at a time, so we can continue building the training / evaluation framework in parallel.

Note: codewise this PR is ready for review - I'll add the missing YAML tags once #2620 is merged :)


`filelock.py` Error

bug

Something isn't working

Hi @liyucheng09, thanks for reporting.

Apparently this issue has to do with your environment setup. One question: is your data in an NFS share? Some people have reported this error when using `fcntl` to write to an NFS share... If this is the case, then it might be that your NFS just may not be set up to provide file locks. You should ask your system administrator, or try these commands in the terminal:
```shell
sudo systemctl enable rpc-statd
sudo systemctl start rpc-statd
```

## Describe the bug

It seems that the `filelock.py` went error.

```
>>> ds=load_dataset('xsum')

^CTraceback (most recent call last):
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 402, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
OSError: [Errno 37] No locks available
```

According to error log, it is OSError, but there is an `except` in the `_acquire` function.

```
    def _acquire(self):
        open_mode = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_TRUNC
        try:
            fd = os.open(self._lock_file, open_mode)
        except (IOError, OSError):
            pass
        else:
            self._lock_file_fd = fd
        return None
```

I don't know why it stucked rather than `pass` directly.

I am not quite familiar with filelock operation, so any help is highly appriciated.

## Steps to reproduce the bug
```python

ds = load_dataset('xsum')
```

## Expected results
A clear and concise description of the expected results.

## Actual results
```
>>> ds=load_dataset('xsum')

^CTraceback (most recent call last):
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 402, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
OSError: [Errno 37] No locks available

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/load.py", line 818, in load_dataset
    use_auth_token=use_auth_token,
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/load.py", line 470, in prepare_module
    with FileLock(lock_path):
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 323, in __enter__
    self.acquire()
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 272, in acquire
    self._acquire()
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 402, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
KeyboardInterrupt
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.0
- Platform: Linux-4.15.0-135-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.13
- PyArrow version: 4.0.1


Fix missing EOL issue in to_json for old versions of pandas

1.10

Next minor release

Some versions of pandas don't add an EOL at the end of the output of `to_json`.
Therefore users could end up having two samples in the same line

Close https://github.com/huggingface/datasets/issues/2615


Support remote data files

enhancement

New feature or request

1.10

Next minor release

> @lhoestq maybe we could also use (if available) the ETag of the remote file in `create_config_id`?

Sure ! We can get the ETag with
```python
headers = get_authentication_headers_for_url(url, use_auth_token=use_auth_token)  # auth for private repos
etag = http_head(url, headers=headers).headers.get("ETag")
```

Since the computation of the `config_id` is done in the `DatasetBuilder.__init__`, then this means that we need to add a new parameter `use_auth_token` in `DatasetBuilder.__init__`

Does that sound good ? We can add this in a following PR

Add support for (streaming) remote data files:

```python
data_files = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{relative_file_path}"
ds = load_dataset("json", split="train", data_files=data_files, streaming=True)
```

cc: @thomwolf


Jsonlines export error

bug

Something isn't working

Thanks for reporting @TevenLeScao! I'm having a look...

(not sure what just happened on the assignations sorry)

For some reason this happens (both `datasets` version are on master) only on Python 3.6 and not Python 3.8.

@TevenLeScao we are using `pandas` to serialize the dataset to JSON Lines. So it must be due to pandas. Could you please check the pandas version causing the issue?

@TevenLeScao I have just checked it: this was a bug in `pandas` and it was fixed in version 1.2: https://github.com/pandas-dev/pandas/pull/36898

Well I though it was me who has taken on this issue... 😅

Sorry, I was also talking to teven offline so I already had the PR ready before noticing x)

I was also already working in my PR... Nevermind. Next time we should pay attention if there is somebody (self-)assigned to an issue and if he/she is still working on it before overtaking it... 😄

The fix is available on `master` @TevenLeScao , thanks for reporting

## Describe the bug
When exporting large datasets in jsonlines (c4 in my case) the created file has an error every 9999 lines: the 9999th and 10000th are concatenated, thus breaking the jsonlines format. This sounds like it is related to batching, which is by 10000 by default

## Steps to reproduce the bug
This what I'm running:

in python:

```
from datasets import load_dataset
ptb = load_dataset("ptb_text_only")
ptb["train"].to_json("ptb.jsonl")
```

then out of python:

```
head -10000 ptb.jsonl
```

## Expected results
Properly separated lines

## Actual results
The last line is a concatenation of two lines

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.1.dev0
- Platform: Linux-5.4.0-1046-gcp-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyArrow version: 4.0.1


Convert numpy scalar to python float in Pearsonr output

1.10

Next minor release

Following of https://github.com/huggingface/datasets/pull/2612


Use ndarray.item instead of ndarray.tolist

1.10

Next minor release

This PR follows up on #2612 to use `numpy.ndarray.item` instead of `numpy.ndarray.tolist` as the latter is somewhat confusing to the developer (even though it works).

Judging from the `numpy` docs, `ndarray.item` is closer to what we want: https://numpy.org/doc/stable/reference/generated/numpy.ndarray.item.html#numpy-ndarray-item

PS. Sorry for the duplicate work here. I should have read the numpy docs more carefully in #2612


Return Python float instead of numpy.float64 in sklearn metrics

1.10

Next minor release

I opened an issue on the `sklearn` repo to understand why `numpy.float64` is the default: https://github.com/scikit-learn/scikit-learn/discussions/20490

It could be surprising at first to use `tolist()` on numpy scalars but it works ^^

did the same for Pearsonr here: https://github.com/huggingface/datasets/pull/2614

This PR converts the return type of all `sklearn` metrics to be Python `float` instead of `numpy.float64`.

The reason behind this is that our Hub evaluation framework relies on converting benchmark-specific metrics to YAML ([example](https://huggingface.co/datasets/autonlp/autonlp-benchmark-raft-neelalex__raft-test-neelalex__raft-predictions-3/blob/main/README.md#L11)) and the `numpy.float64` format produces garbage like:

```python
import yaml
from datasets import load_metric

metric = load_metric("accuracy")
score = metric.compute(predictions=[0,1], references=[0,1])
print(yaml.dump(score["accuracy"])) # output below
# !!python/object/apply:numpy.core.multiarray.scalar
# - !!python/object/apply:numpy.dtype
#   args:
#   - f8
#   - false
#   - true
#   state: !!python/tuple
#   - 3
#   - <
#   - null
#   - null
#   - null
#   - -1
#   - -1
#   - 0
# - !!binary |
#   AAAAAAAA8D8=
```


More consistent naming

As per @stas00's suggestion in #2500, this PR inserts a space between the logo and the lib name (`🤗Datasets` -> `🤗 Datasets`) for consistency with the Transformers lib. Additionally, more consistent names are used for Datasets Hub, etc.


Add missing WikiANN language tags

1.10

Next minor release

Add missing language tags for WikiANN datasets.


Fix potential DuplicatedKeysError

1.10

Next minor release

Fix potential DiplicatedKeysError by ensuring keys are unique.

We should promote as a good practice, that the keys should be programmatically generated as unique, instead of read from data (which might be not unique).


Support streaming JSON files

1.10

Next minor release

Use open in JSON dataset builder, so that it can be patched with xopen for streaming.

Close #2607.


Streaming local gzip compressed JSON line files is not working

Something isn't working

Updating to pyarrow-4.0.1 didn't fix the issue

Here is an exemple dataset with 2 of these compressed JSON files: https://huggingface.co/datasets/thomwolf/github-python

Hi @thomwolf, thanks for reporting.

It seems this might be due to the fact that the JSON Dataset builder uses `pyarrow.json` (`paj.read_json`) to read the data without using the Python standard `open(file,...` (which is the one patched with `xopen` to work in streaming mode).

This has to be fixed.

Sorry for reopening this, but I'm having the same issue as @thomwolf when streaming a gzipped JSON Lines file from the hub. Or is that just not possible by definition?
I installed `datasets`in editable mode from source (so probably includes the fix from #2608 ?):
```
>>> datasets.__version__
'1.9.1.dev0'
```

```
>>> msmarco = datasets.load_dataset("webis/msmarco", "corpus", streaming=True)
Using custom data configuration corpus-174d3b7155eb68db
>>> msmarco_iter = iter(msmarco['train'])
>>> print(next(msmarco_iter))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/media/ssd/TREC/msmarco/datasets/src/datasets/iterable_dataset.py", line 338, in __iter__
    for key, example in self._iter():
  File "/media/ssd/TREC/msmarco/datasets/src/datasets/iterable_dataset.py", line 335, in _iter
    yield from ex_iterable
  File "/media/ssd/TREC/msmarco/datasets/src/datasets/iterable_dataset.py", line 78, in __iter__
    for key, example in self.generate_examples_fn(**self.kwargs):
  File "/home/christopher/.cache/huggingface/modules/datasets_modules/datasets/msmarco/eb63dff8d83107168e973c7a655a6082d37e08d71b4ac39a0afada479c138745/msmarco.py", line 96, in _generate_examples
    with gzip.open(file, "rt", encoding="utf-8") as f:
  File "/usr/lib/python3.6/gzip.py", line 53, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/usr/lib/python3.6/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'https://huggingface.co/datasets/webis/msmarco/resolve/main/msmarco_doc_00.gz'
```

Loading the dataset without streaming set to True, works fine.

Hi ! To make the streaming work, we extend `open` in the dataset builder to work with urls.

Therefore you just need to use `open` before using `gzip.open`:
```diff
- with gzip.open(file, "rt", encoding="utf-8") as f:
+ with gzip.open(open(file, "rb"), "rt", encoding="utf-8") as f:
```

You can see that it is the case for oscar.py and c4.py for example:

https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/datasets/oscar/oscar.py#L358-L358

https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/datasets/c4/c4.py#L88-L88

@lhoestq Sorry I missed that. Thank you Quentin!

## Describe the bug
Using streaming to iterate on local gzip compressed JSON files raise a file not exist error

## Steps to reproduce the bug
```python
from datasets import load_dataset

streamed_dataset = load_dataset('json', split='train', data_files=data_files, streaming=True)

next(iter(streamed_dataset))
```

## Actual results
```
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-27a664e29784> in <module>
----> 1 next(iter(streamed_dataset))

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in __iter__(self)
    336
    337     def __iter__(self):
--> 338         for key, example in self._iter():
    339             if self.features:
    340                 # we encode the example for ClassLabel feature types for example

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in _iter(self)
    333         else:
    334             ex_iterable = self._ex_iterable
--> 335         yield from ex_iterable
    336
    337     def __iter__(self):

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in __iter__(self)
     76
     77     def __iter__(self):
---> 78         for key, example in self.generate_examples_fn(**self.kwargs):
     79             yield key, example
     80

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in wrapper(**kwargs)
    282     def wrapper(**kwargs):
    283         python_formatter = PythonFormatter()
--> 284         for key, table in generate_tables_fn(**kwargs):
    285             batch = python_formatter.format_batch(table)
    286             for i, example in enumerate(_batch_to_examples(batch)):

~/Documents/GitHub/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files, original_files)
     85                         file,
     86                         read_options=self.config.pa_read_options,
---> 87                         parse_options=self.config.pa_parse_options,
     88                     )
     89                 except pa.ArrowInvalid as err:

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/_json.pyx in pyarrow._json.read_json()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/_json.pyx in pyarrow._json._get_reader()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.get_input_stream()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.get_native_file()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.OSFile.__cinit__()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.OSFile._open_readable()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

FileNotFoundError: [Errno 2] Failed to open local file 'gzip://file-000000000000.json::/Users/thomwolf/github-dataset/file-000000000000.json.gz'. Detail: [errno 2] No such file or directory
```

## Environment info
- `datasets` version: 1.9.1.dev0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyArrow version: 1.0.0


[Metrics] addition of wiki_split metrics

enhancement

New feature or request

metric request

Requesting to add a new metric

**Is your feature request related to a problem? Please describe.**
While training the model on sentence split the task in English we require to evaluate the trained model on `Exact Match`, `SARI` and `BLEU` score
like this
![image](https://user-images.githubusercontent.com/26653468/124746876-ff5a3380-df3e-11eb-9a01-4b48db7a6694.png)
While training we require metrics which can give all the output

Currently, we don't have an exact match for text normalized data

**Describe the solution you'd like**
A custom metrics for wiki_split that can calculate these three values and provide it in the form of a single dictionary
For exact match, we can refer to [this](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py)

**Describe alternatives you've considered**
Two metrics are already present one more can be added for an exact match then we can run all three metrics in training script

#self-assign


Make any ClientError trigger retry in streaming mode (e.g. ClientOSError)

1.10

Next minor release

During the FLAX sprint some users have this error when streaming datasets:
```python
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
```
This error must trigger a retry instead of directly crashing

Therefore I extended the error type that triggers the retry to be the base aiohttp error type: `ClientError`
In particular both `ClientOSError` and `ServerDisconnectedError` inherit from `ClientError`.


Add option to delete temporary files (e.g. extracted files) when loading dataset

enhancement

New feature or request

1.10

Next minor release

Hi !
If we want something more general, we could either
1. delete the extracted files after the arrow data generation automatically, or
2. delete each extracted file during the arrow generation right after it has been closed.

Solution 2 is better to save disk space during the arrow generation. Is it what you had in mind ?

The API could look like
```python
load_dataset(..., delete_extracted_files_after_usage=True)
```

In terms of implementation, here are some directions we could take for each solution:
1. get the list of the extracted files from the DownloadManager and then delete them after the dataset is processed. This can be implemented in `download_and_prepare` I guess
2. maybe wrap and mock `open` in the builder to make it delete the file when the file is closed.

Also, if I delete the extracted files they need to be re-extracted again instead of loading from the Arrow cache files

I think we already opened an issue about this topic (suggested by @stas00): duplicated of #2481?

This is in our TODO list... 😅

I think the deletion of each extracted file could be implemented in our CacheManager and ExtractManager (once merged to master: #2295, #2277). 😉

Oh yes sorry, I didn't check if this was a duplicate

Nevermind @thomwolf, I just mentioned the other issue so that both appear linked in GitHub and we do not forget to close both once we make the corresponding Pull Request... That was the main reason! 😄

Ok yes. I think this is an important feature to be able to use large datasets which are pretty much always compressed files.

In particular now this requires to keep the extracted file on the drive if you want to avoid reprocessing the dataset so in my case, this require using always ~400GB of drive instead of just 200GB (which is already significant).

Two nice features would be to:
- allow to delete the extracted files without loosing the ability to load the dataset from the cached arrow-file
- streamlined decompression when only the currently read file is extracted - this might require to read the list of files from the extracted archives before processing them?

Here is a sample dataset with 2 such large compressed JSON files for debugging: https://huggingface.co/datasets/thomwolf/github-python

Note that I'm confirming that with the current master branch of dataset, deleting extracted files (without deleting the arrow cache file) lead to **re-extracting** these files when reloading the dataset instead of directly loading the arrow cache file.

Hi ! That's weird, it doesn't do that on my side (tested on master on my laptop by deleting the `extracted` folder in the download cache directory). You tested with one of the files at https://huggingface.co/datasets/thomwolf/github-python that you have locally ?

Yes it’s when I load local compressed JSON line files with load_dataset(‘json’, data_files=…)

@thomwolf I'm sorry but I can't reproduce this problem. I'm also using:
```python
ds = load_dataset("json", split="train", data_files=data_files, cache_dir=cache_dir)
```
after having removed the extracted files:
```python
assert sorted((cache_dir / "downloads" / "extracted").iterdir()) == []
```

I get the logging message:
```shell
WARNING  datasets.builder:builder.py:531 Reusing dataset json ...
```

>
>
> Do you confirm the extracted folder stays empty after reloading?

Yes, I have the above mentioned assertion on the emptiness of the extracted folder:
```python
assert sorted((cache_dir / "downloads" / "extracted").iterdir()) == []
```

I'm loading a dataset constituted of 44 GB of compressed JSON files.

When loading the dataset with the JSON script, extracting the files create about 200 GB of uncompressed files before creating the 180GB of arrow cache tables

Having a simple way to delete the extracted files after usage (or even better, to stream extraction/delete) would be nice to avoid disk cluter.

I can maybe tackle this one in the JSON script unless you want a more general solution.


Fix DuplicatedKeysError in omp

1.10

Next minor release

Close #2598.


Remove import of transformers

1.10

Next minor release

When pickling a tokenizer within multiprocessing, check that is instance of transformers PreTrainedTokenizerBase without importing transformers.

Related to huggingface/transformers#12549 and #502.


Fix `filter` with multiprocessing in case all samples are discarded

1.10

Next minor release

Fixes #2600

Also I moved the check for `num_proc` larger than dataset size added in #2566 up so that multiprocessing is not used with one process.


Crash when using multiprocessing (`num_proc` > 1) on `filter` and all samples are discarded

bug

Something isn't working

## Describe the bug
If `filter` is applied to a dataset using multiprocessing (`num_proc` > 1) and all sharded datasets are empty afterwards (due to all samples being discarded), the program crashes.

## Steps to reproduce the bug
```python
from datasets import Dataset
data = Dataset.from_dict({'id': [0,1]})
data.filter(lambda x: False, num_proc=2)
```

## Expected results
An empty table should be returned without crashing.

## Actual results
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 185, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2143, in filter
    return self.map(
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1738, in map
    result = concatenate_datasets(transformed_shards)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3267, in concatenate_datasets
    table = concat_tables(tables_to_concat, axis=axis)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/table.py", line 853, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/table.py", line 713, in from_tables
    blocks = to_blocks(tables[0])
IndexError: list index out of range
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.12.11-300.fc34.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.10
- PyArrow version: 3.0.0


Update processing.rst with other export formats

1.10

Next minor release

Add other supported export formats than CSV in the docs.


Unable to download omp dataset

Something isn't working

Hi @erikadistefano , thanks for reporting the issue.

I have created a Pull Request that should fix it.

Once merged into master, feel free to update your installed `datasets` library (either by installing it from our GitHub master branch or waiting until our next release) to be able to load omp dataset.

## Describe the bug
The omp dataset cannot be downloaded because of a DuplicatedKeysError

## Steps to reproduce the bug
from datasets import load_dataset
omp = load_dataset('omp', 'posts_labeled')
print(omp)

## Expected results
This code should download the omp dataset and print the dictionary

## Actual results
Downloading and preparing dataset omp/posts_labeled (download: 1.27 MiB, generated: 13.31 MiB, post-processed: Unknown size, total: 14.58 MiB) to /home/erika_distefano/.cache/huggingface/datasets/omp/posts_labeled/1.1.0/2fe5b067be3bff1d4588d5b0cbb9b5b22ae1b9d5b026a8ff572cd389f862735b...
0 examples [00:00, ? examples/s]2021-07-06 09:43:55.868815: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 990, in _prepare_split
    writer.write(example, key)
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 338, in write
    self.check_duplicate_keys()
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 349, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 3326
Keys should be unique and deterministic in nature

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "hf_datasets.py", line 32, in <module>
    omp = load_dataset('omp', 'posts_labeled')
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 992, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 409, in finalize
    self.check_duplicate_keys()
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 349, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 3326
Keys should be unique and deterministic in nature

## Environment info
- `datasets` version: 1.8.0
- Platform: Ubuntu 18.04.4 LTS
- Python version: 3.6.9
- PyArrow version: 3.0.0


Remove redundant prepare_module

refactoring

Restructuring existing code without changing its external behavior

1.10

Next minor release

I have noticed that after implementing `load_dataset_builder` (#2500), there is a redundant call to `prepare_module`.