Something isn't working

Hi @JTWang2000, thanks for reporting.

However, I cannot reproduce your reported bug:
```python
>>> from datasets import load_dataset

>>> dataset = load_dataset("sst", "default")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 8544
    })
    validation: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 1101
    })
    test: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 2210
    })
})
```

Maybe, the cause is that you have a quite old version of `huggingface_hub`. Could you please try to update it and confirm if the problem persists?
```
pip install -U huggingface_hub
```

Im facing the same issue. I did run the upgrade command but that doesnt seem to resolve the issue

Hi @aneeshjain, could you please specify which `huggingface_hub` version you are using?

Besides that, please run `datasets-cli env` and copy-and-paste its output below.

The problem seems to be with the latest version of `datasets`. After running `pip install -U datasets huggingface_hub`, I get the following:

```bash
python -c "import huggingface_hub; print(f'hbvers={huggingface_hub.__version__}'); import datasets; print(f'dvers={datasets.__version__}')"
hbvers=0.0.8
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/opt/conda/lib/python3.6/site-packages/datasets/data_files.py", line 122, in <module>
    allowed_extensions: Optional[list] = None,
AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
````
Note that pip reports the latest `datasets` version as
```bash
 pip show datasets
Name: datasets
Version: 1.14.0
```
However, if I downgrade datasets with `pip install datasets==1.11.0`, things now work
```bash
python -c "import huggingface_hub; print(f'hbvers={huggingface_hub.__version__}'); import datasets; print(f'dvers={datasets.__version__}')"
hbvers=0.0.8
dvers=1.11.0
````

> Hi @JTWang2000, thanks for reporting.
>
> However, I cannot reproduce your reported bug:
>
> ```python
> >>> from datasets import load_dataset
>
> >>> dataset = load_dataset("sst", "default")
> >>> dataset
> DatasetDict({
>     train: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 8544
>     })
>     validation: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 1101
>     })
>     test: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 2210
>     })
> })
> ```
>
> Maybe, the cause is that you have a quite old version of `huggingface_hub`. Could you please try to update it and confirm if the problem persists?
>
> ```
> pip install -U huggingface_hub
> ```

My problem solved after updating huggingface hub command. Thanks a lot and sorry for the late reply.

@tjruwase, please note that versions of `datsets` and `huggingface_hub` must be compatible one with each other:
- In `datasets` version `1.11.0`, the requirement on `huggingface_hub` was: `huggingface_hub<0.1.0`
  https://github.com/huggingface/datasets/blob/2cc00f372a96133e701275eec4d6b26d15257289/setup.py#L90
  - Therefore, your installed `huggingface_hub` version `0.0.8` was compatible
- In `datasets` version `1.12.0`, the requirement on `huggingface_hub` was: `huggingface_hub>=0.0.14,<0.1.0`
  https://github.com/huggingface/datasets/blob/6c766f9115d686182d76b1b937cb27e099c45d68/setup.py#L104
  - Therefore, your installed `huggingface_hub` version `0.0.8` was no longer compatible
- Currently, in `datasets` version `1.15.1`, the requirement on `huggingface_hub` is: `huggingface_hub>=0.1.0,<1.0.0`
  https://github.com/huggingface/datasets/blob/018100679d21cf27136f0eccb1c50e3a9c968ce2/setup.py#L102

@JTWang2000, thanks for your answer. I close this issue then.

## Describe the bug
When using `pip install datasets`
or use `conda install -c huggingface -c conda-forge datasets`
cannot install datasets

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("sst", "default")
```

## Actual results
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-fbe7981e6e21> in <module>
      1 import torch
      2 import transformers
----> 3 from datasets import load_dataset
      4
      5 dataset = load_dataset("sst", "default")

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/__init__.py in <module>
     35 from .arrow_reader import ArrowReader, ReadInstruction
     36 from .arrow_writer import ArrowWriter
---> 37 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     38 from .combine import interleave_datasets
     39 from .dataset_dict import DatasetDict, IterableDatasetDict

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/builder.py in <module>
     42 )
     43 from .arrow_writer import ArrowWriter, BeamWriter
---> 44 from .data_files import DataFilesDict, _sanitize_patterns
     45 from .dataset_dict import DatasetDict, IterableDatasetDict
     46 from .fingerprint import Hasher

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/data_files.py in <module>
    118
    119 def _exec_patterns_in_dataset_repository(
--> 120     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    121     patterns: List[str],
    122     allowed_extensions: Optional[list] = None,

AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-11.3.1-arm64-arm-64bit
- Python version: 3.8.10
- PyArrow version: 5.0.0


Push to hub capabilities for `Dataset` and `DatasetDict`

Thank you for your reviews! I should have addressed all of your comments, and I added a test to ensure that `private` datasets work correctly too. I have merged the changes in `huggingface_hub`, so the `main` branch can be installed now; and I will release v0.1.0 soon.

As blockers for this PR:
- It's still waiting for #3027 to be addressed as the folder name will dictate the split name
- The `self.split` name is set to `None` when the dataset dict is instantiated as follows:
```py
ds = Dataset.from_dict({"x": [1, 2, 3], "y": [4, 5, 6]})
local_ds = DatasetDict({"random": ds})

local_ds['random'].split  # returns None
```
In order to remove the `split=key` I would need to know of a different way to test here as it relies on the above as a surefire way of constructing a `DatasetDict`.
- Finally, the `threading` parameter is flaky on moon-staging which results in many errors server side. I propose to leave it as an argument instead of having it having it set to `True` so that users may toggle it according to their wish.

Currently it looks like it only saves the last split.
Indeed when writing the data of one split, it deletes all the other files from the other splits
```python
>>> dataset.push_to_hub("lhoestq/squad_titles", shard_size=50<<10)
Pushing split train to the Hub.
Pushing dataset shards to the dataset hub: 100%|█| 31/31 [00:22<00:00,  1.38
Pushing split validation to the Hub.
The repository already exists: the `private` keyword argument will be ignored.
Deleting unused files from dataset repository: 100%|█| 31/31 [00:14<00:00,
Pushing dataset shards to the dataset hub: 100%|█| 4/4 [00:03<00:00,  1.18it
```
Note the "Deleting" part.

I think this PR should fix #3035, so feel free to link it.

Thank you for your comments! I have rebased on `master` to have PR #3221. I've updated all tests to reflect the `-` instead of the `_` in the filenames.

@lhoestq, I have fixed the issue with splits and added a corresponding test.

@mariosasko I have not updated the `load_dataset` method to work differently, so I don't expect #3035 to be resolved with `push_to_hub`.

Only remaining issues before merging:
- Take a good look at the `threading` and if that's something we want to keep.
- As mentioned above:
>The self.split name is set to None when the dataset dict is instantiated as follows:
> ```
> ds = Dataset.from_dict({"x": [1, 2, 3], "y": [4, 5, 6]})
> local_ds = DatasetDict({"random": ds})
>
> local_ds['random'].split  # returns None
> ```
I need to understand how to build a `DatasetDict` from some `Dataset` objects to be able to leverage the `split` parameter in `DatasetDict.push_to_hub`

Cool thanks ! And indeed this won't solve https://github.com/huggingface/datasets/issues/3035 yet

> I need to understand how to build a DatasetDict from some Dataset objects to be able to leverage the split parameter in DatasetDict.push_to_hub

You can use the key in the DatasetDict instead of the `split` attribute

What do you think about bumping the minimum version of pyarrow to 3.0.0 ? This is the minimum required version to write parquet files, which is needed for push_to_hub. That's why our pyarrow 1 CI is failing.

I think it's fine since it's been available for a long time (january 2021) and it's also the version that is installed on Google Colab.

This PR implements a `push_to_hub` method on `Dataset` and `DatasetDict`. This does not currently work in `IterableDatasetDict` nor `IterableDataset` as those are simple dicts and I would like your opinion on how you would like to implement this before going ahead and doing it.

This implementation needs to be used with the following `huggingface_hub` branch in order to work correctly: https://github.com/huggingface/huggingface_hub/pull/415

### Implementation

The `push_to_hub` API is entirely based on HTTP requests rather than a git-based workflow:
- This allows pushing changes without firstly cloning the repository, which reduces the time in half for the `push_to_hub` method.
- Collaboration, as well as the system of branches/merges/rebases is IMO less straightforward than for models and spaces. In the situation where such collaboration is needed, I would *heavily* advocate for the `Repository` helper of the `huggingface_hub` to be used instead of the `push_to_hub` method which will always be, by design, limiting in that regard (even if based on a git-workflow instead of HTTP requests)

In order to overcome the limit of 5GB files set by the HTTP requests, dataset sharding is used.

### Testing

The test suite implemented here makes use of the moon-staging instead of the production setup. As several repositories are created and deleted, it is better to use the staging.

It does not require setting an environment variable or any kind of special attention but introduces a new decorator `with_staging_testing` which patches global variables to use the staging endpoint instead of the production endpoint.

### Examples

The tests cover a lot of examples and behavior.


`ModuleNotFoundError: No module named 'fsspec.exceptions'`

bug

Something isn't working

Thanks for reporting, @VictorSanh.

I'm fixing it.

## Describe the bug
I keep runnig into a fsspec ModuleNotFound error

## Steps to reproduce the bug
```python
>>> from datasets import get_dataset_infos
2021-10-15 15:25:37.863206: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-10-15 15:25:37.863252: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 56, in <module>
    from .utils.streaming_download_manager import StreamingDownloadManager
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/streaming_download_manager.py", line 11, in <module>
    from fsspec.exceptions import FSTimeoutError
ModuleNotFoundError: No module named 'fsspec.exceptions'
```

Yet, I do have `fsspec`:
```bash
hf@victor-scale:~/dev/promptsource$ pip show fsspec
Name: fsspec
Version: 2021.5.0
Summary: File-system specification
Home-page: http://github.com/intake/filesystem_spec
Author: None
Author-email: None
License: BSD
Location: /home/hf/dev/promptsource/.venv/lib/python3.7/site-packages
Requires:
Required-by: datasets
```

With the same version of fsspec and `datasets==1.9.0`, I don't see this problem....

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
I can't even run `datasets-cli env` actually.., but here's my env:
- `datasets` version: 1.13.3
- Platform: Ubuntu 18.04
- Python version: 3.7.10
- PyArrow version: 3.0.0


Fix Audio feature mp3 resampling

Issue #3095 is related to mp3 resampling, not to `cast_column`.

This PR fixes Audio feature mp3 resampling.

Fix #3095.


`cast_column` makes audio decoding fail

Something isn't working

Thanks for reporting, @patrickvonplaten.

I think the issue is related to mp3 resampling, not to `cast_column`.

You can check that `cast_column` works OK with non-mp3 audio files:
```python
from datasets import load_dataset
import datasets
ds = load_dataset("arabic_speech_corpus", split="train")
ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))
print(ds[0]["audio"])
```

I'm fixing it.

## Describe the bug

After changing the sampling rate automatic decoding fails.

## Steps to reproduce the bug
```python
from datasets import load_dataset
import datasets

ds = load_dataset("common_voice", "ab", split="train")

ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))

print(ds[0]["audio"])  # <- this fails currently
```

yields:

```
TypeError: forward() takes 2 positional arguments but 4 were given
```

## Expected results
no failure

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 1.13.2 (master)
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0


Support loading a dataset from SQLite files

enhancement

New feature or request

As requested by @julien-c, we could eventually support loading a dataset from SQLite files, like it is the case for JSON/CSV files.


Error loading json dataset with multiple splits if keys in nested dicts have a different order

bug

Something isn't working

Hi,

even Pandas, which is less strict compared to PyArrow when it comes to reading JSON, doesn't support different orderings:
```python
import io
import pandas as pd

s = """
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
"""

buffer = io.StringIO(s)
df = pd.read_json(buffer, lines=True)

print(df.shape[0]) # 0
```

So we can't even fall back to Pandas in such cases.

It seems the only option is a script that recursively re-orders fields to enforce deterministic order:
```python
with open("train.json", "r") as fin:
    with open("train_reordered.json", "w") as fout:
        for line in fin:
            obj_jsonl = json.loads(line.strip())
            fout.write(json.dumps(obj_jsonl, sort_keys=True) + "\n")
```

## Describe the bug
Loading a json dataset with multiple splits that have nested dicts with keys in different order results in the error below.

If the keys in the nested dicts always have the same order or even if you just load a single split in which the nested dicts don't have the same order, everything works fine.

## Steps to reproduce the bug
Create two json files:

train.json
```
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
```

test.json
```
{"a": {"b": 1, "c": 2}}
{"a": {"b": 3, "c": 4}}
```

```python
from datasets import load_dataset
# Loading the files individually works (even though the keys in train.json don't have the same order)
load_dataset('json', data_files={"test": "test.json"})
load_dataset('json', data_files={"train": "train.json"})
# Loading both splits fails
load_dataset('json', data_files={"train": "train.json", "test": "test.json"})
```

## Expected results
Loading both splits should not give an error whether the nested dicts are have the same order or not.

## Actual results
```
>>> load_dataset('json', data_files={"train": "train.json", "test": "test.json"})
Using custom data configuration default-f1bc76fd07398c4c
Downloading and preparing dataset json/default to /home/dthulke/.cache/huggingface/datasets/json/default-f1bc76fd07398c4c/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 8839.42it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 477.82it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/builder.py", line 1159, in _prepare_split
    writer.write_table(table)
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/arrow_writer.py", line 428, in write_table
    pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
  File "pyarrow/table.pxi", line 1596, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 592, in pyarrow.lib._sanitize_arrays
  File "pyarrow/array.pxi", line 329, in pyarrow.lib.asarray
  File "pyarrow/table.pxi", line 277, in pyarrow.lib.ChunkedArray.cast
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/pyarrow/compute.py", line 297, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 527, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 337, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 120, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<b: int64, c: int64> to struct using function cast_struct
```

## Environment info
- `datasets` version: 1.13.2
- Platform: Linux-4.15.0-147-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyArrow version: 5.0.0


Fix JNLBA dataset

@albertvillanova all tests are passing now. Either you or @lhoestq can review it!

As mentioned in #3089, I've added more tags and also updated the link for dataset which was earlier using a Google Drive link.
I'm having problem with generating dummy data as `datasets-cli dummy_data  ./datasets/jnlpba --auto_generate --match_text_files "*.iob2"` is giving `datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
` error. I'll try to add dummy data manually.


`blog_authorship_corpus` is broken

Something isn't working

Hi @fdtomasi, thanks for reporting.

You are right: the original host data URL does no longer exist.

I've contacted the authors of the dataset to ask them if they host this dataset in another URL.

Hi, @fdtomasi, the URL is fixed.

The fix is already in our master branch and it will be accessible in our next release.

In the meantime, you can include the fix if you install the `datasets` library from the master branch:
```
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```

Awesome thank you so much for the quick fix!

## Describe the bug
The dataset `blog_authorship_corpus` is broken.
By bypassing the checksum checks, the loading does not return any error but the resulting dataset is empty.
I suspect it is because the data download url is broken (http://www.cs.biu.ac.il/~koppel/blogs/blogs.zip).

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("blog_authorship_corpus", split="train", download_mode='force_redownload')
```

## Expected results
No error.

## Actual results
```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
/tmp/ipykernel_5237/1729238701.py in <module>
      2 ds = load_dataset(
      3     "blog_authorship_corpus", split="train",
----> 4     download_mode='force_redownload'
      5 )

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
   1115         ignore_verifications=ignore_verifications,
   1116         try_from_hf_gcs=try_from_hf_gcs,
-> 1117         use_auth_token=use_auth_token,
   1118     )
   1119

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    635                     if not downloaded_from_gcs:
    636                         self._download_and_prepare(
--> 637                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    638                         )
    639                     # Sync info

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    707         if verify_infos:
    708             verify_checksums(
--> 709                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    710             )
    711

/opt/conda/lib/python3.7/site-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['http://www.cs.biu.ac.il/~koppel/blogs/blogs.zip']
```

## Environment info
- `datasets` version: 1.13.2
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.10
- PyArrow version: 5.0.0


Update BibTeX entry

Update BibTeX entry.


JNLPBA Dataset

Something isn't working

# Steps to reproduce

To reproduce:

```python
from datasets import load_dataset
dataset = load_dataset('jnlpba')

dataset['train'].features['ner_tags']
```
Output:
```python
Sequence(feature=ClassLabel(num_classes=3, names=['O', 'B', 'I'], names_file=None, id=None), length=-1, id=None)
```

Since I cannot create a branch here is the updated code:

```python

# coding=utf-8
# Copyright 2020 HuggingFace Datasets Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Lint as: python3
"""Introduction to the Bio-Entity Recognition Task at JNLPBA"""

import os

import datasets

logger = datasets.logging.get_logger(__name__)

_CITATION = """\
@inproceedings{kim2004introduction,
               title={Introduction to the bio-entity recognition task at JNLPBA},
               author={Kim, Jin-Dong and Ohta, Tomoko and Tsuruoka, Yoshimasa and Tateisi, Yuka and Collier, Nigel},
               booktitle={Proceedings of the international joint workshop on natural language processing in biomedicine and its applications},
               pages={70--75},
               year={2004},
               organization={Citeseer}
}
"""

_DESCRIPTION = """\
The data came from the GENIA version 3.02 corpus (Kim et al., 2003). This was formed from a controlled search
on MEDLINE using the MeSH terms human, blood cells and transcription factors. From this search 2,000 abstracts
were selected and hand annotated according to a small taxonomy of 48 classes based on a chemical classification.
Among the classes, 36 terminal classes were used to annotate the GENIA corpus.
"""

_HOMEPAGE = "http://www.geniaproject.org/shared-tasks/bionlp-jnlpba-shared-task-2004"
_TRAIN_URL = "http://www.nactem.ac.uk/GENIA/current/Shared-tasks/JNLPBA/Train/Genia4ERtraining.tar.gz"
_VAL_URL = 'http://www.nactem.ac.uk/GENIA/current/Shared-tasks/JNLPBA/Evaluation/Genia4ERtest.tar.gz'

_URLS = {
    "train": _TRAIN_URL,
    "val": _VAL_URL,
}

_TRAIN_DIRECTORY = "Genia4ERtraining"
_VAL_DIRECTORY = "Genia4ERtest"

_TRAIN_FILE = "Genia4ERtask1.iob2"
_VAL_FILE = "Genia4EReval1.iob2"

class JNLPBAConfig(datasets.BuilderConfig):
    """BuilderConfig for JNLPBA"""

    def __init__(self, **kwargs):
        """BuilderConfig for JNLPBA.
        Args:
          **kwargs: keyword arguments forwarded to super.
        """
        super(JNLPBAConfig, self).__init__(**kwargs)

class JNLPBA(datasets.GeneratorBasedBuilder):
    """JNLPBA dataset."""

    BUILDER_CONFIGS = [
        JNLPBAConfig(name="jnlpba", version=datasets.Version("1.0.0"), description="JNLPBA dataset"),
    ]

    def _info(self):
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            features=datasets.Features(
                {
                    "id": datasets.Value("string"),
                    "tokens": datasets.Sequence(datasets.Value("string")),
                    "ner_tags": datasets.Sequence(
                        datasets.features.ClassLabel(
                            names=[
                                'O',
                                'B-DNA',
                                'I-DNA',
                                'B-RNA',
                                'I-RNA',
                                'B-cell_line',
                                'I-cell_line',
                                'B-cell_type',
                                'I-cell_type',
                                'B-protein',
                                'I-protein',
                            ]
                        )
                    ),
                }
            ),
            supervised_keys=None,
            homepage=_HOMEPAGE,
            citation=_CITATION,
        )

    def _split_generators(self, dl_manager):
        downloaded_files = dl_manager.download_and_extract(_URLS)

        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN,
                                    gen_kwargs={"filepath": os.path.join(downloaded_files['train'], _TRAIN_FILE)}),
            datasets.SplitGenerator(name=datasets.Split.VALIDATION,
                                    gen_kwargs={"filepath": os.path.join(downloaded_files['val'], _VAL_FILE)})
        ]

    def _generate_examples(self, filepath):
        logger.info("⏳ Generating examples from = %s", filepath)
        with open(filepath, encoding="utf-8") as f:
            guid = 0
            tokens = []
            ner_tags = []
            for line in f:
                if line.startswith('###'):
                    continue
                if line == '' or line == '\n':
                    if tokens:
                        yield guid, {
                            "id": str(guid),
                            "tokens": tokens,
                            "ner_tags": ner_tags,
                        }
                        guid += 1
                        tokens = []
                        ner_tags = []
                else:
                    # tokens are tab separated
                    splits = line.split("\t")
                    #print(splits)
                    #print(len(splits))
                    if len(splits) < 2:
                        splits = splits[0].split()
                    tokens.append(splits[0])
                    ner_tags.append(splits[1].rstrip())
            # last example
            yield guid, {
                "id": str(guid),
                "tokens": tokens,
                "ner_tags": ner_tags,
            }
```

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results
The dataset loading script for this dataset is incorrect. This is a biomedical dataset used for named entity recognition. The entities in the [script](https://github.com/huggingface/datasets/blob/master/datasets/jnlpba/jnlpba.py#L81-L83) are: O, B, and I.  The correct entities from the original data file are:

['O',
 'B-DNA',
 'I-DNA',
 'B-RNA',
 'I-RNA',
 'B-cell_line',
 'I-cell_line',
 'B-cell_type',
 'I-cell_type',
 'B-protein',
 'I-protein']

## Actual results
The dataset loader script needs to include the following NER names:

['O',
 'B-DNA',
 'I-DNA',
 'B-RNA',
 'I-RNA',
 'B-cell_line',
 'I-cell_line',
 'B-cell_type',
 'I-cell_type',
 'B-protein',
 'I-protein']

And the [data](https://github.com/huggingface/datasets/blob/master/datasets/jnlpba/jnlpba.py#L46) that is being pulled has been modified from the original dataset and does not include the original NER tags.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Use template column_mapping to transmit_format instead of template features

Use `template.column_mapping` to check for modified columns since `template.features` represent a generic template/column mapping.

Fix #3087

TODO:
- [x] Add a test


Removing label column in a text classification dataset yields to errors

Something isn't working

## Describe the bug

This looks like #3059 but it's not linked to the cache this time. Removing the `label` column from a text classification dataset and then performing any processing will result in an error.

To reproduce:
```py
from datasets import load_dataset
from transformers import AutoTokenizer

raw_datasets = load_dataset("imdb")
raw_datasets = raw_datasets.remove_columns("label")

model_checkpoint = "distilbert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
context_length = 128

def tokenize_pad_and_truncate(texts):
    return tokenizer(texts["text"], truncation=True, padding="max_length", max_length=context_length)

tokenized_datasets = raw_datasets.map(tokenize_pad_and_truncate, batched=True)
```

Traceback:

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-1-ba61bb32f786> in <module>
     12     return tokenizer(texts["text"], truncation=True, padding="max_length", max_length=context_length)
     13
---> 14 tokenized_datasets = raw_datasets.map(tokenize_pad_and_truncate, batched=True)

~/git/datasets/src/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    500                     desc=desc,
    501                 )
--> 502                 for k, dataset in self.items()
    503             }
    504         )

~/git/datasets/src/datasets/dataset_dict.py in <dictcomp>(.0)
    500                     desc=desc,
    501                 )
--> 502                 for k, dataset in self.items()
    503             }
    504         )

~/git/datasets/src/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2051                 new_fingerprint=new_fingerprint,
   2052                 disable_tqdm=disable_tqdm,
-> 2053                 desc=desc,
   2054             )
   2055         else:

~/git/datasets/src/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    501             self: "Dataset" = kwargs.pop("self")
    502         # apply actual function
--> 503         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    504         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    505         for dataset in datasets:

~/git/datasets/src/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    468         }
    469         # apply actual function
--> 470         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    471         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    472         # re-apply format to the output

~/git/datasets/src/datasets/fingerprint.py in wrapper(*args, **kwargs)
    404             # Call actual function
    405
--> 406             out = func(self, *args, **kwargs)
    407
    408             # Update fingerprint of in-place transforms + update in-place history of transforms

~/git/datasets/src/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2243             if os.path.exists(cache_file_name) and load_from_cache_file:
   2244                 logger.warning("Loading cached processed dataset at %s", cache_file_name)
-> 2245                 info = self.info.copy()
   2246                 info.features = features
   2247                 info.task_templates = None

~/git/datasets/src/datasets/info.py in copy(self)
    278
    279     def copy(self) -> "DatasetInfo":
--> 280         return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
    281
    282

~/git/datasets/src/datasets/info.py in __init__(self, description, citation, homepage, license, features, post_processed, supervised_keys, task_templates, builder_name, config_name, version, splits, download_checksums, download_size, post_processing_size, dataset_size, size_in_bytes)

~/git/datasets/src/datasets/info.py in __post_init__(self)
    177                 for idx, template in enumerate(self.task_templates):
    178                     if isinstance(template, TextClassification):
--> 179                         labels = self.features[template.label_column].names
    180                         self.task_templates[idx] = TextClassification(
    181                             text_column=template.text_column, label_column=template.label_column, labels=labels

KeyError: 'label'
```


Remove _resampler from Audio fields

The `_resampler` Audio attribute was implemented to optimize audio resampling, but it should not be cached.

This PR removes `_resampler` from Audio fields, so that it is not returned by `fields()` or `asdict()`.

Fix #3083.


Fixes to `to_tf_dataset`

Hi ! Can you give some details about why you need these changes ?

Hey, sorry, I should have explained! I've been getting a lot of `VisibleDeprecationWarning` from Numpy, due to an issue in the formatter, see #3084 . This is a temporary workaround (since I'm using these methods in the upcoming course) until I can fix that issue, because I couldn't see an obvious fix for the Numpy formatter. If you can see a quick way to fix that, though, that might be even better!


VisibleDeprecationWarning when using `set_format("numpy")`

Something isn't working

I just opened a PR and I verified that the code you provided doesn't show any deprecation warning :)

Code to reproduce:

```
from datasets import load_dataset
dataset = load_dataset("glue", "mnli")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-cased')

def tokenize_function(dataset):
    return tokenizer(dataset['premise'])

tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=dataset['train'].features)

tokenized_datasets.set_format("numpy")

tokenized_datasets['train'][5:8]
```
Outputs:

```
python3.9/site-packages/datasets/formatting/formatting.py:167: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return np.array(array, copy=False, **self.np_array_kwargs)
```


Datasets with Audio feature raise error when loaded from cache due to _resampler parameter

Something isn't working

## Describe the bug
As reported by @patrickvonplaten, when loaded from the cache, datasets containing the Audio feature raise TypeError.

## Steps to reproduce the bug
```python
from datasets import load_dataset

# load first time works
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean")

# load from cache breaks
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean")
```

## Actual results
```
TypeError: __init__() got an unexpected keyword argument '_resampler'
```


Fix error related to huggingface_hub timeout parameter

The `huggingface_hub` package added the parameter `timeout` from version 0.0.19.

This PR bumps this minimal version.

Fix #3080.


[Audio datasets] Adapting all audio datasets

@lhoestq - are there other important speech datasets that I'm forgetting here?

Think PR is good to go otherwise

@lhoestq @albertvillanova - how can we make an exception for the AMI README so that the test doesn't fail? The dataset card definitely should have a data preprocessing section

Hi @patrickvonplaten ,

the data preprocessing section is not defined as a valid section in the readme validation file. After this line:
https://github.com/huggingface/datasets/blob/568db594d51110da9e23d224abded2a976b3c8c7/src/datasets/utils/resources/readme_structure.yaml#L20
feel free to insert (correctly indented of course):
```python
- name: "Dataset Preprocessing"
  allow_empty: true
  allow_empty_text: true
  subsections: null
```
and then the tests should pass.

Thanks a lot @albertvillanova - I've added the feature to all audio datasets and corrected the task of `covost2`

This PR adds the new `Audio(...)` features - see: https://github.com/huggingface/datasets/pull/2324 to the most important audio datasets:

- Librispeech
- Timit
- Common Voice
- AMI
- ... (others I'm forgetting now)

The PR is curently blocked because the following leads to a problem:

```python
from datasets import load_dataset

# load first time works
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean")

# load from cache breaks
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean")
```

As soon as it's unblocked, I'll adapt the other audio datasets as well.


Error related to timeout keyword argument

Something isn't working

## Describe the bug
As reported by @patrickvonplaten, a TypeError is raised when trying to load a dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean")
```

## Actual results
```
TypeError: dataset_info() got an unexpected keyword argument 'timeout'
```


Fix loading a metric with internal import

After refactoring the module factory (#2986), a bug was introduced when loading metrics with internal imports.

This PR adds a new test case and fixes this bug.

Fix #3076.

CC: @sgugger @merveenoyan


Error when loading a metric

Something isn't working

## Describe the bug
As reported by @sgugger, after last release, exception is thrown when loading a metric.

## Steps to reproduce the bug
```python
from datasets import load_metric

metric = load_metric("squad_v2")
```

## Actual results
```
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-1-e612a8cab787> in <module>
      1 from datasets import load_metric
----> 2 metric = load_metric("squad_v2")

d:\projects\huggingface\datasets\src\datasets\load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, revision, script_version, **metric_init_kwargs)
   1336         )
   1337         revision = script_version
-> 1338     metric_module = metric_module_factory(
   1339         path, revision=revision, download_config=download_config, download_mode=download_mode
   1340     ).module_path

d:\projects\huggingface\datasets\src\datasets\load.py in metric_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, **download_kwargs)
   1237                 if not isinstance(e1, FileNotFoundError):
   1238                     raise e1 from None
-> 1239                 raise FileNotFoundError(
   1240                     f"Couldn't find a metric script at {relative_to_absolute_path(combined_path)}. "
   1241                     f"Metric '{path}' doesn't exist on the Hugging Face Hub either."

FileNotFoundError: Couldn't find a metric script at D:\projects\huggingface\datasets\squad_v2\squad_v2.py. Metric 'squad_v2' doesn't exist on the Hugging Face Hub either.
```


Updates LexGLUE and MultiEURLEX README.md files

- Fix leaderboard in LexGLUE.
- Fix an error in the CaseHOLD data example.
- Turn MultiEURLEX dataset statistics table into HTML to nicely render in HF website.


add XCSR dataset

> Hi ! Thanks for adding this dataset :)
>
> Do you know how the translations were done ? Maybe we can mention that in the dataset card.
>
> The rest looks all good to me :) good job with the dataset script and the dataset card !
>
> Just one thing: we try to have dummy_data.zip files that are as small as possible, however here each zip file is 70KB+. It think we can make them even smaller if we remove unnecessary files in them. In particular in the `ar` dummy data zip file, we don't need the data for all languages, but rather only the `ar` files. Could you try to remove the unnecessary files in the dummy data zip files ?

Hi!

Thank you so much for reviewing this PR. I've updated the README to briefly mention the translations and added a link to the paper, where a detailed description of the translation procedure can be found in the appendix.

For the dummy_data.zip files, is it possible to keep all the current files? I tried to remove some of the files, but the removal led to a failure in the local testing. We also think it may be better to keep the current dummy_data.zip files because all the data are useful actually. Thanks a lot!!

Hi @lhoestq, just a gentle ping on this PR. :D

Hi,

I wanted to add the [XCSR ](https://inklab.usc.edu//XCSR/xcsr_datasets) dataset to huggingface! :)

I followed the instructions of adding new dataset to huggingface and have all the required files ready now! It would be super helpful if you can take a look and review them. Thanks in advance for your time and help. Look forward to hearing from you and can't wait to add XCSR to huggingface :D


Import error installing with ppc64le

Something isn't working

This seems to be an issue with importing PyArrow so I posted the problem [here](https://issues.apache.org/jira/browse/ARROW-14323), and I'm closing this issue.

## Describe the bug
Installing the datasets library with a computer running with ppc64le seems to cause an issue when importing the datasets library.

```
python
Python 3.6.13 | packaged by conda-forge | (default, Sep 23 2021, 07:37:44)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datasets
Illegal instruction (core dumped)
```

Error when importing
`Illegal instruction (core dumped)`

## Steps to reproduce the bug
I get this error when installing the library by using conda.  I can't install with pip I believe because pyarrow only has the ppc64le library on conda forge
```
conda create --name transformers_py36_v2 python=3.6
conda activate transformers_py36_v2
conda install datasets
```

## Tracebacks
conda create --name transformers_py36_v2 python=3.6

```
Collecting package metadata (current_repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda

## Package Plan ##

  environment location: /p/home/gerryc/.conda/envs/transformers_py36_v2

  added / updated specs:
    - python=3.6

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-ppc64le::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-ppc64le::_openmp_mutex-4.5-1_gnu
  ca-certificates    conda-forge/linux-ppc64le::ca-certificates-2021.10.8-h1084571_0
  certifi            pkgs/main/linux-ppc64le::certifi-2020.12.5-py36h6ffa863_0
  ld_impl_linux-ppc~ conda-forge/linux-ppc64le::ld_impl_linux-ppc64le-2.36.1-ha35d02b_2
  libffi             conda-forge/linux-ppc64le::libffi-3.4.2-h3b9df90_4
  libgcc-ng          conda-forge/linux-ppc64le::libgcc-ng-11.2.0-h7698a5e_11
  libgomp            conda-forge/linux-ppc64le::libgomp-11.2.0-h7698a5e_11
  libstdcxx-ng       conda-forge/linux-ppc64le::libstdcxx-ng-11.2.0-habdf983_11
  libzlib            conda-forge/linux-ppc64le::libzlib-1.2.11-h339bb43_1013
  ncurses            conda-forge/linux-ppc64le::ncurses-6.2-hea85c5d_4
  openssl            conda-forge/linux-ppc64le::openssl-1.1.1l-h4e0d66e_0
  pip                conda-forge/noarch::pip-21.3-pyhd8ed1ab_0
  python             conda-forge/linux-ppc64le::python-3.6.13-h57873ef_2_cpython
  readline           conda-forge/linux-ppc64le::readline-8.1-h5c45dff_0
  setuptools         pkgs/main/linux-ppc64le::setuptools-58.0.4-py36h6ffa863_0
  sqlite             conda-forge/linux-ppc64le::sqlite-3.36.0-h4e2196e_2
  tk                 conda-forge/linux-ppc64le::tk-8.6.11-h41c6715_1
  wheel              conda-forge/noarch::wheel-0.37.0-pyhd8ed1ab_1
  xz                 conda-forge/linux-ppc64le::xz-5.2.5-h6eb9509_1
  zlib               conda-forge/linux-ppc64le::zlib-1.2.11-h339bb43_1013

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate transformers_py36_v2
#
# To deactivate an active environment, use
#
#     $ conda deactivate
```

conda activate transformers_py36_v2
conda install datasets
```
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda

## Package Plan ##

  environment location: /p/home/gerryc/.conda/envs/transformers_py36_v2

  added / updated specs:
    - datasets

The following NEW packages will be INSTALLED:

  abseil-cpp         conda-forge/linux-ppc64le::abseil-cpp-20210324.2-h3b9df90_0
  aiohttp            conda-forge/linux-ppc64le::aiohttp-3.7.4.post0-py36hc33305d_0
  arrow-cpp          conda-forge/linux-ppc64le::arrow-cpp-5.0.0-py36hf9cf308_8_cpu
  async-timeout      conda-forge/noarch::async-timeout-3.0.1-py_1000
  attrs              conda-forge/noarch::attrs-21.2.0-pyhd8ed1ab_0
  aws-c-cal          conda-forge/linux-ppc64le::aws-c-cal-0.5.11-hb3fac3d_0
  aws-c-common       conda-forge/linux-ppc64le::aws-c-common-0.6.2-h4e0d66e_0
  aws-c-event-stream conda-forge/linux-ppc64le::aws-c-event-stream-0.2.7-h76da5f2_13
  aws-c-io           conda-forge/linux-ppc64le::aws-c-io-0.10.5-hf6a6c7c_0
  aws-checksums      conda-forge/linux-ppc64le::aws-checksums-0.1.11-hfe76d68_7
  aws-sdk-cpp        conda-forge/linux-ppc64le::aws-sdk-cpp-1.8.186-h90855e8_3
  brotlipy           conda-forge/linux-ppc64le::brotlipy-0.7.0-py36hc33305d_1001
  bzip2              conda-forge/linux-ppc64le::bzip2-1.0.8-h4e0d66e_4
  c-ares             conda-forge/linux-ppc64le::c-ares-1.17.2-h4e0d66e_0
  cffi               conda-forge/linux-ppc64le::cffi-1.14.6-py36h021ab3c_1
  chardet            conda-forge/linux-ppc64le::chardet-4.0.0-py36h270354c_1
  colorama           conda-forge/noarch::colorama-0.4.4-pyh9f0ad1d_0
  cryptography       conda-forge/linux-ppc64le::cryptography-3.4.7-py36hc71b123_0
  dataclasses        conda-forge/noarch::dataclasses-0.8-pyh787bdff_2
  datasets           conda-forge/noarch::datasets-1.12.1-pyhd8ed1ab_1
  dill               conda-forge/noarch::dill-0.3.4-pyhd8ed1ab_0
  filelock           conda-forge/noarch::filelock-3.3.0-pyhd8ed1ab_0
  fsspec             conda-forge/noarch::fsspec-2021.10.0-pyhd8ed1ab_0
  gflags             conda-forge/linux-ppc64le::gflags-2.2.2-hb209c28_1004
  glog               conda-forge/linux-ppc64le::glog-0.5.0-h4040248_0
  grpc-cpp           conda-forge/linux-ppc64le::grpc-cpp-1.40.0-h2bf711c_2
  huggingface_hub    conda-forge/noarch::huggingface_hub-0.0.19-pyhd8ed1ab_0
  idna               conda-forge/noarch::idna-2.10-pyh9f0ad1d_0
  idna_ssl           conda-forge/noarch::idna_ssl-1.0.0-0
  importlib-metadata conda-forge/linux-ppc64le::importlib-metadata-4.8.1-py36h270354c_0
  importlib_metadata conda-forge/noarch::importlib_metadata-4.8.1-hd8ed1ab_0
  krb5               conda-forge/linux-ppc64le::krb5-1.19.2-haf43566_2
  libblas            conda-forge/linux-ppc64le::libblas-3.9.0-11_linuxppc64le_openblas
  libbrotlicommon    conda-forge/linux-ppc64le::libbrotlicommon-1.0.9-h4e0d66e_5
  libbrotlidec       conda-forge/linux-ppc64le::libbrotlidec-1.0.9-h4e0d66e_5
  libbrotlienc       conda-forge/linux-ppc64le::libbrotlienc-1.0.9-h4e0d66e_5
  libcblas           conda-forge/linux-ppc64le::libcblas-3.9.0-11_linuxppc64le_openblas
  libcurl            conda-forge/linux-ppc64le::libcurl-7.79.1-he415e40_1
  libedit            conda-forge/linux-ppc64le::libedit-3.1.20191231-h41a240f_2
  libev              conda-forge/linux-ppc64le::libev-4.33-h6eb9509_1
  libevent           conda-forge/linux-ppc64le::libevent-2.1.10-h97db324_4
  libgfortran-ng     conda-forge/linux-ppc64le::libgfortran-ng-11.2.0-hfdc3801_11
  libgfortran5       conda-forge/linux-ppc64le::libgfortran5-11.2.0-he58fbb4_11
  liblapack          conda-forge/linux-ppc64le::liblapack-3.9.0-11_linuxppc64le_openblas
  libnghttp2         conda-forge/linux-ppc64le::libnghttp2-1.43.0-h42039ad_1
  libopenblas        conda-forge/linux-ppc64le::libopenblas-0.3.17-pthreads_h486567c_1
  libprotobuf        conda-forge/linux-ppc64le::libprotobuf-3.18.1-h690f14c_0
  libssh2            conda-forge/linux-ppc64le::libssh2-1.10.0-ha5a9321_2
  libthrift          conda-forge/linux-ppc64le::libthrift-0.15.0-h54f692e_1
  libutf8proc        conda-forge/linux-ppc64le::libutf8proc-2.6.1-h4e0d66e_0
  lz4-c              conda-forge/linux-ppc64le::lz4-c-1.9.3-h3b9df90_1
  multidict          conda-forge/linux-ppc64le::multidict-5.2.0-py36hc33305d_0
  multiprocess       conda-forge/linux-ppc64le::multiprocess-0.70.12.2-py36hc33305d_0
  numpy              conda-forge/linux-ppc64le::numpy-1.19.5-py36h86665d4_1
  orc                conda-forge/linux-ppc64le::orc-1.7.0-hae6b4bd_0
  packaging          conda-forge/noarch::packaging-21.0-pyhd8ed1ab_0
  pandas             conda-forge/linux-ppc64le::pandas-1.1.5-py36hab1a6e6_0
  parquet-cpp        conda-forge/noarch::parquet-cpp-1.5.1-2
  pyarrow            conda-forge/linux-ppc64le::pyarrow-5.0.0-py36h7a46c7e_8_cpu
  pycparser          conda-forge/noarch::pycparser-2.20-pyh9f0ad1d_2
  pyopenssl          conda-forge/noarch::pyopenssl-21.0.0-pyhd8ed1ab_0
  pyparsing          conda-forge/noarch::pyparsing-2.4.7-pyh9f0ad1d_0
  pysocks            conda-forge/linux-ppc64le::pysocks-1.7.1-py36h270354c_3
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.2-pyhd8ed1ab_0
  python-xxhash      conda-forge/linux-ppc64le::python-xxhash-2.0.2-py36hc33305d_0
  python_abi         conda-forge/linux-ppc64le::python_abi-3.6-2_cp36m
  pytz               conda-forge/noarch::pytz-2021.3-pyhd8ed1ab_0
  pyyaml             conda-forge/linux-ppc64le::pyyaml-5.4.1-py36hc33305d_1
  re2                conda-forge/linux-ppc64le::re2-2021.09.01-h3b9df90_0
  requests           conda-forge/noarch::requests-2.25.1-pyhd3deb0d_0
  s2n                conda-forge/linux-ppc64le::s2n-1.0.10-h97db324_0
  six                conda-forge/noarch::six-1.16.0-pyh6c4a22f_0
  snappy             conda-forge/linux-ppc64le::snappy-1.1.8-hb209c28_3
  tqdm               conda-forge/noarch::tqdm-4.62.3-pyhd8ed1ab_0
  typing-extensions  conda-forge/noarch::typing-extensions-3.10.0.2-hd8ed1ab_0
  typing_extensions  conda-forge/noarch::typing_extensions-3.10.0.2-pyha770c72_0
  urllib3            conda-forge/noarch::urllib3-1.26.7-pyhd8ed1ab_0
  xxhash             conda-forge/linux-ppc64le::xxhash-0.8.0-h4e0d66e_3
  yaml               conda-forge/linux-ppc64le::yaml-0.2.5-h6eb9509_0
  yarl               conda-forge/linux-ppc64le::yarl-1.6.3-py36hc33305d_2
  zipp               conda-forge/noarch::zipp-3.6.0-pyhd8ed1ab_0
  zstd               conda-forge/linux-ppc64le::zstd-1.5.0-h65c4b1a_0

The following packages will be UPDATED:

  certifi            pkgs/main::certifi-2020.12.5-py36h6ff~ --> conda-forge::certifi-2021.5.30-py36h270354c_0

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Red Hat Enterprise Linux 8.2 (Ootpa)
- Python version: 3.6
- PyArrow version: pyarrow - 5.0.0 - py36h7a46c7e_8_cpu - conda-forge

Any help would be appreciated! I've been struggling on installing datasets on this machine.


Fix pathlib patches for streaming

Fix issue https://github.com/huggingface/datasets/issues/2866 (for good this time)

`counter` now works in both streaming and non-streaming mode.
And the `AttributeError: 'str' object has no attribute 'as_posix'` related to the patch of Path.open is fixed as well

Note : the patches should only affect the datasets module, not the user's ones ! That's why we should probably use something else than patch.object to patch the Path class' methods.

cc @severo @albertvillanova


Custom plain text dataset, plain json dataset and plain csv dataset are remove from datasets template folder

dataset request

Requesting to add a new dataset

Hi @zixiliuUSC,

As explained in the documentation (https://huggingface.co/docs/datasets/loading.html#json), we support loading any dataset in JSON (as well as CSV, text, Parquet) format:
```python
ds = load_dataset('json', data_files='my_file.json')
```

## Adding a Dataset
- **Name:** text, json, csv
- **Description:** I am developing a customized dataset loading script. The problem is  mainly about my custom dataset is seperate into many files and I only find a dataset loading template in [https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py](https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py) that can handle my circumstance. I'm afraid these templates are too old to use. Could you re-add these three templates to current master branch?


Fix Windows CI with FileNotFoundError when stting up s3_base fixture

Thanks ! Sorry for the inconvenience ^^'

Fix #3069.


bug

Something isn't working

## Describe the bug
After commit 9353fc863d0c99ab0427f83cc5a4f04fcf52f1df, the CI fails on Windows  with FileNotFoundError when stting up s3_base fixture. See: https://app.circleci.com/pipelines/github/huggingface/datasets/8151/workflows/5db8d154-badd-4d3d-b202-ca7a318997a2/jobs/50321

Error summary:
```
ERROR tests/test_arrow_dataset.py::test_dummy_dataset_serialize_s3 - FileNotF...
ERROR tests/test_dataset_dict.py::test_dummy_dataset_serialize_s3 - FileNotFo...
```

Stack trace:
```
______________ ERROR at setup of test_dummy_dataset_serialize_s3 ______________
[gw0] win32 -- Python 3.6.8 C:\tools\miniconda3\python.exe

    @pytest.fixture()
    def s3_base():
        # writable local S3 system
        import shlex
        import subprocess

        # Mocked AWS Credentials for moto.
        old_environ = os.environ.copy()
        os.environ.update(S3_FAKE_ENV_VARS)

>       proc = subprocess.Popen(shlex.split("moto_server s3 -p %s" % s3_port))

tests\s3_fixtures.py:32:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\tools\miniconda3\lib\subprocess.py:729: in __init__
    restore_signals, start_new_session)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <subprocess.Popen object at 0x0000012BB8A4B908>
args = 'moto_server s3 -p 5555', executable = None, preexec_fn = None
close_fds = True, pass_fds = (), cwd = None, env = None
startupinfo = <subprocess.STARTUPINFO object at 0x0000012BB8177630>
creationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = -1
c2pwrite = -1, errread = -1, errwrite = -1, unused_restore_signals = True
unused_start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       unused_restore_signals, unused_start_new_session):
        """Execute program (MS Windows version)"""

        assert not pass_fds, "pass_fds not supported on Windows."

        if not isinstance(args, str):
            args = list2cmdline(args)

        # Process startup details
        if startupinfo is None:
            startupinfo = STARTUPINFO()
        if -1 not in (p2cread, c2pwrite, errwrite):
            startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES
            startupinfo.hStdInput = p2cread
            startupinfo.hStdOutput = c2pwrite
            startupinfo.hStdError = errwrite

        if shell:
            startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
            startupinfo.wShowWindow = _winapi.SW_HIDE
            comspec = os.environ.get("COMSPEC", "cmd.exe")
            args = '{} /c "{}"'.format (comspec, args)

        # Start the process
        try:
            hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                                     # no special security
                                     None, None,
                                     int(not close_fds),
                                     creationflags,
                                     env,
                                     os.fspath(cwd) if cwd is not None else None,
>                                    startupinfo)
E                                    FileNotFoundError: [WinError 2] The system cannot find the file specified

C:\tools\miniconda3\lib\subprocess.py:1017: FileNotFoundError
```


feat: increase streaming retry config

@lhoestq I had 2 runs for more than 2 days each, continuously streaming (they were failing before with 3 retries at 1 sec interval).

They are running on TPU's (so great internet connection) and only had connection errors a few times each (3 & 4). Each time it worked after only 1 retry.
The reason for a higher number of retries is for local connections. It would allow for almost 2mn of a wifi/ethernet disconnection. In practice this should not happen very often.

Let me know if you think it's too much.

Increase streaming config parameters:
* retry interval set to 5 seconds
* max retries set to 20 (so 1mn 40s)


add story_cloze

Thanks for pushing this dataset :)

According to the CI, the file `cloze_test_val__spring2016 - cloze_test_ALL_val.csv` is missing in the dummy data zip file (the zip files seem empty). Feel free to add this file with 4-5 lines and it should be good

And you can fix the YAML tags with
```yaml
pretty_name: Story Cloze Test
```
and filling the other tags task_categories and task_ids

If the dataset doesn exist on paperswithcode, you can just leave
```yaml
paperswithcode_id: null
```

> Thanks @zaidalyafeai, the failing test is due to an issue in the master branch, that has already been fixed.
>
> You can include the fix:
>
> ```
> git checkout add_story_cloze
> git fetch upstream master
> git merge upstream/master
> ```

Thanks @albertvillanova, passed all the tests now.

Thanks Albert, I fixed the suggested comments. This dataset has no train splits, it is only used for evaluation.


Add iter_archive

Added the `iter_archive` method for the StreamingDownloadManager.

It was already implemented in the regular DownloadManager.
Now it can be used to stream from TAR archives as mentioned in https://github.com/huggingface/datasets/issues/2829

I also updated the `food101` dataset as an example.
Any image/audio dataset using TAR archives can be updated to use `iter_archive` in order to be streamable :)

cc @severo


Fix test command after refac

Fix the `datasets-cli` test command after the `prepare_module` change in #2986


Make `interleave_datasets` more robust

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Right now there are few hiccups using `interleave_datasets`. Interleaved dataset iterates until the smallest dataset completes it's iterator. In this way larger datasets may not complete full epoch of iteration.
It creates new problems in calculation of epoch since there are no way to track which dataset from `interleave_datasets` completes how many epoch.

**Describe the solution you'd like**
For `interleave_datasets` module,
- [ ] Add a boolean argument `--stop-iter` in `interleave_datasets` that enables dataset to either iterate infinite amount of time or not. That means it should not return `StopIterator` exception in case `--stop-iter=False`.
- [ ] Internal list variable `iter_cnt` that explains how many times (in steps/epochs) each dataset iterates at a given point.
- [ ] Add an argument `--max-iter` (list type) that explain maximum amount of time each of the dataset can iterate. After complete  `--max-iter` of one dataset, other dataset should continue sampling and when all the dataset finish their respective `--max-iter`, only then return `StopIterator`

Note: I'm new to `datasets` api. May be these features are already there in the datasets.

Since multitask training is the latest trends, I believe this feature would make the `datasets` api more popular.

@lhoestq


Windows CI is unable to test streaming properly because of SSL issues

I think this problem is already fixed:
```python
In [4]: import fsspec
   ...:
   ...: url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"
   ...:
   ...: fsspec.open(url).open()
Out[4]: <File-like object HTTPFileSystem, https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattribu
```

No I'm still having this issue on my windows, and so does the CI

In https://github.com/huggingface/datasets/pull/3041 the windows tests were skipped because of SSL issues with moon-staging.huggingface.co:443

The issue appears only on windows with asyncio. On Linux it works. With requests it works as well. And with the production environment huggingface.co it also works.

to reproduce on windows:
```python
import fsspec

# use any URL to a file in a dataset repo
url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"

fsspec.open(url).open()
```
raises
```python
FileNotFoundError: https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes
```
because of
```python
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host moon-staging.huggingface.co:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')]
```


Update summary on PyPi beyond NLP

More than just NLP now


Feature request : add leave=True to dataset.map to enable tqdm nested bars (and whilst we're at it couldn't we get a way to access directly tqdm underneath?)

New feature or request

@lhoestq, @albertvillanova can we have `**tqdm_kwargs` in `map`? If there are any fields that are important to our tqdm (like iterable or unit), we can pop them before initialising the tqdm object so as to avoid duplicity.

Hi ! Sounds like a good idea :)

Also I think it would be better to have this as an actual parameters instead of kwargs to make it clearer

**A clear and concise description of what you want to happen.**

It would be so nice to be able to nest HuggingFace `Datasets.map() ` progress bars in the grander scheme of things and whilst we're at it why not other functions.

**Describe alternatives you've considered**

By the way is there not a way to directly interact with underlying tqdm module ? **kwargs-ish?

**Additional context**

Furthering tqdm integration #2374 and huggingface/transformers#11797 solutioned by huggingface/transformers#12226 provided with tqdm description as `desc=`

@sgugger @bhavitvyamalik


load_dataset('openwebtext') yields "Compressed file ended before the end-of-stream marker was reached"

bug

Something isn't working

Hi @RylanSchaeffer, thanks for reporting.

I'm sorry, but I was not able to reproduce your problem.

Normally, the reason for this type of error is that, during your download of the data files, this was not fully complete.

Could you please try to load the dataset again but forcing its redownload? Please use:
```python
dataset = load_dataset("openwebtext", download_mode="FORCE_REDOWNLOAD")
```

Let me know if the problem persists.

I close this issue for the moment. Feel free to re-open it again if the problem persists.

## Describe the bug
When I try `load_dataset('openwebtext')`, I receive a "EOFError: Compressed file ended before the end-of-stream marker was reached" error.

## Steps to reproduce the bug
```
from datasets import load_dataset
dataset = load_dataset('openwebtext')
```

## Expected results
I expect the `dataset` variable to be properly constructed.

## Actual results

```
File "/home/rschaef/CoCoSci-Language-Distillation/distillation_v2/ratchet_learning/tasks/base.py", line 37, in create_dataset
    dataset_str,
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/load.py", line 1117, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/builder.py", line 637, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/builder.py", line 704, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/rschaef/.cache/huggingface/modules/datasets_modules/datasets/openwebtext/85b3ae7051d2d72e7c5fdf6dfb462603aaa26e9ed506202bf3a24d261c6c40a1/openwebtext.py", line 61, in _split_generators
    dl_dir = dl_manager.download_and_extract(_URL)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/download_manager.py", line 261, in extract
    partial(cached_path, download_config=download_config), path_or_paths, num_proc=num_proc, disable_tqdm=False
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 197, in map_nested
    return function(data_struct)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 316, in cached_path
    output_path, force_extract=download_config.force_extract
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/extract.py", line 40, in extract
    self.extractor.extract(input_path, output_path, extractor=extractor)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/extract.py", line 179, in extract
    return extractor.extract(input_path, output_path)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/extract.py", line 53, in extract
    tar_file.extractall(output_path)
  File "/usr/lib/python3.6/tarfile.py", line 2010, in extractall
    numeric_owner=numeric_owner)
  File "/usr/lib/python3.6/tarfile.py", line 2052, in extract
    numeric_owner=numeric_owner)
  File "/usr/lib/python3.6/tarfile.py", line 2122, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/usr/lib/python3.6/tarfile.py", line 2171, in makefile
    copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
  File "/usr/lib/python3.6/tarfile.py", line 249, in copyfileobj
    buf = src.read(bufsize)
  File "/usr/lib/python3.6/lzma.py", line 200, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.6/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/usr/lib/python3.6/_compression.py", line 99, in read
    raise EOFError("Compressed file ended before the "
python-BaseException
EOFError: Compressed file ended before the end-of-stream marker was reached
```

## Environment info
- `datasets` version: 1.12.1
- Platform: Linux-4.4.0-173-generic-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.10
- PyArrow version: 5.0.0


Fix task reloading from cache

When reloading a dataset from the cache when doing `map`, the tasks templates were kept instead of being updated regarding the output of the `map` function. This is an issue because we drop the tasks templates that are not compatible anymore after `map`, for example if a column of the template was removed.

This PR fixes this and for convenience introduces a decorator `@transmit_tasks` that takes care of doing this verification, similar to the `@transmit_format` decorator.

This should fix issue https://github.com/huggingface/datasets/issues/3047 cc @sgugger


Dataset wikipedia and Bookcorpusopen cannot be fetched from dataloader.

Something isn't working

Hi ! I think this issue is more related to the `transformers` project. Could you open an issue on https://github.com/huggingface/transformers ?

Anyway I think the issue could be that both wikipedia and bookcorpusopen have an additional "title" column, contrary to wikitext which only has a "text" column. After calling `load_dataset`, can you try doing `dataset = dataset.remove_columns("title")` ?

Removing the "title" column works! Thanks for your advice.

Maybe I should still create an issue to `transformers' to mark this solution?

## Describe the bug
I have used the previous version of `transformers` and `datasets`. The dataset `wikipedia` can be successfully used. Recently, I upgrade them to the newest version and find it raises errors. I also tried other datasets. The `wikitext` works and the `bookcorpusopen` raises the same errors as `wikipedia`.

## Steps to reproduce the bug
Run the `run_mlm_no_trainer.py` and the given script on this [link](https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling). Change the dataset from wikitext to wikipedia or bookcorpusopen. BTW, the library transformers is of version 4.11.3.

## Expected results
The data batchs are fetched from the data loader and train.

## Actual results
The first time to fetch data batch occurs error.
`Traceback (most recent call last):
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 705, in convert_to_tensors
    tensor = as_tensor(value)
ValueError: too many dimensions 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/original_run_mlm_no_trainer.py", line 528, in <module>
    main()
  File "src/original_run_mlm_no_trainer.py", line 488, in main
    for step, batch in enumerate(train_dataloader):
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/accelerate/data_loader.py", line 303, in __iter__
    for batch in super().__iter__():
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/data/data_collator.py", line 41, in __call__
    return self.torch_call(features)
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/data/data_collator.py", line 671, in torch_call
    batch = self.tokenizer.pad(examples, return_tensors="pt", pad_to_multiple_of=self.pad_to_multiple_of)
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2774, in pad
    return BatchEncoding(batch_outputs, tensor_type=return_tensors)
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 210, in __init__
    self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 722, in convert_to_tensors
    "Unable to create tensor, you should probably activate truncation and/or padding "
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length.
`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Linux-5.8.0-59-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.6
- PyArrow version: 5.0.0


Error in per class precision computation

Something isn't working

Hi @tidhamecha2, thanks for reporting.

Indeed, we fixed this issue just one week ago: #3008

The fix will be included in our next version release.

In the meantime, you can incorporate the fix by installing `datasets` from the master branch:
```
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```

## Describe the bug
When trying to get the per class precision values by providing `average=None`, following error is thrown `ValueError: can only convert an array of size 1 to a Python scalar`

## Steps to reproduce the bug
```python
from datasets import load_dataset, load_metric
precision_metric = load_metric("precision")
predictions = [0, 2, 1, 0, 0, 1]
references = [0, 1, 2, 0, 1, 2]
results = precision_metric.compute(predictions=predictions, references=references, average=None)
```

## Expected results
`  {'precision': array([0.66666667, 0.        , 0.        ])}`
as per https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py

## Actual results
```
   output = self._compute(predictions=predictions, references=references, **kwargs)
  File "~/.cache/huggingface/modules/datasets_modules/metrics/precision/94709a71c6fe37171ef49d3466fec24dee9a79846c9f176dff66a649e9811690/precision.py", line 110, in _compute
    sample_weight=sample_weight,
ValueError: can only convert an array of size 1 to a Python scalar
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: linux
- Python version: 3.6.9
- PyArrow version: 5.0.0


Fix meteor metric for version >= 3.6.4

After `nltk` update, the meteor metric expects pre-tokenized inputs (breaking change).

This PR fixes this issue, while maintaining compatibility with older versions.


CI test suite fails after meteor metric update

Something isn't working

## Describe the bug
CI test suite fails: https://app.circleci.com/pipelines/github/huggingface/datasets/8110/workflows/f059ba43-9154-4632-bebb-82318447ddc9/jobs/50010

Stack trace:
```
___________________ LocalMetricTest.test_load_metric_meteor ____________________
[gw1] linux -- Python 3.6.15 /home/circleci/.pyenv/versions/3.6.15/bin/python3.6

self = <tests.test_metric_common.LocalMetricTest testMethod=test_load_metric_meteor>
metric_name = 'meteor'

    def test_load_metric(self, metric_name):
        doctest.ELLIPSIS_MARKER = "[...]"
        metric_module = importlib.import_module(datasets.load.prepare_module(os.path.join("metrics", metric_name))[0])
        metric = datasets.load.import_main_class(metric_module.__name__, dataset=False)
        # check parameters
        parameters = inspect.signature(metric._compute).parameters
        self.assertTrue("predictions" in parameters)
        self.assertTrue("references" in parameters)
        self.assertTrue(all([p.kind != p.VAR_KEYWORD for p in parameters.values()]))  # no **kwargs
        # run doctest
        with self.patch_intensive_calls(metric_name, metric_module.__name__):
            with self.use_local_metrics():
>               results = doctest.testmod(metric_module, verbose=True, raise_on_error=True)

tests/test_metric_common.py:75:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../.pyenv/versions/3.6.15/lib/python3.6/doctest.py:1951: in testmod
    runner.run(test)
../.pyenv/versions/3.6.15/lib/python3.6/doctest.py:1839: in run
    r = DocTestRunner.run(self, test, compileflags, out, False)
../.pyenv/versions/3.6.15/lib/python3.6/doctest.py:1476: in run
    return self.__run(test, compileflags, out)
../.pyenv/versions/3.6.15/lib/python3.6/doctest.py:1382: in __run
    exception)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <doctest.DebugRunner object at 0x7f4c26bd3da0>
out = <built-in method write of _io.TextIOWrapper object at 0x7f51a21852d0>
test = <DocTest datasets_modules.datasets.meteor.6201bb45d2c144ea7963680949d20f523d74a741fa0f8a806f836e6caa5245d7.meteor.Mete...ets_modules/datasets/meteor/6201bb45d2c144ea7963680949d20f523d74a741fa0f8a806f836e6caa5245d7/meteor.py:87 (5 examples)>
example = <doctest.Example object at 0x7f4c26bd3eb8>
exc_info = (<class 'TypeError'>, TypeError('"hypothesis" expects pre-tokenized hypothesis (Iterable[str]): It is a guide to action which ensures that the military always obeys the commands of the party',), <traceback object at 0x7f4cd01afec8>)

    def report_unexpected_exception(self, out, test, example, exc_info):
>       raise UnexpectedException(test, example, exc_info)
E       doctest.UnexpectedException: <DocTest datasets_modules.datasets.meteor.6201bb45d2c144ea7963680949d20f523d74a741fa0f8a806f836e6caa5245d7.meteor.Meteor from /tmp/pytest-of-circleci/pytest-0/popen-gw1/cache/modules/datasets_modules/datasets/meteor/6201bb45d2c144ea7963680949d20f523d74a741fa0f8a806f836e6caa5245d7/meteor.py:87 (5 examples)>

../.pyenv/versions/3.6.15/lib/python3.6/doctest.py:1845: UnexpectedException
```


Update Biosses

Fix variable naming


load_dataset('the_pile_openwebtext2') produces ArrowInvalid, value too large to fit in C integer type

Something isn't working

I encountered the same bug using different datasets.
any suggestions?

## Describe the bug
When loading `the_pile_openwebtext2`, we get the error `pyarrow.lib.ArrowInvalid: Value 2111 too large to fit in C integer type`

## Steps to reproduce the bug
```python
import datasets
ds = datasets.load_dataset('the_pile_openwebtext2')
```

## Expected results
Should download the dataset, convert it to an arrow file, and return a working Dataset object.

## Actual results
The download works, but conversion to the arrow file fails as follows:

```
>>> ds = datasets.load_dataset('the_pile_openwebtext2')
Downloading and preparing dataset openwebtext2/plain_text (download: 27.33 GiB, generated: 63.86 GiB
, post-processed: Unknown size, total: 91.19 GiB) to /home/davidbau/.cache/huggingface/datasets/open
webtext2/plain_text/1.0.0/c48ec73ba3483bac673463f48f67e9a4fd8cb49a9d6ec4fb957f0b424b97cf25...
Traceback (most recent call last):
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/builder.py", line 1133,
 in _prepare_split
    writer.write(example, key)
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
366, in write
    self.write_examples_on_file()
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
311, in write_examples_on_file
    pa_array = pa.array(typed_sequence)
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
115, in __arrow_array__
    out = pa.array(cast_to_python_objects(self.data, only_1d_for_numpy=True), type=type)
  File "pyarrow/array.pxi", line 305, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Value 2111 too large to fit in C integer type
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
```

- Platform: Ubuntu 20.04
- Python version: python 3.9
- PyArrow version: 3.0.0


load_dataset cannot download the data and hangs on forever if cache dir specified

Something isn't working

Issue was environment inconsistency, updating packages did the trick

`conda install -c huggingface -c conda-forge datasets`

> Collecting package metadata (current_repodata.json): done
> Solving environment: |
> The environment is inconsistent, please check the package plan carefully
> The following packages are causing the inconsistency:
>
>   - conda-forge/noarch::datasets==1.12.1=pyhd8ed1ab_1
>   - conda-forge/win-64::multiprocess==0.70.12.2=py38h294d835_0
> done
>
> Package Plan
>
>   environment location: C:\xxx\anaconda3\envs\UnBias-94-1
>
>   added / updated specs:
>     - datasets
>
>
> The following NEW packages will be INSTALLED:
>
>   dill               conda-forge/noarch::dill-0.3.4-pyhd8ed1ab_0
>
> The following packages will be UPDATED:
>
>   ca-certificates    pkgs/main::ca-certificates-2021.9.30-~ --> conda-forge::ca-certificates-2021.10.8-h5b45459_0
>   certifi            pkgs/main::certifi-2021.5.30-py38haa9~ --> conda-forge::certifi-2021.10.8-py38haa244fe_0
>
> The following packages will be SUPERSEDED by a higher-priority channel:
>

## Describe the bug
After updating datasets, a code that ran just fine for ages began to fail. Specifying _datasets.load_dataset_'s _cache_dir_ optional argument on Windows 10 machine results in data download to hang on forever. Same call without cache_dir works just fine. Surprisingly exact same code just runs perfectly fine on Linux docker instance running in cloud.

Unfortunately I updated Windows also at the same time and I can't remember which version of datasets was running in my conda environment prior to the update otherwise I would have tried both to check this out.  :(

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```
cache_dir = 'c:/data/datasets'
dataset = load_dataset('wikipedia', '20200501.en', split='train',cache_dir=cache_dir)
```
Note that exact same code without specifying _cache_dir_ argument works perfectly fine.
```
cache_dir = 'c:/data/datasets'
dataset = load_dataset('wikipedia', '20200501.en', split='train')
```

## Expected results
Downloads the dataset and cache is handled in the _cache_dir_ directory
## Actual results
Data download keeps hanging on forever, **NO TRACEBACK**!

## Environment info
- `datasets` version: 1.12.1
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.8.11
- PyArrow version: 3.0.0


Non-Matching Checksum Error with crd3 dataset

Something isn't working

I got the same error for another dataset (`multi_woz_v22`):

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/dialog_acts.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/test/dialogues_001.json']
```

I'm seeing the same issue as @RylanSchaeffer:
Python 3.7.11, macOs 11.4
datasets==1.14.0

fails on:
```python
dataset = datasets.load_dataset("multi_woz_v22")
```

## Describe the bug
When I try loading the crd3 dataset (https://huggingface.co/datasets/crd3), an error is thrown.

## Steps to reproduce the bug
```python
dataset = load_dataset('crd3', split='train')
```

## Expected results
I expect no error to be thrown.

## Actual results
A non-matching checksum error is thrown.

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/RevanthRameshkumar/CRD3/archive/master.zip']
```

## Environment info

- `datasets` version: 1.12.1
- Platform: Linux-4.4.0-173-generic-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.10
- PyArrow version: 5.0.0


Fix streaming: catch Timeout error

I'm running a large test.
Let's see if I get any error within a few days.

This time it stopped after 8h but correctly raised `ConnectionError: Server Disconnected`.

Traceback:
```
Traceback (most recent call last):
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 1027, in <module>
    main()
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 991, in main
    for batch in tqdm(
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 376, in data_loader_streaming
    for item in dataset:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 179, in __iter__
    key_examples_list = [(key, example)] + [
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 179, in <listcomp>
    key_examples_list = [(key, example)] + [
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 176, in __iter__
    for key, example in iterator:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 225, in __iter__
    for x in self.ex_iterable:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 99, in __iter__
    for key, example in self.generate_examples_fn(**kwargs_with_shuffled_shards):
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 287, in wrapper
    for key, table in generate_tables_fn(**kwargs):
  File "/home/koush/datasets/src/datasets/packaged_modules/json/json.py", line 107, in _generate_tables
    batch = f.read(self.config.chunksize)
  File "/home/koush/datasets/src/datasets/utils/streaming_download_manager.py", line 136, in read_with_retries
    raise ConnectionError("Server Disconnected")
ConnectionError: Server Disconnected
```

Right before this error, the warnings were correctly raised:

```
10/10/2021 06:02:26 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 1sec [1/3]
10/10/2021 06:02:27 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 1sec [2/3]
10/10/2021 06:02:28 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 1sec [3/3
```

I'm going to see what happens if I change the max retries to 20 and the interval to 5.

Also maybe we can raise the Server Disconnected error with more info about what kind of error caused it (client error, time out, etc.)

I have 2 runs:
* [run 1](https://wandb.ai/dalle-mini/dalle-mini/runs/1nj161cl?workspace=user-borisd13) with [this data](https://huggingface.co/datasets/dalle-mini/encoded) that I will remove soon because I now use the 2nd one
* [run 2](https://wandb.ai/dalle-mini/dalle-mini/runs/he9rrc3q?workspace=user-borisd13) with [this data](https://huggingface.co/datasets/dalle-mini/encoded-vqgan_imagenet_f16_16384)
* `load_dataset(dataset_repo, data_files={'train':'data/train/*.jsonl', 'validation':'data/valid/*.jsonl'}, streaming=True)`

They have now been running by a bit more than a day for one run and 15h for the other.

The error logs are not shown in wandb because the script use `pylogging` (not sure why, I should change it) but basically so far with the new settings I had one timeout in each with successful reconnect afterwards.

So I think it's a good idea to have:
* `STREAMING_READ_RETRY_INTERVAL = 5` since before my runs would get 3 errors in a row (with the default 1 second pause)
* `STREAMING_READ_MAX_RETRIES` should also be increased. Since this type of error does not happen a lot, I would still have a large number (at least 10) because a stopped training run may be a big issue if checkpointing/restart is not well implemented which is not always trivial

I agree ! Feel free to open a PR to increase both values

Catches Timeout error during streaming.

fix #3049


TimeoutError during streaming

Something isn't working

## Describe the bug
I got a TimeoutError after streaming for about 10h.

## Steps to reproduce the bug
Very long code but we could do a test of streaming indefinitely data, though error may take a while to appear.

## Expected results
This error was not expected in the code which considers only `ClientError` but not `TimeoutError`.
See [this line](https://github.com/huggingface/datasets/blob/2814fbd0e18150be409f10804670e98d9ecb87d4/src/datasets/utils/streaming_download_manager.py#L129).
Based on the traceback, it looks like the `TimeoutError` was not captured.

## Actual results
```
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/implementations/http.py", line 614, in async_fetch_range
    out = await r.read()
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1032, in read
    self._body = await self.content.read()
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/aiohttp/streams.py", line 370, in read
    block = await self.readany()
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/aiohttp/streams.py", line 392, in readany
    await self._wait("readany")
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/aiohttp/streams.py", line 306, in _wait
    await waiter
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/aiohttp/helpers.py", line 656, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 1027, in <module>
    main()
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 991, in main
    for batch in tqdm(
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 376, in data_loader_streaming
    for item in dataset:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 179, in __iter__
    key_examples_list = [(key, example)] + [
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 179, in <listcomp>
    key_examples_list = [(key, example)] + [
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 176, in __iter__
    for key, example in iterator:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 225, in __iter__
    for x in self.ex_iterable:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 99, in __iter__
    for key, example in self.generate_examples_fn(**kwargs_with_shuffled_shards):
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 287, in wrapper
    for key, table in generate_tables_fn(**kwargs):
  File "/home/koush/datasets/src/datasets/packaged_modules/json/json.py", line 107, in _generate_tables
    batch = f.read(self.config.chunksize)
  File "/home/koush/datasets/src/datasets/utils/streaming_download_manager.py", line 126, in read_with_retries
    out = read(*args, **kwargs)
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/implementations/http.py", line 572, in read
    return super().read(length)
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/spec.py", line 1533, in read
    out = self.cache._fetch(self.loc, self.loc + length)
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/caching.py", line 390, in _fetch
    self.cache = self.fetcher(start, bend)
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/asyn.py", line 69, in sync
    raise FSTimeoutError from return_result
fsspec.exceptions.FSTimeoutError
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.2.dev0
- Platform: Linux-5.4.0-1043-gcp-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 5.0.0


Identify which shard data belongs to

enhancement

New feature or request

Independently of this I think it raises the need to allow multiprocessing during streaming so that we get samples from multiple shards in one batch.

**Is your feature request related to a problem? Please describe.**

I'm training on a large dataset made of multiple sub-datasets.
During training I can observe some jumps in loss which may correspond to different shards.

![image](https://user-images.githubusercontent.com/715491/136668758-521263aa-a9b2-4ad2-8d22-060b6bf86a1c.png)

My suspicion is that either:
* some of the sub-datasets are harder for the model than others
* some of the sub-datasets are not formatted properly

I'd like to identify which shards correspond to those jumps.

**Describe the solution you'd like**

It would be nice to have a key associated to each data sample or data batch containing details on where the data comes from (shard idx + item idx within the shard).
This should be supported both in local and streaming mode.

**Describe alternatives you've considered**
A fix would be for me to add myself details (shard id, sample id) as part of each data sample.
The inconvenient is that it requires users to process/reupload every dataset when they need this feature.