Error when loading a metric

Something isn't working

## Describe the bug
As reported by @sgugger, after last release, exception is thrown when loading a metric.

## Steps to reproduce the bug
```python
from datasets import load_metric

metric = load_metric("squad_v2")
```

## Actual results
```
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-1-e612a8cab787> in <module>
      1 from datasets import load_metric
----> 2 metric = load_metric("squad_v2")

d:\projects\huggingface\datasets\src\datasets\load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, revision, script_version, **metric_init_kwargs)
   1336         )
   1337         revision = script_version
-> 1338     metric_module = metric_module_factory(
   1339         path, revision=revision, download_config=download_config, download_mode=download_mode
   1340     ).module_path

d:\projects\huggingface\datasets\src\datasets\load.py in metric_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, **download_kwargs)
   1237                 if not isinstance(e1, FileNotFoundError):
   1238                     raise e1 from None
-> 1239                 raise FileNotFoundError(
   1240                     f"Couldn't find a metric script at {relative_to_absolute_path(combined_path)}. "
   1241                     f"Metric '{path}' doesn't exist on the Hugging Face Hub either."

FileNotFoundError: Couldn't find a metric script at D:\projects\huggingface\datasets\squad_v2\squad_v2.py. Metric 'squad_v2' doesn't exist on the Hugging Face Hub either.
```


Updates LexGLUE and MultiEURLEX README.md files

- Fix leaderboard in LexGLUE.
- Fix an error in the CaseHOLD data example.
- Turn MultiEURLEX dataset statistics table into HTML to nicely render in HF website.


add XCSR dataset

> Hi ! Thanks for adding this dataset :)
>
> Do you know how the translations were done ? Maybe we can mention that in the dataset card.
>
> The rest looks all good to me :) good job with the dataset script and the dataset card !
>
> Just one thing: we try to have dummy_data.zip files that are as small as possible, however here each zip file is 70KB+. It think we can make them even smaller if we remove unnecessary files in them. In particular in the `ar` dummy data zip file, we don't need the data for all languages, but rather only the `ar` files. Could you try to remove the unnecessary files in the dummy data zip files ?

Hi!

Thank you so much for reviewing this PR. I've updated the README to briefly mention the translations and added a link to the paper, where a detailed description of the translation procedure can be found in the appendix.

For the dummy_data.zip files, is it possible to keep all the current files? I tried to remove some of the files, but the removal led to a failure in the local testing. We also think it may be better to keep the current dummy_data.zip files because all the data are useful actually. Thanks a lot!!

Hi @lhoestq, just a gentle ping on this PR. :D

Hi,

I wanted to add the [XCSR ](https://inklab.usc.edu//XCSR/xcsr_datasets) dataset to huggingface! :)

I followed the instructions of adding new dataset to huggingface and have all the required files ready now! It would be super helpful if you can take a look and review them. Thanks in advance for your time and help. Look forward to hearing from you and can't wait to add XCSR to huggingface :D


Import error installing with ppc64le

bug

Something isn't working

This seems to be an issue with importing PyArrow so I posted the problem [here](https://issues.apache.org/jira/browse/ARROW-14323), and I'm closing this issue.

## Describe the bug
Installing the datasets library with a computer running with ppc64le seems to cause an issue when importing the datasets library.

```
python
Python 3.6.13 | packaged by conda-forge | (default, Sep 23 2021, 07:37:44)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datasets
Illegal instruction (core dumped)
```

Error when importing
`Illegal instruction (core dumped)`

## Steps to reproduce the bug
I get this error when installing the library by using conda.  I can't install with pip I believe because pyarrow only has the ppc64le library on conda forge
```
conda create --name transformers_py36_v2 python=3.6
conda activate transformers_py36_v2
conda install datasets
```

## Tracebacks
conda create --name transformers_py36_v2 python=3.6

```
Collecting package metadata (current_repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda

## Package Plan ##

  environment location: /p/home/gerryc/.conda/envs/transformers_py36_v2

  added / updated specs:
    - python=3.6

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-ppc64le::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-ppc64le::_openmp_mutex-4.5-1_gnu
  ca-certificates    conda-forge/linux-ppc64le::ca-certificates-2021.10.8-h1084571_0
  certifi            pkgs/main/linux-ppc64le::certifi-2020.12.5-py36h6ffa863_0
  ld_impl_linux-ppc~ conda-forge/linux-ppc64le::ld_impl_linux-ppc64le-2.36.1-ha35d02b_2
  libffi             conda-forge/linux-ppc64le::libffi-3.4.2-h3b9df90_4
  libgcc-ng          conda-forge/linux-ppc64le::libgcc-ng-11.2.0-h7698a5e_11
  libgomp            conda-forge/linux-ppc64le::libgomp-11.2.0-h7698a5e_11
  libstdcxx-ng       conda-forge/linux-ppc64le::libstdcxx-ng-11.2.0-habdf983_11
  libzlib            conda-forge/linux-ppc64le::libzlib-1.2.11-h339bb43_1013
  ncurses            conda-forge/linux-ppc64le::ncurses-6.2-hea85c5d_4
  openssl            conda-forge/linux-ppc64le::openssl-1.1.1l-h4e0d66e_0
  pip                conda-forge/noarch::pip-21.3-pyhd8ed1ab_0
  python             conda-forge/linux-ppc64le::python-3.6.13-h57873ef_2_cpython
  readline           conda-forge/linux-ppc64le::readline-8.1-h5c45dff_0
  setuptools         pkgs/main/linux-ppc64le::setuptools-58.0.4-py36h6ffa863_0
  sqlite             conda-forge/linux-ppc64le::sqlite-3.36.0-h4e2196e_2
  tk                 conda-forge/linux-ppc64le::tk-8.6.11-h41c6715_1
  wheel              conda-forge/noarch::wheel-0.37.0-pyhd8ed1ab_1
  xz                 conda-forge/linux-ppc64le::xz-5.2.5-h6eb9509_1
  zlib               conda-forge/linux-ppc64le::zlib-1.2.11-h339bb43_1013

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate transformers_py36_v2
#
# To deactivate an active environment, use
#
#     $ conda deactivate
```

conda activate transformers_py36_v2
conda install datasets
```
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda

## Package Plan ##

  environment location: /p/home/gerryc/.conda/envs/transformers_py36_v2

  added / updated specs:
    - datasets

The following NEW packages will be INSTALLED:

  abseil-cpp         conda-forge/linux-ppc64le::abseil-cpp-20210324.2-h3b9df90_0
  aiohttp            conda-forge/linux-ppc64le::aiohttp-3.7.4.post0-py36hc33305d_0
  arrow-cpp          conda-forge/linux-ppc64le::arrow-cpp-5.0.0-py36hf9cf308_8_cpu
  async-timeout      conda-forge/noarch::async-timeout-3.0.1-py_1000
  attrs              conda-forge/noarch::attrs-21.2.0-pyhd8ed1ab_0
  aws-c-cal          conda-forge/linux-ppc64le::aws-c-cal-0.5.11-hb3fac3d_0
  aws-c-common       conda-forge/linux-ppc64le::aws-c-common-0.6.2-h4e0d66e_0
  aws-c-event-stream conda-forge/linux-ppc64le::aws-c-event-stream-0.2.7-h76da5f2_13
  aws-c-io           conda-forge/linux-ppc64le::aws-c-io-0.10.5-hf6a6c7c_0
  aws-checksums      conda-forge/linux-ppc64le::aws-checksums-0.1.11-hfe76d68_7
  aws-sdk-cpp        conda-forge/linux-ppc64le::aws-sdk-cpp-1.8.186-h90855e8_3
  brotlipy           conda-forge/linux-ppc64le::brotlipy-0.7.0-py36hc33305d_1001
  bzip2              conda-forge/linux-ppc64le::bzip2-1.0.8-h4e0d66e_4
  c-ares             conda-forge/linux-ppc64le::c-ares-1.17.2-h4e0d66e_0
  cffi               conda-forge/linux-ppc64le::cffi-1.14.6-py36h021ab3c_1
  chardet            conda-forge/linux-ppc64le::chardet-4.0.0-py36h270354c_1
  colorama           conda-forge/noarch::colorama-0.4.4-pyh9f0ad1d_0
  cryptography       conda-forge/linux-ppc64le::cryptography-3.4.7-py36hc71b123_0
  dataclasses        conda-forge/noarch::dataclasses-0.8-pyh787bdff_2
  datasets           conda-forge/noarch::datasets-1.12.1-pyhd8ed1ab_1
  dill               conda-forge/noarch::dill-0.3.4-pyhd8ed1ab_0
  filelock           conda-forge/noarch::filelock-3.3.0-pyhd8ed1ab_0
  fsspec             conda-forge/noarch::fsspec-2021.10.0-pyhd8ed1ab_0
  gflags             conda-forge/linux-ppc64le::gflags-2.2.2-hb209c28_1004
  glog               conda-forge/linux-ppc64le::glog-0.5.0-h4040248_0
  grpc-cpp           conda-forge/linux-ppc64le::grpc-cpp-1.40.0-h2bf711c_2
  huggingface_hub    conda-forge/noarch::huggingface_hub-0.0.19-pyhd8ed1ab_0
  idna               conda-forge/noarch::idna-2.10-pyh9f0ad1d_0
  idna_ssl           conda-forge/noarch::idna_ssl-1.0.0-0
  importlib-metadata conda-forge/linux-ppc64le::importlib-metadata-4.8.1-py36h270354c_0
  importlib_metadata conda-forge/noarch::importlib_metadata-4.8.1-hd8ed1ab_0
  krb5               conda-forge/linux-ppc64le::krb5-1.19.2-haf43566_2
  libblas            conda-forge/linux-ppc64le::libblas-3.9.0-11_linuxppc64le_openblas
  libbrotlicommon    conda-forge/linux-ppc64le::libbrotlicommon-1.0.9-h4e0d66e_5
  libbrotlidec       conda-forge/linux-ppc64le::libbrotlidec-1.0.9-h4e0d66e_5
  libbrotlienc       conda-forge/linux-ppc64le::libbrotlienc-1.0.9-h4e0d66e_5
  libcblas           conda-forge/linux-ppc64le::libcblas-3.9.0-11_linuxppc64le_openblas
  libcurl            conda-forge/linux-ppc64le::libcurl-7.79.1-he415e40_1
  libedit            conda-forge/linux-ppc64le::libedit-3.1.20191231-h41a240f_2
  libev              conda-forge/linux-ppc64le::libev-4.33-h6eb9509_1
  libevent           conda-forge/linux-ppc64le::libevent-2.1.10-h97db324_4
  libgfortran-ng     conda-forge/linux-ppc64le::libgfortran-ng-11.2.0-hfdc3801_11
  libgfortran5       conda-forge/linux-ppc64le::libgfortran5-11.2.0-he58fbb4_11
  liblapack          conda-forge/linux-ppc64le::liblapack-3.9.0-11_linuxppc64le_openblas
  libnghttp2         conda-forge/linux-ppc64le::libnghttp2-1.43.0-h42039ad_1
  libopenblas        conda-forge/linux-ppc64le::libopenblas-0.3.17-pthreads_h486567c_1
  libprotobuf        conda-forge/linux-ppc64le::libprotobuf-3.18.1-h690f14c_0
  libssh2            conda-forge/linux-ppc64le::libssh2-1.10.0-ha5a9321_2
  libthrift          conda-forge/linux-ppc64le::libthrift-0.15.0-h54f692e_1
  libutf8proc        conda-forge/linux-ppc64le::libutf8proc-2.6.1-h4e0d66e_0
  lz4-c              conda-forge/linux-ppc64le::lz4-c-1.9.3-h3b9df90_1
  multidict          conda-forge/linux-ppc64le::multidict-5.2.0-py36hc33305d_0
  multiprocess       conda-forge/linux-ppc64le::multiprocess-0.70.12.2-py36hc33305d_0
  numpy              conda-forge/linux-ppc64le::numpy-1.19.5-py36h86665d4_1
  orc                conda-forge/linux-ppc64le::orc-1.7.0-hae6b4bd_0
  packaging          conda-forge/noarch::packaging-21.0-pyhd8ed1ab_0
  pandas             conda-forge/linux-ppc64le::pandas-1.1.5-py36hab1a6e6_0
  parquet-cpp        conda-forge/noarch::parquet-cpp-1.5.1-2
  pyarrow            conda-forge/linux-ppc64le::pyarrow-5.0.0-py36h7a46c7e_8_cpu
  pycparser          conda-forge/noarch::pycparser-2.20-pyh9f0ad1d_2
  pyopenssl          conda-forge/noarch::pyopenssl-21.0.0-pyhd8ed1ab_0
  pyparsing          conda-forge/noarch::pyparsing-2.4.7-pyh9f0ad1d_0
  pysocks            conda-forge/linux-ppc64le::pysocks-1.7.1-py36h270354c_3
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.2-pyhd8ed1ab_0
  python-xxhash      conda-forge/linux-ppc64le::python-xxhash-2.0.2-py36hc33305d_0
  python_abi         conda-forge/linux-ppc64le::python_abi-3.6-2_cp36m
  pytz               conda-forge/noarch::pytz-2021.3-pyhd8ed1ab_0
  pyyaml             conda-forge/linux-ppc64le::pyyaml-5.4.1-py36hc33305d_1
  re2                conda-forge/linux-ppc64le::re2-2021.09.01-h3b9df90_0
  requests           conda-forge/noarch::requests-2.25.1-pyhd3deb0d_0
  s2n                conda-forge/linux-ppc64le::s2n-1.0.10-h97db324_0
  six                conda-forge/noarch::six-1.16.0-pyh6c4a22f_0
  snappy             conda-forge/linux-ppc64le::snappy-1.1.8-hb209c28_3
  tqdm               conda-forge/noarch::tqdm-4.62.3-pyhd8ed1ab_0
  typing-extensions  conda-forge/noarch::typing-extensions-3.10.0.2-hd8ed1ab_0
  typing_extensions  conda-forge/noarch::typing_extensions-3.10.0.2-pyha770c72_0
  urllib3            conda-forge/noarch::urllib3-1.26.7-pyhd8ed1ab_0
  xxhash             conda-forge/linux-ppc64le::xxhash-0.8.0-h4e0d66e_3
  yaml               conda-forge/linux-ppc64le::yaml-0.2.5-h6eb9509_0
  yarl               conda-forge/linux-ppc64le::yarl-1.6.3-py36hc33305d_2
  zipp               conda-forge/noarch::zipp-3.6.0-pyhd8ed1ab_0
  zstd               conda-forge/linux-ppc64le::zstd-1.5.0-h65c4b1a_0

The following packages will be UPDATED:

  certifi            pkgs/main::certifi-2020.12.5-py36h6ff~ --> conda-forge::certifi-2021.5.30-py36h270354c_0

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Red Hat Enterprise Linux 8.2 (Ootpa)
- Python version: 3.6
- PyArrow version: pyarrow - 5.0.0 - py36h7a46c7e_8_cpu - conda-forge

Any help would be appreciated! I've been struggling on installing datasets on this machine.


Fix pathlib patches for streaming

Fix issue https://github.com/huggingface/datasets/issues/2866 (for good this time)

`counter` now works in both streaming and non-streaming mode.
And the `AttributeError: 'str' object has no attribute 'as_posix'` related to the patch of Path.open is fixed as well

Note : the patches should only affect the datasets module, not the user's ones ! That's why we should probably use something else than patch.object to patch the Path class' methods.

cc @severo @albertvillanova


Custom plain text dataset, plain json dataset and plain csv dataset are remove from datasets template folder

dataset request

Requesting to add a new dataset

Hi @zixiliuUSC,

As explained in the documentation (https://huggingface.co/docs/datasets/loading.html#json), we support loading any dataset in JSON (as well as CSV, text, Parquet) format:
```python
ds = load_dataset('json', data_files='my_file.json')
```

## Adding a Dataset
- **Name:** text, json, csv
- **Description:** I am developing a customized dataset loading script. The problem is  mainly about my custom dataset is seperate into many files and I only find a dataset loading template in [https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py](https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py) that can handle my circumstance. I'm afraid these templates are too old to use. Could you re-add these three templates to current master branch?


Fix Windows CI with FileNotFoundError when stting up s3_base fixture

Thanks ! Sorry for the inconvenience ^^'

Fix #3069.


bug

Something isn't working

## Describe the bug
After commit 9353fc863d0c99ab0427f83cc5a4f04fcf52f1df, the CI fails on Windows  with FileNotFoundError when stting up s3_base fixture. See: https://app.circleci.com/pipelines/github/huggingface/datasets/8151/workflows/5db8d154-badd-4d3d-b202-ca7a318997a2/jobs/50321

Error summary:
```
ERROR tests/test_arrow_dataset.py::test_dummy_dataset_serialize_s3 - FileNotF...
ERROR tests/test_dataset_dict.py::test_dummy_dataset_serialize_s3 - FileNotFo...
```

Stack trace:
```
______________ ERROR at setup of test_dummy_dataset_serialize_s3 ______________
[gw0] win32 -- Python 3.6.8 C:\tools\miniconda3\python.exe

    @pytest.fixture()
    def s3_base():
        # writable local S3 system
        import shlex
        import subprocess

        # Mocked AWS Credentials for moto.
        old_environ = os.environ.copy()
        os.environ.update(S3_FAKE_ENV_VARS)

>       proc = subprocess.Popen(shlex.split("moto_server s3 -p %s" % s3_port))

tests\s3_fixtures.py:32:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\tools\miniconda3\lib\subprocess.py:729: in __init__
    restore_signals, start_new_session)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <subprocess.Popen object at 0x0000012BB8A4B908>
args = 'moto_server s3 -p 5555', executable = None, preexec_fn = None
close_fds = True, pass_fds = (), cwd = None, env = None
startupinfo = <subprocess.STARTUPINFO object at 0x0000012BB8177630>
creationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = -1
c2pwrite = -1, errread = -1, errwrite = -1, unused_restore_signals = True
unused_start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       unused_restore_signals, unused_start_new_session):
        """Execute program (MS Windows version)"""

        assert not pass_fds, "pass_fds not supported on Windows."

        if not isinstance(args, str):
            args = list2cmdline(args)

        # Process startup details
        if startupinfo is None:
            startupinfo = STARTUPINFO()
        if -1 not in (p2cread, c2pwrite, errwrite):
            startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES
            startupinfo.hStdInput = p2cread
            startupinfo.hStdOutput = c2pwrite
            startupinfo.hStdError = errwrite

        if shell:
            startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
            startupinfo.wShowWindow = _winapi.SW_HIDE
            comspec = os.environ.get("COMSPEC", "cmd.exe")
            args = '{} /c "{}"'.format (comspec, args)

        # Start the process
        try:
            hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                                     # no special security
                                     None, None,
                                     int(not close_fds),
                                     creationflags,
                                     env,
                                     os.fspath(cwd) if cwd is not None else None,
>                                    startupinfo)
E                                    FileNotFoundError: [WinError 2] The system cannot find the file specified

C:\tools\miniconda3\lib\subprocess.py:1017: FileNotFoundError
```


feat: increase streaming retry config

@lhoestq I had 2 runs for more than 2 days each, continuously streaming (they were failing before with 3 retries at 1 sec interval).

They are running on TPU's (so great internet connection) and only had connection errors a few times each (3 & 4). Each time it worked after only 1 retry.
The reason for a higher number of retries is for local connections. It would allow for almost 2mn of a wifi/ethernet disconnection. In practice this should not happen very often.

Let me know if you think it's too much.

Increase streaming config parameters:
* retry interval set to 5 seconds
* max retries set to 20 (so 1mn 40s)


add story_cloze

Thanks for pushing this dataset :)

According to the CI, the file `cloze_test_val__spring2016 - cloze_test_ALL_val.csv` is missing in the dummy data zip file (the zip files seem empty). Feel free to add this file with 4-5 lines and it should be good

And you can fix the YAML tags with
```yaml
pretty_name: Story Cloze Test
```
and filling the other tags task_categories and task_ids

If the dataset doesn exist on paperswithcode, you can just leave
```yaml
paperswithcode_id: null
```

> Thanks @zaidalyafeai, the failing test is due to an issue in the master branch, that has already been fixed.
>
> You can include the fix:
>
> ```
> git checkout add_story_cloze
> git fetch upstream master
> git merge upstream/master
> ```

Thanks @albertvillanova, passed all the tests now.

Thanks Albert, I fixed the suggested comments. This dataset has no train splits, it is only used for evaluation.


Add iter_archive

Added the `iter_archive` method for the StreamingDownloadManager.

It was already implemented in the regular DownloadManager.
Now it can be used to stream from TAR archives as mentioned in https://github.com/huggingface/datasets/issues/2829

I also updated the `food101` dataset as an example.
Any image/audio dataset using TAR archives can be updated to use `iter_archive` in order to be streamable :)

cc @severo


Fix test command after refac

Fix the `datasets-cli` test command after the `prepare_module` change in #2986


Make `interleave_datasets` more robust

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Right now there are few hiccups using `interleave_datasets`. Interleaved dataset iterates until the smallest dataset completes it's iterator. In this way larger datasets may not complete full epoch of iteration.
It creates new problems in calculation of epoch since there are no way to track which dataset from `interleave_datasets` completes how many epoch.

**Describe the solution you'd like**
For `interleave_datasets` module,
- [ ] Add a boolean argument `--stop-iter` in `interleave_datasets` that enables dataset to either iterate infinite amount of time or not. That means it should not return `StopIterator` exception in case `--stop-iter=False`.
- [ ] Internal list variable `iter_cnt` that explains how many times (in steps/epochs) each dataset iterates at a given point.
- [ ] Add an argument `--max-iter` (list type) that explain maximum amount of time each of the dataset can iterate. After complete  `--max-iter` of one dataset, other dataset should continue sampling and when all the dataset finish their respective `--max-iter`, only then return `StopIterator`

Note: I'm new to `datasets` api. May be these features are already there in the datasets.

Since multitask training is the latest trends, I believe this feature would make the `datasets` api more popular.

@lhoestq


Windows CI is unable to test streaming properly because of SSL issues

I think this problem is already fixed:
```python
In [4]: import fsspec
   ...:
   ...: url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"
   ...:
   ...: fsspec.open(url).open()
Out[4]: <File-like object HTTPFileSystem, https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattribu
```

No I'm still having this issue on my windows, and so does the CI

In https://github.com/huggingface/datasets/pull/3041 the windows tests were skipped because of SSL issues with moon-staging.huggingface.co:443

The issue appears only on windows with asyncio. On Linux it works. With requests it works as well. And with the production environment huggingface.co it also works.

to reproduce on windows:
```python
import fsspec

# use any URL to a file in a dataset repo
url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"

fsspec.open(url).open()
```
raises
```python
FileNotFoundError: https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes
```
because of
```python
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host moon-staging.huggingface.co:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')]
```


Update summary on PyPi beyond NLP

More than just NLP now


Feature request : add leave=True to dataset.map to enable tqdm nested bars (and whilst we're at it couldn't we get a way to access directly tqdm underneath?)

New feature or request

@lhoestq, @albertvillanova can we have `**tqdm_kwargs` in `map`? If there are any fields that are important to our tqdm (like iterable or unit), we can pop them before initialising the tqdm object so as to avoid duplicity.

Hi ! Sounds like a good idea :)

Also I think it would be better to have this as an actual parameters instead of kwargs to make it clearer

**A clear and concise description of what you want to happen.**

It would be so nice to be able to nest HuggingFace `Datasets.map() ` progress bars in the grander scheme of things and whilst we're at it why not other functions.

**Describe alternatives you've considered**

By the way is there not a way to directly interact with underlying tqdm module ? **kwargs-ish?

**Additional context**

Furthering tqdm integration #2374 and huggingface/transformers#11797 solutioned by huggingface/transformers#12226 provided with tqdm description as `desc=`

@sgugger @bhavitvyamalik


load_dataset('openwebtext') yields "Compressed file ended before the end-of-stream marker was reached"

bug

Something isn't working

Hi @RylanSchaeffer, thanks for reporting.

I'm sorry, but I was not able to reproduce your problem.

Normally, the reason for this type of error is that, during your download of the data files, this was not fully complete.

Could you please try to load the dataset again but forcing its redownload? Please use:
```python
dataset = load_dataset("openwebtext", download_mode="FORCE_REDOWNLOAD")
```

Let me know if the problem persists.

I close this issue for the moment. Feel free to re-open it again if the problem persists.

## Describe the bug
When I try `load_dataset('openwebtext')`, I receive a "EOFError: Compressed file ended before the end-of-stream marker was reached" error.

## Steps to reproduce the bug
```
from datasets import load_dataset
dataset = load_dataset('openwebtext')
```

## Expected results
I expect the `dataset` variable to be properly constructed.

## Actual results

```
File "/home/rschaef/CoCoSci-Language-Distillation/distillation_v2/ratchet_learning/tasks/base.py", line 37, in create_dataset
    dataset_str,
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/load.py", line 1117, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/builder.py", line 637, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/builder.py", line 704, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/rschaef/.cache/huggingface/modules/datasets_modules/datasets/openwebtext/85b3ae7051d2d72e7c5fdf6dfb462603aaa26e9ed506202bf3a24d261c6c40a1/openwebtext.py", line 61, in _split_generators
    dl_dir = dl_manager.download_and_extract(_URL)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/download_manager.py", line 261, in extract
    partial(cached_path, download_config=download_config), path_or_paths, num_proc=num_proc, disable_tqdm=False
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 197, in map_nested
    return function(data_struct)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 316, in cached_path
    output_path, force_extract=download_config.force_extract
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/extract.py", line 40, in extract
    self.extractor.extract(input_path, output_path, extractor=extractor)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/extract.py", line 179, in extract
    return extractor.extract(input_path, output_path)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/extract.py", line 53, in extract
    tar_file.extractall(output_path)
  File "/usr/lib/python3.6/tarfile.py", line 2010, in extractall
    numeric_owner=numeric_owner)
  File "/usr/lib/python3.6/tarfile.py", line 2052, in extract
    numeric_owner=numeric_owner)
  File "/usr/lib/python3.6/tarfile.py", line 2122, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/usr/lib/python3.6/tarfile.py", line 2171, in makefile
    copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
  File "/usr/lib/python3.6/tarfile.py", line 249, in copyfileobj
    buf = src.read(bufsize)
  File "/usr/lib/python3.6/lzma.py", line 200, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.6/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/usr/lib/python3.6/_compression.py", line 99, in read
    raise EOFError("Compressed file ended before the "
python-BaseException
EOFError: Compressed file ended before the end-of-stream marker was reached
```

## Environment info
- `datasets` version: 1.12.1
- Platform: Linux-4.4.0-173-generic-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.10
- PyArrow version: 5.0.0


Fix task reloading from cache

When reloading a dataset from the cache when doing `map`, the tasks templates were kept instead of being updated regarding the output of the `map` function. This is an issue because we drop the tasks templates that are not compatible anymore after `map`, for example if a column of the template was removed.

This PR fixes this and for convenience introduces a decorator `@transmit_tasks` that takes care of doing this verification, similar to the `@transmit_format` decorator.

This should fix issue https://github.com/huggingface/datasets/issues/3047 cc @sgugger


Dataset wikipedia and Bookcorpusopen cannot be fetched from dataloader.

Something isn't working

Hi ! I think this issue is more related to the `transformers` project. Could you open an issue on https://github.com/huggingface/transformers ?

Anyway I think the issue could be that both wikipedia and bookcorpusopen have an additional "title" column, contrary to wikitext which only has a "text" column. After calling `load_dataset`, can you try doing `dataset = dataset.remove_columns("title")` ?

Removing the "title" column works! Thanks for your advice.

Maybe I should still create an issue to `transformers' to mark this solution?

## Describe the bug
I have used the previous version of `transformers` and `datasets`. The dataset `wikipedia` can be successfully used. Recently, I upgrade them to the newest version and find it raises errors. I also tried other datasets. The `wikitext` works and the `bookcorpusopen` raises the same errors as `wikipedia`.

## Steps to reproduce the bug
Run the `run_mlm_no_trainer.py` and the given script on this [link](https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling). Change the dataset from wikitext to wikipedia or bookcorpusopen. BTW, the library transformers is of version 4.11.3.

## Expected results
The data batchs are fetched from the data loader and train.

## Actual results
The first time to fetch data batch occurs error.
`Traceback (most recent call last):
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 705, in convert_to_tensors
    tensor = as_tensor(value)
ValueError: too many dimensions 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/original_run_mlm_no_trainer.py", line 528, in <module>
    main()
  File "src/original_run_mlm_no_trainer.py", line 488, in main
    for step, batch in enumerate(train_dataloader):
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/accelerate/data_loader.py", line 303, in __iter__
    for batch in super().__iter__():
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/data/data_collator.py", line 41, in __call__
    return self.torch_call(features)
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/data/data_collator.py", line 671, in torch_call
    batch = self.tokenizer.pad(examples, return_tensors="pt", pad_to_multiple_of=self.pad_to_multiple_of)
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2774, in pad
    return BatchEncoding(batch_outputs, tensor_type=return_tensors)
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 210, in __init__
    self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
  File "/home/zyli/anaconda3/envs/LatestStacking/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 722, in convert_to_tensors
    "Unable to create tensor, you should probably activate truncation and/or padding "
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length.
`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Linux-5.8.0-59-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.6
- PyArrow version: 5.0.0


Error in per class precision computation

Something isn't working

Hi @tidhamecha2, thanks for reporting.

Indeed, we fixed this issue just one week ago: #3008

The fix will be included in our next version release.

In the meantime, you can incorporate the fix by installing `datasets` from the master branch:
```
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```

## Describe the bug
When trying to get the per class precision values by providing `average=None`, following error is thrown `ValueError: can only convert an array of size 1 to a Python scalar`

## Steps to reproduce the bug
```python
from datasets import load_dataset, load_metric
precision_metric = load_metric("precision")
predictions = [0, 2, 1, 0, 0, 1]
references = [0, 1, 2, 0, 1, 2]
results = precision_metric.compute(predictions=predictions, references=references, average=None)
```

## Expected results
`  {'precision': array([0.66666667, 0.        , 0.        ])}`
as per https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py

## Actual results
```
   output = self._compute(predictions=predictions, references=references, **kwargs)
  File "~/.cache/huggingface/modules/datasets_modules/metrics/precision/94709a71c6fe37171ef49d3466fec24dee9a79846c9f176dff66a649e9811690/precision.py", line 110, in _compute
    sample_weight=sample_weight,
ValueError: can only convert an array of size 1 to a Python scalar
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: linux
- Python version: 3.6.9
- PyArrow version: 5.0.0


Fix meteor metric for version >= 3.6.4

After `nltk` update, the meteor metric expects pre-tokenized inputs (breaking change).

This PR fixes this issue, while maintaining compatibility with older versions.


CI test suite fails after meteor metric update

Something isn't working

## Describe the bug
CI test suite fails: https://app.circleci.com/pipelines/github/huggingface/datasets/8110/workflows/f059ba43-9154-4632-bebb-82318447ddc9/jobs/50010

Stack trace:
```
___________________ LocalMetricTest.test_load_metric_meteor ____________________
[gw1] linux -- Python 3.6.15 /home/circleci/.pyenv/versions/3.6.15/bin/python3.6

self = <tests.test_metric_common.LocalMetricTest testMethod=test_load_metric_meteor>
metric_name = 'meteor'

    def test_load_metric(self, metric_name):
        doctest.ELLIPSIS_MARKER = "[...]"
        metric_module = importlib.import_module(datasets.load.prepare_module(os.path.join("metrics", metric_name))[0])
        metric = datasets.load.import_main_class(metric_module.__name__, dataset=False)
        # check parameters
        parameters = inspect.signature(metric._compute).parameters
        self.assertTrue("predictions" in parameters)
        self.assertTrue("references" in parameters)
        self.assertTrue(all([p.kind != p.VAR_KEYWORD for p in parameters.values()]))  # no **kwargs
        # run doctest
        with self.patch_intensive_calls(metric_name, metric_module.__name__):
            with self.use_local_metrics():
>               results = doctest.testmod(metric_module, verbose=True, raise_on_error=True)

tests/test_metric_common.py:75:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../.pyenv/versions/3.6.15/lib/python3.6/doctest.py:1951: in testmod
    runner.run(test)
../.pyenv/versions/3.6.15/lib/python3.6/doctest.py:1839: in run
    r = DocTestRunner.run(self, test, compileflags, out, False)
../.pyenv/versions/3.6.15/lib/python3.6/doctest.py:1476: in run
    return self.__run(test, compileflags, out)
../.pyenv/versions/3.6.15/lib/python3.6/doctest.py:1382: in __run
    exception)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <doctest.DebugRunner object at 0x7f4c26bd3da0>
out = <built-in method write of _io.TextIOWrapper object at 0x7f51a21852d0>
test = <DocTest datasets_modules.datasets.meteor.6201bb45d2c144ea7963680949d20f523d74a741fa0f8a806f836e6caa5245d7.meteor.Mete...ets_modules/datasets/meteor/6201bb45d2c144ea7963680949d20f523d74a741fa0f8a806f836e6caa5245d7/meteor.py:87 (5 examples)>
example = <doctest.Example object at 0x7f4c26bd3eb8>
exc_info = (<class 'TypeError'>, TypeError('"hypothesis" expects pre-tokenized hypothesis (Iterable[str]): It is a guide to action which ensures that the military always obeys the commands of the party',), <traceback object at 0x7f4cd01afec8>)

    def report_unexpected_exception(self, out, test, example, exc_info):
>       raise UnexpectedException(test, example, exc_info)
E       doctest.UnexpectedException: <DocTest datasets_modules.datasets.meteor.6201bb45d2c144ea7963680949d20f523d74a741fa0f8a806f836e6caa5245d7.meteor.Meteor from /tmp/pytest-of-circleci/pytest-0/popen-gw1/cache/modules/datasets_modules/datasets/meteor/6201bb45d2c144ea7963680949d20f523d74a741fa0f8a806f836e6caa5245d7/meteor.py:87 (5 examples)>

../.pyenv/versions/3.6.15/lib/python3.6/doctest.py:1845: UnexpectedException
```


Update Biosses

Fix variable naming


load_dataset('the_pile_openwebtext2') produces ArrowInvalid, value too large to fit in C integer type

Something isn't working

I encountered the same bug using different datasets.
any suggestions?

## Describe the bug
When loading `the_pile_openwebtext2`, we get the error `pyarrow.lib.ArrowInvalid: Value 2111 too large to fit in C integer type`

## Steps to reproduce the bug
```python
import datasets
ds = datasets.load_dataset('the_pile_openwebtext2')
```

## Expected results
Should download the dataset, convert it to an arrow file, and return a working Dataset object.

## Actual results
The download works, but conversion to the arrow file fails as follows:

```
>>> ds = datasets.load_dataset('the_pile_openwebtext2')
Downloading and preparing dataset openwebtext2/plain_text (download: 27.33 GiB, generated: 63.86 GiB
, post-processed: Unknown size, total: 91.19 GiB) to /home/davidbau/.cache/huggingface/datasets/open
webtext2/plain_text/1.0.0/c48ec73ba3483bac673463f48f67e9a4fd8cb49a9d6ec4fb957f0b424b97cf25...
Traceback (most recent call last):
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/builder.py", line 1133,
 in _prepare_split
    writer.write(example, key)
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
366, in write
    self.write_examples_on_file()
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
311, in write_examples_on_file
    pa_array = pa.array(typed_sequence)
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
115, in __arrow_array__
    out = pa.array(cast_to_python_objects(self.data, only_1d_for_numpy=True), type=type)
  File "pyarrow/array.pxi", line 305, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Value 2111 too large to fit in C integer type
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
```

- Platform: Ubuntu 20.04
- Python version: python 3.9
- PyArrow version: 3.0.0


load_dataset cannot download the data and hangs on forever if cache dir specified

Something isn't working

Issue was environment inconsistency, updating packages did the trick

`conda install -c huggingface -c conda-forge datasets`

> Collecting package metadata (current_repodata.json): done
> Solving environment: |
> The environment is inconsistent, please check the package plan carefully
> The following packages are causing the inconsistency:
>
>   - conda-forge/noarch::datasets==1.12.1=pyhd8ed1ab_1
>   - conda-forge/win-64::multiprocess==0.70.12.2=py38h294d835_0
> done
>
> Package Plan
>
>   environment location: C:\xxx\anaconda3\envs\UnBias-94-1
>
>   added / updated specs:
>     - datasets
>
>
> The following NEW packages will be INSTALLED:
>
>   dill               conda-forge/noarch::dill-0.3.4-pyhd8ed1ab_0
>
> The following packages will be UPDATED:
>
>   ca-certificates    pkgs/main::ca-certificates-2021.9.30-~ --> conda-forge::ca-certificates-2021.10.8-h5b45459_0
>   certifi            pkgs/main::certifi-2021.5.30-py38haa9~ --> conda-forge::certifi-2021.10.8-py38haa244fe_0
>
> The following packages will be SUPERSEDED by a higher-priority channel:
>

## Describe the bug
After updating datasets, a code that ran just fine for ages began to fail. Specifying _datasets.load_dataset_'s _cache_dir_ optional argument on Windows 10 machine results in data download to hang on forever. Same call without cache_dir works just fine. Surprisingly exact same code just runs perfectly fine on Linux docker instance running in cloud.

Unfortunately I updated Windows also at the same time and I can't remember which version of datasets was running in my conda environment prior to the update otherwise I would have tried both to check this out.  :(

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```
cache_dir = 'c:/data/datasets'
dataset = load_dataset('wikipedia', '20200501.en', split='train',cache_dir=cache_dir)
```
Note that exact same code without specifying _cache_dir_ argument works perfectly fine.
```
cache_dir = 'c:/data/datasets'
dataset = load_dataset('wikipedia', '20200501.en', split='train')
```

## Expected results
Downloads the dataset and cache is handled in the _cache_dir_ directory
## Actual results
Data download keeps hanging on forever, **NO TRACEBACK**!

## Environment info
- `datasets` version: 1.12.1
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.8.11
- PyArrow version: 3.0.0


Non-Matching Checksum Error with crd3 dataset

Something isn't working

I got the same error for another dataset (`multi_woz_v22`):

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/dialog_acts.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/test/dialogues_001.json']
```

I'm seeing the same issue as @RylanSchaeffer:
Python 3.7.11, macOs 11.4
datasets==1.14.0

fails on:
```python
dataset = datasets.load_dataset("multi_woz_v22")
```

## Describe the bug
When I try loading the crd3 dataset (https://huggingface.co/datasets/crd3), an error is thrown.

## Steps to reproduce the bug
```python
dataset = load_dataset('crd3', split='train')
```

## Expected results
I expect no error to be thrown.

## Actual results
A non-matching checksum error is thrown.

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/RevanthRameshkumar/CRD3/archive/master.zip']
```

## Environment info

- `datasets` version: 1.12.1
- Platform: Linux-4.4.0-173-generic-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.10
- PyArrow version: 5.0.0


Fix streaming: catch Timeout error

I'm running a large test.
Let's see if I get any error within a few days.

This time it stopped after 8h but correctly raised `ConnectionError: Server Disconnected`.

Traceback:
```
Traceback (most recent call last):
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 1027, in <module>
    main()
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 991, in main
    for batch in tqdm(
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 376, in data_loader_streaming
    for item in dataset:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 179, in __iter__
    key_examples_list = [(key, example)] + [
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 179, in <listcomp>
    key_examples_list = [(key, example)] + [
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 176, in __iter__
    for key, example in iterator:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 225, in __iter__
    for x in self.ex_iterable:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 99, in __iter__
    for key, example in self.generate_examples_fn(**kwargs_with_shuffled_shards):
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 287, in wrapper
    for key, table in generate_tables_fn(**kwargs):
  File "/home/koush/datasets/src/datasets/packaged_modules/json/json.py", line 107, in _generate_tables
    batch = f.read(self.config.chunksize)
  File "/home/koush/datasets/src/datasets/utils/streaming_download_manager.py", line 136, in read_with_retries
    raise ConnectionError("Server Disconnected")
ConnectionError: Server Disconnected
```

Right before this error, the warnings were correctly raised:

```
10/10/2021 06:02:26 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 1sec [1/3]
10/10/2021 06:02:27 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 1sec [2/3]
10/10/2021 06:02:28 - WARNING - datasets.utils.streaming_download_manager - Got disconnected from remote data host. Retrying in 1sec [3/3
```

I'm going to see what happens if I change the max retries to 20 and the interval to 5.

Also maybe we can raise the Server Disconnected error with more info about what kind of error caused it (client error, time out, etc.)

I have 2 runs:
* [run 1](https://wandb.ai/dalle-mini/dalle-mini/runs/1nj161cl?workspace=user-borisd13) with [this data](https://huggingface.co/datasets/dalle-mini/encoded) that I will remove soon because I now use the 2nd one
* [run 2](https://wandb.ai/dalle-mini/dalle-mini/runs/he9rrc3q?workspace=user-borisd13) with [this data](https://huggingface.co/datasets/dalle-mini/encoded-vqgan_imagenet_f16_16384)
* `load_dataset(dataset_repo, data_files={'train':'data/train/*.jsonl', 'validation':'data/valid/*.jsonl'}, streaming=True)`

They have now been running by a bit more than a day for one run and 15h for the other.

The error logs are not shown in wandb because the script use `pylogging` (not sure why, I should change it) but basically so far with the new settings I had one timeout in each with successful reconnect afterwards.

So I think it's a good idea to have:
* `STREAMING_READ_RETRY_INTERVAL = 5` since before my runs would get 3 errors in a row (with the default 1 second pause)
* `STREAMING_READ_MAX_RETRIES` should also be increased. Since this type of error does not happen a lot, I would still have a large number (at least 10) because a stopped training run may be a big issue if checkpointing/restart is not well implemented which is not always trivial

I agree ! Feel free to open a PR to increase both values

Catches Timeout error during streaming.

fix #3049


TimeoutError during streaming

Something isn't working

## Describe the bug
I got a TimeoutError after streaming for about 10h.

## Steps to reproduce the bug
Very long code but we could do a test of streaming indefinitely data, though error may take a while to appear.

## Expected results
This error was not expected in the code which considers only `ClientError` but not `TimeoutError`.
See [this line](https://github.com/huggingface/datasets/blob/2814fbd0e18150be409f10804670e98d9ecb87d4/src/datasets/utils/streaming_download_manager.py#L129).
Based on the traceback, it looks like the `TimeoutError` was not captured.

## Actual results
```
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/implementations/http.py", line 614, in async_fetch_range
    out = await r.read()
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1032, in read
    self._body = await self.content.read()
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/aiohttp/streams.py", line 370, in read
    block = await self.readany()
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/aiohttp/streams.py", line 392, in readany
    await self._wait("readany")
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/aiohttp/streams.py", line 306, in _wait
    await waiter
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/aiohttp/helpers.py", line 656, in __exit__
    raise asyncio.TimeoutError from None
asyncio.exceptions.TimeoutError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 1027, in <module>
    main()
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 991, in main
    for batch in tqdm(
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/home/koush/dalle-mini/dev/seq2seq/run_seq2seq_flax.py", line 376, in data_loader_streaming
    for item in dataset:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 179, in __iter__
    key_examples_list = [(key, example)] + [
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 179, in <listcomp>
    key_examples_list = [(key, example)] + [
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 176, in __iter__
    for key, example in iterator:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 225, in __iter__
    for x in self.ex_iterable:
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 99, in __iter__
    for key, example in self.generate_examples_fn(**kwargs_with_shuffled_shards):
  File "/home/koush/datasets/src/datasets/iterable_dataset.py", line 287, in wrapper
    for key, table in generate_tables_fn(**kwargs):
  File "/home/koush/datasets/src/datasets/packaged_modules/json/json.py", line 107, in _generate_tables
    batch = f.read(self.config.chunksize)
  File "/home/koush/datasets/src/datasets/utils/streaming_download_manager.py", line 126, in read_with_retries
    out = read(*args, **kwargs)
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/implementations/http.py", line 572, in read
    return super().read(length)
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/spec.py", line 1533, in read
    out = self.cache._fetch(self.loc, self.loc + length)
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/caching.py", line 390, in _fetch
    self.cache = self.fetcher(start, bend)
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/koush/.pyenv/versions/dev/lib/python3.9/site-packages/fsspec/asyn.py", line 69, in sync
    raise FSTimeoutError from return_result
fsspec.exceptions.FSTimeoutError
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.2.dev0
- Platform: Linux-5.4.0-1043-gcp-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 5.0.0


Identify which shard data belongs to

enhancement

New feature or request

Independently of this I think it raises the need to allow multiprocessing during streaming so that we get samples from multiple shards in one batch.

**Is your feature request related to a problem? Please describe.**

I'm training on a large dataset made of multiple sub-datasets.
During training I can observe some jumps in loss which may correspond to different shards.

![image](https://user-images.githubusercontent.com/715491/136668758-521263aa-a9b2-4ad2-8d22-060b6bf86a1c.png)

My suspicion is that either:
* some of the sub-datasets are harder for the model than others
* some of the sub-datasets are not formatted properly

I'd like to identify which shards correspond to those jumps.

**Describe the solution you'd like**

It would be nice to have a key associated to each data sample or data batch containing details on where the data comes from (shard idx + item idx within the shard).
This should be supported both in local and streaming mode.

**Describe alternatives you've considered**
A fix would be for me to add myself details (shard id, sample id) as part of each data sample.
The inconvenient is that it requires users to process/reupload every dataset when they need this feature.


Loading from cache a dataset for LM built from a text classification dataset sometimes errors

bug

Something isn't working

This has been fixed in 1.15, let me know if you still have this issue

## Describe the bug

Yes, I know, that description sucks. So the problem is arising in the course when we build a masked language modeling dataset using the IMDB dataset. To reproduce (or try since it's a bit fickle).

Create a dataset for masled-language modeling from the IMDB dataset.

```python
from datasets import load_dataset
from transformers import Autotokenizer

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased)
imdb_dataset = load_dataset("imdb", split="train")

def tokenize_function(examples):
    return tokenizer(examples["text"])

tokenized_dataset = imdb_dataset.map(
    tokenize_function, batched=True, remove_columns=["text", "label"]
)

chunk_size = 128

def group_texts(examples):
    # Concatenate all texts.
    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
    # Compute length of concatenated texts
    total_length = len(concatenated_examples[list(examples.keys())[0]])
    # We drop the last chunk if it's smaller than chunk_size
    total_length = (total_length // chunk_size) * chunk_size
    # Split by chunks of max_len.
    result = {
        k: [t[i : i + chunk_size] for i in range(0, total_length, chunk_size)]
        for k, t in concatenated_examples.items()
    }
    # Create a new labels column
    result["labels"] = result["input_ids"].copy()
    return result

lm_dataset = tokenized_dataset.map(group_texts, batched=True)
```

Until now, all is well. The problem comes when you re-execute that code, more specifically:

```python
tokenized_dataset = imdb_dataset.map(
    tokenize_function, batched=True, remove_columns=["text", "label"]
)
lm_dataset = tokenized_dataset.map(group_texts, batched=True)
```

Try several times if the bug doesn't appear instantly, or just each line at a time, ideally in a notebook/Colab and you should get at some point:

```python
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-40-357a56ee3d53> in <module>
----> 1 lm_dataset = tokenized_dataset.map(group_texts, batched=True)

~/git/datasets/src/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1947                 new_fingerprint=new_fingerprint,
   1948                 disable_tqdm=disable_tqdm,
-> 1949                 desc=desc,
   1950             )
   1951         else:

~/git/datasets/src/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    424         }
    425         # apply actual function
--> 426         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    427         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    428         # re-apply format to the output

~/git/datasets/src/datasets/fingerprint.py in wrapper(*args, **kwargs)
    404             # Call actual function
    405
--> 406             out = func(self, *args, **kwargs)
    407
    408             # Update fingerprint of in-place transforms + update in-place history of transforms

~/git/datasets/src/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2138             if os.path.exists(cache_file_name) and load_from_cache_file:
   2139                 logger.warning("Loading cached processed dataset at %s", cache_file_name)
-> 2140                 info = self.info.copy()
   2141                 info.features = features
   2142                 return Dataset.from_file(cache_file_name, info=info, split=self.split)

~/git/datasets/src/datasets/info.py in copy(self)
    278
    279     def copy(self) -> "DatasetInfo":
--> 280         return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
    281
    282

~/git/datasets/src/datasets/info.py in __init__(self, description, citation, homepage, license, features, post_processed, supervised_keys, task_templates, builder_name, config_name, version, splits, download_checksums, download_size, post_processing_size, dataset_size, size_in_bytes)

~/git/datasets/src/datasets/info.py in __post_init__(self)
    177                 for idx, template in enumerate(self.task_templates):
    178                     if isinstance(template, TextClassification):
--> 179                         labels = self.features[template.label_column].names
    180                         self.task_templates[idx] = TextClassification(
    181                             text_column=template.text_column, label_column=template.label_column, labels=labels

KeyError: 'label'
```

It seems that when loading the cache, the dataset tries to access some kind of text classification template (which I imagine comes from the original dataset) and to look at a key that has since been removed.


Fix MedDialog metadata JSON

Fix #2969.


Fix inconsistent caching behaviour in Dataset.map() with multiprocessing #3044

Hi ! Thanks for noticing this inconsistence and suggesting a fix :)

If I understand correctly you try to pass the same fingerprint to each processed shard of the dataset. This can be an issue since each shard is actually a different dataset with different data: they shouldn't have the same fingerprint.

Ideally we want the result after `map` to have this fingerprint. The result after `map` is the concatenation of all the processed shards. In this case what we can do is add the `fingerprint` parameter to `concatenate_datasets` to overwrite the fingerprint here if needed:
https://github.com/huggingface/datasets/blob/03b7f123cc17afc517c0aa2f912bbd90cb266185/src/datasets/arrow_dataset.py#L3588-L3590

then you can pass the fingerprint to `concatenate_datasets` here:
https://github.com/huggingface/datasets/blob/03b7f123cc17afc517c0aa2f912bbd90cb266185/src/datasets/arrow_dataset.py#L2044-L2044

Hi @lhoestq, thanks for the pointers! Not having a unique fingerprint per shard was indeed was indeed a problem.

Let me look into this. I'll be back with a fix soon.

Alright, to clarify about my problem. I using am using `datasets` with large datasets, and want to cache a heavy and non-deterministically fingerprintable function (using `datasets.fingerprint.Hasher`). Using `Dataset.map()` as it is would cause generating a random fingerprint. To circumvent this, I am generating custom deterministic fingerprints, which I pass as an argument to `Dataset.map()`. In that way, a deterministic fingerprint is set, and caching can be used.

This approach works well when using `num_proc==1`, but not so well when using `num_proc>1`. In both cases, `dataset._fingerprint` is effectively set to `new_fingerprint` at the end of the `.map()` call. However, caching is not used when `num_proc>1`, a non deterministically fingerprintable function and `new_fingerprint != null. The reason is that caching operates within `Dataset._map_single` and `new_fingerprint` is not passed here.

This pull request implements a quick fix (+unit test) by passing `new_fingerprint=f"{new_fingerprint}-part{rank+1}-{num_proc}"` to each `_map_single` call. Using a separate name for each call makes sure that each worker uses a different cache file (as you mentioned above).

However, this solution still means that using a different value for `num_proc` will require computing new partial cache files. In the long run, performing the caching within `map()` instead of within `_map_single()` would be a cleaner solution.

Hi @vlievin,

If I understand your example correctly, you are trying to use the `new_fingerprint` param to have a deterministic fingerprint of the transform, which is not hashable due to randomness. Any particular reason why you are not using the `cache_file_name` param instead? I did run your example with the `cache_file_name` specified, and it behaves as expected based on the logs. Internally, `new_fingerprint` is needed to inject the calculated fingerprint into a method by the `fingerprint_transform` decorator, which is then used to compute the cache file name in `Dataset._get_cache_file_path` if the user hasn't specified one.

Hi @lhoestq, I have cleaned up the unit test (incl. styling). It should be ready to merge as such. I am using this branch in my project and everything works fine.

Hi @mariosasko,  the argument `new_fingerprint` allowed me to deterministically cache my transformation when using `num_proc=1`, so I assumed that was the right way to go. But maybe I have misinterpreted how `new_fingerprint` should be used.

But in any case, `map()` should perform consistently with regards to `num_proc`. In my opinion, the behaviour of `Dataset.map()` should perform the same, and this without requiring the user to input `cache_file_name` when `num_proc>1` is set.
But maybe there is a more elegant way to fix this using `cache_file_name` internally for each `_single_map()` call.

So, I think this is a more high level design decision and I will leave it to the maintainers :)

Hi @vlievin,

I appreciate your effort, but `new_fingerprint` behaves as described in the `Dataset.map` docs, and we don't have to follow some artificial consistency in regards to `num_proc`:
https://github.com/huggingface/datasets/blob/adc5cec58dd15ee672016086fefdea34b3143e4f/src/datasets/arrow_dataset.py#L1962-L1963

Additionally, to compute the cache file name, you are using a private method (`dset._get_cache_file_path(new_fingerprint)`); prefixed with `_`), so this is a sign you may be doing something wrong because you are relying on the internals. I suggest you use cache_file_name instead and follow the suffix template docs, which explain how to compute file paths of the created cache files when `num_proc > 1`.

Hi @mariosasko, thanks for the pointer regarding the use of the private method in then unit tests.

Yes, `new_fingerprint` behaves as documented. If you don't think this is an issue, feel free to close this pull request.

Allowing the users to pass the fingerprint themselves for functions that can't be hashed would be a nice improvements. However I agree that as @mariosasko mentioned this is currently not how we want the API to behave for now - since it has to do with the internals of the library.

Though we can discuss what could be the right way of doing it in https://github.com/huggingface/datasets/issues/3044 if you don't mind !

Fix #3044

1. A rough unit test that fails without the fix. It probably doesn't comply with your code standards, but that just to draft the idea.
2. A one liner fix


Inconsistent caching behaviour when using `Dataset.map()` with a `new_fingerprint` and `num_proc>1`

Something isn't working

Following the discussion in #3045 if would be nice to have a way to let users have a nice experience with caching even if the function is not hashable.

Currently a workaround is to make the function picklable. This can be done by implementing a callable class instead, that can be pickled using by implementing a custom `__getstate__` method for example.

However it sounds pretty complicated for a simple thing. Maybe one idea would be to have something similar to streamlit: they allow users to register the hashing of their own objects.

See the documentation about their `hash_funcs` here: https://docs.streamlit.io/library/advanced-features/caching#the-hash_funcs-parameter

Here is the example they give:

```python
class FileReference:
    def __init__(self, filename):
        self.filename = filename

def hash_file_reference(file_reference):
    filename = file_reference.filename
    return (filename, os.path.getmtime(filename))

@st.cache(hash_funcs={FileReference: hash_file_reference})
def func(file_reference):
    ...
```

My solution was to generate a custom hash, and use the hash as a `new_fingerprint` argument to the `map()` method to enable caching. This works, but is quite hacky.

@lhoestq, this approach is very neat, this would make the whole caching mechanic more explicit. I don't have so much time to look into this right now, but I might give it a try in the future.

## Describe the bug
Caching does not work when using `Dataset.map()` with:
1. a function that cannot be deterministically fingerprinted
2. `num_proc>1`
3.  using a custom fingerprint set with the argument `new_fingerprint`.

This means that the dataset will be mapped with the function for each and every call, which does not happen if `num_proc==1`. In that case (`num_proc==1`) subsequent calls will load the transformed dataset from the cache, which is the  expected behaviour. The example can easily be translated into a unit test.

I have a fix and will submit a pull request asap.

## Steps to reproduce the bug
```python
import hashlib
import json
import os
from typing import Dict, Any

import numpy as np
from datasets import load_dataset, Dataset

Batch = Dict[str, Any]
filename = 'example.json'

class Transformation():
    """A transformation with a random state that cannot be fingerprinted"""

    def __init__(self):
        self.state = np.random.random()

    def __call__(self, batch: Batch) -> Batch:
        batch['x'] = [np.random.random() for _ in batch['x']]
        return batch

def generate_dataset():
    """generate a simple dataset"""
    rgn = np.random.RandomState(24)
    data = {
        'data': [{'x': float(y), 'y': -float(y)} for y in
                 rgn.random(size=(1000,))]}
    if not os.path.exists(filename):
        with open(filename, 'w') as f:
            f.write(json.dumps(data))

    return filename

def process_dataset_with_cache(num_proc=1, remove_cache=False,
                               cache_expected_to_exist=False):

    # load the generated dataset
    dset: Dataset = next(
        iter(load_dataset('json', data_files=filename, field='data').values()))
    new_fingerprint = hashlib.md5("static-id".encode("utf8")).hexdigest()

    # get the expected cached path
    cache_path = dset._get_cache_file_path(new_fingerprint)
    if remove_cache and os.path.exists(cache_path):
        os.remove(cache_path)

     # check that the cache exists, and print a statement
    # if was actually expected to exist
    cache_exist = os.path.exists(cache_path)
    print(f"> cache file exists={cache_exist}")
    if cache_expected_to_exist and not cache_exist:
        print("=== Cache does not exist! ====")

    # apply the transformation with the new fingerprint
    dset = dset.map(
        Transformation(),
        batched=True,
        num_proc=num_proc,
        new_fingerprint=new_fingerprint,
        desc="mapping dataset with transformation")

generate_dataset()

for num_proc in [1, 2]:
    print(f"# num_proc={num_proc}, first pass")
    # first pass to generate the cache (always create a new cache here)
    process_dataset_with_cache(remove_cache=True,
                               num_proc=num_proc,
                               cache_expected_to_exist=False)
    print(f"# num_proc={num_proc}, second pass")
    # second pass, expects the cache to exist
    process_dataset_with_cache(remove_cache=False,
                               num_proc=num_proc,
                               cache_expected_to_exist=True)

os.remove(filename)

```

## Expected results
In the above python example, with `num_proc=2`, the **cache file should exist in the second call** of `process_dataset_with_cache`  ("=== Cache does not exist! ====" should not be printed).
When the cache is successfully created, `map()` is called only one time.

## Actual results
In the above python example, with `num_proc=2`, the **cache does not exist in the second call** of `process_dataset_with_cache` (this results in printing "=== Cache does not exist! ====").
Because the cache doesn't exist, the `map()` method is executed a second time and the dataset is not loaded from the cache.

## Environment info
- `datasets` version: 1.12.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 5.0.0


Add PASS dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** PASS
- **Description:** An ImageNet replacement for self-supervised pretraining without humans
- **Data:** https://www.robots.ox.ac.uk/~vgg/research/pass/ https://github.com/yukimasano/PASS

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Improving elasticsearch integration

@lhoestq @albertvillanova Iwas trying to fix the failing tests in circleCI but is there a test elasticsearch instance somewhere? If not, can I launch a docker container to have one?

- adding murmurhash signature to sample in index
- adding optional credentials for remote elasticsearch server
- enabling sample update in index
- upgrade the elasticsearch 7.10.1 python client
- adding ElasticsearchBulider to instantiate a dataset from an index and a filtering query


Load private data files + use glob on ZIP archives for json/csv/etc. module inference

I have an error on windows:
```python
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host moon-staging.huggingface.co:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')]
```
at the `fsspec` call in `xglob`:
```python
fs, *_ = fsspec.get_fs_token_paths(urlpath, storage_options=storage_options)
```

Looks like the windows CI has an SSL issue...

I can reproduce it on my windows machine. On linux it works fine though

I'm just skipping the windows test for now

The Windows CI failure seems unrelated to this PR
```python
ERROR tests/test_arrow_dataset.py::test_dummy_dataset_serialize_s3
```

As mentioned in https://github.com/huggingface/datasets/issues/3032 loading data files from private repository isn't working correctly because of the data files resolved.

#2986 did a refactor of the data files resolver. I added authentication to it.

I also improved it to glob inside ZIP archives to look for json/csv/etc. files and infer which dataset builder (json/csv/etc.) to use.

Fix https://github.com/huggingface/datasets/issues/3032

Note that #2986 needs to get merged first


[save_to_disk] Using `select()` followed by `save_to_disk` saves complete dataset making it hard to create dummy dataset

bug

Something isn't working

Hi,

the `save_to_disk` docstring explains that `flatten_indices` has to be called on a dataset before saving it to save only the shard/slice of the dataset.

That works! Thansk!

Might be worth doing that automatically actually in case the `save_to_disk` is called on a dataset that has an indices mapping :-)

I agree with @patrickvonplaten: this issue is reported recurrently, so better if we implement the `.flatten_indices()` automatically?

That would be great indeed - I don't really see a use case where one would not like to call `.flatten_indices()` before calling `save_to_disk`

## Describe the bug

When only keeping a dummy size of a dataset (say the first 100 samples), and then saving it  to disk to upload it in the following to the hub for easy demo/use - not just the small dataset is saved but the whole dataset with an indices file. The problem with this is that the dataset is still very big.

## Steps to reproduce the bug

E.g. run the following:

```python
from datasets import load_dataset, save_to_disk

nlp = load_dataset("glue", "mnli", split="train")
nlp.save_to_disk("full")

nlp = nlp.select(range(100))
nlp.save_to_disk("dummy")
```

Now one can see that both `"dummy"` and `"full"` have the same size. This shouldn't be the case IMO.

## Expected results

IMO `"dummy"` should be much smaller so that one can easily play around with the dataset on the hub.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.12.2.dev0
- Platform: Linux-5.11.0-34-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 5.0.0


Add sberquad dataset


add sberquad dataset


SberQuad


Protect master branch to force contributions via Pull Requests

enhancement

New feature or request

It would be nice to protect the master from direct commits, but still having a way to merge our own PRs when no review is required (for example when updating a dataset_infos.json file, or minor bug fixes - things that happen quite often actually).
Do you know if there's a way ?

This is done. Now the master branch is protected:
- [x] Require a pull request before merging: all commits must be made to a non-protected branch and submitted via a pull request
  - Required number of approvals before merging: 1
- [x] Require linear history: prevent merge commits from being pushed
- [x] These requirements are not enforced for administrators
- [x] Additionally, the master branch is also protected against deletion and force pushes

CC: @lhoestq @julien-c @thomwolf

In order to have a clearer Git history in the master branch, I propose to protect it so that all contributions must be done through a Pull Request and no direct commits to master are allowed.
- The Pull Request allows to give context, discuss any potential issues and improve the quality of the contribution
- The Pull Request will eventually be squashed and merged into master with a single commit that links to the Pull Request page (with all the context/discussions)

Note that we already implemented a protection in the master branch to avoid *merge* commits and ensure a linear history. This proposal goes one step further by avoiding all kind of direct commits and forcing contributions **only** through Pull Requests.

Please note that we can temporarily deactivate this protection if we need to make a direct commit, e.g. at each new version release.

The only way GitHub allows this kind or protection is by requiring a minimal number (at least one) of approvals of the Pull Request. The inconvenient is that the PR creator cannot approve their own PR: another person must approve it before it can be merged into master. To circumvent this, we could eventually disable this protection in the master branch when an urgent commit is needed (e.g. for a hotfix) and there is no other person available at that time to approve the PR.


`load_dataset` does not work with uploaded arrow file

New feature or request

Hi ! This is not a bug, this is simply not implemented.
`save_to_disk` is for on-disk serialization and was not made compatible for the Hub.
That being said, I agree we actually should make it work with the Hub x)

cc @LysandreJik maybe we can solve this at the same time as adding `push_to_hub`

## Describe the bug

I've preprocessed and uploaded a dataset here: https://huggingface.co/datasets/ami-wav2vec2/ami_headset_single_preprocessed . The dataset is in `.arrow` format.

The dataset can correctly be loaded when doing:

```bash
git lfs install
git clone https://huggingface.co/datasets/ami-wav2vec2/ami_headset_single_preprocessed
```

followed by

```python
from datasets import load_from_disk

ds = load_from_disk("./ami_headset_single_preprocessed")
```

However when I try to directly download the dataset as follows:

```python
from datasets import load_dataset

ds = load_dataset("ami-wav2vec2/ami_headset_single_preprocessed")
```

the following error occurs:

```bash
/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
   1115         ignore_verifications=ignore_verifications,
   1116         try_from_hf_gcs=try_from_hf_gcs,
-> 1117         use_auth_token=use_auth_token,
   1118     )
   1119

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    635                     if not downloaded_from_gcs:
    636                         self._download_and_prepare(
--> 637                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    638                         )
    639                     # Sync info

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    724             try:
    725                 # Prepare split will record examples associated to the split
--> 726                 self._prepare_split(split_generator, **prepare_split_kwargs)
    727             except OSError as e:
    728                 raise OSError(

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _prepare_split(self, split_generator)
   1186                 generator, unit=" tables", leave=False, disable=bool(logging.get_verbosity() == logging.NOTSET)
   1187             ):
-> 1188                 writer.write_table(table)
   1189             num_examples, num_bytes = writer.finalize()
   1190

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in write_table(self, pa_table, writer_batch_size)
    424         # reorder the arrays if necessary + cast to self._schema
    425         # we can't simply use .cast here because we may need to change the order of the columns
--> 426         pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
    427         batches: List[pa.RecordBatch] = pa_table.to_batches(max_chunksize=writer_batch_size)
    428         self._num_bytes += sum(batch.nbytes for batch in batches)

/usr/local/lib/python3.7/dist-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_arrays()

/usr/local/lib/python3.7/dist-packages/pyarrow/table.pxi in pyarrow.lib._sanitize_arrays()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.asarray()

/usr/local/lib/python3.7/dist-packages/pyarrow/table.pxi in pyarrow.lib.ChunkedArray.cast()

/usr/local/lib/python3.7/dist-packages/pyarrow/compute.py in cast(arr, target_type, safe)
    279     else:
    280         options = CastOptions.unsafe(target_type)
--> 281     return call_function("cast", [arr], options)
    282
    283

/usr/local/lib/python3.7/dist-packages/pyarrow/_compute.pyx in pyarrow._compute.call_function()

/usr/local/lib/python3.7/dist-packages/pyarrow/_compute.pyx in pyarrow._compute.Function.call()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Unsupported cast from struct<train: struct<name: string, num_bytes: int64, num_examples: int64, dataset_name: string>, validation: struct<name: string, num_bytes: int64, num_examples: int64, dataset_name: string>, test: struct<name: string, num_bytes: int64, num_examples: int64, dataset_name: string>> to list using function cast_list
```

## Expected results

The dataset should be correctly loaded with `load_dataset` IMO.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.2.dev0
- Platform: Linux-5.11.0-34-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 5.0.0


Errors loading dataset using fs = a gcsfs.GCSFileSystem

bug

Something isn't working

## Describe the bug

Cannot load dataset using a `gcsfs.GCSFileSystem`.  I'm not sure if this should be a bug in `gcsfs` or here...

Basically what seems to be happening is that since datasets saves datasets as folders and folders aren't "real objects" in gcs, gcsfs raises a 404 error.  There are workarounds if you use gcsfs directly to download the file, but as is I can't get `load_from_disk` to work.

## Steps to reproduce the bug
```python
from datasets import load_dataset
# load some dataset
dataset = load_dataset("squad", split="train")

# save it to gcs
import gcsfs
fs = gcsfs.GCSFileSystem(project="my-gs-project")
dataset.save_to_disk("gs://my-bucket/squad", fs=fs)

# try to load it from gcs
from datasets import load_from_disk
dataset2 = load_from_disk("my-bucket/squad", fs=fs)
```

## Expected results
`dataset2` would be a copy of `dataset` but loaded from my bucket.

## Actual results
Long traceback but essentially it's a 404 error from gcsfs saying the object `my-bucket/squad` doesn't exist when this is called:
https://github.com/huggingface/datasets/blob/9c81b7d2e6d9feae69a084a3abda265a4ca07fb5/src/datasets/arrow_dataset.py#L977

This is because there is no actual object called `my-bucket/squad`, there are objects called `my-bucket/squad/dataset.arrow`, etc.

Note that *this* works fine, since it's explicitly saying "download all the objects with this prefix":
```python
            fs.download(src_dataset_path + "/*", dataset_path.as_posix(), recursive=True)
```
For example, I can do a workaround this way:
```python
import tempfile

with tempfile.TemporaryDirectory() as temppath:
    fs.download("gs://my-bucket/squad/*", temppath)
    dataset2 = load_from_disk(temppath)
```

It's unclear to me if it's `gcsfs`'s responsibility to say "hey that's folder not a file, I should try to get objects inside of it not the object itself", or if that's `datasets`'s responsibility... I'm leaning towards the latter since you're never loading a dataset from one file using this function/method, only a dataset folder?

Another minor thing that should maybe should be rolled into this bug...

https://github.com/huggingface/datasets/blob/9c81b7d2e6d9feae69a084a3abda265a4ca07fb5/src/datasets/arrow_dataset.py#L968

These fail if you pass in a `gs://` path, e.g.

```python
dataset2 = load_from_disk("gs://my-bucket/squad", fs=fs)
```
Because at this point, `dataset_info_path` is `gs:/my-bucket/squad/dataset_info.json`, gcsfs throws a:
```
Invalid bucket name: 'gs:'
```
error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: macOS Big Sur 11.6
- Python version: 3.7.12
- PyArrow version: 5.0.0


Actual "proper" install of ruamel.yaml in the windows CI

It was impossible to update the package directly with `pip`. Indeed it was installed with `distutils` which prevents `pip` or `conda` to uninstall it.

I had to `rm` a directory from the `site-packages` python directory, and then do `pip install ruamel.yaml`

It's not that "proper" but I couldn't find better solutions


Error when loading private dataset with "data_files" arg

Something isn't working

We'll do a release tomorrow or on wednesday to make the fix available :)

Thanks for reproting !

## Describe the bug
A clear and concise description of what the bug is.

Private datasets with no loading script can't be loaded using `data_files` parameter.

## Steps to reproduce the bug
```python
from datasets import load_dataset
data_files = {"train": "**/train/*/*.jsonl", "valid": "**/valid/*/*.jsonl"}
dataset = load_dataset('dalle-mini/encoded', data_files=data_files, use_auth_token=True, streaming=True)
```

Same error happens in non-streaming mode.

## Expected results
Files should be loaded (whether in streaming or not).

## Actual results
Error:

```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, return_associated_base_path, data_files, **download_kwargs)
    539                 try:
--> 540                     local_path = cached_path(file_path, download_config=download_config)
    541                 except FileNotFoundError:

8 frames
FileNotFoundError: Couldn't find file at https://huggingface.co/datasets/dalle-mini/encoded/resolve/main/encoded.py

During handling of the above exception, another exception occurred:

HTTPError                                 Traceback (most recent call last)
HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/datasets/dalle-mini/encoded?full=true

During handling of the above exception, another exception occurred:

FileNotFoundError                         Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, return_associated_base_path, data_files, **download_kwargs)
    547                     except Exception:
    548                         raise FileNotFoundError(
--> 549                             f"Couldn't find a directory or a {resource_type} named '{path}'. "
    550                             f"It doesn't exist locally at {expected_dir_for_combined_path_abs} or remotely on {hf_api.endpoint}/datasets"
    551                         )

FileNotFoundError: Couldn't find a directory or a dataset named 'dalle-mini/encoded'. It doesn't exist locally at /content/dalle-mini/encoded or remotely on https://huggingface.co/datasets
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0

@lhoestq


Align tqdm control with cache control

Could you add this function to the documentation please ?

You can add it in `main_classes.rst`, and maybe add a `Tip` section in the `map` section in the `process.rst`

Currently, once disabled with `disable_progress_bar`, progress bars cannot be re-enabled again. To overcome this limitation, this PR introduces the `set_progress_bar_enabled` function that accepts a boolean indicating whether to display progress bars. The goal is to provide a similar API to the existing cache control API. Following the Zen of Python (😄), there should be one and preferably only one obvious way to do it, so I'm also deprecating the aforementioned `disable_progress_bar` function. Additionally, I justify the deprecation with the fact that this function has never been in the docs.

Moreover, similar API changes have recently been introduced to [`tfds`](https://github.com/tensorflow/datasets/blob/a1e8b98f45b0214082b546cc967c67c43fffda55/tensorflow_datasets/core/utils/tqdm_utils.py#L98-L112).

Considering the popularity of the [comment](https://github.com/huggingface/datasets/issues/1627#issuecomment-751383559) I made a while ago, this API (`set_progress_bar_enabled` and `is_progress_bar_enabled`) should be mentioned in the docs, but I'm not sure where to put it exactly. Maybe we can replace the `logging_methods` page under `package_reference` with `utility_methods` and then introduce two subsections on that page: `Logging methods` and `tqdm control`.

Additionally, this PR:
* adds the `disable_tqdm` keyword arg of `Dataset._map_single` to the `ignore_kwargs` list to ignore it when computing the fingerprint (forgot to add it in #2696)
* deletes the unused components in `tqdm_utils.py`, which seem to be inherited from `tfds`
* disables the tqdm output in the test suite. As I see it, this output doesn't seem informative, but let me know if this is not a good idea


Add `remove_columns` to `IterableDataset`

Thanks ! That looks all good :)

I don't think that batching would help. Indeed we're dealing with python iterators that yield elements one by one, so batched `map` needs to accumulate a batch, apply the function, and then yield examples from the batch.

Though once we have parallel processing in `map`, we can reconsider it

Also feel free to check the CI failure - apparently the import of `Union` is missing

Thanks for the review and explaining that!
On top of what you said, I think `remove_columns` is very unlikely to be a bottleneck, so it doesn't matter anyways.

Thank you for reviewing! @mariosasko

I wonder how the checking would work. Is there any checking present in `IterableDataset ` now? What if `.remove_columns()` is applied after some arbitrary `.map()`?

> I wonder how the checking would work. Is there any checking present in IterableDataset now? What if .remove_columns() is applied after some arbitrary .map()?

That's the challenge here indeed ^^ In this case it's not trivial to know the names of the columns. Feel free to open an issue so we can discuss this

Fixes #2944

WIP
* Not tested yet.
* We might want to allow batched remove for efficiency.

@lhoestq Do you think it should have `batched=` and `batch_size=`?


Use standard open-domain validation split in nq_open

I had to run datasets-cli with --ignore_verifications the first time since it was complaining about a missing file, but now it runs without that flag fine. I moved dummy_data.zip to the new folder, but also had to modify the filename of the test file in the zip (should I not have done that?). Finally, I added the pretty name tag.

The nq_open dataset originally drew the validation set from this file:
https://github.com/google-research-datasets/natural-questions/blob/master/nq_open/NQ-open.efficientqa.dev.1.1.sample.jsonl
However, that's the dev set used specifically and only for the efficientqa competition, and it's not the same dev set as is used in every open-domain question answering paper (including the Lee et al paper that introduced the open-domain variant of NQ, cited at the top of the dataset file). This PR changes nq_open to use the standard validation split and bumps the version to 2.0.0 since this is a breaking change.


Properly install ruamel-yaml for windows CI

@lhoestq I would say this does not "properly" install `ruamel-yaml`, but the contrary, you overwrite the previous version without desinstalling it first.

According to `pip` docs:
>  This can break your system if the existing package is of a different version or was installed with a different package manager!

Note that our case fulfills both conditions:
- the installing version (`0.17.16`) is different from the existing one (`0.15.87`)
- you are installing using `pip` (`setuptools`), whereas the exisitng version was installed using `distutils`

That is why I did not fix the issue this way, made a hotfix pinning `huggingface_hub` (#3025), while looking for a permanent solution for the issue.

Yea I did this because we need the latest version of `huggingface_hub` for #2986 and because I didn't want to ssh to the windows worker x)
Maybe it can be fixed by installing it with conda - let me try

Oh yea it may not work since it was first installed with distutils


Resolve data_files by split name

Awesome @lhoestq I like the proposal and it works great on my JSON community dataset. Here is the [log](https://gist.github.com/vblagoje/714babc325bcbdd5de579fd8e1648892).

From my discussion with @borisdayma it would be more general the files match if their paths contains the split name - not only if the filename contains the split name. For example for a dataset like this:
```
train/
└── data.csv
test/
└── data.csv
```

But IMO the default should be
```
data/
├── train.csv
└── test.csv
```
because it allows people to have other directories if they have different subsets of their data (different configurations, not splits)

I just created a PR for this at https://github.com/huggingface/datasets/pull/3221, let me know what you think :)

This issue is about discussing the default behavior when someone loads a dataset that consists in data files. For example:
```python
load_dataset("lhoestq/demo1")
```
should return two splits "train" and "test" since the dataset repostiory is like
```
data/
├── train.csv
└── test.csv
```
Currently it returns only one split "train" which contains the data of both files

I started playing with this idea on this branch btw: `resolve-data_files-by-split-name`
Basically the idea is that if you named you data files after split names then the default pattern is
```python
{
    "train": ["*train*"],
    "test": ["*test*"],
    "validation": ["*dev*", "valid"],
}
```
otherwise it's
```python
{
    "train": ["*"]
}
```

Let me know what you think !
cc @albertvillanova @LysandreJik @vblagoje


added arxiv paper inswiss_judgment_prediction dataset card


Fix Windows test suite

Try a hotfix to restore Windows test suite.

Fix #3024.


Windows test suite fails

Something isn't working

## Describe the bug

There is an error during installation of tests dependencies for Windows: https://app.circleci.com/pipelines/github/huggingface/datasets/7981/workflows/9b6a0114-2b8e-4069-94e5-e844dbbdba4e/jobs/49206

```
ERROR: Cannot uninstall 'ruamel-yaml'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
```


Fix typo


MeDAL dataset: Add further description and update download URL

@lhoestq I'm a bit confused by the error message. I haven't touched the YAML code at all - do you have any insight on that?

I just added the missing `pretty_name` tag in the YAML - sorry about that ;)

Thanks! Seems like it did the trick since the tests are passing. Let me know if there's anything else I can do in this PR!

Added more details in the following sections:
* Dataset Structure
* Data Instances
* Data Splits
* Source Data
* Annotations
* Discussions of Biases
* LIcensing Information


Support loading dataset from multiple zipped CSV data files

Fix partially #3018.

CC: @lewtun