Fix fsspec AbstractFileSystem access

This addresses the issue from #2914 by changing the way fsspec's AbstractFileSystem is accessed.


Having a dependency defining fsspec entrypoint raises an AttributeError when importing datasets

bug

Something isn't working

## Describe the bug
In one of my project, I defined a custom fsspec filesystem with an entrypoint.
My guess is that by doing so, a variable named `spec` is created in the module `fsspec` (created by entering a for loop as there are entrypoints defined, see the loop in question [here](https://github.com/intake/filesystem_spec/blob/0589358d8a029ed6b60d031018f52be2eb721291/fsspec/__init__.py#L55)).
So that `fsspec.spec`, that was previously referring to the `spec` submodule, is now referring to that `spec` variable.
This make the import of datasets failing as it is using that `fsspec.spec`.

## Steps to reproduce the bug
I could reproduce the bug with a dummy poetry project.

Here is the pyproject.toml:
```toml
[tool.poetry]
name = "debug-datasets"
version = "0.1.0"
description = ""
authors = ["Pierre Godard"]

[tool.poetry.dependencies]
python = "^3.8"
datasets = "^1.11.0"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.plugins."fsspec.specs"]
"file2" = "fsspec.implementations.local.LocalFileSystem"
```

The only other file being a `debug_datasets/__init__.py` empty file.

The overall structure of the project is as follows:
```
.
├── pyproject.toml
└── debug_datasets
    └── __init__.py
```

Then, within the project folder run:

```
poetry install
poetry run python
```

And in the python interpreter, try to import `datasets`:

```
import datasets
```

## Expected results
The import should run successfully.

## Actual results

Here is the trace of the error I get:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/godarpi/.cache/pypoetry/virtualenvs/debug-datasets-JuFzTKL--py3.8/lib/python3.8/site-packages/datasets/__init__.py", line 33, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/home/godarpi/.cache/pypoetry/virtualenvs/debug-datasets-JuFzTKL--py3.8/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 48, in <module>
    from .filesystems import extract_path_from_uri, is_remote_filesystem
  File "/home/godarpi/.cache/pypoetry/virtualenvs/debug-datasets-JuFzTKL--py3.8/lib/python3.8/site-packages/datasets/filesystems/__init__.py", line 30, in <module>
    def is_remote_filesystem(fs: fsspec.spec.AbstractFileSystem) -> bool:
AttributeError: 'EntryPoint' object has no attribute 'AbstractFileSystem'
```

## Suggested fix

`datasets/filesystems/__init__.py`, line 30, replace:
```
    def is_remote_filesystem(fs: fsspec.spec.AbstractFileSystem) -> bool:
```
by:
```
    def is_remote_filesystem(fs: fsspec.AbstractFileSystem) -> bool:
```

I will come up with a PR soon if this effectively solves the issue.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.11.0
- Platform: WSL2 (Ubuntu 20.04.1 LTS)
- Python version: 3.8.5
- PyArrow version: 5.0.0
- `fsspec` version: 2021.8.1


timit_asr dataset only includes one text phrase

Something isn't working

Hi @margotwagner,
This bug was fixed in #1995. Upgrading the datasets should work (min v1.8.0 ideally)

Hi @margotwagner,

Yes, as @bhavitvyamalik has commented, this bug was fixed in `datasets` version 1.5.0. You need to update it, as your current version is 1.4.1:
> Environment info
> - `datasets` version: 1.4.1

## Describe the bug
The dataset 'timit_asr' only includes one text phrase. It only includes the transcription "Would such an act of refusal be useful?" multiple times rather than different phrases.

## Steps to reproduce the bug
Note: I am following the tutorial https://huggingface.co/blog/fine-tune-wav2vec2-english

1. Install the dataset and other packages
```python
!pip install datasets>=1.5.0
!pip install transformers==4.4.0
!pip install soundfile
!pip install jiwer
```
2. Load the dataset
```python
from datasets import load_dataset, load_metric

timit = load_dataset("timit_asr")
```
3. Remove columns that we don't want
```python
timit = timit.remove_columns(["phonetic_detail", "word_detail", "dialect_region", "id", "sentence_type", "speaker_id"])
```
4. Write a short function to display some random samples of the dataset.
```python
from datasets import ClassLabel
import random
import pandas as pd
from IPython.display import display, HTML

def show_random_elements(dataset, num_examples=10):
    assert num_examples <= len(dataset), "Can't pick more elements than there are in the dataset."
    picks = []
    for _ in range(num_examples):
        pick = random.randint(0, len(dataset)-1)
        while pick in picks:
            pick = random.randint(0, len(dataset)-1)
        picks.append(pick)

    df = pd.DataFrame(dataset[picks])
    display(HTML(df.to_html()))

show_random_elements(timit["train"].remove_columns(["file"]))
```

## Expected results
10 random different transcription phrases.

## Actual results
10 of the same transcription phrase "Would such an act of refusal be useful?"

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.4.1
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyArrow version: not listed


Update link to Blog in docs footer

Update link.


Fix exception chaining

Fix exception chaining to avoid tracebacks with message: `During handling of the above exception, another exception occurred:`


feat: 🎸 pass additional arguments to get private configs + info

`use_auth_token` can now be passed to the functions to get the configs
or infos of private datasets on the hub


fix anli splits

I can't run the tests for dummy data, facing this error

`ImportError while loading conftest '/home/zaid/tmp/fix_anli_splits/datasets/tests/conftest.py'.
tests/conftest.py:10: in <module>
    from datasets import config
E   ImportError: cannot import name 'config' from 'datasets' (unknown location)`


Update Zenodo metadata with creator names and affiliation

This PR helps in prefilling author data when automatically generating the DOI after each release.


add story_cloze dataset

Will create a new one, this one seems to be missed up.

@lhoestq I have spent some time but I still I can't succeed in correctly testing the dummy_data.


feat: 🎸 add a function to get a dataset config's split names

> Should I add a section in https://github.com/huggingface/datasets/blob/master/docs/source/load_hub.rst? (there is no section for get_dataset_infos)

Yes totally :) This tutorial should indeed mention this, given how fundamental it is

Also: pass additional arguments (use_auth_token) to get private configs + info of private datasets on the hub

Questions:

- [x] I'm not sure how the versions work: I changed 1.12.1.dev0 to 1.12.1.dev1, was it correct?
    -> no: reverted
- [x] Should I add a section in https://github.com/huggingface/datasets/blob/master/docs/source/load_hub.rst? (there is no section for get_dataset_infos)
    -> yes: added


Update BibTeX entry

Update BibTeX entry.


FORCE_REDOWNLOAD does not work

Something isn't working

Hi ! Thanks for reporting. The error seems to happen only if you use compressed files.

The second dataset is prepared in another dataset cache directory than the first - which is normal, since the source file is different. However, it doesn't uncompress the new data file because it finds the old uncompressed data in the extraction cache directory.

If we fix the extraction cache mechanism to uncompress a local file if it changed then it should fix the issue.
Currently the extraction cache mechanism only takes into account the path of the compressed file, which is an issue.

Facing the same issue, is there any way to overtake this issue until it will be fixed?

You can clear your extraction cache in the meantime (by default at `~/.cache/huggingface/datasets/downloads/extracted`)

## Describe the bug
With GenerateMode.FORCE_REDOWNLOAD, the documentation says
    +------------------------------------+-----------+---------+
    |                                    | Downloads | Dataset |
    +====================================+===========+=========+
    | `REUSE_DATASET_IF_EXISTS` (default)| Reuse     | Reuse   |
    +------------------------------------+-----------+---------+
    | `REUSE_CACHE_IF_EXISTS`            | Reuse     | Fresh   |
    +------------------------------------+-----------+---------+
    | `FORCE_REDOWNLOAD`                 | Fresh     | Fresh   |
    +------------------------------------+-----------+---------+

However, the old dataset is loaded even when FORCE_REDOWNLOAD is chosen.

## Steps to reproduce the bug
```python

import pandas as pd
from datasets import load_dataset, GenerateMode
pd.DataFrame(range(5), columns=['numbers']).to_csv('/tmp/test.tsv.gz', index=False)
ee = load_dataset('csv', data_files=['/tmp/test.tsv.gz'], delimiter='\t', split='train', download_mode=GenerateMode.FORCE_REDOWNLOAD)
print(ee)
pd.DataFrame(range(10), columns=['numerals']).to_csv('/tmp/test.tsv.gz', index=False)
ee = load_dataset('csv', data_files=['/tmp/test.tsv.gz'], delimiter='\t', split='train', download_mode=GenerateMode.FORCE_REDOWNLOAD)
print(ee)

```

## Expected results
Dataset({
    features: ['numbers'],
    num_rows: 5
})
Dataset({
    features: ['numerals'],
    num_rows: 10
})

## Actual results
Dataset({
    features: ['numbers'],
    num_rows: 5
})
Dataset({
    features: ['numbers'],
    num_rows: 5
})

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-4.14.181-108.257.amzn1.x86_64-x86_64-with-glibc2.10
- Python version: 3.7.10
- PyArrow version: 3.0.0


Fix xpathopen to accept positional arguments

Fix `xpathopen()` so that it also accepts positional arguments.

Fix #2901.


Add WIT Dataset

dataset request

Requesting to add a new dataset

WikiMedia is now hosting the pixel values directly which should make it a lot easier!
The files can be found here:
https://techblog.wikimedia.org/2021/09/09/the-wikipedia-image-caption-matching-challenge-and-a-huge-release-of-image-data-for-research/
https://analytics.wikimedia.org/published/datasets/one-off/caption_competition/training/image_pixels/

> @hassiahk is working on it #2810

Thank you @bhavitvyamalik! Added this issue so we could track progress 😄 . Just linked the PR as well for visibility.

Hey folks, we are now hosting the merged pixel values + embeddings + metadata ourselves. I gave it a try - [nateraw/wit](https://huggingface.co/datasets/nateraw/wit)

**⚠ - Make sure you add `streaming=True` unless you're prepared to download 400GB of data!**

```python
from datasets import load_dataset

ds = load_dataset('nateraw/wit', streaming=True)
example = next(iter(ds))
```

```python
>>> example = next(iter(ds['train']))
>>> example.keys()
dict_keys(['b64_bytes', 'original_width', 'image_url', 'wit_features', 'original_height', 'metadata_url', 'mime_type', 'caption_attribution_description', 'embedding'])
>>> example['wit_features'].keys()
dict_keys(['hierarchical_section_title', 'language', 'attribution_passes_lang_id', 'context_section_description', 'is_main_image', 'page_title', 'caption_title_and_reference_description', 'caption_alt_text_description', 'caption_reference_description', 'page_url', 'context_page_description', 'section_title', 'page_changed_recently'])
```

## Adding a Dataset
- **Name:** *WIT*
- **Description:** *Wikipedia-based Image Text Dataset*
- **Paper:** *[WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
](https://arxiv.org/abs/2103.01913)*
- **Data:** *https://github.com/google-research-datasets/wit*
- **Motivation:**  (excerpt from their Github README.md)

> - The largest multimodal dataset (publicly available at the time of this writing) by the number of image-text examples.
> - A massively multilingual dataset (first of its kind) with coverage for over 100+ languages.
> - A collection of diverse set of concepts and real world entities.
> - Brings forth challenging real-world test sets.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Incompatibility with pytest

bug

Something isn't working

Sorry, my bad... When implementing `xpathopen`, I just considered the use case in the COUNTER dataset... I'm fixing it!

## Describe the bug

pytest complains about xpathopen / path.open("w")

## Steps to reproduce the bug

Create a test file, `test.py`:

```python
import datasets as ds
def load_dataset():
    ds.load_dataset("counter", split="train", streaming=True)
```

And launch it with pytest:

```bash
python -m pytest test.py
```

## Expected results

It should give something like:

```
collected 1 item

test.py .                                                                                [100%]

======= 1 passed in 3.15s =======
```

## Actual results

```
============================================================================================================================= test session starts ==============================================================================================================================
platform linux -- Python 3.8.11, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/slesage/hf/datasets-preview-backend, configfile: pyproject.toml
plugins: anyio-3.3.1
collected 1 item

tests/queries/test_rows.py .                                                                                [100%]Traceback (most recent call last):
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pytest/__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/_pytest/config/__init__.py", line 162, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/_pytest/main.py", line 316, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/_pytest/main.py", line 304, in wrap_session
    config.hook.pytest_sessionfinish(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/_pytest/terminal.py", line 803, in pytest_sessionfinish
    outcome.get_result()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/_pytest/cacheprovider.py", line 428, in pytest_sessionfinish
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/_pytest/cacheprovider.py", line 188, in set
    f = path.open("w")
TypeError: xpathopen() takes 1 positional argument but 2 were given
```

## Environment info

- `datasets` version: 1.12.0
- Platform: Linux-5.11.0-1017-aws-x86_64-with-glibc2.29
- Python version: 3.8.11
- PyArrow version: 4.0.1


Fix null sequence encoding

The Sequence feature encoding was failing when a `None` sequence was used in a dataset.

Fix https://github.com/huggingface/datasets/issues/2892


Dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Hug emoji

Requesting to add a new dataset


Add OpenAI's HumanEval dataset

I just fixed the class name, and added `[More Information Needed]` in empty sections in case people want to complete the dataset card :)

This PR adds OpenAI's [HumanEval](https://github.com/openai/human-eval) dataset. The dataset consists of 164 handcrafted programming problems with solutions and unittests to verify solution. This dataset is useful to evaluate code generation models.


add multi-proc in `to_csv`

I think you can just add a test `test_dataset_to_csv_multiproc` in `tests/io/test_csv.py` and we'll be good

Hi @lhoestq,
I've added `test_dataset_to_csv` apart from `test_dataset_to_csv_multiproc` as no test was there to check generated CSV file when `num_proc=1`. Please let me know if anything is also required!

This PR extends the multi-proc method used in #2747 for`to_json` to `to_csv` as well.

Results on my machine post benchmarking on `ascent_kb` dataset (giving ~45% improvement when compared to num_proc = 1):
```
Time taken on 1 num_proc, 10000 batch_size  674.2055702209473
Time taken on 4 num_proc, 10000 batch_size  425.6553490161896

Time taken on 1 num_proc, 50000 batch_size  623.5897650718689
Time taken on 4 num_proc, 50000 batch_size  380.0402421951294

Time taken on 4 num_proc, 100000 batch_size  361.7168130874634
```
This is a WIP as writing tests is pending for this PR.

I'm also exploring [this](https://arrow.apache.org/docs/python/csv.html#incremental-writing) approach for which I'm using `pyarrow-5.0.0`.


Use pyarrow.Table.replace_schema_metadata instead of pyarrow.Table.cast

This PR partially addresses #2252.

``update_metadata_with_features`` uses ``Table.cast`` which slows down ``load_from_disk`` (and possibly other methods that use it) for very large datasets. Since ``update_metadata_with_features`` is only updating the schema metadata, it makes more sense to use ``pyarrow.Table.replace_schema_metadata`` which is much faster. This PR adds a ``replace_schema_metadata`` method to all table classes, and modifies  ``update_metadata_with_features`` to use it instead of ``cast``.


Fix COUNTER dataset

Fix filename generating `FileNotFoundError`.

Related to #2866.
CC: @severo.


add mbpp dataset

I think it's fine to have the original schema

This PR adds the mbpp dataset introduced by Google [here](https://github.com/google-research/google-research/tree/master/mbpp) as mentioned in #2816.

The dataset contain two versions: a full and a sanitized one. They have a slightly different schema and it is current state the loading preserves the original schema. An open question is whether to harmonize the two schemas when loading the dataset or to preserve the original one. Since not all fields are overlapping the schema will not be exactly the same.


Error when encoding a dataset with None objects with a Sequence feature

bug

Something isn't working

This has been fixed by https://github.com/huggingface/datasets/pull/2900
We're doing a new release 1.12 today to make the fix available :)

There is an error when encoding a dataset with None objects with a Sequence feature

To reproduce:
```python
from datasets import Dataset, Features, Value, Sequence
data = {"a": [[0], None]}
features = Features({"a": Sequence(Value("int32"))})
dataset = Dataset.from_dict(data, features=features)
```
raises

```python
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-24-40add67f8751> in <module>
      2 data = {"a": [[0], None]}
      3 features = Features({"a": Sequence(Value("int32"))})
----> 4 dataset = Dataset.from_dict(data, features=features)
[...]
~/datasets/features.py in encode_nested_example(schema, obj)
    888         if isinstance(obj, str):  # don't interpret a string as a list
    889             raise ValueError("Got a string but expected a list instead: '{}'".format(obj))
--> 890         return [encode_nested_example(schema.feature, o) for o in obj]
    891     # Object with special encoding:
    892     # ClassLabel will convert from string to int, TranslationVariableLanguages does some checks

TypeError: 'NoneType' object is not iterable
```

Instead, if should run without error, as if the `features` were not passed


Allow dynamic first dimension for ArrayXD

@lhoestq, thanks for your review.

I added test for `to_pylist`, I didn't do that for `to_numpy` because this method shouldn't be called for dynamic dimension ArrayXD - this method will try to make a single numpy array for the whole column which cannot be done for dynamic arrays.

I dig into `to_pandas()` functionality and I found it quite difficult to implement. `PandasArrayExtensionArray` takes single np.array as an argument. It might be a bit of changes to make it work with the list of arrays. Do you mind if we exclude this work from this PR. I added an error message for the case if somebody tries to use dynamic arrays with `to_pandas`

@lhoestq, I just fixed all the tests. Let me know if there is anything else to add.

@lhoestq, any chance you had some time to check out this PR?

Hi ! Sorry for the delay

It looks good to me ! I think the only thing missing is the support for passing a list of numpy arrays to `map` when the first dimension is dynamic.

Currently it raises an error:
```python
from datasets import *
import numpy as np

features=  Features({"a": Array3D(shape=(None, 5, 2), dtype="int32")})
d = Dataset.from_dict({"a": [np.zeros((5,5,2)), np.zeros((2,5,2))]}, features=features)
d = d.map(lambda a: {"a": np.concatenate([a]*2)}, input_columns="a")
print(d[0])
```
raises
```python
Traceback (most recent call last):
  File "playground/ttest.py", line 6, in <module>
    d = d.map(lambda x: {"a": np.concatenate([x]*2)}, input_columns="a")
  File "/home/truent/hf/datasets/src/datasets/arrow_dataset.py", line 1932, in map
    return self._map_single(
  File "/home/truent/hf/datasets/src/datasets/arrow_dataset.py", line 426, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/truent/hf/datasets/src/datasets/fingerprint.py", line 406, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/truent/hf/datasets/src/datasets/arrow_dataset.py", line 2317, in _map_single
    writer.finalize()
  File "/home/truent/hf/datasets/src/datasets/arrow_writer.py", line 443, in finalize
    self.write_examples_on_file()
  File "/home/truent/hf/datasets/src/datasets/arrow_writer.py", line 312, in write_examples_on_file
    pa_array = pa.array(typed_sequence)
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/truent/hf/datasets/src/datasets/arrow_writer.py", line 108, in __arrow_array__
    storage = pa.array(self.data, type.storage_dtype)
  File "pyarrow/array.pxi", line 305, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
```

I think the issue is that here we don't cover the case where self.data is a list of numpy arrays:

https://github.com/huggingface/datasets/blob/55fd140a63b8f03a0e72985647e498f1fc799d3f/src/datasets/arrow_writer.py#L104-L109

We should remove the `isinstance(self.data[0], np.ndarray)` part and add these lines to cover this case:

https://github.com/huggingface/datasets/blob/55fd140a63b8f03a0e72985647e498f1fc799d3f/src/datasets/arrow_writer.py#L112-L113

@lhoestq, thanks, good catch!
Are you able to run this check with fixed dimension ArrayXD?
for below example
```
import numpy as np
from datasets import *

features = Features({"a": Array3D(shape=(2, 5, 2), dtype="int32")})
d = Dataset.from_dict({"a": [np.zeros((2, 5, 2)), np.zeros((2, 5, 2))]}, features=features)
d = d.map(lambda a: {"a": np.array(a) + 1}, input_columns="a")
print(d[0])
```

I am getting:
```
  File "/home/ib/datasets/src/datasets/arrow_writer.py", line 116, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
```

Nevertheless, I tried to fix that. Let me know if that works.

@lhoestq, just resolved the conflicts. Let me know if there is anything left to do with this PR

Hi, thanks a lot for your comments.
Agree, happy to contribute to this topic in future PRs

Hi @rpowalski, thanks for adding this feature!

I wanted to check if you are still interested in documenting this, otherwise I'd be happy to help with it :)

Add support for dynamic first dimension for ArrayXD features. See issue [#887](https://github.com/huggingface/datasets/issues/887).
Following changes allow for `to_pylist` method of `ArrayExtensionArray` to return a list of numpy arrays where fist dimension can vary.

@lhoestq Could you suggest how you want to extend test suit. For now I added only very limited testing.


dataset request

Requesting to add a new dataset


Coc

Requesting to add a new dataset


v1.11.1 release date

question

Further information is requested

@albertvillanova i think this issue is still valid and should not be closed till `>1.11.0` is published :)

Hello, i need to use latest features in one of my packages but there have been no new datasets release since 2 months ago.

When do you plan to publush v1.11.1 release?


#2837 Use cache folder for lockfile

The CI fail about the meteor metric is unrelated to this PR

Fixes #2837

Use a cache folder directory to store the FileLock.

The issue was that the lock file was in a readonly folder.


Hj


Adding an Elastic Search index to a Dataset

bug

Something isn't working

Hi, is this bug deterministic in your poetry env ? I mean, does it always stop at 90% or is it random ?

Also, can you try using another version of Elasticsearch ? Maybe there's an issue with the one of you poetry env

I face similar issue with oscar dataset on remote ealsticsearch instance. It was mainly due to timeout of batch indexing requests and I solve these by adding large request_timeout param in `search.py`

```
       for ok, action in es.helpers.streaming_bulk(
            client=self.es_client,
            index=index_name,
            actions=passage_generator(),
            request_timeout=3600,
        )
        ```

Hi @MotzWanted - are there any errors in the Elasticsearch cluster logs? Since it works in your local environment and the cluster versions are different between your poetry env and your local env, it is possible that it is some difference in the cluster - either settings or the cluster being under a different load etc that has this effect, so it would be useful to see if any errors are thrown in the cluster's logs when you try to ingest.
Which elasticsearch client method is the function `add_elasticsearch_index` from your code using under the hood? Is it `helpers.bulk` or is the indexing performed using something else? You can try adding a timeout to the indexing method to see if this helps. Also, you mention that it stops at around 90%  - do you know if the timeout/hanging happens always when a particular document is being indexed or does it happen randomly at around 90% completeness but on different documents?

## Describe the bug
When trying to index documents from the squad dataset, the connection to ElasticSearch seems to break:

Reusing dataset squad (/Users/andreasmotz/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)
 90%|████████████████████████████████████████████▉     | 9501/10570 [00:01<00:00, 6335.61docs/s]

No error is thrown, but the indexing breaks ~90%.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset
from elasticsearch import Elasticsearch
es = Elasticsearch()
squad = load_dataset('squad', split='validation')
index_name = "corpus"
es_config = {
    "settings": {
        "number_of_shards": 1,
        "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}},
    },
    "mappings": {
        "properties": {
            "idx" : {"type" : "keyword"},
            "title" : {"type" : "keyword"},
            "text": {
                "type": "text",
                "analyzer": "standard",
                "similarity": "BM25"
            },
        }
    },
}
class IndexBuilder:
    """
    Elastic search indexing of a corpus
    """
    def __init__(
        self,
        *args,
        #corpus : None,
        dataset : squad,
        index_name = str,
        query = str,
        config = dict,
        **kwargs,
    ):
        #instantiate HuggingFace dataset
        self.dataset = dataset
        #instantiate ElasticSearch config
        self.config = config
        self.es = Elasticsearch()
        self.index_name = index_name
        self.query = query
    def elastic_index(self):
        print(self.es.info)
        self.es.indices.delete(index=self.index_name, ignore=[400, 404])
        search_index = self.dataset.add_elasticsearch_index(column='context', host='localhost', port='9200', es_index_name=self.index_name, es_index_config=self.config)
        return search_index
    def exact_match_method(self, index):
        scores, retrieved_examples = index.get_nearest_examples('context', query=self.query, k=1)
        return scores, retrieved_examples
if __name__ == "__main__":
    print(type(squad))
    Index = IndexBuilder(dataset=squad, index_name='corpus_index', query='Where was Chopin born?', config=es_config)
    search_index = Index.elastic_index()
    scores, examples = Index.exact_match_method(search_index)
    print(scores, examples)
    for name in squad.column_names:
        print(type(squad[name]))
```

## Environment info
We run the code in Poetry. This might be the issue, since the script runs successfully in our local environment.

Poetry:
- Python version: 3.8
- PyArrow: 4.0.1
- Elasticsearch: 7.13.4
- datasets: 1.10.2

Local:
- Python version: 3.8
- PyArrow: 3.0.0
- Elasticsearch: 7.7.1
- datasets: 1.7.0


Add IC, SI, ER tasks to SUPERB

Sorry for the late PR, uploading 10+GB files to the hub through a VPN was an adventure :sweat_smile:

Thank you so much for adding these subsets @anton-l!

> These datasets either require manual downloads or have broken/unstable links. You can get all necessary archives in this repo: https://huggingface.co/datasets/anton-l/superb_source_data_dumps/tree/main
Are we allowed to make these datasets public or would that violate the terms of their use?

@lewtun These ones all have non-permissive licences, so the mirrored data I linked is open only to the HF org for now. But we could try contacting the authors to ask if they'd like to host these with us.
For example VoxCeleb1 now has direct links (the ones in the script) that don't require form submission and passwords, but they ban IPs after each download for some reason :(

> @lewtun These ones all have non-permissive licences, so the mirrored data I linked is open only to the HF org for now. But we could try contacting the authors to ask if they'd like to host these with us.
> For example VoxCeleb1 now has direct links (the ones in the script) that don't require form submission and passwords, but they ban IPs after each download for some reason :(

I think there would be a lot of value added if the authors would be willing to host their data on the HF Hub! As an end-user of `datasets`, I've found I'm more likely to explore a dataset if I'm able to quickly pull the subsets without needing a manual download. Perhaps we can tell them that the Hub offers several advantages like versioning and interactive exploration (with `datasets-viewer`)?

This PR adds 3 additional classification tasks to SUPERB

#### Intent Classification
Dataset URL seems to be down at the moment :( See the note below.
S3PRL source: https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/fluent_commands/dataset.py
Instructions: https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#ic-intent-classification---fluent-speech-commands

#### Speaker Identification
Manual download script:
```
mkdir VoxCeleb1
cd VoxCeleb1

wget https://thor.robots.ox.ac.uk/~vgg/data/voxceleb/vox1a/vox1_dev_wav_partaa
wget https://thor.robots.ox.ac.uk/~vgg/data/voxceleb/vox1a/vox1_dev_wav_partab
wget https://thor.robots.ox.ac.uk/~vgg/data/voxceleb/vox1a/vox1_dev_wav_partac
wget https://thor.robots.ox.ac.uk/~vgg/data/voxceleb/vox1a/vox1_dev_wav_partad
cat vox1_dev* > vox1_dev_wav.zip
unzip vox1_dev_wav.zip

wget https://thor.robots.ox.ac.uk/~vgg/data/voxceleb/vox1a/vox1_test_wav.zip
unzip vox1_test_wav.zip

# download the official SUPERB train-dev-test split
wget https://raw.githubusercontent.com/s3prl/s3prl/master/s3prl/downstream/voxceleb1/veri_test_class.txt
```
S3PRL source: https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/voxceleb1/dataset.py
Instructions: https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sid-speaker-identification

#### Intent Classification
Manual download requires going through a slow application process, see the note below.
S3PRL source: https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/emotion/IEMOCAP_preprocess.py
Instructions: https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#er-emotion-recognition

#### :warning:  Note
These datasets either require manual downloads or have broken/unstable links. You can get all necessary archives in this repo: https://huggingface.co/datasets/anton-l/superb_source_data_dumps/tree/main


Fix data URLs and metadata in DocRED dataset

The host of `docred` dataset has updated the `dev` data file. This PR:
- Updates the dev URL
- Updates dataset metadata

This PR also fixes the URL of the `train_distant` split, which was wrong.

Fix #2882.


`load_dataset('docred')` results in a `NonMatchingChecksumError`

Something isn't working

Hi @tmpr, thanks for reporting.

Two weeks ago (23th Aug), the host of the source `docred` dataset updated one of the files (`dev.json`): you can see it [here](https://drive.google.com/drive/folders/1c5-0YwnoJx8NS6CV2f-NoTHR__BdkNqw).

Therefore, the checksum needs to be updated.

Normally, in the meantime, you could avoid the error by passing `ignore_verifications=True` to `load_dataset`. However, as the old link points to a non-existing file, the link must be updated too.

I'm fixing all this.

## Describe the bug
I get consistent `NonMatchingChecksumError: Checksums didn't match for dataset source files` errors when trying to execute `datasets.load_dataset('docred')`.

## Steps to reproduce the bug
It is quasi only this code:
```python
import datasets
data = datasets.load_dataset('docred')
```

## Expected results
The DocRED dataset should be loaded without any problems.

## Actual results
```
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-4-b1b83f25a16c> in <module>
----> 1 d = datasets.load_dataset('docred')

~/anaconda3/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
    845
    846     # Download and prepare data
--> 847     builder_instance.download_and_prepare(
    848         download_config=download_config,
    849         download_mode=download_mode,

~/anaconda3/lib/python3.8/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    613                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    614                     if not downloaded_from_gcs:
--> 615                         self._download_and_prepare(
    616                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    617                         )

~/anaconda3/lib/python3.8/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    673         # Checksums verification
    674         if verify_infos:
--> 675             verify_checksums(
    676                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    677             )

~/anaconda3/lib/python3.8/site-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1fDmfUUo5G7gfaoqWWvK81u08m71TK2g7']
```

## Environment info
- `datasets` version: 1.11.0
- Platform: Linux-5.11.0-7633-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 5.0.0

This error also happened on my Windows-partition, after freshly installing python 3.9 and `datasets`.

## Remarks

- I have already called `rm -rf /home/<user>/.cache/huggingface`, i.e., I have tried clearing the cache.
- The problem does not exist for other datasets, i.e., it seems to be DocRED-specific.


Add BIOSSES dataset

Adding the biomedical semantic sentence similarity dataset, BIOSSES, listed in "Biomedical Datasets - BigScience Workshop 2021"


Extend support for streaming datasets that use pathlib.Path stem/suffix

This PR extends the support in streaming mode for datasets that use `pathlib`, by patching the properties `pathlib.Path.stem` and `pathlib.Path.suffix`.

Related to #2876, #2874, #2866.

CC: @severo


In v1.4.1, all TIMIT train transcripts are "Would such an act of refusal be useful?"

Something isn't working

Hi @rcgale, thanks for reporting.

Please note that this bug was fixed on `datasets` version 1.5.0: https://github.com/huggingface/datasets/commit/a23c73e526e1c30263834164f16f1fdf76722c8c#diff-f12a7a42d4673bb6c2ca5a40c92c29eb4fe3475908c84fd4ce4fad5dc2514878

If you update `datasets` version, that should work.

On the other hand, would it be possible for @patrickvonplaten to update the [blog post](https://huggingface.co/blog/fine-tune-wav2vec2-english) with the correct version of `datasets`?

I just proposed a change in the blog post.

I had assumed there was a data format change that broke a previous version of the code, since presumably @patrickvonplaten tested the tutorial with the version they explicitly referenced. But that fix you linked suggests a problem in the code, which surprised me.

I still wonder, though, is there a way for downloads to be invalidated server-side? If the client can announce its version during a download request, perhaps the server could reject known incompatibilities? It would save much valuable time if `datasets` raised an informative error on a known problem ("Error: the requested data set requires `datasets>=1.5.0`."). This kind of API versioning is a prudent move anyhow, as there will surely come a time when you'll need to make a breaking change to data.

Also, thank you for a quick and helpful reply!

## Describe the bug
Using version 1.4.1 of `datasets`, TIMIT transcripts are all the same.

## Steps to reproduce the bug
I was following this tutorial
- https://huggingface.co/blog/fine-tune-wav2vec2-english

But here's a distilled repro:
```python
!pip install datasets==1.4.1
from datasets import load_dataset
timit = load_dataset("timit_asr", cache_dir="./temp")
unique_transcripts = set(timit["train"]["text"])
print(unique_transcripts)
assert len(unique_transcripts) > 1
```
## Expected results
Expected the correct TIMIT data. Or an error saying that this version of `datasets` can't produce it.

## Actual results
Every train transcript was "Would such an act of refusal be useful?" Every test transcript was "The bungalow was pleasantly situated near the shore."

## Environment info
- `datasets` version: 1.4.1
- Platform: Darwin-18.7.0-x86_64-i386-64bit
- Python version: 3.7.9
- PyTorch version (GPU?): 1.9.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: tried both
- Using distributed or parallel set-up in script?: no
-


NotADirectoryError: [WinError 267] During load_from_disk

Something isn't working

## Describe the bug
Trying to load saved dataset or dataset directory from Amazon S3 on a Windows machine fails.
Performing the same operation succeeds on non-windows environment (AWS Sagemaker).

## Steps to reproduce the bug
```python
# Followed https://huggingface.co/docs/datasets/filesystems.html#loading-a-processed-dataset-from-s3

from datasets import load_from_disk
from datasets.filesystems import S3FileSystem

s3_file = "output of save_to_disk"

s3_filesystem = S3FileSystem()

load_from_disk(s3_file, fs=s3_filesystem)
```

## Expected results
load_from_disk succeeds without error

## Actual results
Seems like it succeeds in pulling the file into a windows temp directory, as it exists in my system, but fails to process it.
```
Exception ignored in: <finalize object at 0x26409231ce0; dead>
Traceback (most recent call last):
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\weakref.py", line 566, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\tempfile.py", line 817, in _cleanup
    cls._rmtree(name)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\tempfile.py", line 813, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 613, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 613, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 613, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 2 more times]
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 618, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\tempfile.py", line 805, in onerror
    cls._rmtree(path)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\tempfile.py", line 813, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 599, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 596, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\grassycup\\AppData\\Local\\Temp\\tmp45f_qbma\\tests3bucket\\output\\test_output\\train\\dataset.arrow'
Exception ignored in: <finalize object at 0x264091c7880; dead>
Traceback (most recent call last):
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\weakref.py", line 566, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\tempfile.py", line 817, in _cleanup
    cls._rmtree(name)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\tempfile.py", line 813, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 613, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 613, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 613, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 2 more times]
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 618, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\tempfile.py", line 805, in onerror
    cls._rmtree(path)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\tempfile.py", line 813, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 599, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\Users\grassycup\Anaconda3\envs\hello.world\lib\shutil.py", line 596, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] The directory name is invalid:
'C:\\Users\\grassycup\\AppData\\Local\\Temp\\tmp45f_qbma\\tests3bucket\\output\\test_output\\train\\dataset.arrow'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.11.0
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.8.11
- PyArrow version: 3.0.0


Don't keep the dummy data folder or dataset_infos.json when resolving data files

enhancement

New feature or request

Hi @lhoestq I am new to huggingface datasets, I would like to work on this issue!

Thanks for the help :)

As mentioned in the PR, excluding files named "dummy_data.zip" is actually more general than excluding the files inside a "dummy" folder. I just did the change in the PR, I think we can merge it now

When there's no dataset script, all the data files of a folder or a repository on the Hub are loaded as data files.

There are already a few exceptions:
- files starting with "." are ignored
- the dataset card "README.md" is ignored
- any file named "config.json" is ignored (currently it isn't used anywhere, but it could be used in the future to define splits or configs for example, but not 100% sure)

However any data files in a folder named "dummy" should be ignored as well as they should only be used to test the dataset.
Same for "dataset_infos.json" which should only be used to get the `dataset.info`


Extend support for streaming datasets that use pathlib.Path.glob

I am thinking that ideally we should call `fs.glob()` instead...

Thanks, @lhoestq: the idea of adding the mock filesystem is to avoid network calls and reduce testing time ;)

I have added `rglob` as well and fixed some bugs.

This PR extends the support in streaming mode for datasets that use `pathlib`, by patching the method `pathlib.Path.glob`.

Related to #2874, #2866.

CC: @severo


Add Congolese Swahili speech datasets

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Congolese Swahili speech corpora
- **Data:** https://gamayun.translatorswb.org/data/

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Also related: https://mobile.twitter.com/OktemAlp/status/1435196393631764482


Support streaming datasets that use pathlib

I've tried https://github.com/huggingface/datasets/issues/2866 again, and I get the same error.

```python
import datasets as ds
ds.load_dataset('counter', split="train", streaming=False)
```

@severo Issue #2866 is not fully fixed yet: multiple patches need to be implemented for `pathlib`, as that dataset uses quite a lot of `pathlib` functions... 😅

No worry and no stress, I just wanted to check for that case :) I'm very happy that you're working on issues I'm interested in!

This PR extends the support in streaming mode for datasets that use `pathlib.Path`.

Related to: #2866.
CC: @severo


adding swedish_medical_ner

Hi, what's the current status of this request? It says Changes requested, but I can't see what changes?

Hi, it looks like this PR includes changes to other files that `swedish_medical_ner`.

Feel free to remove these changes, or simply create a new PR that only contains the addition of the dataset

Adding the Swedish Medical NER dataset, listed in "Biomedical Datasets - BigScience Workshop 2021"

Code refactored


adding swedish_medical_ner

Adding the Swedish Medical NER dataset, listed in "Biomedical Datasets - BigScience Workshop 2021"


datasets.config.PYARROW_VERSION has no attribute 'major'

bug

Something isn't working

I have changed line 288 to `if int(datasets.config.PYARROW_VERSION.split(".")[0]) < 3:` just to get around it.

Hi @bwang482,

I'm sorry but I'm not able to reproduce your bug.

Please note that in our current master branch, we made a commit (d03223d4d64b89e76b48b00602aba5aa2f817f1e) that simultaneously modified:
- test_dataset_common.py: https://github.com/huggingface/datasets/commit/d03223d4d64b89e76b48b00602aba5aa2f817f1e#diff-a1bc225bd9a5bade373d1f140e24d09cbbdc97971c2f73bb627daaa803ada002L289 that introduces the usage of `datasets.config.PYARROW_VERSION.major`
- but also changed config.py: https://github.com/huggingface/datasets/commit/d03223d4d64b89e76b48b00602aba5aa2f817f1e#diff-e021fcfc41811fb970fab889b8d245e68382bca8208e63eaafc9a396a336f8f2L40, so that `datasets.config.PYARROW_VERSION.major` exists

Reopening this. Although the `test_dataset_common.py` script works fine now.

Has this got something to do with my pull request not passing `ci/circleci: run_dataset_script_tests_pyarrow` tests?

https://github.com/huggingface/datasets/pull/2873

Hi @bwang482,

If you click on `Details` (on the right of your non passing CI test names: `ci/circleci: run_dataset_script_tests_pyarrow`), you can have more information about the non-passing tests.

For example, for ["ci/circleci: run_dataset_script_tests_pyarrow_1" details](https://circleci.com/gh/huggingface/datasets/46324?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link), you can see that the only non-passing test has to do with the dataset card (missing information in the `README.md` file): `test_changed_dataset_card`
```
=========================== short test summary info ============================
FAILED tests/test_dataset_cards.py::test_changed_dataset_card[swedish_medical_ner]
= 1 failed, 3214 passed, 2874 skipped, 2 xfailed, 1 xpassed, 15 warnings in 175.59s (0:02:55) =
```

Therefore, your PR non-passing test has nothing to do with this issue.

In the test_dataset_common.py script, line 288-289

```
if datasets.config.PYARROW_VERSION.major < 3:
   packaged_datasets = [pd for pd in packaged_datasets if pd["dataset_name"] != "parquet"]
```

which throws the error below. `datasets.config.PYARROW_VERSION` itself return the string '4.0.1'. I have tested this on both datasets.__version_=='1.11.0' and '1.9.0'. I am using Mac OS.

```
import datasets
datasets.config.PYARROW_VERSION.major
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/var/folders/1f/0wqmlgp90qjd5mpj53fnjq440000gn/T/ipykernel_73361/2547517336.py in <module>
      1 import datasets
----> 2 datasets.config.PYARROW_VERSION.major

AttributeError: 'str' object has no attribute 'major'
```

## Environment info
- `datasets` version: 1.11.0
- Platform: Darwin-20.6.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyArrow version: 4.0.1


Fix three typos in two files for documentation

Changed "bacth_size" to "batch_size" (2x)
Changed "intsructions" to "instructions"


Something isn't working

Hi, @Chenfei-Kang.

I'm sorry, but I'm not able to reproduce your bug:
```python
from datasets import load_dataset

ds = load_dataset("glue", 'cola')
ds
```
```
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 8551
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1043
    })
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1063
    })
})
```

Could you please give more details and environment info (platform, PyArrow version)?

> Hi, @Chenfei-Kang.
>
> I'm sorry, but I'm not able to reproduce your bug:
>
> ```python
> from datasets import load_dataset
>
> ds = load_dataset("glue", 'cola')
> ds
> ```
>
> ```
> DatasetDict({
>     train: Dataset({
>         features: ['sentence', 'label', 'idx'],
>         num_rows: 8551
>     })
>     validation: Dataset({
>         features: ['sentence', 'label', 'idx'],
>         num_rows: 1043
>     })
>     test: Dataset({
>         features: ['sentence', 'label', 'idx'],
>         num_rows: 1063
>     })
> })
> ```
>
> Could you please give more details and environment info (platform, PyArrow version)?

Sorry to reply you so late.
platform: pycharm 2021 + anaconda with python 3.7
PyArrow version: 5.0.0
huggingface-hub: 0.0.16
datasets: 1.9.0

- For the platform, we need to know the operating system of your machine. Could you please run the command `datasets-cli env` and copy-and-paste its output below?
- In relation with the error, you just gave us the error type and message (`TypeError: 'NoneType' object is not callable`). Could you please copy-paste the complete stack trace, so that we know exactly which part of the code threw the error?

> * For the platform, we need to know the operating system of your machine. Could you please run the command `datasets-cli env` and copy-and-paste its output below?
> * In relation with the error, you just gave us the error type and message (`TypeError: 'NoneType' object is not callable`). Could you please copy-paste the complete stack trace, so that we know exactly which part of the code threw the error?

1. For the platform, here are the output:
        - datasets` version: 1.11.0
        - Platform: Windows-10-10.0.19041-SP0
        - Python version: 3.7.10
        - PyArrow version: 5.0.0
2. For the code and error：
     ```python
     from datasets import load_dataset, load_metric
     dataset = load_dataset("glue", "cola")
    ```
    ```python
    Traceback (most recent call last):
    ....
    ....
    File "my_file.py", line 2, in <module>
    dataset = load_dataset("glue", "cola")
    File "My environments\lib\site-packages\datasets\load.py", line 830, in load_dataset
    **config_kwargs,
     File "My environments\lib\site-packages\datasets\load.py", line 710, in load_dataset_builder
    **config_kwargs,
    TypeError: 'NoneType' object is not callable
    ```
   Thank you!

One naive question: do you have internet access from the machine where you execute the code?

> For that environment, I am sorry but I can't reproduce the bug: I can load the dataset without any problem.

But I can download other task dataset such as `dataset = load_dataset('squad')`. I don't know what went wrong.  Thank you so much!

## Describe the bug

TypeError: 'NoneType' object is not callable
## Steps to reproduce the bug
```python
from datasets import load_dataset, load_metric
dataset = datasets.load_dataset("glue", 'cola')
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.11.0
- Platform:
- Python version: 3.7
- PyArrow version:


Add Common Objects in 3D (CO3D)

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Common Objects in 3D (CO3D)*
- **Description:** *See blog post [here](https://ai.facebook.com/blog/common-objects-in-3d-dataset-for-3d-reconstruction)*
- **Paper:** *[link to paper](https://arxiv.org/abs/2109.00512)*
- **Data:** *[link to data](https://ai.facebook.com/datasets/co3d-downloads/)*
- **Motivation:** *excerpt from above blog post:*

> As the first data set of its kind, CO3D will aptly enable reconstruction of real-life 3D objects. Indeed, CO3D already provides training data to enable our NeRFormer to tackle the new-view synthesis (NVS) task. Here, photorealistic NVS is a major step on the path to fully immersive AR/VR effects, where objects can be virtually transported across different environments, which will allow connecting users by sharing or recollecting their experiences.
>
> Besides practical applications in AR/VR, we hope that the data set will become a standard testbed for the recent proliferation of methods (including NeRFormer, Implicit Differentiable Renderer, NeRF, and others) that reconstruct 3D scenes by means of an implicit shape model.
>

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add CaSiNo dataset

Hi @lhoestq

Just a request to look at the dataset. Please let me know if any changes are necessary before merging it into the repo. Thank you.

Hey @lhoestq

Thanks for merging it. One question: I still cannot find the dataset on https://huggingface.co/datasets. Does it take some time or did I miss something?

Hi ! It takes a few hours or a day for the list of datasets on the website to be updated ;)

Hi. I request you to add our dataset to the repository.

This data was recently published at NAACL 2021: https://aclanthology.org/2021.naacl-main.254.pdf


"counter" dataset raises an error in normal mode, but not in streaming mode

bug

Something isn't working

Hi @severo, thanks for reporting.

Just note that currently not all canonical datasets support streaming mode: this is one case!

All datasets that use `pathlib` joins (using `/`) instead of `os.path.join` (as in this dataset) do not support streaming mode yet.

OK. Do you think it's possible to detect this, and raise an exception (maybe `NotImplementedError`, or a specific `StreamingError`)?

We should definitely support datasets using `pathlib` in streaming mode...

For non-supported datasets in streaming mode, we have already a request of raising an error/warning: see #2654.

Hi @severo, please note that "counter" dataset will be streamable (at least until it arrives at the missing file, error already in normal mode) once these PRs are merged:
- #2874
- #2876
- #2880

I have tested it. 😉

Now (on master), we get:

```
import datasets as ds
ds.load_dataset('counter', split="train", streaming=False)
```

```
Using custom data configuration default
Downloading and preparing dataset counter/default (download: 1.29 MiB, generated: 2.48 MiB, post-processed: Unknown size, total: 3.77 MiB) to /home/slesage/.cache/huggingface/datasets/counter/default/1.0.0/9f84962fa0f35bec5a34fe0bdff8681838d497008c457f7856c48654476ec0e9...
Traceback (most recent call last):
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 726, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 1124, in _prepare_split
    for key, record in utils.tqdm(
  File "/home/slesage/hf/datasets/.venv/lib/python3.8/site-packages/tqdm/std.py", line 1185, in __iter__
    for obj in iterable:
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/counter/9f84962fa0f35bec5a34fe0bdff8681838d497008c457f7856c48654476ec0e9/counter.py", line 161, in _generate_examples
    with derived_file.open(encoding="utf-8") as f:
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/pathlib.py", line 1222, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/pathlib.py", line 1078, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/slesage/.cache/huggingface/datasets/downloads/extracted/b57aa6db5601a738e57b95c1fd8cced54ff28fc540efcdaf0f6c4f1bb5dfe211/COUNTER/0032p.xml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets/src/datasets/load.py", line 1112, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 636, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets/src/datasets/builder.py", line 728, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: '/home/slesage/.cache/huggingface/datasets/downloads/extracted/b57aa6db5601a738e57b95c1fd8cced54ff28fc540efcdaf0f6c4f1bb5dfe211/COUNTER/0032p.xml'
```

The error is now the same with or without streaming. I close the issue, thanks @albertvillanova and @lhoestq!

Note that we might want to open an issue to fix the "counter" dataset by itself, but I let it up to you.

Fixed here: https://github.com/huggingface/datasets/pull/2894. Thanks @albertvillanova

On master, I get:

```python
>>> import datasets as ds
>>> iterable_dataset = ds.load_dataset('counter', split="train", streaming=True)
>>> rows = list(iterable_dataset.take(100))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets/src/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets/src/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets/src/datasets/iterable_dataset.py", line 273, in __iter__
    yield from islice(self.ex_iterable, self.n)
  File "/home/slesage/hf/datasets/src/datasets/iterable_dataset.py", line 78, in __iter__
    for key, example in self.generate_examples_fn(**self.kwargs):
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/counter/b9e4378dbd3f5ce235d2302e48168c00196e67bbcd13cc7e1f6e69ef82c0cf2a/counter.py", line 153, in _generate_examples
    files = sorted(base_path.glob(r"[0-9][0-9][0-9][0-9].xml"))
TypeError: xpathglob() missing 1 required positional argument: 'pattern'
```

Associated to the above exception, if I create a test and run it with pytest, I get an awful traceback.

- create a file `test_counter.py`

```python
import pytest
from datasets import load_dataset, IterableDataset
from typing import Any, cast

def test_counter() -> Any:
    iterable_dataset = cast(IterableDataset, load_dataset("counter", split="train", streaming=True))
    with pytest.raises(TypeError):
        list(iterable_dataset.take(100))
```

- run the test with pytest

```bash
$ python -m pytest -x test_counter.py
============================================================================================================================= test session starts ==============================================================================================================================
platform linux -- Python 3.9.6, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/slesage/hf/datasets-preview-backend, configfile: pyproject.toml
plugins: anyio-3.3.2, cov-2.12.1
collected 1 item

tests/test_counter.py .                                                                                [100%]Traceback (most recent call last):
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pytest/__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 162, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/_pytest/main.py", line 316, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/_pytest/main.py", line 304, in wrap_session
    config.hook.pytest_sessionfinish(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/_pytest/terminal.py", line 803, in pytest_sessionfinish
    outcome.get_result()
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/_pytest/cacheprovider.py", line 428, in pytest_sessionfinish
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/_pytest/cacheprovider.py", line 188, in set
    f = path.open("w")
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 199, in xpathopen
    return xopen(_as_posix(path), *args, **kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 117, in _as_posix
    path_as_posix = path.as_posix()
AttributeError: 'str' object has no attribute 'as_posix'
```

I opened a PR to fix these issues.
Also in your test you expect a TypeError but I don't know why. On my side it works fine without raising a TypeError

I had the issue (TypeError raised) on my branch, but it's fixed now. Thanks

## Describe the bug

`counter` dataset raises an error on `load_dataset()`, but simply returns an empty iterator in streaming mode.

## Steps to reproduce the bug

```python
>>> import datasets as ds
>>> a = ds.load_dataset('counter', split="train", streaming=False)
Using custom data configuration default
Downloading and preparing dataset counter/default (download: 1.29 MiB, generated: 2.48 MiB, post-processed: Unknown size, total: 3.77 MiB) to /home/slesage/.cache/huggingface/datasets/counter/default/1.0.0/9f84962fa0f35bec5a34fe0bdff8681838d497008c457f7856c48654476ec0e9...
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/datasets/builder.py", line 726, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/datasets/builder.py", line 1124, in _prepare_split
    for key, record in utils.tqdm(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/tqdm/std.py", line 1185, in __iter__
    for obj in iterable:
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/counter/9f84962fa0f35bec5a34fe0bdff8681838d497008c457f7856c48654476ec0e9/counter.py", line 161, in _generate_examples
    with derived_file.open(encoding="utf-8") as f:
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/pathlib.py", line 1222, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/home/slesage/.pyenv/versions/3.8.11/lib/python3.8/pathlib.py", line 1078, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/slesage/.cache/huggingface/datasets/downloads/extracted/b57aa6db5601a738e57b95c1fd8cced54ff28fc540efcdaf0f6c4f1bb5dfe211/COUNTER/0032p.xml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/datasets/load.py", line 1112, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/datasets/builder.py", line 636, in download_and_prepare
    self._download_and_prepare(
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.8/site-packages/datasets/builder.py", line 728, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: '/home/slesage/.cache/huggingface/datasets/downloads/extracted/b57aa6db5601a738e57b95c1fd8cced54ff28fc540efcdaf0f6c4f1bb5dfe211/COUNTER/0032p.xml'
```

```python
>>> import datasets as ds
>>> b = ds.load_dataset('counter', split="train", streaming=True)
Using custom data configuration default
>>> list(b)
[]
```

## Expected results

An exception should be raised in streaming mode

## Actual results

No exception is raised in streaming mode: there is no way to tell if something has broken or if the dataset is simply empty.

## Environment info

- `datasets` version: 1.11.1.dev0
- Platform: Linux-5.11.0-1016-aws-x86_64-with-glibc2.29
- Python version: 3.8.11
- PyArrow version: 4.0.1


Add MultiEURLEX dataset

Hi @lhoestq, we have this new cool multilingual dataset coming at EMNLP 2021. It would be really nice if we could have it in Hugging Face asap. Thanks!

Hi @lhoestq, I adopted most of your suggestions:

- Dummy data files reduced, including the 2 smallest documents per subset JSONL.
- README was updated with the publication URL and instructions on how to download and use label descriptors. Excessive newlines were deleted.

I would prefer to keep the label list in a pure format (original ids), to enable people to combine those with more information or possibly in the future explore the dataset, find inconsistencies and fix those to release a new version.

Thanks for the changes :)

Regarding the labels:

If you use the ClassLabel feature type, the only change is that it will store the ids as integers instead of (currently) string.
The advantage is that if people want to know what id corresponds to which label name, they can use `classlabel.int2str`. It is also the format that helps automate model training for classification in `transformers`.

Let me know if that sounds good to you or if you still want to stick with the labels as they are now.

Hey @lhoestq, thanks for providing this information. This sounds great. I updated my code accordingly to use `ClassLabel`. Could you please provide a minimal example of how `classlabel.int2str` works in practice in my case, where labels are a sequence?

```python
from datasets import load_dataset
dataset = load_dataset('multi_eurlex', 'all_languages')
# Read strs from the labels (list of integers) for the 1st sample of the training split
```

I would like to include this in the README file.

Could you also provide some info on how I could define the supervized key to automate model training, as you said?

Thanks!

Thanks for the update :)

Here is an example of usage:
```python
from datasets import load_dataset
dataset = load_dataset('multi_eurlex', 'all_languages', split='train')
classlabel = dataset.features["labels"].feature
print(dataset[0]["labels"])
# [1, 20, 7, 3, 0]
print(classlabel.int2str(dataset[0]["labels"]))
# ['100160', '100155', '100158', '100147', '100149']
```

The ClassLabel is simply used to define the `id2label` dictionary of classification models, to make the ids match between the model and the dataset. There nothing more to do :p

I think one last thing to do is just update the `dataset_infos.json` file and we'll be good !

**Add new MultiEURLEX Dataset**

MultiEURLEX comprises 65k EU laws in 23 official EU languages (some low-ish resource). Each EU law has been annotated with EUROVOC concepts (labels) by the Publication Office of EU. As with the English EURLEX, the goal is to predict the relevant EUROVOC concepts (labels); this is multi-label classification task (given the text, predict multiple labels).


Fix data URL in ToTTo dataset

1.12

Next minor release

Data source host changed their data URL: google-research-datasets/ToTTo@cebeb43.

Fix #2860.


Update dataset URL

Superseded by PR #2864.

@mrm8488 next time you would like to work on an issue, you can first self-assign it to you (by writing `#self-assign` in a comment on the issue). That way, other people can see you are already working on it and there are not multiple people working on the same issue. 😉


Only retain relevant statistics in certain metrics

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
As I understand, in the `add_batch()` function, the raw predictions and references are kept (in memory?) until `compute()` is called.
https://github.com/huggingface/datasets/blob/e248247518140d5b0527ce2843a1a327e2902059/src/datasets/metric.py#L423-L442

This takes O(n) memory. However, for many (most?) metrics, this is not necessary. E.g., for accuracy, only the # correct and # total need to be recorded.

**Describe the solution you'd like**
Probably an inheritance hierarchy where `"predictions"` and `"references"` are not always the two keys for the final metric computation. Each metric should create and maintain its own relevant statistics, again for example, `"n_correct"` and `"n_total"` for accuracy.

I believe the metrics in AllenNLP (https://github.com/allenai/allennlp/tree/39c40fe38cd2fd36b3465b0b3c031f54ec824160/allennlp/training/metrics) can be used as a good reference.

**Describe alternatives you've considered**
At least `Metric.compute()` shouldn't hard-code `"predictions"` and `"references"` so that custom subclasses may override this behavior.
https://github.com/huggingface/datasets/blob/e248247518140d5b0527ce2843a1a327e2902059/src/datasets/metric.py#L399-L400


fix: 🐛 be more specific when catching exceptions

To give more context: after our discussion, if I understood properly, you are trying to fix a call to `datasets` that takes 15 minutes: https://github.com/huggingface/datasets-preview-backend/issues/17 Is this right?

Yes, that's it. And to do that I'm trying to use https://pypi.org/project/stopit/, which will raise a stopit.TimeoutException exception. But currently, if this exception is raised, it's caught and considered as a "FileNotFoundError" while it should not be caught.

And what about passing the `timeout` parameter instead?

It might be a good idea, but I would have to add a timeout argument to several methods, I'm not sure we want that (I want to ensure all my queries in https://github.com/huggingface/datasets-preview-backend/tree/master/src/datasets_preview_backend/queries resolve in a given time, be it with an error in case of timeout, or with the successful response). The methods are `prepare_module`, `import_main_class`, *builder_cls.*`get_all_exported_dataset_infos`, `load_dataset_builder`, and `load_dataset`

I understand, you are trying to find a fix for your use case. OK.

Just note that it is also an issue for `datasets` users. Once #2859 fixed in `datasets`, you will no longer have this issue...

Closing, since 1. my problem is more #2859, and I was asking for that change in order to make a hack work on my side, 2. if we want to change how exceptions are handled, we surely want to do it on all the codebase, not only in this particular case.

The same specific exception is catched in other parts of the same
function.


Cannot download TOTTO dataset

bug

Something isn't working

Hola @mrm8488, thanks for reporting.

Apparently, the data source host changed their URL one week ago: https://github.com/google-research-datasets/ToTTo/commit/cebeb430ec2a97747e704d16a9354f7d9073ff8f

I'm fixing it.

Error: Couldn't find file at https://storage.googleapis.com/totto/totto_data.zip

`datasets version: 1.11.0`
# How to reproduce:

```py
from datasets import load_dataset
dataset = load_dataset('totto')
```


Loading allenai/c4 in streaming mode does too many HEAD requests

enhancement

New feature or request

This does 60,000+ HEAD requests to get all the ETags of all the data files:
```python
from datasets import load_dataset
load_dataset("allenai/c4", streaming=True)
```
It makes loading the dataset completely impractical.

The ETags are used to compute the config id (it must depend on the data files being used).
Instead of using the ETags, we could simply use the commit hash of the dataset repository on the hub, as well and the glob pattern used to resolve the files (here it's `*` by default, to load all the files of the repository)


Fix s3fs version in CI

The latest s3fs version has new constrains on aiobotocore, and therefore on boto3 and botocore

This PR changes the constrains to avoid the new conflicts

In particular it pins the version of s3fs.


Update: Openwebtext - update size

merging since the CI error in unrelated to this PR and fixed on master

Update the size of the Openwebtext dataset

I also regenerated the dataset_infos.json but the data file checksum didn't change, and the number of examples either (8013769 examples)

related to #2839


fix: 🐛 remove URL's query string only if it's ?dl=1

A lot of URL use the query strings, for example
http://opus.nlpl.eu/download.php?f=Bianet/v1/moses/en-ku.txt.zip, we
must not remove it when trying to detect the protocol. We thus remove it
only in the case of the query string being ?dl=1 which occurs on dropbox
and dl.orangedox.com. Also: add unit tests.

See https://github.com/huggingface/datasets/pull/2843 for the original
discussion.


Fix windows CI CondaError

From this thread: https://github.com/conda/conda/issues/6057

We can fix the conda error
```
CondaError: Cannot link a source that does not exist.
C:\Users\...\Anaconda3\Scripts\conda.exe
```

by doing
```bash
conda update conda
```

before doing any install in the windows CI


Fix caching when moving script

Merging since the CI failure is unrelated to this PR

When caching the result of a `map` function, the hash that is computed depends on many properties of this function, such as all the python objects it uses, its code and also the location of this code.

Using the full path of the python script for the location of the code makes the hash change if a script like `run_mlm.py` is moved.

I changed this by simply using the base name of the script instead of the full path.

Note that this change also affects the hash of the code used from imported modules, but I think it's fine. Indeed it hashes the code of the imported modules anyway, so the location of the python files of the imported modules doesn't matter when computing the hash.

Close https://github.com/huggingface/datasets/issues/2825


Add AMI dataset

Hey @cahya-wirawan,

I played around with the dataset a bit and it looks already very good to me! That's exactly how it should be constructed :-) I can help you a bit with defining the config, etc... on Monday!

@lhoestq - I think the dataset is ready to be merged :-)

At the moment, I don't really see how the failing tests correspond to this PR:
- https://app.circleci.com/pipelines/github/huggingface/datasets/7838/workflows/932a40a2-3e11-48be-84f0-c6434510058e/jobs/48318?invite=true#step-107-18
- https://app.circleci.com/pipelines/github/huggingface/datasets/7838/workflows/932a40a2-3e11-48be-84f0-c6434510058e/jobs/48316?invite=true#step-102-136

could you maybe give it a look? :-)

This is an initial commit for AMI dataset


Fix: linnaeus - fix url

Merging since the CI error is unrelated this this PR

The url was causing a `ConnectionError` because of the "/" at the end

Close https://github.com/huggingface/datasets/issues/2821


Update `column_names` showed as `:func:` in exploring.st

Hi,

One mention of `column_names` in exploring.st was showing it as `:func:` instead of `:attr:`.


Wound segmentation datasets

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Wound segmentation datasets
- **Description:** annotated wound image dataset
- **Paper:** https://www.nature.com/articles/s41598-020-78799-w
- **Data:** https://github.com/uwm-bigdata/wound-segmentation
- **Motivation:** Interesting simple image dataset, useful for segmentation, with visibility due to http://www.miccai.org/special-interest-groups/challenges/ and https://fusc.grand-challenge.org/

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add Open Catalyst Project Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:**  Open Catalyst 2020 (OC20) Dataset
- **Website:** https://opencatalystproject.org/
- **Data:** https://github.com/Open-Catalyst-Project/ocp/blob/master/DATASET.md

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Update README.md

Merging since the CI error is unrelated to this PR and fixed on master

Changed 'Tain' to 'Train'.


fix regex to accept negative timezone

fix #2846


Negative timezone

bug

Something isn't working

## Describe the bug
The load_dataset method do not accept a parquet file with a negative timezone, as it has the following regex:
```
"^(s|ms|us|ns),\s*tz=([a-zA-Z0-9/_+:]*)$"
```
So a valid timestap ```timestamp[us, tz=-03:00]``` returns an error when loading parquet files.

## Steps to reproduce the bug
```python
# Where the timestamp column has a tz of -03:00
datasets = load_dataset('parquet', data_files={'train': train_files, 'validation': validation_files,
                                        'test': test_files}, cache_dir="./cache_teste/")
```

## Expected results
The -03:00 is a valid tz so the regex should accept this without raising an error.

## Actual results
As this regex disaproves a valid tz it raises the following error:
```python
raise ValueError(
                f"{datasets_dtype} is not a validly formatted string representation of a pyarrow timestamp."
                f"Examples include timestamp[us] or timestamp[us, tz=America/New_York]"
                f"See: https://arrow.apache.org/docs/python/generated/pyarrow.timestamp.html#pyarrow.timestamp"
            )
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.11.0
- Platform: Ubuntu 20.04
- Python version: 3.8
- PyArrow version: 5.0.0


[feature request] adding easy to remember `datasets.cache_dataset()` + `datasets.is_dataset_cached()`

enhancement

New feature or request

Often, there is a need to prepare a dataset but not use it immediately, e.g. think tests suite setup, so it'd be really useful to be able to do:

```
if not datasets.is_dataset_cached(ds): datasets.cache_dataset(ds)
```

This can already be done with:
```
builder = load_dataset_builder(ds)
if not os.path.idsir(builder.cache_dir):
    builder.download_and_prepare()
```

but the current way is a way less intuitive and much harder to remember than the proposed API, IMHO.

One more way is to do:

```
_ = load_dataset(ds)
```
but it wastes resources loading the dataset when it's not needed.

this has been discussed at https://huggingface.slack.com/archives/C01229B19EX/p1630021912025800

Thank you!

@lhoestq


Fix: wikicorpus - fix keys

The CI error is unrelated to this PR

... merging !

As mentioned in https://github.com/huggingface/datasets/issues/2552, there is a duplicate keys error in `wikicorpus`.

I fixed that by taking into account the file index in the keys


Fix extraction protocol inference from urls with params

merging since the windows error is just a CircleCI issue

Previously it was unable to infer the compression protocol for files at URLs like
```
https://foo.bar/train.json.gz?dl=1
```
because of the query parameters.

I fixed that, this should allow 10+ datasets to work in streaming mode:
```
  "discovery",
  "emotion",
  "grail_qa",
  "guardian_authorship",
  "pragmeval",
  "simple_questions_v2",
  "versae/adobo",
  "w-nicole/childes_data",
  "w-nicole/childes_data_no_tags_",
  "w-nicole/childes_data_with_tags",
  "w-nicole/childes_data_with_tags_"
```

cc @severo


always requiring the username in the dataset name when there is one

New feature or request

From what I can understand, you want the saved arrow file directory to have username as well instead of just dataset name if it was downloaded with the user prefix?

I don't think the user cares of how this is done, but the 2nd command should fail, IMHO, as its dataset name is invalid:
```
# first run
python -c "from datasets import load_dataset; load_dataset('stas/openwebtext-10k')"
# now run immediately
python -c "from datasets import load_dataset; load_dataset('openwebtext-10k')"
# the second command should fail, but it doesn't fail now.
```

Moreover, if someone were to create `openwebtext-10k` w/o the prefix, they will now get the wrong dataset, if they previously downloaded `stas/openwebtext-10k`.

And if there are 2 users with the same dataset name `foo/ds` and `bar/ds` - currently this won't work to get the correct dataset.

So really there 3 unrelated issues hiding in the current behavior.

This has been fixed now, and we'll do a new release of the library today.

Now the stas/openwebtext-10k dataset is cached at `.cache/huggingface/datasets/stas___openwebtext10k` and openwebtext-10k would be at `.cache/huggingface/datasets/openwebtext10k`. Since they are different, the cache won't fall back on loading the wrong one anymore.

Same for the python script used to generate the dataset: stas/openwebtext-10k is cached at `.cache/huggingface/modules/datasets_modules/datasets/stas___openwebtext10k` and openwebtext-10k would be at `.cache/huggingface/modules/datasets_modules/datasets/openwebtext10k`

Amazing! Thank you for adding this improvement, @lhoestq!

Me and now another person have been bitten by the `datasets`'s non-strictness on requiring a dataset creator's username when it's due.

So both of us started with `stas/openwebtext-10k`, somewhere along the lines lost `stas/` and continued using `openwebtext-10k` and it all was good until we published the software and things broke, since there is no `openwebtext-10k`

So this feature request is asking to tighten the checking and not allow dataset loading if it was downloaded with the user prefix, but then attempted to be used w/o it.

The same in code:

```
# first run
python -c "from datasets import load_dataset; load_dataset('stas/openwebtext-10k')"
# now run immediately
python -c "from datasets import load_dataset; load_dataset('openwebtext-10k')"
# the second command should fail, but it doesn't fail now.
```

Please let me know if I explained myself clearly.

Thank you!


Adding GLUECoS Hinglish and Spanglish code-switching bemchmark

dataset request

Requesting to add a new dataset

Hi @yjernite I am interested in adding this dataset.
In the repo they have also added a code mixed MT task from English to Hinglish [here](https://github.com/microsoft/GLUECoS#code-mixed-machine-translation-task). I think this could be a good dataset addition in itself and then I can add the rest of the GLUECoS tasks as one dataset. What do you think?

## Adding a Dataset
- **Name:** GLUECoS
- **Description:** a Microsoft Benchmark to evaluate code-switching for only two language pairs but a variety of tasks
- **Paper:** https://aclanthology.org/2020.acl-main.329/
- **Data:** https://github.com/microsoft/GLUECoS
- **Motivation:** We currently only have [one other](https://huggingface.co/datasets/lince) dataset for code-switching

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


How can I compute BLEU-4 score use `load_metric` ?

I have found the sacrebleu metric. But, I do not know the difference between it and BLEU-4.
If I want to compute BLEU-4 score, what can i do?


OpenWebText: NonMatchingSplitsSizesError

bug

Something isn't working

Thanks for reporting, I'm updating the verifications metadata

I just regenerated the verifications metadata and noticed that nothing changed: the data file is fine (the checksum didn't change), and the number of examples is still 8013769. Not sure how you managed to get 7982430 examples.

Can you try to delete your cache ( by default at `~/.cache/huggingface/datasets`) and try again please ?
Also, on which platform are you (linux/macos/windows) ?

I'll try without deleting the whole cache (we have large datasets already stored). I was under the impression that `download_mode="force_redownload"` would bypass cache.
Sorry plateform should be linux (Redhat version 8.1)

Hi @thomasw21 , are you still having this issue after clearing your cache ?

Sorry I haven't had time to work on this. I'll close and re-open if I can't figure out why I'm having this issue. Thanks for taking a look !

## Describe the bug

When downloading `openwebtext`, I'm getting:
```
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=39769494896, num_examples=8013769, dataset_name='openwebtext'), 'recorded': SplitInfo(name='train', num_bytes=39611023912, num_examples=7982430, dataset_name='openwebtext')}]
```

I suspect that the file we download from has changed since the size doesn't look like to match with documentation

`Downloading:   0%|          | 0.00/12.9G [00:00<?, ?B/s]` This suggest the total size is 12.9GB, whereas the one documented mentions `Size of downloaded dataset files: 12283.35 MB`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

load_dataset("openwebtext", download_mode="force_redownload")
```

## Expected results

Loading is successful

## Actual results

Loading throws above error.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.2
- Platform: linux (Redhat version 8.1)
- Python version: 3.8
- PyArrow version: 4.0.1


Add error_bad_chunk to the JSON loader

Add the `error_bad_chunk` parameter to the JSON loader.

Setting `error_bad_chunk=False` allows to skip an unparsable chunk of JSON data without raising an error.

Additional note:

In case of an unparsable JSON chunk, the JSON loader no longer tries to load the full JSON (which could take a lot of time in streaming mode) to get the JSON fields that the user may have forgotten to pass. Ex : for squad-like data, the user has to pass `field="data"` to tell the loader to get the list of examples from this field.

TODO: update docs

cc @lvwerra


prepare_module issue when loading from read-only fs

Something isn't working

## Describe the bug

When we use prepare_module from a readonly file system, we create a FileLock using the `local_path`.
This path is not necessarily writable.

`lock_path = local_path + ".lock"`

## Steps to reproduce the bug

Run `load_dataset` on a readonly python loader file.
```python
ds = load_dataset(
        python_loader, data_files={"train": train_path, "test": test_path}
    )
```

where `python_loader` is a path to a file located in a readonly folder.

## Expected results
This should work I think?

## Actual results

```python
    return load_dataset(
  File "/usr/local/lib/python3.8/dist-packages/datasets/load.py", line 711, in load_dataset
    module_path, hash, resolved_file_path = prepare_module(
  File "/usr/local/lib/python3.8/dist-packages/datasets/load.py", line 465, in prepare_module
    with FileLock(lock_path):
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/filelock.py", line 314, in __enter__
    self.acquire()
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/filelock.py", line 263, in acquire
    self._acquire()
  File "/usr/local/lib/python3.8/dist-packages/datasets/utils/filelock.py", line 378, in _acquire
    fd = os.open(self._lock_file, open_mode)
OSError: [Errno 30] Read-only file system: 'YOUR_FILE.py.lock'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.7.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 3.0.0


Optimize Dataset.filter to only compute the indices to keep

Maybe worth updating the docs here as well?

Optimize `Dataset.filter` to only compute the indices of the rows to keep, instead of creating a new Arrow table with the rows to keep. Creating a new table was an issue because it could take a lot of disk space.

This will be useful to process audio datasets for example cc @patrickvonplaten


Update: timit_asr - make the dataset streamable

The TIMIT ASR dataset had two issues that was preventing it from being streamable:

1. it was missing a call to `open` before `pd.read_csv`
2. it was using `os.path.dirname` which is not supported for streaming

I made the dataset streamable by using `open` to load the CSV, and by adding the support for `os.path.dirname` in dataset scripts to stream data

You can now do
```python
from datasets import load_dataset

timit_asr = load_dataset("timit_asr", streaming=True)
print(next(iter(timit_asr["train"])))
```
prints:

```json
{"file": "zip://data/TRAIN/DR4/MMDM0/SI681.WAV::https://data.deepai.org/timit.zip",
"phonetic_detail": {"start": [0, 1960, 2466, 3480, 4000, 5960, 7480, 7880, 9400, 9960, 10680, 13480, 15680, 15880, 16920, 18297, 18882, 19480, 21723, 22516, 24040, 25190, 27080, 28160, 28560, 30120, 31832, 33240, 34640, 35968, 37720],
"utterance": ["h#", "w", "ix", "dcl", "s", "ah", "tcl", "ch", "ix", "n", "ae", "kcl", "t", "ix", "v", "r", "ix", "f", "y", "ux", "zh", "el", "bcl", "b", "iy", "y", "ux", "s", "f", "el", "h#"],
"stop": [1960, 2466, 3480, 4000, 5960, 7480, 7880, 9400, 9960, 10680, 13480, 15680, 15880, 16920, 18297, 18882, 19480, 21723, 22516, 24040, 25190, 27080, 28160, 28560, 30120, 31832, 33240, 34640, 35968, 37720, 39920]},
"sentence_type": "SI", "id": "SI681",
"speaker_id": "MMDM0",
"dialect_region": "DR4",
"text": "Would such an act of refusal be useful?",
"word_detail": {
    "start": [1960, 4000, 9400, 10680, 15880, 18297, 27080, 30120],
    "utterance": ["would", "such", "an", "act", "of", "refusal", "be", "useful"],
    "stop": [4000, 9400, 10680, 15880, 18297, 27080, 30120, 37720]
}}
```

cc @patrickvonplaten @vrindaprabhu


Fix IndexError by ignoring empty RecordBatch

We need to ignore the empty record batches for the interpolation search to work correctly when querying arrow tables

Close #2833

cc @SaulLu


IndexError when accessing first element of a Dataset if first RecordBatch is empty

The computation of the offsets of the underlying Table of a Dataset has some issues if the first RecordBatch is empty.

```python
from datasets import Dataset
import pyarrow as pa

pa_table = pa.Table.from_pydict({"a": [1]})
pa_table2 = pa.Table.from_pydict({"a": []}, schema=pa_table.schema)
ds_table = pa.concat_tables([pa_table2, pa_table])

dataset = Dataset(ds_table)

print([len(b) for b in dataset.data._batches])
# [0, 1]

print(dataset.data._offsets)
# [0 0 1] (should be [0, 1])

dataset[0]
```
raises
```python
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/datasets/table.py in _interpolation_search(arr, x)
     90         else:
     91             i, j = i, k
---> 92     raise IndexError(f"Invalid query '{x}' for size {arr[-1] if len(arr) else 'none'}.")
     93
     94

IndexError: Invalid query '0' for size 1.
```

This can be fixed by ignoring empty batches when computing `table._batches` and `table._offsets`

cc @SaulLu