ImageFolder raises an error with parameters drop_metadata=True and drop_labels=False when metadata.jsonl is present

Something isn't working

## Describe the bug

If you pass `drop_metadata=True` and `drop_labels=False` when a `data_dir` contains at least one `matadata.jsonl` file, you will get a KeyError. This is probably not a very useful case but we shouldn't get an error anyway. Asking users to move metadata files manually outside `data_dir` or pass features manually (when there is a tool that can infer them automatically) don't look like a good idea to me either.

## Steps to reproduce the bug
### Clone an example dataset from the Hub
```bash
git clone https://huggingface.co/datasets/nateraw/test-imagefolder-metadata
```
### Try to load it
```python
from datasets import load_dataset
ds = load_dataset("test-imagefolder-metadata", drop_metadata=True, drop_labels=False)
```
or even just
```python
ds = load_dataset("test-imagefolder-metadata", drop_metadata=True)
```
as `drop_labels=False` is a default value.

## Expected results
A DatasetDict object with two features: `"image"` and `"label"`.

## Actual results
```
Traceback (most recent call last):
  File "/home/polina/workspace/datasets/debug.py", line 18, in <module>
    ds = load_dataset(
  File "/home/polina/workspace/datasets/src/datasets/load.py", line 1732, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 1227, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 1218, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1596, in encode_example
    return encode_nested_example(self, example)
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1165, in encode_nested_example
    {
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1165, in <dictcomp>
    {
  File "/home/polina/workspace/datasets/src/datasets/utils/py_utils.py", line 249, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/home/polina/workspace/datasets/src/datasets/utils/py_utils.py", line 249, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
KeyError: 'label'
```

## Environment info
`datasets` master branch

- `datasets` version: 2.3.3.dev0
- Platform: Linux-5.14.0-1042-oem-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.1
- Pandas version: 1.4.1


Data type is not recognized when using datetime.time

bug

Something isn't working

Hi, thanks for reporting! I'm investigating the issue.

## Describe the bug

Creating a dataset from a pandas dataframe with `datetime.time` format generates an error.

## Steps to reproduce the bug
```python
import pandas as pd
from datetime import time
from datasets import Dataset
df = pd.DataFrame({"feature_name": [time(1, 1, 1)]})
dataset = Dataset.from_pandas(df)
```

## Expected results

The dataset should be created.

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 823, in from_pandas
    return cls(table, info=info, split=split)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 679, in __init__
    inferred_features = Features.from_arrow_schema(arrow_table.schema)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1551, in from_arrow_schema
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1551, in <dictcomp>
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1315, in generate_from_arrow_type
    return Value(dtype=_arrow_to_datasets_dtype(pa_type))
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 83, in _arrow_to_datasets_dtype
    return f"time64[{arrow_type.unit}]"
AttributeError: 'pyarrow.lib.DataType' object has no attribute 'unit'
```

## Environment info

- `datasets` version: 2.3.3.dev0
- Platform: Linux-5.13.0-1031-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.4.2


np arrays get turned into native lists

Something isn't working

If you add the line `dataset2.set_format('np')` before calling `dataset2[0]['tmp']` it should return `np.ndarray`.
I believe internally it will not store it as a list, it is only returning a list when you index it.

```
In [1]: import datasets, numpy as np
In [2]: dataset = datasets.load_dataset("glue", "mrpc")["validation"]
In [3]: dataset2 = dataset.map(lambda x: {"tmp": np.array([0.5])}, batched=False)
In [4]: dataset2[0]["tmp"]
Out[4]: [0.5]

In [5]: dataset2.set_format('np')

In [6]: dataset2[0]["tmp"]
Out[6]: array([0.5])
```

I see, thanks! Any idea if the default numpy → list conversion might cause precision loss?

I'm not super familiar with our datasets works internally, but I think your `np` array will be stored in a `pyarrow` format, and then you take a view of this as a python array. In which case, I think the precision should be preserved.

## Describe the bug
When attaching an `np.array` field, it seems that it automatically gets turned into a list (see below). Why is this happening? Could it lose precision? Is there a way to make sure this doesn't happen?

## Steps to reproduce the bug
```python
>>> import datasets, numpy as np
>>> dataset = datasets.load_dataset("glue", "mrpc")["validation"]
Reusing dataset glue (...)
100%|███████████████████████████████████████████████| 3/3 [00:00<00:00, 1360.61it/s]
>>> dataset2 = dataset.map(lambda x: {"tmp": np.array([0.5])}, batched=False)
100%|██████████████████████████████████████████| 408/408 [00:00<00:00, 10819.97ex/s]
>>> dataset2[0]["tmp"]
[0.5]
>>> type(dataset2[0]["tmp"])
<class 'list'>
```

## Expected results
`dataset2[0]["tmp"]` should be an `np.ndarray`.

## Actual results
It's a list.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: mac, though I'm pretty sure it happens on a linux machine too
- Python version: 3.9.7
- PyArrow version: 6.0.1


contribute data loading for object detection datasets with yolo data format

enhancement

New feature or request

Hi! The `imagefolder` script is already quite complex, so a standalone script sounds better. Also, I suggest we create an org on the Hub (e.g. `hf-loaders`) and store such scripts there for easier maintenance rather than having them as packaged modules (IMO only very generic loaders should be packaged). WDYT @lhoestq @albertvillanova @polinaeterna?

Thank you for the suggestion @mariosasko . I agree with the point, but I have a few doubts

1. How would the user access the script if it's not a part of the core codebase?
2. Could you direct me as to what will be the tasks I have to do to contribute to the code? As per my understanding, it would be like
    1. Create a new org "hf-loaders" and add you (and more HF people) to the org
    2. Add data loader script as a (model?)
    3. Test it with a dataset on HF hub
3. We should maybe brainstorm as to which public datasets have this format (YOLO type) and are the most important ones to test the script with. We can even add the datasets on HF Hub alongside the script

**Is your feature request related to a problem? Please describe.**
At the moment, HF datasets loads [image classification datasets](https://huggingface.co/docs/datasets/image_process) out-of-the-box. There could be a data loader for loading standard object detection datasets ([original discussion here](https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/discussions/2))

**Describe the solution you'd like**
I wrote a [custom script](https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/blob/main/detect_chess_pieces.py) to load dataset which has YOLO data format.

**Describe alternatives you've considered**
The script can either be a standalone dataset builder, or a modified version of `ImageFolder`

**Additional context**
I would be happy to contribute to this, but I would do it at a very slow pace (maybe a month or two) as I have my exams approaching 😄


Fix `embed_storage` on features inside lists/sequences

_The documentation is not available anymore as the PR was closed or merged._

Add a dedicated function for embed_storage to always preserve the embedded/casted arrays (and to have more control over `embed_storage` in general).

Fix #4591

~~(Waiting for #4608 to be merged to mark this PR as ready for review - required for fixing `xgetsize` in private repos)~~ Done!


Ensure ConcatenationTable.cast uses target_schema metadata

Hi @lhoestq, Thanks for the detailed comment. I've tested the suggested approach and can confirm it works for the testcase outlined above! The PR is updated with the changes.

Currently, `ConcatenationTable.cast` does not use target_schema metadata when casting subtables. This causes an issue when using cast_column and the underlying table is a ConcatenationTable.

Code example of where issue arrises:
```
from datasets import Dataset, Image

column1 = [0, 1]
image_paths = ['/images/image1.jpg', '/images/image2.jpg']

ds = Dataset.from_dict({"column1": column1})
ds = ds.add_column("image", image_paths)
ds.cast_column("image", Image()) # Fails here
```
Output
```
...
TypeError: Couldn't cast array of type
string
to
{'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='string', id=None)}
```


Align/fix license metadata info

Thank you thank you! Let's merge and pray? 😱

I just need to add `license_details` to the validator and yup we can merge

fix bad "other-*" licenses and add the corresponding "license_details" when relevant


Release 2.3.0 broke custom iterable datasets

bug

Something isn't working

Apparently, `fsspec` does not allow access to attribute-based modules anymore, such as `fsspec.async`.

However, this is a fairly simple fix:
- Change the import to: `from fsspec import asyn`;
- Change line 18 to: `asyn.iothread[0] = None`;
- Change line 19 to `asyn.loop[0] = None`.

Hi! I think it's easier to replace `import fsspec` with `import fsspec.asyn` and leave the rest unchanged. @gugarosa Are you interested in submitting a PR?

Perfect, it is even better!

Just submitted the PR: #4630.

Thank you!

## Describe the bug
Trying to iterate examples from custom iterable dataset fails to bug introduced in `torch_iterable_dataset.py` since the release of 2.3.0.

## Steps to reproduce the bug
```python
next(iter(custom_iterable_dataset))
```

## Expected results
`next(iter(custom_iterable_dataset))` should return examples from the dataset

## Actual results
```
/usr/local/lib/python3.7/dist-packages/datasets/formatting/dataset_wrappers/torch_iterable_dataset.py in _set_fsspec_for_multiprocess()
     16     See https://github.com/fsspec/gcsfs/issues/379
     17     """
---> 18     fsspec.asyn.iothread[0] = None
     19     fsspec.asyn.loop[0] = None
     20

AttributeError: module 'fsspec' has no attribute 'asyn'
```

## Environment info
- `datasets` version: 2.3.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 8.0.0
- Pandas version: 1.3.5


Preserve member order by MockDownloadManager.iter_archive

Currently, `MockDownloadManager.iter_archive` yields paths to archive members in an order given by `path.rglob("*")`, which migh not be the same order as in the original archive.

See issue in:
- https://github.com/huggingface/datasets/pull/4579#issuecomment-1172135027

This PR fixes the order of the members yielded by `MockDownloadManager.iter_archive` so that it is the same as in the original archive.


codeparrot/github-code failing to load

Something isn't working

I believe the issue is in `codeparrot/github-code`. `base_path` param is missing - https://huggingface.co/datasets/codeparrot/github-code/blob/main/github-code.py#L169

Function definition has changed.
https://github.com/huggingface/datasets/blob/0e1c629cfb9f9ba124537ba294a0ec451584da5f/src/datasets/data_files.py#L547

@mariosasko could you please confirm my finding? And are there any changes that need to be done from my side?

> Good catch ! We recently did a breaking change in `get_patterns_in_dataset_repository`, I think we can revert it

I can't wait for that releasee. Broke my application

This simple workaround should fix: https://huggingface.co/datasets/codeparrot/github-code/discussions/2

`get_patterns_in_dataset_repository` can treat whether `base_path=None`, so we just need to make sure that codeparrot/github-code `_split_generators` calls with such an argument.

I am afraid your suggested change @gugarosa will break compatibility with older datasets versions that don't have `base_path` argument in `get_patterns_in_dataset_repository`, as a workaround while the issue gets resolved in `datasets` can you downgrade your datasets version  to `<=2.1.0` ?
@lvwerra do you think we should adapt the script to check the datasets version before calling `get_patterns_in_dataset_repository`?

Actually I think it's just simpler to fix it in the dataset itself, let me open a PR

EDIT: PR opened here: https://huggingface.co/datasets/codeparrot/github-code/discussions/3

PR is merged, it's working now ! Closing this one :)

> I am afraid your suggested change @gugarosa will break compatibility with older datasets versions that don't have `base_path` argument in `get_patterns_in_dataset_repository`, as a workaround while the issue gets resolved in `datasets` can you downgrade your datasets version to `<=2.1.0` ?
> @lvwerra do you think we should adapt the script to check the datasets version before calling `get_patterns_in_dataset_repository`?

You are definitely right, sorry about it. I always keep forgetting that we need to keep in mind users from past versions, my bad.

## Describe the bug
codeparrot/github-code fails to load with a `TypeError: get_patterns_in_dataset_repository() missing 1 required positional argument: 'base_path'`

## Steps to reproduce the bug
```python
from datasets import load_dataset
```

## Expected results
loaded dataset object

## Actual results
```python
 [3]: dataset = load_dataset("codeparrot/github-code")
No config specified, defaulting to: github-code/all-all
Downloading and preparing dataset github-code/all-all to /home/bebr/.cache/huggingface/datasets/codeparrot___github-code/all-all/0.0.0/a55513bc0f81db773f9896c7aac225af0cff5b323bb9d2f68124f0a8cc3fb817...
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 dataset = load_dataset("codeparrot/github-code")

File ~/miniconda3/envs/fastapi-kube/lib/python3.10/site-packages/datasets/load.py:1679, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1676 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1678 # Download and prepare data
-> 1679 builder_instance.download_and_prepare(
   1680     download_config=download_config,
   1681     download_mode=download_mode,
   1682     ignore_verifications=ignore_verifications,
   1683     try_from_hf_gcs=try_from_hf_gcs,
   1684     use_auth_token=use_auth_token,
   1685 )
   1687 # Build dataset for splits
   1688 keep_in_memory = (
   1689     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1690 )

File ~/miniconda3/envs/fastapi-kube/lib/python3.10/site-packages/datasets/builder.py:704, in DatasetBuilder.download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702         logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703 if not downloaded_from_gcs:
--> 704     self._download_and_prepare(
    705         dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706     )
    707 # Sync info
    708 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/miniconda3/envs/fastapi-kube/lib/python3.10/site-packages/datasets/builder.py:1221, in GeneratorBasedBuilder._download_and_prepare(self, dl_manager, verify_infos)
   1220 def _download_and_prepare(self, dl_manager, verify_infos):
-> 1221     super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)

File ~/miniconda3/envs/fastapi-kube/lib/python3.10/site-packages/datasets/builder.py:771, in DatasetBuilder._download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    769 split_dict = SplitDict(dataset_name=self.name)
    770 split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 771 split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    773 # Checksums verification
    774 if verify_infos and dl_manager.record_checksums:

File ~/.cache/huggingface/modules/datasets_modules/datasets/codeparrot--github-code/a55513bc0f81db773f9896c7aac225af0cff5b323bb9d2f68124f0a8cc3fb817/github-code.py:169, in GithubCode._split_generators(self, dl_manager)
    162 def _split_generators(self, dl_manager):
    164     hfh_dataset_info = HfApi(datasets.config.HF_ENDPOINT).dataset_info(
    165         _REPO_NAME,
    166         timeout=100.0,
    167     )
--> 169     patterns = datasets.data_files.get_patterns_in_dataset_repository(hfh_dataset_info)
    170     data_files = datasets.data_files.DataFilesDict.from_hf_repo(
    171         patterns,
    172         dataset_info=hfh_dataset_info,
    173     )
    175     files = dl_manager.download_and_extract(data_files["train"])

TypeError: get_patterns_in_dataset_repository() missing 1 required positional argument: 'base_path'
```

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.18.7-arch1-1-x86_64-with-glibc2.35
- Python version: 3.10.5
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Something isn't working

Hi! You can use streaming to fetch only a subset of the data:
```python
raw_dataset = load_dataset("librispeech_asr", "clean", split="train.100", streaming=True)
```
Also, we plan to make it possible to download a particular split in the non-streaming mode, but this task is not easy due to how our dataset scripts are structured.

Hi,

That's a great help. Thank you very much.

## Describe the bug
librispeech dataset has to download whole subset when specifing the split to use

## Steps to reproduce the bug
see below
# Sample code to reproduce the bug
```
!pip install datasets
from datasets import load_dataset
raw_dataset = load_dataset("librispeech_asr", "clean", split="train.100")
```

## Expected results
The split "train.clean.100" is downloaded.

## Actual results
All four splits in "clean" subset is downloaded.

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Fix xisfile, xgetsize, xisdir, xlistdir in private repo

_The documentation is not available anymore as the PR was closed or merged._

Added tests for xisfile, xgetsize, xlistdir and xglob for private repos, and also tests for xwalk that was untested

`xisfile` is working in a private repository when passing a chained URL to a file inside an archive, e.g. `zip://a.txt::https://huggingface/datasets/username/dataset_name/resolve/main/data.zip`. However it's not working when passing a simple file `https://huggingface/datasets/username/dataset_name/resolve/main/data.zip`.

This is because the authentication headers are not passed correctly in this case.

This is causing dataset streaming to fail in private parquet repositories, as noted in https://github.com/huggingface/datasets/issues/4605

I fixed `xisfile` and the other functions that behave the same way: xgetsize, xisdir and xlistdir

TODO:
- [x] tests

fix https://github.com/huggingface/datasets/issues/4605


Align more metadata with other repo types (models,spaces)

I just set a default value (None) for the deprecated licenses and languages fields, which should fix most of the CI failures.

Note that the CI should still be red because you edited many dataset cards and they're still missing some content - but this is unrelated to this PR so we can ignore these failures

There's also a follow-up PR to this one, in #4613 – I would suggest to merge all of them at the same time and hope not too many things are broken 🙀 🙀

Alright merging this one now, let's see how broken things get

see also associated PR on the `datasets-tagging` Space: https://huggingface.co/spaces/huggingface/datasets-tagging/discussions/2 (to merge after this one is merged)


Something isn't working

Hi! The GH/no-namespace datasets versioning is synced with the version of the `datasets` lib, which means that the `wikiann` script was modified between the two compared versions. In this scenario, you can ensure reproducibility by pinning the script version, which is done by passing `revision="x.y.z"` (e.g. `revision="2.2.0"`) to `load_dataset.`

## Describe the bug
evaluation result changes after `datasets` version change

## Steps to reproduce the bug
1. Train a model on WikiAnn
2. reload the ckpt -> test accuracy becomes same as eval accuracy
3. such behavior is gone after downgrading `datasets`

https://colab.research.google.com/drive/1kYz7-aZRGdayaq-gDTt30tyEgsKlpYOw?usp=sharing

## Expected results
evaluation result shouldn't change before/after `datasets` version changes

## Actual results
evaluation result changes before/after `datasets` version changes

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: colab
- Python version: 3.7.13
- PyArrow version: 6.0.1

Q. How could the evaluation result change before/after `datasets` version changes?


Dataset Viewer issue for boris/gis_filtered

Yes, this dataset is "gated": you first have to go to https://huggingface.co/datasets/boris/gis_filtered and click "Access repository" (if you accept to share your contact information with the repository authors).

I already did that, it returns error when using streaming

Oh, sorry, I misread. Looking at it. Maybe @huggingface/datasets or @SBrandeis

I could reproduce the error, even though I provided my token and accepted the gate form. It looks like an error from `datasets`

This is indeed a bug in `datasets`. Parquet datasets in gated/private repositories can't be streamed properly, which caused the viewer to fail. I opened a PR at https://github.com/huggingface/datasets/pull/4608

### Link

https://huggingface.co/datasets/boris/gis_filtered/viewer/boris--gis_filtered/train

### Description

When I try to access this from the website I get this error:

Status code:   400
Exception:     ClientResponseError
Message:       401, message='Unauthorized', url=URL('https://huggingface.co/datasets/boris/gis_filtered/resolve/80b805053ce61d4eb487b6b8d9095d775c2c466e/data/train/0000.parquet')

If I try to load with code I also get the same issue:
```python
dataset2_train=load_dataset("boris/gis_filtered", use_auth_token=os.environ["HF_TOKEN"],split="train",streaming=True)
dataset2_validation=load_dataset("boris/gis_filtered", use_auth_token=os.environ["HF_TOKEN"], split="validation",streaming=True)
```

### Owner

No


Update CI Windows orb

This PR tries to fix recurrent random CI failures on Windows.

After 2 runs, it seems to have fixed the issue.

Fix #4603.


CI fails recurrently and randomly on Windows

Something isn't working

As reported by @lhoestq,

The windows CI is currently flaky: some dependencies like `aiobotocore`, `multiprocess` and `seqeval` sometimes fail to install.
In particular it seems that building the wheels fail. Here is an example of logs:

```
Building wheel for seqeval (setup.py): started
  Running command 'C:\tools\miniconda3\envs\py37\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"'; __file__='"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6'
  No parent package detected, impossible to derive `name`
  running bdist_wheel
  running build
  running build_py
  package init file 'seqeval\__init__.py' not found (or not a regular file)
  package init file 'seqeval\metrics\__init__.py' not found (or not a regular file)
  C:\tools\miniconda3\envs\py37\lib\site-packages\setuptools\command\install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    setuptools.SetuptoolsDeprecationWarning,
  installing to build\bdist.win-amd64\wheel
  running install
  running install_lib
  warning: install_lib: 'build\lib' does not exist -- no Python modules to install

  running install_egg_info
  running egg_info
  creating UNKNOWN.egg-info
  writing UNKNOWN.egg-info\PKG-INFO
  writing dependency_links to UNKNOWN.egg-info\dependency_links.txt
  writing top-level names to UNKNOWN.egg-info\top_level.txt
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  reading manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  Copying UNKNOWN.egg-info to build\bdist.win-amd64\wheel\.\UNKNOWN-0.0.0-py3.7.egg-info
  running install_scripts
  creating build\bdist.win-amd64\wheel\UNKNOWN-0.0.0.dist-info\WHEEL
  creating 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6\UNKNOWN-0.0.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
  adding 'UNKNOWN-0.0.0.dist-info/METADATA'
  adding 'UNKNOWN-0.0.0.dist-info/WHEEL'
  adding 'UNKNOWN-0.0.0.dist-info/top_level.txt'
  adding 'UNKNOWN-0.0.0.dist-info/RECORD'
  removing build\bdist.win-amd64\wheel
  Building wheel for seqeval (setup.py): finished with status 'done'
  Created wheel for seqeval: filename=UNKNOWN-0.0.0-py3-none-any.whl size=963 sha256=67eb93a6e1ff4796c5882a13f9fa25bb0d3d103796e2525f9cecf3b2ef26d4b1
  Stored in directory: c:\users\circleci\appdata\local\pip\cache\wheels\05\96\ee\7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7
  WARNING: Built wheel for seqeval is invalid: Wheel has unexpected file name: expected 'seqeval', got 'UNKNOWN'
```


Upgrade setuptools in windows CI

The windows CI is currently flaky: some dependencies like aiobotocore, multiprocess and seqeval sometimes fail to install.
In particular it seems that building the wheels fail. Here is an example of logs

```
Building wheel for seqeval (setup.py): started
  Running command 'C:\tools\miniconda3\envs\py37\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"'; __file__='"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6'
  No parent package detected, impossible to derive `name`
  running bdist_wheel
  running build
  running build_py
  package init file 'seqeval\__init__.py' not found (or not a regular file)
  package init file 'seqeval\metrics\__init__.py' not found (or not a regular file)
  C:\tools\miniconda3\envs\py37\lib\site-packages\setuptools\command\install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    setuptools.SetuptoolsDeprecationWarning,
  installing to build\bdist.win-amd64\wheel
  running install
  running install_lib
  warning: install_lib: 'build\lib' does not exist -- no Python modules to install

  running install_egg_info
  running egg_info
  creating UNKNOWN.egg-info
  writing UNKNOWN.egg-info\PKG-INFO
  writing dependency_links to UNKNOWN.egg-info\dependency_links.txt
  writing top-level names to UNKNOWN.egg-info\top_level.txt
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  reading manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  Copying UNKNOWN.egg-info to build\bdist.win-amd64\wheel\.\UNKNOWN-0.0.0-py3.7.egg-info
  running install_scripts
  creating build\bdist.win-amd64\wheel\UNKNOWN-0.0.0.dist-info\WHEEL
  creating 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6\UNKNOWN-0.0.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
  adding 'UNKNOWN-0.0.0.dist-info/METADATA'
  adding 'UNKNOWN-0.0.0.dist-info/WHEEL'
  adding 'UNKNOWN-0.0.0.dist-info/top_level.txt'
  adding 'UNKNOWN-0.0.0.dist-info/RECORD'
  removing build\bdist.win-amd64\wheel
  Building wheel for seqeval (setup.py): finished with status 'done'
  Created wheel for seqeval: filename=UNKNOWN-0.0.0-py3-none-any.whl size=963 sha256=67eb93a6e1ff4796c5882a13f9fa25bb0d3d103796e2525f9cecf3b2ef26d4b1
  Stored in directory: c:\users\circleci\appdata\local\pip\cache\wheels\05\96\ee\7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7
  WARNING: Built wheel for seqeval is invalid: Wheel has unexpected file name: expected 'seqeval', got 'UNKNOWN'
```

hopefully this fixes the issue


Upgrade pip in WIN CI

The windows CI is currently flaky: some dependencies like aiobotocore, multiprocess and seqeval sometimes fail to install.
In particular it seems that building the wheels fail. Here is an example of logs

```
Building wheel for seqeval (setup.py): started
  Running command 'C:\tools\miniconda3\envs\py37\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"'; __file__='"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6'
  No parent package detected, impossible to derive `name`
  running bdist_wheel
  running build
  running build_py
  package init file 'seqeval\__init__.py' not found (or not a regular file)
  package init file 'seqeval\metrics\__init__.py' not found (or not a regular file)
  C:\tools\miniconda3\envs\py37\lib\site-packages\setuptools\command\install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    setuptools.SetuptoolsDeprecationWarning,
  installing to build\bdist.win-amd64\wheel
  running install
  running install_lib
  warning: install_lib: 'build\lib' does not exist -- no Python modules to install

  running install_egg_info
  running egg_info
  creating UNKNOWN.egg-info
  writing UNKNOWN.egg-info\PKG-INFO
  writing dependency_links to UNKNOWN.egg-info\dependency_links.txt
  writing top-level names to UNKNOWN.egg-info\top_level.txt
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  reading manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  Copying UNKNOWN.egg-info to build\bdist.win-amd64\wheel\.\UNKNOWN-0.0.0-py3.7.egg-info
  running install_scripts
  creating build\bdist.win-amd64\wheel\UNKNOWN-0.0.0.dist-info\WHEEL
  creating 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6\UNKNOWN-0.0.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
  adding 'UNKNOWN-0.0.0.dist-info/METADATA'
  adding 'UNKNOWN-0.0.0.dist-info/WHEEL'
  adding 'UNKNOWN-0.0.0.dist-info/top_level.txt'
  adding 'UNKNOWN-0.0.0.dist-info/RECORD'
  removing build\bdist.win-amd64\wheel
  Building wheel for seqeval (setup.py): finished with status 'done'
  Created wheel for seqeval: filename=UNKNOWN-0.0.0-py3-none-any.whl size=963 sha256=67eb93a6e1ff4796c5882a13f9fa25bb0d3d103796e2525f9cecf3b2ef26d4b1
  Stored in directory: c:\users\circleci\appdata\local\pip\cache\wheels\05\96\ee\7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7
  WARNING: Built wheel for seqeval is invalid: Wheel has unexpected file name: expected 'seqeval', got 'UNKNOWN'
```

I tried to update pip and re-run the CI several times and I couldn't re-experience this issue for now, so I think upgrading pip may solve the issue


Remove multiple config section

documentation

Improvements or additions to documentation

This PR removes docs for a future feature and redirects to #4578 instead. See this [discussion](https://huggingface.slack.com/archives/C034N0A7H09/p1656107063801969) for more details :)


Smooth-BLEU bug fixed

Hi,

the current implementation of smooth-BLEU contains a bug: it smoothes unigrams as well. Consequently, when both the reference and translation consist of totally different tokens, it anyway returns a non-zero value (please see the attached image).

This however contradicts the source paper suggesting the smooth-BLEU _(Chin-Yew Lin, Franz Josef Och. ORANGE: a method for evaluating automatic evaluation metrics for machine translation. COLING 2004.)_ :

> Add one count to the n-gram hit and total ngram count for n > 1. Therefore, for candidate translations with less than n words, they can still get a positive smoothed BLEU score from shorter n-gram matches; however if nothing matches then they will get zero scores.

This pull request aims at fixing this bug.

I made a pull request in the target repository `tensorflow/nmt`, which implements this script, yet the last commit there is dating 19.02.2019 and I doubt whether this will be fixed promptly. Yet, this bug is critical, for instance for summarization datasets with short summaries (e.g. AESLC), since smoothing needs to be applied there. Therefore, the easiest solution that I found is to fork the repo and download this script directly from the forked fixed repo.

Kind,
Akim Tsvigun

<img width="516" alt="Снимок экрана 2022-06-29 в 17 49 27" src="https://user-images.githubusercontent.com/36672861/176466935-ac579e6d-6a93-4111-ab41-9b33056e7d47.png">


Host financial_phrasebank data on the Hub

Fix #4597.


Streaming issue for financial_phrasebank

cc @huggingface/datasets: it seems like https://www.researchgate.net/ is flaky for datasets hosting (I put the "hosted-on-google-drive" tag since it's the same kind of issue I think)

Let's see if their license allows hosting their data on the Hub.

License is Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0).

We can host their data on the Hub.

### Link

https://huggingface.co/datasets/financial_phrasebank/viewer/sentences_allagree/train

### Description

As reported by a community member using [AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/5#62bc217436d0e5d316a768f0), there seems to be a problem streaming this dataset:

```
Server error
Status code:   400
Exception:     Exception
Message:       Give up after 5 attempts with ConnectionError
```

### Owner

No


Dataset Viewer issue for universal_dependencies

dataset-viewer

Related to the dataset viewer on huggingface.co

### Link

https://huggingface.co/datasets/universal_dependencies

### Description

invalid json response body at https://datasets-server.huggingface.co/splits?dataset=universal_dependencies reason: Unexpected token I in JSON at position 0

### Owner

_No response_


Dataset Viewer issue with False positive PII redaction

The value is in the data, it's not an issue with the "dataset-viewer".

<img width="1161" alt="Capture d’écran 2022-06-29 à 10 25 51" src="https://user-images.githubusercontent.com/1676121/176389325-4d2a9a7f-1583-45b8-aa7a-960ffaa6a36a.png">

 Maybe open a PR: https://huggingface.co/datasets/cakiki/rosetta-code/discussions

This was indeed a scraping issue which I assumed was a display issue; sorry about that!

### Link

https://huggingface.co/datasets/cakiki/rosetta-code

### Description

Hello, I just noticed an entry being redacted that shouldn't have been:

`RootMeanSquare@Range[10]` is being displayed as `[email protected][10]`

### Owner

_No response_


load_from_disk suggests incorrect fix when used to load DatasetDict

bug

Something isn't working

Edit: Please feel free to remove this issue. The problem was not the error message but the fact that the DatasetDict.load_from_disk does not support loading nested splits, i.e. if one of the splits is itself a DatasetDict. If nesting splits is an antipattern, perhaps the load_from_disk function can throw a warning indicating that?


Fix error message when using load_from_disk to load DatasetDict

Issue #4594
Issue: When `datasets.load_from_disk` is wrongly used to load a `DatasetDict`, the error message suggests using `datasets.load_from_disk`, which is the same function that generated the error.
Fix: The appropriate function which should be suggested instead is `datasets.dataset_dict.load_from_disk`.
Changes: Change the suggestion to say "Please use `datasets.dataset_dict.load_from_disk` instead."


Issue with jalFaizy/detect_chess_pieces when running datasets-cli test

Hi @faizankshaikh

Please note that we have recently launched the Community feature, specifically targeted to create Discussions (about issues/questions/asking-for-help) on each Dataset on the Hub:
- Blog post: https://huggingface.co/blog/community-update
- Docs: https://huggingface.co/docs/hub/repositories-pull-requests-discussions

The Discussion tab for your "jalFaizy/detect_chess_pieces" dataset is here: https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/discussions
You can use it to ask for help by pinging the Datasets maintainers: see our docs here: https://huggingface.co/docs/datasets/master/en/share#ask-for-a-help-and-reviews

I'm transferring this discussion to your Discussion tab and trying to address it: https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/discussions/1

Thank you @albertvillanova , I will keep that in mind.

Just a quick note - I posted the issue on Github because the dataset viewer suggested me to "open an issue for direct support". Maybe it can be updated with your suggestion

![image](https://user-images.githubusercontent.com/8406903/176397633-7b077d81-2044-4487-b58e-6346b05be5cf.png)

Thank you pointing this out: yes, definitely, we should fix the error message. We are working on this.

### Link

https://huggingface.co/datasets/jalFaizy/detect_chess_pieces

### Description

I am trying to write a appropriate data loader for [a custom dataset](https://huggingface.co/datasets/jalFaizy/detect_chess_pieces) using [this script](https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/blob/main/detect_chess_pieces.py)

When I run the command

`$ datasets-cli test "D:\workspace\HF\detect_chess_pieces" --save_infos --all_configs`

It gives the following error

```
Using custom data configuration default
Traceback (most recent call last):
  File "c:\users\faiza\anaconda3\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\faiza\anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\faiza\anaconda3\Scripts\datasets-cli.exe\__main__.py", line 7, in <module>
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\commands\datasets_cli.py", line 39, in main
    service.run()
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\commands\test.py", line 132, in run
    for j, builder in enumerate(get_builders()):
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\commands\test.py", line 125, in get_builders
    yield builder_cls(
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\builder.py", line 1148, in __init__
    super().__init__(*args, **kwargs)
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\builder.py", line 306, in __init__
    info = self.get_exported_dataset_info()
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\builder.py", line 405, in get_exported_dataset_info
    return self.get_all_exported_dataset_infos().get(self.config.name, DatasetInfo())
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\builder.py", line 390, in get_all_exported_dataset_infos
    return DatasetInfosDict.from_directory(cls.get_imported_module_dir())
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\info.py", line 309, in from_directory
    dataset_infos_dict = {
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\info.py", line 310, in <dictcomp>
    config_name: DatasetInfo.from_dict(dataset_info_dict)
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\info.py", line 272, in from_dict
    return cls(**{k: v for k, v in dataset_info_dict.items() if k in field_names})
  File "<string>", line 20, in __init__
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\info.py", line 160, in __post_init__
    templates = [
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\info.py", line 161, in <listcomp>
    template if isinstance(template, TaskTemplate) else task_template_from_dict(template)
  File "c:\users\faiza\anaconda3\lib\site-packages\datasets\tasks\__init__.py", line 43, in task_template_from_dict
    return template.from_dict(task_template_dict)
AttributeError: 'NoneType' object has no attribute 'from_dict'
```

My assumption is that there is some kind of issue in how the "task_templates" are read, because even if I keep them as None, or not include the argument at all, the same error occurs

### Owner

Yes


Can't push Images to hub with manual Dataset

Something isn't working

Hi, thanks for reporting! This issue stems from the changes introduced in https://github.com/huggingface/datasets/pull/4282 (cc @lhoestq), in which list casts are ignored if they don't change the list type (required to preserve `null` values). And `push_to_hub` does a special cast to embed external image files but doesn't change the types, hence the failure.

## Describe the bug

If I create a dataset including an 'Image' feature manually, when pushing to hub decoded images are not pushed,
instead it looks for image where image local path is/used to be.
This doesn't (at least didn't used to) happen with imagefolder. I want to build dataset manually because it is complicated.

This happens even though the dataset is looking like decoded images:
![image](https://user-images.githubusercontent.com/15624271/176322689-2cc819cf-9d5c-4a8f-9f3d-83ae8ec06f20.png)
and I use `embed_external_files=True` while `push_to_hub` (same with false)
## Steps to reproduce the bug
```python

from PIL import Image
from datasets import Image as ImageFeature
from datasets import Features,Dataset
#manually create dataset
feats=Features(
    {
        "images": [ImageFeature()], #same even if explicitly ImageFeature(decode=True)
        "input_image": ImageFeature(),
    }
)

test_data={"images":[[Image.open("test.jpg"),Image.open("test.jpg"),Image.open("test.jpg")]], "input_image":[Image.open("test.jpg")]}
test_dataset=Dataset.from_dict(test_data,features=feats)
print(test_dataset)

test_dataset.push_to_hub("ceyda/image_test_public",private=False,token="",embed_external_files=True)

# clear cache rm -r ~/.cache/huggingface
# remove "test.jpg" # remove to see that it is looking for image on the local path

test_dataset=load_dataset("ceyda/image_test_public",use_auth_token="")
print(test_dataset)
print(test_dataset['train'][0])
```

## Expected results
should be able to push image bytes if dataset has `Image(decode=True)`

## Actual results

errors because it is trying to decode file from the non existing local path.
```
---->  print(test_dataset['train'][0])

File ~/.local/lib/python3.8/site-packages/datasets/arrow_dataset.py:2154, in Dataset.__getitem__(self, key)
   2152 def __getitem__(self, key):  # noqa: F811
   2153     """Can be used to index columns (by string names) or rows (by integer index or iterable of indices or bools)."""
-> 2154     return self._getitem(
   2155         key,
   2156     )

File ~/.local/lib/python3.8/site-packages/datasets/arrow_dataset.py:2139, in Dataset._getitem(self, key, decoded, **kwargs)
   2137 formatter = get_formatter(format_type, features=self.features, decoded=decoded, **format_kwargs)
   2138 pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
-> 2139 formatted_output = format_table(
   2140     pa_subtable, key, formatter=formatter, format_columns=format_columns, output_all_columns=output_all_columns
   2141 )
   2142 return formatted_output

File ~/.local/lib/python3.8/site-packages/datasets/formatting/formatting.py:532, in format_table(table, key, formatter, format_columns, output_all_columns)
    530 python_formatter = PythonFormatter(features=None)
    531 if format_columns is None:
...
-> 3068     fp = builtins.open(filename, "rb")
   3069     exclusive_fp = True
   3071 try:

FileNotFoundError: [Errno 2] No such file or directory: 'test.jpg'
```

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.0-1074-azure-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Generalize meta_path json file creation in load.py [#4540]

_The documentation is not available anymore as the PR was closed or merged._

@albertvillanova,  Can you please review this PR for Issue #4540

@lhoestq  Thank you for merging the PR . Is there a slack channel for contributing to the datasets library. I would love to work on the library and make meaningful contributions.

Hi ! Sure feel free to join our discord ^^
https://discuss.huggingface.co/t/join-the-hugging-face-discord/11263 so that we can discuss together mor eeasily. Otherwise everything happens on github ;)

# What does this PR do?

## Summary

*In function `_copy_script_and_other_resources_in_importable_dir`, using string split when generating `meta_path` throws error in the edge case raised in #4540.*

## Additions
-

## Changes
- Changed meta_path to use `os.path.splitext` instead of using `str.split` to generalize code.

## Deletions
-
## Issues Addressed :

Fixes #4540


Permission denied: '/home/.cache' when load_dataset with local script

Something isn't working


Host head_qa data on the Hub and fix NonMatchingChecksumError

Hi @albertvillanova ! Thanks for the fix ;)
Can I safely checkout from this branch to build `datasets` or it is preferable to wait until all CI tests pass?
Thanks 🙏

@younesbelkada we have just merged this PR.

This PR:
- Hosts head_qa data on the Hub instead of Google Drive
- Fixes NonMatchingChecksumError

Fix https://huggingface.co/datasets/head_qa/discussions/1


Validate new_fingerprint passed by user

Users can pass the dataset fingerprint they want in `map` and other dataset transforms.

However the fingerprint is used to name cache files so we need to make sure it doesn't contain bad characters as mentioned in https://github.com/huggingface/datasets/issues/1718, and that it's not too long


Host pn_summary data on the Hub instead of Google Drive

Fix #4581.


Host multi_news data on the Hub instead of Google Drive

Host data files of multi_news dataset on the Hub.

They were on Google Drive.

Fix #4580.


Add binary classification task IDs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4584). All of your documentation changes will be reflected on that endpoint.

> Awesome thanks ! Can you add it to https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts first please ? This is where we define the cross libraries tasks taxonomy ;)

Thanks for the tip! Done in https://github.com/huggingface/hub-docs/pull/217

As a precursor to aligning the task IDs in `datasets` and AutoTrain, we need a way to distinguish binary vs multiclass vs multilabel classification.

This PR adds binary classification to the task IDs to enable this.

Related AutoTrain issue: https://github.com/huggingface/autonlp-backend/issues/597

cc @abhishekkrthakur @SBrandeis


<code> implementation of FLAC support using torchaudio

I had added Audio FLAC support with torchaudio given that Librosa and SoundFile can give problems. Also, FLAC is been used as audio from https://mlcommons.org/en/peoples-speech/


add_column should preserve _indexes

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4582). All of your documentation changes will be reflected on that endpoint.

https://github.com/huggingface/datasets/issues/3769#issuecomment-1167146126

doing `.add_column("x",x_data)` also removed any `_indexes` on the dataset, decided this shouldn't be the case.

This was because `add_column` was creating a new `Dataset(...)` and wasn't possible to pass indexes on init.
with this PR now can pass 'indexes' on init through `IndexableMixin`

- [x] Added test


Dataset Viewer issue for pn_summary

Note that I refreshed twice this dataset, and I still have (another) error on one of the splits

```
Status code:   400
Exception:     ClientResponseError
Message:       403, message='Forbidden', url=URL('https://doc-14-4c-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/pgotjmcuh77q0lk7p44rparfrhv459kp/1656403650000/11771870722949762109/*/16OgJ_OrfzUF_i3ftLjFn9kpcyoi7UJeO?e=download')
```

Like the three splits are processed in parallel by the workers, I imagine that the Google hosting is rate-limiting us.

cc @albertvillanova

Exactly, Google Drive bans our loading scripts.

When possible, we should host somewhere else.

### Link

https://huggingface.co/datasets/pn_summary/viewer/1.0.0/validation

### Description

Getting an index error on the `validation` and `test` splits:

```
Server error
Status code:   400
Exception:     IndexError
Message:       list index out of range
```

### Owner

No


Dataset Viewer issue for multi_news

Thanks for reporting, @lewtun.

I forced the refreshing of the preview and it worked OK for train and validation splits.

I guess the error has to do with the data files being hosted at Google Drive: this gives errors when requested automatically using scripts.
We should host them to fix the error. Let's see if the license allows that.

I guess we can host the data: https://github.com/Alex-Fabbri/Multi-News/blob/master/LICENSE.txt

### Link

https://huggingface.co/datasets/multi_news

### Description

Not sure what the index error is referring to here:

```
Status code:   400
Exception:     IndexError
Message:       list index out of range
```

### Owner

No


Support streaming cfq dataset

@lhoestq I've been refactoring a little the code:
- Use less RAM by loading only the required samples: only if its index is in the splits file
- Start yielding "earlier" in streaming mode: for each `split_idx`:
  - either yield from buffer
  - or iterate over samples and either yield or buffer the sample

  The speed gain obviously depends on how the indexes are sorted in the split file:
    - Best case: indices are [1, 2, 3]
    - Worst case (no speed gain): indices are [3, 1, 2] or [3, 2, 1]

Let me know what you think.

I have to update the dummy data so that it aligns with the real data (inside the archive, the samples file `dataset.json` is the last member).

There is an issue when testing `test_load_dataset_cfq` with dummy data:
- `MockDownloadManager.iter_archive` yields FIRST `'cfq/dataset.json'`
- [`Streaming`]`DownloadManager.iter_archive` yields LAST `'cfq/dataset.json'` when using real data tar.gz archive

Note that this issue arises only with dummy data: loading the real dataset works smoothly for all configurations: I recreated the `dataset_infos.json` file to check it (it generated the same file).

Impressive, thank you ! :o

feel free to merge master into this branch, now that the files order is respected. You can merge if the CI is green :)

Support streaming cfq dataset.


[Multi Configs] Use directories to differentiate between subsets/configurations

enhancement

New feature or request

Currently to define several subsets/configurations of your dataset, you need to use a dataset script.

However it would be nice to have a no-code way to to this.

For example we could specify different configurations of a dataset (for example, if a dataset contains different languages) with one directory per configuration.

These structures are not supported right now, but would be nice to have:

```
my_dataset_repository/
├── README.md
├── en/
│   ├── train.csv
│   └── test.csv
└── fr/
    ├── train.csv
    └── test.csv
```

Or with one directory per split:

```
my_dataset_repository/
├── README.md
├── en/
│   ├── train/
│   │   ├── shard_0.csv
│   │   └── shard_1.csv
│   └── test/
│       ├── shard_0.csv
│       └── shard_1.csv
└── fr/
    ├── train/
    │   ├── shard_0.csv
    │   └── shard_1.csv
    └── test/
        ├── shard_0.csv
        └── shard_1.csv
```

cc @stevhliu @albertvillanova


Add authentication tip to `load_dataset`

_The documentation is not available anymore as the PR was closed or merged._

Add an authentication tip similar to the one in transformers' `PreTrainedModel.from_pretrained` to `load_dataset`/`load_dataset_builder`.


I still don't know if the way we implemented data files resolution could support the metadata.jsonl file without bad side effects for the other packaged builders. In particular here if you have a folder of csv/parquet/whatever files and a metadata.jsonl file, it would return
```
split: patterns_dict[split] + [METADATA_PATTERN]
```
which is a bit unexpected and can lead to errors.

Maybe this logic can be specific to imagefolder somehow ? This could be an additional pattern `["metadata.jsonl", "**/metadata.jsonl"]` just for imagefolder, that is only used when `data_files=` is not specified by the user.

I guess it's ok to have patterns that lead to duplicate metadata.jsonl files for imagefolder, since the imagefolder logic only considers the closest metadata file for each image.

What do you think ?

Yes, that's indeed the problem. My solution in https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 that accounts for that (include metadata files only if image files are present; not ideal): https://github.com/huggingface/datasets/blob/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95/src/datasets/data_files.py#L119-L125.
Perhaps a cleaner approach would be to check for metadata files after the packaged module type is inferred as `imagefolder` and append metadata files to already resolved data files (if there are any). WDYT?

@lhoestq

> Perhaps a cleaner approach would be to check for metadata files after the packaged module type is inferred as imagefolder and append metadata files to already resolved data files (if there are any). WDYT?

I decided to go with this approach.

 Not sure if you meant the same thing with this comment:

> Maybe this logic can be specific to imagefolder somehow ? This could be an additional pattern ["metadata.jsonl", "**/metadata.jsonl"] just for imagefolder, that is only used when data_files= is not specified by the user.

It adds more code but is easy to follow IMO.

The CI still struggles but you can merge since at least one of the two WIN CI succeeded

Include `metadata.jsonl` in resolved data files.

Fix #4548

@lhoestq ~~https://github.com/huggingface/datasets/commit/d94336d30eef17fc9abc67f67fa1c139661f4e75 adds support for metadata files placed at the root, and https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 accounts for nested metadata files also, but this results in more complex code. Let me know which one of these two approaches you prefer.~~ Maybe https://github.com/huggingface/datasets/commit/d94336d30eef17fc9abc67f67fa1c139661f4e75 is good enough for now (for the sake of simplicity). https://github.com/huggingface/datasets/commit/4d20618ea7a19bc143ddc5fdff9d79e671fcbb95 breaks the imagefolder tests due to duplicates in the resolved metadata files. One way to fix this would be to resolve the metadata pattern only on parent directories, but this adds even more logic to `_get_data_files_patterns`, so not sure if this is what we should do.


Problem about wmt17 zh-en dataset

bug

Something isn't working

Running into the same error with `wmt17/zh-en`, `wmt18/zh-en` and `wmt19/zh-en`.

It seems that in subset casia2015, some samples are like `{'c[hn]':'xxx', 'en': 'aa'}`.
So when using `data = load_dataset('wmt17', "zh-en")` to load the wmt17 zh-en dataset, which will raise the exception:
```
Traceback (most recent call last):
  File "train.py", line 78, in <module>
    data = load_dataset(args.dataset, "zh-en")
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 1684, in load_dataset
    use_auth_token=use_auth_token,
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 705, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 1221, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 1215, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 533, in finalize
    self.write_examples_on_file()
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 410, in write_examples_on_file
    self.write_batch(batch_examples=batch_examples)
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 503, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 230, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py", line 198, in __arrow_array__
    out = cast_array_to_feature(out, type, allow_number_to_str=not self.trying_type)
  File "/usr/local/lib/python3.7/dist-packages/datasets/table.py", line 1675, in wrapper
    return func(array, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/table.py", line 1846, in cast_array_to_feature
    return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
  File "/usr/local/lib/python3.7/dist-packages/datasets/table.py", line 1675, in wrapper
    return func(array, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/table.py", line 1756, in array_cast
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{pa_type}")
TypeError: Couldn't cast array of type
struct<c[hn]: string, en: string, zh: string>
to
struct<en: string, zh: string>
```

So the solution of this problem is to change the original array manually:
```
if 'c[hn]' in str(array.type):
        py_array = array.to_pylist()
        data_list = []
        for vo in py_array:
            tmp = {
                'en': vo['en'],
            }
            if 'zh' not in vo:
                tmp['zh'] = vo['c[hn]']
            else:
                tmp['zh'] = vo['zh']
            data_list.append(tmp)
        array = pa.array(data_list, type=pa.struct([
            pa.field('en', pa.string()),
            pa.field('zh', pa.string()),
        ]))
```

Therefore, maybe a correct version of original casia2015 file need to be updated


Support streaming mlsum dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4574). All of your documentation changes will be reflected on that endpoint.

Support streaming mlsum dataset.

Fix #4572.


Fix evaluation metadata for ncbi_disease

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4573). All of your documentation changes will be reflected on that endpoint.

This PR fixes the task in the evaluation metadata and removes the metrics info as we've decided this is not a great way to propagate this information downstream.


Dataset Viewer issue for mlsum

Thanks for reporting, @lewtun.

After investigation, it seems that the server https://gitlab.lip6.fr does not allow HTTP Range requests.

We are trying to find a workaround...

### Link

https://huggingface.co/datasets/mlsum/viewer/de/train

### Description

There's seems to be a problem with the download / streaming of this dataset:

```
Server error
Status code:   400
Exception:     BadZipFile
Message:       File is not a zip file
```

### Owner

No


Dataset Viewer issue for gsarti/flores_101

Related to https://github.com/huggingface/datasets/issues/4562#issuecomment-1166911751

I'll assign @albertvillanova

I'm just wondering why we don't have this dataset under:
- the `facebook` namespace
- or the canonical dataset `flores`: why does this only have 2 languages?

### Link

https://huggingface.co/datasets/gsarti/flores_101

### Description

It seems like streaming isn't supported for this dataset:

```
Server Error
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://dl.fbaipublicfiles.com/flores101/dataset/flores101_dataset.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

### Owner

No


Dataset sharding non-contiguous?

Something isn't working

This was silly; I was sure I'd looked for a `contiguous` argument, and was certain there wasn't one the first time I looked :smile:

Sorry about that.

Hi! You can pass `contiguous=True` to `.shard()` get contiguous shards. More info on this and the default behavior can be found in the [docs](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.shard).

EDIT: Answered as you closed the thread 😄

Hahaha I'm sorry; my excuse is: it's Sunday. (Which makes me all the more grateful for your response :smiley:

@mariosasko Sorry for reviving this, but I was curious as to why `contiguous=False` was the default. This might be a personal bias, but I feel that a user would expect the opposite to be the default. :thinking:

This project started as a fork of TFDS, and `contiguous=False` is the default behavior [there](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard).

## Describe the bug
I'm not sure if this is a bug; more likely normal behavior but i wanted to double check.
Is it normal that `datasets.shard` does not produce chunks that, when concatenated produce the original ordering of the sharded dataset?

This might be related to this pull request (https://github.com/huggingface/datasets/pull/4466) but I have to admit I did not properly look into the changes made.

## Steps to reproduce the bug
```python
max_shard_size = convert_file_size_to_int('300MB')
dataset_nbytes = dataset.data.nbytes
num_shards = int(dataset_nbytes / max_shard_size) + 1
num_shards = max(num_shards, 1)
print(f"{num_shards=}")
for shard_index in range(num_shards):
    shard = dataset.shard(num_shards=num_shards, index=shard_index)
    shard.to_parquet(f"tokenized/tokenized-{shard_index:03d}.parquet")
os.listdir('tokenized/')
```

## Expected results
I expected the shards to match the order of the data of the original dataset; i.e. `dataset[10]` being the same as `shard_1[10]` for example

## Actual results
Only the first element is the same; i.e. `dataset[0]` is the same as `shard_1[0]`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.15.0-176-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Dataset Viewer issue for sst2

Hi @lewtun, thanks for reporting.

I have checked locally and refreshed the preview and it seems working smooth now:
```python
In [8]: ds
Out[8]:
DatasetDict({
    train: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 872
    })
    test: Dataset({
        features: ['idx', 'sentence', 'label'],
        num_rows: 1821
    })
})
```

Could you confirm?

Thanks @albertvillanova - it is indeed working now (not sure what caused the error in the first place). Closing this :)

### Link

https://huggingface.co/datasets/sst2

### Description

Not sure what is causing this, however it seems that `load_dataset("sst2")` also hangs (even though it downloads the files without problem):

```
Status code:   400
Exception:     Exception
Message:       Give up after 5 attempts with ConnectionError
```

### Owner

No


XNLI cache reload is very slow

Something isn't working

Hi,
Could you tell us how you are running this code?
I tested on my machine (M1 Mac). And it is running fine both on and off internet.

<img width="1033" alt="Screen Shot 2022-07-03 at 1 32 25 AM" src="https://user-images.githubusercontent.com/8711912/177026364-4ad7cedb-e524-4513-97f7-7961bbb34c90.png">
Tested on both stable and dev version.

Sure, I was running it on a Linux machine.
I found that if I turn the Internet off, it would still try to make a HTTPS call which would slow down the cache loading. If you can't reproduce then we can close the issue.

Hi @Muennighoff! You can set the env variable `HF_DATASETS_OFFLINE` to `1` to avoid this behavior in offline mode. More info is available [here](https://huggingface.co/docs/datasets/master/en/loading#offline).

### Reproduce

Using `2.3.3.dev0`

`from datasets import load_dataset`
`load_dataset("xnli", "en")`
Turn off Internet
`load_dataset("xnli", "en")`

I cancelled the second `load_dataset` eventually cuz it took super long. It would be great to have something to specify e.g. `only_load_from_cache`  and avoid the library trying to download when there is no Internet. If I leave it running it works but takes way longer than when there is Internet. I would expect loading from cache to take the same amount of time regardless of whether there is Internet.
```
---------------------------------------------------------------------------
gaierror                                  Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self)
    174             conn = connection.create_connection(
--> 175                 (self._dns_host, self.port), self.timeout, **extra_kw
    176             )

/opt/conda/lib/python3.7/site-packages/urllib3/util/connection.py in create_connection(address, timeout, source_address, socket_options)
     71
---> 72     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
     73         af, socktype, proto, canonname, sa = res

/opt/conda/lib/python3.7/socket.py in getaddrinfo(host, port, family, type, proto, flags)
    751     addrlist = []
--> 752     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    753         af, socktype, proto, canonname, sa = res

gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

KeyboardInterrupt                         Traceback (most recent call last)
/tmp/ipykernel_33/3594208039.py in <module>
----> 1 load_dataset("xnli", "en")

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1673         revision=revision,
   1674         use_auth_token=use_auth_token,
-> 1675         **config_kwargs,
   1676     )
   1677

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, **config_kwargs)
   1494         download_mode=download_mode,
   1495         data_dir=data_dir,
-> 1496         data_files=data_files,
   1497     )
   1498

/opt/conda/lib/python3.7/site-packages/datasets/load.py in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1182                     download_config=download_config,
   1183                     download_mode=download_mode,
-> 1184                     dynamic_modules_path=dynamic_modules_path,
   1185                 ).get_module()
   1186             elif path.count("/") == 1:  # community dataset on the Hub

/opt/conda/lib/python3.7/site-packages/datasets/load.py in __init__(self, name, revision, download_config, download_mode, dynamic_modules_path)
    506         self.dynamic_modules_path = dynamic_modules_path
    507         assert self.name.count("/") == 0
--> 508         increase_load_count(name, resource_type="dataset")
    509
    510     def download_loading_script(self, revision: Optional[str]) -> str:

/opt/conda/lib/python3.7/site-packages/datasets/load.py in increase_load_count(name, resource_type)
    166     if not config.HF_DATASETS_OFFLINE and config.HF_UPDATE_DOWNLOAD_COUNTS:
    167         try:
--> 168             head_hf_s3(name, filename=name + ".py", dataset=(resource_type == "dataset"))
    169         except Exception:
    170             pass

/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py in head_hf_s3(identifier, filename, use_cdn, dataset, max_retries)
     93     return http_head(
     94         hf_bucket_url(identifier=identifier, filename=filename, use_cdn=use_cdn, dataset=dataset),
---> 95         max_retries=max_retries,
     96     )
     97

/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py in http_head(url, proxies, headers, cookies, allow_redirects, timeout, max_retries)
    445         allow_redirects=allow_redirects,
    446         timeout=timeout,
--> 447         max_retries=max_retries,
    448     )
    449     return response

/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py in _request_with_retry(method, url, max_retries, base_wait_time, max_wait_time, timeout, **params)
    366         tries += 1
    367         try:
--> 368             response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
    369             success = True
    370         except (requests.exceptions.ConnectTimeout, requests.exceptions.ConnectionError) as err:

/opt/conda/lib/python3.7/site-packages/requests/api.py in request(method, url, **kwargs)
     59     # cases, and look like a memory leak in others.
     60     with sessions.Session() as session:
---> 61         return session.request(method=method, url=url, **kwargs)
     62
     63

/opt/conda/lib/python3.7/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    527         }
    528         send_kwargs.update(settings)
--> 529         resp = self.send(prep, **send_kwargs)
    530
    531         return resp

/opt/conda/lib/python3.7/site-packages/requests/sessions.py in send(self, request, **kwargs)
    643
    644         # Send the request
--> 645         r = adapter.send(request, **kwargs)
    646
    647         # Total elapsed time of the request (approximately)

/opt/conda/lib/python3.7/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    448                     decode_content=False,
    449                     retries=self.max_retries,
--> 450                     timeout=timeout
    451                 )
    452

/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    708                 body=body,
    709                 headers=headers,
--> 710                 chunked=chunked,
    711             )
    712

/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    384         # Trigger any extra validation we need to do.
    385         try:
--> 386             self._validate_conn(conn)
    387         except (SocketTimeout, BaseSSLError) as e:
    388             # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.

/opt/conda/lib/python3.7/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
   1038         # Force connect early to allow us to validate the connection.
   1039         if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
-> 1040             conn.connect()
   1041
   1042         if not conn.is_verified:

/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in connect(self)
    356     def connect(self):
    357         # Add certificate verification
--> 358         self.sock = conn = self._new_conn()
    359         hostname = self.host
    360         tls_in_tls = False

/opt/conda/lib/python3.7/site-packages/urllib3/connection.py in _new_conn(self)
    173         try:
    174             conn = connection.create_connection(
--> 175                 (self._dns_host, self.port), self.timeout, **extra_kw
    176             )
    177

KeyboardInterrupt:
```


Add evaluation data for amazon_reviews_multi

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4567). All of your documentation changes will be reflected on that endpoint.


Document link #load_dataset_enhancing_performance points to nowhere

Something isn't working

Hi! This is indeed the link the docstring should point to. Are you interested in submitting a PR to fix this?

https://github.com/huggingface/datasets/blame/master/docs/source/cache.mdx#L93

There seems already an anchor here. Somehow it doesn't work. I am not very familiar with how this online documentation works.

## Describe the bug
A clear and concise description of what the bug is.

![image](https://user-images.githubusercontent.com/11674033/175752806-5b066b92-9d28-4771-9112-5c8606f07741.png)

The [load_dataset_enhancing_performance](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#load_dataset_enhancing_performance) link [here](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.load_from_disk.keep_in_memory) points to nowhere, I guess it should point to https://huggingface.co/docs/datasets/v2.3.2/en/cache#improve-performance?


Add UFSC OCPap dataset

dataset request

Requesting to add a new dataset

I will add this directly on the hub (same as #4486)—in https://huggingface.co/lapix

## Adding a Dataset
- **Name:** UFSC OCPap: Papanicolaou Stained Oral Cytology Dataset (v4)
- **Description:** The UFSC OCPap dataset comprises 9,797 labeled images of 1200x1600 pixels acquired from 5 slides of cancer diagnosed and 3 healthy of oral brush samples, from distinct patients.
- **Paper:** https://dx.doi.org/10.2139/ssrn.4119212
- **Data:**  https://data.mendeley.com/datasets/dr7ydy9xbk/1
- **Motivation:** real data of pap stained oral cytology samples

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Support streaming bookcorpus dataset

_The documentation is not available anymore as the PR was closed or merged._

Support streaming bookcorpus dataset.


Support streaming allocine dataset

Support streaming allocine dataset.

Fix #4562.


Dataset Viewer issue for allocine

I removed my assignment as @huggingface/datasets should be able to answer better than me

Thanks for the quick fix @albertvillanova

Note that the underlying issue is that datasets containing TAR files are not streamable out of the box: they need being iterated with `dl_manager.iter_archive` to avoid performance issues because they access their file content *sequentially* (no random access).

> Note that the underlying issue is that datasets containing TAR files are not streamable out of the box: they need being iterated with `dl_manager.iter_archive` to avoid performance issues because they access their file content _sequentially_ (no random access).

Ah thanks for the clarification! I'll look out for this next time and implement the fix myself :)

### Link

https://huggingface.co/datasets/allocine

### Description

Not sure if this is a problem with `bz2` compression, but I thought these datasets could be streamed:

```
Status code:   400
Exception:     AttributeError
Message:       'TarContainedFile' object has no attribute 'readable'
```

### Owner

No


Add evaluation data to acronym_identification


Add evaluation metadata to imagenet-1k

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4560). All of your documentation changes will be reflected on that endpoint.


Add action names in schema_guided_dstc8 dataset card

As aseked in https://huggingface.co/datasets/schema_guided_dstc8/discussions/1, I added the action names in the dataset card


Add evaluation metadata to wmt14

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4558). All of your documentation changes will be reflected on that endpoint.


Add evaluation metadata to wmt16

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4557). All of your documentation changes will be reflected on that endpoint.

> Just to confirm: we should add this metadata via GitHub and not Hub PRs for canonical datasets right?

yes :)


Dataset Viewer issue for conll2003

### Link

https://huggingface.co/datasets/conll2003/viewer/conll2003/test

### Description

Seems like a cache problem with this config / split:

```
Server error
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/cache/modules/datasets_modules/datasets/conll2003/__init__.py'
```

### Owner

No


Dataset Viewer issue for xtreme

### Link

https://huggingface.co/datasets/xtreme/viewer/PAN-X.de/test

### Description

There seems to be a problem with the cache of this config / split:

```
Server error
Status code:   400
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: '/cache/modules/datasets_modules/datasets/xtreme/349258adc25bb45e47de193222f95e68a44f7a7ab53c4283b3f007208a11bf7e/xtreme.py'
```

### Owner

No


Fix WMT dataset loading issue and docs update (Re-opened)

This PR is a fix for #4354

Changes are made for `wmt14`, `wmt15`, `wmt16`, `wmt17`, `wmt18`, `wmt19` and `wmt_t2t`. And READMEs are updated for the corresponding datasets.

Let me know, if any additional changes are required.

Thanks


Stop dropping columns in to_tf_dataset() before we load batches

@lhoestq Rebasing fixed the test failures, so this should be ready to review now! There's still a failure on Win but it seems unrelated.

Gentle ping @lhoestq ! This is a simple fix (dropping columns after loading a batch from the dataset rather than with `.remove_columns()` to make sure we don't break transforms), and tests are green so we're ready for review!

`to_tf_dataset()` dropped unnecessary columns before loading batches from the dataset, but this is causing problems when using a transform, because the dropped columns might be needed to compute the transform. Since there's no real way to check which columns the transform might need, we skip dropping columns and instead drop keys from the batch after we load it.

cc @amyeroberts and https://github.com/huggingface/notebooks/pull/202


Tell users to upload on the hub directly

Thanks ! I updated the two remaining files

As noted in https://github.com/huggingface/datasets/pull/4534, it is still not clear that it is recommended to add datasets on the Hugging Face Hub directly instead of GitHub, so I updated some docs.

Moreover since users won't be able to get reviews from us on the Hub, I added a paragraph to tell users that they can open a discussion and tag `datasets` maintainers for reviews.

Finally I removed the _previous good reasons_ to add a dataset on GitHub to only keep this one:

> In some rare cases it makes more sense to open a PR on GitHub. For example when you are not the author of the dataset and there is no clear organization / namespace that you can put the dataset under.

Does it sound good to you @albertvillanova @julien-c ?


Perform hidden file check on relative data file path

I'm aware of this behavior, which is tricky to solve due to fsspec's hidden file handling (see https://github.com/huggingface/datasets/issues/4115#issuecomment-1108819538). I've tested some regex patterns to address this, and they seem to work (will push them on Monday; btw they don't break any of fsspec's tests, so maybe we can contribute this as an enhancement to them). Also, perhaps we should include the files starting with `__`  in the results again (we hadn't had issues with this pattern before). WDYT?

I see. Feel free to merge this one if it's good for you btw :)

> Also, perhaps we should include the files starting with __ in the results again (we hadn't had issues with this pattern before)

The point was mainly to ignore `__pycache__` directories for example. Also also for consistency with the iter_files/iter_archive which are already ignoring them

Very elegant solution! Feel free to merge if the CI is green after adding the tests.

Fix #4549


imdb source error

bug

Something isn't working

Thanks for reporting, @Muhtasham.

Indeed IMDB dataset is not accessible from yesterday, because the data is hosted on the data owners servers at Stanford (http://ai.stanford.edu/) and these are down due to a power outage originated by a fire: https://twitter.com/StanfordAILab/status/1539472302399623170?s=20&t=1HU1hrtaXprtn14U61P55w

As a temporary workaroud, you can load the IMDB dataset with this tweak:
```python
ds = load_dataset("imdb", revision="tmp-fix-imdb")
```

## Describe the bug
imdb dataset not loading

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("imdb")
```

## Expected results

## Actual results
```bash
06/23/2022 14:45:18 - INFO - datasets.builder - Dataset not on Hf google storage. Downloading and preparing it from source
06/23/2022 14:46:34 - INFO - datasets.utils.file_utils - HEAD request to http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz timed out, retrying... [1.0]
.....
ConnectionError: Couldn't reach http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz (ConnectTimeout(MaxRetryError("HTTPConnectionPool(host='ai.stanford.edu', port=80): Max retries exceeded with url: /~amaas/data/sentiment/aclImdb_v1.tar.gz (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f2d750cf690>, 'Connection to ai.stanford.edu timed out. (connect timeout=100)'))")))
```
## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


FileNotFoundError when passing a data_file inside a directory starting with double underscores

Something isn't working

I have consistently experienced this bug on GitHub actions when bumping to `2.3.2`

Bug experienced in the `accelerate` CI: https://github.com/huggingface/accelerate/runs/7016055148?check_suite_focus=true

This is related to https://github.com/huggingface/datasets/pull/4505 and the changes from https://github.com/huggingface/datasets/pull/4412


Metadata.jsonl for Imagefolder is ignored if it's in a parent directory to the splits directories/do not have "{split}_" prefix

I agree it would be nice to support this. It doesn't fit really well in the current data_files.py, where files of each splits are separated in different folder though, maybe we have to modify a bit the logic here.

One idea would be to extend `get_patterns_in_dataset_repository` and `get_patterns_locally` to additionally check for `metadata.json`, but feel free to comment if you have better ideas (I feel like we're reaching the limits of what the current implementation IMO, so we could think of a different way of resolving the data files if necessary)

If data contains a single `metadata.jsonl` file for several splits, it won't be included in a dataset's `data_files` and therefore ignored.
This happens when a directory is structured like as follows:
```
train/
  file_1.jpg
  file_2.jpg
test/
  file_3.jpg
  file_4.jpg
metadata.jsonl
```
or like as follows:
```
train_file_1.jpg
train_file_2.jpg
test_file_3.jpg
test_file_4.jpg
metadata.jsonl
```
The same for HF repos.

because it's ignored by the patterns [here](https://github.com/huggingface/datasets/blob/master/src/datasets/data_files.py#L29)

@lhoestq @mariosasko Do you think it's better to add this functionality in `data_files.py` or just specifically in imagefolder/audiofolder code? In `data_files.py` would me more general but I don't know if there are any other cases when that might be needed.


[CI] Fix some warnings

_The documentation is not available anymore as the PR was closed or merged._

There is a CI failure only related to the missing content of the universal_dependencies dataset card, we can ignore this failure in this PR

good catch, I thought I resolved them all sorry

There are some warnings in the CI that are annoying, I tried to remove most of them


[CI] fixing seqeval install in ci by pinning setuptools-scm

The latest setuptools-scm version supported on 3.6 is 6.4.2. However for some reason circleci has version 7, which doesn't work.

I fixed this by pinning the version of setuptools-scm in the circleci job

Fix https://github.com/huggingface/datasets/issues/4544


Make DuplicateKeysError more user friendly [For Issue #2556]

> Nice thanks !
>
> After your changes feel free to mark this PR as "ready for review" ;)

Marking PR ready for review.

@lhoestq Let me know if there is anything else required or if we are good to go ahead and merge.

# What does this PR do?

## Summary

*DuplicateKeysError error does not provide any information regarding the examples which have the same the key.*

*This information is very helpful for debugging the dataset generator script.*

## Additions
-

## Changes
- Changed `DuplicateKeysError Class` in `src/datasets/keyhash.py` to add current index and duplicate_key_indices to error message.
- Changed `check_duplicate_keys` function in `src/datasets/arrow_writer.py` to find indices of examples with duplicate hash if duplicate keys are found.

## Deletions
-

## To do  :
- [x] Find way to find and print path `<Path to Dataset>` in Error message

## Issues Addressed :

Fixes #2556


[CI] seqeval installation fails sometimes on python 3.6

The CI sometimes fails to install seqeval, which cause the `seqeval` metric tests to fail.

The installation fails because of this error:
```
Collecting seqeval
  Downloading seqeval-1.2.2.tar.gz (43 kB)

     |███████▌                        | 10 kB 42.1 MB/s eta 0:00:01
     |███████████████                 | 20 kB 53.3 MB/s eta 0:00:01
     |██████████████████████▌         | 30 kB 67.2 MB/s eta 0:00:01
     |██████████████████████████████  | 40 kB 76.1 MB/s eta 0:00:01
     |████████████████████████████████| 43 kB 10.0 MB/s
  Preparing metadata (setup.py) ... - error
  ERROR: Command errored out with exit status 1:
   command: /home/circleci/.pyenv/versions/3.6.15/bin/python3.6 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-1l96tbyj/seqeval_b31086f711d84743abe6905d2aa9dade/setup.py'"'"'; __file__='"'"'/tmp/pip-install-1l96tbyj/seqeval_b31086f711d84743abe6905d2aa9dade/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-pf54_vqy
       cwd: /tmp/pip-install-1l96tbyj/seqeval_b31086f711d84743abe6905d2aa9dade/
  Complete output (22 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-1l96tbyj/seqeval_b31086f711d84743abe6905d2aa9dade/setup.py", line 56, in <module>
      'Programming Language :: Python :: Implementation :: PyPy'
    File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/setuptools/__init__.py", line 143, in setup
      return distutils.core.setup(**attrs)
    File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/distutils/core.py", line 108, in setup
      _setup_distribution = dist = klass(attrs)
    File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/setuptools/dist.py", line 442, in __init__
      k: v for k, v in attrs.items()
    File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/distutils/dist.py", line 281, in __init__
      self.finalize_options()
    File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/setuptools/dist.py", line 601, in finalize_options
      ep.load()(self, ep.name, value)
    File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2346, in load
      return self.resolve()
    File "/home/circleci/.pyenv/versions/3.6.15/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2352, in resolve
      module = __import__(self.module_name, fromlist=['__name__'], level=0)
    File "/tmp/pip-install-1l96tbyj/seqeval_b31086f711d84743abe6905d2aa9dade/.eggs/setuptools_scm-7.0.2-py3.6.egg/setuptools_scm/__init__.py", line 5
      from __future__ import annotations
      ^
  SyntaxError: future feature annotations is not defined
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/9d/2d/233c79d5b4e5ab1dbf111242299153f3caddddbb691219f363ad55ce783d/seqeval-1.2.2.tar.gz#sha256=f28e97c3ab96d6fcd32b648f6438ff2e09cfba87f05939da9b3970713ec56e6f (from https://pypi.org/simple/seqeval/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

```

for example in https://app.circleci.com/pipelines/github/huggingface/datasets/12665/workflows/93878eb9-a923-4b35-b2e7-c5e9b22f10ad/jobs/75300

Here is a diff of the pip install logs until the error is reached: https://www.diffchecker.com/VkQDLeQT

This could be caused by the latest updates of setuptools-scm


[CI] Fix upstream hub test url

Remaining CI failures are unrelated to this fix, merging

Some tests were still using moon-stagign instead of hub-ci.

I also updated the token to use one dedicated to `datasets`