load_datasets on csv returns everything as a string

Something isn't working

Hi @courtneysprouse, thanks for reporting.

Yes, you are right: by default the "csv" loader loads all columns as strings.

You could tweak this behavior by passing the `feature` argument to `load_dataset`, but it is also true that currently it is not possible to perform some kind of casts, due to lacking of implementation in PyArrow. For example:
```python
import datasets

features = datasets.Features(
    {
        "tokens": datasets.Sequence(datasets.Value("string")),
        "ner_tags": datasets.Sequence(datasets.Value("int32")),
    }
)

new_conll = datasets.load_dataset("csv", data_files="ner_conll.csv", features=features)
```
gives `ArrowNotImplementedError` error:
```
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Unsupported cast from string to list using function cast_list
```

On the other hand, if you just would like to save and afterwards load your dataset, you could use `save_to_disk` and `load_from_disk` instead. These functions preserve all data types.
```python
>>> orig_conll.save_to_disk("ner_conll")

>>> from datasets import load_from_disk

>>> new_conll = load_from_disk("ner_conll")
>>> new_conll
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 14042
    })
    validation: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 3251
    })
    test: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 3454
    })
})
>>> new_conll["train"][0]
{'chunk_tags': [11, 21, 11, 12, 21, 22, 11, 12, 0],
 'id': '0',
 'ner_tags': [3, 0, 7, 0, 0, 0, 7, 0, 0],
 'pos_tags': [22, 42, 16, 21, 35, 37, 16, 21, 7],
 'tokens': ['EU',
  'rejects',
  'German',
  'call',
  'to',
  'boycott',
  'British',
  'lamb',
  '.']}
>>> new_conll["train"].features
{'chunk_tags': Sequence(feature=ClassLabel(num_classes=23, names=['O', 'B-ADJP', 'I-ADJP', 'B-ADVP', 'I-ADVP', 'B-CONJP', 'I-CONJP', 'B-INTJ', 'I-INTJ', 'B-LST', 'I-LST', 'B-NP', 'I-NP', 'B-PP', 'I-PP', 'B-PRT', 'I-PRT', 'B-SBAR', 'I-SBAR', 'B-UCP', 'I-UCP', 'B-VP', 'I-VP'], id=None), length=-1, id=None),
 'id': Value(dtype='string', id=None),
 'ner_tags': Sequence(feature=ClassLabel(num_classes=9, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC'], id=None), length=-1, id=None),
 'pos_tags': Sequence(feature=ClassLabel(num_classes=47, names=['"', "''", '#', '$', '(', ')', ',', '.', ':', '``', 'CC', 'CD', 'DT', 'EX', 'FW', 'IN', 'JJ', 'JJR', 'JJS', 'LS', 'MD', 'NN', 'NNP', 'NNPS', 'NNS', 'NN|SYM', 'PDT', 'POS', 'PRP', 'PRP$', 'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', 'VBN', 'VBP', 'VBZ', 'WDT', 'WP', 'WP$', 'WRB'], id=None), length=-1, id=None),
 'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
```

Hi @albertvillanova!

Thanks so much for your suggestions! That worked!

## Describe the bug
If you use:

`conll_dataset.to_csv("ner_conll.csv")`

It will create a csv file with all of your data as expected, however when you load it with:

`conll_dataset = load_dataset("csv", data_files="ner_conll.csv")`

everything is read in as a string. For example if I look at everything in 'ner_tags' I get back `['[3 0 7 0 0 0 7 0 0]', '[1 2]', '[5 0]']` instead of what I originally saved which was `[[3, 0, 7, 0, 0, 0, 7, 0, 0], [1, 2], [5, 0]]`

I think maybe there is something funky going on with the csv delimiter

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
#load original conll dataset
orig_conll = load_dataset("conll2003")

#save original conll as a csv
orig_conll.to_csv("ner_conll.csv")

#reload conll data as a csv
new_conll = load_dataset("csv", data_files="ner_conll.csv")`
```

## Expected results
A clear and concise description of the expected results.
I would expect the data be returned as the data type I saved it as. I.e. if I save a list of ints
[[3, 0, 7, 0, 0, 0, 7, 0, 0]], I shouldnt get back a string ['[3 0 7 0 0 0 7 0 0]']
I also get back a string when I pass a list of strings ['EU', 'rejects', 'German', 'call', 'to', 'boycott', 'British', 'lamb', '.']

## Actual results
A list of strings `['[3 0 7 0 0 0 7 0 0]', '[1 2]', '[5 0]']`
A string "['EU' 'rejects' 'German' 'call' 'to' 'boycott' 'British' 'lamb' '.']"

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.0-121-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 8.0.0


Support extract 7-zip compressed data files

_The documentation is not available anymore as the PR was closed or merged._

Cool! Can you please remove `Fix #3541` from the description as this PR doesn't add support for streaming/`iter_archive`, so it only partially addresses the issue?

Side note:
I think we can use `libarchive` (`libarchive-c` is a Python package with the bindings) for streaming 7z archives. The only issue with this lib is that it's tricky to install on Windows/Mac.

Fix partially #3541, fix #4670.


Dataset Viewer issue for wmt16

dataset-viewer

Related to the dataset viewer on huggingface.co

Thanks for reporting, @lewtun.

~We can't load the dataset locally, so I think this is an issue with the loading script (not the viewer).~

 We are investigating...

Recently, there was a merged PR related to this dataset:
- #4554

We are looking at this...

Indeed, the above mentioned PR fixed the loading script (it was not working before).

I'm forcing the refresh of the Viewer.

Please note that the above mentioned PR also made an enhancement in the `datasets` library, required by this loading script. This enhancement will only be available to the Viewer once we make our next release.

### Link

https://huggingface.co/datasets/wmt16

### Description

[Reported](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/12#62cb83f14c7f35284e796f9c) by a user of AutoTrain Evaluate. AFAIK this dataset was working 1-2 weeks ago, and I'm not sure how to interpret this error.

```
Status code:   400
Exception:     NotImplementedError
Message:       This is a abstract method
```

Thanks!

### Owner

No


Can't extract files from `.7z` zipfile using `download_and_extract`

bug

Something isn't working

Hi @bhavitvyamalik, thanks for reporting.

Yes, currently we do not support 7zip archive compression: I think we should.

As a workaround, you could uncompress it explicitly, like done in e.g. `samsum` dataset:

https://github.com/huggingface/datasets/blob/fedf891a08bfc77041d575fad6c26091bc0fce52/datasets/samsum/samsum.py#L106-L110

Related to this issue: https://github.com/huggingface/datasets/issues/3541

Sure, let me look into and check what can be done. Will keep you guys updated here!

Initially, I thought of solving this without any external dependency. Almost everywhere I saw `lzma` can be used for this but there is a caveat that lzma doesn’t work with 7z archives but only single files. In my case the 7z archive has multiple files so it didn't work. Is it fine to use external library here?

Hi @bhavitvyamalik, thanks for your investigation.

On Monday, I started a PR that will eventually close this issue as well: I'm linking it to this.
- #4672

Let me know what you think.

## Describe the bug
I'm adding a new dataset which is a `.7z` zip file in Google drive and contains 3 json files inside. I'm able to download the data files using `download_and_extract` but after downloading it throws this error:
```
>>> dataset = load_dataset("./datasets/mantis/")
Using custom data configuration default
Downloading and preparing dataset mantis/default to /Users/bhavitvyamalik/.cache/huggingface/datasets/mantis/default/1.1.0/611affa804ec53e2055a335cc1b8b213bb5a0b5142d919967729d5ee23c6bab4...
Downloading data: 100%|█████████████████████████████████████████████████████████| 77.2M/77.2M [00:23<00:00, 3.28MB/s]
/Users/bhavitvyamalik/.cache/huggingface/datasets/downloads/fc3d70123c9de8407587a59aa426c37819cf2bf016795d33270e8a1d558a34e6
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bhavitvyamalik/Desktop/work/hf/datasets/src/datasets/load.py", line 1745, in load_dataset
    use_auth_token=use_auth_token,
  File "/Users/bhavitvyamalik/Desktop/work/hf/datasets/src/datasets/builder.py", line 595, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/Users/bhavitvyamalik/Desktop/work/hf/datasets/src/datasets/builder.py", line 690, in _download_and_prepare
    ) from None
OSError: Cannot find data file.
Original error:
[Errno 20] Not a directory: '/Users/bhavitvyamalik/.cache/huggingface/datasets/downloads/fc3d70123c9de8407587a59aa426c37819cf2bf016795d33270e8a1d558a34e6/merged_train.json'
```
just before generating the splits. I checked `fc3d70123c9de8407587a59aa426c37819cf2bf016795d33270e8a1d558a34e6` file and it's `7z` zip file (similar to downloaded Google drive file) which means it didn't get unzip. Do I need to unzip it separately and then pass the paths for train,dev,test files in `SplitGenerator`?

## Environment info
- `datasets` version: 1.18.4.dev0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.8
- PyArrow version: 5.0.0


loading oscar-corpus/OSCAR-2201  raises an error

Something isn't working

I had to use the appropriate token for use_auth_token. Thank you.

## Describe the bug
load_dataset('oscar-2201', 'af')

raises an error:
Traceback (most recent call last):
  File "/usr/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "..python3.8/site-packages/datasets/load.py", line 1656, in load_dataset
    builder_instance = load_dataset_builder(
  File ".../lib/python3.8/site-packages/datasets/load.py", line 1439, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File ".../lib/python3.8/site-packages/datasets/load.py", line 1189, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at .../oscar-2201/oscar-2201.py or any data file in the same directory. Couldn't find 'oscar-2201' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/oscar-2201/oscar-2201.py

I've tried other permutations such as :
oscar_22 = load_dataset('oscar-2201', 'af',use_auth_token=True)
oscar_22 = load_dataset('oscar-corpus/OSCAR-2201', 'af',use_auth_token=True)
oscar_22 = load_dataset('oscar-2201', 'af')
oscar_22 = load_dataset('oscar-corpus/OSCAR-2201')

with the same unfortunate result.

## Steps to reproduce the bug
oscar_22 = load_dataset('oscar-2201', 'af',use_auth_token=True)
oscar_22 = load_dataset('oscar-corpus/OSCAR-2201', 'af',use_auth_token=True)
oscar_22 = load_dataset('oscar-2201', 'af')
oscar_22 = load_dataset('oscar-corpus/OSCAR-2201')
# Sample code to reproduce the bug
```

## Expected results
loaded data

## Actual results
Traceback (most recent call last):
  File "/usr/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "..python3.8/site-packages/datasets/load.py", line 1656, in load_dataset
    builder_instance = load_dataset_builder(
  File ".../lib/python3.8/site-packages/datasets/load.py", line 1439, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File ".../lib/python3.8/site-packages/datasets/load.py", line 1189, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at .../oscar-2201/oscar-2201.py or any data file in the same directory. Couldn't find 'oscar-2201' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/oscar-2201/oscar-2201.py

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-5.13.0-37-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Dataset Viewer issue for hungnm/multilingual-amazon-review-sentiment-processed

It seems like a private dataset. The viewer is currently not supported on the private datasets.

### Link

https://huggingface.co/hungnm/multilingual-amazon-review-sentiment

### Description

_No response_

### Owner

Yes


duplicate

This issue or pull request already exists

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Issues with concatenating datasets

bug

Something isn't working

Hi! I agree we should improve the features equality checks to account for this particular case. However, your code fails due to `answer_start` having the dtype `int64` instead of `int32` after loading from JSON (it's not possible to embed type precision info into a JSON file; `save_to_disk` does that for arrow files), which would lead to the concatenation error as PyArrow does not support this sort of type promotion. This can be fixed as follows:
```python
temp = load_dataset("json", data_files={"train": "output.jsonl"}, features=squad["train"].features)
```

That makes sense. I totally missed the `int64` and `int32` part. Thanks for pointing it out! Will close this issue for now.

## Describe the bug
It is impossible to concatenate datasets if a feature is sequence of dict in one dataset and a dict of sequence in another. But based on the document, it should be automatically converted.

> A [datasets.Sequence](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Sequence) with a internal dictionary feature will be automatically converted into a dictionary of lists. This behavior is implemented to have a compatilbity layer with the TensorFlow Datasets library but may be un-wanted in some cases. If you don’t want this behavior, you can use a python list instead of the [datasets.Sequence](https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Sequence).

## Steps to reproduce the bug
```python
from datasets import concatenate_datasets, load_dataset

squad = load_dataset("squad_v2")
squad["train"].to_json("output.jsonl", lines=True)

temp = load_dataset("json", data_files={"train": "output.jsonl"})
concatenate_datasets([temp["train"], squad["train"]])
```

## Expected results
No error executing that code

## Actual results
```
ValueError: The features can't be aligned because the key answers of features {'id': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answers': Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None)} has unexpected type - Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None) (expected either {'text': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'answer_start': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)} or Value("null").
```

## Environment info
- `datasets` version: 2.3.2
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.8.11
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Unable to create dataset having Python dataset script only

Something isn't working

Hi @aleSuglia, thanks for reporting.

We are having a look at it.

We transfer this issue to the Community tab of the corresponding Hub dataset: https://huggingface.co/datasets/Heriot-WattUniversity/dialog-babi/discussions

## Describe the bug
Hi there,

I'm trying to add the following dataset to Huggingface datasets: https://huggingface.co/datasets/Heriot-WattUniversity/dialog-babi/blob/

I'm trying to do so using the CLI commands but seems that this command generates the wrong `dataset_info.json` file (you can find it in the repo already):
```
datasets-cli test Heriot-WattUniversity/dialog-babi/dialog_babi.py --save_infos --all-configs
```
while it errors when I remove the python script:
```
datasets-cli test Heriot-WattUniversity/dialog-babi/ --save_infos --all-configs
```
The error message is the following:
```
FileNotFoundError: Unable to resolve any data file that matches '['**']' at /Users/as2180/workspace/Heriot-WattUniversity/dialog-babi with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

## Environment info
- `datasets` version: 2.3.2
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Add stanford dog dataset

Hi @khushmeeet, thanks for your contribution.

But wouldn't it be better to add this dataset to the Hub?
- https://huggingface.co/docs/datasets/share
- https://huggingface.co/docs/datasets/dataset_script

Hi @albertvillanova

Dataset is added to Hub - https://huggingface.co/datasets/dgrnd4/stanford_dog_dataset

Great, so I guess we can close this issue, as the dataset is already available on the Hub.

OK I read the discussion on:
- #4504

Currently, priority is adding datasets to the Hub, not here on GitHub.

If you would like to contribute the loading script and all the metadata you generated (README + JSON files), you could:
- Either make a PR to the existing dataset on the Hub
- Create a new dataset on the Hub:
  - Either under your personal namespace
  - or even more professionally, under the namespace `stanfordSVL` (Stanford Vision and Learning Lab: https://svl.stanford.edu/)

You can use the Community tab to ping us if you need help or have any questions.

This PR is for adding dataset, related to issue #4504.

We are adding Stanford dog breed dataset. It is a multi class image classification dataset.
Details can be found here - http://vision.stanford.edu/aditya86/ImageNetDogs/

Tests on dummy data is failing currently, which I am looking into.


Add text decorators

_The documentation is not available anymore as the PR was closed or merged._

This PR adds some decoration to text about different modalities to make it more obvious separate guides exist for audio, vision, and text. The goal is to make it easier for users to discover these guides!

![underline](https://user-images.githubusercontent.com/59462357/178044392-9596693e-9a4a-479a-a282-f1edbd90be1a.png)

TODO:

- [x] Open PR to support new Tailwind classes


Fix: conll2003 - fix empty example

As reported in https://huggingface.co/datasets/conll2003/discussions/2#62c45a14f93fc97e8260532f, there was an extra empty example at the end of the dataset


Concurrency bug when using same cache among several jobs

Something isn't working

I can confirm that if I run one job first that processes the dataset, then I can run any jobs in parallel with no problem (no write-concurrency anymore...).

Hi! That's weird. It seems like the error points to the `mkstemp` function, but the official docs state the following:
```
There are no race conditions in the file’s creation, assuming that the platform properly implements the [os.O_EXCL](https://docs.python.org/3/library/os.html#os.O_EXCL) flag for [os.open()](https://docs.python.org/3/library/os.html#os.open)
```
So this could mean your platform doesn't support that flag.

~~Can you please check if wrapping the temp file creation (the line `tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)` in `_map_single`) with the `multiprocess.Lock` fixes the issue?~~
Perhaps wrapping the temp file creation in `_map_single`  with `filelock` could work:
```python
with FileLock(lock_path):
    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)
```
Can you please check if that helps?

## Describe the bug
I used to see this bug with an older version of the datasets. It seems to persist.

This is my concrete scenario: I launch several evaluation jobs on a cluster in which I share the file system and I share the cache directory used by huggingface libraries. The evaluation jobs read the same *.csv files. If my jobs get all scheduled pretty much at the same time, there are all kinds of weird concurrency errors. Sometime it crashes silently. This time I got lucky that it crashed with a stack trace that I can share and maybe you get to the bottom of this. If you don't have a similar setup available, it may be hard to reproduce as you really need two jobs accessing the same file at the same time to see this type of bug.

## Steps to reproduce the bug
I'm running a modified version of `run_glue.py` script adapted to my use case. I've seen the same problem when running some glue datasets as well (so it's not specific to loading the datasets from csv files).

## Expected results
No crash, concurrent access to the (intermediate) files just fine.

## Actual results
Crashes due to races/concurrency bugs.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.18.0-348.23.1.el8_5.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 8.0.0
- Pandas version: 1.1.0

Stack trace that I just got with the crash (I've obfuscated some names, it should still be quite informative):

```
Running tokenizer on dataset:   0%|          | 0/3 [00:00<?, ?ba/s]
Traceback (most recent call last):
  File "../../src/models//run_*******.py", line 600, in <module>
    main()
  File "../../src/models//run_*******.py", line 444, in main
    raw_datasets = raw_datasets.map(
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/dataset_dict.py", line 770, in map
    {
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/dataset_dict.py", line 771, in <dictcomp>
    k: dataset.map(
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2376, in map
    return self._map_single(
  File "/*******/envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 551, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 518, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/*******/envs/tr-crt/lib/python3.8/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2776, in _map_single
    buf_writer, writer, tmp_file = init_buffer_and_writer()
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2696, in init_buffer_and_writer
    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)
  File "/*******//envs/tr-crt/lib/python3.8/tempfile.py", line 541, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/*******//envs/tr-crt/lib/python3.8/tempfile.py", line 250, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: '/*******/cache-transformers//transformers/csv/default-ef9cd184210742a7/0.0.0/51cce309a08df9c4d82ffd9363bbe090bf173197fc01a71b034e8594995a1a58/tmps8l6j5yc'
```

As I ran 100s of experiments last year for an empirical paper, I ran into this type of bugs several times. I found several bandaid/work-arounds, e.g., run one job first that caches the dataset => eliminate concurrency; OR use unique caches => eliminate concurrency (but increase storage space), etc. and it all works fine.

I'd like to help you fixing this bug as it's really annoying to always apply the work arounds. Let me know what other info from my side could help you figure out the issue.

Thanks for your help!


Fix _resolve_single_pattern_locally on Windows with multiple drives

Good catch ! Sorry I forgot (again) about windows paths when writing this x)

Currently, when `_resolve_single_pattern_locally` is called from a different drive than the one in `pattern`, it raises an exception:
```
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\io\parquet.py:35: in __init__
    **kwargs,
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\builder.py:287: in __init__
    sanitize_patterns(data_files), base_path=base_path, use_auth_token=use_auth_token
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:761: in from_local_or_remote
    if not isinstance(patterns_for_key, DataFilesList)
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:723: in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:321: in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:239: in _resolve_single_pattern_locally
    for filepath in glob_iter
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:242: in <listcomp>
    os.path.relpath(filepath, base_path), os.path.relpath(pattern, base_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\popen-gw0\\data6\\dataset.parquet'
start = '/'

...

E                   ValueError: path is on mount 'C:', start on mount 'D:'
```

This PR makes sure that `base_path` is in the same drive as `pattern`.


Transfer CI to GitHub Actions

Thanks a lot @albertvillanova ! I hope we're finally done with flakiness on windows ^^

Also thanks for paying extra attention to billing and avoiding running unnecessary jobs. Though for certain aspects (see my comments), I think it's worth having the extra jobs to make our life easier

~@lhoestq I think you forgot to add your comments?~

I had missed it among all the other comments...

@lhoestq, I'm specially enthusiastic with the fail-fast policy: it was in my TODO list for a long time. I really think it will have a positive impact (I would love to know the spent time saving it will enable, besides the carbon footprint reduction). :wink:

So yes, as you said above, let's give it a try at least. If we encounter any inconvenience, we can easily disable it.

Question: I guess I have to disable CircleCI CI before merging this PR?

This PR transfers CI from CircleCI to GitHub Actions. The implementation in GitHub Actions tries to be as faithful as possible to the implementation in CircleCI and get the same output results (exceptions below).

**IMPORTANT NOTE**: The fast-fail policy (described below) is not finally implemented, so that:
- we can continue merging PRs with CI in red because of some random error returned by the Hub
- it is not annoying for maintainers to have to relaunch failed CI jobs

See comments here: https://github.com/huggingface/datasets/pull/4659#discussion_r918802348

Differences in the implementation in GitHub Actions compared to the CircleCI one:
- This PR introduces some *fail-fast* mechanisms to significantly reduce the total time CI is running, both because of environmental impact and because CI in GitHub Actions billing depends on the minutes per month running time (see [About billing for GitHub Actions](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions)):
  - All tests *depend* on `check_code_quality` job: only if `check_code_quality` passes, then the other test jobs are launched
  - The tests are implemented with a matrix strategy (cross-product: OS and PyArrow versions) and fail-fast: if any of the 4 processes fails, the others are cancelled
- OS dependencies for Linux (see table below)

   | OS dependencies | Passed tests | Skipped tests |
   | --- | ---: | ---: |
   | libsndfile1-dev | 4786 | 3119 |
   | libsndfile1 | 4786 | 3119 |
   | libsndfile1, sox | 4788 | 3117 |

  - This PR replaces `libsndfile1-dev` with `libsndfile1`: the same number of passing tests but less packages installed
  - This PR adds `sox`: required by MP3 tests (2 more tests are passed: 4788 instead of 4786)
- For tests using PyArrow 6, this PR uses 6.0.1 instead of 6.0.0

TO DO:
- [ ] Remove old CircleCI CI: kept for the moment to compare stability and performance

Close #4658.

## Comparison between CircleCI and GitHub Actions

| | | CircleCI | GitHub Actions |
| --- | --- | ---: | ---: |
| Ubuntu, pyarrow-latest ||||
|| Passed tests | 4786 | 4788 |
|| Duration | 11m 0s | 10m 10s |
| Windows, pyarrow-latest ||||
|| Passed tests | 4783 | 4783 |
|| Duration | 29m 59s | 22m 56s |


Transfer CI tests to GitHub Actions

Let's try CI tests using GitHub Actions to see if they are more stable than on CircleCI.


Add SQuAD2.0 Dataset

dataset request

Requesting to add a new dataset

Hey, It's already present [here](https://huggingface.co/datasets/squad_v2)

Hi! This dataset is indeed already available on the Hub. Closing.

## Adding a Dataset
- **Name:** *SQuAD2.0*
- **Description:** *Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.*
- **Paper:** *https://aclanthology.org/P18-2124.pdf*
- **Data:** *https://rajpurkar.github.io/SQuAD-explorer/*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Amazon-QA Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Amazon-QA*
- **Description:** *The dataset is .jsonl format, where each line in the file is a json string that corresponds to a question, existing answers to the question and the extracted review snippets (relevant to the question).*
- **Paper:** *https://github.com/amazonqa/amazonqa/tree/master/paper*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/amazon-qa.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Simple Wikipedia

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Simple Wikipedia*
- **Description:** *Two different versions of the data set now exist. Both were generated by aligning Simple English Wikipedia and English Wikipedia. A complete description of the extraction process can be found in "Simple English Wikipedia: A New Simplification Task", William Coster and David Kauchak (2011).*
- **Paper:** *https://aclanthology.org/P11-2117/*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/SimpleWiki.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Quora Question Triplets Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Quora Question Triplets*
- **Description:** *This dataset consists of over 400,000 lines of potential question duplicate pairs. Each line contains IDs for each question in the pair, the full text for each question, and a binary value that indicates whether the line truly contains a duplicate pair.*
- **Paper:**
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/quora_duplicates_triplets.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Altlex dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Altlex*
- **Description:** *Git repository for software associated with the 2016 ACL paper "Identifying Causal Relations Using Parallel Wikipedia Articles.”*
- **Paper:** *https://aclanthology.org/P16-1135.pdf*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/altlex.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Sentence Compression Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Sentence Compression*
- **Description:** *Large corpus of uncompressed and compressed sentences from news articles.*
- **Paper:** *https://www.aclweb.org/anthology/D13-1155/*
- **Data:** *https://github.com/google-research-datasets/sentence-compression/tree/master/data*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Flickr 30k Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Flickr 30k*
- **Description:** *To produce the denotation graph, we have created an image caption corpus consisting of 158,915 crowd-sourced captions describing 31,783 images. This is an extension of our previous Flickr 8k Dataset. The new images and captions focus on people involved in everyday activities and events.*
- **Paper:** *https://transacl.org/ojs/index.php/tacl/article/view/229/33*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/flickr30k_captions.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add SPECTER dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *SPECTER*
- **Description:** *SPECTER: Document-level Representation Learning using Citation-informed Transformers*
- **Paper:** *https://doi.org/10.18653/v1/2020.acl-main.207*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/specter_train_triples.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add PAQ dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *PAQ*
- **Description:** *This repository contains code and models to support the research paper PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them*
- **Paper:** *https://arxiv.org/abs/2102.07033*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/PAQ_pairs.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add WikiAnswers dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *WikiAnswers*
- **Description:** *The WikiAnswers corpus contains clusters of questions tagged by WikiAnswers users as paraphrases. Each cluster optionally contains an answer provided by WikiAnswers users.*
- **Paper:** *https://dl.acm.org/doi/10.1145/2623330.2623677*
- **Data:** *https://github.com/afader/oqa#wikianswers-corpus*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Reddit dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Reddit comments (2015-2018)*
- **Description:** *Reddit is an American social news aggregation website, where users can post links, and take part in discussions on these posts. These threaded discussions provide a large corpus, which is converted into a conversational dataset using the tools in this directory.*
- **Paper:** *https://arxiv.org/abs/1904.06472*
- **Data:** *https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Set HF_SCRIPTS_VERSION to main

After renaming "master" to "main", the CI fails with
```
AssertionError: 'https://raw.githubusercontent.com/huggingface/datasets/main/datasets/_dummy/_dummy.py' not found in "Couldn't find a dataset script at /home/circleci/datasets/_dummy/_dummy.py or any data file in the same directory. Couldn't find '_dummy' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/_dummy/_dummy.py"
```

This is because in the CI we were still using `HF_SCRIPTS_VERSION=master`. I changed it to "main"


[Minor fix] Typo correction

recieve -> receive


Rename master to main

All the mentions I found on google were simple URLs that will be redirected, so it's fine. I also checked the spaces and we should be good:
- dalle-mini used to install the master branch but [it's no longer the case](https://huggingface.co/spaces/flax-community/dalle-mini/commit/b78c972afd5c2d2bed087be6479fe5c9c6cfa741)
- same for [logo generator](https://huggingface.co/spaces/tom-doerr/logo_generator/commit/a9ea330e518870d0ca8f65abb56f71d86750d8e4)
- I opened a PR to fix [vision-datasets-viewer](https://huggingface.co/spaces/nateraw/vision-datasets-viewer/discussions/1)

Ok let's rename the branch, and then we can merge this PR

This PR renames mentions of "master" by "main" in the code base for several cases:
- set the default dataset script version to "main" if the local installation of `datasets` is a dev installation
- update URLs to this github repository to use "main"
- update the DVC benchmark
- update the github workflows
- update docstrings
- update tests to compare the changes in dataset cards against "main"


Streaming issue for ccdv/pubmed-summarization

Thanks for reporting @lewtun.

I confirm there is an issue with streaming: it does not stream locally.

Oh, after investigation, the source of the issue is in the Hub dataset loading script.

I'm opening a PR on the Hub dataset.

I've opened a PR on their Hub dataset to support streaming: https://huggingface.co/datasets/ccdv/pubmed-summarization/discussions/2

### Link

https://huggingface.co/datasets/ccdv/pubmed-summarization

### Description

This was reported by a [user of AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/7). It seems like streaming doesn't work due to the way the dataset loading script is defined?

```
Status code:   400
Exception:     FileNotFoundError
Message:       https://huggingface.co/datasets/ccdv/pubmed-summarization/resolve/main/train.zip/train.txt
```

### Owner

No


Dataset Viewer issue for kmfoda/booksum

Thanks for reporting, @lewtun.

It works locally in streaming mode:
```
{'bid': 27681,
 'is_aggregate': True,
 'source': 'cliffnotes',
 'chapter_path': 'all_chapterized_books/27681-chapters/chapters_1_to_2.txt',
 'summary_path': 'finished_summaries/cliffnotes/The Last of the Mohicans/section_1_part_0.txt',
 'book_id': 'The Last of the Mohicans.chapters 1-2',
 'summary_id': 'chapters 1-2',
 'content': None,
 'summary': '{"name": "Chapters 1-2", "url": "https://web.archive.org/web/20201101053205/https://www.cliffsnotes.com/literature/l/the-last-of-the-mohicans/summary-and-analysis/chapters-12", "summary": "Before any characters appear, the time and geography are made clear. Though it is the last war that England and France waged for a country that neither would retain, the wilderness between the forces still has to be...
```

I'm forcing the refresh of the preview.

The preview appears as expected once the refresh forced.

### Link

https://huggingface.co/datasets/kmfoda/booksum

### Description

A [user of AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/9) discovered this dataset cannot be streamed due to:

```
Status code:   400
Exception:     ClientResponseError
Message:       401, message='Unauthorized', url=URL('https://huggingface.co/datasets/kmfoda/booksum/resolve/47953f583d6967f086cb16a2f4d2346e9834024d/test.csv')
```

I'm not sure why it says "Unauthorized" since it's just a bunch of CSV files in a repo

### Owner

No


Support all split in streaming mode

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4640). All of your documentation changes will be reflected on that endpoint.

Fix #4637.


Add HaGRID -- HAnd Gesture Recognition Image Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** HaGRID -- HAnd Gesture Recognition Image Dataset
- **Description:** We introduce a large image dataset HaGRID (HAnd Gesture Recognition Image Dataset) for hand gesture recognition (HGR) systems. You can use it for image classification or image detection tasks. Proposed dataset allows to build HGR systems, which can be used in video conferencing services (Zoom, Skype, Discord, Jazz etc.), home automation systems, the automotive sector, etc.
- **Paper:** https://arxiv.org/abs/2206.08219
- **Data:** https://github.com/hukenovs/hagrid

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


The speechocean762 dataset

CircleCL reported two errors, but I didn't find the reason. The error message:
```
_________________ ERROR collecting tests/test_dataset_cards.py _________________
tests/test_dataset_cards.py:53: in <module>
    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
tests/test_dataset_cards.py:35: in get_changed_datasets
    diff_output = check_output(["git", "diff", "--name-only", "origin/master...HEAD"], cwd=repo_path)
../.pyenv/versions/3.6.15/lib/python3.6/subprocess.py:356: in check_output
    **kwargs).stdout
../.pyenv/versions/3.6.15/lib/python3.6/subprocess.py:438: in run
    output=stdout, stderr=stderr)
E   subprocess.CalledProcessError: Command '['git', 'diff', '--name-only', 'origin/master...HEAD']' returned non-zero exit status 128.

=========================== short test summary info ============================
ERROR tests/test_dataset_cards.py - subprocess.CalledProcessError: Command '[...
ERROR tests/test_dataset_cards.py - subprocess.CalledProcessError: Command '[...
= 4011 passed, 2357 skipped, 2 xfailed, 1 xpassed, 116 warnings, 2 errors in 284.32s (0:04:44) =

Exited with code exit status 1
```
I'm not sure if it was caused by this PR ...

I ran `tests/test_dataset_cards.py` in my local environment, and it passed:
```
(venv)$ pytest tests/test_dataset_cards.py
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/zhangjunbo/src/datasets
plugins: forked-1.4.0, datadir-1.3.1, xdist-2.5.0
collected 1531 items

tests/test_dataset_cards.py ..... [100%]
======================= 766 passed, 765 skipped in 2.55s ========================
```

@sanchit-gandhi could you also maybe take a quick look? :-)

[speechocean762](https://www.openslr.org/101/) is a non-native English corpus for pronunciation scoring tasks. It is free for both commercial and non-commercial use.

I believe it will be easier to use if it could be available on Hugging Face.


The "all" split breaks streaming

bug

Something isn't working

Thanks for reporting @cakiki.

Yes, this is a bug. We are investigating it.

@albertvillanova Nice! Let me know if it's something I can fix my self; would love to contribtue!

@cakiki I was working on this but if you would like to contribute, go ahead. I will close my PR. ;)

For the moment I just pushed the test (to see if it impacts other tests).

It impacted the test `test_generator_based_download_and_prepare` and I have fixed this.

So that you can copy the test I implemented in my PR and then implement a fix for this issue that passes the test `tests/test_builder.py::test_builder_as_streaming_dataset`.

Hi @cakiki are you still interested in working on this? Are you planning to open a PR?

Hi @albertvillanova ! Sorry it took so long; I wanted to spend this weekend working on it.

## Describe the bug

Not sure if this is a bug or just the way streaming works, but setting `streaming=True` did not work when setting `split="all"`

## Steps to reproduce the bug
The following works:
```python
ds = load_dataset('super_glue', 'wsc.fixed', split='all')
```
The following throws `ValueError: Bad split: all. Available splits: ['train', 'validation', 'test']`:

```python
ds = load_dataset('super_glue', 'wsc.fixed', split='all', streaming=True)
```

## Expected results
An iterator over all splits.

## Actual results
I had to do the following to achieve the desired result:
```python
from itertools import chain
ds = load_dataset('super_glue', 'wsc.fixed', streaming=True)
it =  chain.from_iterable(ds.values())
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.15.0-176-generic-x86_64-with-glibc2.31
- Python version: 3.10.5
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Add info in docs about behavior of download_config.num_proc

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

I went to override `download_config.num_proc` and was confused about what was happening under the hood. It would be nice to have the behavior documented a bit better so folks know what's happening when they use it.

**Describe the solution you'd like**

- Add note about how the default number of workers is 16. Related code:

https://github.com/huggingface/datasets/blob/7bcac0a6a0fc367cc068f184fa132b8de8dfa11d/src/datasets/download/download_manager.py#L299-L302

- Add note that if the number of workers is higher than the number of files to download, it won't use multiprocessing.

**Describe alternatives you've considered**

maybe it would also be nice to set `num_proc` = `num_files` when `num_proc` > `num_files`.

**Additional context**

...


Dataset Viewer issue for vadis/sv-ident

Thanks for reporting, @e-tornike

Some context:
- #4527

The dataset loads locally in streaming mode:
```python
In [2]: from datasets import load_dataset; ds = load_dataset("vadis/sv-ident", split="validation", streaming=True); item = next(iter(ds)); item
Using custom data configuration default
Out[2]:
{'sentence': 'Im Falle von Umweltbelastungen kann selten eindeutig entschieden werden, ob Unbedenklichkeitswerte bereits erreicht oder überschritten sind, die die menschliche Gesundheit oder andere Wohlfahrts»güter« beeinträchtigen.',
 'is_variable': 0,
 'variable': [],
 'research_data': [],
 'doc_id': '51971',
 'uuid': 'ee3d7f88-1a3e-4a59-997f-e986b544a604',
 'lang': 'de'}
```

~~I have forced the refresh of the split in the preview without success.~~

I have forced the refresh of the split in the preview, and now it works.

Preview seems to work now.

https://huggingface.co/datasets/vadis/sv-ident/viewer/default/validation

OK, thank you @e-tornike.

Apparently, after forcing the refresh, we just had to wait a little until it is effectively refreshed.

I'm closing this issue as it was solved after forcing the refresh of the split in the preview.

### Link

https://huggingface.co/datasets/vadis/sv-ident/viewer/default/validation

### Description

Error message when loading validation split in the viewer:

```
Status code:   400
Exception:     Status400Error
Message:       The split cache is empty.
```

### Owner

_No response_


Can't load the Hausa audio dataset

Could you provide the error details. It is difficult to debug otherwise. Also try other config. `ha` is not a valid.

common_voice_train = load_dataset("common_voice", "ha", split="train+validation")


[data_files] Only match separated split names

_The documentation is not available anymore as the PR was closed or merged._

I ran a script to find affected datasets (just did it on non-private non-gated). Adding "testing" and "evaluation" fixes all of of them except one:
- projecte-aina/cat_manynames:	human_annotated_testset.tsv

Let me open a PR on their repository to fix it
EDIT: pr [here](https://huggingface.co/datasets/projecte-aina/cat_manynames/discussions/2)

Feel free to merge @albertvillanova if it's all good to you :)

Thanks for the feedback @albertvillanova  I took your comments into account :)
- added numbers as supported delimiters
- used list comprehension to create the patterns list
- updated the docs and the tests according to your comments

Let me know what you think !

I ended up removing the patching and the context manager :) merging

As reported in https://github.com/huggingface/datasets/issues/4477, the current pattern matching to infer which file goes into which split is too permissive. For example a file "contest.py" would be considered part of a test split (it contains "test") and "seqeval.py" as well (it contains "eval").

In this PR I made the pattern matching more robust by only matching split names **between separators**. The supported separators are dots, dashes, spaces and underscores.

I updated the docs accordingly.

One detail about the tests: I had to update one test because it was using `PurePath.match` as a reference for globbing, but it doesn't support the `[..]` glob pattern. Therefore I added a `mock_fs` context manager that can be used to easily define a dummy filesystem with certain files in it and run pattern matching tests. Its code comes mostly from test_streaming_download_manager.py

Close https://github.com/huggingface/datasets/issues/4477


'sort' method sorts one column only

Hi ! `ds.sort()` does sort the full dataset, not just one column:
```python
from datasets import *

ds = Dataset.from_dict({"foo": [3, 2, 1], "bar": ["c", "b", "a"]})
print(d.sort("foo").to_pandas()
#    foo bar
# 0    1   a
# 1    2   b
# 2    3   c
```

What made you think it was not the case ? Did you experience a situation where it was only sorting one column ?

Hi! thank you for your quick reply!
I wanted to sort the `cnn_dailymail` dataset by the length of the labels (num of characters). I added a new column to the dataset (`ds.add_column`) with the lengths and then sorted by this new column. Only the new length column was sorted, the reset left in their original order.

That's unexpected, can you share the code you used to get this ?

The 'sort' method changes the order of one column only (the one defined by the argument 'column'), thus creating a mismatch between a sample fields. I would expect it to change the order of the samples as a whole, based on the 'column' order.


Update WinoBias README

I'm adding some information about Winobias that I got from the paper :smile:

I think this makes it a bit clearer!


fix(dataset_wrappers): Fixes access to fsspec.asyn in torch_iterable_dataset.py.

Fix #4612.

Apparently, newest `fsspec` versions do not allow access to attribute-based modules if they are not imported, such as `fsspec.async`.

Thus, @mariosasko suggested to add the missing part to the module import to allow for its access.


Rename repo default branch to main

maintenance

Maintenance tasks

Rename repository default branch to `main` (instead of current `master`).

Once renamed, users will have to manually update their local repos:

- [ ] Upstream:
  ```
  git branch -m master main
  git fetch upstream main
  git branch -u upstream/main main
  git remote set-head upstream -a
  ```

- [ ] Origin:
Rename fork default branch as well at: https://github.com/USERNAME/lam/settings/branches
Then:
  ```
  git fetch origin main
  git remote set-head origin -a
  ```

CC: @sgugger


Fix time type `_arrow_to_datasets_dtype` conversion

Fix #4620

The issue stems from the fact that `pa.array([time_data]).type` returns `DataType(time64[unit])`, which doesn't expose the `unit` attribute, instead of `Time64Type(time64[unit])`. I believe this is a bug in PyArrow. Luckily, the both types have the same `str()`, so in this PR I call `pa.type_for_alias(str(type))` to convert them both to the `Time64Type(time64[unit])` format.

cc @severo


fixed duplicate calculation of spearmanr function in metrics wrapper.

Great, can open a PR in `evaluate` as well to optimize this.

Relatedly, I wanted to add a new metric, Kendall Tau (https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html). If I were to open a PR with the wrapper, description, citation, docstrings, readme, etc. would it make more sense to do that in the `datasets` or `evaluate` repo (or both)?

Thanks!

PR opened in`evaluate` library with same minor adjustment: https://github.com/huggingface/evaluate/pull/176

> If I were to open a PR with the wrapper, description, citation, docstrings, readme, etc. would it make more sense to do that in the datasets or evaluate repo (or both)?

I think you could just add it to `evaluate`, we're not adding new metrics in this repo anymore

During _compute, the scipy.stats spearmanr function was called twice, redundantly, once for calculating the score and once for calculating the p-value, under the conditional branch where return_pvalue=True. I adjusted the _compute function to execute the spearmanr function once, store the results tuple in a temporary variable, and then pass the indexed contents to the expected keys of the returned dictionary.


Add non-commercial licensing info for datasets for which we removed tags

yep plus `license_details` also makes sense for this IMO

We removed several YAML tags saying that certain datasets can't be used for commercial purposes: https://github.com/huggingface/datasets/pull/4613#discussion_r911919753

Reason for this is that we only allow tags that are part of our [supported list of licenses](https://github.com/huggingface/datasets/blob/84fc3ad73c85de4eda5d152dfede7671491449cb/src/datasets/utils/resources/standard_licenses.tsv)

We should update the Licensing Information section of the concerned dataset cards, now that the non-commercial tag doesn't exist anymore for certain datasets


Unpack `dl_manager.iter_files` to allow parallization

Cool thanks ! Yup it sounds like the right solution.

It looks like `_generate_tables` needs to be updated as well to fix the CI

Iterate over data files outside `dl_manager.iter_files` to allow parallelization in streaming mode.

(The issue reported [here](https://discuss.huggingface.co/t/dataset-only-have-n-shard-1-when-has-multiple-shards-in-repo/19887))

PS: Another option would be to override `FilesIterable.__getitem__` to make it indexable and check for that type in `_shard_kwargs` and `n_shards,` but IMO this solution adds too much unnecessary complexity.


Remove all paperswithcode_id: null

> We've been using `null` to specify that we checked on pwc but the dataset doesn't exist there.

@lhoestq maybe it's better to accept it on the Hub side then? Let me know if you want us to do it Hub-side

Yup it's maybe better to support it on the Hub side then indeed, thanks ! Closing this one

On the Hub there is a validation error on the `paperswithcode_id` tag when the value is `null`:

<img width="686" alt="image" src="https://user-images.githubusercontent.com/42851186/177151825-93d341c5-25bd-41ab-96c2-c0b516d51c68.png">

We've been using `null` to specify that we checked on pwc but the dataset doesn't exist there.

To have the validation working again we can simply remove all the `paperswithcode_id: null`.

cc @julien-c


Loading MNIST as Pytorch Dataset

bug

Something isn't working

Hi ! We haven't implemented the conversion from images data to PyTorch tensors yet I think

cc @mariosasko

So I understand:

set_format() does not properly do the conversion to pytorch tensors from PIL images.

So that someone who stumbles on this can use the package:

```python
dataset = load_dataset("mnist", split="train")
def transform_func(examples):
    examples["image"] = [np.array(img) for img in examples["image"]]
    return examples
dataset = dataset.with_transform(transform_func)
dataset[0]
```

This then appears to work with pytorch dataloaders as:
```
dataloader=torch.utils.data.DataLoader(dataset,batch_size=1)
```

and tensorflow as:
```
dataset=dataset.to_tf_dataset(batch_size=1)
```

Hi! `set_transform`/`with_transform` is indeed the correct solution for the conversion. Improving this part of the API is one of the things I'm working on currently, so stay tuned!

## Describe the bug
Conversion of MNIST dataset to pytorch fails with bug

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("mnist", split="train")
dataset.set_format('torch')
dataset[0]
print()
```

## Expected results
Expect to see torch tensors image and label

## Actual results
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2020.3.3\plugins\python\helpers\pydev\pydevd.py", line 1491, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2020.3.3\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/chapm/PycharmProjects/multiviewdata/multiviewdata/huggingface/mnist.py", line 13, in <module>
    dataset[0]
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\arrow_dataset.py", line 2154, in __getitem__
    return self._getitem(
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\arrow_dataset.py", line 2139, in _getitem
    formatted_output = format_table(
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\formatting.py", line 532, in format_table
    return formatter(pa_table, query_type=query_type)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\formatting.py", line 281, in __call__
    return self.format_row(pa_table)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 58, in format_row
    return self.recursive_tensorize(row)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 54, in recursive_tensorize
    return map_nested(self._recursive_tensorize, data_struct, map_list=False)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 356, in map_nested
    mapped = [
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 357, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 309, in _single_map_nested
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 309, in <dictcomp>
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 293, in _single_map_nested
    return function(data_struct)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 51, in _recursive_tensorize
    return self._tensorize(data_struct)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 38, in _tensorize
    if np.issubdtype(value.dtype, np.integer):
AttributeError: 'bytes' object has no attribute 'dtype'
python-BaseException

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Windows-10-10.0.22579-SP0
- Python version: 3.9.2
- PyArrow version: 8.0.0
- Pandas version: 1.4.1


Fix ImageFolder with parameters drop_metadata=True and drop_labels=False (when metadata.jsonl is present)

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq @mariosasko pls take a look at https://github.com/huggingface/datasets/pull/4622/commits/769e4c046a5bd5e3a4dbd09cfad1f4cf60677869. I modified `_generate_examples()` according to the same logic too: removed checking if `metadata_files` are not empty for the case when `self.config.drop_metadata=True` because I think we should be aligned with the config and preserve labels if `self.config.drop_labels=False` (the default value) and `self.config.drop_metadata=True` but `metadata_files` are passed. This is an extremely unlikely use case (when `self.config.drop_metadata=True`, but `metadata_files` are passed to `_generate_examples()`) since users usually do not use `_generate_examples()` alone but I believe it would be consistent to have the same behavior as in `_splits_generators()`. This change requires change in tests too if we suppose that we want to preserve labels (default value of `self.config.drop_labels` is False) when `self.config.drop_metadata=True`, even if `metadata_files` are for some reason provided (as it is done in tests).

wdyt about this change?

@lhoestq it wouldn't raise an error if we check `example.keys() == {"image", "label"}` as test checks only `_generate_examples`, not `encode_example`. and in the implementation of this PR `_generate_examples` would return both `image` and `label` key in the case when `drop_metadata=True` and `drop_labels=False` (default) as it seems that we agreed on that :)

and on the other hand it would raise an error if `label` column is missing in _generate_examples when `drop_metadata=True` and `drop_labels=False`

by "it" i mean tests :D (`test_generate_examples_with_metadata_that_misses_one_image`, `test_generate_examples_with_metadata_in_wrong_location` and `test_generate_examples_drop_metadata`)

Perhaps we could make `self.config.drop_metadata = None` and `self.config.drop_labels = None` the defaults to see explicitly what the user wants. This would then turn into  `self.config.drop_metadata = False` and `self.config.drop_labels = True` if metadata files are present and `self.config.drop_metadata = True` and `self.config.drop_labels = False` if not. And if the user wants to have the `label` column alongside metadata columns, it can do so by passing `drop_labels = False` explicitely (in that scenario we have to check that the `label` column is not already present in metadata files). And maybe we can also improve the logging messages.

I find it problematic that the current implementation drops labels in some scenarios even if `self.config.drop_labels = False`, and the user doesn't have control over this behavior.

Let me know what you think.

Will fix #4621

ImageFolder raises `KeyError: 'label'` with params `drop_metadata=True` and `drop_labels=False` (if there is at least one metadata.jsonl file a data directory). This happens because metadata files are collected inside `analyze()` function regardless of `drop_metadata` value. And then the following condition doesn't pass: https://github.com/huggingface/datasets/blob/master/src/datasets/packaged_modules/imagefolder/imagefolder.py#L167

So I suggest to double check it inside `analyze()` not to collect metadata files if they are not needed. (and labels too, to be consistent)

---
Also, I added a test to check if labels are inferred correctly from directories names in general (because we didn't have it) :)


ImageFolder raises an error with parameters drop_metadata=True and drop_labels=False when metadata.jsonl is present

Something isn't working

## Describe the bug

If you pass `drop_metadata=True` and `drop_labels=False` when a `data_dir` contains at least one `matadata.jsonl` file, you will get a KeyError. This is probably not a very useful case but we shouldn't get an error anyway. Asking users to move metadata files manually outside `data_dir` or pass features manually (when there is a tool that can infer them automatically) don't look like a good idea to me either.

## Steps to reproduce the bug
### Clone an example dataset from the Hub
```bash
git clone https://huggingface.co/datasets/nateraw/test-imagefolder-metadata
```
### Try to load it
```python
from datasets import load_dataset
ds = load_dataset("test-imagefolder-metadata", drop_metadata=True, drop_labels=False)
```
or even just
```python
ds = load_dataset("test-imagefolder-metadata", drop_metadata=True)
```
as `drop_labels=False` is a default value.

## Expected results
A DatasetDict object with two features: `"image"` and `"label"`.

## Actual results
```
Traceback (most recent call last):
  File "/home/polina/workspace/datasets/debug.py", line 18, in <module>
    ds = load_dataset(
  File "/home/polina/workspace/datasets/src/datasets/load.py", line 1732, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 1227, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 1218, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1596, in encode_example
    return encode_nested_example(self, example)
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1165, in encode_nested_example
    {
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1165, in <dictcomp>
    {
  File "/home/polina/workspace/datasets/src/datasets/utils/py_utils.py", line 249, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/home/polina/workspace/datasets/src/datasets/utils/py_utils.py", line 249, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
KeyError: 'label'
```

## Environment info
`datasets` master branch

- `datasets` version: 2.3.3.dev0
- Platform: Linux-5.14.0-1042-oem-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.1
- Pandas version: 1.4.1


Data type is not recognized when using datetime.time

Something isn't working

Hi, thanks for reporting! I'm investigating the issue.

## Describe the bug

Creating a dataset from a pandas dataframe with `datetime.time` format generates an error.

## Steps to reproduce the bug
```python
import pandas as pd
from datetime import time
from datasets import Dataset
df = pd.DataFrame({"feature_name": [time(1, 1, 1)]})
dataset = Dataset.from_pandas(df)
```

## Expected results

The dataset should be created.

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 823, in from_pandas
    return cls(table, info=info, split=split)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 679, in __init__
    inferred_features = Features.from_arrow_schema(arrow_table.schema)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1551, in from_arrow_schema
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1551, in <dictcomp>
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1315, in generate_from_arrow_type
    return Value(dtype=_arrow_to_datasets_dtype(pa_type))
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 83, in _arrow_to_datasets_dtype
    return f"time64[{arrow_type.unit}]"
AttributeError: 'pyarrow.lib.DataType' object has no attribute 'unit'
```

## Environment info

- `datasets` version: 2.3.3.dev0
- Platform: Linux-5.13.0-1031-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.4.2


np arrays get turned into native lists

Something isn't working

If you add the line `dataset2.set_format('np')` before calling `dataset2[0]['tmp']` it should return `np.ndarray`.
I believe internally it will not store it as a list, it is only returning a list when you index it.

```
In [1]: import datasets, numpy as np
In [2]: dataset = datasets.load_dataset("glue", "mrpc")["validation"]
In [3]: dataset2 = dataset.map(lambda x: {"tmp": np.array([0.5])}, batched=False)
In [4]: dataset2[0]["tmp"]
Out[4]: [0.5]

In [5]: dataset2.set_format('np')

In [6]: dataset2[0]["tmp"]
Out[6]: array([0.5])
```

I see, thanks! Any idea if the default numpy → list conversion might cause precision loss?

I'm not super familiar with our datasets works internally, but I think your `np` array will be stored in a `pyarrow` format, and then you take a view of this as a python array. In which case, I think the precision should be preserved.

## Describe the bug
When attaching an `np.array` field, it seems that it automatically gets turned into a list (see below). Why is this happening? Could it lose precision? Is there a way to make sure this doesn't happen?

## Steps to reproduce the bug
```python
>>> import datasets, numpy as np
>>> dataset = datasets.load_dataset("glue", "mrpc")["validation"]
Reusing dataset glue (...)
100%|███████████████████████████████████████████████| 3/3 [00:00<00:00, 1360.61it/s]
>>> dataset2 = dataset.map(lambda x: {"tmp": np.array([0.5])}, batched=False)
100%|██████████████████████████████████████████| 408/408 [00:00<00:00, 10819.97ex/s]
>>> dataset2[0]["tmp"]
[0.5]
>>> type(dataset2[0]["tmp"])
<class 'list'>
```

## Expected results
`dataset2[0]["tmp"]` should be an `np.ndarray`.

## Actual results
It's a list.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: mac, though I'm pretty sure it happens on a linux machine too
- Python version: 3.9.7
- PyArrow version: 6.0.1


contribute data loading for object detection datasets with yolo data format

enhancement

New feature or request

Hi! The `imagefolder` script is already quite complex, so a standalone script sounds better. Also, I suggest we create an org on the Hub (e.g. `hf-loaders`) and store such scripts there for easier maintenance rather than having them as packaged modules (IMO only very generic loaders should be packaged). WDYT @lhoestq @albertvillanova @polinaeterna?

Thank you for the suggestion @mariosasko . I agree with the point, but I have a few doubts

1. How would the user access the script if it's not a part of the core codebase?
2. Could you direct me as to what will be the tasks I have to do to contribute to the code? As per my understanding, it would be like
    1. Create a new org "hf-loaders" and add you (and more HF people) to the org
    2. Add data loader script as a (model?)
    3. Test it with a dataset on HF hub
3. We should maybe brainstorm as to which public datasets have this format (YOLO type) and are the most important ones to test the script with. We can even add the datasets on HF Hub alongside the script

1. Like this: `load_dataset("hf-loaders/yolo", data_files=...)`
2. The steps would be:
    1. Create a new org `hf-community-loaders` (IMO a better name than "hf-loaders") and add me (as an admin)
    2. Create a new dataset repo `yolo` and add the loading script to it (`yolo.py`)
    3. Open a discussion to request our review
4. I like this idea. Another option is to add snippets that describe how to load such datasets using the `yolo` loader.

**Is your feature request related to a problem? Please describe.**
At the moment, HF datasets loads [image classification datasets](https://huggingface.co/docs/datasets/image_process) out-of-the-box. There could be a data loader for loading standard object detection datasets ([original discussion here](https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/discussions/2))

**Describe the solution you'd like**
I wrote a [custom script](https://huggingface.co/datasets/jalFaizy/detect_chess_pieces/blob/main/detect_chess_pieces.py) to load dataset which has YOLO data format.

**Describe alternatives you've considered**
The script can either be a standalone dataset builder, or a modified version of `ImageFolder`

**Additional context**
I would be happy to contribute to this, but I would do it at a very slow pace (maybe a month or two) as I have my exams approaching 😄


Fix `embed_storage` on features inside lists/sequences

Add a dedicated function for embed_storage to always preserve the embedded/casted arrays (and to have more control over `embed_storage` in general).

Fix #4591

~~(Waiting for #4608 to be merged to mark this PR as ready for review - required for fixing `xgetsize` in private repos)~~ Done!


Ensure ConcatenationTable.cast uses target_schema metadata

Hi @lhoestq, Thanks for the detailed comment. I've tested the suggested approach and can confirm it works for the testcase outlined above! The PR is updated with the changes.

Currently, `ConcatenationTable.cast` does not use target_schema metadata when casting subtables. This causes an issue when using cast_column and the underlying table is a ConcatenationTable.

Code example of where issue arrises:
```
from datasets import Dataset, Image

column1 = [0, 1]
image_paths = ['/images/image1.jpg', '/images/image2.jpg']

ds = Dataset.from_dict({"column1": column1})
ds = ds.add_column("image", image_paths)
ds.cast_column("image", Image()) # Fails here
```
Output
```
...
TypeError: Couldn't cast array of type
string
to
{'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='string', id=None)}
```


Align/fix license metadata info

Thank you thank you! Let's merge and pray? 😱

I just need to add `license_details` to the validator and yup we can merge

fix bad "other-*" licenses and add the corresponding "license_details" when relevant


Release 2.3.0 broke custom iterable datasets

bug

Something isn't working

Apparently, `fsspec` does not allow access to attribute-based modules anymore, such as `fsspec.async`.

However, this is a fairly simple fix:
- Change the import to: `from fsspec import asyn`;
- Change line 18 to: `asyn.iothread[0] = None`;
- Change line 19 to `asyn.loop[0] = None`.

Hi! I think it's easier to replace `import fsspec` with `import fsspec.asyn` and leave the rest unchanged. @gugarosa Are you interested in submitting a PR?

Perfect, it is even better!

Just submitted the PR: #4630.

Thank you!

## Describe the bug
Trying to iterate examples from custom iterable dataset fails to bug introduced in `torch_iterable_dataset.py` since the release of 2.3.0.

## Steps to reproduce the bug
```python
next(iter(custom_iterable_dataset))
```

## Expected results
`next(iter(custom_iterable_dataset))` should return examples from the dataset

## Actual results
```
/usr/local/lib/python3.7/dist-packages/datasets/formatting/dataset_wrappers/torch_iterable_dataset.py in _set_fsspec_for_multiprocess()
     16     See https://github.com/fsspec/gcsfs/issues/379
     17     """
---> 18     fsspec.asyn.iothread[0] = None
     19     fsspec.asyn.loop[0] = None
     20

AttributeError: module 'fsspec' has no attribute 'asyn'
```

## Environment info
- `datasets` version: 2.3.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 8.0.0
- Pandas version: 1.3.5


Preserve member order by MockDownloadManager.iter_archive

_The documentation is not available anymore as the PR was closed or merged._

Currently, `MockDownloadManager.iter_archive` yields paths to archive members in an order given by `path.rglob("*")`, which migh not be the same order as in the original archive.

See issue in:
- https://github.com/huggingface/datasets/pull/4579#issuecomment-1172135027

This PR fixes the order of the members yielded by `MockDownloadManager.iter_archive` so that it is the same as in the original archive.


codeparrot/github-code failing to load

Something isn't working

I believe the issue is in `codeparrot/github-code`. `base_path` param is missing - https://huggingface.co/datasets/codeparrot/github-code/blob/main/github-code.py#L169

Function definition has changed.
https://github.com/huggingface/datasets/blob/0e1c629cfb9f9ba124537ba294a0ec451584da5f/src/datasets/data_files.py#L547

@mariosasko could you please confirm my finding? And are there any changes that need to be done from my side?

> Good catch ! We recently did a breaking change in `get_patterns_in_dataset_repository`, I think we can revert it

I can't wait for that releasee. Broke my application

This simple workaround should fix: https://huggingface.co/datasets/codeparrot/github-code/discussions/2

`get_patterns_in_dataset_repository` can treat whether `base_path=None`, so we just need to make sure that codeparrot/github-code `_split_generators` calls with such an argument.

I am afraid your suggested change @gugarosa will break compatibility with older datasets versions that don't have `base_path` argument in `get_patterns_in_dataset_repository`, as a workaround while the issue gets resolved in `datasets` can you downgrade your datasets version  to `<=2.1.0` ?
@lvwerra do you think we should adapt the script to check the datasets version before calling `get_patterns_in_dataset_repository`?

Actually I think it's just simpler to fix it in the dataset itself, let me open a PR

EDIT: PR opened here: https://huggingface.co/datasets/codeparrot/github-code/discussions/3

PR is merged, it's working now ! Closing this one :)

> I am afraid your suggested change @gugarosa will break compatibility with older datasets versions that don't have `base_path` argument in `get_patterns_in_dataset_repository`, as a workaround while the issue gets resolved in `datasets` can you downgrade your datasets version to `<=2.1.0` ?
> @lvwerra do you think we should adapt the script to check the datasets version before calling `get_patterns_in_dataset_repository`?

You are definitely right, sorry about it. I always keep forgetting that we need to keep in mind users from past versions, my bad.

## Describe the bug
codeparrot/github-code fails to load with a `TypeError: get_patterns_in_dataset_repository() missing 1 required positional argument: 'base_path'`

## Steps to reproduce the bug
```python
from datasets import load_dataset
```

## Expected results
loaded dataset object

## Actual results
```python
 [3]: dataset = load_dataset("codeparrot/github-code")
No config specified, defaulting to: github-code/all-all
Downloading and preparing dataset github-code/all-all to /home/bebr/.cache/huggingface/datasets/codeparrot___github-code/all-all/0.0.0/a55513bc0f81db773f9896c7aac225af0cff5b323bb9d2f68124f0a8cc3fb817...
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 dataset = load_dataset("codeparrot/github-code")

File ~/miniconda3/envs/fastapi-kube/lib/python3.10/site-packages/datasets/load.py:1679, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1676 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1678 # Download and prepare data
-> 1679 builder_instance.download_and_prepare(
   1680     download_config=download_config,
   1681     download_mode=download_mode,
   1682     ignore_verifications=ignore_verifications,
   1683     try_from_hf_gcs=try_from_hf_gcs,
   1684     use_auth_token=use_auth_token,
   1685 )
   1687 # Build dataset for splits
   1688 keep_in_memory = (
   1689     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1690 )

File ~/miniconda3/envs/fastapi-kube/lib/python3.10/site-packages/datasets/builder.py:704, in DatasetBuilder.download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702         logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703 if not downloaded_from_gcs:
--> 704     self._download_and_prepare(
    705         dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706     )
    707 # Sync info
    708 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/miniconda3/envs/fastapi-kube/lib/python3.10/site-packages/datasets/builder.py:1221, in GeneratorBasedBuilder._download_and_prepare(self, dl_manager, verify_infos)
   1220 def _download_and_prepare(self, dl_manager, verify_infos):
-> 1221     super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)

File ~/miniconda3/envs/fastapi-kube/lib/python3.10/site-packages/datasets/builder.py:771, in DatasetBuilder._download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    769 split_dict = SplitDict(dataset_name=self.name)
    770 split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 771 split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    773 # Checksums verification
    774 if verify_infos and dl_manager.record_checksums:

File ~/.cache/huggingface/modules/datasets_modules/datasets/codeparrot--github-code/a55513bc0f81db773f9896c7aac225af0cff5b323bb9d2f68124f0a8cc3fb817/github-code.py:169, in GithubCode._split_generators(self, dl_manager)
    162 def _split_generators(self, dl_manager):
    164     hfh_dataset_info = HfApi(datasets.config.HF_ENDPOINT).dataset_info(
    165         _REPO_NAME,
    166         timeout=100.0,
    167     )
--> 169     patterns = datasets.data_files.get_patterns_in_dataset_repository(hfh_dataset_info)
    170     data_files = datasets.data_files.DataFilesDict.from_hf_repo(
    171         patterns,
    172         dataset_info=hfh_dataset_info,
    173     )
    175     files = dl_manager.download_and_extract(data_files["train"])

TypeError: get_patterns_in_dataset_repository() missing 1 required positional argument: 'base_path'
```

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.18.7-arch1-1-x86_64-with-glibc2.35
- Python version: 3.10.5
- PyArrow version: 8.0.0
- Pandas version: 1.4.2


Something isn't working

Hi! You can use streaming to fetch only a subset of the data:
```python
raw_dataset = load_dataset("librispeech_asr", "clean", split="train.100", streaming=True)
```
Also, we plan to make it possible to download a particular split in the non-streaming mode, but this task is not easy due to how our dataset scripts are structured.

Hi,

That's a great help. Thank you very much.

## Describe the bug
librispeech dataset has to download whole subset when specifing the split to use

## Steps to reproduce the bug
see below
# Sample code to reproduce the bug
```
!pip install datasets
from datasets import load_dataset
raw_dataset = load_dataset("librispeech_asr", "clean", split="train.100")
```

## Expected results
The split "train.clean.100" is downloaded.

## Actual results
All four splits in "clean" subset is downloaded.

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Fix xisfile, xgetsize, xisdir, xlistdir in private repo

Added tests for xisfile, xgetsize, xlistdir and xglob for private repos, and also tests for xwalk that was untested

`xisfile` is working in a private repository when passing a chained URL to a file inside an archive, e.g. `zip://a.txt::https://huggingface/datasets/username/dataset_name/resolve/main/data.zip`. However it's not working when passing a simple file `https://huggingface/datasets/username/dataset_name/resolve/main/data.zip`.

This is because the authentication headers are not passed correctly in this case.

This is causing dataset streaming to fail in private parquet repositories, as noted in https://github.com/huggingface/datasets/issues/4605

I fixed `xisfile` and the other functions that behave the same way: xgetsize, xisdir and xlistdir

TODO:
- [x] tests

fix https://github.com/huggingface/datasets/issues/4605


Align more metadata with other repo types (models,spaces)

I just set a default value (None) for the deprecated licenses and languages fields, which should fix most of the CI failures.

Note that the CI should still be red because you edited many dataset cards and they're still missing some content - but this is unrelated to this PR so we can ignore these failures

There's also a follow-up PR to this one, in #4613 – I would suggest to merge all of them at the same time and hope not too many things are broken 🙀 🙀

Alright merging this one now, let's see how broken things get

see also associated PR on the `datasets-tagging` Space: https://huggingface.co/spaces/huggingface/datasets-tagging/discussions/2 (to merge after this one is merged)


Something isn't working

Hi! The GH/no-namespace datasets versioning is synced with the version of the `datasets` lib, which means that the `wikiann` script was modified between the two compared versions. In this scenario, you can ensure reproducibility by pinning the script version, which is done by passing `revision="x.y.z"` (e.g. `revision="2.2.0"`) to `load_dataset.`

## Describe the bug
evaluation result changes after `datasets` version change

## Steps to reproduce the bug
1. Train a model on WikiAnn
2. reload the ckpt -> test accuracy becomes same as eval accuracy
3. such behavior is gone after downgrading `datasets`

https://colab.research.google.com/drive/1kYz7-aZRGdayaq-gDTt30tyEgsKlpYOw?usp=sharing

## Expected results
evaluation result shouldn't change before/after `datasets` version changes

## Actual results
evaluation result changes before/after `datasets` version changes

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: colab
- Python version: 3.7.13
- PyArrow version: 6.0.1

Q. How could the evaluation result change before/after `datasets` version changes?


Dataset Viewer issue for boris/gis_filtered

Yes, this dataset is "gated": you first have to go to https://huggingface.co/datasets/boris/gis_filtered and click "Access repository" (if you accept to share your contact information with the repository authors).

I already did that, it returns error when using streaming

Oh, sorry, I misread. Looking at it. Maybe @huggingface/datasets or @SBrandeis

I could reproduce the error, even though I provided my token and accepted the gate form. It looks like an error from `datasets`

This is indeed a bug in `datasets`. Parquet datasets in gated/private repositories can't be streamed properly, which caused the viewer to fail. I opened a PR at https://github.com/huggingface/datasets/pull/4608

### Link

https://huggingface.co/datasets/boris/gis_filtered/viewer/boris--gis_filtered/train

### Description

When I try to access this from the website I get this error:

Status code:   400
Exception:     ClientResponseError
Message:       401, message='Unauthorized', url=URL('https://huggingface.co/datasets/boris/gis_filtered/resolve/80b805053ce61d4eb487b6b8d9095d775c2c466e/data/train/0000.parquet')

If I try to load with code I also get the same issue:
```python
dataset2_train=load_dataset("boris/gis_filtered", use_auth_token=os.environ["HF_TOKEN"],split="train",streaming=True)
dataset2_validation=load_dataset("boris/gis_filtered", use_auth_token=os.environ["HF_TOKEN"], split="validation",streaming=True)
```

### Owner

No


Update CI Windows orb

_The documentation is not available anymore as the PR was closed or merged._

This PR tries to fix recurrent random CI failures on Windows.

After 2 runs, it seems to have fixed the issue.

Fix #4603.


CI fails recurrently and randomly on Windows

Something isn't working

As reported by @lhoestq,

The windows CI is currently flaky: some dependencies like `aiobotocore`, `multiprocess` and `seqeval` sometimes fail to install.
In particular it seems that building the wheels fail. Here is an example of logs:

```
Building wheel for seqeval (setup.py): started
  Running command 'C:\tools\miniconda3\envs\py37\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"'; __file__='"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6'
  No parent package detected, impossible to derive `name`
  running bdist_wheel
  running build
  running build_py
  package init file 'seqeval\__init__.py' not found (or not a regular file)
  package init file 'seqeval\metrics\__init__.py' not found (or not a regular file)
  C:\tools\miniconda3\envs\py37\lib\site-packages\setuptools\command\install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    setuptools.SetuptoolsDeprecationWarning,
  installing to build\bdist.win-amd64\wheel
  running install
  running install_lib
  warning: install_lib: 'build\lib' does not exist -- no Python modules to install

  running install_egg_info
  running egg_info
  creating UNKNOWN.egg-info
  writing UNKNOWN.egg-info\PKG-INFO
  writing dependency_links to UNKNOWN.egg-info\dependency_links.txt
  writing top-level names to UNKNOWN.egg-info\top_level.txt
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  reading manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  Copying UNKNOWN.egg-info to build\bdist.win-amd64\wheel\.\UNKNOWN-0.0.0-py3.7.egg-info
  running install_scripts
  creating build\bdist.win-amd64\wheel\UNKNOWN-0.0.0.dist-info\WHEEL
  creating 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6\UNKNOWN-0.0.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
  adding 'UNKNOWN-0.0.0.dist-info/METADATA'
  adding 'UNKNOWN-0.0.0.dist-info/WHEEL'
  adding 'UNKNOWN-0.0.0.dist-info/top_level.txt'
  adding 'UNKNOWN-0.0.0.dist-info/RECORD'
  removing build\bdist.win-amd64\wheel
  Building wheel for seqeval (setup.py): finished with status 'done'
  Created wheel for seqeval: filename=UNKNOWN-0.0.0-py3-none-any.whl size=963 sha256=67eb93a6e1ff4796c5882a13f9fa25bb0d3d103796e2525f9cecf3b2ef26d4b1
  Stored in directory: c:\users\circleci\appdata\local\pip\cache\wheels\05\96\ee\7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7
  WARNING: Built wheel for seqeval is invalid: Wheel has unexpected file name: expected 'seqeval', got 'UNKNOWN'
```


Upgrade setuptools in windows CI

The windows CI is currently flaky: some dependencies like aiobotocore, multiprocess and seqeval sometimes fail to install.
In particular it seems that building the wheels fail. Here is an example of logs

```
Building wheel for seqeval (setup.py): started
  Running command 'C:\tools\miniconda3\envs\py37\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"'; __file__='"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6'
  No parent package detected, impossible to derive `name`
  running bdist_wheel
  running build
  running build_py
  package init file 'seqeval\__init__.py' not found (or not a regular file)
  package init file 'seqeval\metrics\__init__.py' not found (or not a regular file)
  C:\tools\miniconda3\envs\py37\lib\site-packages\setuptools\command\install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    setuptools.SetuptoolsDeprecationWarning,
  installing to build\bdist.win-amd64\wheel
  running install
  running install_lib
  warning: install_lib: 'build\lib' does not exist -- no Python modules to install

  running install_egg_info
  running egg_info
  creating UNKNOWN.egg-info
  writing UNKNOWN.egg-info\PKG-INFO
  writing dependency_links to UNKNOWN.egg-info\dependency_links.txt
  writing top-level names to UNKNOWN.egg-info\top_level.txt
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  reading manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  Copying UNKNOWN.egg-info to build\bdist.win-amd64\wheel\.\UNKNOWN-0.0.0-py3.7.egg-info
  running install_scripts
  creating build\bdist.win-amd64\wheel\UNKNOWN-0.0.0.dist-info\WHEEL
  creating 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6\UNKNOWN-0.0.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
  adding 'UNKNOWN-0.0.0.dist-info/METADATA'
  adding 'UNKNOWN-0.0.0.dist-info/WHEEL'
  adding 'UNKNOWN-0.0.0.dist-info/top_level.txt'
  adding 'UNKNOWN-0.0.0.dist-info/RECORD'
  removing build\bdist.win-amd64\wheel
  Building wheel for seqeval (setup.py): finished with status 'done'
  Created wheel for seqeval: filename=UNKNOWN-0.0.0-py3-none-any.whl size=963 sha256=67eb93a6e1ff4796c5882a13f9fa25bb0d3d103796e2525f9cecf3b2ef26d4b1
  Stored in directory: c:\users\circleci\appdata\local\pip\cache\wheels\05\96\ee\7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7
  WARNING: Built wheel for seqeval is invalid: Wheel has unexpected file name: expected 'seqeval', got 'UNKNOWN'
```

hopefully this fixes the issue


Upgrade pip in WIN CI

The windows CI is currently flaky: some dependencies like aiobotocore, multiprocess and seqeval sometimes fail to install.
In particular it seems that building the wheels fail. Here is an example of logs

```
Building wheel for seqeval (setup.py): started
  Running command 'C:\tools\miniconda3\envs\py37\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"'; __file__='"'"'C:\\Users\\circleci\\AppData\\Local\\Temp\\pip-install-h55pfgbv\\seqeval_d6cdb9d23ff6490b98b6c4bcaecb516e\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6'
  No parent package detected, impossible to derive `name`
  running bdist_wheel
  running build
  running build_py
  package init file 'seqeval\__init__.py' not found (or not a regular file)
  package init file 'seqeval\metrics\__init__.py' not found (or not a regular file)
  C:\tools\miniconda3\envs\py37\lib\site-packages\setuptools\command\install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    setuptools.SetuptoolsDeprecationWarning,
  installing to build\bdist.win-amd64\wheel
  running install
  running install_lib
  warning: install_lib: 'build\lib' does not exist -- no Python modules to install

  running install_egg_info
  running egg_info
  creating UNKNOWN.egg-info
  writing UNKNOWN.egg-info\PKG-INFO
  writing dependency_links to UNKNOWN.egg-info\dependency_links.txt
  writing top-level names to UNKNOWN.egg-info\top_level.txt
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  reading manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  writing manifest file 'UNKNOWN.egg-info\SOURCES.txt'
  Copying UNKNOWN.egg-info to build\bdist.win-amd64\wheel\.\UNKNOWN-0.0.0-py3.7.egg-info
  running install_scripts
  creating build\bdist.win-amd64\wheel\UNKNOWN-0.0.0.dist-info\WHEEL
  creating 'C:\Users\circleci\AppData\Local\Temp\pip-wheel-x3cc8ym6\UNKNOWN-0.0.0-py3-none-any.whl' and adding 'build\bdist.win-amd64\wheel' to it
  adding 'UNKNOWN-0.0.0.dist-info/METADATA'
  adding 'UNKNOWN-0.0.0.dist-info/WHEEL'
  adding 'UNKNOWN-0.0.0.dist-info/top_level.txt'
  adding 'UNKNOWN-0.0.0.dist-info/RECORD'
  removing build\bdist.win-amd64\wheel
  Building wheel for seqeval (setup.py): finished with status 'done'
  Created wheel for seqeval: filename=UNKNOWN-0.0.0-py3-none-any.whl size=963 sha256=67eb93a6e1ff4796c5882a13f9fa25bb0d3d103796e2525f9cecf3b2ef26d4b1
  Stored in directory: c:\users\circleci\appdata\local\pip\cache\wheels\05\96\ee\7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7
  WARNING: Built wheel for seqeval is invalid: Wheel has unexpected file name: expected 'seqeval', got 'UNKNOWN'
```

I tried to update pip and re-run the CI several times and I couldn't re-experience this issue for now, so I think upgrading pip may solve the issue


Remove multiple config section

documentation

Improvements or additions to documentation

This PR removes docs for a future feature and redirects to #4578 instead. See this [discussion](https://huggingface.slack.com/archives/C034N0A7H09/p1656107063801969) for more details :)