Unable to create dataset having Python dataset script only

Something isn't working

Hi @aleSuglia, thanks for reporting.

We are having a look at it.

We transfer this issue to the Community tab of the corresponding Hub dataset: https://huggingface.co/datasets/Heriot-WattUniversity/dialog-babi/discussions

## Describe the bug
Hi there,

I'm trying to add the following dataset to Huggingface datasets: https://huggingface.co/datasets/Heriot-WattUniversity/dialog-babi/blob/

I'm trying to do so using the CLI commands but seems that this command generates the wrong `dataset_info.json` file (you can find it in the repo already):
```
datasets-cli test Heriot-WattUniversity/dialog-babi/dialog_babi.py --save_infos --all-configs
```
while it errors when I remove the python script:
```
datasets-cli test Heriot-WattUniversity/dialog-babi/ --save_infos --all-configs
```
The error message is the following:
```
FileNotFoundError: Unable to resolve any data file that matches '['**']' at /Users/as2180/workspace/Heriot-WattUniversity/dialog-babi with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

## Environment info
- `datasets` version: 2.3.2
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.9.9
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Add stanford dog dataset

_The documentation is not available anymore as the PR was closed or merged._

Hi @khushmeeet, thanks for your contribution.

But wouldn't it be better to add this dataset to the Hub?
- https://huggingface.co/docs/datasets/share
- https://huggingface.co/docs/datasets/dataset_script

Hi @albertvillanova

Dataset is added to Hub - https://huggingface.co/datasets/dgrnd4/stanford_dog_dataset

Great, so I guess we can close this issue, as the dataset is already available on the Hub.

OK I read the discussion on:
- #4504

Currently, priority is adding datasets to the Hub, not here on GitHub.

If you would like to contribute the loading script and all the metadata you generated (README + JSON files), you could:
- Either make a PR to the existing dataset on the Hub
- Create a new dataset on the Hub:
  - Either under your personal namespace
  - or even more professionally, under the namespace `stanfordSVL` (Stanford Vision and Learning Lab: https://svl.stanford.edu/)

You can use the Community tab to ping us if you need help or have any questions.

This PR is for adding dataset, related to issue #4504.

We are adding Stanford dog breed dataset. It is a multi class image classification dataset.
Details can be found here - http://vision.stanford.edu/aditya86/ImageNetDogs/

Tests on dummy data is failing currently, which I am looking into.


Add text decorators

This PR adds some decoration to text about different modalities to make it more obvious separate guides exist for audio, vision, and text. The goal is to make it easier for users to discover these guides!

![underline](https://user-images.githubusercontent.com/59462357/178044392-9596693e-9a4a-479a-a282-f1edbd90be1a.png)

TODO:

- [x] Open PR to support new Tailwind classes


Fix: conll2003 - fix empty example

As reported in https://huggingface.co/datasets/conll2003/discussions/2#62c45a14f93fc97e8260532f, there was an extra empty example at the end of the dataset


Concurrency bug when using same cache among several jobs

bug

Something isn't working

I can confirm that if I run one job first that processes the dataset, then I can run any jobs in parallel with no problem (no write-concurrency anymore...).

Hi! That's weird. It seems like the error points to the `mkstemp` function, but the official docs state the following:
```
There are no race conditions in the file’s creation, assuming that the platform properly implements the [os.O_EXCL](https://docs.python.org/3/library/os.html#os.O_EXCL) flag for [os.open()](https://docs.python.org/3/library/os.html#os.open)
```
So this could mean your platform doesn't support that flag.

~~Can you please check if wrapping the temp file creation (the line `tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)` in `_map_single`) with the `multiprocess.Lock` fixes the issue?~~
Perhaps wrapping the temp file creation in `_map_single`  with `filelock` could work:
```python
with FileLock(lock_path):
    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)
```
Can you please check if that helps?

## Describe the bug
I used to see this bug with an older version of the datasets. It seems to persist.

This is my concrete scenario: I launch several evaluation jobs on a cluster in which I share the file system and I share the cache directory used by huggingface libraries. The evaluation jobs read the same *.csv files. If my jobs get all scheduled pretty much at the same time, there are all kinds of weird concurrency errors. Sometime it crashes silently. This time I got lucky that it crashed with a stack trace that I can share and maybe you get to the bottom of this. If you don't have a similar setup available, it may be hard to reproduce as you really need two jobs accessing the same file at the same time to see this type of bug.

## Steps to reproduce the bug
I'm running a modified version of `run_glue.py` script adapted to my use case. I've seen the same problem when running some glue datasets as well (so it's not specific to loading the datasets from csv files).

## Expected results
No crash, concurrent access to the (intermediate) files just fine.

## Actual results
Crashes due to races/concurrency bugs.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.18.0-348.23.1.el8_5.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 8.0.0
- Pandas version: 1.1.0

Stack trace that I just got with the crash (I've obfuscated some names, it should still be quite informative):

```
Running tokenizer on dataset:   0%|          | 0/3 [00:00<?, ?ba/s]
Traceback (most recent call last):
  File "../../src/models//run_*******.py", line 600, in <module>
    main()
  File "../../src/models//run_*******.py", line 444, in main
    raw_datasets = raw_datasets.map(
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/dataset_dict.py", line 770, in map
    {
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/dataset_dict.py", line 771, in <dictcomp>
    k: dataset.map(
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2376, in map
    return self._map_single(
  File "/*******/envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 551, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 518, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/*******/envs/tr-crt/lib/python3.8/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2776, in _map_single
    buf_writer, writer, tmp_file = init_buffer_and_writer()
  File "/*******//envs/tr-crt/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2696, in init_buffer_and_writer
    tmp_file = tempfile.NamedTemporaryFile("wb", dir=os.path.dirname(cache_file_name), delete=False)
  File "/*******//envs/tr-crt/lib/python3.8/tempfile.py", line 541, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/*******//envs/tr-crt/lib/python3.8/tempfile.py", line 250, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
FileNotFoundError: [Errno 2] No such file or directory: '/*******/cache-transformers//transformers/csv/default-ef9cd184210742a7/0.0.0/51cce309a08df9c4d82ffd9363bbe090bf173197fc01a71b034e8594995a1a58/tmps8l6j5yc'
```

As I ran 100s of experiments last year for an empirical paper, I ran into this type of bugs several times. I found several bandaid/work-arounds, e.g., run one job first that caches the dataset => eliminate concurrency; OR use unique caches => eliminate concurrency (but increase storage space), etc. and it all works fine.

I'd like to help you fixing this bug as it's really annoying to always apply the work arounds. Let me know what other info from my side could help you figure out the issue.

Thanks for your help!


Fix _resolve_single_pattern_locally on Windows with multiple drives

Good catch ! Sorry I forgot (again) about windows paths when writing this x)

Currently, when `_resolve_single_pattern_locally` is called from a different drive than the one in `pattern`, it raises an exception:
```
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\io\parquet.py:35: in __init__
    **kwargs,
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\builder.py:287: in __init__
    sanitize_patterns(data_files), base_path=base_path, use_auth_token=use_auth_token
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:761: in from_local_or_remote
    if not isinstance(patterns_for_key, DataFilesList)
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:723: in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:321: in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:239: in _resolve_single_pattern_locally
    for filepath in glob_iter
C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\site-packages\datasets\data_files.py:242: in <listcomp>
    os.path.relpath(filepath, base_path), os.path.relpath(pattern, base_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\popen-gw0\\data6\\dataset.parquet'
start = '/'

...

E                   ValueError: path is on mount 'C:', start on mount 'D:'
```

This PR makes sure that `base_path` is in the same drive as `pattern`.


Transfer CI to GitHub Actions

Thanks a lot @albertvillanova ! I hope we're finally done with flakiness on windows ^^

Also thanks for paying extra attention to billing and avoiding running unnecessary jobs. Though for certain aspects (see my comments), I think it's worth having the extra jobs to make our life easier

~@lhoestq I think you forgot to add your comments?~

I had missed it among all the other comments...

@lhoestq, I'm specially enthusiastic with the fail-fast policy: it was in my TODO list for a long time. I really think it will have a positive impact (I would love to know the spent time saving it will enable, besides the carbon footprint reduction). :wink:

So yes, as you said above, let's give it a try at least. If we encounter any inconvenience, we can easily disable it.

Question: I guess I have to disable CircleCI CI before merging this PR?

This PR transfers CI from CircleCI to GitHub Actions. The implementation in GitHub Actions tries to be as faithful as possible to the implementation in CircleCI and get the same output results (exceptions below).

**IMPORTANT NOTE**: The fast-fail policy (described below) is not finally implemented, so that:
- we can continue merging PRs with CI in red because of some random error returned by the Hub
- it is not annoying for maintainers to have to relaunch failed CI jobs

See comments here: https://github.com/huggingface/datasets/pull/4659#discussion_r918802348

Differences in the implementation in GitHub Actions compared to the CircleCI one:
- This PR introduces some *fail-fast* mechanisms to significantly reduce the total time CI is running, both because of environmental impact and because CI in GitHub Actions billing depends on the minutes per month running time (see [About billing for GitHub Actions](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions)):
  - All tests *depend* on `check_code_quality` job: only if `check_code_quality` passes, then the other test jobs are launched
  - The tests are implemented with a matrix strategy (cross-product: OS and PyArrow versions) and fail-fast: if any of the 4 processes fails, the others are cancelled
- OS dependencies for Linux (see table below)

   | OS dependencies | Passed tests | Skipped tests |
   | --- | ---: | ---: |
   | libsndfile1-dev | 4786 | 3119 |
   | libsndfile1 | 4786 | 3119 |
   | libsndfile1, sox | 4788 | 3117 |

  - This PR replaces `libsndfile1-dev` with `libsndfile1`: the same number of passing tests but less packages installed
  - This PR adds `sox`: required by MP3 tests (2 more tests are passed: 4788 instead of 4786)
- For tests using PyArrow 6, this PR uses 6.0.1 instead of 6.0.0

TO DO:
- [ ] Remove old CircleCI CI: kept for the moment to compare stability and performance

Close #4658.

## Comparison between CircleCI and GitHub Actions

| | | CircleCI | GitHub Actions |
| --- | --- | ---: | ---: |
| Ubuntu, pyarrow-latest ||||
|| Passed tests | 4786 | 4788 |
|| Duration | 11m 0s | 10m 10s |
| Windows, pyarrow-latest ||||
|| Passed tests | 4783 | 4783 |
|| Duration | 29m 59s | 22m 56s |


Transfer CI tests to GitHub Actions

Let's try CI tests using GitHub Actions to see if they are more stable than on CircleCI.


Add SQuAD2.0 Dataset

dataset request

Requesting to add a new dataset

Hey, It's already present [here](https://huggingface.co/datasets/squad_v2)

Hi! This dataset is indeed already available on the Hub. Closing.

## Adding a Dataset
- **Name:** *SQuAD2.0*
- **Description:** *Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.*
- **Paper:** *https://aclanthology.org/P18-2124.pdf*
- **Data:** *https://rajpurkar.github.io/SQuAD-explorer/*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Amazon-QA Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Amazon-QA*
- **Description:** *The dataset is .jsonl format, where each line in the file is a json string that corresponds to a question, existing answers to the question and the extracted review snippets (relevant to the question).*
- **Paper:** *https://github.com/amazonqa/amazonqa/tree/master/paper*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/amazon-qa.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Simple Wikipedia

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Simple Wikipedia*
- **Description:** *Two different versions of the data set now exist. Both were generated by aligning Simple English Wikipedia and English Wikipedia. A complete description of the extraction process can be found in "Simple English Wikipedia: A New Simplification Task", William Coster and David Kauchak (2011).*
- **Paper:** *https://aclanthology.org/P11-2117/*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/SimpleWiki.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Quora Question Triplets Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Quora Question Triplets*
- **Description:** *This dataset consists of over 400,000 lines of potential question duplicate pairs. Each line contains IDs for each question in the pair, the full text for each question, and a binary value that indicates whether the line truly contains a duplicate pair.*
- **Paper:**
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/quora_duplicates_triplets.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Altlex dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Altlex*
- **Description:** *Git repository for software associated with the 2016 ACL paper "Identifying Causal Relations Using Parallel Wikipedia Articles.”*
- **Paper:** *https://aclanthology.org/P16-1135.pdf*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/altlex.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Sentence Compression Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Sentence Compression*
- **Description:** *Large corpus of uncompressed and compressed sentences from news articles.*
- **Paper:** *https://www.aclweb.org/anthology/D13-1155/*
- **Data:** *https://github.com/google-research-datasets/sentence-compression/tree/master/data*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Flickr 30k Dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Flickr 30k*
- **Description:** *To produce the denotation graph, we have created an image caption corpus consisting of 158,915 crowd-sourced captions describing 31,783 images. This is an extension of our previous Flickr 8k Dataset. The new images and captions focus on people involved in everyday activities and events.*
- **Paper:** *https://transacl.org/ojs/index.php/tacl/article/view/229/33*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/flickr30k_captions.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add SPECTER dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *SPECTER*
- **Description:** *SPECTER: Document-level Representation Learning using Citation-informed Transformers*
- **Paper:** *https://doi.org/10.18653/v1/2020.acl-main.207*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/specter_train_triples.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add PAQ dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *PAQ*
- **Description:** *This repository contains code and models to support the research paper PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them*
- **Paper:** *https://arxiv.org/abs/2102.07033*
- **Data:** *https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/PAQ_pairs.jsonl.gz*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add WikiAnswers dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *WikiAnswers*
- **Description:** *The WikiAnswers corpus contains clusters of questions tagged by WikiAnswers users as paraphrases. Each cluster optionally contains an answer provided by WikiAnswers users.*
- **Paper:** *https://dl.acm.org/doi/10.1145/2623330.2623677*
- **Data:** *https://github.com/afader/oqa#wikianswers-corpus*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Add Reddit dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *Reddit comments (2015-2018)*
- **Description:** *Reddit is an American social news aggregation website, where users can post links, and take part in discussions on these posts. These threaded discussions provide a large corpus, which is converted into a conversational dataset using the tools in this directory.*
- **Paper:** *https://arxiv.org/abs/1904.06472*
- **Data:** *https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit*
- **Motivation:** *Dataset for training and evaluating models of conversational response*


Set HF_SCRIPTS_VERSION to main

After renaming "master" to "main", the CI fails with
```
AssertionError: 'https://raw.githubusercontent.com/huggingface/datasets/main/datasets/_dummy/_dummy.py' not found in "Couldn't find a dataset script at /home/circleci/datasets/_dummy/_dummy.py or any data file in the same directory. Couldn't find '_dummy' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/_dummy/_dummy.py"
```

This is because in the CI we were still using `HF_SCRIPTS_VERSION=master`. I changed it to "main"


[Minor fix] Typo correction

recieve -> receive


Rename master to main

All the mentions I found on google were simple URLs that will be redirected, so it's fine. I also checked the spaces and we should be good:
- dalle-mini used to install the master branch but [it's no longer the case](https://huggingface.co/spaces/flax-community/dalle-mini/commit/b78c972afd5c2d2bed087be6479fe5c9c6cfa741)
- same for [logo generator](https://huggingface.co/spaces/tom-doerr/logo_generator/commit/a9ea330e518870d0ca8f65abb56f71d86750d8e4)
- I opened a PR to fix [vision-datasets-viewer](https://huggingface.co/spaces/nateraw/vision-datasets-viewer/discussions/1)

Ok let's rename the branch, and then we can merge this PR

This PR renames mentions of "master" by "main" in the code base for several cases:
- set the default dataset script version to "main" if the local installation of `datasets` is a dev installation
- update URLs to this github repository to use "main"
- update the DVC benchmark
- update the github workflows
- update docstrings
- update tests to compare the changes in dataset cards against "main"


Streaming issue for ccdv/pubmed-summarization

Thanks for reporting @lewtun.

I confirm there is an issue with streaming: it does not stream locally.

Oh, after investigation, the source of the issue is in the Hub dataset loading script.

I'm opening a PR on the Hub dataset.

I've opened a PR on their Hub dataset to support streaming: https://huggingface.co/datasets/ccdv/pubmed-summarization/discussions/2

### Link

https://huggingface.co/datasets/ccdv/pubmed-summarization

### Description

This was reported by a [user of AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/7). It seems like streaming doesn't work due to the way the dataset loading script is defined?

```
Status code:   400
Exception:     FileNotFoundError
Message:       https://huggingface.co/datasets/ccdv/pubmed-summarization/resolve/main/train.zip/train.txt
```

### Owner

No


Dataset Viewer issue for kmfoda/booksum

dataset-viewer

Related to the dataset viewer on huggingface.co

Thanks for reporting, @lewtun.

It works locally in streaming mode:
```
{'bid': 27681,
 'is_aggregate': True,
 'source': 'cliffnotes',
 'chapter_path': 'all_chapterized_books/27681-chapters/chapters_1_to_2.txt',
 'summary_path': 'finished_summaries/cliffnotes/The Last of the Mohicans/section_1_part_0.txt',
 'book_id': 'The Last of the Mohicans.chapters 1-2',
 'summary_id': 'chapters 1-2',
 'content': None,
 'summary': '{"name": "Chapters 1-2", "url": "https://web.archive.org/web/20201101053205/https://www.cliffsnotes.com/literature/l/the-last-of-the-mohicans/summary-and-analysis/chapters-12", "summary": "Before any characters appear, the time and geography are made clear. Though it is the last war that England and France waged for a country that neither would retain, the wilderness between the forces still has to be...
```

I'm forcing the refresh of the preview.

The preview appears as expected once the refresh forced.

### Link

https://huggingface.co/datasets/kmfoda/booksum

### Description

A [user of AutoTrain Evaluate](https://huggingface.co/spaces/autoevaluate/model-evaluator/discussions/9) discovered this dataset cannot be streamed due to:

```
Status code:   400
Exception:     ClientResponseError
Message:       401, message='Unauthorized', url=URL('https://huggingface.co/datasets/kmfoda/booksum/resolve/47953f583d6967f086cb16a2f4d2346e9834024d/test.csv')
```

I'm not sure why it says "Unauthorized" since it's just a bunch of CSV files in a repo

### Owner

No


Support all split in streaming mode

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4640). All of your documentation changes will be reflected on that endpoint.

Fix #4637.


Add HaGRID -- HAnd Gesture Recognition Image Dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** HaGRID -- HAnd Gesture Recognition Image Dataset
- **Description:** We introduce a large image dataset HaGRID (HAnd Gesture Recognition Image Dataset) for hand gesture recognition (HGR) systems. You can use it for image classification or image detection tasks. Proposed dataset allows to build HGR systems, which can be used in video conferencing services (Zoom, Skype, Discord, Jazz etc.), home automation systems, the automotive sector, etc.
- **Paper:** https://arxiv.org/abs/2206.08219
- **Data:** https://github.com/hukenovs/hagrid

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


The speechocean762 dataset

CircleCL reported two errors, but I didn't find the reason. The error message:
```
_________________ ERROR collecting tests/test_dataset_cards.py _________________
tests/test_dataset_cards.py:53: in <module>
    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
tests/test_dataset_cards.py:35: in get_changed_datasets
    diff_output = check_output(["git", "diff", "--name-only", "origin/master...HEAD"], cwd=repo_path)
../.pyenv/versions/3.6.15/lib/python3.6/subprocess.py:356: in check_output
    **kwargs).stdout
../.pyenv/versions/3.6.15/lib/python3.6/subprocess.py:438: in run
    output=stdout, stderr=stderr)
E   subprocess.CalledProcessError: Command '['git', 'diff', '--name-only', 'origin/master...HEAD']' returned non-zero exit status 128.

=========================== short test summary info ============================
ERROR tests/test_dataset_cards.py - subprocess.CalledProcessError: Command '[...
ERROR tests/test_dataset_cards.py - subprocess.CalledProcessError: Command '[...
= 4011 passed, 2357 skipped, 2 xfailed, 1 xpassed, 116 warnings, 2 errors in 284.32s (0:04:44) =

Exited with code exit status 1
```
I'm not sure if it was caused by this PR ...

I ran `tests/test_dataset_cards.py` in my local environment, and it passed:
```
(venv)$ pytest tests/test_dataset_cards.py
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/zhangjunbo/src/datasets
plugins: forked-1.4.0, datadir-1.3.1, xdist-2.5.0
collected 1531 items

tests/test_dataset_cards.py ..... [100%]
======================= 766 passed, 765 skipped in 2.55s ========================
```

[speechocean762](https://www.openslr.org/101/) is a non-native English corpus for pronunciation scoring tasks. It is free for both commercial and non-commercial use.

I believe it will be easier to use if it could be available on Hugging Face.


The "all" split breaks streaming

bug

Something isn't working

Thanks for reporting @cakiki.

Yes, this is a bug. We are investigating it.

@albertvillanova Nice! Let me know if it's something I can fix my self; would love to contribtue!

@cakiki I was working on this but if you would like to contribute, go ahead. I will close my PR. ;)

For the moment I just pushed the test (to see if it impacts other tests).

It impacted the test `test_generator_based_download_and_prepare` and I have fixed this.

So that you can copy the test I implemented in my PR and then implement a fix for this issue that passes the test `tests/test_builder.py::test_builder_as_streaming_dataset`.

Hi @cakiki are you still interested in working on this? Are you planning to open a PR?

Hi @albertvillanova ! Sorry it took so long; I wanted to spend this weekend working on it.

## Describe the bug

Not sure if this is a bug or just the way streaming works, but setting `streaming=True` did not work when setting `split="all"`

## Steps to reproduce the bug
The following works:
```python
ds = load_dataset('super_glue', 'wsc.fixed', split='all')
```
The following throws `ValueError: Bad split: all. Available splits: ['train', 'validation', 'test']`:

```python
ds = load_dataset('super_glue', 'wsc.fixed', split='all', streaming=True)
```

## Expected results
An iterator over all splits.

## Actual results
I had to do the following to achieve the desired result:
```python
from itertools import chain
ds = load_dataset('super_glue', 'wsc.fixed', streaming=True)
it =  chain.from_iterable(ds.values())
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.15.0-176-generic-x86_64-with-glibc2.31
- Python version: 3.10.5
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Add info in docs about behavior of download_config.num_proc

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

I went to override `download_config.num_proc` and was confused about what was happening under the hood. It would be nice to have the behavior documented a bit better so folks know what's happening when they use it.

**Describe the solution you'd like**

- Add note about how the default number of workers is 16. Related code:

https://github.com/huggingface/datasets/blob/7bcac0a6a0fc367cc068f184fa132b8de8dfa11d/src/datasets/download/download_manager.py#L299-L302

- Add note that if the number of workers is higher than the number of files to download, it won't use multiprocessing.

**Describe alternatives you've considered**

maybe it would also be nice to set `num_proc` = `num_files` when `num_proc` > `num_files`.

**Additional context**

...


Dataset Viewer issue for vadis/sv-ident

Thanks for reporting, @e-tornike

Some context:
- #4527

The dataset loads locally in streaming mode:
```python
In [2]: from datasets import load_dataset; ds = load_dataset("vadis/sv-ident", split="validation", streaming=True); item = next(iter(ds)); item
Using custom data configuration default
Out[2]:
{'sentence': 'Im Falle von Umweltbelastungen kann selten eindeutig entschieden werden, ob Unbedenklichkeitswerte bereits erreicht oder überschritten sind, die die menschliche Gesundheit oder andere Wohlfahrts»güter« beeinträchtigen.',
 'is_variable': 0,
 'variable': [],
 'research_data': [],
 'doc_id': '51971',
 'uuid': 'ee3d7f88-1a3e-4a59-997f-e986b544a604',
 'lang': 'de'}
```

~~I have forced the refresh of the split in the preview without success.~~

I have forced the refresh of the split in the preview, and now it works.

Preview seems to work now.

https://huggingface.co/datasets/vadis/sv-ident/viewer/default/validation

OK, thank you @e-tornike.

Apparently, after forcing the refresh, we just had to wait a little until it is effectively refreshed.

I'm closing this issue as it was solved after forcing the refresh of the split in the preview.

### Link

https://huggingface.co/datasets/vadis/sv-ident/viewer/default/validation

### Description

Error message when loading validation split in the viewer:

```
Status code:   400
Exception:     Status400Error
Message:       The split cache is empty.
```

### Owner

_No response_


Can't load the Hausa audio dataset

Could you provide the error details. It is difficult to debug otherwise. Also try other config. `ha` is not a valid.

common_voice_train = load_dataset("common_voice", "ha", split="train+validation")


[data_files] Only match separated split names

_The documentation is not available anymore as the PR was closed or merged._

I ran a script to find affected datasets (just did it on non-private non-gated). Adding "testing" and "evaluation" fixes all of of them except one:
- projecte-aina/cat_manynames:	human_annotated_testset.tsv

Let me open a PR on their repository to fix it
EDIT: pr [here](https://huggingface.co/datasets/projecte-aina/cat_manynames/discussions/2)

Feel free to merge @albertvillanova if it's all good to you :)

Thanks for the feedback @albertvillanova  I took your comments into account :)
- added numbers as supported delimiters
- used list comprehension to create the patterns list
- updated the docs and the tests according to your comments

Let me know what you think !

I ended up removing the patching and the context manager :) merging

As reported in https://github.com/huggingface/datasets/issues/4477, the current pattern matching to infer which file goes into which split is too permissive. For example a file "contest.py" would be considered part of a test split (it contains "test") and "seqeval.py" as well (it contains "eval").

In this PR I made the pattern matching more robust by only matching split names **between separators**. The supported separators are dots, dashes, spaces and underscores.

I updated the docs accordingly.

One detail about the tests: I had to update one test because it was using `PurePath.match` as a reference for globbing, but it doesn't support the `[..]` glob pattern. Therefore I added a `mock_fs` context manager that can be used to easily define a dummy filesystem with certain files in it and run pattern matching tests. Its code comes mostly from test_streaming_download_manager.py

Close https://github.com/huggingface/datasets/issues/4477


'sort' method sorts one column only

Hi ! `ds.sort()` does sort the full dataset, not just one column:
```python
from datasets import *

ds = Dataset.from_dict({"foo": [3, 2, 1], "bar": ["c", "b", "a"]})
print(d.sort("foo").to_pandas()
#    foo bar
# 0    1   a
# 1    2   b
# 2    3   c
```

What made you think it was not the case ? Did you experience a situation where it was only sorting one column ?

Hi! thank you for your quick reply!
I wanted to sort the `cnn_dailymail` dataset by the length of the labels (num of characters). I added a new column to the dataset (`ds.add_column`) with the lengths and then sorted by this new column. Only the new length column was sorted, the reset left in their original order.

That's unexpected, can you share the code you used to get this ?

The 'sort' method changes the order of one column only (the one defined by the argument 'column'), thus creating a mismatch between a sample fields. I would expect it to change the order of the samples as a whole, based on the 'column' order.


Update WinoBias README

I'm adding some information about Winobias that I got from the paper :smile:

I think this makes it a bit clearer!


fix(dataset_wrappers): Fixes access to fsspec.asyn in torch_iterable_dataset.py.

Fix #4612.

Apparently, newest `fsspec` versions do not allow access to attribute-based modules if they are not imported, such as `fsspec.async`.

Thus, @mariosasko suggested to add the missing part to the module import to allow for its access.


Rename repo default branch to main

maintenance

Maintenance tasks

Rename repository default branch to `main` (instead of current `master`).

Once renamed, users will have to manually update their local repos:

- [ ] Upstream:
  ```
  git branch -m master main
  git fetch upstream main
  git branch -u upstream/main main
  git remote set-head upstream -a
  ```

- [ ] Origin:
Rename fork default branch as well at: https://github.com/USERNAME/lam/settings/branches
Then:
  ```
  git fetch origin main
  git remote set-head origin -a
  ```

CC: @sgugger


Fix time type `_arrow_to_datasets_dtype` conversion

Fix #4620

The issue stems from the fact that `pa.array([time_data]).type` returns `DataType(time64[unit])`, which doesn't expose the `unit` attribute, instead of `Time64Type(time64[unit])`. I believe this is a bug in PyArrow. Luckily, the both types have the same `str()`, so in this PR I call `pa.type_for_alias(str(type))` to convert them both to the `Time64Type(time64[unit])` format.

cc @severo


fixed duplicate calculation of spearmanr function in metrics wrapper.

Great, can open a PR in `evaluate` as well to optimize this.

Relatedly, I wanted to add a new metric, Kendall Tau (https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.kendalltau.html). If I were to open a PR with the wrapper, description, citation, docstrings, readme, etc. would it make more sense to do that in the `datasets` or `evaluate` repo (or both)?

Thanks!

PR opened in`evaluate` library with same minor adjustment: https://github.com/huggingface/evaluate/pull/176

> If I were to open a PR with the wrapper, description, citation, docstrings, readme, etc. would it make more sense to do that in the datasets or evaluate repo (or both)?

I think you could just add it to `evaluate`, we're not adding new metrics in this repo anymore

During _compute, the scipy.stats spearmanr function was called twice, redundantly, once for calculating the score and once for calculating the p-value, under the conditional branch where return_pvalue=True. I adjusted the _compute function to execute the spearmanr function once, store the results tuple in a temporary variable, and then pass the indexed contents to the expected keys of the returned dictionary.


Add non-commercial licensing info for datasets for which we removed tags

yep plus `license_details` also makes sense for this IMO

We removed several YAML tags saying that certain datasets can't be used for commercial purposes: https://github.com/huggingface/datasets/pull/4613#discussion_r911919753

Reason for this is that we only allow tags that are part of our [supported list of licenses](https://github.com/huggingface/datasets/blob/84fc3ad73c85de4eda5d152dfede7671491449cb/src/datasets/utils/resources/standard_licenses.tsv)

We should update the Licensing Information section of the concerned dataset cards, now that the non-commercial tag doesn't exist anymore for certain datasets


Unpack `dl_manager.iter_files` to allow parallization

Cool thanks ! Yup it sounds like the right solution.

It looks like `_generate_tables` needs to be updated as well to fix the CI

Iterate over data files outside `dl_manager.iter_files` to allow parallelization in streaming mode.

(The issue reported [here](https://discuss.huggingface.co/t/dataset-only-have-n-shard-1-when-has-multiple-shards-in-repo/19887))

PS: Another option would be to override `FilesIterable.__getitem__` to make it indexable and check for that type in `_shard_kwargs` and `n_shards,` but IMO this solution adds too much unnecessary complexity.


Remove all paperswithcode_id: null

> We've been using `null` to specify that we checked on pwc but the dataset doesn't exist there.

@lhoestq maybe it's better to accept it on the Hub side then? Let me know if you want us to do it Hub-side

Yup it's maybe better to support it on the Hub side then indeed, thanks ! Closing this one

On the Hub there is a validation error on the `paperswithcode_id` tag when the value is `null`:

<img width="686" alt="image" src="https://user-images.githubusercontent.com/42851186/177151825-93d341c5-25bd-41ab-96c2-c0b516d51c68.png">

We've been using `null` to specify that we checked on pwc but the dataset doesn't exist there.

To have the validation working again we can simply remove all the `paperswithcode_id: null`.

cc @julien-c


Loading MNIST as Pytorch Dataset

bug

Something isn't working

Hi ! We haven't implemented the conversion from images data to PyTorch tensors yet I think

cc @mariosasko

So I understand:

set_format() does not properly do the conversion to pytorch tensors from PIL images.

So that someone who stumbles on this can use the package:

```python
dataset = load_dataset("mnist", split="train")
def transform_func(examples):
    examples["image"] = [np.array(img) for img in examples["image"]]
    return examples
dataset = dataset.with_transform(transform_func)
dataset[0]
```

This then appears to work with pytorch dataloaders as:
```
dataloader=torch.utils.data.DataLoader(dataset,batch_size=1)
```

and tensorflow as:
```
dataset=dataset.to_tf_dataset(batch_size=1)
```

Hi! `set_transform`/`with_transform` is indeed the correct solution for the conversion. Improving this part of the API is one of the things I'm working on currently, so stay tuned!

## Describe the bug
Conversion of MNIST dataset to pytorch fails with bug

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("mnist", split="train")
dataset.set_format('torch')
dataset[0]
print()
```

## Expected results
Expect to see torch tensors image and label

## Actual results
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2020.3.3\plugins\python\helpers\pydev\pydevd.py", line 1491, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2020.3.3\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/chapm/PycharmProjects/multiviewdata/multiviewdata/huggingface/mnist.py", line 13, in <module>
    dataset[0]
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\arrow_dataset.py", line 2154, in __getitem__
    return self._getitem(
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\arrow_dataset.py", line 2139, in _getitem
    formatted_output = format_table(
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\formatting.py", line 532, in format_table
    return formatter(pa_table, query_type=query_type)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\formatting.py", line 281, in __call__
    return self.format_row(pa_table)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 58, in format_row
    return self.recursive_tensorize(row)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 54, in recursive_tensorize
    return map_nested(self._recursive_tensorize, data_struct, map_list=False)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 356, in map_nested
    mapped = [
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 357, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 309, in _single_map_nested
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 309, in <dictcomp>
    return {k: _single_map_nested((function, v, types, None, True, None)) for k, v in pbar}
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\utils\py_utils.py", line 293, in _single_map_nested
    return function(data_struct)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 51, in _recursive_tensorize
    return self._tensorize(data_struct)
  File "C:\Users\chapm\PycharmProjects\multiviewdata\venv\lib\site-packages\datasets\formatting\torch_formatter.py", line 38, in _tensorize
    if np.issubdtype(value.dtype, np.integer):
AttributeError: 'bytes' object has no attribute 'dtype'
python-BaseException

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Windows-10-10.0.22579-SP0
- Python version: 3.9.2
- PyArrow version: 8.0.0
- Pandas version: 1.4.1


Fix ImageFolder with parameters drop_metadata=True and drop_labels=False (when metadata.jsonl is present)

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq @mariosasko pls take a look at https://github.com/huggingface/datasets/pull/4622/commits/769e4c046a5bd5e3a4dbd09cfad1f4cf60677869. I modified `_generate_examples()` according to the same logic too: removed checking if `metadata_files` are not empty for the case when `self.config.drop_metadata=True` because I think we should be aligned with the config and preserve labels if `self.config.drop_labels=False` (the default value) and `self.config.drop_metadata=True` but `metadata_files` are passed. This is an extremely unlikely use case (when `self.config.drop_metadata=True`, but `metadata_files` are passed to `_generate_examples()`) since users usually do not use `_generate_examples()` alone but I believe it would be consistent to have the same behavior as in `_splits_generators()`. This change requires change in tests too if we suppose that we want to preserve labels (default value of `self.config.drop_labels` is False) when `self.config.drop_metadata=True`, even if `metadata_files` are for some reason provided (as it is done in tests).

wdyt about this change?

@lhoestq it wouldn't raise an error if we check `example.keys() == {"image", "label"}` as test checks only `_generate_examples`, not `encode_example`. and in the implementation of this PR `_generate_examples` would return both `image` and `label` key in the case when `drop_metadata=True` and `drop_labels=False` (default) as it seems that we agreed on that :)

and on the other hand it would raise an error if `label` column is missing in _generate_examples when `drop_metadata=True` and `drop_labels=False`

by "it" i mean tests :D (`test_generate_examples_with_metadata_that_misses_one_image`, `test_generate_examples_with_metadata_in_wrong_location` and `test_generate_examples_drop_metadata`)

Perhaps we could make `self.config.drop_metadata = None` and `self.config.drop_labels = None` the defaults to see explicitly what the user wants. This would then turn into  `self.config.drop_metadata = False` and `self.config.drop_labels = True` if metadata files are present and `self.config.drop_metadata = True` and `self.config.drop_labels = False` if not. And if the user wants to have the `label` column alongside metadata columns, it can do so by passing `drop_labels = False` explicitely (in that scenario we have to check that the `label` column is not already present in metadata files). And maybe we can also improve the logging messages.

I find it problematic that the current implementation drops labels in some scenarios even if `self.config.drop_labels = False`, and the user doesn't have control over this behavior.

Let me know what you think.

Will fix #4621

ImageFolder raises `KeyError: 'label'` with params `drop_metadata=True` and `drop_labels=False` (if there is at least one metadata.jsonl file a data directory). This happens because metadata files are collected inside `analyze()` function regardless of `drop_metadata` value. And then the following condition doesn't pass: https://github.com/huggingface/datasets/blob/master/src/datasets/packaged_modules/imagefolder/imagefolder.py#L167

So I suggest to double check it inside `analyze()` not to collect metadata files if they are not needed. (and labels too, to be consistent)

---
Also, I added a test to check if labels are inferred correctly from directories names in general (because we didn't have it) :)


ImageFolder raises an error with parameters drop_metadata=True and drop_labels=False when metadata.jsonl is present

Something isn't working

## Describe the bug

If you pass `drop_metadata=True` and `drop_labels=False` when a `data_dir` contains at least one `matadata.jsonl` file, you will get a KeyError. This is probably not a very useful case but we shouldn't get an error anyway. Asking users to move metadata files manually outside `data_dir` or pass features manually (when there is a tool that can infer them automatically) don't look like a good idea to me either.

## Steps to reproduce the bug
### Clone an example dataset from the Hub
```bash
git clone https://huggingface.co/datasets/nateraw/test-imagefolder-metadata
```
### Try to load it
```python
from datasets import load_dataset
ds = load_dataset("test-imagefolder-metadata", drop_metadata=True, drop_labels=False)
```
or even just
```python
ds = load_dataset("test-imagefolder-metadata", drop_metadata=True)
```
as `drop_labels=False` is a default value.

## Expected results
A DatasetDict object with two features: `"image"` and `"label"`.

## Actual results
```
Traceback (most recent call last):
  File "/home/polina/workspace/datasets/debug.py", line 18, in <module>
    ds = load_dataset(
  File "/home/polina/workspace/datasets/src/datasets/load.py", line 1732, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 1227, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/polina/workspace/datasets/src/datasets/builder.py", line 1218, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1596, in encode_example
    return encode_nested_example(self, example)
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1165, in encode_nested_example
    {
  File "/home/polina/workspace/datasets/src/datasets/features/features.py", line 1165, in <dictcomp>
    {
  File "/home/polina/workspace/datasets/src/datasets/utils/py_utils.py", line 249, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/home/polina/workspace/datasets/src/datasets/utils/py_utils.py", line 249, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
KeyError: 'label'
```

## Environment info
`datasets` master branch

- `datasets` version: 2.3.3.dev0
- Platform: Linux-5.14.0-1042-oem-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.1
- Pandas version: 1.4.1


PyArrow Dataset error when calling `load_dataset`

Something isn't working

## Describe the bug

I am fine tuning a wav2vec2 model following the script here using my own dataset: https://github.com/huggingface/transformers/blob/main/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py

Loading my Audio dataset from the hub which was originally generated from disk results in the following PyArrow error:

```sh
File "/home/ubuntu/w2v2/run_speech_recognition_ctc.py", line 227, in main
  raw_datasets = load_dataset(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/load.py", line 1679, in load_dataset
  builder_instance.download_and_prepare(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/builder.py", line 704, in download_and_prepare
  self._download_and_prepare(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/builder.py", line 793, in _download_and_prepare
  self._prepare_split(split_generator, **prepare_split_kwargs)
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/builder.py", line 1268, in _prepare_split
  for key, table in logging.tqdm(
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
  for obj in iterable:
File "/home/ubuntu/.virtualenvs/meval/lib/python3.8/site-packages/datasets/packaged_modules/parquet/parquet.py", line 68, in _generate_tables
  for batch_idx, record_batch in enumerate(
File "pyarrow/_parquet.pyx", line 1309, in iter_batches
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs
```

## Steps to reproduce the bug

I created a dataset from a JSON lines manifest of `audio_filepath`, `text`, and `duration`.

When creating the dataset, I do something like this:

```python
import json
from datasets import Dataset, Audio

# manifest_lines is a list of dicts w/ "audio_filepath", "duration", and "text
for line in manifest_lines:
    line = line.strip()
    if line:
        line_dict = json.loads(line)
        manifest_dict["audio"].append(f"{root_path}/{line_dict['audio_filepath']}")
        manifest_dict["duration"].append(line_dict["duration"])
        manifest_dict["transcription"].append(line_dict["text"])

# Create a HF dataset
dataset = Dataset.from_dict(manifest_dict).cast_column(
    "audio", Audio(sampling_rate=16_000),
)

# From the docs for saving to disk
# https://huggingface.co/docs/datasets/v2.3.2/en/package_reference/main_classes#datasets.Dataset.save_to_disk
def read_audio_file(example):
    with open(example["audio"]["path"], "rb") as f:
        return {"audio": {"bytes": f.read()}}

dataset = dataset.map(read_audio_file, num_proc=70)
dataset.save_to_disk(f"/audio-data/hf/{artifact_name}")
dataset.push_to_hub(f"{org-name}/{artifact_name}", max_shard_size="5GB", private=True)
```

Then when I call `load_dataset()` in my training script, with the same dataset I generated above, and download from the huggingface hub I get the above stack trace.
I am able to load the dataset fine if I use `load_from_disk()`.

## Expected results

`load_dataset()` should behave just like `load_from_disk()` and not cause any errors.

## Actual results

See above

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
I am using the `huggingface/transformers-pytorch-gpu:latest` image
- `datasets` version: 2.3.0
- Platform: Docker/Ubuntu 20.04
- Python version: 3.8
- PyArrow version: 8.0.0


Dataset Viewer issue for shamikbose89/lancaster_newsbooks

It seems like the list of splits could not be obtained:

```python
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names("shamikbose89/lancaster_newsbooks", "default")
Using custom data configuration default
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 354, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/shamikbose89--lancaster_newsbooks/2d1c63d269bf7b9342accce0a95960b1710ab4bc774248878bd80eb96c1afaf7/lancaster_newsbooks.py", line 73, in _split_generators
    data_dir = dl_manager.download_and_extract(_URL)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 916, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 879, in extract
    urlpaths = map_nested(self._extract, path_or_paths, map_tuple=True)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 348, in map_nested
    return function(data_struct)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 884, in _extract
    protocol = _get_extraction_protocol(urlpath, use_auth_token=self.download_config.use_auth_token)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 388, in _get_extraction_protocol
    return _get_extraction_protocol_with_magic_number(f)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/download/streaming_download_manager.py", line 354, in _get_extraction_protocol_with_magic_number
    f.seek(0)
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/fsspec/implementations/http.py", line 684, in seek
    raise ValueError("Cannot seek streaming HTTP file")
ValueError: Cannot seek streaming HTTP file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

ping @huggingface/datasets

Oh, I removed the 'split' key from `kwargs`. I put  it back in, but there's still the same error

### Link

https://huggingface.co/datasets/shamikbose89/lancaster_newsbooks

### Description

Status code:   400
Exception:     ValueError
Message:       Cannot seek streaming HTTP file

I am able to use the dataset loading script locally and it also runs when I'm using the one from the hub, but the viewer still doesn't load

### Owner

Yes


Issue loading TheNoob3131/mosquito-data dataset

I am also getting a ValueError: 'Couldn't cast' at the bottom. Is this because of some delimiter issue? My dataset is on the Huggingface Hub. If you could look at it, that would be greatly appreciated.

Hi @thenerd31, thanks for reporting.

Please note that your issue is not caused by the Hugging Face Datasets library, but it has to do with the specific implementation of your dataset on the Hub.

Therefore, I'm transferring this discussion to your own dataset Community tab: https://huggingface.co/datasets/TheNoob3131/mosquito-data/discussions/1

![image](https://user-images.githubusercontent.com/53668030/179815591-d75fa7d3-3122-485f-a852-b06a68909066.png)

So my dataset is public in the Huggingface Hub, but when I try to load it using the load_dataset command, it shows that it is downloading the files, but throws a ValueError. When I went to my directory to see if the files were downloaded, the folder was blank.

Here is the error below:
ValueError                                Traceback (most recent call last)
Input In [8], in <cell line: 3>()
      1 from datasets import load_dataset
----> 3 dataset = load_dataset("TheNoob3131/mosquito-data", split="train")

File ~\Anaconda3\lib\site-packages\datasets\load.py:1679, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1676 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1678 # Download and prepare data
-> 1679 builder_instance.download_and_prepare(
   1680     download_config=download_config,
   1681     download_mode=download_mode,
   1682     ignore_verifications=ignore_verifications,
   1683     try_from_hf_gcs=try_from_hf_gcs,
   1684     use_auth_token=use_auth_token,
   1685 )
   1687 # Build dataset for splits
   1688 keep_in_memory = (
   1689     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1690 )

Is the dataset in the wrong format or is there some security permission that I should enable?


Make Extractor accept Path as input

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4718). All of your documentation changes will be reflected on that endpoint.

This PR:
- Makes `Extractor` accept instance of `Path` as input
- Removes unnecessary castings of `Path` to `str`


Dataset Viewer issue for LawalAfeez/englishreview-ds-mini

It's currently working, as far as I understand

https://huggingface.co/datasets/LawalAfeez/englishreview-ds-mini/viewer/LawalAfeez--englishreview-ds-mini/train

<img width="1556" alt="Capture d’écran 2022-07-19 à 09 24 01" src="https://user-images.githubusercontent.com/1676121/179761130-2d7980b9-c0f6-4093-8b1d-f0a3872fef3f.png">

---

What was your issue?

### Link

_No response_

### Description

Unable to view the split data

### Owner

_No response_


Support "tags" yaml tag

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4716). All of your documentation changes will be reflected on that endpoint.

Added the "tags" YAML tag, so that users can specify data domain/topics keywords for dataset search


Fix POS tags

CI failures are about missing content in the dataset cards or bad tags, and this is unrelated to this PR. Merging :)

We're now using `part-of-speech` and not `part-of-speech-tagging`, see discussion here: https://github.com/huggingface/datasets/commit/114c09aff2fa1519597b46fbcd5a8e0c0d3ae020#r78794777


Fix named split casting and sorting

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4714). All of your documentation changes will be reflected on that endpoint.

This PR:
- makes `NamedSplit` sortable: so that `sorted()` can be called on them
- removes unnecessary `sorted()` on `dict.keys()`: `dict_keys` view is already like a `set`
- removes unnecessary casting of `NamedSplit` to `str`


Document installation of sox OS dependency for audio

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4713). All of your documentation changes will be reflected on that endpoint.

The `sox` OS package needs being installed manually using the distribution package manager.

This PR adds this explanation to the docs.


Update README.md

Highlight that the licence granted by Amazon only covers non-commercial research use.


Document how to create a dataset loading script for audio/vision

documentation

Improvements or additions to documentation

Currently, in our docs for Audio/Vision/Text, we explain how to:
- Load data
- Process data

However we only explain how to *Create a dataset loading script* for text data.

I think it would be useful that we add the same for Audio/Vision as these have some specificities different from Text.

See, for example:
- #4697

CC: @stevhliu


Add object detection processing tutorial

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4710). All of your documentation changes will be reflected on that endpoint.

Great idea! Now that we have more than one task, it makes sense to separate image classification and object detection so it'll be easier for users to follow.

@lhoestq do we want to do that in this PR, or should we merge it and let @stevhliu reorganize separately?

The following adds a quick guide on how to process object detection datasets with `albumentations`.


WMT21 & WMT22

dataset request

Requesting to add a new dataset

Hi ! That would be awesome to have them indeed, thanks for opening this issue

I just added you to the WMT org on the HF Hub if you're interested in adding those datasets.

Feel free to create a dataset repository for each dataset and upload the data files there :) preferably in ZIP archives instead of TAR archives (the current WMT scripts don't support streaming TAR archives, so it would break the dataset preview). We've also had issues with the `statmt.org` host (data unavailable, slow download speed), that's why I think it's better if we re-host the files on the Hub.

`wmt21` (and wmt22) can be added in this GitHub repository I think, for consistency with the previous ones.
To add it, you can copy paste the code of the previous one (e.g. wmt19), and add the new data:
- in wmt_utils.py, add the new data subsets. You need to provide the download URLs, as well as the target and source languages
- in wmt21.py (renamed from wmt19.py), you can specify the subsets that WMT21 uses (i.e. the one you just added)
- in wmt_utils.py, define the python function that must be used to parse the subsets you added. To do so, you must go in `_generate_examples` and chose the proper `sub_generator` based on the subset name. For example, the `paracrawl_v3` subset uses the `_parse_tmx` function:

https://github.com/huggingface/datasets/blob/ede72d3f9796339701ec59899c7c31d2427046fb/datasets/wmt19/wmt_utils.py#L834-L835

Hopefully the data is in a format that is already supported and there's no need to write a new `_parse_*` function for the new subsets. Let me know if you have questions or if I can help :)

## Adding a Dataset
- **Name:** WMT21 & WMT22
- **Description:** We are going to have three tracks: two small tasks and a large task.
The small tracks evaluate translation between fairly related languages and English (all pairs). The large track uses 101 languages.
- **Paper:** /
- **Data:** https://statmt.org/wmt21/large-scale-multilingual-translation-task.html https://statmt.org/wmt22/large-scale-multilingual-translation-task.html
- **Motivation:** Many more languages than previous WMT versions - Could be very high impact

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).

I could also tackle this. I saw the existing logic for WMT models is a bit complex (datasets are stored on the wmt account & retrieved in separate wmt datasets afaict). How long do you think it would take me? @lhoestq


Fix require torchaudio and refactor test requirements

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4708). All of your documentation changes will be reflected on that endpoint.

Currently there is a bug in `require_torchaudio` (indeed it is requiring `sox` instead):
```python
def require_torchaudio(test_case):
    if find_spec("sox") is None:
...
```

The bug was introduced by:
- #3685
  - Commit: https://github.com/huggingface/datasets/pull/3685/commits/b5a3e7122d49c4dcc9333b1d8d18a833fc04b940

which moved
```python
require_sndfile = pytest.mark.skipif(
    # In Windows and OS X, soundfile installs sndfile
    (sys.platform != "linux" and find_spec("soundfile") is None)
    # In Linux, soundfile throws RuntimeError if sndfile not installed with distribution package manager
    or (sys.platform == "linux" and find_library("sndfile") is None),
    reason="Test requires 'sndfile': `pip install soundfile`; "
    "Linux requires sndfile installed with distribution package manager, e.g.: `sudo apt-get install libsndfile1`",
)
require_sox = pytest.mark.skipif(
    find_library("sox") is None,
    reason="Test requires 'sox'; only available in non-Windows, e.g.: `sudo apt-get install sox`",
)
require_torchaudio = pytest.mark.skipif(find_spec("torchaudio") is None, reason="Test requires 'torchaudio'")
```
to
```python
def require_sndfile(test_case):
    """
    Decorator marking a test that requires soundfile.
    These tests are skipped when soundfile isn't installed.
    """
    if (sys.platform != "linux" and find_spec("soundfile") is None) or (
        sys.platform == "linux" and find_library("sndfile") is None
    ):
        test_case = unittest.skip(
            "test requires 'sndfile': `pip install soundfile`; "
            "Linux requires sndfile installed with distribution package manager, e.g.: `sudo apt-get install libsndfile1`",
        )(test_case)
    return test_case

def require_sox(test_case):
    """
    Decorator marking a test that requires sox.
    These tests are skipped when sox isn't installed.
    """
    if find_library("sox") is None:
        return unittest.skip("test requires 'sox'; only available in non-Windows, e.g.: `sudo apt-get install sox`")(
            test_case
        )
    return test_case

def require_torchaudio(test_case):
    """
    Decorator marking a test that requires torchaudio.
    These tests are skipped when torchaudio isn't installed.
    """
    if find_spec("sox") is None:
        return unittest.skip("test requires 'torchaudio'")(test_case)
    return test_case
```

This PR;
- fixes the bug in `require_torchaudio`
- refactors the test requirements back to using `pytest` instead of `unittest`
  - the text in `pytest.skipif` `reason` can be used if needed in a test body: `require_torchaudio.kwargs["reason"]`


Dataset Viewer issue for TheNoob3131/mosquito-data

Thanks for reporting. I refreshed the dataset viewer and it now works as expected.

https://huggingface.co/datasets/TheNoob3131/mosquito-data

<img width="1135" alt="Capture d’écran 2022-07-18 à 13 15 22" src="https://user-images.githubusercontent.com/1676121/179566497-e47f1a27-fd84-4a8d-9d7f-2e0f2da803df.png">

We will investigate why it occurred in the first place

By chance, could you provide some details about the operations done on the dataset: was it private? gated?

Yes, it was a private dataset, and when I made it public, the Dataset Preview did not work.

However, now when I make the dataset private, it says that the Dataset Preview has been disabled. Why is this?

Thanks for the details. For now, the dataset viewer is always disabled on private datasets (see https://huggingface.co/docs/hub/datasets-viewer for more details)

Hi, it was working fine for a few hours, but then I can't see the dataset viewer again (public dataset). Why is this still happening?
It's the same error too:
![image](https://user-images.githubusercontent.com/53668030/179602465-f220f971-d3aa-49ba-a31b-60510f4c2a89.png)

OK? This is a bug, thanks for help spotting and reproducing it (it occurs when a dataset is switched to private, then to public). We will be working on it, meanwhile, I've restored the dataset viewer manually again.

### Link

_No response_

### Description

Getting this error when trying to view dataset preview:
Message:       401, message='Unauthorized', url=URL('https://huggingface.co/datasets/TheNoob3131/mosquito-data/resolve/8aceebd6c4a359d216d10ef020868bd9e8c986dd/0_Africa_train.csv')

### Owner

_No response_


Fix empty examples in xtreme dataset for bucc18 config

_The documentation is not available anymore as the PR was closed or merged._

I guess the report link is this instead: https://huggingface.co/datasets/xtreme/discussions/1

As reported in https://huggingface.co/muibk, there are empty examples in xtreme/bucc18.de

I applied your fix @mustaszewski

I also used a dict to make the dataset generation much faster


Fix crd3

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4705). All of your documentation changes will be reflected on that endpoint.

As reported in https://huggingface.co/datasets/crd3/discussions/1#62cc377073b2512b81662794, each split of the dataset was containing the same data. This issues comes from a bug in the dataset script

I fixed it and also uploaded the data to hf.co to make the dataset work in streaming mode


Skip tests only for lz4/zstd params if not installed

Currently, if `zstandard` or `lz4` are not installed, `test_compression_filesystems` and `test_streaming_dl_manager_extract_all_supported_single_file_compression_types` are skipped for all compression format parameters.

This PR fixes these tests, so that if `zstandard` or `lz4` are not installed, the tests are skipped only for the corresponding compression parameters (`zstd` or `lz4`), whereas the tests are not skipped for all the other compression parameters (`gzip`, `xz` and `bz2`).

Related to:
- #4688


Make cast in `from_pandas` more robust

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4703). All of your documentation changes will be reflected on that endpoint.

Make the cast in `from_pandas` more robust (as it was done for the packaged modules in https://github.com/huggingface/datasets/pull/4364)

This should be useful in situations like [this one](https://discuss.huggingface.co/t/loading-custom-audio-dataset-and-fine-tuning-model/8836/4).


Domain specific dataset discovery on the Hugging Face hub

enhancement

New feature or request

> Hi! I added a link to this issue in our internal request for adding keywords/topics to the Hub, which is identical to the `topic tags` solution. The `collections` solution seems too complex (as you point out). Regarding the `domain tags` solution, we primarily focus on machine learning, so I'm not sure if it's a good idea to make our current taxonomy more complex.

Thanks, for letting me know. Will you allow the topic tags to be user-generated or only chosen from a list?

Thanks for opening this issue @davanstrien.

As we discussed last week, the tag approach would be in principle the simpler to be implemented, either the domain tag (with closed vocabulary: more reliable but also more rigid), or the topic tag (with open vocabulary: more flexible for user needs)

Hi @davanstrien If i remember correctly this was also discussed inside a hf.co Discussion, would you be able to link it here too?

(where i suggested using `tags: - foo - bar` IIRC.

Thanks a ton!

> Hi @davanstrien If i remember correctly this was also discussed inside a hf.co Discussion, would you be able to link it here too?
>
> (where i suggested using `tags: - foo - bar` IIRC.
>
> Thanks a ton!

This doesn't ring a bell - I did a quick search of https://discuss.huggingface.co but didn't find anything.

The `tags: ` approach sounds like a good option for this. It would be especially nice if these could suggest existing tags, but this probably won't be easily possible through the current interface.

I opened a PR to add "tags" to the YAML validator:
https://github.com/huggingface/datasets/pull/4716

I also added "tags" to the [tagging app](https://huggingface.co/spaces/huggingface/datasets-tagging), with suggestions like "bio" or "newspapers"

Thanks @lhoestq for the initiative.

Just one question: are "tags" already supported on the Hub?

I think they aren't. Thus, the Hub should support them so that they are properly displayed.

> I think they're not displayed, but at least it should enable users to filter by tag in using `huggingface_hub` or using the appropriate query params on the website (not sure if it's possible yet though)

I think this would already be a helpful start. I'm happy to try this out with the datasets added to https://huggingface.co/organizations/biglam and use the `huggingface_hub` to filter those datasets using the tags.

**Is your feature request related to a problem? Please describe.**

## The problem

The datasets hub currently has `8,239` datasets. These datasets span a wide range of different modalities and tasks (currently with a bias towards textual data).

There are various ways of identifying datasets that may be relevant for a particular use case:

- searching
- various filters

Currently, however, there isn't an easy way to identify datasets belonging to a specific domain. For example, I want to browse machine learning datasets related to 'social science' or 'climate change research'.

The ability to identify datasets relating to a specific domain has come up in discussions around the [BigLA](https://github.com/bigscience-workshop/lam/) datasets hackathon https://github.com/bigscience-workshop/lam/discussions/31#discussioncomment-3123610. As part of the hackathon, we're currently collecting datasets related to Libraries, Archives and Museums and making them available via the hub. We currently do this under a Hugging Face organization (https://huggingface.co/biglam). However, going forward, I can see some of these datasets being migrated to sit under an organization that is the custodian of the dataset (for example, a national library the data was originally from). At this point, it becomes more difficult to quickly identify datasets from this domain without relying on search.

This is also related to some existing issues on Github related to metadata on the hub:
- https://github.com/huggingface/datasets/issues/3625
- https://github.com/huggingface/datasets/issues/3877

**Describe the solution you'd like**

### Some possible solutions that may help with this:

#### Enable domain tags (from a controlled vocabulary)
- This would add metadata field to the YAML for the domain a dataset relates to
- Advantages:
	- the list is controlled, allowing it to be more easily integrated into the datasets tag app (https://huggingface.co/space/huggingface/datasets-tagging)
	- the controlled vocabulary could align with an existing controlled vocabulary
	- this additional metadata can be used to perform filtering by domain
- disadvantages
	- choosing the best controlled vocab may be difficult
	- there are many datasets that are likely to fit into the 'machine learning' domain (i.e. there is a long tail of datasets that aren't in more 'generic' machine learning domain

#### Enable topic tags (user-generated)

Enable 'free form' topic tags for datasets and models. This would be closer to GitHub's repository topics which can be chosen from a controlled list (https://github.com/topics/) but can also be more user/org specific. This could potentially be useful for organizations to also manage their own models and datasets as the number they hold in their org grows. For example, they may create 'topic tags' for a specific project, so it's clearer which datasets /models are related to that project.

#### Collections

This solution would likely be the biggest shift and may require significant changes in the hub fronted. Collections could work in several different ways but would include:

Users can curate particular datasets, models, spaces, etc., into a collection. For example, they may create a collection of 'historic newspapers suitable for training language models'. These collections would not be mutually exclusive, i.e. a dataset can belong to zero, one or many collections. Collections can also potentially be nested under other collections.

This is fairly common on other data reposotiores for example the following collections:
<img width="293" alt="Screenshot 2022-07-18 at 11 50 44" src="https://user-images.githubusercontent.com/8995957/179496445-963ed122-5e26-4574-96e8-41081bce3e2b.png">

all belong under a higher level collection (https://bl.iro.bl.uk/collections/353c908d-b495-4413-b047-87236d2573e3?locale=en).

There are different models one could use for how these collections could be created:

- only within an org
- for any dataset/model
- the owner or a dataset/model has to agree to be added to a collection
- a collection owner can have people suggest additions to their collection
- other models....

These collections could be thematic, related to particular training approaches, curate models with particular inference properties etc. Whilst some of these features may duplicate current/or future tag filters on the hub, they offer the advantage of being flexible and not having to predict what users will want to do upfront.

There is also potential for automating the creation of these collections based on existing metadata. For example, one could collect models trained on a collection of datasets so for example, if we had a collection of 'historic newspapers suitable for training language models' that contained 30 datasets, we could create another collection 'historic newspaper language models' that takes any model on the hub whose metadata says it used one or more of those 30 datasets.

There is also the option of exploring ML approaches to suggest models/datasets may be relevant to a particular collection.

This approach is likely to be quite difficult to implement well and would require significant thought. There is also likely to be a benefit in doing quite a bit of upfront work in curating useful collections to demonstrate the benefits of collections.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

It is possible to collate this information externally, i.e. one could link back to the relevant models/datasets from an external platform.

**Additional context**
Add any other context about the feature request here.

I'm cc'ing others involved in the BigLAM hackathon who may also have thoughts @cakiki @clancyoftheoverflow @albertvillanova


Added more information in the README about contributors of the Arabic Speech Corpus

Added more information in the README about contributors and encouraged reading the thesis for more infos


Support extract lz4 compressed data files


Fix Authentification Error while streaming

I fixed a few errors when it occurs while streaming the private dataset on the Huggingface Hub.
```
from datasets import load_dataset

dataset = load_dataset(<repo_id>, use_auth_token=<private_token>, streaming=True)
for d in dataset['train']:
    print(d)
    break # this is for checking
```
This code is an example for streaming private datasets.
when the version of the datasets is 2.2.2, it works well but datasets>2.2.2 occurs error like this,
```
/usr/local/lib/python3.7/dist-packages/aiohttp/client_reqrep.py in raise_for_status(self)
1007 status=self.status,
1008 message=self.reason,
→ 1009 headers=self.headers,
1010 )
1011

ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/.../train-00000-of-00001-168b451062c67c34.parquet')
```
(this is an example on the dataset has `parquet` extenstion)
It seems that the `xisfile `module in `download/streaming_download_manager.py` couldn't recognize the file on "https://huggingface.co/~".

so I add three lines.
With this change, there is no error anymore(but this code is ad-hoc).


Enable streaming dataset to use the "all" split

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4698). All of your documentation changes will be reflected on that endpoint.

@albertvillanova
Adding the validation split causes these two `assert_called_once` assertions to fail with `AssertionError: Expected 'ArrowWriter' to have been called once. Called 2 times`:

https://github.com/huggingface/datasets/blob/main/tests/test_builder.py#L548-L562

It might be better to create a new dummy generator for the streaming tests, WDYT? Alternatively we could test for `self.call_count` equalling 2.

Fixes #4637


Trouble with streaming of vision dataset with TAR archive

Hi @frgfm, thanks for reporting.

As the error message says, streaming mode is not supported out of the box when the dataset contains TAR archive files.

To make the dataset streamable, you have to use `dl_manager.iter_archive`.

There are several examples in other datasets, e.g. food101: https://huggingface.co/datasets/food101/blob/main/food101.py

And yes, as the link you pointed out, for the streaming to be possible, the metadata file must be loaded before all of the images:
- either this is the case when iterating the archive (and you get the metadata file before the images)
- or you have to extract the metadata file by hand and upload it separately to the Hub

Hi @albertvillanova :wave:

Thanks! Yeah I saw that but since I didn't have any metadata, I wasn't sure whether I should create them myself.

So one last question:
What is the metadata supposed to be for archives? The relative path of all files in it?
_(Sorry I'm a bit confused since it's quite hard to debug using the single error message from the data preview :sweat_smile: )_

### Link

https://huggingface.co/datasets/frgfm/imagenette

### Description

Hello there :wave:

Thanks for the amazing work you've done with HF Datasets! I've just started playing with it, and managed to upload my first dataset. But for the second one, I'm having trouble with the preview since there is some archive extraction involved :sweat_smile:

Basically, I get a:
```
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://s3.amazonaws.com/fast-ai-imageclas/imagenette2.tgz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

I've tried several things and checked this issue https://github.com/huggingface/datasets/issues/4181 as well, but no luck so far!

Could you point me in the right direction please? :pray:

### Owner

Yes


Cannot load LinCE dataset

bug

Something isn't working

Hi @finiteautomata, thanks for reporting.

Unfortunately, I'm not able to reproduce your issue:
```python
In [1]: from datasets import load_dataset
   ...: dataset = load_dataset("lince", "ner_spaeng")
Downloading builder script: 20.8kB [00:00, 9.09MB/s]
Downloading metadata: 31.2kB [00:00, 13.5MB/s]
Downloading and preparing dataset lince/ner_spaeng (download: 2.93 MiB, generated: 18.45 MiB, post-processed: Unknown size, total: 21.38 MiB) to .../.cache/huggingface/datasets/lince/ner_spaeng/1.0.0/10d41747f55f0849fa84ac579ea1acfa7df49aa2015b60426bc459c111b3d589...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.08M/3.08M [00:01<00:00, 2.73MB/s]
Dataset lince downloaded and prepared to .../.cache/huggingface/datasets/lince/ner_spaeng/1.0.0/10d41747f55f0849fa84ac579ea1acfa7df49aa2015b60426bc459c111b3d589. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 630.66it/s]

In [2]: dataset
Out[2]:
DatasetDict({
    train: Dataset({
        features: ['idx', 'words', 'lid', 'ner'],
        num_rows: 33611
    })
    validation: Dataset({
        features: ['idx', 'words', 'lid', 'ner'],
        num_rows: 10085
    })
    test: Dataset({
        features: ['idx', 'words', 'lid', 'ner'],
        num_rows: 23527
    })
})
```

Please note that for this dataset, the original data files are not hosted on the Hugging Face Hub, but on https://ritual.uh.edu
And sometimes, the server might be temporarily unavailable, as your error message said (trying to connect to the server timed out):
```
ConnectionError: Couldn't reach https://ritual.uh.edu/lince/libaccess/eyJ1c2VybmFtZSI6ICJodWdnaW5nZmFjZSBubHAiLCAidXNlcl9pZCI6IDExMSwgImVtYWlsIjogImR1bW15QGVtYWlsLmNvbSJ9/ner_spaeng.zip (ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='ritual.uh.edu', port=443): Max retries exceeded with url: /lince/libaccess/eyJ1c2VybmFtZSI6ICJodWdnaW5nZmFjZSBubHAiLCAidXNlcl9pZCI6IDExMSwgImVtYWlsIjogImR1bW15QGVtYWlsLmNvbSJ9/ner_spaeng.zip (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7feb1c45a690>, 'Connection to ritual.uh.edu timed out. (connect timeout=100)'))")))
```
In these cases you could:
- either contact the owners of the data server where the data is hosted to inform them about the issue in their server
- or re-try after waiting some time: usually these issues are just temporary

## Describe the bug

Cannot load LinCE dataset due to a connection error

## Steps to reproduce the bug
```python

from datasets import load_dataset

dataset = load_dataset("lince", "ner_spaeng")
```

A notebook with this code and corresponding error can be found at https://colab.research.google.com/drive/1pgX3bNB9amuUwAVfPFm-XuMV5fEg-cD2

## Expected results

It should load the dataset

## Actual results

```python
---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-2-fc551ddcebef> in <module>()
      1 from datasets import load_dataset
      2
----> 3 dataset = load_dataset("lince", "ner_spaeng")

10 frames
/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1682         ignore_verifications=ignore_verifications,
   1683         try_from_hf_gcs=try_from_hf_gcs,
-> 1684         use_auth_token=use_auth_token,
   1685     )
   1686

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    703                     if not downloaded_from_gcs:
    704                         self._download_and_prepare(
--> 705                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706                         )
    707                     # Sync info

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)
   1219
   1220     def _download_and_prepare(self, dl_manager, verify_infos):
-> 1221         super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
   1222
   1223     def _get_examples_iterable_for_split(self, split_generator: SplitGenerator) -> ExamplesIterable:

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    769         split_dict = SplitDict(dataset_name=self.name)
    770         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 771         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    772
    773         # Checksums verification

/root/.cache/huggingface/modules/datasets_modules/datasets/lince/10d41747f55f0849fa84ac579ea1acfa7df49aa2015b60426bc459c111b3d589/lince.py in _split_generators(self, dl_manager)
    481     def _split_generators(self, dl_manager):
    482         """Returns SplitGenerators."""
--> 483         lince_dir = dl_manager.download_and_extract(f"{_LINCE_URL}/{self.config.name}.zip")
    484         data_dir = os.path.join(lince_dir, self.config.data_dir)
    485         return [

/usr/local/lib/python3.7/dist-packages/datasets/download/download_manager.py in download_and_extract(self, url_or_urls)
    429             extracted_path(s): `str`, extracted paths of given URL(s).
    430         """
--> 431         return self.extract(self.download(url_or_urls))
    432
    433     def get_recorded_sizes_checksums(self):

/usr/local/lib/python3.7/dist-packages/datasets/download/download_manager.py in download(self, url_or_urls)
    313             num_proc=download_config.num_proc,
    314             disable_tqdm=not is_progress_bar_enabled(),
--> 315             desc="Downloading data files",
    316         )
    317         duration = datetime.now() - start_time

/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types, disable_tqdm, desc)
    346     # Singleton
    347     if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 348         return function(data_struct)
    349
    350     disable_tqdm = disable_tqdm or not logging.is_progress_bar_enabled()

/usr/local/lib/python3.7/dist-packages/datasets/download/download_manager.py in _download(self, url_or_filename, download_config)
    333             # append the relative path to the base_path
    334             url_or_filename = url_or_path_join(self._base_path, url_or_filename)
--> 335         return cached_path(url_or_filename, download_config=download_config)
    336
    337     def iter_archive(self, path_or_buf: Union[str, io.BufferedReader]):

/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    195             use_auth_token=download_config.use_auth_token,
    196             ignore_url_params=download_config.ignore_url_params,
--> 197             download_desc=download_config.download_desc,
    198         )
    199     elif os.path.exists(url_or_filename):

/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token, ignore_url_params, download_desc)
    531         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
    532         if head_error is not None:
--> 533             raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
    534         elif response is not None:
    535             raise ConnectionError(f"Couldn't reach {url} (error {response.status_code})")

ConnectionError: Couldn't reach https://ritual.uh.edu/lince/libaccess/eyJ1c2VybmFtZSI6ICJodWdnaW5nZmFjZSBubHAiLCAidXNlcl9pZCI6IDExMSwgImVtYWlsIjogImR1bW15QGVtYWlsLmNvbSJ9/ner_spaeng.zip (ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='ritual.uh.edu', port=443): Max retries exceeded with url: /lince/libaccess/eyJ1c2VybmFtZSI6ICJodWdnaW5nZmFjZSBubHAiLCAidXNlcl9pZCI6IDExMSwgImVtYWlsIjogImR1bW15QGVtYWlsLmNvbSJ9/ner_spaeng.zip (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7feb1c45a690>, 'Connection to ritual.uh.edu timed out. (connect timeout=100)'))")))
```

## Environment info

- `datasets` version: 2.3.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5


Add MANtIS dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4695). All of your documentation changes will be reflected on that endpoint.

This PR adds MANtIS dataset.
Arxiv: [https://arxiv.org/abs/1912.04639](https://arxiv.org/abs/1912.04639)
Github: [https://github.com/Guzpenha/MANtIS](https://github.com/Guzpenha/MANtIS)

README and dataset tags are WIP.


Distributed data parallel training for streaming datasets

enhancement

New feature or request

### Feature request

Any documentations for the the `load_dataset(streaming=True)` for (multi-node multi-GPU) DDP training?

### Motivation

Given a bunch of data files, it is expected to split them onto different GPUs. Is there a guide or documentation?

### Your contribution

Does it requires manually split on data files for each worker in `DatasetBuilder._split_generator()`? What is`IterableDatasetShard` expected to do?


update `samsum` script

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4693). All of your documentation changes will be reflected on that endpoint.

update `samsum` script after #4672 was merged (citation is also updated)


Unable to cast a column with `Image()` by using the `cast_column()` feature

bug

Something isn't working

Hi, thanks for reporting! A PR (https://github.com/huggingface/datasets/pull/4614) has already been opened to address this issue.

## Describe the bug
A clear and concise description of what the bug is.

When I create a dataset, then add a column to the created dataset through the `dataset.add_column` feature and then try to cast a column of the dataset (this column contains image paths) with `Image()` by using the `cast_column()` feature, I get the following error - ``` TypeError: Couldn't cast array of type
string
to
{'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='string', id=None)} ```

When I try and cast the same column, but without doing the `add_column` in the previous step, it works as expected.

## Steps to reproduce the bug
```python
from datasets import Dataset, Image

data_dict = {
    "img_path": ["https://picsum.photos/200/300"]
}

dataset = Dataset.from_dict(data_dict)

#NOTE Comment out this line and use cast_column and it works properly
dataset = dataset.add_column("yeet", [1])

#NOTE This line fails to execute properly if `add_column` is called before
dataset = dataset.cast_column("img_path", Image())

# #NOTE This is my current workaround. This seems to work fine with/without `add_column`. While
# # running this, make sure to comment out the `cast_column` line
# new_features = dataset.features.copy()
# new_features["img_path"] = Image()
# dataset = dataset.cast(new_features)

print(dataset)
print(dataset.features)
print(dataset[0])
```

## Expected results
A clear and concise description of the expected results.

Able to successfully use `cast_column` to cast a column containing img_paths to now be Image() features after modifying the dataset using `add_column` in a previous step

## Actual results
Specify the actual results or traceback.

```
Traceback (most recent call last):
  File "/home/surya/Desktop/hf_bug_test.py", line 14, in <module>
    dataset = dataset.cast_column("img_path", Image())
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1580, in cast_column
    dataset._data = dataset._data.cast(dataset.features.arrow_schema)
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1487, in cast
    new_tables.append(subtable.cast(subschema, *args, **kwargs))
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 834, in cast
    return InMemoryTable(table_cast(self.table, *args, **kwargs))
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1897, in table_cast
    return cast_table_to_schema(table, schema)
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1880, in cast_table_to_schema
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1880, in <listcomp>
    arrays = [cast_array_to_feature(table[name], feature) for name, feature in features.items()]
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1673, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1673, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/surya/anaconda3/envs/snap_test/lib/python3.9/site-packages/datasets/table.py", line 1846, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
string
to
{'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='string', id=None)}
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Ubuntu 20.04.3 LTS
- Python version: 3.9.7
- PyArrow version: 7.0.0