Something isn't working

I agree packaged loaders should support `ClassLabel` feature without throwing an error.

@albertvillanova @mariosasko thank you, with that change now I get
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-9-eeb68eeb9bec>](https://localhost:8080/#) in <module>()
     11 )
     12 # You can make this part faster with num_proc=<some int>
---> 13 sentences = sentences.map(lambda ex: features["label"].str2int(ex["label"]) if ex["label"] is not None else None, features=features)
     14 sentences = sentences.shuffle()

8 frames
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in validate_function_output(processed_inputs, indices)
   2193             if processed_inputs is not None and not isinstance(processed_inputs, (Mapping, pa.Table)):
   2194                 raise TypeError(
-> 2195                     f"Provided `function` which is applied to all elements of table returns a variable of type {type(processed_inputs)}. Make sure provided `function` returns a variable of type `dict` (or a pyarrow table) to update the dataset or `None` if you are only interested in side effects."
   2196                 )
   2197             elif isinstance(indices, list) and isinstance(processed_inputs, Mapping):

TypeError: Provided `function` which is applied to all elements of table returns a variable of type <class 'int'>. Make sure provided `function` returns a variable of type `dict` (or a pyarrow table) to update the dataset or `None` if you are only interested in side effects.
```

the error is raised by [this](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L2221)

```
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in validate_function_output(processed_inputs, indices)
```

@mariosasko changed it like

```python
sentences = sentences.map(lambda ex: {"label" : features["label"].str2int(ex["label"]) if ex["label"] is not None else None}, features=features)
```

to avoid the above errorr.

Any update on this? Is this correct ?
> @mariosasko changed it like
>
> ```python
> sentences = sentences.map(lambda ex: {"label" : features["label"].str2int(ex["label"]) if ex["label"] is not None else None}, features=features)
> ```
>
> to avoid the above errorr.

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.10.0+cu111 (True)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from datasets import load_dataset,Features,Value,ClassLabel

class_names = ["cmn","deu","rus","fra","eng","jpn","spa","ita","kor","vie","nld","epo","por","tur","heb","hun","ell","ind","ara","arz","fin","bul","yue","swe","ukr","bel","que","ces","swh","nno","wuu","nob","zsm","est","kat","pol","lat","urd","sqi","isl","fry","afr","ron","fao","san","bre","tat","yid","uig","uzb","srp","qya","dan","pes","slk","eus","cycl","acm","tgl","lvs","kaz","hye","hin","lit","ben","cat","bos","hrv","tha","orv","cha","mon","lzh","scn","gle","mkd","slv","frm","glg","vol","ain","jbo","tok","ina","nds","mal","tlh","roh","ltz","oss","ido","gla","mlt","sco","ast","jav","oci","ile","ota","xal","tel","sjn","nov","khm","tpi","ang","aze","tgk","tuk","chv","hsb","dsb","bod","sme","cym","mri","ksh","kmr","ewe","kab","ber","tpw","udm","lld","pms","lad","grn","mlg","xho","pnb","grc","hat","lao","npi","cor","nah","avk","mar","guj","pan","kir","myv","prg","sux","crs","ckt","bak","zlm","hil","cbk","chr","nav","lkt","enm","arq","lin","abk","pcd","rom","gsw","tam","zul","awa","wln","amh","bar","hbo","mhr","bho","mrj","ckb","osx","pfl","mgm","sna","mah","hau","kan","nog","sin","glv","dng","kal","liv","vro","apc","jdt","fur","che","haw","yor","crh","pdc","ppl","kin","shs","mnw","tet","sah","kum","ngt","nya","pus","hif","mya","moh","wol","tir","ton","lzz","oar","lug","brx","non","mww","hak","nlv","ngu","bua","aym","vec","ibo","tkl","bam","kha","ceb","lou","fuc","smo","gag","lfn","arg","umb","tyv","kjh","oji","cyo","urh","kzj","pam","srd","lmo","swg","mdf","gil","snd","tso","sot","zza","tsn","pau","som","egl","ady","asm","ori","dtp","cho","max","kam","niu","sag","ilo","kaa","fuv","nch","hoc","iba","gbm","sun","war","mvv","pap","ary","kxi","csb","pag","cos","rif","kek","krc","aii","ban","ssw","tvl","mfe","tah","bvy","bcl","hnj","nau","nst","afb","quc","min","tmw","mad","bjn","mai","cjy","got","hsn","gan","tzl","dws","ldn","afh","sgs","krl","vep","rue","tly","mic","ext","izh","sma","jam","cmo","mwl","kpv","koi","bis","ike","run","evn","ryu","mnc","aoz","otk","kas","aln","akl","yua","shy","fkv","gos","fij","thv","zgh","gcf","cay","xmf","tig","div","lij","rap","hrx","cpi","tts","gaa","tmr","iii","ltg","bzt","syc","emx","gom","chg","osp","stq","frr","fro","nys","toi","new","phn","jpa","rel","drt","chn","pli","laa","bal","hdn","hax","mik","ajp","xqa","pal","crk","mni","lut","ayl","ood","sdh","ofs","nus","kiu","diq","qxq","alt","bfz","klj","mus","srn","guc","lim","zea","shi","mnr","bom","sat","szl"]
features = Features({ 'label': ClassLabel(names=class_names), 'text': Value('string')})
num_labels = features['label'].num_classes
data_files = { "train": "train.csv", "test": "test.csv" }
sentences = load_dataset("loretoparisi/tatoeba-sentences",
                             data_files=data_files,
                             delimiter='\t',
                             column_names=['label', 'text'],
                             features = features
```

ERROR:
```
ClassLabel(num_classes=403, names=['cmn', 'deu', 'rus', 'fra', 'eng', 'jpn', 'spa', 'ita', 'kor', 'vie', 'nld', 'epo', 'por', 'tur', 'heb', 'hun', 'ell', 'ind', 'ara', 'arz', 'fin', 'bul', 'yue', 'swe', 'ukr', 'bel', 'que', 'ces', 'swh', 'nno', 'wuu', 'nob', 'zsm', 'est', 'kat', 'pol', 'lat', 'urd', 'sqi', 'isl', 'fry', 'afr', 'ron', 'fao', 'san', 'bre', 'tat', 'yid', 'uig', 'uzb', 'srp', 'qya', 'dan', 'pes', 'slk', 'eus', 'cycl', 'acm', 'tgl', 'lvs', 'kaz', 'hye', 'hin', 'lit', 'ben', 'cat', 'bos', 'hrv', 'tha', 'orv', 'cha', 'mon', 'lzh', 'scn', 'gle', 'mkd', 'slv', 'frm', 'glg', 'vol', 'ain', 'jbo', 'tok', 'ina', 'nds', 'mal', 'tlh', 'roh', 'ltz', 'oss', 'ido', 'gla', 'mlt', 'sco', 'ast', 'jav', 'oci', 'ile', 'ota', 'xal', 'tel', 'sjn', 'nov', 'khm', 'tpi', 'ang', 'aze', 'tgk', 'tuk', 'chv', 'hsb', 'dsb', 'bod', 'sme', 'cym', 'mri', 'ksh', 'kmr', 'ewe', 'kab', 'ber', 'tpw', 'udm', 'lld', 'pms', 'lad', 'grn', 'mlg', 'xho', 'pnb', 'grc', 'hat', 'lao', 'npi', 'cor', 'nah', 'avk', 'mar', 'guj', 'pan', 'kir', 'myv', 'prg', 'sux', 'crs', 'ckt', 'bak', 'zlm', 'hil', 'cbk', 'chr', 'nav', 'lkt', 'enm', 'arq', 'lin', 'abk', 'pcd', 'rom', 'gsw', 'tam', 'zul', 'awa', 'wln', 'amh', 'bar', 'hbo', 'mhr', 'bho', 'mrj', 'ckb', 'osx', 'pfl', 'mgm', 'sna', 'mah', 'hau', 'kan', 'nog', 'sin', 'glv', 'dng', 'kal', 'liv', 'vro', 'apc', 'jdt', 'fur', 'che', 'haw', 'yor', 'crh', 'pdc', 'ppl', 'kin', 'shs', 'mnw', 'tet', 'sah', 'kum', 'ngt', 'nya', 'pus', 'hif', 'mya', 'moh', 'wol', 'tir', 'ton', 'lzz', 'oar', 'lug', 'brx', 'non', 'mww', 'hak', 'nlv', 'ngu', 'bua', 'aym', 'vec', 'ibo', 'tkl', 'bam', 'kha', 'ceb', 'lou', 'fuc', 'smo', 'gag', 'lfn', 'arg', 'umb', 'tyv', 'kjh', 'oji', 'cyo', 'urh', 'kzj', 'pam', 'srd', 'lmo', 'swg', 'mdf', 'gil', 'snd', 'tso', 'sot', 'zza', 'tsn', 'pau', 'som', 'egl', 'ady', 'asm', 'ori', 'dtp', 'cho', 'max', 'kam', 'niu', 'sag', 'ilo', 'kaa', 'fuv', 'nch', 'hoc', 'iba', 'gbm', 'sun', 'war', 'mvv', 'pap', 'ary', 'kxi', 'csb', 'pag', 'cos', 'rif', 'kek', 'krc', 'aii', 'ban', 'ssw', 'tvl', 'mfe', 'tah', 'bvy', 'bcl', 'hnj', 'nau', 'nst', 'afb', 'quc', 'min', 'tmw', 'mad', 'bjn', 'mai', 'cjy', 'got', 'hsn', 'gan', 'tzl', 'dws', 'ldn', 'afh', 'sgs', 'krl', 'vep', 'rue', 'tly', 'mic', 'ext', 'izh', 'sma', 'jam', 'cmo', 'mwl', 'kpv', 'koi', 'bis', 'ike', 'run', 'evn', 'ryu', 'mnc', 'aoz', 'otk', 'kas', 'aln', 'akl', 'yua', 'shy', 'fkv', 'gos', 'fij', 'thv', 'zgh', 'gcf', 'cay', 'xmf', 'tig', 'div', 'lij', 'rap', 'hrx', 'cpi', 'tts', 'gaa', 'tmr', 'iii', 'ltg', 'bzt', 'syc', 'emx', 'gom', 'chg', 'osp', 'stq', 'frr', 'fro', 'nys', 'toi', 'new', 'phn', 'jpa', 'rel', 'drt', 'chn', 'pli', 'laa', 'bal', 'hdn', 'hax', 'mik', 'ajp', 'xqa', 'pal', 'crk', 'mni', 'lut', 'ayl', 'ood', 'sdh', 'ofs', 'nus', 'kiu', 'diq', 'qxq', 'alt', 'bfz', 'klj', 'mus', 'srn', 'guc', 'lim', 'zea', 'shi', 'mnr', 'bom', 'sat', 'szl'], id=None)
Value(dtype='string', id=None)
Using custom data configuration loretoparisi--tatoeba-sentences-7b2c5e991f398f39
Downloading and preparing dataset csv/loretoparisi--tatoeba-sentences to /root/.cache/huggingface/datasets/csv/loretoparisi--tatoeba-sentences-7b2c5e991f398f39/0.0.0/433e0ccc46f9880962cc2b12065189766fbb2bee57a221866138fb9203c83519...
Downloading data files: 100%
2/2 [00:18<00:00, 8.06s/it]
Downloading data: 100%
391M/391M [00:13<00:00, 35.3MB/s]
Downloading data: 100%
92.4M/92.4M [00:02<00:00, 36.5MB/s]
Failed to read file '/root/.cache/huggingface/datasets/downloads/933132df9905194ea9faeb30cabca8c49318795612f6495fcb941a290191dd5d' with error <class 'ValueError'>: invalid literal for int() with base 10: 'cmn'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_tokens()

TypeError: Cannot cast array data from dtype('O') to dtype('int64') according to the rule 'safe'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
15 frames
/usr/local/lib/python3.7/dist-packages/pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_tokens()

ValueError: invalid literal for int() with base 10: 'cmn'
```

while loading without `features` it loads without errors

```
sentences = load_dataset("loretoparisi/tatoeba-sentences",
                             data_files=data_files,
                             delimiter='\t',
                             column_names=['label', 'text']
                         )
```

but the `label` col seems to be wrong (without the `ClassLabel` object):

```
sentences['train'].features
{'label': Value(dtype='string', id=None),
 'text': Value(dtype='string', id=None)}
```

The dataset was https://huggingface.co/datasets/loretoparisi/tatoeba-sentences

Dataset format is:

```
ces	Nechci vědět, co je tam uvnitř.
ces	Kdo o tom chce slyšet?
deu	Tom sagte, er fühle sich nicht wohl.
ber	Mel-iyi-d anida-t tura ?
hun	Gondom lesz rá rögtön.
ber	Mel-iyi-d anida-tt tura ?
deu	Ich will dich nicht reden hören.
```

### Expected behavior

```shell
correctly load train and test files.
```


Add CMU MoCap Dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4208). All of your documentation changes will be reflected on that endpoint.

- Updated the readme.
- Added dummy_data.zip and ran the all the tests.

The dataset works for "asf/amc" and "avi" formats which have a single download link for the complete dataset. But "c3d" and "mpg" have multiple download links, can we combine and host these links on the Hub since the dataset is free to use ?

Resolves #3457

Dataset Request : Add CMU Graphics Lab Motion Capture dataset [#3457](https://github.com/huggingface/datasets/issues/3457)
This PR adds the CMU MoCap Dataset.

The authors didn't respond even after multiple follow ups, so I ended up crawling the website to get categories, subcategories and description information. Some of the subjects do not have category/subcategory/description as well. I am using a subject to categories, subcategories and description map (metadata file).

Currently the loading of the dataset works for "asf/amc" and "avi" formats since they have a single download link. But "c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?
Any suggestions/inputs on this would be helpful. Thank you.


[Minor edit] Fix typo in class name

Typo: `datasets.DatsetDict` -> `datasets.DatasetDict`


Add Nerval Metric

This PR adds readme.md and ner_val.py to metrics.
Nerval is a python package that helps evaluate NER models. It creates classification report and confusion matrix at entity level.


Fix `convert_file_size_to_int` for kilobits and megabits

_The documentation is not available anymore as the PR was closed or merged._

Minor change to fully align this function with the recent change in Transformers (https://github.com/huggingface/transformers/pull/16891)


Add Recall Metric Card

What this PR mainly does:
- add metric card for recall metric
- update docs in recall python file

Note: I've also included a .json file with all of the metric card information. I've started compiling the relevant information in this type of .json files, and then using a script I wrote to generate the formatted metric card, as well as the docs to go in the .py file. I figured I'd upload the .json because it could be useful, especially if I also make a PR with the script I'm using (let me know if that's something you think would be beneficial!)


Add Precision Metric Card

What this PR mainly does:
- add metric card for precision metric
- update docs in precision python file

Note: I've also included a .json file with all of the metric card information. I've started compiling the relevant information in this type of .json files, and then using a script I wrote to generate the formatted metric card, as well as the docs to go in the .py file. I figured I'd upload the .json because it could be useful, especially if I also make a PR with the script I'm using (let me know if that's something you think would be beneficial!)


Fix some type annotation in doc


Update GH template for dataset viewer issues

You can see rendering at: https://github.com/huggingface/datasets/blob/6b48fedbdafe12a42c7b6edcecc32820af1a4822/.github/ISSUE_TEMPLATE/dataset-viewer.yml

Update template to use new issue forms instead.

With this PR we can check if this new feature is useful for us.

Once validated, we can update the other templates.

CC: @severo


Add to docs how to load from local script

This option was missing from the docs guide (it was only explained in the docstring of `load_dataset`). Although this is an infrequent use case, there might be some users interested in it.

Related to #4192

CC: @stevhliu


Cache miss during reload for datasets using image fetch utilities through map

bug

Something isn't working

Hi ! Maybe one of the objects in the function is not deterministic across sessions ? You can read more about it and how to investigate here: https://huggingface.co/docs/datasets/about_cache

Hi @apsdehal! Can you verify that replacing
```python
def fetch_single_image(image_url, timeout=None, retries=0):
    for _ in range(retries + 1):
        try:
            request = urllib.request.Request(
                image_url,
                data=None,
                headers={"user-agent": get_datasets_user_agent()},
            )
            with urllib.request.urlopen(request, timeout=timeout) as req:
                image = PIL.Image.open(io.BytesIO(req.read()))
            break
        except Exception:
            image = None
    return image
```
with
```python
USER_AGENT = get_datasets_user_agent()

def fetch_single_image(image_url, timeout=None, retries=0):
    for _ in range(retries + 1):
        try:
            request = urllib.request.Request(
                image_url,
                data=None,
                headers={"user-agent": USER_AGENT},
            )
            with urllib.request.urlopen(request, timeout=timeout) as req:
                image = PIL.Image.open(io.BytesIO(req.read()))
            break
        except Exception:
            image = None
    return image
```
fixes the issue?

Thanks @mariosasko. That does fix the issue. In general, I think these image downloading utilities since they are being used by a lot of image dataset should be provided as a part of `datasets` library right to keep the logic consistent and READMEs smaller? If they already exists, that is also great, please point me to those. I saw that `http_get` does exist.

You can find my rationale (and a proposed solution) for why these utilities are not a part of `datasets` here: https://github.com/huggingface/datasets/pull/4100#issuecomment-1097994003.

Makes sense. But, I think as the number of image datasets as grow, more people are copying pasting original code from docs to work as it is while we make fixes to them later. I think we do need a central place for these to avoid that confusion as well as more easier access to image datasets. Should we restart that discussion, possible on slack?

## Describe the bug

It looks like that result of `.map` operation dataset are missing the cache when you reload the script and always run from scratch. In same interpretor session, they are able to find the cache and reload it. But, when you exit the interpretor and reload it, the downloading starts from scratch.

## Steps to reproduce the bug

Using the example provided in `red_caps` dataset.
```python
from concurrent.futures import ThreadPoolExecutor
from functools import partial
import io
import urllib

import PIL.Image

import datasets
from datasets import load_dataset
from datasets.utils.file_utils import get_datasets_user_agent

def fetch_single_image(image_url, timeout=None, retries=0):
    for _ in range(retries + 1):
        try:
            request = urllib.request.Request(
                image_url,
                data=None,
                headers={"user-agent": get_datasets_user_agent()},
            )
            with urllib.request.urlopen(request, timeout=timeout) as req:
                image = PIL.Image.open(io.BytesIO(req.read()))
            break
        except Exception:
            image = None
    return image

def fetch_images(batch, num_threads, timeout=None, retries=0):
    fetch_single_image_with_args = partial(fetch_single_image, timeout=timeout, retries=retries)
    with ThreadPoolExecutor(max_workers=num_threads) as executor:
        batch["image"] = list(executor.map(lambda image_urls: [fetch_single_image_with_args(image_url) for image_url in image_urls], batch["image_url"]))
    return batch

def process_image_urls(batch):
    processed_batch_image_urls = []
    for image_url in batch["image_url"]:
        processed_example_image_urls = []
        image_url_splits = re.findall(r"http\S+", image_url)
        for image_url_split in image_url_splits:
            if "imgur" in image_url_split and "," in image_url_split:
                for image_url_part in image_url_split.split(","):
                    if not image_url_part:
                        continue
                    image_url_part = image_url_part.strip()
                    root, ext = os.path.splitext(image_url_part)
                    if not root.startswith("http"):
                      root = "http://i.imgur.com/" + root
                    root = root.split("#")[0]
                    if not ext:
                      ext = ".jpg"
                    ext = re.split(r"[?%]", ext)[0]
                    image_url_part = root + ext
                    processed_example_image_urls.append(image_url_part)
            else:
                processed_example_image_urls.append(image_url_split)
        processed_batch_image_urls.append(processed_example_image_urls)
    batch["image_url"] = processed_batch_image_urls
    return batch

dset = load_dataset("red_caps", "jellyfish")
dset = dset.map(process_image_urls, batched=True, num_proc=4)
features = dset["train"].features.copy()
features["image"] = datasets.Sequence(datasets.Image())
num_threads = 5
dset = dset.map(fetch_images, batched=True, batch_size=50, features=features, fn_kwargs={"num_threads": num_threads})
```

Run this in an interpretor or as a script twice and see that the cache is missed the second time.

## Expected results
At reload there should not be any cache miss

## Actual results
Every time script is run, cache is missed and dataset is built from scratch.

## Environment info
- `datasets` version: 2.1.1.dev0
- Platform: Linux-4.19.0-20-cloud-amd64-x86_64-with-glibc2.10
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


There is no dataset

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Add remove_columns=True

Any reason why we can't just do `[inputs.copy()]` in this line for in-place operations to not have effects anymore:
https://github.com/huggingface/datasets/blob/bf432011ff9155a5bc16c03956bc63e514baf80d/src/datasets/arrow_dataset.py#L2232.

(in the `batched` case, we can also copy the inputs' values (list objects) to ignore in-place modifications to the inputs' columns)

I think `remove_columns=True` has no meaning, so I'm not a fan of this change.

@mariosasko copy does have a cost associated with it ... and plus you'll have to consider `deepcopy` Imagine columnds that are list of list of list of list .... Though I have to agree that `remove_columns=True` doesn't make sense (but, IMO, neither does it in its current use-case as it should refer to `input_columns`)

Okay closing this PR for the following reasons:
 - `remove_columns=True` was expected to keep the `.update`-like operator for `.map`. I initially thought it would be a good way to ignore function side effects and only keep output of that function (cf. PR description).
 - expected `remove_columns=True` is a bad API according to @mariosasko and introduces unecessary changes for little gain (strictly equivalent to `remove_columns=dset.column_names`)

This should fix all the issue we have with in place operations in mapping functions. This is crucial as where we do some weird things like:
```
def apply(batch):
    batch_size = len(batch["id"])
    batch["text"] = ["potato" for _ range(batch_size)]
    return {}

# Columns are: {"id": int}
dset.map(apply, batched=True, remove_columns="text") # crashes because `text` is not in the original columns
dset.map(apply, batched=True) # mapped datasets has `text` column
```

In this PR we suggest to have `remove_columns=True` so that we ignore the input completely, and just use the output to generate mapped dataset. This means that inplace operations won't have any effects anymore.


Embed image and audio files in `save_to_disk`

Following https://github.com/huggingface/datasets/pull/4184, currently a dataset saved using `save_to_disk` doesn't actually contain the bytes of the image or audio files. Instead it stores the path to your local files.

Adding `embed_external_files` and set it to True by default to save_to_disk would be kind of a breaking change since some users will get bigger Arrow files when updating the lib, but the advantages are nice:

- the resulting dataset is self contained, in case you want to delete your cache for example or share it with someone else
- users also upload these Arrow files to cloud storage via the fs parameter, and in this case they would expect to upload a self-contained dataset
- consistency with push_to_hub

This can be implemented at the same time as sharding for `save_to_disk` for efficiency, and reuse the helpers from `push_to_hub` to embed the external files.

cc @mariosasko


Support lists of multi-dimensional numpy arrays

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4194). All of your documentation changes will be reflected on that endpoint.

Fix #4191.

CC: @SaulLu


Document save_to_disk and push_to_hub on images and audio files

Following https://github.com/huggingface/datasets/pull/4187, I explained in the documentation of `save_to_disk` and `push_to_hub` how they handle image and audio data.


load_dataset can't load local dataset,Unable to find ...

Something isn't working

Hi! :)

I believe that should work unless `dataset_infos.json` isn't actually a dataset. For Hugging Face datasets, there is usually a file named `dataset_infos.json` which contains metadata about the dataset (eg. the dataset citation, license, description, etc). Can you double-check that `dataset_infos.json` isn't just metadata please?

Hi @ahf876828330,

As @stevhliu pointed out, the proper way to load a dataset is not trying to load its metadata file.

In your case, as the dataset script is local, you should better point to your local loading script:
```python
dataset = load_dataset("dataset/opus_books.py")
```

Please, feel free to re-open this issue if the previous code snippet does not work for you.

> Hi! :)
>
> I believe that should work unless `dataset_infos.json` isn't actually a dataset. For Hugging Face datasets, there is usually a file named `dataset_infos.json` which contains metadata about the dataset (eg. the dataset citation, license, description, etc). Can you double-check that `dataset_infos.json` isn't just metadata please?

Yes，you are right!So if I have a metadata dataset local,How can I turn it to a dataset that can be used by the load_dataset() function？Are there some examples?

The metadata file isn't a dataset so you can't turn it into one. You should try @albertvillanova's code snippet above (now merged in the docs [here](https://huggingface.co/docs/datasets/master/en/loading#local-loading-script)), which uses your local loading script `opus_books.py` to:

1. Download the actual dataset.
2. Once the dataset is downloaded, `load_dataset` will load it for you.

Traceback (most recent call last):
  File "/home/gs603/ahf/pretrained/model.py", line 48, in <module>
    dataset = load_dataset("json",data_files="dataset/dataset_infos.json")
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 1675, in load_dataset
    **config_kwargs,
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 1496, in load_dataset_builder
    data_files=data_files,
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 1155, in dataset_module_factory
    download_mode=download_mode,
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 800, in get_module
    data_files = DataFilesDict.from_local_or_remote(patterns, use_auth_token=self.downnload_config.use_auth_token)
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 582, in from_local_or_remote
    if not isinstance(patterns_for_key, DataFilesList)
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 544, in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 194, in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 144, in _resolve_single_pattern_locally
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to find '/home/gs603/ahf/pretrained/dataset/dataset_infos.json' at /home/gs603/ahf/pretrained

![image](https://user-images.githubusercontent.com/33253979/164413285-84ea65ac-9126-408f-9cd2-ce4751a5dd73.png)
![image](https://user-images.githubusercontent.com/33253979/164413338-4735142f-408b-41d9-ab87-8484de2be54f.png)

the code is in the model.py,why I can't use the load_dataset function to load my local dataset?


feat: create an `Array3D` column from a list of arrays of dimension 2

enhancement

New feature or request

Hi @SaulLu, thanks for your proposal.

Just I got a bit confused about the dimensions...
- For the 2D case, you mention it is possible to create an `Array2D` from a list of arrays of dimension 1
- However, you give an example of creating an `Array2D` from arrays of dimension 2:
  - the values of `data_map` are arrays of dimension 2
  - the outer list in `prepare_dataset_2D` should not be taken into account in the dimension counting, as it is used because in `map` you pass `batched=True`

Note that for the 3D alternatives you mention:
- In `prepare_dataset_3D_ter`, you create an `Array3D` from arrays of dimension 3:
  - the array `data_map[index][np.newaxis, :, :]` has dimension 3
  - the outer list in `prepare_dataset_3D_ter` is the one used by `batched=True`
- In `prepare_dataset_3D_bis`, you create an `Array3D` from a list of list of lists:
  - the value of `data_map[index].tolist()` is a list of lists
  - it is enclosed by another list `[data_map[index].tolist()]`, thus giving a list of list of lists
  - the outer list is the one used by `batched=True`

Therefore, if I understand correctly, your request would be to be able to create an `Array3D` from a list of an array of dimension 2:
- In `prepare_dataset_3D`, `data_map[index]` is an array of dimension 2
- it is enclosed by a list `[data_map[index]]`, thus giving a list of an array of dimension 2
- the outer list is the one used by `batched=True`

Please, feel free to tell me if I did not understand you correctly.

Hi @albertvillanova ,

Indeed my message was confusing and you guessed right :smile: : I think would be interesting to be able to create an Array3D from a list of an array of dimension 2.

For the 2D case I should have given as a "similar" example:
```python

data_map_1D = {
    1: np.array([0.2, 0.4]),
    2: np.array([0.1, 0.4]),
}

def prepare_dataset_2D(batch):
    batch["pixel_values"] = [[data_map_1D[index]] for index in batch["id"]]
    return batch

ds_2D = ds.map(
    prepare_dataset_2D,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array2D(shape=(1, 2), dtype="float32")})
)
```

**Is your feature request related to a problem? Please describe.**

It is possible to create an `Array2D` column from a list of arrays of dimension 1. Similarly, I think it might be nice to be able to create a `Array3D` column from a list of lists of arrays of dimension 1.

To illustrate my proposal, let's take the following toy dataset t:
```python
import numpy as np
from datasets import Dataset, features

data_map = {
    1: np.array([[0.2, 0,4],[0.19, 0,3]]),
    2: np.array([[0.1, 0,4],[0.19, 0,3]]),
}

def create_toy_ds():
    my_dict = {"id":[1, 2]}
    return Dataset.from_dict(my_dict)

ds = create_toy_ds()
```

The following 2D processing works without any errors raised:
```python
def prepare_dataset_2D(batch):
    batch["pixel_values"] = [data_map[index] for index in batch["id"]]
    return batch

ds_2D = ds.map(
    prepare_dataset_2D,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array2D(shape=(2, 3), dtype="float32")})
)
```

The following 3D processing doesn't work:
```python
def prepare_dataset_3D(batch):
    batch["pixel_values"] = [[data_map[index]] for index in batch["id"]]
    return batch

ds_3D = ds.map(
    prepare_dataset_3D,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array3D(shape=(1,  2, 3, dtype="float32")})
)
```
The error raised is:
```
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
[<ipython-input-6-676547e4cd41>](https://localhost:8080/#) in <module>()
      3     batched=True,
      4     remove_columns=ds.column_names,
----> 5     features=features.Features({"pixel_values": features.Array3D(shape=(1, 2, 3), dtype="float32")})
      6 )

12 frames
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1971                 new_fingerprint=new_fingerprint,
   1972                 disable_tqdm=disable_tqdm,
-> 1973                 desc=desc,
   1974             )
   1975         else:

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    518             self: "Dataset" = kwargs.pop("self")
    519         # apply actual function
--> 520         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    521         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    522         for dataset in datasets:

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    485         }
    486         # apply actual function
--> 487         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    488         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    489         # re-apply format to the output

[/usr/local/lib/python3.7/dist-packages/datasets/fingerprint.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    456             # Call actual function
    457
--> 458             out = func(self, *args, **kwargs)
    459
    460             # Update fingerprint of in-place transforms + update in-place history of transforms

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in _map_single(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2354                                 writer.write_table(batch)
   2355                             else:
-> 2356                                 writer.write_batch(batch)
   2357                 if update_data and writer is not None:
   2358                     writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py](https://localhost:8080/#) in write_batch(self, batch_examples, writer_batch_size)
    505             col_try_type = try_features[col] if try_features is not None and col in try_features else None
    506             typed_sequence = OptimizedTypedSequence(batch_examples[col], type=col_type, try_type=col_try_type, col=col)
--> 507             arrays.append(pa.array(typed_sequence))
    508             inferred_features[col] = typed_sequence.get_inferred_type()
    509         schema = inferred_features.arrow_schema if self.pa_writer is None else self.schema

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._handle_arrow_array_protocol()

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py](https://localhost:8080/#) in __arrow_array__(self, type)
    175                     storage = list_of_np_array_to_pyarrow_listarray(data, type=pa_type.value_type)
    176                 else:
--> 177                     storage = pa.array(data, pa_type.storage_dtype)
    178                 return pa.ExtensionArray.from_storage(pa_type, storage)
    179

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Can only convert 1-dimensional array values
```

**Describe the solution you'd like**
No error in the second scenario and an identical result to the following snippets.

**Describe alternatives you've considered**
There are other alternatives that work such as:
```python

def prepare_dataset_3D_bis(batch):
    batch["pixel_values"] = [[data_map[index].tolist()] for index in batch["id"]]
    return batch

ds_3D_bis = ds.map(
    prepare_dataset_3D_bis,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array3D(shape=(1, 2, 3), dtype="float32")})
)
```
or
```python
def prepare_dataset_3D_ter(batch):
    batch["pixel_values"] = [data_map[index][np.newaxis, :, :] for index in batch["id"]]
    return batch

ds_3D_ter = ds.map(
    prepare_dataset_3D_ter,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array3D(shape=(1,  2, 3), dtype="float32")})
)
```
But both solutions require the user to be aware that `data_map[index]` is an `np.array` type.

cc @lhoestq as we discuss this offline :smile:


Deprecate `shard_size` in `push_to_hub` in favor of `max_shard_size`

_The documentation is not available anymore as the PR was closed or merged._

This PR adds a `max_shard_size` param to `push_to_hub` and deprecates `shard_size` in favor of this new param to have a more descriptive name (a shard has at most the `shard_size` bytes in `push_to_hub`) for the param and to align the API with [Transformers](https://github.com/huggingface/transformers/blob/ff06b177917384137af2d9585697d2d76c40cdfc/src/transformers/modeling_utils.py#L1350).


Document how to use FAISS index for special operations, by accessing the index itself.

Close #4029.


Support streaming cnn_dailymail dataset

Support streaming cnn_dailymail dataset.

Fix #3969.

CC: @severo


Don't duplicate data when encoding audio or image

I'm not familiar with the concept of streaming vs non-streaming in HF datasets. I just wonder that you have the distinction here. Why doesn't it work to always make use of `bytes`? "using a local file - which is often required for audio" - why would that be?

The `path` would always point to some location in the `cache_dir`? I think this can be problematic. I would have expected that after I did `dataset.save_to_disk(...)` that I can remove the cache dir. But maybe just because I'm not familiar with HF. Or maybe the docs can be improved to clarify this.

We could always load every data file into `bytes` and save it this way the audio as bytes in `arrow` format, but the problem then would be that it makes the `file` column useless, *i.e.* people cannot inspect the audio file locally anymore or else they would need to first save bytes as a file which is not evident. This either breaks backwards compatibility or forces the user to stored 2x the required size locally. There was a longer discussion here: https://github.com/huggingface/datasets/issues/3663

It's a good argument though that `dataset.save_to_disk(...)` should save everything that is needed to the disk and should be independent of other folders, but I do think the arguments of #3663 to not break backwards compatibility and to allow people to inspect the downloaded audio files locally are a bit more important here.

But maybe, we could add a flag, `save_files_as_bytes` or `make_independent`, `make_self_contained` or a better name to `save_to_disk(...)` and `push_to_hub(...)` that would allow to make the resulting folder completely independent.

What do you think @mariosasko @lhoestq @polinaeterna @anton-l ?

For context: you can either store the path to local images or audio files, or the bytes of those files.

If your images and audio files are local files, then the arrow file from `save_to_disk` will store paths to these files.
If you want to include the bytes or your images or audio files instead, you must `read()` those files first.
This can be done by storing the "bytes" instead of the "path" of the images or audio files.

On the other hand, the resulting Parquet files from `push_to_hub` are self-contained, so that anyone can reload the dataset from the Hub. If your dataset contains image or audio data, the Parquet files will store the bytes of your images or audio files.

For now I just updated the documentation: https://github.com/huggingface/datasets/pull/4193. Maybe we can also embed the image and audio bytes in `save_to_disk` when we implement sharding, so that is can be done as efficiently as `push_to_hub`.

Anyway, merging this one :)

Right now if you pass both the `bytes` and a local `path` for audio or image data, then the `bytes` are unnecessarily written in the Arrow file, while we could just keep the local `path`.

This PR discards the `bytes` when the audio or image file exists locally.

In particular it's common for audio datasets builders to provide both the bytes and the local path in order to work for both streaming (using the bytes) and non-streaming mode (using a local file - which is often required for audio).

cc @patrickvonplaten


Fix outdated docstring about default dataset config


Librispeech documentation, clarification on format

The documentation in the code is definitely outdated - thanks for letting me know, I'll remove it in https://github.com/huggingface/datasets/pull/4184 .

You're exactly right `audio` `array` already decodes the audio file to the correct waveform. This is done on the fly, which is also why one should **not** do `ds["audio"]["array"][0]` as this will decode all dataset samples, but instead `ds[0]["audio"]["array"]` see: https://huggingface.co/docs/datasets/audio_process#audio-datasets

So, again to clarify: On disk, only the raw flac file content is stored? Is this also the case after `save_to_disk`?

And is it simple to also store it re-encoded as ogg or mp3 instead?

Hey,

Sorry yeah I was just about to look into this! We actually had an outdated version of Librispeech ASR that didn't save any files, but instead converted the audio files to a byte string, then was then decoded on-the-fly. This however is not very user-friendly so we recently decided to instead show the full path of the audio files with the `path` parameter.

I'm currently changing this for Librispeech here: https://github.com/huggingface/datasets/pull/4184 .
You should be able to see the audio file in the original `flac` format under `path` then. I don't think it's a good idea to convert to MP3 out-of-the-box, but we could maybe think about some kind of convert function for audio datasets cc @lhoestq ?

> I don't think it's a good idea to convert to MP3 out-of-the-box, but we could maybe think about some kind of convert function for audio datasets cc @lhoestq ?

Sure, I would expect that `load_dataset("librispeech_asr")` would give you the original (not re-encoded) data (flac or already decoded). So such re-encoding logic would be some separate generic function. So I could do sth like `dataset.reencode_as_ogg(**ogg_encode_opts).save_to_disk(...)` or so.

A follow-up question: I wonder whether a Parquet dataset is maybe more what we actually want to have? (Following also my comment here: https://github.com/huggingface/datasets/pull/4184#issuecomment-1105045491.) Because I think we actually would prefer to embed the data content in the dataset.

So, instead of `save_to_disk`/`load_from_disk`, we would use `to_parquet`,`from_parquet`? Is there any downside? Are arrow files more efficient?

Related is also the doc update in #4193.

`save_to_disk` saves the dataset as an Arrow file, which is the format we use to load a dataset using memory mapping. This way the dataset does not fill your RAM, but is read from your disk instead.

Therefore you can directly reload a dataset saved with `save_to_disk` using `load_from_disk`.

Parquet files are used for cold storage: to use memory mapping on a Parquet dataset, you first have to convert it to Arrow. We use Parquet to reduce the I/O when pushing/downloading data from the Hugging face Hub. When you load a Parquet file from the Hub, it is converted to Arrow on the fly during the download.

https://github.com/huggingface/datasets/blob/cd3ce34ab1604118351e1978d26402de57188901/datasets/librispeech_asr/librispeech_asr.py#L53

> Note that in order to limit the required storage for preparing this dataset, the audio
> is stored in the .flac format and is not converted to a float32 array. To convert, the audio
> file to a float32 array, please make use of the `.map()` function as follows:
>
> ```python
> import soundfile as sf
> def map_to_array(batch):
>     speech_array, _ = sf.read(batch["file"])
>     batch["speech"] = speech_array
>     return batch
> dataset = dataset.map(map_to_array, remove_columns=["file"])
> ```

Is this still true?

In my case, `ds["train.100"]` returns:
```
Dataset({
    features: ['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'],
    num_rows: 28539
})
```
and taking the first instance yields:
```
{'file': '374-180298-0000.flac',
 'audio': {'path': '374-180298-0000.flac',
  'array': array([ 7.01904297e-04,  7.32421875e-04,  7.32421875e-04, ...,
         -2.74658203e-04, -1.83105469e-04, -3.05175781e-05]),
  'sampling_rate': 16000},
 'text': 'CHAPTER SIXTEEN I MIGHT HAVE TOLD YOU OF THE BEGINNING OF THIS LIAISON IN A FEW LINES BUT I WANTED YOU TO SEE EVERY STEP BY WHICH WE CAME I TO AGREE TO WHATEVER MARGUERITE WISHED',
 'speaker_id': 374,
 'chapter_id': 180298,
 'id': '374-180298-0000'}
```

The `audio` `array` seems to be already decoded. So such convert/decode code as mentioned in the doc is wrong?

But I wonder, is it actually stored as flac on disk, and the decoding is done on-the-fly? Or was it decoded already during the preparation and is stored as raw samples on disk?

Note that I also used `datasets.load_dataset("librispeech_asr", "clean").save_to_disk(...)` and then `datasets.load_from_disk(...)` in this example. Does this change anything on how it is stored on disk?

A small related question: Actually I would prefer to even store it as mp3 or ogg on disk. Is this easy to convert?


[Librispeech] Add 'all' config

Just that I understand: With this change, simply doing `load_dataset("librispeech_asr")` is possible and returns the whole dataset?

And to get the subsets, I do sth like:
```python
ds = load_dataset("librispeech_asr")
train_ds = ds["train"]
dev_clean_ds = ds["dev-clean"]
dev_other_ds = ds["dev-other"]
test_clean_ds = ds["test-clean"]
test_other_ds = ds["test-other"]
```
?

> Just that I understand: With this change, simply doing `load_dataset("librispeech_asr")` is possible and returns the whole dataset?
>
> And to get the subsets, I do sth like:
>
> ```python
> ds = load_dataset("librispeech_asr")
> train_ds = ds["train"]
> dev_clean_ds = ds["dev-clean"]
> dev_other_ds = ds["dev-other"]
> test_clean_ds = ds["test-clean"]
> test_other_ds = ds["test-other"]
> ```
>
> ?

You could do:

```python
ds = load_dataset("librispeech_asr", "all")  # <- note that we have to pass a config
train_ds = ds["train"]
dev_clean_ds = ds["dev-clean"]
dev_other_ds = ds["dev-other"]
test_clean_ds = ds["test-clean"]
test_other_ds = ds["test-other"]
```

So, `load_dataset("librispeech_asr")` is not possible, it must be `load_dataset("librispeech_asr", "all")`?

Why is that?

The docs say:
```
name: `str` name, optional configuration for the dataset that affects the data generated on disk. Different
    `builder_config`s will have their own subdirectories and versions.
    If not provided, uses the first configuration in self.BUILDER_CONFIGS
```
https://github.com/huggingface/datasets/blob/cd3ce34ab1604118351e1978d26402de57188901/src/datasets/builder.py#L228

Or maybe you could just define `DEFAULT_CONFIG_NAME`?

> If not provided, uses the first configuration in self.BUILDER_CONFIGS

Oh crap this is outdated documentation. No it doesn't take the first config by default.

EDIT: opened a PR to fix this: https://github.com/huggingface/datasets/pull/4186

> No it doesn't take the first config by default.

But defining `DEFAULT_CONFIG_NAME` would work?

So should we define `DEFAULT_CONFIG_NAME = "all"` here as well? I think this is a reasonable default config.

Don't most datasets have some default config?

> But defining DEFAULT_CONFIG_NAME would work?
>
> So should we define DEFAULT_CONFIG_NAME = "all" here as well? I think this is a reasonable default config.

Yes that would work, and I also find it reasonable to do it :)

> Don't most datasets have some default config?

Most datasets only have one configuration, so the single configuration is the default one. Then other datasets gave several configurations, and whether they have a default one is decided case-by-case.

e.g. `glue` is a benchmark and doesn't have a default task, one must choose which task of `glue` they want to use explicitely.

Thanks a lot for the feedback!

Using `"all"` now as the default config. I changed the layout a bit so that there is not a single "train", but instead we have multiple "train.clean.100", "train.clean.360", "train.other.500". This way we don't even need to do filtering and it's also cleaner IMO.

@albertz - you should now be able to do the following:

```python
load_dataset("librispeech_asr")  # <- run this once to download, prepare dataset and cache everything

# The following operations will be very fast since all the downloading and processing is already cached
train_1 = load_dataset("librispeech_asr", split="train.clean.100")
print(train_1)
train_2 = load_dataset("librispeech_asr", split="train.clean.100+train.clean.360")
print(train_2)
train_full = load_dataset("librispeech_asr", split="train.clean.100+train.clean.360+train.other.500")
print(train_full)
dev_clean_ds = load_dataset("librispeech_asr", split="validation.clean")
print(dev_clean_ds)
dev_other_ds = load_dataset("librispeech_asr", split="validation.other")
print(dev_other_ds)
test_clean_ds = load_dataset("librispeech_asr", split="test.clean")
print(test_clean_ds)
test_other_ds = load_dataset("librispeech_asr", split="test.other")
print(test_other_ds)
```

Think this way we have the best of both worlds. Also @lhoestq, I think we could highlight better in the docs that it's possible to combine different splits. We do this actually quite a lot for speech. For Common Voice many people include "validation" in the training if the data is too small, e.g.: https://github.com/huggingface/transformers/blob/ff06b177917384137af2d9585697d2d76c40cdfc/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L147

Should we maybe add a short section to the loading tutorial here: https://huggingface.co/docs/datasets/v2.1.0/en/loading#hugging-face-hub ? (Happy to do it)

Is there any advantage or difference in calling `load_dataset` multiple times for each split? Or why not just call `load_dataset` once and then access each split?

Note in our case, we cannot really use the caching mechanism because we have a recipe pipeline used by multiple users (and I think a common cache dir for all users might end up in problems) and we basically would use `load_dataset("librispeech_asr").save_to_disk(...)` and then later `load_from_disk(...)`. (See here: https://github.com/rwth-i6/i6_core/pull/253)

So with `load_from_disk`, we cannot really provide the split this way, so we anyway would do sth like:
```python
ds = datasets.load_from_disk(...)
train = ds["train"]
```
Or with your latest proposal, it would look like:
```python
ds = datasets.load_from_disk(...)
train_ds = datasets.concatenate_datasets(
  [ds["train.clean.100"], ds["train.clean.360"], ds["train.other.500"]])
```
right?

> Is there any advantage or difference in calling `load_dataset` multiple times for each split? Or why not just call `load_dataset` once and then access each split?
>
> Note in our case, we cannot really use the caching mechanism because we have a recipe pipeline used by multiple users (and I think a common cache dir for all users might end up in problems) and we basically would use `load_dataset("librispeech_asr").save_to_disk(...)` and then later `load_from_disk(...)`. (See here: [rwth-i6/i6_core#253](https://github.com/rwth-i6/i6_core/pull/253))
>
> So with `load_from_disk`, we cannot really provide the split this way, so we anyway would do sth like:
>
> ```python
> ds = datasets.load_from_disk(...)
> train = ds["train"]
> ```
>
> Or with your latest proposal, it would look like:
>
> ```python
> ds = datasets.load_from_disk(...)
> train_ds = datasets.concatenate_datasets(
>   [ds["train.clean.100"], ds["train.clean.360"], ds["train.other.500"]])
> ```
>
> right?

I see the use case! The only advantage by calling `datasets` multiple times is that one can easily "merge" splits with `"+"`, but yeah you can do the exact same with `concatenate`.

@lhoestq what do you think is the best approach with `load_from_disk`?

@albertz, you could also define the `cache_dir` when doing `load_dataset(...)` which will then put all the relevant `arrow` files int the cache dir that you defined, e.g.:

```python
load_dataset("librispeech_asr", cache_dir="/easy/to/access/directory")
```

@albertz, I took a read through https://github.com/rwth-i6/i6_core/pull/253 .

I think the best would be the following:

1. Do `ds = load_dataset(..., cache_dir="/dir/that/is/easy/to/access")` <- having merged this PR, this will save all the original `.flac` files in the `cache_dir`
2. Do `ds.save_to_disk("local/path")` this should then only save the `arrow.format` with a `path` string to the audio files which are located in `cache_dir` <- this won't require a lot of memory after https://github.com/huggingface/datasets/pull/4184#discussion_r854132740 is fixed and can be done for each person individually.
3. `ds = datasets.load_from_disk("local/path")` can the be used. An object of `ds` will then have a `path` variable that links to the original audio files in the `cache_dir`. You can change these audio files then easily to `.mp3. You could do this with the `.map(...)` function, e.g. define a function that maps through all audio files, load them and then save them on disk afterward.

@lhoestq - I think this one is good to go

> @albertz, I took a read through [rwth-i6/i6_core#253](https://github.com/rwth-i6/i6_core/pull/253) .
>
> I think the best would be the following:
>
> 1. Do `ds = load_dataset(..., cache_dir="/dir/that/is/easy/to/access")` <- having merged this PR, this will save all the original `.flac` files in the `cache_dir`
> 2. Do `ds.save_to_disk("local/path")` this should then only save the `arrow.format` with a `path` string to the audio files which are located in `cache_dir` <- this won't require a lot of memory after [[Librispeech] Add 'all' config #4184 (comment)](https://github.com/huggingface/datasets/pull/4184#discussion_r854132740) is fixed and can be done for each person individually.
> 3. `ds = datasets.load_from_disk("local/path")` can the be used. An object of `ds` will then have a `path` variable that links to the original audio files in the `cache_dir`. You can change these audio files then easily to `.mp3. You could do this with the `.map(...)` function, e.g. define a function that maps through all audio files, load them and then save them on disk afterward.

Oh, so you say that our current implementation in https://github.com/rwth-i6/i6_core/pull/253 is broken? Because our cache dir is just some temp directory which will be removed afterwards, and we just store what we get out of `save_to_disk`. I think it would be good to clarify that in the doc of `save_to_disk`, that this is not enough and can depend on files from the cache dir. (@dthulke)

So, you say we anyway need to share the cache dir among users? But we would want to make sure that after the initial download and preparation of the data, this is set to readonly, because we want to make sure that other people will not modify the data in any way. Right?

But then, we don't really need the `save_to_disk` and `load_from_disk` at all, right?

@albertz

> Oh, so you say that our current implementation in https://github.com/rwth-i6/i6_core/pull/253 is broken? Because our cache dir is just some temp directory which will be removed afterwards, and we just store what we get out of save_to_disk. I think it would be good to clarify that in the doc of save_to_disk, that this is not enough and can depend on files from the cache dir. (@dthulke)

Oh, I wasn't aware that audio files are handled this way. Then we should have the cache directory as an additional job output, so that we keep the audio files.

> So, you say we anyway need to share the cache dir among users?

No, the cache dir can still be a directory in the job output folder. Then the audio paths in the corresponding dataset column correspond to the flac files in that directory. This way the "output" of the job is contained into the job directory and we don't write files to a global cache directory that is independent of the sisyphus graph.

If we want to share the audio data between different users, we can just link to a central instance of the job (similar to how we do it with the `DownloadLibriSpeechCorpusJob`).

@dthulke - that's a good point actually! So you can do both things:

1. Convert all audio files to bytes. Bytes can be saved by `arrow` so in this case you can do `save_to_disk(...)`, but then you cannot really inspect the audio files locally as they'll just be saved within a large arrow file (this actually used to be the default case but we're changing this now). The problem of this is summarized here a bit:  https://github.com/huggingface/datasets/issues/3663 . You can still do this if you'd like, e.g. you could do:

```python
ds = load_dataset("librispeech_asr")

def read_file(batch):
    with open(batch["file"], "r") as f:
        batch["bytes"] = f.read()
    return batch

ds = ds.map(read_file)
ds.save_to_disk("/path") <- the saved arrow object will now contain everything you need
```

however this is not recommend - it's should be much easier to just save the path to the downloaded audio files.

2. Not convert audio files to bytes, but just leave them in their original file format. Then only the path to the original files will be save in arrow. This will be the default case. This means that when you do `load_dataset(...)` both the orginal audio data and the arrow file will be saved in the `cache_dir` (which can be saved locally for every user or in a shared cache - we actually use a shared cache quite a bit at Hugging Face). When do you do `save_to_disk(...)` now only the `path` will be saved in `arrow` format (after this PR is merged, you'll see that the `arrow files should be very light weight` meaning that `save_to_disk(...)` can be done for every user, but has a dependency on the `cache_dir` (because the audio files live there).

=> Now what you could do as well would be to simply move all the audio files to the folder you want (the `save_to_disk(...)` folder) and then change the path of every sample to this folder (maybe with `map(...)`) and then this folder would be self contained. I do however think it's better to just specific a `cache_dir` and re-use `load_dataset(...)` every time instead of `load_from_disk` or `save_to_disk(...)`. Note that you can even pass the relevant cache files to `load_dataset(...)` here: https://huggingface.co/docs/datasets/v2.1.0/en/package_reference/loading_methods#datasets.load_dataset.data_files in which case you can be 100% sure that nothing is redownloaded.

We discussed storing audio files quite a bit, e.g. see: https://github.com/huggingface/datasets/issues/3663 and had (too many) changes around this topic recently, but we've come to the conclusion that the best is to leave the audio format in the format it was originally (`.flac` for Librispeech) so that the user can easily inspect it / understand the data. Arrow cannot save data is `.flac` so we'll just save a path to the original data. Curious to hear your guys opinion on this as well.

So what I would suggest here is to do the following:

1. Do `load_dataset(..., cache_dir=/a/read-only/folder)`
2.
- Either just re-use `load_dataset(..., cache_dir=...)` which should always re-use the data in the `cache_dir` since the hash of the url matches - so there should never be any duplicated downloading

or

- If you want to store the files in MP3 locally, first convert the files to MP3 in the read-only folder, then take do `ds.save_to_disk(/some/path)` which will save the correct path to the read-only folder to MP3 and then you can easily re-use the small arrow dataset that is saved in `/some/path`

> So what I would suggest here is to do the following:
>
> 1. Do `load_dataset(..., cache_dir=/a/read-only/folder)`
>
> * Either just re-use `load_dataset(..., cache_dir=...)` which should always re-use the data in the `cache_dir` since the hash of the url matches - so there should never be any duplicated downloading
>
> or
>
> * If you want to store the files in MP3 locally, first convert the files to MP3 in the read-only folder, then take do `ds.save_to_disk(/some/path)` which will save the correct path to the read-only folder to MP3 and then you can easily re-use the small arrow dataset that is saved in `/some/path`

Also relevant here: https://github.com/huggingface/datasets/issues/3663

I also added some documentation about how `save_to_disk` handles audio files here: https://github.com/huggingface/datasets/pull/4193

> > So, you say we anyway need to share the cache dir among users?
>
> No, the cache dir can still be a directory in the job output folder.

@dthulke But this is what I mean. When we share the job output folder, it means we share the cache dir among users.

I wonder if `load_dataset(..., cache_dir=job_output_cache_dir)` is always save to do then, that it really would not modify the `job_output_cache_dir`.

We could enforce that by making the `job_output_cache_dir` read-only afterwards. We currently don't do this.

@patrickvonplaten @dthulke But in any case, we actually prefer the data content to be inside the dataset (the arrow files). Lots of small files would be very problematic for our cache manager. We have one main copy of the data on NFS, but accessing the NFS directly by all computing nodes is not feasible, so the cache manager will have copies of the files on the nodes. So it means, whenever we access some file, we query the cache manager DB whether the file is already cached somewhere (some other computing node) and if so, it copies it from the other computing node and not from NFS. This works very well when there are not too many files (but the files can be big). So, we want to have only a few but big files. Even for NFS access this is much better.

I also commented in #3663.

Hey @albertz @dthulke,

Thanks a lot for your input!

We've discussed quite a bit with @lhoestq and we think the best approach is the following:

a)
`load_dataset(...)` will not store both bytes and the files because this would mean that 3x the size of the dataset would often be needed (1. the compressed `tar.gz` file, 2. the extracted file b, 3. the raw bytes in arrow format).

For canonical datasets like librispeech and common voice I think we want to keep the dataset filenames because of i) no breaking changes and ii) reasons explained in #3663

However it's also trivial to write your own datasetset downloading script of librispeech and just not extract the folder e.g. this line: https://huggingface.co/datasets/common_voice/blob/main/common_voice.py#L671

And then it'll be allowed to save the bytes and the dataset will be self-contained out-of-the-box when using `load_dataset(...)`

b) Now, one major problem that you guys uncovered is that `save_to_disk(...)` is currently not necessarily saving a dataset to be self-contained. We will change that asap. This means that after we've corrected this when you do download the canonical librispeech dataset the following will work:

```python
ds = load_dataset("....")  # <- here we have a dependency on the filepathes
ds[0]["audio"]["bytes"] # <- will not work

ds.save_to_disk("/local/path")  # <- now we want to have a self-contained dataset in arrow format, so we load the files into bytes and save it in arrow format

# now you can delete everything besides "/local/path"

ds = load_from_disk("/local/path")  # <- this will work
```

So either option a) where you define your own librispeech data downloading script (you guys could just sign up here: https://huggingface.co/join) and upload a dataset loading script in private mode so that no one can see it and you would always store the audio as bytes or b) where you first load then save to disk then delete cache would work.

Hope that fits in your vision :-)

cc @lhoestq @mariosasko

@patrickvonplaten sounds like a good approach to me. For b) this could even be configurable with a parameter like `embed_external_files` as you have for `push_to_hub` (if people prefer to keep separate audio files).

> However it's also trivial to write your own datasetset downloading script of librispeech and just not extract the folder

I don't exactly understand. In all cases, we need to extract it to prepare the dataset, or not? No matter if we want to store the raw bytes inside the dataset or leaving them as local files. Just in the first case, we can safely delete the extracted files after the dataset preparation.

> `save_to_disk(...)` is currently not necessarily saving a dataset to be self-contained. We will change that asap.

For us, this sounds exactly like what we want.

But regarding not introducing breaking changes, wouldn't this maybe also break some setups for users who don't expect this new behavior?

@albertz I would suggest to move the discussion on implementation details on our side to the following issue: rwth-i6/i6_core/issues/257

I like the idea of adding `embed_external_files` and set it to True by default to `save_to_disk`.
It's indeed a kind of breaking change since some users will get bigger Arrow files when updating the lib, but the advantages are nice:
1. I like the idea of having it self contained, in case you want to delete your cache
2. users also upload these Arrow files to cloud storage via the `fs` parameter, and in this case they would expect to upload a self-contained dataset
3. consistency with `push_to_hub`

If it sounds good to you I'll open an issue to discuss this and track the advancements

Add `"all"` config to Librispeech


Document librispeech configs

I think the main purpose of #4179 was how to be able to load both configs into one, so should we maybe add this part of the code: https://github.com/huggingface/datasets/issues/4179#issuecomment-1102383717

to the doc?

Actually @lhoestq would this work given that they have different split names: https://huggingface.co/datasets/librispeech_asr#data-splits ?

This doc extension does not explain why I can't simply load the whole dataset. Or what workaround I need to get the whole dataset, which is what people usually want for Librispeech.

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq, I can add a `"all"` config to Librispeech have the datasets already cached somewhere

I'm closing this PR then, feel free to continue the discussion in https://github.com/huggingface/datasets/issues/4179

Added an example of how to load one config or the other


Zenodo.org download is not responding

bug

Something isn't working

[Off topic but related: Is the uptime of S3 provably better than Zenodo's?]

Hi @dkajtoch, please note that at HuggingFace we are not hosting this dataset: we are just using a script to download their data file and create a dataset from it.

It was the dataset owners decision to host their data at Zenodo. You can see this on their website: https://marcobaroni.org/composes/sick.html

And yes, you are right: Zenodo is currently having some incidents and people are reporting problems from it.

On the other hand, we could contact the data owners and propose them to host their data at our Hugging Face Hub.

@julien-c I guess so.

Thanks @albertvillanova. I know that the problem lies in the source data. I just wanted to point out that these kind of problems are unavoidable without having one place where data sources are cached. Websites may go down or data sources may move. Having a copy in Hugging Face Hub would be a great solution.

Definitely, @dkajtoch! But we have to ask permission to the data owners. And many dataset licenses directly forbid data redistribution: in those cases we are not allowed to host their data on our Hub.

Ahhh good point! License is the problem :(

## Describe the bug
Source download_url from zenodo.org does not respond.
`_DOWNLOAD_URL = "https://zenodo.org/record/2787612/files/SICK.zip?download=1"`
Other datasets also use zenodo.org to store data and they cannot be downloaded as well.

It would be better to actually use more reliable way to store original data like s3 bucket.

## Steps to reproduce the bug
```python
load_dataset("sick")
```

## Expected results
Dataset should be downloaded.

## Actual results
ConnectionError: Couldn't reach https://zenodo.org/record/2787612/files/SICK.zip?download=1 (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='zenodo.org', port=443): Read timed out. (read timeout=100)")))

## Environment info
- `datasets` version: 2.1.0
- Platform: Darwin-21.4.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyArrow version: 7.0.0
- Pandas version: 1.3.5


FLEURS

dataset bug

A bug in a dataset script provided in the library

Yes, you just have to use `dl_manager.iter_archive` instead of `dl_manager.download_and_extract`.

That's because `download_and_extract` doesn't support TAR archives in streaming mode.

Tried to make it streamable, but I don't think it's really possible. @lhoestq @polinaeterna maybe you guys can check:
https://huggingface.co/datasets/google/fleurs/commit/dcf80160cd77977490a8d32b370c027107f2407b

real quick.

I think the problem is that we cannot ensure that the metadata file is found before the audio. Or is this possible somehow @lhoestq ?

@patrickvonplaten I think the metadata file should be found first because the audio files are contained in a folder next to the metadata files (just as in common voice), so the metadata files should be "on top of the list"   as they are closer to the root in the directories hierarchy

@patrickvonplaten but apparently it doesn't... I don't really know why.

Yeah! Any ideas what could be the reason here? cc @lhoestq ?

The order of the files is determined when the TAR archive is created, depending on the commands the creator ran.
If the metadata file is not at the beginning of the file, that makes streaming completely inefficient. In this case the TAR archive needs to be recreated in an appropriate order.

Actually we could maybe just host the metadata file ourselves and then stream the audio data only. Don't think that this would be a problem for the FLEURS authors (I can ask them :-))

## Dataset viewer issue for '*name of the dataset*'

https://huggingface.co/datasets/google/fleurs

```
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://storage.googleapis.com/xtreme_translations/FLEURS/af_za.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

Am I the one who added this dataset ? Yes

Can I fix this somehow in the script? @lhoestq @severo


Add some iteration method on a dataset column (specific for inference)

enhancement

New feature or request

Thanks for the suggestion ! I agree it would be nice to have something directly in `datasets` to do something as simple as that

cc @albertvillanova @mariosasko @polinaeterna What do you think if we have something similar to pandas `Series` that wouldn't bring everything in memory when doing `dataset["audio"]` ? Currently it returns a list with all the decoded audio data in memory.

It would be a breaking change though, since `isinstance(dataset["audio"], list)` wouldn't work anymore, but we could implement a `Sequence` so that `dataset["audio"][0]` still works and only loads one item in memory.

Your alternative suggestion with `iterate` is also sensible, though maybe less satisfactory in terms of experience IMO

I agree that current behavior (decoding all audio file sin the dataset when accessing `dataset["audio"]`) is not useful, IMHO. Indeed in our docs, we are constantly warning our collaborators not to do that.

Therefore I upvote for a "useful" behavior of `dataset["audio"]`. I don't think the breaking change is important in this case, as I guess no many people use it with its current behavior. Therefore, for me it seems reasonable to return a generator (instead of an in-memeory list) for "special" features, like Audio/Image.

@lhoestq on the other hand I don't understand your proposal about Pandas-like...

I recall I had the same idea while working on the `Image` feature, so I agree implementing something similar to `pd.Series` that lazily brings elements in memory would be beneficial.

@lhoestq @mariosasko Could you please give a link to that new feature of `pandas.Series`? As far as I remember since I worked with pandas for more than 6 years, there was no lazy in-memory feature; it was everything in-memory; that was the reason why other frameworks were created, like Vaex or Dask, e.g.

Yea pandas doesn't do lazy loading. I was referring to pandas.Series to say that they have a dedicated class to represent a column ;)

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

Currently, `dataset["audio"]` will load EVERY element in the dataset in RAM, which can be quite big for an audio dataset.
Having an iterator (or sequence) type of object, would make inference with `transformers` 's `pipeline` easier to use and not so memory hungry.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

For a non breaking change:

```python
for audio in dataset.iterate("audio"):
    # {"array": np.array(...), "sampling_rate":...}
```

For a  breaking change solution (not necessary), changing the type of `dataset["audio"]` to a sequence type so that

```python
pipe = pipeline(model="...")
for out in pipe(dataset["audio"]):
    # {"text":....}
```
could work

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

```python
def iterate(dataset, key):
    for item in dataset:
        yield dataset[key]

for out in pipeline(iterate(dataset, "audio")):
    # {"array": ...}
```

This works but requires the helper function which feels slightly clunky.

**Additional context**
Add any other context about the feature request here.

The context is actually to showcase better integration between  `pipeline` and `datasets` in the Quicktour demo: https://github.com/huggingface/transformers/pull/16723/files

@lhoestq


Dataset librispeech_asr fails to load

bug

Something isn't working

@patrickvonplaten Hi! I saw that you prepared this? :)

Another thing, but maybe this should be a separate issue: As I see from the code, it would try to use up to 16 simultaneous downloads? This is problematic for Librispeech or anything on OpenSLR. On [the homepage](https://www.openslr.org/), it says:

> If you want to download things from this site, please download them one at a time, and please don't use any fancy software-- just download things from your browser or use 'wget'. We have a firewall rule to drop connections from hosts with more than 5 simultaneous connections, and certain types of download software may activate this rule.

Related: https://github.com/tensorflow/datasets/issues/3885

Hey @albertz,

Nice to see you here! It's been a while ;-)

Sorry maybe the docs haven't been super clear here. By `split` we mean one of `train.500`, `train.360`, `train.100`, `validation`, `test`. For Librispeech, you'll have to specific a config (either `other` or `clean`) though:

```py
datasets.load_dataset("librispeech_asr", "clean")
```

should work and give you all splits (being "train", "test", ...) for the clean config of the dataset.

If you need both `"clean"` and `"other"` I think you'll have to do concatenate them as follows:

```py
from datasets import concatenate_datasets, load_dataset

other = load_dataset("librispeech_asr", "other")
clean = load_dataset("librispeech_asr", "clean")

librispeech = concatenate_datasets([other, clean])
```

See https://huggingface.co/docs/datasets/v2.1.0/en/process#concatenate

Downloading one split would be:

```py
from datasets import load_dataset

other = load_dataset("librispeech_asr", "other", split="train.500")
```

cc @lhoestq FYI maybe the docs can be improved here

Ah thanks. But wouldn't it be easier/nicer (and more canonical) to just make it in a way that simply `load_dataset("librispeech_asr")` works?

Pinging @lhoestq here, think this could make sense! Not sure however how the dictionary would then look like

Would it make sense to have `clean` as the default config ?

Also I think `load_dataset("librispeech_asr")` should have raised you an error that says that you need to specify a config

I also opened a PR to improve the doc: https://github.com/huggingface/datasets/pull/4183

> Would it make sense to have `clean` as the default config ?

I think a user would expect that the default would give you the full dataset.

> Also I think `load_dataset("librispeech_asr")` should have raised you an error that says that you need to specify a config

It does raise an error, but this error confused me because I did not understand why I needed a config, or why I could not simply download the whole dataset, which is what people usually do with Librispeech.

+1 for @albertz. Also think lots of people download the whole dataset (`"clean"` + `"other"`) for Librispeech.

Think there are also some people though who:
- a) Don't have the memory to store the whole dataset
- b) Just want to evaluate on one of the two configs

Ok ! Adding the "all" configuration would do the job then, thanks ! In the "all" configuration we can merge all the train.xxx splits into one "train" split, or keep them separate depending on what's the most practical to use (probably put everything in "train" no ?)

I'm not too familiar with how to work with HuggingFace datasets, but people often do some curriculum learning scheme, where they start with train.100, later go over to train.100 + train.360, and then later use the whole train (960h). It would be good if this is easily possible.

Hey @albertz,

opened a PR here. Think by adding the "subdataset" class to each split "train", "dev", "other" as shown here: https://github.com/huggingface/datasets/pull/4184/files#r853272727 it should be easily possible (e.g. with the filter function https://huggingface.co/docs/datasets/v2.1.0/en/package_reference/main_classes#datasets.Dataset.filter )

But also since everything is cached one could also just do:

```python
load_dataset("librispeech", "clean", "train.100")
load_dataset("librispeech", "clean", "train.100+train.360")
load_dataset("librispeech" "all", "train")
```

## Describe the bug
The dataset librispeech_asr (standard Librispeech) fails to load.

## Steps to reproduce the bug
```python
datasets.load_dataset("librispeech_asr")
```

## Expected results
It should download and prepare the whole dataset (all subsets).

In [the doc](https://huggingface.co/datasets/librispeech_asr), it says it has two configurations (clean and other).
However, the dataset doc says that not specifying `split` should just load the whole dataset, which is what I want.

Also, in case of this specific dataset, this is also the standard what the community uses. When you look at any publications with results on Librispeech, they always use the whole train dataset for training.

## Actual results
```
...
  File "/home/az/.cache/huggingface/modules/datasets_modules/datasets/librispeech_asr/1f4602f6b5fed8d3ab3e3382783173f2e12d9877e98775e34d7780881175096c/librispeech_asr.py", line 119, in LibrispeechASR._split_generators
    line: archive_path = dl_manager.download(_DL_URLS[self.config.name])
    locals:
      archive_path = <not found>
      dl_manager = <local> <datasets.utils.download_manager.DownloadManager object at 0x7fc07b426160>
      dl_manager.download = <local> <bound method DownloadManager.download of <datasets.utils.download_manager.DownloadManager object at 0x7fc07b426160>>
      _DL_URLS = <global> {'clean': {'dev': 'http://www.openslr.org/resources/12/dev-clean.tar.gz', 'test': 'http://www.openslr.org/resources/12/test-clean.tar.gz', 'train.100': 'http://www.openslr.org/resources/12/train-clean-100.tar.gz', 'train.360': 'http://www.openslr.org/resources/12/train-clean-360.tar.gz'}, 'other'...
      self = <local> <datasets_modules.datasets.librispeech_asr.1f4602f6b5fed8d3ab3e3382783173f2e12d9877e98775e34d7780881175096c.librispeech_asr.LibrispeechASR object at 0x7fc12a633310>
      self.config = <local> BuilderConfig(name='default', version=0.0.0, data_dir='/home/az/i6/setups/2022-03-20--sis/work/i6_core/datasets/huggingface/DownloadAndPrepareHuggingFaceDatasetJob.TV6Nwm6dFReF/output/data_dir', data_files=None, description=None)
      self.config.name = <local> 'default', len = 7
KeyError: 'default'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.1.0
- Platform: Linux-5.4.0-107-generic-x86_64-with-glibc2.31
- Python version: 3.9.9
- PyArrow version: 6.0.1
- Pandas version: 1.4.2


[feat] Add ImageNet dataset

_The documentation is not available anymore as the PR was closed or merged._

Thanks for the comments. I believe I have addressed all of them and also decreased the size of the dummy data file, so it should be ready for a re-review. I also made a change to allow adding synset mapping and valprep script in config in case we add ImageNet 21k some time later.

@lhoestq I have updated the PR to address all of the review comments.

To use the dataset download the tar file
[imagenet_object_localization_patched2019.tar.gz](https://www.kaggle.com/competitions/imagenet-object-localization-challenge/data?select=imagenet_object_localization_patched2019.tar.gz) from Kaggle and then point the datasets library to it by using:

```py
from datasets import load_dataset

dataset = load_dataset("imagenet",
data_dir="/path/to/imagenet_object_localization_patched2019.tar.gz")
```

Currently train and validation splits are supported.


Adding missing subsets to the `SemEval-2018 Task 1` dataset

This dataset for the [1st task of SemEval-2018](https://competitions.codalab.org/competitions/17751) competition was missing all subtasks except for subtask 5. I added another two subtasks (subtask 1 and 2), which are each comprised of 12 additional data subsets: for each language in En, Es, Ar, there are 4 datasets, broken down by emotions (anger, fear, joy, sadness).

## Remaining questions

I wasn't able to find any documentation about how one should make PRs to modify datasets. Because of that, I just did my best to integrate the new data into the code, and tested locally that this worked. I'm sorry if I'm not respecting your contributing guidelines – if they are documented somewhere, I'd appreciate if you could send a pointer!

Not sure how `dataset_infos.json` and `dummy` should be updated. My understanding is that they were automatically generated at the time of the original dataset creation?


Very slow between two operations

Something isn't working

Hello, in the processing stage, I use two operations. The first one : map + filter, is very fast and it uses the full cores, while the socond step is very slow and did not use full cores.

Also, there is a significant lag between them.  Am I missing something ?

 ```
raw_datasets = raw_datasets.map(split_func,
                batched=False,
                num_proc=args.preprocessing_num_workers,
                load_from_cache_file=not args.overwrite_cache,
                desc = "running split para ==>")\
                .filter(lambda example: example['text1']!='' and example['text2']!='',
                    num_proc=args.preprocessing_num_workers, desc="filtering ==>")

    processed_datasets = raw_datasets.map(
        preprocess_function,
        batched=True,
        num_proc=args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not args.overwrite_cache,
        desc="Running tokenizer on dataset===>",
    )
```


Add WIT Dataset

Hi! Coming in late with some context.

There are two versions of the WIT dataset:
1. The original source dataset managed by Wikimedia. It has more information, raw image representations, and each row corresponds to an image linked to all of its captions wherever it happens in Wikipedia (in multiple languages)
2. The Google version, corresponding to the data script in this PR, which duplicates image instances and requires the user to download the images themselves from the provided URL (note that a basic implementation will have them download the same picture several time. @thomasw21 using our download manager instead of `urllib` could help with that, but it wouldn't be required if people had access to the first version)

The Wikimedia folks were really interested in us hosting a ready-to-go streaming version of this dataset where users don't have to download the version themselves, which is why we have the pre-processed versions on an HF bucket, with the raw images and a pre-computed embedding (don't remember the model, we can keep it ). That's the data script currently in https://github.com/huggingface/datasets/pull/2981 . It's nearly ready to go, the one thing we should do is move the raw data from our HF google Cloud bucket to the Hub.

How do you want to move forward? IMO the best way would be to have a WIT dataset under the Wikimedia org with both configurations, but it depends on everyone's timelines

Okay after offline discussion. We'll improve this versions and push it to the hub under `google` namespace.

> which duplicates image instances and requires the user to download the images themselves from the provided URL (note that a basic implementation will have them download the same picture several time. @thomasw21 using our download manager instead of urllib could help with that, but it wouldn't be required if people had access to the first version)

Ah interesting wasn't aware of this duplication issue, concretely it'll just mean that our dataset in bigger than expected ... I think this should be handled after this loading script (though I have to figure our how to spawn a dl_manager).

> The Wikimedia folks were really interested in us hosting a ready-to-go streaming version of this dataset where users don't have to download the version themselves, which is why we have the pre-processed versions on an HF bucket, with the raw images and a pre-computed embedding (don't remember the model, we can keep it ). That's the data script currently in https://github.com/huggingface/datasets/pull/2981 . It's nearly ready to go, the one thing we should do is move the raw data from our HF google Cloud bucket to the Hub.

Similarly a script will be written and pushed to `wikimedia` organisation.

@mariosasko can you make one last review concerning the text description changes? Then I'll handle putting it under `google` namespace and close this PR.

Closing as this has been migrated to the hub under `google` namespace: https://huggingface.co/datasets/google/wit

closes #2981 #2810

@nateraw @hassiahk I've listed you guys as co-author as you've contributed previously to this dataset


Fix when map function modifies input in-place

When `function` modifies input in-place, the guarantee that columns in `remove_columns` are contained in `input` doesn't hold true anymore. Therefore we need to relax way we pop elements by checking if that column exists.


Stream private zipped images

oops looks like some tests are failing sorry, will fix them tomorrow

EDIT: not today but asap hopefully

cc @mariosasko this is ready for review, let me know what you think !

As mentioned in https://github.com/huggingface/datasets/issues/4139 it's currently not possible to stream private/gated zipped images from the Hub.

This is because `Image.decode_example` does not handle authentication. Indeed decoding requires to access and download the file from the private repository.

In this PR I added authentication to `Image.decode_example` via a `token_per_repo_id` optional argument. I first wanted to just pass `use_auth_token` but a single `Image` instance can be responsible of decoding images from a combination of several datasets together (from `interleave_datasets` for example). Therefore I just used a dictionary `repo_id` -> `token` instead.

I'm getting the `repo_id` from the dataset builder (I replaced the `namepace` attribute with `repo_id`)

I did the same for `Audio.decode_example`

cc @SBrandeis @severo


Update assin2 dataset_infos.json

Following comments in https://github.com/huggingface/datasets/issues/4003 we found that it was outdated and casing an error when loading the dataset


to_tf_dataset rewrite

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4170). All of your documentation changes will be reflected on that endpoint.

[Magic is now banned](https://www.youtube.com/watch?v=WIn58XoY728#t=36s) by decree of @sgugger. This is honestly much cleaner, and the functionality will make much more sense in `transformers` anyway!

@gante I renamed the default collator to `minimal_tf_collate_fn`!

@lhoestq @sgugger @gante

I think this should now be ready, it looks good in testing! I'll try a few more notebooks today and tomorrow to be sure before I merge. Key changes are:

- No column autodetection magic (will make a separate PR to add this as a `transformers` function)
- Drops non-numerical features automatically (this is more of a 'DataLoader' method, we'll have a separate method to expose 'raw' datasets to `tf.data`)
- Better autodetection of numerical features.
- Shouldn't randomly crash mid-function :skull:

We definitely have some questions still to resolve about how to handle making a 'DataLoader' dataset versus a 'raw' dataset - see [the Notion doc](https://www.notion.so/huggingface2/Splitting-to_tf_dataset-c2e0773c4bec484384064b30ed634383) if you're interested. Still, since this PR is just fixes/improvements to an existing method which never supported non-numerical features anyway, we can merge it before we've resolved those issues, and then think about how to name and split things afterwards.

P.S. I'll take out the region comments at the end before I merge, I promise! They're just helpful while I'm editing it

+1 for the tests

> Drops non-numerical features automatically

Can you give more details on how this work and the rationale as well ? This is not explained in the docs

Also why are you adding `error_on_missing` and `auto_fix_label_names ` ? The rationale is not clear to me. In particular I think it is sensible enough to expect users to not ask columns that don't exist, and to rename a label column when required.

@lhoestq I rewrote those parts - they were causing some other issues too! `error_on_missing` and `auto_fix_label_names` have been removed. The new logic is to simply drop (before batch collation) all columns the user doesn't ask for, but not to raise errors if the user asked for columns not in the dataset, as they may be added by the collator. Hopefully this cleans it up and matches the documentation better!

This PR rewrites almost all of `to_tf_dataset()`, which makes it kind of hard to list all the changes, but the most critical ones are:

- Much better stability and no more dropping unexpected column names (Sorry @NielsRogge)
- Doesn't clobber custom transforms on the data (Sorry @NielsRogge again)
- Much better handling of the situation when the `collate_fn` adds columns that aren't in the dataset.
- Better inference of shapes and data types
- Lots of hacky special-casing code removed
- Can return string columns (as `tf.String`)
- Most arguments have default values, calling the method should be much simpler
- ~~Can accept a `model` argument and only return columns that are valid inputs to that model~~
- Drops the `dummy_labels` argument - this was a workaround for Keras issues that have been resolved by changes in `transformers`. Also remove it from tests and the Overview notebook.

I still have a couple of TODOs remaining and some testing to do, so don't merge yet, but it should be mostly ready for review at this point!


Timit_asr dataset cannot be previewed recently

Thanks for reporting. The bug has already been detected, and we hope to fix it soon.

TIMIT is now a dataset that requires manual download, see #4145

Therefore it might take a bit more time to fix it

> TIMIT is now a dataset that requires manual download, see #4145
>
> Therefore it might take a bit more time to fix it

Thank you for your quickly response. Exactly, I also found the manual download issue in the morning. But when I used *list_datasets()* to check the available datasets, *'timit_asr'* is still in the list. So I am a little bit confused. If *'timit_asr'* need to be manually downloaded, does that mean we can **not** automatically download it **any more** in the future?

Yes exactly. If you try to load the dataset it will ask you to download it manually first, and to pass the downloaded and extracted data like `load_dataset("timir_asr", data_dir="path/to/extracted/data")`

The URL we were using was coming from a host that doesn't have the permission to redistribute the data, and the dataset owners (LDC) notified us about it.

## Dataset viewer issue for '*timit_asr*'

**Link:** *https://huggingface.co/datasets/timit_asr*

Issue: The timit-asr dataset cannot be previewed recently.

Am I the one who added this dataset ? Yes-No
No


Add code examples to API docs

documentation

Improvements or additions to documentation

_The documentation is not available anymore as the PR was closed or merged._

> Do you think it is clearer to make every code example fully reproducible so when users copy the code they can actually run it and get an output? This seems quite repetitive - maybe even unnecessary - but it is definitely clearer.

I think it's ok to be repetitive to get more clarity. Many users come from `transformers` and may have little experience with some processing methods (especially torch users).

> Should we showcase a function with more than one parameter to highlight different use-cases (it's pretty basic right now, but I'd be happy to add more)?

Maybe let's do it case by case, depending on whether there are parameters that are likely to be used often ?

> For the class_encode_column function, let me know if there is a simpler dataset with fewer columns (currently using winograd_wsc) so it is easier for users to see what changed.

You can try with `boolq`, it has a boolean column that can be converted to labels

> Where possible, I try to show the input before and the output after using a function like flatten for example. Do you think this is too much and just showing the usage (ie, >>> ds.flatten()) will be sufficient?

No I don't think it's too much, it's nice this way thanks :)

Updated each code example so they are fully reproducible (where applicable)! The next step will be to identify some functions where we can show off some parameters that are useful or commonly used. Some useful parameters can be:

- use `map(batched=True)` to process batches of examples.
- set a seed in `shuffle`.
- set `shuffle` and `seed` in `train_test_split`.

Let me know if you think of anything else related to the functions in `arrow_dataset.py`!

Cool thanks ! I think you can also do `num_proc` for `map`

This PR adds code examples for functions related to the base Datasets class to highlight usage. Most of the examples use the `rotten_tomatoes` dataset since it is nice and small. Several things I would appreciate feedback on:

- Do you think it is clearer to make every code example fully reproducible so when users copy the code they can actually run it and get an output? This seems quite repetitive - maybe even unnecessary - but it is definitely clearer. Personally, I think we might be able to get away with not including this since users probably want to try the function on their own dataset. For example:

   ```py
   >>> from datasets import load_dataset
   >>> ds = load_dataset("rotten_tomatoes", split="validation")
   >>> code example goes here
   ```

- Should we showcase a function with more than one parameter to highlight different use-cases (it's pretty basic right now, but I'd be happy to add more)?
- For the `class_encode_column` function, let me know if there is a simpler dataset with fewer columns (currently using `winograd_wsc`) so it is easier for users to see what changed.
- Where possible, I try to show the input before and the output after using a function like `flatten` for example. Do you think this is too much and just showing the usage (ie, `>>> ds.flatten()`) will be sufficient?

Thanks :)


Avoid rate limit in update hub repositories

I also set GIT_LFS_SKIP_SMUDGE=1 to speed up git clones

use http.extraHeader to avoid rate limit


Fix exact match

Clarify docs and add clarifying example to the exact_match metric


Fix google bleu typos, examples


Fix duplicate key in multi_news

To merge after this job succeeded: https://github.com/huggingface/datasets/runs/6012207928


Optional Content Warning for Datasets

enhancement

New feature or request

Hi! You can use the `extra_gated_prompt` YAML field in a dataset card for displaying custom messages/warnings that the user must accept before gaining access to the actual dataset. This option also keeps the viewer hidden until the user agrees to terms.

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

We now have hate speech datasets on the hub, like this one: https://huggingface.co/datasets/HannahRoseKirk/HatemojiBuild

I'm wondering if there is an option to select a content warning message that appears before the dataset preview? Otherwise, people immediately see hate speech when clicking on this dataset.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

Implementation of a content warning message that separates users from the dataset preview until they click out of the warning.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

Possibly just a way to remove the dataset preview completely? I think I like the content warning option better, though.

**Additional context**
Add any other context about the feature request here.