NyssaW/github-issues huggingface dataset:
Fix script of Arabic Billion Words dataset to return all data

The script has a bug and only parses and generates a portion of the entire dataset.

This PR fixes the loading script so that is properly parses the entire dataset. 

Current implementation generates the same number of examples as reported in the [original paper](https://arxiv.org/abs/1611.04033) for all configurations except for one:
- For "Youm7" we generate more examples (1172136) than the ones reported by the paper (1025027)

|                | Number of examples | Number of examples according to the source |
|:---------------|-------------------:|-----:|
| Alittihad      |             349342 |349342 |
| Almasryalyoum  |             291723 |291723 |
| Almustaqbal    |             446873 |446873 |
| Alqabas        |             817274 |817274 |
| Echoroukonline |             139732 |139732 |
| Ryiadh         |             858188 | 858188 |
| Sabanews       |              92149 |92149 |
| SaudiYoum      |             888068 |888068 |
| Techreen       |             314597 |314597 |
| Youm7          |            1172136 |1025027 |

Fix #3126.

Comments:
Make inspect.get_dataset_config_names always return a non-empty list of configs

**Is your feature request related to a problem? Please describe.**

Currently, some datasets have a configuration, while others don't. It would be simpler for the user to always have configuration names to refer to

**Describe the solution you'd like**

In that sense inspect.get_dataset_config_names should always return at least one configuration name, be it `default` or `Check___region_1` (for community datasets like `Check/region_1`).

https://github.com/huggingface/datasets/blob/c5747a5e1dde2670b7f2ca6e79e2ffd99dff85af/src/datasets/inspect.py#L161


Comments:
Hi @severo, I guess this issue requests not only to be able to access the configuration name (by using `inspect.get_dataset_config_names`), but the configuration itself as well (I mean you use the name to get the configuration afterwards, maybe using `builder_cls.builder_configs`), is this right?

Yes, maybe the issue could be reformulated. As a user, I want to avoid having to manage special cases:
- I want to be able to get the names of a dataset's configs, and use them in the rest of the API (get the data, get the split names, etc).
- I don't want to have to manage datasets with named configs (`glue`) differently from datasets without named configs (`acronym_identification`, `Check/region_1`)
Add ADE20k

## Adding a Dataset
- **Name:** ADE20k (actually it's called the MIT Scene Parsing Benchmark, it's actually a subset of ADE20k but a lot of authors still call it ADE20k)
- **Description:** A semantic segmentation dataset, consisting of 150 classes.
- **Paper:** http://people.csail.mit.edu/bzhou/publication/scene-parse-camera-ready.pdf
- **Data:** http://sceneparsing.csail.mit.edu/
- **Motivation:** I am currently adding Transformer-based semantic segmentation models that achieve SOTA on this dataset. It would be great to directly access this dataset using HuggingFace Datasets, in order to make example scripts in HuggingFace Transformers.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
I think we can close this issue since PR [#3607](https://github.com/huggingface/datasets/pull/3607) solves this.
Support Audio feature for TAR archives in sequential access

Add Audio feature support for TAR archived files in sequential access.

Fix #3128.

Comments:
Also do you think we can adapt `cast_column` to keep the same value for this new parameter when the user only wants to change the sampling rate ?

Thanks for your comments, @lhoestq, I will address them afterwards.

But, I think it is more important/urgent first address the current blocking non-passing test: https://github.com/huggingface/datasets/runs/4143579241?check_suite_focus=true
- I am thinking of a way of solving it, but if you have any hint, it will be more than welcome! 😅 

Basically:
```
{'audio': '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_dataset_with_audio_featur1/data/test_audio_44100.wav'}
``` 
becomes
```
{'audio': {'bytes': None, 'path': '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_dataset_with_audio_featur1/data/test_audio_44100.wav'}}
```
after a `map`, which is what was stored in the Arrow file. However we expect it remains invariant after this `map`.

@lhoestq, @mariosasko I finally proposed another implementation different from my last one:
- Before: store Audio always a struct<path: string, bytes: binary>, where bytes can be None
- Now, depending on the examples, either store Audio as a struct (as before), or as a string.

Please note that the main motivation for this change was the issue mentioned above: https://github.com/huggingface/datasets/pull/3129#issuecomment-964347056


Until here we had the assumption that a Features object always has an associated, deterministic, pyarrow schema. This is useful to ensure that we are able to concatenate two datasets that have the same features for example.

By breaking this assumption for the Audio type, how can we ensure that we can concatenate two audio datasets if one has Audio as a struct and the other a string ?

Oh I noticed that the Audio feature type has a private attribute `_storage_dtype`, so the assumption still holds, since they are now different feature types depending on the this attribute :)
(i mean different from the python equal operator point of view)

I think this PR is ready, @lhoestq, @mariosasko.  

Nit: We should also mention the new storage structure in the `Features` docstring [here](https://github.com/huggingface/datasets/blob/b29fb550c31de337b952035a7584147e0f18c0cf/src/datasets/features/features.py#L966) for users to know what type of value to return in their dataset scripts (we also have a link to that docstring in the `ADD_NEW_DATASET` template).
datasets-cli: convertion of a tfds dataset to a huggingface one.

### Discussed in https://github.com/huggingface/datasets/discussions/3079

<div type='discussions-op-text'>

<sup>Originally posted by **vitalyshalumov** October 14, 2021</sup>
I'm trying to convert a tfds dataset to a huggingface one.

I've tried:

1.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/mnist/3.0.1/

2.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/


and other permutations.
The script appears to be running and finishing without an error but when looking in the  huggingface/datasets/ folder nothing is created.


</div>

Comments:
Hi,

the MNIST dataset is already available on the Hub. You can use it as follows:
```python
import datasets
dataset_dict = datasets.load_dataset("mnist")
```

As for the conversion of TFDS datasets to HF datasets, we will be working on it in the coming months, so stay tuned.
"arabic_billion_words" dataset does not create the full dataset

## Describe the bug
When running: 
raw_dataset = load_dataset('arabic_billion_words','Alittihad')
the correct dataset file is pulled from the url.
But, the generated dataset includes just a small portion of the data included in the file.
This is true for all other portions of the "arabic_billion_words" dataset ('Almasryalyoum',.....)

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
raw_dataset = load_dataset('arabic_billion_words','Alittihad')

#The screen message
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: 20.62 MiB, post-processed: Unknown size, total: 352.74 MiB) 

## Expected results
over 100K sentences

## Actual results
only 11K sentences

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1



Comments:
Thanks for reporting, @vitalyshalumov.

Apparently the script to parse the data has a bug, and does not generate the entire dataset.

I'm fixing it.
More efficient nested features encoding

Nested encoding of features wastes a lot of time on operations which are effectively doing nothing when lists are used.
For example, if in the input we have a list of integers, `encoded_nested_example` will iterate over it and apply `encoded_nested_example` on every element even though it just return the int as is.

A similar issue is handled at an earlier stage when casting pytorch/tensorflow/pandas objects to python lists/numpy arrays:
https://github.com/huggingface/datasets/blob/c98c23c4260edadab00f997d1a5d66b7f2e93ce9/src/datasets/features/features.py#L149-L156
https://github.com/huggingface/datasets/blob/c98c23c4260edadab00f997d1a5d66b7f2e93ce9/src/datasets/features/features.py#L212-L228

In this pull request I suggest to use the same approach in `encoded_nested_example`.
In my setup there was a major speedup with this change: loading the data was at least x4 faster. 

Comments:
@lhoestq @albertvillanova @mariosasko
Can you please check this out?

Thanks, done!
Segmentation fault when loading datasets from file

## Describe the bug
Custom dataset loading sometimes segfaults and kills the process if chunks contain a variety of features/

## Steps to reproduce the bug

Download an example file:
```
wget https://gist.githubusercontent.com/TevenLeScao/11e2184394b3fa47d693de2550942c6b/raw/4232704d08fbfcaf93e5b51def9e5051507651ad/tiny_kelm.jsonl
```
Then in Python:
```
import datasets
tiny_kelm = datasets.load_dataset("json", data_files="tiny_kelm.jsonl", chunksize=100000)
```

## Expected results
a `tiny_kelm` functional dataset

## Actual results
☠️ `Segmentation fault (core dumped)` ☠️

## Environment info
- `datasets` version: 1.14.0
- Platform: Linux-5.11.0-38-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0

Comments:
Hi ! I created an issue on Arrow's JIRA after making a minimum reproducible example

https://issues.apache.org/jira/browse/ARROW-14439

```python
import io

import pyarrow.json as paj

batch = b'{"a": [], "b": 1}\n{"b": 1}'
block_size = 12

paj.read_json(
    io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
)
```

I don't see a way to workaround this properly now without hurting the performance of the JSON loader significantly though

The issue has been fixed in pyarrow 6.0.0, please update pyarrow :)

The issue was due to missing fields in the JSON data of type list. Now it's working fine and missing list fields are replaced with empty lists
Use huggingface_hub.HfApi to list datasets/metrics

Delete `datasets.inspect.HfApi` and use `huggingface_hub.HfApi` instead.

WIP until https://github.com/huggingface/huggingface_hub/pull/429 is merged, then wait for the new release of  `huggingface_hub`, update the `huggingface_hub` version in `setup.py` and merge this PR.

cc: @lhoestq 

Comments:
Correctly update metadata to preserve features when concatenating datasets with axis=1

This PR correctly updates metadata to preserve higher-level feature types (e.g. `ClassLabel`) in `datasets.concatenate_datasets` when `axis=1`. Previously, we would delete the feature metadata in `datasets.concatenate_datasets` if `axis=1` and restore the feature types from the arrow table schema in `Dataset.__init__`. However, this approach only works for simple feature types (e.g. `Value`).

Fixes #3111

Comments:
Add OpenSLR 83 - Crowdsourced high-quality UK and Ireland English Dialect speech

## Adding a Dataset
- **Name:** *openslr**
- **Description:** *Data set which contains male and female recordings of English from various dialects of the UK and Ireland.*
- **Paper:** *https://www.openslr.org/resources/83/about.html*
- **Data:** *Eleven separate data files can be found via https://www.openslr.org/resources/83/*
- **Motivation:** *Increase english ASR data with UK and Irish dialects*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

The *openslr* dataset already exists, this will add additional subset, *SLR83*.

Comments:
Ugh. The index files for SLR83 are CSV, not TSV. I need to add logic to process these index files.
CI error at each release commit

After 1.12.0, there is a recurrent CI error at each release commit: https://app.circleci.com/pipelines/github/huggingface/datasets/8289/workflows/665d954d-e409-4602-8202-e678594d2946/jobs/51110

```
____________________ LoadTest.test_load_dataset_canonical _____________________
[gw0] win32 -- Python 3.6.8 C:\tools\miniconda3\python.exe

self = <tests.test_load.LoadTest testMethod=test_load_dataset_canonical>

    def test_load_dataset_canonical(self):
        scripts_version = os.getenv("HF_SCRIPTS_VERSION", SCRIPTS_VERSION)
        with self.assertRaises(FileNotFoundError) as context:
            datasets.load_dataset("_dummy")
        self.assertIn(
            f"https://raw.githubusercontent.com/huggingface/datasets/{scripts_version}/datasets/_dummy/_dummy.py",
>           str(context.exception),
        )
E       AssertionError: 'https://raw.githubusercontent.com/huggingface/datasets/1.14.0/datasets/_dummy/_dummy.py' not found in "Couldn't find a dataset script at C:\\Users\\circleci\\datasets\\_dummy\\_dummy.py or any data file in the same directory. Couldn't find '_dummy' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/_dummy/_dummy.py"

tests\test_load.py:358: AssertionError
```



Comments:
Update doc links to point to new docs

This PR:
* updates the README links and the ADD_NEW_DATASET template to point to the new docs (the new docs don't have a section with the list of all the possible features, so I added that info to the `Features` docstring, which is then referenced in the ADD_NEW_DATASET template)
* fixes some broken links in the `.rst` files (fixed with the `make linkcheck` tool)

Comments:
Loading Data from HDF files

**Is your feature request related to a problem? Please describe.**
More often than not I come along big HDF datasets, and currently there is no straight forward way to feed them to a dataset.

**Describe the solution you'd like**
I would love to see a `from_h5` method that gets an interface implemented by the user on how items are extracted from dataset (in case of multiple datasets containing elements like arrays and metadata and etc).

**Describe alternatives you've considered**
Currently I manually load hdf files using `h5py` and implement PyTorch dataset interface. For small h5 files I load them into a pandas dataframe and use `from_pandas` function in the `datasets` package to load them, but for big datasets this is not feasible.

**Additional context**
HDF files are widespread throughout different domains and are one of the go to's for many researchers/scientists/engineers who work with numerical data. Given `datasets`' usecases have outgrown NLP use cases, it will make a lot of sense focusing on things like supporting HDF files.


Comments:
I'm currently working on bringing [Ecoset](https://www.pnas.org/doi/10.1073/pnas.2011417118) to huggingface datasets and I would second this request...

I would also like this support or something similar. Geospatial datasets come in netcdf which is derived from hdf5, or zarr. I've gotten zarr stores to work with datasets and streaming, but it takes awhile to convert the data to zarr if it's not stored in that natively. 

@mariosasko , I would like to contribute on this "good second issue" . Is there anything in the works for this Issue or can I go ahead ? 


Hi @VijayKalmath! As far as I know, nobody is working on it, so feel free to take over. Also, before you start, I suggest you comment `#self-assign` on this issue to assign it to yourself.

#self-assign

Hey @mariosasko can you assign this issue to me !!

So basically, we just need to load HDF5 files to Parquet?

e.g. Like this? https://stackoverflow.com/questions/46157709/converting-hdf5-to-parquet-without-loading-into-memory

#self-assign
Hi! I'd love to take a shot at implementing this feature to allow loading HDF5 datasets via Hugging Face Datasets. Will keep you updated here!

Hey @mariosasko,
I've opened a PR to address this: #7625
It adds a new h5folder loader to support .h5 files via load_dataset("h5folder", data_dir=...).
Feedback welcome!
concatenate_datasets removes ClassLabel typing.

## Describe the bug

When concatenating two datasets, we lose typing of ClassLabel columns.

I can work on this if this is a legitimate bug,

## Steps to reproduce the bug
```python
import datasets
from datasets import Dataset, ClassLabel, Value, concatenate_datasets

DS_LEN = 100
my_dataset = Dataset.from_dict(
    {
        "sentence": [f"{chr(i % 10)}" for i in range(DS_LEN)],
        "label": [i % 2 for i in range(DS_LEN)]
    }
)
my_predictions = Dataset.from_dict(
    {
        "pred": [(i + 1) % 2 for i in range(DS_LEN)]
    }
)

my_dataset = my_dataset.cast(datasets.Features({"sentence": Value("string"), "label": ClassLabel(2, names=["POS", "NEG"])}))
print("Original")
print(my_dataset)
print(my_dataset.features)


concat_ds = concatenate_datasets([my_dataset, my_predictions], axis=1)
print("Concatenated")
print(concat_ds)
print(concat_ds.features)


```

## Expected results
The features of `concat_ds` should contain ClassLabel.

## Actual results

On master, I get: 
```
{'sentence': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None), 'pred': Value(dtype='int64', id=None)}
```

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1



Comments:
Something like this would fix it I think: https://github.com/huggingface/datasets/compare/master...Dref360:HF-3111/concatenate_types?expand=1
Stream TAR-based dataset using iter_archive

I converted all the dataset based on TAR archive to use iter_archive instead, so that they can be streamable.

It means that around 80 datasets become streamable :)

Comments:
I'm creating a new branch `stream-tar-audio` just for the audio datasets since they need https://github.com/huggingface/datasets/pull/3129 to be merged first

The CI fails are only related to missing sections or tags in the dataset cards - which is unrelated to this PR
Add Google BLEU (aka GLEU) metric

This PR adds the NLTK implementation of Google BLEU metric. This is also a part of an effort to resolve an unfortunate naming collision between GLEU for machine translation and GLEU for grammatical error correction.
I used [this page](https://huggingface.co/docs/datasets/add_metric.html)  for reference. Please, point me to the right direction if I missed anything.

Comments:
download_mode=`force_redownload` does not work on removed datasets

## Describe the bug

If a cached dataset is removed from the library, I don't see how to delete it programmatically. I thought that using `force_redownload` would try to refresh the cache, then raise an exception, but it reuses the cache instead.

## Steps to reproduce the bug

_requires to already have `wit` in the cache_: see https://github.com/huggingface/datasets/pull/2981

```python
import datasets as ds
dataset = ds.load_dataset("wit", split="train", download_mode='force_redownload')
```

## Expected results

It should raise an exception, since the dataset does not exist anymore.

## Actual results

It uses the cached result

```
Using the latest cached version of the module from /home/slesage/.cache/huggingface/modules/datasets_modules/datasets/wit/107afbffd48e058b19101bddc47fbee25fa68eb6d50a733e262875f1285a5171 (last modified on Wed Sep 29 08:21:10 2021) since it couldn't be found locally at wit, or remotely on the Hugging Face Hub.
```

## Environment info

- `datasets` version: 1.13.4.dev0
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1

Comments:
Missing Zenodo 1.13.3 release

After `datasets` 1.13.3 release, this does not appear in Zenodo releases: https://zenodo.org/record/5570305

TODO:
- [x] Contact Zenodo support
- [x] Check it is fixed

Comments:
Zenodo has fixed on their side the 1.13.3 release: https://zenodo.org/record/5589150
Replace FSTimeoutError with parent TimeoutError

PR #3050 introduced a dependency on `fsspec.FSTiemoutError`. Note that this error only exists from `fsspec` version `2021.06.0` (June 2021).

To fix #3097, there are 2 alternatives:
- Either pinning `fsspec` to versions newer or equal to `2021.06.0`
- Or replacing `fsspec.FSTimeoutError` wth its parent `asyncio.TimeoutError`, which exists from Python 3.8.0 (Sep 2018).

This PR implements the second approach.

Fix #3097.

Comments:
`ModuleNotFoundError: No module named 'fsspec.exceptions'`

## Describe the bug
I keep runnig into a fsspec ModuleNotFound error

## Steps to reproduce the bug
```python
>>> from datasets import get_dataset_infos
2021-10-15 15:25:37.863206: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-10-15 15:25:37.863252: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 56, in <module>
    from .utils.streaming_download_manager import StreamingDownloadManager
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/streaming_download_manager.py", line 11, in <module>
    from fsspec.exceptions import FSTimeoutError
ModuleNotFoundError: No module named 'fsspec.exceptions'
```

Yet, I do have `fsspec`:
```bash
hf@victor-scale:~/dev/promptsource$ pip show fsspec
Name: fsspec
Version: 2021.5.0
Summary: File-system specification
Home-page: http://github.com/intake/filesystem_spec
Author: None
Author-email: None
License: BSD
Location: /home/hf/dev/promptsource/.venv/lib/python3.7/site-packages
Requires: 
Required-by: datasets
```

With the same version of fsspec and `datasets==1.9.0`, I don't see this problem....

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
I can't even run `datasets-cli env` actually.., but here's my env:
- `datasets` version: 1.13.3
- Platform: Ubuntu 18.04
- Python version: 3.7.10
- PyArrow version: 3.0.0


Comments:
Thanks for reporting, @VictorSanh.

I'm fixing it.
`cast_column` makes audio decoding fail

## Describe the bug

After changing the sampling rate automatic decoding fails.

## Steps to reproduce the bug
```python
from datasets import load_dataset
import datasets

ds = load_dataset("common_voice", "ab", split="train")

ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))

print(ds[0]["audio"])  # <- this fails currently
```

yields:

```
TypeError: forward() takes 2 positional arguments but 4 were given
```

## Expected results
no failure

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 1.13.2 (master)
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0


Comments:
cc @anton-l @albertvillanova 

Thanks for reporting, @patrickvonplaten.

I think the issue is related to mp3 resampling, not to `cast_column`.

You can check that `cast_column` works OK with non-mp3 audio files:
```python
from datasets import load_dataset
import datasets
ds = load_dataset("arabic_speech_corpus", split="train")
ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))
print(ds[0]["audio"])
```

I'm fixing it.
Support loading a dataset from SQLite files

As requested by @julien-c, we could eventually support loading a dataset from SQLite files, like it is the case for JSON/CSV files.

Comments:
for reference Kaggle has a good number of open source datasets stored in sqlite

Alternatively a tutorial or tool on how to convert from sqlite to parquet would be cool too

Hello, could we leverage [`pandas.read_sql`](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html) for this? 

This would be basically the same as [`CSVBuilder`](https://github.com/huggingface/datasets/blob/7380140accf522a4363bb56c0b77a4190f49bed6/src/datasets/packaged_modules/csv/csv.py#L127)
, but uses `pandas.read_sql(..., chunksize=1)` instead of `pandas.read_csv(..., iterator=True)`

I'm happy to work on this :) 

self-assign
Fix JNLBA dataset

As mentioned in #3089, I've added more tags and also updated the link for dataset which was earlier using a Google Drive link. 
I'm having problem with generating dummy data as `datasets-cli dummy_data  ./datasets/jnlpba --auto_generate --match_text_files "*.iob2"` is giving `datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
` error. I'll try to add dummy data manually.

Comments:
Fix #3089.

@albertvillanova all tests are passing now. Either you or @lhoestq can review it!
Fixes to `to_tf_dataset`


Comments:
Hi ! Can you give some details about why you need these changes ?

Hey, sorry, I should have explained! I've been getting a lot of `VisibleDeprecationWarning` from Numpy, due to an issue in the formatter, see #3084 . This is a temporary workaround (since I'm using these methods in the upcoming course) until I can fix that issue, because I couldn't see an obvious fix for the Numpy formatter. If you can see a quick way to fix that, though, that might be even better!
VisibleDeprecationWarning when using `set_format("numpy")`

Code to reproduce:

```
from datasets import load_dataset
dataset = load_dataset("glue", "mnli")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-cased')

def tokenize_function(dataset):
    return tokenizer(dataset['premise'])

tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=dataset['train'].features)

tokenized_datasets.set_format("numpy")

tokenized_datasets['train'][5:8]
```
Outputs:

```
python3.9/site-packages/datasets/formatting/formatting.py:167: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return np.array(array, copy=False, **self.np_array_kwargs)
```

Comments:
I just opened a PR and I verified that the code you provided doesn't show any deprecation warning :)
Datasets with Audio feature raise error when loaded from cache due to _resampler parameter

## Describe the bug
As reported by @patrickvonplaten, when loaded from the cache, datasets containing the Audio feature raise TypeError.

## Steps to reproduce the bug
```python
from datasets import load_dataset

# load first time works
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 

# load from cache breaks
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 
```

## Actual results
```
TypeError: __init__() got an unexpected keyword argument '_resampler'
```


Comments:
[Audio datasets] Adapting all audio datasets

This PR adds the new `Audio(...)` features - see: https://github.com/huggingface/datasets/pull/2324 to the most important audio datasets:

- Librispeech
- Timit
- Common Voice
- AMI
- ... (others I'm forgetting now)

The PR is curently blocked because the following leads to a problem:

```python
from datasets import load_dataset

# load first time works
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 

# load from cache breaks
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 
```

As soon as it's unblocked, I'll adapt the other audio datasets as well. 

Comments:
@lhoestq - are there other important speech datasets that I'm forgetting here? 

Think PR is good to go otherwise

@lhoestq @albertvillanova - how can we make an exception for the AMI README so that the test doesn't fail? The dataset card definitely should have a data preprocessing section

Hi @patrickvonplaten ,

the data preprocessing section is not defined as a valid section in the readme validation file. After this line:
https://github.com/huggingface/datasets/blob/568db594d51110da9e23d224abded2a976b3c8c7/src/datasets/utils/resources/readme_structure.yaml#L20
feel free to insert (correctly indented of course):
```python
- name: "Dataset Preprocessing"
  allow_empty: true
  allow_empty_text: true
  subsections: null
```
and then the tests should pass.

Thanks a lot @albertvillanova - I've added the feature to all audio datasets and corrected the task of `covost2`
Error related to timeout keyword argument

## Describe the bug
As reported by @patrickvonplaten, a TypeError is raised when trying to load a dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean") 
```

## Actual results
```
TypeError: dataset_info() got an unexpected keyword argument 'timeout'
```



Comments:
Error when loading a metric

## Describe the bug
As reported by @sgugger, after last release, exception is thrown when loading a metric.

## Steps to reproduce the bug
```python
from datasets import load_metric

metric = load_metric("squad_v2")
```

## Actual results
```
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-1-e612a8cab787> in <module>
      1 from datasets import load_metric
----> 2 metric = load_metric("squad_v2")

d:\projects\huggingface\datasets\src\datasets\load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, revision, script_version, **metric_init_kwargs)
   1336         )
   1337         revision = script_version
-> 1338     metric_module = metric_module_factory(
   1339         path, revision=revision, download_config=download_config, download_mode=download_mode
   1340     ).module_path

d:\projects\huggingface\datasets\src\datasets\load.py in metric_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, **download_kwargs)
   1237                 if not isinstance(e1, FileNotFoundError):
   1238                     raise e1 from None
-> 1239                 raise FileNotFoundError(
   1240                     f"Couldn't find a metric script at {relative_to_absolute_path(combined_path)}. "
   1241                     f"Metric '{path}' doesn't exist on the Hugging Face Hub either."

FileNotFoundError: Couldn't find a metric script at D:\projects\huggingface\datasets\squad_v2\squad_v2.py. Metric 'squad_v2' doesn't exist on the Hugging Face Hub either.
```



Comments:
add XCSR dataset

Hi,

I wanted to add the [XCSR ](https://inklab.usc.edu//XCSR/xcsr_datasets) dataset to huggingface! :) 

I followed the instructions of adding new dataset to huggingface and have all the required files ready now! It would be super helpful if you can take a look and review them. Thanks in advance for your time and help. Look forward to hearing from you and can't wait to add XCSR to huggingface :D 

Comments:
> Hi ! Thanks for adding this dataset :)
> 
> Do you know how the translations were done ? Maybe we can mention that in the dataset card.
> 
> The rest looks all good to me :) good job with the dataset script and the dataset card !
> 
> Just one thing: we try to have dummy_data.zip files that are as small as possible, however here each zip file is 70KB+. It think we can make them even smaller if we remove unnecessary files in them. In particular in the `ar` dummy data zip file, we don't need the data for all languages, but rather only the `ar` files. Could you try to remove the unnecessary files in the dummy data zip files ?

Hi! 

Thank you so much for reviewing this PR. I've updated the README to briefly mention the translations and added a link to the paper, where a detailed description of the translation procedure can be found in the appendix.

For the dummy_data.zip files, is it possible to keep all the current files? I tried to remove some of the files, but the removal led to a failure in the local testing. We also think it may be better to keep the current dummy_data.zip files because all the data are useful actually. Thanks a lot!!

Hi @lhoestq, just a gentle ping on this PR. :D
Fix pathlib patches for streaming

Fix issue https://github.com/huggingface/datasets/issues/2866 (for good this time)

`counter` now works in both streaming and non-streaming mode.
And the `AttributeError: 'str' object has no attribute 'as_posix'` related to the patch of Path.open is fixed as well

Note : the patches should only affect the datasets module, not the user's ones ! That's why we should probably use something else than patch.object to patch the Path class' methods.

cc @severo @albertvillanova 

Comments:
Custom plain text dataset, plain json dataset and plain csv dataset are remove from datasets template folder

## Adding a Dataset
- **Name:** text, json, csv
- **Description:** I am developing a customized dataset loading script. The problem is  mainly about my custom dataset is seperate into many files and I only find a dataset loading template in [https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py](https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py) that can handle my circumstance. I'm afraid these templates are too old to use. Could you re-add these three templates to current master branch?



Comments:
Hi @zixiliuUSC, 

As explained in the documentation (https://huggingface.co/docs/datasets/loading.html#json), we support loading any dataset in JSON (as well as CSV, text, Parquet) format:
```python
ds = load_dataset('json', data_files='my_file.json')
```
feat: increase streaming retry config

Increase streaming config parameters:
* retry interval set to 5 seconds
* max retries set to 20 (so 1mn 40s)

Comments:
@lhoestq I had 2 runs for more than 2 days each, continuously streaming (they were failing before with 3 retries at 1 sec interval).

They are running on TPU's (so great internet connection) and only had connection errors a few times each (3 & 4). Each time it worked after only 1 retry.
The reason for a higher number of retries is for local connections. It would allow for almost 2mn of a wifi/ethernet disconnection. In practice this should not happen very often.

Let me know if you think it's too much.
add story_cloze


Comments:
Thanks for pushing this dataset :)

According to the CI, the file `cloze_test_val__spring2016 - cloze_test_ALL_val.csv` is missing in the dummy data zip file (the zip files seem empty). Feel free to add this file with 4-5 lines and it should be good

And you can fix the YAML tags with
```yaml
pretty_name: Story Cloze Test
```
and filling the other tags task_categories and task_ids

If the dataset doesn exist on paperswithcode, you can just leave
```yaml
paperswithcode_id: null
```

@lhoestq can't fix the last test fails.

> Thanks @zaidalyafeai, the failing test is due to an issue in the master branch, that has already been fixed.
> 
> You can include the fix:
> 
> ```
> git checkout add_story_cloze
> git fetch upstream master
> git merge upstream/master
> ```

Thanks @albertvillanova, passed all the tests now. 

Thanks Albert, I fixed the suggested comments. This dataset has no train splits, it is only used for evaluation.
Add iter_archive

Added the `iter_archive` method for the StreamingDownloadManager.

It was already implemented in the regular DownloadManager.
Now it can be used to stream from TAR archives as mentioned in https://github.com/huggingface/datasets/issues/2829

I also updated the `food101` dataset as an example.
Any image/audio dataset using TAR archives can be updated to use `iter_archive` in order to be streamable :)

cc @severo 

Fix #2829.

Comments:
Make `interleave_datasets` more robust

**Is your feature request related to a problem? Please describe.**
Right now there are few hiccups using `interleave_datasets`. Interleaved dataset iterates until the smallest dataset completes it's iterator. In this way larger datasets may not complete full epoch of iteration. 
It creates new problems in calculation of epoch since there are no way to track which dataset from `interleave_datasets` completes how many epoch.

**Describe the solution you'd like**
For `interleave_datasets` module, 
- [ ] Add a boolean argument `--stop-iter` in `interleave_datasets` that enables dataset to either iterate infinite amount of time or not. That means it should not return `StopIterator` exception in case `--stop-iter=False`.
- [ ] Internal list variable `iter_cnt` that explains how many times (in steps/epochs) each dataset iterates at a given point.
- [ ] Add an argument `--max-iter` (list type) that explain maximum amount of time each of the dataset can iterate. After complete  `--max-iter` of one dataset, other dataset should continue sampling and when all the dataset finish their respective `--max-iter`, only then return `StopIterator`

Note: I'm new to `datasets` api. May be these features are already there in the datasets. 

Since multitask training is the latest trends, I believe this feature would make the `datasets` api more popular.

@lhoestq 

Comments:
Hi @lhoestq  Any response on this issue?

Hi ! Sorry for the late response

I agree `interleave_datasets` would benefit a lot from having more flexibility. If I understand correctly it would be nice to be able to define stopping strategies like `stop="first_exhausted"` (default) or `stop="all_exhausted"`. If you'd like to contribute this feature I'd be happy to give you some pointers :)

Also one can already set the max number of iterations per dataset by doing `dataset.take(n)` on the dataset that should only have `n` samples.

Regarding the `iter_cnt` counter, I think this requires a bit more thoughts, since we might have to be able to backpropagate the the counter if `map` or other transforms have been applied after `interleave_datasets`. 

@sbmaruf I just notice that (1)`interleave_datasets` only samples indices once and reuse for all epochs, and (2) it's limited by the smallest dataset. Do you figure out an alternative way to achieve the same purpose?
Windows CI is unable to test streaming properly because of SSL issues

In https://github.com/huggingface/datasets/pull/3041 the windows tests were skipped because of SSL issues with moon-staging.huggingface.co:443

The issue appears only on windows with asyncio. On Linux it works. With requests it works as well. And with the production environment huggingface.co it also works.

to reproduce on windows:
```python
import fsspec

# use any URL to a file in a dataset repo
url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"

fsspec.open(url).open()
```
raises
```python
FileNotFoundError: https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes
```
because of
```python
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host moon-staging.huggingface.co:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')]
```

Comments:
I think this problem is already fixed:
```python
In [4]: import fsspec
   ...:
   ...: url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"
   ...:
   ...: fsspec.open(url).open()
Out[4]: <File-like object HTTPFileSystem, https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattribu
```

No I'm still having this issue on my windows, and so does the CI
Feature request : add leave=True to dataset.map to enable tqdm nested bars (and whilst we're at it couldn't we get a way to access directly tqdm underneath?) 

**A clear and concise description of what you want to happen.**

It would be so nice to be able to nest HuggingFace `Datasets.map() ` progress bars in the grander scheme of things and whilst we're at it why not other functions.

**Describe alternatives you've considered**

By the way is there not a way to directly interact with underlying tqdm module ? **kwargs-ish?

**Additional context**

Furthering tqdm integration #2374 and huggingface/transformers#11797 solutioned by huggingface/transformers#12226 provided with tqdm description as `desc=`

@sgugger @bhavitvyamalik

Comments:
@lhoestq, @albertvillanova can we have `**tqdm_kwargs` in `map`? If there are any fields that are important to our tqdm (like iterable or unit), we can pop them before initialising the tqdm object so as to avoid duplicity.

Hi ! Sounds like a good idea :)

Also I think it would be better to have this as an actual parameters instead of kwargs to make it clearer
Fix task reloading from cache

When reloading a dataset from the cache when doing `map`, the tasks templates were kept instead of being updated regarding the output of the `map` function. This is an issue because we drop the tasks templates that are not compatible anymore after `map`, for example if a column of the template was removed.

This PR fixes this and for convenience introduces a decorator `@transmit_tasks` that takes care of doing this verification, similar to the `@transmit_format` decorator.

This should fix issue https://github.com/huggingface/datasets/issues/3047 cc @sgugger 

Comments:
Error in per class precision computation 

## Describe the bug
When trying to get the per class precision values by providing `average=None`, following error is thrown `ValueError: can only convert an array of size 1 to a Python scalar`

## Steps to reproduce the bug
```python
from datasets import load_dataset, load_metric
precision_metric = load_metric("precision")
predictions = [0, 2, 1, 0, 0, 1]
references = [0, 1, 2, 0, 1, 2]
results = precision_metric.compute(predictions=predictions, references=references, average=None)
```

## Expected results
`  {'precision': array([0.66666667, 0.        , 0.        ])}`
as per https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py

## Actual results
```
   output = self._compute(predictions=predictions, references=references, **kwargs)
  File "~/.cache/huggingface/modules/datasets_modules/metrics/precision/94709a71c6fe37171ef49d3466fec24dee9a79846c9f176dff66a649e9811690/precision.py", line 110, in _compute
    sample_weight=sample_weight,
ValueError: can only convert an array of size 1 to a Python scalar
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: linux
- Python version: 3.6.9
- PyArrow version: 5.0.0


Comments:
Hi @tidhamecha2, thanks for reporting.

Indeed, we fixed this issue just one week ago: #3008

The fix will be included in our next version release.

In the meantime, you can incorporate the fix by installing `datasets` from the master branch:
```
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```
load_dataset('the_pile_openwebtext2') produces ArrowInvalid, value too large to fit in C integer type

## Describe the bug
When loading `the_pile_openwebtext2`, we get the error `pyarrow.lib.ArrowInvalid: Value 2111 too large to fit in C integer type`

## Steps to reproduce the bug
```python
import datasets
ds = datasets.load_dataset('the_pile_openwebtext2')
```

## Expected results
Should download the dataset, convert it to an arrow file, and return a working Dataset object.

## Actual results
The download works, but conversion to the arrow file fails as follows:

```
>>> ds = datasets.load_dataset('the_pile_openwebtext2')
Downloading and preparing dataset openwebtext2/plain_text (download: 27.33 GiB, generated: 63.86 GiB
, post-processed: Unknown size, total: 91.19 GiB) to /home/davidbau/.cache/huggingface/datasets/open
webtext2/plain_text/1.0.0/c48ec73ba3483bac673463f48f67e9a4fd8cb49a9d6ec4fb957f0b424b97cf25...
Traceback (most recent call last):
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/builder.py", line 1133,
 in _prepare_split
    writer.write(example, key)
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
366, in write
    self.write_examples_on_file()
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
311, in write_examples_on_file
    pa_array = pa.array(typed_sequence)
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/davidbau/.conda/envs/tenv/lib/python3.9/site-packages/datasets/arrow_writer.py", line
115, in __arrow_array__
    out = pa.array(cast_to_python_objects(self.data, only_1d_for_numpy=True), type=type)
  File "pyarrow/array.pxi", line 305, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Value 2111 too large to fit in C integer type
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
```

- Platform: Ubuntu 20.04
- Python version: python 3.9
- PyArrow version: 3.0.0


Comments:
I encountered the same bug using different datasets.
any suggestions?

+1, can reproduce here!

I get the same error
Platform: Windows 10
Python: python 3.8.8
PyArrow: 5.0

I was getting a similar error `pyarrow.lib.ArrowInvalid: Integer value 528 not in range: -128 to 127` - AFAICT, this is because the type specified for `reddit_scores` is `datasets.Sequence(datasets.Value("int8"))`, but the actual values can be well outside the max range for 8-bit integers.

I worked around this by downloading the `the_pile_openwebtext2.py` and editing it to use local files and drop reddit scores as a column (not needed for my purposes).

Addressed in https://huggingface.co/datasets/the_pile_openwebtext2/discussions/4