More fine-grained taxonomy of error types

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

Exceptions like `FileNotFoundError` can be raised by different parts of the code, and it's hard to detect which one did

**Describe the solution you'd like**

Give a specific exception type for every group of similar errors

**Describe alternatives you've considered**

Rely on the error message, using regex


Make inspect.get_dataset_config_names always return a non-empty list of configs

New feature or request

dataset-viewer

Related to the dataset viewer on huggingface.co

Hi @severo, I guess this issue requests not only to be able to access the configuration name (by using `inspect.get_dataset_config_names`), but the configuration itself as well (I mean you use the name to get the configuration afterwards, maybe using `builder_cls.builder_configs`), is this right?

Yes, maybe the issue could be reformulated. As a user, I want to avoid having to manage special cases:
- I want to be able to get the names of a dataset's configs, and use them in the rest of the API (get the data, get the split names, etc).
- I don't want to have to manage datasets with named configs (`glue`) differently from datasets without named configs (`acronym_identification`, `Check/region_1`)

**Is your feature request related to a problem? Please describe.**

Currently, some datasets have a configuration, while others don't. It would be simpler for the user to always have configuration names to refer to

**Describe the solution you'd like**

In that sense inspect.get_dataset_config_names should always return at least one configuration name, be it `default` or `Check___region_1` (for community datasets like `Check/region_1`).

https://github.com/huggingface/datasets/blob/c5747a5e1dde2670b7f2ca6e79e2ffd99dff85af/src/datasets/inspect.py#L161


bug

Something isn't working

Hi,

Did you try to run the code multiple times (GitHub URLs can be down sometimes for various reasons)? I can access `https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py`, so this code is working without an error on my side.

Additionally, can you please run the `datasets-cli env` command because it seems to me that you are using the `datasets` version different from `1.12.1`?

Same issue when running `metric = datasets.load_metric("accuracy")`.
Error info is:
```
metric = datasets.load_metric("accuracy")
Traceback (most recent call last):

  File "<ipython-input-2-d25db38b26c5>", line 1, in <module>
    metric = datasets.load_metric("accuracy")

  File "D:\anaconda3\lib\site-packages\datasets\load.py", line 610, in load_metric
    module_path, _ = prepare_module(

  File "D:\anaconda3\lib\site-packages\datasets\load.py", line 330, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)

  File "D:\anaconda3\lib\site-packages\datasets\utils\file_utils.py", line 288, in cached_path
    output_path = get_from_cache(

  File "D:\anaconda3\lib\site-packages\datasets\utils\file_utils.py", line 605, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/accuracy/accuracy.py
```

 My `datasets-cli env` result is as follows:
- `datasets` version: 1.11.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 6.0.0

@yananchen1989 did you find a way to solve this?

It seems to be able to solve this issue by adding the equivalent `accuracy.py` locally.
change `metric = datasets.load_metric("accuracy")` to  `metric = datasets.load_metric(path = "./accuracy.py")`.
Copy `accuracy.py` from browser at [accuracy.py](https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/accuracy/accuracy.py)

datasets version: 1.12.1

`metric = datasets.load_metric('rouge')`

The error:

> ConnectionError                           Traceback (most recent call last)
> <ipython-input-3-dd10a0c5212f> in <module>
> ----> 1 metric = datasets.load_metric('rouge')
>
> /usr/local/lib/python3.6/dist-packages/datasets/load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, script_version, **metric_init_kwargs)
>     613         download_config=download_config,
>     614         download_mode=download_mode,
> --> 615         dataset=False,
>     616     )
>     617     metric_cls = import_main_class(module_path, dataset=False)
>
> /usr/local/lib/python3.6/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, **download_kwargs)
>     328                 file_path = hf_github_url(path=path, name=name, dataset=dataset, version=script_version)
>     329                 try:
> --> 330                     local_path = cached_path(file_path, download_config=download_config)
>     331                 except FileNotFoundError:
>     332                     if script_version is not None:
>
> /usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
>     296             use_etag=download_config.use_etag,
>     297             max_retries=download_config.max_retries,
> --> 298             use_auth_token=download_config.use_auth_token,
>     299         )
>     300     elif os.path.exists(url_or_filename):
>
> /usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
>     603             raise FileNotFoundError("Couldn't find file at {}".format(url))
>     604         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
> --> 605         raise ConnectionError("Couldn't reach {}".format(url))
>     606
>     607     # Try a second time
>
> ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py

Is there any remedy to solve the connection issue ?


Support Audio feature in streaming mode

enhancement

New feature or request

Currently, Audio feature is only supported for non-streaming datasets.

Due to the large size of many speech datasets, we should also support Audio feature in streaming mode.


Add ADE20k

dataset request

Requesting to add a new dataset

vision

Vision datasets

## Adding a Dataset
- **Name:** ADE20k (actually it's called the MIT Scene Parsing Benchmark, it's actually a subset of ADE20k but a lot of authors still call it ADE20k)
- **Description:** A semantic segmentation dataset, consisting of 150 classes.
- **Paper:** http://people.csail.mit.edu/bzhou/publication/scene-parse-camera-ready.pdf
- **Data:** http://sceneparsing.csail.mit.edu/
- **Motivation:** I am currently adding Transformer-based semantic segmentation models that achieve SOTA on this dataset. It would be great to directly access this dataset using HuggingFace Datasets, in order to make example scripts in HuggingFace Transformers.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Support Audio feature for TAR archives in sequential access

enhancement

New feature or request

Currently, Audio feature accesses each audio file by their file path.

However, streamed TAR archive files do not allow random access to their archived files.

Therefore, we should enhance the Audio feature to support TAR archived files in sequential access.


datasets-cli: convertion of a tfds dataset to a huggingface one.

Hi,

the MNIST dataset is already available on the Hub. You can use it as follows:
```python
import datasets
dataset_dict = datasets.load_dataset("mnist")
```

As for the conversion of TFDS datasets to HF datasets, we will be working on it in the coming months, so stay tuned.

### Discussed in https://github.com/huggingface/datasets/discussions/3079

Originally posted by **vitalyshalumov** October 14, 2021
I'm trying to convert a tfds dataset to a huggingface one.

I've tried:

1.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/mnist/3.0.1/

2.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/

and other permutations.
The script appears to be running and finishing without an error but when looking in the  huggingface/datasets/ folder nothing is created.


"arabic_billion_words" dataset does not create the full dataset

bug

Something isn't working

Thanks for reporting, @vitalyshalumov.

Apparently the script to parse the data has a bug, and does not generate the entire dataset.

I'm fixing it.

## Describe the bug
When running:
raw_dataset = load_dataset('arabic_billion_words','Alittihad')
the correct dataset file is pulled from the url.
But, the generated dataset includes just a small portion of the data included in the file.
This is true for all other portions of the "arabic_billion_words" dataset ('Almasryalyoum',.....)

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
raw_dataset = load_dataset('arabic_billion_words','Alittihad')

#The screen message
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: 20.62 MiB, post-processed: Unknown size, total: 352.74 MiB)

## Expected results
over 100K sentences

## Actual results
only 11K sentences

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1


Segmentation fault when loading datasets from file

Something isn't working

Hi ! I created an issue on Arrow's JIRA after making a minimum reproducible example

https://issues.apache.org/jira/browse/ARROW-14439

```python
import io

import pyarrow.json as paj

batch = b'{"a": [], "b": 1}\n{"b": 1}'
block_size = 12

paj.read_json(
    io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
)
```

I don't see a way to workaround this properly now without hurting the performance of the JSON loader significantly though

The issue has been fixed in pyarrow 6.0.0, please update pyarrow :)

The issue was due to missing fields in the JSON data of type list. Now it's working fine and missing list fields are replaced with empty lists

## Describe the bug
Custom dataset loading sometimes segfaults and kills the process if chunks contain a variety of features/

## Steps to reproduce the bug

Download an example file:
```
wget https://gist.githubusercontent.com/TevenLeScao/11e2184394b3fa47d693de2550942c6b/raw/4232704d08fbfcaf93e5b51def9e5051507651ad/tiny_kelm.jsonl
```
Then in Python:
```
import datasets
tiny_kelm = datasets.load_dataset("json", data_files="tiny_kelm.jsonl", chunksize=100000)
```

## Expected results
a `tiny_kelm` functional dataset

## Actual results
☠ `Segmentation fault (core dumped)` ☠

## Environment info
- `datasets` version: 1.14.0
- Platform: Linux-5.11.0-38-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0


OSError with a custom dataset loading script

Something isn't working

Hi,

there is a difference in how the `data_dir` is zipped between the `classla/janes_tag` and the `classla/reldi_hr` dataset. After unzipping, for the former, the data files (`*.conllup`) are in the root directory (root -> data files), and for the latter, they are inside the `data` directory (root -> `data` -> data files).

This can be fixed by removing the `os.path.join` call in https://huggingface.co/datasets/classla/janes_tag/blob/main/janes_tag.py#L86

Let me know if this works for you.

Hi Mario,

I had already tried that before, but it didn't work. I have now recreated the `classla/janes_tag`  zip file so that it also contains the `data` directory, but I am still getting the same error.

Hi,

I just tried to download the `classla/janes_tag` dataset, and this time the zip file is extracted correctly. However, the script is now throwing the IndexError, probably due to a bug in the `_generate_examples`.

Let me know if you are still getting the same error.

Hi,

could you try to download the dataset with a different `cache_dir` like so:
```python
import datasets
dataset = datasets.load_dataset('classla/janes_tag', split='validation', cache_dir="path/to/different/cache/dir")
```
If this works, then most likely the cached extracted data is causing issues. This data is stored at `~/.cache/huggingface/datasets/downloads/extracted` and needs to be deleted, and then it should work (you can easily locate the directory with the path given in the `OSError` message). Additionally, I'd suggest you to update `datasets` to the newest version with:
```
pip install -U datasets
```

Thank you, deleting the `~/.cache/huggingface/datasets/downloads/extracted` directory helped. However, I am still having problems.

There was indeed a bug in the script that was throwing an `IndexError`, which I have now corrected (added the condition to skip the lines starting with '# text') and it is working locally, but still throws an error when I try to load the dataset from HuggingFace. I literally copied and pasted the `_generate_examples` function and ran it on the `dev_all.conllup` file, which I even re-downloaded from the repository to be certain that the files are exactly the same. I also deleted everything again just in case, but it didn't help. The code works locally, but throws an `IndexError` when loading from `datasets.`

Hi,

Did some investigation.

To fix the dataset script on the Hub, append the following labels to the `names` list of the `upos_tags` field:
```'INTJ NOUN', 'AUX PRON', 'PART ADV', 'PRON ADP', 'INTJ INTJ', 'VERB NOUN', 'NOUN AUX'```.

This step is required to avoid an error due to missing labels in the following step which is:
```python
load_dataset("classla/janes_tag", split="validation", download_mode="force_redownload")
```
This will generate and cache the dataset, so specifying `download_mode` will not be required anymore unless you update the script/data on the Hub.

## Describe the bug
I am getting an OS error when trying to load the newly uploaded dataset classla/janes_tag. What puzzles me is that I have already uploaded a very similar dataset - classla/reldi_hr - with no issues. The loading scripts for the two datasets are almost identical and they have the same directory structure, yet I am only getting an error with janes_tag.

## Steps to reproduce the bug
```python
dataset = datasets.load_dataset('classla/janes_tag', split='validation')
```

## Expected results
Dataset correctly loaded.

## Actual results

Traceback (most recent call last):
  File "C:/mypath/test.py", line 91, in <module>
    load_and_print('janes_tag')
  File "C:/mypath/test.py", line 32, in load_and_print
    dataset = datasets.load_dataset('classla/{}'.format(ds_name), split='validation')
  File "C:\mypath\venv\lib\site-packages\datasets\load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 704, in _download_and_prepare
    ) from None
OSError: Cannot find data file.
Original error:
[Errno 2] No such file or directory: 'C:\\mypath\\.cache\\huggingface\\datasets\\downloads\\2c9996e44bdc5af9c89bffb9e6d7a3e42fdb2f56bacab45de13b20f3032ea7ca\\data\\train_all.conllup'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.5
- PyArrow version: 3.0.0


Add OpenSLR 83 - Crowdsourced high-quality UK and Ireland English Dialect speech

dataset request

Requesting to add a new dataset

Ugh. The index files for SLR83 are CSV, not TSV. I need to add logic to process these index files.

## Adding a Dataset
- **Name:** *openslr**
- **Description:** *Data set which contains male and female recordings of English from various dialects of the UK and Ireland.*
- **Paper:** *https://www.openslr.org/resources/83/about.html*
- **Data:** *Eleven separate data files can be found via https://www.openslr.org/resources/83/*
- **Motivation:** *Increase english ASR data with UK and Irish dialects*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

The *openslr* dataset already exists, this will add additional subset, *SLR83*.


CI error at each release commit

bug

Something isn't working

After 1.12.0, there is a recurrent CI error at each release commit: https://app.circleci.com/pipelines/github/huggingface/datasets/8289/workflows/665d954d-e409-4602-8202-e678594d2946/jobs/51110

```
____________________ LoadTest.test_load_dataset_canonical _____________________
[gw0] win32 -- Python 3.6.8 C:\tools\miniconda3\python.exe

self = <tests.test_load.LoadTest testMethod=test_load_dataset_canonical>

    def test_load_dataset_canonical(self):
        scripts_version = os.getenv("HF_SCRIPTS_VERSION", SCRIPTS_VERSION)
        with self.assertRaises(FileNotFoundError) as context:
            datasets.load_dataset("_dummy")
        self.assertIn(
            f"https://raw.githubusercontent.com/huggingface/datasets/{scripts_version}/datasets/_dummy/_dummy.py",
>           str(context.exception),
        )
E       AssertionError: 'https://raw.githubusercontent.com/huggingface/datasets/1.14.0/datasets/_dummy/_dummy.py' not found in "Couldn't find a dataset script at C:\\Users\\circleci\\datasets\\_dummy\\_dummy.py or any data file in the same directory. Couldn't find '_dummy' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/_dummy/_dummy.py"

tests\test_load.py:358: AssertionError
```


load_from_disk in DatasetsDict/Dataset not working with PyArrowHDFS wrapper implementing fsspec.spec.AbstractFileSystem

Something isn't working

Hi ! Can you try again with pyarrow 6.0.0 ? I think it includes some changes regarding filesystems compatibility with fsspec.

Hi @lhoestq! I ended up using `fsspec.implementations.arrow.HadoopFileSystem` which doesn't have the problem I described with pyarrow 5.0.0.

I'll try again with `PyArrowHDFS` once I update arrow to 6.0.0.

Thanks!

## Describe the bug
Passing a PyArrowHDFS implementation of fsspec.spec.AbstractFileSystem (in the `fs` param required by `load_from_disk` methods in `DatasetDict` (in datasets_dict.py) and `Dataset` (in arrow_dataset.py) results in an error when calling the download method in the `fs` parameter.

## Steps to reproduce the bug

The documentation for the `fs` parameter states:

```
fs (:class:`~filesystems.S3FileSystem` or ``fsspec.spec.AbstractFileSystem``, optional, default ``None``):
                Instance of the remote filesystem used to download the files from.
```

`PyArrowHDFS` from [fsspec](https://filesystem-spec.readthedocs.io/en/latest/_modules/fsspec/implementations/hdfs.html) implements `fsspec.spec.AbstractFileSystem`. However, when using it as shown below, I get an error.

```python
from fsspec.implementations.hdfs import PyArrowHDFS
...
transformed_corpus_path = "/user/my_user/clickbait/transformed_ds/"
fs = PyArrowHDFS(host, port, user, kerb_ticket=kerb_ticket)
dss = DatasetDict.load_from_disk(transformed_corpus_path, fs, True)
```

## Expected results

Previous to load from disk, I have managed to successfully store in HDFS the data and meta-information of a DatasetDict by doing:
```python
transformed_corpus_path = "/user/my_user/clickbait/transformed_ds/"
fs = PyArrowHDFS(host, port, user, kerb_ticket=kerb_ticket)
my_datasets.save_to_disk(transformed_corpus_path, fs=fs)
```

As I have 3 datasets in the DatasetDict named `my_datasets`, the previous Python code creates the following contents in HDFS:

```sh
$ hadoop fs -ls "/user/my_user/clickbait/transformed_ds/"
Found 4 items
-rw-------   3 my_user users         43 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/dataset_dict.json
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/test
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/train
drwx------   - my_user users          0 2021-10-19 03:08 /user/my_user/clickbait/transformed_ds/validation
```

I would expect to recover on `dss` the Arrow-backed datasets I previously saved in HDFS calling the `save_to_disk` method on the `DatasetDict` object when invoking `DatasetDict.load_from_disk(...)` as described above.

## Actual results

However, when trying to recover the saved datasets, I get this error:

```
...
  File "/home/fperez/dev/neuromancer/neuromancer/corpus.py", line 186, in load_transformed_corpus_from_disk
    dss = DatasetDict.load_from_disk(transformed_corpus_path, fs, True)
  File "/home/fperez/anaconda3/envs/neuromancer/lib/python3.9/site-packages/datasets/dataset_dict.py", line 748, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(dataset_dict_split_path, fs, keep_in_memory=keep_in_memory)
  File "/home/fperez/anaconda3/envs/neuromancer/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1048, in load_from_disk
    fs.download(src_dataset_path, dataset_path.as_posix(), recursive=True)
  File "pyarrow/_hdfsio.pyx", line 438, in pyarrow._hdfsio.HadoopFileSystem.download
TypeError: download() got an unexpected keyword argument 'recursive'
```

Examining the [signature of the download method in pyarrow 5.0.0](https://github.com/apache/arrow/blob/54d2bd89c99df72fa091b025452f85dd5d88e3cf/python/pyarrow/_hdfsio.pyx#L438) we can see that there's no download parameter:

```python
    def download(self, path, stream, buffer_size=None):
        with self.open(path, 'rb') as f:
            f.download(stream, buffer_size=buffer_size)
```

## Environment info
- `datasets` version: 1.13.3
- Platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 5.0.0


Loading Data from HDF files

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
More often than not I come along big HDF datasets, and currently there is no straight forward way to feed them to a dataset.

**Describe the solution you'd like**
I would love to see a `from_h5` method that gets an interface implemented by the user on how items are extracted from dataset (in case of multiple datasets containing elements like arrays and metadata and etc).

**Describe alternatives you've considered**
Currently I manually load hdf files using `h5py` and implement PyTorch dataset interface. For small h5 files I load them into a pandas dataframe and use `from_pandas` function in the `datasets` package to load them, but for big datasets this is not feasible.

**Additional context**
HDF files are widespread throughout different domains and are one of the go to's for many researchers/scientists/engineers who work with numerical data. Given `datasets`' usecases have outgrown NLP use cases, it will make a lot of sense focusing on things like supporting HDF files.


bug

Something isn't working

I am very unsure on why you tagged me here. I am not a maintainer of the Datasets library and have no idea how to help you.

Ok got it, tensor full of NaNs, cf.

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
315 # This check fails with FloatArrays with nans, which is not what we want, so account for that:

Actually this is is a live bug, documented yet still live so reopening

## Describe the bug
Despite having batches way under 2Gb when running `datasets.map()`, after processing correctly the data of the first batch without fuss and irrespective of writer_batch_size (say 2,4,8,16,32,64 and 128 in my case), it returns the following error :

> OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

Note that I always run `batch_size=writer_batch_size` :

## Steps to reproduce the bug
```python
datasets.map(lambda example : {"column_name" : function(arguments)}, batched=False, remove_columns = datasets.column_names, batch_size=batch_size, writer_batch_size=batch_size, disable_nullable=True, num_proc=None, desc="blablabla")
```
## Introspecting CUDA memory during bug

Placed within `function(arguments)` the following statement to introspect memory usage, merely a little over 1/4 of 2Gb
`print(torch.cuda.memory_summary(device=device, abbreviated=False))`

> |===========================================================================|
 |                  PyTorch CUDA memory summary, device ID 0                 |
 |---------------------------------------------------------------------------|
 |            CUDA OOMs: 0            |        cudaMalloc retries: 0         |
 |===========================================================================|
 |        Metric         | Cur Usage  | Peak Usage | Tot Alloc  | Tot Freed  |
 |---------------------------------------------------------------------------|
 | Allocated memory      |  541418 KB |  545725 KB |  555695 KB |   14276 KB |
 |       from large pool |  540672 KB |  544431 KB |  544431 KB |    3759 KB |
 |       from small pool |     746 KB |    1714 KB |   11264 KB |   10517 KB |
 |---------------------------------------------------------------------------|
 | Active memory         |  541418 KB |  545725 KB |  555695 KB |   14276 KB |
 |       from large pool |  540672 KB |  544431 KB |  544431 KB |    3759 KB |
 |       from small pool |     746 KB |    1714 KB |   11264 KB |   10517 KB |
 |---------------------------------------------------------------------------|
 | GPU reserved memory   |  598016 KB |  598016 KB |  598016 KB |       0 B  |
 |       from large pool |  595968 KB |  595968 KB |  595968 KB |       0 B  |
 |       from small pool |    2048 KB |    2048 KB |    2048 KB |       0 B  |
 |---------------------------------------------------------------------------|
 | Non-releasable memory |   36117 KB |   52292 KB |  274275 KB |  238158 KB |
|       from large pool |   34816 KB |   51537 KB |  261713 KB |  226897 KB |
 |       from small pool |    1301 KB |    2045 KB |   12562 KB |   11261 KB |
 |---------------------------------------------------------------------------|
 | Allocations           |     198    |     224    |     478    |     280    |
 |       from large pool |      74    |      75    |      75    |       1    |
 |       from small pool |     124    |     150    |     403    |     279    |
 |---------------------------------------------------------------------------|
 | Active allocs         |     198    |     224    |     478    |     280    |
 |       from large pool |      74    |      75    |      75    |       1    |
 |       from small pool |     124    |     150    |     403    |     279    |
 |---------------------------------------------------------------------------|
 | GPU reserved segments |      21    |      21    |      21    |       0    |
 |       from large pool |      20    |      20    |      20    |       0    |
 |       from small pool |       1    |       1    |       1    |       0    |
 |---------------------------------------------------------------------------|
 | Non-releasable allocs |      18    |      23    |     166    |     148    |
 |       from large pool |      17    |      18    |      19    |       2    |
 |       from small pool |       1    |       6    |     147    |     146    |
 |===========================================================================|

## Expected results
Efficiently process the datasets and write it down to disk.

## Actual results
--------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2390                             else:
-> 2391                                 writer.write(example)
   2392                 else:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write(self, example, key, writer_batch_size)
    367
--> 368             self.write_examples_on_file()
    369

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
    316                 if not isinstance(pa_array[0], pa.lib.FloatScalar):
--> 317                     raise OverflowError(
    318                         "There was an overflow in the {}. Try to reduce writer_batch_size to have batches smaller than 2GB".format(

OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

During handling of the above exception, another exception occurred:

OverflowError                             Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_16268/2456940807.py in <module>
      3 #tracker = OfflineEmissionsTracker(country_iso_code="FRA", project_name='xxx'+time_stamp,output_dir='./codecarbon')
      4 #tracker.start()
----> 5 process_datasets(source_datasets_paths, dataset_dir, LM_tokenizer, LMhead_model, datasets_selection=['wikipedia'], from_scratch=True,
      6                  clean_sentences=False, negative_sampling=False, translate=False, tokenize=False, generate_embeddings=True, concatenate_embeddings=False,
      7                  max_sample=10000, padding='do_not_pad', truncation=True, cpu_batch_size=1000, gpu_batch_size=2, cpu_writer_batch_size=1000, gpu_writer_batch_size=2, disable_nullable=True, num_proc=None) #

~\xxx\xxx.py in process_datasets(source_datasets_paths, dataset_dir, LM_tokenizer, LMhead_model, datasets_selection, from_scratch, clean_sentences, translate, negative_sampling, tokenize, generate_embeddings, concatenate_embeddings, max_sample, padding, truncation, cpu_batch_size, gpu_batch_size, cpu_writer_batch_size, gpu_writer_batch_size, disable_nullable, num_proc)
    481             for column in tqdm(dataset.column_names, desc=f'Processing column', leave=False):
    482                 if "xxx_" in column:
--> 483                     dataset = dataset.map(lambda example :
    484                                                 {"embeddings_"+str(column).replace("translated_",""):function(input_ids=example[column],
    485                                                                                token_type_ids=example[column.replace("input_ids","token_type_ids")],

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2034
   2035         if num_proc is None or num_proc == 1:
-> 2036             return self._map_single(
   2037                 function=function,
   2038                 with_indices=with_indices,

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in wrapper(*args, **kwargs)
    501             self: "Dataset" = kwargs.pop("self")
    502         # apply actual function
--> 503         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    504         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    505         for dataset in datasets:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in wrapper(*args, **kwargs)
    468         }
    469         # apply actual function
--> 470         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    471         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    472         # re-apply format to the output

~\anaconda3\envs\xxx\lib\site-packages\datasets\fingerprint.py in wrapper(*args, **kwargs)
    404             # Call actual function
    405
--> 406             out = func(self, *args, **kwargs)
    407
    408             # Update fingerprint of in-place transforms + update in-place history of transforms

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2425                 if update_data:
   2426                     if writer is not None:
-> 2427                         writer.finalize()
   2428                     if tmp_file is not None:
   2429                         tmp_file.close()

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in finalize(self, close_stream)
    440             # Re-intializing to empty list for next batch
    441             self.hkey_record = []
--> 442         self.write_examples_on_file()
    443         if self.pa_writer is None:
    444             if self._schema is not None:

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
    315                 # This check fails with FloatArrays with nans, which is not what we want, so account for that:
    316                 if not isinstance(pa_array[0], pa.lib.FloatScalar):
--> 317                     raise OverflowError(
    318                         "There was an overflow in the {}. Try to reduce writer_batch_size to have batches smaller than 2GB".format(
    319                             type(pa_array)

OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.8.11
- PyArrow version: 3.0.0

##Next steps
Testing on Linux.
@albertvillanova


concatenate_datasets removes ClassLabel typing.

Something isn't working

Something like this would fix it I think: https://github.com/huggingface/datasets/compare/master...Dref360:HF-3111/concatenate_types?expand=1

## Describe the bug

When concatenating two datasets, we lose typing of ClassLabel columns.

I can work on this if this is a legitimate bug,

## Steps to reproduce the bug
```python
import datasets
from datasets import Dataset, ClassLabel, Value, concatenate_datasets

DS_LEN = 100
my_dataset = Dataset.from_dict(
    {
        "sentence": [f"{chr(i % 10)}" for i in range(DS_LEN)],
        "label": [i % 2 for i in range(DS_LEN)]
    }
)
my_predictions = Dataset.from_dict(
    {
        "pred": [(i + 1) % 2 for i in range(DS_LEN)]
    }
)

my_dataset = my_dataset.cast(datasets.Features({"sentence": Value("string"), "label": ClassLabel(2, names=["POS", "NEG"])}))
print("Original")
print(my_dataset)
print(my_dataset.features)

concat_ds = concatenate_datasets([my_dataset, my_predictions], axis=1)
print("Concatenated")
print(concat_ds)
print(concat_ds.features)

```

## Expected results
The features of `concat_ds` should contain ClassLabel.

## Actual results

On master, I get:
```
{'sentence': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None), 'pred': Value(dtype='int64', id=None)}
```

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1


download_mode=`force_redownload` does not work on removed datasets

Something isn't working

## Describe the bug

If a cached dataset is removed from the library, I don't see how to delete it programmatically. I thought that using `force_redownload` would try to refresh the cache, then raise an exception, but it reuses the cache instead.

## Steps to reproduce the bug

_requires to already have `wit` in the cache_: see https://github.com/huggingface/datasets/pull/2981

```python
import datasets as ds
dataset = ds.load_dataset("wit", split="train", download_mode='force_redownload')
```

## Expected results

It should raise an exception, since the dataset does not exist anymore.

## Actual results

It uses the cached result

```
Using the latest cached version of the module from /home/slesage/.cache/huggingface/modules/datasets_modules/datasets/wit/107afbffd48e058b19101bddc47fbee25fa68eb6d50a733e262875f1285a5171 (last modified on Wed Sep 29 08:21:10 2021) since it couldn't be found locally at wit, or remotely on the Hugging Face Hub.
```

## Environment info

- `datasets` version: 1.13.4.dev0
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1


Missing Zenodo 1.13.3 release

Something isn't working

Zenodo has fixed on their side the 1.13.3 release: https://zenodo.org/record/5589150

After `datasets` 1.13.3 release, this does not appear in Zenodo releases: https://zenodo.org/record/5570305

TODO:
- [x] Contact Zenodo support
- [x] Check it is fixed


Unsuitable project description in PyPI

Currently, `datasets` project description appearing in PyPI shows the release instructions addressed to core maintainers: https://pypi.org/project/datasets/1.13.3/


Something isn't working

Hi @JTWang2000, thanks for reporting.

However, I cannot reproduce your reported bug:
```python
>>> from datasets import load_dataset

>>> dataset = load_dataset("sst", "default")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 8544
    })
    validation: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 1101
    })
    test: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 2210
    })
})
```

Maybe, the cause is that you have a quite old version of `huggingface_hub`. Could you please try to update it and confirm if the problem persists?
```
pip install -U huggingface_hub
```

Im facing the same issue. I did run the upgrade command but that doesnt seem to resolve the issue

Hi @aneeshjain, could you please specify which `huggingface_hub` version you are using?

Besides that, please run `datasets-cli env` and copy-and-paste its output below.

The problem seems to be with the latest version of `datasets`. After running `pip install -U datasets huggingface_hub`, I get the following:

```bash
python -c "import huggingface_hub; print(f'hbvers={huggingface_hub.__version__}'); import datasets; print(f'dvers={datasets.__version__}')"
hbvers=0.0.8
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/opt/conda/lib/python3.6/site-packages/datasets/data_files.py", line 122, in <module>
    allowed_extensions: Optional[list] = None,
AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'
````
Note that pip reports the latest `datasets` version as
```bash
 pip show datasets
Name: datasets
Version: 1.14.0
```
However, if I downgrade datasets with `pip install datasets==1.11.0`, things now work
```bash
python -c "import huggingface_hub; print(f'hbvers={huggingface_hub.__version__}'); import datasets; print(f'dvers={datasets.__version__}')"
hbvers=0.0.8
dvers=1.11.0
````

> Hi @JTWang2000, thanks for reporting.
>
> However, I cannot reproduce your reported bug:
>
> ```python
> >>> from datasets import load_dataset
>
> >>> dataset = load_dataset("sst", "default")
> >>> dataset
> DatasetDict({
>     train: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 8544
>     })
>     validation: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 1101
>     })
>     test: Dataset({
>         features: ['sentence', 'label', 'tokens', 'tree'],
>         num_rows: 2210
>     })
> })
> ```
>
> Maybe, the cause is that you have a quite old version of `huggingface_hub`. Could you please try to update it and confirm if the problem persists?
>
> ```
> pip install -U huggingface_hub
> ```

My problem solved after updating huggingface hub command. Thanks a lot and sorry for the late reply.

@tjruwase, please note that versions of `datsets` and `huggingface_hub` must be compatible one with each other:
- In `datasets` version `1.11.0`, the requirement on `huggingface_hub` was: `huggingface_hub<0.1.0`
  https://github.com/huggingface/datasets/blob/2cc00f372a96133e701275eec4d6b26d15257289/setup.py#L90
  - Therefore, your installed `huggingface_hub` version `0.0.8` was compatible
- In `datasets` version `1.12.0`, the requirement on `huggingface_hub` was: `huggingface_hub>=0.0.14,<0.1.0`
  https://github.com/huggingface/datasets/blob/6c766f9115d686182d76b1b937cb27e099c45d68/setup.py#L104
  - Therefore, your installed `huggingface_hub` version `0.0.8` was no longer compatible
- Currently, in `datasets` version `1.15.1`, the requirement on `huggingface_hub` is: `huggingface_hub>=0.1.0,<1.0.0`
  https://github.com/huggingface/datasets/blob/018100679d21cf27136f0eccb1c50e3a9c968ce2/setup.py#L102

@JTWang2000, thanks for your answer. I close this issue then.

## Describe the bug
When using `pip install datasets`
or use `conda install -c huggingface -c conda-forge datasets`
cannot install datasets

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("sst", "default")
```

## Actual results
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-fbe7981e6e21> in <module>
      1 import torch
      2 import transformers
----> 3 from datasets import load_dataset
      4
      5 dataset = load_dataset("sst", "default")

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/__init__.py in <module>
     35 from .arrow_reader import ArrowReader, ReadInstruction
     36 from .arrow_writer import ArrowWriter
---> 37 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
     38 from .combine import interleave_datasets
     39 from .dataset_dict import DatasetDict, IterableDatasetDict

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/builder.py in <module>
     42 )
     43 from .arrow_writer import ArrowWriter, BeamWriter
---> 44 from .data_files import DataFilesDict, _sanitize_patterns
     45 from .dataset_dict import DatasetDict, IterableDatasetDict
     46 from .fingerprint import Hasher

~/miniforge3/envs/actor/lib/python3.8/site-packages/datasets/data_files.py in <module>
    118
    119 def _exec_patterns_in_dataset_repository(
--> 120     dataset_info: huggingface_hub.hf_api.DatasetInfo,
    121     patterns: List[str],
    122     allowed_extensions: Optional[list] = None,

AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-11.3.1-arm64-arm-64bit
- Python version: 3.8.10
- PyArrow version: 5.0.0


`ModuleNotFoundError: No module named 'fsspec.exceptions'`

Something isn't working

Thanks for reporting, @VictorSanh.

I'm fixing it.

## Describe the bug
I keep runnig into a fsspec ModuleNotFound error

## Steps to reproduce the bug
```python
>>> from datasets import get_dataset_infos
2021-10-15 15:25:37.863206: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2021-10-15 15:25:37.863252: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/builder.py", line 56, in <module>
    from .utils.streaming_download_manager import StreamingDownloadManager
  File "/home/hf/dev/promptsource/.venv/lib/python3.7/site-packages/datasets/utils/streaming_download_manager.py", line 11, in <module>
    from fsspec.exceptions import FSTimeoutError
ModuleNotFoundError: No module named 'fsspec.exceptions'
```

Yet, I do have `fsspec`:
```bash
hf@victor-scale:~/dev/promptsource$ pip show fsspec
Name: fsspec
Version: 2021.5.0
Summary: File-system specification
Home-page: http://github.com/intake/filesystem_spec
Author: None
Author-email: None
License: BSD
Location: /home/hf/dev/promptsource/.venv/lib/python3.7/site-packages
Requires:
Required-by: datasets
```

With the same version of fsspec and `datasets==1.9.0`, I don't see this problem....

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
I can't even run `datasets-cli env` actually.., but here's my env:
- `datasets` version: 1.13.3
- Platform: Ubuntu 18.04
- Python version: 3.7.10
- PyArrow version: 3.0.0


`cast_column` makes audio decoding fail

Something isn't working

Thanks for reporting, @patrickvonplaten.

I think the issue is related to mp3 resampling, not to `cast_column`.

You can check that `cast_column` works OK with non-mp3 audio files:
```python
from datasets import load_dataset
import datasets
ds = load_dataset("arabic_speech_corpus", split="train")
ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))
print(ds[0]["audio"])
```

I'm fixing it.

## Describe the bug

After changing the sampling rate automatic decoding fails.

## Steps to reproduce the bug
```python
from datasets import load_dataset
import datasets

ds = load_dataset("common_voice", "ab", split="train")

ds = ds.cast_column("audio", datasets.features.Audio(sampling_rate=16_000))

print(ds[0]["audio"])  # <- this fails currently
```

yields:

```
TypeError: forward() takes 2 positional arguments but 4 were given
```

## Expected results
no failure

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 1.13.2 (master)
- Platform: Linux-5.11.0-1019-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0


Support loading a dataset from SQLite files

enhancement

New feature or request

As requested by @julien-c, we could eventually support loading a dataset from SQLite files, like it is the case for JSON/CSV files.


Error loading json dataset with multiple splits if keys in nested dicts have a different order

bug

Something isn't working

Hi,

even Pandas, which is less strict compared to PyArrow when it comes to reading JSON, doesn't support different orderings:
```python
import io
import pandas as pd

s = """
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
"""

buffer = io.StringIO(s)
df = pd.read_json(buffer, lines=True)

print(df.shape[0]) # 0
```

So we can't even fall back to Pandas in such cases.

It seems the only option is a script that recursively re-orders fields to enforce deterministic order:
```python
with open("train.json", "r") as fin:
    with open("train_reordered.json", "w") as fout:
        for line in fin:
            obj_jsonl = json.loads(line.strip())
            fout.write(json.dumps(obj_jsonl, sort_keys=True) + "\n")
```

## Describe the bug
Loading a json dataset with multiple splits that have nested dicts with keys in different order results in the error below.

If the keys in the nested dicts always have the same order or even if you just load a single split in which the nested dicts don't have the same order, everything works fine.

## Steps to reproduce the bug
Create two json files:

train.json
```
{"a": {"c": 8, "b": 5}}
{"a": {"b": 7, "c": 6}}
```

test.json
```
{"a": {"b": 1, "c": 2}}
{"a": {"b": 3, "c": 4}}
```

```python
from datasets import load_dataset
# Loading the files individually works (even though the keys in train.json don't have the same order)
load_dataset('json', data_files={"test": "test.json"})
load_dataset('json', data_files={"train": "train.json"})
# Loading both splits fails
load_dataset('json', data_files={"train": "train.json", "test": "test.json"})
```

## Expected results
Loading both splits should not give an error whether the nested dicts are have the same order or not.

## Actual results
```
>>> load_dataset('json', data_files={"train": "train.json", "test": "test.json"})
Using custom data configuration default-f1bc76fd07398c4c
Downloading and preparing dataset json/default to /home/dthulke/.cache/huggingface/datasets/json/default-f1bc76fd07398c4c/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 8839.42it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 477.82it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/builder.py", line 1159, in _prepare_split
    writer.write_table(table)
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/datasets/arrow_writer.py", line 428, in write_table
    pa_table = pa.Table.from_arrays([pa_table[name] for name in self._schema.names], schema=self._schema)
  File "pyarrow/table.pxi", line 1596, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 592, in pyarrow.lib._sanitize_arrays
  File "pyarrow/array.pxi", line 329, in pyarrow.lib.asarray
  File "pyarrow/table.pxi", line 277, in pyarrow.lib.ChunkedArray.cast
  File "/home/dthulke/venvs/venv_torch_transformers/lib/python3.6/site-packages/pyarrow/compute.py", line 297, in cast
    return call_function("cast", [arr], options)
  File "pyarrow/_compute.pyx", line 527, in pyarrow._compute.call_function
  File "pyarrow/_compute.pyx", line 337, in pyarrow._compute.Function.call
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 120, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from struct<b: int64, c: int64> to struct using function cast_struct
```

## Environment info
- `datasets` version: 1.13.2
- Platform: Linux-4.15.0-147-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyArrow version: 5.0.0


`blog_authorship_corpus` is broken

Something isn't working

Hi @fdtomasi, thanks for reporting.

You are right: the original host data URL does no longer exist.

I've contacted the authors of the dataset to ask them if they host this dataset in another URL.

Hi, @fdtomasi, the URL is fixed.

The fix is already in our master branch and it will be accessible in our next release.

In the meantime, you can include the fix if you install the `datasets` library from the master branch:
```
pip install -U git+ssh://git@github.com/huggingface/datasets.git@master#egg=datasest
```
or
```
pip install -U git+https://github.com/huggingface/datasets.git@master#egg=datasets
```

Awesome thank you so much for the quick fix!

## Describe the bug
The dataset `blog_authorship_corpus` is broken.
By bypassing the checksum checks, the loading does not return any error but the resulting dataset is empty.
I suspect it is because the data download url is broken (http://www.cs.biu.ac.il/~koppel/blogs/blogs.zip).

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("blog_authorship_corpus", split="train", download_mode='force_redownload')
```

## Expected results
No error.

## Actual results
```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
/tmp/ipykernel_5237/1729238701.py in <module>
      2 ds = load_dataset(
      3     "blog_authorship_corpus", split="train",
----> 4     download_mode='force_redownload'
      5 )

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, task, streaming, **config_kwargs)
   1115         ignore_verifications=ignore_verifications,
   1116         try_from_hf_gcs=try_from_hf_gcs,
-> 1117         use_auth_token=use_auth_token,
   1118     )
   1119

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    635                     if not downloaded_from_gcs:
    636                         self._download_and_prepare(
--> 637                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    638                         )
    639                     # Sync info

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    707         if verify_infos:
    708             verify_checksums(
--> 709                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    710             )
    711

/opt/conda/lib/python3.7/site-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['http://www.cs.biu.ac.il/~koppel/blogs/blogs.zip']
```

## Environment info
- `datasets` version: 1.13.2
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.10
- PyArrow version: 5.0.0


JNLPBA Dataset

Something isn't working

# Steps to reproduce

To reproduce:

```python
from datasets import load_dataset
dataset = load_dataset('jnlpba')

dataset['train'].features['ner_tags']
```
Output:
```python
Sequence(feature=ClassLabel(num_classes=3, names=['O', 'B', 'I'], names_file=None, id=None), length=-1, id=None)
```

Since I cannot create a branch here is the updated code:

```python

# coding=utf-8
# Copyright 2020 HuggingFace Datasets Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Lint as: python3
"""Introduction to the Bio-Entity Recognition Task at JNLPBA"""

import os

import datasets

logger = datasets.logging.get_logger(__name__)

_CITATION = """\
@inproceedings{kim2004introduction,
               title={Introduction to the bio-entity recognition task at JNLPBA},
               author={Kim, Jin-Dong and Ohta, Tomoko and Tsuruoka, Yoshimasa and Tateisi, Yuka and Collier, Nigel},
               booktitle={Proceedings of the international joint workshop on natural language processing in biomedicine and its applications},
               pages={70--75},
               year={2004},
               organization={Citeseer}
}
"""

_DESCRIPTION = """\
The data came from the GENIA version 3.02 corpus (Kim et al., 2003). This was formed from a controlled search
on MEDLINE using the MeSH terms human, blood cells and transcription factors. From this search 2,000 abstracts
were selected and hand annotated according to a small taxonomy of 48 classes based on a chemical classification.
Among the classes, 36 terminal classes were used to annotate the GENIA corpus.
"""

_HOMEPAGE = "http://www.geniaproject.org/shared-tasks/bionlp-jnlpba-shared-task-2004"
_TRAIN_URL = "http://www.nactem.ac.uk/GENIA/current/Shared-tasks/JNLPBA/Train/Genia4ERtraining.tar.gz"
_VAL_URL = 'http://www.nactem.ac.uk/GENIA/current/Shared-tasks/JNLPBA/Evaluation/Genia4ERtest.tar.gz'

_URLS = {
    "train": _TRAIN_URL,
    "val": _VAL_URL,
}

_TRAIN_DIRECTORY = "Genia4ERtraining"
_VAL_DIRECTORY = "Genia4ERtest"

_TRAIN_FILE = "Genia4ERtask1.iob2"
_VAL_FILE = "Genia4EReval1.iob2"

class JNLPBAConfig(datasets.BuilderConfig):
    """BuilderConfig for JNLPBA"""

    def __init__(self, **kwargs):
        """BuilderConfig for JNLPBA.
        Args:
          **kwargs: keyword arguments forwarded to super.
        """
        super(JNLPBAConfig, self).__init__(**kwargs)

class JNLPBA(datasets.GeneratorBasedBuilder):
    """JNLPBA dataset."""

    BUILDER_CONFIGS = [
        JNLPBAConfig(name="jnlpba", version=datasets.Version("1.0.0"), description="JNLPBA dataset"),
    ]

    def _info(self):
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            features=datasets.Features(
                {
                    "id": datasets.Value("string"),
                    "tokens": datasets.Sequence(datasets.Value("string")),
                    "ner_tags": datasets.Sequence(
                        datasets.features.ClassLabel(
                            names=[
                                'O',
                                'B-DNA',
                                'I-DNA',
                                'B-RNA',
                                'I-RNA',
                                'B-cell_line',
                                'I-cell_line',
                                'B-cell_type',
                                'I-cell_type',
                                'B-protein',
                                'I-protein',
                            ]
                        )
                    ),
                }
            ),
            supervised_keys=None,
            homepage=_HOMEPAGE,
            citation=_CITATION,
        )

    def _split_generators(self, dl_manager):
        downloaded_files = dl_manager.download_and_extract(_URLS)

        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN,
                                    gen_kwargs={"filepath": os.path.join(downloaded_files['train'], _TRAIN_FILE)}),
            datasets.SplitGenerator(name=datasets.Split.VALIDATION,
                                    gen_kwargs={"filepath": os.path.join(downloaded_files['val'], _VAL_FILE)})
        ]

    def _generate_examples(self, filepath):
        logger.info("⏳ Generating examples from = %s", filepath)
        with open(filepath, encoding="utf-8") as f:
            guid = 0
            tokens = []
            ner_tags = []
            for line in f:
                if line.startswith('###'):
                    continue
                if line == '' or line == '\n':
                    if tokens:
                        yield guid, {
                            "id": str(guid),
                            "tokens": tokens,
                            "ner_tags": ner_tags,
                        }
                        guid += 1
                        tokens = []
                        ner_tags = []
                else:
                    # tokens are tab separated
                    splits = line.split("\t")
                    #print(splits)
                    #print(len(splits))
                    if len(splits) < 2:
                        splits = splits[0].split()
                    tokens.append(splits[0])
                    ner_tags.append(splits[1].rstrip())
            # last example
            yield guid, {
                "id": str(guid),
                "tokens": tokens,
                "ner_tags": ner_tags,
            }
```

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
```

## Expected results
The dataset loading script for this dataset is incorrect. This is a biomedical dataset used for named entity recognition. The entities in the [script](https://github.com/huggingface/datasets/blob/master/datasets/jnlpba/jnlpba.py#L81-L83) are: O, B, and I.  The correct entities from the original data file are:

['O',
 'B-DNA',
 'I-DNA',
 'B-RNA',
 'I-RNA',
 'B-cell_line',
 'I-cell_line',
 'B-cell_type',
 'I-cell_type',
 'B-protein',
 'I-protein']

## Actual results
The dataset loader script needs to include the following NER names:

['O',
 'B-DNA',
 'I-DNA',
 'B-RNA',
 'I-RNA',
 'B-cell_line',
 'I-cell_line',
 'B-cell_type',
 'I-cell_type',
 'B-protein',
 'I-protein']

And the [data](https://github.com/huggingface/datasets/blob/master/datasets/jnlpba/jnlpba.py#L46) that is being pulled has been modified from the original dataset and does not include the original NER tags.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Removing label column in a text classification dataset yields to errors

Something isn't working

## Describe the bug

This looks like #3059 but it's not linked to the cache this time. Removing the `label` column from a text classification dataset and then performing any processing will result in an error.

To reproduce:
```py
from datasets import load_dataset
from transformers import AutoTokenizer

raw_datasets = load_dataset("imdb")
raw_datasets = raw_datasets.remove_columns("label")

model_checkpoint = "distilbert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
context_length = 128

def tokenize_pad_and_truncate(texts):
    return tokenizer(texts["text"], truncation=True, padding="max_length", max_length=context_length)

tokenized_datasets = raw_datasets.map(tokenize_pad_and_truncate, batched=True)
```

Traceback:

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-1-ba61bb32f786> in <module>
     12     return tokenizer(texts["text"], truncation=True, padding="max_length", max_length=context_length)
     13
---> 14 tokenized_datasets = raw_datasets.map(tokenize_pad_and_truncate, batched=True)

~/git/datasets/src/datasets/dataset_dict.py in map(self, function, with_indices, input_columns, batched, batch_size, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    500                     desc=desc,
    501                 )
--> 502                 for k, dataset in self.items()
    503             }
    504         )

~/git/datasets/src/datasets/dataset_dict.py in <dictcomp>(.0)
    500                     desc=desc,
    501                 )
--> 502                 for k, dataset in self.items()
    503             }
    504         )

~/git/datasets/src/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   2051                 new_fingerprint=new_fingerprint,
   2052                 disable_tqdm=disable_tqdm,
-> 2053                 desc=desc,
   2054             )
   2055         else:

~/git/datasets/src/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    501             self: "Dataset" = kwargs.pop("self")
    502         # apply actual function
--> 503         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    504         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    505         for dataset in datasets:

~/git/datasets/src/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    468         }
    469         # apply actual function
--> 470         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    471         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    472         # re-apply format to the output

~/git/datasets/src/datasets/fingerprint.py in wrapper(*args, **kwargs)
    404             # Call actual function
    405
--> 406             out = func(self, *args, **kwargs)
    407
    408             # Update fingerprint of in-place transforms + update in-place history of transforms

~/git/datasets/src/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2243             if os.path.exists(cache_file_name) and load_from_cache_file:
   2244                 logger.warning("Loading cached processed dataset at %s", cache_file_name)
-> 2245                 info = self.info.copy()
   2246                 info.features = features
   2247                 info.task_templates = None

~/git/datasets/src/datasets/info.py in copy(self)
    278
    279     def copy(self) -> "DatasetInfo":
--> 280         return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
    281
    282

~/git/datasets/src/datasets/info.py in __init__(self, description, citation, homepage, license, features, post_processed, supervised_keys, task_templates, builder_name, config_name, version, splits, download_checksums, download_size, post_processing_size, dataset_size, size_in_bytes)

~/git/datasets/src/datasets/info.py in __post_init__(self)
    177                 for idx, template in enumerate(self.task_templates):
    178                     if isinstance(template, TextClassification):
--> 179                         labels = self.features[template.label_column].names
    180                         self.task_templates[idx] = TextClassification(
    181                             text_column=template.text_column, label_column=template.label_column, labels=labels

KeyError: 'label'
```


VisibleDeprecationWarning when using `set_format("numpy")`

Something isn't working

I just opened a PR and I verified that the code you provided doesn't show any deprecation warning :)

Code to reproduce:

```
from datasets import load_dataset
dataset = load_dataset("glue", "mnli")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-cased')

def tokenize_function(dataset):
    return tokenizer(dataset['premise'])

tokenized_datasets = dataset.map(tokenize_function, batched=True, remove_columns=dataset['train'].features)

tokenized_datasets.set_format("numpy")

tokenized_datasets['train'][5:8]
```
Outputs:

```
python3.9/site-packages/datasets/formatting/formatting.py:167: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return np.array(array, copy=False, **self.np_array_kwargs)
```


Datasets with Audio feature raise error when loaded from cache due to _resampler parameter

Something isn't working

## Describe the bug
As reported by @patrickvonplaten, when loaded from the cache, datasets containing the Audio feature raise TypeError.

## Steps to reproduce the bug
```python
from datasets import load_dataset

# load first time works
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean")

# load from cache breaks
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean")
```

## Actual results
```
TypeError: __init__() got an unexpected keyword argument '_resampler'
```


Error related to timeout keyword argument

Something isn't working

## Describe the bug
As reported by @patrickvonplaten, a TypeError is raised when trying to load a dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean")
```

## Actual results
```
TypeError: dataset_info() got an unexpected keyword argument 'timeout'
```


Error when loading a metric

Something isn't working

## Describe the bug
As reported by @sgugger, after last release, exception is thrown when loading a metric.

## Steps to reproduce the bug
```python
from datasets import load_metric

metric = load_metric("squad_v2")
```

## Actual results
```
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-1-e612a8cab787> in <module>
      1 from datasets import load_metric
----> 2 metric = load_metric("squad_v2")

d:\projects\huggingface\datasets\src\datasets\load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, revision, script_version, **metric_init_kwargs)
   1336         )
   1337         revision = script_version
-> 1338     metric_module = metric_module_factory(
   1339         path, revision=revision, download_config=download_config, download_mode=download_mode
   1340     ).module_path

d:\projects\huggingface\datasets\src\datasets\load.py in metric_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, **download_kwargs)
   1237                 if not isinstance(e1, FileNotFoundError):
   1238                     raise e1 from None
-> 1239                 raise FileNotFoundError(
   1240                     f"Couldn't find a metric script at {relative_to_absolute_path(combined_path)}. "
   1241                     f"Metric '{path}' doesn't exist on the Hugging Face Hub either."

FileNotFoundError: Couldn't find a metric script at D:\projects\huggingface\datasets\squad_v2\squad_v2.py. Metric 'squad_v2' doesn't exist on the Hugging Face Hub either.
```


Import error installing with ppc64le

Something isn't working

This seems to be an issue with importing PyArrow so I posted the problem [here](https://issues.apache.org/jira/browse/ARROW-14323), and I'm closing this issue.

## Describe the bug
Installing the datasets library with a computer running with ppc64le seems to cause an issue when importing the datasets library.

```
python
Python 3.6.13 | packaged by conda-forge | (default, Sep 23 2021, 07:37:44)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datasets
Illegal instruction (core dumped)
```

Error when importing
`Illegal instruction (core dumped)`

## Steps to reproduce the bug
I get this error when installing the library by using conda.  I can't install with pip I believe because pyarrow only has the ppc64le library on conda forge
```
conda create --name transformers_py36_v2 python=3.6
conda activate transformers_py36_v2
conda install datasets
```

## Tracebacks
conda create --name transformers_py36_v2 python=3.6

```
Collecting package metadata (current_repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda

## Package Plan ##

  environment location: /p/home/gerryc/.conda/envs/transformers_py36_v2

  added / updated specs:
    - python=3.6

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-ppc64le::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-ppc64le::_openmp_mutex-4.5-1_gnu
  ca-certificates    conda-forge/linux-ppc64le::ca-certificates-2021.10.8-h1084571_0
  certifi            pkgs/main/linux-ppc64le::certifi-2020.12.5-py36h6ffa863_0
  ld_impl_linux-ppc~ conda-forge/linux-ppc64le::ld_impl_linux-ppc64le-2.36.1-ha35d02b_2
  libffi             conda-forge/linux-ppc64le::libffi-3.4.2-h3b9df90_4
  libgcc-ng          conda-forge/linux-ppc64le::libgcc-ng-11.2.0-h7698a5e_11
  libgomp            conda-forge/linux-ppc64le::libgomp-11.2.0-h7698a5e_11
  libstdcxx-ng       conda-forge/linux-ppc64le::libstdcxx-ng-11.2.0-habdf983_11
  libzlib            conda-forge/linux-ppc64le::libzlib-1.2.11-h339bb43_1013
  ncurses            conda-forge/linux-ppc64le::ncurses-6.2-hea85c5d_4
  openssl            conda-forge/linux-ppc64le::openssl-1.1.1l-h4e0d66e_0
  pip                conda-forge/noarch::pip-21.3-pyhd8ed1ab_0
  python             conda-forge/linux-ppc64le::python-3.6.13-h57873ef_2_cpython
  readline           conda-forge/linux-ppc64le::readline-8.1-h5c45dff_0
  setuptools         pkgs/main/linux-ppc64le::setuptools-58.0.4-py36h6ffa863_0
  sqlite             conda-forge/linux-ppc64le::sqlite-3.36.0-h4e2196e_2
  tk                 conda-forge/linux-ppc64le::tk-8.6.11-h41c6715_1
  wheel              conda-forge/noarch::wheel-0.37.0-pyhd8ed1ab_1
  xz                 conda-forge/linux-ppc64le::xz-5.2.5-h6eb9509_1
  zlib               conda-forge/linux-ppc64le::zlib-1.2.11-h339bb43_1013

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate transformers_py36_v2
#
# To deactivate an active environment, use
#
#     $ conda deactivate
```

conda activate transformers_py36_v2
conda install datasets
```
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda

## Package Plan ##

  environment location: /p/home/gerryc/.conda/envs/transformers_py36_v2

  added / updated specs:
    - datasets

The following NEW packages will be INSTALLED:

  abseil-cpp         conda-forge/linux-ppc64le::abseil-cpp-20210324.2-h3b9df90_0
  aiohttp            conda-forge/linux-ppc64le::aiohttp-3.7.4.post0-py36hc33305d_0
  arrow-cpp          conda-forge/linux-ppc64le::arrow-cpp-5.0.0-py36hf9cf308_8_cpu
  async-timeout      conda-forge/noarch::async-timeout-3.0.1-py_1000
  attrs              conda-forge/noarch::attrs-21.2.0-pyhd8ed1ab_0
  aws-c-cal          conda-forge/linux-ppc64le::aws-c-cal-0.5.11-hb3fac3d_0
  aws-c-common       conda-forge/linux-ppc64le::aws-c-common-0.6.2-h4e0d66e_0
  aws-c-event-stream conda-forge/linux-ppc64le::aws-c-event-stream-0.2.7-h76da5f2_13
  aws-c-io           conda-forge/linux-ppc64le::aws-c-io-0.10.5-hf6a6c7c_0
  aws-checksums      conda-forge/linux-ppc64le::aws-checksums-0.1.11-hfe76d68_7
  aws-sdk-cpp        conda-forge/linux-ppc64le::aws-sdk-cpp-1.8.186-h90855e8_3
  brotlipy           conda-forge/linux-ppc64le::brotlipy-0.7.0-py36hc33305d_1001
  bzip2              conda-forge/linux-ppc64le::bzip2-1.0.8-h4e0d66e_4
  c-ares             conda-forge/linux-ppc64le::c-ares-1.17.2-h4e0d66e_0
  cffi               conda-forge/linux-ppc64le::cffi-1.14.6-py36h021ab3c_1
  chardet            conda-forge/linux-ppc64le::chardet-4.0.0-py36h270354c_1
  colorama           conda-forge/noarch::colorama-0.4.4-pyh9f0ad1d_0
  cryptography       conda-forge/linux-ppc64le::cryptography-3.4.7-py36hc71b123_0
  dataclasses        conda-forge/noarch::dataclasses-0.8-pyh787bdff_2
  datasets           conda-forge/noarch::datasets-1.12.1-pyhd8ed1ab_1
  dill               conda-forge/noarch::dill-0.3.4-pyhd8ed1ab_0
  filelock           conda-forge/noarch::filelock-3.3.0-pyhd8ed1ab_0
  fsspec             conda-forge/noarch::fsspec-2021.10.0-pyhd8ed1ab_0
  gflags             conda-forge/linux-ppc64le::gflags-2.2.2-hb209c28_1004
  glog               conda-forge/linux-ppc64le::glog-0.5.0-h4040248_0
  grpc-cpp           conda-forge/linux-ppc64le::grpc-cpp-1.40.0-h2bf711c_2
  huggingface_hub    conda-forge/noarch::huggingface_hub-0.0.19-pyhd8ed1ab_0
  idna               conda-forge/noarch::idna-2.10-pyh9f0ad1d_0
  idna_ssl           conda-forge/noarch::idna_ssl-1.0.0-0
  importlib-metadata conda-forge/linux-ppc64le::importlib-metadata-4.8.1-py36h270354c_0
  importlib_metadata conda-forge/noarch::importlib_metadata-4.8.1-hd8ed1ab_0
  krb5               conda-forge/linux-ppc64le::krb5-1.19.2-haf43566_2
  libblas            conda-forge/linux-ppc64le::libblas-3.9.0-11_linuxppc64le_openblas
  libbrotlicommon    conda-forge/linux-ppc64le::libbrotlicommon-1.0.9-h4e0d66e_5
  libbrotlidec       conda-forge/linux-ppc64le::libbrotlidec-1.0.9-h4e0d66e_5
  libbrotlienc       conda-forge/linux-ppc64le::libbrotlienc-1.0.9-h4e0d66e_5
  libcblas           conda-forge/linux-ppc64le::libcblas-3.9.0-11_linuxppc64le_openblas
  libcurl            conda-forge/linux-ppc64le::libcurl-7.79.1-he415e40_1
  libedit            conda-forge/linux-ppc64le::libedit-3.1.20191231-h41a240f_2
  libev              conda-forge/linux-ppc64le::libev-4.33-h6eb9509_1
  libevent           conda-forge/linux-ppc64le::libevent-2.1.10-h97db324_4
  libgfortran-ng     conda-forge/linux-ppc64le::libgfortran-ng-11.2.0-hfdc3801_11
  libgfortran5       conda-forge/linux-ppc64le::libgfortran5-11.2.0-he58fbb4_11
  liblapack          conda-forge/linux-ppc64le::liblapack-3.9.0-11_linuxppc64le_openblas
  libnghttp2         conda-forge/linux-ppc64le::libnghttp2-1.43.0-h42039ad_1
  libopenblas        conda-forge/linux-ppc64le::libopenblas-0.3.17-pthreads_h486567c_1
  libprotobuf        conda-forge/linux-ppc64le::libprotobuf-3.18.1-h690f14c_0
  libssh2            conda-forge/linux-ppc64le::libssh2-1.10.0-ha5a9321_2
  libthrift          conda-forge/linux-ppc64le::libthrift-0.15.0-h54f692e_1
  libutf8proc        conda-forge/linux-ppc64le::libutf8proc-2.6.1-h4e0d66e_0
  lz4-c              conda-forge/linux-ppc64le::lz4-c-1.9.3-h3b9df90_1
  multidict          conda-forge/linux-ppc64le::multidict-5.2.0-py36hc33305d_0
  multiprocess       conda-forge/linux-ppc64le::multiprocess-0.70.12.2-py36hc33305d_0
  numpy              conda-forge/linux-ppc64le::numpy-1.19.5-py36h86665d4_1
  orc                conda-forge/linux-ppc64le::orc-1.7.0-hae6b4bd_0
  packaging          conda-forge/noarch::packaging-21.0-pyhd8ed1ab_0
  pandas             conda-forge/linux-ppc64le::pandas-1.1.5-py36hab1a6e6_0
  parquet-cpp        conda-forge/noarch::parquet-cpp-1.5.1-2
  pyarrow            conda-forge/linux-ppc64le::pyarrow-5.0.0-py36h7a46c7e_8_cpu
  pycparser          conda-forge/noarch::pycparser-2.20-pyh9f0ad1d_2
  pyopenssl          conda-forge/noarch::pyopenssl-21.0.0-pyhd8ed1ab_0
  pyparsing          conda-forge/noarch::pyparsing-2.4.7-pyh9f0ad1d_0
  pysocks            conda-forge/linux-ppc64le::pysocks-1.7.1-py36h270354c_3
  python-dateutil    conda-forge/noarch::python-dateutil-2.8.2-pyhd8ed1ab_0
  python-xxhash      conda-forge/linux-ppc64le::python-xxhash-2.0.2-py36hc33305d_0
  python_abi         conda-forge/linux-ppc64le::python_abi-3.6-2_cp36m
  pytz               conda-forge/noarch::pytz-2021.3-pyhd8ed1ab_0
  pyyaml             conda-forge/linux-ppc64le::pyyaml-5.4.1-py36hc33305d_1
  re2                conda-forge/linux-ppc64le::re2-2021.09.01-h3b9df90_0
  requests           conda-forge/noarch::requests-2.25.1-pyhd3deb0d_0
  s2n                conda-forge/linux-ppc64le::s2n-1.0.10-h97db324_0
  six                conda-forge/noarch::six-1.16.0-pyh6c4a22f_0
  snappy             conda-forge/linux-ppc64le::snappy-1.1.8-hb209c28_3
  tqdm               conda-forge/noarch::tqdm-4.62.3-pyhd8ed1ab_0
  typing-extensions  conda-forge/noarch::typing-extensions-3.10.0.2-hd8ed1ab_0
  typing_extensions  conda-forge/noarch::typing_extensions-3.10.0.2-pyha770c72_0
  urllib3            conda-forge/noarch::urllib3-1.26.7-pyhd8ed1ab_0
  xxhash             conda-forge/linux-ppc64le::xxhash-0.8.0-h4e0d66e_3
  yaml               conda-forge/linux-ppc64le::yaml-0.2.5-h6eb9509_0
  yarl               conda-forge/linux-ppc64le::yarl-1.6.3-py36hc33305d_2
  zipp               conda-forge/noarch::zipp-3.6.0-pyhd8ed1ab_0
  zstd               conda-forge/linux-ppc64le::zstd-1.5.0-h65c4b1a_0

The following packages will be UPDATED:

  certifi            pkgs/main::certifi-2020.12.5-py36h6ff~ --> conda-forge::certifi-2021.5.30-py36h270354c_0

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Red Hat Enterprise Linux 8.2 (Ootpa)
- Python version: 3.6
- PyArrow version: pyarrow - 5.0.0 - py36h7a46c7e_8_cpu - conda-forge

Any help would be appreciated! I've been struggling on installing datasets on this machine.


Custom plain text dataset, plain json dataset and plain csv dataset are remove from datasets template folder

dataset request

Requesting to add a new dataset

Hi @zixiliuUSC,

As explained in the documentation (https://huggingface.co/docs/datasets/loading.html#json), we support loading any dataset in JSON (as well as CSV, text, Parquet) format:
```python
ds = load_dataset('json', data_files='my_file.json')
```

## Adding a Dataset
- **Name:** text, json, csv
- **Description:** I am developing a customized dataset loading script. The problem is  mainly about my custom dataset is seperate into many files and I only find a dataset loading template in [https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py](https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py) that can handle my circumstance. I'm afraid these templates are too old to use. Could you re-add these three templates to current master branch?


bug

Something isn't working

## Describe the bug
After commit 9353fc863d0c99ab0427f83cc5a4f04fcf52f1df, the CI fails on Windows  with FileNotFoundError when stting up s3_base fixture. See: https://app.circleci.com/pipelines/github/huggingface/datasets/8151/workflows/5db8d154-badd-4d3d-b202-ca7a318997a2/jobs/50321

Error summary:
```
ERROR tests/test_arrow_dataset.py::test_dummy_dataset_serialize_s3 - FileNotF...
ERROR tests/test_dataset_dict.py::test_dummy_dataset_serialize_s3 - FileNotFo...
```

Stack trace:
```
______________ ERROR at setup of test_dummy_dataset_serialize_s3 ______________
[gw0] win32 -- Python 3.6.8 C:\tools\miniconda3\python.exe

    @pytest.fixture()
    def s3_base():
        # writable local S3 system
        import shlex
        import subprocess

        # Mocked AWS Credentials for moto.
        old_environ = os.environ.copy()
        os.environ.update(S3_FAKE_ENV_VARS)

>       proc = subprocess.Popen(shlex.split("moto_server s3 -p %s" % s3_port))

tests\s3_fixtures.py:32:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\tools\miniconda3\lib\subprocess.py:729: in __init__
    restore_signals, start_new_session)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <subprocess.Popen object at 0x0000012BB8A4B908>
args = 'moto_server s3 -p 5555', executable = None, preexec_fn = None
close_fds = True, pass_fds = (), cwd = None, env = None
startupinfo = <subprocess.STARTUPINFO object at 0x0000012BB8177630>
creationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = -1
c2pwrite = -1, errread = -1, errwrite = -1, unused_restore_signals = True
unused_start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       unused_restore_signals, unused_start_new_session):
        """Execute program (MS Windows version)"""

        assert not pass_fds, "pass_fds not supported on Windows."

        if not isinstance(args, str):
            args = list2cmdline(args)

        # Process startup details
        if startupinfo is None:
            startupinfo = STARTUPINFO()
        if -1 not in (p2cread, c2pwrite, errwrite):
            startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES
            startupinfo.hStdInput = p2cread
            startupinfo.hStdOutput = c2pwrite
            startupinfo.hStdError = errwrite

        if shell:
            startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
            startupinfo.wShowWindow = _winapi.SW_HIDE
            comspec = os.environ.get("COMSPEC", "cmd.exe")
            args = '{} /c "{}"'.format (comspec, args)

        # Start the process
        try:
            hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                                     # no special security
                                     None, None,
                                     int(not close_fds),
                                     creationflags,
                                     env,
                                     os.fspath(cwd) if cwd is not None else None,
>                                    startupinfo)
E                                    FileNotFoundError: [WinError 2] The system cannot find the file specified

C:\tools\miniconda3\lib\subprocess.py:1017: FileNotFoundError
```


Make `interleave_datasets` more robust

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Right now there are few hiccups using `interleave_datasets`. Interleaved dataset iterates until the smallest dataset completes it's iterator. In this way larger datasets may not complete full epoch of iteration.
It creates new problems in calculation of epoch since there are no way to track which dataset from `interleave_datasets` completes how many epoch.

**Describe the solution you'd like**
For `interleave_datasets` module,
- [ ] Add a boolean argument `--stop-iter` in `interleave_datasets` that enables dataset to either iterate infinite amount of time or not. That means it should not return `StopIterator` exception in case `--stop-iter=False`.
- [ ] Internal list variable `iter_cnt` that explains how many times (in steps/epochs) each dataset iterates at a given point.
- [ ] Add an argument `--max-iter` (list type) that explain maximum amount of time each of the dataset can iterate. After complete  `--max-iter` of one dataset, other dataset should continue sampling and when all the dataset finish their respective `--max-iter`, only then return `StopIterator`

Note: I'm new to `datasets` api. May be these features are already there in the datasets.

Since multitask training is the latest trends, I believe this feature would make the `datasets` api more popular.

@lhoestq


Windows CI is unable to test streaming properly because of SSL issues

I think this problem is already fixed:
```python
In [4]: import fsspec
   ...:
   ...: url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"
   ...:
   ...: fsspec.open(url).open()
Out[4]: <File-like object HTTPFileSystem, https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattribu
```

No I'm still having this issue on my windows, and so does the CI

In https://github.com/huggingface/datasets/pull/3041 the windows tests were skipped because of SSL issues with moon-staging.huggingface.co:443

The issue appears only on windows with asyncio. On Linux it works. With requests it works as well. And with the production environment huggingface.co it also works.

to reproduce on windows:
```python
import fsspec

# use any URL to a file in a dataset repo
url = "https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes"

fsspec.open(url).open()
```
raises
```python
FileNotFoundError: https://moon-staging.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/my-dataset-16242824690709/resolve/main/.gitattributes
```
because of
```python
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host moon-staging.huggingface.co:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)')]
```


Feature request : add leave=True to dataset.map to enable tqdm nested bars (and whilst we're at it couldn't we get a way to access directly tqdm underneath?)

New feature or request

@lhoestq, @albertvillanova can we have `**tqdm_kwargs` in `map`? If there are any fields that are important to our tqdm (like iterable or unit), we can pop them before initialising the tqdm object so as to avoid duplicity.

Hi ! Sounds like a good idea :)

Also I think it would be better to have this as an actual parameters instead of kwargs to make it clearer

**A clear and concise description of what you want to happen.**

It would be so nice to be able to nest HuggingFace `Datasets.map() ` progress bars in the grander scheme of things and whilst we're at it why not other functions.

**Describe alternatives you've considered**

By the way is there not a way to directly interact with underlying tqdm module ? **kwargs-ish?

**Additional context**

Furthering tqdm integration #2374 and huggingface/transformers#11797 solutioned by huggingface/transformers#12226 provided with tqdm description as `desc=`

@sgugger @bhavitvyamalik


load_dataset('openwebtext') yields "Compressed file ended before the end-of-stream marker was reached"

bug

Something isn't working

Hi @RylanSchaeffer, thanks for reporting.

I'm sorry, but I was not able to reproduce your problem.

Normally, the reason for this type of error is that, during your download of the data files, this was not fully complete.

Could you please try to load the dataset again but forcing its redownload? Please use:
```python
dataset = load_dataset("openwebtext", download_mode="FORCE_REDOWNLOAD")
```

Let me know if the problem persists.

I close this issue for the moment. Feel free to re-open it again if the problem persists.

## Describe the bug
When I try `load_dataset('openwebtext')`, I receive a "EOFError: Compressed file ended before the end-of-stream marker was reached" error.

## Steps to reproduce the bug
```
from datasets import load_dataset
dataset = load_dataset('openwebtext')
```

## Expected results
I expect the `dataset` variable to be properly constructed.

## Actual results

```
File "/home/rschaef/CoCoSci-Language-Distillation/distillation_v2/ratchet_learning/tasks/base.py", line 37, in create_dataset
    dataset_str,
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/load.py", line 1117, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/builder.py", line 637, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/builder.py", line 704, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/rschaef/.cache/huggingface/modules/datasets_modules/datasets/openwebtext/85b3ae7051d2d72e7c5fdf6dfb462603aaa26e9ed506202bf3a24d261c6c40a1/openwebtext.py", line 61, in _split_generators
    dl_dir = dl_manager.download_and_extract(_URL)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/download_manager.py", line 261, in extract
    partial(cached_path, download_config=download_config), path_or_paths, num_proc=num_proc, disable_tqdm=False
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 197, in map_nested
    return function(data_struct)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 316, in cached_path
    output_path, force_extract=download_config.force_extract
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/extract.py", line 40, in extract
    self.extractor.extract(input_path, output_path, extractor=extractor)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/extract.py", line 179, in extract
    return extractor.extract(input_path, output_path)
  File "/home/rschaef/CoCoSci-Language-Distillation/cocosci/lib/python3.6/site-packages/datasets/utils/extract.py", line 53, in extract
    tar_file.extractall(output_path)
  File "/usr/lib/python3.6/tarfile.py", line 2010, in extractall
    numeric_owner=numeric_owner)
  File "/usr/lib/python3.6/tarfile.py", line 2052, in extract
    numeric_owner=numeric_owner)
  File "/usr/lib/python3.6/tarfile.py", line 2122, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/usr/lib/python3.6/tarfile.py", line 2171, in makefile
    copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
  File "/usr/lib/python3.6/tarfile.py", line 249, in copyfileobj
    buf = src.read(bufsize)
  File "/usr/lib/python3.6/lzma.py", line 200, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.6/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/usr/lib/python3.6/_compression.py", line 99, in read
    raise EOFError("Compressed file ended before the "
python-BaseException
EOFError: Compressed file ended before the end-of-stream marker was reached
```

## Environment info
- `datasets` version: 1.12.1
- Platform: Linux-4.4.0-173-generic-x86_64-with-Ubuntu-16.04-xenial
- Python version: 3.6.10
- PyArrow version: 5.0.0