DataCollatorWithPadding: TypeError

Something isn't working

>
> Input:
>
> ```
> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
> data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
> ```
>
> Output:
>
> ```
> TypeError                                 Traceback (most recent call last)
> /tmp/ipykernel_42/1563280798.py in <module>
>       1 checkpoint = 'bert-base-uncased'
>       2 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
> ----> 3 data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="pt")
> TypeError: __init__() got an unexpected keyword argument 'return_tensors'
> ```
>

The issue is due to the older version of transformers and datasets. It has been resolved by upgrading their versions.

`# upgrade transformers and datasets to latest versions`
`!pip install --upgrade transformers`
`!pip install --upgrade datasets`

Cheers!

Hi,
I am following the HuggingFace course. I am now at Fine-tuning [https://huggingface.co/course/chapter3/3?fw=tf](https://huggingface.co/course/chapter3/3?fw=tf).  When I set up `DataCollatorWithPadding` as following I got an error while trying to reproduce the course code in Kaggle. This error occurs with either a CPU-only-device or a GPU-device.

Input:
```checkpoint = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
```

Output:
```---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_42/1563280798.py in <module>
      1 checkpoint = 'bert-base-uncased'
      2 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
----> 3 data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="pt")
TypeError: __init__() got an unexpected keyword argument 'return_tensors'
```

When I call `help` method, it too confirms that there is no argument `return_tensors`.
Input:
```
help(DataCollatorWithPadding.__init__)
```
Output:
```
Help on function __init__ in module transformers.data.data_collator:

__init__(self, tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, padding: Union[bool, str, transformers.file_utils.PaddingStrategy] = True, max_length: Union[int, NoneType] = None, pad_to_multiple_of: Union[int, NoneType] = None) -> None
```

But, the source file *[Data Collator - docs](https://huggingface.co/transformers/main_classes/data_collator.html#datacollatorwithpadding)* says that there is such an argument. By default, it returns Pytorch tensors while I need TF tensors.

Where do I miss?
Please help me.


fix old_val typo in f-string

This PR is to correct a typo in #3277  that @Carlosbogo revieled in a comment.

Related closed issue : #3257

Sorry about that 😅.


Add wikipedia tags

Add the missing tags to the wikipedia dataset card.

I also added the missing languages code in our language codes list.

This should also fix the code snippet that is presented on the Hub to load the dataset: fix https://github.com/huggingface/datasets/issues/3292


❓ Dataset loading script from Hugging Face Hub

dataset request

Requesting to add a new dataset

dataset-viewer

Related to the dataset viewer on huggingface.co

Hi ! In the next version of `datasets`, your train and test splits will be correctly separated (changes from #3027) if you create a dataset repository with only your CSV files.

Also it seems that you overwrite the `data_files` and `data_dir` arguments in your code, when you instantiate the AGNewsConfig objects. Those parameters are not necessary since you already know which files you want to load.

You can find an example on how to specify which file the dataset has to download in this [example script](https://huggingface.co/datasets/lhoestq/custom_squad/blob/main/custom_squad.py#L101-L107):
```python
_URLS = {
    "train": "train-v1.1.json",  # you can use a URL or a relative path from the python script to your file in the repository
    "dev": "dev-v1.1.json",
}
```
```python
    def _split_generators(self, dl_manager):
        downloaded_files = dl_manager.download_and_extract(_URLS)

        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"filepath": downloaded_files["train"]}),
            datasets.SplitGenerator(name=datasets.Split.VALIDATION, gen_kwargs={"filepath": downloaded_files["dev"]}),
        ]
```

Also I think the viewer will be updated when you fix the dataset script, let me know if it doesn't

Hi @lhoestq,

Thanks a lot for the super quick answer!

Your suggestion solves my issue. I am now able to load the dataset properly 🚀
However, the dataviewer is not working yet.

Really, thanks a lot for your help and consideration!

Best,
Pietro

Great ! We'll take a look at the viewer to fix it

@lhoestq  I think I am having a related problem.
My call to load_dataset() looks like this:

```
    datasets = load_dataset(
        os.path.abspath(layoutlmft.data.datasets.xfun.__file__),
        f"xfun.{data_args.lang}",
        additional_langs=data_args.additional_langs,
        keep_in_memory=True,
    )

```

My _split_generation code is:

```
    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""

        downloaded_file = dl_manager.download_and_extract("https://guillaumejaume.github.io/FUNSD/dataset.zip")
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN, gen_kwargs={"filepath": f"{downloaded_file}/dataset/training_data/"}
            ),
            datasets.SplitGenerator(
                name=datasets.Split.TEST, gen_kwargs={"filepath": f"{downloaded_file}/dataset/testing_data/"}
            ),
        ]

```
However I get the error "TypeError: _generate_examples() got an unexpected keyword argument 'filepath'"
The path looks right and I see the data in the path so I think the only problem I have is that it doesn't like the key "filepath".  However, the documentation (example [here](https://huggingface.co/datasets/lhoestq/custom_squad/blob/main/custom_squad.py#L101-L107)) seems to show that this is the correct parameter.

Here is the full stack trace:

```
Downloading and preparing dataset xfun/xfun.en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /Users/caseygre/.cache/huggingface/datasets/xfun/xfun.en/0.0.0/96b8cb7c57f6f822f0ab37ae3be7b82d84ac57062e774c9361ccf0a4b9ef61cc...
Traceback (most recent call last):
  File "/Users/caseygre/PycharmProjects/aegis-ml-new/unilm/venv-LayoutLM/lib/python3.9/site-packages/datasets/builder.py", line 574, in download_and_prepare
    self._download_and_prepare(
  File "/Users/caseygre/PycharmProjects/aegis-ml-new/unilm/venv-LayoutLM/lib/python3.9/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/caseygre/PycharmProjects/aegis-ml-new/unilm/venv-LayoutLM/lib/python3.9/site-packages/datasets/builder.py", line 975, in _prepare_split
    generator = self._generate_examples(**split_generator.gen_kwargs)
TypeError: _generate_examples() got an unexpected keyword argument 'filepath'
python-BaseException
```

Hi ! The `gen_kwargs` dictionary is passed to `_generate_examples`, so in your case it must be defined this way:
```python
def _generate_examples(self, filepath):
    ...
```

And here is an additional tip: you can use `os.path.join(downloaded_file, "dataset/testing_data")` instead of `f"downloaded_file}/dataset/testing_data/"` to get compatibility with Windows and streaming.

Indeed Windows uses a backslash separator, not a slash, and streaming uses chained URLs (like `zip://dataset/testing_data::https://https://guillaumejaume.github.io/FUNSD/dataset.zip` for example)

Thanks for you quick reply @lhoestq and so sorry for my very delayed response.
We have gotten around the error another way but I will try to duplicate this when I can.  We may have had "filepaths" instead of "filepath" in our def of _generate_examples() and not noticed the difference.  If I find a more useful answer for others I will add to this ticket so they know what the issue was.
Note: we do have our own _generate_examples() defined with the same def as Quentin has.  (But one version does have "filepaths".)

Hi there,

I am trying to add my custom `ag_news` with its own loading script on the Hugging Face datasets hub. In particular, I would like to test the addition of a second configuration to the existing `ag_news` dataset. Once it works in my hub, I plan to make a PR to the original dataset. However, in trying to do so I have encountered certain problems as detailed below.

Issues I have encountered:
- Without a loading script, the train and test files are loaded together into a unique `dataset.Dataset` -> so I wrote a loading script. Also, I need a loading script otherwise I cannot specify multiple configurations
- Once my loading script is working locally, I do not manage to make it work on the hub. In particular, I would like to be able to load the dataset like this
```python
load_dataset("pietrolesci/ag_news", name="my_configuration")
```

Apparently, the `load_dataset` is able to pick up the loading script from the hub and run it. However, it errors because it is unable to find the files. The structure of my hub repo is the following
```
ag_news.py
train.csv
test.csv
```
and the loading script I specify  `data_dir=Path(__file__).parent` and `data_files=DataFilesDict({"train": "train.csv", "test": "test.csv"})`. In the documentation I could not find info regarding loading a dataset from the hub using a loading script present on the hub.

Any suggestion is very much appreciated.

Best,
Pietro

Link to the hub repo: https://huggingface.co/datasets/pietrolesci/ag_news

BONUS: how can I make the data viewer work in this specific case? :)


Add option to find unique elements in nested sequences when calling `Dataset.unique`

enhancement

New feature or request

It would be nice to have an option to flatten nested sequences to find unique elements stored in them when calling `Dataset.unique`. Currently, `Dataset.unique` only supports finding unique sequences and not unique elements in that situation.


Agnews dataset viewer is not working

Hi ! Thanks for reporting
We've already fixed the code that generates the preview for this dataset, we'll release the fix soon :)

## Dataset viewer issue for '*name of the dataset*'

**Link:** https://huggingface.co/datasets/ag_news

Hi there, the `ag_news` dataset viewer is not working.

Am I the one who added this dataset? No


.map() cache is wrongfully reused - only happens when the mapping function is imported

bug

Something isn't working

## Describe the bug
When `.map` is used with a mapping function that is imported, the cache is reused even if the mapping function has been modified.
The reason for this is that `dill` that is used for creating the fingerprint [pickles imported functions by reference](https://stackoverflow.com/a/67851411).

I guess it is not a widespread case, but it can still lead to unwanted results unnoticeably.

## Steps to reproduce the bug
Create files `a.py` and `b.py`:
```python
# a.py
from datasets import load_dataset

def main():
    squad = load_dataset("squad")
    squad.map(mapping_func, batched=True)

def mapping_func(examples):
    ID_LENGTH = 4
    examples["id"] = [id_[:ID_LENGTH] for id_ in examples["id"]]
    return examples

if __name__ == "__main__":
    main()
```
```python
# b.py
from datasets import load_dataset
from a import mapping_func

def main():
    squad = load_dataset("squad")
    squad.map(mapping_func, batched=True)

if __name__ == "__main__":
    main()
```
Run `python b.py` twice: In the first run you will see tqdm bars showing that the data is processed, and in the second run you will see "Loading cached processed dataset at...".
Now change `ID_LENGTH` to another number in order to change the mapping function, and run `python b.py` again. You'll see that `.map` loads from the cache the result of the previous mapping function.

## Expected results
Run `python a.py` twice: In the first run you will see tqdm bars showing that the data is processed, and in the second run you will see "Loading cached processed dataset at...".
Now change `ID_LENGTH` to another number in order to change the mapping function, and run `python a.py` again. You'll see that the dataset is being processed and that there's no reuse of the previous mapping function result.

## Workaround
Put the mapping function inside a dummy class as a static method:
```python
# a.py
class MappingFuncClass:
    @staticmethod
    def mapping_func(examples):
        ID_LENGTH = 4
        examples["id"] = [id_[:ID_LENGTH] for id_ in examples["id"]]
        return examples
```
```python
# b.py
from datasets import load_dataset
from a import MappingFuncClass

def main():
    squad = load_dataset("squad")
    squad.map(MappingFuncClass.mapping_func, batched=True)

if __name__ == "__main__":
    main()
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 4.0.1


Fix temporary dataset_path creation for URIs related to remote fs

Hi ! Thanks for the fix :)

I think this should be `extract_path_from_uri` 's responsibility to strip the extra `/` from a badly formatted path like `hdfs:///absolute/path` (or raise an error). Do you think you can simply do the changes in `extract_path_from_uri` ? This way this fix will be available for all the other parts of the lib that need to extract the inner path from an URI of a remote filesystem

Then we can also keep your test cases but simply apply them to `extract_path_from_uri` instead

Hi @lhoestq! No problem! Thanks for your interest! :)

I think stripping the 3rd `/` in `hdfs:///absolute/path` inside `extract_path_from_uri` is not the solution. When I provide `hdfs:///absolute/path` to `extract_path_from_uri` we want `/absolute/path` to be returned, as it does now (at least in the case of URIs with `hdfs` schemas, for `s3` is different as it should start with a bucket name).

The problem comes in line 1041 in the original code below:

https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1038-L1042

Lets assume the following parameters for line 1041 after `extract_path_from_uri` has removed the `hdfs` schema part and the `://` from `hdfs:///absolute/path`, and `get_temporary_cache_files_directory()` returns `/tmp/a1b2b3c4`, as it is shown below:

```python
src_dataset_path = '/absolute/path'
tmp_dir = '/tmp/a1b2b3c4'
dataset_path = Path(tmp_dir, src_dataset_path)
```

After passing those paths to the `Path` object, `dataset_path` contains only `/absolute/path`; that is, it has lost the temporary directory path. This is because, when two (or more) absolute paths are passed to the `Path` function, only the last one is taken. However, if the contents of those variables are:

```python
src_dataset_path = 'relative/path'
tmp_dir = '/tmp/a1b2b3c4'
dataset_path = Path(tmp_dir, src_dataset_path)
```

then `dataset_path` contains `/tmp/a1b2b3c4/relative/path` as expected.

Absolute paths are allowed in hdfs URIs, so that's why I added the extra function `build_local_temp_path` in the PR; so in case the second argument is an absolute path, it still will create the correct absolute path by concatenating the temp dir and the path passed by converting it to a relative path (and it also works for windows paths too.) It also allows to add the tests, checking that the main combinations are ok.

I've checked all the places where the result of `extract_path_from_uri` is used, and as far as I've seen this is the only place where it is concatenated with another possible absolute path, so no need to add `build_local_temp_path` anywhere else.

This aims to close #3295


Temporary dataset_path for remote fs URIs not built properly in arrow_dataset.py::load_from_disk

Something isn't working

Hi ! Good catch and thanks for opening a PR :)

I just responded in your PR

## Describe the bug
When trying to build a temporary dataset path from a remote URI in this block of code:

https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1038-L1042

the result is not the expected when passing an absolute path in an URI like `hdfs:///absolute/path`.

## Steps to reproduce the bug
```python
dataset_path = "hdfs:///absolute/path"
src_dataset_path = extract_path_from_uri(dataset_path)
tmp_dir = get_temporary_cache_files_directory()
dataset_path = Path(tmp_dir, src_dataset_path)
print(dataset_path)
```

## Expected results
With the code above, we would expect a value in `dataset_path` similar to:
`/tmp/tmpnwxyvao5/absolute/path`

## Actual results
However, we get a `dataset_path` value like:
`/absolute/path`

This is because this line here: https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1041
returns the last absolute path when two absolute paths (the one in `tmp_dir` and the one extracted from the URI in `src_dataset_path`) are passed as arguments.

## Environment info
- `datasets` version: 1.13.3
- Platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 5.0.0


Add Natural Adversarial Objects dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Natural Adversarial Objects (NAO)
- **Description:** Natural Adversarial Objects (NAO) is a new dataset to evaluate the robustness of object detection models. NAO contains 7,934 images and 9,943 objects that are unmodified and representative of real-world scenarios, but cause state-of-the-art detection models to misclassify with high confidence.
- **Paper:** https://arxiv.org/abs/2111.04204v1
- **Data:** https://drive.google.com/drive/folders/15P8sOWoJku6SSEiHLEts86ORfytGezi8
- **Motivation:** interesting object detection dataset useful for miscclassifications

cc @NielsRogge

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Pin version exclusion for Markdown

As Markdown version 3.3.5 has a bug, it is better to exclude it in case the users have it previously installed in their environment.

Related to #3289, #3286.


Not able to load 'wikipedia' dataset

bug

Something isn't working

Hi ! Indeed it looks like the code snippet on the Hugging face Hub doesn't show the second parameter

![image](https://user-images.githubusercontent.com/42851186/142649237-45ba55c5-1a64-4c30-8692-2c8120572f92.png)

Thanks for reporting, I'm taking a look

## Describe the bug
I am following the instruction for loading the wikipedia dataset using datasets. However getting the below error.

## Steps to reproduce the bug
from datasets import load_dataset
dataset = load_dataset("wikipedia")
```

## Expected results
A clear and concise description of the expected results.

## Actual results
~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    339                         "Config name is missing."
    340                         "\nPlease pick one among the available configs: %s" % list(self.builder_configs.keys())
--> 341                         + "\nExample of usage:\n\t`{}`".format(example_of_usage)
    342                     )
    343                 builder_config = self.BUILDER_CONFIGS[0]

ValueError: Config name is missing.
Please pick one among the available configs: ['20200501.aa', '20200501.ab', '20200501.ace', '20200501.ady', '20200501.af', '20200501.ak', '20200501.als', '20200501.am', '20200501.an', '20200501.ang', '20200501.ar', '20200501.arc', '20200501.arz', '20200501.as', '20200501.ast', '20200501.atj', '20200501.av', '20200501.ay', '20200501.az', '20200501.azb', '20200501.ba', '20200501.bar', '20200501.bat-smg', '20200501.bcl', '20200501.be', '20200501.be-x-old', '20200501.bg', '20200501.bh', '20200501.bi', '20200501.bjn', '20200501.bm', '20200501.bn', '20200501.bo', '20200501.bpy', '20200501.br', '20200501.bs', '20200501.bug', '20200501.bxr', '20200501.ca', '20200501.cbk-zam', '20200501.cdo', '20200501.ce', '20200501.ceb', '20200501.ch', '20200501.cho', '20200501.chr', '20200501.chy', '20200501.ckb', '20200501.co', '20200501.cr', '20200501.crh', '20200501.cs', '20200501.csb', '20200501.cu', '20200501.cv', '20200501.cy', '20200501.da', '20200501.de', '20200501.din', '20200501.diq', '20200501.dsb', '20200501.dty', '20200501.dv', '20200501.dz', '20200501.ee', '20200501.el', '20200501.eml', '20200501.en', '20200501.eo', '20200501.es', '20200501.et', '20200501.eu', '20200501.ext', '20200501.fa', '20200501.ff', '20200501.fi', '20200501.fiu-vro', '20200501.fj', '20200501.fo', '20200501.fr', '20200501.frp', '20200501.frr', '20200501.fur', '20200501.fy', '20200501.ga', '20200501.gag', '20200501.gan', '20200501.gd', '20200501.gl', '20200501.glk', '20200501.gn', '20200501.gom', '20200501.gor', '20200501.got', '20200501.gu', '20200501.gv', '20200501.ha', '20200501.hak', '20200501.haw', '20200501.he', '20200501.hi', '20200501.hif', '20200501.ho', '20200501.hr', '20200501.hsb', '20200501.ht', '20200501.hu', '20200501.hy', '20200501.ia', '20200501.id', '20200501.ie', '20200501.ig', '20200501.ii', '20200501.ik', '20200501.ilo', '20200501.inh', '20200501.io', '20200501.is', '20200501.it', '20200501.iu', '20200501.ja', '20200501.jam', '20200501.jbo', '20200501.jv', '20200501.ka', '20200501.kaa', '20200501.kab', '20200501.kbd', '20200501.kbp', '20200501.kg', '20200501.ki', '20200501.kj', '20200501.kk', '20200501.kl', '20200501.km', '20200501.kn', '20200501.ko', '20200501.koi', '20200501.krc', '20200501.ks', '20200501.ksh', '20200501.ku', '20200501.kv', '20200501.kw', '20200501.ky', '20200501.la', '20200501.lad', '20200501.lb', '20200501.lbe', '20200501.lez', '20200501.lfn', '20200501.lg', '20200501.li', '20200501.lij', '20200501.lmo', '20200501.ln', '20200501.lo', '20200501.lrc', '20200501.lt', '20200501.ltg', '20200501.lv', '20200501.mai', '20200501.map-bms', '20200501.mdf', '20200501.mg', '20200501.mh', '20200501.mhr', '20200501.mi', '20200501.min', '20200501.mk', '20200501.ml', '20200501.mn', '20200501.mr', '20200501.mrj', '20200501.ms', '20200501.mt', '20200501.mus', '20200501.mwl', '20200501.my', '20200501.myv', '20200501.mzn', '20200501.na', '20200501.nah', '20200501.nap', '20200501.nds', '20200501.nds-nl', '20200501.ne', '20200501.new', '20200501.ng', '20200501.nl', '20200501.nn', '20200501.no', '20200501.nov', '20200501.nrm', '20200501.nso', '20200501.nv', '20200501.ny', '20200501.oc', '20200501.olo', '20200501.om', '20200501.or', '20200501.os', '20200501.pa', '20200501.pag', '20200501.pam', '20200501.pap', '20200501.pcd', '20200501.pdc', '20200501.pfl', '20200501.pi', '20200501.pih', '20200501.pl', '20200501.pms', '20200501.pnb', '20200501.pnt', '20200501.ps', '20200501.pt', '20200501.qu', '20200501.rm', '20200501.rmy', '20200501.rn', '20200501.ro', '20200501.roa-rup', '20200501.roa-tara', '20200501.ru', '20200501.rue', '20200501.rw', '20200501.sa', '20200501.sah', '20200501.sat', '20200501.sc', '20200501.scn', '20200501.sco', '20200501.sd', '20200501.se', '20200501.sg', '20200501.sh', '20200501.si', '20200501.simple', '20200501.sk', '20200501.sl', '20200501.sm', '20200501.sn', '20200501.so', '20200501.sq', '20200501.sr', '20200501.srn', '20200501.ss', '20200501.st', '20200501.stq', '20200501.su', '20200501.sv', '20200501.sw', '20200501.szl', '20200501.ta', '20200501.tcy', '20200501.te', '20200501.tet', '20200501.tg', '20200501.th', '20200501.ti', '20200501.tk', '20200501.tl', '20200501.tn', '20200501.to', '20200501.tpi', '20200501.tr', '20200501.ts', '20200501.tt', '20200501.tum', '20200501.tw', '20200501.ty', '20200501.tyv', '20200501.udm', '20200501.ug', '20200501.uk', '20200501.ur', '20200501.uz', '20200501.ve', '20200501.vec', '20200501.vep', '20200501.vi', '20200501.vls', '20200501.vo', '20200501.wa', '20200501.war', '20200501.wo', '20200501.wuu', '20200501.xal', '20200501.xh', '20200501.xmf', '20200501.yi', '20200501.yo', '20200501.za', '20200501.zea', '20200501.zh', '20200501.zh-classical', '20200501.zh-min-nan', '20200501.zh-yue', '20200501.zu']
Example of usage:
	`load_dataset('wikipedia', '20200501.aa')`

I think the other parameter is missing in the load_dataset function that is not shown in the instruction.


Use f-strings in the dataset scripts

Uses f-strings to format the .py files in the dataset folder


Make several audio datasets streamable

Reading FLAC (for `librispeech_asr`) works OK for me (`soundfile` version: `0.10.3`):
```python
In [2]: ds = load_dataset("datasets/librispeech_asr/librispeech_asr.py", "clean", streaming=True, split="train.100")

In [3]: item = next(iter(ds))

In [4]: item.keys()
Out[4]: dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])

In [5]: item["file"]
Out[5]: '374-180298-0000.flac'

In [6]: item["audio"].keys()
Out[6]: dict_keys(['path', 'array', 'sampling_rate'])

In [7]: item["audio"]["sampling_rate"]
Out[7]: 16000

In [8]: item["audio"]["path"]
Out[8]: '374-180298-0000.flac'

In [9]: item["audio"]["array"].shape
Out[9]: (232480,)
```

Oh cool ! I think this might have come from an issue with my local `soundfile` installation then

I'll do `multilingual_librispeech` in a separate PR since it requires the data to be in another format (in particular separate the train/dev/test splits in different files)

<s>Needs https://github.com/huggingface/datasets/pull/3129 to be merged first</s>

Make those audio datasets streamable:
- [x] common_voice
- [x] openslr
- [x] vivos
- [x] librispeech_asr <s>(still has some issues to read FLAC)</s> *actually it's ok*
- [ ] <s>multilingual_librispeech (yet to be converted)</S> *TODO in a separate PR*


Unpin markdown for build_docs now that it's fixed

`markdown`'s bug has been fixed, so this PR reverts #3286


Allow datasets with indices table when concatenating along axis=1

Calls `flatten_indices` on the datasets with indices table in `concatenate_datasets` to fix issues when concatenating along `axis=1`.

cc @lhoestq: I decided to flatten all the datasets instead of flattening all the datasets except the largest one in the end. The latter approach fails on the following example:
```python
a = Dataset.from_dict({"a": [10, 20, 30, 40]})
b = Dataset.from_dict({"b": [10, 20, 30, 40, 50, 60]})  # largest dataset
a = a.select([1, 2, 3])
b = b.select([1, 2, 3])
concatenate_datasets([a, b], axis=1) # fails at line concat_tables(...) because the real length of b's data is 6 and a's length is 3 after flattening (was 4 before flattening)
```

Also, it requires additional re-ordering of indices to prepare them for working with the indices table of the largest dataset. IMO not worth when we save only one `flatten_indices` call. (feel free to check the code of that approach at https://github.com/huggingface/datasets/commit/6acd10481c70950dcfdbfd2bab0bf0c74ad80bcb if you are interested)

Fixes #3273


Add The Pile dataset and PubMed Central subset

Add:
- The complete final version of The Pile dataset: "all" config
- PubMed Central subset of The Pile: "pubmed_central" config

Close #1675, close bigscience-workshop/data_tooling#74.

CC: @StellaAthena, @lewtun


Fix build_docs CI

Because of https://github.com/Python-Markdown/markdown/issues/1196 we have to temporarily pin `markdown` to 3.3.4 for the docs to build without issues


Add IEMOCAP dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** IEMOCAP
- **Description:** acted, multimodal and multispeaker database
- **Paper:** https://sail.usc.edu/iemocap/Busso_2008_iemocap.pdf
- **Data:** https://sail.usc.edu/iemocap/index.html
- **Motivation:** Useful multimodal dataset

cc @anton-l

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add VoxLingua107 dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** VoxLingua107
- **Description:** VoxLingua107 is a speech dataset for training spoken language identification models. The dataset consists of short speech segments automatically extracted from YouTube videos and labeled according the language of the video title and description, with some post-processing steps to filter out false positives.
- **Paper:** https://arxiv.org/abs/2011.12998
- **Data:** http://bark.phon.ioc.ee/voxlingua107/
- **Motivation:** Nice audio classification dataset

cc @anton-l

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add Speech Commands dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Speech commands
- **Description:** A Dataset for Limited-Vocabulary Speech Recognition
- **Paper:** https://arxiv.org/abs/1804.03209
- **Data:** https://www.tensorflow.org/datasets/catalog/speech_commands,  Available:
http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz
- **Motivation:** Nice dataset for audio classification training

cc @anton-l

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Hi ! Thanks for reporting :)
I think this is because the dataset is behind an access page. We can fix the dataset viewer

If you also have this error when you use the `datasets` library in python, you should probably pass `use_auth_token=True` to the `load_dataset()` function to use your account to access the dataset.

Ah ok, I didn't realise about the login page. I'll try `use_auth_token=True` and see if that solves it.

Regards!

Hi,

Using `use_auth_token=True` and downloading the credentials with `huggingface-cli login` (stored in .huggingface/token) solved the issue.

Should I leave the issue open until you fix the Dataset viewer issue?

Cool ! Yes let's keep this issue open until the viewer is fixed - I'll close it when this is fixed. Thanks

The error I get when trying to load OSCAR 21.09 is this
```
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
```

The URL I get in the browser is this
```
https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/blob/main/OSCAR-2109.py
```

Maybe URL is the issue? (resolve vs blob)

> The error I get when trying to load OSCAR 21.09 is this
>
> ```
> ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
> ```
>
> The URL I get in the browser is this
>
> ```
> https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/blob/main/OSCAR-2109.py
> ```
>
> Maybe URL is the issue? (resolve vs blob)

You need to download your login credentials. See `huggingface-cli login` documentation and when loading the dataset use `use_auth_token=True`:
`
load_dataset(corpus, language, split=None, use_auth_token=True, cache_dir=cache_folder)`

## Dataset viewer issue for '*oscar-corpus/OSCAR-2109*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/oscar-corpus/OSCAR-2109)*

*The dataset library cannot download any language from the oscar-corpus/OSCAR-2109 dataset. By entering the URL in your browser I can access the file.*

```
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
```

Am I the one who added this dataset ? No

Using the older version of [OSCAR](https://huggingface.co/datasets/oscar) I don't have any issues downloading languages with the dataset library.


[Datasets] Improve Covost 2

It's currently quite confusing to understand the manual data download instruction of Covost and not very user-friendly.

Currenty the user has to:

1. Go on Common Voice website
2. Find the correct dataset which is **not** mentioned in the error message
3. Download it
4. Untar it
5. Create a language id folder (why? this folder does not exist in the `.tar` downloaded file)
6. pass the folder containing the created language id folder

This PR improves this to:

1. Go on Common Voice website
2. Find the correct dataset which **is** mentioned in the error message
3. Download it
4. Untar it
5. pass the untared folder

**Note**: This PR is not at all time-critical


Fix bookcorpusopen RAM usage

Each document is a full book, so the default arrow writer batch size of 10,000 is too big, and it can fill up RAM quickly before flushing the first batch on disk. I changed its batch size to 256 to use maximum 100MB of memory

Fix #3167.


Minor Typo Fix - Precision to Recall


Add The Pile subsets

Add The Pile subsets:
- pubmed
- ubuntu_irc
- europarl
- hacker_news
- nih_exporter

Close bigscience-workshop/data_tooling#301.


COCO 🥥 on the 🤗 Hub?

TO-DO:
- [x] Add unlabeled 2017 splits, train and validation splits of 2015
- [ ] Add Class Labels as list instead

This is a draft PR since I ran into few small problems.
I referred to this TFDS code: https://github.com/tensorflow/datasets/blob/2538a08c184d53b37bfcf52cc21dd382572a88f4/tensorflow_datasets/object_detection/coco.py
cc: @mariosasko


Update clue benchmark

Fix #3374


Support streaming zipped dataset repo by passing only repo name

Fix #3373.


NonMatchingChecksumError for the CLUE:cluewsc2020, chid, c3 and tnews

Hi, it seems like there are updates in cluewsc2020, chid, c3 and tnews, since i could not load them due to the checksum error.


Support streaming zipped CSV dataset repo by passing only repo name

enhancement

New feature or request

Given a community 🤗 dataset repository containing only a zipped CSV file (only raw data, no loading script), I would like to load it in streaming mode without passing `data_files`:
```
ds_name = "bigscience-catalogue-data/vietnamese_poetry_from_fsoft_ai_lab"
ds = load_dataset(ds_name, split="train", streaming=True, use_auth_token=True)
item = next(iter(ds))
```

Currently, it gives a `FileNotFoundError` because there is no glob (no "\*" after "zip://": "zip://*") in the passed URL:
```
'zip://::https://huggingface.co/datasets/bigscience-catalogue-data/vietnamese_poetry_from_fsoft_ai_lab/resolve/e5d45f1bd9a8a798cc14f0a45ebc1ce91907c792/poems_dataset.zip'
```


[SEO improvement] Add Dataset Metadata to make datasets indexable

New feature or request

Some people who host datasets on github seem to include a table of metadata at the end of their README.md to make the dataset indexable by [Google Dataset Search](https://datasetsearch.research.google.com/) (See [here](https://github.com/google-research/google-research/tree/master/goemotions#dataset-metadata) and [here](https://github.com/cvdfoundation/google-landmark#dataset-metadata)). This could be a useful addition to canonical datasets; perhaps even community datasets.

I'll include a screenshot (as opposed to markdown) as an example so as not to have a github issue indexed as a dataset:

> ![image](https://user-images.githubusercontent.com/3664563/144496173-953428cf-633a-4571-b75b-f099c6b2ed65.png)

**_PS: It might very well be the case that this is already covered by some other markdown magic I'm not aware of._**


feat: add americas nli dataset

This PR adds the [Americas NLI](https://arxiv.org/abs/2104.08726) dataset, extension of XNLI to 10 low-resource indigenous languages spoken in the Americas: Ashaninka, Aymara, Bribri, Guarani, Nahuatl, Otomi, Quechua, Raramuri, Shipibo-Konibo, and Wixarika.

One odd thing (not sure) is that I had to set

`datasets-cli dummy_data ./datasets/americas_nli/ --auto_generate --n_lines 7500`

`n_lines` very large to successfully generate the dummy files for all the subsets. Happy to get some guidance here.

Otherwise, I hope everything is in order :)

e: missed a step, onto fixing the tests
e2: there you go -- hope it's ok to have added more languages with their ISO codes to `languages.json`, need those tests to pass :laughing:


Document a training loop for streaming dataset

I added some docs about streaming dataset. In particular I added two subsections:
- one on how to use `map` for preprocessing
- one on how to use a streaming dataset in a pytorch training loop

cc @patrickvonplaten @stevhliu if you have some comments
cc @Rocketknight1 later we can add the one for TF and I might need your help ^^'


[Audio] Allow resampling for audio datasets in streaming mode

New feature or request

Many audio datasets like Common Voice always need to be resampled. This can very easily be done in non-streaming mode as follows:

```python
from datasets import load_dataset

ds = load_dataset("common_voice", "ab", split="test")

ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
```

However in streaming mode it fails currently:

```python
from datasets import load_dataset

ds = load_dataset("common_voice", "ab", split="test", streaming=True)

ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
```

with the following error:

```
AttributeError: 'IterableDataset' object has no attribute 'cast_column'
```

It would be great if we could add such a feature (I'm not 100% sure though how complex this would be)


Fix dict source_datasets tagset validator

Currently, the `source_datasets` tag validation does not support passing a dict with configuration keys.

This PR:
- Extends `tagset_validator` to support regex tags
- Uses `tagset_validator` to validate dict `source_datasets`


Fix typo in other-structured-to-text task tag

Fix typo in task tag:
- `other-stuctured-to-text` (before)
- `other-structured-to-text` (now)


Add multimodal datasets

dataset request

Requesting to add a new dataset

Epic issue to track the addition of multimodal datasets:
- [ ] #2526
- [ ] #1842
- [ ] #1810

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

@VictorSanh feel free to add and sort by priority any interesting dataset. I have added the multimodal dataset requests which were already present as issues.


Add task tags for multimodal datasets

enhancement

New feature or request

## **Is your feature request related to a problem? Please describe.**

Currently, task tags are either exclusively related to text or speech processing:
- https://github.com/huggingface/datasets/blob/master/src/datasets/utils/resources/tasks.json

## **Describe the solution you'd like**

We should also add tasks related to:
- multimodality
- image
- video

CC: @VictorSanh @lewtun @lhoestq @merveenoyan @SBrandeis


Use the Audio feature in the AutomaticSpeechRecognition template

This updates the ASR template and all supported datasets to use the `Audio` feature


Update URL of Jeopardy! dataset

Updates the URL of the Jeopardy! dataset.

Fix #3361


Adapt image datasets

This PR:
* adapts the ImageClassification template to use the new Image feature
* adapts the following datasets to use the new Image feature:
  * beans (+ fixes streaming)
  * cast_vs_dogs (+ fixes streaming)
  * cifar10
  * cifar100
  * fashion_mnist
  * mnist
  * head_qa

cc @nateraw


Jeopardy _URL access denied

bug

Something isn't working

## Describe the bug
http://skeeto.s3.amazonaws.com/share/JEOPARDY_QUESTIONS1.json.gz returns Access Denied now.

However, https://drive.google.com/file/d/0BwT5wj_P7BKXb2hfM3d2RHU1ckE/view?usp=sharing from the original Reddit post https://www.reddit.com/r/datasets/comments/1uyd0t/200000_jeopardy_questions_in_a_json_file/ may work.

## Steps to reproduce the bug
```shell
> python
Python 3.7.12 (default, Sep  5 2021, 08:34:29)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
```
```python
>>> from datasets import load_dataset
>>> load_dataset("jeopardy")
```

## Expected results
The download completes.

## Actual results
```shell
Downloading: 4.18kB [00:00, 1.60MB/s]
Downloading: 2.03kB [00:00, 1.04MB/s]
Using custom data configuration default
Downloading and preparing dataset jeopardy/default (download: 12.13 MiB, generated: 34.46 MiB, post-processed: Unknown size, total: 46.59 MiB) to /Users/mike/.cache/huggingface/datasets/jeopardy/default/0.1.0/25ee3e4a73755e637b8810f6493fd36e4523dea3ca8a540529d0a6e24c7f9810...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 675, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/mike/.cache/huggingface/modules/datasets_modules/datasets/jeopardy/25ee3e4a73755e637b8810f6493fd36e4523dea3ca8a540529d0a6e24c7f9810/jeopardy.py", line 72, in _split_generators
    filepath = dl_manager.download_and_extract(_DATA_URL)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 197, in download
    download_func, url_or_urls, map_tuple=True, num_proc=download_config.num_proc, disable_tqdm=False
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 197, in map_nested
    return function(data_struct)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 305, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 594, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach http://skeeto.s3.amazonaws.com/share/JEOPARDY_QUESTIONS1.json.gz
```

---

```shell
> curl http://skeeto.s3.amazonaws.com/share/JEOPARDY_QUESTIONS1.json.gz
```
```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>70Y9R36XNPEQXMGV</RequestId><HostId>G6F5AK4qo7JdaEdKGMtS0P6gdLPeFOdEfSEfvTOZEfk9km0/jAfp08QLfKSTFFj1oWIKoAoBehM=</HostId></Error>
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: macOS Catalina 10.15.7
- Python version: 3.7.12
- PyArrow version: 6.0.1


Add The Pile USPTO subset

Add:
- USPTO subset of The Pile: "uspto" config

Close bigscience-workshop/data_tooling#297.

CC: @StellaAthena


Add The Pile Free Law subset

@albertvillanova Is there a specific reason you’re adding the Pile under “the” instead of under “pile”? That does not appear to be consistent with other datasets.

Hi @StellaAthena,

I asked myself the same question, but at the end I decided to be consistent with previously added Pile subsets:
- #2817

I guess the reason is to stress that the definite article is always used before the name of the dataset (your site says: "The Pile. An 800GB Dataset of Diverse Text for Language Modeling"). Other datasets are not usually preceded by the definite article, like "the SQuAD" or "the GLUE" or "the Common Voice"...

CC: @lhoestq

Add:
- Free Law subset of The Pile: "free_law" config

Close bigscience-workshop/data_tooling#75.

CC: @StellaAthena


add new field, and get errors

Hi,

could you please post this question on our [Forum](https://discuss.huggingface.co/) as we keep issues for bugs and feature requests?

> Hi,
>
> could you please post this question on our [Forum](https://discuss.huggingface.co/) as we keep issues for bugs and feature requests?

ok.

after adding new field **tokenized_examples["example_id"]**, and get errors below,
I think it is due to changing data to tensor, and **tokenized_examples["example_id"]** is string list
**all fields**
```
***************** train_dataset 1: Dataset({
    features: ['attention_mask', 'end_positions', 'example_id', 'input_ids', 'start_positions', 'token_type_ids'],
    num_rows: 87714
})
```

**Errors**
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 705, in convert_to_tensors
    tensor = as_tensor(value)
ValueError: too many dimensions 'str'
```


Update README.md

After having worked a bit with the dataset.
As far as I know, it is solely in English (en-US). There are only a few mails in Spanish, French or German (less than a dozen I would estimate).


to_tf_dataset() refactor

Also, please don't merge yet - I need to make sure all the code samples and notebooks have a collate_fn specified, since we're removing the ability for this method to work without one!

This is the promised cleanup to `to_tf_dataset()` now that the course is out of the way! The main changes are:

- A collator is always required (there was way too much hackiness making things like labels work without it)
- Lots of cleanup and a lot of code moved to `_get_output_signature`
- Should now handle it gracefully when the data collator adds unexpected columns


Extend support for streaming datasets that use pd.read_excel

This PR fixes error:
```
ValueError: Cannot seek streaming HTTP file
```

CC: @severo


Remove duplicate name from dataset cards

Remove duplicate name from dataset card for:
- ajgt_twitter_ar
- emotone_ar


add one field  "example_id", but I can't see it in the "comput_loss" function

Hi ! Your function looks fine, I used to map `squad` locally and it indeed added the `example_id` field correctly.

However I think that in the `compute_loss` method only a subset of the fields are available: the model inputs. Since `example_id` is not a model input (it's not passed as a parameter to the model), the data loader doesn't need to return it by default.

However you can disable this behavior by setting `remove_unused_columns` to `False` to your training arguments. In this case in `compute_loss` you will get the full item with all the fields.

Note that since the model doesn't take `example_id` as input, you will have to remove it from the inputs when `model(**inputs)` is called

Hi, I have set **args.remove_unused_columns=False** and  **training_args.remove_unused_columns=False**, but the field doesn't been contained yet.
```
def main():
    argp = HfArgumentParser(TrainingArguments)
    # The HfArgumentParser object collects command-line arguments into an object (and provides default values for unspecified arguments).
    # In particular, TrainingArguments has several keys that you'll need/want to specify (when you call run.py from the command line):
    # --do_train
    #     When included, this argument tells the script to train a model.
    #     See docstrings for "--task" and "--dataset" for how the training dataset is selected.
    # --do_eval
    #     When included, this argument tells the script to evaluate the trained/loaded model on the validation split of the selected dataset.
    # --per_device_train_batch_size <int, default=8>
    #     This is the training batch size.
    #     If you're running on GPU, you should try to make this as large as you can without getting CUDA out-of-memory errors.
    #     For reference, with --max_length=128 and the default ELECTRA-small model, a batch size of 32 should fit in 4gb of GPU memory.
    # --num_train_epochs <float, default=3.0>
    #     How many passes to do through the training data.
    # --output_dir <path>
    #     Where to put the trained model checkpoint(s) and any eval predictions.
    #     *This argument is required*.

    argp.add_argument('--model', type=str,
                      default='google/electra-small-discriminator',
                      help="""This argument specifies the base model to fine-tune.
        This should either be a HuggingFace model ID (see https://huggingface.co/models)
        or a path to a saved model checkpoint (a folder containing config.json and pytorch_model.bin).""")
    argp.add_argument('--task', type=str, choices=['nli', 'qa'], required=True,
                      help="""This argument specifies which task to train/evaluate on.
        Pass "nli" for natural language inference or "qa" for question answering.
        By default, "nli" will use the SNLI dataset, and "qa" will use the SQuAD dataset.""")
    argp.add_argument('--dataset', type=str, default=None,
                      help="""This argument overrides the default dataset used for the specified task.""")
    argp.add_argument('--max_length', type=int, default=128,
                      help="""This argument limits the maximum sequence length used during training/evaluation.
        Shorter sequence lengths need less memory and computation time, but some examples may end up getting truncated.""")
    argp.add_argument('--max_train_samples', type=int, default=None,
                      help='Limit the number of examples to train on.')
    argp.add_argument('--max_eval_samples', type=int, default=None,
                      help='Limit the number of examples to evaluate on.')

    argp.remove_unused_columns = False
    training_args, args = argp.parse_args_into_dataclasses()
    args.remove_unused_columns=False
    training_args.remove_unused_columns=False
```

```
**************** train_dataset: Dataset({
    features: ['id', 'title', 'context', 'question', 'answers'],
    num_rows: 87599
})

**************** train_dataset_featurized: Dataset({
    features: ['attention_mask', 'end_positions', 'input_ids', 'start_positions', 'token_type_ids'],
    num_rows: 87714
})
```

Hi, I print the value, all are set to False, but don't work.
```
********************* training_args: TrainingArguments(
_n_gpu=1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_find_unused_parameters=None,
debug=[],
deepspeed=None,
disable_tqdm=False,
do_eval=False,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_steps=None,
evaluation_strategy=IntervalStrategy.NO,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
gradient_accumulation_steps=1,
greater_is_better=None,
group_by_length=False,
ignore_data_skip=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=5e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=-1,
log_level=-1,
log_level_replica=-1,
log_on_each_node=True,
logging_dir=./re_trained_model/runs/Dec01_14-15-08_399b9290604c,
logging_first_step=False,
logging_steps=500,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_type=SchedulerType.LINEAR,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
no_cuda=False,
num_train_epochs=3.0,
output_dir=./re_trained_model,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=8,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=re_trained_model,
push_to_hub_organization=None,
push_to_hub_token=None,
remove_unused_columns=False,
report_to=['tensorboard'],
resume_from_checkpoint=None,
run_name=./re_trained_model,
save_on_each_node=False,
save_steps=500,
save_strategy=IntervalStrategy.STEPS,
save_total_limit=None,
seed=42,
sharded_ddp=[],
skip_memory_metrics=True,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_legacy_prediction_loop=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
)
```
```
********************* args: Namespace(dataset='squad', max_eval_samples=None, max_length=128, max_train_samples=None, model='google/electra-small-discriminator', remove_unused_columns=False, task='qa')
2021-12-01 14:15:10,048 - WARNING - datasets.builder - Reusing dataset squad (/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)
Some weights of the model checkpoint at google/electra-small-discriminator were not used when initializing ElectraForQuestionAnswering: ['discriminator_predictions.dense_prediction.weight', 'discriminator_predictions.dense_prediction.bias', 'discriminator_predictions.dense.weight', 'discriminator_predictions.dense.bias']
- This IS expected if you are initializing ElectraForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing ElectraForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of ElectraForQuestionAnswering were not initialized from the model checkpoint at google/electra-small-discriminator and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Preprocessing data... (this takes a little bit, should only happen once per dataset)
```

Hmmm, it might be because the default data collator removes all the fields with `string` type:

https://github.com/huggingface/transformers/blob/4c0dd199c8305903564c2edeae23d294edd4b321/src/transformers/data/data_collator.py#L107-L112

I guess you also need a custom data collator that doesn't remove them.

can you give a tutorial about how to do this?

I overwrite **get_train_dataloader**, and remove **_remove_unused_columns**, but it doesn't work.

```
    def get_train_dataloader(self) -> DataLoader:
        """
        Returns the training :class:`~torch.utils.data.DataLoader`.

        Will use no sampler if :obj:`self.train_dataset` does not implement :obj:`__len__`, a random sampler (adapted
        to distributed training if necessary) otherwise.

        Subclass and override this method if you want to inject some custom behavior.
        """
        if self.train_dataset is None:
            raise ValueError("Trainer: training requires a train_dataset.")

        train_dataset = self.train_dataset
        # if is_datasets_available() and isinstance(train_dataset, datasets.Dataset):
        #     train_dataset = self._remove_unused_columns(train_dataset, description="training")

        if isinstance(train_dataset, torch.utils.data.IterableDataset):
            if self.args.world_size > 1:
                train_dataset = IterableDatasetShard(
                    train_dataset,
                    batch_size=self.args.train_batch_size,
                    drop_last=self.args.dataloader_drop_last,
                    num_processes=self.args.world_size,
                    process_index=self.args.process_index,
                )

            return DataLoader(
                train_dataset,
                batch_size=self.args.train_batch_size,
                collate_fn=self.data_collator,
                num_workers=self.args.dataloader_num_workers,
                pin_memory=self.args.dataloader_pin_memory,
            )

        train_sampler = self._get_train_sampler()

        return DataLoader(
            train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler,
            collate_fn=self.data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.dataloader_num_workers,
            pin_memory=self.args.dataloader_pin_memory,
        )
```

Hi, it works now, thank you.
1. **args.remove_unused_columns=False** and **training_args.remove_unused_columns=False**
2. overwrite **get_train_dataloader**, and remove **_remove_unused_columns**
3. add new fields, and can be got in **inputs**.

Hi,  I add one field **example_id**, but I can't see it in the **comput_loss** function, how can I do this? below is the information of inputs

```
*********************** inputs: {'attention_mask': tensor([[1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        ...,
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0]], device='cuda:0'), 'end_positions': tensor([ 25,  97,  93,  44,  25, 112, 109, 134], device='cuda:0'), 'input_ids': tensor([[ 101, 2054, 2390,  ...,    0,    0,    0],
        [ 101, 2054, 2515,  ...,    0,    0,    0],
        [ 101, 2054, 2106,  ...,    0,    0,    0],
        ...,
        [ 101, 2339, 2001,  ...,    0,    0,    0],
        [ 101, 2054, 2515,  ...,    0,    0,    0],
        [ 101, 2054, 2003,  ...,    0,    0,    0]], device='cuda:0'), 'start_positions': tensor([ 20,  90,  89,  41,  25,  96, 106, 132], device='cuda:0'), 'token_type_ids': tensor([[0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        ...,
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0]], device='cuda:0')}
```

```
# This function preprocesses a question answering dataset, tokenizing the question and context text
# and finding the right offsets for the answer spans in the tokenized context (to use as labels).
# Adapted from https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py
def prepare_train_dataset_qa(examples, tokenizer, max_seq_length=None):
    questions = [q.lstrip() for q in examples["question"]]
    max_seq_length = tokenizer.model_max_length
    # tokenize both questions and the corresponding context
    # if the context length is longer than max_length, we split it to several
    # chunks of max_length
    tokenized_examples = tokenizer(
        questions,
        examples["context"],
        truncation="only_second",
        max_length=max_seq_length,
        stride=min(max_seq_length // 2, 128),
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length"
    )

    # Since one example might give us several features if it has a long context,
    # we need a map from a feature to its corresponding example.
    sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")
    # The offset mappings will give us a map from token to character position
    # in the original context. This will help us compute the start_positions
    # and end_positions to get the final answer string.
    offset_mapping = tokenized_examples.pop("offset_mapping")

    tokenized_examples["start_positions"] = []
    tokenized_examples["end_positions"] = []

    tokenized_examples["example_id"] = []

    for i, offsets in enumerate(offset_mapping):
        input_ids = tokenized_examples["input_ids"][i]
        # We will label features not containing the answer the index of the CLS token.
        cls_index = input_ids.index(tokenizer.cls_token_id)
        sequence_ids = tokenized_examples.sequence_ids(i)
        # from the feature idx to sample idx
        sample_index = sample_mapping[i]
        # get the answer for a feature
        answers = examples["answers"][sample_index]

        tokenized_examples["example_id"].append(examples["id"][sample_index])

        if len(answers["answer_start"]) == 0:
            tokenized_examples["start_positions"].append(cls_index)
            tokenized_examples["end_positions"].append(cls_index)
        else:
            # Start/end character index of the answer in the text.
            start_char = answers["answer_start"][0]
            end_char = start_char + len(answers["text"][0])

            # Start token index of the current span in the text.
            token_start_index = 0
            while sequence_ids[token_start_index] != 1:
                token_start_index += 1

            # End token index of the current span in the text.
            token_end_index = len(input_ids) - 1
            while sequence_ids[token_end_index] != 1:
                token_end_index -= 1

            # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).
            if not (offsets[token_start_index][0] <= start_char and
                    offsets[token_end_index][1] >= end_char):
                tokenized_examples["start_positions"].append(cls_index)
                tokenized_examples["end_positions"].append(cls_index)
            else:
                # Otherwise move the token_start_index and token_end_index to the two ends of the answer.
                # Note: we could go after the last offset if the answer is the last word (edge case).
                while token_start_index < len(offsets) and \
                        offsets[token_start_index][0] <= start_char:
                    token_start_index += 1
                tokenized_examples["start_positions"].append(
                    token_start_index - 1)
                while offsets[token_end_index][1] >= end_char:
                    token_end_index -= 1
                tokenized_examples["end_positions"].append(token_end_index + 1)

    return tokenized_examples
```

_Originally posted by @yanllearnn in https://github.com/huggingface/datasets/issues/3333#issuecomment-983457161_


Make LABR dataset streamable

Fix LABR dataset to make it streamable.

Related to: #3350.


Add VCTK dataset

Hello @patrickvonplaten, I hope it's okay to ping you with a (dumb) question!

I've been trying to get `dl_manager.download_and_extract(_DL_URL)` to work with no avail. I verified that this is a problem on two different machines (lab server, GCP), so I doubt it's an issue with network connectivity. Here is the full trace.

```
(venv) (base) jaketae@jake-gpu1:~/documents/datasets$ datasets-cli test datasets/vctk --save_infos --all_configs
Testing builder 'main' (1/1)
Downloading and preparing dataset vctk/main to /home/jaketae/.cache/huggingface/datasets/vctk/main/0.9.2/2bfa52a93469fa9d6d4b1831c6511db5442b9f4e48620aef2bc3890d7a5268a8...
Traceback (most recent call last):
  File "/home/jaketae/documents/datasets/venv/bin/datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/home/jaketae/documents/datasets/src/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/home/jaketae/documents/datasets/src/datasets/commands/test.py", line 146, in run
    builder.download_and_prepare(
  File "/home/jaketae/documents/datasets/src/datasets/builder.py", line 593, in download_and_prepare
    self._download_and_prepare(
  File "/home/jaketae/documents/datasets/src/datasets/builder.py", line 659, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/jaketae/.cache/huggingface/modules/datasets_modules/datasets/vctk/2bfa52a93469fa9d6d4b1831c6511db5442b9f4e48620aef2bc3890d7a5268a8/vctk.py", line 76, in _split_generators
    root_path = dl_manager.download_and_extract(_DL_URL)
  File "/home/jaketae/documents/datasets/src/datasets/utils/download_manager.py", line 283, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/jaketae/documents/datasets/src/datasets/utils/download_manager.py", line 195, in download
    downloaded_path_or_paths = map_nested(
  File "/home/jaketae/documents/datasets/src/datasets/utils/py_utils.py", line 234, in map_nested
    return function(data_struct)
  File "/home/jaketae/documents/datasets/src/datasets/utils/download_manager.py", line 216, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/jaketae/documents/datasets/src/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/home/jaketae/documents/datasets/src/datasets/utils/file_utils.py", line 608, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url}")
ConnectionError: Couldn't reach https://datashare.is.ed.ac.uk/bitstream/handle/10283/3443/VCTK-Corpus-0.92.zip
```

On my local, however, the URL correctly points to the download zip file. My admittedly naive guess is that the website is web-crawler or scraper proof (requiring specific headers, etc.), but I also think I might have just missed a very basic step in the process.

Apologies for the delayed PR, and TIA for the help!

Hey @jaketae,

Hmm, yeah I don't know really either - the link also works correctly for me when doing:

```
wget https://datashare.is.ed.ac.uk/bitstream/handle/10283/3443/VCTK-Corpus-0.92.zip
```

I think however that I had a similar problem previously with Edinburgh's (`.ed.ac.uk`) websites which I solved with the following hack. Not sure if this could be the same problem here...
https://github.com/huggingface/datasets/blob/e1104ad5d3e83f8b1571e0d6fef4fdabf0a1fde5/datasets/ami/ami.py#L364

The AMI dataset is stored under a different website though it seems: `"https://groups.inf.ed.ac.uk/ami/AMICorpusMirror//amicorpus/{}/audio/{}"`

so not 100p sure if this solves the problem

Hi @patrickvonplaten,

Thanks for the feedback! Sadly, disabling multi-processing didn't cut it for me.

I've been looking at VCTK code in [`torchaudio`](https://pytorch.org/audio/stable/_modules/torchaudio/datasets/vctk.html) and [`tfds`](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/audio/vctk.py). I don't think they're using a hack to accomplish this, so I'll try to look into it to see if I can pinpoint the cause. I'll keep you in the loop here. Thank you!

Hi @patrickvonplaten,

After more investigation, I found that simply increasing `etag_timeout` in `get_from_cache` from 10 to 100 solved it. However, unless I'm missing something, an issue is that `etag_timeout` is basically hard-coded as a default parameter because `cached_path`, which calls `get_from_cache` has no way of modifying the default.

https://github.com/huggingface/datasets/blob/b25ac1d62670e7b339ed552ecc37846d2abd30c7/src/datasets/utils/file_utils.py#L298-L310

https://github.com/huggingface/datasets/blob/b25ac1d62670e7b339ed552ecc37846d2abd30c7/src/datasets/utils/file_utils.py#L497-L510

I can think of two solutions.

* Simply increase the default to 100
* Allow `etag_timeout` to be modifiable on a per-dataset basis by integrating it to `download_config` (maybe this is already supported?)

Thank you!

Fixes #1837.


Avoid content-encoding issue while streaming datasets

This PR will fix streaming of datasets served with gzip content-encoding:
```
ClientPayloadError: 400, message='Can not decode content-encoding: gzip'
```

Fix #2918.

CC: @severo


raise exception instead of using assertions.

@mariosasko - Thanks for the review & suggestions. Updated as per the suggestions.

fix for the remaining files https://github.com/huggingface/datasets/issues/3171


Match key names to correspond with filename

Thanks for the suggestion! I think the current checked-in `CHECKPOINT_URLS` is already not working. I believe anyone who tried using the new ckpts (`BLEURT-20-X`) can't unless this fix is in. The zip file from bleurt side unzips to directory name matching the filename (capitalized for new ones). For example without current changes I'd get the following error

```
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-5-f6832fe20f84> in <module>()
      1 predictions = ["hello there", "general kenobi"]
      2 references = ["hello there", "general kenobi"]
----> 3 bleurt = datasets.load_metric("bleurt", "bleurt-20")
      4 results = bleurt.compute(predictions=predictions, references=references)

4 frames
/usr/local/lib/python3.7/dist-packages/bleurt/checkpoint.py in read_bleurt_config(path)
     84   """Reads and checks config file from a BLEURT checkpoint."""
     85   assert tf.io.gfile.exists(path), \
---> 86       "Could not find BLEURT checkpoint {}".format(path)
     87   config_path = os.path.join(path, CONFIG_FILE)
     88   assert tf.io.gfile.exists(config_path), \

AssertionError: Could not find BLEURT checkpoint /root/.cache/huggingface/metrics/bleurt/bleurt-20/downloads/extracted/e34c60f1a05394ecda54e253a10413ca7b5d59f9a23f3cc73258c6b78ffa2f50/bleurt-20
```
inspecting specified path I see that directory name is `BLEURT-20` instead of `bleurt-20`.
Other solution similar to your suggestion is meddle with `dl_manager.download_and_extract` to unzip to paths with lowering all the paths but I imagine this will affect other parts of the library.

In order to properly locate downloaded ckpt files key name needs to match filename. Correcting change introduced in #3235


iter_archive for zip files

And also don't always try streaming with Google Drive - it can have issues because of how Google Drive works (with quotas, restrictions, etc.) and it can indeed cause `BlockSizeError`.

Feel free to host your test data elsewhere, such as in a dataset repository on https://huggingface.co (see [here](https://huggingface.co/docs/datasets/upload_dataset.html#upload-your-files) for a tutorial on how to upload files)

* In this PR, I added the option to iterate through zipfiles for `download_manager.py` only.
* Next PR will be the same applied to `streaming_download_manager.py`.
* Related issue #3272.
## Comments :
* There is no `.isreg()` equivalent in zipfile library to check if file is Regular so I used `.is_dir()` instead to skip directories.
* For now I got `streaming_download_manager.py`  working for local zip files, but not for urls. I get the following error when I test it on an archive in google drive, so still working on it.  `BlockSizeError: Got more bytes so far (>2112) than requested (22)`

## Tasks :
- [x] download_manager.py
- [ ] streaming_download_manager.py


Failed to convert `string` with pyarrow for QED since 1.15.0

Something isn't working

Scratch that, probably the old and incompatible usage of dataset builder from promptsource.

## Describe the bug
Loading QED was fine until 1.15.0.
related: bigscience-workshop/promptsource#659, bigscience-workshop/promptsource#670

Not sure where the root cause is, but here are some candidates:
- #3158
- #3120
- #3196
- #2891

## Steps to reproduce the bug
```python
load_dataset("qed")
```

## Expected results
Loading completed.

## Actual results
```shell
ArrowInvalid: Could not convert in with type str: tried to convert to boolean
Traceback:
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/streamlit/script_runner.py", line 354, in _run_script
    exec(code, module.__dict__)
File "/Users/s0s0cr3/Documents/GitHub/promptsource/promptsource/app.py", line 260, in <module>
    dataset = get_dataset(dataset_key, str(conf_option.name) if conf_option else None)
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/streamlit/caching.py", line 543, in wrapped_func
    return get_or_create_cached_value()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/streamlit/caching.py", line 527, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/Users/s0s0cr3/Documents/GitHub/promptsource/promptsource/utils.py", line 49, in get_dataset
    builder_instance.download_and_prepare()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/builder.py", line 1106, in _prepare_split
    num_examples, num_bytes = writer.finalize()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/arrow_writer.py", line 456, in finalize
    self.write_examples_on_file()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/arrow_writer.py", line 325, in write_examples_on_file
    pa_array = pa.array(typed_sequence)
File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/arrow_writer.py", line 121, in __arrow_array__
    out = pa.array(cast_to_python_objects(self.data, only_1d_for_numpy=True), type=type)
File "pyarrow/array.pxi", line 305, in pyarrow.lib.array
File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.0, 1.16.1
- Platform: macOS 1.15.7 or above
- Python version: 3.7.12 and 3.9
- PyArrow version: 3.0.0, 5.0.0, 6.0.1


Failed to download species_800 from Google Drive zip file

Something isn't working

Hi,

the dataset is downloaded normally on my machine. Maybe the URL was down at the time of your download. Could you try again?

> Hi,
>
> the dataset is downloaded normally on my machine. Maybe the URL was down at the time of your download. Could you try again?

I have tried that many times with both load_dataset() and a browser almost simultaneously. The browser always works for me while load_dataset() fails.

@mariosasko
> the dataset is downloaded normally on my machine. Maybe the URL was down at the time of your download. Could you try again?

I've tried yet again just a moment ago. This time I realize that, the step `(... post-processed: Unknown size, total: 20.89 MiB) to /Users/mike/.cache/huggingface/datasets/species800/species_800/1.0.0/532167f0bb8fbc0d77d6d03c4fd642c8c55527b9c5f2b1da77f3d00b0e559976...` and the one after seem unstable. If I want to retry, I will have to delete it (and probably other cache lock files). It **_sometimes_** works.

But I didn't try `download_mode="force_redownload"` yet.

Anyway, I suppose this isn't really a pressing issue for the time being, so I'm going to close this. Thank you.

## Describe the bug
One can manually download the zip file on Google Drive, but `load_dataset()` cannot.
related: #3248

## Steps to reproduce the bug
```shell
> python
Python 3.7.12 (default, Sep  5 2021, 08:34:29)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
```
```python
>>> from datasets import load_dataset
>>> s800 = load_dataset("species_800")
```

## Expected results
species_800 downloaded.

## Actual results
```shell
Downloading: 5.68kB [00:00, 1.22MB/s]
Downloading: 2.70kB [00:00, 691kB/s]
Downloading and preparing dataset species800/species_800 (download: 17.36 MiB, generated: 3.53 MiB, post-processed: Unknown size, total: 20.89 MiB) to /Users/mike/.cache/huggingface/datasets/species800/species_800/1.0.0/532167f0bb8fbc0d77d6d03c4fd642c8c55527b9c5f2b1da77f3d00b0e559976...
  0%|                                                                                | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 675, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/mike/.cache/huggingface/modules/datasets_modules/datasets/species_800/532167f0bb8fbc0d77d6d03c4fd642c8c55527b9c5f2b1da77f3d00b0e559976/species_800.py", line 104, in _split_generators
    downloaded_files = dl_manager.download_and_extract(urls_to_download)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 284, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 197, in download
    download_func, url_or_urls, map_tuple=True, num_proc=download_config.num_proc, disable_tqdm=False
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 209, in map_nested
    for obj in utils.tqdm(iterable, disable=disable_tqdm)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 209, in <listcomp>
    for obj in utils.tqdm(iterable, disable=disable_tqdm)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 143, in _single_map_nested
    return function(data_struct)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 217, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 305, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/Users/mike/Library/Caches/pypoetry/virtualenvs/promptsource-hsdAcWsQ-py3.7/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 594, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://drive.google.com/u/0/uc?id=1OletxmPYNkz2ltOr9pyT0b0iBtUWxslh&export=download/
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14,0 1.15.0, 1.16.1
- Platform: macOS Catalina 10.15.7
- Python version: 3.7.12
- PyArrow version: 6.0.1


Add ArrayXD docs

Documents support for dynamic first dimension in `ArrayXD` from #2891, and explain the `ArrayXD` feature in general.

Let me know if I'm missing anything @lhoestq :)


Better error message when download fails

From our discussions in https://github.com/huggingface/datasets/issues/3269 and https://github.com/huggingface/datasets/issues/3282 it would be nice to have better messages if a download fails.

In particular the error now shows:
- the error from the HEAD request if there's one
- otherwise the response code of the HEAD request

I also added an error to tell users to pass `use_auth_token` when the Hugging Face Hub returns 401 (Unauthorized).

While paying around with this I also fixed a minor issue with the `force_download` parameter that was not always taken into account


Fix ASSET dataset data URLs

> Hi @tianjianjiang, thanks for the fix.
> The links should also be updated in the `dataset_infos.json` file.
> The failing tests are due to the missing tag in the header of the `README.md` file:

Hi @albertvillanova, thank you for the info! My apologies for the messy PR.

Change the branch name "master" to "main" in the data URLs, since facebookresearch has changed that.


Mirror the canonical datasets to the Hugging Face Hub

enhancement

New feature or request

I created a GitHub project to keep track of what needs to be done:
https://github.com/huggingface/datasets/projects/3

I also store my code in a (private for now) repository at https://github.com/huggingface/mirror_canonical_datasets_on_hub

- [ ] create a repo on https://hf.co/datasets for every canonical dataset
- [ ] on every commit related to a dataset, update the hf.co repo

See https://github.com/huggingface/moon-landing/pull/1562

@SBrandeis: I let you edit this description if needed to precise the intent.


Fix JSON ClassLabel casting for integers

Loading a JSON dataset with ClassLabel feature types currently fails if the JSON data already has integers. Indeed currently it tries to convert the strings to integers without even checking if the data are not integers already.

For example this currently fails:
```python
from datasets import load_dataset, Features, ClassLabel

path = "data.json"
f = Features({"a": ClassLabel(names=["neg", "pos"])})
d = load_dataset("json", data_files=path, features=f)
```
data.json
```json
{"a": 0}
{"a": 1}
```

I fixed that by adding a line that checks the type of the JSON data before trying to convert them

cc @albertvillanova let me know if it sounds good to you


to_tf_dataset fails on TPU

bug

Something isn't working

This might be related to https://github.com/tensorflow/tensorflow/issues/38762 , what do you think @Rocketknight1 ?
> Dataset.from_generator is expected to not work with TPUs as it uses py_function underneath which is incompatible with Cloud TPU 2VM setup. If you would like to read from large datasets, maybe try to materialize it on disk and use TFRecordDataest instead.

Hi @lhoestq @nbroad1881, I think it's very similar, yes. Unfortunately `to_tf_dataset` uses `tf.numpy_function` which can't be compiled - this is a necessary evil to load from the underlying Arrow dataset. We need to update the notebooks/examples to clarify that this won't work, or to identify a workaround. You may be able to get it to work on an actual cloud TPU VM, but those are quite new and we haven't tested it yet.

Thank you for the explanation. I didn't realize the nuances of `tf.numpy_function`. In this scenario, would it be better to use `export(format='tfrecord')` ?  It's not quite the same, but for very large datasets that don't fit in memory it looks like it is the only option. I haven't used `export` before, but I do recall reading that there are suggestions for how big and how many tfrecords there should be to not bottleneck the TPU. It might be nice if there were a way for the `export` method to split the files up into appropriate chunk sizes depending on the size of the dataset and the number of devices. And if that is too much, it would be nice to be able to specify the number of files that would be created when using `export`. Well... maybe the user should just do the chunking themselves and call `export` a bunch of times.  Whatever the case, you have been helpful. Thanks Tensorflow boy ;-)

Yeah, this is something we really should have a proper guide on. I'll make a note to test some things and make a 'TF TPU best practices' notebook at some point, but in the meantime I think your solution of exporting TFRecords will probably work.

Using `to_tf_dataset` to create a dataset and then putting it in `model.fit` results in an internal error on TPUs. I've only tried on Colab and Kaggle TPUs, not GCP TPUs.

## Steps to reproduce the bug
I made a colab to show the error. https://colab.research.google.com/drive/12x_PFKzGouFxqD4OuWfnycW_1TaT276z?usp=sharing

## Expected results
dataset from `to_tf_dataset` works in `model.fit`
Right below the first error in the colab I use `tf.data.Dataset.from_tensor_slices` and `model.fit` works just fine. This is the desired outcome.

## Actual results
```
InternalError: 5 root error(s) found.
  (0) INTERNAL: {{function_node __inference_train_function_30558}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1638231897.932218653","description":"Failed to pick subchannel","file":"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3151,"referenced_errors":[{"created":"@1638231897.932216754","description":"failed to connect to all addresses","file":"third_party/grpc/src/core/lib/transport/error_utils.cc","file_line":161,"grpc_status":14}]}
	 [[{{node StatefulPartitionedCall}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
Executing non-communication op <MultiDeviceIteratorGetNextFromShard> originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic.
	 [[RemoteCall]]
	 [[IteratorGetNextAsOptional]]
	 [[tpu_compile_succeeded_assert/_14023832043698465348/_7/_439]]
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0
- Tensorflow 2.7.0
- `transformers` 4.12.5


[WIP] Add doctests for tutorials

Opening a PR as discussed with @LysandreJik for some help with doctest issues. The goal is to add doctests for each of the tutorials in the documentation to make sure the code samples work as shown.

### Issues

A doctest has been added in the docstring of the `load_dataset_builder` function in `load.py` to handle variable outputs with the `ELLIPSIS` directive. When I run doctest on the `load_hub.rst` file, doctest should recognize the expected output from the docstring, and the corresponding code sample in `load_hub.rst` should pass. I am having the same issue with handling tracebacks in the `load_dataset` function.

From the docstring:
```
>>> dataset_builder.cache_dir #doctest: +ELLIPSIS
/Users/.../.cache/huggingface/datasets/imdb/plain_text/1.0.0/...
```
Test result:
```
Failed example:
    dataset_builder.cache_dir
Expected:
    /Users/.../.cache/huggingface/datasets/imdb/plain_text/1.0.0/...
Got:
    /Users/steven/.cache/huggingface/datasets/imdb/plain_text/1.0.0/2fdd8b9bcadd6e7055e742a706876ba43f19faee861df134affd7a3f60fc38a1
```

I am able to get the doctest to pass by adding the doctest directives (`ELLIPSIS` and `NORMALIZE_WHITESPACE`) to the code samples in the `rst` file directly. But my understanding is that these directives should also work in the docstrings of the functions. I am running the test from the root of the directory:

```
python -m doctest -v docs/source/load_hub.rst
```


Typing of Dataset.__getitem__ could be improved.

Something isn't working

Hi ! Thanks for the suggestion, I didn't know about this decorator.

If you are interesting in contributing, feel free to open a pull request to add the overload methods for each typing combination :) To assign you to this issue, you can comment `#self-assign` in this thread.

`Dataset.__getitem__` is defined right here: https://github.com/huggingface/datasets/blob/e6f1352fe19679de897f3d962e616936a17094f5/src/datasets/arrow_dataset.py#L1840

## Describe the bug

The newly added typing for Dataset.__getitem__ is Union[Dict, List]. This makes tools like mypy a bit awkward to use as we need to check the type manually. We could use type overloading to make this easier. [Documentation](https://docs.python.org/3/library/typing.html#typing.overload)

## Steps to reproduce the bug
Let's have a file `test.py`

```python
from typing import List, Dict, Any

from datasets import Dataset

ds = Dataset.from_dict({
    'a': [1,2,3],
    'b': ["1", "2", "3"]
})

one_colum: List[str] = ds['a']
some_index: Dict[Any, Any] = ds[1]
```

## Expected results

Running `mypy test.py` should not give any error.

## Actual results

```
test.py:10: error: Incompatible types in assignment (expression has type "Union[Dict[Any, Any], List[Any]]", variable has type "List[str]")
test.py:11: error: Incompatible types in assignment (expression has type "Union[Dict[Any, Any], List[Any]]", variable has type "Dict[Any, Any]")
Found 2 errors in 1 file (checked 1 source file)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 6.0.1


Add support for multiple dynamic dimensions and to_pandas conversion for dynamic arrays

Add support for multiple dynamic dimensions (e.g. `(None, None, 3)` for arbitrary sized images) and `to_pandas()` conversion for dynamic arrays.

TODOs:
* [ ] Cleaner code
* [ ] Formatting issues (if NumPy doesn't allow broadcasting even though dtype is np.object)
* [ ] Fix some issues with zero-dim tensors
* [ ] Tests


WIP:add Speech commands dataset

OMG I hope it's more or less clear what I mean...

@lhoestq
Hi Quentin! Thank you for your feedback and suggestions! 🤗

Yes, that was actually what I wanted to do next - I mean the steaming stuff :)
Also, I need to make some changes to the readme (to account for the updated features set).

Hopefully, I will be done by tomorrow afternoon if that's ok.

I'm confused with the correct train / val / test splits of both versions of this dataset. But know I got the solution on how to address that and will do this tomorrow.


Integrate Polars library

enhancement

New feature or request

If possible, a neat API could be something like `Dataset.to_polars()`, as well as `Dataset.set_format("polars")`

Note they use a "custom" implementation of Arrow: [Arrow2](https://github.com/jorgecarleitao/arrow2).

Check potential integration of the Polars library: https://github.com/pola-rs/polars
- Benchmark: https://h2oai.github.io/db-benchmark/

CC: @thomwolf @lewtun