WMT19 Dataset for Kazakh-English is not formatted correctly

Hi ! Thanks for reporting

By looking at the raw `news-commentary-v14.en-kk.tsv` file, it looks like there are at least 17 lines with this issue.
Moreover these issues are not always the same:
- L97 is only `kk` text and must be appended at the end of the `kk` text of the **next** line
- L2897 is only `kk` text and must be appended at the end of the `kk` text of the **previous** line
- L1247 and L1248 are only `kk` texts and must be inserted at the **beginning** of the `kk` text of the next line
- (and there are many others)

It would be nice to have a corrected version of this file ! The file is available in the `wmt/news-commentary` repository on the Datasets Hub here:
https://huggingface.co/datasets/wmt/news-commentary/tree/main/v14/training

Then maybe we can notify the WMT authors and host the corrected version somewhere

In addition to the bug of languages being switched from Issue @415, there are incorrect translations in the dataset because the English-Kazakh translations have a one off formatting error.

The News Commentary v14 parallel data set for kk-en from http://www.statmt.org/wmt19/translation-task.html has a bug here:

> Line 94. The Swiss National Bank, for its part, has been battling with the deflationary effects of the franc’s dramatic appreciation over the past few years.	Швейцарияның Ұлттық банкі өз тарапынан, соңғы бірнеше жыл ішінде франк құнының қатты өсуінің дефляциялық әсерімен күресіп келеді.
>
> Line 95. Дефляциялық күштер 2008 жылы терең және ұзаққа созылған жаһандық дағдарысқа байланысты орын алған ірі экономикалық және қаржылық орын алмасулардың арқасында босатылды.  Жеке қарыз қаражаты үлесінің қысқаруы орталық банктің рефляцияға жұмсалған күш-жігеріне тұрақты соққан қарсы желдей болды.
>
> Line 96. The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008. Private deleveraging became a steady headwind to central bank efforts to reflate.	2009 жылы, алдыңғы қатарлы экономикалардың шамамен үштен бірі бағаның төмендеуін көрсетті, бұл соғыстан кейінгі жоғары деңгей болды.

As you can see, line 95 has only the Kazakh translation which should be part of line 96. This causes all of the following English-Kazakh translation pairs to be one off rendering ALL of those translations incorrect. This issue was not fixed when the dataset was imported to Huggingface. By running this code

```
import datasets
from datasets import load_dataset
dataset = load_dataset('wmt19', 'kk-en')
for key in dataset['train']['translation']:
    if 'The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008.' in key['kk']:
        print(key['en'])
        print(key['kk'])
        break
```
we get:
> 2009 жылы, алдыңғы қатарлы экономикалардың шамамен үштен бірі бағаның төмендеуін көрсетті, бұл соғыстан кейінгі жоғары деңгей болды.
> The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008. Private deleveraging became a steady headwind to central bank efforts to reflate.

which shows that the issue still persists in the Huggingface dataset. The Kazakh sentence matches up to the next English sentence in the dataset instead of the current one.

Please let me know if there's you have any ideas to fix this one-off error from the dataset or if this can be fixed by Huggingface.


Request to remove S2ORC dataset

Hello @kyleclo! Currently, we are getting the data from your bucket, so if you remove it the HF script won't work anymore :)

Until you solve things on your end, @lhoestq suggested we just return a warning message when people try to load that dataset from HF. What would you like it to say?

Hi @kyleclo, as of today, you have not removed your bucket data yet, and therefore HuggingFace can download it from there.

Is it OK? Are you planning to eventually delete it? Thank you.

Hi!  Sorry I missed @yjernite 's previous message, thanks for responding!

Is there an option where we can keep our data in our bucket, but the HF script no longer pulls data from it?

Hi!  I was wondering if it's possible to remove [S2ORC](https://huggingface.co/datasets/s2orc) from hosting on Huggingface's platform?  Unfortunately, there are some legal considerations about how we make this data available.  Happy to add back to Huggingface's platform once we work out those hurdles!  Thanks!


Trouble loading wiki_movies

Hi ! `wiki_movies` was added in `datasets==1.2.0`. However it looks like you have `datasets==1.1.2`.

To use `wiki_movies`, please update `datasets` with
```
pip install --upgrade datasets
```

Thanks a lot! That solved it and I was able to upload a model trained on it as well :)

Hello,
I am trying to load_dataset("wiki_movies") and it gives me this error -

`FileNotFoundError: Couldn't find file locally at wiki_movies/wiki_movies.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/wiki_movies/wiki_movies.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/wiki_movies/wiki_movies.py`

Trying to do `python run_mlm.py \
    --model_name_or_path roberta-base \
    --dataset_name wiki_movies \` also gives the same error.

Is this something on my end? From what I can tell, this dataset was re-added by @lhoestq a few months ago.
Thank you!


citation, homepage, and license fields of `dataset_info.json` are duplicated many times

enhancement

New feature or request

good first issue

Good for newcomers

Thanks for reporting :)
Maybe we can concatenate fields only if they are different.

Currently this is done here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/src/datasets/info.py#L180-L196

This can be a good first contribution to the library.
Please comment if you'd like to improve this and open a PR :)

This happens after a `map` operation when `num_proc` is set to `>1`. I tested this by cleaning up the json before running the `map` op on the dataset so it's unlikely it's coming from an earlier concatenation.

Example result:
```
"citation": "@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n
```

@lhoestq and I believe this is happening due to the fields being concatenated `num_proc` times.


Move Dataset.to_csv to csv module

refactoring

Restructuring existing code without changing its external behavior

Move the implementation of `Dataset.to_csv` to module `datasets.io.csv`.


MIAM dataset - new citation details

Hi !
Looks like there's a unicode error in the new citation in the miam.py file.
Could you try to fix it ? Not sure from which character it comes from though

You can test if it works on your side with
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_miam
```

Hi @lhoestq, I have updated the citations to reference an OpenReview preprint.


Fix deprecated warning message and docstring

documentation

Improvements or additions to documentation

I have a question: what about `dictionary_encode_column_`?
- It is deprecated in Dataset, but it recommends using a non-existing method instead: `Dataset.dictionary_encode_column` does not exist.
- It is NOT deprecated in DatasetDict.

`dictionary_encode_column_ ` should be deprecated since it never worked correctly. It will be removed in a major release.
This has to be deprecated in `DatasetDict` as well.
And `Dataset.dictionary_encode_column` doesn't exist indeed.

Thanks @lhoestq. I have fixed deprecated for `dictionary_encode_column_`.

Fix deprecated warnings:
- Use deprecated Sphinx directive in docstring
- Fix format of deprecated message
- Raise FutureWarning


load_from_disk takes a long time to load local dataset

Hi !
Can you share more information about the features of your dataset ? You can get them by printing `my_dataset.features`
Can you also share the code of your `map` function ?

It is actually just the tokenized `wikipedia` dataset with `input_ids`, `attention_mask`, etc, with one extra column which is a list of integers. The `text` column is removed during tokenization.

```
def add_len_and_seq(example):
    end_idx = example['input_ids'].index(SEP)
    example['actual_len'] = end_idx-1
    seq_len = len(example['input_ids'])

    example['seq'] = [PAD_ID] + [np.uint8(example['some_integer'])]*(end_idx-1) + [PAD_ID]*(seq_len-end_idx)

    return example
```

Is `PAD_ID` a python integer ? You need all the integers in `example['seq']` to have the same type.
Does this work if you remove the `np.uint8` and use python integers instead ?

yup I casted it to `np.uint8` outside the function where it was defined. It was originally using python integers.

Strangely, even when I manually created `np.arrays` of specific `dtypes`, the types in the final `dataset_info.json` that gets written are still `int64`.

Update: I tried creating lists of `int8`s and got the same result.

Yes this is a known issue: #625
We're working on making the precision kept for numpy :)
To specify the precision of the integers, currently one needs to specify the output features with `.map(..., features=output_features)`

Do you know what step is taking forever in the code ?
What happens if you interrupt the execution of the dataset loading ?

After a synchronous discussion, we found that the cache file sizes have an enormous effect on the loading speed: smaller cache files result in faster load times. `num_proc` controls the number of cache files that are being written and is inversely proportional to the individual file size. In other words, increase `num_proc` for smaller cache files :)

Maybe this can be highlighted somewhere in the docs.

I have an extremely large tokenized dataset (24M examples) that loads in a few minutes. However, after adding a column similar to `input_ids` (basically a list of integers) and saving the dataset to disk, the load time goes to >1 hour. I've even tried using `np.uint8` after seeing #1985 but it doesn't seem to be helping (the total size seems to be smaller though).

Does anyone know what could be the issue? Or does the casting of that column to `int8` need to happen in the function that writes the arrow table instead of in the `map` where I create the list of integers?

Tagging @lhoestq since you seem to be working on these issues and PRs :)


SQuAD version

Hi ! This is 1.1 as specified by the download urls here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/datasets/squad/squad.py#L50-L55

Hi~
I want train on squad dataset. What's the version of the squad? Is it 1.1 or 1.0? I'm new in QA, I don't find some descriptions about it.


fixes issue #1110 by descending further if `obj["_type"]` is a dict

Check metrics


CoNLL 2003 dataset not including German

dataset request

Requesting to add a new dataset

Hello, thanks for all the work on developing and maintaining this amazing platform, which I am enjoying working with!

I was wondering if there is a reason why the German CoNLL 2003 dataset is not included in the [repository](https://github.com/huggingface/datasets/tree/master/datasets/conll2003), since a copy of it could be found in some places on the internet such as GitHub? I could help adding the German data to the hub, unless there are some copyright issues that I am unaware of...

This is considering that many work use the union of CoNLL 2002 and 2003 datasets for comparing cross-lingual NER transfer performance in `en`, `de`, `es`, and `nl`. E.g., [XLM-R](https://www.aclweb.org/anthology/2020.acl-main.747.pdf).

## Adding a Dataset
- **Name:** CoNLL 2003 German
- **Paper:** https://www.aclweb.org/anthology/W03-0419/
- **Data:** https://github.com/huggingface/datasets/tree/master/datasets/conll2003


Fix: Allows a feature to be named "_type"

Nice thank you !
This looks like a pretty simple yet effective fix ;)
Could you just add a test in `test_features.py` to make sure that you can create `features` with a `_type` field and that it is possible to convert it as a dict and reload it ?
```python
from datasets import Features, Value

# We usually use `asdict` on a `DatasetInfo` object which is a dataclass instance that contains the features.
# So we need the conversion of features to dict to work.
# You can test that using `dataclasses._asdict_inner`.
# This is the function used by `dataclasses.asdict` to convert a dataclass instance attribute to a dict
from dataclasses import _asdict_inner

f = Features({"_type": Value("string")})
reloaded_f = Features.from_dict(_asdict_inner(f, dict))
assert reloaded_f == f
```

Sure, i will add a test.
One question: are the posted benchmarks reliable? The extra type check seems to add quite some overhead judging by the relative differences. Do you think this is an issue?

The benchmark has a bit of noise, the values are fine ;)
especially in the change you did since the overhead added is negligible.

Ok, i added the test you described above.

I avoided importing the private `_asdict_inner` method and directly used the `DatasetInfo` class, if this is ok with you. Thanks a lot for your support during this PR!

This PR tries to fix issue #1110. Sorry for taking so long to come back to this.

It's a simple fix, but i am not sure if it works for all possible types of `obj`. Let me know what you think @lhoestq


How to disable making arrow tables in load_dataset ?

Hi ! We plan to add streaming features in the future.

This should allow to load a dataset instantaneously without generating the arrow table. The trade-off is that accessing examples from a streaming dataset must be done in an iterative way, and with an additional (but hopefully minor) overhead.
What do you think about this ?

If you have ideas or suggestions of what you expect from such features as a user, feel free to share them, this is really valuable to us !

People mainly want this feature either because it takes too much time too make arrow tables, or they occupy too much memory on the disk. I think both the problem can be solved if we provide arrow tables themselves on datasets hub. Can we do this currently @lhoestq ?

@lhoestq I think the ```try_from_hf_gcs``` provide the same functionality. What all datasets are available on HF GCS? Are all the datasets on huggingFace datasets hub are made available on GCS, automatically?

Only datasets like wikipedia, wiki40b, wiki_dpr and natural questions are available already processed on the HF google storage. This is used to download directly the arrow file instead of building it from the original data files.

@lhoestq How can we make sure that the data we upload on HuggingFace hub is available in form of preprocessed arrow files ?

We're still working on this :) This will be available soon
Users will be able to put their processed arrow files on the Hub

Hi! You can now use `Dataset.push_to_hub` to store preprocessed files on the Hub.

And to avoid downloading preprocessed files, you can use streaming by setting `streaming=True` in `load_dataset`.

Is there a way to disable the construction of arrow tables, or to make them on the fly as the dataset is being used ?


Fix copy snippet in docs

With this change the lines starting with `...` in the code blocks can be properly copied to clipboard.


Add machine translated multilingual STS benchmark dataset

Hello dear maintainer, are there any comments or questions about this PR?

@iamollas thanks for the feedback. I did not see the template.
I improved it...

also see here https://github.com/PhilipMay/stsb-multi-mt


Add documentaton for dataset README.md files

Hi ! We are using the [datasets-tagging app](https://github.com/huggingface/datasets-tagging) to select the tags to add.

We are also adding the full list of tags in #2107
This covers multilinguality, language_creators, licenses, size_categories and task_categories.

In general if you want to add a tag that doesn't exist (for example for a custom license) you must make it start with `other-` and then a custom tag name.

edit (@theo-m) if you ever find yourself resorting to adding an `other-*` tag, please do ping us somewhere so we can think about adding it to the "official" list :)

@lhoestq hmm - ok thanks for the answer.
To be honest I am not sure if this issue can be closed now.
I just wanted to point out that this should either be documented or linked in the documentation.
If you feel like it is (will be) please just close this.

We're still working on the validation+documentation in this.
Feel free to keep this issue open till we've added them

@lhoestq what is the status on this? Did you add documentation?

Hi ! There's the tagging app at https://huggingface.co/datasets/tagging/ that you can use.
It shows the list of all the tags you can use.

It is based on all the tag sets defined in this folder:
https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources

@lhoestq is there something like this form Models?

I don't think so. Feel free to take a look at the tags of other models (example [here](https://huggingface.co/bert-base-uncased/blob/main/README.md)). But we should definitely have some docs or an app to write the tags. Feel free to open an issue in the `transformers` repo or in the `huggingface_hub` repo so we can discuss this

Hi,
the dataset README files have special headers.
Somehow a documenation of the allowed values and tags is missing.
Could you add that?

Just to give some concrete questions that should be answered imo:
- which values can be passted to multilinguality?
- what should be passed to language_creators?
- which values should licenses have? What do I say when it is a custom license? Should I add a link?
- how should I choose size_categories ? What are valid ranges?
- what are valid task_categories?

Thanks
Philip


change bibtex template to author instead of authors

Trailing whitespace was removed. So more changes in diff than just this fix.

Hi,
IMO when using BibTex Author should be used instead of Authors.
See here: http://www.bibtex.org/Using/de/

Thanks
Philip


Update metadata if dataset features are modified

@lhoestq I'll try to add a test later if you think this approach with the wrapper is good.

Awesome thank you !
Yes this approach with a wrapper is good :)

@lhoestq Added a test. To verify that this change fixes the problem, replace:
```
!pip install datasets==1.5
```
with:
```
!pip install git+https://github.com/mariosasko/datasets-1.git@update-metadata
```
in the first cell of the notebook that is attached to the linked issue.

The CI failure is unrelated I think (building the docs locally doesn't throw an error).

The CI fail for the docs has been fixed on master.
Merging :)

This PR adds a decorator that updates the dataset metadata if a previously executed transform modifies its features.
Fixes #2083


change user permissions to -rw-r--r--

I tried this with `ade_corpus_v2` dataset. `ade_corpus_v2-train.arrow` (downloaded dataset) and `cache-25d41a4d3c2d8a25.arrow` (ran a mapping function on the dataset) both had file permission with octal value of `0644`.

Fix for #2065


Fix max_wait_time in requests

it was handled as a min time, not max cc @SBrandeis


Requesting to add a new dataset

## Adding a Dataset
- **Name:** CUAD - Contract Understanding Atticus Dataset
- **Description:** As one of the only large, specialized NLP benchmarks annotated by experts, CUAD can serve as a challenging research benchmark for the broader NLP community.
- **Paper:** https://arxiv.org/abs/2103.06268
- **Data:** https://github.com/TheAtticusProject/cuad/
- **Motivation:** good domain specific datasets are valuable

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


`concatenate_datasets` throws error when changing the order of datasets to concatenate

Hi,

this bug is related to `Dataset.{remove_columns, rename_column, flatten}` not propagating the change to the schema metadata when the info features are updated, so this line is the culprit:
```python
common_voice_train = common_voice_train.remove_columns(['client_id', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment'])

```
The order is important because the resulting dataset inherits the schema metadata of the first dataset passed to the `concatenate_datasets(...)` function (`pa.concat_tables` [docs](https://arrow.apache.org/docs/python/generated/pyarrow.concat_tables.html)). I'll try to fix this ASAP.

Hey,

I played around with the `concatenate_datasets(...)` function: https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate_datasets#datasets.concatenate_datasets

and noticed that when the order in which the datasets are concatenated changes an error is thrown where it should not IMO.

Here is a google colab to reproduce the error: https://colab.research.google.com/drive/17VTFU4KQ735-waWZJjeOHS6yDTfV5ekK?usp=sharing


Updated card using information from data statement and datasheet

I updated and clarified the REFreSD [data card](https://github.com/mcmillanmajora/datasets/blob/refresd_card/datasets/refresd/README.md) with information from the Eleftheria's [website](https://elbria.github.io/post/refresd/). I added brief descriptions where the initial card referred to the paper, and I also recreated some of the tables in the paper to show relevant dataset statistics.

I'll email Eleftheria to see if she has any comments on the card.


Fix docstrings issues

Fix docstring issues.


Multidimensional arrays in a Dataset

Hi !

This is actually supported ! but not yet in `from_pandas`.
You can use `from_dict` for now instead:
```python
from datasets import Dataset, Array2D, Features, Value
import pandas as pd
import numpy as np

dataset = {
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
}
dataset = Dataset.from_dict(dataset)
```

This will work but to use it with the torch formatter you must specify the `Array2D` feature type in order to tell the shape:
```python
from datasets import Dataset, Array2D, Features, Value
import pandas as pd
import numpy as np

dataset = {
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
}
dataset = Dataset.from_dict(dataset, features=Features({
    "bbox": Array2D(shape=(3, 4), dtype="int64"),
    "input_ids": Value("int64")
}))
dataset.set_format("torch")
print(dataset[0]['bbox'])
# tensor([[1, 2, 3, 4],
#         [1, 2, 3, 4],
#         [1, 2, 3, 4]])
```
If you don't specify the `Array2D` feature type, then the inferred type will be Sequence(Sequence(Value("int64"))) and therefore the torch formatter will return list of tensors

Thanks for the explanation.
With my original DataFrame, I did
```
dataset = dataset.to_dict("list")
```
and then the rest of the transformation from dictionary works just fine.

Hi,

I'm trying to put together a `datasets.Dataset` to be used with LayoutLM which is available in `transformers`. This model requires as input the bounding boxes of each of the token of a sequence. This is when I realized that `Dataset` does not support multi-dimensional arrays as a value for a column in a row.

The following code results in conversion error in pyarrow (`pyarrow.lib.ArrowInvalid: ('Can only convert 1-dimensional array values', 'Conversion failed for column bbox with type object')`)

```
from datasets import Dataset
import pandas as pd
import numpy as np

dataset = pd.DataFrame({
    'bbox': [
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]]),
        np.array([[1,2,3,4],[1,2,3,4],[1,2,3,4]])
    ],
    'input_ids': [1, 2, 3, 4]
})
dataset = Dataset.from_pandas(dataset)
```

Since I wanted to use pytorch for the downstream training task, I also tried a few ways to directly put in a column of 2-D pytorch tensor in a formatted dataset, but I can only have a list of 1-D tensors, or a list of arrays, or a list of lists.

```
import torch
from datasets import Dataset
import pandas as pd

dataset = pd.DataFrame({
    'bbox': [
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]],
        [[1,2,3,4],[1,2,3,4],[1,2,3,4]]
    ],
    'input_ids': [1, 2, 3, 4]
})
dataset = Dataset.from_pandas(dataset)

def test(examples):
    return {'bbbox': torch.Tensor(examples['bbox'])}
dataset = dataset.map(test)
print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])

dataset.set_format(type='torch', columns=['input_ids', 'bbox'], output_all_columns=True)
print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])

def test2(examples):
    return {'bbbox': torch.stack(examples['bbox'])}
dataset = dataset.map(test2)

print(dataset[0]['bbox'])
print(dataset[0]['bbbox'])
```

Is is possible to support n-D arrays/tensors in datasets?
It seems that it can also be useful for this [feature request](https://github.com/huggingface/datasets/issues/263).


Refactorize Metric.compute signature to force keyword arguments only

Minor refactoring of Metric.compute signature to force the use of keyword arguments, by using the single star syntax.


MemoryError when computing WER metric

metric bug

A bug in a metric script

Hi ! Thanks for reporting.
We're indeed using `jiwer` to compute the WER.

Maybe instead of calling `jiwer.wer` once for all the preditions/references we can compute the WER iteratively to avoid memory issues ? I'm not too familial with `jiwer` but this must be possible.

Currently the code to compute the WER is defined here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/metrics/wer/wer.py#L93-L94

Hi,

I've just pushed a pull request that is related to this issue https://github.com/huggingface/datasets/pull/2169. It's not iterative, but it should avoid memory errors. It's based on the editdistance python library. An iterative implementation should be as easy as storing scores and words stepwise and dividing at the end.

I see, this was solved by other thread. Ok, let me know if you want to switch the implementation for any reason :)

Thanks for diving into this anyway ^^'
As you said this actually got solved a few days ago

Someone created an issue https://github.com/jitsi/jiwer/issues/40 at jiwer which shows that this is still a problem in the current version. Would be curious to figure out how this can be fixed by jiwer... :) I assume that it runs of out memory because it's trying to compute the WER over (too many) test samples?

Hi !

It's computed iteratively so not sure what could go wrong

https://github.com/huggingface/datasets/blob/8afd0ba8c27800a55ea69d9fcd702dc97d9c16d8/metrics/wer/wer.py#L100-L106

@NiklasHoltmeyer what version of `datasets` are you running ?

One possible explanation might be that it is the user who is passing all the sentences in a single element to `wer.compute`?

As current implementation iterates over the elements of `predictions` and `references`, this can be problematic if `predictions` and `references` contain a single huge element each.

This could be the case, for example, with a single string with all sentences:
```python
result["predicted"] = "One sentence. Other sentence."
```
or with a __double__ nested list of sentence lists
```python
result["predicted"] = [[ ["One sentence."], ["Other sentence"] ]]
```

The user should check the dimensions of the data structure passed to `predictions` and `references`.

Hi all,

in my case I was using and older version of datasets and, as @albertvillanova points out, passing the full list of sentences for the metric calculation. The problem was in the way jiwer implements WER, as it tries to compute WER for the full list at once instead of doing it element-wise. I think that with the latest implementation of datasets, or by using the alternative WER function that I've contributed on this [pull request](https://github.com/huggingface/datasets/pull/2169) there shouldn't be memory errors.

@lhoestq i was using Datasets==1.5.0 with 1.6.1 it worked (atleast the first run) but 1.5.0 is not compatible with my preprocessing. i cant save my dataset to a parquet file while using the latest datasets version

->
```
  File "../preprocess_dataset.py", line 132, in <module>
    pq.write_table(train_dataset.data, f'{resampled_data_dir}/{data_args.dataset_config_name}.train.parquet')
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 1674, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 588, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
TypeError: Argument 'table' has incorrect type (expected pyarrow.lib.Table, got ConcatenationTable)
```

if i do
```
import pyarrow.parquet as pq
...
...
pq.write_table(train_dataset.data, 'train.parquet')
pq.write_table(eval_dataset.data, 'eval.parquet')
```

while using 1.6.1. and its working with 1.5.0

Hi ! You can pass dataset.data.table instead of dataset.data to pq.write_table

This seems to be working so far! Thanks!

Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.


Bump huggingface_hub version

`0.0.2 => 0.0.6`


Issue: Dataset download error

dataset bug

A bug in a dataset script provided in the library

Hi @XuhuiZhou, thanks for reporting this issue.

Indeed, the old links are no longer valid (404 Not Found error), and the script must be updated with the new links to Google Drive.

It would be nice to update the urls indeed !

To do this, you just need to replace the urls in `iwslt2017.py` and then update the dataset_infos.json file with
```
datasets-cli test ./datasets/iwslt2017 --all_configs --save_infos --ignore_verifications
```

Is this a command to update my local files or fix the file Github repo in general? (I am not so familiar with the datasets-cli command here)

I also took a brief look at the **Sharing your dataset** section, looks like I could fix that locally and push it to the repo? I guess we are "canonical" category?

This command will update your local file. Then you can open a Pull Request to push your fix to the github repo :)
And yes you are right, it is a "canonical" dataset, i.e. a dataset script defined in this github repo (as opposed to dataset repositories of users on the huggingface hub)

Hi, thanks for the answer.

I gave a try to the problem today. But I encountered an upload error:

```
git push -u origin fix_link_iwslt
Enter passphrase for key '/home2/xuhuizh/.ssh/id_rsa':
ERROR: Permission to huggingface/datasets.git denied to XuhuiZhou.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

Any insight here?

By the way, when I run the datasets-cli command, it shows the following error, but does not seem to be the error coming from `iwslt.py`

```
Traceback (most recent call last):
  File "/home2/xuhuizh/anaconda3/envs/UMT/bin/datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/home2/xuhuizh/projects/datasets/src/datasets/commands/datasets_cli.py", line 35, in main
    service.run()
  File "/home2/xuhuizh/projects/datasets/src/datasets/commands/test.py", line 141, in run
    try_from_hf_gcs=False,
  File "/home2/xuhuizh/projects/datasets/src/datasets/builder.py", line 579, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home2/xuhuizh/projects/datasets/src/datasets/builder.py", line 639, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/home2/xuhuizh/projects/datasets/src/datasets/utils/info_utils.py", line 32, in verify_checksums
    raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
datasets.utils.info_utils.ExpectedMoreDownloadedFiles: {'https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz'}
```

Hi ! To create a PR on this repo your must fork it and create a branch on your fork. See how to fork the repo [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#start-by-preparing-your-environment).
And to make the command work without the `ExpectedMoreDownloadedFiles` error, you just need to use the `--ignore_verifications` flag.

Hi @XuhuiZhou,

As @lhoestq has well explained, you need to fork HF's repository, create a feature branch in your fork, push your changes to it and then open a Pull Request to HF's upstream repository. This is so because at HuggingFace Datasets we follow a development model called "Fork and Pull Model". You can find more information here:
- [Understanding the GitHub flow](https://guides.github.com/introduction/flow/)
- [Forking Projects](https://guides.github.com/activities/forking/)

Alternatively, if you find all these steps too complicated, you can use the GitHub official command line tool: [GitHub CLI](https://cli.github.com/). Once installed, in order to create a Pull Request, you only need to use this command:
```shell
gh pr create --web
```
This utility will automatically create the fork, push your changes and open a Pull Request, under the hood.

The download link in `iwslt2017.py` file does not seem to work anymore.

For example, `FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnted/texts/zh/en/zh-en.tgz`

Would be nice if we could modify it script and use the new downloadable link?


ConnectionError: Couldn't reach common_voice.py

Hi @LifaSun, thanks for reporting this issue.

Sometimes, GitHub has some connectivity problems. Could you confirm that the problem persists?

@albertvillanova Thanks! It works well now.

When I run:
from datasets import load_dataset, load_metric

common_voice_train = load_dataset("common_voice", "zh-CN", split="train+validation")
common_voice_test = load_dataset("common_voice", "zh-CN", split="test")

Got:
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/master/datasets/common_voice/common_voice.py

Version:
1.4.1

Thanks!  @lhoestq @LysandreJik @thomwolf


Fix size categories in YAML Tags

> It would be great if there was a way to make the task categories consistent too. For this, the streamlit app can look into all the datasets and check for existing categories and show them in the list. This may add some consistency.

We can also update the task lists here: https://github.com/huggingface/datasets-tagging/blob/main/task_set.json

Hi @lhoestq,

Thanks for approving.
How do I add the new categories to the tagging app? What I have added is till `1T` and not `1M`.

I'll also check the task list :)

Thanks,
Gunjan

I think you can change it here: https://github.com/huggingface/datasets-tagging/blob/main/tagging_app.py#L412-L423

Hi @lhoestq,

I have made a PR for size categories on `datasets-tagging`

For tags, I have thought of adding more tags and categories, based on what I know about the existing datasets, any list will not be exhaustive because the contributors can be very specific or very general. Hence, there could be a continuous process of evaluating existing tags and adding more and more.

```json
{
    "image-classification": {
        "description": "image classification tasks",
        "options": [
            "multi-class-classification",
            "multi-label-classification",
            "other"
        ]
    },
    "conditional-text-generation": {
        "description": "data-to-text and text transduction tasks such as translation or summarization",
        "options": [
            "machine-translation",
            "sentence-splitting-fusion",
            "extractive-and-abstractive-summarization",
            "abstractive-summarization",
            "extractive-summarization",
            "multi-document-summarization",
            "table-to-text",
            "text-simplification",
            "explanation-generation",
            "stuctured-to-text",
            "other"
        ]
    },
    "conditional-speech-generation": {
        "description": "speech generation tasks",
        "options": [
            "text-to-speech",
            "speech-translation",
            "other"
        ]
    },

    "conditional-structure-generation":{
        "description": "text or speech to structured data",
        "options":[
            "knowlege-graph-mining",
            "code-generation",
        ]
    },
    "question-answering": {
        "description": "question answering tasks",
        "options": [
            "open-domain-qa",
            "closed-domain-qa",
            "multiple-choice-qa",
            "extractive-qa",
            "abstractive-qa",
            "conversational-qa",
            "multi-document-qa",
            "other"
        ]
    },
    "speech-classification": {
        "description": "speech to label tasks",
        "options": [
            "other"
        ]
    },
    "sequence-modeling": {
        "description": "such as language, speech or dialogue modeling",
        "options": [
            "dialogue-modeling",
            "language-modeling",
            "speech-modeling",
            "multi-turn",
            "slot-filling",
            "other"
        ]
    },
    "speech-recognition": {
        "description": "speech to text tasks",
        "options": [
            "automatic-speech-recognition",
            "other"
        ]
    },
    "structure-prediction": {
        "description": "predicting structural properties of the text, such as syntax",
        "options": [
            "coreference-resolution",
            "named-entity-recognition",
            "part-of-speech-tagging",
            "parsing",
            "sentence-segmentation",
            "single-span-prediction",
            "multi-span-prediction",
            "clause-or-phrase-segmentation",
            "dependency-parsing",
            "constituency-parsing",
            "other"
        ]
    },

    "text-classification": {
        "description": "predicting a class index or boolean value",
        "options": [
            "acceptability-classification",
            "entity-linking-classification",
            "relation-extraction",
            "common-sense-reasoning",
            "fact-checking",
            "intent-classification",
            "multi-class-classification",
            "multi-label-classification",
            "natural-language-inference",
            "semantic-similarity-classification",
            "sentiment-classification",
            "topic-classification",
            "emotion-classification",
            "token-classification",
            "word-sense-disambiguation",
            "offense-classification",
            "hate-speech-classification",
            "language-classification",
            "bias-classification",
            "other"
        ]
    },
    "text-retrieval": {
        "description": "information or text retrieval tasks",
        "options": [
            "document-retrieval",
            "utterance-retrieval",
            "entity-linking-retrieval",
            "fact-checking-retrieval",
            "other"
        ]
    },
    "text-scoring": {
        "description": "text scoring tasks, predicting a real valued score for some text",
        "options": [
            "semantic-similarity-scoring",
            "sentiment-scoring",
            "other"
        ]
    },
    "other": {
        "description": "raw data or other task families",
        "options": [
            "data-mining",
            "raw-text",
            "raw-speech",
            "raw-image",
            "other"
        ]
    }
}
```
I'll sort this when adding it to the .json. Also, I'll change categories according to this if this seems okay to you and commit it to this PR.

I'll also fix spelling others, and some categories which are partially correct, for e.g. `other-machine-translation` to the correct tag.
Lastly, with the options also we can add a description to make it easier for the users to understand what we mean by each option. Example, for "emotion-classification", we can explain what kinds of data we are talking about, or what we mean by "single-span-prediction", etc.

Good idea thank you ! Can you open a PR on datasets-tagging for the tasks as well ?
Also you can update the dataset card with the new tasks categories in another PR if you don't mind

Hi @lhoestq,

Thanks, what all do I need to add to merge this PR?

We can merge this one once the PR on dataset sizes is merged on `datasets-tagging` ;)

Hi @lhoestq,

One problem with this approach is that for datasets like `ccaligned_multilingual`, the infos won't be complete because we don't have all configs. In that case, people might face trouble finding the datatset using the tag. Although, they probably won't be checking the size tag for a dataset like that.

What do you think?

CC @theo-m

For datasets like `ccaligned_multilingual` it's important to have all the tags for users to search and find it. Currently is has the full list of tags (without the config names). So you can actually find the dataset, but you don't know what tag correspond to what configuration.

This PR fixes several `size_categories` in YAML tags and makes them consistent. Additionally, I have added a few more categories after `1M`, up to `1T`. I would like to add that to the streamlit app also.

This PR also adds a couple of infos that I found missing.

The code for generating this:
```python
for dataset in sorted(os.listdir('./datasets/')):
    if '.' not in dataset and dataset not in ['c4', 'csv', 'downloads', 'cc100', 'ccaligned_multilingual', 'celeb_a', 'chr_en', 'emea', 'glue']:
        infos = {}
        stats = {}
        st = ''
        with open(f'datasets/{dataset}/README.md') as f:
            d = f.read()
        start_dash = d.find('---') + 3
        end_dash = d[start_dash:].find('---') + 3
        rest_text = d[end_dash + 3:]
        try:
            full_yaml = OmegaConf.create(d[start_dash:end_dash])
            readme = OmegaConf.to_container(full_yaml['size_categories'], resolve=True)
        except Exception as e:
            print(e)
            continue
        try:
            with open(f'datasets/{dataset}/dataset_infos.json') as f:
                data = json.load(f)
        except Exception as e:
            print(e)
            continue  # Skip those without infos.
        done_set = set([])
        num_keys = len(data.keys())
        for keys in data:
            # dataset = load_dataset('opus100', f'{dirs}')
            total = 0
            for split in data[keys]['splits']:
                total = total + data[keys]['splits'][split]['num_examples']
            if total < 1000:
                st += "- n<1K" + '\n'
                infos[keys] = ["n<1K"]
            elif total >= 1000 and total < 10000:
                infos[keys] = ["1K<n<10K"]
            elif total >= 10000 and total < 100000:
                infos[keys] = ["10K<n<100K"]
            elif total >= 100000 and total < 1000000:
                infos[keys] = ["100K<n<1M"]
            elif total >= 1000000 and total < 10000000:
                infos[keys] = ["1M<n<10M"]
            elif total >= 10000000 and total < 100000000:
                infos[keys] = ["10M<n<100M"]
            elif total >= 100000000 and total < 1000000000:
                infos[keys] = ["100M<n<1B"]
            elif total >= 1000000000 and total < 10000000000:
                infos[keys] = ["1B<n<10B"]
            elif total >= 10000000000 and total < 100000000000:
                infos[keys] = ["10B<n<100B"]
            elif total >= 100000000000 and total < 1000000000000:
                infos[keys] = ["100B<n<1T"]
            else:
                infos[keys] = ["n>1T"]
            done_set = done_set.union(infos[keys])
        if (isinstance(readme, list) and list(infos.values())[0] != readme) or (isinstance(readme, dict) and readme != infos):

            print('-' * 30)
            print(done_set)
            print(f"Changing Full YAML for {dataset}")
            print(OmegaConf.to_yaml(full_yaml))

            if len(done_set) == 1:
                full_yaml['size_categories'] = list(done_set)
            else:
                full_yaml['size_categories'] = dict([(k, v) for k, v in sorted(infos.items(), key=lambda x: x[0])])

            full_yaml_string = OmegaConf.to_yaml(full_yaml)
            print('-' * 30)
            print(full_yaml_string)
            inp = input('Do you wish to continue?(Y/N)')
            if inp == 'Y':
                with open(f'./datasets/{dataset}/README.md', 'w') as f:
                    f.write('---\n')
                    f.write(full_yaml_string)
                    f.write('---')
                    f.write(rest_text)
            else:
                break
```

Note that the lower-bound is inclusive. I'm unsure if this is how it is done in the tagging app.

EDIT:
It would be great if there was a way to make the task categories consistent too. For this, the streamlit app can look into all the datasets and check for existing categories and show them in the list. This may add some consistency.

EDIT:
I understand this will not work for cases where only the infos for some of the configs are present, for example: `ccaligned_multingual` has only 5 out of several configs present, and infos has only information about them.  Hence, I have skipped a few datasets in the code, if there are more such datasets, then I'll ignore them too.


Fixes check of TF_AVAILABLE and TORCH_AVAILABLE

# What is this PR doing

This PR implements the checks if `Tensorflow` and `Pytorch` are available the same way as `transformers` does it. I added the additional checks for the different `Tensorflow` and `torch` versions.  #2068


Fix docstring issues

I think I will stop pushing to this PR, so that it can me merged for today release.

I will open another PR for further fixing docs.

Do you agree, @lhoestq ?

Fix docstring issues.


Multiprocessing is slower than single process

bug

Something isn't working

```python
# benchmark_filter.py
import logging
import sys
import time

from datasets import load_dataset, set_caching_enabled

if __name__ == "__main__":
    set_caching_enabled(False)
    logging.basicConfig(level=logging.DEBUG)

    bc = load_dataset("bookcorpus")

    now = time.time()
    try:
        bc["train"].filter(lambda x: len(x["text"]) < 64, num_proc=int(sys.argv[1]))
    except Exception as e:
        print(f"cancelled: {e}")
    elapsed = time.time() - now

    print(elapsed)
```

Running `python benchmark_filter.py 1` (20min+) is faster than `python benchmark_filter.py 2` (2hrs+)


ArrowInvalid issue for squad v2 dataset

Hi ! This error happens when you use `map` in batched mode and then your function doesn't return the same number of values per column.

Indeed since you're using `map` in batched mode, `prepare_validation_features` must take a batch as input (i.e. a dictionary of multiple rows of the dataset), and return a batch.

However it seems like `tokenized_examples` doesn't have the same number of elements in each field. One field seems to have `1180` elements while `candidate_attention_mask` only has `1178`.

Hello, I am using the huggingface official question answering example notebook (https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb).

In the prepare_validation_features function, I made some modifications to tokenize a new set of quesions with the original contexts and save them in three different list called candidate_input_dis, candidate_attetion_mask and candidate_token_type_ids. When I try to run the next cell for dataset.map, I got the following error:

`ArrowInvalid: Column 1 named candidate_attention_mask expected length 1180 but got length 1178`

My code is as follows:

```
def generate_candidate_questions(examples):
  val_questions = examples["question"]
  candididate_questions = random.sample(datasets["train"]["question"], len(val_questions))
  candididate_questions = [x[:max_length] for x in candididate_questions]
  return candididate_questions

def prepare_validation_features(examples, use_mixing=False):
  pad_on_right = tokenizer.padding_side == "right"
  tokenized_examples = tokenizer(
      examples["question" if pad_on_right else "context"],
      examples["context" if pad_on_right else "question"],
      truncation="only_second" if pad_on_right else "only_first",
      max_length=max_length,
      stride=doc_stride,
      return_overflowing_tokens=True,
      return_offsets_mapping=True,
      padding="max_length",
  )
  if use_mixing:
    candidate_questions = generate_candidate_questions(examples)
    tokenized_candidates = tokenizer(
        candidate_questions if pad_on_right else examples["context"],
        examples["context"] if pad_on_right else candidate_questions,
        truncation="only_second" if pad_on_right else "only_first",
        max_length=max_length,
        stride=doc_stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )

  sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")

  tokenized_examples["example_id"] = []

  if use_mixing:
    tokenized_examples["candidate_input_ids"] = tokenized_candidates["input_ids"]
    tokenized_examples["candidate_attention_mask"] = tokenized_candidates["attention_mask"]
    tokenized_examples["candidate_token_type_ids"] = tokenized_candidates["token_type_ids"]

  for i in range(len(tokenized_examples["input_ids"])):
      sequence_ids = tokenized_examples.sequence_ids(i)
      context_index = 1 if pad_on_right else 0

      sample_index = sample_mapping[i]
      tokenized_examples["example_id"].append(examples["id"][sample_index])

      tokenized_examples["offset_mapping"][i] = [
          (o if sequence_ids[k] == context_index else None)
          for k, o in enumerate(tokenized_examples["offset_mapping"][i])
      ]

  return tokenized_examples

validation_features = datasets["validation"].map(
    lambda xs: prepare_validation_features(xs, True),
    batched=True,
    remove_columns=datasets["validation"].column_names
)
```

I guess this might happen because of the batched=True. I see similar issues in this repo related to arrow table length mismatch error, but in their cases, the numbers vary a lot. In my case, this error always happens when the expected length and unexpected length are very close. Thanks for the help!


Add and fix docstring for NamedSplit

Maybe we should add some other split classes?

Add and fix docstring for `NamedSplit`, which was missing.


PyTorch not available error on SageMaker GPU docker though it is installed

Hey @sivakhno,

how does your `requirements.txt` look like to install the `datasets` library and which version of it are you running? Can you try to install `datasets>=1.4.0`

Hi @philschmid - thanks for suggestion. I am using `datasets==1.4.1`.
I have also tried using `torch=1.6.0` (docker `763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.6.0-gpu-py3 `), but the error is the same.

Could paste the code you use the start your training job and the fine-tuning script you run?

@sivakhno this should be now fixed in `datasets>=1.5.0`.

@philschmid Recently released tensorflow-macos seems to be missing.

I get en error when running data loading using SageMaker SDK

```
  File "main.py", line 34, in <module>
    run_training()
  File "main.py", line 25, in run_training
    dm.setup('fit')
  File "/opt/conda/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 92, in wrapped_fn
    return fn(*args, **kwargs)
  File "/opt/ml/code/data_module.py", line 103, in setup
    self.dataset[split].set_format(type="torch", columns=self.columns)
  File "/opt/conda/lib/python3.6/site-packages/datasets/fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 995, in set_format
    _ = get_formatter(type, **format_kwargs)
File "/opt/conda/lib/python3.6/site-packages/datasets/formatting/__init__.py", line 114, in get_formatter
    raise _FORMAT_TYPES_ALIASES_UNAVAILABLE[format_type]
ValueError: PyTorch needs to be installed to be able to return PyTorch tensors.
```

when trying to execute dataset loading using this notebook  https://github.com/PyTorchLightning/pytorch-lightning/blob/master/notebooks/04-transformers-text-classification.ipynb, specifically lines

```
self.columns = [c for c in self.dataset[split].column_names if c in self.loader_columns]
self.dataset[split].set_format(type="torch", columns=self.columns)
```

The SageMaker docker image used is 763104351884.dkr.ecr.eu-central-1.amazonaws.com/pytorch-training:1.4.0-gpu-py3 .

By running container interactively I have checked that torch loading completes successfully by executing `https://github.com/huggingface/datasets/blob/master/src/datasets/config.py#L39`.

Also as a first line in the data loading module I have

```
import os
os.environ["USE_TF"] = "0"
os.environ["USE_TORCH"] = "1"
````

But unfortunately the error stills persists. Any suggestions would be appreciated as I am stack.
Many Thanks!


Multiprocessing windows error

Hi ! Thanks for reporting.
This looks like a bug, could you try to provide a minimal code example that reproduces the issue ? This would be very helpful !

Otherwise I can try to run the wav2vec2 code above on my side but probably not this week..

```
from datasets import load_dataset

dataset = load_dataset('glue', 'mrpc', split='train')

updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)

```

I was able to copy some of the shell
This is repeating every half second
Win 10, Anaconda with python 3.8, datasets installed from main branche
```

  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    _check_not_importing_main()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 134, in _check_not_importing_main
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    exitcode = _main(fd, parent_sentinel)
    raise RuntimeError('''
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 125, in _main
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.    return _run_module_code(code, init_globals, run_name,
    prepare(preparation_data)

  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 236, in prepare
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\test.py", line 6, in <module>
    updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1370, in map
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 119, in Pool
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 212, in __init__
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\test.py", line 6, in <module>
    self._repopulate_pool()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 303, in _repopulate_pool
    updated_dataset = dataset.map(lambda example: {'sentence1': 'My sentence: ' + example['sentence1']}, num_proc=4)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1370, in map
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 326, in _repopulate_pool_static
    with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 119, in Pool
    w.start()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\process.py", line 121, in start
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 212, in __init__
    self._popen = self._Popen(self)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\context.py", line 327, in _Popen
    self._repopulate_pool()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 303, in _repopulate_pool
    return Popen(process_obj)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\popen_spawn_win32.py", line 45, in __init__
    return self._repopulate_pool_static(self._ctx, self.Process,
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\pool.py", line 326, in _repopulate_pool_static
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 134, in _check_not_importing_main
    w.start()
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\process.py", line 121, in start
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...
```

Thanks this is really helpful !
I'll try to reproduce on my side and come back to you

if __name__ == '__main__':

This line before calling the map function stops the error but the script still repeats endless

Indeed you needed `if __name__ == '__main__'` since accoding to [this stackoverflow post](https://stackoverflow.com/a/18205006):

> On Windows the subprocesses will import (i.e. execute) the main module at start. You need to insert an if __name__ == '__main__': guard in the main module to avoid creating subprocesses recursively.

Regarding the hanging issue, can you try to update `dill` and `multiprocess` ?

```
Traceback (most recent call last):
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 791, in move
    os.rename(src, real_dst)
FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\tmpx9fl_jg8' -> 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\cache-9b4f203a63742dfc.arrow'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\multiprocess\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\Codes\Python Apps\asr\cvtrain.py", line 243, in <module>
    common_voice_train = common_voice_train.map(remove_special_characters, remove_columns=["sentence"])
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1339, in map
    return self._map_single(
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 203, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\fingerprint.py", line 337, in wrapper
    out = func(self, *args, **kwargs)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\site-packages\datasets\arrow_dataset.py", line 1646, in _map_single
    shutil.move(tmp_file.name, cache_file_name)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 805, in move
    copy_function(src, real_dst)
  File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  0%|                                                                                | 0/27771 [00:00<?, ?ex/s]
 File "C:\Users\flozi\anaconda3\envs\wav2vec\lib\shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
OSError: [Errno 22] Invalid argument: 'D:\\huggingfacecache\\common_voice\\de\\6.1.0\\0041e06ab061b91d0a23234a2221e87970a19cf3a81b20901474cffffeb7869f\\cache-9b4f203a63742dfc.arrow'
```

I was adding freeze support before calling the mapping function like this
if __name__ == '__main__':
    freeze_support()
   dataset.map(....)

Usually OSError of an arrow file on windows means that the file is currently opened as a dataset object, so you can't overwrite it until the dataset object falls out of scope.
Can you make sure that there's no dataset object that loaded the `cache-9b4f203a63742dfc.arrow` file ?

Now I understand
The error occures because the script got restarted in another thread, so the object is already loaded.
Still don't have an idea why a new thread starts the whole script again

As described here https://huggingface.co/blog/fine-tune-xlsr-wav2vec2

When using the num_proc argument on windows the whole Python environment crashes and hanging in loop.
For example at the map_to_array part.
An error occures because the cache file already exists and windows throws and error. After this the log crashes into an loop


Fix docstring rendering of Dataset/DatasetDict.from_csv args

Fix the docstring rendering of Dataset/DatasetDict.from_csv args.


Only user permission of saved cache files, not group

enhancement

New feature or request

good first issue

Good for newcomers

Hi ! Thanks for reporting.

Currently there's no way to specify this.

When loading/processing a dataset, the arrow file is written using a temporary file. Then once writing is finished, it's moved to the cache directory (using `shutil.move` [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1646))

That means it keeps the permissions specified by the `tempfile.NamedTemporaryFile` object, i.e. `-rw-------` instead of `-rw-r--r--`. Improving this could be a nice first contribution to the library :)

Hi @lhoestq,
I looked into this and yes you're right. The `NamedTemporaryFile` is always created with mode 0600, which prevents group from reading the file. Should we change the permissions of `tmp_file.name` [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1871) and [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1590), post creation to 0644 inorder for group and others to read it?

Good idea :) we could even update the permissions after the file has been moved by shutil.move [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1899) and [here](https://github.com/huggingface/datasets/blob/f6b8251eb975f66a568356d2a40d86442c03beb9/src/datasets/arrow_dataset.py#L1646) actually.
Apparently they set the default 0600 for temporary files for security reasons, so let's update the umask only after the file has been moved

Would it be possible to actually set the umask based on a user provided argument? For example, a popular usecase my team has is using a shared file-system for processing datasets. This may involve writing/deleting other files, or changing filenames, which a -rw-r--r-- wouldn't fix.

Note that you can get the cache files of a dataset with the `cache_files` attributes.
Then you can `chmod` those files and all the other cache files in the same directory.

Moreover we can probably keep the same permissions after each transform. This way you just need to set the permissions once after doing `load_dataset` for example, and then all the new transformed cached files will have the same permissions.
What do you think ?

This means we'll check the permissions of other `cache_files` already created for a dataset before setting permissions for new  `cache_files`?

You can just check the permission of `dataset.cache_files[0]` imo

> This way you just need to set the permissions once after doing load_dataset for example, and then all the new transformed cached files will have the same permissions.

I was referring to this. Ensuring that newly generated `cache_files` have the same permissions

Yes exactly

I imagine users can first do `load_dataset`, then chmod on the arrow files. After that all the new cache files could have the same permissions as the first arrow files. Opinions on this ?

Sounds nice but I feel this is a sub-part of the approach mentioned by @siddk. Instead of letting the user set new permissions by itself first and then making sure newly generated files have same permissions why don't we ask the user initially only what they want? What are your thoughts?

Yes sounds good. Should this be a parameter in `load_dataset` ? Or an env variable ? Or use the value of `os.umask` ?

Ideally it should be a parameter in `load_dataset` but I'm not sure how important it is for the users (considering only important things should go into `load_dataset` parameters)

I think it's fairly important; for context, our team uses a shared file-system where many folks run experiments based on datasets that are cached by other users.

For example, I might start a training run, downloading a dataset. Then, a couple of days later, a collaborator using the same repository might want to use the same dataset on the same shared filesystem, but won't be able to under the default permissions.

Being able to specify directly in the top-level `load_dataset()` call seems important, but an equally valid option would be to just inherit from the running user's `umask` (this should probably be the default anyway).

So basically, argument that takes a custom set of permissions, and by default, use the running user's umask!

Maybe let's start by defaulting to the user's umask !
Do you want to give it a try @bhavitvyamalik ?

Yeah sure! Instead of using default `0o644` should I first extract umask of current user and then use `os.umask` on it? We can do it inside `Dataset` class so that all folders/files created during the call use running user's umask

You can get the umask using `os.umask` and then I guess you can just use `os.chmod` as in your previous PR, but with the right permissions depending on the umask.

FWIW, we have this issue with other caches - e.g. `transformers` model files. So probably will need to backport this into `transformers` as well.

thanks @thomwolf for the pointer.

Hi @stas00,
For this should we use the same umask code in the respective model directory inside `TRANSFORMERS_CACHE`?

That sounds very right to me, @bhavitvyamalik

The cluster I am working on does not allow me to change the permission of the files with os.chmod. I was wondering if there is any workaround for this? My cache is in a GCP bucket and I can't change file permissions once I mount it.

@vmurahari3 what error do you have exactly ?

I get a permission denied error on https://github.com/huggingface/datasets/blob/b8363e0539c6f0cb5de49af32962cf2eb4c47395/src/datasets/arrow_dataset.py#L2799. I suspect I don't have permissions to change group permissions. I am mounting a GCP bucket through [gcsfuse](https://github.com/GoogleCloudPlatform/gcsfuse).

What @lhoestq is asking for is the full multi-line traceback  - it's almost never enough to show the last line - a full stack is needed to get the context. Thank you!

I wonder if a workaround is to try/except and then issue a warning if this fails?

Hello,

It seems when a cached file is saved from calling `dataset.map` for preprocessing, it gets the user permissions and none of the user's group permissions. As we share data files across members of our team, this is causing a bit of an issue as we have to continually reset the permission of the files. Do you know any ways around this or a way to correctly set the permissions?


Fix ted_talks_iwslt version error

This PR fixes the bug where the version argument would be passed twice if the dataset configuration was created on the fly.

Fixes #2059


[Common Voice] Adapt dataset script so that no manual data download is actually needed

This PR changes the dataset script so that no manual data dir is needed anymore.


docs: fix missing quotation

The json code misses a quote


Cannot load udpos subsets from xtreme dataset using load_dataset()

Good for newcomers

@lhoestq Adding "_" to the class labels in the dataset script will fix the issue.

The bigger issue IMO is that the data files are in conll format, but the examples are tokens, not sentences.

Hi ! Thanks for reporting @adzcodez

> @lhoestq Adding "_" to the class labels in the dataset script will fix the issue.
>
> The bigger issue IMO is that the data files are in conll format, but the examples are tokens, not sentences.

You're right: "_" should be added to the list of labels, and the examples must be sequences of tokens, not singles tokens.

@lhoestq Can you please label this issue with the "good first issue" label? I'm not sure I'll find time to fix this.

To resolve it, the user should:
1. add `"_"` to the list of labels
2. transform the udpos subset to the conll format (I think the preprocessing logic can be borrowed from [the original repo](https://github.com/google-research/xtreme/blob/58a76a0d02458c4b3b6a742d3fd4ffaca80ff0de/utils_preprocess.py#L187-L204))
3. update the dummy data
4. update the dataset info
5. [optional] add info about the data fields structure of the udpos subset to the dataset readme

I tried fixing this issue, but its working fine in the dev version : "1.6.2.dev0"

I think somebody already fixed it.

Hi,

after #2326, the lines with pos tags equal to `"_"` are filtered out when generating the dataset, so this fixes the KeyError described above. However, the udpos subset should be in the conll format i.e. it should yield sequences of tokens and not single tokens, so it would be great to see this fixed (feel free to borrow the logic from [here](https://github.com/google-research/xtreme/blob/58a76a0d02458c4b3b6a742d3fd4ffaca80ff0de/utils_preprocess.py#L187-L204) if you decide to work on this).

Hello,

I am trying to load the udpos English subset from xtreme dataset, but this faces an error during loading. I am using datasets v1.4.1, pip install. I have tried with other udpos languages which also fail, though loading a different subset altogether (such as XNLI) has no issue. I have also tried on Colab and faced the same error.

Reprex is:

`from datasets import load_dataset `
`dataset = load_dataset('xtreme', 'udpos.English')`

The error is:
`KeyError: '_'`

The full traceback is:
KeyError                                  Traceback (most recent call last)
<ipython-input-5-7181359ea09d> in <module>
      1 from datasets import load_dataset
----> 2 dataset = load_dataset('xtreme', 'udpos.English')

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
    738
    739     # Download and prepare data
--> 740     builder_instance.download_and_prepare(
    741         download_config=download_config,
    742         download_mode=download_mode,

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    576                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    577                     if not downloaded_from_gcs:
--> 578                         self._download_and_prepare(
    579                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    580                         )

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    654             try:
    655                 # Prepare split will record examples associated to the split
--> 656                 self._prepare_split(split_generator, **prepare_split_kwargs)
    657             except OSError as e:
    658                 raise OSError(

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\builder.py in _prepare_split(self, split_generator)
    977                 generator, unit=" examples", total=split_info.num_examples, leave=False, disable=not_verbose
    978             ):
--> 979                 example = self.info.features.encode_example(record)
    980                 writer.write(example)
    981         finally:

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example)
    946     def encode_example(self, example):
    947         example = cast_to_python_objects(example)
--> 948         return encode_nested_example(self, example)
    949
    950     def encode_batch(self, batch):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    840     # Nested structures: we allow dict, list/tuples, sequences
    841     if isinstance(schema, dict):
--> 842         return {
    843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in <dictcomp>(.0)
    841     if isinstance(schema, dict):
    842         return {
--> 843             k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
    844         }
    845     elif isinstance(schema, (list, tuple)):

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_nested_example(schema, obj)
    868     # ClassLabel will convert from string to int, TranslationVariableLanguages does some checks
    869     elif isinstance(schema, (ClassLabel, TranslationVariableLanguages, Value, _ArrayXD)):
--> 870         return schema.encode_example(obj)
    871     # Other object should be directly convertible to a native Arrow type (like Translation and Translation)
    872     return obj

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in encode_example(self, example_data)
    647         # If a string is given, convert to associated integer
    648         if isinstance(example_data, str):
--> 649             example_data = self.str2int(example_data)
    650
    651         # Allowing -1 to mean no label.

~\Anaconda3\envs\mlenv\lib\site-packages\datasets\features.py in str2int(self, values)
    605                 if value not in self._str2int:
    606                     value = value.strip()
--> 607                 output.append(self._str2int[str(value)])
    608             else:
    609                 # No names provided, try to integerize

KeyError: '_'


Filtering refactor

I thought at first that the multiproc test was not relevant now that we do stuff only in memory, but I think there's something that's actually broken, my tiny benchmark on bookcorpus runs forever (2hrs+) when I add `num_proc=4` as a kwarg, will investigate 👀

I'm not familiar with the caching you describe for `.map`, I'll look it up.

turns out the multi proc issue is also on master, I won't fix it in this PR but opened #2071 to track the problem.

tracemalloc outputs from this script:

```python
import logging
import sys
import time
import tracemalloc

from datasets import load_dataset, set_caching_enabled

if __name__ == "__main__":
    set_caching_enabled(False)
    logging.basicConfig(level=logging.DEBUG)

    tracemalloc.start()
    bc = load_dataset("bookcorpus")

    now = time.time()
    try:
        snapshot1 = tracemalloc.take_snapshot()
        bc["train"].filter(lambda x: len(x["text"]) < 64, num_proc=int(sys.argv[1]))
    except Exception as e:
        print(f"cancelled: {e}")
        exit(1)
    snapshot2 = tracemalloc.take_snapshot()
    tracemalloc.stop()
    elapsed = time.time() - now

    print(elapsed)
    top_stats = snapshot2.compare_to(snapshot1, "lineno")

    print("[ Top 10 differences ]")
    for stat in top_stats[:10]:
        print(stat)

```

This branch:

```
    ssh://theo@35.205.12.130:22/home/theo/.local/share/miniconda3/envs/datasets/bin/python -u benchmark_filter.py 1
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com:443
    DEBUG:urllib3.connectionpool:https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/datasets/bookcorpus/bookcorpus.py HTTP/1.1" 200 0
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com:443
    DEBUG:urllib3.connectionpool:https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/master/datasets/bookcorpus/bookcorpus.py HTTP/1.1" 200 0
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com:443
    DEBUG:urllib3.connectionpool:https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/master/datasets/bookcorpus/dataset_infos.json HTTP/1.1" 200 0
    WARNING:datasets.builder:Reusing dataset bookcorpus (/home/theo/.cache/huggingface/datasets/bookcorpus/plain_text/1.0.0/af844be26c089fb64810e9f2cd841954fd8bd596d6ddd26326e4c70e2b8c96fc)
      0%|                                                 | 0/74005 [00:00<?, ?ba/s]2021-03-23 10:23:20.051255: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2021-03-23 10:23:20.051304: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
    100%|████████████████████████████████████| 74005/74005 [12:16<00:00, 100.54ba/s]
    815.6356580257416
    [ Top 10 differences ]
    <frozen importlib._bootstrap_external>:580: size=38.0 MiB (+33.7 MiB), count=326226 (+307928), average=122 B
    <frozen importlib._bootstrap>:219: size=7643 KiB (+7553 KiB), count=26372 (+25473), average=297 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/torch/__init__.py:427: size=1291 KiB (+1291 KiB), count=5924 (+5924), average=223 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/abc.py:85: size=1039 KiB (+1026 KiB), count=3428 (+3384), average=310 B
    <frozen importlib._bootstrap_external>:64: size=917 KiB (+891 KiB), count=5300 (+5132), average=177 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/collections/__init__.py:456: size=720 KiB (+709 KiB), count=3403 (+3349), average=217 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/tf_export.py:346: size=607 KiB (+607 KiB), count=3962 (+3962), average=157 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/linecache.py:137: size=998 KiB (+487 KiB), count=9551 (+4517), average=107 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/tf_decorator.py:241: size=367 KiB (+367 KiB), count=5225 (+5225), average=72 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/decorator_utils.py:114: size=359 KiB (+359 KiB), count=330 (+330), average=1114 B
```

On master:
```
    ssh://theo@35.205.12.130:22/home/theo/.local/share/miniconda3/envs/datasets/bin/python -u benchmark_filter.py 1
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): s3.amazonaws.com:443
    DEBUG:urllib3.connectionpool:https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/datasets/bookcorpus/bookcorpus.py HTTP/1.1" 200 0
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com:443
    DEBUG:urllib3.connectionpool:https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/master/datasets/bookcorpus/bookcorpus.py HTTP/1.1" 200 0
    DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): raw.githubusercontent.com:443
    DEBUG:urllib3.connectionpool:https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/master/datasets/bookcorpus/dataset_infos.json HTTP/1.1" 200 0
    WARNING:datasets.builder:Reusing dataset bookcorpus (/home/theo/.cache/huggingface/datasets/bookcorpus/plain_text/1.0.0/af844be26c089fb64810e9f2cd841954fd8bd596d6ddd26326e4c70e2b8c96fc)
      0%|                                                 | 0/74005 [00:00<?, ?ba/s]2021-03-23 12:26:47.219622: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2021-03-23 12:26:47.219669: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    DEBUG:tensorflow:Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client.
    100%|███████████████████████████████████| 74005/74005 [1:02:17<00:00, 19.80ba/s]
    3738.870892047882
    [ Top 10 differences ]
    <frozen importlib._bootstrap_external>:580: size=38.0 MiB (+33.7 MiB), count=326221 (+307919), average=122 B
    <frozen importlib._bootstrap>:219: size=7648 KiB (+7557 KiB), count=26455 (+25555), average=296 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/torch/__init__.py:427: size=1291 KiB (+1291 KiB), count=5924 (+5924), average=223 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/abc.py:85: size=1039 KiB (+1026 KiB), count=3429 (+3385), average=310 B
    <frozen importlib._bootstrap_external>:64: size=917 KiB (+891 KiB), count=5300 (+5132), average=177 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/collections/__init__.py:456: size=720 KiB (+709 KiB), count=3403 (+3349), average=217 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/tf_export.py:346: size=607 KiB (+607 KiB), count=3962 (+3962), average=157 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/linecache.py:137: size=1000 KiB (+489 KiB), count=9569 (+4535), average=107 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/tf_decorator.py:241: size=367 KiB (+367 KiB), count=5225 (+5225), average=72 B
    /home/theo/.local/share/miniconda3/envs/datasets/lib/python3.8/site-packages/tensorflow/python/util/decorator_utils.py:114: size=359 KiB (+359 KiB), count=330 (+330), average=1114 B
```

I'm not concluding much, it seems nothing is really happening to memory on `pyarrow::Table.filter`?

Cool ! Maybe it increases the memory a bit but what's brought in memory is not the resulting Table but something else (not sure what though).
What's the length of the resulting dataset ?
You can also take a look at `pyarrow.total_allocated_memory()` to show how much memory is being used by pyarrow

```diff
diff --git a/benchmarks/benchmark_filter.py b/benchmarks/benchmark_filter.py
index 4b9efd4e..a862c204 100644
--- a/benchmarks/benchmark_filter.py
+++ b/benchmarks/benchmark_filter.py
@@ -1,6 +1,9 @@
 import logging
 import sys
 import time
+import tracemalloc
+
+import pyarrow as pa

 from datasets import load_dataset, set_caching_enabled

@@ -9,13 +12,28 @@ if __name__ == "__main__":
     set_caching_enabled(False)
     logging.basicConfig(level=logging.DEBUG)

+    tracemalloc.start()
     bc = load_dataset("bookcorpus")

     now = time.time()
     try:
+        snapshot1 = tracemalloc.take_snapshot()
+        pamem1 = pa.total_allocated_bytes()
         bc["train"].filter(lambda x: len(x["text"]) < 64, num_proc=int(sys.argv[1]))
+        pamem2 = pa.total_allocated_bytes()
+        snapshot2 = tracemalloc.take_snapshot()
     except Exception as e:
         print(f"cancelled: {e}")
+        exit(1)
+    tracemalloc.stop()
     elapsed = time.time() - now

     print(elapsed)
+    top_stats = snapshot2.compare_to(snapshot1, "lineno")
+
+    print("[ Top 10 differences ]")
+    for stat in top_stats[:10]:
+        print(stat)
+
+    print("[ pyarrow reporting ]")
+    print(f"before: ({pamem1}) after: ({pamem2})")
```

this yields 0-0, does not seem like a good tool 😛 and the documentation is [quite mysterious.](https://arrow.apache.org/docs/python/generated/pyarrow.total_allocated_bytes.html)

Personally if I use your script to benchmark on this branch
```python
bc = load_dataset("bookcorpus", split="train[:1%]")
bc = bc.filter(lambda x: len(x["text"]) < 64)
```

then I get
```
[ pyarrow reporting ]
before: (0) after: (15300672)
```

Maybe you got 0-0 because the filter output is directly garbage collected, since you didn't do
```python
bc["train"] = bc["train"].filter(...)
```
Can you try again on your side just to make sure ?

Even if the documentation doesn't say much, `pa.total_allocated_bytes` if pretty useful, and also very consistent.
It tracks the number of bytes used for arrow data.

> Maybe you got 0-0 because the filter output is directly garbage collected, since you didn't do
>
> ```python
> bc["train"] = bc["train"].filter(...)
> ```
Nice catch! I get 1.74GB for this branch

Looks like we may need to write the filtered table on the disk then.

The other option is to slice the table to keep only the good rows and concatenate them but this is too slow at the moment since slicing is O(n) until #1803 is fixed. I'll work on this issue this afternoon

From investigation it looks like the lib's `Table.filter` cannot send its output to memorymap, asked a question on the mailing list, see [here](https://lists.apache.org/thread.html/r8cd8591ce83a967eb0097a7f31785ac2f3ee95ea371c8c5beb0720ad%40%3Cuser.arrow.apache.org%3E)

fix https://github.com/huggingface/datasets/issues/2032

benchmarking is somewhat inconclusive, currently running on `book_corpus` with:

```python
    bc = load_dataset("bookcorpus")
    now = time.time()
    bc.filter(lambda x: len(x["text"]) < 64)
    elapsed = time.time() - now
    print(elapsed)
```

this branch does it in 233 seconds, master in 1409 seconds.


Error while following docs to load the `ted_talks_iwslt` dataset

@skyprince999 as you authored the PR for this dataset, any comments?

This has been fixed in #2064  by @mariosasko (thanks again !)

The fix is available on the master branch and we'll do a new release very soon :)

I am currently trying to load the `ted_talks_iwslt` dataset into google colab.

The [docs](https://huggingface.co/datasets/ted_talks_iwslt) mention the following way of doing so.

```python
dataset = load_dataset("ted_talks_iwslt", language_pair=("it", "pl"), year="2014")
```

Executing it results in the error attached below.

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-7dcc67154ef9> in <module>()
----> 1 dataset = load_dataset("ted_talks_iwslt", language_pair=("it", "pl"), year="2014")

4 frames
/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
    730         hash=hash,
    731         features=features,
--> 732         **config_kwargs,
    733     )
    734

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in __init__(self, writer_batch_size, *args, **kwargs)
    927
    928     def __init__(self, *args, writer_batch_size=None, **kwargs):
--> 929         super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
    930         # Batch size used by the ArrowWriter
    931         # It defines the number of samples that are kept in memory before writing them

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in __init__(self, cache_dir, name, hash, features, **config_kwargs)
    241             name,
    242             custom_features=features,
--> 243             **config_kwargs,
    244         )
    245

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    337             if "version" not in config_kwargs and hasattr(self, "VERSION") and self.VERSION:
    338                 config_kwargs["version"] = self.VERSION
--> 339             builder_config = self.BUILDER_CONFIG_CLASS(**config_kwargs)
    340
    341         # otherwise use the config_kwargs to overwrite the attributes

/root/.cache/huggingface/modules/datasets_modules/datasets/ted_talks_iwslt/024d06b1376b361e59245c5878ab8acf9a7576d765f2d0077f61751158e60914/ted_talks_iwslt.py in __init__(self, language_pair, year, **kwargs)
    219             description=description,
    220             version=datasets.Version("1.1.0", ""),
--> 221             **kwargs,
    222         )
    223

TypeError: __init__() got multiple values for keyword argument 'version'
```

How to resolve this?

PS: Thanks a lot @huggingface team for creating this great library!


Is it possible to convert a `tfds` to HuggingFace `dataset`?

I was having some weird bugs with `C4`dataset version of HuggingFace, so I decided to try to download `C4`from `tfds`. I would like to know if it is possible to convert a tfds dataset to HuggingFace dataset format :)

I can also open a new issue reporting the bug I'm receiving with `datasets.load_dataset('c4','en')` in the future if you think that it would be useful.

Thanks!


update link to ZEST dataset

Updating the link as the original one is no longer working.