Unexpected behavior when loading cached csv file?

I read a csv file from disk and forgot so specify the right delimiter. When i read the csv file again specifying the right delimiter it had no effect since it was using the cached dataset. I am not sure if this is unwanted behavior since i can always specify `download_mode="force_redownload"`. But i think it would be nice if the information what `delimiter` or what `column_names` were used would influence the identifier of the cached dataset.

Small snippet to reproduce the behavior:
```python
import datasets

with open("dummy_data.csv", "w") as file:
    file.write("test,this;text\n")

print(datasets.load_dataset("csv", data_files="dummy_data.csv", split="train").column_names)
# ["test", "this;text"]

print(datasets.load_dataset("csv", data_files="dummy_data.csv", split="train", delimiter=";").column_names)
# still ["test", "this;text"]
```

By the way, thanks a lot for this amazing library! :)


Better error message for uninitialized metric

When calling `metric.compute()` without having called `metric.add` or `metric.add_batch` at least once, the error was quite cryptic. I added a better error message

Fix #729


Allow custom split names in text dataset

The `text` dataset used to return only splits like train, test and validation. Other splits were ignored.
Now any split name is allowed.

I did the same for `json`, `pandas` and `csv`

Fix #735


Properly delete metrics when a process is killed

Tests are flaky when using metrics in distributed setup.
There is because of one test that make sure that using two possibly incompatible metric computation (same exp id) either works or raises the right error.
However if the error is raised, all the processes of the metric are killed, and the open files (arrow + lock files) are not closed correctly. This causes PermissionError on Windows when deleting the temporary directory.

To fix that I added a `finally` clause in the function passed to multiprocess to properly close the files when the process exits.


[ROUGE] Add description to Rouge metric

Add information about case sensitivity to ROUGE.


Adding CC-100: Monolingual Datasets from Web Crawl Data

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** CC-100: Monolingual Datasets from Web Crawl Data
- **Description:** https://twitter.com/alex_conneau/status/1321507120848625665
- **Paper:** https://arxiv.org/abs/1911.02116
- **Data:** http://data.statmt.org/cc-100/
- **Motivation:** A large scale multi-lingual language modeling dataset. Text is de-duplicated and filtered by how "Wikipedia-like" it is, hopefully helping avoid some of the worst parts of the common crawl.

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Fix metric with cache dir

The cache_dir provided by the user was concatenated twice and therefore causing FileNotFound errors.
The tests didn't cover the case of providing `cache_dir=` for metrics because of a stupid issue (it was not using the right parameter).

I remove the double concatenation and I fixed the tests

Fix #728


Using `Dataset.map` with `n_proc>1` print multiple progress bars

When using `Dataset.map` with `n_proc > 1`, only one of the processes should print a progress bar (to make the output readable). Right now, `n_proc` progress bars are printed.


Fix custom builder caching

The cache directory of a dataset didn't take into account additional parameters that the user could specify such as `features` or any parameter of the builder configuration kwargs (ex: `encoding` for the `text` dataset).

To fix that, the cache directory name now has a suffix that depends on all of them.

Fix #730
Fix #750


How to choose proper download_mode in function load_dataset?

Hi, I am a beginner to datasets and I try to use datasets to load my csv file.
my csv file looks like this

```
text,label
"Effective but too-tepid biopic",3
"If you sometimes like to go to the movies to have fun , Wasabi is a good place to start .",4
"Emerges as something rare , an issue movie that 's so honest and keenly observed that it does n't feel like one .",5
```

First I try to use this command to load my csv file .

``` python
dataset=load_dataset('csv', data_files=['sst_test.csv'])
```

It seems good, but when i try to overwrite the convert_options to convert  'label' columns from int64 to float32 like this.

``` python
import pyarrow as pa
from pyarrow import csv
read_options = csv.ReadOptions(block_size=1024*1024)
parse_options = csv.ParseOptions()
convert_options = csv.ConvertOptions(column_types={'text': pa.string(), 'label': pa.float32()})
dataset = load_dataset('csv', data_files=['sst_test.csv'], read_options=read_options,
                       parse_options=parse_options, convert_options=convert_options)
```

It keeps the same:

```shell
Dataset(features: {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}, num_rows: 2210)
```

I think this issue is caused by the parameter "download_mode" Default to REUSE_DATASET_IF_EXISTS because after I delete the cache_dir, it seems right.

Is it a bug? How to choose proper download_mode to avoid this issue?


Add a `lazy_map` method to `Dataset` and `DatasetDict`

enhancement

New feature or request

The library is great, but it would be even more awesome with a `lazy_map` method implemented on `Dataset` and `DatasetDict`. This would apply a function on a give item but when the item is requested. Two use cases:

1. load image on the fly
2. apply a random function and get different outputs at each epoch (like data augmentation or randomly masking a part of a sentence for BERT-like objectives).


Add option for named splits when using ds.train_test_split

New feature or request

### Feature Request 🚀

Can we add a way to name your splits when using the `.train_test_split` function?

In almost every use case I've come across, I have a `train` and a `test` split in my `DatasetDict`, and I want to create a `validation` split. Therefore, its kinda useless to get a `test` split back from `train_test_split`, as it'll just overwrite my real `test` split that I intended to keep.

### Workaround

this is my hack for dealin with this, for now :slightly_smiling_face:

```python
from datasets import load_dataset

ds = load_dataset('imdb')
ds['train'], ds['validation'] = ds['train'].train_test_split(.1).values()
```


[GEM] add DART data-to-text generation dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** DART
- **Description:** DART consists of 82,191 examples across different domains with each input being a semantic RDF triple set derived from data records in tables and the tree ontology of the schema, annotated with sentence descriptions that cover all facts in the triple set.
- **Paper:** https://arxiv.org/abs/2007.02871v1
- **Data:** https://github.com/Yale-LILY/dart
- **Motivation:** the dataset will likely be included in the GEM benchmark

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


[GEM] Add DART data-to-text generation dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** DART
- **Description:** DART consists of 82,191 examples across different domains with each input being a semantic RDF triple set derived from data records in tables and the tree ontology of the schema, annotated with sentence descriptions that cover all facts in the triple set.
- **Paper:** https://arxiv.org/abs/2007.02871v1
- **Data:** https://github.com/Yale-LILY/dart
- **Motivation:** It will likely be included in the GEM generation evaluation benchmark

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Adding Issue Template for Dataset Requests

adding .github/ISSUE_TEMPLATE/add-dataset.md


Fixed errors in bertscore related to custom baseline

[bertscore version 0.3.6 ](https://github.com/Tiiiger/bert_score) added support for custom baseline files. This update added extra argument `baseline_path` to BERTScorer class as well as some extra boolean parameters `use_custom_baseline` in functions like `get_hash(model, num_layers, idf, rescale_with_baseline, use_custom_baseline)`.

This PR fix those matching errors in bertscore metric implementation.


[GEM] Add Czech Restaurant data-to-text generation dataset

Requesting to add a new dataset

- Paper: https://www.aclweb.org/anthology/W19-8670.pdf
- Data: https://github.com/UFAL-DSG/cs_restaurant_dataset
- The dataset will likely be part of the GEM benchmark


Downloaded datasets are not usable offline

I've been trying to use the IMDB dataset offline, but after downloading it and turning off the internet it still raises  an error from the ```requests``` library trying to reach for the online dataset.
Is this the intended behavior ?
(Sorry, I wrote the the first version of this issue while still on nlp 0.3.0).


Add meta-data to the HANS dataset

good first issue

Good for newcomers

dataset bug

A bug in a dataset script provided in the library

The current version of the [HANS dataset](https://github.com/huggingface/datasets/blob/master/datasets/hans/hans.py) is missing the additional information provided for each example, including the sentence parses, heuristic and subcase.


(Load dataset failure) ConnectionError: Couldn’t reach https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/cnn_dailymail/cnn_dailymail.py

Hey, I want to load the cnn-dailymail dataset for fine-tune.
I write the code like this
from datasets import load_dataset

test_dataset = load_dataset(“cnn_dailymail”, “3.0.0”, split=“train”)

And I got the following errors.

Traceback (most recent call last):
File “test.py”, line 7, in
test_dataset = load_dataset(“cnn_dailymail”, “3.0.0”, split=“test”)
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\load.py”, line 589, in load_dataset
module_path, hash = prepare_module(
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\load.py”, line 268, in prepare_module
local_path = cached_path(file_path, download_config=download_config)
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\utils\file_utils.py”, line 300, in cached_path
output_path = get_from_cache(
File “C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\datasets\utils\file_utils.py”, line 475, in get_from_cache
raise ConnectionError(“Couldn’t reach {}”.format(url))
ConnectionError: Couldn’t reach https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/cnn_dailymail/cnn_dailymail.py

How can I fix this ?


Process 0 very slow when using num_procs with map to tokenizer

<img width="721" alt="image" src="https://user-images.githubusercontent.com/17930170/97066109-776d0d00-15ed-11eb-8bba-bb4d2e0fcc33.png">
The code I am using is
```

        dataset = load_dataset("text", data_files=[file_path], split='train')
        dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                                truncation=True, max_length=args.block_size), num_proc=8)
        dataset.set_format(type='torch', columns=['input_ids'])
        dataset.save_to_disk(file_path+'.arrow')
```


CUDA out of memory

In your dataset ,cuda run out of memory as long as the trainer begins:
however, without changing any other element/parameter,just switch dataset to `LineByLineTextDataset`,everything becames OK.


Start community-provided dataset docs

Continuation of #736 with clean fork.

#### Old description
This is what I did to get the pseudo-labels updated. Not sure if it generalizes, but I figured I would write it down. It was pretty easy because all I had to do was make properly formatted directories and change URLs.

In slack @thomwolf called it a user-namespace dataset, but the docs call it community dataset.
I think the first naming is clearer, but I didn't address that here.

I didn't add metadata, will try that.


Start community-provided dataset docs V2


Use full released xsum dataset

#672 Fix xsum to expand coverage and include IDs
Code based on parser from older version of `datasets/xsum/xsum.py`
@lhoestq


Fix doc links to viewer

It seems #733 forgot some links in the doc :)


Clicking on a metric in the search page points to datasets page giving "Missing dataset" warning

Hi! Sorry if this isn't the right place to talk about the website, I just didn't exactly where to write this.

Searching a metric in https://huggingface.co/metrics gives the right results but clicking on a metric (E.g ROUGE) points to https://huggingface.co/datasets/rouge. Clicking on a metric without searching points to the right page.

Thanks for all the great work!


Error loading ms_marco v2.1 using load_dataset()

Code:
`dataset = load_dataset('ms_marco', 'v2.1')`

Error:
```
`---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-16-34378c057212> in <module>()
      9
     10 # Downloading and loading a dataset
---> 11 dataset = load_dataset('ms_marco', 'v2.1')

10 frames
/usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx)
    353         """
    354         try:
--> 355             obj, end = self.scan_once(s, idx)
    356         except StopIteration as err:
    357             raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Unterminated string starting at: line 1 column 388988661 (char 388988660)
`
```


load_dataset doesn't include `features` in its hash

It looks like the function `load_dataset` does not include what's passed in the `features` argument when creating a hash for a given  dataset. As a result, if a user includes new features from an already downloaded dataset, those are ignored.

Example: some models on the hub have a different ordering for the labels than what `datasets` uses for MNLI so I'd like to do something along the lines of:
```
dataset = load_dataset("glue", "mnli")
features = dataset["train"].features
features["label"] = ClassLabel(names = ['entailment', 'contradiction', 'neutral'])  # new label order
dataset = load_dataset("glue", "mnli", features=features)
```


[XGLUE] Adding new dataset

dataset request

Requesting to add a new dataset

XGLUE is a multilingual GLUE like dataset propesed in this [paper](https://arxiv.org/pdf/2004.01401.pdf).

I'm planning on adding the dataset to the library myself in a couple of weeks.
Also tagging @JetRunner @qiweizhen in case I need some guidance


New version of CompGuessWhat?! with refined annotations

This pull request introduces a few fixes to the annotations for VisualGenome in the CompGuessWhat?! original split.


Add Quail question answering dataset

QuAIL is a multi-domain RC dataset featuring news, blogs, fiction and user stories. Each domain is represented by 200 texts,    which gives us a 4-way data split. The texts are 300-350 word excerpts from CC-licensed texts that were hand-picked so as to make sense to human readers without larger context. Domain diversity mitigates the issue of possible overlap between training and test data of large pre-trained models, which the current SOTA systems are based on. For instance, BERT is trained on Wikipedia + BookCorpus, and was tested on Wikipedia-based SQuAD (Devlin, Chang, Lee, & Toutanova, 2019).

https://text-machine-lab.github.io/blog/2020/quail/ @annargrs


dataset(ngt): add ngt dataset initial loading script

Currently only making the paths to the annotation ELAN (eaf) file and videos available.
This is the first accessible way to download this dataset, which is not manual file-by-file.

Only downloading the necessary files, the annotation files are very small, 20MB for all of them, but the video files are large, 100GB in total, saved in `mpg` format.
I do not intend to actually store these as an uncompressed array of frames, because it will be huge.

Future updates may add pose estimation files for all videos, making it easier to work with this data


Fix emotion description

Fixes the description of the emotion dataset to reflect the class names observed in the data, not the ones described in the paper.

I also took the liberty to make use of `ClassLabel` for the emotion labels.


Dataset Explorer Doesn't Work for squad_es and squad_it

https://huggingface.co/nlp/viewer/?dataset=squad_es
https://huggingface.co/nlp/viewer/?dataset=squad_it

Both pages show "OSError: [Errno 28] No space left on device".


load_dataset for CSV files not working

Similar to #622, I've noticed there is a problem when trying to load a CSV file with datasets.

`
from datasets import load_dataset
`
`
dataset = load_dataset("csv", data_files=["./sample_data.csv"], delimiter="\t", column_names=["title", "text"], script_version="master")
`

Displayed error:
`
...
ArrowInvalid: CSV parse error: Expected 2 columns, got 1
`

I should mention that when I've tried to read data from `https://github.com/lhoestq/transformers/tree/custom-dataset-in-rag-retriever/examples/rag/test_data/my_knowledge_dataset.csv` it worked without a problem. I've read that there might be some problems with /r character, so I've removed them from the custom dataset, but the problem still remains.

I've added a colab reproducing the bug, but unfortunately I cannot provide the dataset.
https://colab.research.google.com/drive/1Qzu7sC-frZVeniiWOwzoCe_UHZsrlxu8?usp=sharing

Are there any work around for it ?
Thank you


Add OCNLI, a new CLUE dataset

OCNLI stands for Original Chinese Natural Language Inference. It is a corpus for
            Chinese Natural Language Inference, collected following closely the procedures of MNLI,
            but with enhanced strategies aiming for more challenging inference pairs. We want to
            emphasize we did not use human/machine translation in creating the dataset, and thus
            our Chinese texts are original and not translated.


Creating dataset consumes too much memory

Moving this issue from https://github.com/huggingface/datasets/pull/722 here, because it seems like a general issue.

Given the following dataset example, where each example saves a sequence of 260x210x3 images (max length 400):
```python
    def _generate_examples(self, base_path, split):
        """ Yields examples. """

        filepath = os.path.join(base_path, "annotations", "manual", "PHOENIX-2014-T." + split + ".corpus.csv")
        images_path = os.path.join(base_path, "features", "fullFrame-210x260px", split)

        with open(filepath, "r", encoding="utf-8") as f:
            data = csv.DictReader(f, delimiter="|", quoting=csv.QUOTE_NONE)
            for row in data:
                frames_path = os.path.join(images_path, row["video"])[:-7]
                np_frames = []
                for frame_name in os.listdir(frames_path):
                    frame_path = os.path.join(frames_path, frame_name)
                    im = Image.open(frame_path)
                    np_frames.append(np.asarray(im))
                    im.close()

                yield row["name"], {"video": np_frames}
```

The dataset creation process goes out of memory on a machine with 500GB RAM.
I was under the impression that the "generator" here is exactly for that, to avoid memory constraints.

However, even if you want the entire dataset in memory, it would be in the worst case
`260x210x3 x 400 max length x 7000 samples` in bytes (uint8) = 458.64 gigabytes
So I'm not sure why it's taking more than 500GB.

And the dataset creation fails after 170 examples on a machine with 120gb RAM, and after 672 examples on a machine with 500GB RAM.

---

## Info that might help:
Iterating over examples is extremely slow.
![image](https://user-images.githubusercontent.com/5757359/96359590-3c666780-111d-11eb-9347-1f833ad982a9.png)
If I perform this iteration in my own, custom loop (Without saving to file), it runs at 8-9 examples/sec

And you can see at this state it is using 94% of the memory:
![image](https://user-images.githubusercontent.com/5757359/96359606-7afc2200-111d-11eb-8c11-0afbdba1a6a3.png)

And it is only using one CPU core, which is probably why it's so slow:
![image](https://user-images.githubusercontent.com/5757359/96359630-a3841c00-111d-11eb-9ba0-7fd3cdf51d26.png)


Fix TREC urls

The old TREC urls are now redirections.
I updated the urls to the new ones, since we don't support redirections for downloads.

Fix #737


Add wiki dpr multiset embeddings

There are two DPR encoders, one trained on Natural Questions and one trained on a multiset/hybrid dataset.
Previously only the embeddings from the encoder trained on NQ were available. I'm adding the ones from the encoder trained on the multiset/hybrid dataset.
In the configuration you can now specify `embeddings_name="nq"` or `embeddings_name="multiset"`


Replace seqeval code with original classification_report for simplicity

Recently, the original seqeval has enabled us to get per type scores and overall scores as a dictionary.

This PR replaces the current code with the original function(`classification_report`) to simplify it.

Also, the original code has been updated to fix #352.
- Related issue: https://github.com/chakki-works/seqeval/pull/38

```python
from datasets import load_metric
metric = load_metric("seqeval")
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
metric.compute(predictions=y_pred, references=y_true)
# Output: {'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.8}
```


Trec Dataset Connection Error

**Datasets Version:**
1.1.2

**Python Version:**
3.6/3.7

**Code:**
```python
from datasets import load_dataset
load_dataset("trec")
```

**Expected behavior:**
Download Trec dataset and load Dataset object

**Current Behavior:**
Get a connection error saying it couldn't reach http://cogcomp.org/Data/QA/QC/train_5500.label (but the link doesn't seem broken)

<details>
  <summary>Error Logs</summary>

Using custom data configuration default
Downloading and preparing dataset trec/default (download: 350.79 KiB, generated: 403.39 KiB, post-processed: Unknown size, total: 754.18 KiB) to /root/.cache/huggingface/datasets/trec/default/1.1.0/ca4248481ad244f235f4cf277186cad2ee8769f975119a2bbfc41b8932b88bd7...
---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-8-66bf1242096e> in <module>()
----> 1 load_dataset("trec")

10 frames
/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag)
    473         elif response is not None and response.status_code == 404:
    474             raise FileNotFoundError("Couldn't find file at {}".format(url))
--> 475         raise ConnectionError("Couldn't reach {}".format(url))
    476
    477     # Try a second time

ConnectionError: Couldn't reach http://cogcomp.org/Data/QA/QC/train_5500.label

</details>


Start community-provided dataset docs

This is one I did to get the pseudo-labels updated. Not sure if it generalizes, but I figured I would write it down. It was pretty easy because all I had to do was make properly formatted directories and change URLs.

+ In slack @thomwolf  called it a `user-namespace` dataset, but the docs call it `community dataset`.
I think the first naming is clearer, but I didn't address that here.

+ I didn't add metadata, will try that.


Throw error when an unexpected key is used in data_files

I have found that only "train", "validation" and "test" are valid keys in the `data_files` argument. When you use any other ones, those attached files are silently ignored - leading to unexpected behaviour for the users.

So the following, unintuitively, returns only one key (namely `train`).

```python
datasets = load_dataset("text", data_files={"train": train_f, "valid": valid_f})
print(datasets.keys())
# dict_keys(['train'])
```

whereas using `validation` instead, does return the expected result:

```python
datasets = load_dataset("text", data_files={"train": train_f, "validation": valid_f})
print(datasets.keys())
# dict_keys(['train', 'validation'])
```

I would like to see more freedom in which keys one can use, but if that is not possible at least an error should be thrown when using an unexpected key.


Fix GLUE metric description

Small typo: the description says translation instead of prediction.


Update link to dataset viewer

Change 404 error links in quick tour to working ones


dataset(wlasl): initial loading script

takes like 9-10 hours to download all of the videos for the dataset, but it does finish :)


dataset(aslg_pc12): initial loading script

This contains the only current public part of this corpus.

The rest of the corpus is not yet been made public, but this sample is still being used by researchers.


Possible caching bug

bug

Something isn't working

The following code with `test1.txt` containing just "🤗🤗🤗":
```
dataset = datasets.load_dataset('text', data_files=['test1.txt'], split="train", encoding="latin_1")
print(dataset[0])
dataset = datasets.load_dataset('text', data_files=['test1.txt'], split="train", encoding="utf-8")
print(dataset[0])
```
produces this output:
```
Downloading and preparing dataset text/default-15600e4d83254059 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155...
Dataset text downloaded and prepared to /home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155. Subsequent calls will reuse this data.
{'text': 'ð\x9f¤\x97ð\x9f¤\x97ð\x9f¤\x97'}
Using custom data configuration default
Reusing dataset text (/home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155)
{'text': 'ð\x9f¤\x97ð\x9f¤\x97ð\x9f¤\x97'}
```
Just changing the order (and deleting the temp files):
```
dataset = datasets.load_dataset('text', data_files=['test1.txt'], split="train", encoding="utf-8")
print(dataset[0])
dataset = datasets.load_dataset('text', data_files=['test1.txt'], split="train", encoding="latin_1")
print(dataset[0])
```
produces this:
```
Using custom data configuration default
Downloading and preparing dataset text/default-15600e4d83254059 (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155...
Dataset text downloaded and prepared to /home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155. Subsequent calls will reuse this data.
{'text': '🤗🤗🤗'}
Using custom data configuration default
Reusing dataset text (/home/arne/.cache/huggingface/datasets/text/default-15600e4d83254059/0.0.0/52cefbb2b82b015d4253f1aeb1e6ee5591124a6491e834acfe1751f765925155)
{'text': '🤗🤗🤗'}
```

Is it intended that the cache path does not depend on the config entries?

tested with datasets==1.1.2 and python==3.8.5


Better error message when one forgets to call `add_batch` before `compute`

When using metrics, if for some reason a user forgets to call `add_batch` to a metric before `compute` (with no arguments), the error message is a bit cryptic and could probably be made clearer.

## Reproducer

```python
import datasets
import torch
from datasets import Metric

class GatherMetric(Metric):
    def _info(self):
        return datasets.MetricInfo(
            description="description",
            citation="citation",
            inputs_description="kwargs",
            features=datasets.Features({
                'predictions': datasets.Value('int64'),
                'references': datasets.Value('int64'),
            }),
            codebase_urls=[],
            reference_urls=[],
            format='numpy'
        )

    def _compute(self, predictions, references):
        return {"predictions": predictions, "labels": references}

metric = GatherMetric(cache_dir="test-metric")
inputs = torch.randint(0, 2, (1024,))
targets = torch.randint(0, 2, (1024,))

batch_size = 8
for i in range(0, 1024, batch_size):
     pass # User forgets to call `add_batch`
result = metric.compute()
```

## Stack trace:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-267729d187fa> in <module>
      3      pass
      4     # metric.add_batch(predictions=inputs[i:i+batch_size], references=targets[i:i+batch_size])
----> 5 result = metric.compute()

~/git/datasets/src/datasets/metric.py in compute(self, *args, **kwargs)
    380         if predictions is not None:
    381             self.add_batch(predictions=predictions, references=references)
--> 382         self._finalize()
    383
    384         self.cache_file_name = None

~/git/datasets/src/datasets/metric.py in _finalize(self)
    343         elif self.process_id == 0:
    344             # Let's acquire a lock on each node files to be sure they are finished writing
--> 345             file_paths, filelocks = self._get_all_cache_files()
    346
    347             # Read the predictions and references

~/git/datasets/src/datasets/metric.py in _get_all_cache_files(self)
    280         filelocks = []
    281         for process_id, file_path in enumerate(file_paths):
--> 282             filelock = FileLock(file_path + ".lock")
    283             try:
    284                 filelock.acquire(timeout=self.timeout)

TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
```


Passing `cache_dir` to a metric does not work

When passing `cache_dir` to a custom metric, the folder is concatenated to itself at some point and this results in a FileNotFoundError:

## Reproducer

```python
import datasets
import torch
from datasets import Metric

class GatherMetric(Metric):
    def _info(self):
        return datasets.MetricInfo(
            description="description",
            citation="citation",
            inputs_description="kwargs",
            features=datasets.Features({
                'predictions': datasets.Value('int64'),
                'references': datasets.Value('int64'),
            }),
            codebase_urls=[],
            reference_urls=[],
            format='numpy'
        )

    def _compute(self, predictions, references):
        return {"predictions": predictions, "labels": references}

metric = GatherMetric(cache_dir="test-metric")
inputs = torch.randint(0, 2, (1024,))
targets = torch.randint(0, 2, (1024,))

batch_size = 8
for i in range(0, 1024, batch_size):
    metric.add_batch(predictions=inputs[i:i+batch_size], references=targets[i:i+batch_size])
result = metric.compute()
```

## Stack trace:

```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
~/git/datasets/src/datasets/metric.py in _finalize(self)
    349                 reader = ArrowReader(path=self.data_dir, info=DatasetInfo(features=self.features))
--> 350                 self.data = Dataset(**reader.read_files([{"filename": f} for f in file_paths]))
    351             except FileNotFoundError:

~/git/datasets/src/datasets/arrow_reader.py in read_files(self, files, original_instructions)
    227         # Prepend path to filename
--> 228         pa_table = self._read_files(files)
    229         files = copy.deepcopy(files)

~/git/datasets/src/datasets/arrow_reader.py in _read_files(self, files)
    166         for f_dict in files:
--> 167             pa_table: pa.Table = self._get_dataset_from_filename(f_dict)
    168             pa_tables.append(pa_table)

~/git/datasets/src/datasets/arrow_reader.py in _get_dataset_from_filename(self, filename_skip_take)
    291         )
--> 292         mmap = pa.memory_map(filename)
    293         f = pa.ipc.open_stream(mmap)

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.memory_map()

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.MemoryMappedFile._open()

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

FileNotFoundError: [Errno 2] Failed to open local file 'test-metric/gather_metric/default/test-metric/gather_metric/default/default_experiment-1-0.arrow'. Detail: [errno 2] No such file or directory

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-17-e42d43cc981f> in <module>
      2 for i in range(0, 1024, batch_size):
      3     metric.add_batch(predictions=inputs[i:i+batch_size], references=targets[i:i+batch_size])
----> 4 result = metric.compute()

~/git/datasets/src/datasets/metric.py in compute(self, *args, **kwargs)
    380         if predictions is not None:
    381             self.add_batch(predictions=predictions, references=references)
--> 382         self._finalize()
    383
    384         self.cache_file_name = None

~/git/datasets/src/datasets/metric.py in _finalize(self)
    351             except FileNotFoundError:
    352                 raise ValueError(
--> 353                     "Error in finalize: another metric instance is already using the local cache file. "
    354                     "Please specify an experiment_id to avoid colision between distributed metric instances."
    355                 )

ValueError: Error in finalize: another metric instance is already using the local cache file. Please specify an experiment_id to avoid colision between distributed metric instances.
```

The code works when we remove the `cache_dir=...` from the metric.


Parallel downloads progress bar flickers

When there are parallel downloads using the download manager, the tqdm progress bar flickers since all the progress bars are on the same line.

To fix that we could simply specify `position=i` for i=0 to n the number of files to download when instantiating the tqdm progress bar.

Another way would be to have one "master" progress bar that tracks the number of finished downloads, and then one progress bar per process that show the current downloads.


"Checksums didn't match for dataset source files" error while loading openwebtext dataset

Hi,
I have encountered this problem during loading the openwebtext dataset:
```
>>> dataset = load_dataset('openwebtext')
Downloading and preparing dataset openwebtext/plain_text (download: 12.00 GiB, generated: 37.04 GiB, post-processed: Unknown size, total: 49.03 GiB) to /home/admin/.cache/huggingface/datasets/openwebtext/plain_text/1.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/builder.py", line 536, in _download_and_prepare
    self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
  File "/home/admin/workspace/anaconda3/envs/torch1.6-py3.7/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 39, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://zenodo.org/record/3834942/files/openwebtext.tar.xz']
```
I think this problem is caused because the released dataset has changed. Or I should download the dataset manually?

Sorry for release the unfinised issue by mistake.


pretty print dataset objects

Currently, if I do:
```
from datasets import load_dataset
load_dataset("wikihow", 'all', data_dir="/hf/pegasus-datasets/wikihow/")
```
I get:
```

DatasetDict({'train': Dataset(features: {'text': Value(dtype='string', id=None),
'headline': Value(dtype='string', id=None), 'title': Value(dtype='string',
id=None)}, num_rows: 157252), 'validation': Dataset(features: {'text':
Value(dtype='string', id=None), 'headline': Value(dtype='string', id=None),
'title': Value(dtype='string', id=None)}, num_rows: 5599), 'test':
Dataset(features: {'text': Value(dtype='string', id=None), 'headline':
Value(dtype='string', id=None), 'title': Value(dtype='string', id=None)},
num_rows: 5577)})
```

This is not very readable.

Can we either have a better `__repr__` or have a custom method to nicely pprint the dataset object?

Here is my very simple attempt. With this PR, it produces:
```
DatasetDict({
  train:   Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 157252
  })
  validation:   Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 5599
  })
  test:   Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 5577
  })
})
```
I did omit the data types on purpose to make it more readable, but it shouldn't be too difficult to integrate those too.

note that this PR also fixes the inconsistency in output that in master misses enclosing `{}` for Dataset, but it is there for `DatasetDict` - or perhaps it was by design.

I'm totally not attached to this format, just wanting something more readable. One approach could be to serialize to `json.dumps` or something similar. It'd make the indentation simpler.

Thank you.


need to redirect /nlp to /datasets and remove outdated info

It looks like the website still has all the `nlp` data, e.g.: https://huggingface.co/nlp/viewer/?dataset=wikihow&config=all

should probably redirect to: https://huggingface.co/datasets/wikihow

also for some reason the new information is slightly borked. If you look at the old one it was nicely formatted and had the links marked up, the new one is just a jumble of text in one chunk and no markup for links (i.e. not clickable).


Adding pseudo-labels to datasets

I recently [uploaded pseudo-labels](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/precomputed_pseudo_labels.md) for CNN/DM, XSUM and WMT16-en-ro to s3, and thom mentioned I should add them to this repo.
Since pseudo-labels are just a large model's generations on an existing dataset, what is the right way to structure this contribution.
I read https://huggingface.co/docs/datasets/add_dataset.html, but it doesn't really cover this type of contribution.

I could, for example, make a new directory, `xsum_bart_pseudolabels` for each set of pseudolabels or add some sort of parametrization to `xsum.py`: https://github.com/huggingface/datasets/blob/5f4c6e830f603830117877b8990a0e65a2386aa6/datasets/xsum/xsum.py

What do you think @lhoestq ?


datasets(RWTH-PHOENIX-Weather 2014 T): add initial loading script

This is the first sign language dataset in this repo as far as I know.
Following an old issue I opened https://github.com/huggingface/datasets/issues/302.

I added the dataset official REAMDE file, but I see it's not very standard, so it can be removed.


feat(dl_manager): add support for ftp downloads

I am working on a new dataset (#302) and encounter a problem downloading it.

```python
# This is the official download link from https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/
_URL = "ftp://wasserstoff.informatik.rwth-aachen.de/pub/rwth-phoenix/2016/phoenix-2014-T.v3.tar.gz"

dl_manager.download_and_extract(_URL)
```

I get an error:

> ValueError: unable to parse ftp://wasserstoff.informatik.rwth-aachen.de/pub/rwth-phoenix/2016/phoenix-2014-T.v3.tar.gz as a URL or as a local path

I checked, and indeed you don't consider `ftp` as a remote file.
https://github.com/huggingface/datasets/blob/4c2af707a6955cf4b45f83ac67990395327c5725/src/datasets/utils/file_utils.py#L188

Adding `ftp` to that list does not immediately solve the issue, so there probably needs to be some extra work.


OSError: Cannot find data file when not using the dummy dataset in RAG

## Environment info

    transformers version: 3.3.1
    Platform: Linux-4.19
    Python version: 3.7.7
    PyTorch version (GPU?): 1.6.0
    Tensorflow version (GPU?): No
    Using GPU in script?: Yes
    Using distributed or parallel set-up in script?: No

## To reproduce

Steps to reproduce the behaviour:
```
import os
os.environ['HF_DATASETS_CACHE'] = '/workspace/notebooks/POCs/cache'

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration

tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq")
retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=False)
```

Plese note that I'm using the whole dataset: **use_dummy_dataset=False**
After around 4 hours (downloading and some other things) this is returned:

```
Downloading and preparing dataset wiki_dpr/psgs_w100.nq.exact (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /workspace/notebooks/POCs/cache/wiki_dpr/psgs_w100.nq.exact/0.0.0/14b973bf2a456087ff69c0fd34526684eed22e48e0dfce4338f9a22b965ce7c2...

---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    459             try:
--> 460                 return pickle.load(fid, **pickle_kwargs)
    461             except Exception:

UnpicklingError: pickle data was truncated

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    552                 # Prepare split will record examples associated to the split
--> 553                 self._prepare_split(split_generator, **prepare_split_kwargs)
    554             except OSError:

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _prepare_split(self, split_generator)
    840             for key, record in utils.tqdm(
--> 841                 generator, unit=" examples", total=split_info.num_examples, leave=False, disable=not_verbose
    842             ):

/opt/conda/lib/python3.7/site-packages/tqdm/notebook.py in __iter__(self, *args, **kwargs)
    217         try:
--> 218             for obj in super(tqdm_notebook, self).__iter__(*args, **kwargs):
    219                 # return super(tqdm...) will not catch exception

/opt/conda/lib/python3.7/site-packages/tqdm/std.py in __iter__(self)
   1128         try:
-> 1129             for obj in iterable:
   1130                 yield obj

~/.cache/huggingface/modules/datasets_modules/datasets/wiki_dpr/14b973bf2a456087ff69c0fd34526684eed22e48e0dfce4338f9a22b965ce7c2/wiki_dpr.py in _generate_examples(self, data_file, vectors_files)
    131                         break
--> 132                     vecs = np.load(open(vectors_files.pop(0), "rb"), allow_pickle=True)
    133                     vec_idx = 0

/opt/conda/lib/python3.7/site-packages/numpy/lib/npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    462                 raise IOError(
--> 463                     "Failed to interpret file %s as a pickle" % repr(file))
    464     finally:

OSError: Failed to interpret file <_io.BufferedReader name='/workspace/notebooks/POCs/cache/downloads/f34d5f091294259b4ca90e813631e69a6ded660d71b6cbedf89ddba50df94448'> as a pickle

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-10-f28df370ac47> in <module>
      1 # ln -s /workspace/notebooks/POCs/cache /root/.cache/huggingface/datasets
----> 2 retriever = RagRetriever.from_pretrained("facebook/rag-token-nq", index_name="exact", use_dummy_dataset=False)

/opt/conda/lib/python3.7/site-packages/transformers/retrieval_rag.py in from_pretrained(cls, retriever_name_or_path, **kwargs)
    307         generator_tokenizer = rag_tokenizer.generator
    308         return cls(
--> 309             config, question_encoder_tokenizer=question_encoder_tokenizer, generator_tokenizer=generator_tokenizer
    310         )
    311

/opt/conda/lib/python3.7/site-packages/transformers/retrieval_rag.py in __init__(self, config, question_encoder_tokenizer, generator_tokenizer)
    298         self.config = config
    299         if self._init_retrieval:
--> 300             self.init_retrieval()
    301
    302     @classmethod

/opt/conda/lib/python3.7/site-packages/transformers/retrieval_rag.py in init_retrieval(self)
    324
    325         logger.info("initializing retrieval")
--> 326         self.index.init_index()
    327
    328     def postprocess_docs(self, docs, input_strings, prefix, n_docs, return_tensors=None):

/opt/conda/lib/python3.7/site-packages/transformers/retrieval_rag.py in init_index(self)
    238                 split=self.dataset_split,
    239                 index_name=self.index_name,
--> 240                 dummy=self.use_dummy_dataset,
    241             )
    242             self.dataset.set_format("numpy", columns=["embeddings"], output_all_columns=True)

/opt/conda/lib/python3.7/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    609         download_config=download_config,
    610         download_mode=download_mode,
--> 611         ignore_verifications=ignore_verifications,
    612     )
    613

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    474                     if not downloaded_from_gcs:
    475                         self._download_and_prepare(
--> 476                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    477                         )
    478                     # Sync info

/opt/conda/lib/python3.7/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    553                 self._prepare_split(split_generator, **prepare_split_kwargs)
    554             except OSError:
--> 555                 raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))
    556
    557         if verify_infos:

OSError: Cannot find data file.

```

Thanks


Fix train_test_split output format

There was an issue in the `transmit_format` wrapper that returned bad formats when using train_test_split.
This was due to `column_names` being handled as a List[str] instead of Dict[str, List[str]] when the dataset transform (train_test_split) returns a DatasetDict (one set of column names per split).

This should fix @timothyjlaurent 's issue in #620 and fix #676

I added tests for `transmit_format` so that it doesn't happen again


Don't use tqdm 4.50.0

tqdm 4.50.0 introduced permission errors on windows
see [here](https://app.circleci.com/pipelines/github/huggingface/datasets/235/workflows/cfb6a39f-68eb-4802-8b17-2cd5e8ea7369/jobs/1111) for the error details.

For now I just added `<4.50.0` in the setup.py
Hopefully we can find what's wrong with this version soon


Fixes #712 Error in the Overview.ipynb notebook by adding `with_details=True` parameter to `list_datasets` function in Cell 3 of **overview** notebook


Fixes #712 Attribute error in cell 3 of the overview notebook

Fixes the Attribute error in cell 3 of the overview notebook


Use python read for text dataset

As mentioned in #622 the pandas reader used for text dataset doesn't work properly when there are \r characters in the text file.

Instead I switched to pure python using `open` and `read`.
From my benchmark on a 100MB text file, it's the same speed as the previous pandas reader.


Add the official dependabot implementation

This will keep dependencies up to date. This will require a pr label `dependencies` being created in order to function correctly.


Fix reading text files with carriage return symbols

The new pandas-based text reader isn't able to work properly with files that contain carriage return symbols (`\r`).

It fails with the following error message:

```
...
  File "pandas/_libs/parsers.pyx", line 847, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 874, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 918, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 905, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2042, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Buffer overflow caught - possible malformed input file.
```

___
I figured out the pandas uses those symbols as line terminators and this eventually causes the error. Explicitly specifying the `lineterminator` fixes that issue and everything works fine.

Please, consider this PR as it seems to be a common issue to solve.


Error in the notebooks/Overview.ipynb notebook

Hi,

I got the following error in **cell number 3** while exploring the **Overview.ipynb** notebook in google colab. I used the [link ](https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb) provided in the main README file to open it in colab.

```python
# You can access various attributes of the datasets before downloading them
squad_dataset = list_datasets()[datasets.index('squad')]

pprint(squad_dataset.__dict__)  # It's a simple python dataclass
```

Error message
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-8dc805c4949c> in <module>()
      2 squad_dataset = list_datasets()[datasets.index('squad')]
      3
 ----> 4 pprint(squad_dataset.__dict__)  # It's a simple python dataclass

AttributeError: 'str' object has no attribute '__dict__'
```

The object `squad_dataset` is a `str` not a `dataclass` .


fix README typos/ consistency


How to use similarity settings  other then "BM25" in Elasticsearch index ?

**QUESTION : How should we use other similarity algorithms supported by Elasticsearch other than "BM25"  ?**
**ES Reference**
https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html
**HF doc reference:**
https://huggingface.co/docs/datasets/faiss_and_ea.html

**context :**
========

I used the latest Elasticsearch server  version 7.9.2
When I set DFR  which is one of the other similarity algorithms supported by elasticsearch  in the mapping, I get an error

For example DFR that I had tried in the first instance in mappings as below.,
`"mappings": {"properties": {"text": {"type": "text", "analyzer": "standard", "similarity": "DFR"}}},`

I get the following error
RequestError: RequestError(400, 'mapper_parsing_exception', 'Unknown Similarity type [DFR] for field [text]')

The other thing as another option I had tried was to declare "similarity": "my_similarity" within settings and then assigning "my_similarity" inside the mappings as below

`es_config = {
        "settings": {
            "number_of_shards": 1,
             **"similarity":  "my_similarity"**: {
          "type": "DFR",
          "basic_model": "g",
          "after_effect": "l",
          "normalization": "h2",
          "normalization.h2.c": "3.0"
        } ,
            "analysis": {"analyzer": {"stop_standard": {"type": "standard", " stopwords": "_english_"}}},

        },
        "mappings": {"properties": {"text": {"type": "text", "analyzer": "standard", "similarity": "my_similarity"}}},
    }`

For this , I got the following error
RequestError: RequestError(400, 'illegal_argument_exception', 'unknown setting [index.similarity] please check that any required plugins are installed, or check the breaking changes documentation for removed settings')


Datasets performance slow? - 6.4x slower than in memory dataset

I've been very excited about this amazing datasets project. However, I've noticed that the performance can be substantially slower than using an in-memory dataset.

Now, this is expected I guess, due to memory mapping data using arrow files, and you don't get anything for free. But I was surprised at how much slower.

For example, in the `yelp_polarity` dataset (560000 datapoints, or 17500 batches of 32), it was taking me 3:31 to just get process the data and get it on the GPU (no model involved). Whereas, the equivalent in-memory dataset would finish in just 0:33.

Is this expected? Given that one of the goals of this project is also accelerate dataset processing, this seems a bit slower than I would expect. I understand the advantages of being able to work on datasets that exceed memory, and that's very exciting to me, but thought I'd open this issue to discuss.

For reference I'm running a AMD Ryzen Threadripper 1900X 8-Core Processor CPU, with 128 GB of RAM and an NVME SSD Samsung 960 EVO. I'm running with an RTX Titan 24GB GPU.

I can see with `iotop` that the dataset gets quickly loaded into the system read buffers, and thus doesn't incur any additional IO reads. Thus in theory, all the data *should* be in RAM, but in my benchmark code below it's still 6.4 times slower.

What am I doing wrong? And is there a way to force the datasets to completely load into memory instead of being memory mapped in cases where you want maximum performance?

At 3:31 for 17500 batches, that's 12ms per batch. Does this 12ms just become insignificant as a proportion of forward and backward passes in practice, and thus it's not worth worrying about this in practice?

In any case, here's my code `benchmark.py`. If you run it with an argument of `memory` it will copy the data into memory before executing the same test.

``` py
import sys
from datasets import load_dataset
from transformers import DataCollatorWithPadding, BertTokenizerFast
from torch.utils.data import DataLoader
from tqdm import tqdm

if __name__ == '__main__':
    tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
    collate_fn = DataCollatorWithPadding(tokenizer, padding=True)

    ds = load_dataset('yelp_polarity')

    def do_tokenize(x):
        return tokenizer(x['text'], truncation=True)

    ds = ds.map(do_tokenize, batched=True)
    ds.set_format('torch', ['input_ids', 'token_type_ids', 'attention_mask'])

    if len(sys.argv) == 2 and sys.argv[1] == 'memory':
        # copy to memory - probably a faster way to do this - but demonstrates the point
        # approximately 530 batches per second - 17500 batches in 0:33
        print('using memory')
        _ds = [data for data in tqdm(ds['train'])]
    else:
        # approximately 83 batches per second - 17500 batches in 3:31
        print('using datasets')
        _ds = ds['train']

    dl = DataLoader(_ds, shuffle=True, collate_fn=collate_fn, batch_size=32, num_workers=4)

    for data in tqdm(dl):
        for k, v in data.items():
            data[k] = v.to('cuda')
```

For reference, my conda environment is [here](https://gist.github.com/05b6101518ff70ed42a858b302a0405d)

Once again, I'm very excited about this library, and how easy it is to load datasets, and to do so without worrying about system memory constraints.

Thanks for all your great work.


Requirements should specify pyarrow<1

I was looking at the docs on [Perplexity](https://huggingface.co/transformers/perplexity.html) via GPT2. When you load datasets and try to load Wikitext, you get the error,

```
module 'pyarrow' has no attribute 'PyExtensionType'
```
I traced it back to datasets having installed PyArrow 1.0.1 but there's not pinning in the setup file.

https://github.com/huggingface/datasets/blob/e86a2a8f869b91654e782c9133d810bb82783200/setup.py#L68

Downgrading by installing `pip install "pyarrow<1"` resolved the issue.


Fix config creation for data files with NamedSplit

During config creation, we need to iterate through the data files of all the splits to compute a hash.
To make sure the hash is unique given a certain combination of files/splits, we sort the split names.
However the `NamedSplit` objects can't be passed to `sorted` and currently it raises an error: we need to sort the string of their names instead.

Fix #705


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1 (installed from master)
- `datasets` version: 1.0.2 (installed as a dependency from transformers)
- Platform: Linux-4.15.0-118-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.9

I'm testing my own text classification dataset using [this example](https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-generic-text-classification-script-in-tensorflow) from transformers. The dataset is split into train / dev / test, and in csv format, containing just a text and a label columns, using comma as sep. Here's a sample:
```
text,label
"Registra-se a presença do acadêmico <name> . <REL_SEP> Ao me deparar com a descrição de dois autores no polo ativo da ação junto ao PJe , margem esquerda foi informado pela procuradora do reclamante que se trata de uma reclamação trabalhista individual . <REL_SEP> Diante disso , face a ausência injustificada do autor <name> , determina-se o ARQUIVAMENTO do presente processo , com relação a este , nos termos do [[ art . 844 da CLT ]] . <REL_SEP> CUSTAS AUTOR - DISPENSADO <REL_SEP> Custas pelo autor no importe de R $326,82 , calculadas sobre R $16.341,03 , dispensadas na forma da lei , em virtude da concessão dos benefícios da Justiça Gratuita , ora deferida . <REL_SEP> Cientes os presentes . <REL_SEP> Audiência encerrada às 8h42min . <REL_SEP> <name> <REL_SEP> Juíza do Trabalho <REL_SEP> Ata redigida por << <name> >> , Secretário de Audiência .",NO_RELATION
```

However, @Santosh-Gupta reported in #7351 that he had the exact same problem using the ChemProt dataset. His colab notebook is referenced in the following section.

## To reproduce

Steps to reproduce the behavior:

1. Created a new conda environment using conda env -n transformers python=3.7
2. Cloned transformers master, `cd` into it and installed using pip install --editable .  -r examples/requirements.txt
3. Installed tensorflow with `pip install tensorflow`
3. Ran `run_tf_text_classification.py` with the following parameters:

```
--train_file <DATASET_PATH>/train.csv \
--dev_file <DATASET_PATH>/dev.csv \
--test_file <DATASET_PATH>/test.csv \
--label_column_id 1 \
--model_name_or_path neuralmind/bert-base-portuguese-cased \
--output_dir <OUTPUT_PATH> \
--num_train_epochs 4 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--do_train \
--do_eval \
--do_predict \
--logging_steps 1000 \
--evaluate_during_training \
--save_steps 1000 \
--overwrite_output_dir \
--overwrite_cache
```

I have also copied [@Santosh-Gupta 's colab notebook](https://colab.research.google.com/drive/11APei6GjphCZbH5wD9yVlfGvpIkh8pwr?usp=sharing) as a reference.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Here is the stack trace:

```
2020-10-02 07:33:41.622011: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
/media/discoD/repositorios/transformers_pedro/src/transformers/training_args.py:333: FutureWarning: The `evaluate_during_training` argument is deprecated in favor of `evaluation_strategy` (which has more options)
  FutureWarning,
2020-10-02 07:33:43.471648: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-10-02 07:33:43.471791: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.472664: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.7085GHz coreCount: 15 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 238.66GiB/s
2020-10-02 07:33:43.472684: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-02 07:33:43.472765: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-02 07:33:43.472809: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-02 07:33:43.472848: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-02 07:33:43.474209: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-02 07:33:43.474276: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-02 07:33:43.561219: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-02 07:33:43.561397: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.562345: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.563219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-02 07:33:43.563595: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-02 07:33:43.570091: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3591830000 Hz
2020-10-02 07:33:43.570494: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x560842432400 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-02 07:33:43.570511: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-02 07:33:43.570702: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.571599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.7085GHz coreCount: 15 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 238.66GiB/s
2020-10-02 07:33:43.571633: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-10-02 07:33:43.571645: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-10-02 07:33:43.571654: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-10-02 07:33:43.571664: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-10-02 07:33:43.571691: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-10-02 07:33:43.571704: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-10-02 07:33:43.571718: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-10-02 07:33:43.571770: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.572641: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:43.573475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-10-02 07:33:47.139227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-02 07:33:47.139265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2020-10-02 07:33:47.139272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2020-10-02 07:33:47.140323: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:47.141248: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:47.142085: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-10-02 07:33:47.142854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5371 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-10-02 07:33:47.146317: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5608b95dc5c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-10-02 07:33:47.146336: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1070, Compute Capability 6.1
10/02/2020 07:33:47 - INFO - __main__ -   n_replicas: 1, distributed training: False, 16-bits training: False
10/02/2020 07:33:47 - INFO - __main__ -   Training/evaluation parameters TFTrainingArguments(output_dir='/media/discoD/models/datalawyer/pedidos/transformers_tf', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=True, evaluate_during_training=True, evaluation_strategy=<EvaluationStrategy.STEPS: 'steps'>, prediction_loss_only=False, per_device_train_batch_size=4, per_device_eval_batch_size=4, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=4.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Oct02_07-33-43_user-XPS-8700', logging_first_step=False, logging_steps=1000, save_steps=1000, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, dataloader_num_workers=0, past_index=-1, run_name='/media/discoD/models/datalawyer/pedidos/transformers_tf', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=False, tpu_name=None, xla=False)
10/02/2020 07:33:53 - INFO - filelock -   Lock 140407857405776 acquired on /home/user/.cache/huggingface/datasets/e0f1e9ed46db1e2429189f06b479cbd4075c0976104c1aacf8f77d9a53d2ad87.03756fef6da334f50a7ff73608e21b5018229944ca250416ce7352e25d84a552.py.lock
10/02/2020 07:33:53 - INFO - filelock -   Lock 140407857405776 released on /home/user/.cache/huggingface/datasets/e0f1e9ed46db1e2429189f06b479cbd4075c0976104c1aacf8f77d9a53d2ad87.03756fef6da334f50a7ff73608e21b5018229944ca250416ce7352e25d84a552.py.lock
Using custom data configuration default
Traceback (most recent call last):
  File "run_tf_text_classification.py", line 283, in <module>
    main()
  File "run_tf_text_classification.py", line 222, in main
    max_seq_length=data_args.max_seq_length,
  File "run_tf_text_classification.py", line 43, in get_tfds
    ds = datasets.load_dataset("csv", data_files=files)
  File "/media/discoD/anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/load.py", line 604, in load_dataset
    **config_kwargs,
  File "/media/discoD/anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/builder.py", line 158, in __init__
    **config_kwargs,
  File "/media/discoD/anaconda3/envs/transformers/lib/python3.7/site-packages/datasets/builder.py", line 269, in _create_builder_config
    for key in sorted(data_files.keys()):
TypeError: '<' not supported between instances of 'NamedSplit' and 'NamedSplit'
```

## Expected behavior

Should be able to run the text-classification example as described in [https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-generic-text-classification-script-in-tensorflow](https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-generic-text-classification-script-in-tensorflow)

Originally opened this issue at transformers' repository: [https://github.com/huggingface/transformers/issues/7535](https://github.com/huggingface/transformers/issues/7535). @jplu instructed me to open here, since according to [this](https://github.com/huggingface/transformers/issues/7535#issuecomment-702778885) evidence, the problem is from datasets.

Thanks!


Fix remote tests for new datasets

When adding a new dataset, the remote tests fail because they try to get the new dataset from the master branch (i.e., where the dataset doesn't exist yet)
To fix that I reverted to the use of the HF API that fetch the available datasets on S3 that is synced with the master branch


Add hotpot QA

Added the [HotpotQA](https://github.com/hotpotqa/hotpot) multi-hop question answering dataset.


Complete rouge kwargs

In #701 we noticed that some kwargs were missing for rouge


Add rouge 2 and rouge Lsum to rouge metric outputs

Continuation of #700

Rouge 2 and Rouge Lsum were missing in Rouge's outputs.
Rouge Lsum is also useful to evaluate Rouge L for sentences with `\n`

Fix #617


Add rouge-2 in rouge_types for metric calculation

The description of the ROUGE metric says,
```
_KWARGS_DESCRIPTION = """
Calculates average rouge scores for a list of hypotheses and references
Args:
    predictions: list of predictions to score. Each predictions
        should be a string with tokens separated by spaces.
    references: list of reference for each prediction. Each
        reference should be a string with tokens separated by spaces.
Returns:
    rouge1: rouge_1 f1,
    rouge2: rouge_2 f1,
    rougeL: rouge_l f1,
    rougeLsum: rouge_l precision
"""
```

but the `rouge_types` argument defaults to  `rouge_types = ["rouge1", "rougeL"]`, this PR updates and add `rouge2` to the list so as to reflect the description card.


XNLI dataset is not loading

`dataset = datasets.load_dataset(path='xnli')`

showing below error
```
/opt/conda/lib/python3.7/site-packages/nlp/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     36     if len(bad_urls) > 0:
     37         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 38         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     39     logger.info("All the checksums matched successfully" + for_verification_name)
     40

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip']
```

I think URL is now changed to "https://cims.nyu.edu/~sbowman/xnli/XNLI-MT-1.0.zip"


Update README.md

Hey I was just telling my subscribers to check out your repositories
Thank you


Elasticsearch index docs

I added the docs for ES indexes.

I also added a `load_elasticsearch_index` method to load an index that has already been built.

I checked the tests for the ES index and we have tests that mock ElasticSearch.
I think this is good for now but at some point it would be cool to have an end-to-end test with a real ES running.


Update XNLI download link

The old link isn't working anymore. I updated it with the new official link.
Fix #690


Use GitHub instead of aws in remote dataset tests

Recently we switched from aws s3 to github to download dataset scripts.
However in the tests, the dummy data were still downloaded from s3.
So I changed that to download them from github instead, in the MockDownloadManager.

Moreover I noticed that `anli`'s dummy data were quite heavy (18MB compressed, i.e. the entire dataset) so I replaced them with dummy data with few examples.


Rachel ker add dataset/mlsum


Update README.md


Add UI filter to filter datasets based on task

enhancement

New feature or request

This is great work, so huge shoutout to contributors and huggingface.

The [/nlp/viewer](https://huggingface.co/nlp/viewer/) is great and the [/datasets](https://huggingface.co/datasets) page is great. I was wondering if in both or either places we can have a filter that selects if a dataset is good for the following tasks (non exhaustive list)

- Classification
	- Multi label
	- Multi class
- Q&A
- Summarization
- Translation

I believe this feature might have some value, for folks trying to find datasets for a particular task, and then testing their model capabilities.

Thank you :)


XNLI dataset: NonMatchingChecksumError

Hi,
I tried to download "xnli" dataset in colab using
`xnli = load_dataset(path='xnli')`
but got 'NonMatchingChecksumError' error

`NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-27-a87bedc82eeb> in <module>()
----> 1 xnli = load_dataset(path='xnli')

3 frames
/usr/local/lib/python3.6/dist-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     37     if len(bad_urls) > 0:
     38         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 39         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     40     logger.info("All the checksums matched successfully" + for_verification_name)
     41

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip']`

The same code worked well several days ago in colab but stopped working now. Thanks!


Switch to pandas reader for text dataset

Following the discussion in #622 , it appears that there's no appropriate ways to use the payrrow csv reader to read text files because of the separator.

In this PR I switched to pandas to read the file.

Moreover pandas allows to read the file by chunk, which means that you can build the arrow dataset from a text file that is bigger than RAM (we used to have to shard text files an mentioned in https://github.com/huggingface/datasets/issues/610#issuecomment-691672919)

From a test that I did locally on a 1GB text file, the pyarrow reader used to run in 150ms while the new one takes 650ms (multithreading off for pyarrow). This is probably due to chunking since I am having the same speed difference by calling `read()` and calling `read(chunksize)` + `readline()` to read the text file.


Disable tokenizers parallelism in multiprocessed map

It was reported in #620 that using multiprocessing with a tokenizers shows this message:
```
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
```
This message is shown when TOKENIZERS_PARALLELISM is unset.
Moreover if it is set to `true`, then the program just hangs.

To hide the message (if TOKENIZERS_PARALLELISM is unset) and avoid hanging (if TOKENIZERS_PARALLELISM is `true`), then I set TOKENIZERS_PARALLELISM to `false` when forking the process. After forking is gets back to its original value.

Also I added a warning if TOKENIZERS_PARALLELISM was `true` and is set to `false`:
```
Setting TOKENIZERS_PARALLELISM=false for forked processes.
```

cc @n1t0


`ArrowInvalid` occurs while running `Dataset.map()` function

It seems to fail to process the final batch. This [colab](https://colab.research.google.com/drive/1_byLZRHwGP13PHMkJWo62Wp50S_Z2HMD?usp=sharing) can reproduce the error.

Code:

```python
# train_ds = Dataset(features: {
#     'title': Value(dtype='string', id=None),
#     'score': Value(dtype='float64', id=None)
# }, num_rows: 99999)

# suggested in #665
class PicklableTokenizer(BertJapaneseTokenizer):
    def __getstate__(self):
        state = dict(self.__dict__)
        state['do_lower_case'] = self.word_tokenizer.do_lower_case
        state['never_split'] = self.word_tokenizer.never_split
        del state['word_tokenizer']
        return state

    def __setstate(self):
        do_lower_case = state.pop('do_lower_case')
        never_split = state.pop('never_split')
        self.__dict__ = state
        self.word_tokenizer = MecabTokenizer(
            do_lower_case=do_lower_case, never_split=never_split
        )

t = PicklableTokenizer.from_pretrained('bert-base-japanese-whole-word-masking')

encoded = train_ds.map(
    lambda examples: {'tokens': t.encode(examples['title'], max_length=1000)}, batched=True, batch_size=1000
)
```

Error Message:

```
 99% 99/100 [00:22<00:00, 39.07ba/s]
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
<timed exec> in <module>

/usr/local/lib/python3.6/site-packages/datasets/arrow_dataset.py in map(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint)
   1242                 fn_kwargs=fn_kwargs,
   1243                 new_fingerprint=new_fingerprint,
-> 1244                 update_data=update_data,
   1245             )
   1246         else:

/usr/local/lib/python3.6/site-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    151             "output_all_columns": self._output_all_columns,
    152         }
--> 153         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    154         if new_format["columns"] is not None:
    155             new_format["columns"] = list(set(new_format["columns"]) & set(out.column_names))

/usr/local/lib/python3.6/site-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    161             # Call actual function
    162
--> 163             out = func(self, *args, **kwargs)
    164
    165             # Update fingerprint of in-place transforms + update in-place history of transforms

/usr/local/lib/python3.6/site-packages/datasets/arrow_dataset.py in _map_single(self, function, with_indices, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, update_data)
   1496                     if update_data:
   1497                         batch = cast_to_python_objects(batch)
-> 1498                         writer.write_batch(batch)
   1499             if update_data:
   1500                 writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file

/usr/local/lib/python3.6/site-packages/datasets/arrow_writer.py in write_batch(self, batch_examples, writer_batch_size)
    271             typed_sequence = TypedSequence(batch_examples[col], type=col_type, try_type=col_try_type)
    272             typed_sequence_examples[col] = typed_sequence
--> 273         pa_table = pa.Table.from_pydict(typed_sequence_examples)
    274         self.write_table(pa_table)
    275

/usr/local/lib/python3.6/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_pydict()

/usr/local/lib/python3.6/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_arrays()

/usr/local/lib/python3.6/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.validate()

/usr/local/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Column 4 named tokens expected length 999 but got length 1000
```


Dataset browser url is still https://huggingface.co/nlp/viewer/

Might be worth updating to https://huggingface.co/datasets/viewer/


Add features parameter to CSV

Add support for the `features` parameter when loading a csv dataset:

```python
from datasets import load_dataset, Features

features = Features({...})
csv_dataset = load_dataset("csv", data_files=["path/to/my/file.csv"], features=features)
```

I added tests to make sure that it is also compatible with the caching system

Fix #623


Fix column order issue in cast

Previously, the order of the columns in the features passes to `cast_` mattered.
However even though features passed to `cast_` had the same order as the dataset features, it could fail because the schema that was built was always in alphabetical order.
This issue was reported by @lewtun  in #623

To fix that I fixed the schema to follow the order of the arrow table columns.
I also added the possibility to give features that are not ordered the same way as the dataset features.


Fix wrong delimiter in text dataset

The delimiter is set to the bell character as it is used nowhere is text files usually.
However in the text dataset the delimiter was set to `\b` which is backspace in python, while the bell character is `\a`.
I replace \b by \a

Hopefully it fixes issues mentioned by some users in #622


Update navbar chapter titles color

Consistency with the color change that was done in transformers at https://github.com/huggingface/transformers/pull/7423
It makes the background-color of the chapter titles in the docs navbar darker, to differentiate them from the inner sections.

see changes [here](https://691-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html)


Adding missing @property (+2 small flake8 fixes).

Fixes #678


Fix bug related to boolean in GAP dataset.

### Why I did
The value in `row["A-coref"]` and `row["B-coref"]` is `'TRUE'` or `'FALSE'`.
This type is `string`, then `bool('FALSE')` is equal to `True` in Python.
So, both rows are transformed into `True` now.

So, I modified this problem.

### What I did
I modified `bool(row["A-coref"])` and `bool(row["B-coref"])` to `row["A-coref"] == "TRUE"` and `row["B-coref"] == "TRUE"`.

Thank you!


Fix negative ids when slicing with an array

```python
from datasets import Dataset

d = ds.Dataset.from_dict({"a": range(10)})
print(d[[0, -1]])
# OverflowError
```

raises an error because of the negative id.

This PR fixes that.
Fix #668


The download instructions for c4 datasets are not contained in the error message

The manual download instructions are not clear
```The dataset c4 with config en requires manual data.
 Please follow the manual download instructions: <bound method C4.manual_download_instructions of <datasets_modules.datasets.c4.830b0c218bd41fed439812c8dd19dbd4767d2a3faa385eb695cf8666c982b1b3.c4.C4 object at 0x7ff8c5969760>>.
 Manual data can be loaded with `datasets.load_dataset(c4, data_dir='<path/to/manual/data>')
```

Either `@property`  could be added to C4.manual_download_instrcutions (or make it a real property), or the manual_download_instructions function needs to be called I think.

Let me know if you want a PR for this, but I'm not sure which possible fix is the correct one.


Move cache dir root creation in builder's init

We use lock files in the builder initialization but sometimes the cache directory where they're supposed to be was not created. To fix that I moved the builder's cache dir root creation in the builder's init.

Fix #671


train_test_split returns empty dataset item

I try to split my dataset by `train_test_split`, but after that the item in `train` and `test` `Dataset` is empty.
The codes:
```
yelp_data = datasets.load_from_disk('/home/ssd4/huanglianzhe/test_yelp')
    print(yelp_data[0])
    yelp_data = yelp_data.train_test_split(test_size=0.1)
    print(yelp_data)
    print(yelp_data['test'])
    print(yelp_data['test'][0])
```
The outputs:
```
{'stars': 2.0, 'text': 'xxxx'}
Loading cached split indices for dataset at /home/ssd4/huanglianzhe/test_yelp/cache-f9b22d8b9d5a7346.arrow and /home/ssd4/huanglianzhe/test_yelp/cache-4aa26fa4005059d1.arrow
DatasetDict({'train': Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 7219009), 'test': Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 802113)})
Dataset(features: {'stars': Value(dtype='float64', id=None), 'text': Value(dtype='string', id=None)}, num_rows: 802113)
{}    # yelp_data['test'][0] is empty
```


Add custom dataset to NLP?

Is it possible to add a custom dataset such as a .csv to the NLP library?

Thanks.


load_dataset() won't download in Windows

I don't know if this is just me or Windows. Maybe other Windows users can chime in if they don't have this problem. I've been trying to get some of the tutorials working on Windows, but when I use the load_dataset() function, it just stalls and the script keeps running indefinitely without downloading anything. I've waited upwards of 18 hours to download the 'multi-news' dataset (which isn't very big), and still nothing. I've tried running it through different IDE's and the command line, but it had the same behavior. I've also tried it with all virus and malware protection turned off. I've made sure python and all IDE's are exceptions to the firewall and all the requisite permissions are enabled.

Additionally, I checked to see if other packages could download content such as an nltk corpus, and they could. I've also run the same script using Ubuntu and it downloaded fine (and quickly). When I copied the downloaded datasets from my Ubuntu drive to my Windows .cache folder it worked fine by reusing the already-downloaded dataset, but it's cumbersome to do that for every dataset I want to try in my Windows environment.

Could this be a bug, or is there something I'm doing wrong or not thinking of?

Thanks.


blog_authorship_corpus crashed

This is just to report that When I pick blog_authorship_corpus in
https://huggingface.co/nlp/viewer/?dataset=blog_authorship_corpus
I get this:
![image](https://user-images.githubusercontent.com/7553188/94349542-4364f300-0013-11eb-897d-b25660a449f0.png)


Questions about XSUM

Hi there ✋

I'm looking into your `xsum` dataset and I have several questions on that.
So here is how I loaded the data:
```
>>> data = datasets.load_dataset('xsum', version='1.0.1')
>>> data['train']
Dataset(features: {'document': Value(dtype='string', id=None), 'summary': Value(dtype='string', id=None)}, num_rows: 204017)
>>> data['test']
Dataset(features: {'document': Value(dtype='string', id=None), 'summary': Value(dtype='string', id=None)}, num_rows: 11333)
```

The first issue is, the instance counts don’t match what I see on [the dataset's website](https://github.com/EdinburghNLP/XSum/tree/master/XSum-Dataset#what-builds-the-xsum-dataset) (11,333 vs 11,334 for test set; 204,017 vs 204,045 for training set)
```
 … training (90%, 204,045), validation (5%, 11,332), and test (5%, 11,334) set.
```
Any thoughts why? Perhaps @mariamabarham could help here, since she recently had a PR on this dataaset https://github.com/huggingface/datasets/pull/289  (reviewed by @patrickvonplaten)

Another issue is that the instances don't seem to have IDs. The original datasets provides IDs for the instances: https://github.com/EdinburghNLP/XSum/blob/master/XSum-Dataset/XSum-TRAINING-DEV-TEST-SPLIT-90-5-5.json but to be able to use them, the dataset sizes need to match.

CC @jbragg


[BUG] No such file or directory

This happens when both
1. Huggingface datasets cache dir does not exist
2. Try to load a local dataset script

builder.py throws an error when trying to create a filelock in a directory (cache/datasets) that does not exist
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L177

Tested on v1.0.2

@lhoestq


Fix SQuAD metric kwargs description

The `answer_start` field was missing in the kwargs docstring.

This should fix #657

FYI another fix was proposed by @tshrjn in #658 and suggests to remove this field.
However IMO `answer_start` is useful to match the squad dataset format for consistency, even though it is not used in the metric computation. I think it's better to keep it this way, so that you can just give references=squad["answers"] to .compute().

Let me know what sounds the best for you


How to skip a example when running dataset.map

in processing func, I process examples and detect some invalid examples, which I did not want it to be added into train dataset. However I did not find how to skip this recognized invalid example when doing dataset.map.


OverflowError when slicing with an array containing negative ids

```python
from datasets import Dataset

d = ds.Dataset.from_dict({"a": range(10)})

print(d[0])
# {'a': 0}

print(d[-1])
# {'a': 9}

print(d[[0, -1]])
# OverflowError
```
results in
```
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-5-863dc3555598> in <module>
----> 1 d[[0, -1]]

~/Desktop/hf/nlp/src/datasets/arrow_dataset.py in __getitem__(self, key)
   1070             format_columns=self._format_columns,
   1071             output_all_columns=self._output_all_columns,
-> 1072             format_kwargs=self._format_kwargs,
   1073         )
   1074

~/Desktop/hf/nlp/src/datasets/arrow_dataset.py in _getitem(self, key, format_type, format_columns, output_all_columns, format_kwargs)
   1025                 indices = key
   1026
-> 1027             indices_array = pa.array([int(i) for i in indices], type=pa.uint64())
   1028
   1029             # Check if we need to convert indices

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

OverflowError: can't convert negative value to unsigned int
```


Loss not decrease with Datasets and Transformers

HI,

The following script is used to fine-tune a BertForSequenceClassification model on SST2.

The script is adapted from [this colab](https://colab.research.google.com/github/huggingface/datasets/blob/master/notebooks/Overview.ipynb) that presents an example of fine-tuning BertForQuestionAnswering using squad dataset. In that colab, loss works fine. When I adapt it to SST2, the loss fails to decrease as it should. I attach the adapted script below and appreciate anyone pointing out what I miss?

```python
import torch
from datasets import load_dataset
from transformers import BertForSequenceClassification
from transformers import BertTokenizerFast
# Load our training dataset and tokenizer
dataset = load_dataset("glue", 'sst2')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
del dataset["test"] # let's remove it in this demo

# Tokenize our training dataset
def convert_to_features(example_batch):
    encodings = tokenizer(example_batch["sentence"])
    encodings.update({"labels": example_batch["label"]})
    return encodings

encoded_dataset = dataset.map(convert_to_features, batched=True)
# Format our dataset to outputs torch.Tensor to train a pytorch model
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'labels']
encoded_dataset.set_format(type='torch', columns=columns)

# Instantiate a PyTorch Dataloader around our dataset
# Let's do dynamic batching (pad on the fly with our own collate_fn)
def collate_fn(examples):
    return tokenizer.pad(examples, return_tensors='pt')

dataloader = torch.utils.data.DataLoader(encoded_dataset['train'], collate_fn=collate_fn, batch_size=8)
# Now let's train our model
device = 'cuda' if torch.cuda.is_available() else 'cpu'
# Let's load a pretrained Bert model and a simple optimizer
model = BertForSequenceClassification.from_pretrained('bert-base-cased', return_dict=True)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)
model.train().to(device)
for i, batch in enumerate(dataloader):
    batch.to(device)
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    model.zero_grad()
    print(f'Step {i} - loss: {loss:.3}')

```
In case needed.

- datasets == 1.0.2
- transformers == 3.2.0


Does both 'bookcorpus' and 'wikipedia' belong to the same datasets which Google used for pretraining BERT?