kardwalker/my_dataset huggingface dataset:
GitHub issue #244: Add Allociné Dataset
State: closed | Created: 2020-06-05T19:19:26Z

This is a french binary sentiment classification dataset, which was used to train this model: https://huggingface.co/tblard/tf-allocine.

Basically, it's a french "IMDB" dataset, with more reviews.

More info on [this repo](https://github.com/TheophileBlard/french-sentiment-analysis-with-bert).
GitHub issue #242: UnicodeDecodeError when downloading GLUE-MNLI
State: closed | Created: 2020-06-05T16:30:01Z

When I run
```python
dataset = nlp.load_dataset('glue', 'mnli')
```
I get an encoding error (could it be because I'm using Windows?) :
```python
# Lots of error log lines later...
~\Miniconda3\envs\nlp\lib\site-packages\tqdm\std.py in __iter__(self)
   1128         try:
-> 1129             for obj in iterable:
   1130                 yield obj

~\Miniconda3\envs\nlp\lib\site-packages\nlp\datasets\glue\5256cc2368cf84497abef1f1a5f66648522d5854b225162148cb8fc78a5a91cc\glue.py in _generate_examples(self, data_file, split, mrpc_files)
    529 
--> 530                 for n, row in enumerate(reader):
    531                     if is_cola_non_test:

~\Miniconda3\envs\nlp\lib\csv.py in __next__(self)
    110             self.fieldnames
--> 111         row = next(self.reader)
    112         self.line_num = self.reader.line_num

~\Miniconda3\envs\nlp\lib\encodings\cp1252.py in decode(self, input, final)
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24 

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 6744: character maps to <undefined>
```
Anyway this can be solved by specifying to decode in UTF when reading the csv file. I am proposing a PR if that's okay.
GitHub issue #241: Fix empty cache dir
State: closed | Created: 2020-06-05T15:45:22Z

If the cache dir of a dataset is empty, the dataset fails to load and throws a FileNotFounfError. We could end up with empty cache dir because there was a line in the code that created the cache dir without using a temp dir. Using a temp dir is useful as it gets renamed to the real cache dir only if the full process is successful.

So I removed this bad line, and I also reordered things a bit to make sure that we always use a temp dir. I also added warning if we still end up with empty cache dirs in the future.

This should fix #239
GitHub issue #240: Deterministic dataset loading
State: closed | Created: 2020-06-05T09:03:26Z

When calling:
```python 
import nlp
dataset = nlp.load_dataset("trivia_qa", split="validation[:1%]")
```

the resulting dataset is not deterministic over different google colabs. 
After talking to @thomwolf, I suspect the reason to be the use of `glob.glob` in line:

https://github.com/huggingface/nlp/blob/2e0a8639a79b1abc848cff5c669094d40bba0f63/datasets/trivia_qa/trivia_qa.py#L180

which seems to return an ordering of files that depends on the filesystem:
https://stackoverflow.com/questions/6773584/how-is-pythons-glob-glob-ordered

I think we should go through all the dataset scripts and make sure to have deterministic behavior.

A simple solution for `glob.glob()` would be to just replace it with `sorted(glob.glob())` to have everything sorted by name. 

What do you think @lhoestq?
GitHub issue #239: [Creating new dataset] Not found dataset_info.json
State: closed | Created: 2020-06-05T06:15:04Z

Hi, I am trying to create Toronto Book Corpus. #131 

I ran
`~/nlp % python nlp-cli test datasets/bookcorpus --save_infos --all_configs`
but this doesn't create `dataset_info.json` and try to use it
```
INFO:nlp.load:Checking datasets/bookcorpus/bookcorpus.py for additional imports.
INFO:filelock:Lock 139795325778640 acquired on datasets/bookcorpus/bookcorpus.py.lock
INFO:nlp.load:Found main folder for dataset datasets/bookcorpus/bookcorpus.py at /home/yisiang/miniconda3/envs/ml/lib/python3.7/site-packages/nlp/datasets/bookcorpus
INFO:nlp.load:Found specific version folder for dataset datasets/bookcorpus/bookcorpus.py at /home/yisiang/miniconda3/envs/ml/lib/python3.7/site-packages/nlp/datasets/bookcorpus/8e84759446cf68d0b0deb3417e60cc331f30a3bbe58843de18a0f48e87d1efd9
INFO:nlp.load:Found script file from datasets/bookcorpus/bookcorpus.py to /home/yisiang/miniconda3/envs/ml/lib/python3.7/site-packages/nlp/datasets/bookcorpus/8e84759446cf68d0b0deb3417e60cc331f30a3bbe58843de18a0f48e87d1efd9/bookcorpus.py
INFO:nlp.load:Couldn't find dataset infos file at datasets/bookcorpus/dataset_infos.json
INFO:nlp.load:Found metadata file for dataset datasets/bookcorpus/bookcorpus.py at /home/yisiang/miniconda3/envs/ml/lib/python3.7/site-packages/nlp/datasets/bookcorpus/8e84759446cf68d0b0deb3417e60cc331f30a3bbe58843de18a0f48e87d1efd9/bookcorpus.json
INFO:filelock:Lock 139795325778640 released on datasets/bookcorpus/bookcorpus.py.lock
INFO:nlp.builder:Overwrite dataset info from restored data version.
INFO:nlp.info:Loading Dataset info from /home/yisiang/.cache/huggingface/datasets/book_corpus/plain_text/1.0.0
Traceback (most recent call last):
  File "nlp-cli", line 37, in <module>
    service.run()
  File "/home/yisiang/miniconda3/envs/ml/lib/python3.7/site-packages/nlp/commands/test.py", line 78, in run
    builders.append(builder_cls(name=config.name, data_dir=self._data_dir))
  File "/home/yisiang/miniconda3/envs/ml/lib/python3.7/site-packages/nlp/builder.py", line 610, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "/home/yisiang/miniconda3/envs/ml/lib/python3.7/site-packages/nlp/builder.py", line 152, in __init__
    self.info = DatasetInfo.from_directory(self._cache_dir)
  File "/home/yisiang/miniconda3/envs/ml/lib/python3.7/site-packages/nlp/info.py", line 157, in from_directory
    with open(os.path.join(dataset_info_dir, DATASET_INFO_FILENAME), "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/yisiang/.cache/huggingface/datasets/book_corpus/plain_text/1.0.0/dataset_info.json'
```
btw, `ls /home/yisiang/.cache/huggingface/datasets/book_corpus/plain_text/1.0.0/` show me nothing is in the directory.

I have also pushed the script to my fork [bookcorpus.py](https://github.com/richardyy1188/nlp/blob/bookcorpusdev/datasets/bookcorpus/bookcorpus.py).
GitHub issue #238: [Metric] Bertscore : Warning : Empty candidate sentence; Setting recall to be 0.
State: closed | Created: 2020-06-05T02:14:47Z

When running BERT-Score, I'm meeting this warning :

> Warning: Empty candidate sentence; Setting recall to be 0.

Code :

```
import nlp
metric = nlp.load_metric("bertscore")
scores = metric.compute(["swag", "swags"], ["swags", "totally something different"], lang="en", device=0)
```

---

**What am I doing wrong / How can I hide this warning ?**
GitHub issue #236: CompGuessWhat?! dataset 
State: closed | Created: 2020-06-04T19:45:50Z

Hello,

Thanks for the amazing library that you put together. I'm Alessandro Suglia, the first author of CompGuessWhat?!, a recently released dataset for grounded language learning accepted to ACL 2020 ([https://compguesswhat.github.io](https://compguesswhat.github.io)).

This pull-request adds the CompGuessWhat?! splits that have been extracted from the original dataset. This is only part of our evaluation framework because there is also an additional split of the dataset that has a completely different set of games. I didn't integrate it yet because I didn't know what would be the best practice in this case. Let me clarify the scenario.

In our paper, we have a main dataset (let's call it `compguesswhat-gameplay`) and a zero-shot dataset (let's call it `compguesswhat-zs-gameplay`). In the current code of the pull-request, I have only integrated `compguesswhat-gameplay`. I was thinking that it would be nice to have the `compguesswhat-zs-gameplay` in the same dataset class by simply specifying some particular option to the `nlp.load_dataset()` factory. For instance:

```python

cgw = nlp.load_dataset("compguesswhat")
cgw_zs = nlp.load_dataset("compguesswhat", zero_shot=True)
```

The other option would be to have a separate dataset class. Any preferences?
GitHub issue #235: Add experimental datasets
State: closed | Created: 2020-06-04T15:54:56Z

## Adding an *experimental datasets* folder

After using the 🤗nlp library for some time, I find that while it makes it super easy to create new memory-mapped datasets with lots of cool utilities, a lot of what I want to do doesn't work well with the current `MockDownloader` based testing paradigm, making it hard to share my work with the community.

My suggestion would be to add a **datasets\_experimental** folder so we can start making these new datasets public without having to completely re-think testing for every single one. We would allow contributors to submit dataset PRs in this folder, but require an explanation for why the current testing suite doesn't work for them. We can then aggregate the feedback and periodically see what's missing from the current tests.

I have added a **datasets\_experimental** folder to the repository and S3 bucket with two initial datasets: ELI5 (explainlikeimfive) and a Wikipedia Snippets dataset to support indexing (wiki\_snippets)

### ELI5
#### Dataset description
This allows people to download the [ELI5: Long Form Question Answering](https://arxiv.org/abs/1907.09190) dataset, along with two variants based on the r/askscience and r/AskHistorians. Full Reddit dumps for each month are downloaded from [pushshift](https://files.pushshift.io/reddit/), filtered for submissions and comments from the desired subreddits, then deleted one at a time to save space. The resulting dataset is split into a training, validation, and test dataset for r/explainlikeimfive, r/askscience, and r/AskHistorians respectively, where each item is a question along with all of its high scoring answers.

#### Issues with the current testing
1. the list of files to be downloaded is not pre-defined, but rather determined by parsing an index web page at run time. This is necessary as the name and compression type of the dump files changes from month to month as the pushshift website is maintained.  Currently, the dummy folder requires the user to know which files will be downloaded.
2. to save time, the script works on the compressed files using the corresponding python packages rather than first running `download\_and\_extract` then filtering the extracted files.  

### Wikipedia Snippets
#### Dataset description
This script creates a *snippets* version of a source Wikipedia dataset: each article is split into passages of fixed length which can then be indexed using ElasticSearch or a dense indexer. The script currently handles all **wikipedia** and **wiki40b** source datasets, and allows the user to choose the passage length and how much overlap they want across passages. In addition to the passage text, each snippet also has the article title, list of titles of sections covered by the text, and information to map the passage back to the initial dataset at the paragraph and character level.

#### Issues with the current testing
1. The DatasetBuilder needs to call `nlp.load_dataset()`. Currently, testing is not recursive (the test doesn't know where to find the dummy data for the source dataset)
GitHub issue #234: Huggingface NLP, Uploading custom dataset
State: closed | Created: 2020-06-04T05:59:06Z

Hello,

Does anyone know how we can call our custom dataset using the nlp.load command? Let's say that I have a dataset based on the same format as that of squad-v1.1, how am I supposed to load it using huggingface nlp.

Thank you!
GitHub issue #233: Fail to download c4 english corpus
State: closed | Created: 2020-06-04T01:06:38Z

i run following code to download c4 English corpus.

```
dataset = nlp.load_dataset('c4', 'en', beam_runner='DirectRunner'
, data_dir='/mypath')
```

and i met failure as follows

```
Downloading and preparing dataset c4/en (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/adam/.cache/huggingface/datasets/c4/en/2.3.0...
Traceback (most recent call last):
  File "download_corpus.py", line 38, in <module>
    , data_dir='/home/adam/data/corpus/en/c4')
  File "/home/adam/anaconda3/envs/adam/lib/python3.7/site-packages/nlp/load.py", line 520, in load_dataset
    save_infos=save_infos,
  File "/home/adam/anaconda3/envs/adam/lib/python3.7/site-packages/nlp/builder.py", line 420, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/adam/anaconda3/envs/adam/lib/python3.7/site-packages/nlp/builder.py", line 816, in _download_and_prepare
    dl_manager, verify_infos=False, pipeline=pipeline,
  File "/home/adam/anaconda3/envs/adam/lib/python3.7/site-packages/nlp/builder.py", line 457, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/adam/anaconda3/envs/adam/lib/python3.7/site-packages/nlp/datasets/c4/f545de9f63300d8d02a6795e2eb34e140c47e62a803f572ac5599e170ee66ecc/c4.py", line 175, in _split_generators
    dl_manager.download_checksums(_CHECKSUMS_URL)
AttributeError: 'DownloadManager' object has no attribute 'download_checksums

```
can i get any advice?
GitHub issue #232: Nlp cli fix endpoints
State: closed | Created: 2020-06-03T14:10:39Z

With this PR users will be able to upload their own datasets and metrics.

As mentioned in #181, I had to use the new endpoints and revert the use of dataclasses (just in case we have changes in the API in the future).

We now distinguish commands for datasets and commands for metrics:
```bash
nlp-cli upload_dataset <path/to/dataset>
nlp-cli upload_metric <path/to/metric>
nlp-cli s3_datasets {rm, ls}
nlp-cli s3_metrics {rm, ls}
```

Does it sound good to you @julien-c @thomwolf ?
GitHub issue #228: Not able to access the XNLI dataset
State: closed | Created: 2020-06-03T12:25:14Z

When I try to access the XNLI dataset, I get the following error. The option of plain_text get selected automatically and then I get the following error.

```
FileNotFoundError: [Errno 2] No such file or directory: '/home/sasha/.cache/huggingface/datasets/xnli/plain_text/1.0.0/dataset_info.json'
Traceback:
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp_viewer/run.py", line 86, in <module>
    dts, fail = get(str(option.id), str(conf_option.name) if conf_option else None)
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp_viewer/run.py", line 72, in get
    builder_instance = builder_cls(name=conf)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/builder.py", line 610, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/builder.py", line 152, in __init__
    self.info = DatasetInfo.from_directory(self._cache_dir)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/info.py", line 157, in from_directory
    with open(os.path.join(dataset_info_dir, DATASET_INFO_FILENAME), "r") as f:
```

Is it possible to see if the dataset_info.json is correctly placed?
GitHub issue #227: Should we still have to force to install apache_beam to download wikipedia ?
State: closed | Created: 2020-06-03T09:33:20Z

Hi, first thanks to @lhoestq 's revolutionary work, I successfully downloaded processed wikipedia according to the doc. 😍😍😍

But at the first try, it tell me to install `apache_beam` and `mwparserfromhell`, which I thought wouldn't be used according to #204 , it was kind of confusing me at that time.

Maybe we should not force users to install these ? Or we just add them to`nlp`'s dependency ?
GitHub issue #225: [ROUGE] Different scores with `files2rouge`
State: closed | Created: 2020-06-01T00:50:36Z

It seems that the ROUGE score of `nlp` is lower than the one of `files2rouge`.

Here is a self-contained notebook to reproduce both scores : https://colab.research.google.com/drive/14EyAXValB6UzKY9x4rs_T3pyL7alpw_F?usp=sharing

---

`nlp` : (Only mid F-scores)

>rouge1 0.33508031962733364
rouge2 0.14574333776191592
rougeL 0.2321187823256159

`files2rouge` :

>Running ROUGE...
===========================
1 ROUGE-1 Average_R: 0.48873 (95%-conf.int. 0.41192 - 0.56339)
1 ROUGE-1 Average_P: 0.29010 (95%-conf.int. 0.23605 - 0.34445)
1 ROUGE-1 Average_F: 0.34761 (95%-conf.int. 0.29479 - 0.39871)
===========================
1 ROUGE-2 Average_R: 0.20280 (95%-conf.int. 0.14969 - 0.26244)
1 ROUGE-2 Average_P: 0.12772 (95%-conf.int. 0.08603 - 0.17752)
1 ROUGE-2 Average_F: 0.14798 (95%-conf.int. 0.10517 - 0.19240)
===========================
1 ROUGE-L Average_R: 0.32960 (95%-conf.int. 0.26501 - 0.39676)
1 ROUGE-L Average_P: 0.19880 (95%-conf.int. 0.15257 - 0.25136)
1 ROUGE-L Average_F: 0.23619 (95%-conf.int. 0.19073 - 0.28663)

---

When using longer predictions/gold, the difference is bigger.  
**How can I reproduce same score as `files2rouge` ?**

@lhoestq
GitHub issue #224: [Feature Request/Help] BLEURT model -> PyTorch
State: closed | Created: 2020-05-30T18:30:40Z

Hi, I am interested in porting google research's new BLEURT learned metric to PyTorch (because I wish to do something experimental with language generation and backpropping through BLEURT). I noticed that you guys don't have it yet so I am partly just asking if you plan to add it (@thomwolf said you want to do so on Twitter).

I had a go of just like manually using the checkpoint that they publish which includes the weights. It seems like the architecture is exactly aligned with the out-of-the-box BertModel in transformers just with a single linear layer on top of the CLS embedding. I loaded all the weights to the PyTorch model but I am not able to get the same numbers as the BLEURT package's python api. Here is my colab notebook where I tried  https://colab.research.google.com/drive/1Bfced531EvQP_CpFvxwxNl25Pj6ptylY?usp=sharing . If you have any pointers on what might be going wrong that would be much appreciated!

Thank you muchly!
GitHub issue #223: [Feature request] Add FLUE dataset 
State: closed | Created: 2020-05-30T08:52:15Z

Hi,

I think it would be interesting to add the FLUE dataset for francophones or anyone wishing to work on French.

In other requests, I read that you are already working on some datasets, and I was wondering if FLUE was planned.

If it is not the case, I can provide each of the cleaned FLUE datasets (in the form of a directly exploitable dataset rather than in the original xml formats which require additional processing, with the French part for cases where the dataset is based on a multilingual dataframe, etc.).
GitHub issue #222: Colab Notebook breaks when downloading the squad dataset
State: closed | Created: 2020-05-29T22:55:59Z

When I run the notebook in Colab
https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb
breaks when running this cell:
![image](https://user-images.githubusercontent.com/338917/83311709-ffd1b800-a1dd-11ea-8394-3a87df0d7f8b.png)
GitHub issue #221: Fix tests/test_dataset_common.py
State: closed | Created: 2020-05-29T14:12:15Z

When I run the command `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_arcd` while working on #220. I get the error ` unexpected keyword argument "'download_and_prepare_kwargs'"` at the level of  `load_dataset`. Indeed, this [function](https://github.com/huggingface/nlp/blob/master/src/nlp/load.py#L441) no longer has the argument `download_and_prepare_kwargs` but rather `download_config`. So here I change the tests accordingly.
GitHub issue #217: Multi-task dataset mixing
State: open | Created: 2020-05-29T09:22:26Z

It seems like many of the best performing models on the GLUE benchmark make some use of multitask learning (simultaneous training on multiple tasks).

The [T5 paper](https://arxiv.org/pdf/1910.10683.pdf) highlights multiple ways of mixing the tasks together during finetuning:
- **Examples-proportional mixing** - sample from tasks proportionally to their dataset size
- **Equal mixing** - sample uniformly from each task
- **Temperature-scaled mixing** - The generalized approach used by multilingual BERT which uses a temperature T, where the mixing rate of each task is raised to the power 1/T and renormalized. When T=1 this is equivalent to equal mixing, and becomes closer to equal mixing with increasing T.

Following this discussion https://github.com/huggingface/transformers/issues/4340 in [transformers](https://github.com/huggingface/transformers), @enzoampil suggested that the `nlp` library might be a better place for this functionality.

Some method for combining datasets could be implemented ,e.g.
```
dataset = nlp.load_multitask(['squad','imdb','cnn_dm'], temperature=2.0, ...)
```

We would need a few additions:
- Method of identifying the tasks - how can we support adding a string to each task as an identifier: e.g. 'summarisation: '?
- Method of combining the metrics - a standard approach is to use the specific metric for each task and add them together for a combined score.

It would be great to support common use cases such as pretraining on the GLUE benchmark before fine-tuning on each GLUE task in turn. 

I'm willing to write bits/most of this I just need some guidance on the interface and other library details so I can integrate it properly.
GitHub issue #216: ❓ How to get ROUGE-2 with the ROUGE metric ?
State: closed | Created: 2020-05-28T23:47:32Z

I'm trying to use ROUGE metric, but I don't know how to get the ROUGE-2 metric.

---

I compute scores with :

```python
import nlp

rouge = nlp.load_metric('rouge')
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        rouge.add([lp], [lg])
score = rouge.compute()
```

then : _(print only the F-score for readability)_

```python
for k, s in score.items():
    print(k, s.mid.fmeasure)
```

It gives :

>rouge1 0.7915168355671788
rougeL 0.7915168355671788

---

**How can I get the ROUGE-2 score ?**

Also, it's seems weird that ROUGE-1 and ROUGE-L scores are the same. Did I made a mistake ?

@lhoestq
GitHub issue #215: NonMatchingSplitsSizesError when loading blog_authorship_corpus
State: closed | Created: 2020-05-28T22:55:19Z

Getting this error when i run `nlp.load_dataset('blog_authorship_corpus')`. 

```
raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', 
num_bytes=610252351, num_examples=532812, dataset_name='blog_authorship_corpus'), 
'recorded': SplitInfo(name='train', num_bytes=616473500, num_examples=536323, 
dataset_name='blog_authorship_corpus')}, {'expected': SplitInfo(name='validation', 
num_bytes=37500394, num_examples=31277, dataset_name='blog_authorship_corpus'), 
'recorded': SplitInfo(name='validation', num_bytes=30786661, num_examples=27766, 
dataset_name='blog_authorship_corpus')}]
```

Upon checking it seems like there is a disparity between the information in `datasets/blog_authorship_corpus/dataset_infos.json` and what was downloaded. Although I can get away with this by passing `ignore_verifications=True` in `load_dataset`, I'm thinking doing so might give problems later on.
GitHub issue #214: [arrow_dataset.py] add new filter function
State: closed | Created: 2020-05-28T16:21:40Z

The `.map()` function is super useful, but can IMO a bit tedious when filtering certain examples.
I think, filtering out examples is also a very common operation people would like to perform on datasets.

This PR is a proposal to add a `.filter()` function in the same spirit than the `.map()` function.

Here is a sample code you can play around with:

```python
ds = nlp.load_dataset("squad", split="validation[:10%]")


def remove_under_idx_5(example, idx):
    return idx < 5


def only_keep_examples_with_is_in_context(example):
    return "is" in example["context"]


result_keep_only_first_5 = ds.filter(remove_under_idx_5, with_indices=True, load_from_cache_file=False)
result_keep_examples_with_is_in_context = ds.filter(only_keep_examples_with_is_in_context, load_from_cache_file=False)

print("Original number of examples: {}".format(len(ds)))
print("First five examples number of examples: {}".format(len(result_keep_only_first_5)))
print("Is in context examples number of examples: {}".format(len(result_keep_examples_with_is_in_context)))
```
GitHub issue #213: better message if missing beam options
State: closed | Created: 2020-05-28T15:06:57Z

WDYT @yjernite ?
For example:
```python
dataset = nlp.load_dataset('wikipedia', '20200501.aa')
```
Raises:
```
MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
	`load_dataset('wikipedia', '20200501.aa', beam_runner='DirectRunner')`
```
GitHub issue #211: [Arrow writer, Trivia_qa] Could not convert TagMe with type str: converting to null type
State: closed | Created: 2020-05-28T14:38:14Z

Running the following code 

```
import nlp
ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
ds.map(lambda x: x, load_from_cache_file=False)
```

triggers a `ArrowInvalid: Could not convert TagMe with type str: converting to null type` error.

On the other hand if we remove a certain column of `trivia_qa` which seems responsible for the bug, it works:

```
import nlp
ds = nlp.load_dataset("trivia_qa", "rc", split="validation[:1%]")  # this might take 2.3 min to download but it's cached afterwards...
ds.map(lambda x: x, remove_columns=["entity_pages"], load_from_cache_file=False)
```

. Seems quite hard to debug what's going on here... @lhoestq @thomwolf - do you have a good first guess what the problem could be?

**Note** BTW: I think this could be a good test to check that the datasets work correctly: Take a tiny portion of the dataset and check that it can be written correctly.
GitHub issue #209: Add a Google Drive exception for small files
State: closed | Created: 2020-05-28T10:40:17Z

I tried to use the ``nlp`` library to load personnal datasets. I mainly copy-paste the code for ``multi-news`` dataset because my files are stored on Google Drive. 

One of my dataset is small (< 25Mo) so it can be verified by Drive without asking the authorization to the user. This makes the download starts directly. 

Currently the ``nlp`` raises a error: ``ConnectionError: Couldn't reach https://drive.google.com/uc?export=download&id=1DGnbUY9zwiThTdgUvVTSAvSVHoloCgun`` while the url is working. So I just add a new exception as you have already done for  ``firebasestorage.googleapis.com`` : 

```
elif (response.status_code == 400 and "firebasestorage.googleapis.com" in url) or (response.status_code == 405 and "drive.google.com" in url)
```

I make an example of the error that you can run on [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1ae_JJ9uvUt-9GBh0uGZhjbF5aXkl-BPv?usp=sharing)

I avoid the error by adding an exception but there is maybe a proper way to do it.

Many thanks :hugs:
Best,
GitHub issue #207: Remove test set from NLP viewer
State: closed | Created: 2020-05-27T18:32:07Z

While the new [NLP viewer](https://huggingface.co/nlp/viewer/) is a great tool, I think it would be best to outright remove the option of looking at the test sets. At the very least, a warning should be displayed to users before showing the test set. Newcomers to the field might not be aware of best practices, and small things like this can help increase awareness.
GitHub issue #206: [Question] Combine 2 datasets which have the same columns
State: closed | Created: 2020-05-27T16:25:52Z

Hi,

I am using ``nlp`` to load personal datasets. I created summarization datasets in multi-languages based on wikinews. I have one dataset for english and one for german (french is getting to be ready as well). I want to keep these datasets independent because they need different pre-processing (add different task-specific prefixes for T5 : *summarize:* for english and *zusammenfassen:* for german)

My issue is that I want to train T5 on the combined english and german datasets to see if it improves results. So I would like to combine 2 datasets (which have the same columns) to make one and train T5 on it. I was wondering if there is a proper way to do it? I assume that it can be done by combining all examples of each dataset but maybe you have a better solution.

Hoping this is clear enough,

Thanks a lot 😊
Best
GitHub issue #205: Better arrow dataset iter
State: closed | Created: 2020-05-27T16:20:21Z

I tried to play around with `tf.data.Dataset.from_generator` and I found out that the `__iter__` that we have for `nlp.arrow_dataset.Dataset` ignores the format that has been set (torch or tensorflow).
With these changes I should be able to come up with a `tf.data.Dataset` that uses lazy loading, as asked in #193.
GitHub issue #204: Add Dataflow support + Wikipedia + Wiki40b
State: closed | Created: 2020-05-27T12:32:49Z

# Add Dataflow support + Wikipedia + Wiki40b

## Support datasets processing with Apache Beam

Some datasets are too big to be processed on a single machine, for example: wikipedia, wiki40b, etc. Apache Beam allows to process datasets on many execution engines like Dataflow, Spark, Flink, etc.

To process such datasets with Beam, I added a command to run beam pipelines `nlp-cli run_beam path/to/dataset/script`. Then I used it to process the english + french wikipedia, and the english of wiki40b.
The processed arrow files are on GCS and are the result of a Dataflow job.

I added a markdown documentation file in `docs` that explains how to use it properly.

## Load already processed datasets

Now that we have those datasets already processed, I made it possible to load datasets that are already processed. You can do `load_dataset('wikipedia', '20200501.en')` and it will download the processed files from the Hugging Face GCS directly into the user's cache and be ready to use !

The Wikipedia dataset was already asked in #187 and this PR should soon allow to add Natural Questions as asked in #129 

## Other changes in the code

To make things work, I had to do a few adjustments:
- add a `ship_files_with_pipeline` method to the `DownloadManager`. This is because beam pipelines can be run in the cloud and therefore need to have access to your downloaded data. I used it in the wikipedia script:
    ```python
    if not pipeline.is_local():
            downloaded_files = dl_manager.ship_files_with_pipeline(downloaded_files, pipeline)
    ```
- add parquet to arrow conversion. This is because the output of beam pipelines are parquet files so we need to convert them to arrow and have the arrow files on GCS
- add a test script with a dummy beam dataset
- minor adjustments to allow read/write operations on remote files using `apache_beam.io.filesystems.FileSystems` if we want (it can be connected to gcp, s3, hdfs, etc...)
GitHub issue #203: Raise an error if no config name for datasets like glue
State: closed | Created: 2020-05-27T09:03:58Z

Some datasets like glue (see #130) and scientific_papers (see #197) have many configs.
For example for glue there are cola, sst2, mrpc etc.

Currently if a user does `load_dataset('glue')`, then Cola is loaded by default and it can be confusing. Instead, we should raise an error to let the user know that he has to pick one of the available configs (as proposed in #152). For example for glue, the message looks like:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
	`load_dataset('glue', 'cola')`
```

The error is raised if the config name is missing and if there are >=2 possible configs.
GitHub issue #202: Mistaken `_KWARGS_DESCRIPTION` for XNLI metric
State: closed | Created: 2020-05-27T08:34:42Z

Hi!

The [`_KWARGS_DESCRIPTION`](https://github.com/huggingface/nlp/blob/7d0fa58641f3f462fb2861dcdd6ce7f0da3f6a56/metrics/xnli/xnli.py#L45) for the XNLI metric uses `Args` and `Returns` text from [BLEU](https://github.com/huggingface/nlp/blob/7d0fa58641f3f462fb2861dcdd6ce7f0da3f6a56/metrics/bleu/bleu.py#L58) metric:

```
_KWARGS_DESCRIPTION = """
Computes XNLI score which is just simple accuracy.
Args:
    predictions: list of translations to score.
        Each translation should be tokenized into a list of tokens.
    references: list of lists of references for each translation.
        Each reference should be tokenized into a list of tokens.
    max_order: Maximum n-gram order to use when computing BLEU score.
    smooth: Whether or not to apply Lin et al. 2004 smoothing.
Returns:
    'bleu': bleu score,
    'precisions': geometric mean of n-gram precisions,
    'brevity_penalty': brevity penalty,
    'length_ratio': ratio of lengths,
    'translation_length': translation_length,
    'reference_length': reference_length
"""
```

But it should be something like:

```
_KWARGS_DESCRIPTION = """
Computes XNLI score which is just simple accuracy.
Args:
    predictions: Predicted labels.
    references: Ground truth labels.
Returns:
    'accuracy': accuracy
```
GitHub issue #200: [ArrowWriter] Set schema at first write example
State: closed | Created: 2020-05-26T21:59:48Z

Right now if the schema was not specified when instantiating `ArrowWriter`, then it could be set with the first `write_table` for example (it calls `self._build_writer()` to do so).

I noticed that it was not done if the first example is added via `.write`, so I added it for coherence.
GitHub issue #198: Index outside of table length
State: closed | Created: 2020-05-26T21:09:40Z

The offset input box warns of numbers larger than a limit (like 2000) but then the errors start at a smaller value than that limit (like 1955).

> ValueError: Index (2000) outside of table length (2000).
> Traceback:
> File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
>     exec(code, module.__dict__)
> File "/home/sasha/nlp_viewer/run.py", line 116, in <module>
>     v = d[item][k]
> File "/home/sasha/.local/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
>     output_all_columns=self._output_all_columns,
> File "/home/sasha/.local/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 290, in _getitem
>     raise ValueError(f"Index ({key}) outside of table length ({self._data.num_rows}).")
GitHub issue #197: Scientific Papers only downloading Pubmed
State: closed | Created: 2020-05-26T15:18:47Z

Hi!

I have been playing around with this module, and I am a bit confused about the `scientific_papers` dataset. I thought that it would download two separate datasets, arxiv and pubmed. But when I run the following:

```
dataset = nlp.load_dataset('scientific_papers', data_dir='.', cache_dir='.')
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.05k/5.05k [00:00<00:00, 2.66MB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.90k/4.90k [00:00<00:00, 2.42MB/s]
Downloading and preparing dataset scientific_papers/pubmed (download: 4.20 GiB, generated: 2.33 GiB, total: 6.53 GiB) to ./scientific_papers/pubmed/1.1.1...
Downloading: 3.62GB [00:40, 90.5MB/s]
Downloading: 880MB [00:08, 101MB/s]
Dataset scientific_papers downloaded and prepared to ./scientific_papers/pubmed/1.1.1. Subsequent calls will reuse this data.
```

only a pubmed folder is created. There doesn't seem to be something for arxiv. Are these two datasets merged? Or have I misunderstood something?

Thanks!
GitHub issue #193: [Tensorflow] Use something else than `from_tensor_slices()`
State: closed | Created: 2020-05-26T07:19:14Z

In the example notebook, the TF Dataset is built using `from_tensor_slices()` :

```python
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'start_positions', 'end_positions']
train_tf_dataset.set_format(type='tensorflow', columns=columns)
features = {x: train_tf_dataset[x] for x in columns[:3]} 
labels = {"output_1": train_tf_dataset["start_positions"]}
labels["output_2"] = train_tf_dataset["end_positions"]
tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)
```

But according to [official tensorflow documentation](https://www.tensorflow.org/guide/data#consuming_numpy_arrays), this will load the entire dataset to memory.

**This defeats one purpose of this library, which is lazy loading.**

Is there any other way to load the `nlp` dataset into TF dataset lazily ?

---

For example, is it possible to use [Arrow dataset](https://www.tensorflow.org/io/api_docs/python/tfio/arrow/ArrowDataset) ? If yes, is there any code example ?
GitHub issue #192: [Question] Create Apache Arrow dataset from raw text file
State: closed | Created: 2020-05-25T16:42:47Z

Hi guys, I have gathered and preprocessed about 2GB of COVID papers from CORD dataset @ Kggle. I have seen you have a text dataset as "Crime and punishment" in Apache arrow format. Do you have any script to do it from a raw txt file (preprocessed as for BERT like) or any guide?
Is the worth of send it to you and add it to the NLP library?
Thanks, Manu
GitHub issue #189: [Question] BERT-style multiple choice formatting
State: closed | Created: 2020-05-25T05:11:05Z

Hello, I am wondering what the equivalent formatting of a dataset should be to allow for multiple-choice answering prediction, BERT-style. Previously, this was done by passing a list of `InputFeatures` to the dataloader instead of a list of `InputFeature`, where `InputFeatures` contained lists of length equal to the number of answer choices in the MCQ instead of single items. I'm a bit confused on what the output of my feature conversion function should be when using `dataset.map()` to ensure similar behavior.

Thanks!
GitHub issue #186: Weird-ish: Not creating unique caches for different phases
State: closed | Created: 2020-05-23T06:40:58Z

Sample code:

```python
import nlp
dataset = nlp.load_dataset('boolq')

def func1(x):
    return x

def func2(x):
    return None

train_output = dataset["train"].map(func1)
valid_output = dataset["validation"].map(func1)
print()
print(len(train_output), len(valid_output))
# Output: 9427 9427
```

The map method in both cases seem to be pointing to the same cache, so the latter call based on the validation data will return the processed train data cache.

What's weird is that the following doesn't seem to be an issue:

```python
train_output = dataset["train"].map(func2)
valid_output = dataset["validation"].map(func2)
print()
print(len(train_output), len(valid_output))
# 9427 3270
```
GitHub issue #185: [Commands] In-detail instructions to create dummy data folder
State: closed | Created: 2020-05-22T12:26:25Z

### Dummy data command 

This PR adds a new command `python nlp-cli dummy_data <path_to_dataset_folder>` that gives in-detail instructions on how to add the dummy data files. 

It would be great if you can try it out by moving the current dummy_data folder of any dataset in `./datasets` with `mv datasets/<dataset_script>/dummy_data datasets/<dataset_name>/dummy_data_copy` and running the command `python nlp-cli dummy_data ./datasets/<dataset_name>` to see if you like the instructions. 

### CONTRIBUTING.md
Also the CONTRIBUTING.md is made cleaner including a new section on "How to add a dataset". 

### Current PRs 
It would be nice if we can try out if this command helps current PRs, *e.g.* #169  to add a dataset. I comment on those PRs.
GitHub issue #184: Use IndexError instead of ValueError when index out of range
State: closed | Created: 2020-05-22T10:43:42Z

**`default __iter__ needs IndexError`**.

When I want to create a wrapper of arrow dataset to adapt to fastai,
I don't know how to initialize it, so I didn't use inheritance but use object composition.
I wrote sth like this.
```
clas HF_dataset():
  def __init__(self, arrow_dataset):
    self.dset = arrow_dataset
  def __getitem__(self, i):
    return self.my_get_item(self.dset)
```
But `for sample in my_dataset:` gave me `ValueError(f"Index ({key}) outside of table length ({self._data.num_rows}).")` . This is because default `__iter__` will stop when it catched `IndexError`.

You can also see my [work](https://github.com/richardyy1188/Pretrain-MLM-and-finetune-on-GLUE-with-fastai/blob/master/GLUE_with_fastai.ipynb) that uses fastai2 to show/load batches from huggingface/nlp GLUE datasets

So I hope we can use `IndexError` instead to let other people who want to wrap it for any purpose won't be caught by this caveat.

BTW, I super appreciate your work, both transformers and nlp save my life. 💖💖💖💖💖💖💖
GitHub issue #183: [Bug] labels of glue/ax are all -1 
State: closed | Created: 2020-05-22T08:43:36Z

```
ax = nlp.load_dataset('glue', 'ax')
for i in range(30): print(ax['test'][i]['label'], end=', ')
```
```
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 
```
GitHub issue #181: Cannot upload my own dataset
State: closed | Created: 2020-05-21T16:45:52Z

I look into `nlp-cli` and `user.py` to learn how to upload my own data.

It is supposed to work like this
- Register to get username, password at huggingface.co
- `nlp-cli login` and type username, passworld
- I have a single file to upload at `./ttc/ttc_freq_extra.csv`
- `nlp-cli upload ttc/ttc_freq_extra.csv`

But I got this error.

```
2020-05-21 16:33:52.722464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
About to upload file /content/ttc/ttc_freq_extra.csv to S3 under filename ttc/ttc_freq_extra.csv and namespace korakot
Proceed? [Y/n] y
Uploading... This might take a while if files are large
Traceback (most recent call last):
  File "/usr/local/bin/nlp-cli", line 33, in <module>
    service.run()
  File "/usr/local/lib/python3.6/dist-packages/nlp/commands/user.py", line 234, in run
    token=token, filename=filename, filepath=filepath, organization=self.args.organization
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 141, in presign_and_upload
    urls = self.presign(token, filename=filename, organization=organization)
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 132, in presign
    return PresignedUrl(**d)
TypeError: __init__() got an unexpected keyword argument 'cdn'
```
GitHub issue #179: [Feature request] separate split name and split instructions
State: closed | Created: 2020-05-21T14:10:51Z

Currently, the name of an nlp.NamedSplit is parsed in arrow_reader.py and used as the instruction.

This makes it impossible to have several training sets, which can occur when:
- A dataset corresponds to a collection of sub-datasets
- A dataset was built in stages, adding new examples at each stage

Would it be possible to have two separate fields in the Split class, a name /instruction and a unique ID that is used as the key in the builder's split_dict ?
GitHub issue #178: [Manual data] improve error message for manual data in general
State: closed | Created: 2020-05-20T18:10:45Z

`nlp.load("xsum")` now leads to the following error message:

![Screenshot from 2020-05-20 20-05-28](https://user-images.githubusercontent.com/23423619/82481825-3587ea00-9ad6-11ea-9ca2-5794252c6ac7.png)

I guess the manual download instructions for `xsum` can also be improved.
GitHub issue #175: [Manual data dir] Error message: nlp.load_dataset('xsum') -> TypeError
State: closed | Created: 2020-05-20T17:00:32Z

v 0.1.0 from pip

```python
import nlp
xsum = nlp.load_dataset('xsum')
```

Issue is `dl_manager.manual_dir`is `None`

```python

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-42-8a32f066f3bd> in <module>
----> 1 xsum = nlp.load_dataset('xsum')

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519 

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    397         split_dict = SplitDict(dataset_name=self.name)
    398         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 399         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    400         # Checksums verification
    401         if verify_infos:

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/datasets/xsum/5c5fca23aaaa469b7a1c6f095cf12f90d7ab99bcc0d86f689a74fd62634a1472/xsum.py in _split_generators(self, dl_manager)
    102         with open(dl_path, "r") as json_file:
    103             split_ids = json.load(json_file)
--> 104         downloaded_path = os.path.join(dl_manager.manual_dir, "xsum-extracts-from-downloads")
    105         return [
    106             nlp.SplitGenerator(

~/miniconda3/envs/nb/lib/python3.7/posixpath.py in join(a, *p)
     78     will be discarded.  An empty last part will result in a path that
     79     ends with a separator."""
---> 80     a = os.fspath(a)
     81     sep = _get_sep(a)
     82     path = a

TypeError: expected str, bytes or os.PathLike object, not NoneType


```
GitHub issue #173: Rm extracted test dirs
State: closed | Created: 2020-05-20T13:30:48Z

All the dummy data used for tests were duplicated. For each dataset, we had one zip file but also its extracted directory. I removed all these directories

Furthermore instead of extracting next to the dummy_data.zip file, we extract in the temp `cached_dir` used for tests, so that all the extracted directories get removed after testing.

Finally there was a bug in the `mock_download_manager` that would let it create directories with invalid names, as in #172. I fixed that by encoding url arguments. I had to rename the dummy data for `scientific_papers` and `cnn_dailymail` (the aws tests don't pass for those 2 in this PR, but they will once aws will be synced, as the local ones do)

Let me know if it sounds good to you @patrickvonplaten . I'm still not entirely familiar with the mock downloader