Tokenized BLEU considered harmful - Discussion on community-based process

Generic discussion on the library

Metric discussion

Discussions on the metrics

I second this request. The bottom line is that **scores produced with different reference tokenizations are not comparable**. To discourage (even inadvertent) cheating, the user should never touch the reference. The `v13a` tokenization standard is not ideal, but at least it has been consistently used at matrix.statmt.org, facilitating comparisons.

Sacrebleu exposes [all its data sources](https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/dataset.py) and additionally provides [an API](https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/__init__.py) to accessing the references, which seem to fit within the spirit of your codebase.

Didn't we have a slide and discussion at WMT admitting that, for production-quality models, BLEU doesn't correlate with human eval anyway?

Yes, there are slides like that at WMT every year :) BLEU correlates with human judgment only at coarse levels, and it seems to be getting worse when people try to use it to do model selection among high-performing neural systems.

However, the point isn't whether BLEU is a good metric, but whether your BLEU score can be compared to other BLEU scores. They only can be compared if you use the same reference tokenization (similar to how you [can't compare LM perplexities across different segmentations](https://sjmielke.com/comparing-perplexities.htm)). sacrebleu was an attempt to get everyone to use WMT's reference tokenization (meaning, your system has to first remove its own tokenization) so that you could just compare across papers. This also prevents scores from being gamed.

I do not consider as a sufficient solution switching this library's default metric from BLEU to the wrapper around SacreBLEU.

As currently implemented, the wrapper allows end users to toggle SacreBLEU options, but doesn't pass along the SacreBLEU signature. As @mjpost showed in [Post18](https://www.aclweb.org/anthology/W18-6319.pdf), it's simply not credible to assume that people will stick to the defaults, therefore, the signature is necessary to be explicit about what options were used.

In addition to the `v13a` or `intl` options for the SacreBLEU `tokenize` argument, which was pointed out earlier, papers frequently differ on whether they lowercase text before scoring (`lowercase`) and the smoothing method used (`smooth_method`). BLEU scores can differ substantially (over 1 BLEU) just by changing these options.

Losing the SacreBLEU signature is a regression in reproducibility and clarity.

(Perhaps this should belong in a separate issue?)

Thanks for sharing your thoughts. This is a very important discussion.

Also one of the first items on our mid-term roadmap (we will try to clean it and share it soon) is to introduce mechanisms to get high-quality traceability and reproducibility for all the processes related to the library.

So having the signature for `sacrebleu` is really important!

Regarding BLEU, I guess we can just remove it from the canonical metrics included in the repo itself (it won't prevent people to add it as "user-metrics" but at least we won't be promoting it).

On a more general note (definitely too large for the scope of this issue) we are wondering, with @srush in particular, how we could handle the selection of metrics/datasets with the most community-based and bottom-up approach possible. If you have opinions on this, please share!

Yeah, I would love to have discussions about ways this project can have an community-based, transparent process to arrive at strong default metrics. @kpu / @mjpost do you have any suggestions of how that might work or pointers to places where this is done right? Perhaps this question can be template for what is likely to be repeated for other datasets.

I think @bittlingmayer is referring to Figure 6 in http://statmt.org/wmt19/pdf/53/WMT02.pdf .  When you look at Appendix A there are some cases where metrics fall apart at the high end and some where they correlate well.  en-zh is arguably production-quality.

This could evolve into a metrics Bazaar where the value add is really the packaging and consistency: it installs/compiles the metrics for me, gives a reproducible name to use in publication (involve the authors; you don't want a different sacrebleu hash system), a version number, and evaluation of the metrics like http://ufallab.ms.mff.cuni.cz/~bojar/wmt19-metrics-task-package.tgz but run when code changes rather than once a year.

While a Bazaar setup works for models / datasets, I am not sure it is ideal for metrics ? Ideal from my perspective would be to have tasks with metrics moderated by experts who document, cite, and codify known pitchfalls (as above^) and make it non-trivial for beginners to mess it up.

@srush @thomwolf

ModelFront could provide (automated, "QE-based") evaluation for all the pretrained translation models you host.  Not bottom-up and not valid for claiming SoTA, but independent, practical for builders and not top-down.

For that I would also suggest some diverse benchmarks (so split it out into datasets with only user-generated data, or only constants, or only UI strings, or only READMEs) which tease out known trade-offs.  Even hypothetical magic eval is limited if we always reduce it to a single number.

Realistically people want to know how a model compares to an API like Google Translate, Microsoft Translator, DeepL or Yandex (especially for a language pair like EN:RU, or for the many languages that only Yandex supports), and that could be done too.

Very important discussion.
I am trying to understand the effects of tokenization.
I wanted to ask which is a good practice.
Sacrebleu should be used on top of the tokenized output, or detokenized(raw) text?

Use sacrebleu on detokenized output and raw unmodified references.

https://github.com/huggingface/nlp/blob/7d1526dfeeb29248d832f1073192dbf03ad642da/metrics/bleu/bleu.py#L76 assumes the inputs are tokenized by the user.  This is bad practice because the user's tokenizer is usually not the same as the one used by `mteval-v13a.pl`, the closest thing we have to a standard.  Moreover, tokenizers are like window managers: they can be endlessly customized and nobody has quite the same options.

As @mjpost reported in https://www.aclweb.org/anthology/W18-6319.pdf BLEU configurations can vary by 1.8.  Yet people are incorrectly putting non-comparable BLEU scores in the same table, such as Table 1 in https://arxiv.org/abs/2004.04902 .

There are a few use cases for tokenized BLEU like Thai.  For Chinese, people seem to use character BLEU for better or worse.

The default easy option should be the one that's correct more often.  And that is sacrebleu.  Please don't make it easy for people to run what is usually the wrong option; it definitely shouldn't be `bleu`.

Also, I know this is inherited from TensorFlow and, paging @lmthang, they should discourage it too.


Update README.md

small typo


Fix print statement in READ.md

print statement was throwing generator object instead of printing names of available datasets/metrics


Update README.md

the readme got removed, closing this one


[Question] Using/adding a local dataset

Hi @zphang,

So you can just give the local path to a dataset script file and it should work.

Here is an example:
- you can download one of the scripts in the `datasets` folder of the present repo (or clone the repo)
- then you can load it with `load_dataset('PATH/TO/YOUR/LOCAL/SCRIPT.py')`

Does it make sense?

Could you give a more concrete example, please?

I looked up wikitext dataset script from the repo. Should I just overwrite the `data_file` on line 98 to point to the local dataset directory? Would it work for different configurations of wikitext (wikitext2, wikitext103 etc.)?

Or maybe we can use DownloadManager to specify local dataset location? In that case, where do we use DownloadManager instance?

Thanks

Hi @MaveriQ , although what I am doing is to commit a new dataset, but I think looking at imdb script might help.
You may want to use `dl_manager.download_custom`, give it a url(arbitrary string), a custom_download(arbitrary function) and return a path, and finally use _get sample to fetch a sample.

The download manager supports local directories. You can specify a local directory instead of a url and it should work.

Closing this one.
Feel free to re-open if you have other questions :)

Users may want to either create/modify a local copy of a dataset, or use a custom-built dataset with the same `Dataset` API as externally downloaded datasets.

It appears to be possible to point to a local dataset path rather than downloading the external ones, but I'm not exactly sure how to go about doing this.

A notebook/example script demonstrating this would be very helpful.


[Feature Request] Add the OpenWebText dataset

dataset request

Requesting to add a new dataset

We're experimenting with hosting the OpenWebText corpus on Zenodo for easier downloading. https://zenodo.org/record/3834942#.Xs1w8i-z2J8

The OpenWebText dataset is an open clone of OpenAI's WebText dataset. It can be used to train ELECTRA as is specified in the [README](https://www.github.com/google-research/electra).

More information and the download link are available [here](https://skylion007.github.io/OpenWebTextCorpus/).


[Feature request] Add Toronto BookCorpus dataset

Requesting to add a new dataset

As far as I understand, `wikitext` is refer to `WikiText-103` and `WikiText-2` that created by researchers in Salesforce, and mostly used in traditional language modeling.

You might want to say `wikipedia`, a dump from wikimedia foundation.

Also I would like to have Toronto BookCorpus too ! Though it involves copyright problem...

Hi, @lhoestq, just a reminder that this is solved by #248 .😉

I know the copyright/distribution of this one is complex, but it would be great to have! That, combined with the existing `wikitext`, would provide a complete dataset for pretraining models like BERT.


Loading GLUE dataset loads CoLA by default

dataset bug

A bug in a dataset script provided in the library

As a follow-up to this: It looks like the actual GLUE task name is supplied as the `name` argument. Is there a way to check what `name`s/sub-datasets are available under a grouping like GLUE? That information doesn't seem to be readily available in info from `nlp.list_datasets()`.

Edit: I found the info under `Glue.BUILDER_CONFIGS`

Yes so the first config is loaded by default when no `name` is supplied but for GLUE this should probably throw an error indeed.

We can probably just add an `__init__` at the top of the `class Glue(nlp.GeneratorBasedBuilder)` in the `glue.py` script which does this check:
```
class Glue(nlp.GeneratorBasedBuilder):
    def __init__(self, *args, **kwargs):
        assert 'name' in kwargs and kwargs[name] is not None, "Glue has to be called with a configuration name"
        super(Glue, self).__init__(*args, **kwargs)
```

An error is raised if the sub-dataset is not specified :)
```
ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
	`load_dataset('glue', 'cola')`
```

If I run:

```python
dataset = nlp.load_dataset('glue')
```
The resultant dataset seems to be CoLA be default, without throwing any error. This is in contrast to calling:

```python
metric = nlp.load_metric("glue")
```
which throws an error telling the user that they need to specify a task in GLUE. Should the same apply for loading datasets?


[Feature request] Add Google Natural Question dataset

dataset request

Requesting to add a new dataset

Indeed, I think this one is almost ready cc @lhoestq

I'm doing the latest adjustments to make the processing of the dataset run on Dataflow

Is there an update to this? It will be very beneficial for the QA community!

Still work in progress :)
The idea is to have the dataset already processed somewhere so that the user only have to download the processed files. I'm also doing it for wikipedia.

Super appreciate your hard work !!
I'll cross my fingers and hope easily loadable wikipedia dataset will come soon.

Quick update on NQ: due to some limitations I met using apache beam + parquet I was not able to use the dataset in a nested parquet structure in python to convert it to our Apache Arrow format yet.
However we had planned to change this conversion step anyways so we'll make just sure that it enables to process and convert the NQ dataset to arrow.

Would be great to have https://github.com/google-research-datasets/natural-questions as an alternative to SQuAD.


Some error inside nlp.load_dataset()

Google colab has an old version of Apache Arrow built-in.
Be sure you execute the "pip install" cell and restart the notebook environment if the colab asks for it.

First of all, nice work!

I am going through [this overview notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb)

In simple step `dataset = nlp.load_dataset('squad', split='validation[:10%]')`

I get an error, which is connected with some inner code, I think:
```
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-8-d848d3a99b8c> in <module>()
      1 # Downloading and loading a dataset
      2
----> 3 dataset = nlp.load_dataset('squad', split='validation[:10%]')

8 frames

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    414             try:
    415                 # Prepare split will record examples associated to the split
--> 416                 self._prepare_split(split_generator, **prepare_split_kwargs)
    417             except OSError:
    418                 raise OSError("Cannot find data file. " + (self.MANUAL_DOWNLOAD_INSTRUCTIONS or ""))

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _prepare_split(self, split_generator)
    585         fname = "{}-{}.arrow".format(self.name, split_generator.name)
    586         fpath = os.path.join(self._cache_dir, fname)
--> 587         examples_type = self.info.features.type
    588         writer = ArrowWriter(data_type=examples_type, path=fpath, writer_batch_size=self._writer_batch_size)
    589

/usr/local/lib/python3.6/dist-packages/nlp/features.py in type(self)
    460     @property
    461     def type(self):
--> 462         return get_nested_type(self)
    463
    464     @classmethod

/usr/local/lib/python3.6/dist-packages/nlp/features.py in get_nested_type(schema)
    370     # Nested structures: we allow dict, list/tuples, sequences
    371     if isinstance(schema, dict):
--> 372         return pa.struct({key: get_nested_type(value) for key, value in schema.items()})
    373     elif isinstance(schema, (list, tuple)):
    374         assert len(schema) == 1, "We defining list feature, you should just provide one example of the inner type"

/usr/local/lib/python3.6/dist-packages/nlp/features.py in <dictcomp>(.0)
    370     # Nested structures: we allow dict, list/tuples, sequences
    371     if isinstance(schema, dict):
--> 372         return pa.struct({key: get_nested_type(value) for key, value in schema.items()})
    373     elif isinstance(schema, (list, tuple)):
    374         assert len(schema) == 1, "We defining list feature, you should just provide one example of the inner type"

/usr/local/lib/python3.6/dist-packages/nlp/features.py in get_nested_type(schema)
    379         # We allow to reverse list of dict => dict of list for compatiblity with tfds
    380         if isinstance(inner_type, pa.StructType):
--> 381             return pa.struct(dict((f.name, pa.list_(f.type, schema.length)) for f in inner_type))
    382         return pa.list_(inner_type, schema.length)
    383

/usr/local/lib/python3.6/dist-packages/nlp/features.py in <genexpr>(.0)
    379         # We allow to reverse list of dict => dict of list for compatiblity with tfds
    380         if isinstance(inner_type, pa.StructType):
--> 381             return pa.struct(dict((f.name, pa.list_(f.type, schema.length)) for f in inner_type))
    382         return pa.list_(inner_type, schema.length)
    383

TypeError: list_() takes exactly one argument (2 given)
```


Update Overview.ipynb

update notebook


remove webis

Remove webis from dataset folder.

Our first dataset script that only lives on AWS :-) https://s3.console.aws.amazon.com/s3/buckets/datasets.huggingface.co/nlp/datasets/webis/tl_dr/?region=us-east-1 @julien-c @jplu


[Newsroom] add newsroom

I checked it with the data link of the mail you forwarded @thomwolf => works well!


Xsum, require manual download of some files


[Tests] Local => aws

For each dataset, If there exist a `dataset_info.json`, then the command `nlp-cli test path/to/my/dataset --al_configs` is successful only if the `dataset_infos.json` is correct. The infos are correct if the size and checksums of the downloaded file are correct, and if the number of examples in each split are correct.

Note: the `test` command is supposed to test the script, that's why it runs the script even if the cached files already exist. Let me know if it's good to you.

> For each dataset, If there exist a `dataset_info.json`, then the command `nlp-cli test path/to/my/dataset --al_configs` is successful only if the `dataset_infos.json` is correct. The infos are correct if the size and checksums of the downloaded file are correct, and if the number of examples in each split are correct.
>
> Note: the `test` command is supposed to test the script, that's why it runs the script even if the cached files already exist. Let me know if it's good to you.

Does it have to download the whole data to check if the checksums are correct? I guess so no?

> > For each dataset, If there exist a `dataset_info.json`, then the command `nlp-cli test path/to/my/dataset --al_configs` is successful only if the `dataset_infos.json` is correct. The infos are correct if the size and checksums of the downloaded file are correct, and if the number of examples in each split are correct.
> > Note: the `test` command is supposed to test the script, that's why it runs the script even if the cached files already exist. Let me know if it's good to you.
>
> Does it have to download the whole data to check if the checksums are correct? I guess so no?

Yes it has to download them all (unless they were already downloaded in which case it just uses the cached downloaded files).

## Change default Test from local => aws

As a default we set` aws=True`, `Local=False`, `slow=False`

### 1. RUN_AWS=1 (default)
This runs 4 tests per dataset script.

a) Does the dataset script have a valid etag / Can it be reached on AWS?
b) Can we load its `builder_class`?
c) Can we load **all** dataset configs?
d) _Most importantly_: Can we load the dataset?

Important - we currently only test the first config of each dataset to reduce test time. Total test time is around 1min20s.

### 2. RUN_LOCAL=1 RUN_AWS=0

***This should be done when debugging dataset scripts of the ./datasets folder***

This only runs 1 test per dataset test, which is equivalent to aws d) - Can we load the dataset from the local `datasets` directory?

### 3. RUN_SLOW=1

We should set up to run these tests maybe 1 time per week ? @thomwolf

The `slow` tests include two more important tests.

e) Can we load the dataset with all possible configs? This test will probably fail at the moment because a lot of dummy data is missing. We should add the dummy data step by step to be sure that all configs work.

f) Test that the actual dataset can be loaded. This will take quite some time to run, but is important to make sure that the "real" data can be loaded. It will also test whether the dataset script has the correct checksums file which is currently not tested with `aws=True`. @lhoestq - is there an easy way to check cheaply whether the `dataset_info.json` is correct for each dataset script?


Final cleanup of readme and metrics


make style


🐛 `map` not working

I didn't assign the output 🤦♂

```python
dataset.map(test)
```

should be :

```python
dataset = dataset.map(test)
```

I'm trying to run a basic example (mapping function to add a prefix).
[Here is the colab notebook I'm using.](https://colab.research.google.com/drive/1YH4JCAy0R1MMSc-k_Vlik_s1LEzP_t1h?usp=sharing)

```python
import nlp

dataset = nlp.load_dataset('squad', split='validation[:10%]')

def test(sample):
    sample['title'] = "test prefix @@@ " + sample["title"]
    return sample

print(dataset[0]['title'])
dataset.map(test)
print(dataset[0]['title'])
```
Output :
> Super_Bowl_50
Super_Bowl_50

Expected output :
> Super_Bowl_50
test prefix @@@ Super_Bowl_50


🐛 Colab : type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'

It's strange, after installing `nlp` on Colab, the `pyarrow` version seems fine from `pip` but not from python :

```python
import pyarrow

!pip show pyarrow
print("version = {}".format(pyarrow.__version__))
```

> Name: pyarrow
Version: 0.17.0
Summary: Python library for Apache Arrow
Home-page: https://arrow.apache.org/
Author: None
Author-email: None
License: Apache License, Version 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: numpy
Required-by: nlp, feather-format
>
> version = 0.14.1

Ok I just had to restart the runtime after installing `nlp`. After restarting, the version of `pyarrow` is fine.

I'm trying to load CNN/DM dataset on Colab.

[Colab notebook](https://colab.research.google.com/drive/11Mf7iNhIyt6GpgA1dBEtg3cyMHmMhtZS?usp=sharing)

But I meet this error :

> AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'


❓ How to apply a map to all subsets ?

I'm working with CNN/DM dataset, where I have 3 subsets : `train`, `test`, `validation`.

Should I apply my map function on the subsets one by one ?

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
for corpus in ['train', 'test', 'validation']:
         cnn_dm[corpus] = cnn_dm[corpus].map(my_func)
```

Or is there a better way to do this ?


❓ How to remove specific rows of a dataset ?

I saw on the [example notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb#scrollTo=efFhDWhlvSVC) how to remove a specific column :

```python
dataset.drop('id')
```

But I didn't find how to remove a specific row.

**For example, how can I remove all sample with `id` < 10 ?**


🐛 Trying to use ROUGE metric : pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323

metric bug

A bug in a metric script

Can you share your data files or a minimally reproducible example?

Sure, [here is a Colab notebook](https://colab.research.google.com/drive/1uiS89fnHMG7HV_cYxp3r-_LqJQvNNKs9?usp=sharing) reproducing the error.

> ArrowInvalid: Column 1 named references expected length 36 but got length 56

This is because `add` takes as input a batch of elements and you provided only one. I think we should have `add` for one prediction/reference and `add_batch` for a batch of predictions/references. This would make it more coherent with the way we use Arrow.

Let me do this change

Thanks for noticing though. I was mainly used to do `.compute` directly ^^

I'm trying to use rouge metric.

I have to files : `test.pred.tokenized` and `test.gold.tokenized` with each line containing a sentence.
I tried :

```python
import nlp

rouge = nlp.load_metric('rouge')
with open("test.pred.tokenized") as p, open("test.gold.tokenized") as g:
    for lp, lg in zip(p, g):
            rouge.add(lp, lg)
```

But I meet following error :

> pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323

---

Full stack-trace :

```
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/me/.venv/transformers/lib/python3.6/site-packages/nlp/metric.py", line 224, in add
    self.writer.write_batch(batch)
  File "/home/me/.venv/transformers/lib/python3.6/site-packages/nlp/arrow_writer.py", line 148, in write_batch
    pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
  File "pyarrow/table.pxi", line 1550, in pyarrow.lib.Table.from_pydict
  File "pyarrow/table.pxi", line 1503, in pyarrow.lib.Table.from_arrays
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323
```

(`nlp` installed from source)


I could access the info by first accessing the different splits :

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
print(cnn_dm['train'].info)
```

Information seems to be duplicated between the subsets :

```python
print(cnn_dm["train"].info == cnn_dm["test"].info == cnn_dm["validation"].info)
# True
```

Is it expected ?

Good point @Colanim ! What happens under the hood when running:

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
```

is that for every split in `cnn_dailymail`, a different dataset object (which all holds the same info) is created. This has the advantages that the datasets are easily separable in a training setup.
Also note that you can load e.g. only the `train` split of the dataset via:

```python
cnn_dm_train = nlp.load_dataset('cnn_dailymail', split="train")
print(cnn_dm_train.info)
```

I think we should make the `info` object slightly different when creating the dataset for each split - at the moment it contains for example the variable `splits` which should maybe be renamed to `split` and contain only one `SplitInfo` object ...

I'm trying to access the information of CNN/DM dataset :

```python
cnn_dm = nlp.load_dataset('cnn_dailymail')
print(cnn_dm.info)
```

returns :

> AttributeError: 'dict' object has no attribute 'info'


Couldn't reach CNN/DM dataset

Installing from source (instead of Pypi package) solved the problem.

I can't get CNN / DailyMail dataset.

```python
import nlp

assert "cnn_dailymail" in [dataset.id for dataset in nlp.list_datasets()]
cnn_dm = nlp.load_dataset('cnn_dailymail')
```

[Colab notebook](https://colab.research.google.com/drive/1zQ3bYAVzm1h0mw0yWPqKAg_4EUlSx5Ex?usp=sharing)

gives following error :

```
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/cnn_dailymail/cnn_dailymail.py
```


Adding docstrings and some doc

Some doc


Qa4mre - add dataset

Added dummy data test only for the first config. Will do the rest later.
I had to do add some minor hacks to an important function to make it work.
There might be a cleaner way to handle it - can you take a look @thomwolf ?


[Clean-up] remove under construction datastes


fix reddit tifu dummy data


[Reclor] fix reclor

- That's probably one me. Could have made the manual data test more flexible. @mariamabarham


convert can use manual dir as second argument

@mariamabarham


add writer_batch_size to GeneratorBasedBuilder

You can now specify `writer_batch_size` in the builder arguments or directly in `load_dataset`


Add data dir test command

Nice - I think we can merge this. I will update the checksums for `wikihow` then as well


[New structure on AWS] Adapt paths

Some small changes so that we have the correct paths. @julien-c


Add trivia_q

Currently tested only for one config to pass tests. Needs to add more dummy data later.


[Manual downloads] add logic proposal for manual downloads and add wikihow

> Wikihow is an example that needs to manually download two files as stated in: https://github.com/mahnazkoupaee/WikiHow-Dataset.
>
> The user can then store these files under a hard-coded name: `wikihowAll.csv` and `wikihowSep.csv` in this case in a directory of his choice, e.g. `~/wikihow/manual_dir`.
>
> The dataset can then be loaded via:
>
> ```python
> import nlp
> nlp.load_dataset("wikihow", data_dir="~/wikihow/manual_dir")
> ```
>
> I added/changed so that there are explicit error messages when using manually downloaded files.

wouldn't be nicer if we can have `manual_dir/wikihow`?

> > Wikihow is an example that needs to manually download two files as stated in: https://github.com/mahnazkoupaee/WikiHow-Dataset.
> > The user can then store these files under a hard-coded name: `wikihowAll.csv` and `wikihowSep.csv` in this case in a directory of his choice, e.g. `~/wikihow/manual_dir`.
> > The dataset can then be loaded via:
> > ```python
> > import nlp
> > nlp.load_dataset("wikihow", data_dir="~/wikihow/manual_dir")
> > ```
> >
> >
> > I added/changed so that there are explicit error messages when using manually downloaded files.
>
> wouldn't be nicer if we can have `manual_dir/wikihow`?

Sure, I mean the user can decide whatever he likes best :-) The path one puts in `data_dir` will be used as the path to the manual dir. `nlp.load_dataset("wikihow", data_dir="~/manual_dir/wikihow")` would work as well as any other path ;-)

Wikihow is an example that needs to manually download two files as stated in: https://github.com/mahnazkoupaee/WikiHow-Dataset.

The user can then store these files under a hard-coded name: `wikihowAll.csv` and `wikihowSep.csv` in this case in a directory of his choice, e.g. `~/wikihow/manual_dir`.

The dataset can then be loaded via:

```python
import nlp
nlp.load_dataset("wikihow", data_dir="~/wikihow/manual_dir")
```

I added/changed so that there are explicit error messages when using manually downloaded files.


Run save infos

Haha that cornell dialogue dataset - that ran for 3h on my computer as well. The `generate_examples` method in this script is one of the most inefficient code samples I've ever seen :D

Indeed it's been 3 hours already
```73111 examples [3:07:48,  2.40 examples/s]```

I replaced the old checksum file with the new `dataset_infos.json` by running the script on almost all the datasets we have. The only one that is still running on my side is the cornell dialog


[Reddit] add reddit

- Everything worked fine @mariamabarham. Made my computer nearly crash, but all seems to be working :-)


Add per type scores in seqeval metric

LGTM :-) Some small suggestions to shorten the code a bit :-)

Can you put the kwargs as normal kwargs instead of a dict? (And add them to the kwargs description As well)

Yes and there is a dynamically generated doc string in the metric script KWARGS DESCRIPTION

This PR add a bit more detail in the seqeval metric. Now the usage and output are:

```python
import nlp
met = nlp.load_metric('metrics/seqeval')
references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
predictions =  [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
met.compute(predictions, references)

#Output: {'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.8}
```

It is also possible to compute scores for non IOB notations, POS tagging for example hasn't this kind of notation. Add `suffix` parameter:

```python
import nlp
met = nlp.load_metric('metrics/seqeval')
references = [['O', 'O', 'O', 'MISC', 'MISC', 'MISC', 'O'], ['PER', 'PER', 'O']]
predictions =  [['O', 'O', 'MISC', 'MISC', 'MISC', 'MISC', 'O'], ['PER', 'PER', 'O']]
met.compute(predictions, references, metrics_kwargs={"suffix": True})

#Output: {'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.9}
```


[Cmrc 2018] fix cmrc2018


Webis tl-dr

> Should that rather be in an organization scope, @thomwolf @patrickvonplaten ?

I'm a bit indifferent - both would be fine for me!

@jplu - if creating the dummy_data is too tedious, I can do it as well :-)

Yeah I think naming it webis/tl_dr would be best @jplu if that works for you

> There is dummy_data here, no ?

Some paths were wrong - the structure is really confusing and the error messages don't really help either - I have to think about how to make this easier to understand!

Hope it was ok that I fiddled with your PR !

> Some paths were wrong - the structure is really confusing and the error message don't really help either - I have to think about how to make this easier to understand!

Oh ok! I haven't noticed that sorry :(

> Hope it was ok that I fiddled with your PR !

Of course it was ok :)

@julien-c Looks like what you have in mind?

```python
import nlp
nlp.load_dataset("datasets/webis", "tl_dr")

#Output: Downloading and preparing dataset webis/tl_dr (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/jplu/.cache/huggingface/datasets/webis/tl_dr/1.0.0...
```

Merging this for now. Maybe we can see whether to rename it in a different PR @julien-c ?

Hi,
Author here of the webis-tldr corpus. Any plans on integrating this dataset into the hub? I remember we could access it in the previous versions of the library. If there is a particular issue that I can help with, do let me know.

Thanks!

Hi @shahbazsyed, this dataset _is_ inside the hub but it's namespaced by the organization name `webis`.

You can load it following the steps described in https://huggingface.co/datasets/webis/tl_dr

Here's a Colab showcasing that it works: https://colab.research.google.com/drive/11IrzRVpnMLJZ8_UFFHLR8FhiajjAHRUU?usp=sharing

The reason the code is in S3 and not in this repo is that the dataset is namespaced under the `webis` organization. We don't have a lot of namespaced datasets yet but this should become the main way we add more datasets in the future.
Let us know if that's an issue for you. Thank you!

Add the Webid TL:DR dataset.


[Csv] add tests for csv dataset script

@thomwolf - can you check and merge if ok?

Adds dummy data tests for csv.


lm1b

I might have a different version of `isort` than others. It seems like I'm always reordering the imports of others. But isn't really a problem...

Add lm1b dataset.


Replace checksums files by Dataset infos json

> Ok, really clean!
> I like the logic (not a huge fan of using `_asdict_inner` but it makes sense).
> I think it's a nice improvement!
>
> How should we update the files in the repo? Run a big job on a server or on somebody's computer who has most of the datasets already downloaded?

Maybe we can split the updates among us...IMO most datasets run very quickly.
I think I've downloaded 50 datasets and 80% are loaded in <5min,  15% in <1h and then `wmt` which is still downloading (since 12h).
I deleted my cache because the `wmt` downloads require quite a lot of space, so I only have parts of the `wmt` datasets on my computer.

@mariamabarham I guess you have downloaded most of the datasets no?

### Better verifications when loading a dataset

I replaced the `urls_checksums` directory that used to contain `checksums.txt` and `cached_sizes.txt`, by a single file `dataset_infos.json`. It's just a dict `config_name` -> `DatasetInfo`.

It simplifies and improves how verifications of checksums and splits sizes are done, as they're all stored in `DatasetInfo` (one per config). Also, having already access to `DatasetInfo` enables to check disk space before running `download_and_prepare` for a given config.

The dataset infos json file is user readable, you can take a look at the squad one that I generated in this PR.

### Renaming

According to these changes, I did some renaming:
`save_checksums` -> `save_infos`
`ignore_checksums` -> `ignore_verifications`

for example, when you are creating a dataset you have to run
```nlp-cli test path/to/my/dataset --save_infos --all_configs```
instead of
```nlp-cli test path/to/my/dataset --save_checksums --all_configs```

### And now, the fun part

We'll have to rerun the `nlp-cli test ... --save_infos --all_configs` for all the datasets

-----------------

feedback appreciated !


Librispeech

@jplu - I changed this weird archieve - iter method to something simpler. It's only one file to download anyways so I don't see the point of using weird iter methods...It's a huge file though :D 30 million lines of text. Took me quite some time to download :D

Add librispeech dataset and remove some useless content.


Cleanup notebooks and various fixes

Fixes on dataset (more flexible) metrics (fix) and general clean ups


[WIP] add wmt14

WMT14 takes forever to download :-/

- WMT is the first dataset that uses an abstract class IMO, so I had to modify the `load_dataset_module` a bit.


[Paracrawl] add paracrawl

- Huge dataset - took ~1h to download
- Also this PR reformats all dataset scripts and adds `datasets` to `make style`


Add download gg drive

awesome - so no manual downloaded needed here?

Yes exactly. It works like a standard download

We can now add datasets that download from google drive


Add list and inspect methods - cleanup hf_api

Add a bunch of methods to easily list and inspect the processing scripts up-loaded on S3:
```python
nlp.list_datasets()
nlp.list_metrics()
# Copy and prepare the scripts at `local_path` for easy inspection/modification.
nlp.inspect_dataset(path, local_path)
# Copy and prepare the scripts at `local_path` for easy inspection/modification.
nlp.inspect_metric(path, local_path)
```

Also clean up the `HfAPI` to use `dataclasses` for better user-experience


Add wiki40b

Looks good to me. I have not really looked too much into the Beam Datasets yet though - so I think you can merge whenever you think is good for Beam datasets :-)

This one is a beam dataset that downloads files using tensorflow.
I tested it on a small config and it works fine


Add Flores

Beautiful language for sure!


[Load => load_dataset] change naming

Rename leftovers @thomwolf


Add boolq

Awesome :-) Thanks for adding the function to the Mock DL Manager

I just added the dummy data for this dataset.
This one was uses `tf.io.gfile.copy` to download the data but I added the support for custom download in the mock_download_manager. I also had to add a `tensorflow` dependency for tests.


[TedHrLr] add left dummy data


New datasets


[Datasets] add ted_hrlr

@thomwolf - After looking at `xnli` I think it's better to leave the translation features and add a `translation` key to make them work in our framework.

The result looks like this:
![Screenshot from 2020-05-12 18-34-43](https://user-images.githubusercontent.com/23423619/81721933-ee1faf00-9480-11ea-9e95-d6557cbd0ce0.png)

you can see that each split has a `translation` key which value is the nlp.features.Translation object.

That's a simple change. If it's ok for you, I will add dummy data for the other configs and treat the other translation scripts in the same way.


add tests

Tests for py_utils functions and for the BaseReader used to read from arrow and parquet.
I also removed unused utils functions.


Add nbytes + nexamples check

Looks good to me! Should we hard code those numbers in the config classes and make sure that when loading a dataset that the numbers match?

### Save size and number of examples
Now when you do `save_checksums`, it also create `cached_sizes.txt` right next to the checksum file.
This new file stores the bytes sizes and the number of examples of each split that has been prepared and stored in the cache. Example:

```
# Cached sizes: <full_config_name> <num_bytes> <num_examples>
hansards/house/1.0.0/test 22906629 122290
hansards/house/1.0.0/train 191459584 947969
hansards/senate/1.0.0/test 5711686 25553
hansards/senate/1.0.0/train 40324278 182135
```

### Check processing output

If there is a `caches_sizes.txt`, then each time we run `download_and_prepare` it will make sure that the sizes match. You can set `ignore_checksums=True` if you don't want that to happen.

### Fill Dataset Info

All the split infos and the checksums are now stored correctly in DatasetInfo after `download_and_prepare`

### Check space on disk before running `download_and_prepare`

Check if the space is lower than the sum of the sizes of the files in `checksums.txt` and `cached_files.txt`. This is not ideal though as it considers the files for all configs.

TODO:
A better way to do it would be to have save the `DatasetInfo` instead of the `checksums.txt` and `cached_sizes.txt`, in order to have one file per dataset config (and therefore consider only the sizes of the files for one config and not all of them). It can also be the occasion to factorize all the `download_and_prepare` verifications. Maybe next PR ?


[Convert] add new pattern


[Tests] skip beam dataset tests for now

@lhoestq - I moved the wkipedia file to the "correct" folder.

For now we will skip tests for Beam Datasets


pin flake 8

Flake 8's new version does not like our format. Pinning the version for now.


WIP adding metrics

It's all about my metric stuff so I'll probably merge it unless you want to have a look.

Took the occasion to remove the old doc and requirements.txt

Adding the following metrics as identified by @mariamabarham:

1. BLEU:  BiLingual Evaluation Understudy: https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py,  https://github.com/chakki-works/sumeval/blob/master/sumeval/metrics/bleu.py (multilingual)
2. GLEU:  Google-BLEU:  https://github.com/cnap/gec-ranking/blob/master/scripts/compute_gleu
3. Sacrebleu: https://pypi.org/project/sacrebleu/1.4.8/ (pypi package), https://github.com/mjpost/sacrebleu (github implementation)
4. ROUGE: Recall-Oriented Understudy for Gisting Evaluation: https://github.com/google-research/google-research/tree/master/rouge, https://github.com/chakki-works/sumeval/blob/master/sumeval/metrics/rouge.py (multilingual)
5. Seqeval: https://github.com/chakki-works/seqeval (github implementation), https://pypi.org/project/seqeval/0.0.12/ (pypi package)
6. Coval:  coreference evaluation package for the CoNLL and ARRAU datasets https://github.com/ns-moosavi/coval
7. SQuAD v1 evaluation script
8. SQuAD V2 evaluation script: https://worksheets.codalab.org/rest/bundles/0x6b567e1cf2e041ec80d7098f031c5c9e/contents/blob/
9. GLUE
10. XNLI

Not now:
1. Perplexity: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/perplexity.py
2. Spearman: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/spearman_correlation.py
3. F1_measure: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/f1_measure.py
4. Pearson_corelation: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/pearson_correlation.py
5. AUC: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/auc.py
6. Entropy: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/entropy.py


fix overflow check

I did some tests and unfortunately the test
```
pa_array.nbytes > MAX_BATCH_BYTES
```
doesn't work. Indeed for a StructArray, `nbytes` can be less 2GB even if there is an overflow (it loops...).

I don't think we can do a proper overflow test for the limit of 2GB...

For now I replaced it with a sanity check on the first element.


JSON script

The tests for the Wikipedia dataset do not pass anymore with the error:
```
To be able to use this dataset, you need to install the following dependencies ['mwparserfromhell'] using 'pip install mwparserfromhell' for instance'
```

This was an issue on master. You can just rebase from master.

Perfect! Indeed, it worked^^ Thanks @lhoestq

Currently the dummy_data tests are always green because in a PR the dataset is not yet synchronized with aws. This PR fixes this: https://github.com/huggingface/nlp/pull/140 .

Could you test `json` locally or wait until the PR: https://github.com/huggingface/nlp/pull/140 is merged ? :-)

Ok, I will wait #140 to be merged and then rebase :)

Add a JSONS script to read JSON datasets from files.


[README dummy data tests] README to better understand how the dummy data structure works

In this PR a README.md is added to tests to shine more light on how the dummy data structure works. I try to explain the different possible cases. IMO the best way to understand the logic is to checkout the dummy data structure of the different datasets I mention in the README.md since those are the "edge cases".

@mariamabarham @thomwolf @lhoestq @jplu - I'd be happy to checkout the dummy data structure and get some feedback on possible improvements.


Fix arrow writer for big datasets using writer_batch_size

After a quick chat with Yacine : the 2Go test may not be sufficient actually, as I'm looking at the size of the array and not the size of the current_rows. If the test doesn't do the job I think I'll remove it and lower the batch size a bit to be sure that it never exceeds 2Go. I'll do more tests later

This PR fixes Yacine's bug.
According to [this](https://github.com/apache/arrow/blob/master/docs/source/cpp/arrays.rst#size-limitations-and-recommendations), it is not recommended to have pyarrow arrays bigger than 2Go.

Therefore I set a default batch size of 100 000 examples per batch. In general it shouldn't exceed 2Go. If it does, I reduce the batch_size on the fly, and I notify the user with a warning.


adding RACE, QASC, Super_glue and Tiny_shakespear datasets

I think rebasing to master will solve the quality test and the datasets that don't have a testing structure yet because of the manual download - maybe you can put them in `datasets under construction`? Then would also make it easier for me to see how to add tests for them :-)


fix cache dir in builder tests

Nice, is that the reason one cannot rerun the tests without deleting the cache?

Yes exactly. It was not using the temporary dir for tests.

minor fix


[CSV] re-add csv

Re-adding csv under the datasets under construction to keep circle ci happy - will have to see how to include it in the tests.

@lhoestq noticed that I accidently deleted it in https://github.com/huggingface/nlp/pull/63#discussion_r422263729.


[Tests] Test files locally

This PR adds a `aws` and a `local` decorator to the tests so that tests now run on the local datasets.

By default, the `aws` is deactivated and `local` is activated and `slow` is deactivated, so that only 1 test per dataset runs on circle ci.

**When local is activated all folders in `./datasets` are tested.**

**Important** When adding a dataset, we should no longer upload it to AWS. The steps are:
1. Open a PR
2. Add a dataset as described in `datasets/README.md`
3. If all tests pass, push to master

Currently we have 49 functional datasets in our code base.

We have 6 datasets "under-construction" that don't pass the tests - so I put them in a folder "datasets_under_construction" - it would be nice to open a PR to fix them and put them in the `datasets` folder.

**Important** when running tests locally, the datasets are cached so to rerun them delete your local cache via:
`rm -r ~/.cache/huggingface/datasets/*`

@thomwolf @mariamabarham @lhoestq


[Datasets] ReadME


fix math dataset and xcopa

- fixes math dataset and xcopa, uploaded both of the to S3


[Datasets] Make master ready for datasets adding

Add all relevant files so that datasets can now be added on master


[Dataset scripts] add all datasets scripts

As mentioned, we can have the canonical datasets in the master. For now I also want to include all the data as present on S3 to make the synchronization easier when uploading new datastes.

@mariamabarham @lhoestq @thomwolf - what do you think?

If this is ok for you, I can sync up the master with the `add_dataset` branch: https://github.com/huggingface/nlp/pull/37 so that master is up to date.


[Cached Path] Better error message

IMO returning `None` in this function only leads to confusion and is never helpful.


[Load] rename setup_module to prepare_module

rename setup_module to prepare_module due to issues with pytests `setup_module` function.
See: PR #59.


Update to simplify some datasets conversion

> Also we should convert `tf.io.gfile.exists` into `os.path.exists` , `tf.io.gfile.listdir`into `os.listdir` and `tf.io.gfile.glob` into `glob.glob` (will need to add `import glob`)

We should probably open a new PR about this

I think it might be a good idea to both change the supervised keys to a named tuple and also handle the translation features specifically.

Just noticed that `pyarrow` apparently does not have a `is_boolean` function. Or do I have the wrong `pyarrow` version?

Ah, it was a typo `pa.types.is_boolean` is the correct name. Will fix in: https://github.com/huggingface/nlp/pull/59

This PR updates the encoding of `Values` like `integers`, `boolean` and `float` to use python casting and avoid having to cast in the dataset scripts, as mentioned here: https://github.com/huggingface/nlp/pull/37#discussion_r420176626

We could also change (not included in this PR yet):
- `supervized_keys` to make them a NamedTuple instead of a dataclass, and
- handle specifically the `Translation` features.
as mentioned here: https://github.com/huggingface/nlp/pull/37#discussion_r421740236

@patrickvonplaten @mariamabarham tell me if you want these two last changes as well.


Fix tests

Very weird bug indeed! I think the problem was that when importing `setup_module` we overwrote `pytest's` setup_module function. I think this is the relevant code in pytest: https://github.com/pytest-dev/pytest/blob/9d2eabb397b059b75b746259daeb20ee5588f559/src/_pytest/python.py#L460.

Also PR: #25 introduced some renaming: `DatasetBuilder.builder_config` -> `DatasetBuilder.config` so that we will have to change most of the dataset scripts (Just replace the "builder_config" with "config").

I think the renaming is a good idea and I can do the fix with a bash regex, but will have to re-upload most of the datasets. @thomwolf @mariamabarham

> Also PR: #25 introduced some renaming: `DatasetBuilder.builder_config` -> `DatasetBuilder.config` so that we will have to change most of the dataset scripts (Just replace the "builder_config" with "config").
>
> I think the renaming is a good idea and I can do the fix with a bash regex, but will have to re-upload most of the datasets. @thomwolf @mariamabarham

I think if it only needs a re-uploading, we can rename it, `DatasetBuilder.config` is easier and sounds better

Ok seems to be fine. Most tests work - merging.

@patrickvonplaten I've broken a bit the tests with #25 while simplifying and re-organizing the `load.py` and `download_manager.py` scripts.

I'm trying to fix them here but I have a weird error, do you think you can have a look?
```bash
(datasets) MacBook-Pro-de-Thomas:datasets thomwolf$ python -m pytest -sv ./tests/test_dataset_common.py::DatasetTest::test_builder_class_snli
============================================================================= test session starts =============================================================================
platform darwin -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /Users/thomwolf/miniconda2/envs/datasets/bin/python
cachedir: .pytest_cache
rootdir: /Users/thomwolf/Documents/GitHub/datasets
plugins: xdist-1.31.0, forked-1.1.3
collected 1 item

tests/test_dataset_common.py::DatasetTest::test_builder_class_snli ERROR

=================================================================================== ERRORS ====================================================================================
____________________________________________________________ ERROR at setup of DatasetTest.test_builder_class_snli ____________________________________________________________

file_path = <module 'tests.test_dataset_common' from '/Users/thomwolf/Documents/GitHub/datasets/tests/test_dataset_common.py'>
download_config = DownloadConfig(cache_dir=None, force_download=False, resume_download=False, local_files_only=False, proxies=None, user_agent=None, extract_compressed_file=True, force_extract=True)
download_kwargs = {}

    def setup_module(file_path: str, download_config: Optional[DownloadConfig] = None, **download_kwargs,) -> DatasetBuilder:
        r"""
            Download/extract/cache a dataset to add to the lib from a path or url which can be:
                - a path to a local directory containing the dataset processing python script
                - an url to a S3 directory with a dataset processing python script

            Dataset codes are cached inside the lib to allow easy import (avoid ugly sys.path tweaks)
            and using cloudpickle (among other things).

            Return: tuple of
                the unique id associated to the dataset
                the local path to the dataset
        """
        if download_config is None:
            download_config = DownloadConfig(**download_kwargs)
        download_config.extract_compressed_file = True
        download_config.force_extract = True

>       name = list(filter(lambda x: x, file_path.split("/")))[-1] + ".py"
E       AttributeError: module 'tests.test_dataset_common' has no attribute 'split'

src/nlp/load.py:169: AttributeError
============================================================================== warnings summary ===============================================================================
/Users/thomwolf/miniconda2/envs/datasets/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15
  /Users/thomwolf/miniconda2/envs/datasets/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================================================================== short test summary info ===========================================================================
ERROR tests/test_dataset_common.py::DatasetTest::test_builder_class_snli - AttributeError: module 'tests.test_dataset_common' has no attribute 'split'
========================================================================= 1 warning, 1 error in 3.63s =========================================================================
```


Aborted PR - Fix tests

Wait I messed up my branch, let me clean this.


Better cached path

I should have read this PR before doing my own: https://github.com/huggingface/nlp/pull/62 :D
will close mine. Looks great :-)

> Awesome, this is really nice!
>
> By the way, we should improve the `cached_path` method of the `transformers` repo similarly, don't you think (@patrickvonplaten in particular).

Yeah, we should do the same in `transformers` I think - will note it down.

### Changes:
- The `cached_path` no longer returns None if the file is missing/the url doesn't work. Instead, it can raise `FileNotFoundError` (missing file), `ConnectionError` (no cache and unreachable url) or `ValueError` (parsing error)
- Fix requests to firebase API that doesn't handle HEAD requests...
- Allow custom download in datasets script: it allows to use `tf.io.gfile.copy` for example, to download from google storage. I added an example: the `boolq` script


[Dataset] Tester add mock function

need to add an empty `extract()` function to make `hansard` dataset test work.


Beam datasets

Right now the changes are a bit hard to read as the one from #25 are also included. You can wait until #25 is merged before looking at the implementation details

Nice!! I tested it a bit and works quite well. I will do a my review once the #25 will be merged because there are several overlaps.

At least I can share my thoughts on your **Next** section:
1) I don't think it is a good thing to rely on tfds preprocessed datasets uploaded in their online storage, because they might be updated or deleted at any moment by Google and then possibly break our own processing.
2) Improves the pipeline is always a good direction, but in the meantime we might also share the preprocessed dataset in S3 storage. Which might be another way to see 1), instead of downloading Google preprocessed datasets, using our own ones.
3) Apache Beam can be easily integrated in Spark, so I don't see the need to replace Beam by Spark.

Ok I've merged #25 so you can rebase or merge if you want.

I fully agree with @jplu notes for the "next section".

Don't hesitate to use some credit on Google Dataflow if you think it would be useful to give it a try.

Pr is ready for review !

New minor changes:
- re-added the csv dataset builder (it was on my branch from #25 but disappeared from master)
- move the csv script and the wikipedia script to "under construction" for now
- some renaming in the `nlp-cli test` command

# Beam datasets

## Intro

Beam Datasets are using beam pipelines for preprocessing (basically lots of `.map` over objects called PCollections).
The advantage of apache beam is that you can choose which type of runner you want to use to preprocess your data. The main runners are:
- the `DirectRunner` to run the pipeline locally (default). However I encountered memory issues for big datasets (like the french or english wikipedia). Small dataset work fine
- Google Dataflow. I didn't play with it.
- Spark or Flink, two well known data processing frameworks. I tried to use the Spark/Flink local runners provided by apache beam for python and wasn't able to make them work properly though...

## From tfds beam datasets to our own beam datasets

Tensorflow datasets used beam and a complicated pipeline to shard the TFRecords files.
To allow users to download beam datasets and not having to preprocess them, they also allow to download the already preprocessed datasets from their google storage (the beam pipeline doesn't run in that case).

On our side, we replace TFRecords by something else. Arrow or Parquet do the job but I chose Parquet as: 1) there is a builtin apache beam parquet writer that is quite convenient, and 2) reading parquet from the pyarrow library is also simple and effective (there is a mmap option !)

Moreover we don't shard datasets in many many files like tfds (they were doing probably doing that mainly because of the limit of 2Gb per TFRecord file). Therefore we have a simpler pipeline that saves each split into one parquet file. We also removed the utilities to use their google storage (for now maybe ? we'll have to discuss it).

## Main changes

- Added a BeamWriter to save the output of beam pipelines into parquet files and fill dataset infos
- Create a ParquetReader and refactor a bit the arrow_reader.py

\> **With this, we can now try to add beam datasets from tfds**

I already added the wikipedia one, and I will also try to add the Wiki40b dataset

## Test the wikipedia script

You can download and run the beam pipeline for wikipedia (using the `DirectRunner` by default) like this:

```
>>> import nlp
>>> nlp.load("datasets/nlp/wikipedia", dataset_config="20200501.frr")
```

This wikipedia dataset (lang: frr, North Frisian) is a small one (~10Mb), but feel free to try bigger ones (and fill 20Gb of swap memory if you try the english one lol)

## Next

Should we allow to download preprocessed datasets from the tfds google storage ?
Should we try to optimize the beam pipelines to run locally without memory issues ?
Should we try other data processing frameworks for big datasets, like spark ?

## About this PR

It should be merged after #25

-----------------

I'd be happy to have your feedback and your ideas to improve the processing of big datasets like wikipedia :)


[Tests] Improved Error message for dummy folder structure

Improved Error message


[Features] Typo in generate_from_dict

Change `isinstance` test in features when generating features from dict.


allow dummy folder structure to handle dict of lists

`esnli.py` needs that extension of the dummy data testing.


[Testing] Improved testing structure

Awesome!
Let's have this in the doc at the end :-)

This PR refactors the test design a bit and puts the mock download manager in the `utils` files as it is just a test helper class.

as @mariamabarham pointed out, creating a dummy folder structure can be quite hard to grasp.
This PR tries to change that to some extent.

It follows the following logic for the `dummy` folder structure now:
1.) The data bulider has no config -> the  `dummy` folder structure is:
`dummy/<version>/dummy_data.zip`
2) The data builder has >= 1 configs -> the `dummy` folder structure is:
`dummy/<config_name_1>/<version>/dummy_data.zip`
`dummy/<config_name_2>/<version>/dummy_data.zip`

Now, the difficult part is how to create the `dummy_data.zip` file. There are two cases:
A) The `data_urs` parameter inserted into the `download_and_extract` fn is a **string**:
-> the `dummy_data.zip` file zips the folder:
`dummy_data/<relative_path_of_folder_structure_of_url>`
B) The `data_urs` parameter inserted into the `download_and_extract` fn is a **dict**:
-> the `dummy_data.zip` file zips the folder:
`dummy_data/<relative_path_of_folder_structure_of_url_behind _key_1>`
`dummy_data/<relative_path_of_folder_structure_of_url_behind _key_2>`

By relative folder structure I mean `url_path.split('./')[-1]`. As an example the dataset **xquad** by deepmind has the following url path behind the key `de`: `https://github.com/deepmind/xquad/blob/master/xquad.de.json`
-> This means that the relative url path should be `xquad.de.json`.

@mariamabarham B) is a change from how is was before and I think is makes more sense.
While before the `dummy_data.zip` file for xquad with config `de` looked like:
`dummy_data/de` it would now look like `dummy_data/xquad.de.json`. I think this is better and easier to understand.

Therefore there are currently 6 tests that would have to have changed their dummy folder structure, but which can easily be done (30min).

I also added a function: `print_dummy_data_folder_structure` that prints out the expected structures when testing which should be quite helpful.


[Tests] test only for fast test as a default

Test failure is not related to change in test file.

Test only for one config on circle ci to speed up testing. Add all config test as a slow test.
@mariamabarham @thomwolf


fix flatten nested


[Command Convert] remove tensorflow import

Remove all tensorflow import statements.


[PyArrow Feature] fix py arrow bool

To me it seems that `bool` can only be accessed with `bool_` when looking at the pyarrow types: https://arrow.apache.org/docs/python/api/datatypes.html.


[Features] Strip str key before dict look-up

The dataset `anli.py` currently fails because it tries to look up a key `1\n` in a dict that only has the key `1`. Added an if statement to strip key if it cannot be found in dict.


[Load] Separate Module kwargs and builder kwargs.

Kwargs for the `load_module` fn should be passed with `module_xxxx` to `builder_kwargs` of `load` fn.

This is a follow-up PR of: https://github.com/huggingface/nlp/pull/41


[Tests] Fix tests for datasets with no config

Forgot to fix `None` problem for datasets that have no config this in PR: https://github.com/huggingface/nlp/pull/42


[Checksums] If no configs exist prevent to run over empty list

Whoops I fixed it directly on master before checking that you have done it in this PR. We may close it

Yeah, I saw :-) But I think we should add this as well since some datasets have an empty list of configs and then as the code is now it would fail.

In this PR, I just make sure that the code jumps in the correct else if "there are no configs" as is the case for some datasets @mariamabarham

Sorry, I thought you meant a different commit . Just saw this one: https://github.com/huggingface/nlp/commit/7c644f284e2303b57612a6e7c904fe13906d893f
.

All good then :-)

`movie_rationales` e.g. has no configs.


[Tests] allow tests for builders without config

Some dataset scripts have no configs - the tests have to be adapted for this case.
In this case the dummy data will be saved as:
- natural_questions
  -> dummy
  -> -> 1.0.0 (version num)
  -> -> -> dummy_data.zip


[Load module] allow kwargs into load module

Currenly it is not possible to force a re-download of the dataset script.

This simple change allows to pass ``force_reload=True`` as ``builder_kwargs`` in the ``load.py`` function.


Update remote checksums instead of overwrite

When the user uploads a dataset on S3, checksums are also uploaded with the `--upload_checksums` parameter.

If the user uploads the dataset in several steps, then the remote checksums file was previously overwritten. Now it's going to be updated with the new checksums.


[Test] improve slow testing


[Checksums] Error for some datasets

@lhoestq - could you take a look? It's not very urgent though!

Fixed with 06882b4

Now your command works :)
Note that you can also do
```
nlp-cli test datasets/nlp/xnli --save_checksums
```
So that it will save the checksums directly in the right directory.

The checksums command works very nicely for `squad`. But for `crime_and_punish` and `xnli`,
the same bug happens:

When running:
```
python nlp-cli nlp-cli test xnli --save_checksums
```

leads to:

```
  File "nlp-cli", line 33, in <module>
    service.run()
  File "/home/patrick/python_bin/nlp/commands/test.py", line 61, in run
    ignore_checksums=self._ignore_checksums,
  File "/home/patrick/python_bin/nlp/builder.py", line 383, in download_and_prepare
    self._download_and_prepare(dl_manager=dl_manager, download_config=download_config)
  File "/home/patrick/python_bin/nlp/builder.py", line 627, in _download_and_prepare
    dl_manager=dl_manager, max_examples_per_split=download_config.max_examples_per_split,
  File "/home/patrick/python_bin/nlp/builder.py", line 431, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/patrick/python_bin/nlp/datasets/xnli/8bf4185a2da1ef2a523186dd660d9adcf0946189e7fa5942ea31c63c07b68a7f/xnli.py", line 95, in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
  File "/home/patrick/python_bin/nlp/utils/download_manager.py", line 246, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/patrick/python_bin/nlp/utils/download_manager.py", line 186, in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
  File "/home/patrick/python_bin/nlp/utils/download_manager.py", line 166, in _record_sizes_checksums
    self._recorded_sizes_checksums[url] = get_size_checksum(path)
  File "/home/patrick/python_bin/nlp/utils/checksums_utils.py", line 81, in get_size_checksum
    with open(path, "rb") as f:
TypeError: expected str, bytes or os.PathLike object, not tuple
```


[Datasets ToDo-List] add datasets

Note:
```
nlp-cli test datasets/nlp/<your-dataset-folder> --save_checksums --all_configs
```
directly saves the checksums in the right place, and runs for all the dataset configurations.

@patrickvonplaten can you provide the add the link to the PR for the dummy data?

https://github.com/huggingface/nlp/pull/15 - But it's probably best to checkout into this branch and look how the dummy data strtucture is for `squad` for example.

> are lock files supposed to stay ?

Not sure! I think the checksum command creates them, so I just uploaded them as well.

We can trash the `lock` file, they are dummy file that are only used to avoid concurrent access when the library is run.
You can read the filelock readme and code, it's a very simple single-file library: https://github.com/benediktschmitt/py-filelock

The testing design was slightly changed as explained in https://github.com/huggingface/nlp/pull/51 .
If creating the dummy folder is too confusing it helps to upload everything else to AWS, then run the test and check the INFO when testing on how to create the dummy folder structure.

Closing because we can now work on master

## Description

This PR acts as a dashboard to see which datasets are added to the library and work.

Cicle-ci should always be green so that we can be sure that newly added datasets are functional.
This PR should not be merged.

## Progress

**For the following datasets the test commands**:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_<your-dataset-name>
```
and
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_dataset_all_configs_<your-dataset-name>
```

**passes**.

- [x] Squad
- [x] Sentiment140
- [x] XNLI
- [x] Crime_and_Punish
- [x] movie_rationales
- [x] ai2_arc
- [x] anli
- [x] event2Mind
- [x] Fquad
- [x] blimp
- [x] empathetic_dialogues
- [x] cosmos_qa
- [x] xquad
- [x] blog_authorship_corpus
- [x] SNLI
- [x] break_data
- [x] SQuAD v2
- [x] cfq
- [x] eraser_multi_rc
- [x] Glue
- [x] Tydiqa
- [x] wiki_qa
- [x] wikitext
- [x] winogrande
- [x] wiqa
- [x] esnli
- [x] civil_comments
- [x] commonsense_qa
- [x] com_qa
- [x] coqa
- [x] wiki_split
- [x] cos_e
- [x] xcopa
- [x] quarel
- [x] quartz
- [x] squad_it
- [x] quoref
- [x] squad_pt
- [x] cornell_movie_dialog
- [x] SciQ
- [x] Scifact
- [x] hellaswag
- [x] ted_multi (in translate)
- [x] Aeslc (summarization)
- [x] drop
- [x] gap
- [x] hansard
- [x] opinosis
- [x] MLQA
- [x] math_dataset

## How-To-Add a dataset

**Before adding a dataset make sure that your branch is up to date**:
1. `git checkout add_datasets`
2. `git pull`

**Add a dataset via the `convert_dataset.sh` bash script:**

Running `bash convert_dataset.sh <file/to/tfds/datascript.py>` (*e.g.* `bash convert_dataset.sh ../tensorflow-datasets/tensorflow_datasets/text/movie_rationales.py`) will automatically run all the steps mentioned in **Add a dataset manually** below.

Make sure that you run `convert_dataset.sh` from the root folder of `nlp`.

The conversion script should work almost always for step 1): "convert dataset script from tfds to nlp format" and 2) "create checksum file" and step 3) "make style".

It can also sometimes automatically run step 4) "create the correct dummy data from tfds", but this will only work if a) there is either no config name or only one config name and b) the `tfds testing/test_data/fake_example` is in the correct form.

Nevertheless, the script should always be run in the beginning until an error occurs to be more efficient.

If the conversion script does not work or fails at some step, then you can run the steps manually as follows:

**Add a dataset manually**

Make sure you run all of the following commands from the root of your `nlp` git clone.
Also make sure that you changed to this branch:
```
git checkout add_datasets
```

1) the tfds datascript file should be converted to `nlp` style:

```
python nlp-cli convert --tfds_path <path/to/tensorflow_datasets/text/your_dataset_name>.py --nlp_directory datasets/nlp
```

This will convert the tdfs script and create a folder with the correct name.

2) the checksum file should be added. Use the command:
```
python nlp-cli test datasets/nlp/<your-dataset-folder> --save_checksums --all_configs
```

A checksums.txt file should be created in your folder and the structure should look as follows:

squad/
├── squad.py/
└── urls_checksums/
...........└── checksums.txt

Delete the created `*.lock` file afterward - it should not be uploaded to AWS.

3) run black and isort on your newly added datascript files so that they look nice:

```
make style
```

4) the dummy data should be added. For this it might be useful to take a look into the structure of other examples as shown in the PR here and at `<path/to/tensorflow_datasets/testing/test_data/test_data/fake_examples>` whether the same  data can be used.

5)  the data can be uploaded to AWS using the command
```
aws s3 cp datasets/nlp/<your-dataset-folder> s3://datasets.huggingface.co/nlp/<your-dataset-folder> --recursive
```

6) check whether all works as expected using:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_<your-dataset-name>
```
and
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_dataset_all_configs_<your-dataset-name>
```

7) push to this PR and rerun the circle ci workflow to check whether circle ci stays green.

8) Edit this commend and tick off your newly added dataset :-)

## TODO-list

Maybe we can add a TODO-list here for everybody that feels like adding new datasets so that we will not add the same datasets.

Here a link to available datasets: https://docs.google.com/spreadsheets/d/1zOtEqOrnVQwdgkC4nJrTY6d-Av02u0XFzeKAtBM2fUI/edit#gid=0

Patrick:

- [ ] boolq - *weird download link*
- [ ] c4 - *beam dataset*


Metrics - refactoring, adding support for download and distributed metrics

Ok, this one seems to be ready to merge.

> Really cool, I love it! I would just raise a tiny point, the distributive version of the metrics might not work properly with TF because it is a different way to do, why not to add a "framework" detection and raise warning when TF is used, saying something like "not available yet in TF switch to non distributive metric computation".
>
> What do you think?

Good point @jplu I'm not sure how you should do distributed metrics evaluation for TF.
There is only one python script, right?
Maybe it's just the same as in the not-distributed case?

I think non-distributed case should work in TF for both cases indeed, but this needs to be tested.

Refactoring metrics to have a similar loading API than the datasets and improving the import system.

# Import system
The import system has ben upgraded. There are now three types of imports allowed:
1. `library` imports (identified as "absolute imports")
```python
import seqeval
```
=> we'll test all the imports before running the scripts and if one cannot be imported we'll display an error message like this one:
`ImportError: To be able to use this metric/dataset, you need to install the following dependencies ['seqeval'] using 'pip install seqeval' for instance'`

2. `internal` imports (identified as "relative imports")
```python
import .c4_utils
```
=> we'll assume this point to a file in the same directory/S3-directory as the main script and download this file.

2. `external` imports (identified as "relative imports" with a comment starting with `# From:`)
```python
from .nmt_bleu import compute_bleu  # From: https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py
```
=> we'll assume this point to the URL of a python script (if it's a link to a github file, we'll take the raw file automatically).
=> the script is downloaded and renamed to the import name (here above renamed from `bleu.py` to `nmt_bleu.py`). Renaming the file can be necessary if the distant file has the same name as the dataset/metric processing script. If you forgot to rename the distant script and it has the same name as the dataset/metric, you'll have an explicit error message asking to rename the import anyway.

# Hosting metrics

Metrics are hosted on a S3 bucket like the dataset processing scripts.

# Metrics scripts

Metrics scripts have a lot in common with datasets processing scripts. They also have a `metric.info` including citations, descriptions and links to relevant pages.

Metrics have more documentation to supply to ensure they are used well.

Four examples are already included for reference in [./metrics](./metrics): BLEU, ROUGE, SacreBLEU and SeqEVAL.

# Automatic support for distributed/multi-processing metric computation

We've also added support for automatic distributed/multi-processing metric computation (e.g. when using DistributedDataParallel). We leverage our own dataset format for smart caching in this case.

Here is a quick gist of a standard use of metrics (the simplest usage):
```python
import nlp
bleu_metric = nlp.load_metric('bleu')

# If you only have a single iteration, you can easily compute the score like this
predictions = model(inputs)
score = bleu_metric.compute(predictions, references)

# If you have a loop, you can "add" your predictions and references at each iteration instead of having to save them yourself (the metric object store them efficiently for you)
for batch in dataloader:
    model_input, targets = batch
    predictions = model(model_inputs)
    bleu.add(predictions, targets)
score = bleu_metric.compute()  # Compute the score from all the stored predictions/references
```

Here is a quick gist of a use in a distributed torch setup (should work for any python multi-process setup actually). It's pretty much identical to the second example above:
```python
import nlp
# You need to give the total number of parallel python processes (num_process) and the id of each process (process_id)
bleu = nlp.load_metric('bleu', process_id=torch.distributed.get_rank(),b num_process=torch.distributed.get_world_size())

for batch in dataloader:
    model_input, targets = batch
    predictions = model(model_inputs)
    bleu.add(predictions, targets)
score = bleu_metric.compute()  # Compute the score on the first node by default (can be set to compute on each node as well)
```


[Tests] fix typo

@lhoestq - currently the slow test fail with:

```
_____________________________________________________________________________________ DatasetTest.test_load_real_dataset_xnli _____________________________________________________________________________________

self = <tests.test_dataset_common.DatasetTest testMethod=test_load_real_dataset_xnli>, dataset_name = 'xnli'

    @slow
    def test_load_real_dataset(self, dataset_name):
        with tempfile.TemporaryDirectory() as temp_data_dir:
>           dataset = load(dataset_name, data_dir=temp_data_dir)

tests/test_dataset_common.py:153:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../python_bin/nlp/load.py:497: in load
    dbuilder.download_and_prepare(**download_and_prepare_kwargs)
../../python_bin/nlp/builder.py:383: in download_and_prepare
    self._download_and_prepare(dl_manager=dl_manager, download_config=download_config)
../../python_bin/nlp/builder.py:627: in _download_and_prepare
    dl_manager=dl_manager, max_examples_per_split=download_config.max_examples_per_split,
../../python_bin/nlp/builder.py:431: in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
../../python_bin/nlp/datasets/xnli/8bf4185a2da1ef2a523186dd660d9adcf0946189e7fa5942ea31c63c07b68a7f/xnli.py:95: in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
../../python_bin/nlp/utils/download_manager.py:246: in download_and_extract
    return self.extract(self.download(url_or_urls))
../../python_bin/nlp/utils/download_manager.py:186: in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
../../python_bin/nlp/utils/download_manager.py:166: in _record_sizes_checksums
    self._recorded_sizes_checksums[url] = get_size_checksum(path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = ('', '/tmp/tmpkajlg9yc/downloads/c0f7773c480a3f2d85639d777e0e17e65527460310d80760fd3fc2b2f2960556.c952a63cb17d3d46e412ceb7dbcd656ce2b15cc9ef17f50c28f81c48a7c853b5')

    def get_size_checksum(path: str) -> Tuple[int, str]:
        """Compute the file size and the sha256 checksum of a file"""
        m = sha256()
>       with open(path, "rb") as f:
E       TypeError: expected str, bytes or os.PathLike object, not tuple

../../python_bin/nlp/utils/checksums_utils.py:81: TypeError
```

- the checksums probably need to be updated no? And we should also think about how to write a test for the checksums.


[Tests] add slow tests

This PR adds a slow test that downloads the "real" dataset. The test is decorated as "slow" so that it will not automatically run on circle ci.

Before uploading a dataset, one should test that this test passes, manually by running

```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_<your-dataset-script-name>
```

This PR should be merged after PR: #33


Big cleanup/refactoring for clean serialization

Great! I think when this merged, we can merge sure that Circle Ci stays happy when uploading new datasets.

This PR cleans many base classes to re-build them as `dataclasses`. We can thus use a simple serialization workflow for `DatasetInfo`, including it's `Features` and `SplitDict` based on `dataclasses` `asdict()`.

The resulting code is a lot shorter, can be easily serialized/deserialized, dataset info are human-readable and we can get rid of the `dataclass_json` dependency.

The scripts have breaking changes and the conversion tool is updated.

Example of dataset info in SQuAD script now:
```python
  def _info(self):
    return nlp.DatasetInfo(
        description=_DESCRIPTION,
        features=nlp.Features({
            "id":
                nlp.Value('string'),
            "title":
                nlp.Value('string'),
            "context":
                nlp.Value('string'),
            "question":
                nlp.Value('string'),
            "answers":
                nlp.Sequence({
                    "text": nlp.Value('string'),
                    "answer_start": nlp.Value('int32'),
                }),
        }),
        # No default supervised_keys (as we have to pass both question
        # and context as input).
        supervised_keys=None,
        homepage="https://rajpurkar.github.io/SQuAD-explorer/",
        citation=_CITATION,
    )
```

Example of serialized dataset info:
```bash
{
    "description": "Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.\n",
    "citation": "@article{2016arXiv160605250R,\n             author = {{Rajpurkar}, Pranav and {Zhang}, Jian and {Lopyrev},\n                                 Konstantin and {Liang}, Percy},\n                title = \"{SQuAD: 100,000+ Questions for Machine Comprehension of Text}\",\n            journal = {arXiv e-prints},\n                 year = 2016,\n                    eid = {arXiv:1606.05250},\n                pages = {arXiv:1606.05250},\narchivePrefix = {arXiv},\n             eprint = {1606.05250},\n}\n",
    "homepage": "https://rajpurkar.github.io/SQuAD-explorer/",
    "license": "",
    "features": {
        "id": {
            "dtype": "string",
            "_type": "Value"
        },
        "title": {
            "dtype": "string",
            "_type": "Value"
        },
        "context": {
            "dtype": "string",
            "_type": "Value"
        },
        "question": {
            "dtype": "string",
            "_type": "Value"
        },
        "answers": {
            "feature": {
                "text": {
                    "dtype": "string",
                    "_type": "Value"
                },
                "answer_start": {
                    "dtype": "int32",
                    "_type": "Value"
                }
            },
            "length": -1,
            "_type": "Sequence"
        }
    },
    "supervised_keys": null,
    "name": "squad",
    "version": {
        "version_str": "1.0.0",
        "description": "New split API (https://tensorflow.org/datasets/splits)",
        "nlp_version_to_prepare": null,
        "major": 1,
        "minor": 0,
        "patch": 0
    },
    "splits": {
        "train": {
            "name": "train",
            "num_bytes": 79426386,
            "num_examples": 87599,
            "dataset_name": "squad"
        },
        "validation": {
            "name": "validation",
            "num_bytes": 10491883,
            "num_examples": 10570,
            "dataset_name": "squad"
        }
    },
    "size_in_bytes": 0,
    "download_size": 35142551,
    "download_checksums": []
}
```


Fix map caching notebooks

Previously, caching results with `.map()` didn't work in notebooks.
To reuse a result, `.map()` serializes the functions with `dill.dumps` and then it hashes it.

The problem is that when using `dill.dumps` to serialize a function, it also saves its origin (filename + line no.) and the origin of all the `globals` this function needs. However for notebooks and shells, the filename looks like \<ipython-input-13-9ed2afe61d25\> and the line no. changes often.

To fix the problem, I added a new dispatch function for code objects that ignore the origin of the code if it comes from a notebook or a python shell.

I tested these cases in a notebook:
- lambda functions
- named functions
- methods
- classmethods
- staticmethods
- classes that implement `__call__`

The caching now works as expected for all of them :)
I also tested the caching in the demo notebook and it works fine !


[Circle ci] Install a virtual env before running tests

Install a virtual env before running tests to not running into sudo issues when dynamically downloading files.

Same number of tests now pass / fail as on my local computer:
![Screenshot from 2020-05-01 12-14-44](https://user-images.githubusercontent.com/23423619/80798814-8a0a0a80-8ba5-11ea-8db8-599d33bbfccd.png)


add metrics which require download files from github

To download files from github, I copied the `load_dataset_module` and its dependencies (without the builder) in `load.py` to `metrics/metric_utils.py`. I made the following changes:

- copy the needed files in a folder`metric_name`
- delete all other files that are not needed

For metrics that require an external import, I first create a `<metric_name>_imports.py` file which contains all external urls. Then I create a `<metric_name>.py` in which I will load the external files using `<metric_name>_imports.py`


Hf_api small changes

From Patrick:
```python
from nlp import hf_api
api = hf_api.HfApi()
api.dataset_list()
```

works :-)


[Circle ci] Adds circle ci config

@thomwolf can you take a look and set up circle ci on:
https://app.circleci.com/projects/project-dashboard/github/huggingface

I think for `nlp` only admins can set it up, which I guess is you :-)


[Cleanup] Removes all files in testing except test_dataset_common

As far as I know, all files in `tests` were old `tfds test files` so I removed them. We can still look them up on the other library.


[Tests] Clean tests

the abseil testing library (https://abseil.io/docs/python/quickstart.html) is better than the one I had before, so I decided to switch to that and changed the `setup.py` config file.
Abseil has more support and a cleaner API for parametrized testing I think.

I added a list of all dataset scripts that are currently on AWS, but will replace that once the
API is integrated into this lib.

One can now easily test for just a single function for a single dataset with:
`tests/test_dataset_common.py::DatasetTest::test_load_dataset_wikipedia`

NOTE: This PR is rebased on PR #29 so should be merged after.


Add script csv datasets

Very interesting thoughts, we should think deeper about all what you raised indeed.

Ok here is a proposal for a more general API and workflow.

# New `ArrowBasedBuilder`

For all the formats that can be directly and efficiently loaded by Arrow (CSV, JSON, Parquet, Arrow), we don't really want to have to go through a conversion to python and back to Arrow. This new builder has a `_generate_tables` method to yield `Arrow.Tables` instead of single examples.
The tables can be directly casted in Arrow so it's not necessary to supply  `Features`, they can be deduced from the `Table` column.

# Central role of the `BuilderConfig` to store all the arguments necessary for the Dataset creation.

`BuilderConfig` provide a few defaults fields `name`, `version`,  `description`,  `data_files` and `data_dir` which can be used to store values necessary for the creation of the dataset. It can be freely extended to store additional information (see the example for `CsvConfig`).

On the contrary, `DatasetInfo` is designed as an organized and delimited information storage class with predefined fields.

`DatasetInfo` now store two names:
- `builder_name`: Name of the builder script used to create the dataset
- `config_name`: Name of the configuration used to create the dataset.

# Refactoring `load()` arguments and all the chain of processing including the `DownloadManager`

`load()` now accept a selection of arguments which are used to update the `BuilderConfig` and some kwargs which are used to handle the download process.

Supplying a `BuilderConfig` as `config` will override the config provided in the dataset. Supplying a `str` will get the associated config from the dataset. Default is to fetch the first config of the dataset.

Giving additional arguments to `load()` will override the arguments in the `BuilderConfig`.

# CSV script

The `csv.py` script is provided as an example, usage is:
```python
bbc = nlp.load('/Users/thomwolf/Documents/GitHub/datasets/datasets/nlp/csv',
               name='bbc',
               version="1.0.1",
               split='train',
               data_files={'train': ['/Users/thomwolf/Documents/GitHub/datasets/datasets/dummy_data/csv/test.csv']},
               skip_rows=10,
               download_mode='force_redownload')
```

# Checksums

We now don't raise an error if the checksum file is not found.

# `DownloadConfig`

We now have a download configuration class to handle all the specific arguments for file caching like proxies, using only local files or user-agents.

Ok merging this for now.

One general note is that it's a bit hard to handle the `ClassLabel` generally in both `nlp` and `Arrow` since a class label typically need some metadata for the class names. For now, I raise a `NotImplementedError` when an `ArrowBuilder` output a table with a `DictionaryType` is encountered (which could be a simple equivalent for a `ClassLabel` Feature in Arrow tables).

In general and if we need this in the future for some Beam Datasets for instance, I think we should use one of the `metadata` fields in the `Arrow` type or table's schema to store the relation with indices and class names.

So ping me if you meet Beam datasets which uses `ClassLabels` (cc @lhoestq @patrickvonplaten @mariamabarham).

This is a PR allowing to create datasets from local CSV files. A usage might be:

```python
import nlp
ds = nlp.load(
    path="csv",
    name="bbc",
    dataset_files={
        nlp.Split.TRAIN: ["datasets/dummy_data/csv/train.csv"],
        nlp.Split.TEST: [""datasets/dummy_data/csv/test.csv""]
    },
    csv_kwargs={
        "skip_rows": 0,
        "delimiter": ",",
        "quote_char": "\"",
        "header_as_column_names": True
    }
)
```

```
Downloading and preparing dataset bbc/1.0.0 (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/jplu/.cache/huggingface/datasets/bbc/1.0.0...
Dataset bbc downloaded and prepared to /home/jplu/.cache/huggingface/datasets/bbc/1.0.0. Subsequent calls will reuse this data.
{'test': Dataset(schema: {'category': 'string', 'text': 'string'}, num_rows: 49), 'train': Dataset(schema: {'category': 'string', 'text': 'string'}, num_rows: 99), 'validation': Dataset(schema: {'category': 'string', 'text': 'string'}, num_rows: 0)}
```

How it is read:

- `path`: the `csv` word means "I want to create a CSV dataset"
- `name`: the name of this dataset is `bbc`
- `dataset_files`: this is a dictionary where each key is the list of files corresponding to the key split.
- `csv_kwargs`: this is the keywords arguments to "explain" how to read the CSV files
  * `skip_rows`: number of rows have to be skipped, starting from the beginning of the file
  * `delimiter`: which delimiter is used to separate the columns
  * `quote_char`: which quote char is used to represent a column where the delimiter appears in one of them
  * `header_as_column_names`: will use the first row (header) of the file as name for the features. Otherwise the names will be automatically generated as `f1`, `f2`, etc... Will be applied after the `skip_rows` parameter.

**TODO**:  for now the `csv.py` is copied each time we create a new dataset as `ds_name.py`, this behavior will be modified to have only the `csv.py` script copied only once and not for all the CSV datasets.


Add checksums

Looks good to me :-)

Just would prefer to get rid of the `_DYNAMICALLY_IMPORTED_MODULE` attribute and replace it by a `get_imported_module()` function. Maybe there is something I'm not seeing here though - what do you think?

> * I'm not sure I understand the general organization of checksums. I see we have a checksum folder with potentially several checksum files but I also see that checksum files can potentially contain several checksums. Could you explain a bit more how this is organized?

It should look like this:
squad/
├── squad.py/
└── urls_checksums/
...........└── checksums.txt

In checksums.txt, the format is one line per (url, size, checksum)

I don't have a strong opinion between `urls_checksums/checksums.txt` or directly `checksums.txt` (not inside the `urls_checksums` folder), let me know what you think.

> * Also regarding your comment on checksum files for "canonical" datasets. I understand we can just create these with `nlp-cli test` and then upload them manually to our S3, right?

Yes you're right

Update of the commands:

- nlp-cli test \<dataset\> : Run download_and_prepare and verify checksums
  * --name \<name\> : run only for the name
  * --all_configs : run for all configs
  * --save_checksums : instead of verifying checksums, compute and save them
  * --ignore_checksums : don't do checksums verification

- nlp-cli upload \<dataset_folder\> : Upload a dataset
  * --upload_checksums : compute and upload checksums for uploaded files

TODO:
- don't overwrite checksums files on S3, to let the user upload a dataset in several steps if needed

Question:
- One idea from @patrickvonplaten : shall we upload checksums everytime we upload files ? (and therefore remove the upload_checksums parameter)

### Checksums files

They are stored next to the dataset script in urls_checksums/checksums.txt.
They are used to check the integrity of the datasets downloaded files.
I kept the same format as tensorflow-datasets.
There is one checksums file for all configs.

### Load a dataset

When you do `load("squad")`, it will also download the checksums file and put it next to the script in nlp/datasets/hash/urls_checksums/checksums.txt.
It also verifies that the downloaded files checksums match the expected ones.

You can ignore checksum tests with `load("squad", ignore_checksums=True)` (under the hood it just adds `ignore_checksums=True` in the `DownloadConfig`)

### Test a dataset

There is a new command `nlp-cli test squad` that runs `download_and_prepare` to see if it runs ok, and that verifies that all the checksums match. Allowed arguments are `--name`, `--all_configs`, `--ignore_checksums` and `--register_checksums`.

### Register checksums

1. If the dataset has external dataset files

The command `nlp-cli test squad --register_checksums --all_configs` runs `download_and_prepare` on all configs to see if it runs ok, and it creates the checksums file.
You can also register one config at a time using `--name` instead ; the checksums file will be completed and not overwritten.

If the script is a local script, the checksum file is moved to urls_checksums/checksums.txt next to the local script, to enable the user to upload both the script and the checksums file afterwards with `nlp-cli upload squad`.

2. If the dataset files are all inside the directory of the dataset script

The user can directly do `nlp-cli upload squad --register_checksums`, as there is no need to download anything.
In this case however, all the dataset must be uploaded at once.

--

PS : it doesn't allow to register checksums for canonical datasets, the file has to be added manually on S3 for now (I guess ?)

Also I feel like we must be sure that this processes would not constrain too much any user from uploading its dataset.
Let me know what you think :)


Add metrics

This PR is a draft for adding metrics (sacrebleu and seqeval are added)

use case examples:
`import nlp`
**sacrebleu:**
```
refs = [['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
        ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.']]
sys = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']
sacrebleu = nlp.load_metrics('sacrebleu')
print(sacrebleu.score)
```

**seqeval:**
```
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
seqeval = nlp.load_metrics('seqeval')
print(seqeval.accuracy_score(y_true, y_pred)
print(seqeval.f1_score(y_true, y_pred)
```
_examples are taken from the corresponding web page_

your comments and suggestions are more than welcomed


adding bleu score code

this PR add the BLEU score metric to the lib. It can be tested by running the following code.

` from nlp.metrics import bleu

hyp1 = "It is a guide to action which ensures that the military always obeys the commands of the party"
ref1a = "It is a guide to action that ensures that the military  forces always being under the commands of the party "
 ref1b = "It is the guiding principle which guarantees the military force always being under the command of the Party"
ref1c = "It is the practical guide for the army always to heed the directions of the party"

list_of_references = [[ref1a, ref1b, ref1c]]
hypotheses = [hyp1]
bleu = bleu.bleu_score(list_of_references, hypotheses,4, smooth=True)
print(bleu) `