kardwalker/my_dataset huggingface dataset:
GitHub issue #172: Clone not working on Windows environment
State: closed | Created: 2020-05-20T00:45:14Z

Cloning in a windows environment is not working because of use of special character '?' in folder name ..
Please consider changing the folder name ....
Reference to folder -
nlp/datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs/dailymail/stories/

error log:
fatal: cannot create directory at 'datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs': Invalid argument
GitHub issue #171: fix squad metric format
State: closed | Created: 2020-05-19T18:37:36Z

The format of the squad metric was wrong.
This should fix #143 

I tested with
```python3
predictions = [
    {'id': '56be4db0acb8001400a502ec', 'prediction_text': 'Denver Broncos'}
]
references = [
    {'answers': [{'text': 'Denver Broncos'}], 'id': '56be4db0acb8001400a502ec'}
]
```
GitHub issue #170: Rename anli dataset
State: closed | Created: 2020-05-19T16:26:57Z

What we have now as the `anli` dataset is actually the αNLI dataset from the ART challenge dataset. This name is confusing because `anli` is also the name of adversarial NLI (see [https://github.com/facebookresearch/anli](https://github.com/facebookresearch/anli)).

I renamed the current `anli` dataset by `art`.
GitHub issue #169: Adding Qanta (Quizbowl) Dataset
State: closed | Created: 2020-05-19T16:03:01Z

This PR adds the qanta question answering datasets from [Quizbowl: The Case for Incremental Question Answering](https://arxiv.org/abs/1904.04792) and [Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples](https://www.aclweb.org/anthology/Q19-1029/) (adversarial fold)

This partially continues a discussion around fixing dummy data from https://github.com/huggingface/nlp/issues/161

I ran the following code to double check that it works and did some sanity checks on the output. The majority of the code itself is from our `allennlp` version of the dataset reader.

```python
import nlp
# Default is full question
data = nlp.load_dataset('./datasets/qanta') 
# Four configs
# Primarily useful for training
data = nlp.load_dataset('./datasets/qanta', 'mode=sentences,char_skip=25')  
# Primarily used in evaluation
data = nlp.load_dataset('./datasets/qanta', 'mode=first,char_skip=25')  
data = nlp.load_dataset('./datasets/qanta', 'mode=full,char_skip=25')  
# Primarily useful in evaluation and "live" play
data = nlp.load_dataset('./datasets/qanta', 'mode=runs,char_skip=25')  
```
GitHub issue #167: [Tests] refactor tests
State: closed | Created: 2020-05-19T11:43:32Z

This PR separates AWS and Local tests to remove these ugly statements in the script:
```python
        if "/" not in dataset_name:
            logging.info("Skip {} because it is a canonical dataset")
            return
```

To run a `aws` test, one should now run the following command: 

```python 
pytest -s tests/test_dataset_common.py::AWSDatasetTest::test_builder_class_wmt14
```

The same `local` test, can be run with:
```python 
pytest -s tests/test_dataset_common.py::LocalDatasetTest::test_builder_class_wmt14
```
GitHub issue #166: Add a method to shuffle a dataset
State: closed | Created: 2020-05-19T10:08:46Z

Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think?
GitHub issue #165: ANLI
State: closed | Created: 2020-05-19T07:50:57Z

Can I recommend the following:

For ANLI, use https://github.com/facebookresearch/anli. As that paper says, "Our dataset is not
to be confused with abductive NLI (Bhagavatula et al., 2019), which calls itself αNLI, or ART.". 

Indeed, the paper cited under what is currently called anli says in the abstract "We introduce a challenge dataset, ART".

The current naming will confuse people :)
GitHub issue #163: [Feature request] Add cos-e v1.0
State: closed | Created: 2020-05-18T22:05:26Z

I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
GitHub issue #161: Discussion on version identifier & MockDataLoaderManager for test data
State: open | Created: 2020-05-18T20:31:30Z

Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
GitHub issue #157: nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
State: closed | Created: 2020-05-18T16:46:38Z

I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
GitHub issue #156: SyntaxError with WMT datasets
State: closed | Created: 2020-05-18T14:38:18Z

The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook
GitHub issue #153: Meta-datasets (GLUE/XTREME/...) – Special care to attributions and citations
State: open | Created: 2020-05-18T07:24:22Z

Meta-datasets are interesting in terms of standardized benchmarks but they also have specific behaviors, in particular in terms of attribution and authorship. It's very important that each specific dataset inside a meta dataset is properly referenced and the citation/specific homepage/etc are very visible and accessible and not only the generic citation of the meta-dataset itself.

Let's take GLUE as an example:

The configuration has the citation for each dataset included (e.g. [here](https://github.com/huggingface/nlp/blob/master/datasets/glue/glue.py#L154-L161)) but it should be copied inside the dataset info so that, when people access `dataset.info.citation` they get both the citation for GLUE and the citation for the specific datasets inside GLUE that they have loaded.
GitHub issue #150: Add WNUT 17 NER dataset
State: closed | Created: 2020-05-17T22:19:04Z

Hi,

this PR adds the WNUT 17 dataset to `nlp`.

> Emerging and Rare entity recognition
> This shared task focuses on identifying unusual, previously-unseen entities in the context of emerging discussions. Named entities form the basis of many modern approaches to other tasks (like event clustering and summarisation), but recall on them is a real problem in noisy text - even among annotators. This drop tends to be due to novel entities and surface forms. Take for example the tweet “so.. kktny in 30 mins?” - even human experts find entity kktny hard to detect and resolve. This task will evaluate the ability to detect and classify novel, emerging, singleton named entities in noisy text.
> 
> The goal of this task is to provide a definition of emerging and of rare entities, and based on that, also datasets for detecting these entities.

More information about the dataset can be found on the [shared task page](https://noisy-text.github.io/2017/emerging-rare-entities.html).

Dataset is taken is taken from their [GitHub repository](https://github.com/leondz/emerging_entities_17), because the data provided in this repository contains minor fixes in the dataset format.

## Usage

Then the WNUT 17 dataset can be used in `nlp` like this:

```python
import nlp

wnut_17 = nlp.load_dataset("./datasets/wnut_17/wnut_17.py")

print(wnut_17)
```

This outputs:

```txt
'train': Dataset(schema: {'id': 'string', 'tokens': 'list<item: string>', 'labels': 'list<item: string>'}, num_rows: 3394)
'validation': Dataset(schema: {'id': 'string', 'tokens': 'list<item: string>', 'labels': 'list<item: string>'}, num_rows: 1009)
'test': Dataset(schema: {'id': 'string', 'tokens': 'list<item: string>', 'labels': 'list<item: string>'}, num_rows: 1287)
```

Number are identical with the ones in [this paper](https://www.ijcai.org/Proceedings/2019/0702.pdf) and are the same as using the `dataset` reader in Flair.

## Features

The following feature format is used to represent a sentence in the WNUT 17 dataset:

| Feature | Example | Description
| ---- | ---- | -----------------
| `id` | `0` | Number (id) of current sentence
| `tokens` | `["AHFA", "extends", "deadline"]` | List of tokens (strings) for a sentence
| `labels` | `["B-group", "O", "O"]` | List of labels (outer span)

The following labels are used in WNUT 17:

```txt
O
B-corporation
I-corporation
B-location
I-location
B-product
I-product
B-person
I-person
B-group
I-group
B-creative-work
I-creative-work
```
GitHub issue #148: _download_and_prepare() got an unexpected keyword argument 'verify_infos'
State: closed | Created: 2020-05-17T01:48:53Z

# Reproduce
In Colab,
```
%pip install -q  nlp
%pip install -q apache_beam mwparserfromhell

dataset = nlp.load_dataset('wikipedia')
```
get
```
Downloading and preparing dataset wikipedia/20200501.aa (download: Unknown size, generated: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/wikipedia/20200501.aa/1.0.0...

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-6-52471d2a0088> in <module>()
----> 1 dataset = nlp.load_dataset('wikipedia')

1 frames

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519 

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

TypeError: _download_and_prepare() got an unexpected keyword argument 'verify_infos'
```
GitHub issue #147: Error with sklearn train_test_split
State: closed | Created: 2020-05-17T00:28:24Z

It would be nice if we could use sklearn `train_test_split` to quickly generate subsets from the dataset objects returned by `nlp.load_dataset`. At the moment the code:

```python
data = nlp.load_dataset('imdb', cache_dir=data_cache)
f_half, s_half = train_test_split(data['train'], test_size=0.5, random_state=seed)
```
throws:
```
ValueError: Can only get row(s) (int or slice) or columns (string).
```
It's not a big deal, since there are other ways to split the data, but it would be a cool thing to have.
GitHub issue #146: Add BERTScore to metrics
State: closed | Created: 2020-05-16T22:09:39Z

This PR adds [BERTScore](https://arxiv.org/abs/1904.09675) to metrics.
Here is an example of how to use it.

```sh
import nlp
bertscore = nlp.load_metric('metrics/bertscore') # or simply nlp.load_metric('bertscore') after this is added to huggingface's s3 bucket
predictions = ['example', 'fruit']
references = [['this is an example.', 'this is one example.'], ['apple']]
results = bertscore.compute(predictions, references, lang='en')
print(results)
```
GitHub issue #144: [AWS tests] AWS test should not run for canonical datasets
State: closed | Created: 2020-05-16T13:39:30Z

AWS tests should in general not run for canonical datasets. Only local tests will run in this case. This way a PR is able to pass when adding a new dataset.

This PR changes to logic to the following: 

1) All datasets that are present in `nlp/datasets` are tested only locally. This way when one adds a canonical dataset, the PR includes his dataset in the tests.

2) All datasets that are only present on AWS, such as `webis/tl_dr` atm are tested only on AWS. 

I think the testing structure might need a bigger refactoring and better documentation very soon. 

Merging for now to unblock new PRs @thomwolf @mariamabarham .
GitHub issue #143: ArrowTypeError in squad metrics
State: closed | Created: 2020-05-16T12:06:37Z

`squad_metric.compute` is giving following error
```
ArrowTypeError: Could not convert [{'text': 'Denver Broncos'}, {'text': 'Denver Broncos'}, {'text': 'Denver Broncos'}] with type list: was not a dict, tuple, or recognized null value for conversion to struct type
```

This is how my predictions and references look like
```
predictions[0]
# {'id': '56be4db0acb8001400a502ec', 'prediction_text': 'Denver Broncos'}
```

```
references[0]
# {'answers': [{'text': 'Denver Broncos'},
  {'text': 'Denver Broncos'},
  {'text': 'Denver Broncos'}],
 'id': '56be4db0acb8001400a502ec'}
```

These are structured as per the `squad_metric.compute` help string.
GitHub issue #142: [WMT] Add all wmt
State: closed | Created: 2020-05-16T11:28:46Z

This PR adds all wmt datasets scripts. At the moment the script is **not** functional for the language pairs "cs-en", "ru-en", "hi-en" because apparently it takes up to a week to get the manual data for these datasets: see http://ufal.mff.cuni.cz/czeng. 

The datasets are fully functional though for the "big" language pairs "de-en" and "fr-en". 

Overall I think the scripts are very messy and might need a big refactoring at some point.

For now I think there are good to merge (most dataset configs can be used). I will add "cs", "ru" and "hi" when the manual data is available.
GitHub issue #140: [Tests] run local tests as default
State: closed | Created: 2020-05-16T10:56:06Z

This PR also enables local tests by default

I think it's safer for now to enable both local and aws tests for every commit. The problem currently is that when we do a PR to add a dataset, the dataset is not yet on AWS on therefore not tested on the PR itself. Thus the PR will always be green even if the datasets are not correct. This PR aims at fixing this.

## Suggestion on how to commit to the repo from now on:
Now since the repo is "online", I think we should adopt a couple of best practices:
1) - No direct committing to the repo anymore. Every change should be opened in a PR and be well documented so that we can find it later
2) - Every PR has to be reviewed by at least x people (I guess @thomwolf you should decide here) because we now have to be much more careful when doing changes to the API for backward compatibility, etc...
GitHub issue #138: Consider renaming to nld
State: closed | Created: 2020-05-15T20:23:27Z

Hey :)

Just making a thread here recording what I said on Twitter, as it's impossible to follow discussion there. It's also just really not a good way to talk about this sort of thing.

The issue is that modules go into the global namespace, so you shouldn't use variable names that conflict with module names. This means the package makes `nlp` a bad variable name everywhere in the codebase. I've always used `nlp` as the canonical variable name of spaCy's `Language` objects, and this is a convention that a lot of other code has followed (Stanza, flair, etc). And actually, your `transformers` library uses `nlp` as the name for its `Pipeline` instance in your readme.

If you stick with the `nlp` name for this package, if anyone uses it then they should rewrite all of that code. If `nlp` is a bad choice of variable anywhere, it's a bad choice of variable everywhere --- because you shouldn't have to notice whether some other function uses a module when you're naming variables within a function. You want to have one convention that you can stick to everywhere.

If people use your `nlp` package and continue to use the `nlp` variable name, they'll find themselves with confusing bugs. There will be many many bits of code cut-and-paste from tutorials that give confusing results when combined with the data loading from the `nlp` library. The problem will be especially bad for shadowed modules (people might reasonably have a module named `nlp.py` within their codebase) and notebooks, as people might run notebook cells for data loading out-of-order.

I don't think it's an exaggeration to say that if your library becomes popular, we'll all be answering issues around this about once a week for the next few years. That seems pretty unideal, so I do hope you'll reconsider.

I suggest `nld` as a better name. It more accurately represents what the package actually does. It's pretty unideal to have a package named `nlp` that doesn't do any processing, and contains data about natural language generation or other non-NLP tasks. The name is equally short, and is sort of a visual pun on `nlp`, since a d is a rotated p.
GitHub issue #133: [Question] Using/adding a local dataset
State: closed | Created: 2020-05-15T16:26:06Z

Users may want to either create/modify a local copy of a dataset, or use a custom-built dataset with the same `Dataset` API as externally downloaded datasets.

It appears to be possible to point to a local dataset path rather than downloading the external ones, but I'm not exactly sure how to go about doing this.

A notebook/example script demonstrating this would be very helpful.
GitHub issue #132: [Feature Request] Add the OpenWebText dataset
State: closed | Created: 2020-05-15T15:57:29Z

The OpenWebText dataset is an open clone of OpenAI's WebText dataset. It can be used to train ELECTRA as is specified in the [README](https://www.github.com/google-research/electra).

More information and the download link are available [here](https://skylion007.github.io/OpenWebTextCorpus/).
GitHub issue #130: Loading GLUE dataset loads CoLA by default
State: closed | Created: 2020-05-15T14:55:50Z

If I run:

```python
dataset = nlp.load_dataset('glue')
```
The resultant dataset seems to be CoLA be default, without throwing any error. This is in contrast to calling:

```python
metric = nlp.load_metric("glue")
```
which throws an error telling the user that they need to specify a task in GLUE. Should the same apply for loading datasets?
GitHub issue #126: remove webis
State: closed | Created: 2020-05-15T11:25:20Z

Remove webis from dataset folder.

Our first dataset script that only lives on AWS :-) https://s3.console.aws.amazon.com/s3/buckets/datasets.huggingface.co/nlp/datasets/webis/tl_dr/?region=us-east-1 @julien-c @jplu
GitHub issue #123: [Tests] Local => aws
State: closed | Created: 2020-05-15T09:12:25Z

## Change default Test from local => aws

As a default we set` aws=True`, `Local=False`, `slow=False`

### 1. RUN_AWS=1 (default)
This runs 4 tests per dataset script.

a) Does the dataset script have a valid etag / Can it be reached on AWS? 
b) Can we load its `builder_class`?
c) Can we load **all** dataset configs?
d) _Most importantly_: Can we load the dataset? 

Important - we currently only test the first config of each dataset to reduce test time. Total test time is around 1min20s.

### 2. RUN_LOCAL=1 RUN_AWS=0

***This should be done when debugging dataset scripts of the ./datasets folder***

This only runs 1 test per dataset test, which is equivalent to aws d) - Can we load the dataset from the local `datasets` directory?

### 3. RUN_SLOW=1

We should set up to run these tests maybe 1 time per week ? @thomwolf 

The `slow` tests include two more important tests. 

e) Can we load the dataset with all possible configs? This test will probably fail at the moment because a lot of dummy data is missing. We should add the dummy data step by step to be sure that all configs work.

f) Test that the actual dataset can be loaded. This will take quite some time to run, but is important to make sure that the "real" data can be loaded. It will also test whether the dataset script has the correct checksums file which is currently not tested with `aws=True`. @lhoestq - is there an easy way to check cheaply whether the `dataset_info.json` is correct for each dataset script?
GitHub issue #120: 🐛 `map` not working
State: closed | Created: 2020-05-15T06:43:08Z

I'm trying to run a basic example (mapping function to add a prefix).  
[Here is the colab notebook I'm using.](https://colab.research.google.com/drive/1YH4JCAy0R1MMSc-k_Vlik_s1LEzP_t1h?usp=sharing)

```python
import nlp

dataset = nlp.load_dataset('squad', split='validation[:10%]')

def test(sample):
    sample['title'] = "test prefix @@@ " + sample["title"]
    return sample

print(dataset[0]['title'])
dataset.map(test)
print(dataset[0]['title'])
```
Output :
> Super_Bowl_50
Super_Bowl_50

Expected output :
> Super_Bowl_50
test prefix @@@ Super_Bowl_50
GitHub issue #119: 🐛 Colab : type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'
State: closed | Created: 2020-05-15T02:27:26Z

I'm trying to load CNN/DM dataset on Colab.

[Colab notebook](https://colab.research.google.com/drive/11Mf7iNhIyt6GpgA1dBEtg3cyMHmMhtZS?usp=sharing)

But I meet this error :

> AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'
GitHub issue #118: ❓ How to apply a map to all subsets ?
State: closed | Created: 2020-05-15T01:58:52Z

I'm working with CNN/DM dataset, where I have 3 subsets : `train`, `test`, `validation`.

Should I apply my map function on the subsets one by one ?

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
for corpus in ['train', 'test', 'validation']:
         cnn_dm[corpus] = cnn_dm[corpus].map(my_func)
```

Or is there a better way to do this ?
GitHub issue #117: ❓ How to remove specific rows of a dataset ?
State: closed | Created: 2020-05-15T01:25:06Z

I saw on the [example notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb#scrollTo=efFhDWhlvSVC) how to remove a specific column :

```python
dataset.drop('id')
```

But I didn't find how to remove a specific row. 

**For example, how can I remove all sample with `id` < 10 ?**
GitHub issue #116: 🐛 Trying to use ROUGE metric : pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323
State: closed | Created: 2020-05-15T01:12:06Z

I'm trying to use rouge metric.

I have to files : `test.pred.tokenized` and `test.gold.tokenized` with each line containing a sentence.  
I tried :

```python
import nlp

rouge = nlp.load_metric('rouge')
with open("test.pred.tokenized") as p, open("test.gold.tokenized") as g:
    for lp, lg in zip(p, g):
            rouge.add(lp, lg)
```

But I meet following error :

> pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323

---

Full stack-trace :

```
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/me/.venv/transformers/lib/python3.6/site-packages/nlp/metric.py", line 224, in add
    self.writer.write_batch(batch)
  File "/home/me/.venv/transformers/lib/python3.6/site-packages/nlp/arrow_writer.py", line 148, in write_batch
    pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
  File "pyarrow/table.pxi", line 1550, in pyarrow.lib.Table.from_pydict
  File "pyarrow/table.pxi", line 1503, in pyarrow.lib.Table.from_arrays
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323
```

(`nlp` installed from source)
GitHub issue #115: AttributeError: 'dict' object has no attribute 'info'
State: closed | Created: 2020-05-15T00:29:47Z

I'm trying to access the information of CNN/DM dataset :

```python
cnn_dm = nlp.load_dataset('cnn_dailymail')
print(cnn_dm.info)
```

returns :

> AttributeError: 'dict' object has no attribute 'info'
GitHub issue #114: Couldn't reach CNN/DM dataset
State: closed | Created: 2020-05-15T00:16:17Z

I can't get CNN / DailyMail dataset.

```python
import nlp

assert "cnn_dailymail" in [dataset.id for dataset in nlp.list_datasets()]
cnn_dm = nlp.load_dataset('cnn_dailymail')
```

[Colab notebook](https://colab.research.google.com/drive/1zQ3bYAVzm1h0mw0yWPqKAg_4EUlSx5Ex?usp=sharing)

gives following error :

```
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/cnn_dailymail/cnn_dailymail.py
```
GitHub issue #103: [Manual downloads] add logic proposal for manual downloads and add wikihow
State: closed | Created: 2020-05-14T13:30:36Z

Wikihow is an example that needs to manually download two files as stated in: https://github.com/mahnazkoupaee/WikiHow-Dataset. 

The user can then store these files under a hard-coded name: `wikihowAll.csv` and `wikihowSep.csv` in this case in a directory of his choice, e.g. `~/wikihow/manual_dir`.

The dataset can then be loaded via:

```python
import nlp
nlp.load_dataset("wikihow", data_dir="~/wikihow/manual_dir")
```

I added/changed so that there are explicit error messages when using manually downloaded files.
GitHub issue #100: Add per type scores in seqeval metric
State: closed | Created: 2020-05-14T09:37:52Z

This PR add a bit more detail in the seqeval metric. Now the usage and output are:

```python
import nlp
met = nlp.load_metric('metrics/seqeval')
references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
predictions =  [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
met.compute(predictions, references)

#Output: {'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.8}
```

It is also possible to compute scores for non IOB notations, POS tagging for example hasn't this kind of notation. Add `suffix` parameter:

```python
import nlp
met = nlp.load_metric('metrics/seqeval')
references = [['O', 'O', 'O', 'MISC', 'MISC', 'MISC', 'O'], ['PER', 'PER', 'O']]
predictions =  [['O', 'O', 'MISC', 'MISC', 'MISC', 'MISC', 'O'], ['PER', 'PER', 'O']]
met.compute(predictions, references, metrics_kwargs={"suffix": True})

#Output: {'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.9}
```
GitHub issue #95: Replace checksums files by Dataset infos json
State: closed | Created: 2020-05-13T19:36:16Z

### Better verifications when loading a dataset

I replaced the `urls_checksums` directory that used to contain `checksums.txt` and `cached_sizes.txt`, by a single file `dataset_infos.json`. It's just a dict `config_name` -> `DatasetInfo`.

It simplifies and improves how verifications of checksums and splits sizes are done, as they're all stored in `DatasetInfo` (one per config). Also, having already access to `DatasetInfo` enables to check disk space before running `download_and_prepare` for a given config.

The dataset infos json file is user readable, you can take a look at the squad one that I generated in this PR.

### Renaming

According to these changes, I did some renaming:
`save_checksums` -> `save_infos`
`ignore_checksums` -> `ignore_verifications`

for example, when you are creating a dataset you have to run
```nlp-cli test path/to/my/dataset --save_infos --all_configs```
instead of
```nlp-cli test path/to/my/dataset --save_checksums --all_configs```

### And now, the fun part

We'll have to rerun the `nlp-cli test ... --save_infos --all_configs` for all the datasets

-----------------

feedback appreciated !
GitHub issue #89: Add list and inspect methods - cleanup hf_api
State: closed | Created: 2020-05-13T09:30:15Z

Add a bunch of methods to easily list and inspect the processing scripts up-loaded on S3:
```python
nlp.list_datasets()
nlp.list_metrics()
# Copy and prepare the scripts at `local_path` for easy inspection/modification.
nlp.inspect_dataset(path, local_path) 
# Copy and prepare the scripts at `local_path` for easy inspection/modification.
nlp.inspect_metric(path, local_path) 
```

Also clean up the `HfAPI` to use `dataclasses` for better user-experience
GitHub issue #82: [Datasets] add ted_hrlr
State: closed | Created: 2020-05-12T16:46:50Z

@thomwolf - After looking at `xnli` I think it's better to leave the translation features and add a `translation` key to make them work in our framework. 

The result looks like this:
![Screenshot from 2020-05-12 18-34-43](https://user-images.githubusercontent.com/23423619/81721933-ee1faf00-9480-11ea-9e95-d6557cbd0ce0.png)

you can see that each split has a `translation` key which value is the nlp.features.Translation object. 

That's a simple change. If it's ok for you, I will add dummy data for the other configs and treat the other translation scripts in the same way.
GitHub issue #80: Add nbytes + nexamples check
State: closed | Created: 2020-05-12T16:18:43Z

### Save size and number of examples
Now when you do `save_checksums`, it also create `cached_sizes.txt` right next to the checksum file.
This new file stores the bytes sizes and the number of examples of each split that has been prepared and stored in the cache. Example:

```
# Cached sizes: <full_config_name> <num_bytes> <num_examples>
hansards/house/1.0.0/test 22906629 122290
hansards/house/1.0.0/train 191459584 947969
hansards/senate/1.0.0/test 5711686 25553
hansards/senate/1.0.0/train 40324278 182135
```

### Check processing output 

If there is a `caches_sizes.txt`, then each time we run `download_and_prepare` it will make sure that the sizes match. You can set `ignore_checksums=True` if you don't want that to happen.

### Fill Dataset Info

All the split infos and the checksums are now stored correctly in DatasetInfo after `download_and_prepare`

### Check space on disk before running `download_and_prepare`

Check if the space is lower than the sum of the sizes of the files in `checksums.txt` and `cached_files.txt`. This is not ideal though as it considers the files for all configs.

TODO:
A better way to do it would be to have save the `DatasetInfo` instead of the `checksums.txt` and `cached_sizes.txt`, in order to have one file per dataset config (and therefore consider only the sizes of the files for one config and not all of them). It can also be the occasion to factorize all the `download_and_prepare` verifications. Maybe next PR ?
GitHub issue #75: WIP adding metrics
State: closed | Created: 2020-05-12T09:52:00Z

Adding the following metrics as identified by @mariamabarham:

1. BLEU:  BiLingual Evaluation Understudy: https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py,  https://github.com/chakki-works/sumeval/blob/master/sumeval/metrics/bleu.py (multilingual)
2. GLEU:  Google-BLEU:  https://github.com/cnap/gec-ranking/blob/master/scripts/compute_gleu
3. Sacrebleu: https://pypi.org/project/sacrebleu/1.4.8/ (pypi package), https://github.com/mjpost/sacrebleu (github implementation)
4. ROUGE: Recall-Oriented Understudy for Gisting Evaluation: https://github.com/google-research/google-research/tree/master/rouge, https://github.com/chakki-works/sumeval/blob/master/sumeval/metrics/rouge.py (multilingual)
5. Seqeval: https://github.com/chakki-works/seqeval (github implementation), https://pypi.org/project/seqeval/0.0.12/ (pypi package)
6. Coval:  coreference evaluation package for the CoNLL and ARRAU datasets https://github.com/ns-moosavi/coval
7. SQuAD v1 evaluation script
8. SQuAD V2 evaluation script: https://worksheets.codalab.org/rest/bundles/0x6b567e1cf2e041ec80d7098f031c5c9e/contents/blob/
9. GLUE
10. XNLI


Not now:
1. Perplexity: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/perplexity.py
2. Spearman: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/spearman_correlation.py
3. F1_measure: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/f1_measure.py
4. Pearson_corelation: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/pearson_correlation.py
5. AUC: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/auc.py 
6. Entropy: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/entropy.py
GitHub issue #74: fix overflow check
State: closed | Created: 2020-05-12T09:38:01Z

I did some tests and unfortunately the test
```
pa_array.nbytes > MAX_BATCH_BYTES
```
doesn't work. Indeed for a StructArray, `nbytes` can be less 2GB even if there is an overflow (it loops...).

I don't think we can do a proper overflow test for the limit of 2GB...

For now I replaced it with a sanity check on the first element.
GitHub issue #72: [README dummy data tests] README to better understand how the dummy data structure works
State: closed | Created: 2020-05-11T22:19:03Z

In this PR a README.md is added to tests to shine more light on how the dummy data structure works. I try to explain the different possible cases. IMO the best way to understand the logic is to checkout the dummy data structure of the different datasets I mention in the README.md since those are the "edge cases". 

@mariamabarham @thomwolf @lhoestq @jplu - I'd be happy to checkout the dummy data structure and get some feedback on possible improvements.
GitHub issue #71: Fix arrow writer for big datasets using writer_batch_size
State: closed | Created: 2020-05-11T14:45:36Z

This PR fixes Yacine's bug.
According to [this](https://github.com/apache/arrow/blob/master/docs/source/cpp/arrays.rst#size-limitations-and-recommendations), it is not recommended to have pyarrow arrays bigger than 2Go.

Therefore I set a default batch size of 100 000 examples per batch. In general it shouldn't exceed 2Go. If it does, I reduce the batch_size on the fly, and I notify the user with a warning.
GitHub issue #68: [CSV] re-add csv
State: closed | Created: 2020-05-08T17:38:29Z

Re-adding csv under the datasets under construction to keep circle ci happy - will have to see how to include it in the tests.

@lhoestq noticed that I accidently deleted it in https://github.com/huggingface/nlp/pull/63#discussion_r422263729.
GitHub issue #67: [Tests] Test files locally
State: closed | Created: 2020-05-08T15:02:43Z

This PR adds a `aws` and a `local` decorator to the tests so that tests now run on the local datasets. 

By default, the `aws` is deactivated and `local` is activated and `slow` is deactivated, so that only 1 test per dataset runs on circle ci. 

**When local is activated all folders in `./datasets` are tested.**

**Important** When adding a dataset, we should no longer upload it to AWS. The steps are:
1. Open a PR
2. Add a dataset as described in `datasets/README.md`
3. If all tests pass, push to master

Currently we have 49 functional datasets in our code base. 

We have 6 datasets "under-construction" that don't pass the tests - so I put them in a folder "datasets_under_construction" - it would be nice to open a PR to fix them and put them in the `datasets` folder.

**Important** when running tests locally, the datasets are cached so to rerun them delete your local cache via:
`rm -r ~/.cache/huggingface/datasets/*` 

@thomwolf @mariamabarham @lhoestq
GitHub issue #63: [Dataset scripts] add all datasets scripts
State: closed | Created: 2020-05-08T10:50:15Z

As mentioned, we can have the canonical datasets in the master. For now I also want to include all the data as present on S3 to make the synchronization easier when uploading new datastes. 

@mariamabarham @lhoestq @thomwolf - what do you think? 

If this is ok for you, I can sync up the master with the `add_dataset` branch: https://github.com/huggingface/nlp/pull/37 so that master is up to date.
GitHub issue #60: Update to simplify some datasets conversion
State: closed | Created: 2020-05-07T22:02:24Z

This PR updates the encoding of `Values` like `integers`, `boolean` and `float` to use python casting and avoid having to cast in the dataset scripts, as mentioned here: https://github.com/huggingface/nlp/pull/37#discussion_r420176626

We could also change (not included in this PR yet):
- `supervized_keys` to make them a NamedTuple instead of a dataclass, and
- handle specifically the `Translation` features.
as mentioned here: https://github.com/huggingface/nlp/pull/37#discussion_r421740236

@patrickvonplaten @mariamabarham tell me if you want these two last changes as well.
GitHub issue #57: Better cached path
State: closed | Created: 2020-05-07T18:36:00Z

### Changes:
- The `cached_path` no longer returns None if the file is missing/the url doesn't work. Instead, it can raise `FileNotFoundError` (missing file), `ConnectionError` (no cache and unreachable url) or `ValueError` (parsing error)
- Fix requests to firebase API that doesn't handle HEAD requests...
- Allow custom download in datasets script: it allows to use `tf.io.gfile.copy` for example, to download from google storage. I added an example: the `boolq` script
GitHub issue #55: Beam datasets
State: closed | Created: 2020-05-07T11:04:32Z

# Beam datasets

## Intro

Beam Datasets are using beam pipelines for preprocessing (basically lots of `.map` over objects called PCollections).
The advantage of apache beam is that you can choose which type of runner you want to use to preprocess your data. The main runners are:
- the `DirectRunner` to run the pipeline locally (default). However I encountered memory issues for big datasets (like the french or english wikipedia). Small dataset work fine
- Google Dataflow. I didn't play with it.
- Spark or Flink, two well known data processing frameworks. I tried to use the Spark/Flink local runners provided by apache beam for python and wasn't able to make them work properly though...

## From tfds beam datasets to our own beam datasets

Tensorflow datasets used beam and a complicated pipeline to shard the TFRecords files.
To allow users to download beam datasets and not having to preprocess them, they also allow to download the already preprocessed datasets from their google storage (the beam pipeline doesn't run in that case).

On our side, we replace TFRecords by something else. Arrow or Parquet do the job but I chose Parquet as: 1) there is a builtin apache beam parquet writer that is quite convenient, and 2) reading parquet from the pyarrow library is also simple and effective (there is a mmap option !)

Moreover we don't shard datasets in many many files like tfds (they were doing probably doing that mainly because of the limit of 2Gb per TFRecord file). Therefore we have a simpler pipeline that saves each split into one parquet file. We also removed the utilities to use their google storage (for now maybe ? we'll have to discuss it).

## Main changes

- Added a BeamWriter to save the output of beam pipelines into parquet files and fill dataset infos
- Create a ParquetReader and refactor a bit the arrow_reader.py

\> **With this, we can now try to add beam datasets from tfds**

I already added the wikipedia one, and I will also try to add the Wiki40b dataset

## Test the wikipedia script

You can download and run the beam pipeline for wikipedia (using the `DirectRunner` by default) like this:

```
>>> import nlp
>>> nlp.load("datasets/nlp/wikipedia", dataset_config="20200501.frr")
```

This wikipedia dataset (lang: frr, North Frisian) is a small one (~10Mb), but feel free to try bigger ones (and fill 20Gb of swap memory if you try the english one lol)

## Next

Should we allow to download preprocessed datasets from the tfds google storage ?
Should we try to optimize the beam pipelines to run locally without memory issues ?
Should we try other data processing frameworks for big datasets, like spark ?


## About this PR

It should be merged after #25 

-----------------

I'd be happy to have your feedback and your ideas to improve the processing of big datasets like wikipedia :)
GitHub issue #51: [Testing] Improved testing structure
State: closed | Created: 2020-05-06T12:03:07Z

This PR refactors the test design a bit and puts the mock download manager in the `utils` files as it is just a test helper class.

as @mariamabarham pointed out, creating a dummy folder structure can be quite hard to grasp.
This PR tries to change that to some extent.

It follows the following logic for the `dummy` folder structure now:
1.) The data bulider has no config -> the  `dummy` folder structure is:
`dummy/<version>/dummy_data.zip`
2) The data builder has >= 1 configs -> the `dummy` folder structure is: 
`dummy/<config_name_1>/<version>/dummy_data.zip`
`dummy/<config_name_2>/<version>/dummy_data.zip`

Now, the difficult part is how to create the `dummy_data.zip` file. There are two cases:
A) The `data_urs` parameter inserted into the `download_and_extract` fn is a **string**:
-> the `dummy_data.zip` file zips the folder: 
`dummy_data/<relative_path_of_folder_structure_of_url>`
B) The `data_urs` parameter inserted into the `download_and_extract` fn is a **dict**:
-> the `dummy_data.zip` file zips the folder: 
`dummy_data/<relative_path_of_folder_structure_of_url_behind _key_1>`
`dummy_data/<relative_path_of_folder_structure_of_url_behind _key_2>`

By relative folder structure I mean `url_path.split('./')[-1]`. As an example the dataset **xquad** by deepmind has the following url path behind the key `de`: `https://github.com/deepmind/xquad/blob/master/xquad.de.json` 
-> This means that the relative url path should be `xquad.de.json`.


@mariamabarham B) is a change from how is was before and I think is makes more sense. 
While before the `dummy_data.zip` file for xquad with config `de` looked like:
`dummy_data/de` it would now look like `dummy_data/xquad.de.json`. I think this is better and easier to understand. 

Therefore there are currently 6 tests that would have to have changed their dummy folder structure, but which can easily be done (30min). 

I also added a function: `print_dummy_data_folder_structure` that prints out the expected structures when testing which should be quite helpful.
GitHub issue #38: [Checksums] Error for some datasets
State: closed | Created: 2020-05-04T08:00:16Z

The checksums command works very nicely for `squad`. But for `crime_and_punish` and `xnli`, 
the same bug happens:

When running: 
```
python nlp-cli nlp-cli test xnli --save_checksums
```

leads to:

```
  File "nlp-cli", line 33, in <module>
    service.run()
  File "/home/patrick/python_bin/nlp/commands/test.py", line 61, in run
    ignore_checksums=self._ignore_checksums,
  File "/home/patrick/python_bin/nlp/builder.py", line 383, in download_and_prepare
    self._download_and_prepare(dl_manager=dl_manager, download_config=download_config)
  File "/home/patrick/python_bin/nlp/builder.py", line 627, in _download_and_prepare
    dl_manager=dl_manager, max_examples_per_split=download_config.max_examples_per_split,
  File "/home/patrick/python_bin/nlp/builder.py", line 431, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/patrick/python_bin/nlp/datasets/xnli/8bf4185a2da1ef2a523186dd660d9adcf0946189e7fa5942ea31c63c07b68a7f/xnli.py", line 95, in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
  File "/home/patrick/python_bin/nlp/utils/download_manager.py", line 246, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/patrick/python_bin/nlp/utils/download_manager.py", line 186, in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
  File "/home/patrick/python_bin/nlp/utils/download_manager.py", line 166, in _record_sizes_checksums
    self._recorded_sizes_checksums[url] = get_size_checksum(path)
  File "/home/patrick/python_bin/nlp/utils/checksums_utils.py", line 81, in get_size_checksum
    with open(path, "rb") as f:
TypeError: expected str, bytes or os.PathLike object, not tuple
```
GitHub issue #34: [Tests] add slow tests
State: closed | Created: 2020-05-03T11:01:22Z

This PR adds a slow test that downloads the "real" dataset. The test is decorated as "slow" so that it will not automatically run on circle ci.

Before uploading a dataset, one should test that this test passes, manually by running 

```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_<your-dataset-script-name>
```

This PR should be merged after PR: #33
GitHub issue #32: Fix map caching notebooks
State: closed | Created: 2020-05-01T11:55:26Z

Previously, caching results with `.map()` didn't work in notebooks.
To reuse a result, `.map()` serializes the functions with `dill.dumps` and then it hashes it.

The problem is that when using `dill.dumps` to serialize a function, it also saves its origin (filename + line no.) and the origin of all the `globals` this function needs. However for notebooks and shells, the filename looks like \<ipython-input-13-9ed2afe61d25\> and the line no. changes often.

To fix the problem, I added a new dispatch function for code objects that ignore the origin of the code if it comes from a notebook or a python shell.

I tested these cases in a notebook:
- lambda functions
- named functions
- methods
- classmethods
- staticmethods
- classes that implement `__call__`

The caching now works as expected for all of them :)
I also tested the caching in the demo notebook and it works fine !
GitHub issue #31: [Circle ci] Install a virtual env before running tests
State: closed | Created: 2020-05-01T10:11:17Z

Install a virtual env before running tests to not running into sudo issues when dynamically downloading files. 

Same number of tests now pass / fail as on my local computer: 
![Screenshot from 2020-05-01 12-14-44](https://user-images.githubusercontent.com/23423619/80798814-8a0a0a80-8ba5-11ea-8db8-599d33bbfccd.png)