jamesLeeeeeee/datasets-github-issues huggingface dataset:
[Tests] refactor tests
Author: patrickvonplaten | Pull request: yes | State: closed

This PR separates AWS and Local tests to remove these ugly statements in the script:
```python
        if "/" not in dataset_name:
            logging.info("Skip {} because it is a canonical dataset")
            return
```

To run a `aws` test, one should now run the following command: 

```python 
pytest -s tests/test_dataset_common.py::AWSDatasetTest::test_builder_class_wmt14
```

The same `local` test, can be run with:
```python 
pytest -s tests/test_dataset_common.py::LocalDatasetTest::test_builder_class_wmt14
```
ANLI
Author: douwekiela | Pull request: no | State: closed

Can I recommend the following:

For ANLI, use https://github.com/facebookresearch/anli. As that paper says, "Our dataset is not
to be confused with abductive NLI (Bhagavatula et al., 2019), which calls itself αNLI, or ART.". 

Indeed, the paper cited under what is currently called anli says in the abstract "We introduce a challenge dataset, ART".

The current naming will confuse people :)
[Feature request] Add cos-e v1.0
Author: sarahwie | Pull request: no | State: closed

I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).
Discussion on version identifier & MockDataLoaderManager for test data
Author: EntilZha | Pull request: no | State: open

Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done.
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
Author: saahiluppal | Pull request: no | State: closed

I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a
SyntaxError with WMT datasets
Author: tomhosking | Pull request: no | State: closed

The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook
Meta-datasets (GLUE/XTREME/...) – Special care to attributions and citations
Author: thomwolf | Pull request: no | State: open

Meta-datasets are interesting in terms of standardized benchmarks but they also have specific behaviors, in particular in terms of attribution and authorship. It's very important that each specific dataset inside a meta dataset is properly referenced and the citation/specific homepage/etc are very visible and accessible and not only the generic citation of the meta-dataset itself.

Let's take GLUE as an example:

The configuration has the citation for each dataset included (e.g. [here](https://github.com/huggingface/nlp/blob/master/datasets/glue/glue.py#L154-L161)) but it should be copied inside the dataset info so that, when people access `dataset.info.citation` they get both the citation for GLUE and the citation for the specific datasets inside GLUE that they have loaded.
Add WNUT 17 NER dataset
Author: stefan-it | Pull request: yes | State: closed

Hi,

this PR adds the WNUT 17 dataset to `nlp`.

> Emerging and Rare entity recognition
> This shared task focuses on identifying unusual, previously-unseen entities in the context of emerging discussions. Named entities form the basis of many modern approaches to other tasks (like event clustering and summarisation), but recall on them is a real problem in noisy text - even among annotators. This drop tends to be due to novel entities and surface forms. Take for example the tweet “so.. kktny in 30 mins?” - even human experts find entity kktny hard to detect and resolve. This task will evaluate the ability to detect and classify novel, emerging, singleton named entities in noisy text.
> 
> The goal of this task is to provide a definition of emerging and of rare entities, and based on that, also datasets for detecting these entities.

More information about the dataset can be found on the [shared task page](https://noisy-text.github.io/2017/emerging-rare-entities.html).

Dataset is taken is taken from their [GitHub repository](https://github.com/leondz/emerging_entities_17), because the data provided in this repository contains minor fixes in the dataset format.

## Usage

Then the WNUT 17 dataset can be used in `nlp` like this:

```python
import nlp

wnut_17 = nlp.load_dataset("./datasets/wnut_17/wnut_17.py")

print(wnut_17)
```

This outputs:

```txt
'train': Dataset(schema: {'id': 'string', 'tokens': 'list<item: string>', 'labels': 'list<item: string>'}, num_rows: 3394)
'validation': Dataset(schema: {'id': 'string', 'tokens': 'list<item: string>', 'labels': 'list<item: string>'}, num_rows: 1009)
'test': Dataset(schema: {'id': 'string', 'tokens': 'list<item: string>', 'labels': 'list<item: string>'}, num_rows: 1287)
```

Number are identical with the ones in [this paper](https://www.ijcai.org/Proceedings/2019/0702.pdf) and are the same as using the `dataset` reader in Flair.

## Features

The following feature format is used to represent a sentence in the WNUT 17 dataset:

| Feature | Example | Description
| ---- | ---- | -----------------
| `id` | `0` | Number (id) of current sentence
| `tokens` | `["AHFA", "extends", "deadline"]` | List of tokens (strings) for a sentence
| `labels` | `["B-group", "O", "O"]` | List of labels (outer span)

The following labels are used in WNUT 17:

```txt
O
B-corporation
I-corporation
B-location
I-location
B-product
I-product
B-person
I-person
B-group
I-group
B-creative-work
I-creative-work
```
_download_and_prepare() got an unexpected keyword argument 'verify_infos'
Author: richarddwang | Pull request: no | State: closed

# Reproduce
In Colab,
```
%pip install -q  nlp
%pip install -q apache_beam mwparserfromhell

dataset = nlp.load_dataset('wikipedia')
```
get
```
Downloading and preparing dataset wikipedia/20200501.aa (download: Unknown size, generated: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/wikipedia/20200501.aa/1.0.0...

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-6-52471d2a0088> in <module>()
----> 1 dataset = nlp.load_dataset('wikipedia')

1 frames

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519 

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

TypeError: _download_and_prepare() got an unexpected keyword argument 'verify_infos'
```
Error with sklearn train_test_split
Author: ClonedOne | Pull request: no | State: closed

It would be nice if we could use sklearn `train_test_split` to quickly generate subsets from the dataset objects returned by `nlp.load_dataset`. At the moment the code:

```python
data = nlp.load_dataset('imdb', cache_dir=data_cache)
f_half, s_half = train_test_split(data['train'], test_size=0.5, random_state=seed)
```
throws:
```
ValueError: Can only get row(s) (int or slice) or columns (string).
```
It's not a big deal, since there are other ways to split the data, but it would be a cool thing to have.
Add BERTScore to metrics
Author: felixgwu | Pull request: yes | State: closed

This PR adds [BERTScore](https://arxiv.org/abs/1904.09675) to metrics.
Here is an example of how to use it.

```sh
import nlp
bertscore = nlp.load_metric('metrics/bertscore') # or simply nlp.load_metric('bertscore') after this is added to huggingface's s3 bucket
predictions = ['example', 'fruit']
references = [['this is an example.', 'this is one example.'], ['apple']]
results = bertscore.compute(predictions, references, lang='en')
print(results)
```
[AWS tests] AWS test should not run for canonical datasets
Author: patrickvonplaten | Pull request: yes | State: closed

AWS tests should in general not run for canonical datasets. Only local tests will run in this case. This way a PR is able to pass when adding a new dataset.

This PR changes to logic to the following: 

1) All datasets that are present in `nlp/datasets` are tested only locally. This way when one adds a canonical dataset, the PR includes his dataset in the tests.

2) All datasets that are only present on AWS, such as `webis/tl_dr` atm are tested only on AWS. 

I think the testing structure might need a bigger refactoring and better documentation very soon. 

Merging for now to unblock new PRs @thomwolf @mariamabarham .
ArrowTypeError in squad metrics
Author: patil-suraj | Pull request: no | State: closed

`squad_metric.compute` is giving following error
```
ArrowTypeError: Could not convert [{'text': 'Denver Broncos'}, {'text': 'Denver Broncos'}, {'text': 'Denver Broncos'}] with type list: was not a dict, tuple, or recognized null value for conversion to struct type
```

This is how my predictions and references look like
```
predictions[0]
# {'id': '56be4db0acb8001400a502ec', 'prediction_text': 'Denver Broncos'}
```

```
references[0]
# {'answers': [{'text': 'Denver Broncos'},
  {'text': 'Denver Broncos'},
  {'text': 'Denver Broncos'}],
 'id': '56be4db0acb8001400a502ec'}
```

These are structured as per the `squad_metric.compute` help string.
[WMT] Add all wmt
Author: patrickvonplaten | Pull request: yes | State: closed

This PR adds all wmt datasets scripts. At the moment the script is **not** functional for the language pairs "cs-en", "ru-en", "hi-en" because apparently it takes up to a week to get the manual data for these datasets: see http://ufal.mff.cuni.cz/czeng. 

The datasets are fully functional though for the "big" language pairs "de-en" and "fr-en". 

Overall I think the scripts are very messy and might need a big refactoring at some point.

For now I think there are good to merge (most dataset configs can be used). I will add "cs", "ru" and "hi" when the manual data is available.
[Tests] run local tests as default
Author: patrickvonplaten | Pull request: yes | State: closed

This PR also enables local tests by default

I think it's safer for now to enable both local and aws tests for every commit. The problem currently is that when we do a PR to add a dataset, the dataset is not yet on AWS on therefore not tested on the PR itself. Thus the PR will always be green even if the datasets are not correct. This PR aims at fixing this.

## Suggestion on how to commit to the repo from now on:
Now since the repo is "online", I think we should adopt a couple of best practices:
1) - No direct committing to the repo anymore. Every change should be opened in a PR and be well documented so that we can find it later
2) - Every PR has to be reviewed by at least x people (I guess @thomwolf you should decide here) because we now have to be much more careful when doing changes to the API for backward compatibility, etc...
Consider renaming to nld
Author: honnibal | Pull request: no | State: closed

Hey :)

Just making a thread here recording what I said on Twitter, as it's impossible to follow discussion there. It's also just really not a good way to talk about this sort of thing.

The issue is that modules go into the global namespace, so you shouldn't use variable names that conflict with module names. This means the package makes `nlp` a bad variable name everywhere in the codebase. I've always used `nlp` as the canonical variable name of spaCy's `Language` objects, and this is a convention that a lot of other code has followed (Stanza, flair, etc). And actually, your `transformers` library uses `nlp` as the name for its `Pipeline` instance in your readme.

If you stick with the `nlp` name for this package, if anyone uses it then they should rewrite all of that code. If `nlp` is a bad choice of variable anywhere, it's a bad choice of variable everywhere --- because you shouldn't have to notice whether some other function uses a module when you're naming variables within a function. You want to have one convention that you can stick to everywhere.

If people use your `nlp` package and continue to use the `nlp` variable name, they'll find themselves with confusing bugs. There will be many many bits of code cut-and-paste from tutorials that give confusing results when combined with the data loading from the `nlp` library. The problem will be especially bad for shadowed modules (people might reasonably have a module named `nlp.py` within their codebase) and notebooks, as people might run notebook cells for data loading out-of-order.

I don't think it's an exaggeration to say that if your library becomes popular, we'll all be answering issues around this about once a week for the next few years. That seems pretty unideal, so I do hope you'll reconsider.

I suggest `nld` as a better name. It more accurately represents what the package actually does. It's pretty unideal to have a package named `nlp` that doesn't do any processing, and contains data about natural language generation or other non-NLP tasks. The name is equally short, and is sort of a visual pun on `nlp`, since a d is a rotated p.
[Question] Using/adding a local dataset
Author: zphang | Pull request: no | State: closed

Users may want to either create/modify a local copy of a dataset, or use a custom-built dataset with the same `Dataset` API as externally downloaded datasets.

It appears to be possible to point to a local dataset path rather than downloading the external ones, but I'm not exactly sure how to go about doing this.

A notebook/example script demonstrating this would be very helpful.
[Feature Request] Add the OpenWebText dataset
Author: LysandreJik | Pull request: no | State: closed

The OpenWebText dataset is an open clone of OpenAI's WebText dataset. It can be used to train ELECTRA as is specified in the [README](https://www.github.com/google-research/electra).

More information and the download link are available [here](https://skylion007.github.io/OpenWebTextCorpus/).
Loading GLUE dataset loads CoLA by default
Author: zphang | Pull request: no | State: closed

If I run:

```python
dataset = nlp.load_dataset('glue')
```
The resultant dataset seems to be CoLA be default, without throwing any error. This is in contrast to calling:

```python
metric = nlp.load_metric("glue")
```
which throws an error telling the user that they need to specify a task in GLUE. Should the same apply for loading datasets?
[Tests] Local => aws
Author: patrickvonplaten | Pull request: yes | State: closed

## Change default Test from local => aws

As a default we set` aws=True`, `Local=False`, `slow=False`

### 1. RUN_AWS=1 (default)
This runs 4 tests per dataset script.

a) Does the dataset script have a valid etag / Can it be reached on AWS? 
b) Can we load its `builder_class`?
c) Can we load **all** dataset configs?
d) _Most importantly_: Can we load the dataset? 

Important - we currently only test the first config of each dataset to reduce test time. Total test time is around 1min20s.

### 2. RUN_LOCAL=1 RUN_AWS=0

***This should be done when debugging dataset scripts of the ./datasets folder***

This only runs 1 test per dataset test, which is equivalent to aws d) - Can we load the dataset from the local `datasets` directory?

### 3. RUN_SLOW=1

We should set up to run these tests maybe 1 time per week ? @thomwolf 

The `slow` tests include two more important tests. 

e) Can we load the dataset with all possible configs? This test will probably fail at the moment because a lot of dummy data is missing. We should add the dummy data step by step to be sure that all configs work.

f) Test that the actual dataset can be loaded. This will take quite some time to run, but is important to make sure that the "real" data can be loaded. It will also test whether the dataset script has the correct checksums file which is currently not tested with `aws=True`. @lhoestq - is there an easy way to check cheaply whether the `dataset_info.json` is correct for each dataset script?
🐛 `map` not working
Author: astariul | Pull request: no | State: closed

I'm trying to run a basic example (mapping function to add a prefix).  
[Here is the colab notebook I'm using.](https://colab.research.google.com/drive/1YH4JCAy0R1MMSc-k_Vlik_s1LEzP_t1h?usp=sharing)

```python
import nlp

dataset = nlp.load_dataset('squad', split='validation[:10%]')

def test(sample):
    sample['title'] = "test prefix @@@ " + sample["title"]
    return sample

print(dataset[0]['title'])
dataset.map(test)
print(dataset[0]['title'])
```
Output :
> Super_Bowl_50
Super_Bowl_50

Expected output :
> Super_Bowl_50
test prefix @@@ Super_Bowl_50
🐛 Colab : type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'
Author: astariul | Pull request: no | State: closed

I'm trying to load CNN/DM dataset on Colab.

[Colab notebook](https://colab.research.google.com/drive/11Mf7iNhIyt6GpgA1dBEtg3cyMHmMhtZS?usp=sharing)

But I meet this error :

> AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'
❓ How to apply a map to all subsets ?
Author: astariul | Pull request: no | State: closed

I'm working with CNN/DM dataset, where I have 3 subsets : `train`, `test`, `validation`.

Should I apply my map function on the subsets one by one ?

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
for corpus in ['train', 'test', 'validation']:
         cnn_dm[corpus] = cnn_dm[corpus].map(my_func)
```

Or is there a better way to do this ?
❓ How to remove specific rows of a dataset ?
Author: astariul | Pull request: no | State: closed

I saw on the [example notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb#scrollTo=efFhDWhlvSVC) how to remove a specific column :

```python
dataset.drop('id')
```

But I didn't find how to remove a specific row. 

**For example, how can I remove all sample with `id` < 10 ?**
🐛 Trying to use ROUGE metric : pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323
Author: astariul | Pull request: no | State: closed

I'm trying to use rouge metric.

I have to files : `test.pred.tokenized` and `test.gold.tokenized` with each line containing a sentence.  
I tried :

```python
import nlp

rouge = nlp.load_metric('rouge')
with open("test.pred.tokenized") as p, open("test.gold.tokenized") as g:
    for lp, lg in zip(p, g):
            rouge.add(lp, lg)
```

But I meet following error :

> pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323

---

Full stack-trace :

```
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/me/.venv/transformers/lib/python3.6/site-packages/nlp/metric.py", line 224, in add
    self.writer.write_batch(batch)
  File "/home/me/.venv/transformers/lib/python3.6/site-packages/nlp/arrow_writer.py", line 148, in write_batch
    pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
  File "pyarrow/table.pxi", line 1550, in pyarrow.lib.Table.from_pydict
  File "pyarrow/table.pxi", line 1503, in pyarrow.lib.Table.from_arrays
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323
```

(`nlp` installed from source)
AttributeError: 'dict' object has no attribute 'info'
Author: astariul | Pull request: no | State: closed

I'm trying to access the information of CNN/DM dataset :

```python
cnn_dm = nlp.load_dataset('cnn_dailymail')
print(cnn_dm.info)
```

returns :

> AttributeError: 'dict' object has no attribute 'info'
Couldn't reach CNN/DM dataset
Author: astariul | Pull request: no | State: closed

I can't get CNN / DailyMail dataset.

```python
import nlp

assert "cnn_dailymail" in [dataset.id for dataset in nlp.list_datasets()]
cnn_dm = nlp.load_dataset('cnn_dailymail')
```

[Colab notebook](https://colab.research.google.com/drive/1zQ3bYAVzm1h0mw0yWPqKAg_4EUlSx5Ex?usp=sharing)

gives following error :

```
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/cnn_dailymail/cnn_dailymail.py
```
[Manual downloads] add logic proposal for manual downloads and add wikihow
Author: patrickvonplaten | Pull request: yes | State: closed

Wikihow is an example that needs to manually download two files as stated in: https://github.com/mahnazkoupaee/WikiHow-Dataset. 

The user can then store these files under a hard-coded name: `wikihowAll.csv` and `wikihowSep.csv` in this case in a directory of his choice, e.g. `~/wikihow/manual_dir`.

The dataset can then be loaded via:

```python
import nlp
nlp.load_dataset("wikihow", data_dir="~/wikihow/manual_dir")
```

I added/changed so that there are explicit error messages when using manually downloaded files.