kardwalker/my_dataset huggingface dataset:
GitHub issue #320: Blog Authorship Corpus, Non Matching Splits Sizes Error, nlp viewer
State: closed | Created: 2020-06-29T07:36:35Z

Selecting `blog_authorship_corpus` in the nlp viewer throws the following error: 

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=610252351, num_examples=532812, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='train', num_bytes=614706451, num_examples=535568, dataset_name='blog_authorship_corpus')}, {'expected': SplitInfo(name='validation', num_bytes=37500394, num_examples=31277, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='validation', num_bytes=32553710, num_examples=28521, dataset_name='blog_authorship_corpus')}]
Traceback:
File "/home/sasha/streamlit/lib/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 172, in <module>
    dts, fail = get(str(option.id), str(conf_option.name) if conf_option else None)
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 132, in get
    builder_instance.download_and_prepare()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
```
@srush @lhoestq
GitHub issue #319: Nested sequences with dicts
State: closed | Created: 2020-06-27T23:45:17Z

Am pretty much finished [adding a dataset](https://github.com/ghomasHudson/nlp/blob/DocRED/datasets/docred/docred.py) for [DocRED](https://github.com/thunlp/DocRED), but am getting an error when trying to add a nested `nlp.features.sequence(nlp.features.sequence({key:value,...}))`. 

The original data is in this format:
```python
{
  'title': "Title of wiki page",
  'vertexSet': [
                  [
                    { 'name': "mention_name", 
                      'sent_id': "mention in which sentence", 
                      'pos': ["postion of mention in a sentence"], 
                      'type': "NER_type"},
                    {another mention}
                  ], 
                  [another entity]
                ]
    ...
}
```
So to represent this I've attempted to write:
```
...
features=nlp.Features({
    "title": nlp.Value("string"),
    "vertexSet": nlp.features.Sequence(nlp.features.Sequence({
        "name": nlp.Value("string"),
        "sent_id": nlp.Value("int32"),
        "pos": nlp.features.Sequence(nlp.Value("int32")),
        "type": nlp.Value("string"),
    })),
    ...
    }),
...
```
This is giving me the error:
```
pyarrow.lib.ArrowTypeError: Could not convert [{'pos': [[0,2], [2,4], [3,5]], "type": ["ORG", "ORG", "ORG"], "name": ["Lark Force", "Lark Force", "Lark Force", "sent_id": [0, 3, 4]}..... with type list: was not a dict, tuple, or recognized null value for conversion to struct type
```
Do we expect the pyarrow stuff to break when doing this deeper nesting? I've checked that it still works when you do `nlp.features.Sequence(nlp.features.Sequence(nlp.Value("string"))` or `nlp.features.Sequence({key:value,...})` just not nested sequences with a dict.

If it's not possible, I can always convert it to a shallower structure. I'd rather not change the DocRED authors' structure if I don't have to though.
GitHub issue #318: Multitask
State: closed | Created: 2020-06-27T13:27:29Z

Following our discussion in #217, I've implemented a first working version of `MultiDataset`.

There's a function `build_multitask()` which takes either individual `nlp.Dataset`s or `dicts` of splits and constructs `MultiDataset`(s). I've added a notebook with example usage.

I've implemented many of the `nlp.Dataset` methods (cache_files, columns, nbytes, num_columns, num_rows, column_names, schema, shape). Some of the other methods are complicated as they change the number of examples. These raise `NotImplementedError`s at the moment.

This will need some tests which I haven't written yet.

There's definitely room for improvements but I think the general approach is sound.
GitHub issue #317: Adding a dataset with multiple subtasks
State: closed | Created: 2020-06-26T23:14:19Z

I intent to add the datasets of the MT Quality Estimation shared tasks to `nlp`. However, they have different subtasks -- such as word-level, sentence-level and document-level quality estimation, each of which having different language pairs, and some of the data reused in different subtasks.

For example, in [QE 2019,](http://www.statmt.org/wmt19/qe-task.html) we had the same English-Russian and English-German data for word-level and sentence-level QE. 

I suppose these datasets could have both their word and sentence-level labels inside `nlp.Features`; but what about other subtasks? Should they be considered a different dataset altogether?

I read the discussion on #217 but the case of QE seems a lot simpler.
GitHub issue #315: [Question] Best way to batch a large dataset?
State: open | Created: 2020-06-25T22:30:20Z

I'm training on large datasets such as Wikipedia and BookCorpus. Following the instructions in [the tutorial notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb), I see the following recommended for TensorFlow:

```python
train_tf_dataset = train_tf_dataset.filter(remove_none_values, load_from_cache_file=False)
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'start_positions', 'end_positions']
train_tf_dataset.set_format(type='tensorflow', columns=columns)
features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]} 
labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
### Question about this last line ###
tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)
```

This code works for something like WikiText-2. However, scaling up to WikiText-103, the last line takes 5-10 minutes to run. I assume it is because tf.data.Dataset.from_tensor_slices() is pulling everything into memory, not lazily loading. This approach won't scale up to datasets 25x larger such as Wikipedia.

So I tried manual batching using `dataset.select()`:

```python
idxs = np.random.randint(len(dataset), size=bsz)
batch = dataset.select(idxs).map(lambda example: {"input_ids": tokenizer(example["text"])})
tf_batch = tf.constant(batch["ids"], dtype=tf.int64)
```

This appears to create a new Apache Arrow dataset with every batch I grab, and then tries to cache it. The runtime of `dataset.select([0, 1])` appears to be much worse than `dataset[:2]`. So using `select()` doesn't seem to be performant enough for a training loop.

Is there a performant scalable way to lazily load batches of nlp Datasets?
GitHub issue #313: Add MWSC
State: closed | Created: 2020-06-25T09:22:02Z

Adding the [Modified Winograd Schema Challenge](https://github.com/salesforce/decaNLP/blob/master/local_data/schema.txt) dataset which formed part of the [decaNLP](http://decanlp.com/) benchmark. Not sure how much use people would find for it it outside of the benchmark, but it is general purpose.

Code is heavily borrowed from the [decaNLP repo](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L773-L877).

There's a few (possibly overly opinionated) design choices I made:

- I used the train/test/dev split [buried in the decaNLP code](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L852-L855)
- I split out each example into the 2 alternatives. Originally the data uses the format:
    ```
    The city councilmen refused the demonstrators a permit because they [feared/advocated] violence. 
    Who [feared/advocated] violence? 
    councilmen/demonstrators
    ```
    I split into the 2 variants:
    ```
    The city councilmen refused the demonstrators a permit because they feared violence. 
    Who feared violence? 
    councilmen/demonstrators
    
    The city councilmen refused the demonstrators a permit because they advocated violence. 
    Who advocated violence? 
    councilmen/demonstrators
    ```
    I can't see any use for having the options combined into a single example (splitting them is [the way decaNLP processes](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L846-L850)) them. You can't train on both versions with them combined, and splitting the examples later would be a pain to do. I think [winogrande.py](https://github.com/huggingface/nlp/blob/master/datasets/winogrande/winogrande.py) presents the data in this way?

- I've not used the decaNLP framing (appending the options to the question e.g. `Who feared violence? 
 -- councilmen or demonstrators?`) but left it more generic by adding the options as a new key: `"options":["councilmen","demonstrators"]` This should be an easy thing to change using `map` if needed by a specific application.

Dataset is working as-is but if anyone has any thoughts/preferences on the design decisions here I'm definitely open to different choices.
GitHub issue #312: [Feature request] Add `shard()` method to dataset
State: closed | Created: 2020-06-24T22:48:33Z

Currently, to shard a dataset into 10 pieces on different ranks, you can run

```python
rank = 3 # for example
size = 10
dataset = nlp.load_dataset('wikitext', 'wikitext-2-raw-v1', split=f"train[{rank*10}%:{(rank+1)*10}%]")
```

However, this breaks down if you have a number of ranks that doesn't divide cleanly into 100, such as 64 ranks. Is there interest in adding a method shard() that looks like this?

```python
rank = 3
size = 64
dataset = nlp.load_dataset("wikitext", "wikitext-2-raw-v1", split="train").shard(rank=rank, size=size)
```

TensorFlow has a similar API: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard. I'd be happy to contribute this code.
GitHub issue #311: Add qa_zre
State: closed | Created: 2020-06-24T22:17:22Z

Adding the QA-ZRE dataset from ["Zero-Shot Relation Extraction via Reading Comprehension"](http://nlp.cs.washington.edu/zeroshot/).

A common processing step seems to be replacing the `XXX` placeholder with the `subject`. I've left this out as it's something you could easily do with `map`.
GitHub issue #310: add wikisql
State: closed | Created: 2020-06-24T18:00:35Z

Adding the [WikiSQL](https://github.com/salesforce/WikiSQL) dataset.

Interesting things to note:
- Have copied the function (`_convert_to_human_readable`) which converts the SQL query to a human-readable (string) format as this is what most people will want when actually using this dataset for NLP applications.
- `conds` was originally a tuple but is converted to a dictionary to support differing types.

Would be nice to add the logical_form metrics too at some point.
GitHub issue #305: Importing downloaded package repository fails
State: closed | Created: 2020-06-23T21:09:05Z

The `get_imports` function in `src/nlp/load.py` has a feature to download a package as a zip archive of the github repository and import functions from the unpacked directory. This is used for example in the `metrics/coval.py` file, and would be useful to add BLEURT (@ankparikh).

Currently however, the code seems to have trouble with imports within the package. For example:
```
import nlp
coval = nlp.load_metric('coval')
```
yields:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yacine/Code/nlp/src/nlp/load.py", line 432, in load_metric
    metric_cls = import_main_class(module_path, dataset=False)
  File "/home/yacine/Code/nlp/src/nlp/load.py", line 57, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/yacine/anaconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/yacine/Code/nlp/src/nlp/metrics/coval/a78807df33ac45edbb71799caf2b3b47e55df4fd690267808fe963a5e8b30952/coval.py", line 21, in <module>
    from .coval_backend.conll import reader  # From: https://github.com/ns-moosavi/coval
  File "/home/yacine/Code/nlp/src/nlp/metrics/coval/a78807df33ac45edbb71799caf2b3b47e55df4fd690267808fe963a5e8b30952/coval_backend/conll/reader.py", line 2, in <module>
    from conll import mention
ModuleNotFoundError: No module named 'conll'
```

Not sure what the fix would be there.
GitHub issue #304: Problem while printing doc string when instantiating multiple metrics.
State: closed | Created: 2020-06-23T19:32:05Z

When I load more than one metric and try to print doc string of a particular metric,. It shows the doc strings of all imported metric one after the other which looks quite confusing and clumsy.
Attached [Colab](https://colab.research.google.com/drive/13H0ZgyQ2se0mqJ2yyew0bNEgJuHaJ8H3?usp=sharing) Notebook for problem clarification..
GitHub issue #302: Question - Sign Language Datasets
State: closed | Created: 2020-06-23T14:53:40Z

An emerging field in NLP is SLP - sign language processing.

I was wondering about adding datasets here, specifically because it's shaping up to be large and easily usable.
The metrics for sign language to text translation are the same.

So, what do you think about (me, or others) adding datasets here?


An example dataset would be [RWTH-PHOENIX-Weather 2014 T](https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/)
For every item in the dataset, the data object includes:
1. video_path - path to mp4 file
2. pose_path - a path to `.pose` file with human pose landmarks
3. openpose_path - a path to a `.json` file with human pose landmarks
4. gloss - string
5. text - string
6. video_metadata - height, width, frames, framerate


------

To make it a tad more complicated - what if sign language libraries add requirements to `nlp`? for example, sign language is commonly annotated using `ilex`, `eaf`, or `srt` files, which are all loadable as text, but there is no reason for the dataset to parse that file by itself, if libraries exist to do so.
GitHub issue #301: Setting cache_dir gives error on wikipedia download
State: closed | Created: 2020-06-23T11:31:44Z

First of all thank you for a super handy library! I'd like to download large files to a specific drive so I set `cache_dir=my_path`. This works fine with e.g. imdb and squad. But on wikipedia I get an error:
```
nlp.load_dataset('wikipedia', '20200501.de', split = 'train', cache_dir=my_path)
```
```
OSError                                   Traceback (most recent call last)
<ipython-input-2-23551344d7bc> in <module>
      1 import nlp
----> 2 nlp.load_dataset('wikipedia', '20200501.de', split = 'train', cache_dir=path)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526 

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    385                     with utils.temporary_assignment(self, "_cache_dir", tmp_data_dir):
    386                         reader = ArrowReader(self._cache_dir, self.info)
--> 387                         reader.download_from_hf_gcs(self._cache_dir, self._relative_data_dir(with_version=True))
    388                         downloaded_info = DatasetInfo.from_directory(self._cache_dir)
    389                         self.info.update(downloaded_info)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/arrow_reader.py in download_from_hf_gcs(self, cache_dir, relative_data_dir)
    231             remote_dataset_info = os.path.join(remote_cache_dir, "dataset_info.json")
    232             downloaded_dataset_info = cached_path(remote_dataset_info)
--> 233             os.rename(downloaded_dataset_info, os.path.join(cache_dir, "dataset_info.json"))
    234             if self._info is not None:
    235                 self._info.update(self._info.from_directory(cache_dir))

OSError: [Errno 18] Invalid cross-device link: '/home/local/NTU/nn/.cache/huggingface/datasets/025fa4fd4f04aaafc9e939260fbc8f0bb190ce14c61310c8ae1ddd1dcb31f88c.9637f367b6711a79ca478be55fe6989b8aea4941b7ef7adc67b89ff403020947' -> '/data/nn/nlp/wikipedia/20200501.de/1.0.0.incomplete/dataset_info.json'
```
GitHub issue #300: Fix bertscore references
State: closed | Created: 2020-06-23T09:38:59Z

I added some type checking for metrics. There was an issue where a metric could interpret a string a a list. A `ValueError` is raised if a string is given instead of a list.

Moreover I added support for both strings and lists of strings for `references` in `bertscore`, as it is the case in the original code.

Both ways work:
```
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, [lg])
score = scorer.compute(lang="en")
```

```
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, lg)
score = scorer.compute(lang="en")
```

This should fix #295 and #238
GitHub issue #297: Error in Demo for Specific Datasets
State: closed | Created: 2020-06-23T00:38:42Z

Selecting `natural_questions` or `newsroom` dataset in the online demo results in an error similar to the following.

![image](https://user-images.githubusercontent.com/60150701/85347842-ac861900-b4ae-11ea-98c4-a53a00934783.png)
GitHub issue #296: snli -1 labels
State: closed | Created: 2020-06-22T23:33:30Z

I'm trying to train a model on the SNLI dataset. Why does it have so many -1 labels?
```
import nlp
from collections import Counter
data = nlp.load_dataset('snli')['train']
print(Counter(data['label']))
Counter({0: 183416, 2: 183187, 1: 182764, -1: 785})
```
GitHub issue #295: Improve input warning for evaluation metrics
State: closed | Created: 2020-06-22T17:28:57Z

Hi, 

I am the author of `bert_score`. Recently, we received [ an issue ](https://github.com/Tiiiger/bert_score/issues/62) reporting a problem in using `bert_score` from the `nlp` package (also see #238 in this repo).  After looking into this, I realized that the problem arises from the format `nlp.Metric` takes input. 

Here is a minimal example:
```python
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, lg)
score = scorer.compute(lang="en")
```

The problem in the above code is that `scorer.add()` expects a list of strings as input for the references. As a result, the `scorer` here would take a list of characters in `lg` to be the references. The correct implementation would be calling
```python
scorer.add(lp, [lg])
```

I just want to raise this issue to you to prevent future user errors of a similar kind. I assume some simple type checking can prevent this from happening?

Thanks!
GitHub issue #290: ConnectionError - Eli5 dataset download
State: closed | Created: 2020-06-19T13:40:33Z

Hi, I have a problem with downloading Eli5 dataset. When typing `nlp.load_dataset('eli5')`, I get ConnectionError: Couldn't reach https://storage.googleapis.com/huggingface-nlp/cache/datasets/eli5/LFQA_reddit/1.0.0/explain_like_im_five-train_eli5.arrow

I would appreciate if you could help me with this issue.
GitHub issue #289: update xsum
State: closed | Created: 2020-06-19T12:28:32Z

This PR makes the following update to the xsum dataset:

- Manual download is not required anymore

- dataset can be loaded as follow: `nlp.load_dataset('xsum')`


**Important** 
Instead of using on outdated url to download the data:  "https://raw.githubusercontent.com/EdinburghNLP/XSum/master/XSum-Dataset/XSum-TRAINING-DEV-TEST-SPLIT-90-5-5.json" 

a more up-to-date url stored here: https://s3.amazonaws.com/datasets.huggingface.co/summarization/xsum.tar.gz is used
, so that the user does not need to manually download the data anymore. 
There might be slight breaking changes here for xsum.
GitHub issue #284: Fix manual download instructions
State: closed | Created: 2020-06-18T15:59:57Z

This PR replaces the static `DatasetBulider` variable `MANUAL_DOWNLOAD_INSTRUCTIONS` by a property function `manual_download_instructions()`. 

Some datasets like XTREME and all WMT need the manual data dir only for a small fraction of the possible configs.

After some brainstorming with @mariamabarham and @lhoestq, we came to the conclusion that having a property function `manual_download_instructions()` gives us more flexibility to decide on a per config basis in the dataset builder if manual download instructions are needed.

Also this PR should unblock solves a bug with `wmt16 - ro-en` 
@sshleifer from this branch you should be able to succesfully run

```python 
import nlp 
ds = nlp.load_dataset('./datasets/wmt16', 'ro-en')
```

and once this PR is merged S3 should be synched so that 

```python
import nlp
ds = nlp.load_dataset("wmt16", "ro-en")
```

works as well.

**Important**: Since `MANUAL_DOWNLOAD_INSTRUCTIONS` was not really exposed to the user, this PR should not be a problem regarding backward compatibility.
GitHub issue #283: Consistent formatting of citations
State: closed | Created: 2020-06-18T14:48:45Z

The citations are all of a different format, some have "```" and have text inside, others are proper bibtex. 

Can we make it so that they all are proper citations, i.e. parse by the bibtex spec:

https://bibtexparser.readthedocs.io/en/master/
GitHub issue #282: Update dataset_info from gcs
State: closed | Created: 2020-06-18T13:41:15Z

Some datasets are hosted on gcs (wikipedia for example). In this PR I make sure that, when a user loads such datasets, the file_instructions are built using the dataset_info.json from gcs and not from the info extracted from the local `dataset_infos.json` (the one that contain the info for each config). Indeed local files may end up outdated.

Furthermore, to avoid outdated dataset_infos.json, I now make sure that each time you run `load_dataset` it also tries to update the file locally.
GitHub issue #281: Private/sensitive data
State: closed | Created: 2020-06-18T09:47:27Z

Hi all,
Thanks for this fantastic library, it makes it very easy to do prototyping for NLP projects interchangeably between TF/Pytorch. 

Unfortunately, there is data that cannot easily be shared publicly as it may contain sensitive information. 
Is there support/a plan to support such data with NLP, e.g. by reading it from local sources?

Use case flow could look like this: use NLP to prototype an approach on similar, public data and apply the resulting prototype on sensitive/private data without the need to rethink data processing pipelines. 

Many thanks for your responses ahead of time and kind regards,
MFreidank
GitHub issue #280: Error with SquadV2 Metrics
State: closed | Created: 2020-06-17T19:10:54Z

I can't seem to import squad v2 metrics. 

**squad_metric = nlp.load_metric('squad_v2')**

**This throws me an error.:**


```
ImportError                               Traceback (most recent call last)
<ipython-input-8-170b6a170555> in <module>
----> 1 squad_metric = nlp.load_metric('squad_v2')

~/env/lib64/python3.6/site-packages/nlp/load.py in load_metric(path, name, process_id, num_process, data_dir, experiment_id, in_memory, download_config, **metric_init_kwargs)
    426     """
    427     module_path = prepare_module(path, download_config=download_config, dataset=False)
--> 428     metric_cls = import_main_class(module_path, dataset=False)
    429     metric = metric_cls(
    430         name=name,

~/env/lib64/python3.6/site-packages/nlp/load.py in import_main_class(module_path, dataset)
     55     """
     56     importlib.invalidate_caches()
---> 57     module = importlib.import_module(module_path)
     58 
     59     if dataset:

/usr/lib64/python3.6/importlib/__init__.py in import_module(name, package)
    124                 break
    125             level += 1
--> 126     return _bootstrap._gcd_import(name[level:], package, level)
    127 
    128 

/usr/lib64/python3.6/importlib/_bootstrap.py in _gcd_import(name, package, level)

/usr/lib64/python3.6/importlib/_bootstrap.py in _find_and_load(name, import_)

/usr/lib64/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/usr/lib64/python3.6/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib64/python3.6/importlib/_bootstrap_external.py in exec_module(self, module)

/usr/lib64/python3.6/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

~/env/lib64/python3.6/site-packages/nlp/metrics/squad_v2/a15e787c76889174874386d3def75321f0284c11730d2a57e28fe1352c9b5c7a/squad_v2.py in <module>
     16 
     17 import nlp
---> 18 from .evaluate import evaluate
     19 
     20 _CITATION = """\

ImportError: cannot import name 'evaluate'
```
GitHub issue #279: Dataset Preprocessing Cache with .map() function not working as expected
State: closed | Created: 2020-06-17T17:17:21Z

I've been having issues with reproducibility when loading and processing datasets with the `.map` function. I was only able to resolve them by clearing all of the cache files on my system. 

Is there a way to disable using the cache when processing a dataset? As I make minor processing changes on the same dataset, I want to be able to be certain the data is being re-processed rather than loaded from a cached file. 

Could you also help me understand a bit more about how the caching functionality is used for pre-processing? E.g. how is it determined when to load from a cache vs. reprocess. 
I was particularly having an issue where the correct dataset splits were loaded, but as soon as I applied the `.map()` function to each split independently, they somehow all exited this process having been converted to the test set.
Thanks!
GitHub issue #278: MemoryError when loading German Wikipedia
State: closed | Created: 2020-06-17T15:06:21Z

Hi, first off let me say thank you for all the awesome work you're doing at Hugging Face across all your projects (NLP, Transformers, Tokenizers) - they're all amazing contributions to us working with NLP models :)

I'm trying to download the German Wikipedia dataset as follows:

```
wiki = nlp.load_dataset("wikipedia", "20200501.de", split="train")
```

However, when I do so, I get the following error:

```
Downloading and preparing dataset wikipedia/20200501.de (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/ubuntu/.cache/huggingface/datasets/wikipedia/20200501.de/1.0.0...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/load.py", line 520, in load_dataset
    save_infos=save_infos,
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/builder.py", line 433, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/builder.py", line 824, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
	`load_dataset('wikipedia', '20200501.de', beam_runner='DirectRunner')`
```

So, following on from the example usage at the bottom, I tried specifying `beam_runner='DirectRunner`, however when I do this after about 20 min after the data has all downloaded, I get a `MemoryError` as warned.

This isn't an issue for the English or French Wikipedia datasets (I've tried both), as neither seem to require that `beam_runner` be specified. Can you please clarify why this is an issue for the German dataset?

My nlp version is 0.2.1.

Thank you!
GitHub issue #277: Empty samples in glue/qqp
State: closed | Created: 2020-06-17T05:54:52Z

```
qqp = nlp.load_dataset('glue', 'qqp')
print(qqp['train'][310121])
print(qqp['train'][362225])
```
```
{'question1': 'How can I create an Android app?', 'question2': '', 'label': 0, 'idx': 310137}
{'question1': 'How can I develop android app?', 'question2': '', 'label': 0, 'idx': 362246}
```
Notice that question 2 is empty string. 
BTW, I have checked and these two are the only naughty ones in all splits of qqp.
GitHub issue #276: Fix metric compute (original_instructions missing)
State: closed | Created: 2020-06-16T08:52:01Z

When loading arrow data we added in cc8d250 a way to specify the instructions that were used to store them with the loaded dataset.
However metrics load data the same way but don't need instructions (we use one single file).

In this PR I just make `original_instructions` optional when reading files to load a `Dataset` object.

This should fix #269
GitHub issue #275: NonMatchingChecksumError when loading pubmed dataset
State: closed | Created: 2020-06-16T07:31:51Z

I get this error when i run `nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')`.
The error is:

```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-2-7742dea167d0> in <module>()
----> 1 df = nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')
      2 df = pd.DataFrame(df)
      3 gc.collect()

3 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522 

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    431                 verify_infos = not save_infos and not ignore_verifications
    432                 self._download_and_prepare(
--> 433                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    434                 )
    435                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    468         # Checksums verification
    469         if verify_infos:
--> 470             verify_checksums(self.info.download_checksums, dl_manager.get_recorded_sizes_checksums())
    471         for split_generator in split_generators:
    472             if str(split_generator.split_info.name).lower() == "all":

/usr/local/lib/python3.6/dist-packages/nlp/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums)
     34     bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
     35     if len(bad_urls) > 0:
---> 36         raise NonMatchingChecksumError(str(bad_urls))
     37     logger.info("All the checksums matched successfully.")
     38 

NonMatchingChecksumError: ['https://drive.google.com/uc?id=1b3rmCSIoh6VhD4HKWjI4HOW-cSwcwbeC&export=download', 'https://drive.google.com/uc?id=1lvsqvsFi3W-pE1SqNZI0s8NR9rC1tsja&export=download']
```
I'm currently working on google colab.

That is quite strange because yesterday it was fine.
GitHub issue #271: Fix allociné dataset configuration
State: closed | Created: 2020-06-13T10:12:10Z

This is a patch for #244. According to the [live nlp viewer](url), the Allociné dataset must be loaded with :
```python
dataset = load_dataset('allocine', 'allocine')
```
This is redundant, as there is only one "dataset configuration", and should only be:
```python
dataset = load_dataset('allocine')
```

This is my mistake, because the code for [`allocine.py`](https://github.com/huggingface/nlp/blob/master/datasets/allocine/allocine.py) was inspired by [`imdb.py`](https://github.com/huggingface/nlp/blob/master/datasets/imdb/imdb.py), which also force the user to specify the "dataset configuration" (even if there is only one).

I believe this PR should solve this issue, making the Allociné dataset more convenient to use.
GitHub issue #270: c4 dataset is not viewable in nlpviewer demo
State: closed | Created: 2020-06-13T08:26:16Z

I get the following error when I try to view the c4 dataset in [nlpviewer](https://huggingface.co/nlp/viewer/)

```python
ModuleNotFoundError: No module named 'langdetect'
Traceback:
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp_viewer/run.py", line 54, in <module>
    configs = get_confs(option.id)
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp_viewer/run.py", line 48, in get_confs
    builder_cls = nlp.load.import_main_class(module_path, dataset=True)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/load.py", line 57, in import_main_class
    module = importlib.import_module(module_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/datasets/c4/88bb1b1435edad3fb772325710c4a43327cbf4a23b9030094556e6f01e14ec19/c4.py", line 29, in <module>
    from .c4_utils import (
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/datasets/c4/88bb1b1435edad3fb772325710c4a43327cbf4a23b9030094556e6f01e14ec19/c4_utils.py", line 29, in <module>
    import langdetect
```
GitHub issue #269: Error in metric.compute: missing `original_instructions` argument
State: closed | Created: 2020-06-13T06:26:54Z

I'm running into an error using metrics for computation in the latest master as well as version 0.2.1. Here is a minimal example:

```python
import nlp
rte_metric = nlp.load_metric('glue', name="rte")
rte_metric.compute(
    [0, 0, 1, 1],
    [0, 1, 0, 1],
)
```

```
    181             # Read the predictions and references
    182             reader = ArrowReader(path=self.data_dir, info=None)
--> 183             self.data = reader.read_files(node_files)
    184 
    185             # Release all of our locks

TypeError: read_files() missing 1 required positional argument: 'original_instructions'
```

I believe this might have been introduced with cc8d2508b75f7ba0e5438d0686ee02dcec43c7f4, which added the `original_instructions` argument. Elsewhere, an empty-string default is provided--perhaps that could be done here too?
GitHub issue #267: How can I load/find WMT en-romanian?
State: closed | Created: 2020-06-12T01:09:37Z

I believe it is from `wmt16`

When I run

```python
wmt = nlp.load_dataset('wmt16')
```
I get:
```python
AssertionError: The dataset wmt16 with config cs-en requires manual data. 
 Please follow the manual download instructions:   Some of the wmt configs here, require a manual download.
  Please look into wmt.py to see the exact path (and file name) that has to
  be downloaded.
  . 
 Manual data can be loaded with `nlp.load(wmt16, data_dir='<path/to/manual/data>')
```
There is no wmt.py,as the error message suggests, and wmt16.py doesn't have manual download instructions.

Any idea how to do this?

Thanks in advance!
GitHub issue #266: Add sort, shuffle, test_train_split and select methods
State: closed | Created: 2020-06-11T16:22:20Z

Add a bunch of methods to reorder/split/select rows in a dataset:
- `dataset.select(indices)`: Create a new dataset with rows selected following the list/array of indices (which can have a different size than the dataset and contain duplicated indices, the only constrain is that all the integers in the list must be smaller than the dataset size, otherwise we're indexing outside the dataset...)
- `dataset.sort(column_name)`: sort a dataset according to a column (has to be a column with a numpy compatible type)
- `dataset.shuffle(seed)`: shuffle a dataset rows
- `dataset.train_test_split(test_size, train_size)`: Return a dictionary with two random train and test subsets (`train` and `test` ``Dataset`` splits)

All these methods are **not** in-place which means they return new ``Dataset``.
This is the default behavior in the library.

Fix #147 #166 #259
GitHub issue #265: Add pyarrow warning colab
State: closed | Created: 2020-06-11T15:57:51Z

When a user installs `nlp` on google colab, then google colab doesn't update pyarrow, and the runtime needs to be restarted to use the updated version of pyarrow.

This is an issue because `nlp` requires the updated version to work correctly.

In this PR I added en error that is shown to the user in google colab if the user tries to `import nlp` without having restarted the runtime. The error tells the user to restart the runtime.
GitHub issue #264: Fix small issues creating dataset
State: closed | Created: 2020-06-11T15:20:16Z

Fix many small issues mentioned in #249:
- don't force to install apache beam for commands
- fix None cache dir when using `dl_manager.download_custom`
- added new extras in `setup.py` named `dev` that contains tests and quality dependencies
- mock dataset sizes when running tests with dummy data
- add a note about the naming convention of datasets (camel case - snake case) in CONTRIBUTING.md

This should help users create their datasets.
Next step is the `add_dataset.md` docs :)
GitHub issue #262: Add new dataset ANLI Round 1
State: closed | Created: 2020-06-11T04:14:57Z

Adding new dataset [ANLI](https://github.com/facebookresearch/anli/).

I'm not familiar with how to add new dataset. Let me know if there is any issue. I only include round 1 data here. There will be round 2, round 3 and more in the future with potentially different format. I think it will be better to separate them.
GitHub issue #259: documentation missing how to split a dataset
State: closed | Created: 2020-06-10T13:18:13Z

I am trying to understand how to split a dataset ( as arrow_dataset). 
I know I can do something like this to access a split which is already in the original dataset : 

`ds_test = nlp.load_dataset('imdb, split='test') `

But how can I split ds_test into a test and a validation set (without reading the data into memory and keeping the arrow_dataset as container)?
I guess it has something to do with the module split :-) but there is no real documentation in the code but only a reference to a longer description: 

> See the  [guide on splits](https://github.com/huggingface/nlp/tree/master/docs/splits.md)  for more information.

But the guide seems to be missing.

To clarify: I know that this has been modelled after the dataset of tensorflow and that some of the documentation there can be used [like this one](https://www.tensorflow.org/datasets/splits). But to come back to the example above: I cannot simply split the testset doing this: 
`ds_test = nlp.load_dataset('imdb, split='test'[:5000]) `
`ds_val = nlp.load_dataset('imdb, split='test'[5000:])`

because the imdb test data is sorted by class (probably not a good idea anyway)
GitHub issue #258: Why is dataset after tokenization far more larger than the orginal one ?
State: closed | Created: 2020-06-10T01:27:07Z

I tokenize wiki dataset by `map` and cache the results.
```
def tokenize_tfm(example):
    example['input_ids'] = hf_fast_tokenizer.convert_tokens_to_ids(hf_fast_tokenizer.tokenize(example['text']))
    return example
wiki = nlp.load_dataset('wikipedia', '20200501.en', cache_dir=cache_dir)['train']
wiki.map(tokenize_tfm, cache_file_name=cache_dir/"wikipedia/20200501.en/1.0.0/tokenized_wiki.arrow")
```
and when I see their size
```
ls -l --block-size=M
17460M  wikipedia-train.arrow
47511M  tokenized_wiki.arrow
```
The tokenized one is over 2x size of original one.
Is there something I did wrong ?
GitHub issue #254: [Feature request] Be able to remove a specific sample of the dataset
State: closed | Created: 2020-06-09T02:22:13Z

As mentioned in #117, it's currently not possible to remove a sample of the dataset.

But it is a important use case : After applying some preprocessing, some samples might be empty for example. We should be able to remove these samples from the dataset, or at least mark them as `removed` so when iterating the dataset, we don't iterate these samples.

I think it should be a feature. What do you think ?

---

Any work-around in the meantime ?
GitHub issue #252: NonMatchingSplitsSizesError error when reading the IMDB dataset
State: closed | Created: 2020-06-08T12:26:24Z

Hi!

I am trying to load the `imdb` dataset with this line:

`dataset = nlp.load_dataset('imdb', data_dir='/A/PATH', cache_dir='/A/PATH')`

but I am getting the following error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/load.py", line 517, in load_dataset
    save_infos=save_infos,
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/builder.py", line 363, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/builder.py", line 421, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/mounts/Users/cisintern/antmarakis/anaconda3/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
nlp.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=33442202, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=5929447, num_examples=4537, dataset_name='imdb')}, {'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
```

Am I overlooking something? Thanks!
GitHub issue #251: Better access to all dataset information
State: closed | Created: 2020-06-08T11:56:50Z

Moves all the dataset info down one level from `dataset.info.XXX` to `dataset.XXX`
This way it's easier to access `dataset.feature['label']` for instance

Also, add the original split instructions used to create the dataset in `dataset.split`
Ex:
```
from nlp import load_dataset
stsb = load_dataset('glue', name='stsb', split='train')
stsb.split
>>> NamedSplit('train')
```
GitHub issue #248: add Toronto BooksCorpus
State: closed | Created: 2020-06-07T12:54:56Z

1. I knew there is a branch `toronto_books_corpus`
 - After I downloaded it, I found it is all non-english, and only have one row. 
- It seems that it cites the wrong paper
- according to papar using it, it is called `BooksCorpus` but not `TornotoBooksCorpus`

2. It use a text mirror in google drive
- `bookscorpus.py` include a function `download_file_from_google_drive` , maybe you will want to put it elsewhere.
- text mirror is found in this [comment on the issue](https://github.com/soskek/bookcorpus/issues/24#issuecomment-556024973), and it said to have the same statistics as the one in the paper.
- You may want to download it and put it on your gs in case of it disappears someday.

3. Copyright ?
The paper has said

> **The BookCorpus Dataset.** In order to train our sentence similarity model we collected a corpus of 11,038 books ***from the web***. These are __**free books written by yet unpublished authors**__. We only included books that had more than 20K words in order to filter out perhaps noisier shorter stories. The dataset has books in 16 different genres, e.g., Romance (2,865 books), Fantasy (1,479), Science fiction (786), Teen (430), etc. Table 2 highlights the summary statistics of our book corpus.

and we have changed the form (not books), so I don't think it should have that problems. Or we can state that use it at your own risk or only for academic use. I know @thomwolf should know these things more.

This should solved #131
GitHub issue #247: Make all dataset downloads deterministic by applying `sorted` to glob and os.listdir
State: closed | Created: 2020-06-06T11:02:10Z

This PR makes all datasets loading deterministic by applying `sorted()` to all `glob.glob` and `os.listdir` statements.

Are there other "non-deterministic" functions apart from `glob.glob()` and `os.listdir()` that you can think of @thomwolf @lhoestq @mariamabarham @jplu ?

**Important** 
It does break backward compatibility for these datasets because
1. When loading the complete dataset the order in which the examples are saved is different now
2. When loading only part of a split, the examples themselves might be different.

@patrickvonplaten - the nlp / longformer notebook has to be updated since the examples might now be different
GitHub issue #246: What is the best way to cache a dataset? 
State: closed | Created: 2020-06-06T11:02:07Z

For example if I want to use streamlit with a nlp dataset:

```
@st.cache
def load_data():
    return nlp.load_dataset('squad')
```
This code raises the error "uncachable object"

Right now I just fixed with a constant for my specific case:
```
    @st.cache(hash_funcs={pyarrow.lib.Buffer: lambda b: 0})
```
But I was curious to know what is the best way in general