ERROR:root:mwparserfromhell

It looks like it comes from `mwparserfromhell`.

Would it be possible to get the bad `section` that causes this issue ? The `section` string is from `datasets/wikipedia.py:L548` ? You could just add a `try` statement and print the section if the line `section_text.append(section.strip_code().strip())` crashes.

It will help us know if we have to fix it on our side or if it is a `mwparserfromhell` issue.

Hi,

Thank you for you answer.
I have try to print the bad section using `try` and `except`, but it is a bit weird as the error seems to appear 3 times for instance, but the two first error does not print anything (as if the function did not go in the `except` part).
For the third one, I got that (I haven't display the entire text) :

> error : ==== Parque nacional Cajas ====
> {{AP|Parque nacional Cajas}}
> [[Archivo:Ecuador cajas national park.jpg|thumb|left|300px|Laguna del Cajas]]
> El parque nacional Cajas está situado en los [[Cordillera de los Andes|Andes]], al sur del [[Ecuador]], en la provincia de [[Provincia de Azuay|Azuay]], a 33
>  [[km]] al noroccidente de la ciudad de [[Cuenca (Ecuador)|Cuenca]]. Los accesos más comunes al parque inician todos en Cuenca: Desde allí, la vía Cuenca-Mol
> leturo atraviesa en Control de [[Surocucho]] en poco más de 30 minutos de viaje; más adelante, esta misma carretera pasa a orillas de la laguna La Toreadora donde están el Centro Administrativo y de Información del parque. Siguiendo de largo hacia [[Molleturo]], por esta vía se conoce el sector norte del Cajas y se serpentea entre varias lagunas mayores y menores.
> Para acceder al parque desde la costa, la vía Molleturo-Cuenca es también la mejor opción.

How can I display the link instead of the text ? I suppose it will help you more

The error appears several times as Apache Beam retries to process examples up to 4 times irc.

I just tried to run this text into `mwparserfromhell` but it worked without the issue.

I used this code (from the `wikipedia.py` script):
```python
import mwparserfromhell as parser
import re
import six

raw_content = r"""==== Parque nacional Cajas ====
{{AP|Parque nacional Cajas}}
[[Archivo:Ecuador cajas national park.jpg|thumb|left|300px|Laguna del Cajas]]
El parque nacional Cajas está situado en los [[Cordillera de los Andes|Andes]], al sur del [[Ecuador]], en la provincia de [[Provincia de Azuay|Azuay]], a 33
[[km]] al noroccidente de la ciudad de [[Cuenca (Ecuador)|Cuenca]]. Los accesos más comunes al parque inician todos en Cuenca: Desde allí, la vía Cuenca-Mol
leturo atraviesa en Control de [[Surocucho]] en poco más de 30 minutos de viaje; más adelante, esta misma carretera pasa a orillas de la laguna La Toreadora donde están el Centro Administrativo y de Información del parque. Siguiendo de largo hacia [[Molleturo]], por esta vía se conoce el sector norte del Cajas y se serpentea entre varias lagunas mayores y menores.
"""

wikicode = parser.parse(raw_content)

# Filters for references, tables, and file/image links.
re_rm_wikilink = re.compile("^(?:File|Image|Media):", flags=re.IGNORECASE | re.UNICODE)

def rm_wikilink(obj):
    return bool(re_rm_wikilink.match(six.text_type(obj.title)))

def rm_tag(obj):
    return six.text_type(obj.tag) in {"ref", "table"}

def rm_template(obj):
    return obj.name.lower() in {"reflist", "notelist", "notelist-ua", "notelist-lr", "notelist-ur", "notelist-lg"}

def try_remove_obj(obj, section):
    try:
        section.remove(obj)
    except ValueError:
        # For unknown reasons, objects are sometimes not found.
        pass

section_text = []
for section in wikicode.get_sections(flat=True, include_lead=True, include_headings=True):
    for obj in section.ifilter_wikilinks(matches=rm_wikilink, recursive=True):
        try_remove_obj(obj, section)
    for obj in section.ifilter_templates(matches=rm_template, recursive=True):
        try_remove_obj(obj, section)
    for obj in section.ifilter_tags(matches=rm_tag, recursive=True):
        try_remove_obj(obj, section)

    section_text.append(section.strip_code().strip())
```

Not sure why we're having this issue. Maybe could you get also the file that's causing that ?

thanks for your answer.
How can I know which file is causing the issue ?
I am trying to load the spanish wikipedia data.

Because of the way Apache Beam works we indeed don't have access to the file name at this point in the code.
We'll have to use some tricks I think :p

You can append `filepath` to `title` in `wikipedia.py:L512` for example. [[EDIT: it's L494 my bad]]
Then just do `try:...except:` on the  call of `_parse_and_clean_wikicode` L500 I guess.

Thanks for diving into this ! I tried it myself but I run out of memory on my laptop
As soon as we have the name of the file it should be easier to find what's wrong.

Thanks for your help.

I tried to print the "title" of the document inside the` except (mwparserfromhell.parser.ParserError) as e`,the title displayed was : "Campeonato Mundial de futsal de la AMF 2015". (Wikipedia ES) Is it what you were looking for ?

Thanks a lot @Shiro-LK !

I was able to reproduce the issue. It comes from [this table on wikipedia](https://es.wikipedia.org/wiki/Campeonato_Mundial_de_futsal_de_la_AMF_2015#Clasificados) that can't be parsed.

The file in which the problem occurs comes from the wikipedia dumps, and it can be downloaded [here](https://dumps.wikimedia.org/eswiki/20200501/eswiki-20200501-pages-articles-multistream6.xml-p6424816p7924815.bz2)

Parsing the file this way raises the parsing issue:

```python
import mwparserfromhell as parser
from tqdm.auto import tqdm
import bz2
import six
import logging
import codecs
import xml.etree.cElementTree as etree

filepath = "path/to/eswiki-20200501-pages-articles-multistream6.xml-p6424816p7924815.bz2"

def _extract_content(filepath):
    """Extracts article content from a single WikiMedia XML file."""
    logging.info("generating examples from = %s", filepath)
    with open(filepath, "rb") as f:
        f = bz2.BZ2File(filename=f)
        if six.PY3:
            # Workaround due to:
            # https://github.com/tensorflow/tensorflow/issues/33563
            utf_f = codecs.getreader("utf-8")(f)
        else:
            utf_f = f
        # To clear root, to free-up more memory than just `elem.clear()`.
        context = etree.iterparse(utf_f, events=("end",))
        context = iter(context)
        unused_event, root = next(context)
        for unused_event, elem in tqdm(context, total=949087):
            if not elem.tag.endswith("page"):
                continue
            namespace = elem.tag[:-4]
            title = elem.find("./{0}title".format(namespace)).text
            ns = elem.find("./{0}ns".format(namespace)).text
            id_ = elem.find("./{0}id".format(namespace)).text
            # Filter pages that are not in the "main" namespace.
            if ns != "0":
                root.clear()
                continue
            raw_content = elem.find("./{0}revision/{0}text".format(namespace)).text
            root.clear()

            if "Campeonato Mundial de futsal de la AMF 2015" in title:
                yield (id_, title, raw_content)

for id_, title, raw_content in _extract_content(filepath):
    wikicode = parser.parse(raw_content)
```

The copied the raw content that can't be parsed [here](https://pastebin.com/raw/ZbmevLyH).

The minimal code to reproduce is:
```python
import mwparserfromhell as parser
import requests

raw_content = requests.get("https://pastebin.com/raw/ZbmevLyH").content.decode("utf-8")
wikicode = parser.parse(raw_content)

```

I will create an issue on mwparserfromhell's repo to see if we can fix that

This going to be fixed in the next `mwparserfromhell` release :)

Hi,

I am trying to download some wikipedia data but I got this error for spanish "es" (but there are maybe some others languages which have the same error I haven't tried all of them ).

`ERROR:root:mwparserfromhell ParseError: This is a bug and should be reported. Info: C tokenizer exited with non-empty token stack.`

The code I have use was :
`dataset = load_dataset('wikipedia', '20200501.es', beam_runner='DirectRunner')`


Blog Authorship Corpus, Non Matching Splits Sizes Error, nlp viewer

I wonder if this means downloading failed? That corpus has a really slow server.

This dataset seems to have a decoding problem that results in inconsistencies in the number of generated examples.
See #215.
That's why we end up with a `NonMatchingSplitsSizesError `.

Selecting `blog_authorship_corpus` in the nlp viewer throws the following error:

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=610252351, num_examples=532812, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='train', num_bytes=614706451, num_examples=535568, dataset_name='blog_authorship_corpus')}, {'expected': SplitInfo(name='validation', num_bytes=37500394, num_examples=31277, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='validation', num_bytes=32553710, num_examples=28521, dataset_name='blog_authorship_corpus')}]
Traceback:
File "/home/sasha/streamlit/lib/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 172, in <module>
    dts, fail = get(str(option.id), str(conf_option.name) if conf_option else None)
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 132, in get
    builder_instance.download_and_prepare()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
```
@srush @lhoestq


Nested sequences with dicts

Oh yes, this is a backward compatibility feature with tensorflow_dataset in which a `Sequence` or `dict` is converted in a `dict` of `lists`, unfortunately it is not very intuitive, see here: https://github.com/huggingface/nlp/blob/master/src/nlp/features.py#L409

To avoid this behavior, you can just define the list in the feature with a simple list or a tuple (which is also simpler to write).
In your case, the features could be as follow:
``` python
...
features=nlp.Features({
    "title": nlp.Value("string"),
    "vertexSet": [[{
        "name": nlp.Value("string"),
        "sent_id": nlp.Value("int32"),
        "pos": nlp.features.Sequence(nlp.Value("int32")),
        "type": nlp.Value("string"),
    }]],
    ...
    }),
...
```

Am pretty much finished [adding a dataset](https://github.com/ghomasHudson/nlp/blob/DocRED/datasets/docred/docred.py) for [DocRED](https://github.com/thunlp/DocRED), but am getting an error when trying to add a nested `nlp.features.sequence(nlp.features.sequence({key:value,...}))`.

The original data is in this format:
```python
{
  'title': "Title of wiki page",
  'vertexSet': [
                  [
                    { 'name': "mention_name",
                      'sent_id': "mention in which sentence",
                      'pos': ["postion of mention in a sentence"],
                      'type': "NER_type"},
                    {another mention}
                  ],
                  [another entity]
                ]
    ...
}
```
So to represent this I've attempted to write:
```
...
features=nlp.Features({
    "title": nlp.Value("string"),
    "vertexSet": nlp.features.Sequence(nlp.features.Sequence({
        "name": nlp.Value("string"),
        "sent_id": nlp.Value("int32"),
        "pos": nlp.features.Sequence(nlp.Value("int32")),
        "type": nlp.Value("string"),
    })),
    ...
    }),
...
```
This is giving me the error:
```
pyarrow.lib.ArrowTypeError: Could not convert [{'pos': [[0,2], [2,4], [3,5]], "type": ["ORG", "ORG", "ORG"], "name": ["Lark Force", "Lark Force", "Lark Force", "sent_id": [0, 3, 4]}..... with type list: was not a dict, tuple, or recognized null value for conversion to struct type
```
Do we expect the pyarrow stuff to break when doing this deeper nesting? I've checked that it still works when you do `nlp.features.Sequence(nlp.features.Sequence(nlp.Value("string"))` or `nlp.features.Sequence({key:value,...})` just not nested sequences with a dict.

If it's not possible, I can always convert it to a shallower structure. I'd rather not change the DocRED authors' structure if I don't have to though.


Multitask

It's definitely going in the right direction ! Thanks for giving it a try

I really like the API.
IMO it's fine right now if we don't have all the dataset transforms (map, filter, etc.) as it can be done before building the multitask dataset, but it will be important to have them in the end.
All the formatting methods could easily be added though.

I think there are some parts that will require some work with apache arrow like slicing. I can find a way to do it using pyarrow tables concatenation (I did something similar when implementing `__getitem__` with an input that is a list of indices [here](https://github.com/huggingface/nlp/pull/322/files#diff-73270df8d7f08c62a27e40806e1a5fb0R463-R469)). It is very fast and it allows to have the same output format as a normal Dataset.

Also maybe we should check that not only the columns but also the schemas match ?
And maybe add the `seed` of the shuffling step as an argument ?

Maybe we should remove the methods that are not implemented for now, WDYT @thomwolf ?

That's an interesting first draft, thanks a lot for that and the user facing API is really nice.

I think we should dive more into this and the questions of #217 before merging the first version though.

In particular, the typical way to do multi-tasking is usually to sample a task and then sample a batch within the selected task. I think we should probably stay be closer to this traditional approach, or at least make it very easy to do, rather than go to close to the T5 approach which is very specific to this paper.

In this regard, it seems important to find some way to address the remarks of @zphang. I'm still wondering if we should not adopt more of a sampling approach rather than an iteration approach.

@thomwolf Thanks! I mainly wanted to get something working quickly for my own MTL research. I agree with a lot of the points you made so I'll convert this pull request back to a draft.

For your specific point about 'batch-level' multitask mixing, it would be a pretty trivial change to add a `batch_size` parameter and ensure every `batch_size` examples are from the same task. This would certainly work, but would add a notion of 'batches' to a Dataset, which does feel like a 'Sampler-level' concept and not a Dataset one. There's also the possibility of wanting some specific task-level sampling functionality (e.g. applying `SortishSampler` to each task) which would only work with this kind of 2 step sampling approach. My first proposal in the transformers repo was actually a Sampler https://github.com/huggingface/transformers/issues/4340. I wonder whether functionality at the sampler-level has a place in the vision for the `nlp` repo?

I imagine following a sampling approach you'd have to abandon maintaining the same user-facing API as a standard dataset (A shame because replacing a single dataset seamlessly with a multitask one is a really nice user-experience).

Random half-Idea: You could have a class which accepts a list of any iterables (either a Dataset or a DataLoader which already is doing the batching). Not sure what interface you'd present though. hmmm.

There's definitely more discussion to have.

Following our discussion in #217, I've implemented a first working version of `MultiDataset`.

There's a function `build_multitask()` which takes either individual `nlp.Dataset`s or `dicts` of splits and constructs `MultiDataset`(s). I've added a notebook with example usage.

I've implemented many of the `nlp.Dataset` methods (cache_files, columns, nbytes, num_columns, num_rows, column_names, schema, shape). Some of the other methods are complicated as they change the number of examples. These raise `NotImplementedError`s at the moment.

This will need some tests which I haven't written yet.

There's definitely room for improvements but I think the general approach is sound.


Adding a dataset with multiple subtasks

For one dataset you can have different configurations that each have their own `nlp.Features`.
We imagine having one configuration per subtask for example.
They are loaded with `nlp.load_dataset("my_dataset", "my_config")`.

For example the `glue` dataset has many configurations. It is a bit different from your case though because each configuration is a dataset by itself (sst2, mnli).
Another example is `wikipedia` that has one configuration per language.

I intent to add the datasets of the MT Quality Estimation shared tasks to `nlp`. However, they have different subtasks -- such as word-level, sentence-level and document-level quality estimation, each of which having different language pairs, and some of the data reused in different subtasks.

For example, in [QE 2019,](http://www.statmt.org/wmt19/qe-task.html) we had the same English-Russian and English-German data for word-level and sentence-level QE.

I suppose these datasets could have both their word and sentence-level labels inside `nlp.Features`; but what about other subtasks? Should they be considered a different dataset altogether?

I read the discussion on #217 but the case of QE seems a lot simpler.


add AG News dataset

Thanks @jxmorris12 for adding this adding.
Can you please add a small description of the PR?

adds support for the AG-News topic classification dataset


[Question] Best way to batch a large dataset?

generic discussion

Generic discussion on the library

Update: I think I've found a solution.

```python
output_types = {"input_ids": tf.int64, "token_type_ids": tf.int64, "attention_mask": tf.int64}
def train_dataset_gen():
    for i in range(len(train_dataset)):
        yield train_dataset[i]
tf_dataset = tf.data.Dataset.from_generator(train_dataset_gen, output_types=output_types)
```

loads WikiText-2 in 20 ms, and WikiText-103 in 20 ms. It appears to be lazily loading via indexing train_dataset.

Yes this is the current best solution. We should probably show it in the tutorial notebook.

Note that this solution unfortunately doesn't allow to train on TPUs (yet). See #193

This approach still seems quite slow. When using TFRecords with a similar training loop, I get ~3.0-3.5 it/s on multi-node, multi-GPU training. I notice a pretty severe performance regression when scaling, with observed performance numbers. Since the allreduce step takes less than 100ms/it and I've achieved 80% scaling efficiency up to 64 GPUs, it must be the data pipeline.

| Nodes | GPUs | Iterations/Second |
| --- | --- | --- |
| 1 | 2 | 2.01 |
| 1 | 8 | 0.81 |
| 2 | 16 | 0.37 |

Here are performance metrics over 10k steps. The iteration speed appears to follow some sort of caching pattern. I would love to use `nlp` in my project, but a slowdown from 3.0 it/s to 0.3 it/s is too great to stomach.

<img width="1361" alt="Screen Shot 2020-07-02 at 8 29 22 AM" src="https://user-images.githubusercontent.com/4564897/86378156-2f8d3900-bc3e-11ea-918b-c395c3df5377.png">

An interesting alternative to investigate here would be to use the tf.io library which has some support for Arrow to TF conversion: https://www.tensorflow.org/io/api_docs/python/tfio/arrow/ArrowDataset

There are quite a few types supported, including lists so if the unsupported columns are dropped then we could maybe have a zero-copy mapping from Arrow to TensorFlow, including tokenized inputs and 1D tensors like the ones we mostly use in NLP: https://github.com/tensorflow/io/blob/322b3170c43ecac5c6af9e39dbd18fd747913e5a/tensorflow_io/arrow/python/ops/arrow_dataset_ops.py#L44-L72

Here is an introduction on Arrow to TF using tf.io: https://medium.com/tensorflow/tensorflow-with-apache-arrow-datasets-cdbcfe80a59f

Interesting. There's no support for strings, but it does enable int and floats so that would work for tokenized inputs.

ArrowStreamDataset requires loading from a "record batch iterator", which can be instantiated from in-memory arrays as described here: https://arrow.apache.org/docs/python/ipc.html.

But the nlp.Dataset stores its data as a `pyarrow.lib.Table`, and the underlying features are `pyarrow.lib.ChunkedArray`. I can't find any documentation about lazily creating a record batch iterator from a ChunkedArray or a Table. Have you had any success?

I can't find [any uses](https://grep.app/search?q=ArrowDataset&filter[lang][0]=Python) of tfio.arrow.ArrowDataset on GitHub.

You can use `to_batches` maybe?
https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_batches

Also note that since #322 it is now possible to do
```python
ids = [1, 10, 42, 100]
batch = dataset[ids]
```
From my experience it is quite fast but it can take lots of memory for large batches (haven't played that much with it).
Let me know if you think there could be a better way to implement it. (current code is [here](https://github.com/huggingface/nlp/blob/78628649962671b4aaa31a6b24e7275533416845/src/nlp/arrow_dataset.py#L463))

Thanks @lhoestq! That format is much better to work with.

I put together a benchmarking script. This doesn't measure the CPU-to-GPU efficiency, nor how it scales with multi-GPU multi-node training where many processes are making the same demands on the same dataset. But it does show some interesting results:

```python
import nlp
import numpy as np
import tensorflow as tf
import time

dset = nlp.load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
dset = dset.filter(lambda ex: len(ex["text"]) > 0)
bsz = 1024
n_batches = 100

def single_item_gen():
    for i in range(len(dset)):
        yield dset[i]

def sequential_batch_gen():
    for i in range(0, len(dset), bsz):
        yield dset[i:i+bsz]

def random_batch_gen():
    for i in range(len(dset)):
        indices = list(np.random.randint(len(dset), size=(bsz,)))
        yield dset[indices]

output_types = {"text": tf.string}
single_item = tf.data.Dataset.from_generator(single_item_gen, output_types=output_types).batch(bsz)
interleaved = tf.data.Dataset.range(10).interleave(
    lambda idx: tf.data.Dataset.from_generator(single_item_gen, output_types=output_types),
    cycle_length=10,
)
sequential_batch = tf.data.Dataset.from_generator(sequential_batch_gen, output_types=output_types)
random_batch = tf.data.Dataset.from_generator(random_batch_gen, output_types=output_types)

def iterate(tf_dset):
    start = time.perf_counter()
    for i, batch in enumerate(tf_dset.take(n_batches)):
        pass
    elapsed = time.perf_counter() - start
    print(f"{tf_dset} took {elapsed:.3f} secs")

iterate(single_item)
iterate(interleaved)
iterate(sequential_batch)
iterate(random_batch)
```

Results:
```
<BatchDataset shapes: {text: <unknown>}, types: {text: tf.string}> took 23.005 secs
<InterleaveDataset shapes: {text: <unknown>}, types: {text: tf.string}> took 0.135 secs
<FlatMapDataset shapes: {text: <unknown>}, types: {text: tf.string}> took 0.074 secs
<FlatMapDataset shapes: {text: <unknown>}, types: {text: tf.string}> took 0.550 secs
```

- Batching a generator which fetches a single item is terrible.
- Interleaving performs well on a single process, but doesn't scale well to multi-GPU training. I believe the bottleneck here is in Arrow dataset locking or something similar. The numbers from the table above are with interleaving.
- The sequential access dominates the random access (7x faster). Is there any way to bring random access times closer to sequential access? Maybe re-indexing the dataset after shuffling each pass over the data.

Hey @jarednielsen

Thanks for this very interesting analysis!! IMHO to read text data one should use `tf.data.TextLineDataset`. It would be interesting to compare what you have done with simply load with a `TextLineDataset` and see if there is a difference.

A good example can be found here https://www.tensorflow.org/tutorials/load_data/text

Thanks! I'm not actually loading in raw text data, that was just the synthetic data I created for this benchmark. A more realistic use case would be a dataset of tokenized examples, which would be a dict of lists of integers. TensorFlow's TextLineDataset greedily loads the dataset into the graph itself, which can lead to out-of-memory errors - one of the main reason I'm so drawn to the `nlp` library is its zero-copy no-RAM approach to dataset loading and mapping.

It's quite helpful for running a preprocessing pipeline - a sample ELECTRA pipeline I've built is here: https://github.com/jarednielsen/deep-learning-models/blob/nlp/models/nlp/common/preprocess.py.

Sorry, I think I badly expressed myself, my bad. What I suggested is to compare with the usual loading textual data in pure TF with `TextLineDataset` with `nlp`. I know it is not recommended with very large datasets to use it, but I was curious to see how it behaves compared to a processing with `nlp` on smaller datasets.

BTW your script looks very interesting, thanks for sharing!!

I'm training on large datasets such as Wikipedia and BookCorpus. Following the instructions in [the tutorial notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb), I see the following recommended for TensorFlow:

```python
train_tf_dataset = train_tf_dataset.filter(remove_none_values, load_from_cache_file=False)
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'start_positions', 'end_positions']
train_tf_dataset.set_format(type='tensorflow', columns=columns)
features = {x: train_tf_dataset[x].to_tensor(default_value=0, shape=[None, tokenizer.max_len]) for x in columns[:3]}
labels = {"output_1": train_tf_dataset["start_positions"].to_tensor(default_value=0, shape=[None, 1])}
labels["output_2"] = train_tf_dataset["end_positions"].to_tensor(default_value=0, shape=[None, 1])
### Question about this last line ###
tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)
```

This code works for something like WikiText-2. However, scaling up to WikiText-103, the last line takes 5-10 minutes to run. I assume it is because tf.data.Dataset.from_tensor_slices() is pulling everything into memory, not lazily loading. This approach won't scale up to datasets 25x larger such as Wikipedia.

So I tried manual batching using `dataset.select()`:

```python
idxs = np.random.randint(len(dataset), size=bsz)
batch = dataset.select(idxs).map(lambda example: {"input_ids": tokenizer(example["text"])})
tf_batch = tf.constant(batch["ids"], dtype=tf.int64)
```

This appears to create a new Apache Arrow dataset with every batch I grab, and then tries to cache it. The runtime of `dataset.select([0, 1])` appears to be much worse than `dataset[:2]`. So using `select()` doesn't seem to be performant enough for a training loop.

Is there a performant scalable way to lazily load batches of nlp Datasets?


Fixed singlular very minor spelling error

Thank you BatJeti! The storm-joker, aka the typo, finally got caught!

An instance of "independantly" was changed to "independently". That's all.


Add MWSC

Adding the [Modified Winograd Schema Challenge](https://github.com/salesforce/decaNLP/blob/master/local_data/schema.txt) dataset which formed part of the [decaNLP](http://decanlp.com/) benchmark. Not sure how much use people would find for it it outside of the benchmark, but it is general purpose.

Code is heavily borrowed from the [decaNLP repo](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L773-L877).

There's a few (possibly overly opinionated) design choices I made:

- I used the train/test/dev split [buried in the decaNLP code](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L852-L855)
- I split out each example into the 2 alternatives. Originally the data uses the format:
    ```
    The city councilmen refused the demonstrators a permit because they [feared/advocated] violence.
    Who [feared/advocated] violence?
    councilmen/demonstrators
    ```
    I split into the 2 variants:
    ```
    The city councilmen refused the demonstrators a permit because they feared violence.
    Who feared violence?
    councilmen/demonstrators

    The city councilmen refused the demonstrators a permit because they advocated violence.
    Who advocated violence?
    councilmen/demonstrators
    ```
    I can't see any use for having the options combined into a single example (splitting them is [the way decaNLP processes](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L846-L850)) them. You can't train on both versions with them combined, and splitting the examples later would be a pain to do. I think [winogrande.py](https://github.com/huggingface/nlp/blob/master/datasets/winogrande/winogrande.py) presents the data in this way?

- I've not used the decaNLP framing (appending the options to the question e.g. `Who feared violence?
 -- councilmen or demonstrators?`) but left it more generic by adding the options as a new key: `"options":["councilmen","demonstrators"]` This should be an easy thing to change using `map` if needed by a specific application.

Dataset is working as-is but if anyone has any thoughts/preferences on the design decisions here I'm definitely open to different choices.


[Feature request] Add `shard()` method to dataset

Hi Jared,
Interesting, thanks for raising this question. You can also do that after loading with `dataset.select()` or `dataset.filter()` which let you keep only a specific subset of rows in a dataset.
What is your use-case for sharding?

Thanks for the pointer to those functions! It's still a little more verbose since you have to manually calculate which ids each rank would keep, but definitely works.

My use case is multi-node, multi-GPU training and avoiding global batches of duplicate elements. I'm using horovod. You can shuffle indices, or set random seeds, but explicitly sharding the dataset up front is the safest and clearest way I've found to do so.

Currently, to shard a dataset into 10 pieces on different ranks, you can run

```python
rank = 3 # for example
size = 10
dataset = nlp.load_dataset('wikitext', 'wikitext-2-raw-v1', split=f"train[{rank*10}%:{(rank+1)*10}%]")
```

However, this breaks down if you have a number of ranks that doesn't divide cleanly into 100, such as 64 ranks. Is there interest in adding a method shard() that looks like this?

```python
rank = 3
size = 64
dataset = nlp.load_dataset("wikitext", "wikitext-2-raw-v1", split="train").shard(rank=rank, size=size)
```

TensorFlow has a similar API: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard. I'd be happy to contribute this code.


Add qa_zre

Adding the QA-ZRE dataset from ["Zero-Shot Relation Extraction via Reading Comprehension"](http://nlp.cs.washington.edu/zeroshot/).

A common processing step seems to be replacing the `XXX` placeholder with the `subject`. I've left this out as it's something you could easily do with `map`.


add wikisql

Adding the [WikiSQL](https://github.com/salesforce/WikiSQL) dataset.

Interesting things to note:
- Have copied the function (`_convert_to_human_readable`) which converts the SQL query to a human-readable (string) format as this is what most people will want when actually using this dataset for NLP applications.
- `conds` was originally a tuple but is converted to a dictionary to support differing types.

Would be nice to add the logical_form metrics too at some point.


Add narrative qa

Does it make sense to download the full stories? I remember attempting to implement this dataset a while ago and ended up with something like:
```python
    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""

        dl_dir = dl_manager.download_and_extract(_DOWNLOAD_URL)
        data_dir = os.path.join(dl_dir, "narrativeqa-master")

        urls = {"test":{}, "train": {},"valid":{}}
        with open(os.path.join(data_dir,"documents.csv")) as f_in:
            csv_reader = csv.reader(f_in)
            next(csv_reader) # discard header row
            for i,row in enumerate(csv_reader):
                if i > 1572:
                    break
                if row != []:
                    urls[row[1]][row[0]] = row[3]

        url_files = {}
        for key in urls.keys():
            url_files[key] = dl_manager.download_and_extract(urls[key])

        return [
            nlp.SplitGenerator(
                name=nlp.Split.TRAIN,
                gen_kwargs={
                    "data_dir":data_dir,
                    "split":"train",
                    "doc_id_to_path":url_files["train"]
                    }
            ),
        ....
```
It does end up cluttering your huggingface cache dir though.

Also since there doesn't seem to be any meaning in the order of answer_1 and answer_2, it might make sense to combine them (see [squad.py](https://github.com/huggingface/nlp/blob/8b0ffc85e4e52ae1f18d31be99b6c70b82c991ca/datasets/squad/squad.py#L86-L88)):
```python
"answers": nlp.features.Sequence({
    "text": nlp.Value("string"),
    "tokenized": nlp.features.Sequence(nlp.Value("string"))
})
```
(the tokenized features should also probably be lists of strings not just strings - see [natural_questions.py](https://github.com/huggingface/nlp/blob/4cd34287300a1135ce7b22f6dd209ca305c71b3a/datasets/natural_questions/natural_questions.py#L83))

Again, this is a personal preference thing, but it might be useful to combine the document-related features:
```python
{
    "document": {
        "id": nlp.Value("string"),
        "kind": nlp.Value("string"),
        "url": nlp.Value("string"),
        "file_size": nlp.Value("int32"),
        "word_count": nlp.Value("int32"),
        "start": nlp.Value("string"),
        "end": nlp.Value("string"),
        "wiki_url": nlp.Value("string"),
        "wiki_title": nlp.Value("string"),
        "summary": nlp.features.Sequence({
            "text": nlp.Value("string"),
            "tokens": nlp.features.Sequence(nlp.Value("string"))
        }),
        "text": nlp.Value("string"),
    },
    "question": nlp.features.Sequence({
        "text": nlp.Value("string"),
        "tokens": nlp.features.Sequence(nlp.Value("string"))
    }),
    "answers": nlp.features.Sequence({
        "text": nlp.Value("string"),
        "tokens": nlp.features.Sequence(nlp.Value("string"))
    })
}
```

Did you manage to fix the dummy data @Varal7 ?

@lhoestq do you think it's acceptable for the `dl_manager` to go grab all the individual stories from project gutenburg? I've got a working version of that but it does clutter up your huggingface cache somewhat.

The real value (and original purpose) of this dataset is doing question answering on the full text.

> @lhoestq do you think it's acceptable for the `dl_manager` to go grab all the individual stories from project gutenburg? I've got a working version of that but it does clutter up your huggingface cache somewhat.
>
> The real value (and original purpose) of this dataset is doing question answering on the full text.

What's the problem exactly with the cache ?

Nothing, just that because each story is a separate download it gets a bit messy as all 1573 files are under `~/.cache/hugginface/datasets` rather than organized under a subdir.

Probably doesn't matter to the end user though.

Yea I agree it's a mess. I just created #393 to make things easier.

I got the PR merged to have a cleaner the cache directory (everything is downloaded inside the 'downloads' sub-directory).
Feel free to download all the stories then @ghomasHudson @Varal7  x)
If you have the possibility of downloading a compressed file with most of the stories at once it would be better though.

Looks good @lhoestq . The problem I'm having at the moment is that stories from project Gutenberg occasionally fail. All books are out of copyright so we should be able to host them.

Here's a zip file of the full text if we have anywhere to put them: https://drive.google.com/file/d/17jOR7NqvzDwSlPXrlHaYV-PGI8JG-KY5/view?usp=sharing

I put the zip file here @ghomasHudson
https://storage.googleapis.com/huggingface-nlp/datasets/narrative_qa/narrativeqa_full_text.zip

Sorry for the delay

Test cases for dummy data don't pass

Only contains data for summaries (not whole story)


Specify utf-8 encoding for MRPC files

Fixes #307, again probably a Windows-related issue.


Specify encoding for MRPC

Same as #242, but with MRPC: on Windows, I get a `UnicodeDecodeError` when I try to download the dataset:
```python
dataset = nlp.load_dataset('glue', 'mrpc')
```

```python
Downloading and preparing dataset glue/mrpc (download: Unknown size, generated: Unknown size, total: Unknown size) to C:\Users\Python\.cache\huggingface\datasets\glue\mrpc\1.0.0...
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
~\Miniconda3\envs\nlp\lib\site-packages\nlp\builder.py in incomplete_dir(dirname)
    369                 try:
--> 370                     yield tmp_dir
    371                     if os.path.isdir(dirname):

~\Miniconda3\envs\nlp\lib\site-packages\nlp\builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    430                 verify_infos = not save_infos and not ignore_verifications
--> 431                 self._download_and_prepare(
    432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs

~\Miniconda3\envs\nlp\lib\site-packages\nlp\builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    482                 # Prepare split will record examples associated to the split
--> 483                 self._prepare_split(split_generator, **prepare_split_kwargs)
    484             except OSError:

~\Miniconda3\envs\nlp\lib\site-packages\nlp\builder.py in _prepare_split(self, split_generator)
    663         generator = self._generate_examples(**split_generator.gen_kwargs)
--> 664         for key, record in utils.tqdm(generator, unit=" examples", total=split_info.num_examples, leave=False):
    665             example = self.info.features.encode_example(record)

~\Miniconda3\envs\nlp\lib\site-packages\tqdm\notebook.py in __iter__(self, *args, **kwargs)
    217         try:
--> 218             for obj in super(tqdm_notebook, self).__iter__(*args, **kwargs):
    219                 # return super(tqdm...) will not catch exception

~\Miniconda3\envs\nlp\lib\site-packages\tqdm\std.py in __iter__(self)
   1128         try:
-> 1129             for obj in iterable:
   1130                 yield obj

~\Miniconda3\envs\nlp\lib\site-packages\nlp\datasets\glue\7fc58099eb3983a04c8dac8500b70d27e6eceae63ffb40d7900c977897bb58c6\glue.py in _generate_examples(self, data_file, split, mrpc_files)
    514             examples = self._generate_example_mrpc_files(mrpc_files=mrpc_files, split=split)
--> 515             for example in examples:
    516                 yield example["idx"], example

~\Miniconda3\envs\nlp\lib\site-packages\nlp\datasets\glue\7fc58099eb3983a04c8dac8500b70d27e6eceae63ffb40d7900c977897bb58c6\glue.py in _generate_example_mrpc_files(self, mrpc_files, split)
    576                 reader = csv.DictReader(f, delimiter="\t", quoting=csv.QUOTE_NONE)
--> 577                 for n, row in enumerate(reader):
    578                     is_row_in_dev = [row["#1 ID"], row["#2 ID"]] in dev_ids

~\Miniconda3\envs\nlp\lib\csv.py in __next__(self)
    110             self.fieldnames
--> 111         row = next(self.reader)
    112         self.line_num = self.reader.line_num

~\Miniconda3\envs\nlp\lib\encodings\cp1252.py in decode(self, input, final)
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1180: character maps to <undefined>
```
The fix is the same: specify `utf-8` encoding when opening the file. The previous fix didn't work as MRPC's download process is different from the others in GLUE.
I am going to propose a new PR :)


add pg19 dataset

@lucidrains - Thanks a lot for making the PR - PG19 is a super important dataset! Thanks for making it. Many people are asking for PG-19, so it would be great to have that in the library as soon as possible @thomwolf .

I'm looking forward to our first deep learning written novel already lol. It's definitely happening

Oh I just noticed but as we changed the urls to download the files, we have to update `dataset_infos.json`.
Could you re-rurn `nlp-cli test ./datasets/pg19 --save_infos` ?

@lhoestq - I think it's good to merge no?

`dataset_infos.json` is still not up to date with the new urls (we can see that there are urls like `gs://deepmind-gutenberg/train/*` instead of `https://storage.googleapis.com/deepmind-gutenberg/train/*` in the json file)

Can you check that you re-ran the command to update the json file, and that you pushed the changes @lucidrains ?

@lhoestq ohhh, I made the change in this commit https://github.com/lucidrains/nlp/commit/f3e23d823ad9942031be80b7c4e4212c592cd90c , that's interesting that the pull request didn't pick it up. maybe it's because I did it on another machine, let me check and get back to you!

@lhoestq wrong branch 😅 thanks for catching!

https://github.com/huggingface/nlp/issues/274

Add functioning PG19 dataset with dummy data

`cos_e.py` was just auto-linted by `make style`


Importing downloaded package repository fails

metric bug

A bug in a metric script

The `get_imports` function in `src/nlp/load.py` has a feature to download a package as a zip archive of the github repository and import functions from the unpacked directory. This is used for example in the `metrics/coval.py` file, and would be useful to add BLEURT (@ankparikh).

Currently however, the code seems to have trouble with imports within the package. For example:
```
import nlp
coval = nlp.load_metric('coval')
```
yields:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yacine/Code/nlp/src/nlp/load.py", line 432, in load_metric
    metric_cls = import_main_class(module_path, dataset=False)
  File "/home/yacine/Code/nlp/src/nlp/load.py", line 57, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/yacine/anaconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/yacine/Code/nlp/src/nlp/metrics/coval/a78807df33ac45edbb71799caf2b3b47e55df4fd690267808fe963a5e8b30952/coval.py", line 21, in <module>
    from .coval_backend.conll import reader  # From: https://github.com/ns-moosavi/coval
  File "/home/yacine/Code/nlp/src/nlp/metrics/coval/a78807df33ac45edbb71799caf2b3b47e55df4fd690267808fe963a5e8b30952/coval_backend/conll/reader.py", line 2, in <module>
    from conll import mention
ModuleNotFoundError: No module named 'conll'
```

Not sure what the fix would be there.


Problem while printing doc string when instantiating multiple metrics.

A bug in a metric script

When I load more than one metric and try to print doc string of a particular metric,. It shows the doc strings of all imported metric one after the other which looks quite confusing and clumsy.
Attached [Colab](https://colab.research.google.com/drive/13H0ZgyQ2se0mqJ2yyew0bNEgJuHaJ8H3?usp=sharing) Notebook for problem clarification..


allow to move files across file systems

Users are allowed to use the `cache_dir` that they want.
Therefore it can happen that we try to move files across filesystems.
We were using `os.rename` that doesn't allow that, so I changed some of them to `shutil.move`.

This should fix #301


Question - Sign Language Datasets

enhancement

New feature or request

generic discussion

Generic discussion on the library

Even more complicating -

As I see it, datasets can have "addons".
For example, the WebNLG dataset is a dataset for data-to-text. However, a work of mine and other works enriched this dataset with text plans / underlying text structures. In that case, I see a need to load the dataset "WebNLG" with "plans" addon.

Same for sign language - if there is a dataset of videos, one addon can be to run OpenPose, another to run ARKit4 pose estimation, and another to run PoseNet, or even just a video embedding addon. (which are expensive to run individually for everyone who wants to use these data)

This is something I dabbled with my own implementation to a [research datasets library](https://github.com/AmitMY/meta-scholar/) and I love to get the discussion going on these topics.

This is a really cool idea !
The example for data objects you gave for the RWTH-PHOENIX-Weather 2014 T dataset can totally fit inside the library.

For your point about formats like `ilex`, `eaf`, or `srt`, it is possible to use any library in your dataset script.
However most user probably won't need these libraries, as most datasets don't need them, and therefore it's unlikely that we will have them in the minimum requirements to use `nlp` (we want to keep it as light-weight as possible). If a user wants to load your dataset and doesn't have the libraries you need, an error is raised asking the user to install them.

More generally, we plan to have something like a `requirements.txt` per dataset. This could also be a place for addons as you said. What do you think ?

Thanks, Quentin, I think a `requirements.txt` per dataset will be a good thing.
I will work on adding this dataset next week, and once we sort all of the kinks, I'll add more.

An emerging field in NLP is SLP - sign language processing.

I was wondering about adding datasets here, specifically because it's shaping up to be large and easily usable.
The metrics for sign language to text translation are the same.

So, what do you think about (me, or others) adding datasets here?

An example dataset would be [RWTH-PHOENIX-Weather 2014 T](https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/)
For every item in the dataset, the data object includes:
1. video_path - path to mp4 file
2. pose_path - a path to `.pose` file with human pose landmarks
3. openpose_path - a path to a `.json` file with human pose landmarks
4. gloss - string
5. text - string
6. video_metadata - height, width, frames, framerate

------

To make it a tad more complicated - what if sign language libraries add requirements to `nlp`? for example, sign language is commonly annotated using `ilex`, `eaf`, or `srt` files, which are all loadable as text, but there is no reason for the dataset to parse that file by itself, if libraries exist to do so.


Setting cache_dir gives error on wikipedia download

Whoops didn't mean to close this one.
I did some changes, could you try to run it from the master branch ?

First of all thank you for a super handy library! I'd like to download large files to a specific drive so I set `cache_dir=my_path`. This works fine with e.g. imdb and squad. But on wikipedia I get an error:
```
nlp.load_dataset('wikipedia', '20200501.de', split = 'train', cache_dir=my_path)
```
```
OSError                                   Traceback (most recent call last)
<ipython-input-2-23551344d7bc> in <module>
      1 import nlp
----> 2 nlp.load_dataset('wikipedia', '20200501.de', split = 'train', cache_dir=path)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    385                     with utils.temporary_assignment(self, "_cache_dir", tmp_data_dir):
    386                         reader = ArrowReader(self._cache_dir, self.info)
--> 387                         reader.download_from_hf_gcs(self._cache_dir, self._relative_data_dir(with_version=True))
    388                         downloaded_info = DatasetInfo.from_directory(self._cache_dir)
    389                         self.info.update(downloaded_info)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/arrow_reader.py in download_from_hf_gcs(self, cache_dir, relative_data_dir)
    231             remote_dataset_info = os.path.join(remote_cache_dir, "dataset_info.json")
    232             downloaded_dataset_info = cached_path(remote_dataset_info)
--> 233             os.rename(downloaded_dataset_info, os.path.join(cache_dir, "dataset_info.json"))
    234             if self._info is not None:
    235                 self._info.update(self._info.from_directory(cache_dir))

OSError: [Errno 18] Invalid cross-device link: '/home/local/NTU/nn/.cache/huggingface/datasets/025fa4fd4f04aaafc9e939260fbc8f0bb190ce14c61310c8ae1ddd1dcb31f88c.9637f367b6711a79ca478be55fe6989b8aea4941b7ef7adc67b89ff403020947' -> '/data/nn/nlp/wikipedia/20200501.de/1.0.0.incomplete/dataset_info.json'
```


Fix bertscore references

I added some type checking for metrics. There was an issue where a metric could interpret a string a a list. A `ValueError` is raised if a string is given instead of a list.

Moreover I added support for both strings and lists of strings for `references` in `bertscore`, as it is the case in the original code.

Both ways work:
```
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, [lg])
score = scorer.compute(lang="en")
```

```
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, lg)
score = scorer.compute(lang="en")
```

This should fix #295 and #238


remove some print in snli file

I guess you can just rebase from master to fix the CI

This PR removes unwanted  `print` statements in some files such as `snli.py`


Add searchable datasets

Looks very cool! Only looked at it superficially though

Alright I think I've checked all your comments, thanks :)

Moreover I just added a way to serialize faiss indexes.
This is important because for big datasets the index construction can take some time.

Examples:

```python
ds = nlp.load_dataset('crime_and_punish', split='train')
ds_with_embeddings = ds.map(lambda example: {'embeddings': embed(example['line']}))
ds_with_embeddings.add_faiss_index(column='embeddings')
# query
scores, retrieved_examples = ds_with_embeddings.get_nearest_examples('embeddings', embed('my new query'), k=10)
# save index
ds_with_embeddings.get_index('embeddings').save('my_index.faiss')
```

```python
ds = nlp.load_dataset('crime_and_punish', split='train')
# load index
faiss_index = nlp.search.FaissIndex.load('my_index.faiss')
ds.add_faiss_index('embeddings', faiss_index=faiss_index)
# query
scores, retrieved_examples = ds.get_nearest_examples('embeddings', embed('my new query'), k=10)
```

Let me know what you think

Nice!

Here are a few comments:

I think it would be good to separate (1) the name of the column we use for indexing and (2) the name of the index itself, at least in our head. As I understand it, once the index is created, the column we used to create it is irrelevant so the column name will only be relevant in the `add_faiss_index` and we should be able to supply a different index name, e.g. `my_faiss_index`. When we reload an index, we don't really care about the column that was used to create it, right? so it's maybe better to have an `index_name` (which default to the column name for a simple user experience but it can also be something else and this should be clear in our head when we define the API).

I'm wondering if we should not have a triple of methods for each retrieval engine: `add_xxx_index`, `save_xxx_index` and `load_xxx_index` when `xxx` can be `faiss` or `elasticsearch`. I'm not a fan of exposing `nlp.search.FaissIndex` unless you think there is a strong reason to have the user learn this abstraction.

Last but not least, I think we should already think about hosting index on our S3. I would maybe go for something like this: host the index serialized with the cached dataset on user-provided namespaces:
```python
wiki_indexed = load_dataset('thom/wiki_indexed_with_dpr_faiss')
```

I agree, I just changed to using `index_name` and having add/save/load methods

To summarize:

```python
ds = nlp.load_dataset('crime_and_punish', split='train')
ds_with_embeddings = ds.map(lambda example: {'embeddings': embed(example['line']}))
ds_with_embeddings.add_faiss_index(column='embeddings')
# query
scores, retrieved_examples = ds_with_embeddings.get_nearest_examples('embeddings', embed('my new query'), k=10)
# save index
ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss')
```

```python
ds = nlp.load_dataset('crime_and_punish', split='train')
# load index
ds.load_faiss_index('embeddings', 'my_index.faiss')
# query
scores, retrieved_examples = ds.get_nearest_examples('embeddings', embed('my new query'), k=10)
```

Good to me. I understand that for now there is no check that the index matches the dataset on loading.
Maybe just add a basic test on the number of examples?

Looks like the CI is having troubles to pass because of `tests/test_dataset_common.py::AWSDatasetTest::test_builder_configs_{<insert_rando_dataset_name_here>}`, `requests.exceptions.ConnectionError` :/

# Better support for Numpy format + Add Indexed Datasets

I was working on adding Indexed Datasets but in the meantime I had to also add more support for Numpy arrays in the lib.

## Better support for Numpy format

New features:
- New fast method to convert Numpy arrays from Arrow structure (up to x100 speed up) using Pandas.
- Allow to output Numpy arrays in batched `.map`, which was the only missing part to fully support Numpy arrays.

Pandas offers fast zero-copy Numpy arrays conversion from Arrow structures.
Using it we can speed up the reading of memory-mapped Numpy array stored in Arrow format.

With these changes you can easily compute embeddings of texts using `.map()`. For example:
```python
def embed(text):
    tokenized_example = tokenizer.encode(text, return_tensors="pt")
    embeddings = bert_encoder(tokenized_examples).numpy()
    return embeddings
dset_with_embeddings = dset.map(lambda example: {"embeddings": embed(example["text])})
```
And then reading the embeddings from the arrow format is be very fast.

PS1: Note that right now only 1d arrays are supported.
PS2: It seems possible to do without pandas but it will require more _trickery_.
PS3: I did a simple benchmark with google colab that you can view here:
https://colab.research.google.com/drive/1QlLTR6LRwYOKGJ-hTHmHyolE3wJzvfFg?usp=sharing

## Add Indexed Datasets

For many retrieval tasks it is convenient to index a dataset to be able to run fast queries.
For example for models like DPR, REALM, RAG etc. that are models for Open Domain QA, the retrieval step is very important.

Therefore I added two ways to add an index to a column of a dataset:
1) You can index it using a Dense Index like Faiss. It is used to index vectors.
    Faiss is a library for efficient similarity search and clustering of dense vectors.
    It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM.
2) You can index it using a Sparse Index like Elasticsearch. It is used to index text and run queries based on BM25 similarity.

Example of usage:

```python
ds = nlp.load_dataset('crime_and_punish', split='train')
ds_with_embeddings = ds.map(lambda example: {'embeddings': embed(example['line']}))  # `embed` outputs a `np.array`
ds_with_embeddings.add_vector_index(column='embeddings')
scores, retrieved_examples = ds_with_embeddings.get_nearest(column='embeddings', query=embed('my new query'), k=10)
```

```python
ds = nlp.load_dataset('crime_and_punish', split='train')
es_client = elasticsearch.Elasticsearch()
ds.add_text_index(column='line', es_client=es_client, index_name="my_es_index")
scores, retrieved_examples = ds.get_nearest(column='line', query='my new query', k=10)
```

PS4: Faiss allows to specify many options for the [index](https://github.com/facebookresearch/faiss/wiki/The-index-factory) and for [GPU settings](https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU). I made sure that the user has full control over those settings.

## Tests

I added tests for Faiss, Elasticsearch and indexed datasets.
I had to edit the CI config because all the test scripts were not being run by CircleCI.

------------------

I'd be really happy to have some feedbacks :)


Error in Demo for Specific Datasets

Thanks for reporting these errors :)

I can actually see two issues here.

First, datasets like `natural_questions` require apache_beam to be processed. Right now the import is not at the right place so we have this error message. However, even the imports are fixed, the nlp viewer doesn't actually have the resources to process NQ right now so we'll have to wait until we have a version that we've already processed on our google storage (that's what we've done for wikipedia for example).

Second, datasets like `newsroom` require manual downloads as we're not allowed to redistribute the data ourselves (if I'm not wrong). An error message should be displayed saying that we're not allowed to show the dataset.

I can fix the first issue with the imports but for the second one I think we'll have to see with @srush to show a message for datasets that require manual downloads (it can be checked whether a dataset requires manual downloads if `dataset_builder_instance.manual_download_instructions is not None`).

I added apache-beam to the viewer. We can think about how to add newsroom.

We don't plan to host the source files of newsroom ourselves for now.
You can still get the dataset if you follow the download instructions given by `dataset = load_dataset('newsroom')` though.
The viewer also shows the instructions now.

Closing this one. If you have other questions, feel free to re-open :)

Selecting `natural_questions` or `newsroom` dataset in the online demo results in an error similar to the following.

![image](https://user-images.githubusercontent.com/60150701/85347842-ac861900-b4ae-11ea-98c4-a53a00934783.png)


snli -1 labels

@jxmorris12  , we use `-1` to label examples for which `gold label`  is missing (`gold label = -` in the original dataset).

Thanks @mariamabarham! so the original dataset is missing some labels? That is weird. Is standard practice just to discard those examples training/eval?

Yes the original dataset is missing some labels maybe  @sleepinyourhat , @gangeli can correct me if I'm wrong
For my personal opinion at least if you want your model to learn to predict no answer (-1) you can leave it their but otherwise you can discard them.

I'm trying to train a model on the SNLI dataset. Why does it have so many -1 labels?
```
import nlp
from collections import Counter
data = nlp.load_dataset('snli')['train']
print(Counter(data['label']))
Counter({0: 183416, 2: 183187, 1: 182764, -1: 785})
```


Improve input warning for evaluation metrics

Hi,

I am the author of `bert_score`. Recently, we received [ an issue ](https://github.com/Tiiiger/bert_score/issues/62) reporting a problem in using `bert_score` from the `nlp` package (also see #238 in this repo).  After looking into this, I realized that the problem arises from the format `nlp.Metric` takes input.

Here is a minimal example:
```python
import nlp

scorer = nlp.load_metric("bertscore")
with open("pred.txt") as p, open("ref.txt") as g:
    for lp, lg in zip(p, g):
        scorer.add(lp, lg)
score = scorer.compute(lang="en")
```

The problem in the above code is that `scorer.add()` expects a list of strings as input for the references. As a result, the `scorer` here would take a list of characters in `lg` to be the references. The correct implementation would be calling
```python
scorer.add(lp, [lg])
```

I just want to raise this issue to you to prevent future user errors of a similar kind. I assume some simple type checking can prevent this from happening?

Thanks!


Cannot load arxiv dataset on MacOS?

dataset bug

A bug in a dataset script provided in the library

I couldn't replicate this issue on my macbook :/
Could you try to play with different encodings in `with open(path, encoding=...) as f` in scientific_papers.py:L108 ?

I was able to track down the file causing the problem by adding the following to `scientific_papers.py` (starting at line 116):

```python
                from json import JSONDecodeError
                try:
                    d = json.loads(line)
                    summary = "\n".join(d["abstract_text"])
                except JSONDecodeError:
                    print(path, line)
```

For me it was at: `/Users/johngiorgi/.cache/huggingface/datasets/f87fd498c5003cbe253a2af422caa1e58f87a4fd74cb3e67350c635c8903b259/arxiv-dataset/train.txt` with `"article_id": "1407.3051"`.

Not really 100% sure at the moment, but it looks like this specific substring from `"article_text"` may be causing the problem?

```
"after the missing - mass scale adjustment , the validity of the corrections was tested in the @xmath85 productions at 1.69 gev/@xmath1 . in fig .  [", "fig : calibrations ] ( a ) , we show the missing - mass spectrum in the @xmath86 region in the @xmath87 reaction at 1.69 gev/@xmath1 . a fitting result with a lorentzian function for the @xmath86  ( dashed line ) and the three - body phas
```

perhaps because it appears to be truncated. I (think) I can recreate the problem by doing the following:

```python
import json

# A minimal example of the json file that causes the error
invalid_json = '{"article_id": "1407.3051", "article_text": ["the missing - mass resolution was obtained to be 2.8 @xmath3 0.1  mev/@xmath4  ( fwhm ) , which corresponds to the missing - mass resolution of 3.2  @xmath3  0.2  mev/@xmath4  ( fwhm ) at the @xmath6 cusp region in the @xmath0 reaction .", "this resolution is at least by a factor of 2 better than the previous measurement with the same reaction ( 3.2@xmath595.5 mev/@xmath4 in @xmath84 )  @xcite .", "after the missing - mass scale adjustment , the validity of the corrections was tested in the @xmath85 productions at 1.69 gev/@xmath1 . in fig .  [", "fig : calibrations ] ( a ) , we show the missing - mass spectrum in the @xmath86 region in the @xmath87 reaction at 1.69 gev/@xmath1 . a fitting result with a lorentzian function for the @xmath86  ( dashed line ) and the three - body phas'
# The line of code from `scientific_papers.py` which appears to cause the error
json.loads(invalid_json)
```

This is as far as I get before I am stumped.

I just checked inside `train.txt` and this line isn't truncated for me (line 163577).
Could you try to clear your cache and re-download the dataset ?

Ah the turn-it-off-turn-it-on again solution! That did it, thanks a lot :)

I am having trouble loading the `"arxiv"` config from the `"scientific_papers"` dataset on MacOS. When I try loading the dataset with:

```python
arxiv = nlp.load_dataset("scientific_papers", "arxiv")
```

I get the following stack trace:

```bash
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-2-8e00c55d5a59> in <module>
----> 1 arxiv = nlp.load_dataset("scientific_papers", "arxiv")

~/miniconda3/envs/t2t/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526

~/miniconda3/envs/t2t/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    430                 verify_infos = not save_infos and not ignore_verifications
    431                 self._download_and_prepare(
--> 432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    433                 )
    434                 # Sync info

~/miniconda3/envs/t2t/lib/python3.7/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    481             try:
    482                 # Prepare split will record examples associated to the split
--> 483                 self._prepare_split(split_generator, **prepare_split_kwargs)
    484             except OSError:
    485                 raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))

~/miniconda3/envs/t2t/lib/python3.7/site-packages/nlp/builder.py in _prepare_split(self, split_generator)
    662
    663         generator = self._generate_examples(**split_generator.gen_kwargs)
--> 664         for key, record in utils.tqdm(generator, unit=" examples", total=split_info.num_examples, leave=False):
    665             example = self.info.features.encode_example(record)
    666             writer.write(example)

~/miniconda3/envs/t2t/lib/python3.7/site-packages/tqdm/std.py in __iter__(self)
   1106                 fp_write=getattr(self.fp, 'write', sys.stderr.write))
   1107
-> 1108         for obj in iterable:
   1109             yield obj
   1110             # Update and possibly print the progressbar.

~/miniconda3/envs/t2t/lib/python3.7/site-packages/nlp/datasets/scientific_papers/107a416c0e1958cb846f5934b5aae292f7884a5b27e86af3f3ef1a093e058bbc/scientific_papers.py in _generate_examples(self, path)
    114                 # "section_names": list[str], list of section names.
    115                 # "sections": list[list[str]], list of sections (list of paragraphs)
--> 116                 d = json.loads(line)
    117                 summary = "\n".join(d["abstract_text"])
    118                 # In original paper, <S> and </S> are not used in vocab during training

~/miniconda3/envs/t2t/lib/python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    346             parse_int is None and parse_float is None and
    347             parse_constant is None and object_pairs_hook is None and not kw):
--> 348         return _default_decoder.decode(s)
    349     if cls is None:
    350         cls = JSONDecoder

~/miniconda3/envs/t2t/lib/python3.7/json/decoder.py in decode(self, s, _w)
    335
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

~/miniconda3/envs/t2t/lib/python3.7/json/decoder.py in raw_decode(self, s, idx)
    351         """
    352         try:
--> 353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
    355             raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Unterminated string starting at: line 1 column 46983 (char 46982)

163502 examples [02:10, 2710.68 examples/s]
```

I am not sure how to trace back to the specific JSON file that has the "Unterminated string". Also, I do not get this error on colab so I suspect it may be MacOS specific. Copy pasting the relevant lines from `transformers-cli env` below:

- Platform: Darwin-19.5.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)

Any ideas?


Don't test community datasets

This PR disables testing for community datasets on aws.

It should fix the CI that is currently failing.


Update metadata for x_stance dataset

Great! Thanks @jvamvas for these updates.

I have fixed a warning. The remaining test failure is due to an unrelated dataset.

We just fixed the other dataset on master. Could you rebase from master and push to rerun the CI ?

Thank you for featuring the x_stance dataset in your library. This PR updates some metadata:
- Citation: Replace preprint with proceedings
- URL: Use a URL with long-term availability


break statement not required

I guess,test failing due to connection error?

If I'm not wrong this function returns None if no main class was found.
I think it makes things less clear not to have a return at the end of the function.
I guess we can have one return in the for loop instead of the break statement, AND one return at the end to explicitly return None.
What do you think ?


ConnectionError - Eli5 dataset download

It should ne fixed now, thanks for reporting this one :)
It was an issue on our google storage.

Let me now if you're still facing this issue.

Hi, I have a problem with downloading Eli5 dataset. When typing `nlp.load_dataset('eli5')`, I get ConnectionError: Couldn't reach https://storage.googleapis.com/huggingface-nlp/cache/datasets/eli5/LFQA_reddit/1.0.0/explain_like_im_five-train_eli5.arrow

I would appreciate if you could help me with this issue.


update xsum

Looks cool!
@mariamabarham can you add a detailed description here what exactly is changed and how the user can load xsum now?

This is a super useful PR :-) @sshleifer - maybe you can take a look at the updated version of xsum if you can use it for your use case. Now, one should be able to just load it with:

```python
nlp.load_datasets("xsum", ....)  # no manual dir required anymore
```

This PR makes the following update to the xsum dataset:

- Manual download is not required anymore

- dataset can be loaded as follow: `nlp.load_dataset('xsum')`

**Important**
Instead of using on outdated url to download the data:  "https://raw.githubusercontent.com/EdinburghNLP/XSum/master/XSum-Dataset/XSum-TRAINING-DEV-TEST-SPLIT-90-5-5.json"

a more up-to-date url stored here: https://s3.amazonaws.com/datasets.huggingface.co/summarization/xsum.tar.gz is used
, so that the user does not need to manually download the data anymore.
There might be slight breaking changes here for xsum.


Error at the first example in README: AttributeError: module 'dill' has no attribute '_dill'

It looks like the bug comes from `dill`. Which version of `dill` are you using ?

Thank you. It is version 0.2.6, which version is better?

0.2.6 is three years old now, maybe try a more recent one, e.g. the current 0.3.2 if you can?

Thanks guys! I upgraded dill and it works.

/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:469: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:470: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:471: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:472: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:473: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:476: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/Users/parasol_tree/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
  return f(*args, **kwds)
/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "/Users/parasol_tree/Resource/019 - Github/AcademicEnglishToolkit /test.py", line 7, in <module>
    import nlp
  File "/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/nlp/__init__.py", line 27, in <module>
    from .arrow_dataset import Dataset
  File "/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/nlp/arrow_dataset.py", line 31, in <module>
    from nlp.utils.py_utils import dumps
  File "/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/nlp/utils/__init__.py", line 20, in <module>
    from .download_manager import DownloadManager, GenerateMode
  File "/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/nlp/utils/download_manager.py", line 25, in <module>
    from .py_utils import flatten_nested, map_nested, size_str
  File "/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/nlp/utils/py_utils.py", line 244, in <module>
    class Pickler(dill.Pickler):
  File "/Users/parasol_tree/anaconda3/lib/python3.6/site-packages/nlp/utils/py_utils.py", line 247, in Pickler
    dispatch = dill._dill.MetaCatchingDict(dill.Pickler.dispatch.copy())
AttributeError: module 'dill' has no attribute '_dill'


fix squad_v2 metric

Fix #280
The imports were wrong


Add ANLI dataset.

Awesome!! Thanks @easonnie.
Let's wait for additional reviews maybe from @lhoestq @patrickvonplaten @jplu

I completed all the steps in https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset and push the code for ANLI. Please let me know if there are any errors.


Consistent formatting of citations


Fix manual download instructions

But I get
```python
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py
```
When I try from jupyter on brutasse or my mac. (the jupyter server is run from transformers).

Both machines can run
```bash
aws s3 ls s3://datasets.huggingface.co/nlp/datasets/wmt16/
```
but it seems one must be in the nlp directory to run the command?

(I ran `pip install -e . ` on this branch in both situations.)

`https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py` looks very weird.

(Also, S3 is not a file-system, it's a flat key-value store)

> But I get
>
> ```python
> ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py
> ```
>
> When I try from jupyter on brutasse or my mac. (the jupyter server is run from transformers).
>
> Both machines can run
>
> ```shell
> aws s3 ls s3://datasets.huggingface.co/nlp/datasets/wmt16/
> ```
>
> but it seems one must be in the nlp directory to run the command?
>
> (I ran `pip install -e . ` on this branch in both situations.)

As soon as it is on master, the dataset script wmt16.py will be synced on S3 and you'll be able to do `load_dataset("wmt16")`

This PR replaces the static `DatasetBulider` variable `MANUAL_DOWNLOAD_INSTRUCTIONS` by a property function `manual_download_instructions()`.

Some datasets like XTREME and all WMT need the manual data dir only for a small fraction of the possible configs.

After some brainstorming with @mariamabarham and @lhoestq, we came to the conclusion that having a property function `manual_download_instructions()` gives us more flexibility to decide on a per config basis in the dataset builder if manual download instructions are needed.

Also this PR should unblock solves a bug with `wmt16 - ro-en`
@sshleifer from this branch you should be able to succesfully run

```python
import nlp
ds = nlp.load_dataset('./datasets/wmt16', 'ro-en')
```

and once this PR is merged S3 should be synched so that

```python
import nlp
ds = nlp.load_dataset("wmt16", "ro-en")
```

works as well.

**Important**: Since `MANUAL_DOWNLOAD_INSTRUCTIONS` was not really exposed to the user, this PR should not be a problem regarding backward compatibility.


Consistent formatting of citations

The citations are all of a different format, some have "```" and have text inside, others are proper bibtex.

Can we make it so that they all are proper citations, i.e. parse by the bibtex spec:

https://bibtexparser.readthedocs.io/en/master/


Update dataset_info from gcs

Some datasets are hosted on gcs (wikipedia for example). In this PR I make sure that, when a user loads such datasets, the file_instructions are built using the dataset_info.json from gcs and not from the info extracted from the local `dataset_infos.json` (the one that contain the info for each config). Indeed local files may end up outdated.

Furthermore, to avoid outdated dataset_infos.json, I now make sure that each time you run `load_dataset` it also tries to update the file locally.


Private/sensitive data

Hi @MFreidank, you should already be able to load a dataset from local sources, indeed. (ping @lhoestq and @jplu)

We're also thinking about the ability to host private datasets on a hosted bucket with permission management, but that's further down the road.

Hi @MFreidank, it is possible to load a dataset from your local storage, but only CSV/TSV and JSON are supported. To load a dataset in JSON format:

```
nlp.load_dataset(path="json", data_files={nlp.Split.TRAIN: ["path/to/train.json"], nlp.Split.TEST: ["path/to/test.json"]})
```

For CSV/TSV datasets, you have to replace `json` by `csv`.

Hi @julien-c @jplu,
Thanks for sharing this solution with me, it helps, this is what I was looking for.
If not already there and only missed by me, this could be a great addition in the docs.

Closing my issue as resolved, thanks again.

Hi all,
Thanks for this fantastic library, it makes it very easy to do prototyping for NLP projects interchangeably between TF/Pytorch.

Unfortunately, there is data that cannot easily be shared publicly as it may contain sensitive information.
Is there support/a plan to support such data with NLP, e.g. by reading it from local sources?

Use case flow could look like this: use NLP to prototype an approach on similar, public data and apply the resulting prototype on sensitive/private data without the need to rethink data processing pipelines.

Many thanks for your responses ahead of time and kind regards,
MFreidank


Error with SquadV2 Metrics

I can't seem to import squad v2 metrics.

**squad_metric = nlp.load_metric('squad_v2')**

**This throws me an error.:**

```
ImportError                               Traceback (most recent call last)
<ipython-input-8-170b6a170555> in <module>
----> 1 squad_metric = nlp.load_metric('squad_v2')

~/env/lib64/python3.6/site-packages/nlp/load.py in load_metric(path, name, process_id, num_process, data_dir, experiment_id, in_memory, download_config, **metric_init_kwargs)
    426     """
    427     module_path = prepare_module(path, download_config=download_config, dataset=False)
--> 428     metric_cls = import_main_class(module_path, dataset=False)
    429     metric = metric_cls(
    430         name=name,

~/env/lib64/python3.6/site-packages/nlp/load.py in import_main_class(module_path, dataset)
     55     """
     56     importlib.invalidate_caches()
---> 57     module = importlib.import_module(module_path)
     58
     59     if dataset:

/usr/lib64/python3.6/importlib/__init__.py in import_module(name, package)
    124                 break
    125             level += 1
--> 126     return _bootstrap._gcd_import(name[level:], package, level)
    127
    128

/usr/lib64/python3.6/importlib/_bootstrap.py in _gcd_import(name, package, level)

/usr/lib64/python3.6/importlib/_bootstrap.py in _find_and_load(name, import_)

/usr/lib64/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/usr/lib64/python3.6/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib64/python3.6/importlib/_bootstrap_external.py in exec_module(self, module)

/usr/lib64/python3.6/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

~/env/lib64/python3.6/site-packages/nlp/metrics/squad_v2/a15e787c76889174874386d3def75321f0284c11730d2a57e28fe1352c9b5c7a/squad_v2.py in <module>
     16
     17 import nlp
---> 18 from .evaluate import evaluate
     19
     20 _CITATION = """\

ImportError: cannot import name 'evaluate'
```


Dataset Preprocessing Cache with .map() function not working as expected

When you're processing a dataset with `.map`, it checks whether it has already done this computation using a hash based on the function and the input (using some fancy serialization with `dill`). If you found that it doesn't work as expected in some cases, let us know !

Given that, you can still force to re-process using `.map(my_func, load_from_cache_file=False)` if you want to.

I am curious about the problem you have with splits. It makes me think about #160 that was an issue of version 0.1.0. What version of `nlp` are you running ? Could you give me more details ?

Thanks, that's helpful! I was running 0.1.0, but since upgraded to 0.2.1. I can't reproduce the issue anymore as I've cleared the cache & everything now seems to be running fine since the upgrade. I've added some checks to my code, so if I do encounter it again I will reopen this issue.

Just checking in, the cache sometimes still does not work when I make changes in my processing function in version `1.2.1`. The changes made to my data processing function only propagate to the dataset when I use `load_from_cache_file=False` or clear the cache. Is this a system-specific issue?

Hi @sarahwie
The data are reloaded from the cache if the hash of the function you provide is the same as a computation you've done before. The hash is computed by recursively looking at the python objects of the function you provide.

If you think there's an issue, can you share the function you used or a google colab please ?

I can't reproduce it, so I'll close for now.

I've been having issues with reproducibility when loading and processing datasets with the `.map` function. I was only able to resolve them by clearing all of the cache files on my system.

Is there a way to disable using the cache when processing a dataset? As I make minor processing changes on the same dataset, I want to be able to be certain the data is being re-processed rather than loaded from a cached file.

Could you also help me understand a bit more about how the caching functionality is used for pre-processing? E.g. how is it determined when to load from a cache vs. reprocess.
I was particularly having an issue where the correct dataset splits were loaded, but as soon as I applied the `.map()` function to each split independently, they somehow all exited this process having been converted to the test set.
Thanks!


MemoryError when loading German Wikipedia

Hi !

As you noticed, "big" datasets like Wikipedia require apache beam to be processed.
However users usually don't have an apache beam runtime available (spark, dataflow, etc.) so our goal for this library is to also make available processed versions of these datasets, so that users can just download and use them right away.

This is the case for english and french wikipedia right now: we've processed them ourselves and now they are available from our google storage. However we've not processed the german one (yet).

Hi @lhoestq

Thank you for your quick reply. I thought this might be the case, that the processing was done for some languages and not for others. Is there any set timeline for when other languages (German, Italian) will be processed?

Given enough memory, is it possible to process the data ourselves by specifying the `beam_runner`?

Adding them is definitely in our short term objectives. I'll be working on this early next week :)

Although if you have an apache beam runtime feel free to specify the beam runner. You can find more info [here](https://github.com/huggingface/nlp/blob/master/docs/beam_dataset.md) on how to make it work on Dataflow but you can adapt it for Spark or any other beam runtime (by changing the `runner`).

However if you don't have a beam runtime and even if you have enough memory, I discourage you to use the `DirectRunner` on the german or italian wikipedia. According to Apache Beam documentation it was made for testing purposes and therefore it is memory-inefficient.

I added the German and the Italian Wikipedia to our google cloud storage:
First update the `nlp` package to 0.3.0:
```bash
pip install nlp --upgrade
```
and then
```python
from nlp import load_dataset
wiki_de = load_dataset("wikipedia", "20200501.de")
wiki_it = load_dataset("wikipedia", "20200501.it")
```
The datasets are downloaded and directly ready to use (no processing).

Hi @lhoestq

Wow, thanks so much, that's **really** incredible! I was considering looking at creating my own Beam Dataset, as per the doc you linked, but instead opted to process the data myself using `wikiextractor`. However, now that this is available, I'll definitely switch across and use it.

Thanks so much for the incredible work, this really helps out our team considerably!

Have a great (and well-deserved ;) weekend ahead!

P.S. I'm not sure if I should close the issue here - if so I'm happy to do so.

Thanks for your message, glad I could help :)
Closing this one.

Hi, first off let me say thank you for all the awesome work you're doing at Hugging Face across all your projects (NLP, Transformers, Tokenizers) - they're all amazing contributions to us working with NLP models :)

I'm trying to download the German Wikipedia dataset as follows:

```
wiki = nlp.load_dataset("wikipedia", "20200501.de", split="train")
```

However, when I do so, I get the following error:

```
Downloading and preparing dataset wikipedia/20200501.de (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/ubuntu/.cache/huggingface/datasets/wikipedia/20200501.de/1.0.0...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/load.py", line 520, in load_dataset
    save_infos=save_infos,
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/builder.py", line 433, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/builder.py", line 824, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
	`load_dataset('wikipedia', '20200501.de', beam_runner='DirectRunner')`
```

So, following on from the example usage at the bottom, I tried specifying `beam_runner='DirectRunner`, however when I do this after about 20 min after the data has all downloaded, I get a `MemoryError` as warned.

This isn't an issue for the English or French Wikipedia datasets (I've tried both), as neither seem to require that `beam_runner` be specified. Can you please clarify why this is an issue for the German dataset?

My nlp version is 0.2.1.

Thank you!


Empty samples in glue/qqp

We are only wrapping the original dataset.

Maybe try to ask on the GLUE mailing list or reach out to the original authors?

Tanks for the suggestion, I'll try to ask GLUE benchmark.
I'll first close the issue,  post the following up here afterwards, and reopen the issue if needed.

```
qqp = nlp.load_dataset('glue', 'qqp')
print(qqp['train'][310121])
print(qqp['train'][362225])
```
```
{'question1': 'How can I create an Android app?', 'question2': '', 'label': 0, 'idx': 310137}
{'question1': 'How can I develop android app?', 'question2': '', 'label': 0, 'idx': 362246}
```
Notice that question 2 is empty string.
BTW, I have checked and these two are the only naughty ones in all splits of qqp.


Fix metric compute (original_instructions missing)

Awesome! This is working now:

```python
import nlp
seqeval = nlp.load_metric("seqeval")
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]

results = seqeval.compute(y_true, y_pred)
```

I heavily need this fix for an upcoming `nlp` integration PR for Transformers (token classification example)  😅

Haha nice ! We'll ship this fix with the next release that will probably come out on thursday :)

When loading arrow data we added in cc8d250 a way to specify the instructions that were used to store them with the loaded dataset.
However metrics load data the same way but don't need instructions (we use one single file).

In this PR I just make `original_instructions` optional when reading files to load a `Dataset` object.

This should fix #269


NonMatchingChecksumError when loading pubmed dataset

For some reason the files are not available for unauthenticated users right now (like the download service of this package). Instead of downloading the right files, it downloads the html of the error.
According to the error it should be back again in 24h.

![image](https://user-images.githubusercontent.com/42851186/84751599-096c6580-afbd-11ea-97f3-ee4aef791711.png)

I get this error when i run `nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')`.
The error is:

```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-2-7742dea167d0> in <module>()
----> 1 df = nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')
      2 df = pd.DataFrame(df)
      3 gc.collect()

3 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    431                 verify_infos = not save_infos and not ignore_verifications
    432                 self._download_and_prepare(
--> 433                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    434                 )
    435                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    468         # Checksums verification
    469         if verify_infos:
--> 470             verify_checksums(self.info.download_checksums, dl_manager.get_recorded_sizes_checksums())
    471         for split_generator in split_generators:
    472             if str(split_generator.split_info.name).lower() == "all":

/usr/local/lib/python3.6/dist-packages/nlp/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums)
     34     bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
     35     if len(bad_urls) > 0:
---> 36         raise NonMatchingChecksumError(str(bad_urls))
     37     logger.info("All the checksums matched successfully.")
     38

NonMatchingChecksumError: ['https://drive.google.com/uc?id=1b3rmCSIoh6VhD4HKWjI4HOW-cSwcwbeC&export=download', 'https://drive.google.com/uc?id=1lvsqvsFi3W-pE1SqNZI0s8NR9rC1tsja&export=download']
```
I'm currently working on google colab.

That is quite strange because yesterday it was fine.


PG-19

dataset request

Requesting to add a new dataset

Sounds good! Do you want to give it a try?

Ok, I'll see if I can figure it out tomorrow!

Got around to this today, and so far so good, I'm able to download and load pg19 locally. However, I think there may be an issue with the dummy data, and testing in general.

The problem lies in the fact that each book from pg19 actually resides as its own text file in a google cloud folder that denotes the split, where the book id is the name of the text file. https://console.cloud.google.com/storage/browser/deepmind-gutenberg/train/ I don't believe there's anywhere else (even in the supplied metadata), where the mapping of id -> split can be found.

Therefore I end up making a network call `tf.io.gfile.listdir` to get all the files within each of the split directories. https://github.com/lucidrains/nlp/commit/adbacbd85decc80db2347d0882e7dab4faa6fd03#diff-cece8f166a85dd927caf574ba303d39bR78

Does this network call need to be eventually stubbed out for testing?

Ohh nevermind, I think I can use `download_custom` here with `listdir` as the custom function. Ok, I'll keep trying to make the dummy data work!

Hi, and thanks for all your open-sourced work, as always!

I was wondering if you would be open to adding PG-19 to your collection of datasets. https://github.com/deepmind/pg19 It is often used for benchmarking long-range language modeling.


update cos_e to add cos_e v1.0

This PR updates the cos_e dataset to add  v1.0 as requested here #163
@nazneenrajani


asd


Fix allociné dataset configuration

Actually when there is only one configuration, then you don't need to specify the configuration in `load_dataset`. You can run:
```python
dataset = load_dataset('allocine')
```
and it works.

Maybe we should take that into account in the nlp viewer @srush ?

@lhoestq Just to understand the exact semantics. Are you suggesting that if there is exactly 1 configuration I should not show the configuration menu and just treat it as if there were 0 configurations?

The configuration menu is fine imo.
It was more about the code snippet presented in the viewer.
For example for Allociné it currently shows this snippet to load the dataset:
```python
!pip install nlp
from nlp import load_dataset
dataset = load_dataset('allocine', 'allocine')
```
However for datasets with one or zero configurations, the second argument in `load_dataset` is optional. For Allociné, that has one configuration, we can expect to show instead:
```python
!pip install nlp
from nlp import load_dataset
dataset = load_dataset('allocine')
```

> Actually when there is only one configuration, then you don't need to specify the configuration in `load_dataset`. You can run:
>
> ```python
> dataset = load_dataset('allocine')
> ```
>
> and it works.
>
> Maybe we should take that into account in the nlp viewer @srush ?

Oh ok, I didn't expect it would work!

Anyway, I think it's intrinsically better to simply remove the optional parameter.
The dummy data folder architecture seems also more logical this way.

Fixed in the viewer. Checked that allocine works.

This is a patch for #244. According to the [live nlp viewer](url), the Allociné dataset must be loaded with :
```python
dataset = load_dataset('allocine', 'allocine')
```
This is redundant, as there is only one "dataset configuration", and should only be:
```python
dataset = load_dataset('allocine')
```

This is my mistake, because the code for [`allocine.py`](https://github.com/huggingface/nlp/blob/master/datasets/allocine/allocine.py) was inspired by [`imdb.py`](https://github.com/huggingface/nlp/blob/master/datasets/imdb/imdb.py), which also force the user to specify the "dataset configuration" (even if there is only one).

I believe this PR should solve this issue, making the Allociné dataset more convenient to use.