How to reuse functionality of a (generic) dataset?

Hi @ArneBinder, we have a few "generic" datasets which are intended to load data files with a predefined format:
- csv: https://github.com/huggingface/nlp/tree/master/datasets/csv
- json: https://github.com/huggingface/nlp/tree/master/datasets/json
- text: https://github.com/huggingface/nlp/tree/master/datasets/text

You can find more details about this way to load datasets here in the documentation: https://huggingface.co/nlp/loading_datasets.html#from-local-files

Maybe your brat loading script could be shared in a similar fashion?

> Maybe your brat loading script could be shared in a similar fashion?

@thomwolf that was also my first idea and I think I will tackle that in the next days. I separated the code and created a real abstract class `AbstractBrat` to allow to inherit from that (I've just seen that the dataset_loader loads the first non abstract class), now `Brat` is very similar in its functionality to https://github.com/huggingface/nlp/tree/master/datasets/text but inherits from `AbstractBrat`.

However, it is still not clear to me how to add a specific dataset (as explained in https://huggingface.co/nlp/add_dataset.html) to your repo that uses this format/abstract class, i.e. re-using the `features` entry of the  `DatasetInfo` object and `_generate_examples()`. Again, by doing so, the only remaining entries/functions to define would be `_DESCRIPTION`, `_CITATION`, `homepage` and `_URL` (which is all copy-paste stuff) and `_split_generators()`.

In a lack of better ideas, I tried sth like below, but of course it does not work outside `nlp` (`AbstractBrat` is currently defined in [datasets/brat.py](https://github.com/ArneBinder/nlp/blob/5e81fb8710546ee7be3353a7f02a3045e9a8351e/datasets/brat/brat.py)):
```python
from __future__ import absolute_import, division, print_function

import os

import nlp

from datasets.brat.brat import AbstractBrat

_CITATION = """
@inproceedings{lauscher2018b,
  title = {An argument-annotated corpus of scientific publications},
  booktitle = {Proceedings of the 5th Workshop on Mining Argumentation},
  publisher = {Association for Computational Linguistics},
  author = {Lauscher, Anne and Glava\v{s}, Goran and Ponzetto, Simone Paolo},
  address = {Brussels, Belgium},
  year = {2018},
  pages = {40–46}
}
"""

_DESCRIPTION = """\
This dataset is an extension of the Dr. Inventor corpus (Fisas et al., 2015, 2016) with an annotation layer containing
fine-grained argumentative components and relations. It is the first argument-annotated corpus of scientific
publications (in English), which allows for joint analyses of argumentation and other rhetorical dimensions of
scientific writing.
"""

_URL = "http://data.dws.informatik.uni-mannheim.de/sci-arg/compiled_corpus.zip"

class Sciarg(AbstractBrat):

    VERSION = nlp.Version("1.0.0")

    def _info(self):

        brat_features = super()._info().features
        return nlp.DatasetInfo(
            # This is the description that will appear on the datasets page.
            description=_DESCRIPTION,
            # nlp.features.FeatureConnectors
            features=brat_features,
            # If there's a common (input, target) tuple from the features,
            # specify them here. They'll be used if as_supervised=True in
            # builder.as_dataset.
            #supervised_keys=None,
            # Homepage of the dataset for documentation
            homepage="https://github.com/anlausch/ArguminSci",
            citation=_CITATION,
        )

    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""
        # TODO: Downloads the data and defines the splits
        # dl_manager is a nlp.download.DownloadManager that can be used to
        # download and extract URLs
        dl_dir = dl_manager.download_and_extract(_URL)
        data_dir = os.path.join(dl_dir, "compiled_corpus")
        print(f'data_dir: {data_dir}')
        return [
            nlp.SplitGenerator(
                name=nlp.Split.TRAIN,
                # These kwargs will be passed to _generate_examples
                gen_kwargs={
                    "directory": data_dir,
                },
            ),
        ]
```

Nevertheless, many thanks for tackling the dataset accessibility problem with this great library!

As temporary fix I've created [ArneBinder/nlp-formats](https://github.com/ArneBinder/nlp-formats) (contributions welcome).

I have written a generic dataset for corpora created with the Brat annotation tool ([specification](https://brat.nlplab.org/standoff.html), [dataset code](https://github.com/ArneBinder/nlp/blob/brat/datasets/brat/brat.py)). Now I wonder how to use that to create specific dataset instances. What's the recommended way to reuse formats and loading functionality for datasets with a common format?

In my case, it took a bit of time to create the Brat dataset and I think others would appreciate to not have to think about that again. Also, I assume there are other formats (e.g. conll) that are widely used, so having this would really ease dataset onboarding and adoption of the library.


[FEATURE REQUEST] Multiprocessing with for dataset.map, dataset.filter

enhancement

New feature or request

Yes that's definitely something we plan to add ^^

Yes, that would be nice. We could take a look at what tensorflow `tf.data` does under the hood for instance.

So `tf.data.Dataset.map()` returns a `ParallelMapDataset` if `num_parallel_calls is not None` [link](https://github.com/tensorflow/tensorflow/blob/2b96f3662bd776e277f86997659e61046b56c315/tensorflow/python/data/ops/dataset_ops.py#L1623).

There, `num_parallel_calls` is turned into a tensor and and fed to `gen_dataset_ops.parallel_map_dataset` where it looks like tensorflow takes over.

We could start with something simple like a thread or process pool that `imap`s over some shards.

Multiprocessing was added in #552 . You can set the number of processes with `.map(..., num_proc=...)`. It also works for `filter`

Closing this one, but feel free to reo-open if you have other questions

@lhoestq Great feature implemented! Do you have plans to add it to official tutorials [Processing data in a Dataset](https://huggingface.co/docs/datasets/processing.html?highlight=save#augmenting-the-dataset)? It took me sometime to find this parallel processing api.

Thanks for the heads up !

I just added a paragraph about multiprocessing:
https://huggingface.co/docs/datasets/master/processing.html#multiprocessing

It would be nice to be able to speed up `dataset.map` or `dataset.filter`. Perhaps this is as easy as sharding the dataset sending each shard to a process/thread/dask pool and using the new `nlp.concatenate_dataset()` function to join them all together?


Correct data structure for PAN-X task in XTREME dataset?

Thanks for noticing ! This looks more reasonable indeed.
Feel free to open a PR

Hi @lhoestq
I made the proposed changes to the `xtreme.py` script. I noticed that I also need to change the schema in the `dataset_infos.json` file.  More specifically the `"features"` part of the PAN-X.LANG dataset:

```json
"features":{
   "word":{
      "dtype":"string",
      "id":null,
      "_type":"Value"
   },
   "ner_tag":{
      "dtype":"string",
      "id":null,
      "_type":"Value"
   },
   "lang":{
      "dtype":"string",
      "id":null,
      "_type":"Value"
   }
}
```
To fit the code above the fields `"word"`, `"ner_tag"`, and `"lang"` would become `"words"`, `ner_tags"` and `"langs"`. In addition the `dtype` should be changed from `"string"` to `"list"`.

 I made this changes but when trying to test this locally with `dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')` I face the issue that the `dataset_info.json` file is always overwritten by a downloaded version with the old settings, which then throws an error because the schema does not match. This makes it hard to test the changes locally. Do you have any suggestions on how to deal with that?

Hi !

You have to point to your local script.
First clone the repo and then:

```python
dataset = load_dataset("./datasets/xtreme", "PAN-X.en")
```
The "xtreme" directory contains "xtreme.py".

You also have to change the features definition in the `_info` method. You could use:

```python
features = nlp.Features({
    "words": [nlp.Value("string")],
    "ner_tags": [nlp.Value("string")],
    "langs": [nlp.Value("string")],
})
```

Hope this helps !
Let me know if you have other questions.

Thanks, I am making progress. I got a new error `NonMatchingSplitsSizesError ` (see traceback below), which I suspect is due to the fact that number of rows in the dataset changed (one row per word --> one row per sentence) as well as the number of bytes due to the slightly updated data structure.

```python
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='validation', num_bytes=1756492, num_examples=80536, dataset_name='xtreme'), 'recorded': SplitInfo(name='validation', num_bytes=1837109, num_examples=10000, dataset_name='xtreme')}, {'expected': SplitInfo(name='test', num_bytes=1752572, num_examples=80326, dataset_name='xtreme'), 'recorded': SplitInfo(name='test', num_bytes=1833214, num_examples=10000, dataset_name='xtreme')}, {'expected': SplitInfo(name='train', num_bytes=3496832, num_examples=160394, dataset_name='xtreme'), 'recorded': SplitInfo(name='train', num_bytes=3658428, num_examples=20000, dataset_name='xtreme')}]
```
I can fix the error by replacing the values in the `datasets_infos.json` file, which I tested for English. However, to update this for all 40 datasets manually is slightly painful. Is there a better way to update the expected values for all datasets?

You can update the json file by calling
```
nlp-cli test ./datasets/xtreme --save_infos --all_configs
```

One more thing about features. I mentioned

```python
features = nlp.Features({
    "words": [nlp.Value("string")],
    "ner_tags": [nlp.Value("string")],
    "langs": [nlp.Value("string")],
})
```

but it's actually not consistent with the way we write datasets. Something like this is simpler to read and more consistent with the way we define datasets:

```python
features = nlp.Features({
    "words": nlp.Sequence(nlp.Value("string")),
    "ner_tags": nlp.Sequence(nlp.Value("string")),
    "langs": nlp.Sequence(nlp.Value("string")),
})
```

Sorry about that

Closing this since PR #437 fixed the problem and has been merged to `master`.

Hi 🤗  team!

## Description of the problem
Thanks to the fix from #416 I am now able to load the NER task in the XTREME dataset as follows:

```python
from nlp import load_dataset
# AmazonPhotos.zip is located in data/
dataset = load_dataset("xtreme", "PAN-X.en", data_dir='./data')
dataset_train = dataset['train']
```

However, I am not sure that `load_dataset()` is returning the correct data structure for NER.

Currently, every row in `dataset_train` is of the form
```python
{'word': str, 'ner_tag': str, 'lang': str}
```
but I think we actually want something like
```python
{'words': List[str], 'ner_tags': List[str], 'langs': List[str]}
```
so that each row corresponds to a _sequence_ of words associated with each example. With the current data structure I do not think it is possible to transform `dataset_train` into a form suitable for training because we do not know the boundaries between examples.

Indeed, [this line](https://github.com/google-research/xtreme/blob/522434d1aece34131d997a97ce7e9242a51a688a/third_party/utils_tag.py#L58) in the XTREME repo, processes the texts as lists of sentences, tags, and languages.

## Proposed solution
Replace
```python
with open(filepath) as f:
    data = csv.reader(f, delimiter="\t", quoting=csv.QUOTE_NONE)
    for id_, row in enumerate(data):
        if row:
            lang, word = row[0].split(":")[0], row[0].split(":")[1]
            tag = row[1]
            yield id_, {"word": word, "ner_tag": tag, "lang": lang}
```
from  [these lines](https://github.com/huggingface/nlp/blob/ce7d3a1d630b78fe27188d1706f3ea980e8eec43/datasets/xtreme/xtreme.py#L881-L887) of the `_generate_examples()` function with something like

```python
guid_index = 1
with open(filepath, encoding="utf-8") as f:
    words = []
    ner_tags = []
    langs = []
    for line in f:
        if line.startswith("-DOCSTART-") or line == "" or line == "\n":
            if words:
                yield guid_index, {"words": words, "ner_tags": ner_tags, "langs": langs}
                guid_index += 1
                words = []
                ner_tags = []
        else:
            # pan-x data is tab separated
            splits = line.split("\t")
            # strip out en: prefix
            langs.append(splits[0][:2])
            words.append(splits[0][3:])
            if len(splits) > 1:
                labels.append(splits[-1].replace("\n", ""))
            else:
                # examples have no label in test set
                labels.append("O")
```
If you agree, me or @lvwerra would be happy to implement this and create a PR.


Addition of google drive links to dl_manager

I think the problem is the way you wrote your urls. Try the following structure to see `https://drive.google.com/uc?export=download&id=your_file_id` .

@lhoestq

Oh sorry, I think `_get_drive_url` is doing that.

Have you tried to use `dl_manager.download_and_extract(_get_drive_url(_TRAIN_URL)`? it should work with google drive links.

Hello there, I followed the template to create a download script of my own, which works fine for me, although I had to shun the dl_manager because it was downloading nothing from the drive links and instead use gdown.

This is the script for me:

```python
class EmoConfig(nlp.BuilderConfig):
    """BuilderConfig for SQUAD."""

    def __init__(self, **kwargs):
        """BuilderConfig for EmoContext.
    Args:
      **kwargs: keyword arguments forwarded to super.
    """
        super(EmoConfig, self).__init__(**kwargs)

_TEST_URL = "https://drive.google.com/file/d/1Hn5ytHSSoGOC4sjm3wYy0Dh0oY_oXBbb/view?usp=sharing"
_TRAIN_URL = "https://drive.google.com/file/d/12Uz59TYg_NtxOy7SXraYeXPMRT7oaO7X/view?usp=sharing"

class EmoDataset(nlp.GeneratorBasedBuilder):
    """ SemEval-2019 Task 3: EmoContext Contextual Emotion Detection in Text. Version 1.0.0 """

    VERSION = nlp.Version("1.0.0")
    force = False

    def _info(self):
        return nlp.DatasetInfo(
            description=_DESCRIPTION,
            features=nlp.Features(
                {
                    "text": nlp.Value("string"),
                    "label": nlp.features.ClassLabel(names=["others", "happy", "sad", "angry"]),
                }
            ),
            supervised_keys=None,
            homepage="https://www.aclweb.org/anthology/S19-2005/",
            citation=_CITATION,
        )

    def _get_drive_url(self, url):
        base_url = 'https://drive.google.com/uc?id='
        split_url = url.split('/')
        return base_url + split_url[5]

    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""
        if(not os.path.exists("emo-train.json") or self.force):
            gdown.download(self._get_drive_url(_TRAIN_URL), "emo-train.json", quiet = True)
        if(not os.path.exists("emo-test.json") or self.force):
            gdown.download(self._get_drive_url(_TEST_URL), "emo-test.json", quiet = True)
        return [
            nlp.SplitGenerator(
                name=nlp.Split.TRAIN,
                gen_kwargs={
                    "filepath": "emo-train.json",
                    "split": "train",
                },
            ),
            nlp.SplitGenerator(
                name=nlp.Split.TEST,
                gen_kwargs={"filepath": "emo-test.json", "split": "test"},
            ),
        ]

    def _generate_examples(self, filepath, split):
        """ Yields examples. """
        with open(filepath, 'rb') as f:
            data = json.load(f)
            for id_, text, label in zip(data["text"].keys(), data["text"].values(), data["Label"].values()):
                yield id_, {
                    "text": text,
                    "label": label,
                }
```

Can someone help me in adding gdrive links to be used with default dl_manager or adding gdown as another dl_manager, because I'd like to add this dataset to nlp's official database.


Something is wrong with WMT 19 kk-en dataset

dataset bug

A bug in a dataset script provided in the library

The translation in the `train` set does not look right:

```
>>>import nlp
>>>from nlp import load_dataset
>>>dataset = load_dataset('wmt19', 'kk-en')
>>>dataset["train"]["translation"][0]
{'kk': 'Trumpian Uncertainty', 'en': 'Трамптық белгісіздік'}
>>>dataset["validation"]["translation"][0]
{'kk': 'Ақша-несие саясатының сценарийін қайта жазсақ', 'en': 'Rewriting the Monetary-Policy Script'}
```


from_dict delete?

`from_dict` was added in #350 that was unfortunately not included in the 0.3.0 release. It's going to be included in the next release that will be out pretty soon though.
Right now if you want to use `from_dict` you have to install the package from the master branch
```
pip install git+https://github.com/huggingface/nlp.git
```

> `from_dict` was added in #350 that was unfortunately not included in the 0.3.0 release. It's going to be included in the next release that will be out pretty soon though.
> Right now if you want to use `from_dict` you have to install the package from the master branch
>
> ```
> pip install git+https://github.com/huggingface/nlp.git
> ```
OK, thank you.

AttributeError: type object 'Dataset' has no attribute 'from_dict'


Is there a way to download only NQ dev?

Unfortunately it's not possible to download only the dev set of NQ.

I think we could add a way to download only the test set by adding a custom configuration to the processing script though.

Ok, got it. I think this could be a valuable feature - especially for large datasets like NQ, but potentially also others.
For us, it will in this case make the difference of using the library or keeping the old downloads of the raw dev datasets.
However, I don't know if that fits into your plans with the library and can also understand if you don't want to support this.

I don't think we could force this behavior generally since the dataset script authors are free to organize the file download as they want (sometimes the mapping between split and files can be very much nontrivial) but we can add an additional configuration for Natural Question indeed as @lhoestq indicate.

Maybe I missed that in the docs, but is there a way to only download the dev set of natural questions (~1 GB)?
As we want to benchmark QA models on different datasets, I would like to avoid downloading the 41GB of training data.

I tried
```
dataset = nlp.load_dataset('natural_questions', split="validation", beam_runner="DirectRunner")
```
But this still triggered a big download of presumably the whole dataset. Is there any way of doing this or are splits / slicing options only available after downloading?

Thanks!


Unable to load XTREME dataset from disk

Hi @lewtun, you have to provide the full path to the downloaded file for example `/home/lewtum/..`

I was able to repro. Opening a PR to fix that.
Thanks for reporting this issue !

Hi 🤗  team!

## Description of the problem
Following the [docs](https://huggingface.co/nlp/loading_datasets.html?highlight=xtreme#manually-downloading-files) I'm trying to load the `PAN-X.fr` dataset from the [XTREME](https://github.com/google-research/xtreme) benchmark.

I have manually downloaded the `AmazonPhotos.zip` file from [here](https://www.amazon.com/clouddrive/share/d3KGCRCIYwhKJF0H3eWA26hjg2ZCRhjpEQtDL70FSBN?_encoding=UTF8&%2AVersion%2A=1&%2Aentries%2A=0&mgh=1) and am running into a `FileNotFoundError` when I point to the location of the dataset.

As far as I can tell, the problem is that `AmazonPhotos.zip` decompresses to `panx_dataset` and `load_dataset()` is not looking in the correct path:

```
# path where load_dataset is looking for fr.tar.gz
/root/.cache/huggingface/datasets/9b8c4f1578e45cb2539332c79738beb3b54afbcd842b079cabfd79e3ed6704f6/
# path where it actually exists
/root/.cache/huggingface/datasets/9b8c4f1578e45cb2539332c79738beb3b54afbcd842b079cabfd79e3ed6704f6/panx_dataset/
```

## Steps to reproduce the problem

1. Manually download the XTREME benchmark from [here](https://www.amazon.com/clouddrive/share/d3KGCRCIYwhKJF0H3eWA26hjg2ZCRhjpEQtDL70FSBN?_encoding=UTF8&%2AVersion%2A=1&%2Aentries%2A=0&mgh=1)

2. Run the following code snippet
```python
from nlp import load_dataset
# AmazonPhotos.zip is in the root of the folder
dataset = load_dataset("xtreme", "PAN-X.fr", data_dir='./')
```

3. Here is the stack trace
```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-4-26786bb5fa93> in <module>
----> 1 dataset = load_dataset("xtreme", "PAN-X.fr", data_dir='./')

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    430                 verify_infos = not save_infos and not ignore_verifications
    431                 self._download_and_prepare(
--> 432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    433                 )
    434                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    464         split_dict = SplitDict(dataset_name=self.name)
    465         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 466         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    467         # Checksums verification
    468         if verify_infos:

/usr/local/lib/python3.6/dist-packages/nlp/datasets/xtreme/b8c2ed3583a7a7ac60b503576dfed3271ac86757628897e945bd329c43b8a746/xtreme.py in _split_generators(self, dl_manager)
    725             panx_dl_dir = dl_manager.extract(panx_path)
    726             lang = self.config.name.split(".")[1]
--> 727             lang_folder = dl_manager.extract(os.path.join(panx_dl_dir, lang + ".tar.gz"))
    728             return [
    729                 nlp.SplitGenerator(

/usr/local/lib/python3.6/dist-packages/nlp/utils/download_manager.py in extract(self, path_or_paths)
    196         """
    197         return map_nested(
--> 198             lambda path: cached_path(path, extract_compressed_file=True, force_extract=False), path_or_paths,
    199         )
    200

/usr/local/lib/python3.6/dist-packages/nlp/utils/py_utils.py in map_nested(function, data_struct, dict_only, map_tuple)
    170                 return tuple(mapped)
    171     # Singleton
--> 172     return function(data_struct)
    173
    174

/usr/local/lib/python3.6/dist-packages/nlp/utils/download_manager.py in <lambda>(path)
    196         """
    197         return map_nested(
--> 198             lambda path: cached_path(path, extract_compressed_file=True, force_extract=False), path_or_paths,
    199         )
    200

/usr/local/lib/python3.6/dist-packages/nlp/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
    203     elif urlparse(url_or_filename).scheme == "":
    204         # File, but it doesn't exist.
--> 205         raise FileNotFoundError("Local file {} doesn't exist".format(url_or_filename))
    206     else:
    207         # Something unknown

FileNotFoundError: Local file /root/.cache/huggingface/datasets/9b8c4f1578e45cb2539332c79738beb3b54afbcd842b079cabfd79e3ed6704f6/fr.tar.gz doesn't exist
```

## OS and hardware
```
- `nlp` version: 0.3.0
- Platform: Linux-4.15.0-72-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


train_test_split error: 'dict' object has no attribute 'deepcopy'

It was fixed in 2ddd18d139d3047c9c3abe96e1e7d05bb360132c.
Could you pull the latest changes from master @morganmcg1 ?

`train_test_split` is giving me an error when I try and call it:

`'dict' object has no attribute 'deepcopy'`

## To reproduce

```
dataset = load_dataset('glue', 'mrpc', split='train')
dataset = dataset.train_test_split(test_size=0.2)
```

## Full Stacktrace
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-feb740dbec9a> in <module>
      1 dataset = load_dataset('glue', 'mrpc', split='train')
----> 2 dataset = dataset.train_test_split(test_size=0.2)

~/anaconda3/envs/fastai2_me/lib/python3.7/site-packages/nlp/arrow_dataset.py in train_test_split(self, test_size, train_size, shuffle, seed, generator, keep_in_memory, load_from_cache_file, train_cache_file_name, test_cache_file_name, writer_batch_size)
   1032                     "writer_batch_size": writer_batch_size,
   1033                 }
-> 1034                 train_kwargs = cache_kwargs.deepcopy()
   1035                 train_kwargs["split"] = "train"
   1036                 test_kwargs = cache_kwargs.deepcopy()

AttributeError: 'dict' object has no attribute 'deepcopy'
```


MissingBeamOptions for Wikipedia 20200501.en

Fixed. Could you try again @mitchellgordon95 ?
It was due a file not being updated on S3.

We need to make sure all the datasets scripts get updated properly @julien-c

I found the same issue with almost any language other than English. (For English, it works). Will someone need to update the file on S3 again?

This is because only some languages are already preprocessed (en, de, fr, it) and stored on our google storage.
We plan to have a systematic way to preprocess more wikipedia languages in the future.

For the other languages you have to process them on your side using apache beam. That's why the lib asks for a Beam runner.

There may or may not be a regression for the pre-processed Wikipedia dataset. This was working fine 10 commits ago (without having Apache Beam available):

```
nlp.load_dataset('wikipedia', "20200501.en", split='train')
```

And now, having pulled master, I get:

```
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, total: 34.06 GiB) to /home/hltcoe/mgordon/.cache/huggingface/datasets/wikipedia/20200501.en/1.0.0/76b0b2747b679bb0ee7a1621e50e5a6378477add0c662668a324a5bc07d516dd...
Traceback (most recent call last):
  File "scripts/download.py", line 11, in <module>
    fire.Fire(download_pretrain)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "scripts/download.py", line 6, in download_pretrain
    nlp.load_dataset('wikipedia', "20200501.en", split='train')
  File "/exp/mgordon/nlp/src/nlp/load.py", line 534, in load_dataset
    save_infos=save_infos,
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 460, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 870, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, S
park, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
        `load_dataset('wikipedia', '20200501.en', beam_runner='DirectRunner')`
```


Faster Shuffling?

I think the slowness here probably come from the fact that we are copying from and to python.

@lhoestq for all the `select`-based methods I think we should stay in Arrow format and update the writer so that it can accept Arrow tables or batches as well. What do you think?

> @lhoestq for all the `select`-based methods I think we should stay in Arrow format and update the writer so that it can accept Arrow tables or batches as well. What do you think?

I just tried with `writer.write_table` with tables of 1000 elements and it's slower that the solution in #405

On my side (select 10 000 examples):
- Original implementation: 12s
- Batched solution: 100ms
- solution using arrow tables: 350ms

I'll try with arrays and record batches to see if we can make it work.

I tried using `.take` from pyarrow recordbatches but it doesn't improve the speed that much:
```python
import nlp
import numpy as np

dset = nlp.Dataset.from_file("dummy_test_select.arrow")  # dummy dataset with 100000 examples like {"a": "h"*512}
indices = np.random.randint(0, 100_000, 1000_000)
```

```python
%%time
batch_size = 10_000
writer = ArrowWriter(schema=dset.schema, path="dummy_path",
                     writer_batch_size=1000, disable_nullable=False)
for i in tqdm(range(0, len(indices), batch_size)):
    table = pa.concat_tables(dset._data.slice(int(i), 1) for i in indices[i : min(len(indices), i + batch_size)])
    batch = table.to_pydict()
    writer.write_batch(batch)
writer.finalize()
# 9.12s
```

```python
%%time
batch_size = 10_000
writer = ArrowWriter(schema=dset.schema, path="dummy_path",
                     writer_batch_size=1000, disable_nullable=False)
for i in tqdm(range(0, len(indices), batch_size)):
    batch_indices = indices[i : min(len(indices), i + batch_size)]
    # First, extract only the indices that we need with a mask
    mask = [False] * len(dset)
    for k in batch_indices:
        mask[k] = True
    t_batch = dset._data.filter(pa.array(mask))
    # Second, build the list of indices for the filtered table, and taking care of duplicates
    rev_positions = {}
    duplicates = 0
    for i, j in enumerate(sorted(batch_indices)):
        if j in rev_positions:
            duplicates += 1
        else:
            rev_positions[j] = i - duplicates
    rev_map = [rev_positions[j] for j in batch_indices]
    # Third, use `.take` from the combined recordbatch
    t_combined = t_batch.combine_chunks()  # load in memory
    recordbatch = t_combined.to_batches()[0]
    table = pa.Table.from_arrays(
        [recordbatch[c].take(pa.array(rev_map)) for c in range(len(dset._data.column_names))],
        schema=writer.schema
    )
    writer.write_table(table)
writer.finalize()
# 3.2s
```

Shuffling is now significantly faster thanks to #513
Feel free to play with it now :)

Closing this one, but feel free to re-open if you have other questions

Consider shuffling bookcorpus:

```
dataset = nlp.load_dataset('bookcorpus', split='train')
dataset.shuffle()
```
According to tqdm, this will take around 2.5 hours on my machine to complete (even with the faster version of select from #405). I've also tried with `keep_in_memory=True` and `writer_batch_size=1000`.

But I can also just write the lines to a text file:

```
batch_size = 100000
with open('tmp.txt', 'w+') as out_f:
    for i in tqdm(range(0, len(dataset), batch_size)):
        batch = dataset[i:i+batch_size]['text']
        print("\n".join(batch), file=out_f)
```

Which completes in a couple minutes, followed by `shuf tmp.txt > tmp2.txt` which completes in under a minute. And finally,

```
dataset = nlp.load_dataset('text', data_files='tmp2.txt')
```

Which completes in under 10 minutes. I read up on Apache Arrow this morning, and it seems like the columnar data format is not especially well-suited to shuffling rows, since moving items around requires a lot of book-keeping.

Is shuffle inherently slow, or am I just using it wrong? And if it is slow, would it make sense to try converting the data to a row-based format on disk and then shuffling? (Instead of calling select with a random permutation, as is currently done.)


Memory issue when doing select

As noticed in #389, the following code loads the entire wikipedia in memory.

```python
import nlp
w = nlp.load_dataset("wikipedia", "20200501.en", split="train")
w.select([0])
```

This is caused by [this line](https://github.com/huggingface/nlp/blob/master/src/nlp/arrow_dataset.py#L626) for some reason, that tries to serialize the function with all the wikipedia data with it.

It's not the case with `.map` or `.filter`.
However functions that are based on `.select` like `.shuffle`, `.shard`, `.train_test_split`, `.sort` are affected.


🌟 [Metric Request] WOOD score

metric request

Requesting to add a new metric

WOOD score paper : https://arxiv.org/pdf/2007.06898.pdf

Abstract :

>Models that surpass human performance on several popular benchmarks display significant degradation in performance on exposure to Out of Distribution (OOD) data. Recent research has shown that models overfit to spurious biases and ‘hack’ datasets, in lieu of learning generalizable features like humans. In order to stop the inflation in model performance – and thus overestimation in AI systems’ capabilities – we propose a simple and novel evaluation metric, WOOD Score, that encourages generalization during evaluation.


🐛 [Dataset] Cannot download wmt14, wmt15 and wmt17

similar slow download speed here for nlp.load_dataset('wmt14', 'fr-en')
`
Downloading: 100%|██████████████████████████████████████████████████████████| 658M/658M [1:00:42<00:00, 181kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████| 918M/918M [1:39:38<00:00, 154kB/s]
Downloading:   2%|▉                                                       | 40.9M/2.37G [04:48<5:03:06, 128kB/s]
`
Could we just download a specific subdataset in 'wmt14', such as 'newstest14'?

> The code runs but the download speed is extremely slow, the same behaviour is not observed on wmt16 and wmt18

The original source for the files may provide slow download speeds.
We can probably host these files ourselves.

> When trying to download wmt17 zh-en, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz

Looks like the file`UNv1.0.en-zh.tar.gz` is missing, or the url changed. We need to fix that

> Could we just download a specific subdataset in 'wmt14', such as 'newstest14'?

Right now I don't think it's possible. Maybe @patrickvonplaten knows more about it

Yeah, the download speed is sadly always extremely slow :-/.
I will try to check out the `wmt17 zh-en` bug :-)

Maybe this can be used - https://stuncorpusprod.blob.core.windows.net/corpusfiles/UNv1.0.en-zh.tar.gz.00

1. I try downloading `wmt14`, `wmt15`, `wmt17`, `wmt19` with the following code:
```
nlp.load_dataset('wmt14','de-en')
nlp.load_dataset('wmt15','de-en')
nlp.load_dataset('wmt17','de-en')
nlp.load_dataset('wmt19','de-en')
```
The code runs but the download speed is **extremely slow**, the same behaviour is not observed on `wmt16` and `wmt18`

2. When trying to download `wmt17 zh-en`, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz


Conversion through to_pandas output numpy arrays for lists instead of python objects

To convert from arrow type we have three options: to_numpy, to_pandas and to_pydict/to_pylist.

- to_numpy and to_pandas return numpy arrays instead of lists but are very fast.
- to_pydict/to_pylist can be 100x slower and become the bottleneck for reading data, but at least they return lists.

Maybe we can have to_pydict/to_pylist as the default and use to_numpy or to_pandas when the format (set by `set_format`) is 'numpy' or 'pandas'

In a related question, the conversion through to_pandas output numpy arrays for the lists instead of python objects.

Here is an example:
```python
>>> dataset._data.slice(key, 1).to_pandas().to_dict("list")
{'sentence1': ['Amrozi accused his brother , whom he called " the witness " , of deliberately distorting his evidence .'], 'sentence2': ['Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .'], 'label': [1], 'idx': [0], 'input_ids': [array([  101,  7277,  2180,  5303,  4806,  1117,  1711,   117,  2292,
        1119,  1270,   107,  1103,  7737,   107,   117,  1104,  9938,
        4267, 12223, 21811,  1117,  2554,   119,   102])], 'token_type_ids': [array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0])], 'attention_mask': [array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1])]}
>>> type(dataset._data.slice(key, 1).to_pandas().to_dict("list")['input_ids'][0])
<class 'numpy.ndarray'>
>>> dataset._data.slice(key, 1).to_pydict()
{'sentence1': ['Amrozi accused his brother , whom he called " the witness " , of deliberately distorting his evidence .'], 'sentence2': ['Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .'], 'label': [1], 'idx': [0], 'input_ids': [[101, 7277, 2180, 5303, 4806, 1117, 1711, 117, 2292, 1119, 1270, 107, 1103, 7737, 107, 117, 1104, 9938, 4267, 12223, 21811, 1117, 2554, 119, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
```


1080


NLp


[dataset] Structure of MLQA seems unecessary nested

Same for the RACE dataset: https://github.com/huggingface/nlp/blob/master/datasets/race/race.py

Should we scan all the datasets to remove this pattern of un-necessary nesting?

You're right, I think we don't need to use the nested dictionary.

The features of the MLQA dataset comprise several nested dictionaries with a single element inside (for `questions` and `ids`): https://github.com/huggingface/nlp/blob/master/datasets/mlqa/mlqa.py#L90-L97

Should we keep this @mariamabarham @patrickvonplaten? Was this added for compatibility with tfds?

```python
            features=nlp.Features(
                {
                    "context": nlp.Value("string"),
                    "questions": nlp.features.Sequence({"question": nlp.Value("string")}),
                    "answers": nlp.features.Sequence(
                        {"text": nlp.Value("string"), "answer_start": nlp.Value("int32"),}
                    ),
                    "ids": nlp.features.Sequence({"idx": nlp.Value("string")})
```


Iyy!!!


to_pandas conversion doesn't always work

**Edit**: other topic previously in this message moved to a new issue: https://github.com/huggingface/nlp/issues/387

Could you try to update pyarrow to >=0.17.0 ? It should fix the `to_pandas` bug

Also I'm not sure that structures like list<struct> are fully supported in the lib (none of the datasets use that).
It can cause issues when using dataset transforms like `filter` for example

For some complex nested types, the conversion from Arrow to python dict through pandas doesn't seem to be possible.

Here is an example using the official SQUAD v2 JSON file.

This example was found while investigating #373.

```python
>>> squad = load_dataset('json', data_files={nlp.Split.TRAIN: ["./train-v2.0.json"]}, download_mode=nlp.GenerateMode.FORCE_REDOWNLOAD, version="1.0.0", field='data')
>>> squad['train']
Dataset(schema: {'title': 'string', 'paragraphs': 'list<item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>>'}, num_rows: 442)
>>> squad['train'][0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/thomwolf/Documents/GitHub/datasets/src/nlp/arrow_dataset.py", line 589, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/Users/thomwolf/Documents/GitHub/datasets/src/nlp/arrow_dataset.py", line 529, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pandas().to_dict("list"))
  File "pyarrow/array.pxi", line 559, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1367, in pyarrow.lib.Table._to_pandas
  File "/Users/thomwolf/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 766, in table_to_blockmanager
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
  File "/Users/thomwolf/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 1101, in _table_to_blocks
    list(extension_columns.keys()))
  File "pyarrow/table.pxi", line 881, in pyarrow.lib.table_to_blocks
  File "pyarrow/error.pxi", line 105, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Not implemented type for Arrow list to pandas: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>
```

cc @lhoestq would we have a way to detect this from the schema maybe?

Here is the schema for this pretty complex JSON:
```python
>>> squad['train'].schema
title: string
paragraphs: list<item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>>
  child 0, item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>
      child 0, qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>
          child 0, item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>
              child 0, question: string
              child 1, id: string
              child 2, answers: list<item: struct<text: string, answer_start: int64>>
                  child 0, item: struct<text: string, answer_start: int64>
                      child 0, text: string
                      child 1, answer_start: int64
              child 3, is_impossible: bool
              child 4, plausible_answers: list<item: struct<text: string, answer_start: int64>>
                  child 0, item: struct<text: string, answer_start: int64>
                      child 0, text: string
                      child 1, answer_start: int64
      child 1, context: string
```


TypeError when computing bertscore

I am not able to reproduce this issue on my side.
Could you give us more details about the inputs you used ?

I do get another error though:
```
~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/bert_score/utils.py in bert_cos_score_idf(model, refs, hyps, tokenizer, idf_dict, verbose, batch_size, device, all_layers)
    371         return sorted(list(set(l)), key=lambda x: len(x.split(" ")))
    372
--> 373     sentences = dedup_and_sort(refs + hyps)
    374     embs = []
    375     iter_range = range(0, len(sentences), batch_size)

ValueError: operands could not be broadcast together with shapes (0,) (2,)
```
That's because it gets numpy arrays as input and not lists. See #387

The other issue was fixed by #403

Do you still get this issue @willywsm1013 ?

Hi,

I installed nlp 0.3.0 via pip, and my python version is 3.7.
When I tried to compute bertscore with the code:
```
import nlp
bertscore = nlp.load_metric('bertscore')
# load hyps and refs
...
print (bertscore.compute(hyps, refs, lang='en'))
```

I got the following error.
```
Traceback (most recent call last):
  File "bert_score_evaluate.py", line 16, in <module>
    print (bertscore.compute(hyps, refs, lang='en'))
  File "/home/willywsm/anaconda3/envs/torcher/lib/python3.7/site-packages/nlp/metric.py", line 200, in compute
    output = self._compute(predictions=predictions, references=references, **metrics_kwargs)
  File "/home/willywsm/anaconda3/envs/torcher/lib/python3.7/site-packages/nlp/metrics/bertscore/fb176889831bf0ce995ed197edc94b2e9a83f647a869bb8c9477dbb2d04d0f08/bertscore.py", line 105, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() takes 3 positional arguments but 4 were given
```

It seems like there is something wrong with get_hash() function?


Segmentation fault when loading local JSON dataset as of #372

I've seen this sort of thing before -- it might help to delete the directory -- I've also noticed that there is an error with the json Dataloader for any data I've tried to load. I've replaced it with this, which skips over the data feature population step:

```python
import os

import pyarrow.json as paj

import nlp as hf_nlp

from nlp import DatasetInfo, BuilderConfig, SplitGenerator, Split, utils
from nlp.arrow_writer import ArrowWriter

class JSONDatasetBuilder(hf_nlp.ArrowBasedBuilder):
    BUILDER_CONFIG_CLASS = BuilderConfig

    def _info(self):
        return DatasetInfo()

    def _split_generators(self, dl_manager):
        """ We handle string, list and dicts in datafiles
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [SplitGenerator(name=Split.TRAIN, gen_kwargs={"files": files})]
        splits = []
        for split_name in [Split.TRAIN, Split.VALIDATION, Split.TEST]:
            if split_name in self.config.data_files:
                files = self.config.data_files[split_name]
                if isinstance(files, str):
                    files = [files]
                splits.append(SplitGenerator(name=split_name, gen_kwargs={"files": files}))
        return splits

    def _prepare_split(self, split_generator):
        fname = "{}-{}.arrow".format(self.name, split_generator.name)
        fpath = os.path.join(self._cache_dir, fname)

        writer = ArrowWriter(path=fpath)

        generator = self._generate_tables(**split_generator.gen_kwargs)
        for key, table in utils.tqdm(generator, unit=" tables", leave=False):
            writer.write_table(table)
        num_examples, num_bytes = writer.finalize()

        split_generator.split_info.num_examples = num_examples
        split_generator.split_info.num_bytes = num_bytes

    def _generate_tables(self, files):
        for i, file in enumerate(files):
            pa_table = paj.read_json(
                file
            )
            yield i, pa_table

```

Yes, deleting the directory solves the error whenever I try to rerun.

By replacing the json-loader, you mean the cached file in my `site-packages` directory? e.g. `/home/XXX/.cache/lib/python3.7/site-packages/nlp/datasets/json/(...)/json.py`

When I was testing this out before the #372 PR was merged I had issues installing it properly locally. Since the `json.py` script was downloaded instead of actually using the one provided in the local install. Manually updating that file seemed to solve it, but it didn't seem like a proper solution. Especially when having to run this on a remote compute cluster with no access to that directory.

I see, diving in the JSON file for SQuAD it's a pretty complex structure.

The best solution for you, if you have a dataset really similar to SQuAD would be to copy and modify the SQuAD data processing script. We will probably add soon an option to be able to specify file path to use instead of the automatic URL encoded in the script but in the meantime you can:
- copy the [squad script](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py)  in a new script for your dataset
- in the new script replace [these `urls_to_download `](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py#L99-L102) by `urls_to_download=self.config.data_files`
- load the dataset with `dataset = load_dataset('path/to/your/new/script', data_files={nlp.Split.TRAIN: "./datasets/train-v2.0.json"})`

This way you can reuse all the processing logic of the SQuAD loading script.

This seems like a more sensible solution! Thanks, @thomwolf. It's been a little daunting to understand what these scripts actually do, due to the level of abstraction and central documentation.

Am I correct in assuming that the `_generate_examples()` function is the actual procedure for how the data is loaded from file? Meaning that essentially with a file containing another format, that is the only function that requires re-implementation? I'm working with a lot of datasets that, due to licensing and privacy, cannot be published. As this library is so neatly integrated with the transformers library and gives easy access to public sets such as SQUAD and increased performance, it is very neat to be able to load my private sets as well. As of now, I have just been working on scripts for translating all my data into the SQUAD-format before using the json script, but I see that it might not be necessary after all.

Yes `_generate_examples()` is the main entry point. If you change the shape of the returned dictionary you also need to update the `features` in the `_info`.

I'm currently writing the doc so it should be easier soon to use the library and know how to add your datasets.

Could you try to update pyarrow to >=0.17.0 @vegarab ?
I don't have any segmentation fault with my version of pyarrow (0.17.1)

I tested with
```python
import nlp
s = nlp.load_dataset("json", data_files="train-v2.0.json", field="data", split="train")
s[0]
# {'title': 'Normans', 'paragraphs': [{'qas': [{'question': 'In what country is Normandy located?', 'id':...
```

Also if you want to have your own dataset script, we now have a new documentation !
See here:
https://huggingface.co/nlp/add_dataset.html

@lhoestq
For some reason, I am not able to reproduce the segmentation fault, on pyarrow==0.16.0. Using the exact same environment and file.

Anyhow, I discovered that pyarrow>=0.17.0 is required to read in a JSON file where the pandas structs contain lists. Otherwise, pyarrow complains when attempting to cast the struct:
```py
import nlp
>>> s = nlp.load_dataset("json", data_files="datasets/train-v2.0.json", field="data", split="train")
Using custom data configuration default
>>> s[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 558, in __getitem__
    format_kwargs=self._format_kwargs,
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 498, in _getitem
    outputs = self._unnest(self._data.slice(key, 1).to_pandas().to_dict("list"))
  File "pyarrow/array.pxi", line 559, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1367, in pyarrow.lib.Table._to_pandas
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 766, in table_to_blockmanager
    blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
  File "/home/vegarab/.conda/envs/torch/lib/python3.7/site-packages/pyarrow/pandas_compat.py", line 1101, in _table_to_blocks
    list(extension_columns.keys()))
  File "pyarrow/table.pxi", line 881, in pyarrow.lib.table_to_blocks
  File "pyarrow/error.pxi", line 105, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Not implemented type for Arrow list to pandas: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>
>>> s
Dataset(schema: {'title': 'string', 'paragraphs': 'list<item: struct<qas: list<item: struct<question: string, id: string, answers: list<item: struct<text: string, answer_start: int64>>, is_impossible: bool, plausible_answers: list<item: struct<text: string, answer_start: int64>>>>, context: string>>'}, num_rows: 35)
```

Upgrading to >=0.17.0 provides the same dataset structure, but accessing the records is possible without the same exception.

Very happy to see some extended documentation!

#376 seems to be reporting the same issue as mentioned above.

This issue helped me a lot, thanks.
Hope this issue will be fixed soon.

The last issue was closed (#369) once the #372 update was merged. However, I'm still not able to load a SQuAD formatted JSON file. Instead of the previously recorded pyarrow error, I now get a segmentation fault.

```
dataset = nlp.load_dataset('json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
```
causes
```
Using custom data configuration default
Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/XXX/.cache/huggingface/datasets/json/default/0.0.0...
0 tables [00:00, ? tables/s]Segmentation fault (core dumped)
```
where `./datasets/train-v2.0.json` is downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/.
This is consistent with other SQuAD-formatted JSON files.

When attempting to load the dataset again, I get the following:
```
Using custom data configuration default
Traceback (most recent call last):
  File "dataloader.py", line 6, in <module>
    'json', data_files={nlp.Split.TRAIN: ["./datasets/train-v2.0.json"]}, field='data')
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 382, in download_and_prepare
    with incomplete_dir(self._cache_dir) as tmp_data_dir:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 368, in incomplete_dir
    os.makedirs(tmp_dir)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/XXX/.cache/huggingface/datasets/json/default/0.0.0.incomplete'
```

(Not sure if you wanted this in the previous issue #369 or not as it was closed.)


can't load local dataset: pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries

I am able to reproduce this with the official SQuAD `train-v2.0.json` file downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/

I am facing this issue in transformers library 3.0.2 while reading a csv using datasets.
Is this fixed in latest version?
I updated the latest version 4.0.1 but still getting this error. What could cause this error?

Trying to load a local SQuAD-formatted dataset (from a JSON file, about 60MB):
```
dataset = nlp.load_dataset(path='json', data_files={nlp.Split.TRAIN: ["./path/to/file.json"]})
```
causes
```
Traceback (most recent call last):
  File "dataloader.py", line 9, in <module>
    ["./path/to/file.json"]})
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 483, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 719, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False):
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/datasets/json/88c1bc5c68489f7eda549ed05a5a738527c613b3e7a4ee3524d9d233353a949b/json.py", line 53, in _generate_tables
    file, read_options=self.config.pa_read_options, parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 191, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```

I haven't been able to find any reports of this specific pyarrow error here or elsewhere.


load_metric can't acquire lock anymore

I found that, in the same process (or the same interactive session), if I do

import nlp

m1 = nlp.load_metric('glue', 'mrpc')
m2 = nlp.load_metric('glue', 'sst2')

I will get the same error `ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id'`.

I can't load metric (glue) anymore after an error in a previous run. I even removed the whole cache folder `/home/XXX/.cache/huggingface/`, and the issue persisted. What are the steps to fix this?

    Traceback (most recent call last):
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/metric.py", line 101, in __init__
        self.filelock.acquire(timeout=1)
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/filelock.py", line 278, in acquire
        raise Timeout(self._lock_file)
    filelock.Timeout: The file lock '/home/XXX/.cache/huggingface/metrics/glue/1.0.0/1-glue-0.arrow.lock' could not be acquired.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "examples_huggingface_nlp.py", line 268, in <module>
        main()
      File "examples_huggingface_nlp.py", line 242, in main
        dataset, metric = get_dataset_metric(glue_task)
      File "examples_huggingface_nlp.py", line 77, in get_dataset_metric
        metric = nlp.load_metric('glue', glue_config, experiment_id=1)
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/load.py", line 440, in load_metric
        **metric_init_kwargs,
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/metric.py", line 104, in __init__
        "Cannot acquire lock, caching file might be used by another process, "
    ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id' for this run.
    I0709 15:54:41.008838 139854118430464 filelock.py:318] Lock 139852058030936 released on /home/XXX/.cache/huggingface/metrics/glue/1.0.0/1-glue-0.arrow.lock


How to augment data ?

Using batched map is probably the easiest way at the moment.
What kind of augmentation would you like to do ?

Some samples in the dataset are too long, I want to divide them in several samples.

Using batched map is the way to go then.
We'll make it clearer in the docs that map could be used for augmentation.

Let me know if you think there should be another way to do it. Or feel free to close the issue otherwise.

It just feels awkward to use map to augment data. Also it means it's not possible to augment data in a non-batched way.

But to be honest I have no idea of a good API...

Or for non-batched samples, how about returning a tuple ?

```python
def aug(sample):
    # Simply copy the existing data to have x2 amount of data
    return sample, sample

dataset = dataset.map(aug)
```

It feels really natural and easy, but :

* it means the behavior with batched data is different
* I don't know how doable it is backend-wise

@lhoestq

As we're working with arrow's columnar format we prefer to play with batches that are dictionaries instead of tuples.
If we have tuple it implies to re-format the data each time we want to write to arrow, which can lower the speed of map for example.

It's also a matter of coherence, as we don't want users to be confused whether they have to return dictionaries for some functions and tuples for others when they're doing batches.

Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
```


[dateset subset missing]  xtreme paws-x

You're right, thanks for pointing it out. We will update it

I tried nlp.load_dataset('xtreme', 'PAWS-X.es') but get the value error
It turns out that the subset for Spanish is missing
https://github.com/google-research-datasets/paws/tree/master/pawsx


🐛 [Metrics] ROUGE is non-deterministic

> Hi, can you give a full self-contained example to reproduce this behavior?

There is a notebook in the post ;)

> If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.
>
> Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.
>
> Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :
>
> > ['0.3350', '0.1470', '0.2329']
> > ['0.3358', '0.1451', '0.2332']
>
> Why ROUGE is not deterministic ?

This is because of rouge's `BootstrapAggregator` that uses sampling to get confidence intervals (low, mid, high).
You can get deterministic scores per sentence pair by using
```python
score = rouge.compute(rouge_types=["rouge1", "rouge2", "rougeL"], use_agregator=False)
```
Or you can set numpy's random seed if you still want to use the aggregator.

Maybe we can set all the random seeds of numpy/torch etc. while running `metric.compute` ?

Now if you re-run the notebook, the two printed results are the same @colanim
```
['0.3356', '0.1466', '0.2318']
['0.3356', '0.1466', '0.2318']
```
However across sessions, the results may change (as numpy's random seed can be different). You can prevent that by setting your seed:
```python
rouge = nlp.load_metric('rouge', seed=42)
```

If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

> ['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ?


[Feature request] Add dataset.ragged_map() function for many-to-many transformations

Actually `map(batched=True)` can already change the size of the dataset.
It can accept examples of length `N` and returns a batch of length `M` (can be null or greater than `N`).

I'll make that explicit in the doc that I'm currently writing.

You're two steps ahead of me :) In my testing, it also works if `M` < `N`.

A batched map of different length seems to work if you directly overwrite all of the original keys, but fails if any of the original keys are preserved.

For example,
```python
# Create a dummy dataset
dset = load_dataset("wikitext", "wikitext-2-raw-v1")["test"]
dset = dset.map(lambda ex: {"length": len(ex["text"]), "foo": 1})

# Do an allreduce on each batch, overwriting both keys
dset.map(lambda batch: {"length": [sum(batch["length"])], "foo": [1]})
# Dataset(schema: {'length': 'int64', 'foo': 'int64'}, num_rows: 5)

# Now attempt an allreduce without touching the `foo` key
dset.map(lambda batch: {"length": [sum(batch["length"])]})
# This fails with the error message below
```

```bash
  File "/path/to/nlp/src/nlp/arrow_dataset.py", line 728, in map
    arrow_schema = pa.Table.from_pydict(test_output).schema
  File "pyarrow/io.pxi", line 1532, in pyarrow.lib.Codec.detect
  File "pyarrow/table.pxi", line 1503, in pyarrow.lib.Table.from_arrays
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named foo expected length 1 but got length 2
```

Adding the `remove_columns=["length", "foo"]` argument to `map()` solves the issue. Leaving the above error for future visitors. Perfect, thank you!

`dataset.map()` enables one-to-one transformations. Input one example and output one example. This is helpful for tokenizing and cleaning individual lines.
`dataset.filter()` enables one-to-(one-or-none) transformations. Input one example and output either zero/one example. This is helpful for removing portions from the dataset.
However, some dataset transformations are many-to-many. Consider constructing BERT training examples from a dataset of sentences, where you map `["a", "b", "c"] -> ["a[SEP]b", "a[SEP]c", "b[SEP]c", "c[SEP]b", ...]`

I propose a more general `ragged_map()` method that takes in a batch of examples of length `N` and return a batch of examples `M`. This is different from the `map(batched=True)` method, which takes examples of length `N` and returns a batch of length `N`, processing individual examples in parallel. I don't have a clear vision of how this would be implemented efficiently and lazily, but would love to hear the community's feedback on this.

My specific use case is creating an end-to-end ELECTRA data pipeline. I would like to take the raw WikiText data and generate training examples from this using the `ragged_map()` method, then export to TFRecords and train quickly. This would be a reproducible pipeline with no bash scripts. Currently I'm relying on scripts like https://github.com/google-research/electra/blob/master/build_pretraining_dataset.py, which are less general.


ArrowBasedBuilder _prepare_split parse_schema breaks on nested structures

Hi, it depends on what it is in your `dataset_builder.py` file. Can you share it?

If you are just loading `json` files, you can also directly use the `json` script (which will find the schema/features from your JSON structure):

```python
from nlp import load_dataset
ds = load_dataset("json", data_files=rel_datafiles)
```

The behavior I'm seeing is from the `json` script.
I hacked this together to overcome the error with the `JSON` dataloader

```
class DatasetBuilder(hf_nlp.ArrowBasedBuilder):
    BUILDER_CONFIG_CLASS = BuilderConfig

    def _info(self):
        return DatasetInfo()

    def _split_generators(self, dl_manager):
        """ We handle string, list and dicts in datafiles
        """
        if isinstance(self.config.data_files, (str, list, tuple)):
            files = self.config.data_files
            if isinstance(files, str):
                files = [files]
            return [SplitGenerator(name=Split.TRAIN, gen_kwargs={"files": files})]
        splits = []
        for split_name in [Split.TRAIN, Split.VALIDATION, Split.TEST]:
            if split_name in self.config.data_files:
                files = self.config.data_files[split_name]
                if isinstance(files, str):
                    files = [files]
                splits.append(SplitGenerator(name=split_name, gen_kwargs={"files": files}))
        return splits

    def _prepare_split(self, split_generator):
        fname = "{}-{}.arrow".format(self.name, split_generator.name)
        fpath = os.path.join(self._cache_dir, fname)

        writer = ArrowWriter(path=fpath)

        generator = self._generate_tables(**split_generator.gen_kwargs)
        for key, table in utils.tqdm(generator, unit=" tables", leave=False):
            writer.write_table(table)
        num_examples, num_bytes = writer.finalize()

        split_generator.split_info.num_examples = num_examples
        split_generator.split_info.num_bytes = num_bytes
        # this is where the error is coming from
        # def parse_schema(schema, schema_dict):
        #     for field in schema:
        #         if pa.types.is_struct(field.type):
        #             schema_dict[field.name] = {}
        #             parse_schema(field.type, schema_dict[field.name])
        #         elif pa.types.is_list(field.type) and pa.types.is_struct(field.type.value_type):
        #             schema_dict[field.name] = {}
        #             parse_schema(field.type.value_type, schema_dict[field.name])
        #         else:
        #             schema_dict[field.name] = Value(str(field.type))
        #
        # parse_schema(writer.schema, features)
        # self.info.features = Features(features)

    def _generate_tables(self, files):
        for i, file in enumerate(files):
            pa_table = paj.read_json(
                file
            )
            yield i, pa_table
```

So I basically just don't populate the `self.info.features` though this doesn't seem to cause any problems in my downstream applications.

The other workaround I was doing was to just use pyarrow.json to build a table and then to create the Dataset with its constructor or from_table methods. `load_dataset` has nice split logic, so I'd prefer to use that.

Also noticed that if you for example in a loader script

```
from nlp import ArrowBasedBuilder

class MyBuilder(ArrowBasedBuilder):
...

```
and use that in the subclass, it will be on the module's __dict__ and will be selected before the `MyBuilder` subclass, and it will raise `NotImplementedError` on its `_generate_examples` method... In the code it check for abstract classes but Builder and ArrowBasedBuilder aren't abstract classes, they're regular classes with `@abstract_methods`.

Indeed this is part of a more general limitation which is the fact that we should generate and update the `features` from the auto-inferred Arrow schema when they are not provided (also happen when a user change the schema using `map()`, the features should be auto-generated and guessed as much as possible to keep the `features` synced with the underlying Arrow table schema).

We will try to solve this soon.

I tried using the Json dataloader to load some JSON lines files. but get an exception in the parse_schema function.

```
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-23-9aecfbee53bd> in <module>
     55 from nlp import load_dataset
     56
---> 57 ds = load_dataset("../text2struct/model/dataset_builder.py", data_files=rel_datafiles)
     58
     59

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    430                 verify_infos = not save_infos and not ignore_verifications
    431                 self._download_and_prepare(
--> 432                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    433                 )
    434                 # Sync info

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    481             try:
    482                 # Prepare split will record examples associated to the split
--> 483                 self._prepare_split(split_generator, **prepare_split_kwargs)
    484             except OSError:
    485                 raise OSError("Cannot find data file. " + (self.manual_download_instructions or ""))

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in _prepare_split(self, split_generator)
    736                     schema_dict[field.name] = Value(str(field.type))
    737
--> 738         parse_schema(writer.schema, features)
    739         self.info.features = Features(features)
    740

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/builder.py in parse_schema(schema, schema_dict)
    734                     parse_schema(field.type.value_type, schema_dict[field.name])
    735                 else:
--> 736                     schema_dict[field.name] = Value(str(field.type))
    737
    738         parse_schema(writer.schema, features)

<string> in __init__(self, dtype, id, _type)

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in __post_init__(self)
     55
     56     def __post_init__(self):
---> 57         self.pa_type = string_to_arrow(self.dtype)
     58
     59     def __call__(self):

~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/nlp/features.py in string_to_arrow(type_str)
     32         if str(type_str + "_") not in pa.__dict__:
     33             raise ValueError(
---> 34                 f"Neither {type_str} nor {type_str + '_'} seems to be a pyarrow data type. "
     35                 f"Please make sure to use a correct data type, see: "
     36                 f"https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions"

ValueError: Neither list<item: string> nor list<item: string>_ seems to be a pyarrow data type. Please make sure to use a correct data type, see: https://arrow.apache.org/docs/python/api/datatypes.html#factory-functions
```

If I create the dataset imperatively, using a pyarrow table, the dataset is created correctly. If I override the `_prepare_split` method to avoid calling the validate schema, the dataset can load as well.


can't load SNLI dataset

I just added the processed files of `snli` on our google storage, so that when you do `load_dataset` it can download the processed files from there :)

We are thinking about having available those processed files for more datasets in the future, because sometimes files aren't available (like for `snli`), or the download speed is too slow, or sometimes the files take time to be processed.

Closing this one. Feel free to re-open if you have other questions :)

`nlp` seems to load `snli` from some URL based on nlp.stanford.edu. This subdomain is frequently down -- including right now, when I'd like to load `snli` in a Colab notebook, but can't.

Is there a plan to move these datasets to huggingface servers for a more stable solution?

Btw, here's the stack trace:

```
File "/content/nlp/src/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/content/nlp/src/nlp/builder.py", line 466, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/content/nlp/src/nlp/datasets/snli/e417f6f2e16254938d977a17ed32f3998f5b23e4fcab0f6eb1d28784f23ea60d/snli.py", line 76, in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 217, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in download
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/py_utils.py", line 190, in map_nested
    return function(data_struct)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in <lambda>
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 198, in cached_path
    local_files_only=download_config.local_files_only,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://nlp.stanford.edu/projects/snli/snli_1.0.zip
```


[Dataset requests] New datasets for Text Classification

help wanted

Extra attention is needed

dataset request

Requesting to add a new dataset

- `nlp` has MR! It's called `rotten_tomatoes`
- SST is part of GLUE, or is that just SST-2?
- `nlp` also has `ag_news`, a popular news classification dataset

I'd also like to see:
- the Yahoo Answers topic classification dataset
- the Kaggle Fake News classification dataset

Thanks @jxmorris12 for pointing this out.

In glue we only have SST-2 maybe we can add separately SST-1.

This is the homepage for the Amazon dataset: https://www.kaggle.com/datafiniti/consumer-reviews-of-amazon-products

Is there an easy way to download kaggle datasets programmatically? If so, I can add this one!

Hi @jxmorris12 for now I think our `dl_manager` does not download from Kaggle.
@thomwolf , @lhoestq

Pretty sure the quora dataset is the same one I implemented here: https://github.com/huggingface/nlp/pull/366

Great list. Any idea if Amazon Reviews has been added?

- ~40 GB of text (sadly no emoji)
- popular MLM pre-training dataset before bigger datasets like WebText https://arxiv.org/abs/1808.01371
- turns out that binarizing the 1-5 star rating leads to great Pos/Neg/Neutral dataset, T5 paper claims to get very high accuracy (98%!) on this with small amount of finetuning https://arxiv.org/abs/2004.14546

Apologies if it's been included (great to see where) and if not, it's one of the better medium/large NLP dataset for semi-supervised learning, albeit a bit out of date.

Thanks!!

cc @sshleifer

On the Amazon Reviews dataset, the original UCSD website has noted these are now updated to include product reviews through 2018 -- actually quite recent compared to many other datasets. Almost certainly the largest NLP dataset out there with labels!
https://jmcauley.ucsd.edu/data/amazon/

Any chance someone has time to onboard this dataset in a HF way?

cc @sshleifer

We are missing a few datasets for Text Classification which is an important field.

Namely, it would be really nice to add:
- TREC-6 dataset (see here for instance: https://pytorchnlp.readthedocs.io/en/latest/source/torchnlp.datasets.html#torchnlp.datasets.trec_dataset)  **[done]**
- Yelp-5
- Movie review (Movie Review (MR) dataset [156]) **[done (same as rotten_tomatoes)]**
- SST (Stanford Sentiment Treebank) **[include in glue]**
- Multi-Perspective Question Answering (MPQA) dataset **[require authentication (indeed manual download)]**
- Amazon. This is a popular corpus of product reviews collected from the Amazon website [159]. It contains labels for both binary classification and multi-class (5-class) classification
- 20 Newsgroups. The 20 Newsgroups dataset  **[done]**
- Sogou News dataset **[done]**
- Reuters news. The Reuters-21578 dataset [165] **[done]**
- DBpedia. The DBpedia dataset [170]
- Ohsumed. The Ohsumed collection [171] is a subset of the MEDLINE database
- EUR-Lex. The EUR-Lex dataset
- WOS. The Web Of Science (WOS) dataset **[done]**
- PubMed. PubMed [173]
- TREC-QA. TREC-QA
- Quora. The Quora dataset [180]

All these datasets are cited in https://arxiv.org/abs/2004.03705


'cp950' codec error from load_dataset('xtreme', 'tydiqa')

This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
See issues #242 and #307

It should be in `xtreme.py:L755`:
```python
        if self.config.name == "tydiqa" or self.config.name.startswith("MLQA") or self.config.name == "SQuAD":
            with open(filepath) as f:
                data = json.load(f)
```

Could you try to add the encoding parameter:
```python
open(filepath, encoding='utf-8')
```

Hello @jerryIsHere :) Did it work ?
If so we may change the dataset script to force the utf-8 encoding

@lhoestq sorry for being that late, I found 4 copy of xtreme.py. I did the changes as what has been told to all of them.
The problem is not solved

Could you provide a better error message so that we can make sure it comes from the opening of the `tydiqa`'s json files ?

@lhoestq
The error message is same as before:
Exception has occurred: UnicodeDecodeError
'cp950' codec can't decode byte 0xe2 in position 111: illegal multibyte sequence
  File "D:\python\test\test.py", line 3, in <module>
    dataset = load_dataset('xtreme', 'tydiqa')

![image](https://user-images.githubusercontent.com/50871412/87748794-7c216880-c829-11ea-94f0-7caeacb4d865.png)

I said that I found 4 copy of xtreme.py and add the 「, encoding='utf-8'」 parameter to the open() function
these python script was found under this directory
C:\Users\USER\AppData\Local\Programs\Python\Python37\Lib\site-packages\nlp\datasets\xtreme

Hi there !
I encountered the same issue with the IMDB dataset on windows. It threw an error about charmap not being able to decode a symbol during the first time I tried to download it. I checked on a remote linux machine I have, and it can't be reproduced.
I added ```encoding='UTF-8'``` to both lines that have ```open``` in ```imdb.py```  (108 and 114) and it worked for me.
Thank you !

> Hi there !
> I encountered the same issue with the IMDB dataset on windows. It threw an error about charmap not being able to decode a symbol during the first time I tried to download it. I checked on a remote linux machine I have, and it can't be reproduced.
> I added `encoding='UTF-8'` to both lines that have `open` in `imdb.py` (108 and 114) and it worked for me.
> Thank you !

Hello !
Glad you managed to fix this issue on your side.
Do you mind opening a PR for IMDB ?

> This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
> Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
> See issues #242 and #307

Sorry for not responding for about a month.
I have just found that it is necessary to change / add the environment variable as what was told in #242.
Everything works after I add the new environment variable and restart my PC.

I think the encoding issue for windows isn't limited to the open() function call specific to few dataset, but actually in the entire library, depends on the machine / os you use.

Since #481 we shouldn't have other issues with encodings as they need to be set to "utf-8" be default.

Closing this one, but feel free to re-open if you gave other questions

![image](https://user-images.githubusercontent.com/50871412/86744744-67481680-c06c-11ea-8612-b77eba92a392.png)

I guess the error is related to python source encoding issue that my PC is trying to decode the source code with wrong encoding-decoding tools, perhaps :
https://www.python.org/dev/peps/pep-0263/

I guess the error was triggered by the code " module = importlib.import_module(module_path)" at line 57 in the source code:  nlp/src/nlp/load.py / (https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/src/nlp/load.py#L51)

Any ideas?

p.s. tried the same code on colab, that runs perfectly


Supporting documents in ELI5

Hi @saverymax ! For licensing reasons, the original team was unable to release pre-processed CommonCrawl documents. Instead, they provided a script to re-create them from a CommonCrawl dump, but it unfortunately requires access to a medium-large size cluster:
https://github.com/facebookresearch/ELI5#downloading-support-documents-from-the-commoncrawl

In order to make the task accessible to people who may not have access to this kind of infrastructure, we suggest to use Wikipedia as a knowledge source rather than the full CommonCrawl. The following blog post shows how you can create Wikipedia support documents and get a performance that is on par with a system that uses CommonCrawl pages.
https://yjernite.github.io/lfqa.html#task_description

Hope that helps, using ElasticSearch to index Wiki40b and create the documents should take about 4 hours. Let us know if you have any trouble with the blog post though!

Hi, thanks for the quick response. The blog post is quite an interesting working example, thanks for sharing it.
Two follow-up points/questions about my original question:

1. Yes, I read that the facebook team could not share the CommonCrawl b/c of licensing reasons. They state "No, we are not allowed to host processed Reddit or CommonCrawl data," which indicates they could also not share the Reddit data for licensing reasons. But it seems that HuggingFace is able to share the Reddit data, so why not a subset of CommonCrawl?

2. Thanks for the suggestion about ElasticSearch and Wiki40b. This is good to know about performance. I definitely could do the indexing and querying myself. What I like about the ELI5 dataset though, at least what is suggested by the paper, is that to create the dataset they had already selected the top 100 web sources and made a single support document from those. Though it doesn't appear to be too sophisticated an approach, having a single support document pre-computed (without having to run the facebook code or a replacement with another dataset) is super useful for my work, especially since I'm not working on developing the latest and greatest retrieval model. Of course, I don't expect HF NLP datasets to be perfectly tailored to my use-case. I know there is overhead to any project, I'm just illustrating a use-case of ELI5 which is not possible with the data provided as-is. If it's for licensing reasons, that is perfectly acceptable a reason, and I appreciate your response.

I was attempting to use the ELI5 dataset, when I realized that huggingface does not provide the supporting documents (the source documents from the common crawl). Without the supporting documents, this makes the dataset about as useful for my project as a block of cheese, or some other more apt metaphor.  According to facebook, the entire document collection is quite large. However, it would still be helpful to at least include a subset of the supporting documents i.e., having some data is better than having a block of cheese, in my case at least.

If you choose not to include them, it would be helpful to have documentation mentioning this specifically. It is especially confusing because the hf nlp ELI5 dataset has the key `'document'` but there are no documents to be found :(


Features should be updated when `map()` changes schema

`dataset.column_names` are being updated but `dataset.features` aren't indeed...

`dataset.map()` can change the schema and column names.

We should update the features in this case (with what is possible to infer).


[Feature request] Export Arrow dataset to TFRecords

The TFRecord generation process is error-prone and requires complex separate Python scripts to download and preprocess the data. I propose to combine the user-friendly features of `nlp` with the speed and efficiency of TFRecords. Sample API:

```python
# use these existing methods
ds = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
ds = ds.map(lambda ex: tokenizer(ex))
ds.set_format("tensorflow", columns=["input_ids", "token_type_ids", "attention_mask"])
# then add this method
ds.export(folder="/my/tfrecords", prefix="myrecord", num_shards=8, format="tfrecord")
```
which would create files like so:
```bash
/my/tfrecords/myrecord_1.tfrecord
/my/tfrecords/myrecord_2.tfrecord
...
```

I would be happy to contribute this method. We could use a similar approach for PyTorch. Thoughts?


[Dataset requests] New datasets for Open Question Answering

help wanted

Extra attention is needed

dataset request

Requesting to add a new dataset

We are still a few datasets missing for Open-Question Answering which is currently a field in strong development.

Namely, it would be really nice to add:
- WebQuestions (Berant et al., 2013) [done]
- CuratedTrec (Baudis et al. 2015) [not open-source]
- MS-MARCO (NGuyen et al. 2016) [done]
- SearchQA (Dunn et al. 2017) [done]
- FEVER (Thorne et al. 2018) - [ done]

All these datasets are cited in http://arxiv.org/abs/2005.11401