konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Nested sequences with dicts

Am pretty much finished [adding a dataset](https://github.com/ghomasHudson/nlp/blob/DocRED/datasets/docred/docred.py) for [DocRED](https://github.com/thunlp/DocRED), but am getting an error when trying to add a nested `nlp.features.sequence(nlp.features.sequence({key:value,...}))`. 

The original data is in this format:
```python
{
  'title': "Title of wiki page",
  'vertexSet': [
                  [
                    { 'name': "mention_name", 
                      'sent_id': "mention in which sentence", 
                      'pos': ["postion of mention in a sentence"], 
                      'type': "NER_type"},
                    {another mention}
                  ], 
                  [another entity]
                ]
    ...
}
```
So to represent this I've attempted to write:
```
...
features=nlp.Features({
    "title": nlp.Value("string"),
    "vertexSet": nlp.features.Sequence(nlp.features.Sequence({
        "name": nlp.Value("string"),
        "sent_id": nlp.Value("int32"),
        "pos": nlp.features.Sequence(nlp.Value("int32")),
        "type": nlp.Value("string"),
    })),
    ...
    }),
...
```
This is giving me the error:
```
pyarrow.lib.ArrowTypeError: Could not convert [{'pos': [[0,2], [2,4], [3,5]], "type": ["ORG", "ORG", "ORG"], "name": ["Lark Force", "Lark Force", "Lark Force", "sent_id": [0, 3, 4]}..... with type list: was not a dict, tuple, or recognized null value for conversion to struct type
```
Do we expect the pyarrow stuff to break when doing this deeper nesting? I've checked that it still works when you do `nlp.features.Sequence(nlp.features.Sequence(nlp.Value("string"))` or `nlp.features.Sequence({key:value,...})` just not nested sequences with a dict.

If it's not possible, I can always convert it to a shallower structure. I'd rather not change the DocRED authors' structure if I don't have to though.

Oh yes, this is a backward compatibility feature with tensorflow_dataset in which a `Sequence` or `dict` is converted in a `dict` of `lists`, unfortunately it is not very intuitive, see here: https://github.com/huggingface/nlp/blob/master/src/nlp/features.py#L409

To avoid this behavior, you can just define the list in the feature with a simple list or a tuple (which is also simpler to write).
In your case, the features could be as follow:
``` python
...
features=nlp.Features({
    "title": nlp.Value("string"),
    "vertexSet": [[{
        "name": nlp.Value("string"),
        "sent_id": nlp.Value("int32"),
        "pos": nlp.features.Sequence(nlp.Value("int32")),
        "type": nlp.Value("string"),
    }]],
    ...
    }),
...
```
Adding a dataset with multiple subtasks

I intent to add the datasets of the MT Quality Estimation shared tasks to `nlp`. However, they have different subtasks -- such as word-level, sentence-level and document-level quality estimation, each of which having different language pairs, and some of the data reused in different subtasks.

For example, in [QE 2019,](http://www.statmt.org/wmt19/qe-task.html) we had the same English-Russian and English-German data for word-level and sentence-level QE. 

I suppose these datasets could have both their word and sentence-level labels inside `nlp.Features`; but what about other subtasks? Should they be considered a different dataset altogether?

I read the discussion on #217 but the case of QE seems a lot simpler.

For one dataset you can have different configurations that each have their own `nlp.Features`.
We imagine having one configuration per subtask for example.
They are loaded with `nlp.load_dataset("my_dataset", "my_config")`.

For example the `glue` dataset has many configurations. It is a bit different from your case though because each configuration is a dataset by itself (sst2, mnli).
Another example is `wikipedia` that has one configuration per language.
Add MWSC

Adding the [Modified Winograd Schema Challenge](https://github.com/salesforce/decaNLP/blob/master/local_data/schema.txt) dataset which formed part of the [decaNLP](http://decanlp.com/) benchmark. Not sure how much use people would find for it it outside of the benchmark, but it is general purpose.

Code is heavily borrowed from the [decaNLP repo](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L773-L877).

There's a few (possibly overly opinionated) design choices I made:

- I used the train/test/dev split [buried in the decaNLP code](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L852-L855)
- I split out each example into the 2 alternatives. Originally the data uses the format:
    ```
    The city councilmen refused the demonstrators a permit because they [feared/advocated] violence. 
    Who [feared/advocated] violence? 
    councilmen/demonstrators
    ```
    I split into the 2 variants:
    ```
    The city councilmen refused the demonstrators a permit because they feared violence. 
    Who feared violence? 
    councilmen/demonstrators
    
    The city councilmen refused the demonstrators a permit because they advocated violence. 
    Who advocated violence? 
    councilmen/demonstrators
    ```
    I can't see any use for having the options combined into a single example (splitting them is [the way decaNLP processes](https://github.com/salesforce/decaNLP/blob/1e9605f246b9e05199b28bde2a2093bc49feeeaa/text/torchtext/datasets/generic.py#L846-L850)) them. You can't train on both versions with them combined, and splitting the examples later would be a pain to do. I think [winogrande.py](https://github.com/huggingface/nlp/blob/master/datasets/winogrande/winogrande.py) presents the data in this way?

- I've not used the decaNLP framing (appending the options to the question e.g. `Who feared violence? 
 -- councilmen or demonstrators?`) but left it more generic by adding the options as a new key: `"options":["councilmen","demonstrators"]` This should be an easy thing to change using `map` if needed by a specific application.

Dataset is working as-is but if anyone has any thoughts/preferences on the design decisions here I'm definitely open to different choices.

Looks good to me
[Feature request] Add `shard()` method to dataset

Currently, to shard a dataset into 10 pieces on different ranks, you can run

```python
rank = 3 # for example
size = 10
dataset = nlp.load_dataset('wikitext', 'wikitext-2-raw-v1', split=f"train[{rank*10}%:{(rank+1)*10}%]")
```

However, this breaks down if you have a number of ranks that doesn't divide cleanly into 100, such as 64 ranks. Is there interest in adding a method shard() that looks like this?

```python
rank = 3
size = 64
dataset = nlp.load_dataset("wikitext", "wikitext-2-raw-v1", split="train").shard(rank=rank, size=size)
```

TensorFlow has a similar API: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shard. I'd be happy to contribute this code.

Hi Jared,
Interesting, thanks for raising this question. You can also do that after loading with `dataset.select()` or `dataset.filter()` which let you keep only a specific subset of rows in a dataset.
What is your use-case for sharding?

Thanks for the pointer to those functions! It's still a little more verbose since you have to manually calculate which ids each rank would keep, but definitely works.

My use case is multi-node, multi-GPU training and avoiding global batches of duplicate elements. I'm using horovod. You can shuffle indices, or set random seeds, but explicitly sharding the dataset up front is the safest and clearest way I've found to do so.
add wikisql

Adding the [WikiSQL](https://github.com/salesforce/WikiSQL) dataset.

Interesting things to note:
- Have copied the function (`_convert_to_human_readable`) which converts the SQL query to a human-readable (string) format as this is what most people will want when actually using this dataset for NLP applications.
- `conds` was originally a tuple but is converted to a dictionary to support differing types.

Would be nice to add the logical_form metrics too at some point.

That's great work @ghomasHudson !
add pg19 dataset

https://github.com/huggingface/nlp/issues/274

Add functioning PG19 dataset with dummy data

`cos_e.py` was just auto-linted by `make style`

@lucidrains - Thanks a lot for making the PR - PG19 is a super important dataset! Thanks for making it. Many people are asking for PG-19, so it would be great to have that in the library as soon as possible @thomwolf .

@mariamabarham yup! around 11GB!

I'm looking forward to our first deep learning written novel already lol. It's definitely happening

Good to merge IMO.

Oh I just noticed but as we changed the urls to download the files, we have to update `dataset_infos.json`.
Could you re-rurn `nlp-cli test ./datasets/pg19 --save_infos` ?

@lhoestq on it!

should be good!

@lhoestq - I think it's good to merge no?

`dataset_infos.json` is still not up to date with the new urls (we can see that there are urls like `gs://deepmind-gutenberg/train/*` instead of `https://storage.googleapis.com/deepmind-gutenberg/train/*` in the json file)

Can you check that you re-ran the command to update the json file, and that you pushed the changes @lucidrains ?

@lhoestq ohhh, I made the change in this commit https://github.com/lucidrains/nlp/commit/f3e23d823ad9942031be80b7c4e4212c592cd90c , that's interesting that the pull request didn't pick it up. maybe it's because I did it on another machine, let me check and get back to you!

@lhoestq wrong branch 😅 thanks for catching! 

Awesome thanks 🎉
Importing downloaded package repository fails

The `get_imports` function in `src/nlp/load.py` has a feature to download a package as a zip archive of the github repository and import functions from the unpacked directory. This is used for example in the `metrics/coval.py` file, and would be useful to add BLEURT (@ankparikh).

Currently however, the code seems to have trouble with imports within the package. For example:
```
import nlp
coval = nlp.load_metric('coval')
```
yields:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yacine/Code/nlp/src/nlp/load.py", line 432, in load_metric
    metric_cls = import_main_class(module_path, dataset=False)
  File "/home/yacine/Code/nlp/src/nlp/load.py", line 57, in import_main_class
    module = importlib.import_module(module_path)
  File "/home/yacine/anaconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/yacine/Code/nlp/src/nlp/metrics/coval/a78807df33ac45edbb71799caf2b3b47e55df4fd690267808fe963a5e8b30952/coval.py", line 21, in <module>
    from .coval_backend.conll import reader  # From: https://github.com/ns-moosavi/coval
  File "/home/yacine/Code/nlp/src/nlp/metrics/coval/a78807df33ac45edbb71799caf2b3b47e55df4fd690267808fe963a5e8b30952/coval_backend/conll/reader.py", line 2, in <module>
    from conll import mention
ModuleNotFoundError: No module named 'conll'
```

Not sure what the fix would be there.
Problem while printing doc string when instantiating multiple metrics.

When I load more than one metric and try to print doc string of a particular metric,. It shows the doc strings of all imported metric one after the other which looks quite confusing and clumsy.
Attached [Colab](https://colab.research.google.com/drive/13H0ZgyQ2se0mqJ2yyew0bNEgJuHaJ8H3?usp=sharing) Notebook for problem clarification..
Question - Sign Language Datasets

An emerging field in NLP is SLP - sign language processing.

I was wondering about adding datasets here, specifically because it's shaping up to be large and easily usable.
The metrics for sign language to text translation are the same.

So, what do you think about (me, or others) adding datasets here?


An example dataset would be [RWTH-PHOENIX-Weather 2014 T](https://www-i6.informatik.rwth-aachen.de/~koller/RWTH-PHOENIX-2014-T/)
For every item in the dataset, the data object includes:
1. video_path - path to mp4 file
2. pose_path - a path to `.pose` file with human pose landmarks
3. openpose_path - a path to a `.json` file with human pose landmarks
4. gloss - string
5. text - string
6. video_metadata - height, width, frames, framerate


------

To make it a tad more complicated - what if sign language libraries add requirements to `nlp`? for example, sign language is commonly annotated using `ilex`, `eaf`, or `srt` files, which are all loadable as text, but there is no reason for the dataset to parse that file by itself, if libraries exist to do so.

Even more complicating - 

As I see it, datasets can have "addons".
For example, the WebNLG dataset is a dataset for data-to-text. However, a work of mine and other works enriched this dataset with text plans / underlying text structures. In that case, I see a need to load the dataset "WebNLG" with "plans" addon.

Same for sign language - if there is a dataset of videos, one addon can be to run OpenPose, another to run ARKit4 pose estimation, and another to run PoseNet, or even just a video embedding addon. (which are expensive to run individually for everyone who wants to use these data)

This is something I dabbled with my own implementation to a [research datasets library](https://github.com/AmitMY/meta-scholar/) and I love to get the discussion going on these topics.

This is a really cool idea !
The example for data objects you gave for the RWTH-PHOENIX-Weather 2014 T dataset can totally fit inside the library.

For your point about formats like `ilex`, `eaf`, or `srt`, it is possible to use any library in your dataset script.
However most user probably won't need these libraries, as most datasets don't need them, and therefore it's unlikely that we will have them in the minimum requirements to use `nlp` (we want to keep it as light-weight as possible). If a user wants to load your dataset and doesn't have the libraries you need, an error is raised asking the user to install them.

More generally, we plan to have something like a `requirements.txt` per dataset. This could also be a place for addons as you said. What do you think ?

Thanks, Quentin, I think a `requirements.txt` per dataset will be a good thing.
I will work on adding this dataset next week, and once we sort all of the kinks, I'll add more.
Setting cache_dir gives error on wikipedia download

First of all thank you for a super handy library! I'd like to download large files to a specific drive so I set `cache_dir=my_path`. This works fine with e.g. imdb and squad. But on wikipedia I get an error:
```
nlp.load_dataset('wikipedia', '20200501.de', split = 'train', cache_dir=my_path)
```
```
OSError                                   Traceback (most recent call last)
<ipython-input-2-23551344d7bc> in <module>
      1 import nlp
----> 2 nlp.load_dataset('wikipedia', '20200501.de', split = 'train', cache_dir=path)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    522         download_mode=download_mode,
    523         ignore_verifications=ignore_verifications,
--> 524         save_infos=save_infos,
    525     )
    526 

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    385                     with utils.temporary_assignment(self, "_cache_dir", tmp_data_dir):
    386                         reader = ArrowReader(self._cache_dir, self.info)
--> 387                         reader.download_from_hf_gcs(self._cache_dir, self._relative_data_dir(with_version=True))
    388                         downloaded_info = DatasetInfo.from_directory(self._cache_dir)
    389                         self.info.update(downloaded_info)

~/anaconda3/envs/fastai2/lib/python3.7/site-packages/nlp/arrow_reader.py in download_from_hf_gcs(self, cache_dir, relative_data_dir)
    231             remote_dataset_info = os.path.join(remote_cache_dir, "dataset_info.json")
    232             downloaded_dataset_info = cached_path(remote_dataset_info)
--> 233             os.rename(downloaded_dataset_info, os.path.join(cache_dir, "dataset_info.json"))
    234             if self._info is not None:
    235                 self._info.update(self._info.from_directory(cache_dir))

OSError: [Errno 18] Invalid cross-device link: '/home/local/NTU/nn/.cache/huggingface/datasets/025fa4fd4f04aaafc9e939260fbc8f0bb190ce14c61310c8ae1ddd1dcb31f88c.9637f367b6711a79ca478be55fe6989b8aea4941b7ef7adc67b89ff403020947' -> '/data/nn/nlp/wikipedia/20200501.de/1.0.0.incomplete/dataset_info.json'
```

Whoops didn't mean to close this one.
I did some changes, could you try to run it from the master branch ?

Now it works, thanks!
Error in Demo for Specific Datasets

Selecting `natural_questions` or `newsroom` dataset in the online demo results in an error similar to the following.

![image](https://user-images.githubusercontent.com/60150701/85347842-ac861900-b4ae-11ea-98c4-a53a00934783.png)


Thanks for reporting these errors :)

I can actually see two issues here.

First, datasets like `natural_questions` require apache_beam to be processed. Right now the import is not at the right place so we have this error message. However, even the imports are fixed, the nlp viewer doesn't actually have the resources to process NQ right now so we'll have to wait until we have a version that we've already processed on our google storage (that's what we've done for wikipedia for example).

Second, datasets like `newsroom` require manual downloads as we're not allowed to redistribute the data ourselves (if I'm not wrong). An error message should be displayed saying that we're not allowed to show the dataset.

I can fix the first issue with the imports but for the second one I think we'll have to see with @srush to show a message for datasets that require manual downloads (it can be checked whether a dataset requires manual downloads if `dataset_builder_instance.manual_download_instructions is not None`).



I added apache-beam to the viewer. We can think about how to add newsroom. 

We don't plan to host the source files of newsroom ourselves for now.
You can still get the dataset if you follow the download instructions given by `dataset = load_dataset('newsroom')` though.
The viewer also shows the instructions now.

Closing this one. If you have other questions, feel free to re-open :)
snli -1 labels

I'm trying to train a model on the SNLI dataset. Why does it have so many -1 labels?
```
import nlp
from collections import Counter
data = nlp.load_dataset('snli')['train']
print(Counter(data['label']))
Counter({0: 183416, 2: 183187, 1: 182764, -1: 785})
```


@jxmorris12  , we use `-1` to label examples for which `gold label`  is missing (`gold label = -` in the original dataset). 

Thanks @mariamabarham! so the original dataset is missing some labels? That is weird. Is standard practice just to discard those examples training/eval?

Yes the original dataset is missing some labels maybe  @sleepinyourhat , @gangeli can correct me if I'm wrong 
For my personal opinion at least if you want your model to learn to predict no answer (-1) you can leave it their but otherwise you can discard them. 

thanks @mariamabarham  :)
Update metadata for x_stance dataset

Thank you for featuring the x_stance dataset in your library. This PR updates some metadata:
- Citation: Replace preprint with proceedings
- URL: Use a URL with long-term availability


Great! Thanks @jvamvas for these updates.


I have fixed a warning. The remaining test failure is due to an unrelated dataset.

We just fixed the other dataset on master. Could you rebase from master and push to rerun the CI ?
break statement not required


I guess,test failing due to connection error?

We just fixed the other dataset on master. Could you rebase from master and push to rerun the CI ?

If I'm not wrong this function returns None if no main class was found.
I think it makes things less clear not to have a return at the end of the function.
I guess we can have one return in the for loop instead of the break statement, AND one return at the end to explicitly return None.
What do you think ?
ConnectionError - Eli5 dataset download

Hi, I have a problem with downloading Eli5 dataset. When typing `nlp.load_dataset('eli5')`, I get ConnectionError: Couldn't reach https://storage.googleapis.com/huggingface-nlp/cache/datasets/eli5/LFQA_reddit/1.0.0/explain_like_im_five-train_eli5.arrow

I would appreciate if you could help me with this issue.

It should ne fixed now, thanks for reporting this one :)
It was an issue on our google storage.

Let me now if you're still facing this issue.

It works now, thanks for prompt help!
update xsum

This PR makes the following update to the xsum dataset:

- Manual download is not required anymore

- dataset can be loaded as follow: `nlp.load_dataset('xsum')`


**Important** 
Instead of using on outdated url to download the data:  "https://raw.githubusercontent.com/EdinburghNLP/XSum/master/XSum-Dataset/XSum-TRAINING-DEV-TEST-SPLIT-90-5-5.json" 

a more up-to-date url stored here: https://s3.amazonaws.com/datasets.huggingface.co/summarization/xsum.tar.gz is used
, so that the user does not need to manually download the data anymore. 
There might be slight breaking changes here for xsum. 

Looks cool!
@mariamabarham can you add a detailed description here what exactly is changed and how the user can load xsum now?

And a rebase should solve the conflicts

This is a super useful PR :-) @sshleifer - maybe you can take a look at the updated version of xsum if you can use it for your use case. Now, one should be able to just load it with:

```python 
nlp.load_datasets("xsum", ....)  # no manual dir required anymore
```
Fix manual download instructions

This PR replaces the static `DatasetBulider` variable `MANUAL_DOWNLOAD_INSTRUCTIONS` by a property function `manual_download_instructions()`. 

Some datasets like XTREME and all WMT need the manual data dir only for a small fraction of the possible configs.

After some brainstorming with @mariamabarham and @lhoestq, we came to the conclusion that having a property function `manual_download_instructions()` gives us more flexibility to decide on a per config basis in the dataset builder if manual download instructions are needed.

Also this PR should unblock solves a bug with `wmt16 - ro-en` 
@sshleifer from this branch you should be able to succesfully run

```python 
import nlp 
ds = nlp.load_dataset('./datasets/wmt16', 'ro-en')
```

and once this PR is merged S3 should be synched so that 

```python
import nlp
ds = nlp.load_dataset("wmt16", "ro-en")
```

works as well.

**Important**: Since `MANUAL_DOWNLOAD_INSTRUCTIONS` was not really exposed to the user, this PR should not be a problem regarding backward compatibility.

Verified that this works, thanks!

But I get
```python
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py
```
When I try from jupyter on brutasse or my mac. (the jupyter server is run from transformers).


Both machines can run
```bash
aws s3 ls s3://datasets.huggingface.co/nlp/datasets/wmt16/
```
but it seems one must be in the nlp directory to run the command?

(I ran `pip install -e . ` on this branch in both situations.)




`https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py` looks very weird.

(Also, S3 is not a file-system, it's a flat key-value store)

Good to merge I think @lhoestq 

> But I get
> 
> ```python
> ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/datasets/./datasets/wmt16/wmt16.py
> ```
> 
> When I try from jupyter on brutasse or my mac. (the jupyter server is run from transformers).
> 
> Both machines can run
> 
> ```shell
> aws s3 ls s3://datasets.huggingface.co/nlp/datasets/wmt16/
> ```
> 
> but it seems one must be in the nlp directory to run the command?
> 
> (I ran `pip install -e . ` on this branch in both situations.)

As soon as it is on master, the dataset script wmt16.py will be synced on S3 and you'll be able to do `load_dataset("wmt16")`
Private/sensitive data

Hi all,
Thanks for this fantastic library, it makes it very easy to do prototyping for NLP projects interchangeably between TF/Pytorch. 

Unfortunately, there is data that cannot easily be shared publicly as it may contain sensitive information. 
Is there support/a plan to support such data with NLP, e.g. by reading it from local sources?

Use case flow could look like this: use NLP to prototype an approach on similar, public data and apply the resulting prototype on sensitive/private data without the need to rethink data processing pipelines. 

Many thanks for your responses ahead of time and kind regards,
MFreidank

Hi @MFreidank, you should already be able to load a dataset from local sources, indeed. (ping @lhoestq and @jplu)

We're also thinking about the ability to host private datasets on a hosted bucket with permission management, but that's further down the road.

Hi @MFreidank, it is possible to load a dataset from your local storage, but only CSV/TSV and JSON are supported. To load a dataset in JSON format:

```
nlp.load_dataset(path="json", data_files={nlp.Split.TRAIN: ["path/to/train.json"], nlp.Split.TEST: ["path/to/test.json"]})
```

For CSV/TSV datasets, you have to replace `json` by `csv`.

Hi @julien-c @jplu,
Thanks for sharing this solution with me, it helps, this is what I was looking for. 
If not already there and only missed by me, this could be a great addition in the docs.

Closing my issue as resolved, thanks again.
Dataset Preprocessing Cache with .map() function not working as expected

I've been having issues with reproducibility when loading and processing datasets with the `.map` function. I was only able to resolve them by clearing all of the cache files on my system. 

Is there a way to disable using the cache when processing a dataset? As I make minor processing changes on the same dataset, I want to be able to be certain the data is being re-processed rather than loaded from a cached file. 

Could you also help me understand a bit more about how the caching functionality is used for pre-processing? E.g. how is it determined when to load from a cache vs. reprocess. 
I was particularly having an issue where the correct dataset splits were loaded, but as soon as I applied the `.map()` function to each split independently, they somehow all exited this process having been converted to the test set.
Thanks!

When you're processing a dataset with `.map`, it checks whether it has already done this computation using a hash based on the function and the input (using some fancy serialization with `dill`). If you found that it doesn't work as expected in some cases, let us know !

Given that, you can still force to re-process using `.map(my_func, load_from_cache_file=False)` if you want to.

I am curious about the problem you have with splits. It makes me think about #160 that was an issue of version 0.1.0. What version of `nlp` are you running ? Could you give me more details ?

Thanks, that's helpful! I was running 0.1.0, but since upgraded to 0.2.1. I can't reproduce the issue anymore as I've cleared the cache & everything now seems to be running fine since the upgrade. I've added some checks to my code, so if I do encounter it again I will reopen this issue.

Just checking in, the cache sometimes still does not work when I make changes in my processing function in version `1.2.1`. The changes made to my data processing function only propagate to the dataset when I use `load_from_cache_file=False` or clear the cache. Is this a system-specific issue?

Hi @sarahwie 
The data are reloaded from the cache if the hash of the function you provide is the same as a computation you've done before. The hash is computed by recursively looking at the python objects of the function you provide.

If you think there's an issue, can you share the function you used or a google colab please ?

I can't reproduce it, so I'll close for now.
Empty samples in glue/qqp

```
qqp = nlp.load_dataset('glue', 'qqp')
print(qqp['train'][310121])
print(qqp['train'][362225])
```
```
{'question1': 'How can I create an Android app?', 'question2': '', 'label': 0, 'idx': 310137}
{'question1': 'How can I develop android app?', 'question2': '', 'label': 0, 'idx': 362246}
```
Notice that question 2 is empty string. 
BTW, I have checked and these two are the only naughty ones in all splits of qqp.

We are only wrapping the original dataset.

Maybe try to ask on the GLUE mailing list or reach out to the original authors?

Tanks for the suggestion, I'll try to ask GLUE benchmark.
I'll first close the issue,  post the following up here afterwards, and reopen the issue if needed.
Fix metric compute (original_instructions missing)

When loading arrow data we added in cc8d250 a way to specify the instructions that were used to store them with the loaded dataset.
However metrics load data the same way but don't need instructions (we use one single file).

In this PR I just make `original_instructions` optional when reading files to load a `Dataset` object.

This should fix #269 

Awesome! This is working now:

```python
import nlp 
seqeval = nlp.load_metric("seqeval") 
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']] 
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]                                                                                                                                                                                    

results = seqeval.compute(y_true, y_pred)
```

I heavily need this fix for an upcoming `nlp` integration PR for Transformers (token classification example)  😅

Haha nice ! We'll ship this fix with the next release that will probably come out on thursday :)
NonMatchingChecksumError when loading pubmed dataset

I get this error when i run `nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')`.
The error is:

```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-2-7742dea167d0> in <module>()
----> 1 df = nlp.load_dataset('scientific_papers', 'pubmed', split = 'train[:50%]')
      2 df = pd.DataFrame(df)
      3 gc.collect()

3 frames
/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    518         download_mode=download_mode,
    519         ignore_verifications=ignore_verifications,
--> 520         save_infos=save_infos,
    521     )
    522 

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    431                 verify_infos = not save_infos and not ignore_verifications
    432                 self._download_and_prepare(
--> 433                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    434                 )
    435                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    468         # Checksums verification
    469         if verify_infos:
--> 470             verify_checksums(self.info.download_checksums, dl_manager.get_recorded_sizes_checksums())
    471         for split_generator in split_generators:
    472             if str(split_generator.split_info.name).lower() == "all":

/usr/local/lib/python3.6/dist-packages/nlp/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums)
     34     bad_urls = [url for url in expected_checksums if expected_checksums[url] != recorded_checksums[url]]
     35     if len(bad_urls) > 0:
---> 36         raise NonMatchingChecksumError(str(bad_urls))
     37     logger.info("All the checksums matched successfully.")
     38 

NonMatchingChecksumError: ['https://drive.google.com/uc?id=1b3rmCSIoh6VhD4HKWjI4HOW-cSwcwbeC&export=download', 'https://drive.google.com/uc?id=1lvsqvsFi3W-pE1SqNZI0s8NR9rC1tsja&export=download']
```
I'm currently working on google colab.

That is quite strange because yesterday it was fine.


For some reason the files are not available for unauthenticated users right now (like the download service of this package). Instead of downloading the right files, it downloads the html of the error.
According to the error it should be back again in 24h.

![image](https://user-images.githubusercontent.com/42851186/84751599-096c6580-afbd-11ea-97f3-ee4aef791711.png)
PG-19

Hi, and thanks for all your open-sourced work, as always!

I was wondering if you would be open to adding PG-19 to your collection of datasets. https://github.com/deepmind/pg19 It is often used for benchmarking long-range language modeling.

Sounds good! Do you want to give it a try?

Ok, I'll see if I can figure it out tomorrow!

Got around to this today, and so far so good, I'm able to download and load pg19 locally. However, I think there may be an issue with the dummy data, and testing in general.

The problem lies in the fact that each book from pg19 actually resides as its own text file in a google cloud folder that denotes the split, where the book id is the name of the text file. https://console.cloud.google.com/storage/browser/deepmind-gutenberg/train/ I don't believe there's anywhere else (even in the supplied metadata), where the mapping of id -> split can be found.

Therefore I end up making a network call `tf.io.gfile.listdir` to get all the files within each of the split directories. https://github.com/lucidrains/nlp/commit/adbacbd85decc80db2347d0882e7dab4faa6fd03#diff-cece8f166a85dd927caf574ba303d39bR78

Does this network call need to be eventually stubbed out for testing?

Ohh nevermind, I think I can use `download_custom` here with `listdir` as the custom function. Ok, I'll keep trying to make the dummy data work!
Fix allociné dataset configuration

This is a patch for #244. According to the [live nlp viewer](url), the Allociné dataset must be loaded with :
```python
dataset = load_dataset('allocine', 'allocine')
```
This is redundant, as there is only one "dataset configuration", and should only be:
```python
dataset = load_dataset('allocine')
```

This is my mistake, because the code for [`allocine.py`](https://github.com/huggingface/nlp/blob/master/datasets/allocine/allocine.py) was inspired by [`imdb.py`](https://github.com/huggingface/nlp/blob/master/datasets/imdb/imdb.py), which also force the user to specify the "dataset configuration" (even if there is only one).

I believe this PR should solve this issue, making the Allociné dataset more convenient to use.

Actually when there is only one configuration, then you don't need to specify the configuration in `load_dataset`. You can run:
```python
dataset = load_dataset('allocine')
```
and it works.

Maybe we should take that into account in the nlp viewer @srush ?

@lhoestq Just to understand the exact semantics. Are you suggesting that if there is exactly 1 configuration I should not show the configuration menu and just treat it as if there were 0 configurations? 

The configuration menu is fine imo.
It was more about the code snippet presented in the viewer.
For example for Allociné it currently shows this snippet to load the dataset:
```python
!pip install nlp
from nlp import load_dataset
dataset = load_dataset('allocine', 'allocine')
```
However for datasets with one or zero configurations, the second argument in `load_dataset` is optional. For Allociné, that has one configuration, we can expect to show instead:
```python
!pip install nlp
from nlp import load_dataset
dataset = load_dataset('allocine')
```

> Actually when there is only one configuration, then you don't need to specify the configuration in `load_dataset`. You can run:
> 
> ```python
> dataset = load_dataset('allocine')
> ```
> 
> and it works.
> 
> Maybe we should take that into account in the nlp viewer @srush ?

Oh ok, I didn't expect it would work! 

Anyway, I think it's intrinsically better to simply remove the optional parameter. 
The dummy data folder architecture seems also more logical this way.


Fixed in the viewer. Checked that allocine works.

Awesome thanks :)

Closing this.
c4 dataset is not viewable in nlpviewer demo

I get the following error when I try to view the c4 dataset in [nlpviewer](https://huggingface.co/nlp/viewer/)

```python
ModuleNotFoundError: No module named 'langdetect'
Traceback:
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp_viewer/run.py", line 54, in <module>
    configs = get_confs(option.id)
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp_viewer/run.py", line 48, in get_confs
    builder_cls = nlp.load.import_main_class(module_path, dataset=True)
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/load.py", line 57, in import_main_class
    module = importlib.import_module(module_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/datasets/c4/88bb1b1435edad3fb772325710c4a43327cbf4a23b9030094556e6f01e14ec19/c4.py", line 29, in <module>
    from .c4_utils import (
File "/home/sasha/.local/lib/python3.7/site-packages/nlp/datasets/c4/88bb1b1435edad3fb772325710c4a43327cbf4a23b9030094556e6f01e14ec19/c4_utils.py", line 29, in <module>
    import langdetect
```

C4 is too large to be shown in the viewer
Error in metric.compute: missing `original_instructions` argument

I'm running into an error using metrics for computation in the latest master as well as version 0.2.1. Here is a minimal example:

```python
import nlp
rte_metric = nlp.load_metric('glue', name="rte")
rte_metric.compute(
    [0, 0, 1, 1],
    [0, 1, 0, 1],
)
```

```
    181             # Read the predictions and references
    182             reader = ArrowReader(path=self.data_dir, info=None)
--> 183             self.data = reader.read_files(node_files)
    184 
    185             # Release all of our locks

TypeError: read_files() missing 1 required positional argument: 'original_instructions'
```

I believe this might have been introduced with cc8d2508b75f7ba0e5438d0686ee02dcec43c7f4, which added the `original_instructions` argument. Elsewhere, an empty-string default is provided--perhaps that could be done here too?
How can I load/find WMT en-romanian?

I believe it is from `wmt16`

When I run

```python
wmt = nlp.load_dataset('wmt16')
```
I get:
```python
AssertionError: The dataset wmt16 with config cs-en requires manual data. 
 Please follow the manual download instructions:   Some of the wmt configs here, require a manual download.
  Please look into wmt.py to see the exact path (and file name) that has to
  be downloaded.
  . 
 Manual data can be loaded with `nlp.load(wmt16, data_dir='<path/to/manual/data>')
```
There is no wmt.py,as the error message suggests, and wmt16.py doesn't have manual download instructions.

Any idea how to do this?

Thanks in advance!




I will take a look :-)
Add sort, shuffle, test_train_split and select methods

Add a bunch of methods to reorder/split/select rows in a dataset:
- `dataset.select(indices)`: Create a new dataset with rows selected following the list/array of indices (which can have a different size than the dataset and contain duplicated indices, the only constrain is that all the integers in the list must be smaller than the dataset size, otherwise we're indexing outside the dataset...)
- `dataset.sort(column_name)`: sort a dataset according to a column (has to be a column with a numpy compatible type)
- `dataset.shuffle(seed)`: shuffle a dataset rows
- `dataset.train_test_split(test_size, train_size)`: Return a dictionary with two random train and test subsets (`train` and `test` ``Dataset`` splits)

All these methods are **not** in-place which means they return new ``Dataset``.
This is the default behavior in the library.

Fix #147 #166 #259 

Nice !

Also it looks like we can have a train_test_split method for free:
```python
train_indices, test_indices = train_test_split(range(len(dataset)))
train = dataset.sort(indices=train_indices)
test = dataset.sort(indices=test_indices)
```

and a shuffling method for free:
```python
shuffled_indices = shuffle(range(len(dataset)))
shuffled_dataset = dataset.sort(indices=shuffled_indices)
```

Maybe we can have a specific API for train_test_split and shuffle. They are two features asked quite often (see #147, #166)

Ok, I think this one is ready to merge.

@patrickvonplaten @jplu @mariamabarham @joeddav @n1t0 @julien-c you may want to give it a look, it adds a bunch of methods to reorder/split/select rows in a dataset:
- `dataset.select(indices)`: Create a new dataset with rows selected following the list/array of indices (which can have a different size than the dataset and contain duplicated indices, the only constrain is that all the integers in the list must be smaller than the dataset size, otherwise we're indexing outside the dataset...)
- `dataset.sort(column_name)`: sort a dataset according to a column (has to be a column with a numpy compatible type)
- `dataset.shuffle(seed)`: shuffle a dataset rows
- `dataset.train_test_split(test_size, train_size)`: Return a dictionary with two random train and test subsets (`train` and `test` ``Dataset`` splits)

All these methods are **not** in-place which means they return new ``Dataset``, which is the default behavior in the library.

> Might be a solution to put 0.25 and 0.75 as default values for respectively `test_size` and `train_size`. WDYT?

According to sklearn documentation, it is indeed set to 0.25 and 0.75 if both `test_size` and `train_size` are None.
Let me add it.

I think we're good to go now :) @joeddav @thomwolf @jplu
Add new dataset ANLI Round 1

Adding new dataset [ANLI](https://github.com/facebookresearch/anli/).

I'm not familiar with how to add new dataset. Let me know if there is any issue. I only include round 1 data here. There will be round 2, round 3 and more in the future with potentially different format. I think it will be better to separate them.

Hello ! Thanks for adding this one :)

This looks great, you just have to do the last steps to make the CI pass.
I can see that two things are missing:
1. the dummy data that is used to test that the script is working as expected
2. the json file with all the infos about the dataset

You can see the steps to help you create the dummy data and generate the dataset_infos.json file right [here](https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset)
Why is dataset after tokenization far more larger than the orginal one ?

I tokenize wiki dataset by `map` and cache the results.
```
def tokenize_tfm(example):
    example['input_ids'] = hf_fast_tokenizer.convert_tokens_to_ids(hf_fast_tokenizer.tokenize(example['text']))
    return example
wiki = nlp.load_dataset('wikipedia', '20200501.en', cache_dir=cache_dir)['train']
wiki.map(tokenize_tfm, cache_file_name=cache_dir/"wikipedia/20200501.en/1.0.0/tokenized_wiki.arrow")
```
and when I see their size
```
ls -l --block-size=M
17460M  wikipedia-train.arrow
47511M  tokenized_wiki.arrow
```
The tokenized one is over 2x size of original one.
Is there something I did wrong ?

Hi ! This is because `.map` added the new column `input_ids` to the dataset, and so all the other columns were kept. Therefore the dataset size increased a lot.
 If you want to only keep the `input_ids` column, you can stash the other ones by specifying `remove_columns=["title", "text"]` in the arguments of `.map`

Hi ! Thanks for your reply.

But since size of `input_ids` < size of `text`, I am wondering why
size of `input_ids` + `text` > 2x the size of `text`  🤔

Hard to tell... This is probably related to the way apache arrow compresses lists of integers, that may be different from the compression of strings.

Thanks for your point. 😀, It might be answer.
Since this is hard to know, I'll close this issue.
But if somebody knows more details, please comment below ~ 😁
[Feature request] Add a feature to dataset

Is there a straightforward way to add a field to the arrow_dataset, prior to performing map?

Do you have an example of what you would like to do? (you can just add a field in the output of the unction you give to map and this will add this field in the output table)

Given another source of data loaded in, I want to pre-add it to the dataset so that it aligns with the indices of the arrow dataset prior to performing map.

E.g. 
```
new_info = list of length dataset['train']

dataset['train'] = dataset['train'].map(lambda x: some_function(x, new_info[index of x]))

def some_function(x, new_info_x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x
    return x
```
I was thinking to instead create a new field in the arrow dataset so that instance x contains all the necessary information when map function is applied (since I don't have index information to pass to map function).

This is what I have so far: 

```
import pyarrow as pa
from nlp.arrow_dataset import Dataset

aug_dataset = dataset['train'][:]
aug_dataset['new_info'] = new_info

#reformat as arrow-table
schema = dataset['train'].schema

# this line doesn't work:
schema.append(pa.field('new_info', pa.int32()))

table = pa.Table.from_pydict(
    aug_dataset,
    schema=schema
)
dataset['train'] = Dataset(table) 
```

Maybe you can use `with_indices`?

```python
new_info = list of length dataset['train']

def some_function(indice, x):
    # adds new_info[index of x] as a field to x
    x['new_info'] = new_info_x[indice]
    return x

dataset['train'] = dataset['train'].map(some_function, with_indices=True)
```

Oh great. That should work. I missed that in the documentation- thanks :)
[Feature request] Be able to remove a specific sample of the dataset

As mentioned in #117, it's currently not possible to remove a sample of the dataset.

But it is a important use case : After applying some preprocessing, some samples might be empty for example. We should be able to remove these samples from the dataset, or at least mark them as `removed` so when iterating the dataset, we don't iterate these samples.

I think it should be a feature. What do you think ?

---

Any work-around in the meantime ?

Oh yes you can now do that with the `dataset.filter()` method that was added in #214