NyssaW/github-issues huggingface dataset:
Index outside of table length

The offset input box warns of numbers larger than a limit (like 2000) but then the errors start at a smaller value than that limit (like 1955).

> ValueError: Index (2000) outside of table length (2000).
> Traceback:
> File "/home/sasha/.local/lib/python3.7/site-packages/streamlit/ScriptRunner.py", line 322, in _run_script
>     exec(code, module.__dict__)
> File "/home/sasha/nlp_viewer/run.py", line 116, in <module>
>     v = d[item][k]
> File "/home/sasha/.local/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 338, in __getitem__
>     output_all_columns=self._output_all_columns,
> File "/home/sasha/.local/lib/python3.7/site-packages/nlp/arrow_dataset.py", line 290, in _getitem
>     raise ValueError(f"Index ({key}) outside of table length ({self._data.num_rows}).")

Comments:
Sounds like something related to the nlp viewer @srush 

Fixed.
Scientific Papers only downloading Pubmed

Hi!

I have been playing around with this module, and I am a bit confused about the `scientific_papers` dataset. I thought that it would download two separate datasets, arxiv and pubmed. But when I run the following:

```
dataset = nlp.load_dataset('scientific_papers', data_dir='.', cache_dir='.')
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.05k/5.05k [00:00<00:00, 2.66MB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.90k/4.90k [00:00<00:00, 2.42MB/s]
Downloading and preparing dataset scientific_papers/pubmed (download: 4.20 GiB, generated: 2.33 GiB, total: 6.53 GiB) to ./scientific_papers/pubmed/1.1.1...
Downloading: 3.62GB [00:40, 90.5MB/s]
Downloading: 880MB [00:08, 101MB/s]
Dataset scientific_papers downloaded and prepared to ./scientific_papers/pubmed/1.1.1. Subsequent calls will reuse this data.
```

only a pubmed folder is created. There doesn't seem to be something for arxiv. Are these two datasets merged? Or have I misunderstood something?

Thanks!

Comments:
Hi so there are indeed two configurations in the datasets as you can see [here](https://github.com/huggingface/nlp/blob/master/datasets/scientific_papers/scientific_papers.py#L81-L82).

You can load either one with:
```python
dataset = nlp.load_dataset('scientific_papers', 'pubmed')
dataset = nlp.load_dataset('scientific_papers', 'arxiv')
```

This issues is actually related to a similar user-experience issue with GLUE. When several configurations are available and the first configuration is loaded by default (see issue #152 and #130), it seems to be unexpected for users.

I think we should maybe raise a (very explicit) error when there are several configurations available and the user doesn't specify one.

What do you think @lhoestq @patrickvonplaten @mariamabarham ?

Yes, it looks like the right thing to do 

Now if you don't specify which part you want, it raises an error:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['pubmed', 'arxiv']
Example of usage:
	`load_dataset('scientific_papers', 'pubmed')`
```
Add Dataset: Qanta

Fixes dummy data for #169 @EntilZha

Comments:
@lhoestq - the config name is rather special here: *E.g.* `mode=first,char_skip=25`. It includes `=` and `,` - will that be a problem for windows folders, you think? 

Apart from that good to merge for me.

It's ok to have `=` and `,`.
Windows doesn't like things like `?`, `:`, `/` etc.

I'll add some lines to raise an error if the config name is invalid.

Thanks for fixing things up! I'm curious to take a look at the zip files now to know the format for future reference.
[Question] Create Apache Arrow dataset from raw text file

Hi guys, I have gathered and preprocessed about 2GB of COVID papers from CORD dataset @ Kggle. I have seen you have a text dataset as "Crime and punishment" in Apache arrow format. Do you have any script to do it from a raw txt file (preprocessed as for BERT like) or any guide?
Is the worth of send it to you and add it to the NLP library?
Thanks, Manu


Comments:
We store every dataset in the Arrow format. This is convenient as it supports nested types and memory mapping. If you are curious feel free to check the [pyarrow documentation](https://arrow.apache.org/docs/python/)

You can use this library to load your covid papers by creating a dataset script. You can find inspiration from the ones we've already written in `/datasets`. Here is a link to the steps to [add a dataset](https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset)

Hello @mrm8488 and @lhoestq 

Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Thanks :)

> Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Sure, to get a dataset in arrow format you can either:
- [load from local files (txt, json, csv)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-local-files)
- OR [load from python data (dict, pandas)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-in-memory-data)
- OR [create your own dataset script](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#using-a-custom-dataset-loading-script)


> > Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?
> 
> Sure, to get a dataset in arrow format you can either:
> 
>     * [load from local files (txt, json, csv)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-local-files)
> 
>     * OR [load from python data (dict, pandas)](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#from-in-memory-data)
> 
>     * OR [create your own dataset script](https://huggingface.co/nlp/loading_datasets.html?highlight=csv#using-a-custom-dataset-loading-script)

Links were broken. 

Updated links provided as below
- [load from local files (txt, json, csv)](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-or-remote-files)
- [load from python data (dict, pandas)](https://huggingface.co/docs/datasets/loading_datasets.html#from-in-memory-data)
- [create your own dataset script](https://huggingface.co/docs/datasets/loading_datasets.html#using-a-custom-dataset-loading-script)
add squad Spanish v1 and v2

This PR add the Spanish Squad versions 1 and 2 datasets. 
Fixes #164 

Comments:
Nice ! :) 
Can we group them into one dataset with two versions, instead of having two datasets ?

Yes sure, I can use the version as config name

@lhoestq can you check? I grouped them

Awesome :) feel free to merge after fixing the test in the CI

@mariamabarham - feel free to merge when you're ready. I only checked the dummy files. I did not run the SLOW tests.
[Question] BERT-style multiple choice formatting

Hello, I am wondering what the equivalent formatting of a dataset should be to allow for multiple-choice answering prediction, BERT-style. Previously, this was done by passing a list of `InputFeatures` to the dataloader instead of a list of `InputFeature`, where `InputFeatures` contained lists of length equal to the number of answer choices in the MCQ instead of single items. I'm a bit confused on what the output of my feature conversion function should be when using `dataset.map()` to ensure similar behavior.

Thanks!

Comments:
Hi @sarahwie, can you details this a little more?

I'm not sure I understand what you refer to and what you mean when you say "Previously, this was done by passing a list of InputFeatures to the dataloader instead of a list of InputFeature"

I think I've resolved it. For others' reference: to convert from using the [`MultipleChoiceDataset` class](https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/examples/multiple-choice/utils_multiple_choice.py#L82)/[`run_multiple_choice.py`](https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/examples/multiple-choice/run_multiple_choice.py) script in Huggingface Transformers, I've done the following for hellaswag:

1. converted the `convert_examples_to_features()` function to only take one input and return a dictionary rather than a list:
```
def convert_examples_to_features(example, tokenizer, max_length):

    choices_inputs = defaultdict(list)
    for ending_idx, ending in enumerate(example['endings']['ending']):
        text_a = example['ctx']
        text_b = ending

        inputs = tokenizer.encode_plus(
            text_a,
            text_b,
            add_special_tokens=True,
            max_length=max_length,
            pad_to_max_length=True,
            return_overflowing_tokens=True,
        )
        if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
            logger.info(
                "Attention! you are cropping tokens (swag task is ok). "
                "If you are training ARC and RACE and you are poping question + options,"
                "you need to try to use a bigger max seq length!"
            )

        for key in inputs:
            choices_inputs[key].append(inputs[key])
            
        choices_inputs['label'] = int(example['label'])

    return choices_inputs
```
2. apply this directly (instance-wise) to dataset, convert dataset to torch tensors. Dataset is then ready to be passed to `Trainer` instance.

```
dataset['train'] = dataset['train'].map(lambda x: convert_examples_to_features(x, tokenizer, max_length), batched=False)
columns = ['input_ids', 'token_type_ids', 'attention_mask', 'label']
dataset['train'].set_format(type='torch', columns=columns)
```
When will the remaining math_dataset modules be added as dataset objects

Currently only the algebra_linear_1d is supported. Is there a timeline for making the other modules supported. If no timeline is established, how can I help?

Comments:
On a similar note it would be nice to differentiate between train-easy, train-medium, and train-hard

Hi @tylerroost, we don't have a timeline for this at the moment.
If you want to give it a look we would be happy to review a PR on it.
Also, the library is one week old so everything is quite barebones, in particular the doc.
You should expect some bumps on the road.

To get you started, you can check the datasets scripts in the `./datasets` folder on the repo and find the one on math_datasets that will need to be modified. Then you should check the original repository on the math_dataset to see where the other files to download are located and what is the expected format for the various parts of the dataset.

To get a general overview on how datasets scripts are written and used, you can read the nice tutorial on how to add a new dataset for TensorFlow Dataset [here](https://www.tensorflow.org/datasets/add_dataset), our API is not exactly identical but it can give you a high-level overview.

Thanks I'll give it a look
Weird-ish: Not creating unique caches for different phases

Sample code:

```python
import nlp
dataset = nlp.load_dataset('boolq')

def func1(x):
    return x

def func2(x):
    return None

train_output = dataset["train"].map(func1)
valid_output = dataset["validation"].map(func1)
print()
print(len(train_output), len(valid_output))
# Output: 9427 9427
```

The map method in both cases seem to be pointing to the same cache, so the latter call based on the validation data will return the processed train data cache.

What's weird is that the following doesn't seem to be an issue:

```python
train_output = dataset["train"].map(func2)
valid_output = dataset["validation"].map(func2)
print()
print(len(train_output), len(valid_output))
# 9427 3270
```

Comments:
Looks like a duplicate of #120.
This is already fixed on master. We'll do a new release on pypi soon

Good catch, it looks fixed.
[Commands] In-detail instructions to create dummy data folder

### Dummy data command 

This PR adds a new command `python nlp-cli dummy_data <path_to_dataset_folder>` that gives in-detail instructions on how to add the dummy data files. 

It would be great if you can try it out by moving the current dummy_data folder of any dataset in `./datasets` with `mv datasets/<dataset_script>/dummy_data datasets/<dataset_name>/dummy_data_copy` and running the command `python nlp-cli dummy_data ./datasets/<dataset_name>` to see if you like the instructions. 

### CONTRIBUTING.md
Also the CONTRIBUTING.md is made cleaner including a new section on "How to add a dataset". 

### Current PRs 
It would be nice if we can try out if this command helps current PRs, *e.g.* #169  to add a dataset. I comment on those PRs.

Comments:
awesome !
Use IndexError instead of ValueError when index out of range

**`default __iter__ needs IndexError`**.

When I want to create a wrapper of arrow dataset to adapt to fastai,
I don't know how to initialize it, so I didn't use inheritance but use object composition.
I wrote sth like this.
```
clas HF_dataset():
  def __init__(self, arrow_dataset):
    self.dset = arrow_dataset
  def __getitem__(self, i):
    return self.my_get_item(self.dset)
```
But `for sample in my_dataset:` gave me `ValueError(f"Index ({key}) outside of table length ({self._data.num_rows}).")` . This is because default `__iter__` will stop when it catched `IndexError`.

You can also see my [work](https://github.com/richardyy1188/Pretrain-MLM-and-finetune-on-GLUE-with-fastai/blob/master/GLUE_with_fastai.ipynb) that uses fastai2 to show/load batches from huggingface/nlp GLUE datasets

So I hope we can use `IndexError` instead to let other people who want to wrap it for any purpose won't be caught by this caveat.

BTW, I super appreciate your work, both transformers and nlp save my life. 💖💖💖💖💖💖💖



Comments:
[Bug] labels of glue/ax are all -1 

```
ax = nlp.load_dataset('glue', 'ax')
for i in range(30): print(ax['test'][i]['label'], end=', ')
```
```
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 
```

Comments:
This is the test set given by the Glue benchmark. The labels are not provided, and therefore set to -1.

Ah, yeah. Why it didn’t occur to me. 😂
Thank you for your comment.
[Feature request] separate split name and split instructions

Currently, the name of an nlp.NamedSplit is parsed in arrow_reader.py and used as the instruction.

This makes it impossible to have several training sets, which can occur when:
- A dataset corresponds to a collection of sub-datasets
- A dataset was built in stages, adding new examples at each stage

Would it be possible to have two separate fields in the Split class, a name /instruction and a unique ID that is used as the key in the builder's split_dict ?

Comments:
If your dataset is a collection of sub-datasets, you should probably consider having one config per sub-dataset. For example for Glue, we have sst2, mnli etc.
If you want to have multiple train sets (for example one per stage). The easiest solution would be to name them `nlp.Split("train_stage1")`, `nlp.Split("train_stage2")`, etc. or something like that.

Thanks for the tip! I ended up setting up three different versions of the dataset with their own configs.

for the named splits, I was trying with `nlp.Split("train-stage1")`, which fails. Changing to `nlp.Split("train_stage1")` works :) I looked for examples of what works in the code comments, it may be worth adding some examples of valid/invalid names in there?
[Manual data] improve error message for manual data in general

`nlp.load("xsum")` now leads to the following error message:

![Screenshot from 2020-05-20 20-05-28](https://user-images.githubusercontent.com/23423619/82481825-3587ea00-9ad6-11ea-9ca2-5794252c6ac7.png)

I guess the manual download instructions for `xsum` can also be improved.

Comments:
[Manual data dir] Error message: nlp.load_dataset('xsum') -> TypeError

v 0.1.0 from pip

```python
import nlp
xsum = nlp.load_dataset('xsum')
```

Issue is `dl_manager.manual_dir`is `None`

```python

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-42-8a32f066f3bd> in <module>
----> 1 xsum = nlp.load_dataset('xsum')

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519 

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    397         split_dict = SplitDict(dataset_name=self.name)
    398         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 399         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    400         # Checksums verification
    401         if verify_infos:

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/datasets/xsum/5c5fca23aaaa469b7a1c6f095cf12f90d7ab99bcc0d86f689a74fd62634a1472/xsum.py in _split_generators(self, dl_manager)
    102         with open(dl_path, "r") as json_file:
    103             split_ids = json.load(json_file)
--> 104         downloaded_path = os.path.join(dl_manager.manual_dir, "xsum-extracts-from-downloads")
    105         return [
    106             nlp.SplitGenerator(

~/miniconda3/envs/nb/lib/python3.7/posixpath.py in join(a, *p)
     78     will be discarded.  An empty last part will result in a path that
     79     ends with a separator."""
---> 80     a = os.fspath(a)
     81     sep = _get_sep(a)
     82     path = a

TypeError: expected str, bytes or os.PathLike object, not NoneType


```


Comments:
Rm extracted test dirs

All the dummy data used for tests were duplicated. For each dataset, we had one zip file but also its extracted directory. I removed all these directories

Furthermore instead of extracting next to the dummy_data.zip file, we extract in the temp `cached_dir` used for tests, so that all the extracted directories get removed after testing.

Finally there was a bug in the `mock_download_manager` that would let it create directories with invalid names, as in #172. I fixed that by encoding url arguments. I had to rename the dummy data for `scientific_papers` and `cnn_dailymail` (the aws tests don't pass for those 2 in this PR, but they will once aws will be synced, as the local ones do)

Let me know if it sounds good to you @patrickvonplaten . I'm still not entirely familiar with the mock downloader

Comments:
Thanks for cleaning up the extracted dummy data folders! Instead of changing the file_utils we could also just put these folders under `.gitignore` (or maybe already done?).

Awesome! I guess you might have to add the changes for the MockDLManager now in a different file though because of my last PR - sorry!
Clone not working on Windows environment

Cloning in a windows environment is not working because of use of special character '?' in folder name ..
Please consider changing the folder name ....
Reference to folder -
nlp/datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs/dailymail/stories/

error log:
fatal: cannot create directory at 'datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs': Invalid argument



Comments:
Should be fixed on master now :)

Thanks @lhoestq 👍  Now I can uninstall WSL and get back to work with windows.🙂
fix squad metric format

The format of the squad metric was wrong.
This should fix #143 

I tested with
```python3
predictions = [
    {'id': '56be4db0acb8001400a502ec', 'prediction_text': 'Denver Broncos'}
]
references = [
    {'answers': [{'text': 'Denver Broncos'}], 'id': '56be4db0acb8001400a502ec'}
]
```

Comments:
One thing for SQuAD is that I wanted to be able to use the SQuAD dataset directly in the metrics and I'm not sure it will be possible with this format.

(maybe it's not really possible in general though)

This is kinda related to one thing I had in mind which is that we may want to be able to dump our model predictions in a `Dataset` as well so that we don't keep them in memory (and we can export them in a nice format later as well when we will have a serialization formats).

Maybe this is overkill though, I haven't fully wraped my head around this.

I'm also perfectly fine with merging this PR in the current state and working on a larger scope later.

This is the format needed to run the official script directly. The format of the squad dataset is different from the input of the metric. 

> One thing for SQuAD is that I wanted to be able to use the SQuAD dataset directly in the metrics and I'm not sure it will be possible with this format.
> 
> (maybe it's not really possible in general though)

Ok I see. I'll try to use the same format

Ok with this update I changed the format to fit the squad dataset format.
Now you can do:
```python
squad_dset = nlp.load_dataset("squad")
squad_metric = nlp.load_metric("/Users/quentinlhoest/Desktop/hf/nlp-bis/metrics/squad")
predictions = [
    {"id": v["id"], "prediction_text": v["answers"]["text"][0]}  # take first possible answer
    for v in squad_dset["validation"]
]
squad_metric.compute(predictions, squad_dset["validation"])
```
Rename anli dataset

What we have now as the `anli` dataset is actually the αNLI dataset from the ART challenge dataset. This name is confusing because `anli` is also the name of adversarial NLI (see [https://github.com/facebookresearch/anli](https://github.com/facebookresearch/anli)).

I renamed the current `anli` dataset by `art`.

Comments:
Adding Qanta (Quizbowl) Dataset

This PR adds the qanta question answering datasets from [Quizbowl: The Case for Incremental Question Answering](https://arxiv.org/abs/1904.04792) and [Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples](https://www.aclweb.org/anthology/Q19-1029/) (adversarial fold)

This partially continues a discussion around fixing dummy data from https://github.com/huggingface/nlp/issues/161

I ran the following code to double check that it works and did some sanity checks on the output. The majority of the code itself is from our `allennlp` version of the dataset reader.

```python
import nlp
# Default is full question
data = nlp.load_dataset('./datasets/qanta') 
# Four configs
# Primarily useful for training
data = nlp.load_dataset('./datasets/qanta', 'mode=sentences,char_skip=25')  
# Primarily used in evaluation
data = nlp.load_dataset('./datasets/qanta', 'mode=first,char_skip=25')  
data = nlp.load_dataset('./datasets/qanta', 'mode=full,char_skip=25')  
# Primarily useful in evaluation and "live" play
data = nlp.load_dataset('./datasets/qanta', 'mode=runs,char_skip=25')  
```

Comments:
Hi @EntilZha - sorry for waiting so long until taking action here. We created a new command and a new recipe of how to add dummy_data. Can you maybe rebase to `master` as explained in 7. of https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-contribute-to-nlp and check that your dummy data is correct following the instructions here: https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset ? 

If the tests described in 5. of https://github.com/huggingface/nlp/blob/master/CONTRIBUTING.md#how-to-add-a-dataset pass we can merge the PR :-) 

I updated to the most recent master and followed the steps, but still having the similar error where it can't find the correct file since the path to the directory is given, rather than the individual files within them. This still something wrong about how I'm inputting the data or how the tests are reading it?

It's the dummy_data structure. You actually have to call the dummy data file name `dummy_data` (not .json anything). So there should not be a `dummy_data` folder but for each config only a `dummy_data` which contains your json dummy data. Can you maybe try once more - if it doesn't work I do it for you :-). 

Would that work if there are multiple files? In my case, I'm including something similar to squad 1.0/2.0 where we have the main dataset + an additional challenge set in different files. Would I have the zip decompress to two files in that case?

This dataset was actually a special case. It helped us improve the  dummy data instructions :-), see #195 .Close this PR and merge #194.
[Tests] refactor tests

This PR separates AWS and Local tests to remove these ugly statements in the script:
```python
        if "/" not in dataset_name:
            logging.info("Skip {} because it is a canonical dataset")
            return
```

To run a `aws` test, one should now run the following command: 

```python 
pytest -s tests/test_dataset_common.py::AWSDatasetTest::test_builder_class_wmt14
```

The same `local` test, can be run with:
```python 
pytest -s tests/test_dataset_common.py::LocalDatasetTest::test_builder_class_wmt14
```

Comments:
Nice !
Add a method to shuffle a dataset

Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think?

Comments:
+1 for the naming convention

About the `shuffle` method, from my understanding it should be done in `Dataloader` (better separation between dataset processing - usage)

+1 for shuffle in `Dataloader`. 
Some `Dataloader` just store idxs of dataset and just shuffle those idxs, which might(?) be faster than do shuffle in dataset, especially when doing shuffle every epoch.

Also +1 for the naming convention.

As you might already know the issue of dataset shuffling came up in the nlp code [walkthrough](https://youtu.be/G3pOvrKkFuk?t=3204) by Yannic Kilcher


We added the `.shuffle` method :)

Closing this one.
ANLI

Can I recommend the following:

For ANLI, use https://github.com/facebookresearch/anli. As that paper says, "Our dataset is not
to be confused with abductive NLI (Bhagavatula et al., 2019), which calls itself αNLI, or ART.". 

Indeed, the paper cited under what is currently called anli says in the abstract "We introduce a challenge dataset, ART".

The current naming will confuse people :)

Comments:
Add Spanish POR and NER Datasets

Hi guys,
In order to cover multilingual support a little step could be adding standard Datasets used for Spanish NER and POS tasks.
I can provide it in raw and preprocessed formats.

Comments:
Hello @mrm8488, are these datasets official datasets published in an NLP/CL/ML venue?

What about this one: https://github.com/ccasimiro88/TranslateAlignRetrieve?
[Feature request] Add cos-e v1.0

I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).

Comments:
Sounds good, @mariamabarham do you want to give a look?
I think we should have two configurations so we can allow either version of the dataset to be loaded with the `1.0` version being the default maybe.

Cc some authors of the great cos-e: @nazneenrajani @bmccann

cos_e v1.0 is related to CQA v1.0 but only CQA v1.11 dataset is available on their website. Indeed their is lots of ids in cos_e v1, which are  not in CQA v1.11  or the other way around.
@sarahwie, @thomwolf, @nazneenrajani,  @bmccann  do you know where I can  find CQA v1.0


@mariamabarham I'm also not sure where to find CQA 1.0. Perhaps it's not possible to include this version of the dataset. I'll close the issue if that's the case.

I do have a copy of the dataset. I can upload it to our repo.

Great @nazneenrajani. let me know once done.
Thanks

@mariamabarham @sarahwie I added them to the cos-e repo https://github.com/salesforce/cos-e/tree/master/data/v1.0

You can now do
```python
from nlp import load_dataset
cos_e = load_dataset("cos_e", "v1.0")
```
Thanks @mariamabarham !

Thanks!

@mariamabarham Just wanted to note that default behavior `cos_e = load_dataset("cos_e")` now loads `v1.0`. Not sure if this is intentional (but the flag specification does work as intended). 

> @mariamabarham Just wanted to note that default behavior `cos_e = load_dataset("cos_e")` now loads `v1.0`. Not sure if this is intentional (but the flag specification does work as intended).

In the new version of `nlp`, if you try `cos_e = load_dataset("cos_e")` it throws this error:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['v1.0', 'v1.11']
Example of usage:
	`load_dataset('cos_e', 'v1.0')`
```
For datasets with at least two configurations, we now force the user to pick one (no default)
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a

Comments:
You can just run: 
`val = nlp.load_dataset('squad')` 

if you want to have just the validation script you can also do:

`val = nlp.load_dataset('squad', split="validation")`

If you want to load a local dataset, make sure you include a `./` before the folder name. 

This happens by just doing run all cells on colab ... I assumed the colab example is broken. 

Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
```
!pip uninstall -y -qq pyarrow
!pip uninstall -y -qq nlp
!pip install -qq git+https://github.com/huggingface/nlp.git
```

> Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
> 
> ```
> !pip uninstall -y -qq pyarrow
> !pip uninstall -y -qq nlp
> !pip install -qq git+https://github.com/huggingface/nlp.git
> ```

Tried, having the same error.

Can you post a link here of your colab? I'll make a copy of it and see what's wrong

This should be fixed in the current version of the notebook. You can try it again

Also see: https://github.com/huggingface/nlp/issues/222

I am getting this error when running this command
```
val = nlp.load_dataset('squad', split="validation")
```

FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/dataset_info.json'

Can anybody help?

It seems like your download was corrupted :-/ Can you run the following command: 

```
rm -r /root/.cache/huggingface/datasets
```

to delete the cache completely and rerun the download? 

I tried the notebook again today and it worked without barfing. 👌
Meta-datasets (GLUE/XTREME/...) – Special care to attributions and citations

Meta-datasets are interesting in terms of standardized benchmarks but they also have specific behaviors, in particular in terms of attribution and authorship. It's very important that each specific dataset inside a meta dataset is properly referenced and the citation/specific homepage/etc are very visible and accessible and not only the generic citation of the meta-dataset itself.

Let's take GLUE as an example:

The configuration has the citation for each dataset included (e.g. [here](https://github.com/huggingface/nlp/blob/master/datasets/glue/glue.py#L154-L161)) but it should be copied inside the dataset info so that, when people access `dataset.info.citation` they get both the citation for GLUE and the citation for the specific datasets inside GLUE that they have loaded.

Comments:
As @yoavgo suggested, there should be the possibility to call a function like  nlp.bib that outputs all bibtex ref from the datasets and models actually used and eventually nlp.bib.forreadme that would output the same info + versions numbers so they can be included in a readme.md file.

Actually, double checking with @mariamabarham, we already have this feature I think.

It's like this currently:
```python
>>> from nlp import load_dataset
>>> 
>>> dataset = load_dataset('glue', 'cola', split='train')
>>> print(dataset.info.citation)
@article{warstadt2018neural,
  title={Neural Network Acceptability Judgments},
  author={Warstadt, Alex and Singh, Amanpreet and Bowman, Samuel R},
  journal={arXiv preprint arXiv:1805.12471},
  year={2018}
}
@inproceedings{wang2019glue,
  title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding},
  author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},
  note={In the Proceedings of ICLR.},
  year={2019}
}

Note that each GLUE dataset has its own citation. Please see the source to see
the correct citation for each contained dataset.
```

What do you think @dseddah?

Looks good but why would there be a difference between the ref in the source and the one to be printed? 

Yes, I think we should remove this warning @mariamabarham.

It's probably a relic of tfds which didn't have the same way to access citations.
[Feature request] Add Ubuntu Dialogue Corpus dataset

https://github.com/rkadlec/ubuntu-ranking-dataset-creator or http://dataset.cs.mcgill.ca/ubuntu-corpus-1.0/

Comments:
@AlphaMycelium the Ubuntu Dialogue Corpus [version 2]( https://github.com/rkadlec/ubuntu-ranking-dataset-creator) is added. Note that it requires a manual download by following the download instructions in the [repos]( https://github.com/rkadlec/ubuntu-ranking-dataset-creator).
Maybe we can close this issue for now?
_download_and_prepare() got an unexpected keyword argument 'verify_infos'

# Reproduce
In Colab,
```
%pip install -q  nlp
%pip install -q apache_beam mwparserfromhell

dataset = nlp.load_dataset('wikipedia')
```
get
```
Downloading and preparing dataset wikipedia/20200501.aa (download: Unknown size, generated: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/wikipedia/20200501.aa/1.0.0...

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-6-52471d2a0088> in <module>()
----> 1 dataset = nlp.load_dataset('wikipedia')

1 frames

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519 

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

TypeError: _download_and_prepare() got an unexpected keyword argument 'verify_infos'
```

Comments:
Same error for dataset 'wiki40b'

Should be fixed on master :)
Error with sklearn train_test_split

It would be nice if we could use sklearn `train_test_split` to quickly generate subsets from the dataset objects returned by `nlp.load_dataset`. At the moment the code:

```python
data = nlp.load_dataset('imdb', cache_dir=data_cache)
f_half, s_half = train_test_split(data['train'], test_size=0.5, random_state=seed)
```
throws:
```
ValueError: Can only get row(s) (int or slice) or columns (string).
```
It's not a big deal, since there are other ways to split the data, but it would be a cool thing to have.

Comments:
Indeed. Probably we will want to have a similar method directly in the library

Related: #166
Add BERTScore to metrics

This PR adds [BERTScore](https://arxiv.org/abs/1904.09675) to metrics.
Here is an example of how to use it.

```sh
import nlp
bertscore = nlp.load_metric('metrics/bertscore') # or simply nlp.load_metric('bertscore') after this is added to huggingface's s3 bucket
predictions = ['example', 'fruit']
references = [['this is an example.', 'this is one example.'], ['apple']]
results = bertscore.compute(predictions, references, lang='en')
print(results)
```

Comments:
[AWS tests] AWS test should not run for canonical datasets

AWS tests should in general not run for canonical datasets. Only local tests will run in this case. This way a PR is able to pass when adding a new dataset.

This PR changes to logic to the following: 

1) All datasets that are present in `nlp/datasets` are tested only locally. This way when one adds a canonical dataset, the PR includes his dataset in the tests.

2) All datasets that are only present on AWS, such as `webis/tl_dr` atm are tested only on AWS. 

I think the testing structure might need a bigger refactoring and better documentation very soon. 

Merging for now to unblock new PRs @thomwolf @mariamabarham .

Comments:
ArrowTypeError in squad metrics

`squad_metric.compute` is giving following error
```
ArrowTypeError: Could not convert [{'text': 'Denver Broncos'}, {'text': 'Denver Broncos'}, {'text': 'Denver Broncos'}] with type list: was not a dict, tuple, or recognized null value for conversion to struct type
```

This is how my predictions and references look like
```
predictions[0]
# {'id': '56be4db0acb8001400a502ec', 'prediction_text': 'Denver Broncos'}
```

```
references[0]
# {'answers': [{'text': 'Denver Broncos'},
  {'text': 'Denver Broncos'},
  {'text': 'Denver Broncos'}],
 'id': '56be4db0acb8001400a502ec'}
```

These are structured as per the `squad_metric.compute` help string.

Comments:
There was an issue in the format, thanks.
Now you can do
```python3
squad_dset = nlp.load_dataset("squad")
squad_metric = nlp.load_metric("/Users/quentinlhoest/Desktop/hf/nlp-bis/metrics/squad")
predictions = [
    {"id": v["id"], "prediction_text": v["answers"]["text"][0]}  # take first possible answer
    for v in squad_dset["validation"]
]
squad_metric.compute(predictions, squad_dset["validation"])
```

and the expected format is 
```
Args:
    predictions: List of question-answers dictionaries with the following key-values:
        - 'id': id of the question-answer pair as given in the references (see below)
        - 'prediction_text': the text of the answer
    references: List of question-answers dictionaries with the following key-values:
        - 'id': id of the question-answer pair (see above),
        - 'answers': a Dict {'text': list of possible texts for the answer, as a list of strings}
```
[WMT] Add all wmt

This PR adds all wmt datasets scripts. At the moment the script is **not** functional for the language pairs "cs-en", "ru-en", "hi-en" because apparently it takes up to a week to get the manual data for these datasets: see http://ufal.mff.cuni.cz/czeng. 

The datasets are fully functional though for the "big" language pairs "de-en" and "fr-en". 

Overall I think the scripts are very messy and might need a big refactoring at some point.

For now I think there are good to merge (most dataset configs can be used). I will add "cs", "ru" and "hi" when the manual data is available. 

Comments:
[Tests] run local tests as default

This PR also enables local tests by default

I think it's safer for now to enable both local and aws tests for every commit. The problem currently is that when we do a PR to add a dataset, the dataset is not yet on AWS on therefore not tested on the PR itself. Thus the PR will always be green even if the datasets are not correct. This PR aims at fixing this.

## Suggestion on how to commit to the repo from now on:
Now since the repo is "online", I think we should adopt a couple of best practices:
1) - No direct committing to the repo anymore. Every change should be opened in a PR and be well documented so that we can find it later
2) - Every PR has to be reviewed by at least x people (I guess @thomwolf you should decide here) because we now have to be much more careful when doing changes to the API for backward compatibility, etc...


Comments:
You are right and I think those are usual best practice :) I'm 100% fine with this^^

Merging this for now to unblock other PRs.
[Question] Using/adding a local dataset

Users may want to either create/modify a local copy of a dataset, or use a custom-built dataset with the same `Dataset` API as externally downloaded datasets.

It appears to be possible to point to a local dataset path rather than downloading the external ones, but I'm not exactly sure how to go about doing this.

A notebook/example script demonstrating this would be very helpful.

Comments:
Hi @zphang,

So you can just give the local path to a dataset script file and it should work.

Here is an example:
- you can download one of the scripts in the `datasets` folder of the present repo (or clone the repo)
- then you can load it with `load_dataset('PATH/TO/YOUR/LOCAL/SCRIPT.py')`

Does it make sense?

Could you give a more concrete example, please? 

I looked up wikitext dataset script from the repo. Should I just overwrite the `data_file` on line 98 to point to the local dataset directory? Would it work for different configurations of wikitext (wikitext2, wikitext103 etc.)?

Or maybe we can use DownloadManager to specify local dataset location? In that case, where do we use DownloadManager instance?

Thanks

Hi @MaveriQ , although what I am doing is to commit a new dataset, but I think looking at imdb script might help.
You may want to use `dl_manager.download_custom`, give it a url(arbitrary string), a custom_download(arbitrary function) and return a path, and finally use _get sample to fetch a sample.

The download manager supports local directories. You can specify a local directory instead of a url and it should work.

Closing this one.
Feel free to re-open if you have other questions :)
[Feature Request] Add the OpenWebText dataset

The OpenWebText dataset is an open clone of OpenAI's WebText dataset. It can be used to train ELECTRA as is specified in the [README](https://www.github.com/google-research/electra).

More information and the download link are available [here](https://skylion007.github.io/OpenWebTextCorpus/).

Comments:
We're experimenting with hosting the OpenWebText corpus on Zenodo for easier downloading. https://zenodo.org/record/3834942#.Xs1w8i-z2J8

Closing since it's been added in #660
[Feature request] Add Toronto BookCorpus dataset

I know the copyright/distribution of this one is complex, but it would be great to have! That, combined with the existing `wikitext`, would provide a complete dataset for pretraining models like BERT.

Comments:
As far as I understand, `wikitext` is refer to `WikiText-103` and `WikiText-2` that created by researchers in Salesforce, and mostly used in traditional language modeling.

You might want to say `wikipedia`, a dump from wikimedia foundation.

Also I would like to have Toronto BookCorpus too ! Though it involves copyright problem...

Hi, @lhoestq, just a reminder that this is solved by #248 .😉
Loading GLUE dataset loads CoLA by default

If I run:

```python
dataset = nlp.load_dataset('glue')
```
The resultant dataset seems to be CoLA be default, without throwing any error. This is in contrast to calling:

```python
metric = nlp.load_metric("glue")
```
which throws an error telling the user that they need to specify a task in GLUE. Should the same apply for loading datasets?

Comments:
As a follow-up to this: It looks like the actual GLUE task name is supplied as the `name` argument. Is there a way to check what `name`s/sub-datasets are available under a grouping like GLUE? That information doesn't seem to be readily available in info from `nlp.list_datasets()`.

Edit: I found the info under `Glue.BUILDER_CONFIGS`

Yes so the first config is loaded by default when no `name` is supplied but for GLUE this should probably throw an error indeed.

We can probably just add an `__init__` at the top of the `class Glue(nlp.GeneratorBasedBuilder)` in the `glue.py` script which does this check:
```
class Glue(nlp.GeneratorBasedBuilder):
    def __init__(self, *args, **kwargs):
        assert 'name' in kwargs and kwargs[name] is not None, "Glue has to be called with a configuration name"
        super(Glue, self).__init__(*args, **kwargs)
```

An error is raised if the sub-dataset is not specified :)
```
ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
	`load_dataset('glue', 'cola')`
```
[Feature request] Add Google Natural Question dataset

Would be great to have https://github.com/google-research-datasets/natural-questions as an alternative to SQuAD.

Comments:
Indeed, I think this one is almost ready cc @lhoestq 

I'm doing the latest adjustments to make the processing of the dataset run on Dataflow

Is there an update to this? It will be very beneficial for the QA community!

Still work in progress :)
The idea is to have the dataset already processed somewhere so that the user only have to download the processed files. I'm also doing it for wikipedia.

Super appreciate your hard work !!
I'll cross my fingers and hope easily loadable wikipedia dataset will come soon. 

Quick update on NQ: due to some limitations I met using apache beam + parquet I was not able to use the dataset in a nested parquet structure in python to convert it to our Apache Arrow format yet.
However we had planned to change this conversion step anyways so we'll make just sure that it enables to process and convert the NQ dataset to arrow.

NQ was added in #427 🎉
🐛 `map` not working

I'm trying to run a basic example (mapping function to add a prefix).  
[Here is the colab notebook I'm using.](https://colab.research.google.com/drive/1YH4JCAy0R1MMSc-k_Vlik_s1LEzP_t1h?usp=sharing)

```python
import nlp

dataset = nlp.load_dataset('squad', split='validation[:10%]')

def test(sample):
    sample['title'] = "test prefix @@@ " + sample["title"]
    return sample

print(dataset[0]['title'])
dataset.map(test)
print(dataset[0]['title'])
```
Output :
> Super_Bowl_50
Super_Bowl_50

Expected output :
> Super_Bowl_50
test prefix @@@ Super_Bowl_50

Comments:
I didn't assign the output 🤦‍♂️

```python
dataset.map(test)
```

should be :

```python
dataset = dataset.map(test)
```