konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
mlsum

Hello, 

The tests for the load_real_data fail, as there is no default language subset to download it looks for a file that does not exist. This bug does not happen when using the load_dataset function, as it asks you to specify a language if you do not, so I submit this PR anyway. The dataset is avalaible on : https://gitlab.lip6.fr/scialom/mlsum_data

Thanks @RachelKer for this PR.

I think the dummy_data structure does not also match. In the `_split_generator` you have something like `os.path.join(downloaded_files["validation"], lang+'_val.jsonl')` but in you dummy_data you have `os.path.join(downloaded_files["validation"], lang+"_val.zip",  lang+'_val.jsonl')`. I think ` jsonl` files should be directly  in the `dummy_data` folder without the sub-folder 

@lhoestq 

Hi @RachelKer :)
Thanks for adding MLSUM !

To fix the CI I think you just have to rebase from master

Great, I think it is working now. Thanks :)

It looks like your PR does tons of changes in other datasets. 
Maybe this is because of the merge from master ?

Hmm, I see, sorry I messed up somewhere. Maybe it's easier if we close the pull request and I do another one ?

Yea if it's easier for you feel free to re-open a PR
[FEATURE REQUEST] Multiprocessing with for dataset.map, dataset.filter

It would be nice to be able to speed up `dataset.map` or `dataset.filter`. Perhaps this is as easy as sharding the dataset sending each shard to a process/thread/dask pool and using the new `nlp.concatenate_dataset()` function to join them all together?

Yes that's definitely something we plan to add ^^

Yes, that would be nice. We could take a look at what tensorflow `tf.data` does under the hood for instance.

So `tf.data.Dataset.map()` returns a `ParallelMapDataset` if `num_parallel_calls is not None` [link](https://github.com/tensorflow/tensorflow/blob/2b96f3662bd776e277f86997659e61046b56c315/tensorflow/python/data/ops/dataset_ops.py#L1623).

There, `num_parallel_calls` is turned into a tensor and and fed to `gen_dataset_ops.parallel_map_dataset` where it looks like tensorflow takes over.

We could start with something simple like a thread or process pool that `imap`s over some shards.
 

Multiprocessing was added in #552 . You can set the number of processes with `.map(..., num_proc=...)`. It also works for `filter`

Closing this one, but feel free to reo-open if you have other questions

@lhoestq Great feature implemented! Do you have plans to add it to official tutorials [Processing data in a Dataset](https://huggingface.co/docs/datasets/processing.html?highlight=save#augmenting-the-dataset)? It took me sometime to find this parallel processing api.

Thanks for the heads up !

I just added a paragraph about multiprocessing:
https://huggingface.co/docs/datasets/master/processing.html#multiprocessing
Change features vs schema logic

## New logic for `nlp.Features` in datasets

Previously, it was confusing to have `features` and pyarrow's `schema` in `nlp.Dataset`.
However `features` is supposed to be the front-facing object to define the different fields of a dataset, while `schema` is only used to write arrow files.

Changes:
- Remove `schema` field in `nlp.Dataset`
- Make `features` the source of truth to read/write examples
- `features` can no longer be `None` in `nlp.Dataset`
- Update `features` after each dataset transform such as `nlp.Dataset.map`

Todo: change the tests to take these changes into account

I had to make `SplitDict` serializable to be able to copy `DatasetInfo` objects properly.
Serialization was also asked in #389 

One thing I forgot to say here, is that we also want to use the features arguments of `load_dataset` (which goes in the builder’s config) to override the default features of a dataset script.
Something is wrong with WMT 19 kk-en dataset

The translation in the `train` set does not look right:

```
>>>import nlp
>>>from nlp import load_dataset
>>>dataset = load_dataset('wmt19', 'kk-en')
>>>dataset["train"]["translation"][0]
{'kk': 'Trumpian Uncertainty', 'en': 'Трамптық белгісіздік'}
>>>dataset["validation"]["translation"][0]
{'kk': 'Ақша-несие саясатының сценарийін қайта жазсақ', 'en': 'Rewriting the Monetary-Policy Script'}
```
from_dict delete?

AttributeError: type object 'Dataset' has no attribute 'from_dict'

`from_dict` was added in #350 that was unfortunately not included in the 0.3.0 release. It's going to be included in the next release that will be out pretty soon though.
Right now if you want to use `from_dict` you have to install the package from the master branch
```
pip install git+https://github.com/huggingface/nlp.git
```

> `from_dict` was added in #350 that was unfortunately not included in the 0.3.0 release. It's going to be included in the next release that will be out pretty soon though.
> Right now if you want to use `from_dict` you have to install the package from the master branch
> 
> ```
> pip install git+https://github.com/huggingface/nlp.git
> ```
OK, thank you.
Is there a way to download only NQ dev?

Maybe I missed that in the docs, but is there a way to only download the dev set of natural questions (~1 GB)? 
As we want to benchmark QA models on different datasets, I would like to avoid downloading the 41GB of training data. 

I tried
```
dataset = nlp.load_dataset('natural_questions', split="validation", beam_runner="DirectRunner")
```
But this still triggered a big download of presumably the whole dataset. Is there any way of doing this or are splits / slicing options only available after downloading?

Thanks!

Unfortunately it's not possible to download only the dev set of NQ.

I think we could add a way to download only the test set by adding a custom configuration to the processing script though.

Ok, got it. I think this could be a valuable feature - especially for large datasets like NQ, but potentially also others. 
For us, it will in this case make the difference of using the library or keeping the old downloads of the raw dev datasets.  
However, I don't know if that fits into your plans with the library and can also understand if you don't want to support this.

I don't think we could force this behavior generally since the dataset script authors are free to organize the file download as they want (sometimes the mapping between split and files can be very much nontrivial) but we can add an additional configuration for Natural Question indeed as @lhoestq indicate.
train_test_split error: 'dict' object has no attribute 'deepcopy'

`train_test_split` is giving me an error when I try and call it:

`'dict' object has no attribute 'deepcopy'`

## To reproduce

```
dataset = load_dataset('glue', 'mrpc', split='train')
dataset = dataset.train_test_split(test_size=0.2)
```

## Full Stacktrace
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-feb740dbec9a> in <module>
      1 dataset = load_dataset('glue', 'mrpc', split='train')
----> 2 dataset = dataset.train_test_split(test_size=0.2)

~/anaconda3/envs/fastai2_me/lib/python3.7/site-packages/nlp/arrow_dataset.py in train_test_split(self, test_size, train_size, shuffle, seed, generator, keep_in_memory, load_from_cache_file, train_cache_file_name, test_cache_file_name, writer_batch_size)
   1032                     "writer_batch_size": writer_batch_size,
   1033                 }
-> 1034                 train_kwargs = cache_kwargs.deepcopy()
   1035                 train_kwargs["split"] = "train"
   1036                 test_kwargs = cache_kwargs.deepcopy()

AttributeError: 'dict' object has no attribute 'deepcopy'
```

It was fixed in 2ddd18d139d3047c9c3abe96e1e7d05bb360132c.
Could you pull the latest changes from master @morganmcg1 ?

Thanks @lhoestq, works fine now!
MissingBeamOptions for Wikipedia 20200501.en

There may or may not be a regression for the pre-processed Wikipedia dataset. This was working fine 10 commits ago (without having Apache Beam available):

```
nlp.load_dataset('wikipedia', "20200501.en", split='train')
```

And now, having pulled master, I get:

```
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, total: 34.06 GiB) to /home/hltcoe/mgordon/.cache/huggingface/datasets/wikipedia/20200501.en/1.0.0/76b0b2747b679bb0ee7a1621e50e5a6378477add0c662668a324a5bc07d516dd...
Traceback (most recent call last):
  File "scripts/download.py", line 11, in <module>
    fire.Fire(download_pretrain)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "scripts/download.py", line 6, in download_pretrain
    nlp.load_dataset('wikipedia', "20200501.en", split='train')
  File "/exp/mgordon/nlp/src/nlp/load.py", line 534, in load_dataset
    save_infos=save_infos,
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 460, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 870, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, S
park, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
        `load_dataset('wikipedia', '20200501.en', beam_runner='DirectRunner')`
```

Fixed. Could you try again @mitchellgordon95 ?
It was due a file not being updated on S3.

We need to make sure all the datasets scripts get updated properly @julien-c 

Works for me! Thanks.

I found the same issue with almost any language other than English. (For English, it works). Will someone need to update the file on S3 again?

This is because only some languages are already preprocessed (en, de, fr, it) and stored on our google storage.
We plan to have a systematic way to preprocess more wikipedia languages in the future.

For the other languages you have to process them on your side using apache beam. That's why the lib asks for a Beam runner.
add web_questions

add Web Question dataset
#336 

Maybe @patrickvonplaten you can help with the dummy_data structure? it still broken

What does the `nlp-cli dummy_data` command returns ?

`test.json` -> `test` 
and 
`train.json` -> `train`

as shown by the `nlp-cli dummy_data` command ;-)

LGTM for merge @lhoestq - I let you merge if you want to.
Concatenate datasets

I'm constructing the "WikiBooks" dataset, which is a concatenation of Wikipedia & BookCorpus. So I implemented the `Dataset.from_concat()` method, which concatenates two datasets with the same schema.

This would also be useful if someone wants to pretrain on a large generic dataset + their own custom dataset. Not in love with the method name, so would love to hear suggestions.

Usage:
```python
from nlp import Dataset, load_dataset

data1, data2 = {"id": [0, 1, 2]}, {"id": [3, 4, 5]}
dset1, dset2 = Dataset.from_dict(data1), Dataset.from_dict(data2)
dset_concat = Dataset.from_concat([dset1, dset2])
print(dset_concat)
# Dataset(schema: {'id': 'int64'}, num_rows: 6)
```

Looks cool :)

I feel like 
```python
concatenated_dataset = dataset1.concatenate(dataset2)
```
could be more natural. What do you think ?

Also could you also concatenate the `nlp.Dataset._data_files` ?
```python
return cls(table, info=info, split=split, data_files=self._data_files + other_dataset._data_files)
```

I feel like "WikiBooks" would be a multi task dataset that could fit in the #217 discussion.
Not sure concatenate should be the solution for a multi task dataset.

Thanks for the suggestion! `dset1.concatenate(dset2)` does feel more natural. Although this seems to be a different "class" of transformation function than map() or filter(), acting on two datasets rather than on one. I would prefer the function signature treat both datasets symmetrically.

Python lists have `list1 + list2` or `list1.extend(list2)`.
NumPy has `np.concatenate((arr1, arr2))`.
Pandas has `pd.join((df1, df2))`.
PyTorch has `ConcatDataset((dset1, dset2))`.

Given the symmetrical treatment and clear communication that this creates a new object, rather than a simple chaining on the first, my preference is now for `nlp.concatenate((dset1, dset2))`. This would place the function in the same API class as `nlp.load_dataset`. Does that work?

The multi-task discussion is interesting, thanks for pointing me to that! I'll be focusing on T5 in a few weeks, so I'm sure I'll have many opinions then :). For now, I think a simple concatenate feature is important and orthogonal to that discussion. For example, a user may want to create a custom dataset that joins Wikipedia with their own custom text.

> Given the symmetrical treatment and clear communication that this creates a new object, rather than a simple chaining on the first, my preference is now for `nlp.concatenate((dset1, dset2))`. This would place the function in the same API class as `nlp.load_dataset`. Does that work?

Yep I like this idea. Maybe `nlp.concatenate_datasets()` ?

> For now, I think a simple concatenate feature is important and orthogonal to that discussion. For example, a user may want to create a custom dataset that joins Wikipedia with their own custom text.

I agree :)

Great, just updated!
🐛 [Dataset] Cannot download wmt14, wmt15 and wmt17

1. I try downloading `wmt14`, `wmt15`, `wmt17`, `wmt19` with the following code:
```
nlp.load_dataset('wmt14','de-en')
nlp.load_dataset('wmt15','de-en')
nlp.load_dataset('wmt17','de-en')
nlp.load_dataset('wmt19','de-en')
```
The code runs but the download speed is **extremely slow**, the same behaviour is not observed on `wmt16` and `wmt18`

2. When trying to download `wmt17 zh-en`, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz

similar slow download speed here for nlp.load_dataset('wmt14', 'fr-en')
`
Downloading: 100%|██████████████████████████████████████████████████████████| 658M/658M [1:00:42<00:00, 181kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████| 918M/918M [1:39:38<00:00, 154kB/s]
Downloading:   2%|▉                                                       | 40.9M/2.37G [04:48<5:03:06, 128kB/s]
`
Could we just download a specific subdataset in 'wmt14', such as 'newstest14'? 

> The code runs but the download speed is extremely slow, the same behaviour is not observed on wmt16 and wmt18

The original source for the files may provide slow download speeds.
We can probably host these files ourselves.

> When trying to download wmt17 zh-en, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz

Looks like the file`UNv1.0.en-zh.tar.gz` is missing, or the url changed. We need to fix that

> Could we just download a specific subdataset in 'wmt14', such as 'newstest14'?

Right now I don't think it's possible. Maybe @patrickvonplaten knows more about it


Yeah, the download speed is sadly always extremely slow :-/. 
I will try to check out the `wmt17 zh-en` bug :-) 

Maybe this can be used - https://stuncorpusprod.blob.core.windows.net/corpusfiles/UNv1.0.en-zh.tar.gz.00 

These issues seem to be fixed now.
Conversion through to_pandas output numpy arrays for lists instead of python objects

In a related question, the conversion through to_pandas output numpy arrays for the lists instead of python objects.

Here is an example:
```python
>>> dataset._data.slice(key, 1).to_pandas().to_dict("list")
{'sentence1': ['Amrozi accused his brother , whom he called " the witness " , of deliberately distorting his evidence .'], 'sentence2': ['Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .'], 'label': [1], 'idx': [0], 'input_ids': [array([  101,  7277,  2180,  5303,  4806,  1117,  1711,   117,  2292,
        1119,  1270,   107,  1103,  7737,   107,   117,  1104,  9938,
        4267, 12223, 21811,  1117,  2554,   119,   102])], 'token_type_ids': [array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0])], 'attention_mask': [array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1])]}
>>> type(dataset._data.slice(key, 1).to_pandas().to_dict("list")['input_ids'][0])
<class 'numpy.ndarray'>
>>> dataset._data.slice(key, 1).to_pydict()
{'sentence1': ['Amrozi accused his brother , whom he called " the witness " , of deliberately distorting his evidence .'], 'sentence2': ['Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .'], 'label': [1], 'idx': [0], 'input_ids': [[101, 7277, 2180, 5303, 4806, 1117, 1711, 117, 2292, 1119, 1270, 107, 1103, 7737, 107, 117, 1104, 9938, 4267, 12223, 21811, 1117, 2554, 119, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
```

To convert from arrow type we have three options: to_numpy, to_pandas and to_pydict/to_pylist.

- to_numpy and to_pandas return numpy arrays instead of lists but are very fast.
- to_pydict/to_pylist can be 100x slower and become the bottleneck for reading data, but at least they return lists.

Maybe we can have to_pydict/to_pylist as the default and use to_numpy or to_pandas when the format (set by `set_format`) is 'numpy' or 'pandas'
Update dataset loading and features - Add TREC dataset

This PR:
- add a template for a new dataset script
- update the caching structure so that the path to the cached data files is also a function of the dataset loading script hash. This way when you update a loading script the data will be automatically updated instead of falling back to the previous version (which is usually a outdated). This makes it in particular easier to iterate when writing a new dataset loading script.
- fix a bug in the `ClassLabel` feature and make it more flexible so that its methods `str2int` and `int2str` can also accept list, numpy arrays and PyTorch/TensorFlow tensors.
- add the TREC-6 dataset

I just copied the files that are on google storage to follow the new `_relative_data_dir ` format. It should be good to merge now :)

Well actually it seems there are some merge conflicts to fix first
[dataset] Structure of MLQA seems unecessary nested

The features of the MLQA dataset comprise several nested dictionaries with a single element inside (for `questions` and `ids`): https://github.com/huggingface/nlp/blob/master/datasets/mlqa/mlqa.py#L90-L97

Should we keep this @mariamabarham @patrickvonplaten? Was this added for compatibility with tfds?

```python
            features=nlp.Features(
                {
                    "context": nlp.Value("string"),
                    "questions": nlp.features.Sequence({"question": nlp.Value("string")}),
                    "answers": nlp.features.Sequence(
                        {"text": nlp.Value("string"), "answer_start": nlp.Value("int32"),}
                    ),
                    "ids": nlp.features.Sequence({"idx": nlp.Value("string")})
```

Same for the RACE dataset: https://github.com/huggingface/nlp/blob/master/datasets/race/race.py

Should we scan all the datasets to remove this pattern of un-necessary nesting?

You're right, I think we don't need to use the nested dictionary.
TypeError when computing bertscore

Hi, 

I installed nlp 0.3.0 via pip, and my python version is 3.7.
When I tried to compute bertscore with the code:
```
import nlp 
bertscore = nlp.load_metric('bertscore')  
# load hyps and refs 
...
print (bertscore.compute(hyps, refs, lang='en'))
```

I got the following error.
```
Traceback (most recent call last):
  File "bert_score_evaluate.py", line 16, in <module>
    print (bertscore.compute(hyps, refs, lang='en'))
  File "/home/willywsm/anaconda3/envs/torcher/lib/python3.7/site-packages/nlp/metric.py", line 200, in compute
    output = self._compute(predictions=predictions, references=references, **metrics_kwargs)
  File "/home/willywsm/anaconda3/envs/torcher/lib/python3.7/site-packages/nlp/metrics/bertscore/fb176889831bf0ce995ed197edc94b2e9a83f647a869bb8c9477dbb2d04d0f08/bertscore.py", line 105, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() takes 3 positional arguments but 4 were given
```

It seems like there is something wrong with get_hash() function?

I am not able to reproduce this issue on my side.
Could you give us more details about the inputs you used ?

I do get another error though:
```
~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/bert_score/utils.py in bert_cos_score_idf(model, refs, hyps, tokenizer, idf_dict, verbose, batch_size, device, all_layers)
    371         return sorted(list(set(l)), key=lambda x: len(x.split(" ")))
    372 
--> 373     sentences = dedup_and_sort(refs + hyps)
    374     embs = []
    375     iter_range = range(0, len(sentences), batch_size)

ValueError: operands could not be broadcast together with shapes (0,) (2,)
```
That's because it gets numpy arrays as input and not lists. See #387 

The other issue was fixed by #403 

Do you still get this issue @willywsm1013 ?
Add dataset post processing for faiss indexes

# Post processing of datasets for faiss indexes

Now that we can have datasets with embeddings (see `wiki_pr` for example), we can allow users to load the dataset + get the Faiss index that comes with it to do nearest neighbors queries.

## Implementation proposition

- Faiss indexes have to be added to the `nlp.Dataset` object, and therefore it's in a different scope that what are doing the `_split_generators` and `_generate_examples` methods of `nlp.DatasetBuilder`. Therefore I added a new method for post processing of the `nlp.Dataset` object called `_post_process` (name could change)
- The role of `_post_process` is to apply dataset transforms (filter/map etc.) or indexing functions (add_faiss_index) to modify/enrich the `nlp.Dataset` object. It is not part of the `download_and_prepare` process (that is focused on arrow files creation) so the post processing is run inside the `as_dataset` method.
- `_post_process` can generate new files (cached files from dataset transforms or serialized faiss indexes) and their names are defined by `_post_processing_resources`
- as we know what are the post processing resources, we can download them automatically from google storage instead of computing them if they're available (as we do for arrow files)

I'd happy to discuss these choices !

## The `wiki_dpr` index

It takes 1h20 and ~7GB of memory to compute. The final index is 1.42GB and takes ~1.5GB of memory.
This is pretty cool given that a naive flat index would take 170GB of memory to store the 21M vectors of dim 768.
I couldn't use directly the Faiss `index_factory` as I needed to set the metric to inner product.

## Example of usage

```python
import nlp
dset = nlp.load_dataset(
    "wiki_dpr",
    "psgs_w100_with_nq_embeddings",
    split="train",
    with_index=True
)
print(len(dset), dset.list_indexes())  # (21015300, ['embeddings'])
```

(it also works with the dataset configuration without the embeddings because I added the index file in google storage for this one too)

## Demo

You can also check a demo on google colab that shows how to use it with the DPRQuestionEncoder from transformers:
https://colab.research.google.com/drive/1FakNU8W5EPMcWff7iP1H6REg3XSS0YLp?usp=sharing


I changed the `wiki_dpr` script to ignore the last 24 examples for now. Hopefully we'll have the full version soon.
The datasets_infos.json and the data on GCS are updated.

And I also added a check to make sure we don't have post processing resources in sub-directories.

I added a dummy config that can be loaded with:
```python
wiki = load_dataset("wiki_dpr", "dummy_psgs_w100_no_embeddings", with_index=True, split="train")
```
It's only 6MB of arrow files and 30MB of index
can't load local dataset: pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries

Trying to load a local SQuAD-formatted dataset (from a JSON file, about 60MB):
```
dataset = nlp.load_dataset(path='json', data_files={nlp.Split.TRAIN: ["./path/to/file.json"]})
```
causes
```
Traceback (most recent call last):
  File "dataloader.py", line 9, in <module>
    ["./path/to/file.json"]})
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 483, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 719, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False):
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/datasets/json/88c1bc5c68489f7eda549ed05a5a738527c613b3e7a4ee3524d9d233353a949b/json.py", line 53, in _generate_tables
    file, read_options=self.config.pa_read_options, parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 191, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```

I haven't been able to find any reports of this specific pyarrow error here or elsewhere. 

I am able to reproduce this with the official SQuAD `train-v2.0.json` file downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/

I am facing this issue in transformers library 3.0.2 while reading a csv using datasets.
Is this fixed in latest version? 
I updated the latest version 4.0.1 but still getting this error. What could cause this error?
load_metric can't acquire lock anymore

I can't load metric (glue) anymore after an error in a previous run. I even removed the whole cache folder `/home/XXX/.cache/huggingface/`, and the issue persisted. What are the steps to fix this?

    Traceback (most recent call last):
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/metric.py", line 101, in __init__
        self.filelock.acquire(timeout=1)
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/filelock.py", line 278, in acquire
        raise Timeout(self._lock_file)
    filelock.Timeout: The file lock '/home/XXX/.cache/huggingface/metrics/glue/1.0.0/1-glue-0.arrow.lock' could not be acquired.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "examples_huggingface_nlp.py", line 268, in <module>
        main()
      File "examples_huggingface_nlp.py", line 242, in main
        dataset, metric = get_dataset_metric(glue_task)
      File "examples_huggingface_nlp.py", line 77, in get_dataset_metric
        metric = nlp.load_metric('glue', glue_config, experiment_id=1)
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/load.py", line 440, in load_metric
        **metric_init_kwargs,
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/metric.py", line 104, in __init__
        "Cannot acquire lock, caching file might be used by another process, "
    ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id' for this run.
    I0709 15:54:41.008838 139854118430464 filelock.py:318] Lock 139852058030936 released on /home/XXX/.cache/huggingface/metrics/glue/1.0.0/1-glue-0.arrow.lock


I found that, in the same process (or the same interactive session), if I do

import nlp

m1 = nlp.load_metric('glue', 'mrpc')
m2 = nlp.load_metric('glue', 'sst2')

I will get the same error `ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id'`.
Add quora dataset

Added the [Quora question pairs dataset](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs).

Implementation Notes:
- I used the original version provided on the quora website. There's also a [Kaggle competition](https://www.kaggle.com/c/quora-question-pairs) which has a nice train/test split but I can't find an easy way to download it.
- I've made the questions into a list:
    ```python
    {
        "questions": [
            {"id":0, "text": "Is this an example question?"},
            {"id":1, "text": "Is this a sample question?"},
        ],
    ...
    }
    ```
    rather than:
    ```python
    {
        "question1": "Is this an example question?",
        "question2": "Is this a sample question?"
        "qid0": 0
        "qid1": 1
    ...
    }
    ```
    Not sure if this was the right call.
- Can't find a good citation for this dataset

Tests seem to be failing because of pandas

Kaggle needs authentification to download datasets. We don't have a way to handle that in the lib for now
How to augment data ?

Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
```

Using batched map is probably the easiest way at the moment.
What kind of augmentation would you like to do ?

Some samples in the dataset are too long, I want to divide them in several samples.

Using batched map is the way to go then.
We'll make it clearer in the docs that map could be used for augmentation.

Let me know if you think there should be another way to do it. Or feel free to close the issue otherwise.

It just feels awkward to use map to augment data. Also it means it's not possible to augment data in a non-batched way.

But to be honest I have no idea of a good API...

Or for non-batched samples, how about returning a tuple ?

```python
def aug(sample):
    # Simply copy the existing data to have x2 amount of data
    return sample, sample

dataset = dataset.map(aug)
```

It feels really natural and easy, but :

* it means the behavior with batched data is different
* I don't know how doable it is backend-wise

@lhoestq 

As we're working with arrow's columnar format we prefer to play with batches that are dictionaries instead of tuples.
If we have tuple it implies to re-format the data each time we want to write to arrow, which can lower the speed of map for example.

It's also a matter of coherence, as we don't want users to be confused whether they have to return dictionaries for some functions and tuples for others when they're doing batches.
add MS MARCO dataset

This PR adds the MS MARCO dataset as requested in this issue #336. MS mARCO has multiple task including:

- Passage and Document Retrieval

- Keyphrase Extraction

- QA and NLG

This PR only adds the 2 versions of the  QA and NLG task dataset which was realeased with the original paper here https://arxiv.org/pdf/1611.09268.pdf 

Tests are failing because of the dummy data. I tried to fix it without success. Can you please have a look at it? @patrickvonplaten , @lhoestq 

The dummy data for v2.1 is missing as far as I can see. I think running the  dummy data command should work correctly here. 

Also, it might be that the structure of the dummy data is wrong - looking at `generate_examples` the structure does not look too easy.

The fact that the dummy data for v2.1 is missing shouldn't make the test fails I think. But as you mention the dummy data structure of v1.1 is wrong. I tried to rename files but it does not solve the issue.

Is MS mARCO added to nlp library?I am not able to view it?

> Is MS mARCO added to nlp library?I am not able to view it?

Hi @parthplc ,the PR is  not merged yet. The dummy data structure is still failing. Maybe @patrickvonplaten can help with it.

Dataset is fixed and should be ready for use. @mariamabarham @lhoestq feel free to merge whenever!

> Dataset is fixed and should be ready for use. @mariamabarham @lhoestq feel free to merge whenever!

thanks
🐛 [Metrics] ROUGE is non-deterministic

If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

> ['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ?

Hi, can you give a full self-contained example to reproduce this behavior?

> Hi, can you give a full self-contained example to reproduce this behavior?

There is a notebook in the post ;)

> If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.
> 
> Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.
> 
> Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :
> 
> > ['0.3350', '0.1470', '0.2329']
> > ['0.3358', '0.1451', '0.2332']
> 
> Why ROUGE is not deterministic ?

This is because of rouge's `BootstrapAggregator` that uses sampling to get confidence intervals (low, mid, high).
You can get deterministic scores per sentence pair by using
```python
score = rouge.compute(rouge_types=["rouge1", "rouge2", "rougeL"], use_aggregator=False)
```
Or you can set numpy's random seed if you still want to use the aggregator.

Maybe we can set all the random seeds of numpy/torch etc. while running `metric.compute` ?

We should probably indeed!

Now if you re-run the notebook, the two printed results are the same @colanim
```
['0.3356', '0.1466', '0.2318']
['0.3356', '0.1466', '0.2318']
```
However across sessions, the results may change (as numpy's random seed can be different). You can prevent that by setting your seed:
```python
rouge = nlp.load_metric('rouge', seed=42)
```

> 

Minor nit: Note that "aggregator" is misspelled in this command. Should be `use_aggregator=False`.  

Thanks, I fixed the code snippet
Starting to add some real doc

Adding a lot of documentation for:
- load a dataset
- explore the dataset object
- process data with the dataset
- add a new dataset script
- share a dataset script
- full package reference

This version of the doc can be explored here: https://2219-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html

Also:
- fix a bug in `train_test_split`
- update the `csv` script
- add a verbose argument to the dataset processing methods

Still missing:
- doc for the metrics
- how to directly upload a community provided dataset with the CLI
- clean up more docstrings
- add the `features` argument to `load_dataset` (should be another PR)

Ok this is starting to be really big so it's probably good to merge this first version of the doc and continue in another PR :)

This first version of the doc can be explored here: https://2219-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html
Add hashes to cnn_dailymail

The URL hashes are helpful for comparing results from other sources.

Looks you to me :)

Could you also update the json file that goes with the dataset script by doing 
```
nlp-cli test ./datasets/cnn_dailymail --save_infos --all_configs
```
It will update the features metadata and the size of the dataset with your changes.

@lhoestq I ran that command.

Thanks for the helpful repository!
can't load SNLI dataset

`nlp` seems to load `snli` from some URL based on nlp.stanford.edu. This subdomain is frequently down -- including right now, when I'd like to load `snli` in a Colab notebook, but can't.

Is there a plan to move these datasets to huggingface servers for a more stable solution?

Btw, here's the stack trace:

```
File "/content/nlp/src/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/content/nlp/src/nlp/builder.py", line 466, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/content/nlp/src/nlp/datasets/snli/e417f6f2e16254938d977a17ed32f3998f5b23e4fcab0f6eb1d28784f23ea60d/snli.py", line 76, in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 217, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in download
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/py_utils.py", line 190, in map_nested
    return function(data_struct)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in <lambda>
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 198, in cached_path
    local_files_only=download_config.local_files_only,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://nlp.stanford.edu/projects/snli/snli_1.0.zip
```

I just added the processed files of `snli` on our google storage, so that when you do `load_dataset` it can download the processed files from there :)

We are thinking about having available those processed files for more datasets in the future, because sometimes files aren't available (like for `snli`), or the download speed is too slow, or sometimes the files take time to be processed.

Closing this one. Feel free to re-open if you have other questions :)

Thank you!
More faiss control

Allow users to specify a faiss index they created themselves, as sometimes indexes can be composite for examples

> Ok, so we're getting rid of the `FaissGpuOptions`?

We support `device=...` because it's simple, but faiss GPU options can be used in so many ways (you can set different gpu options for the different parts of your index for example) that it's probably better to let the user create and configure its index and then use `custom_index=...`
Hyperpartisan news detection

Adding the hyperpartisan news detection dataset from PAN. This contains news article text, labelled with whether they're hyper-partisan and why kinds of biases they display.

Implementation notes:
- As with many PAN tasks, the data is hosted on [Zenodo](https://zenodo.org/record/1489920) and must be requested before use. I've used the manual download stuff for this, although the dataset is provided under a Creative Commons Attribution 4.0 International License, so we could host a version if we wanted to?
- The 'bias' attribute doesn't exist for the 'byarticle' configuration. I've added an empty string to the class labels to deal with this. Is there a more standard value for empty data?
- Should we always subclass `nlp.BuilderConfig`?



Thank you so much for working on this! This is awesome!

How much would it help you if we would remove the manual request?

We are naturally interested in getting some broad idea of how many people and who are using our dataset. But if you consider hosting the dataset yourself, I would rather remove this small barrier on our side (so that we then still get the download count from your library).

This is an interesting aspect indeed!
Do you want to send me an email (see my homepage) and I'll invite you on our slack channel to talk about that?
@ghomasHudson wanna reach out to me as well? I tried to find your email to invite you without success.
Search qa

This PR adds the Search QA dataset used in **SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine**. The dataset has the following config name:

- raw_jeopardy: raw data

- train_test_val:  which is the splitted version

#336 

Could you rebase from master just to make sure we won't break anything for `fever` pls @mariamabarham ?
[Dataset requests] New datasets for Open Question Answering

We are still a few datasets missing for Open-Question Answering which is currently a field in strong development.

Namely, it would be really nice to add:
- WebQuestions (Berant et al., 2013) [done]
- CuratedTrec (Baudis et al. 2015) [not open-source]
- MS-MARCO (NGuyen et al. 2016) [done]
- SearchQA (Dunn et al. 2017) [done]
- FEVER (Thorne et al. 2018) - [ done]

 

All these datasets are cited in http://arxiv.org/abs/2005.11401
Add wiki_dpr

Presented in the [Dense Passage Retrieval paper](https://arxiv.org/pdf/2004.04906.pdf), this dataset consists in 21M passages from the english wikipedia along with their 768-dim embeddings computed using DPR's context encoder.

Note on the implementation:
- There are two configs: with and without the embeddings (73GB vs 14GB)
- I used a non-fixed-size sequence of floats to describe the feature format of the embeddings. I wanted to use fixed-size sequences but I had issues with reading the arrow file afterwards (for example `dataset[0]` was crashing)
- I added the case for lists of urls as input of the download_manager

The two configurations don't have the same sizes, I may change that so that they both have 21015300 examples for convenience, even though it's supposed to have 21015324 examples in total.

One configuration only has 21015300 examples because it seems that the embeddings of the last 24 examples are missing.

It's ok to merge now imo. I'll make another PR if we find a way to have the missing embeddings
Add package path to sys when downloading package as github archive

This fixes the `coval.py` metric so that imports within the downloaded module work correctly. We can use a similar trick to add the BLEURT metric (@ankparikh)

@thomwolf not sure how you feel about adding to the `PYTHONPATH` from the script. This is the only way I could make it work with my understanding of `importlib` but there might be a more elegant method.

This PR fixes https://github.com/huggingface/nlp/issues/305

Sorry for the long diff, everything after the imports comes from `black` for code quality :/ 

 I think it's fine and I can't think of another way to make the import work anyways.

Maybe we can have the `sys.path` behavior inside `prepare_module` instead ? Currently it seems to come out of nowhere in the code ^^'
We could check if external imports have a `__init__.py` and if it is the case then we can add to directory to the `PYTHONPATH`
Blog Authorship Corpus, Non Matching Splits Sizes Error, nlp viewer

Selecting `blog_authorship_corpus` in the nlp viewer throws the following error: 

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=610252351, num_examples=532812, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='train', num_bytes=614706451, num_examples=535568, dataset_name='blog_authorship_corpus')}, {'expected': SplitInfo(name='validation', num_bytes=37500394, num_examples=31277, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='validation', num_bytes=32553710, num_examples=28521, dataset_name='blog_authorship_corpus')}]
Traceback:
File "/home/sasha/streamlit/lib/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 172, in <module>
    dts, fail = get(str(option.id), str(conf_option.name) if conf_option else None)
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 591, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/streamlit/lib/streamlit/caching.py", line 575, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 132, in get
    builder_instance.download_and_prepare()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/builder.py", line 488, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/nlp/utils/info_utils.py", line 70, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
```
@srush @lhoestq 

I wonder if this means downloading failed? That corpus has a really slow server.

This dataset seems to have a decoding problem that results in inconsistencies in the number of generated examples.
See #215.
That's why we end up with a `NonMatchingSplitsSizesError `.