NyssaW/github-issues huggingface dataset:
add DatasetDict

## Add DatasetDict

### Overview

When you call `load_dataset` it can return a dictionary of datasets if there are several splits (train/test for example).
If you wanted to apply dataset transforms you had to iterate over each split and apply the transform.

Instead of returning a dict, it now returns a `nlp.DatasetDict` object which inherits from dict and contains the same data as before, except that now users can call dataset transforms directly from the output, and they'll be applied on each split.

Before:
```python
from nlp import load_dataset

squad = load_dataset("squad")
print(squad.keys())
# dict_keys(['train', 'validation'])
squad = {
    split_name: dataset.map(my_func) for split_name, dataset in squad.items()
}
print(squad.keys())
# dict_keys(['train', 'validation'])
```

Now:
```python
from nlp import load_dataset

squad = load_dataset("squad")
print(squad.keys())
# dict_keys(['train', 'validation'])
squad = squad.map(my_func)
print(squad.keys())
# dict_keys(['train', 'validation'])
```

### Dataset transforms

`nlp.DatasetDict` implements the following dataset transforms:
- map
- filter
- sort
- shuffle

### Arguments

The arguments of the methods are the same except for split-specific arguments like `cache_file_name`.
For such arguments, the expected input is a dictionary `{split_name: argument_value}`
It concerns:
- `cache_file_name` in map, filter, sort, shuffle
- `seed` and `generator` in shuffle

Comments:
I did the changes in the docstrings and I added a type check in each `DatasetDict` method to make sure all values are of type `Dataset`

Awesome, do you mind adding these in the doc as well?

I added it to the docs (processing + main classes)

I'm trying to follow along with the following about datasets from the docs:

https://huggingface.co/nlp/loading_datasets.html
https://huggingface.co/nlp/processing.html

However the train_test_split method no longer works as it is expecting a dataset, rather than a datsetdict. How would I got about splitting a CSV into a train and test set? 

I'm trying to utilize the Trainer() class, but am having trouble converting my data from a csv into dataset objects to pass in.
mlsum

Hello, 

The tests for the load_real_data fail, as there is no default language subset to download it looks for a file that does not exist. This bug does not happen when using the load_dataset function, as it asks you to specify a language if you do not, so I submit this PR anyway. The dataset is avalaible on : https://gitlab.lip6.fr/scialom/mlsum_data

Comments:
Thanks @RachelKer for this PR.

I think the dummy_data structure does not also match. In the `_split_generator` you have something like `os.path.join(downloaded_files["validation"], lang+'_val.jsonl')` but in you dummy_data you have `os.path.join(downloaded_files["validation"], lang+"_val.zip",  lang+'_val.jsonl')`. I think ` jsonl` files should be directly  in the `dummy_data` folder without the sub-folder 

@lhoestq 

Hi @RachelKer :)
Thanks for adding MLSUM !

To fix the CI I think you just have to rebase from master

Great, I think it is working now. Thanks :)

It looks like your PR does tons of changes in other datasets. 
Maybe this is because of the merge from master ?

Hmm, I see, sorry I messed up somewhere. Maybe it's easier if we close the pull request and I do another one ?

Yea if it's easier for you feel free to re-open a PR
Allow sequence features for beam + add processed Natural Questions

## Allow Sequence features for Beam Datasets + add Natural Questions

### The issue

The steps of beam datasets processing is the following:
- download the source files and send them in a remote storage (gcs)
- process the files using a beam runner (dataflow)
- save output in remote storage (gcs)
- convert output to arrow in remote storage (gcs)

However it wasn't possible to process `natural_questions` because apache beam's processing outputs parquet files, and it's not yet possible to read parquet files with list features.

### The proposed solution

To allow sequence features for beam I added a workaround that serializes the values using `json.dumps`, so that we end up with strings instead of the original features. Then when the arrow file is created, the serialized objects are transformed back to normal with `json.loads`. Not sure if there's a better way to do it.

### Natural Questions

I was able to process NQ with it, and so I added the json infos file in this PR too.
The processed arrow files are also stored in gcs.
It allows you to load NQ with

```python
from nlp import load_dataset
nq = load_dataset("natural_questions")  # download the 90GB arrow files from gcs and return the dataset
```

### Tests

I added a test case to make sure it works as expected.
Note that the CI will fail because I am updating `natural_questions.py`: it's not synced with the script on S3. It will be synced as soon as this PR is merged.
```
=========================== short test summary info ============================
FAILED tests/test_hf_gcp.py::TestDatasetOnHfGcp::test_script_synced_with_s3_natural_questions/default
```

Comments:
[FEATURE REQUEST] Multiprocessing with for dataset.map, dataset.filter

It would be nice to be able to speed up `dataset.map` or `dataset.filter`. Perhaps this is as easy as sharding the dataset sending each shard to a process/thread/dask pool and using the new `nlp.concatenate_dataset()` function to join them all together?

Comments:
Yes that's definitely something we plan to add ^^

Yes, that would be nice. We could take a look at what tensorflow `tf.data` does under the hood for instance.

So `tf.data.Dataset.map()` returns a `ParallelMapDataset` if `num_parallel_calls is not None` [link](https://github.com/tensorflow/tensorflow/blob/2b96f3662bd776e277f86997659e61046b56c315/tensorflow/python/data/ops/dataset_ops.py#L1623).

There, `num_parallel_calls` is turned into a tensor and and fed to `gen_dataset_ops.parallel_map_dataset` where it looks like tensorflow takes over.

We could start with something simple like a thread or process pool that `imap`s over some shards.
 

Multiprocessing was added in #552 . You can set the number of processes with `.map(..., num_proc=...)`. It also works for `filter`

Closing this one, but feel free to reo-open if you have other questions

@lhoestq Great feature implemented! Do you have plans to add it to official tutorials [Processing data in a Dataset](https://huggingface.co/docs/datasets/processing.html?highlight=save#augmenting-the-dataset)? It took me sometime to find this parallel processing api.

Thanks for the heads up !

I just added a paragraph about multiprocessing:
https://huggingface.co/docs/datasets/master/processing.html#multiprocessing
Change features vs schema logic

## New logic for `nlp.Features` in datasets

Previously, it was confusing to have `features` and pyarrow's `schema` in `nlp.Dataset`.
However `features` is supposed to be the front-facing object to define the different fields of a dataset, while `schema` is only used to write arrow files.

Changes:
- Remove `schema` field in `nlp.Dataset`
- Make `features` the source of truth to read/write examples
- `features` can no longer be `None` in `nlp.Dataset`
- Update `features` after each dataset transform such as `nlp.Dataset.map`

Todo: change the tests to take these changes into account

Comments:
I had to make `SplitDict` serializable to be able to copy `DatasetInfo` objects properly.
Serialization was also asked in #389 

One thing I forgot to say here, is that we also want to use the features arguments of `load_dataset` (which goes in the builder’s config) to override the default features of a dataset script.
Something is wrong with WMT 19 kk-en dataset

The translation in the `train` set does not look right:

```
>>>import nlp
>>>from nlp import load_dataset
>>>dataset = load_dataset('wmt19', 'kk-en')
>>>dataset["train"]["translation"][0]
{'kk': 'Trumpian Uncertainty', 'en': 'Трамптық белгісіздік'}
>>>dataset["validation"]["translation"][0]
{'kk': 'Ақша-несие саясатының сценарийін қайта жазсақ', 'en': 'Rewriting the Monetary-Policy Script'}
```

Comments:
from_dict delete?

AttributeError: type object 'Dataset' has no attribute 'from_dict'

Comments:
`from_dict` was added in #350 that was unfortunately not included in the 0.3.0 release. It's going to be included in the next release that will be out pretty soon though.
Right now if you want to use `from_dict` you have to install the package from the master branch
```
pip install git+https://github.com/huggingface/nlp.git
```

> `from_dict` was added in #350 that was unfortunately not included in the 0.3.0 release. It's going to be included in the next release that will be out pretty soon though.
> Right now if you want to use `from_dict` you have to install the package from the master branch
> 
> ```
> pip install git+https://github.com/huggingface/nlp.git
> ```
OK, thank you.
Is there a way to download only NQ dev?

Maybe I missed that in the docs, but is there a way to only download the dev set of natural questions (~1 GB)? 
As we want to benchmark QA models on different datasets, I would like to avoid downloading the 41GB of training data. 

I tried
```
dataset = nlp.load_dataset('natural_questions', split="validation", beam_runner="DirectRunner")
```
But this still triggered a big download of presumably the whole dataset. Is there any way of doing this or are splits / slicing options only available after downloading?

Thanks!

Comments:
Unfortunately it's not possible to download only the dev set of NQ.

I think we could add a way to download only the test set by adding a custom configuration to the processing script though.

Ok, got it. I think this could be a valuable feature - especially for large datasets like NQ, but potentially also others. 
For us, it will in this case make the difference of using the library or keeping the old downloads of the raw dev datasets.  
However, I don't know if that fits into your plans with the library and can also understand if you don't want to support this.

I don't think we could force this behavior generally since the dataset script authors are free to organize the file download as they want (sometimes the mapping between split and files can be very much nontrivial) but we can add an additional configuration for Natural Question indeed as @lhoestq indicate.
train_test_split error: 'dict' object has no attribute 'deepcopy'

`train_test_split` is giving me an error when I try and call it:

`'dict' object has no attribute 'deepcopy'`

## To reproduce

```
dataset = load_dataset('glue', 'mrpc', split='train')
dataset = dataset.train_test_split(test_size=0.2)
```

## Full Stacktrace
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-feb740dbec9a> in <module>
      1 dataset = load_dataset('glue', 'mrpc', split='train')
----> 2 dataset = dataset.train_test_split(test_size=0.2)

~/anaconda3/envs/fastai2_me/lib/python3.7/site-packages/nlp/arrow_dataset.py in train_test_split(self, test_size, train_size, shuffle, seed, generator, keep_in_memory, load_from_cache_file, train_cache_file_name, test_cache_file_name, writer_batch_size)
   1032                     "writer_batch_size": writer_batch_size,
   1033                 }
-> 1034                 train_kwargs = cache_kwargs.deepcopy()
   1035                 train_kwargs["split"] = "train"
   1036                 test_kwargs = cache_kwargs.deepcopy()

AttributeError: 'dict' object has no attribute 'deepcopy'
```

Comments:
It was fixed in 2ddd18d139d3047c9c3abe96e1e7d05bb360132c.
Could you pull the latest changes from master @morganmcg1 ?

Thanks @lhoestq, works fine now!
MissingBeamOptions for Wikipedia 20200501.en

There may or may not be a regression for the pre-processed Wikipedia dataset. This was working fine 10 commits ago (without having Apache Beam available):

```
nlp.load_dataset('wikipedia', "20200501.en", split='train')
```

And now, having pulled master, I get:

```
Downloading and preparing dataset wikipedia/20200501.en (download: 16.99 GiB, generated: 17.07 GiB, total: 34.06 GiB) to /home/hltcoe/mgordon/.cache/huggingface/datasets/wikipedia/20200501.en/1.0.0/76b0b2747b679bb0ee7a1621e50e5a6378477add0c662668a324a5bc07d516dd...
Traceback (most recent call last):
  File "scripts/download.py", line 11, in <module>
    fire.Fire(download_pretrain)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/home/hltcoe/mgordon/.conda/envs/huggingface/lib/python3.6/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "scripts/download.py", line 6, in download_pretrain
    nlp.load_dataset('wikipedia', "20200501.en", split='train')
  File "/exp/mgordon/nlp/src/nlp/load.py", line 534, in load_dataset
    save_infos=save_infos,
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 460, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/exp/mgordon/nlp/src/nlp/builder.py", line 870, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, S
park, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory).
Example of usage:
        `load_dataset('wikipedia', '20200501.en', beam_runner='DirectRunner')`
```

Comments:
Fixed. Could you try again @mitchellgordon95 ?
It was due a file not being updated on S3.

We need to make sure all the datasets scripts get updated properly @julien-c 

Works for me! Thanks.

I found the same issue with almost any language other than English. (For English, it works). Will someone need to update the file on S3 again?

This is because only some languages are already preprocessed (en, de, fr, it) and stored on our google storage.
We plan to have a systematic way to preprocess more wikipedia languages in the future.

For the other languages you have to process them on your side using apache beam. That's why the lib asks for a Beam runner.
Make select() faster by batching reads

Here's a benchmark:

```
dataset = nlp.load_dataset('bookcorpus', split='train')

start = time.time()
dataset.select(np.arange(1000), reader_batch_size=1, load_from_cache_file=False)
end = time.time()
print(f'{end - start}')

start = time.time()
dataset.select(np.arange(1000), reader_batch_size=1000, load_from_cache_file=False)
end = time.time()
print(f'{end - start}')
```

Without batching, select takes around 1.27 seconds. With batching, it takes around 0.01 seconds. The slowness was upsetting me because dataset.shuffle() was supposed to take ~27 hours for bookcorpus. Now with the fix it takes ~2.5 hours (which still is pretty slow, but I'll open a separate issue for that).

Comments:
Add seed in metrics

With #361 we noticed that some metrics were not deterministic.
In this PR I allow the user to specify numpy's seed when instantiating a metric with `load_metric`.
The seed is set only when `compute` is called, and reset afterwards.

Moreover when calling `compute` with the same metric instance (i.e. same experiment_id), the metric will always return the same results given the same inputs. This is the case even if the seed is was not specified by the user, as the previous seed is going to be reused.

However, instantiating twice a metric (two different experiments) without specifying a seed can create different results.

Comments:
add web_questions

add Web Question dataset
#336 

Maybe @patrickvonplaten you can help with the dummy_data structure? it still broken

Comments:
What does the `nlp-cli dummy_data` command returns ?

`test.json` -> `test` 
and 
`train.json` -> `train`

as shown by the `nlp-cli dummy_data` command ;-)

LGTM for merge @lhoestq - I let you merge if you want to.
Add contiguous sharding

This makes dset.shard() play nice with nlp.concatenate_datasets(). When I originally wrote the shard() method, I was thinking about a distributed training scenario, but https://github.com/huggingface/nlp/pull/389 also uses it for splitting the dataset for distributed preprocessing.

Usage:
```
nlp.concatenate_datasets([dset.shard(n, i, contiguous=True) for i in range(n)])
```

Comments:
Memory issue when doing select

As noticed in #389, the following code loads the entire wikipedia in memory.

```python
import nlp
w = nlp.load_dataset("wikipedia", "20200501.en", split="train")
w.select([0])
```

This is caused by [this line](https://github.com/huggingface/nlp/blob/master/src/nlp/arrow_dataset.py#L626) for some reason, that tries to serialize the function with all the wikipedia data with it.

It's not the case with `.map` or `.filter`.
However functions that are based on `.select` like `.shuffle`, `.shard`, `.train_test_split`, `.sort` are affected.


Comments:
Style change detection

Another [PAN task](https://pan.webis.de/clef20/pan20-web/style-change-detection.html). This time about identifying when the style/author changes in documents.

- There's the possibility of adding the [PAN19](https://zenodo.org/record/3577602) and PAN18 style change detection tasks too (these are datasets whose labels are a subset of PAN20's). These would probably make more sense as separate datasets (like wmt is now)
- I've converted the integer 0,1 values to a boolean
- Using manually downloaded data again. This might be changed at some point following the discussion in https://github.com/huggingface/nlp/pull/349.

Comments:
Concatenate datasets

I'm constructing the "WikiBooks" dataset, which is a concatenation of Wikipedia & BookCorpus. So I implemented the `Dataset.from_concat()` method, which concatenates two datasets with the same schema.

This would also be useful if someone wants to pretrain on a large generic dataset + their own custom dataset. Not in love with the method name, so would love to hear suggestions.

Usage:
```python
from nlp import Dataset, load_dataset

data1, data2 = {"id": [0, 1, 2]}, {"id": [3, 4, 5]}
dset1, dset2 = Dataset.from_dict(data1), Dataset.from_dict(data2)
dset_concat = Dataset.from_concat([dset1, dset2])
print(dset_concat)
# Dataset(schema: {'id': 'int64'}, num_rows: 6)
```

Comments:
Looks cool :)

I feel like 
```python
concatenated_dataset = dataset1.concatenate(dataset2)
```
could be more natural. What do you think ?

Also could you also concatenate the `nlp.Dataset._data_files` ?
```python
return cls(table, info=info, split=split, data_files=self._data_files + other_dataset._data_files)
```

I feel like "WikiBooks" would be a multi task dataset that could fit in the #217 discussion.
Not sure concatenate should be the solution for a multi task dataset.

Thanks for the suggestion! `dset1.concatenate(dset2)` does feel more natural. Although this seems to be a different "class" of transformation function than map() or filter(), acting on two datasets rather than on one. I would prefer the function signature treat both datasets symmetrically.

Python lists have `list1 + list2` or `list1.extend(list2)`.
NumPy has `np.concatenate((arr1, arr2))`.
Pandas has `pd.join((df1, df2))`.
PyTorch has `ConcatDataset((dset1, dset2))`.

Given the symmetrical treatment and clear communication that this creates a new object, rather than a simple chaining on the first, my preference is now for `nlp.concatenate((dset1, dset2))`. This would place the function in the same API class as `nlp.load_dataset`. Does that work?

The multi-task discussion is interesting, thanks for pointing me to that! I'll be focusing on T5 in a few weeks, so I'm sure I'll have many opinions then :). For now, I think a simple concatenate feature is important and orthogonal to that discussion. For example, a user may want to create a custom dataset that joins Wikipedia with their own custom text.

> Given the symmetrical treatment and clear communication that this creates a new object, rather than a simple chaining on the first, my preference is now for `nlp.concatenate((dset1, dset2))`. This would place the function in the same API class as `nlp.load_dataset`. Does that work?

Yep I like this idea. Maybe `nlp.concatenate_datasets()` ?

> For now, I think a simple concatenate feature is important and orthogonal to that discussion. For example, a user may want to create a custom dataset that joins Wikipedia with their own custom text.

I agree :)

Great, just updated!
🐛 [Dataset] Cannot download wmt14, wmt15 and wmt17

1. I try downloading `wmt14`, `wmt15`, `wmt17`, `wmt19` with the following code:
```
nlp.load_dataset('wmt14','de-en')
nlp.load_dataset('wmt15','de-en')
nlp.load_dataset('wmt17','de-en')
nlp.load_dataset('wmt19','de-en')
```
The code runs but the download speed is **extremely slow**, the same behaviour is not observed on `wmt16` and `wmt18`

2. When trying to download `wmt17 zh-en`, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz

Comments:
similar slow download speed here for nlp.load_dataset('wmt14', 'fr-en')
`
Downloading: 100%|██████████████████████████████████████████████████████████| 658M/658M [1:00:42<00:00, 181kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████| 918M/918M [1:39:38<00:00, 154kB/s]
Downloading:   2%|▉                                                       | 40.9M/2.37G [04:48<5:03:06, 128kB/s]
`
Could we just download a specific subdataset in 'wmt14', such as 'newstest14'? 

> The code runs but the download speed is extremely slow, the same behaviour is not observed on wmt16 and wmt18

The original source for the files may provide slow download speeds.
We can probably host these files ourselves.

> When trying to download wmt17 zh-en, I got the following error:
> ConnectionError: Couldn't reach https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz

Looks like the file`UNv1.0.en-zh.tar.gz` is missing, or the url changed. We need to fix that

> Could we just download a specific subdataset in 'wmt14', such as 'newstest14'?

Right now I don't think it's possible. Maybe @patrickvonplaten knows more about it


Yeah, the download speed is sadly always extremely slow :-/. 
I will try to check out the `wmt17 zh-en` bug :-) 

Maybe this can be used - https://stuncorpusprod.blob.core.windows.net/corpusfiles/UNv1.0.en-zh.tar.gz.00 

These issues seem to be fixed now.
Conversion through to_pandas output numpy arrays for lists instead of python objects

In a related question, the conversion through to_pandas output numpy arrays for the lists instead of python objects.

Here is an example:
```python
>>> dataset._data.slice(key, 1).to_pandas().to_dict("list")
{'sentence1': ['Amrozi accused his brother , whom he called " the witness " , of deliberately distorting his evidence .'], 'sentence2': ['Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .'], 'label': [1], 'idx': [0], 'input_ids': [array([  101,  7277,  2180,  5303,  4806,  1117,  1711,   117,  2292,
        1119,  1270,   107,  1103,  7737,   107,   117,  1104,  9938,
        4267, 12223, 21811,  1117,  2554,   119,   102])], 'token_type_ids': [array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0])], 'attention_mask': [array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1])]}
>>> type(dataset._data.slice(key, 1).to_pandas().to_dict("list")['input_ids'][0])
<class 'numpy.ndarray'>
>>> dataset._data.slice(key, 1).to_pydict()
{'sentence1': ['Amrozi accused his brother , whom he called " the witness " , of deliberately distorting his evidence .'], 'sentence2': ['Referring to him as only " the witness " , Amrozi accused his brother of deliberately distorting his evidence .'], 'label': [1], 'idx': [0], 'input_ids': [[101, 7277, 2180, 5303, 4806, 1117, 1711, 117, 2292, 1119, 1270, 107, 1103, 7737, 107, 117, 1104, 9938, 4267, 12223, 21811, 1117, 2554, 119, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
```

Comments:
To convert from arrow type we have three options: to_numpy, to_pandas and to_pydict/to_pylist.

- to_numpy and to_pandas return numpy arrays instead of lists but are very fast.
- to_pydict/to_pylist can be 100x slower and become the bottleneck for reading data, but at least they return lists.

Maybe we can have to_pydict/to_pylist as the default and use to_numpy or to_pandas when the format (set by `set_format`) is 'numpy' or 'pandas'
Update dataset loading and features - Add TREC dataset

This PR:
- add a template for a new dataset script
- update the caching structure so that the path to the cached data files is also a function of the dataset loading script hash. This way when you update a loading script the data will be automatically updated instead of falling back to the previous version (which is usually a outdated). This makes it in particular easier to iterate when writing a new dataset loading script.
- fix a bug in the `ClassLabel` feature and make it more flexible so that its methods `str2int` and `int2str` can also accept list, numpy arrays and PyTorch/TensorFlow tensors.
- add the TREC-6 dataset

Comments:
I just copied the files that are on google storage to follow the new `_relative_data_dir ` format. It should be good to merge now :)

Well actually it seems there are some merge conflicts to fix first
[dataset] Structure of MLQA seems unecessary nested

The features of the MLQA dataset comprise several nested dictionaries with a single element inside (for `questions` and `ids`): https://github.com/huggingface/nlp/blob/master/datasets/mlqa/mlqa.py#L90-L97

Should we keep this @mariamabarham @patrickvonplaten? Was this added for compatibility with tfds?

```python
            features=nlp.Features(
                {
                    "context": nlp.Value("string"),
                    "questions": nlp.features.Sequence({"question": nlp.Value("string")}),
                    "answers": nlp.features.Sequence(
                        {"text": nlp.Value("string"), "answer_start": nlp.Value("int32"),}
                    ),
                    "ids": nlp.features.Sequence({"idx": nlp.Value("string")})
```

Comments:
Same for the RACE dataset: https://github.com/huggingface/nlp/blob/master/datasets/race/race.py

Should we scan all the datasets to remove this pattern of un-necessary nesting?

You're right, I think we don't need to use the nested dictionary.
TypeError when computing bertscore

Hi, 

I installed nlp 0.3.0 via pip, and my python version is 3.7.
When I tried to compute bertscore with the code:
```
import nlp 
bertscore = nlp.load_metric('bertscore')  
# load hyps and refs 
...
print (bertscore.compute(hyps, refs, lang='en'))
```

I got the following error.
```
Traceback (most recent call last):
  File "bert_score_evaluate.py", line 16, in <module>
    print (bertscore.compute(hyps, refs, lang='en'))
  File "/home/willywsm/anaconda3/envs/torcher/lib/python3.7/site-packages/nlp/metric.py", line 200, in compute
    output = self._compute(predictions=predictions, references=references, **metrics_kwargs)
  File "/home/willywsm/anaconda3/envs/torcher/lib/python3.7/site-packages/nlp/metrics/bertscore/fb176889831bf0ce995ed197edc94b2e9a83f647a869bb8c9477dbb2d04d0f08/bertscore.py", line 105, in _compute
    hashcode = bert_score.utils.get_hash(model_type, num_layers, idf, rescale_with_baseline)
TypeError: get_hash() takes 3 positional arguments but 4 were given
```

It seems like there is something wrong with get_hash() function?

Comments:
I am not able to reproduce this issue on my side.
Could you give us more details about the inputs you used ?

I do get another error though:
```
~/.virtualenvs/hf-datasets/lib/python3.7/site-packages/bert_score/utils.py in bert_cos_score_idf(model, refs, hyps, tokenizer, idf_dict, verbose, batch_size, device, all_layers)
    371         return sorted(list(set(l)), key=lambda x: len(x.split(" ")))
    372 
--> 373     sentences = dedup_and_sort(refs + hyps)
    374     embs = []
    375     iter_range = range(0, len(sentences), batch_size)

ValueError: operands could not be broadcast together with shapes (0,) (2,)
```
That's because it gets numpy arrays as input and not lists. See #387 

The other issue was fixed by #403 

Do you still get this issue @willywsm1013 ?
Add dataset post processing for faiss indexes

# Post processing of datasets for faiss indexes

Now that we can have datasets with embeddings (see `wiki_pr` for example), we can allow users to load the dataset + get the Faiss index that comes with it to do nearest neighbors queries.

## Implementation proposition

- Faiss indexes have to be added to the `nlp.Dataset` object, and therefore it's in a different scope that what are doing the `_split_generators` and `_generate_examples` methods of `nlp.DatasetBuilder`. Therefore I added a new method for post processing of the `nlp.Dataset` object called `_post_process` (name could change)
- The role of `_post_process` is to apply dataset transforms (filter/map etc.) or indexing functions (add_faiss_index) to modify/enrich the `nlp.Dataset` object. It is not part of the `download_and_prepare` process (that is focused on arrow files creation) so the post processing is run inside the `as_dataset` method.
- `_post_process` can generate new files (cached files from dataset transforms or serialized faiss indexes) and their names are defined by `_post_processing_resources`
- as we know what are the post processing resources, we can download them automatically from google storage instead of computing them if they're available (as we do for arrow files)

I'd happy to discuss these choices !

## The `wiki_dpr` index

It takes 1h20 and ~7GB of memory to compute. The final index is 1.42GB and takes ~1.5GB of memory.
This is pretty cool given that a naive flat index would take 170GB of memory to store the 21M vectors of dim 768.
I couldn't use directly the Faiss `index_factory` as I needed to set the metric to inner product.

## Example of usage

```python
import nlp
dset = nlp.load_dataset(
    "wiki_dpr",
    "psgs_w100_with_nq_embeddings",
    split="train",
    with_index=True
)
print(len(dset), dset.list_indexes())  # (21015300, ['embeddings'])
```

(it also works with the dataset configuration without the embeddings because I added the index file in google storage for this one too)

## Demo

You can also check a demo on google colab that shows how to use it with the DPRQuestionEncoder from transformers:
https://colab.research.google.com/drive/1FakNU8W5EPMcWff7iP1H6REg3XSS0YLp?usp=sharing


Comments:
I changed the `wiki_dpr` script to ignore the last 24 examples for now. Hopefully we'll have the full version soon.
The datasets_infos.json and the data on GCS are updated.

And I also added a check to make sure we don't have post processing resources in sub-directories.

I added a dummy config that can be loaded with:
```python
wiki = load_dataset("wiki_dpr", "dummy_psgs_w100_no_embeddings", with_index=True, split="train")
```
It's only 6MB of arrow files and 30MB of index
Make the json script more flexible

Fix https://github.com/huggingface/nlp/issues/359
Fix https://github.com/huggingface/nlp/issues/369

JSON script now can accept JSON files containing a single dict with the records as a list in one attribute to the dict (previously it only accepted JSON files containing records as rows of dicts in the file).

In this case, you should indicate using `field=XXX` the name of the field in the JSON structure which contains the records you want to load. The records can be a dict of lists or a list of dicts.

E.g. to load the SQuAD dataset JSON (without using the `squad` specific dataset loading script), in which the data rows are in the `data` field of the JSON dict, you can do:
```python
from nlp import load_dataset
dataset = load_dataset('json', data_files='/PATH/TO/JSON', field='data')
```

Comments:
can't load local dataset: pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries

Trying to load a local SQuAD-formatted dataset (from a JSON file, about 60MB):
```
dataset = nlp.load_dataset(path='json', data_files={nlp.Split.TRAIN: ["./path/to/file.json"]})
```
causes
```
Traceback (most recent call last):
  File "dataloader.py", line 9, in <module>
    ["./path/to/file.json"]})
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/load.py", line 524, in load_dataset
    save_infos=save_infos,
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 483, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/builder.py", line 719, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False):
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/XXX/.conda/envs/torch/lib/python3.7/site-packages/nlp/datasets/json/88c1bc5c68489f7eda549ed05a5a738527c613b3e7a4ee3524d9d233353a949b/json.py", line 53, in _generate_tables
    file, read_options=self.config.pa_read_options, parse_options=self.config.pa_parse_options,
  File "pyarrow/_json.pyx", line 191, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```

I haven't been able to find any reports of this specific pyarrow error here or elsewhere. 

Comments:
I am able to reproduce this with the official SQuAD `train-v2.0.json` file downloaded directly from https://rajpurkar.github.io/SQuAD-explorer/

I am facing this issue in transformers library 3.0.2 while reading a csv using datasets.
Is this fixed in latest version? 
I updated the latest version 4.0.1 but still getting this error. What could cause this error?
load_metric can't acquire lock anymore

I can't load metric (glue) anymore after an error in a previous run. I even removed the whole cache folder `/home/XXX/.cache/huggingface/`, and the issue persisted. What are the steps to fix this?

    Traceback (most recent call last):
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/metric.py", line 101, in __init__
        self.filelock.acquire(timeout=1)
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/filelock.py", line 278, in acquire
        raise Timeout(self._lock_file)
    filelock.Timeout: The file lock '/home/XXX/.cache/huggingface/metrics/glue/1.0.0/1-glue-0.arrow.lock' could not be acquired.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "examples_huggingface_nlp.py", line 268, in <module>
        main()
      File "examples_huggingface_nlp.py", line 242, in main
        dataset, metric = get_dataset_metric(glue_task)
      File "examples_huggingface_nlp.py", line 77, in get_dataset_metric
        metric = nlp.load_metric('glue', glue_config, experiment_id=1)
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/load.py", line 440, in load_metric
        **metric_init_kwargs,
      File "/home/XXX/miniconda3/envs/ML-DL-py-3.7/lib/python3.7/site-packages/nlp/metric.py", line 104, in __init__
        "Cannot acquire lock, caching file might be used by another process, "
    ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id' for this run.
    I0709 15:54:41.008838 139854118430464 filelock.py:318] Lock 139852058030936 released on /home/XXX/.cache/huggingface/metrics/glue/1.0.0/1-glue-0.arrow.lock


Comments:
I found that, in the same process (or the same interactive session), if I do

import nlp

m1 = nlp.load_metric('glue', 'mrpc')
m2 = nlp.load_metric('glue', 'sst2')

I will get the same error `ValueError: Cannot acquire lock, caching file might be used by another process, you should setup a unique 'experiment_id'`.
Add quora dataset

Added the [Quora question pairs dataset](https://www.quora.com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs).

Implementation Notes:
- I used the original version provided on the quora website. There's also a [Kaggle competition](https://www.kaggle.com/c/quora-question-pairs) which has a nice train/test split but I can't find an easy way to download it.
- I've made the questions into a list:
    ```python
    {
        "questions": [
            {"id":0, "text": "Is this an example question?"},
            {"id":1, "text": "Is this a sample question?"},
        ],
    ...
    }
    ```
    rather than:
    ```python
    {
        "question1": "Is this an example question?",
        "question2": "Is this a sample question?"
        "qid0": 0
        "qid1": 1
    ...
    }
    ```
    Not sure if this was the right call.
- Can't find a good citation for this dataset

Comments:
Tests seem to be failing because of pandas

Kaggle needs authentification to download datasets. We don't have a way to handle that in the lib for now
How to augment data ?

Is there any clean way to augment data ?

For now my work-around is to use batched map, like this :

```python
def aug(samples):
    # Simply copy the existing data to have x2 amount of data
    for k, v in samples.items():
        samples[k].extend(v)
    return samples

dataset = dataset.map(aug, batched=True)
```

Comments:
Using batched map is probably the easiest way at the moment.
What kind of augmentation would you like to do ?

Some samples in the dataset are too long, I want to divide them in several samples.

Using batched map is the way to go then.
We'll make it clearer in the docs that map could be used for augmentation.

Let me know if you think there should be another way to do it. Or feel free to close the issue otherwise.

It just feels awkward to use map to augment data. Also it means it's not possible to augment data in a non-batched way.

But to be honest I have no idea of a good API...

Or for non-batched samples, how about returning a tuple ?

```python
def aug(sample):
    # Simply copy the existing data to have x2 amount of data
    return sample, sample

dataset = dataset.map(aug)
```

It feels really natural and easy, but :

* it means the behavior with batched data is different
* I don't know how doable it is backend-wise

@lhoestq 

As we're working with arrow's columnar format we prefer to play with batches that are dictionaries instead of tuples.
If we have tuple it implies to re-format the data each time we want to write to arrow, which can lower the speed of map for example.

It's also a matter of coherence, as we don't want users to be confused whether they have to return dictionaries for some functions and tuples for others when they're doing batches.
add MS MARCO dataset

This PR adds the MS MARCO dataset as requested in this issue #336. MS mARCO has multiple task including:

- Passage and Document Retrieval

- Keyphrase Extraction

- QA and NLG

This PR only adds the 2 versions of the  QA and NLG task dataset which was realeased with the original paper here https://arxiv.org/pdf/1611.09268.pdf 

Tests are failing because of the dummy data. I tried to fix it without success. Can you please have a look at it? @patrickvonplaten , @lhoestq 

Comments:
The dummy data for v2.1 is missing as far as I can see. I think running the  dummy data command should work correctly here. 

Also, it might be that the structure of the dummy data is wrong - looking at `generate_examples` the structure does not look too easy.

The fact that the dummy data for v2.1 is missing shouldn't make the test fails I think. But as you mention the dummy data structure of v1.1 is wrong. I tried to rename files but it does not solve the issue.

Is MS mARCO added to nlp library?I am not able to view it?

> Is MS mARCO added to nlp library?I am not able to view it?

Hi @parthplc ,the PR is  not merged yet. The dummy data structure is still failing. Maybe @patrickvonplaten can help with it.

Dataset is fixed and should be ready for use. @mariamabarham @lhoestq feel free to merge whenever!

> Dataset is fixed and should be ready for use. @mariamabarham @lhoestq feel free to merge whenever!

thanks
🐛 [Metrics] ROUGE is non-deterministic

If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.

Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.

Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :

> ['0.3350', '0.1470', '0.2329']
['0.3358', '0.1451', '0.2332']

---

Why ROUGE is not deterministic ?

Comments:
Hi, can you give a full self-contained example to reproduce this behavior?

> Hi, can you give a full self-contained example to reproduce this behavior?

There is a notebook in the post ;)

> If I run the ROUGE metric 2 times, with same predictions / references, the scores are slightly different.
> 
> Refer to [this Colab notebook](https://colab.research.google.com/drive/1wRssNXgb9ldcp4ulwj-hMJn0ywhDOiDy?usp=sharing) for reproducing the problem.
> 
> Example of F-score for ROUGE-1, ROUGE-2, ROUGE-L in 2 differents run :
> 
> > ['0.3350', '0.1470', '0.2329']
> > ['0.3358', '0.1451', '0.2332']
> 
> Why ROUGE is not deterministic ?

This is because of rouge's `BootstrapAggregator` that uses sampling to get confidence intervals (low, mid, high).
You can get deterministic scores per sentence pair by using
```python
score = rouge.compute(rouge_types=["rouge1", "rouge2", "rougeL"], use_aggregator=False)
```
Or you can set numpy's random seed if you still want to use the aggregator.

Maybe we can set all the random seeds of numpy/torch etc. while running `metric.compute` ?

We should probably indeed!

Now if you re-run the notebook, the two printed results are the same @colanim
```
['0.3356', '0.1466', '0.2318']
['0.3356', '0.1466', '0.2318']
```
However across sessions, the results may change (as numpy's random seed can be different). You can prevent that by setting your seed:
```python
rouge = nlp.load_metric('rouge', seed=42)
```

> 

Minor nit: Note that "aggregator" is misspelled in this command. Should be `use_aggregator=False`.  

Thanks, I fixed the code snippet
Starting to add some real doc

Adding a lot of documentation for:
- load a dataset
- explore the dataset object
- process data with the dataset
- add a new dataset script
- share a dataset script
- full package reference

This version of the doc can be explored here: https://2219-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html

Also:
- fix a bug in `train_test_split`
- update the `csv` script
- add a verbose argument to the dataset processing methods

Still missing:
- doc for the metrics
- how to directly upload a community provided dataset with the CLI
- clean up more docstrings
- add the `features` argument to `load_dataset` (should be another PR)

Comments:
Ok this is starting to be really big so it's probably good to merge this first version of the doc and continue in another PR :)

This first version of the doc can be explored here: https://2219-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html
Add hashes to cnn_dailymail

The URL hashes are helpful for comparing results from other sources.

Comments:
Looks you to me :)

Could you also update the json file that goes with the dataset script by doing 
```
nlp-cli test ./datasets/cnn_dailymail --save_infos --all_configs
```
It will update the features metadata and the size of the dataset with your changes.

@lhoestq I ran that command.

Thanks for the helpful repository!
Add text dataset

Usage:

```python
from nlp import load_dataset
dset = load_dataset("text", data_files="/path/to/file.txt")["train"]
```


I created a dummy_data.zip which contains three files: `train.txt`, `test.txt`, `dev.txt`. Each of these contains two lines. It passes

```bash
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_text
```

but I would like a second set of eyes to ensure I did it right.


Comments:
can't load SNLI dataset

`nlp` seems to load `snli` from some URL based on nlp.stanford.edu. This subdomain is frequently down -- including right now, when I'd like to load `snli` in a Colab notebook, but can't.

Is there a plan to move these datasets to huggingface servers for a more stable solution?

Btw, here's the stack trace:

```
File "/content/nlp/src/nlp/builder.py", line 432, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/content/nlp/src/nlp/builder.py", line 466, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/content/nlp/src/nlp/datasets/snli/e417f6f2e16254938d977a17ed32f3998f5b23e4fcab0f6eb1d28784f23ea60d/snli.py", line 76, in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 217, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in download
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/py_utils.py", line 190, in map_nested
    return function(data_struct)
  File "/content/nlp/src/nlp/utils/download_manager.py", line 156, in <lambda>
    lambda url: cached_path(url, download_config=self._download_config,), url_or_urls,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 198, in cached_path
    local_files_only=download_config.local_files_only,
  File "/content/nlp/src/nlp/utils/file_utils.py", line 356, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://nlp.stanford.edu/projects/snli/snli_1.0.zip
```

Comments:
I just added the processed files of `snli` on our google storage, so that when you do `load_dataset` it can download the processed files from there :)

We are thinking about having available those processed files for more datasets in the future, because sometimes files aren't available (like for `snli`), or the download speed is too slow, or sometimes the files take time to be processed.

Closing this one. Feel free to re-open if you have other questions :)

Thank you!
More faiss control

Allow users to specify a faiss index they created themselves, as sometimes indexes can be composite for examples

Comments:
> Ok, so we're getting rid of the `FaissGpuOptions`?

We support `device=...` because it's simple, but faiss GPU options can be used in so many ways (you can set different gpu options for the different parts of your index for example) that it's probably better to let the user create and configure its index and then use `custom_index=...`