devopsmarc/my-issues-dataset huggingface dataset:
When will the remaining math_dataset modules be added as dataset objects 
 Currently only the algebra_linear_1d is supported. Is there a timeline for making the other modules supported. If no timeline is established, how can I help? 
 Hi @tylerroost, we don't have a timeline for this at the moment.
If you want to give it a look we would be happy to review a PR on it.
Also, the library is one week old so everything is quite barebones, in particular the doc.
You should expect some bumps on the road.

To get you started, you can check the datasets scripts in the `./datasets` folder on the repo and find the one on math_datasets that will need to be modified. Then you should check the original repository on the math_dataset to see where the other files to download are located and what is the expected format for the various parts of the dataset.

To get a general overview on how datasets scripts are written and used, you can read the nice tutorial on how to add a new dataset for TensorFlow Dataset [here](https://www.tensorflow.org/datasets/add_dataset), our API is not exactly identical but it can give you a high-level overview.
Weird-ish: Not creating unique caches for different phases 
 Sample code:

```python
import nlp
dataset = nlp.load_dataset('boolq')

def func1(x):
    return x

def func2(x):
    return None

train_output = dataset["train"].map(func1)
valid_output = dataset["validation"].map(func1)
print()
print(len(train_output), len(valid_output))
# Output: 9427 9427
```

The map method in both cases seem to be pointing to the same cache, so the latter call based on the validation data will return the processed train data cache.

What's weird is that the following doesn't seem to be an issue:

```python
train_output = dataset["train"].map(func2)
valid_output = dataset["validation"].map(func2)
print()
print(len(train_output), len(valid_output))
# 9427 3270
``` 
 Looks like a duplicate of #120.
This is already fixed on master. We'll do a new release on pypi soon
[Bug] labels of glue/ax are all -1  
 ```
ax = nlp.load_dataset('glue', 'ax')
for i in range(30): print(ax['test'][i]['label'], end=', ')
```
```
-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 
``` 
 This is the test set given by the Glue benchmark. The labels are not provided, and therefore set to -1.
Cannot upload my own dataset 
 I look into `nlp-cli` and `user.py` to learn how to upload my own data.

It is supposed to work like this
- Register to get username, password at huggingface.co
- `nlp-cli login` and type username, passworld
- I have a single file to upload at `./ttc/ttc_freq_extra.csv`
- `nlp-cli upload ttc/ttc_freq_extra.csv`

But I got this error.

```
2020-05-21 16:33:52.722464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
About to upload file /content/ttc/ttc_freq_extra.csv to S3 under filename ttc/ttc_freq_extra.csv and namespace korakot
Proceed? [Y/n] y
Uploading... This might take a while if files are large
Traceback (most recent call last):
  File "/usr/local/bin/nlp-cli", line 33, in <module>
    service.run()
  File "/usr/local/lib/python3.6/dist-packages/nlp/commands/user.py", line 234, in run
    token=token, filename=filename, filepath=filepath, organization=self.args.organization
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 141, in presign_and_upload
    urls = self.presign(token, filename=filename, organization=organization)
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 132, in presign
    return PresignedUrl(**d)
TypeError: __init__() got an unexpected keyword argument 'cdn'
``` 
 It's my misunderstanding. I cannot just upload a csv. I need to write a dataset loading script too.
Cannot upload my own dataset 
 I look into `nlp-cli` and `user.py` to learn how to upload my own data.

It is supposed to work like this
- Register to get username, password at huggingface.co
- `nlp-cli login` and type username, passworld
- I have a single file to upload at `./ttc/ttc_freq_extra.csv`
- `nlp-cli upload ttc/ttc_freq_extra.csv`

But I got this error.

```
2020-05-21 16:33:52.722464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
About to upload file /content/ttc/ttc_freq_extra.csv to S3 under filename ttc/ttc_freq_extra.csv and namespace korakot
Proceed? [Y/n] y
Uploading... This might take a while if files are large
Traceback (most recent call last):
  File "/usr/local/bin/nlp-cli", line 33, in <module>
    service.run()
  File "/usr/local/lib/python3.6/dist-packages/nlp/commands/user.py", line 234, in run
    token=token, filename=filename, filepath=filepath, organization=self.args.organization
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 141, in presign_and_upload
    urls = self.presign(token, filename=filename, organization=organization)
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 132, in presign
    return PresignedUrl(**d)
TypeError: __init__() got an unexpected keyword argument 'cdn'
``` 
 I now try with the sample `datasets/csv` folder. 

    nlp-cli upload csv

The error is still the same

```
2020-05-21 17:20:56.394659: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
About to upload file /content/csv/csv.py to S3 under filename csv/csv.py and namespace korakot
About to upload file /content/csv/dummy/0.0.0/dummy_data.zip to S3 under filename csv/dummy/0.0.0/dummy_data.zip and namespace korakot
Proceed? [Y/n] y
Uploading... This might take a while if files are large
Traceback (most recent call last):
  File "/usr/local/bin/nlp-cli", line 33, in <module>
    service.run()
  File "/usr/local/lib/python3.6/dist-packages/nlp/commands/user.py", line 234, in run
    token=token, filename=filename, filepath=filepath, organization=self.args.organization
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 141, in presign_and_upload
    urls = self.presign(token, filename=filename, organization=organization)
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 132, in presign
    return PresignedUrl(**d)
TypeError: __init__() got an unexpected keyword argument 'cdn'
```
Cannot upload my own dataset 
 I look into `nlp-cli` and `user.py` to learn how to upload my own data.

It is supposed to work like this
- Register to get username, password at huggingface.co
- `nlp-cli login` and type username, passworld
- I have a single file to upload at `./ttc/ttc_freq_extra.csv`
- `nlp-cli upload ttc/ttc_freq_extra.csv`

But I got this error.

```
2020-05-21 16:33:52.722464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
About to upload file /content/ttc/ttc_freq_extra.csv to S3 under filename ttc/ttc_freq_extra.csv and namespace korakot
Proceed? [Y/n] y
Uploading... This might take a while if files are large
Traceback (most recent call last):
  File "/usr/local/bin/nlp-cli", line 33, in <module>
    service.run()
  File "/usr/local/lib/python3.6/dist-packages/nlp/commands/user.py", line 234, in run
    token=token, filename=filename, filepath=filepath, organization=self.args.organization
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 141, in presign_and_upload
    urls = self.presign(token, filename=filename, organization=organization)
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 132, in presign
    return PresignedUrl(**d)
TypeError: __init__() got an unexpected keyword argument 'cdn'
``` 
 We haven't tested the dataset upload feature yet cc @julien-c 
This is on our short/mid-term roadmap though
Cannot upload my own dataset 
 I look into `nlp-cli` and `user.py` to learn how to upload my own data.

It is supposed to work like this
- Register to get username, password at huggingface.co
- `nlp-cli login` and type username, passworld
- I have a single file to upload at `./ttc/ttc_freq_extra.csv`
- `nlp-cli upload ttc/ttc_freq_extra.csv`

But I got this error.

```
2020-05-21 16:33:52.722464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
About to upload file /content/ttc/ttc_freq_extra.csv to S3 under filename ttc/ttc_freq_extra.csv and namespace korakot
Proceed? [Y/n] y
Uploading... This might take a while if files are large
Traceback (most recent call last):
  File "/usr/local/bin/nlp-cli", line 33, in <module>
    service.run()
  File "/usr/local/lib/python3.6/dist-packages/nlp/commands/user.py", line 234, in run
    token=token, filename=filename, filepath=filepath, organization=self.args.organization
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 141, in presign_and_upload
    urls = self.presign(token, filename=filename, organization=organization)
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 132, in presign
    return PresignedUrl(**d)
TypeError: __init__() got an unexpected keyword argument 'cdn'
``` 
 Even if I fix the `TypeError: __init__() got an unexpected keyword argument 'cdn'` error, it looks like it still uploads to                                                                                                                               `https://s3.amazonaws.com/models.huggingface.co/bert/<namespace>/<dataset_name>` instead of `https://s3.amazonaws.com/datasets.huggingface.co/nlp/<namespace>/<dataset_name>`
Cannot upload my own dataset 
 I look into `nlp-cli` and `user.py` to learn how to upload my own data.

It is supposed to work like this
- Register to get username, password at huggingface.co
- `nlp-cli login` and type username, passworld
- I have a single file to upload at `./ttc/ttc_freq_extra.csv`
- `nlp-cli upload ttc/ttc_freq_extra.csv`

But I got this error.

```
2020-05-21 16:33:52.722464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
About to upload file /content/ttc/ttc_freq_extra.csv to S3 under filename ttc/ttc_freq_extra.csv and namespace korakot
Proceed? [Y/n] y
Uploading... This might take a while if files are large
Traceback (most recent call last):
  File "/usr/local/bin/nlp-cli", line 33, in <module>
    service.run()
  File "/usr/local/lib/python3.6/dist-packages/nlp/commands/user.py", line 234, in run
    token=token, filename=filename, filepath=filepath, organization=self.args.organization
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 141, in presign_and_upload
    urls = self.presign(token, filename=filename, organization=organization)
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 132, in presign
    return PresignedUrl(**d)
TypeError: __init__() got an unexpected keyword argument 'cdn'
``` 
 @lhoestq The endpoints in https://github.com/huggingface/nlp/blob/master/src/nlp/hf_api.py should be (depending on the type of file):
```
POST   /api/datasets/presign
GET    /api/datasets/listObjs
DELETE /api/datasets/deleteObj
POST   /api/metrics/presign  
GET    /api/metrics/listObjs
DELETE /api/metrics/deleteObj
```

In addition to this, @thomwolf cleaned up the objects with dataclasses but you should revert this and re-align to the hf_api that's in this branch of transformers: https://github.com/huggingface/transformers/pull/4632 (so that potential new JSON attributes in the API output don't break existing versions of any library)
Cannot upload my own dataset 
 I look into `nlp-cli` and `user.py` to learn how to upload my own data.

It is supposed to work like this
- Register to get username, password at huggingface.co
- `nlp-cli login` and type username, passworld
- I have a single file to upload at `./ttc/ttc_freq_extra.csv`
- `nlp-cli upload ttc/ttc_freq_extra.csv`

But I got this error.

```
2020-05-21 16:33:52.722464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
About to upload file /content/ttc/ttc_freq_extra.csv to S3 under filename ttc/ttc_freq_extra.csv and namespace korakot
Proceed? [Y/n] y
Uploading... This might take a while if files are large
Traceback (most recent call last):
  File "/usr/local/bin/nlp-cli", line 33, in <module>
    service.run()
  File "/usr/local/lib/python3.6/dist-packages/nlp/commands/user.py", line 234, in run
    token=token, filename=filename, filepath=filepath, organization=self.args.organization
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 141, in presign_and_upload
    urls = self.presign(token, filename=filename, organization=organization)
  File "/usr/local/lib/python3.6/dist-packages/nlp/hf_api.py", line 132, in presign
    return PresignedUrl(**d)
TypeError: __init__() got an unexpected keyword argument 'cdn'
``` 
 New commands are
```
nlp-cli upload_dataset <path/to/dataset>
nlp-cli upload_metric <path/to/metric>
nlp-cli s3_datasets {rm, ls}
nlp-cli s3_metrics {rm, ls}
```
Closing this issue.
[Feature request] separate split name and split instructions 
 Currently, the name of an nlp.NamedSplit is parsed in arrow_reader.py and used as the instruction.

This makes it impossible to have several training sets, which can occur when:
- A dataset corresponds to a collection of sub-datasets
- A dataset was built in stages, adding new examples at each stage

Would it be possible to have two separate fields in the Split class, a name /instruction and a unique ID that is used as the key in the builder's split_dict ? 
 If your dataset is a collection of sub-datasets, you should probably consider having one config per sub-dataset. For example for Glue, we have sst2, mnli etc.
If you want to have multiple train sets (for example one per stage). The easiest solution would be to name them `nlp.Split("train_stage1")`, `nlp.Split("train_stage2")`, etc. or something like that.
[Feature request] separate split name and split instructions 
 Currently, the name of an nlp.NamedSplit is parsed in arrow_reader.py and used as the instruction.

This makes it impossible to have several training sets, which can occur when:
- A dataset corresponds to a collection of sub-datasets
- A dataset was built in stages, adding new examples at each stage

Would it be possible to have two separate fields in the Split class, a name /instruction and a unique ID that is used as the key in the builder's split_dict ? 
 Thanks for the tip! I ended up setting up three different versions of the dataset with their own configs.

for the named splits, I was trying with `nlp.Split("train-stage1")`, which fails. Changing to `nlp.Split("train_stage1")` works :) I looked for examples of what works in the code comments, it may be worth adding some examples of valid/invalid names in there?
Add a method to shuffle a dataset 
 Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think? 
 +1 for the naming convention

About the `shuffle` method, from my understanding it should be done in `Dataloader` (better separation between dataset processing - usage)
Add a method to shuffle a dataset 
 Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think? 
 +1 for shuffle in `Dataloader`. 
Some `Dataloader` just store idxs of dataset and just shuffle those idxs, which might(?) be faster than do shuffle in dataset, especially when doing shuffle every epoch.

Also +1 for the naming convention.
Add a method to shuffle a dataset 
 Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think? 
 As you might already know the issue of dataset shuffling came up in the nlp code [walkthrough](https://youtu.be/G3pOvrKkFuk?t=3204) by Yannic Kilcher
[Feature request] Add cos-e v1.0 
 I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf). 
 Sounds good, @mariamabarham do you want to give a look?
I think we should have two configurations so we can allow either version of the dataset to be loaded with the `1.0` version being the default maybe.

Cc some authors of the great cos-e: @nazneenrajani @bmccann
[Feature request] Add cos-e v1.0 
 I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf). 
 cos_e v1.0 is related to CQA v1.0 but only CQA v1.11 dataset is available on their website. Indeed their is lots of ids in cos_e v1, which are  not in CQA v1.11  or the other way around.
@sarahwie, @thomwolf, @nazneenrajani,  @bmccann  do you know where I can  find CQA v1.0
[Feature request] Add cos-e v1.0 
 I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf). 
 @mariamabarham I'm also not sure where to find CQA 1.0. Perhaps it's not possible to include this version of the dataset. I'll close the issue if that's the case.
[Feature request] Add cos-e v1.0 
 I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf). 
 You can now do
```python
from nlp import load_dataset
cos_e = load_dataset("cos_e", "v1.0")
```
Thanks @mariamabarham !
[Feature request] Add cos-e v1.0 
 I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf). 
 @mariamabarham Just wanted to note that default behavior `cos_e = load_dataset("cos_e")` now loads `v1.0`. Not sure if this is intentional (but the flag specification does work as intended).
[Feature request] Add cos-e v1.0 
 I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf). 
 > @mariamabarham Just wanted to note that default behavior `cos_e = load_dataset("cos_e")` now loads `v1.0`. Not sure if this is intentional (but the flag specification does work as intended).

In the new version of `nlp`, if you try `cos_e = load_dataset("cos_e")` it throws this error:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['v1.0', 'v1.11']
Example of usage:
	`load_dataset('cos_e', 'v1.0')`
```
For datasets with at least two configurations, we now force the user to pick one (no default)
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 usually you can replace `download` in your dataset script with `download_and_prepare()` - could you share the code for your dataset here? :-)
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 I have an initial version here: https://github.com/EntilZha/nlp/tree/master/datasets/qanta Thats pretty close to what I'll do as a PR, but still want to do some more sanity checks/tests (just got tests passing).

I figured out how to get all tests passing by adding a download command and some finagling with the data zip https://github.com/EntilZha/nlp/blob/master/tests/utils.py#L127
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 I'm quite positive that you can just replace the `dl_manager.download()` statements here: https://github.com/EntilZha/nlp/blob/4d46443b65f1f756921db8275594e6af008a1de7/datasets/qanta/qanta.py#L194 with `dl_manager.download_and_extract()` even though you don't extract anything. I would prefer to avoid adding more functions to the MockDataLoadManager and keep it as simple as possible (It's already to complex now IMO). 

Could you check if you can replace the `download()` function?
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 I might be doing something wrong, but swapping those two gives this error:
```
>       with open(path) as f:
E       IsADirectoryError: [Errno 21] Is a directory: 'datasets/qanta/dummy/mode=first,char_skip=25/2018.4.18/dummy_data-zip-extracted/dummy_data'

src/nlp/datasets/qanta/3d965403133687b819905ead4b69af7bcee365865279b2f797c79f809b4490c3/qanta.py:280: IsADirectoryError

During handling of the above exception, another exception occurred:
```

So it seems like the directory name is getting passed. Is this not functioning as expected, or is there some caching happening maybe? I deleted the dummy files and re-ran the import script with no changes. I'm digging a bit in with a debugger, but no clear reason yet
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 From what I can tell here: https://github.com/huggingface/nlp/blob/master/tests/utils.py#L115

1. `data_url` is the correct http link
2. `path_to_dummy_data` is a directory, which is causing the issue

That path comes from `download_dummy_data`, which I think assumes that the data comes from the zip file, but isn't aware of individual files. So it seems like it data manager needs to be aware if the url its getting is for a file or a zip/directory, and pass this information along. This might happen in `download_dummy_data`, but probably better to happen in `download_and_extract`? Maybe a simple check to see if `os.path.basename` returns the dummy data zip filename, if not then join paths with the basename of the url?
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 I think the dataset script works correctly. Just the dummy data structure seems to be wrong. I will soon add more commands that should make the create of the dummy data easier.

I'd recommend that you won't concentrate too much on the dummy data.
If you manage to load the dataset correctly via:

```python 
# use local path to qanta
nlp.load_dataset("./datasets/qanta")
```

then feel free to open a PR and we will look into the dummy data problem together :-) 

Also please make sure that the Version is in the format 1.0.0 (three numbers separated by two points) - not a date.
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 The script loading seems to work fine so I'll work on getting a PR open after a few sanity checks on the data.

On version, we currently have it versioned with YYYY.MM.DD scheme so it would be nice to not change that, but will it cause issues?
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 > The script loading seems to work fine so I'll work on getting a PR open after a few sanity checks on the data.
> 
> On version, we currently have it versioned with YYYY.MM.DD scheme so it would be nice to not change that, but will it cause issues?

It would cause issues for sure for the tests....not sure if it would also cause issues otherwise.

I would prefer to keep the same version style as we have for other models. You could for example simply add version 1.0.0 and add a comment with the date you currently use for the versioning.

 What is your opinion regarding the version here @lhoestq @mariamabarham @thomwolf ?
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 Maybe use the YYYY.MM.DD as the config name ? That's what we are doing for wikipedia
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 > Maybe use the YYYY.MM.DD as the config name ? That's what we are doing for wikipedia

I'm not sure if this will work because the name should be unique and it seems that he has multiple config name in his data with the same version.
As @patrickvonplaten  suggested, I think  you can add a comment about the version in the data description.
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 Actually maybe our versioning format (inherited from tfds) is too strong for what we use it for?
We could allow any string maybe?

I see it more and more like an identifier for the user that we will back with a serious hashing/versioning system.- so we could let the user quite free on it.
Discussion on version identifier & MockDataLoaderManager for test data 
 Hi, I'm working on adding a dataset and ran into an error due to `download` not being defined on `MockDataLoaderManager`, but being defined in `nlp/utils/download_manager.py`. The readme step running this: `RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_localmydatasetname` triggers the error. If I can get something to work, I can include it in my data PR once I'm done. 
 I'm good with either putting it in description, adding it to the config, or loosening version formatting. I mostly don't have a full conceptual grasp of what each identifier ends up meaning in the datasets code so hard to evaluate the best approach.

For background, the multiple formats is a consequence of:

1. Each example is one multi-sentence trivia question
2. For training, its better to treat each sentence as an example
3. For evaluation, should test on: (1) first sentence, (2) full question, and (3) partial questions (does the model get the question right having seen the first half)

We use the date format for version since: (1) we expect some degree of updates since new questions come in every year and (2) the timestamp itself matches the Wikipedia dump that it is dependent on (so similar to the Wikipedia dataset).

perhaps this is better discussed in https://github.com/huggingface/nlp/pull/169 or update title?
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)" 
 I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a 
 You can just run: 
`val = nlp.load_dataset('squad')` 

if you want to have just the validation script you can also do:

`val = nlp.load_dataset('squad', split="validation")`
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)" 
 I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a 
 If you want to load a local dataset, make sure you include a `./` before the folder name.
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)" 
 I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a 
 This happens by just doing run all cells on colab ... I assumed the colab example is broken.
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)" 
 I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a 
 Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
```
!pip uninstall -y -qq pyarrow
!pip uninstall -y -qq nlp
!pip install -qq git+https://github.com/huggingface/nlp.git
```
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)" 
 I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a 
 > Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
> 
> ```
> !pip uninstall -y -qq pyarrow
> !pip uninstall -y -qq nlp
> !pip install -qq git+https://github.com/huggingface/nlp.git
> ```

Tried, having the same error.
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)" 
 I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a 
 Can you post a link here of your colab? I'll make a copy of it and see what's wrong
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)" 
 I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a 
 This should be fixed in the current version of the notebook. You can try it again
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)" 
 I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a 
 I am getting this error when running this command
```
val = nlp.load_dataset('squad', split="validation")
```

FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/dataset_info.json'

Can anybody help?
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)" 
 I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a 
 It seems like your download was corrupted :-/ Can you run the following command: 

```
rm -r /root/.cache/huggingface/datasets
```

to delete the cache completely and rerun the download?
SyntaxError with WMT datasets 
 The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook 
 Jeez - don't know what happened there :D Should be fixed now! 

Thanks a lot for reporting this @tomhosking !
SyntaxError with WMT datasets 
 The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook 
 Hi @patrickvonplaten!

I'm now getting the below error:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-3206959998b9> in <module>
      1 import nlp
      2 
----> 3 dataset = nlp.load_dataset('wmt14')
      4 print(dataset['train'][0])

~/.local/lib/python3.6/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    507     # Instantiate the dataset builder
    508     builder_instance = builder_cls(
--> 509         cache_dir=cache_dir, name=name, version=version, data_dir=data_dir, data_files=data_files, **config_kwargs,
    510     )
    511 

TypeError: Can't instantiate abstract class Wmt with abstract methods _subsets
```
SyntaxError with WMT datasets 
 The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook 
 To correct this error I think you need the master branch of `nlp`. Can you try to install `nlp` with. `WMT` was not included at the beta release of the library. 

Can you try:
`pip install git+https://github.com/huggingface/nlp.git`

and check again?
SyntaxError with WMT datasets 
 The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook 
 That works, thanks :)

The WMT datasets are listed in by `list_datasets()` in the beta release on pypi - it would be good to only show datasets that are actually supported by that version?
SyntaxError with WMT datasets 
 The following snippet produces a syntax error:

```
import nlp

dataset = nlp.load_dataset('wmt14')
print(dataset['train'][0])
```

```
Traceback (most recent call last):

  File "/home/tom/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-8-3206959998b9>", line 3, in <module>
    dataset = nlp.load_dataset('wmt14')

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 505, in load_dataset
    builder_cls = import_main_class(module_path, dataset=True)

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/load.py", line 56, in import_main_class
    module = importlib.import_module(module_path)

  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 994, in _gcd_import

  File "<frozen importlib._bootstrap>", line 971, in _find_and_load

  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt14.py", line 21, in <module>
    from .wmt_utils import Wmt, WmtConfig

  File "/home/tom/.local/lib/python3.6/site-packages/nlp/datasets/wmt14/c258d646f4f5870b0245f783b7aa0af85c7117e06aacf1e0340bd81935094de2/wmt_utils.py", line 659
    <<<<<<< HEAD
     ^
SyntaxError: invalid syntax
```

Python version:
`3.6.9 (default, Apr 18 2020, 01:56:04)  [GCC 8.4.0]`
Running on Ubuntu 18.04, via a Jupyter notebook 
 Usually, the idea is that a dataset can be added without releasing a new version. The problem in the case of `WMT` was that some "core" code of the library had to be changed as well. 

@thomwolf @lhoestq @julien-c - How should we go about this. If we add a dataset that also requires "core" code changes, how do we handle the versioning? The moment a dataset is on AWS it will actually be listed with `list_datasets()` in all earlier versions...

Is there a way to somehow insert the `pip version` to the HfApi() and get only the datasets that were available for this version (at the date of the release of the version) @julien-c ?