ahmedfarazsyk/datasets-github-issues huggingface dataset:
[Feature request] separate split name and split instructions
Author: yjernite
State: closed

Currently, the name of an nlp.NamedSplit is parsed in arrow_reader.py and used as the instruction.

This makes it impossible to have several training sets, which can occur when:
- A dataset corresponds to a collection of sub-datasets
- A dataset was built in stages, adding new examples at each stage

Would it be possible to have two separate fields in the Split class, a name /instruction and a unique ID that is used as the key in the builder's split_dict ?

Comments: If your dataset is a collection of sub-datasets, you should probably consider having one config per sub-dataset. For example for Glue, we have sst2, mnli etc.
If you want to have multiple train sets (for example one per stage). The easiest solution would be to name them `nlp.Split("train_stage1")`, `nlp.Split("train_stage2")`, etc. or something like that.

Thanks for the tip! I ended up setting up three different versions of the dataset with their own configs.

for the named splits, I was trying with `nlp.Split("train-stage1")`, which fails. Changing to `nlp.Split("train_stage1")` works :) I looked for examples of what works in the code comments, it may be worth adding some examples of valid/invalid names in there?
[Manual data dir] Error message: nlp.load_dataset('xsum') -> TypeError
Author: sshleifer
State: closed

v 0.1.0 from pip

```python
import nlp
xsum = nlp.load_dataset('xsum')
```

Issue is `dl_manager.manual_dir`is `None`

```python

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-42-8a32f066f3bd> in <module>
----> 1 xsum = nlp.load_dataset('xsum')

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519 

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    397         split_dict = SplitDict(dataset_name=self.name)
    398         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 399         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    400         # Checksums verification
    401         if verify_infos:

~/miniconda3/envs/nb/lib/python3.7/site-packages/nlp/datasets/xsum/5c5fca23aaaa469b7a1c6f095cf12f90d7ab99bcc0d86f689a74fd62634a1472/xsum.py in _split_generators(self, dl_manager)
    102         with open(dl_path, "r") as json_file:
    103             split_ids = json.load(json_file)
--> 104         downloaded_path = os.path.join(dl_manager.manual_dir, "xsum-extracts-from-downloads")
    105         return [
    106             nlp.SplitGenerator(

~/miniconda3/envs/nb/lib/python3.7/posixpath.py in join(a, *p)
     78     will be discarded.  An empty last part will result in a path that
     79     ends with a separator."""
---> 80     a = os.fspath(a)
     81     sep = _get_sep(a)
     82     path = a

TypeError: expected str, bytes or os.PathLike object, not NoneType


```
Clone not working on Windows environment
Author: codehunk628
State: closed

Cloning in a windows environment is not working because of use of special character '?' in folder name ..
Please consider changing the folder name ....
Reference to folder -
nlp/datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs/dailymail/stories/

error log:
fatal: cannot create directory at 'datasets/cnn_dailymail/dummy/3.0.0/3.0.0/dummy_data-zip-extracted/dummy_data/uc?export=download&id=0BwmD_VLjROrfM1BxdkxVaTY2bWs': Invalid argument



Comments: Should be fixed on master now :)

Thanks @lhoestq 👍  Now I can uninstall WSL and get back to work with windows.🙂
Add a method to shuffle a dataset
Author: thomwolf
State: closed
Labels: name: generic discussion

Could maybe be a `dataset.shuffle(generator=None, seed=None)` signature method.

Also, we could maybe have a clear indication of which method modify in-place and which methods return/cache a modified dataset. I kinda like torch conversion of having an underscore suffix for all the methods which modify a dataset in-place. What do you think?

Comments: +1 for the naming convention

About the `shuffle` method, from my understanding it should be done in `Dataloader` (better separation between dataset processing - usage)

+1 for shuffle in `Dataloader`. 
Some `Dataloader` just store idxs of dataset and just shuffle those idxs, which might(?) be faster than do shuffle in dataset, especially when doing shuffle every epoch.

Also +1 for the naming convention.

As you might already know the issue of dataset shuffling came up in the nlp code [walkthrough](https://youtu.be/G3pOvrKkFuk?t=3204) by Yannic Kilcher


We added the `.shuffle` method :)

Closing this one.
ANLI
Author: douwekiela
State: closed

Can I recommend the following:

For ANLI, use https://github.com/facebookresearch/anli. As that paper says, "Our dataset is not
to be confused with abductive NLI (Bhagavatula et al., 2019), which calls itself αNLI, or ART.". 

Indeed, the paper cited under what is currently called anli says in the abstract "We introduce a challenge dataset, ART".

The current naming will confuse people :)
Add Spanish POR and NER Datasets
Author: mrm8488
State: closed
Labels: name: dataset request

Hi guys,
In order to cover multilingual support a little step could be adding standard Datasets used for Spanish NER and POS tasks.
I can provide it in raw and preprocessed formats.

Comments: Hello @mrm8488, are these datasets official datasets published in an NLP/CL/ML venue?

What about this one: https://github.com/ccasimiro88/TranslateAlignRetrieve?
[Feature request] Add cos-e v1.0
Author: sarahwie
State: closed
Labels: name: dataset request

I noticed the second release of cos-e (v1.11) is included in this repo. I wanted to request inclusion of v1.0, since this is the version on which results are reported on in [the paper](https://www.aclweb.org/anthology/P19-1487/), and v1.11 has noted [annotation](https://github.com/salesforce/cos-e/issues/2) [issues](https://arxiv.org/pdf/2004.14546.pdf).

Comments: Sounds good, @mariamabarham do you want to give a look?
I think we should have two configurations so we can allow either version of the dataset to be loaded with the `1.0` version being the default maybe.

Cc some authors of the great cos-e: @nazneenrajani @bmccann

cos_e v1.0 is related to CQA v1.0 but only CQA v1.11 dataset is available on their website. Indeed their is lots of ids in cos_e v1, which are  not in CQA v1.11  or the other way around.
@sarahwie, @thomwolf, @nazneenrajani,  @bmccann  do you know where I can  find CQA v1.0


@mariamabarham I'm also not sure where to find CQA 1.0. Perhaps it's not possible to include this version of the dataset. I'll close the issue if that's the case.

I do have a copy of the dataset. I can upload it to our repo.

Great @nazneenrajani. let me know once done.
Thanks

@mariamabarham @sarahwie I added them to the cos-e repo https://github.com/salesforce/cos-e/tree/master/data/v1.0

You can now do
```python
from nlp import load_dataset
cos_e = load_dataset("cos_e", "v1.0")
```
Thanks @mariamabarham !

Thanks!

@mariamabarham Just wanted to note that default behavior `cos_e = load_dataset("cos_e")` now loads `v1.0`. Not sure if this is intentional (but the flag specification does work as intended). 

> @mariamabarham Just wanted to note that default behavior `cos_e = load_dataset("cos_e")` now loads `v1.0`. Not sure if this is intentional (but the flag specification does work as intended).

In the new version of `nlp`, if you try `cos_e = load_dataset("cos_e")` it throws this error:
```
ValueError: Config name is missing.
Please pick one among the available configs: ['v1.0', 'v1.11']
Example of usage:
	`load_dataset('cos_e', 'v1.0')`
```
For datasets with at least two configurations, we now force the user to pick one (no default)
nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"
Author: saahiluppal
State: closed

I'm trying to load datasets from nlp but there seems to have error saying 
"TypeError: list_() takes exactly one argument (2 given)"

gist can be found here
https://gist.github.com/saahiluppal/c4b878f330b10b9ab9762bc0776c0a6a

Comments: You can just run: 
`val = nlp.load_dataset('squad')` 

if you want to have just the validation script you can also do:

`val = nlp.load_dataset('squad', split="validation")`

If you want to load a local dataset, make sure you include a `./` before the folder name. 

This happens by just doing run all cells on colab ... I assumed the colab example is broken. 

Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
```
!pip uninstall -y -qq pyarrow
!pip uninstall -y -qq nlp
!pip install -qq git+https://github.com/huggingface/nlp.git
```

> Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
> 
> ```
> !pip uninstall -y -qq pyarrow
> !pip uninstall -y -qq nlp
> !pip install -qq git+https://github.com/huggingface/nlp.git
> ```

Tried, having the same error.

Can you post a link here of your colab? I'll make a copy of it and see what's wrong

This should be fixed in the current version of the notebook. You can try it again

Also see: https://github.com/huggingface/nlp/issues/222

I am getting this error when running this command
```
val = nlp.load_dataset('squad', split="validation")
```

FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/dataset_info.json'

Can anybody help?

It seems like your download was corrupted :-/ Can you run the following command: 

```
rm -r /root/.cache/huggingface/datasets
```

to delete the cache completely and rerun the download? 

I tried the notebook again today and it worked without barfing. 👌
Meta-datasets (GLUE/XTREME/...) – Special care to attributions and citations
Author: thomwolf
State: open
Labels: name: generic discussion

Meta-datasets are interesting in terms of standardized benchmarks but they also have specific behaviors, in particular in terms of attribution and authorship. It's very important that each specific dataset inside a meta dataset is properly referenced and the citation/specific homepage/etc are very visible and accessible and not only the generic citation of the meta-dataset itself.

Let's take GLUE as an example:

The configuration has the citation for each dataset included (e.g. [here](https://github.com/huggingface/nlp/blob/master/datasets/glue/glue.py#L154-L161)) but it should be copied inside the dataset info so that, when people access `dataset.info.citation` they get both the citation for GLUE and the citation for the specific datasets inside GLUE that they have loaded.

Comments: As @yoavgo suggested, there should be the possibility to call a function like  nlp.bib that outputs all bibtex ref from the datasets and models actually used and eventually nlp.bib.forreadme that would output the same info + versions numbers so they can be included in a readme.md file.

Actually, double checking with @mariamabarham, we already have this feature I think.

It's like this currently:
```python
>>> from nlp import load_dataset
>>> 
>>> dataset = load_dataset('glue', 'cola', split='train')
>>> print(dataset.info.citation)
@article{warstadt2018neural,
  title={Neural Network Acceptability Judgments},
  author={Warstadt, Alex and Singh, Amanpreet and Bowman, Samuel R},
  journal={arXiv preprint arXiv:1805.12471},
  year={2018}
}
@inproceedings{wang2019glue,
  title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding},
  author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},
  note={In the Proceedings of ICLR.},
  year={2019}
}

Note that each GLUE dataset has its own citation. Please see the source to see
the correct citation for each contained dataset.
```

What do you think @dseddah?

Looks good but why would there be a difference between the ref in the source and the one to be printed? 

Yes, I think we should remove this warning @mariamabarham.

It's probably a relic of tfds which didn't have the same way to access citations.
[Feature request] Add Ubuntu Dialogue Corpus dataset
Author: danth
State: closed
Labels: name: dataset request

https://github.com/rkadlec/ubuntu-ranking-dataset-creator or http://dataset.cs.mcgill.ca/ubuntu-corpus-1.0/

Comments: @AlphaMycelium the Ubuntu Dialogue Corpus [version 2]( https://github.com/rkadlec/ubuntu-ranking-dataset-creator) is added. Note that it requires a manual download by following the download instructions in the [repos]( https://github.com/rkadlec/ubuntu-ranking-dataset-creator).
Maybe we can close this issue for now?
_download_and_prepare() got an unexpected keyword argument 'verify_infos'
Author: richarddwang
State: closed
Labels: name: dataset bug

# Reproduce
In Colab,
```
%pip install -q  nlp
%pip install -q apache_beam mwparserfromhell

dataset = nlp.load_dataset('wikipedia')
```
get
```
Downloading and preparing dataset wikipedia/20200501.aa (download: Unknown size, generated: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/wikipedia/20200501.aa/1.0.0...

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-6-52471d2a0088> in <module>()
----> 1 dataset = nlp.load_dataset('wikipedia')

1 frames

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519 

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

TypeError: _download_and_prepare() got an unexpected keyword argument 'verify_infos'
```

Comments: Same error for dataset 'wiki40b'

Should be fixed on master :)
Error with sklearn train_test_split
Author: ClonedOne
State: closed
Labels: name: enhancement

It would be nice if we could use sklearn `train_test_split` to quickly generate subsets from the dataset objects returned by `nlp.load_dataset`. At the moment the code:

```python
data = nlp.load_dataset('imdb', cache_dir=data_cache)
f_half, s_half = train_test_split(data['train'], test_size=0.5, random_state=seed)
```
throws:
```
ValueError: Can only get row(s) (int or slice) or columns (string).
```
It's not a big deal, since there are other ways to split the data, but it would be a cool thing to have.

Comments: Indeed. Probably we will want to have a similar method directly in the library

Related: #166
ArrowTypeError in squad metrics
Author: patil-suraj
State: closed
Labels: name: metric bug

`squad_metric.compute` is giving following error
```
ArrowTypeError: Could not convert [{'text': 'Denver Broncos'}, {'text': 'Denver Broncos'}, {'text': 'Denver Broncos'}] with type list: was not a dict, tuple, or recognized null value for conversion to struct type
```

This is how my predictions and references look like
```
predictions[0]
# {'id': '56be4db0acb8001400a502ec', 'prediction_text': 'Denver Broncos'}
```

```
references[0]
# {'answers': [{'text': 'Denver Broncos'},
  {'text': 'Denver Broncos'},
  {'text': 'Denver Broncos'}],
 'id': '56be4db0acb8001400a502ec'}
```

These are structured as per the `squad_metric.compute` help string.

Comments: There was an issue in the format, thanks.
Now you can do
```python3
squad_dset = nlp.load_dataset("squad")
squad_metric = nlp.load_metric("/Users/quentinlhoest/Desktop/hf/nlp-bis/metrics/squad")
predictions = [
    {"id": v["id"], "prediction_text": v["answers"]["text"][0]}  # take first possible answer
    for v in squad_dset["validation"]
]
squad_metric.compute(predictions, squad_dset["validation"])
```

and the expected format is 
```
Args:
    predictions: List of question-answers dictionaries with the following key-values:
        - 'id': id of the question-answer pair as given in the references (see below)
        - 'prediction_text': the text of the answer
    references: List of question-answers dictionaries with the following key-values:
        - 'id': id of the question-answer pair (see above),
        - 'answers': a Dict {'text': list of possible texts for the answer, as a list of strings}
```
[Question] Using/adding a local dataset
Author: zphang
State: closed

Users may want to either create/modify a local copy of a dataset, or use a custom-built dataset with the same `Dataset` API as externally downloaded datasets.

It appears to be possible to point to a local dataset path rather than downloading the external ones, but I'm not exactly sure how to go about doing this.

A notebook/example script demonstrating this would be very helpful.

Comments: Hi @zphang,

So you can just give the local path to a dataset script file and it should work.

Here is an example:
- you can download one of the scripts in the `datasets` folder of the present repo (or clone the repo)
- then you can load it with `load_dataset('PATH/TO/YOUR/LOCAL/SCRIPT.py')`

Does it make sense?

Could you give a more concrete example, please? 

I looked up wikitext dataset script from the repo. Should I just overwrite the `data_file` on line 98 to point to the local dataset directory? Would it work for different configurations of wikitext (wikitext2, wikitext103 etc.)?

Or maybe we can use DownloadManager to specify local dataset location? In that case, where do we use DownloadManager instance?

Thanks

Hi @MaveriQ , although what I am doing is to commit a new dataset, but I think looking at imdb script might help.
You may want to use `dl_manager.download_custom`, give it a url(arbitrary string), a custom_download(arbitrary function) and return a path, and finally use _get sample to fetch a sample.

The download manager supports local directories. You can specify a local directory instead of a url and it should work.

Closing this one.
Feel free to re-open if you have other questions :)
[Feature Request] Add the OpenWebText dataset
Author: LysandreJik
State: closed
Labels: name: dataset request

The OpenWebText dataset is an open clone of OpenAI's WebText dataset. It can be used to train ELECTRA as is specified in the [README](https://www.github.com/google-research/electra).

More information and the download link are available [here](https://skylion007.github.io/OpenWebTextCorpus/).

Comments: We're experimenting with hosting the OpenWebText corpus on Zenodo for easier downloading. https://zenodo.org/record/3834942#.Xs1w8i-z2J8

Closing since it's been added in #660
[Feature request] Add Toronto BookCorpus dataset
Author: jarednielsen
State: closed
Labels: name: dataset request

I know the copyright/distribution of this one is complex, but it would be great to have! That, combined with the existing `wikitext`, would provide a complete dataset for pretraining models like BERT.

Comments: As far as I understand, `wikitext` is refer to `WikiText-103` and `WikiText-2` that created by researchers in Salesforce, and mostly used in traditional language modeling.

You might want to say `wikipedia`, a dump from wikimedia foundation.

Also I would like to have Toronto BookCorpus too ! Though it involves copyright problem...

Hi, @lhoestq, just a reminder that this is solved by #248 .😉
Loading GLUE dataset loads CoLA by default
Author: zphang
State: closed
Labels: name: dataset bug

If I run:

```python
dataset = nlp.load_dataset('glue')
```
The resultant dataset seems to be CoLA be default, without throwing any error. This is in contrast to calling:

```python
metric = nlp.load_metric("glue")
```
which throws an error telling the user that they need to specify a task in GLUE. Should the same apply for loading datasets?

Comments: As a follow-up to this: It looks like the actual GLUE task name is supplied as the `name` argument. Is there a way to check what `name`s/sub-datasets are available under a grouping like GLUE? That information doesn't seem to be readily available in info from `nlp.list_datasets()`.

Edit: I found the info under `Glue.BUILDER_CONFIGS`

Yes so the first config is loaded by default when no `name` is supplied but for GLUE this should probably throw an error indeed.

We can probably just add an `__init__` at the top of the `class Glue(nlp.GeneratorBasedBuilder)` in the `glue.py` script which does this check:
```
class Glue(nlp.GeneratorBasedBuilder):
    def __init__(self, *args, **kwargs):
        assert 'name' in kwargs and kwargs[name] is not None, "Glue has to be called with a configuration name"
        super(Glue, self).__init__(*args, **kwargs)
```

An error is raised if the sub-dataset is not specified :)
```
ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
	`load_dataset('glue', 'cola')`
```
[Feature request] Add Google Natural Question dataset
Author: elyase
State: closed
Labels: name: dataset request

Would be great to have https://github.com/google-research-datasets/natural-questions as an alternative to SQuAD.

Comments: Indeed, I think this one is almost ready cc @lhoestq 

I'm doing the latest adjustments to make the processing of the dataset run on Dataflow

Is there an update to this? It will be very beneficial for the QA community!

Still work in progress :)
The idea is to have the dataset already processed somewhere so that the user only have to download the processed files. I'm also doing it for wikipedia.

Super appreciate your hard work !!
I'll cross my fingers and hope easily loadable wikipedia dataset will come soon. 

Quick update on NQ: due to some limitations I met using apache beam + parquet I was not able to use the dataset in a nested parquet structure in python to convert it to our Apache Arrow format yet.
However we had planned to change this conversion step anyways so we'll make just sure that it enables to process and convert the NQ dataset to arrow.

NQ was added in #427 🎉