SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Discussion on version identifier & MockDataLoaderManager for test data

Comment:
I'm good with either putting it in description, adding it to the config, or loosening version formatting. I mostly don't have a full conceptual grasp of what each identifier ends up meaning in the datasets code so hard to evaluate the best approach.

For background, the multiple formats is a consequence of:

1. Each example is one multi-sentence trivia question
2. For training, its better to treat each sentence as an example
3. For evaluation, should test on: (1) first sentence, (2) full question, and (3) partial questions (does the model get the question right having seen the first half)

We use the date format for version since: (1) we expect some degree of updates since new questions come in every year and (2) the timestamp itself matches the Wikipedia dump that it is dependent on (so similar to the Wikipedia dataset).

perhaps this is better discussed in https://github.com/huggingface/nlp/pull/169 or update title?
Issue: caching in map causes same result to be returned for train, validation and test

Comment:
Hi @dpressel, 

thanks for posting your issue! Can you maybe add a complete code snippet that we can copy paste to reproduce the error? For example, I'm not sure where the variable `train_set` comes from in your code and it seems like you are loading multiple datasets at once?
Issue: caching in map causes same result to be returned for train, validation and test

Comment:
Hi, the full example was listed in the PR above, but here is the exact link:

https://github.com/dpressel/mead-baseline/blob/3c1aa3ca062cb23f303ca98ac40b6652b37ee971/api-examples/layers-classify-hf-datasets.py

The problem is coming from
```
            if cache_file_name is None:
                # we create a unique hash from the function, current dataset file and the mapping args
                cache_kwargs = {
                    "with_indices": with_indices,
                    "batched": batched,
                    "batch_size": batch_size,
                    "remove_columns": remove_columns,
                    "keep_in_memory": keep_in_memory,
                    "load_from_cache_file": load_from_cache_file,
                    "cache_file_name": cache_file_name,
                    "writer_batch_size": writer_batch_size,
                    "arrow_schema": arrow_schema,
                    "disable_nullable": disable_nullable,
                }
                cache_file_name = self._get_cache_file_path(function, cache_kwargs)
```
The cached value is always the same, but I was able to change that by just renaming the function each time which seems to fix the issue.
Issue: caching in map causes same result to be returned for train, validation and test

Comment:
Ok, I think @lhoestq has already found a solution :-) Maybe you can chime in @lhoestq
Issue: caching in map causes same result to be returned for train, validation and test

Comment:
> Ok, I think @lhoestq has already found a solution :-) Maybe you can chime in @lhoestq

Oh, awesome!  I see the PR, Ill check it out
Issue: caching in map causes same result to be returned for train, validation and test

Comment:
The PR should prevent the cache from losing track of the of the dataset type (based on the location of its data). Not sure about your second problem though (cache off).
Issue: caching in map causes same result to be returned for train, validation and test

Comment:
Yes, with caching on, it seems to work without the function renaming hack, I mentioned this also in the PR. Thanks!
Issue: nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

Comment:
You can just run: 
`val = nlp.load_dataset('squad')` 

if you want to have just the validation script you can also do:

`val = nlp.load_dataset('squad', split="validation")`
Issue: nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

Comment:
If you want to load a local dataset, make sure you include a `./` before the folder name.
Issue: nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

Comment:
This happens by just doing run all cells on colab ... I assumed the colab example is broken.
Issue: nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

Comment:
Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
```
!pip uninstall -y -qq pyarrow
!pip uninstall -y -qq nlp
!pip install -qq git+https://github.com/huggingface/nlp.git
```
Issue: nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

Comment:
> Oh I see you might have a wrong version of pyarrow install on the colab -> could you try the following. Add these lines to the beginning of your notebook, restart the runtime and run it again:
> 
> ```
> !pip uninstall -y -qq pyarrow
> !pip uninstall -y -qq nlp
> !pip install -qq git+https://github.com/huggingface/nlp.git
> ```

Tried, having the same error.
Issue: nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

Comment:
Can you post a link here of your colab? I'll make a copy of it and see what's wrong
Issue: nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

Comment:
This should be fixed in the current version of the notebook. You can try it again
Issue: nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

Comment:
I am getting this error when running this command
```
val = nlp.load_dataset('squad', split="validation")
```

FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/dataset_info.json'

Can anybody help?
Issue: nlp.load_dataset() gives "TypeError: list_() takes exactly one argument (2 given)"

Comment:
It seems like your download was corrupted :-/ Can you run the following command: 

```
rm -r /root/.cache/huggingface/datasets
```

to delete the cache completely and rerun the download?
Issue: SyntaxError with WMT datasets

Comment:
Jeez - don't know what happened there :D Should be fixed now! 

Thanks a lot for reporting this @tomhosking !
Issue: SyntaxError with WMT datasets

Comment:
Hi @patrickvonplaten!

I'm now getting the below error:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-3206959998b9> in <module>
      1 import nlp
      2 
----> 3 dataset = nlp.load_dataset('wmt14')
      4 print(dataset['train'][0])

~/.local/lib/python3.6/site-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    507     # Instantiate the dataset builder
    508     builder_instance = builder_cls(
--> 509         cache_dir=cache_dir, name=name, version=version, data_dir=data_dir, data_files=data_files, **config_kwargs,
    510     )
    511 

TypeError: Can't instantiate abstract class Wmt with abstract methods _subsets
```
Issue: SyntaxError with WMT datasets

Comment:
To correct this error I think you need the master branch of `nlp`. Can you try to install `nlp` with. `WMT` was not included at the beta release of the library. 

Can you try:
`pip install git+https://github.com/huggingface/nlp.git`

and check again?
Issue: SyntaxError with WMT datasets

Comment:
That works, thanks :)

The WMT datasets are listed in by `list_datasets()` in the beta release on pypi - it would be good to only show datasets that are actually supported by that version?
Issue: SyntaxError with WMT datasets

Comment:
Usually, the idea is that a dataset can be added without releasing a new version. The problem in the case of `WMT` was that some "core" code of the library had to be changed as well. 

@thomwolf @lhoestq @julien-c - How should we go about this. If we add a dataset that also requires "core" code changes, how do we handle the versioning? The moment a dataset is on AWS it will actually be listed with `list_datasets()` in all earlier versions...

Is there a way to somehow insert the `pip version` to the HfApi() and get only the datasets that were available for this version (at the date of the release of the version) @julien-c ?
Issue: SyntaxError with WMT datasets

Comment:
We plan to have something like a `requirements.txt` per dataset to prevent user from loading dataset with old version of `nlp` or any other libraries. Right now the solution is just to keep `nlp` up to date when you want to load a dataset that leverages the latests features of `nlp`.

For datasets that are on AWS but that use features that are not released yet we should be able to filter those from the `list_dataset` as soon as we have the `requirements.txt` feature on (filter datasets that need a future version of `nlp`).

Shall we rename this issue to be more explicit about the problem ?
Something like `Specify the minimum version of the nlp library required for each dataset` ?
Issue: Meta-datasets (GLUE/XTREME/...) – Special care to attributions and citations

Comment:
As @yoavgo suggested, there should be the possibility to call a function like  nlp.bib that outputs all bibtex ref from the datasets and models actually used and eventually nlp.bib.forreadme that would output the same info + versions numbers so they can be included in a readme.md file.
Issue: Meta-datasets (GLUE/XTREME/...) – Special care to attributions and citations

Comment:
Actually, double checking with @mariamabarham, we already have this feature I think.

It's like this currently:
```python
>>> from nlp import load_dataset
>>> 
>>> dataset = load_dataset('glue', 'cola', split='train')
>>> print(dataset.info.citation)
@article{warstadt2018neural,
  title={Neural Network Acceptability Judgments},
  author={Warstadt, Alex and Singh, Amanpreet and Bowman, Samuel R},
  journal={arXiv preprint arXiv:1805.12471},
  year={2018}
}
@inproceedings{wang2019glue,
  title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding},
  author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.},
  note={In the Proceedings of ICLR.},
  year={2019}
}

Note that each GLUE dataset has its own citation. Please see the source to see
the correct citation for each contained dataset.
```

What do you think @dseddah?
Issue: Meta-datasets (GLUE/XTREME/...) – Special care to attributions and citations

Comment:
Looks good but why would there be a difference between the ref in the source and the one to be printed?
Issue: Meta-datasets (GLUE/XTREME/...) – Special care to attributions and citations

Comment:
Yes, I think we should remove this warning @mariamabarham.

It's probably a relic of tfds which didn't have the same way to access citations.
Issue: [Feature request] Add Ubuntu Dialogue Corpus dataset

Comment:
@AlphaMycelium the Ubuntu Dialogue Corpus [version 2]( https://github.com/rkadlec/ubuntu-ranking-dataset-creator) is added. Note that it requires a manual download by following the download instructions in the [repos]( https://github.com/rkadlec/ubuntu-ranking-dataset-creator).
Maybe we can close this issue for now?
Issue: ArrowTypeError in squad metrics

Comment:
There was an issue in the format, thanks.
Now you can do
```python3
squad_dset = nlp.load_dataset("squad")
squad_metric = nlp.load_metric("/Users/quentinlhoest/Desktop/hf/nlp-bis/metrics/squad")
predictions = [
    {"id": v["id"], "prediction_text": v["answers"]["text"][0]}  # take first possible answer
    for v in squad_dset["validation"]
]
squad_metric.compute(predictions, squad_dset["validation"])
```

and the expected format is 
```
Args:
    predictions: List of question-answers dictionaries with the following key-values:
        - 'id': id of the question-answer pair as given in the references (see below)
        - 'prediction_text': the text of the answer
    references: List of question-answers dictionaries with the following key-values:
        - 'id': id of the question-answer pair (see above),
        - 'answers': a Dict {'text': list of possible texts for the answer, as a list of strings}
```
Issue: Consider renaming to nld

Comment:
I would suggest `nlds`. NLP is a very general, broad and ambiguous term, the library is not about NLP (as in processing) per se, it is about accessing Natural Language related datasets. So the name should reflect its purpose.
Issue: Consider renaming to nld

Comment:
Chiming in to second everything @honnibal said, and to add that I think the current name is going to impact the discoverability of this library. People who are looking for "NLP Datasets" through a search engine are going to see a library called `nlp` and think it's too broad. People who are looking to do NLP in python are going to search "Python NLP" and end up here, confused that this is a collection of datasets.

The names of the other huggingface libraries work because they're the only game in town: there are not very many robust, distinct libraries for `tokenizers` or `transformers` in python, for example. But there are several options for NLP in python, and adding this as a possible search result for "python nlp" when datasets are likely not what someone is searching for adds noise and frustrates potential users.
Issue: Consider renaming to nld

Comment:
I'm also not sure whether the naming of `nlp` is the problem itself, as long as it comes with the appropriate identifier, so maybe something like `huggingface_nlp`? This is analogous to what @honnibal and spacy are doing for `spacy-transformers`. Of course, this is a "step back" from the recent changes/renaming of transformers, but may be some middle ground between a complete rebranding, and keeping it identifiable.
Issue: Consider renaming to nld

Comment:
Interesting, thanks for sharing your thoughts.

As we’ll move toward a first non-beta release, we will pool the community of contributors/users of the library for their opinions on a good final name (like when we renamed the beautifully (?) named `pytorch-pretrained-bert`)

In the meantime, using `from nlp import load_dataset, load_metric` should work 😉
Issue: Consider renaming to nld

Comment:
I feel like we are conflating two distinct subjects here:

1. @honnibal's point is that using `nlp` as a package name might break existing code and bring developer usability issues in the future
2. @pmbaumgartner's point is that the `nlp` package name is too broad and shouldn't be used by a package that exposes only datasets and metrics

(let me know if I mischaracterize your point)

I'll chime in to say that the first point is a bit silly IMO. As Python developers due to the limitations of the import system we already have to share:
- a single flat namespace for packages
- which also conflicts with local modules i.e. local files

If we add the constraint that this flat namespace also be shared with variable names this gets untractable pretty fast :)

I also think all Python software developers/ML engineers/scientists are capable of at least a subset of:
- importing only the methods that they need like @thomwolf suggested
- aliasing their import
- renaming a local variable
Issue: Consider renaming to nld

Comment:
By the way, `nlp` will very likely not be only about datasets, and not even just about datasets and metrics.

I see it as a laboratory for testing several long-term ideas about how we could do NLP in terms of research as well as open-source and community sharing, most of these ideas being too experimental/big to fit in `transformers`.

Some of the directions we would like to explore are about sharing, traceability and more experimental models, as well as seeing a model as the community-based process of creating a composite entity from data, optimization, and code.

We'll see how these ideas end up being implemented and we'll better know how we should define the library when we start to dive into these topics. I'll try to get the `nlp` team to draft a roadmap on these topics at some point.
Issue: Consider renaming to nld

Comment:
> If we add the constraint that this flat namespace also be shared with variable names this gets untractable pretty fast :)

I'm sort of confused by your point here. The namespace *is* shared by variable names. You should not use local variables that are named the same as modules, because then you cannot use the module within the scope of your function.

For instance,

```python

import nlp
import transformers

nlp = transformers.pipeline("sentiment-analysis")
```

This is a bug: you've just overwritten the module, so now you can't use it. Or instead:

```python

import transformers

nlp = transformers.pipeline("sentiment-analysis")
# (Later, e.g. in a notebook)
import nlp
```

This is also a bug: you've overwritten your variable with an import.

If you have a module named `nlp`, you should avoid using `nlp` as a variable, or you'll have bugs in some contexts and inconsistencies in other contexts. You'll have situations where you need to import differently in one module vs another, or name variables differently in one context vs another, which is bad.

> importing only the methods that they need like @thomwolf suggested

Okay but the same logic applies to naming the module *literally anything else*. There's absolutely no point in having a module name that's 3 letters if you always plan to do `import from`! It would be entirely better to name it `nlp_datasets` if you don't want people to do `import nlp`.

And finally:

> By the way, nlp will very likely not be only about datasets, and not even just about datasets and metrics.

So...it isn't a datasets library? https://twitter.com/Thom_Wolf/status/1261282491622731781

I'm confused 😕
Issue: Consider renaming to nld

Comment:
Dropping by as I noticed that the library has been renamed `datasets` so I wonder if the conversation above is settled (`nlp` not used anymore) :)
Issue: Consider renaming to nld

Comment:
I'd argue that `datasets` is worse than `nlp`. Datasets should be a user specific decision and not encapsulate all of python (`pip install datasets`). If this package contained every dataset in the world (NLP / vision / etc) then it would make sense =/
Issue: Consider renaming to nld

Comment:
I can't speak for the HF team @jramapuram, but as member of the community it looks to me that HF wanted to avoid the past path of changing names as scope broadened over time:

Remember
https://github.com/huggingface/pytorch-openai-transformer-lm
https://github.com/huggingface/pytorch-pretrained-BERT
https://github.com/huggingface/pytorch-transformers
and now
https://github.com/huggingface/transformers

;) 

Jokes aside, seems that the library is growing in a multi-modal direction (https://github.com/huggingface/datasets/pull/363) so the current name is not that implausible. Possibly HF ambition is really to grow its community and bring here a large chunk of datasets of the world (including tabular / vision / audio?).
Issue: Consider renaming to nld

Comment:
Yea I see your point. However, wouldn't scoping solve the entire problem? 

```python
import huggingface.datasets as D
import huggingface.transformers as T
```

Calling something `datasets` is akin to saying I'm going to name my package `python` --> `import python`
Issue: Consider renaming to nld

Comment:
Sorry to reply to an old thread, but the name issue really makes troubles recently in my project.

I'd never known in advance there's a package called "datasets". My first thought is that such a general term may be safe to arbitrarily use. Avoiding such a common name because of its ambiguity is quite weird.

As we know in python it's not easy to differentiate system-wide and project-wide import like in C and C++.

On the contrary I fully understand the challenge to rename a popular library. So it seems to provide a "huggingface" wrapper library as suggested above by @jramapuram may be a happy medium for both developers and users.

Best Regards.
Issue: [Question] Using/adding a local dataset

Comment:
Hi @zphang,

So you can just give the local path to a dataset script file and it should work.

Here is an example:
- you can download one of the scripts in the `datasets` folder of the present repo (or clone the repo)
- then you can load it with `load_dataset('PATH/TO/YOUR/LOCAL/SCRIPT.py')`

Does it make sense?
Issue: [Question] Using/adding a local dataset

Comment:
Could you give a more concrete example, please? 

I looked up wikitext dataset script from the repo. Should I just overwrite the `data_file` on line 98 to point to the local dataset directory? Would it work for different configurations of wikitext (wikitext2, wikitext103 etc.)?

Or maybe we can use DownloadManager to specify local dataset location? In that case, where do we use DownloadManager instance?

Thanks
Issue: [Question] Using/adding a local dataset

Comment:
Hi @MaveriQ , although what I am doing is to commit a new dataset, but I think looking at imdb script might help.
You may want to use `dl_manager.download_custom`, give it a url(arbitrary string), a custom_download(arbitrary function) and return a path, and finally use _get sample to fetch a sample.
Issue: [Question] Using/adding a local dataset

Comment:
The download manager supports local directories. You can specify a local directory instead of a url and it should work.
Issue: [Feature request] Add Toronto BookCorpus dataset

Comment:
As far as I understand, `wikitext` is refer to `WikiText-103` and `WikiText-2` that created by researchers in Salesforce, and mostly used in traditional language modeling.

You might want to say `wikipedia`, a dump from wikimedia foundation.

Also I would like to have Toronto BookCorpus too ! Though it involves copyright problem...
Issue: Loading GLUE dataset loads CoLA by default

Comment:
As a follow-up to this: It looks like the actual GLUE task name is supplied as the `name` argument. Is there a way to check what `name`s/sub-datasets are available under a grouping like GLUE? That information doesn't seem to be readily available in info from `nlp.list_datasets()`.

Edit: I found the info under `Glue.BUILDER_CONFIGS`
Issue: Loading GLUE dataset loads CoLA by default

Comment:
Yes so the first config is loaded by default when no `name` is supplied but for GLUE this should probably throw an error indeed.

We can probably just add an `__init__` at the top of the `class Glue(nlp.GeneratorBasedBuilder)` in the `glue.py` script which does this check:
```
class Glue(nlp.GeneratorBasedBuilder):
    def __init__(self, *args, **kwargs):
        assert 'name' in kwargs and kwargs[name] is not None, "Glue has to be called with a configuration name"
        super(Glue, self).__init__(*args, **kwargs)
```
Issue: Loading GLUE dataset loads CoLA by default

Comment:
An error is raised if the sub-dataset is not specified :)
```
ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
	`load_dataset('glue', 'cola')`
```
Issue: [Feature request] Add Google Natural Question dataset

Comment:
Still work in progress :)
The idea is to have the dataset already processed somewhere so that the user only have to download the processed files. I'm also doing it for wikipedia.
Issue: [Feature request] Add Google Natural Question dataset

Comment:
Super appreciate your hard work !!
I'll cross my fingers and hope easily loadable wikipedia dataset will come soon.
Issue: [Feature request] Add Google Natural Question dataset

Comment:
Quick update on NQ: due to some limitations I met using apache beam + parquet I was not able to use the dataset in a nested parquet structure in python to convert it to our Apache Arrow format yet.
However we had planned to change this conversion step anyways so we'll make just sure that it enables to process and convert the NQ dataset to arrow.