SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: OpenBookQA has missing and inconsistent field names

Comment:
I don't have a strong opinion on this, besides the fact that whatever decision we agree on, should be applied to all datasets.

There is always the tension between:
- preserving each dataset original structure (which has the advantage of not forcing users to learn other structure for the same dataset),
-  and on the other hand performing some kind of standardization/harmonization depending on the task (this has the advantage that once learnt, the same structure applies to all datasets; this has been done for e.g.  POS tagging: all datasets have been adapted to a certain "standard" structure).
   - Another advantage: datasets can easily be interchanged (or joined) to be used by the same model

Recently, in the BigScience BioMedical hackathon, they adopted a different approach:
- they implement a "source" config, respecting the original structure as much as possible
- they implement additional config for each task, with a "standard" nested structure per task, which is most useful for users.
Issue: OpenBookQA has missing and inconsistent field names

Comment:
@albertvillanova, thanks for the detailed answer and the new perspectives. I understand the friction for the best design approach much better now.  Ultimately, it is essential to include all the missing fields and the correct data first. Whatever approach is determined to be optimal is important but not as crucial once all the data is there, and users can create lambda functions to create whatever structure serves them best.
Issue: OpenBookQA has missing and inconsistent field names

Comment:
Datasets are not tracked in this repository anymore. I think we must move this thread to the [discussions tab of the dataset](https://huggingface.co/datasets/openbookqa/discussions)
Issue: A typo in docs of datasets.disable_progress_bar

Comment:
Hi! Thanks for catching and reporting the typo, a PR has been opened to fix it :)
Issue: error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered

Comment:
It would help a lot to be able to preview the dataset - I'd like to see if the pronunciations are in the dataset, eg. for ["word"](https://en.wiktionary.org/wiki/word),

Pronunciation
([Received Pronunciation](https://en.wikipedia.org/wiki/Received_Pronunciation)) [IPA](https://en.wiktionary.org/wiki/Wiktionary:International_Phonetic_Alphabet)([key](https://en.wiktionary.org/wiki/Appendix:English_pronunciation)): /wɜːd/
([General American](https://en.wikipedia.org/wiki/General_American)) [enPR](https://en.wiktionary.org/wiki/Appendix:English_pronunciation): wûrd, [IPA](https://en.wiktionary.org/wiki/Wiktionary:International_Phonetic_Alphabet)([key](https://en.wiktionary.org/wiki/Appendix:English_pronunciation)): /wɝd/
Issue: error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered

Comment:
Hi @i-am-neo, thanks for reporting.

Normally this dataset should be private and not accessible for public use. @cakiki, @lvwerra, any reason why is it public? I see many other Wikimedia datasets are also public.

Also note that last commit "Add metadata" (https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_en_wiktionary_filtered/commit/dc2f458dab50e00f35c94efb3cd4009996858609) introduced buggy data files (`data/file-01.jsonl.gz.lock`, `data/file-01.jsonl.gz.lock.lock`). The same bug appears in other datasets as well.

@i-am-neo, please note that in the near future we are planning to make public all datasets used for the BigScience project (at least all of them whose license allows to do that). Once public, they will be accessible for all the NLP community.
Issue: error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered

Comment:
Ah this must be a bug introduced at creation time since the repos were created programmatically; I'll go ahead and make them private; sorry about that!
Issue: error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered

Comment:
All datasets are private now. 

Re:that bug I think we're currently avoiding it by avoiding verifications. (i.e. `ignore_verifications=True`)
Issue: error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered

Comment:
Thanks a lot, @cakiki.

@i-am-neo, I'm closing this issue for now because the dataset is not publicly available yet. Just stay tuned, as we will soon release all the BigScience open-license datasets.
Issue: error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered

Comment:
Thanks for letting me know, @albertvillanova @cakiki.
Any chance of having a subset alpha version in the meantime? 
I only need two dicts out of wiktionary: 1) phoneme(as key): word, and 2) word(as key): its phonemes.

Would like to use it for a mini-poc [Robust ASR](https://github.com/huggingface/transformers/issues/13162#issuecomment-1096881290) decoding, cc @patrickvonplaten. 

(Patrick, possible to email you so as not to litter github with comments? I have some observations after experiments training hubert on some YT AMI-like data (11.44% wer).  Also wonder if a robust ASR is on your/HG's roadmap).  Thanks!
Issue: error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered

Comment:
Hey @i-am-neo,

Cool to hear that you're working on Robust ASR! Feel free to drop me a mail :-)
Issue: error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered

Comment:
@i-am-neo This particular subset of the dataset was taken from the [CirrusSearch dumps](https://dumps.wikimedia.org/other/cirrussearch/current/)
You're specifically after the [enwiktionary-20220425-cirrussearch-content.json.gz](https://dumps.wikimedia.org/other/cirrussearch/current/enwiktionary-20220425-cirrussearch-content.json.gz) file
Issue: error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered

Comment:
thanks @cakiki !  <del>I could access the gz file yesterday (but neglected to tuck it away somewhere safe), and today the link is throwing a 404. Can you help? </del>  Never mind, got it!
Issue: data leakage in `webis/conclugen` dataset

Comment:
Hi @xflashxx, thanks for reporting.

Please note that this dataset was generated and shared by Webis Group: https://huggingface.co/webis

We are contacting the dataset owners to inform them about the issue you found. We'll keep you updated of their reply.
Issue: data leakage in `webis/conclugen` dataset

Comment:
Thanks for reporting this @xflashxx. I'll have a look and get back to you on this.
Issue: data leakage in `webis/conclugen` dataset

Comment:
Hi @xflashxx and @albertvillanova,

I have updated the files with de-duplicated splits. Apparently the debate portals from which part of the examples were sourced had unique timestamps for some examples (up to 6%; updated counts in the README) without any actual content updated that lead to "new" items. The length of `ids_validation` and `ids_testing` is zero.

Regarding impact on scores:
1. We employed automatic evaluation (on a separate set of 1000 examples) only to justify the exclusion of the smaller models for manual evaluation (due to budget constraints). I am confident the ranking still stands (unsurprisingly, the bigger models doing better than those trained on the smaller splits). We also highlight this in the paper. 

2. The examples used for manual evaluation have no overlap with any splits (also because they do not have any ground truth as we applied the trained models on an unlabeled sample to test its practical usage). I've added these two files to the dataset repository.

Hope this helps!
Issue: data leakage in `webis/conclugen` dataset

Comment:
Thanks @shahbazsyed for your fast fix.

As a side note:
- Your email appearing as Point of Contact in the dataset README has a typo: @uni.leipzig.de instead of @uni-leipzig.de
- Your commits on the Hub are not linked to your profile on the Hub: this is because we use the email address to make this link; the email address used in your commit author and the email address set on your Hub account settings.
Issue: conll2003 dataset loads original data.

Comment:
Thanks for reporting @sue99.

Unfortunately. I'm not able to reproduce your problem:
```python
In [1]: import datasets
   ...: from datasets import load_dataset
   ...: dataset = load_dataset("conll2003")

In [2]: dataset
Out[2]: 
DatasetDict({
    train: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 14042
    })
    validation: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 3251
    })
    test: Dataset({
        features: ['id', 'tokens', 'pos_tags', 'chunk_tags', 'ner_tags'],
        num_rows: 3454
    })
})

In [3]: dataset["train"][0]
Out[3]: 
{'id': '0',
 'tokens': ['EU',
  'rejects',
  'German',
  'call',
  'to',
  'boycott',
  'British',
  'lamb',
  '.'],
 'pos_tags': [22, 42, 16, 21, 35, 37, 16, 21, 7],
 'chunk_tags': [11, 21, 11, 12, 21, 22, 11, 12, 0],
 'ner_tags': [3, 0, 7, 0, 0, 0, 7, 0, 0]}
```

Just guessing: might be the case that you are calling `load_dataset` from a working directory that contains a local folder named `conll2003` (containing the raw data files)? If that is the case, `datasets` library gives precedence to the local folder over the dataset on the Hub.
Issue: The data preview of XGLUE

Comment:
Thanks for reporting @czq1999.

Note that the dataset viewer uses the dataset in streaming mode and that not all datasets support streaming yet.

That is the case for XGLUE dataset (as the error message points out): this must be refactored to support streaming.
Issue: NonMatchingChecksumError when attempting to download GLUE

Comment:
Hi :)

I think your issue may be related to the older `nlp` library. I was able to download `glue` with the latest version of `datasets`. Can you try updating with:

```py
pip install -U datasets
```

Then you can download:

```py
from datasets import load_dataset
ds = load_dataset("glue", "rte")
```
Issue: NonMatchingChecksumError when attempting to download GLUE

Comment:
This appears to work. Thank you!

On Wed, Apr 27, 2022, 1:18 PM Steven Liu ***@***.***> wrote:

> Hi :)
>
> I think your issue may be related to the older nlp library. I was able to
> download glue with the latest version of datasets. Can you try updating
> with:
>
> pip install -U datasets
>
> Then you can download:
>
> from datasets import load_datasetds = load_dataset("glue", "rte")
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/4241#issuecomment-1111267650>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACJUEKLUP2EL7ES3RRWJRPTVHFZHBANCNFSM5UPJBYXA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
Issue: Dataset caching policy

Comment:
Hi @loretoparisi, thanks for reporting.

There is an option to force the redownload of the data files (and thus not using previously download and cached data files): `load_dataset(..., download_mode="force_redownload")`.

Please, let me know if this fixes your problem.

I can confirm you that your dataset loads without any problem for me:
```python
In [2]: ds = load_dataset("loretoparisi/tatoeba-sentences", data_files={"train": "train.csv", "test": "test.csv"}, delimiter="\t", column_names=['label', 'text'])

In [3]: ds
Out[3]: 
DatasetDict({
    train: Dataset({
        features: ['label', 'text'],
        num_rows: 8256449
    })
    test: Dataset({
        features: ['label', 'text'],
        num_rows: 2061204
    })
})
```
Issue: Dataset caching policy

Comment:
@albertvillanova thank you, it seems it still does not work using:

```python
sentences = load_dataset(
     "loretoparisi/tatoeba-sentences",
     data_files=data_files,
     delimiter='\t', 
     column_names=['label', 'text'],
     download_mode="force_redownload"
)
```
[This](https://colab.research.google.com/drive/1EA6FWo5pHxU8rPHHRn24NlHqRPiOlPTr?usp=sharing) is my notebook!

The problem is that the download file's revision for `test.csv` is not correctly parsed

![Schermata 2022-04-27 alle 18 09 41](https://user-images.githubusercontent.com/163333/165563507-0be53eb6-8f61-49b0-b959-306e59281de3.png)

If you download that file `test.csv` from the repo, the line `\\N` is not there anymore (it was there at the first file upload).

My impression is that the Apache Arrow file is still cached - so server side, despite of enabling a forced download. For what I can see I get those two arrow files, but I cannot grep the bad line (`\\N`) since are binary files:

```
!ls -l /root/.cache/huggingface/datasets/csv/loretoparisi--tatoeba-sentences-efeff8965c730a2c/0.0.0/433e0ccc46f9880962cc2b12065189766fbb2bee57a221866138fb9203c83519
!ls -l /root/.cache/huggingface/datasets/csv/loretoparisi--tatoeba-sentences-efeff8965c730a2c/0.0.0/433e0ccc46f9880962cc2b12065189766fbb2bee57a221866138fb9203c83519/csv-test.arrow
!head /root/.cache/huggingface/datasets/csv/loretoparisi--tatoeba-sentences-efeff8965c730a2c/0.0.0/433e0ccc46f9880962cc2b12065189766fbb2bee57a221866138fb9203c83519/dataset_info.json
```
Issue: Dataset caching policy

Comment:
SOLVED! The problem was the with the file itself, using caching parameter helped indeed.
Thanks for helping!
Issue: Common Voice 8 doesn't show datasets viewer

Comment:
Thanks for reporting. I understand it's an error in the dataset script. To reproduce:

```python
>>> import datasets as ds
>>> split_names = ds.get_dataset_split_names("mozilla-foundation/common_voice_8_0", use_auth_token="**********")
Downloading builder script: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.9k/10.9k [00:00<00:00, 10.9MB/s]
Downloading extra modules: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.98k/2.98k [00:00<00:00, 3.36MB/s]
Downloading extra modules: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 53.1k/53.1k [00:00<00:00, 650kB/s]
No config specified, defaulting to: common_voice/en
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-preview-backend/libs/libmodels/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 280, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_8_0/720589e6e5ad674019008b719053303a71716db1b27e63c9846df02fdf93f2f3/common_voice_8_0.py", line 153, in _split_generators
    self._log_download(self.config.name, bundle_version, hf_auth_token)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_8_0/720589e6e5ad674019008b719053303a71716db1b27e63c9846df02fdf93f2f3/common_voice_8_0.py", line 139, in _log_download
    email = HfApi().whoami(auth_token)["email"]
KeyError: 'email'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/libs/libmodels/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 323, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-preview-backend/libs/libmodels/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 285, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```
Issue: Common Voice 8 doesn't show datasets viewer

Comment:
Thanks for reporting @patrickvonplaten and thanks for the investigation @severo.

Unfortunately I'm not able to reproduce the error.

I think the error has to do with authentication with `huggingface_hub`, because the exception is thrown from these code lines: https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0/blob/main/common_voice_8_0.py#L137-L139
```python
from huggingface_hub import HfApi, HfFolder

if isinstance(auth_token, bool):
    email = HfApi().whoami(auth_token)
email = HfApi().whoami(auth_token)["email"]
```

Could you please verify the previous code with the `auth_token` you pass to `load_dataset(..., use_auth_token=auth_token,...`?
Issue: Common Voice 8 doesn't show datasets viewer

Comment:
OK, thanks for digging a bit into it. Indeed, the error occurs with the dataset-viewer, but not with a normal user token, because we use an app token, and it does not have a related email!

```python
>>> from huggingface_hub import HfApi, HfFolder
>>> auth_token = "hf_app_******"
>>> t = HfApi().whoami(auth_token)
>>> t
{'type': 'app', 'name': 'dataset-preview-backend'}
>>> t["email"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'email'
```

Note also that the doc (https://huggingface.co/docs/huggingface_hub/package_reference/hf_api#huggingface_hub.HfApi.whoami) does not state that `whoami` should return an `email` key.

@SBrandeis @julien-c: do you think the app token should have an email associated, like the users?
Issue: Common Voice 8 doesn't show datasets viewer

Comment:
We can workaround this with
```python
email = HfApi().whoami(auth_token).get("email", "system@huggingface.co")
```
in the common voice scripts
Issue: Common Voice 8 doesn't show datasets viewer

Comment:
Hmmm, does this mean that any person who downloads the common voice dataset will be logged as "system@huggingface.co"? If so, it would defeat the purpose of sending the user's email to the commonvoice API, right?
Issue: Common Voice 8 doesn't show datasets viewer

Comment:
I agree with @severo: we cannot set our system email as default, allowing anybody not authenticated to by-pass the Common Voice usage policy.

Additionally, looking at the code, I think we should implement a more robust way to send user email to Common Voice: currently anybody can tweak the script and send somebody else email instead.

CC: @patrickvonplaten @lhoestq @SBrandeis @julien-c
Issue: Common Voice 8 doesn't show datasets viewer

Comment:
Hmm I don't agree here. 

Anybody can always just bypass the system by setting whatever email. As soon as someone has access to the downloading script it's trivial to tweak the code to not send the "correct" email but to just whatever and it would work.

Note that someone only has visibility on the code after having "signed" the access-mechanism so I think we can expect the users to have agreed to not do anything malicious. 

I'm fine with both @lhoestq's solution or we find a way that forces the user to be logged in + being able to load the data for the datasets viewer. Wdyt @lhoestq @severo @albertvillanova ?
Issue: Common Voice 8 doesn't show datasets viewer

Comment:
> Additionally, looking at the code, I think we should implement a more robust way to send user email to Common Voice: currently anybody can tweak the script and send somebody else email instead.

Yes, I agree we can forget about this @patrickvonplaten. After having had a look at Common Voice website, I've seen they only require sending an email (no auth is inplace on their side, contrary to what I had previously thought). Therefore, currently we impose stronger requirements than them: we require the user having logged in and accepted the access mechanism.

Currently the script as it is already requires the user being logged in:
```python
HfApi().whoami(auth_token)
```
throws an exception if None/invalid auth_token is passed.

On the other hand, we should agree on the way to allow the viewer to stream the data.
Issue: Why the `conll2003` dataset on huggingface only contains the `en` subset? Where is the  German data?

Comment:
Thanks for reporting @beyondguo.

Indeed, we generate this dataset from this raw data file URL: https://data.deepai.org/conll2003.zip
And that URL only contains the English version.
Issue: Why the `conll2003` dataset on huggingface only contains the `en` subset? Where is the  German data?

Comment:
The German data requires payment

The [original task page](https://www.clips.uantwerpen.be/conll2003/ner/) states "The German data is a collection of articles from the Frankfurter Rundschau. The named entities have been annotated by people of the University of Antwerp. Only the annotations are available here. In order to build these data sets you need access to the ECI Multilingual Text Corpus. It can be ordered from the Linguistic Data Consortium (2003 non-member price: US$ 35.00)."

Inflation since 2003 has also affected LDC's prices, and today the dataset [LDC94T5](https://catalog.ldc.upenn.edu/LDC94T5) is available under license for $75 a copy. The [license](https://catalog.ldc.upenn.edu/license/eci-slash-mci-user-agreement.pdf) includes a non-distribution condition, which is probably why the data has not turned up openly.

The ACL hold copyright of this data; I'll mail them and anyone I can find at ECI to see if they'll open this up now. After all, it worked with Microsoft 3DMM, why not here too, after 28 years? :)
Issue: Dictionary Feature

Comment:
Hi @jordiae,

Instead of the `Sequence` feature, you can use just a regular list: put the dict between `[` and `]`:
```python
"list_of_dict_feature": [
    {
        "key1_in_dict": datasets.Value("string"),
        "key2_in_dict": datasets.Value("int32"),
        ...
    }
],
```

Feel free to re-open this issue if that does not work for your use case.
Issue: Dictionary Feature

Comment:
> Hi @jordiae,
> 
> Instead of the `Sequence` feature, you can use just a regular list: put the dict between `[` and `]`:
> 
> ```python
> "list_of_dict_feature": [
>     {
>         "key1_in_dict": datasets.Value("string"),
>         "key2_in_dict": datasets.Value("int32"),
>         ...
>     }
> ],
> ```
> 
> Feel free to re-open this issue if that does not work for your use case.

Thank you
Issue: Big_Patent dataset broken

Comment:
Thanks for reporting. The issue seems not to be directly related to the dataset viewer or the `datasets` library, but instead to it being hosted on Google Drive.

See related issues: https://github.com/huggingface/datasets/issues?q=is%3Aissue+is%3Aopen+drive.google.com

To quote [@lhoestq](https://github.com/huggingface/datasets/issues/4075#issuecomment-1087362551):

> PS: if possible, please try to not use Google Drive links in your dataset script, since Google Drive has download quotas and is not always reliable.
Issue: Big_Patent dataset broken

Comment:
We should find out if the dataset license allows redistribution and contact the data owners to propose them to host their data on our Hub.
Issue: DatasetDict containing Datasets with different features when pushed to hub gets remapped features

Comment:
Hi @pietrolesci, thanks for reporting.

Please note that this is a design purpose: a `DatasetDict` has the same features for all its datasets. Normally, a `DatasetDict` is composed of several sub-datasets each corresponding to a different **split**.

To handle sub-datasets with different features, we use another approach: use different **configurations** instead of **splits**.

However, for the moment `push_to_hub` does not support specifying different configurations. IMHO, we should implement this.
Issue: DatasetDict containing Datasets with different features when pushed to hub gets remapped features

Comment:
Hi @albertvillanova,

Thanks a lot for your reply! I got it now. The strange thing for me was to have it correctly working (i.e., DatasetDict with different features in some datasets) locally and not on the Hub. It would be great to have configuration supported by `push_to_hub`. Personally, this latter functionality allowed me to iterate rather quickly on dataset curation.

Again, thanks for your time @albertvillanova!

Best,
Pietro