error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered
State: closed
Labels: description: A bug in a dataset script provided in the library
name: dataset bug

## Describe the bug
Error generated when attempting to download dataset

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("bigscience-catalogue-lm-data/lm_en_wiktionary_filtered")
```

## Expected results
A clear and concise description of the expected results.

## Actual results
```
ExpectedMoreDownloadedFiles               Traceback (most recent call last)

[<ipython-input-62-4ac5cf959477>](https://localhost:8080/#) in <module>()
      1 from datasets import load_dataset
      2 
----> 3 dataset = load_dataset("bigscience-catalogue-lm-data/lm_en_wiktionary_filtered")

3 frames

[/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py](https://localhost:8080/#) in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     31         return
     32     if len(set(expected_checksums) - set(recorded_checksums)) > 0:
---> 33         raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
     34     if len(set(recorded_checksums) - set(expected_checksums)) > 0:
     35         raise UnexpectedDownloadedFile(str(set(recorded_checksums) - set(expected_checksums)))

ExpectedMoreDownloadedFiles: {'/home/leandro/catalogue_data/datasets/lm_en_wiktionary_filtered/data/file-01.jsonl.gz', '/home/leandro/catalogue_data/datasets/lm_en_wiktionary_filtered/data/file-01.jsonl.gz.lock'}
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1


Comments:
It would help a lot to be able to preview the dataset - I'd like to see if the pronunciations are in the dataset, eg. for ["word"](https://en.wiktionary.org/wiki/word),

Pronunciation
([Received Pronunciation](https://en.wikipedia.org/wiki/Received_Pronunciation)) [IPA](https://en.wiktionary.org/wiki/Wiktionary:International_Phonetic_Alphabet)([key](https://en.wiktionary.org/wiki/Appendix:English_pronunciation)): /wɜːd/
([General American](https://en.wikipedia.org/wiki/General_American)) [enPR](https://en.wiktionary.org/wiki/Appendix:English_pronunciation): wûrd, [IPA](https://en.wiktionary.org/wiki/Wiktionary:International_Phonetic_Alphabet)([key](https://en.wiktionary.org/wiki/Appendix:English_pronunciation)): /wɝd/

Hi @i-am-neo, thanks for reporting.

Normally this dataset should be private and not accessible for public use. @cakiki, @lvwerra, any reason why is it public? I see many other Wikimedia datasets are also public.

Also note that last commit "Add metadata" (https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_en_wiktionary_filtered/commit/dc2f458dab50e00f35c94efb3cd4009996858609) introduced buggy data files (`data/file-01.jsonl.gz.lock`, `data/file-01.jsonl.gz.lock.lock`). The same bug appears in other datasets as well.

@i-am-neo, please note that in the near future we are planning to make public all datasets used for the BigScience project (at least all of them whose license allows to do that). Once public, they will be accessible for all the NLP community.

Ah this must be a bug introduced at creation time since the repos were created programmatically; I'll go ahead and make them private; sorry about that!

All datasets are private now. 

Re:that bug I think we're currently avoiding it by avoiding verifications. (i.e. `ignore_verifications=True`)

Thanks a lot, @cakiki.

@i-am-neo, I'm closing this issue for now because the dataset is not publicly available yet. Just stay tuned, as we will soon release all the BigScience open-license datasets.  

Thanks for letting me know, @albertvillanova @cakiki.
Any chance of having a subset alpha version in the meantime? 
I only need two dicts out of wiktionary: 1) phoneme(as key): word, and 2) word(as key): its phonemes.

Would like to use it for a mini-poc [Robust ASR](https://github.com/huggingface/transformers/issues/13162#issuecomment-1096881290) decoding, cc @patrickvonplaten. 

(Patrick, possible to email you so as not to litter github with comments? I have some observations after experiments training hubert on some YT AMI-like data (11.44% wer).  Also wonder if a robust ASR is on your/HG's roadmap).  Thanks!

Hey @i-am-neo,

Cool to hear that you're working on Robust ASR! Feel free to drop me a mail :-)

@i-am-neo This particular subset of the dataset was taken from the [CirrusSearch dumps](https://dumps.wikimedia.org/other/cirrussearch/current/)
You're specifically after the [enwiktionary-20220425-cirrussearch-content.json.gz](https://dumps.wikimedia.org/other/cirrussearch/current/enwiktionary-20220425-cirrussearch-content.json.gz) file

thanks @cakiki !  <del>I could access the gz file yesterday (but neglected to tuck it away somewhere safe), and today the link is throwing a 404. Can you help? </del>  Never mind, got it!

thanks @patrickvonplaten.  will do - getting my observations together.