Builder Configuration Update Required on Common Voice Dataset
State: closed | Author: aasem (NONE)
Labels: name: bug

Missing language in Common Voice dataset

**Link:** https://huggingface.co/datasets/common_voice

I tried to call the Urdu dataset using `load_dataset("common_voice", "ur", split="train+validation")` but couldn't due to builder configuration not found. I checked the source file here for the languages support:

https://github.com/huggingface/datasets/blob/master/datasets/common_voice/common_voice.py

and Urdu isn't included there. I assume a quick update will fix the issue as Urdu speech is now available at the Common Voice dataset.

Am I the one who added this dataset? No


Comments: Hi @aasem, thanks for reporting.

Please note that currently Commom Voice is hosted on our Hub as a community dataset by the Mozilla Foundation. See all Common Voice versions here: https://huggingface.co/mozilla-foundation

Maybe we should add an explaining note in our "legacy" Common Voice canonical script? What do you think @lhoestq @mariosasko ?

Thank you, @albertvillanova, for the quick response. I am not sure about the exact flow but I guess adding the following lines under the `_Languages` dictionary definition in [common_voice.py](https://github.com/huggingface/datasets/blob/master/datasets/common_voice/common_voice.py) might resolve the issue. I guess the dataset is recently made available so the file needs updating.

```
"ur": {
        "Language": "Urdu",
        "Date": "2022-01-19",
        "Size": "68 MB",
        "Version": "ur_3h_2022-01-19",
        "Validated_Hr_Total": 1,
        "Overall_Hr_Total": 3,
        "Number_Of_Voice": 48,
    },
```


@aasem for compliance reasons, we are no longer updating the `common_voice.py` script.

We agreed with Mozilla Foundation to use their community datasets instead, which will ask you to accept their terms of use:
```
You need to share your contact information to access this dataset.

This repository is publicly accessible, but you have to register to access its content — don't worry, it's just one click!

By clicking on “Access repository” below, you accept that your contact information (email address and username) can be shared with the repository authors. This will let the authors get in touch for instance if some parts of the repository's contents need to be taken down for licensing reasons.

By clicking on “Access repository” below, you also agree to not attempt to determine the identity of speakers in the Common Voice dataset.

You will immediately be granted access to the contents of the dataset. 
```

In order to use e.g. their Common Voice dataset version 8.0, please:
- First visit their dataset page: https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
- Accept their term of use by clicking "Access repository"
- You can then load their dataset with:
  ```python
  load_dataset("mozilla-foundation/common_voice_8_0", "ur", split="train+validation")
  ```

@albertvillanova 
>Maybe we should add an explaining note in our "legacy" Common Voice canonical script?

Yes, I agree we should have a deprecation notice in the canonical script to redirect users to the new script.

@albertvillanova, 
I now get the following error after downloading my access token from the huggingface and passing it to `load_dataset` call:

`AttributeError: 'DownloadManager' object has no attribute 'download_config'`

Any quick pointer on how it might be resolved?

@aasem What version of `datasets` are you using? We renamed that attribute from `_download_config` to `download_conig` fairly recently, so updating to the newest version should resolve the issue:
```
pip install -U datasets
```

Thanks a lot, @mariosasko. That completely resolved the issue.