Common voice
State: closed | Author: BirgerMoell (CONTRIBUTOR)

Started filling out information about the dataset and a dataset card.

To do
Create tagging file
Update the common_voice.py file with more information

Comments: Does it make sense to make the domains as the different languages?
A problem is that you need to download the datasets from the browser.
One idea would be to either contact Mozilla regarding API access to the dataset or make use of a headless browser for downloading the datasets (might be hard since we have to figure out how to host them). An even  more creative idea would be to host the dataset inside a torrent and figure out a way to download specific datasets from within that torrent.

Here is some information about the download authorization. They are hosting the data on S3.

https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

Here is an example of how a download link looks.

https://mozilla-common-voice-datasets.s3.dualstack.us-west-2.amazonaws.com/cv-corpus-6.1-2020-12-11/nl.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQ3GQRTO3ND4UAQXB%2F20210217%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20210217T080740Z&X-Amz-Expires=43200&X-Amz-Security-Token=FwoGZXIvYXdzEGIaDCC6ALh%2FwIK9ovvRdCKSBCs5WaSJNsZ2h0SnhpnWFv4yiAJHJTe%2BY6pBcCqadRMs0RABHeQ2n1QDACJ5V9WOqIHfMfT0AI%2Bfe6iFkTGLgRrJOMYpgV%2FmIBcXCjeb72r4ZvudMA8tprkSxZsEh53bJkIDQx1tXqfpz0yoefM0geD3461suEGhHnLIyiwffrUpRg%2BkNZN9%2FLZZXpF5F2pogieKKV533Jetkd1xlWOR%2Bem9R2bENu2RV563XX3JvbWxSYN9IHkVT1xwd4ZiOpUtX7%2F2RoluJUKw%2BUPpyml3J%2FOPPGdr7CyPLjqNxdq9ceRi8lRybty64XvNYZGt45VNTQ3pkTTz4VpUCJAGkgxq95Ve%2BOwW%2Fsc8JtblTFKrH11vej62NB7C0n7JPPS4SLKXHKW%2B7ZbybcNf3BnsAVouPdsGTMslcgkD81b9trnjyXJdOZkzdHUf2KcWVXVceEsZnMhcCZQ1cJpI7qXPEk8QrKCQcNByPLHmPIEdHpj9IrIBKDkl2qO7VX7CCB65WDt2eZRltOcNHXWVFXFktMdQOQztI1j0XSZz2iOX4jPKKaqz193VEytlAqmehNi8pePOnxkP9Z1SP7d3I6rayuBF3phmpHxw499tY3ECYYgoCnJ6QSFa3KxMjFmEpQlmjxuwEMHd4CDL2FJYGcCiIxbCcL1r8ZE3%2BbGdcu7PRsVCHX3Huh%2FqGIaF4h40FgteN6teyKCHKOebs4EGMipb9xmEMZ9ZbVopz4bkhLdMTrjKon9w624Xem0MTPqN7XY%2BB6lRgrW8rd4%3D&X-Amz-Signature=28eabdfce72a472a70b0f9e1e2c37fe1471b5ec8ed60614fbe900bfa97ae1ac8&X-Amz-SignedHeaders=host

It could be that we simply need to make a http-request with the right parameters and we can download the datasets.

> Wow, this looks great already! It's really a difficult dataset so thanks a lot for opening a PR.
> I think the tagging tool is not too important for now and we can take a look at that later!
> 
> At the moment, it would be very good to correctly generate some dummy data for all the possible languages. I think the structure of the `.tsv` file as you've noted in the PR is the one we want to use as the structure for `features = datasets.Features(`
> 
> The splits `'Train"`, `"Test"`, `"Validation"` look great to me! Because this is a special dataset that also has files called `"Invalidated"` I think the best option is to also add those as splits, _i.e._ `"other"`, `"invalidated"`, `"reported"`, `"validated"` . Those split names can be gives as shown here for example:
> 
> https://github.com/huggingface/datasets/blob/28be129db862ec89a87ac9349c64df6b6118aff4/datasets/librispeech_asr/librispeech_asr.py#L124
> 
> Also putting @lhoestq in cc here to hear his opinion on the different splits. @lhoestq Common Voicie is a crowd collected dataset where if a collected data sample did not receive enough "up_votes" from the community -> then it is (If I understood it correctly) marked as invalid -> hence the file `"invalidated.tsv"`. I think this is still useful data, so I would include it what do you think?
> 
> @BirgerMoell let me know if you have any more questions :-)

I think reporting is a separate feature. People can help annotate the data and then they can report things while annotating.
https://commonvoice.mozilla.org/sv-SE/listen

Here is the interface that shows reporting and the thumbs up and down which gives upvotes and downvotes.
<img src="https://i.imgur.com/utWjszt.png" height="800px">


I added splits and features. I'm not sure how you want me to generate dummy data for all the languages?

Hey @BirgerMoell,

I tweaked your dataset file a bit to have a first working version. To test this dataset downloading script, you can do the following:

- 1) Download the Common Voice Georgian dataset from https://commonvoice.mozilla.org/en/datasets (It's pretty small which is why I chose it)
- 2) Run the following command using this branch: 

```python
from datasets import load_dataset

ds = load_dataset("./../datasets/datasets/common_voice", "Georgian", data_dir="./cv-corpus-6.1-2020-12-11/ka/", split="train")
```

Note that I'm loading a local version of the dataset script (`"./../datasets/datasets/common_voice/"` points to the folder in your branch) and that I also insert the downloaded data with the `data_dir` arg.

-> You'll see that the data is correctly loaded and that `ds` contains all the information we need.

Now there are a lot of different datasets on Common Voice, so it probably takes too much time to test all of those, but maybe you can test whether the current script works as well *e.g.* for Swedish, 3,4 other languages.

It would be very nice if we can use the exact same structure for all languages, meaning that we don't have to change the `datasets.Features(...)` structure depending on the language, but can use the exact same one for every language.

If everything works as expected we can then go over to cleaning the script and seeing how to add dummy data tests for it.