Bug in downloading common_voice data and snall chunk of it to one's own hub
Opened by capsabogdan (closed)

### Describe the bug

I'm trying to load the common voice dataset. Currently there is no implementation to download just par tof the data, and I need just one part of it, without downloading the entire dataset



Help please?


![image](https://user-images.githubusercontent.com/48530104/203260511-26df766f-6013-4eaf-be26-8aa13794def2.png)


### Steps to reproduce the bug

So here is what I have done:
1. Download common_voice data 
2. Trim part of it and publish it to my own repo.
3. Download data from my own repo, but am getting this error.

### Expected behavior

There shouldn't be an error in downloading part of the data and publishing it to one's own repo

### Environment info

common_voice 11

Comments:
Sounds like one of the file is not a valid one, can you make sure you uploaded valid mp3 files ?

Well I just sharded the original commonVoice dataset and pushed a small chunk of it in a private rep

What did go wrong?

Holen Sie sich Outlook für iOS<https://aka.ms/o0ukef>
________________________________
Von: Quentin Lhoest ***@***.***>
Gesendet: Tuesday, November 22, 2022 3:03:40 PM
An: huggingface/datasets ***@***.***>
Cc: capsabogdan ***@***.***>; Author ***@***.***>
Betreff: Re: [huggingface/datasets] Bug in downloading common_voice data and snall chunk of it to one's own hub (Issue #5276)


Sounds like one of the file is not a valid one, can you make sure you uploaded valid mp3 files ?

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5276#issuecomment-1323727434>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALSIFOAPAL2V4TBJTSPMAULWJTHDZANCNFSM6AAAAAASHQJ63U>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


It should be all good then !
Could you share a link to your repository for me to investigate what went wrong ?

https://huggingface.co/datasets/DTU54DL/common-voice-test16k

Am Di., 22. Nov. 2022 um 16:43 Uhr schrieb Quentin Lhoest <
***@***.***>:

> It should be all good then !
> Could you share a link to your repository for me to investigate what went
> wrong ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5276#issuecomment-1323876682>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOEUJRZWXAM7DYA5VJDWJTS3NANCNFSM6AAAAAASHQJ63U>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


I see ! This is a bug with MP3 files.

When we store audio data in parquet, we store the bytes and the file name. From the file name extension we know if it's a WAV, an MP3 or else. But here it looks like the paths are all None.

It looks like it comes from here:

https://github.com/huggingface/datasets/blob/7feeb5648a63b6135a8259dedc3b1e19185ee4c7/src/datasets/features/audio.py#L212

Cc @polinaeterna maybe we should simply put the file name instead of None values ?

@lhoestq I remember we wanted to avoid storing redundant data but maybe it's not that crucial indeed to store one more string value. 
Or we can store paths only for mp3s, considering that for other formats we don't have such a problem with reading from bytes without format specified.  

It doesn't cost much to always store the file name IMO

thanks for the help!

can I do anything on my side? we are doing a DL project and we need the
data really quick.

thanks
bogdan

> Message ID: ***@***.***>
>


I opened a pull requests here: https://github.com/huggingface/datasets/pull/5285, we'll do a new release soon with this fix.

Otherwise if you're really in a hurry you can install `datasets` from this PR

[image: image.png]

> Message ID: ***@***.***>
>


any idea on what's going wrong here?

thanks

Am So., 27. Nov. 2022 um 13:53 Uhr schrieb Bogdan Capsa <
***@***.***>:

> [image: image.png]
>
>> Message ID: ***@***.***>
>>
>


hi @capsabogdan! 
could you please share more specifically what problem do you have now?

I have attached this screenshot above . can u pls help? So can not pip from pull request

![image](https://user-images.githubusercontent.com/48530104/204354027-6173e6d1-e3d4-4085-a363-e924cfe1a7f4.png)


The pull request has been merged on `main`.
You can install `datasets` from `main` using
```
pip install git+https://github.com/huggingface/datasets.git
```

I've tried to load this dataset DTU54DL/common-voice-test16k, but am
getting the same error.

So the bug fix will fix only if I upload a new dataset, or also loading
previously uploaded datasets?

thanks

Am Mo., 28. Nov. 2022 um 19:51 Uhr schrieb Quentin Lhoest <
***@***.***>:

> The pull request has been merged on main.
> You can install datasets from main using
>
> pip install git+https://github.com/huggingface/datasets.git
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5276#issuecomment-1329587334>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOCNYYIGHM2EX3ZIO6DWKT5MXANCNFSM6AAAAAASHQJ63U>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


> So the bug fix will fix only if I upload a new dataset, or also loading
previously uploaded datasets?

You have to reupload the dataset, sorry for the inconvenience

thank you so much for the help! works like a charm!

Am Di., 29. Nov. 2022 um 12:15 Uhr schrieb Quentin Lhoest <
***@***.***>:

> So the bug fix will fix only if I upload a new dataset, or also loading
> previously uploaded datasets?
>
> You have to reupload the dataset, sorry for the inconvenience
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5276#issuecomment-1330468393>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALSIFOBKEFZO57BAKY4IGW3WKXQUZANCNFSM6AAAAAASHQJ63U>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>