MustC Speech Translation
Author: patrickvonplaten
State: open
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** *IWSLT19*
- **Description:** *The Speech Translation Task addresses the translation of English audio into German and Portuguese text.*
- **Hompage:** *https://sites.google.com/view/iwslt-evaluation-2019/speech-translation*
- **Data:** *https://sites.google.com/view/iwslt-evaluation-2019/speech-translation* - all data under "Allowed Training Data" and "Development and Evalutaion Data for TED/How2"
- **Motivation:** Important speech dataset

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: Hi @patrickvonplaten  I would like to work on this dataset. 

Thanks! 

That's awesome! Actually, I just noticed that this dataset might become a bit too big!

MuST-C is the main dataset used for IWSLT19 and should probably be added as a standalone dataset. Would you be interested also in adding `datasets/MuST-C` instead?

Description: 
_MuST-C is a multilingual speech translation corpus whose size and quality facilitates the training of end-to-end systems for speech translation from English into several languages. For each target language, MuST-C comprises several hundred hours of audio recordings from English TED Talks, which are automatically aligned at the sentence level with their manual transcriptions and translations._

Paper: https://www.aclweb.org/anthology/N19-1202.pdf

Dataset: https://ict.fbk.eu/must-c/ (One needs to fill out a short from to download the data, but it's very easy).

It would be awesome if you're interested in adding this datates. I'm very happy to guide you through the PR! I think the easiest way to start would probably be to read [this README on how to add a dataset](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) and open a PR. Think you can copy & paste some code from:

- Librispeech_asr: https://github.com/huggingface/datasets/blob/master/datasets/librispeech_asr/librispeech_asr.py
- Flores Translation: https://github.com/huggingface/datasets/blob/master/datasets/flores/flores.py

Think all the rest can be handled on the PR :-) 

Hi @patrickvonplaten 
I have tried downloading this dataset, but the connection seems to reset all the time. I have tried it via the browser, wget, and using gdown . But it gives me an error message. _"The server is busy or down, pls try again"_ (rephrasing the message here)

I have completed adding 4 datasets in the previous data sprint (including the IWSLT dataset #1676 ) ...so just checking if you are able to download it at your end. Otherwise will write to the dataset authors to update the links. 






Let me check tomorrow! Thanks for leaving this message!

cc @patil-suraj for notification 

@skyprince999, I think I'm getting the same error you're getting :-/

```
Sorry, you can't view or download this file at this time.

Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.
```

It would be great if you could write the authors to see whether they can fix it.
Also cc @lhoestq - do you think we could mirror the dataset? 

Also there are huge those datasets. Think downloading MuST-C v1.2 amounts to ~ 1000GB... because there are 14 possible configs each around 60-70GB. I think users mostly will only use one of the 14 configs so that they would only need, in theory, will have to download ~60GB which is ok. But I think this functionality doesn't exist yet in `datasets` no? cc @lhoestq 

> Also cc @lhoestq - do you think we could mirror the dataset?

Yes we can mirror it if the authors are fine with it. You can create a dataset repo on huggingface.co (possibly under the relevant org) and add the mirrored data files.

> I think users mostly will only use one of the 14 configs so that they would only need, in theory, will have to download ~60GB which is ok. But I think this functionality doesn't exist yet in datasets no? cc @lhoestq

If there are different download links for each configuration we can make the dataset builder download only the files related to the requested configuration.

I have written to the dataset authors, highlighting this issue. Waiting for their response. 

Update on 25th Feb: 
The authors have replied back, they are updating the download link and will revert back shortly! 

```
first of all thanks a lot for being interested in MuST-C and for building the data-loader.

Before answering your request, I'd like to clarify that the creation, maintenance, and expansion of MuST-c are not supported by any funded project, so this means that we need to find economic support for all these activities. This also includes permanently moving all the data to AWS or GCP.  We are working at this with the goal of facilitating the use of MuST-C, but this is not something that can happen today. We hope to have some news ASAP and you will be among the first to be informed.

I hope you understand our situation.
```



Awesome, actually @lhoestq let's just ask the authors if we should host the dataset no? They could just use our links then as well for their website - what do you think? Is it fine to use our AWS dataset storage also as external links? 

Yes definitely. Shall we suggest them to create a dataset repository under their org on huggingface.co ? @julien-c 
The dataset is around 1TB

Sounds good! 

Order of magnitude is storage costs ~$20 per TB per month (not including bandwidth). 

Happy to provide this to the community as I feel this is an important dataset. Let us know what the authors want to do!



Great! @skyprince999, do you think you could ping the authors here or link to this thread? I think it could be a cool idea to host the dataset on our side then

Done. They replied back, and they want to have a call over a meet/ skype. Is that possible ? 
Btw @patrickvonplaten you are looped in that email (_pls check you gmail account_)  

Hello! Any news on this?

@gegallego  there were some concerns regarding dataset usage & attribution by a for-profit company, so couldn't take it forward. Also the download links were unstable. 
But I guess if you want to test the fairseq benchmarks, you can connect with them directly for downloading the dataset.  

Yes, that dataset is not easy to download... I had to copy it to my Google Drive and use `rsync` to be able to download it.
However, we could add the dataset with a manual download, right?

yes that is possible. I couldn't unfortunately complete this PR, If you would like to add it, please feel free to do it. 

> https://ict.fbk.eu/must-c/

@patrickvonplaten 

The above link does not work anymore. Can you share a new link?