Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation (2024)

Abstract: This paper describes our system submission to the International Conference on Spoken Language Translation (IWSLT 2024) for Irish-toEnglish speech translation. We built end-to-end systems based on Whisper, and employed a number of data augmentation techniques, such as speech back-translation and noise augmentation. We investigate the effect of using synthetic audio data and discuss several methods for enriching signal diversity.

heading: 1 Introduction
text: Resource scarcity and the scattered nature of the data are crucial challenges for low-resource languages (Lankford et al., 2021; Haddow et al., 2022; Lovenia et al., 2024). In this sense, Irish is considered a low-resource language and significantly lacking in speech and language tools and resources (Barry et al., 2022; Lynn, 2022). Researchers have been employing various data augmentation techniques to improve the quality of low-resource textual machine translation (MT) systems. Among these techniques is using synthetic data generated by back-translation (Sennrich et al., 2016; Edunov et al., 2018; Dowling et al., 2019; Poncelas et al., 2019; Haque et al., 2020), or large language models (Moslem et al., 2022). Similarly, in the area of speech, Lee et al. (2023) showed that models trained solely on synthetic audio datasets can generalize their performance to human voice data. Nevertheless, Guo et al. (2023) revealed a consistent decrease in the diversity of the outputs of language models trained on synthetic textual data. We observe that leveraging synthetic audio data generated by text-to-speech (TTS) models can be beneficial for training speech translation models, especially for low-resource languages. However, it can lack the diversity found in authentic audio signals in terms of pitch, speed, and background noise.
Speech translation systems can be cascaded systems or end-to-end systems (Agarwal et al., 2023).
Cascaded systems use two models, one for automatic speech recognition (ASR) and one for textual machine translation (MT). End-to-end speech translation systems use one model for the whole process; hence, it is more challenging. In this work, we present end-to-end speech translation models.
In addition to describing our system submitted to IWSLT 2024, this work presents the following contributions:
• Showcasing “speech back-translation” as an effective data augmentation technique for speech translation. In other words, just as back-translation can improve the output quality of text-to-text MT, generating source-side synthetic audio data can considerably enhance the performance of speech translation systems, especially for low-resource languages.
• Introducing a collection of datasets for Irishto-English speech translation, three of which comprise 196 hours of synthetic audio.
• Exploring diverse training settings and data processing techniques such as noise augmentation and voice audio detection (VAD).
• Releasing versions of Whisper models, specifically fine-tuned for Irish-to-English speech translation.

heading: 2 Authentic Data
text: The organizers of the IWSLT shared task, provided the IWSLT-2023 dataset, which consists of training, dev, and test portions. We used both the training and dev portions for training, and the test portion for evaluation. We also used the Irish portion of the FLEURS datasets. Moreover, we employed the bilingual audio-text data available at the Bitesize website for teaching Irish.1
1https://huggingface.co/datasets/ymoslem/ BitesizeIrish-GA-EN
313

heading: 3 Synthetic Data
text: This section explains diverse approaches for creating synthetic data for speech translation. We describe each approach, as well as its advantages and disadvantages.

heading: 3.1 Machine Translation
text: When both audio and transcription are available, but there is no translation, forward MT can be useful as a data augmentation technique. However, there is the risk of feeding incorrect target translations into the training process. Forward MT is more sensitive to the quality of the system used to produce the synthetic data. Compared to backtranslation, biases and errors in synthetic data are intuitively more problematic in forward-translation, since they directly affect the gold labels (Bogoychev and Sennrich, 2019). Hence, the used MT system must be of high quality.
We automatically translated the Irish portion of the Spoken Words dataset into English using the Google Translation API. For quality considerations, we decided to use this dataset for training only, but not for evaluation. The dataset consists of 10,925 utterances. Some words are spoken by multiple narrators.2

heading: 3.2 Synthetic Audio Data
text: OPUS (Tiedemann, 2012) hosts several bilingual textual datasets. We extracted portions of the
2https://huggingface.co/datasets/ymoslem/ SpokenWords-GA-EN-MTed
Tatoeba, Wikimedia, and EUbookshop datasets, comprising 1,983, 7,545 and 33,634 segments, respectively. We extensively filter the datasets based on the following criteria: removing duplicates, removing segments longer than 30 words,3 language detection with fastText (Joulin et al., 2017) (both sides), and Seamless toxicity filtering (Barrault et al., 2023). Finally, we used Azure Speech service to generate two sets of audio data, one with a female voice (OrlaNeural) and the other with a male voice (ColmNeural). As an outcome of this process, we introduce three new datasets, Tatoeba-SpeechIrish,4 Wikimedia-Speech-Irish,5 and EUbookshopSpeech-Irish,6 which together comprise 196 hours of synthetic audio. Table 1 illustrates the statistics of our datasets.

heading: 3.3 Audio Signal Processing Augmentation
text: Synthetic audio data generated by TTS models can have different characteristics than authentic audio. In addition to quality considerations, we observe that among the features that distinguish data generated by TTS systems from authentic data are: 1) lack of noise, and 2) silence differences.
Lack of noise: TTS systems try to mimic studio settings, and produce very clean audio signals.
3https://github.com/ymoslem/MT-Preparation 4https://huggingface.co/datasets/ymoslem/
Tatoeba-Speech-Irish 5https://huggingface.co/datasets/ymoslem/ Wikimedia-Speech-Irish 6https://huggingface.co/datasets/ymoslem/ EUbookshop-Speech-Irish
However, authentic audio signals can include all sorts of environmental noise, ranging from white noise to background voices of people and cars. Even in studio settings, some breath signals can occur unless the audio is extensively edited.
Silence variances: All the synthetic audio signals we generated start at a similar point, with almost no silence at the beginning of the audio (probably to facilitate mixing tracks). However, authentic audio signals can start at any point depending on the recording and processing settings, or whether a signal is truncated from a longer one.
Figure 1 illustrates an example sentence from the Common Voice dataset uttered by a human female (non-studio settings) and its synthetic equivalent generated by Azure TTS system.7 The Irish sentence represented here is “Go raibh maith agaibh as ucht na fíorchaoin fáilte a d’fhear sibh romham.” It can be translated into English as “Thank you all for that very generous welcome.” The authentic signal has more noise (both white background noise and sounds of starting/stopping the recording software), while the synthetic signal does not show any noise occurrence. Moreover, unlike the authentic signal, the synthetic data starts almost immediately. Another observation is that this specific authentic signal has a lower volume than synthetic signals.
7Voice name: “Microsoft Server Speech Text to Speech Voice (ga-IE, OrlaNeural)”

heading: 3.3.1 Voice Activity Detection
text: One of the most common audio preprocessing techniques is Voice Audio Detection (VAD). The main idea of VAD is to remove low-amplitude samples from an audio signal. Low-amplitude samples might represent science or noise samples of audio signals, which usually occur at the beginning and end of an audio signal, but can also happen in the midst of longer audio signals. In its basic form, this can be achieved by removing any sample below an absolute value of a threshold (e.g. ± 0.001). However, advanced models like Silero VAD8 can be used as part of the torchaudio framework, and include more sophisticated options (e.g. minimum silence duration) to avoid removing important lowamplitude samples like breath and natural silent durations.
During training, data processed with VAD can either substitute the original data or augment it, i.e. both processed and unprocessed data can be used during training. In one of our experiments (cf. Section 4), we used basic VAD with a threshold of ± 0.001 as a data augmentation technique. When basic VAD is used (i.e. without taking a minimum silence duration into account), this can also speed up the audio signal; in other words, the utterance is spoken faster. At inference time of all the models, we used Silero VAD within Faster-Whisper based on CTranslate2 (Klein et al., 2020).
8https://github.com/snakers4/silero-vad

heading: 3.3.2 Noise Augmentation
text: Mimicking the effect of white noise can take diverse forms, ranging from using real noise to generating random arrays. To simulate light white noise, we generated a random array with a distribution scale 0.002 and added it to all the audio signals in the dataset.

heading: 4 Experiments
text: Our experiments fine-tune Whisper (Radford et al., 2022) for the task of Irish-to-English speech translation. We experiment with a number of data augmentation techniques, such as speech back-translation (source-side synthetic audio data generation), and audio data augmentation with noise and VAD.

heading: 4.1 Speech Back-Translation
text: By the term “speech back-translation”, we refer to generating source-side synthetic audio for data augmentation of speech translation systems, in the same manner that back-translation is employed in text-to-text MT systems. Section 3.2 explains how we created these synthetic audio datasets. In this set of experiments, we built 3 systems by finetuning Whisper Medium. We use different types of datasets as outlined by Table 1.
• Model A: It uses the authentic data only, namely IWSLT-2023 dataset, FLEURS, Bitesize, and SpokenWords.
• Model B: It uses the same authentic data used in Model A as well as two synthetic audio datasets, namely Tatoeba-Speech-Irish, and Wikimedia-Speech-Irish.
• Model B++: In addition to the authentic and two synthetic datasets used in the aforementioned models, Model B++ uses a third synthetic dataset, namely EUbookshop-SpeechIrish.

heading: 4.2 Noise and VAD Augmentation
text: • Model C: It uses the same data as Model B, as well as two versions of the data augmented with basic VAD, and white noise. In other words, we fine-tuned Whisper-Medium on all the authentic data and two synthetic data as well as two augmented datasets, one with lowamplitude sample removal, and one with noise augmentation, as described in Section 3.

heading: 4.3 Training Arguments
text: We tried different learning rates and warm-up values. Specifically, we experimented with warm-up ratios 0%, 1%, and 3% out of 3000 steps, which corresponds to 0, 30, 90 warm-up steps, respectively. As Table 5 and Table 4 demonstrate, when finetuning Whisper Small, changing the warm-up ratio does not seem to lead to a consistent improvement for the first two sizes of data used in Model A and Model B. However, increasing the warm-up ratio to 3% when the size of data is larger as in Model C, seems to slightly improve the performance. For the learning rate, we used 1e-4 across all the experiments for the sake of consistency. The batch size was decided based on the compute capacity of one A100-SXM4-80GB GPU. Hence, we used a batch size of 64 examples when fine-tuning Whisper Small and a batch size of 16 examples when fine-tuning Whisper Medium. The max length of generation was set to 225. As this is an Irish-toEnglish translation task, both the tokenizer language and model generation language were set to English. We train the main models with Whisper Medium for at least two epochs, and save the best performing checkpoint based on the chrF++ score on the validation dataset. Section 5 elaborates on the results of these experiments.

heading: 4.4 Training Epochs
text: As we reported in the previous section, we used 3000 steps for all the experiments with Whisper Small, as further training did not seem to improve the output quality when more than one epoch of data is already reached. However, Whisper Medium was trained with a smaller batch size due to computing constrains. We wanted to see the effect of training for at least two epochs. Hence, we report different step milestones in Table 6. In deep learning training in general, it is a common practice to use early stopping. However, for lowresource languages, a smaller value for early stopping can result in the model not seeing the whole data, which can affect the robustness of the model. This is especially true if we are not sure if the validation dataset is well-representative of the task that the model will be actually required to tackle in the real world. While there is no one rule that applies to all cases, we recommend taking this point into consideration when training generic models for low-resource languages.

heading: 5 Evaluation and Results
text: To evaluate our systems, we calculated BLEU (Papineni et al., 2002), chrF++ (Popović, 2017), and TER (Snover et al., 2006), as implemented in the sacreBLEU library9 (Post, 2018). For semantic evaluation, we used an embedding-based approach, calculating and comparing cosine similarity between the vector embeddings of each reference and the equivalent translation generated by the model. We report the average of semantic similarity across utterances. We used two models with SentenceTransformers (Reimers and Gurevych, 2019), “allmpnet-base-v2” (Semantic 1) and “all-MiniLML12-v2” (Wang et al., 2020) (Semantic 2). As we fine-tuned all the models for approximately two epochs, we report the evaluation of the best performing checkpoint.
For inference, we used Faster-Whisper 10 with the default VAD arguments. We also compared the results without VAD, and found that applying VAD at inference time is better for all the models (cf. Appendix A). We used 5 for “beam size” and 2 for “no repeat ngram size”.
As Table 2 shows, after fine-tuning Whisper Medium on both the authentic and synthetic audio data (Model B), there are consistent improvements across all metrics compared to when we finetuned it on the authentic audio data only (Model A). Moreover, Model B++ that uses three synthetic datasets outperforms Model B that uses only two synthetic datasets. This demonstrates that augmented authentic audio data with high-quality synthetic audio data can enhance end-to-end speech translation systems, especially for low-resource languages like Irish.
Model C uses the same training data as Model B
9https://github.com/mjpost/sacrebleu 10https://github.com/SYSTRAN/faster-whisper
as well as two augmented versions, one version that applies basic VAD, removing low-amplitude samples (cf. Section 3.3.1) and another version that injects white background noise into the data (cf. Section 3.3.2). Although Model C that uses noise and VAD augmented data still outperforms Model A that uses authentic training data only, both Model B and B++ that combines authentic data with synthetic data outperform Model C.
While the choice of augmentation techniques were based on manual observation of the characteristics of the authentic data and the synthetic data, the achieved improvements encourage further investigation. In the future, we would like to conduct more experiments that employ other data augmentation techniques. Moreover, we would like to measure the effect of adding synthetic audio data compared to augmenting the authentic data only. Finally, as the main purpose of this research is to understand the best practices of using synthetic audio data (i.e. data generated by TTS models) to improve speech translation quality, we will conduct further study on mimicking authentic data characteristics to enhance the effect of data augmentation with synthetic audio data.

heading: A Appendix: Arguments
text: A.1 Inference VAD
Argument Type Value Argument Type Value
threshold float 0.5 min_silence_duration_ms int 2000
min_speech_duration_ms int 250 window_size_samples int 1024
max_speech_duration_s float float("inf") speech_pad_ms int 400
Table 3: Default VAD values of Faster-Whisper.
A.2 Training Warm-up Ratio
Whisper Model Datasets Data Size Warm-up BLEU ↑ chrF++ ↑ WER ↓ Semantic 1 ↑ Semantic 2 ↑
A.3 Training Epochs

heading: Future Work
text: In the future, we would like to conduct more experiments that employ other data augmentation techniques. Moreover, we would like to measure the effect of adding synthetic audio data compared to augmenting the authentic data only. Finally, as the main purpose of this research is to understand the best practices of using synthetic audio data (i. e. data generated by TTS models) to improve speech translation quality, we will conduct further study on mimicking authentic data characteristics to enhance the effect of data augmentation with synthetic audio data