Pause-Aware Automatic Dubbing using LLM and Voice Cloning (2024)

Abstract: Automatic dubbing aims to translate the speech of a video into another language, ensuring the new speech naturally fits the original video. This paper details Huawei Translation Services Center’s (HW-TSC) submission for IWSLT 2024’s automatic dubbing task, under an unconstrained setting. Our system’s machine translation (MT) component utilizes a Transformerbased MT model and an LLM-based post-editor to produce translations of varying lengths. The text-to-speech (TTS) component employs a VITS-based TTS model and a voice cloning module to emulate the original speaker’s vocal timbre. For enhanced dubbing synchrony, we introduce a parsing-informed pause selector. Finally, we rerank multiple results based on lip-sync error distance (LSE-D) and character error rate (CER). Our system achieves LSE-D of 10.75 and 12.19 on subset1 and subset2 of DE-EN test sets respectively, superior to last year’s best system.

heading: 1 Introduction
text: The task of automatic dubbing is to translate spoken language in a video into another language such that the translated speech can be seamlessly blended with the original video. A unique aspect of dubbing is isochrony, which refers to the property that the speech translation is time-aligned with the original speaker’s visual cues. The spoken words should match the speaker’s lip movements, ensuring the audio is heard when the lips move and is silent when they don’t.
To address this challenge, a unified model that simultaneously processes translations and speech timing is optimal, allowing for adjustments in translation to fit timing constraints. Chronopoulou et al. (2023) accomplish this by simply binning target phoneme durations and interleaving them with target phonemes during training and inference. Pal et al. (2023) enhance this approach by predicting
the durations of phonemes as target factors. However, these methods fail to utilize pre-trained machine translation (MT) models and large language models (LLM) that are trained on massive text corpora. Moreover, constructing large-scale datasets with phoneme duration labels is challenging, thus limiting the translation quality. Therefore, a disentangled approach that considers MT and dubbing synchrony separately can achieve better results. Our system (Rao et al., 2023) from last year first generated a set of translation candidates and later reranked them based on speech overlaps, achieving better mean opinion scores (MOS) than the baseline systems. Therefore, this year we extend last year’s system by using more advanced pre-trained models and a more sophisticated pause-aware dubbing pipeline.
Specifically, our method comprises the following key components:
• A Transformer-based MT (Machine Translation) model, which is a fine-tuned version of NLLB-1.3B on the CoVoST2 dataset (Changhan Wang, 2020).
• An LLM-based post-editor that modifies the lengths of translations.
• A VITS-based (Kong et al., 2023) TTS model that is non-autoregressive and supports speed control.
• A voice cloning (VC) module based on OpenVoice (Qin et al., 2023), ensuring that the input speech and output speech share the same tone color.
• A pause-aware dubbing pipeline that identifies potential split points using sentence parsing.
• A reranking method based on LSE-D and CER.
In this paper, we provide detailed analyses of the components mentioned above. Our system
1 60
achieves an LSE-D of 10.75 on subset1 and 12.19 on subset2 of the DE-EN test sets, respectively, outperforming last year’s best system. Additionally, we take into account the vocal timbre of the speech, which can enhance the perceptual quality.

heading: 2 Methods

heading: 2.1 System Design
text: Figure 1 (a) shows the naive automatic dubbing system which assumes that the speech of the video does not have obvious pauses. First, an automatic speech recognition (ASR) model transcribes the source speech. The result of this step is provided by the organizer. Then, an MT model translates the source German (DE) text into the target English (EN) text, followed by an LLM that is prompted to change the length of the translation. Utilizing MeloTTS 1, the target speech is synthesized and its duration is compared with the original speech to determine the speed factor. Finally, we regenerate the target speech, convert the tone color, and fill the audio into the original video based on timestamps from the voice activity detection (VAD) 2 system.
Figure 1 (b) illustrates the pause-aware dubbing system. Unlike the naive system, it integrates a pause selector. This selector generates an index of potential word positions that best align with the pauses in the original speech. To avoid unnatural sentence breaks, sentence parsing is employed to determine groups of words that should remain together. Finally, the TTS model is utilized to produce audio clips for each text segment, with the speed factor calculated for each independently.
1https://github.com/myshell-ai/MeloTTS 2https://github.com/snakers4/silero-vad
Algorithm 1 Pause Selector Require: tpause, textMT = {w1, ..., wn}
split = {t1, ..., tn−1}, Tsrc, Ttts 1: PP, V P,NP = Parsing(textMT ) 2: index = SplitPoint(PUNC,PP, V P,NP )
3: i = argmin(abs( tpause Tsrc − splitiTtts )) i ∈ index
4: return i

heading: 2.2 Pause Selector
text: Algorithm 1 provides the details of the pause selector. Given the time of the pause (tpause) predicted by VAD, the translation (textMT ), the word-level timestamps (split) of synthetic speech predicted by a CTC-based aligner from WhisperX (Bain et al., 2023), and the duration of source and generated speech (Tsrc) and (Ttts), we first use sentence parsing 3 to obtain the prepositional phrases (PP), verb phrases (VP), and noun phrases (NP). The possible split index can be only after these phrases and punctuations. Then, we select the best index that minimizes the distance between the normalized word time by duration and the normalized time of the pause.

heading: 2.3 LLM-based Post-Editor
text: LLM (Touvron et al., 2023; Zeng et al., 2023) is known for its exceptional zero-shot and few-shot capabilities, meaning it can perform downstream tasks using a prompt that describes the task or a few examples. In the context of automatic dubbing, we use LLM to generate translations with different lengths so that we can select the one that results in the best lip-sync accuracy. The input prompt for the LLM is shown in Table 1. We first describe
3https://github.com/Halvani/ Constituent-Treelib
2
the task and the role of the LLM as a translator and a proofreader. Then, we instruct it to make the translation more complex or simple. Finally, we provide the source and translated text. We use "complex" and "simple" as indicators of output length as they contribute to better stability than "longer" and "shorter".

heading: 2.4 TTS and VC
text: We use MeloTTS, which is based on the architecture of VITS (Kim et al., 2021; Kong et al., 2023). VITS leverages variational autoencoder, adversarial learning, normalizing flow, and stochastic duration predictor to generate realistic speech in an end-to-end manner without relying on external word alignment and a vocoder. To convert the voice into the desired tone color, we adopt OpenVoice (Qin et al., 2023), which disentangles the tone color information in the encoder. The target speaker embedding is integrated into the decoder.

heading: 2.5 Rerank
text: We use LSE-D and CER to select the final synthetic speech from multiple candidates. The CER is computed between the original ASR transcription and the transcription of the generated speech. For subset1, there are no obvious pauses, so we only use system (a) as shown in Figure 1. For subset2, which contains notable pauses, we use both systems (a) and (b) in Figure 1. For the same translation, we only use system (b) if we do not observe a decline in CER and note an improvement in LSE-D compared to system (a). To rank multiple translations of different lengths, we use the average rank determined by LSE-D and CER and select the translation with the lowest rank. Note that we use CER rather than word error rate to mitigate the influence of the ASR model’s limited ability to recognize out-of-vocabulary words. During rerank, we considered four translations: the original translation, the translation using LLM-based post editor without indicating the output length, the "complex" translation, and the "simple" translation.

heading: 3 Experimental Setups
text: We fine-tuned the NLLB-1.3B 4 model for 20 epochs on the CoVoST2 (Changhan Wang, 2020) DE-EN subset, using a learning rate of 3×10−5 and a batch size of 512. For the LLM-based post-editor,
4https://huggingface.co/facebook/ nllb-200-distilled-1.3B
when employing the “complex” indicator, we sampled three answers and selected the one with the highest Comet score (Rei et al., 2020) compared to the original translation. For the “short” indicator, we sampled only once. When adjusting the speech speed, we set the lower bound to 0.75× and the upper bound to 2.5×. We adopted several evaluation metrics: the BLEU score and the Comet score to evaluate MT quality, and the lip-sync error distance (LSE-D) (Chung and Zisserman, 2017) 5 and ASR character error rate (CER) to measure dubbing performance. We used the Wav2Vec2base model 6, fine-tuned on LibriSpeech, as the ASR model, which utilizes a character-level vocabulary. We opted not to use a more advanced ASR model since the less robust model is more sensitive to speech quality. During rerank, we considered four translations: the original translation, the translation using LLM-based post editor without indicating the output length, the "complex" translation, and the "simple" translation. Additionally, we attempted to enhance the speech by applying denoising and audio super-resolution techniques 7, which remove noise and upscale the audio from 16kHz to 44.1kHz.

heading: 4 Experimental Results

heading: 4.1 Performance of MT and LLM-based Post-Editor
text: As shown in Table 2, the NLLB-1.3B model, fine-tuned on the target-domain CoVoST2 dataset, achieves high translation quality with BLEU scores of 46.37 and 44.03 on subset1 and subset2, respectively, and Comet scores of 89.29 and 88.01, respectively. When using an LLM to post-process the translations, we observe a decrease in BLEU scores, especially for longer translations. However, we find that the Comet scores are similar to those of the unmodified translations, indicating that the LLM effectively performs paraphrasing without changing the meaning of the translations.

heading: 4.2 Results for Pause-Aware Automatic Dubbing
text: For subset2, we observe that the pause-aware automatic dubbing pipeline (Dubbing (b)) contributes
5When computing LSE-D, we used the video with subtitles.
6https://huggingface.co/facebook/ wav2vec2-base-960h
7https://github.com/resemble-ai/ resemble-enhance
3
to a significantly lower LSE-D than the naive pipeline (Dubbing (a)). For instance, with pauseaware dubbing, the LSE-D decreases from 13.88 to 12.27 for the original translation. However, there is an increase in CER. The possible reason could be that the pauses in the translation may be unnatural, or the TTS model’s ability to generate speech for incomplete sentences is limited. Therefore, we combine the two systems. For the same translation, we only use system (b) if we do not observe a decline in CER and note an improvement in LSE-D compared to system (a). This combination method (Dubbing (a + b)) results in the lowest CERs, and the LSE-D is also notably better than the naive system (a).

heading: 4.3 Results for Rerank
text: LSE-D measures the synchronization of speech with video, while CER assesses speech intelligibility. Employing either metric for reranking could enhance the results according to their respective evaluations. Using their average rank can achieve a balance between them. For subset1 and subset2, the final dubbed videos achieve LSE-D scores of 10.75 and 12.19, respectively, and CERs of 4.62% and 3.02%, respectively. It is worth noting that the CER for longer speeches tends to be lower due to more contextual information, while the LSE-D tends to be higher as it is more difficult to align the pauses.

heading: 4.4 Alternative Systems
text: We carried out ablation studies and provided alternative systems in our submission. When VC is not used, the LSE-D is similar to the complete system. The CER is notably lower because the sole TTS model provides better speech quality, whereas the VC model can introduce some noise. How-
ever, without VC, using a female’s voice for a male speaker is unreasonable. Our TTS model operates at a sample rate of 16kHz. To improve the subjective listening experience, we adopted an audio super-resolution model to enhance it to 44.1kHz. Perceptually, higher frequencies contribute to better quality. However, we found that audio superresolution negatively impacts the LSE-D and CER, although we do not observe noticeable distortion in the audio samples.

heading: 5 Discussion
text: Compared to last year’s system, which utilized a length-aware MT system that employed a length tag to indicate the desired output length, this year’s approach aims to enhance translation quality by fine-tuning a pre-trained MT model rather than training one from scratch. Although we attempted to incorporate length tags in the fine-tuning process, we found that they failed to produce translations with varying lengths due to the limited number of epochs and fine-tuning data. Consequently, we used an LLM which has robust rewriting capabilities.
We submitted a single entry for the EnglishChinese subtask, which presents significantly greater challenges than the German-English subtask due to factors such as long-form video, speaker changes, and background music. To address these challenges, we enhanced our automatic dubbing system with an open-source diarization model (Desplanques et al., 2020), a source separation tool (Takahashi and Mitsufuji, 2017), and a TTS API 8. However, given the complexity of the task and the lack of labeled test set, we have not provided a detailed analysis. In movie dub-
8https://github.com/rany2/edge-tts
4
bing, it is crucial that the emotion of the dubbed speech matches that of the original speech, therefore, expressive TTS models are preferred. We evaluated the Seamless Expressive model (Barrault et al., 2023), however, we observed that the speech quality was inconsistent, and for non-English languages, the speech did not sound native.

heading: 6 Conclusion
text: In this paper, we propose a novel pause-aware automatic dubbing system that ensures translated speech signals are not only accurate but also maintain the timbre of the original speech. The key components involve a novel pause selector, informed by parsing, to align dubbing with the video’s pace, a VC model to convert the tone color, and an LLM to provide translation candidates. For future work, we plan to carry out more systematic experiments on long-form, movie-like videos and provide more expressive dubbed videos.

heading: Future Work
text: For future work, we plan to carry out more systematic experiments on long-form, movie-like videos and provide more expressive dubbed videos