From Speaker to Dubber: Movie Dubbing with Prosody and Duration Consistency Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 08:18:28

Abstract: Movie Dubbing aims to convert scripts into speeches that align with the given movie clip in both temporal and emotional aspects while preserving the vocal timbre of one brief reference audio.
The wide variations in emotion, pace, and environment that dubbed speech must exhibit to achieve real alignment make dubbing a complex task.
Considering the limited scale of the movie dubbing datasets (due to copyright) and the interference from background noise, directly learning from movie dubbing datasets limits the pronunciation quality of learned models.
To address this problem, we propose a two-stage dubbing method that allows the model to first learn pronunciation knowledge before practicing it in movie dubbing.
In the first stage, we introduce a multi-task approach to pre-train a phoneme encoder on a large-scale text-speech corpus for learning clear and natural phoneme pronunciations.
For the second stage, we devise a prosody consistency learning module to bridge the emotional expression with the phoneme-level dubbing prosody attributes (pitch and energy).
Finally, we design a duration consistency reasoning module to align the dubbing duration with the lip movement.
Extensive experiments demonstrate that our method outperforms several state-of-the-art methods on two primary benchmarks.
The source code and model checkpoints will be released to the public.
The demos are available at https://speaker2dubber.github.io/.

Main paper: 
 introduction: 
 Movie Dubbing, also known as Visual Voice Cloning (V2C) [4] , aims to convert scripts into speeches utilizing a specified voice guided by a short reference audio while maintaining lip synchronization and reflecting the character's emotions in the given movie clip (as shown in Figure 1 (a) ). It requires the ability to bridge the visual, natural language, and speech modalities, which brings significant challenges.Unlike traditional voice cloning (VC) [1, 3] or text-to-speech (TTS) [19, 25, 28, 33] tasks solely depending on the input text for modeling [38] , movie dubbing requires maintaining consistency between its variation information (e.g., prosody attributes, duration of each phoneme) and the performances of movie characters. This transforms the modeling of duration and prosody in movie dubbing from a one-to-many to a one-to-one mapping problem while retaining the requirement for accurate pronunciation.Firstly, regarding duration consistency, movie dubbing needs to match the total duration of the video and further synchronize the lip movements. To this end, previous works incorporate lip motion into the prediction of the duration for each phoneme [4] or utilize an attention-based upsampling module mapping the phoneme sequence to the video frame length to ensure overall duration consistency [5, 12] . However, the former standalone duration prediction for each phoneme overlooks constraints imposed by the total length of the video, while the latter breaks the completeness of a speech unit, leading to unclear pronunciation.Secondly, in terms of prosody modeling, to reflect the emotions expressed by characters in the video, previous works [4, 12, 23] model the prosody attributes (e.g., pitch and energy) through the emotional expressions of movie characters at the mel-spectrogram frame-level. The excessively fine prosody modeling scale degrades the model's generalization capability [28] , resulting in unstable prosody fluctuations. Furthermore, previous approaches that directly fuse timbre features with phoneme sequences also fail to achieve satisfactory voice cloning effects [4, 5, 12, 23] .Besides, the required audiovisual consistency in movie dubbing emphasizes the comprehensive application of accurate and clear pronunciation. However, movie dubbing datasets are limited in scale (due to copyright issues) and usually involve complicated pronunciation and prosody. The presence of background sounds and ambient noise inherent in movies further increases the difficulty of modeling accurate and clear pronunciation. Existing works [4, 5, 12, 23] try to simultaneously learn pronunciation from scratch and achieve the alignment between dubbing and movie clip solely relying on movie dubbing datasets (as shown in Figure 1 (b) blue arrows), leads to poor pronunciation quality. This demonstrates that the model attempting to leap directly from "baby" to dubber solely depends on movie dubbing datasets is quite difficult and inefficient.To address the above-mentioned problems, we propose a novel two-stage dubbing method (as shown in Figure 1 (b) green arrows) that better aligns with the practicing process from speaker to dubber. The two-stage framework allows the model to first learn the pronunciation knowledge via multi-task speaker pre-training and then practice synchronizing with movie content to achieve prosody and duration consistency. Specifically, in the first stage, we introduce the multi-task pre-training to learn universal pronunciation representation from a large-scale text-to-speech corpus. It overcomes the limitation of movie dubbing datasets and enhances the pronunciation clarity of dubbing. In the second stage, we design a prosody consistency learning module to bridge the visual emotion features with phoneme-level prosody attributes. Furthermore, timbre-adaptive layer normalization (TALN) is introduced to maintain the timbre consistency with reference audio. Last, we propose a duration consistency reasoning module to predict the optimal monotonic alignment between phoneme-level acoustic features and video frame-level lip motion. This module aims to synchronize dubbing with the movie clip in terms of lip movement and overall duration. Finally, the mel-decoder [34] is used to predict the mel-spectrogram of dubbing, and the vocoder [22] converts the mel-spectrogram into a waveform in the time domain.The main contributions are summarized as follows:• We propose a two-stage framework for movie dubbing that enables the model to learn pronunciation from large-scale text-to-speech corpus before practicing in movie dubbing, therefore improving the quality of dubbing pronunciation. • To better achieve prosody consistency, we propose a prosody consistency learning module to bridge the character's emotion with the phoneme-level prosody attributes and utilize TALN to preserve the vocal timbre of reference audio. • To achieve duration consistency, we propose a duration consistency reasoning module to predict the optimal alignment between the phoneme-level acoustics features and lip movement in the video.• Extensive experiments demonstrate that our proposed method outperforms the current state-of-the-art, validating the effectiveness of each module. 
related works 2.1 speech synthesis: 
 Speech synthesis, or text-to-speech, plays a crucial role in various applications, including virtual assistants, navigation systems, accessibility tools, and entertainment media. In recent years, numerous advancements have been made in speech synthesis. WaveNet [40] and Tacotron [35] architectures demonstrate remarkable performance in generating high-quality and expressive speech in an autoregressive way. Then, the Fastspeech series [33, 34] achieve high-quality and rapid speech synthesis by expanding phoneme sequences to the same length as mel-spectrogram using Montreal forced alignment (MFA) [26] and length regulator. Recently, a proliferation of models based on diffusion [14, 31] or generative flow [9, 20] , aiming to generate natural and human-level speech [8, 13, 16, 19, 42] . To further improve the naturalness of synthesized speech, the NaturalSpeech series [17, 36, 38] employ elaborate designed neural speech codecs for speech attribute factorization and a factorized diffusion model to build fine-grained speech attributes. Despite the impressive progress, these methods cannot be directly applied to the V2C task as they lack the required prosody and duration consistency modeling with input move clip.Pre-trained language models (e.g., BERT [7] ) achieve state-of-theart performance in solving natural language processing tasks. Since training a TTS model is like learning a language from scratch [24] , recently many works [11, 15, 24, 29, 45] employ additional BERT encoders to enhance the pronunciation quality and expressiveness of the generated speech. In order to enable the BERT to operate at the phoneme level sequence avoiding the inconsistency between phoneme and character information, MP-BERT [45] merges the phoneme and sup-phoneme sequences into a new sequence, PL-BERT [24] combines whole-word masked phoneme and grapheme prediction as pre-train strategy, both achieve promising results. Pretraining of masked language models provides abundant phonemelevel contextual information which can enhance the naturalness of speech. Therefore, we incorporate this pre-train strategy into the movie dubbing task to enhance the naturalness of dubbing.The V2C task [4] introduces video information as one of the conditions for speech synthesis, requiring the generated dubbing to align with the video content in terms of lip movements, emotions, and duration. This eliminates the one-to-many problem in traditional speech synthesis tasks and significantly increases the complexity of the task. Building on this, a hierarchical prosody learning model is proposed [5] , which adapts the prosody of dubbing by learning at three levels: lip movements, character facial information, and global scene information, at the level of video frames. Recent works [12, 23] try to solve the dubbing task using human-face as the timbre identity features using the non-autoregressive model or diffusion model. Although previous works make progress in improving the consistency between dubbing audio and movie clips, the complexity and limited scale of movie dubbing datasets constrain the pronunciation accuracy of learned models.  
methods 3.1 overview: 
 The target of the overall movie dubbing task is:EQUATIONwhere the Ã𝐷𝑢𝑏 is the generated dubbing and 𝐴 𝑅𝑒 𝑓 ,𝑇 𝑠 , 𝑉 𝑅𝑒 𝑓 are the reference audio, scripts, and reference video, i.e., movie clip respectively.The main architecture of the proposed method is shown in Figure 2. Our method involves two training stages: multi-task speaker pre-training (MTSP) (top) and dubbing training with prosody and duration consistency learning (bottom). In the first stage, we conduct a multi-task speaker pre-training for the phoneme encoder. Since conventional speech and movie dubbing differ significantly in tone and prosody but share common pronunciation semantics, pretraining the phoneme encoder is the optimal choice. Compared to movie dubbing datasets, the recording environment of text-speech corpus is often a noise-free recording studio. The prosody and pronunciation in the corpus are smooth and clear. Therefore it can offer rich pronunciation knowledge for movie dubbing.In the second stage, we freeze the pre-trained phoneme encoder and train the model using the movie dubbing datasets. We propose the Prosody Consistency Learning (PCL) module and the Duration Consistency Reasoning (DCR) module. These modules enable the model to better practice pronunciation skills thus achieving prosody and duration consistency with the movie clip. They respectively align the emotion and lip movement of the character with the phoneme-level prosody and duration of the dubbing. Additionally, in the PCL module, we employ a mel-timbre encoder and timbreadaptive layer normalization (TALN) to extract and integrate the timbre feature of the reference audio, thereby ensuring timbre consistency. We detail each module below.The pre-training consists of two tasks: the TTS task and the MLM prediction task. The former enables the model learns to extract accurate pronunciation representations from high-quality text-speech corpus and the latter facilitates the model to learn contextual relationships between phonemes and better handle unseen text. TTS task. We adopt an architecture similar to the FastSpeech2 [33] in this stage, where the model primarily consists of three main components for generating the mel-spectrogram of speech, namely the phoneme encoder, variance adaptor, and mel-decoder. Specifically, after the text is converted into phoneme sequence by an opensource grapheme-to-phoneme tool1 (G2P), it serves as the input to the model. Subsequently, a phoneme encoder composed of stacked Feed-Forward-Transformer (FFT) blocks [34] is utilized to extract phoneme embeddings, and the variance adaptor is employed to model the prosody attributes of speech, namely pitch, energy, and the duration of each phoneme. Then transform them into acoustic features with the desired length as the mel-spectrogram as follows:EQUATIONwhere 𝑇 𝑜 , 𝑇 𝑝 , and 𝑇 𝑒 are the original text, phoneme sequence, and phoneme embedding, respectively. 𝐿 𝑝 and 𝐿 𝑚𝑒𝑙 denote the length of the phoneme sequence and desired mel-spectrogram, respectively. 𝑑 𝑚 denotes the hidden size of the model. 𝐷 𝑝 , 𝑃 𝑝 , and 𝐸 𝑝 denote the duration and prosody attributes of each phoneme, namely pitch and energy. 𝑇 𝑚𝑒𝑙 denotes the expended phoneme-level acoustic sequence with prosody attributes.It is worth noting that in the original TTS task, the duration, pitch, and energy of each phoneme are predicted by the variance adaptor directly from the phoneme sequence. However, in the movie dubbing task, the duration and the prosody attributes of dubbing need to be bridged with visual information, such as the lip movements and emotional expressions of characters. Since no matching visual information is available in the text-speech corpus, we directly use the ground truth duration, pitch, and energy information in the TTS pre-training task. Furthermore, to enable the model to better adapt to a multi-speaker environment during the pre-training phase, we also employ the mel-timbre encoder and TALN at this stage. We provide a detailed explanation of these components in Section 3.3. The omission here does not affect the description of the pre-training process.Then, we predict the mel-spectrogram of target speech and calculate the loss of this task:EQUATIONwhere the mel-decoder is composed of stacked FFT blocks [34] similar to the phoneme encoder, 𝑀 and M denote the ground truth mel-spectrogram and the predicted one, respectively. Subscript 𝑡 denotes the 𝑡-th frame. MLM prediction task. The MLM prediction task enables the model to enhance the naturalness of dubbing audio by learning phoneme contextual relationships in a self-supervised manner. Specifically, we convert the text in the corpus into phoneme sequences and then randomly mask the sequences with a predetermined masking ratio. After inputting the phoneme sequences into the phoneme encoder, we predict the masked input phoneme tokens from the hidden states of the last layer using a linear projection along with a softmax function. The loss function is the cross-entropy loss commonly used for multi-class prediction. The training target of the MLM prediction task is:EQUATIONwhere the 𝑇 𝑚𝑎𝑠𝑘 is the masked phoneme sequence and 𝐶𝐸 (•) denotes the cross-entropy loss.Loss function of the first stage. During the multi-task speaker pre-training, the TTS task and MLM prediction task are conducted separately. The masking in the MLM prediction task does not affect the encoding of phonemes in the TTS task. The total loss for the first stage is as follows:EQUATIONwhere the 𝛼 1 and 𝛼 2 are pre-defined hyper parameters.The Prosody Consistency Learning (PCL) module consists of two parts: 1) Bridging the emotional states of characters in movie clips with the phoneme-level prosody attributes of dubbing and 2) accurately replicating the timbre of the reference audio.Bridging emotion with dubbing. In movie dubbing, the consistency between the dubbing's prosody and the characters' emotions is crucial. A professional dubber always observes the characters' facial expressions in the movie to adjust their pronunciation prosody.To enable the model to achieve this, we model the phoneme-level prosody attributes pitch and energy based on the facial information of the characters in the video. Following [5] , we exact the face region of each frame of movie clip 𝑉 𝑓 𝑎𝑐𝑒 via S 3 FD face detection [46] , then utilize an emotion face-alignment network (EmoFAN) [39] to encode the face region to emotion features 𝑉 𝑒𝑚𝑜 :EQUATIONwhere the 𝐿 𝑣 denotes the frame number of the reference video.To model the pitch and energy of each phoneme based on the emotion from video frames, we utilize multi-head cross-modal attention to bridge the character's emotion features with prosody attributes of each phoneme:EQUATIONwhere 𝑘 denotes 𝑘-th head's output,𝑊 * 𝑗 are learnable parameter matrix, 𝜉 𝑝ℎ𝑜,𝑝𝑖𝑡𝑐ℎ and 𝜉 𝑝ℎ𝑜,𝑒𝑛𝑒𝑟𝑔𝑦 are phoneme-level prosody feature for pitch and energy, respectively. After bridging the prosody of dubbing with the character's emotion, we predict the pitch energy of each phoneme using the pitch and energy predictor and transfer them to pitch and energy embedding then add to the phoneme sequence:EQUATIONwhere 𝑃 𝑝ℎ𝑜 and 𝐸 𝑝ℎ𝑜 are the predicted pitch and energy and 𝑇 𝑎 ∈ R 𝐿 𝑝 ×𝑑 𝑚 are the phoneme-length acoustics feature with prosody attributes. Timbre consistency. To keep the timbre consistency within the same character and ensure a coherent and consistent understanding of the characters by the audience, we propose a mel-timbre encoder to extract the timbre feature from reference audio.The mel-timbre encoder comprises three parts: (1) a fully connected layer to transform each frame of the mel-spectrogram into the hidden sequence of acoustics features, (2) a temporal processing module to capture contextual information using a stacked 1-D convolutional neural network, and (3) a multi-head attention module to aggregate global information of vocal timbre. Due to the issue of poor generalization of learnable timbre encoder in movie dubbing datasets, we pre-train the mel-timbre encoder on the multi-speaker text-speech corpus [44] using GE2E loss [41] and froze its weights during both stages of training unlike [28] .Inspired by the style transfer in the image and speech generation domain [18, 28] , we utilize a Timbre-Adaptive Layer Norm to fuse the timbre feature into phoneme encoding and mel-spectrogram generation by predicting the gain and bias of the input vector sequence giving the timbre feature:EQUATIONwhere 𝑥 and 𝐸 𝑡𝑖𝑚𝑏𝑟𝑒 are the input sequence and timbre feature, 𝜇, 𝜎 are the mean and variance of 𝑥, 𝑔𝑎𝑖𝑛(•) and 𝑏𝑖𝑎𝑠 (•) are the prediction function of gain and bias, respectively. The proposed TALN is applied in every FFT block of both the phoneme encoder and mel-decoder to integrate the vocal timbre feature.The proposed Duration Consistency Reasoning module (DCR) contains two steps: (1) reasoning the phoneme-lip alignment and (2) expanding to the length of desired mel-spectrogram.Reasoning the phoneme-lip alignment. To exact lip motion features from the reference video, we first extract the lip region 𝑉 𝐿𝑖𝑝 ∈ R 𝐿 𝑣 ×𝐻 𝐿𝑖𝑝 ×𝑊 𝐿𝑖𝑝 ×𝐶 from the video, then exploit a lip motion encoder to obtain the lip movement representation 𝐸 𝐿𝑖𝑝 ∈ R 𝐿 𝑣 ×𝑑 𝑚𝑜𝑑𝑒𝑙 . After obtaining the lip motion features from the movie clip, we can calculate the upper triangular similarity matrix between phoneme-level acoustic features and lip-motion features:EQUATIONwhere the 𝑠 𝑖,𝑗 is the cosine similarity between 𝑖-th phoneme-level acoustic feature and 𝑗-th frame's lip motion feature.The alignment between lip phoneme and lip motion should satisfy surjectivity, which means the alignment path must begin at 𝑠 11 and end at 𝑠 𝐿 𝑝 𝐿 𝑣 . Additionally, this alignment also needs to satisfy monotonicity, each step of the alignment path can only move to the right or diagonally down to the right, thus we can use dynamic programming algorithms to find the optimal alignment:EQUATIONEQUATION)where 𝐴 is the alignment matrix and 𝐴 * is the optimal alignment between phoneme-level acoustic and lip-motion features which simultaneously satisfy monotonicity and surjectivity.Expanding to the mel-length. The duration of the video corresponds strictly to the duration of the audio, hence, the number of video frames corresponds one-to-one with the length of the corresponding mel-spectrogram, maintaining a fixed ratio relationship [5, 12] :EQUATIONwhere the 𝑠𝑟 and ℎ𝑠 are the sampling rate and hop size when processing the audio and 𝐹 𝑃𝑆 denotes the Frames Per Second of the video. With this fixed coefficient 𝑛, we can extend the obtained phonemelip alignment to phoneme-mel-spectrogram alignment, acquiring the duration of each phoneme. Then we expand the acoustics feature sequence of phoneme length to the required mel-spectrogram length using length regulator [34] for mel-spectrogram generation and audio generation: Ã𝐷𝑢𝑏 = Vocoder(MelDecoder(LR(𝑇 𝑎 , 𝐴 * × 𝑛), 𝐸 𝑡𝑖𝑚𝑏𝑟𝑒 )), (14) where the LR(•) denotes the length regulator.The total loss function of this training stage is:EQUATIONwhere 𝜆 * are pre-defined hyper-parameters, L 𝑚𝑒𝑙 , L 𝑝𝑖𝑡𝑐ℎ , and L 𝑒𝑛𝑒𝑟𝑔𝑦 are the L1 Loss to the mel-spectrogram and the prediction of the pitch and energy of each phoneme respectively. For the alignment in duration consistency reasoning, since the dynamic programming algorithm does not have any learnable parameters, we optimize the lip motion encoder using ground truth alignment, gradually making the similarity matrix approach the ideal alignment. For detailed information refer to Appendix. 
experiments: 
 We primarily evaluate our method on two dubbing datasets: V2C-Animation [4] and GRID [6] . Here, we present our experimental results from various aspects, including dataset description, implementation details, evaluation metrics, qualitative and quantitative analysis, as well as ablation studies.V2C-Animation dataset [4] is currently the only publicly available movie dubbing dataset. Specifically, it consists of 10,217 videoaudio-text triplets cropped from 26 Disney animated movies, totaling 153 different speakers, with complete speaker and emotion annotations. The dataset extracts the middle audio track from movie clips as dubbing data but still exists some environmental sounds and noise that cannot be eliminated. Due to the authenticity of this dataset which is from publicly available movies, it's the most challenging dataset for the V2C task currently. Therefore, our main experimental results are validated on this dataset.GRID dataset [6] is a basic benchmark for multi-speaker dubbing.The whole dataset has 33 speakers, each with 1,000 short English samples. All participants are recorded in a noise-free studio with a unified screen background. The V2C-Animation dataset and the Table 1 : Results on V2C-Animation benchmark. The method with "*" refers to a variant taking video embedding as an additional input following [4] . For the Dub 1.0 setting, we use the ground truth audio as reference audio, for the Dub 2.0 setting, we use the non-ground truth audio from the same speaker within the dataset as the reference audio which is more aligned with practical usage in dubbing. The same setup is applied to the GRID benchmark. GRID dataset correspond respectively to dubbing for animated films and live-action recording, covering a wide range of application scenarios.LibriTTS dataset [44] is a multi-speaker English corpus derived from LibriSpeech [30] . LibriTTS comprises 110 hours of audio from 1,141 speakers along with their corresponding text transcripts. It filters out mismatched or significantly noisy samples found in Lib-riSpeech and is widely used in speech synthesis or automatic speech recognition (ASR) tasks. Due to its advantages in speech quality and scale as well as the similar multi-speaker settings, we employ LibriTTS as our speaker pre-training dataset in the first stage.The video frames are sampled at 25 FPS and all audios are resampled to 22.05kHz. For all audio data, we convert the raw waveform into mel-spectrograms following [4, 5, 28, 33, 34] with FFT size of 1024, hop size of 256, window size of 1024, and frequency bins of 80.The ground truth of phoneme duration is extracted by Montreal Forced Aligner [26] . We use continuous wavelet transform (CWT) to decompose the continuous pitch series into pitch spectrograms to get the phoneme-level pitch [10, 37] . For energy extraction, we compute the mean L2-norm of the amplitude of each short-time Fourier transform (STFT) frame within a phoneme duration [33] .For the MLM prediction task of the multi-task speaker pretraining in the first stage, we use a 15% random masking rate and train with a batch size of 16 for 100 epochs together with the TTS task on the LibriTTS-Clean-100 dataset. The weight in Equation 5are set to 𝛼 1 = 1, 𝛼 2 = 0.1. In the second stage of training, we freeze the pre-trained phoneme encoder and train the rest part of the model on the dubbing datasets. We use a pre-trained universal speaker version of HiFi-GAN [22] as vocoder to convert the melspectrogram to time-domain waveforms. The weight in Equation 15are set to 𝜆 1 = 1, 𝜆 2 = 0.1, 𝜆 3 = 0.1, 𝜆 4 = 0.2. An Adam [21] with 𝛽 1 = 0.9, 𝛽 2 = 0.98, 𝜖 = 10 -9 is used as the optimizer in both the training stages. The learning rate is set to 0.00625. Both training and inference are implemented with PyTorch on a GeForce RTX 4090 GPU. For a fair comparison, all comparison models are re-trained on the same dataset. For more details of training and implementation please refer to Appendix.Objective metrics. To measure the difference between the generated dubbing and the ground truth, we adopt the Mel Cepstral Distortion Dynamic Time Warping (MCD-DTW) metric following [4] . To further assess the duration consistency between the generated dubbing and the video, we utilize the MCD-DTW-SL metric which adjusts the weights based on duration consistency [4] . Furthermore, to evaluate the timbre consistency between the generated dubbing and the reference audio, we employ the speaker encoder cosine similarity (SECS) following [2, 3] to compute the similarity of speaker identity. To assess the pronunciation quality of the generated dubbing, we utilize the state-of-the-art automatic speech recognition (ASR) model whisper2 [32] from OpenAI for dubbing recognition and computing the word error rate (WER) 3against the script to evaluate the accuracy of the generated dubbing. In addition, we utilize a speech emotion recognition model [43] to evaluate the emotion accuracy (EMO-ACC) of the generated dubbing (For the V2C-Animation benchmark only because there is no emotion label in GRID dataset). Subjective metrics. For subjective evaluation, we conduct human evaluations of mean opinion score (MOS) in aspects of naturalness (NMOS) and similarity (SMOS). Both metrics are rated on a 1-to-5 scale and reported with the 95% confidence intervals (CI). Following the settings in [4, 5] , all participants are asked to assess the dubbing quality of 30 randomly selected audio samples from each test set. In addition to SMOS and NMOS, we also utilize Comparative MOS (CMOS) with 7 points (from -3 to 3) to compare different models.Results on V2C-Animation benchmark. As shown in Table 1 , our model achieves the best performance across almost all metrics, except for the EMO-ACC metric in the dub 2.0 setting, where it slightly lagged behind the current SOTA dubbing model HPM-Dubbing [5] . In both dub settings, our method achieves the best performance on SECS. It indicates that our method performs better in extracting and cloning vocal timbre. Since the V2C-Animation [5] 3.57±0.17 3.54±0.12 -0.18 3.77±0.20 3.74±0.13 -0.14 Face-TTS [23] 3.18±0. 13 In the subjective evaluation, we randomly select 15 samples from the generated dubbing of each dub setting for human study. Table 3 shows the results. All objective evaluations of our model in the V2C benchmark achieve the highest scores. It demonstrates that our model can generate dubbing closer to realistic dubbing in both naturalness and similarity. In addition, our model outperforms the current state-of-the-art dubbing model [5] in CMOS evaluation with a margin of +0.18. Results on GRID benchmark. As shown in Table 2 , our model achieves the best performance across all metrics on the GRID benchmark. Unlike V2C-Animation, samples in GRID are recorded in a studio environment which does not involve exaggerated prosody variation and background noise. Therefore, the pronunciation accuracy of all comparison methods is generally better on the GRID benchmark compared to the V2C-Animation benchmark. Nevertheless, our model still achieves the best pronunciation clarity (see WER) and the best SECS in both dub settings. The lowest MCD-DTW and MCD-DTW-SL demonstrate the ability of our model to generate dubbing closer to ground truth. Moreover, our model also achieves the highest scores in naturalness and similarity in subjective evaluations and surpasses other models in comparative evaluations with a margin of +0.14. Results of Zero-shot test. In addition to the evaluation on two benchmarks, we also conduct a zero-shot experiment to verify the robustness of our method. In the zero-shot experiment, we utilize scripts and movie clips from the V2C-Animation dataset and reference audio from the GRID dataset (i.e., out of domain) to simulate the application of generating customized dubbed videos in real-world scenarios. Due to the absence of corresponding ground truth in this test, we only calculate WER and SECS for objective evaluation to assess pronunciation quality and timbre consistency.As shown in Table 4 , our model achieves the best pronunciation clarity and timbre consistency in the zero-shot test. It demonstrates that our method can maintain stable dubbing synthesis when facing reference audio from out-of-domain sources. The subjective evaluation results also demonstrate the superiority of our approach over other models. Our model surpasses the state-of-the-art model in naturalness, similarity, and comparison tests.We visualize the mel-spectrograms of ground-truth and synthesized audios by our model and the other two state-of-the-art methods in Figure 3 . The red and white bounding boxes represent regions where different models exhibit significant differences in duration consistency and pronunciation details compared to the ground truth. Through the observation of the red bounding box, it is evident that our model outperforms others in maintaining duration consistency. Reference: Bossbaby@BossBaby Reference: Zootopia@Nick Reference: Frozen@Anna Reference From GRID: GRID@S2 Reference From GRID: GRID@S3 Input The phoneme and pause durations are notably closer to the ground truth dubbing. This phenomenon is more pronounced in the V2C-Animation benchmark due to its complex speaking speed variation. Additionally, from the clearer spectrum lines in the white bounding box, it can be observed that the dubbing generated by our model exhibits clearer and more natural pronunciation details.Character Character Character Character CharacterTo further investigate the specific effects of each module in our proposed method, we conduct ablation studies on the Dub 1.0 setting on the V2C-Animation benchmark. Effectiveness of PCL. The results are presented in Row A-C of Table 5 . We find that the incorporation of TALN enables the model to better clone the timbre of the reference audio. Furthermore, since TALN does not directly affect phoneme features as [4] , the pronunciation accuracy of the model is also improved. The PCL module utilizes emotional information of characters to model the phonemelevel prosody attributes of dubbing, enhancing the model's ability to capture character emotions. Compared to mel-spectrogram framelevel prosody modeling (Row A-B), the phoneme-level modeling used in PCL improves the EMO-ACC. It demonstrates a better alignment between dubbing prosody and the emotion of the character. Besides, the integration of PCL also reduces the distance between the generated mel-spectrogram and the ground truth, demonstrating its effectiveness in improving dubbing quality. Effectiveness of MTSP. The results are presented in Row D-F of Table 5 . The results indicate that the TTS pre-training task can significantly enhance the pronunciation quality, and improve the WER by an absolute margin of 35.02%. The integration of the TTS pre-training task enables our model to generate dubbing with pronunciation accuracy even reaching the level of ground truth. (We do not fine-tune the ASR model on the movie dubbing datasets, so it may even outperform the ground truth.) Additionally, although the MLM pre-training task does not contribute to pronunciation quality improvement as noticeably as the TTS pre-training task, their combination can lead to better performance improvements for the model. The improvement in pronunciation quality also reduces the deviation between synthesized and ground truth dubbing. Duration Consistency Reasoning v.s. Duration Prediction.We compare the performance of our model using duration prediction (Row G) and the proposed DCR (Row H) respectively. The proposed duration consistency reasoning method considers the relationships between dubbing-lip motion and video-audio synchronization. Unlike the duration prediction method, it does not predict the duration of each phoneme separately, thus avoiding inconsistencies in the overall duration compared to the video. The drops on MCD-DTW-SL demonstrate the effectiveness of DCR. 
conclusions: 
 In this work, we propose a two-stage dubbing method to improve the pronunciation quality of dubbing and achieve better audiovisual consistency. In the first stage, we conduct multi-task pre-training on a large-scale text-speech corpus to enable the model to learn universal pronunciation knowledge. In the second stage, the proposed prosody and duration consistency learning module bridges the phoneme-level duration and prosody of dubbing with the movie clip, towards consistency in both aspects. Experimental results demonstrate our method outperforms the current state-of-the-art (SOTA) on two primary benchmarks across multiple settings, validating the effectiveness of our method. 
 
 Appendix: https://github.com/Kyubyong/g2phttps://huggingface.co/openai/whisper-largehttps://github.com/jitsi/jiwer 


Decision: This paper proposes a two-stage TTS framework to "dub" (generate speech) for silent videos. In the first stage, the phoneme encoder is pretrained on a large corpus with timbre-adaptive layer normalization (TALN) and masked language modeling (MLM). In the second stage, facial features are used to predict the pitch and energy, then the duration predictor is replaced by one that considers lip motions. Reviewers find the task to be novel, a good to ACM MM and after some clarification questions (answered in the rebuttal) recommend acceptance of the paper. 

Reasons to accept:
- This paper has novelty and fits the theme of MM
- The presentation of the paper is clear overall, and reviewers tended to increase their scores in response to the rebuttal
- Conclusions are supported by reasonable ablations and experiments

Reasons to reject:
- The paper is quite dense and authors should further declutter results and focus the presentation.
- The baselines may not be strong enough and even though the proposed method outperforms the tested two-stage baselines (I assume these results will make it to the camera ready), the benefit is less strong. As such, the impact is a bit reduced.

Reviews: review: paper_summary: paper_summary: This paper proposes a two-staged TTS framework to generate speech for silence videos. In the first stage, the phoneme encoder is pretrained on a large corpus with timber-adaptive layer normalization (TALN) and masked language modeling (MLM). In the second stage, facial features are used to predict the pitch and energy, then the duration predictor is replaced by one that considers lip motions.
strength_weakness: strength_weakness: 1. This paper has moderate novelty and fits the theme of MM.
2. The presentation of most parts of the paper is good.
3. This paper has sufficient experiments.
limitations: limitations: Overall:
1. Although the experiments are sufficient, some of the results need more explanations and don't quite fit the common sense knowledge.
2. The baselines may not be strong enough. It seems that the baselines in the paper are all trained 'one-stagedly', which means they are directly trained on a small dubbing dataset so they can be easily outperformed. But these baselines can also be trained 'two-stagedly', which means also including a pretraining process on the large dataset. I believe these 'two-stagedly' trained baselines are actually the state-of-the-art approaches for movie dubbing in pactice. Please consider adding the comparison with these approaches to show the real improvement for the movie dubbing industry.

Detailed:
1. About the emotion encoder, attention mechanism is used to capture the emotion features in the video of each phoneme. However, it seems that all frames of the video are used as the keys and values for the  attention mechanism. In the following proposed duration consistency reasoning module, you have already found the matches between the frames and phonemes. So why not use the matched frames as the keys and values for each phoneme in attention mechanism? Or simpler, use max pooling among the matched frames for each phoneme. Please consider adding an ablation study about the structure of emotion encoder.
2. Please add a short description of each baseline used in the experiment in related work.
3. In Table 1 and 2, please consider directly using 'Using the ground truth audio as reference' and 'Using another audio as reference' instead of 'Dub 1.0' and 'Dub 2.0'.
4. In Table 1 and 4, how can WER reach beyond 100% (even 200%)?
5. In Table 1, 2 and 4, the speaker similarities are very low for the many baselines. Please elaborate more on the training setup for the baselines and why the speaker similarities are very low.
6. In Table 2, why are the results of FastSpeech 2 identical under 1.0 and 2.0?
7. The font sizes in Table 3, 4 and 5 are too small.
8. Table 5, 'DCR v.s. Duration Prediction' is not quite suitable for a method name.
paper_summary: paper_summary: This paper offers an interesting research direction in the domain of movie dubbing, with a commendable level of technical detail and a well-thought-out experimental framework. Further refinement in the areas of originality, methodological depth, experimental validation, and social implications will enhance the manuscript's contribution to the field.
strength_weakness: strength_weakness: Innovation: The paper introduces a novel two-stage framework for movie dubbing that emphasizes prosody and duration consistency, which is a unique approach in the domain of audiovisual synthesis.
Technical Depth: The authors have provided a detailed account of the multi-task pre-training and the prosody and duration consistency learning modules, demonstrating a deep understanding of the challenges in dubbing tasks.
Experimental Design: The paper presents a comprehensive set of experiments on two primary benchmarks, comparing the proposed method with state-of-the-art techniques, which is commendable.
limitations: limitations: 1).The Prosody Consistency Learning (PCL) module's use of character emotional features to model dubbing prosody is intriguing. The authors should elaborate on how emotional features are mapped to prosodic attributes and how this process accommodates a range of emotional states.
2). There are some typos in the article.
paper_summary: paper_summary: This paper tackles the challenge of Movie Dubbing, which aims to generate speech from scripts that aligns temporally and emotionally with a movie clip while maintaining the speaker's voice from a brief reference audio. The authors identify the complexities arising from diverse emotions, pacing, and background noise, coupled with limited data availability due to copyright restrictions. To overcome these hurdles, a two-stage dubbing method is proposed.
The first stage focuses on learning clear phoneme pronunciations using a multi-task approach and a large-scale text-to-speech corpus. In the second stage, a prosody consistency learning module bridges emotional expression with phoneme-level dubbing prosody (pitch and energy). Additionally, a duration consistency reasoning module aligns dubbing duration with lip movements. Experiments on benchmark datasets demonstrate the superiority of this method over existing state-of-the-art approaches. The authors plan to release their source code and model checkpoints for public use.
strength_weakness: strength_weakness: The paper tackles the challenging task of movie dubbing, considering multiple factors like temporal alignment, emotional expression, speaker timbre preservation, and background noise.
The proposed two-stage method effectively separates the learning of pronunciation from the adaptation to specific movie dubbing contexts. This allows the model to first acquire general pronunciation knowledge before fine-tuning for the nuances of movie dubbing.
The introduction of dedicated modules for prosody consistency learning and duration consistency reasoning addresses crucial aspects of natural and synchronized dubbing, contributing to the model's effectiveness.
The paper reports superior performance compared to state-of-the-art methods on benchmark datasets, demonstrating the practical value of the proposed approach.
The commitment to releasing source code and model checkpoints promotes reproducibility and encourages further research in the field.
limitations: limitations: Generalization to Unseen Scenarios: While the paper demonstrates promising results on benchmark datasets, its generalizability to unseen movie genres, languages, or emotional expressions remains unclear. Further evaluation on diverse and challenging scenarios would strengthen the paper's claims.

Real-World Evaluation: The evaluation focuses on objective metrics and benchmark datasets. Investigating the model's performance in real-world dubbing scenarios and gathering subjective feedback from human evaluators would provide a more comprehensive assessment of its practical utility.
paper_summary: paper_summary: This paper attempts the task of movie dubbing. As there is limited data for this task, the authors propose a two-staged pipeline as the solution to leverage the abundance of TTS data. In the first stage, the authors pre-train a phoneme encoder from large-scale high-quality text-speech corpus.  While in the second stage, the authors introduce the prosody consistency learning and duration consistency reasoning to better align the generated speech with prosody, timbre, duration of movie characters. Afterwards, the authors evaluate the proposed methods on two benchmark datasets, V2C-Animation and GRID with both subjective and objective metrics. Both qualitative and quantitative results demonstrate the proposed methods achieve the state-of-the-art performance. Besides, the authors conduct ablation study to show the effectiveness of proposed sub-modules.
strength_weakness: strength_weakness: 1. The solutions are well-motivated. Considering that the movie dubbing data is limited, the authors propose to leverage the text-speech corpus. Besides, several sub-modules are proposed to maintain the consistency of prosody, durations, and timbre. 
2. The performance, especially in terms of quantitative aspect, is strong. In Table 1-4, the authors have shown the state-of-the-art performance of proposed approach. Through the provided generated samples, the proposed approach does perform better than baselines. 
3. The experiments are comprehensive. The authors conduct experiments on two benchmark datasets with different settings under various evaluation metrics. Moreover, detailed ablation studies have been conducted as shown in Table 5.
4. The writing is clear and smooth to the readers and the pipeline is well illustrated. Although I have some concerns about (a) why the proposed method has lower WER than GT methods and (b) how to ensure the output from energy predictor and pitch predictor learn the actual energy and pitch and (c) where the performance gain on timbre consistency comes from, I am glad I have found the answers from the later sections.
limitations: limitations: 1. Although the quantitative results are strong in the main paper, after I listen to the provided samples, I still think the generated samples by the proposed method have identity/timbre shift compared to the ground-truth.  The improvement on emotion accuracy is limited, which also is reflected through the generated samples.  Therefore, I suggest to include the discussions on limitations of proposed method. 
2. As there are many modules in the proposed architecture, I think it is better to clear indicate which module is trainable or frozen in Figure 2 as well as the main text. For example, Mel-timbre encoder is frozen in Prosody consistency learning but not highlighted in Figure 2.
3. The details on train/valid/test splits of benchmark datasets are absent.