StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing (2024)

Abstract: Given a script, the challenge in Movie Dubbing (Visual Voice Cloning, V2C) is to generate speech that aligns well with the video in both time and emotion, based on the tone of a reference audio track. Existing stateof-the-art V2C models break the phonemes in the script according to the divisions between video frames, which solves the temporal alignment problem but leads to incomplete phoneme pronunciation and poor identity stability. To address this problem, we propose StyleDubber, which switches dubbing learning from the frame level to phoneme level. It contains three main components: (1) A multimodal style adaptor operating at the phoneme level to learn pronunciation style from the reference audio, and generate intermediate representations informed by the facial emotion presented in the video; (2) An utterance-level style learning module, which guides both the mel-spectrogram decoding and the refining processes from the intermediate embeddings to improve the overall style expression; And (3) a phoneme-guided lip aligner to maintain lip sync. Extensive experiments on two of the primary benchmarks, V2C and Grid, demonstrate the favorable performance of the proposed method as compared to the current stateof-the-art. The code will be made available at https://github.com/GalaxyCong/StyleDubber.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 6767–6779 August 11-16, 2024 ©2024 Association for Computational Linguistics
Given a script, the challenge in Movie Dubbing (Visual Voice Cloning, V2C) is to generate speech that aligns well with the video in both time and emotion, based on the tone of a reference audio track. Existing stateof-the-art V2C models break the phonemes in the script according to the divisions between video frames, which solves the temporal alignment problem but leads to incomplete phoneme pronunciation and poor identity stability. To address this problem, we propose StyleDubber, which switches dubbing learning from the frame level to phoneme level. It contains three main components: (1) A multimodal style adaptor operating at the phoneme level to learn pronunciation style from the reference audio, and generate intermediate representations informed by the facial emotion presented in the video; (2) An utterance-level style learning module, which guides both the mel-spectrogram decoding and the refining processes from the intermediate embeddings to improve the overall style expression; And (3) a phoneme-guided lip aligner to maintain lip sync. Extensive experiments on two of the primary benchmarks, V2C and Grid, demonstrate the favorable performance of the proposed method as compared to the current stateof-the-art. The code will be made available at https://github.com/GalaxyCong/StyleDubber.

heading: 1 Introduction
text: Movie Dubbing (Chen et al., 2022a), also known as Visual Voice Cloning (V2C), aims to convert a script into speech with the voice characteristics specified by the reference audio, while maintaining lip-sync with a video clip, and reflecting the character’s emotions depicted therein (see Figure 1 (a)). V2C is more challenging than conventional text-to-speech (TTS) (Shen et al., 2018a; Ren et al.,
∗Corresponding authors
M odel
…
She's in here. I just have to remember.
Silent video
Reference audio
Text / Subtitle
Speech
(a) Visual Voice Cloning (V2C)
(b) Our StyleDubber aims to learn desired style while keeping lip-sync
Phoneme:{N AA1 T S OW1 sp D K L OW1 Z OW1 IY1 DH AA1 …}
A lig
ni ng
v id
eo Im
ita tin
g st
yl e
Pronunciation habit HH AH0 L OW1
Reference audio
… … … … … ……
Details in phoneme level Emotional expression
Pronunciation habit HH AH0 L OW1 …
… Personal characteristic
1) ⼈类语⾳的⻛格特征通常包括两个层次：（1） 句⼦整体层级的⻛格特征，这类⻛格特征与句⼦ 内容、结构等相关程度较低，通常在同⼀个句⼦ 内部各处较为⼀致，典型如语⾳的情感、说话⼈ ⾳⾊等；（2）句内局部层级的⻛格特征，这类 ⻛格在同⼀个句⼦内部不同位置表现出的效果往 往不同，如语⾳的节奏、重⾳等。
🎬 Visually🔊 Acoustic 🔊 Acoustic
Powerful smoke voice
Version1Figure 1: (a) Illustration of the V2C task. (b) Our StyleDubber learns speech styles on two levels: phonemelevel focuses on pronunciation details, while utterancelevel emphasizes the overall consistency like timbre.
2021), and has obvious applications in the film industry and audio AIGC, including broadening the audience for existing video.
Existing methods broadly fall into two groups. One group of methods focus primarily on achieving audio-visual sync. For example, a duration aligner is introduced in (Hu et al., 2021; Cong et al., 2023) to explicitly control the speed and pause-duration of speaker content by mapping textual phonemes to video frames. Then, an upsampling process is used to expand the video frame sequence to the length of mel-spectrogram frame sequence by multiplying by a fixed coefficient. However, the frame level alignment makes it hard to learn complete phoneme pronunciations, and often leads to seemingly mumbled pronunciations. The other family of methods focuses on maintaining identity consistency between the generated speech and the reference audio. To enable the model to handle a multi-speaker environment, a speaker encoder is used to extract identity embeddings through averaging and nor-
1 6767
malizing per speaker embeddings (Hassid et al., 2022). In contrast, (Lee et al., 2023) and (Hu et al., 2021) try to learn desired speaker voices based on facial appearances. Although humans’ faces can reflect some vocal attributes (e.g., age and identity) to some extent, they rarely encode speech styles, such as pronunciation habits or accents.
According to (Zhou et al., 2022; Li et al., 2022b), human speech can be perceived as a compound of multi-acoustic factors: (1) unique characteristics, such as timbre, which can be reflected on utterance level (see left panel of Figure 1 (b)); (2) pronunciation habits, such as the rhythm and regional accent, which are usually reflected at the phoneme level (see pink rectangles in Figure 1 (b)). We also note that one’s voice can be affected by emotions. For example, the voice can be significantly different when one gets angry. Based on these observations, we propose to learn phoneme level representations from the speaker’s pronunciation habits reflected in the reference audio, and take both facial expressions and overall timbre characteristics at the utterance level of the reference audio into consideration when generating speech.
In light of the above, we propose StyleDubber, which learns a desired style at the phoneme and utterance levels instead of the conventional video frame level. Specifically, a multimodal phoneme adaptor (MPA) is proposed to capture the pronunciation styles at the phoneme level. By leveraging the cross-attention relevance between textual phonemes of the script and the reference audio as well as visual emotions, MPA learns the reference style and then generates intermediate speech representations with consideration of the required emotion. Our model also introduce an utterance-level style learning (USL) module to strengthen personal characteristics during both the mel-spectrogram decoding and refining processes from the above intermediate representations. For the temporal alignment between the resulting speech and the video, we propose a Phoneme-guided Lip Aligner (PLA) to synchronize lip-motion and phoneme embeddings. At last, HiFiGAN (Kong et al., 2020) is used as a vocoder to convert the predicted melspectrogram to the time-domain waves of dubbing.
The main contributions are summarized as:
• We propose StyleDubber, a style-adaptative dubbing model, which imitates a desired personal style in phoneme and utterance levels. It enhances speech generation in terms of speech
clarity and its temporal alignment with video.
• At the phoneme level, we design a multimodal style adaptor, which learns styled pronunciation of textual phonemes and considers facial expressions when generating intermediate speech representations. At the utterance level, our model learns to impose timbre into resulting mel-spectrograms.
• Extensive experimental results show that our model performs favorably compared to current state-of-the-art methods.

heading: 2 Related Work
text: Text to Speech is a longstanding problem, but recent models represent a dramatic improvement (Liu et al., 2024; Tan et al., 2024; Casanova et al., 2022; Wang et al., 2023a; Huang et al., 2023a,b; Ju et al., 2024). FastSpeech2 (Ren et al., 2021), for example, alleviates the one-to-many text-to-speech mapping problem by explicitly modeling variation information. Min et al. (2021), in contrast, improves generalization through episodic meta-learning and generative adversarial networks. Recently Le et al. (2023) proposed a non-autoregressive flowmatching model for mono or cross-lingual zeroshot text-to-speech synthesis. Despite the impressive speech they generate, these methods cannot be applied to the V2C task as they lack the required emotion modelling and lip sync. Visual Voice Cloning is proposed to address the problem of film dubbing (Chen et al., 2022a) and has attracted a lot of attention in cross-modality alignment field (Tu et al., 2022, 2023, 2024; Li et al., 2022a; Liu et al., 2023; Wang et al., 2023b; Xiao et al., 2023, 2022). Then, Cong et al. (2023) proposed a hierarchical prosody dubbing model by associating with lip, face, and scene and focus on frame-level prosody learning (Hu et al., 2021). To handle multi-speaker scenes, Hassid et al. (2022) matches identities by normalizing each speaker to the unit norm, and Lu et al. (2022) adopts a lookup table to match the d-vector. Recently, FaceTTS (Lee et al., 2023) used biometric information extracted directly from the face image as style to improve identity modelling using a score-based diffusion model. Unlike the above methods, StyleDubber address the challenge of insufficient identity information by introducing adaptive utterance-level embedding and detailed pronunciation variations based on the reference audio and video.
2
Human Pronunciation Modeling aims to learn individual pronunciation variations, which is crucial to generate comprehensible, natural, and acceptable speech (Miller, 1998). Compared with fixed speaker representations, phoneme-dependent methods (Li et al., 2022b; Fu et al., 2019) can better control speech and describe more pronunciation features, as phonemes are the basic sound units in a language (Lubis et al., 2023). Recently, Zhou et al. (2022) analysed the correlation between local pronunciation content and speaker embeddings at the quasi-phoneme level by reference attention. Here, in contrast, we propose a multimodal style adaptor to capture the fine-grained pronunciation variation, which not only imitates the reference style acoustically, but also conveys emotional expression by reference transformer.

heading: 3 Proposed Method

heading: 3.1 Overview
text: Our StyleDubber aims to generate a desired dubbing speech Ŷ , given a reference audio Ra, a phoneme sequence Tp converted from the given script, and a video frame sequence Vl:
Ŷ = StyleDubber(Ra, Tp, Vl). (1)
The main architecture of the model is shown in Figure 2. Unlike existing prosody dubbing methods, our model learns speech style from phoneme level and utterance level, inspired by human tonal phonetics. First, the textual phoneme sequence is
converted from raw text. A phoneme encoder is then used to extract phoneme embeddings. These embeddings are fed into our Multimodal Phonemelevel Adaptor (MPA), which learns to capture and apply phoneme-level pronunciation styles to generate intermediate speech representations, meanwhile taking facial expressions into consideration. Next, our Phoneme-guided Lip Aligner (PLA) predicts the duration for each phoneme by associating lip motion sequence. The duration and intermediate dubbing representation are fed to our Utterancelevel Style Learning (USL) module, which learns overall style at the utterance level and applys it during mel-spectrograms decoding and refining processes. We detail each module below.

heading: 3.2 Multimodal Phoneme-level Adaptor
text: Our Multimodal Phoneme-level Adaptor (MPA) contains three steps: (1) learn acoustic style from reference audio; (2) perceive visual emotion from silent movies; (3) generate intermediate speech representations for textual phonemes of the input script, with reference to the captured acoustic styles and emotions in the last two steps.
Learn acoustic style. We extract reference melspectrogram Rmel from reference audio Ra by Short-time Fourier transform (STFT), and the montreal forced aligner (McAuliffe et al., 2017) is used to clip phoneme. Then, we capture the style feature Sp via a mel-style encoder E spk down(·):
Sp = E spk down(Rmel), (2)
3
where Espkdown(·) comprises a mel-style encoder (Min et al., 2021) and four 1D convolutional downsample layers (Zhou et al., 2022). On the other hand, embeddings of textual phoneme sequence TH ∈ RNp×Dm are extracted by a phoneme encoder TH = Epho(Tp) (Cong et al., 2023), where Np denotes the length of phoneme sequence. Next, we propose the acoustic reference transformer RA→L (Acoustic to Language) to calculate the relevance between a textual phoneme embedding and each style feature by crossmodal transformer:
R [0] A→L = R [0] L , R̂ [i] A→L = CM [i],mul A→L(LN(R [i−1] A→L),LN(R [0] A )) + LN(R [i−1] A→L), R [i] A→L = fθ[i] A→L (LN(R̂[i]A→L) + LN(R̂ [i] A→L), (3) where i = {1, ..., D} denotes the number of feedforwardly layers, LN(·) denotes the layer normalization, and fθ is a positionwise feed-forward sublayer parametrized by θ. CM[i],mulA→L (·) is a multihead attention between Sp and TH , as follows:
CM[i],mulA→L = softmax( THSp
⊤ √
dSp )Sp, (4)
where the textual phoneme embedding TH is used as query and the style feature Sp is used as key and value. Unlike crossmodal transformer in (Tsai et al., 2019), our acoustic reference transformer removes the repeatedly reinforcing and MFCCs frame-level operation, and only focuses on interaction between quasi-phoneme scale of reference audio and script phoneme, which is more conducive to human pronunciation habits.
Unlike using cross-entropy loss as style classifier (Zhou et al., 2022), we constrain Espkdown via a style consistency loss:
Lspk = 1 n ·
n∑
j
(1− cos_sim(ϕ(Tj), A(Sp)j)), (5)
where ϕ(·) is a function outputting the embedding by the pre-trained GE2E model (Wan et al., 2018), A(Sp) outputs a style vector via average pooling, and cos_sim(·) is the cosine similarity function. T represents the ground truth audio, n is batch size. Perceive visual emotion. We first use the S3FD model (Zhang et al., 2017) to detect facial region from each frame of video, and then an emotion face-alignment network (EmoFAN) (Toisoul et al., 2021) is used to extract emotion features Fp ∈ RNv×Dm from face regions. The Nv represents the number of video frames. Similar to style
extraction, emotional feature Se is obtained by a downsampling equipped encoder:
Se = E emo down(Fp), (6)
where Se ∈ RNdv×Dm and Ndv is length after down-sample. The difference from Espkdown(·) is that Eemodown(·) has two 1D convolutional downsample layers. Next, an emotion reference transformer ZV→L (Visual to Language) is proposed to analyze the correlations between the emotional feature and textual phoneme. The ZV→L has same architecture with RA→L. The CM [i],mul V→L (·) is multihead attention to calculate correlation between Se and TH :
CM[i],mulV →L = softmax( THSe
⊤ √
dSe )Se, (7)
where key and value are emotional features Se to assist script phoneme in selecting related visual emotion expression. Finally, we regard the output ZDV→L and R D A→L of the last layers of emotion reference transformer and acoustic reference transformer as context visual emotion and acoustic style, respectively. The cross-entropy emotional classification loss Lemo is used to constrain Eemodown(·). Generate intermediate speech representations. We first concatenate the phoneme-level context visual emotion and acoustic style in channel dimension, and then feed it into self-attention blocks SA(·) to fuse these embeddings:
Gfus = SA([ZDV →L, RDA→L]), (8)
where Gfus ∈ RNp×Dm is the fused multimodal context embedding which is with the same length as the textual phonemes. Finally, we combine textual phoneme embedding and multimodal context embedding Opho = Gfus ⊕ TH , which is viewed as the intermediate dubbing representations.

heading: 3.3 Phoneme-guided Lip Aligner
text: The Phoneme-guided Lip Aligner (PLA) consists of two steps: 1) Monotonic attention is used to learn the contextual aligning feature between lip motion and textual phoneme embedding; 2) Liptext duration predictor aims to output the duration of each phoneme based on the contextual aligning feature.
Monotonic Attention. The lip-movement hidden representation LH = Elip(Vl) ∈ RNv×Dm is obtained using the same lip-motion encoder in (Cong et al., 2023). Then, we encourage PLA to use
4
textual phoneme embedding to capture related lip motion by multi-head attention with monotonic constraint:
Clip = softmax( THLH ⊤ √
dLH )LH , (9)
where the textual phoneme embedding TH serves as query, and the lip motion embedding LH serves as key and value. Clip ∈ RNp×Dm captures the dependency between lip and textual phoneme. Monotonic Alignment Loss (MAL) (Chen et al., 2020) is used to ensure proper alignment over the time:
Lm = log(− ∑kp+β l=kp−β ∑P−1
p=0 Mp,l∑L−1 l=0 ∑P−1 p=0 Mp,l ), (10)
where β is a hyper-parameter to control bandwidth, k is the slop for length of phoneme P and corresponding of lip length L, and Mp,l is the masked attention weight matrix with p-th row and l-th column. To this end, Lmon aims to constrain attention weights to diagonal area to satisfy monotonicity.
Duration predictor. Since the total dubbing times TotalLength can known by multiplying time coefficient with video frames Nv in advance (Hu et al., 2021), we transform the alignment problem into inferring the relative time of a phoneme over its total duration. We first use the duration predictor (Ren et al., 2021) to learn the duration from lip-phoneme context Clip and re-scale it into relative duration by using TotalLength divide predicted sum:
dp = TotalLength · ESoftplus(C k lip)∑Np−1
k=0 ESoftplus(Clip) , (11)
where dp ∈ RNp×1 represents the relative duration for each phoneme unit. ESoftplus(·) represents the duration predictor, which consist of 2-layer 1D convolutional with softplus activate function (Song et al., 2021). In this case, we obtain how many melframes correspond to lip-phoneme context Clip to ensure the boundary of phoneme unit will not be broken, while syncing with the whole video.
Loss function. The duration loss is optimized with MSE loss, following (Ren et al., 2021):
Ld = MSE(dp, log(gd)), (12)
where log(gd) represents the ground-truth duration in the log domain.

heading: 3.4 Utterance-level Style Learning
text: We also consider the utterance-level information of reference audio to enhance global style characteristics. Specifically, we use the GE2E model (Wan et al., 2018) to extract the timbre vector Vs as utterance-level condition, which aggregates global style information to guide the decoding and refinement of mel-spectrograms from intermediate speech representations by affine transform. Mel-Decoder. We use transformer-based meldecoder (Cong et al., 2023) to decode intermediate speech representations Opho into a spectrogram hidden sequence:
R̂ = DecoderUSLN(LR(Opho, dp), Vs), (13)
where LR(·) is the length regulator (Ren et al., 2021) to expand Opho to mel-length based on predicted duration dp. R̂ ∈ RNlr×256 denotes a spectrogram hidden sequence, Nlr is the predicted total mel-legnth. During decoding, we replace the original layer norm (Ba et al., 2016) in each Feed-Forward Transformer (FFT) block with our utterance-level style learning normalization (USLN):
USLN(h, Vs) = γ(Vs) · hn + δ(Vs), (14)
where hn = (h− µ)/σ is normalized features by the mean µ and variance σ of input feature h. The γ(Vs) and δ(Vs) represent the learnable gain and bias of the overall style vector by affine transform, respectively, which can adaptively perform scaling and shifting to improve style expression. Refine mel-spectrogram. We introduce the aforementioned USLN to MelPostNet (Shen et al., 2018b) to inject the style information from timbre vector Vs during refining the final melspectrograms stage:
M̂ = POSTUSLN(R̂, Vs), (15)
where M̂ ∈ RNlr×80 denotes the predicted melspectrograms with 80 channels.

heading: 3.5 Training
text: Our model is trained in an end-to-end fashion via optimizing the sum of all losses. The total loss L can be formulated as:
L = λ1Lspk + λ2Lemo + λ3Lr + λ4Lm + λ5Ld, (16)
where Lr is the reconstruction loss to calculate L1 differences between the predicted and ground-truth mel-spectrograms.
5
Finally, the generated mel-spectrograms M̂ are converted to time-domain wave Ŷ via the widely used vocoder HiFiGAN.

heading: 4 Experiments
text: We evaluate our method on two primary V2C datasets, V2C-Animation and GRID. Below, we first provide implementation details. Then, we briefly introduce the datasets and evaluation metrics, followed by quantitative and qualitative results. Ablation studies are also conducted to thoroughly evaluate our model.

heading: 4.1 Implementation Details
text: The video frames are sampled at 25 FPS and all audios are resampled to 22.05kHz. The groundtruth of phoneme duration is extracted by montreal forced aligner (McAuliffe et al., 2017). The window length, frame size, and hop length in STFT are 1024, 1024, and 256, respectively. The lip region is resized to 96 × 96 and pretrained on ResNet-18, following (Martinez et al., 2020; Ma et al., 2021). We use 8 heads for multi-head attention in PLA and the hidden size is 512. The duration predictor consists of 2-layer 1D convolution with kernel size 1. The weights in Eq. 16 are set to λ1 = 25.0, λ2 = 0.1, λ3 = 5.0, λ4 = 2.0, λ5 = 5.0. For downsampling encoder in Espkdown, we use 4 convolutions containing [128, 256, 512, 512] filters with shape
3 × 1 respectively, each followed by an average pooling layer with kernel size 2. In Eemodown, 2 convolutions are used to download to quasi phonemelevel, containing [128, 256] filters with shape 3 × 1. In RA→L and ZV→L, the dimensionality of all reference attention hidden size is set to 128 implemented by a 1D temporal convolutional layer. We set the batch size to 32 and 64 on V2C-Animation and Grid dataset, respectively. For training, we use Adam (Kingma and Ba, 2015) optimizer with β1 = 0.9, β2 = 0.98, ϵ=10−9 to optimize our model. The learning rate is set to 0.00625. Both training and inference are implemented with PyTorch on a GeForce RTX 4090 GPU.

heading: 4.2 Dataset
text: V2C-Animation dataset (Chen et al., 2022a) is currently the only publicly available movie dubbing dataset for multi-speaker. Specifically, it contains 153 diverse characters extracted from 26 Disney cartoon movies, specified with speaker identity and emotion annotations. The whole dataset has 10,217 video clips, and the audio samples are sampled at 22,050Hz with 16 bits. In practice, (Chen et al., 2022a) removes video clips less than 1s. In this work, all experiments are conducted on the V2C denoise version. We will publish this version. GRID dataset (Cooke et al., 2006) is a basic benchmark for multi-speaker dubbing. The whole dataset has 33 speakers, each with 1000 short English sam-
6
ples. All participants are recorded in a noise-free studio with a unified screen background. The train set consists of 32,670 samples, 900 sentences from each speaker. In the test set, there are 100 samples of each speaker.

heading: 4.3 Evaluation Metrics
text: Objective metrics. To measure whether the generated speech carries the desired speaker identity and emotion, speaker identity similarity (SPK-SIM) is calculated by SECS (Casanova et al., 2021), and emotion accuracy (EMO-ACC) is employed by pretrained speech emotion recognition model (Ye et al., 2023). Besides, we adopt the Mel Cepstral Distortion Dynamic Time Warping (MCD-DTW) to measure the difference between generated speech and real speech. We also adopt the metric MCD-DTWSL, which is MCD-DTW weighted by duration consistency (Chen et al., 2022a). The Word Error Rate (WER) (Morris et al., 2004) is used to measure pronunciation accuracy by the publicly available whisper (Radford et al., 2023) as the ASR model. Besides, we use the ASV model (WavLMTDNN (Chen et al., 2022b)) to comprehensively evaluate identity similarity (see Appendix D), following NaturalSpeech 3 (Ju et al., 2024). Subjective metrics. We also provide subjective evaluation results via conducting a human study using a 5-scale mean opinion score (MOS) in two aspects: naturalness and similarity. Following the settings in (Chen et al., 2022a), all participants are asked to assess the sound quality of 25 randomly selected audio samples from each test set.

heading: 4.4 Performance Evaluations
text: We evaluate our method in three experimental settings as shown in Table 3. The first setting is the same as in (Chen et al., 2022a), which uses target audio as reference audio from test set. However, this is impractical in real-world applications. Thus, we design two new and more reasonable settings: “Dub 2.0” uses non-ground truth audio of the same speaker as reference audio; “Dub 3.0” uses the audio of unseen characters (from another dataset) as reference audio. We compare with six recent related baselines to comprehensively analyze. Furthermore, we will release the detailed configuration
for all experiment settings for the GRID and V2C Animation datasets.
Results under Dub 1.0 setting. As shown in Table 1, our method achieves the best performance on almost all metrics on both GRID and V2C-Animation benchmarks. Our method only performs slightly worse in terms of EMO-ACC than the SOTA movie dubbing model HPMDubbing (Cong et al., 2023). Regarding identity accuracy (see SPK-SIM), our method outperforms other models with an absolute margin of 27.27% over the 2nd best method. In terms of MCD-DTW and MCD-DTW-SL, our method achieves 6.11% and 24.38% improvements, respectively. This indicates our method can achieve better speech quality and better duration consistency.
Results under Dub 2.0 setting. We report the V2C results in Table 2. Despite Dub 2.0 is much more challenging than 1.0, our method still outperforms other methods on six metrics. The SPK-SIM and WER significantly improve 7.4% and 6.98% than the fastspeech 2 (TTS-textonly) and meta-stylespeech method, respectively. Additionally, the proposed method (StyleDubber) achieves the lowest MCD-DTW compared to all baselines, which indicates our method achieves minimal acoustic difference even in challenging setting 2.0. Furthermore, the lowest MCD-DTWSL shows that our method achieves almost the same duration sync as the groud-truth video. Finally, the human subjective evaluation results (see MOS-N and MOS-S) also show that our StyleDubber can generate speeches that are closer to realistic speech in both naturalness and similarity.
Results under Dub 3.0 setting. Since there is no target audio at this setting, we only compare SPKSIM and WER, and make subjective evaluations. As shown in Table 4, our StyleDubber achieves the best generation quality in all four metrics, largely outperforming the baselines. The higher SPK-SIM and MOS-S (mean opinion score of similarity) indi-
7
StyleDubberwav_pred_Ralph@Ralph_00_1703_00
wav_pred_Zootopia@Judy_00_0153_00.wav
# Methods WER ↓ SPK-SIM ↑ MCD-DTW ↓ MCD-DTW-SL ↓ EMO-ACC ↑ 1 w/o MPA 39.74 77.26 9.90 9.98 41.18 2 w/o USL 32.81 47.07 10.23 10.43 42.77 3 w/o PLA 35.90 80.77 9.59 11.47 45.48 4 Quasi-phoneme v.s. frame 33.62 80.76 9.75 9.84 43.83 5 w/o RA→L 38.21 77.79 9.81 9.90 42.61 6 w/o ZV→L 33.87 79.30 9.82 9.91 41.49 7 w/o U-MelDecoder 32.26 47.31 10.31 10.41 42.80 8 w/o U-Post 31.73 80.79 9.52 9.61 44.33 9 Full model 31.49 82.26 9.37 9.46 45.62
Table 5: Ablation study of the proposed method on the V2C benchmark dataset with 1.0 setting, respectively.
cate the better generalization ability of our methods to learn style adation across unseen speakers. Besides, our method also maintains good pronunciation (see WER). Overall, our StyleDubber achieves impressive results in challenging scenarios.

heading: 4.5 Qualitative Results
text: We visualize the mel-spectrogram of reference audio, ground-truth audio, and synthesized audios by ours and the other two state-of-the-art methods in Figure 3. We highlight the regions in melspectrograms where significant differences are observed among these methods in reconstruction details (green boxes), and pause (blue boxes), respectively. We find that our method is more similar to the ground-truth mel-spectrogram, which has clearer and distinct horizontal lines in the spectrum to benefit the fine-grained pronunciation expression of speakers (see green box). By observing the blue boxes, we find that our method can learn natural pauses to achieve better sync by aligning phonemes and lip motion.

heading: 4.6 Ablation Studies
text: To further study the influence of the individual components in StyleDubber, we perform the comprehensive ablation analysis using V2C 1.0 version. Effectiveness of MPA, USL, and PLA. The results are presented in Row 1-3 of Table 5. It shows that all these three modules contribute significantly to the overall performance, and each module has a
different focus. After removing the MPA, the MCD-DTW and WER severely drop. This reflects that the MPA achieves minimal difference in acoustic characteristics from the target speech and better pronunciation by phoneme modeling with other modalities. In contrast, the SPK-SIM is most affected by USL, which indicates decoding mel-spectrograms by introducing global style is more beneficial to identity recognition. Finally, the performance of MCD-DTW-SL drops the most when removing the PLA. This can be attributed to the better alignment between video and phoneme sequences.
Quasi-phoneme v.s. frame. To prove the impact of regulating the temporal granularity to quasiphoneme-scale, we remove the downsample operation and retrain the frame-level information as input of RA→L and ZV→L. As shown in Row 4 of Table 5, all metrics have some degree of degradation, which means quasi-phoneme level acoustic and emotion representation is more conducive for script phoneme to capture desired information.
Effectiveness of RA→L and ZV→L. To study the effect on each reference transformer in MPA, we remove ZV→L and ZA→L, respectively. As shown in Row 5-6 of Table 5, ZV→L has a significant effect on improving emotions, while ZA→L more focus on local acoustic information to strengthen style and pronunciation.
Effectiveness of U-MelDecoder and U-post. To prove the effect of each module in USL, we remove the utterance-level style learning on mel-decoder and post-net, respectively. In other words, it still keeps an autoregressive manner by transformerbased decoder, and we just cut off the red arrow in Figure 2 (c). As shown in Row 7-8 of Table 5, when removing the U-post, the performance also drops but is not as large as removing the U-MelDecoder. This indicates that U-MelDecoder is critical to the generation of spectrum, while U-post only works
8
on refining spectrum in 80 channels so that the impact is relatively small.

heading: 5 Conclusion
text: In this work, we propose StyleDubber for movie dubbing, which imitates the speaker’s voice at both phoneme and utterance levels while aligning with a reference video. StyleDubber uses a multimodal phoneme-level adaptor to improve pronunciation that captures speech style while considering the visual emotion. Moreover, a phoneme-guided lip aligner is devised to synchronize vision and speech without destroying the phoneme unit. The proposed model sets new state-of-the-art on the V2C and GRID benchmarks under three settings.

heading: 6 Limitation
text: We follow the task definition of Visual Voice Cloning (V2C), which focuses on generating audio only. Truly solving the larger problem would require changing the video to reflect the updated audio. In future, we will add this capability to better support tasks like cross-language video translation.

heading: 7 Ethics Statement
text: The existence of V2C methods lowers the barrier to high-quality and expressive visual voice cloning. In the long term this technology might enable broader consumption of factual and fictional video content. This could have employment implications, not least for current film voice actors. There is also a risk that V2C might be used to generate fake video depicting people apparently saying things they have never said. This is achievable already by an impersonator using entry-level video editing software, so the marginal impact of V2C on this problem is small. The licence for StyleDubber will explicitly prohibit this application, but the efficacy of such bans is limited, not least by the availability of other software that achieves the same purpose.

heading: Acknowledgements
text: This work was supported in part by National Key R&D Program of China under Grant (2023YFB4502800), National Natural Science Foundation of China: 62322211, 61931008, 62236008, 62336008, U21B2038, 62225207, Fundamental Research Funds for the Central Universities (E2ET1104), “Pionee” and “Leading Goose” R&D Program of Zhejiang Province (2024C01023, 2023C01030). Yuankai Qi, Amin Beheshti, Anton
van den Hengel, and Ming-Hsuan Yang are not supported by the aforementioned fundings.

heading: A The challenges in V2C benchmark
text: The V2C benchmark significantly differs from traditional TTS benchmark , and it is more challenging in the following aspects: (1) The data scale of V2C dataset is much smaller in terms of either the number of data items or speech length (see Figure 4 (a)-(b)). There are only 9374 video clips in V2C, and most of its audio is shorter than 5s. In contrast, FS2 and Stylespeech are trained on LJspeech and LibriTTS with 13,100 and 149,753 samples, most of which are longer than 5s. Although LJspeech also looks similar in size to V2C, it is a singlespeaker dataset, so V2C allocates fewer samples to each speaker. (2) V2C has the largest variance of pitch compared to TTS tasks due to exaggerated expressions of cartoon characters (see Figure 1 (c) and more details in Tab. 2 of V2C-Net). (3) The audio of V2C contains background noise or music, like car whistle and alarm clock sound, et al. Signal-to-noise Ratio (SNR) of V2C is the lowest (Figure 4 (d)). In summary, unlike the large-scale clean TTS datasets, V2C is much more challenging, and the well-known TTS methods suffer performance degradation. In this work, all experiments are conducted on the V2C denoise version. We will publish this version.

heading: B Baselines
text: We compare our method against six closely related methods with available codes. 1) StyleSpeech (Min et al., 2021) is a multi-speaker voice clone method that synthesizes speech in the style of the target speaker via meta-learning; 2) FastSpeech2 (Ren et al., 2021) is a popular multi-speaker TTS method
for explicitly modeling energy and pitch in speech; 3) Zero-shot TTS (Zhou et al., 2022) is a contentdependent fine-grained speaker method for zeroshot speaker adaptation. 4) V2C-Net (Chen et al., 2022a) is the first visual voice cloning model for movie dubbing; 5) HPMDubbing (Cong et al., 2023) is a hierarchical prosody modeling for movie dubbing, which bridges video representations and speech attributes from three levels: lip, facial expression, and scene. 6) Face-TTS (Lee et al., 2023) is a novel face-styled speech synthesis within a diffusion model, which leverages face images to provide a robust characteristic of speakers. In addition, for a fair comparison, for the pure TTS method, we adopt the setting as (Chen et al., 2022a), which takes video embedding as an additional input, before the duration predictor to predict the duration.

heading: C Whisper test on V2C-Animation dataset
text: In Table 6, the results show that large-v3 achieved the lowest WER, thus we re-selected large-v3 as the measurement tool to get more convincing results on the V2C-Animation dataset. Note that Whisper-Large V3 has not been fine-tuned in the V2C-Animation dataset, there is still some gap (WER in GT is 22.55 %), but it is enough to serve as a fair comparison. All results (Dub1.0, 2,0, and 3.0) still reflect that our StyleDubber is the best (see Table 1,2,4) and is more conducive to
12
the clarity of movie dubbing. Considering the inference speed and computation memory, the Grid dataset still retains the original “Whisper-base” as the test benchmark. The “Whisper-base” achieves the 22.41 % GT WER on the GRID test as similar to the VDTTS (Hassid et al., 2022) result in Table 2 (GRID evaluation).

heading: D WavLM-TDNN Similarity Testing
text: We employ the SOTA speaker verification model, WavLM-TDNN, to evaluate the speaker similarity between the prompt (i.e., the reference audio in the V2C task) and synthesized speech, following VALL-E (Wang et al., 2023a), VoiceBox (Le et al., 2023), and NaturalSpeech 3 (Ju et al., 2024). WavLM-TDNN achieved the top rank at the VoxSRC Challenge 2021 and 2022 leaderboards and it is suitable as the SPK-SIM metric for the challenging V2C-nimation dataset (Chen et al., 2022a). The similarity score predicted by WavLMTDNN is in the range of [-1; 1], where a larger value indicates a higher similarity of input samples. Specifically, two metrics need to be calculated: (1) SIM-R represents the similarity with resynthesized audio, which is not comparable across models using different vocoders; (2) SIM-O is used to measure similarity against the original reference audio. Note that the report results based on the GE2E model (Table 1, 2, 4) are compared with the original waveform.
As shown in Table 7,8, the results on two datasets show that even if the similarity measurement method is replaced, our StyleDubber still achieves the best performance in Sim-O and Sim-R. The result proves the effectiveness of StyleDubber, which proposes multi-scale style learning at phoneme and utterance levels and captures precise pronunciation in acoustic details and visual emo-
tion for dubbing. Besides, we have several findings: (1) Compared with the GT result on the GRID dataset (0.87), the Sim-O result of V2C-Animation is lower (0.79), which may be due to the influence of noise and background music. In contrast, the GRID dataset is recorded in a studio environment. (2) Changing the measurement metric has relatively little impact on the GRID dataset. V2C has a very obvious decline in WavLM-TDNN similarity, which is not captured by the GE2E model (GE2E score can reach more than 80%). In the future, we will investigate more robust timbre extractors and use the denoising diffusion probabilistic models to further improve the generated wave quality.
13

heading: Future Work
text: In the future, we will investigate more robust timbre extractors and use the denoising diffusion probabilistic models to further improve the generated wave quality. 13