Neural Dubber: Dubbing for Videos According to Scripts
Authors: Chenxu Hu
Qiao Tian
Tingle Li
Wang Yuping
Yuxuan Wang
Hang Zhao
Venue: NeurIPS 2021 Poster (2021)
Keywords: text to speech
speech
audio-visual
synchronization
multi-modal

Abstract: Dubbing is a post-production process of re-recording actors’ dialogues, which is extensively used in filmmaking and video production. It is usually performed manually by professional voice actors who read lines with proper prosody, and in synchronization with the pre-recorded videos. In this work, we propose Neural Dubber, the first neural network model to solve a novel automatic video dubbing (AVD) task: synthesizing human speech synchronized with the given video from the text. Neural Dubber is a multi-modal text-to-speech (TTS) model that utilizes the lip movement in the video to control the prosody of the generated speech. Furthermore, an image-based speaker embedding (ISE) module is developed for the multi-speaker setting, which enables Neural Dubber to generate speech with a reasonable timbre according to the speaker’s face. Experiments on the chemistry lecture single-speaker dataset and LRS2 multi-speaker dataset show that Neural Dubber can generate speech audios on par with state-of-the-art TTS models in terms of speech quality. Most importantly, both qualitative and quantitative evaluations show that Neural Dubber can control the prosody of synthesized speech by the video, and generate high-fidelity speech temporally synchronized with the video.

Neural Dubber: Dubbing for Videos According to Scripts
Chenxu Hu', Qiao Tian', Tingle Li Yuping Wang?, Yuxuan Wang', Hang Zhao!
IIS, Tsinghua University "ByteDance SSanghai Qi Zhi Institute https:/ httppss////inghaaaaaaa warinnngaaamar-s-aee mars- lab. bibb.githubb...... Cariub..o/Neuuie io/Neurallubber/
Abstract
Dubbing is post-production process of re-recording actors' dialogues, which is extensively used in filmmaking and video production. It is usually performed manually by professional voice actors who read lines with proper prosody, and in synchronization with the pre-recorded videos. this work, we propose Neural Dubber, the first neural network model to solve novel automatic video dubbing (AVD) task: synthesizing human speech synchronized with the given video from the text. Neural Dubber is multi-modal text-to-speech (TTS) model that utilizes the lip movement in the video to control the prosody of the generated speech. mmage-based speaker embedding (ISE) module is developed for Furthermore, setting. which enables Neural Dubber to generate speech with a the multi sueli-sppeaerr reasonable timbre according to the speaker's face. Experiments on the chemistry lecture single-speaker dataset and LRS2 multi-speaker dataset show that Neural Dubber can generate speech audios on par with state-of-the-art TTS models in terms of speech quality. Most importantly, both qualitative and quantitative evaluations show that Neural Dubber can control the prosody of synthesized speech by the video, and generate high-fidelity speech temporally synchronized with the video.
Introduction
Dubbing is a post-production process of re-recording actors' dialogues in a controlled environment (i.c., sound studio), which is extensively used in filmmaking and video production. There two common application scenarios for dubbing. The first one replacing previous dialogues because poor sound quality is very common for speech recorded challenging to record high-quality audio. The second one is replacing the actor' voices in foreign- language films with those of other performers speaking the audience's 's language. For example, an English video needs to be dubbed into Chinese if it is shown in China
noise location the itself is too
In this paper, we mainly focus on the first application scenario, also known "automated dialogue replacement (ADR)", in which the professional voice actor watches original performance in the pre-recorded video, and re-records each line to match the lip movement with proper prosody such as stress, intonation and rhythm, which allows their speech to be synchronized with the pre-recorded video. In this scenario, the lip motion (viseme) in the video is consistent with the given scripts (phoneme), and the pre-recorded high-definition video can not modified during the ADR process.
While dubbing is impressive ability of professional voice actors, we aim achieve this ability computationally. We name this novel task automatic video dubbing (AVD): synthesizing human speech that is temporally synchronized with the given video according to the corresponding text. The main challenges the task are two-fold: (1) temporal synchronization between synthesized speech
Corresponding to hangzhao @ mail wwsinghuaedd..c
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

and video, i.e., the synthesized speech should be synchronized with the lip movement of the speaker in the given video; (2) the content of the speech should be consistent with the input text.
Text to speech (TTS) is a task closely related to dubbing, which aims at converting given texts into natural and intelligible speech. However, several limitations prevent TTS from being applied in the dubbing problem: 1) TTS is a one-to-many mapping problem (i.e., multiple speech variations can be spoken from the same text) [38]. so it is hard to control the variations (e.g., prosody, pitch and duration) in synthesized speech during generation; 2) with only text as input, TTS can not utilize the visual information from the video to control speech synthesis, which greatly limits its applications in dubbing scenarios where the synthesized speech are required to be synchronized with the video.
We introduce Neural Dubber, the first model to solve the AVD task. Neural Dubber is modal speech synthesis model, which generates high-quality and lip-synced speech from the given text and video. In order control the duration and prosody of synthesized speech, Neural Dubber works in a non-autoreeressiive way following [38]. The problem of length mismatch between phoneme sequence and mel-speetrogram sequence in non-autoregressive TTS is usually solved by up-sampling the phoneme sequence according to the predicted phoneme duration. Meanwhile, a phoneme duration predictor is needed, where the ground truth is usually obtained from another model [39, 38] or itself during training [24]. However, due to the natural correspondence between lip movement and text [10], we do not need to get phoneme duration target in advance like previous methods [39, 38, 24]. Instead, we use the text-video aligner which adopts an attention module between the video frames and phonemes, and then upsample the text-video context sequence according to the length ratio of mel-spectrogram sequence and video frame sequence. The text-video aligner not only solves the length mismatch problem, but also allows the lip movement in the video to control the prosody of the generated speech explicitly by the attention between video frames and phonemes.
In the real dubbing scenario, voice actors need alter the timbre and tone according to different performers in the video. In order to better simulate the real case in the AVD task, we propose the image-based speaker embedding (ISE) module, which aims synthesize speech with different timbres conditioning on the speakers' face in the multi-speaker setting. To the best of our knowledge, this is the first attempt to predict a speaker embedding from a face image with the goal of generating speech with a reasonable timbre that is consistent with the speaker's 's facial features (e.g., gender and age). This is achieved by taking advantage of the natural co-occurrence of faces and speech in videos without the supervision of speaker identity. With ISE, Neural Dubber can synthesize speech with a reasonable timbre according to the speaker's 's face. other words, Neural Dubber can use different face images to control the timbre of the synthesized speech.
We conduct experiments on the chemistry lecture dataset from Lip2Wav [35] for the single-speaker AVD. and the LRS2 [1] dataset for the multi-speaker AVD. The results of extensive quantitative and qualitative evaluations show that in terms of speech quality, Neural Dubber on par with state-of- the-art TTS models [51, 41, 38]. Furthermore, Neural Dubber can synthesize speech temporally synchronized with the lip movement video. In the multi-speaker setting, we demonstrate that the ISE enables Neural Dubber to generate speech with reasonable timbre based on the speaker's 's face, resulting Neural Dubber outperforming FastSpeech by a big margin in term of audio quality. We attach some audio files and video clips generated by our model in the supplementary materials.
2 Related Work
Text to Speech. Text to Speech (TTS) [3, 51, 51,39], which aims to synthesize intelligible, natural and high-quality speech from the input text, has seen tremendous progress in recent years. Specifically, the prevalent methods have shifted from concatenative synthesis [19], parametric synthesis [53] to end-to-end neural network-based synthesis [33, 41, 51], where the quality of the synthesized speech is improved by large margin and is close to that of the human counterpart. The general paradigm of end-to-end neural network-based methods usually first generate the acoustic feature (e.g., mel- spectrogram) from text using encoder-decoder architecture autoregressively [51, 41, 34, 28] or non- autoregressively [39, 38, 24], then reconstruct the waveform signal using vocoder [16, 30, 37, 26, 54]. When it comes to the recent multi-speaker TTS system [20], the speaker embedding is often extracted using speaker verification system, and is fed to the decoder of the TTS system in order to encourage the model to obtain timbre inclination for the speaker of interest. Different from TTS, Neural
2

---

Video Script
Temporal Syuchronization
Automatic
Video
Dubbing
Temporal Synchronization
Figure 1: The schematic diagram of the automatic video dubbing (AVD) task. Given the video script and the video as input, the AVD task aims to synthesize speech that temporally synchronized with the video. This is scene where two people are talking with each other. The face picture is gray to indicate that the person was not talking at that time.
Dubber is conditioned not only on texts but also on videos, intending to synthesize natural speech given both of them.
Talking Face Generation. Talking face generation has a long history in computer vision, ranging from baseee-aseed models [14, 58] neural synthesis of 2D [43, 46, 52] 3D [21, 40, 45] face. Recently neural synthesis approaches have been proposed to generate realistic 2D video of talking heads. Concretely speaking, Chung et al. [9] first generates lower face animation using cropped frontal images. After then Zhou et al. [57] further disentangles identity from speech using generative adversarial networks (GANs). Wav2Lip [36] tries to explore the problem of visual dubbing, i.e.. lip-syncing a talking head video of an arbitrary person to match target speech segment. From our perspective, however, such methods can not generate high-fidelity face and lip given speech, spawning the results are of low resolution and look uncanny sometimes. Besides, audios in most talking face pipelines need to be prepared in advance, thus, strictly speaking, this does not belong to dubbing (re-recording)", but to the face synchronization while given audio. In contrast to the aforementioned works, Neural Dubber is not required to prepare audio beforehand and modify the lip motion, but generates speech audio synchronized with the video from scripts.
Lip to Speech Synthesis. Given a video, the lip to speech task aims at synthesizing the corre- sponding speech audio by directly judging from the lip motion. While the conventional method [22] exploits the visual features extracted from active appearance models, recent end-to-end methods have also shed some light on In particular, Vid2Speech [15] and Lipper [27] generate low-dimensional linear predictive coding features to synthesize speech in the constrained scene. Vougioukas et al. [49] using the GANs-based method exert for quality gains. Lip2Wav [35] has achieved promising results in real-life collapse the multi-speaker setting. This is possibly because the word error rate in lip reading task [2, 4, 10, 12] is still high, let alone the lip to speech synthesis. In Neural Dubber, the textual information is provided, allowing us to concentrate more on the alignment between the phoneme and lip motion in video, instead of decoding speech from lip motion directly.
scenarios, but it is still somewhat incongruous and prone to
3 Method
In this section, we first introduce the novel automatic video dubbing (AVD) task; we then describe the overall architecture of our proposed Neural Dubber; finally we detail the main components in Neural Dubber.
hhttps:/een, hattp::/een.wLLieedia.oo htt.ipeiaa.rggvvili.//. org/viks/Dubbing (f (filmmaking)
3

---

Hmel
Image-hased Speaker Embedding
Mel-spectrogram Decoder
Positional
Upsample
Trainable MLP
Encoding
Hcon
Variance Adaptor
Face Feature
Text- Aligner
Scaled Productd Attention
Image-based
Face Feature
Fx Block
FFT Block x 2
Speaker
Extractor
Positional
Embrelding
Lacodling
Module
Phoneme Embedding
Feature Extractor
H pho Hlvdd
Phonemes
Video Frames (Mouth Region) One Face Image
(a) Neural Dubber
(b) Text- VidcoVidoo Aligner
(c) ISE Module
Figure 2: The architecture of Neural Dubber.
3.1 Automatic Video Dubbing
Given a sentence I and a corresponding video clip (without audio) V, the goal of automatic video dubbing (AVD) to synthesize natural and intelligible speech S whose content consistent with T, and whose prosody is synchronized with the lip movement of the active speaker in the video V. Compared the traditional speech synthesis task which only generates natural and intelligible speech S given the sentence T, AVD task is more difficult due to the synchronization requirement.
3.2
Neural Dubber
3.2.1 Design Overview
Our Neural Dubber aims to solve the AVD task. Concretely, we formulate the problem as fol- lows: given phoneme sequence Sp (P1, P...... Pry) and video frame sequence S, (41,22....,,rr we need to predict target mel-spectrogram sequence Sm = (Y,, Y2.... Yrm).
The overall model architecture of Neural Dubber is shown in Figure 2. First, we apply phoneme encoder fp and video encoder fo process the phonemes and images respectively. Note that the images we feed to the video encoder only contain mouth region of the speaker following [10, 32, 42]. use S to represent these images. After the encoding, raw phonemes turn into a sequence of hidden representations Hpho (Sp) 5)55= T while images of mouth region turn into sequence of hidden representations Hvid (S TT.xd Then we feed Hpho and Havid into the text-video aligner (which will be described in Section 3.2.3) and get the expanded sequence Hmet E RT'm xd with the same length as the target mel-spectrogram sequence Sm, Meanwhile, face image randomly selected from the video frames input into image-based speaker embedding (ISE) module (which will be described in Section 3.2.4) to generate image-based based speaker embedding. We add H.mel and ISE together and feed them into the variance adaptor add some variance information (e.g., pitch and energy). Finally, we use the mel-spectrogram decoder to convert the adapted hidden sequence into mel-spectrogram sequence following [39]. Different from FastSpeech 2 [38], our variance adaptor consists of pitch and energy predictors without duration predictor, because we solve the problem of length mismatch between the phoneme and mel-spectrogram sequence text-video aligner and the input of variance adaptor is long the mel-spectrogram sequence.
3.2.2 Phoneme and Video Encoders
The phoneme encoder and video encoder are shown in Figure 2a, which are enclosed in dashed box. The function of the phoneme encoder and video encoder is to transform the original phoneme and image sequences into hidden representation sequences which contain high-level semantics. The phoneme encoder is similar to that in FastSpeech [39], which consists of an embedding layer

---

Transformer (FFT) blocks. The video encoder consists of a feature extractor and K FFT blocks. The feature extractor CNN backbone that generates feature representation for every input mouth image. And then we use the FFT blocks to capture the dynamics of the mouth region because FFT is based on self-attention [48] and ID convolution where self-attention and 1D convolution are suitable for capturing long-term and short-term dynamics respectively.
and N
3.2.3 Text- Video Aligner
The most challenging aspect of the AVD task is alignment: (1) the content of the generated speech should come from the input phonemes; (2) the prosody of the generated speech should be aligned with the video in time. So it does not make sense to produce speech solely from phonemes, nor video. In our design, the text-video aligner (Figure 2b) aims to find the correspondence between text and lip movement first, so that synchronized speech can be generated in the later stage.
In the tixeo aligner, an attention based module learns the alignment between the phoneme sequence and the video frame sequence, and produces the text-video context sequence. Then upsampling operation is performed to change the length of the text-video context sequence Hoon from T, to Tm..
In practice, we adopt the popular Scaled Dot-Product Attention [48] as the attention module, where Hevdd is used as the query, and phh is used as both the key and the value.
Attention(Q... (,, V) Attention (Huid, Hyho: Hpho)
(1)
HuiaH? pho
Softmax
Hpho
H.
(2)
Vd
AHHpho R... xxx
where is the matrix of attention weights. After the attention module, we get the text-video context sequence, i.e., the expanded sequence of phoneme hidden representation by linear combination. We use residual connection [18] to add the Hesd for efficient training. However, we use dropout layer with large dropout rate to prevent meelsppectrggrams from being generated directly from visual information. The attention weight A obtained after softmax is the main determinant of the speed and prosody of the synthesized speech like the attention weight between spectrograms and phonemes in [51, 41, 28]. The sequence of video hidden representations used as the query. As consequence, the attention weight controlled by the video explicitly, and the temporal alignment between video frames and phonemes is achieved. The obtained monotonic alignment between video frames and phonemes contributes to the synchronization between the synthesized speech and the video on fine-gre ine-grained (phoneme) level.
There is natural temporal correspondence between the speech audio and the video. In other words, once the aligmment between video frames and phonemes is achieved, the alignment between mel- spectrogram frames and phonemes be obtained. In practice, the length of mel-sppectoorrams sequence is n times that video frame sequence. We denote the n as
Tmel TV FPS E
sr/hs
where denotes the sampling rate of the audio and hs denotes hop size set when transforming the raw waveform into mel-spectrograms. We upsample the text-video context sequence Heon to Hyel with scale factor is Actually, use the upsampling method with nearest mode.
Heon = {C1, Cz...., Cret E Toxd
(5)
Hmet Upsample (Heon, n) E xm.
After that, the length of the text-video context sequence expanded that of the mel-spectrograms sequence. Thus, the length mismatch problem between the phoneme and mel-spectrograms sequence solved without the supervision of fine grained alignment between phonemes and spectrograms. Because of the attention between video frames and phonemes, the speed and part of prosody of synthesized speech are controlled by input video explicitly, which makes the synthesized speech and input video well synchronized.
5

---

Monotonic Alignment Constraint In text to speech (TTS) task, the monotonic and diagonal aligmments in the attention weights between text and speech are important ensure the quality of synthesized speech [51, 41, 44, 6]. In Neural Dubber, multi-modal TTS model, the monotonic and diagonal alignments between video frames and phonemes are also critical. So we adopt a diagonal constraint on the attention weights to guide the text-video attention module to learn right alignments following [6]. We formulate the diagonal attention rate r as
min((s+bb,Tp) m Aa,
Tm
where k Tp. b is hyperparameter for bandwidth of the diagonal area. We add the diagonal Te
constraint loss which is defined as LDC -r to our final loss for better alignments.
3.2.4 Image-based Speaker Embedding Module
How much can we infer about the way people speak from their appearances? In the real dubbing scenario, voice actors need to alter the timbre according to different performers. In order to better simulate the real case in AVD task, we aim to symthesize speech with different timbre conditioning on the speakers' faces in multi-speaker setting. There have been many works [29, 23, 13] researching the correlation between voice and speakers' face recently, but none of them learn the joint speaker- face embeddings solve the multi-speaker text to speech task. In this work, we propose image- image-based speaker embedding (ISE) module (Figure 2c), a new generates embedding that encapsulates the characteristics of the speaker's voice from an image of his/her face. The ISE module is trained with other components of Neural Dubber from scratch in self-supervised manner, utilizing the natural co-occurrence of faces and speech audio in videos, but without the supervision of speaker identity. We randomly select a face image I! from S
speaker embedding module,
{ 14,) }. and obtain a high-level face feature by feeding the selected face image into pre-trained and fixed face recognition network [31, 5]. Then we feed the face feature to trainable MLP and gain the ISE. The predicted ISE directly broadcasted and added to Himel so as to control the timbre of synthesized speech. Our model learns face-voice correlations which allow it to produce speech that coincides with various voice attributes of the speakers (e.g., gender and age) inferred from their faces.
4 Experiments and Results
4.1 Datasets
Single-speaker Dataset In the single-speaker setting, we conduct experiments on the chemistry lecture dataset from Lip2Wav [35]. With large vocabulary size and lot of head movements, the dataset is originally used for the unconstrained single-speaker lip to speech synthesis. To make it fit the AVD task, we collect the official transcripts from YouTube. We need corresponding sentence-level text and audio clips for training, so we segment the long videos into sentence- clips according to the start and end timestamp of each sentence in the transcripts. Some segmented sentence-level video clips contain frames that only capture the PowerPoint but not lecturer face which can not be used for training. So we conduct data cleaning to remove them. Finally, the dataset contains 6,640 samples, with the total video length of approximately 9 hours. We randomly split the dataset into sets: 6240 samples for training, 200 samples for validation, and 200 samples for testing. In the following subsections, we refer to this dataset chem for short.
Multi-speaker Dataset In multi mullii-ppeakrr setting, we conduct experiments on the LRS2 [1] dataset, which consists of thousands of sentences spoken by various speakers on BBC channels. This dataset suits the AVD task well, because each sample includes both the text and video pair. Note that we only train on the training set of the LRS2 dataset, which only contains data of approximately 29 hours. Compared to other multi-speaker speech synthesis datasets [55], this dataset is quite small for multi-speaker speech generation and does not provide the speaker identity for each sample. The ISE module aids Neural Dubber in solving these problems.

---

4.2 Data Pre- Pre-processing
The video frames are sampled at 25 FPS. We detect and crop the face from the video frames using S FD [56] face detection following [35]. The images input to the video encoder are resized to 96 96 in dimension, which only cover the mouth region of the face, as shown in Figure 2a. The face image input to the ISE module is 224 224 in dimension and covers the whole face of the speaker. In order to alleviate the mispronunciation problem, we convert the text sequences into the phoneme sequences [3, 28, 38] with an open-source poppeee-to-pponeme tool. For the speech audio, we transform the raw waveform into seesspecss following [41]. The frame size and hop size are set to 640 samples (40 ms) and 160 samples (10 ms) with respect to the sample rate 16 kHz.
4.3 Model Configuration
Neural Dubber
Our Neural Dubber consists of 4 feed-forward Transformer (FFT) blocks [39] in the phoneme encoder, the mel-spectrogran decoder, and 2 FFT blocks in the video encoder. The feature extractor in the video encoder is the ResNet18 [18] except for the first 2D convolution layer being replaced by 3D convolutions [32]. The variance adaptor contains pitch predictor and energy predictor. The configurations of the FFT block, the mel-spectrogram decoder, the pitch predictor and the energy predictor are the same as those in FastSpeech 2 [38]. In the text-video aligner, the hidden size of the scaled dot-product attention is set to 256, the number of the repeat operation is set according to Equation (4). In the ISE module, the face feature extractor we use is a pre-trained and fixed ResNet50 trained on the VGGFace2 [5] dataset. The face feature is a 4096-D feature that is extracted from the penultimate layer (i.e., one layer prior to the classification layer) of the network.
Baseline Since automatic video dubbing is new task that we propose, none of the previous works focused on solving this task. So we propose a baseline model based on the Tacotron [51] system with some modifications which make it fit to the new AVD task. We call this baseline model Video-based Tacotron. In order to make use of the information in video, we concatenate the spectrogram frames with the corresponding hidden representation of video frames, and use it the decoder input:
Y =-=== Y,-1 H
where Yi_; is the decoder input, ① represents the concatenation operation, Hvid is the hidden representation of video frames, which is obtained in the same way in Neural Dubber described in the Section 3.2.1 and n is same as that in Equation (4). The Video-based Tacotron implementation is based on an open-source Tacotron repository where the attention is replaced with the location- sensitive attention [7] according to [41] for better results. We set the reduction factor r to 2 and change the vocoder Parallel WaveGAN [54] for fair comparison.
4.4 Training and Inference
We train Neural Dubber on 1 NVIDIA V100 GPU. We use the Adam optimizer [25] with 31 0.9, 32 0.98, 10-2 and follow the same learning rate schedule in [48]. Our model is optimized with the loss similar to that [38]. We set the batchsize to 18 and on chem dataset and LRS2 dataset respectively. It takes 200k/300k steps for training until convergence on the chem/LRS2 dataset. In this work, we use Parallel WaveGAN [54] as the vocoder to transform the generated mel-spectrograms into audio samples. We train two Parallel WaveGAN vocoders the training set of chem dataset and LRS2 dataset respectively, following an open-source implementation 4, Each Parallel WaveGan vocoder is trained on NVIDIA V100 GPU for 1000K steps. In the inference process, the output mel-spectrograms of Neural Dubber are transformed into audio samples using the pre-trained Parallel WaveGAN.
4.5 Evaluation
4.5.1
Metrics
Since the AVD task aims to synthesize human speech synchronized with the video from text, the audio quality and the audio-visual synchronization (av sync) are the important evaluation criteria.
httrpp///gittibbboom///ttttt///r com/Efatchhorr///eeettttt
'https:// tettps://githar com/kan- http//anw-aaaaalll/aaalllllllaae
7

---

Human Evaluation We conduct the mean opinion score (MOS) [8] evaluation on the test set to measure the audio quality and the av syne. We randomly select 30 video clips from the test set, where each video clip is scored by at least 20 raters, who are all native English speakers. We overlay the synthesized speech on the original video before showing it to the rater, following [35]. The text and the video are consistent among different systems, so that all raters only examine the audio quality and the av syne without other interference factors. For each video clip, the raters are asked to rate scores of 1-5 from bad to excellent (higher score indicates better quality) on the audio quality and the av sync, respectively. We perform the MOS evaluation on Amazon Mechanical Turk (MTurk).
Quantitative Evaluation In order to measure the synchronization between the generated speech and the video quantitatively, we use the pre-trained SyneNet [11], which is publicly available* following [35]. The method can explicitly test for synchronization between speech audio and lip movements in unconstrained videos in the wild [11, 36]. We adopt two metries: Lip Sync Error Distance (LSE-D) and Lip Syne Error Confidence (LSE-C) from Wav2Lip [36]. The two metrics can be automatically calculated by the pre-trained SyncNet model. LSE-D denotes the minimal distance between the audio and the video features for different offset values. A lower LSE-D means the speech audio and video are more synchronized. LSE-C denotes the confidence that the audio and the video are synchronized with certain time offset. A lower LSE-C means that some parts of the video are completely out of sync, where the audio and the video are uncorrelated.
Speakers
F2
F4
F5
M5
F6
M6
F7
M7
ND
F8
MB
F9
M9
F10
M10
F11
M11
FS2
F12
VI
Figure 3: Mel-spectrograms of audios synthesized by some systems: Ground Truth (GT), Neural Dubber (ND), FastSpeech 2 (FS2) and Video-based Tacotron (VT).
Figure 4: Speaker embedding visualization.
4.5.2 Single-speaker AVD
We first conduct MOS evaluation on the chem single-speaker dataset, compare the audio quality and the av sync of the video clips generated by Neural Dubber with other systems, including 1) GT, the ground-truth video clips; GT (Mel PWG), where we first convert the ground-truth audio into mel-spectrograms, and then convert it back to audio using Parallel WaveGAN [54] (PWG); 3) FastSpeech 2 [38] (Mel+ + PWG): 4) Video-based Tacotron (Mel PWG). Note that the systems in 2). 3), 4) and Neural Dubber use the same pre-trained Parallel WaveGAN for a fair comparison. In addition, we compare Neural Dubber with those systems on the test set using the LSE-D and LSE-C metrics. The results for single-speaker AVD are shown in Table 1. It can be seen that Neural Dubber can surpass the Video-based Tacotron baseline and on par with FastSpeech 2 in terms of audio quality, which demonstrates that Neural Dubber can synthesize high-quality speech. Furthermore, in terms of the av sync, Neural Dubber outperforms FastSpeech and Video-based Tacotron by a big margin and matches GT (Mel PWG) system in both qualitative and quantitative evaluations, which shows that Neural Dubber can control the prosody of speech and generate speech synchronized with the video. For FastSpeech 2 and Video-based Tacotron, the LSE-D is high and the LSE-C is low,
bhtttp:://giee http://.....ooooooo///nnnnnttttt
8

---

indicating that they can not generate speech synchronized with the video. We also show qualitative comparison in Figure which contains speettrrsss of audios generated by the above systems. It shows that the prosody of the audio generated by Neural Dubber is closed to that of ground truth recording, i.e., well synchronized with the video.
Method
Audio Quality
AV Syne
LSE-D L
LSE-C f
GT GT (Mel + PWG)
3.93 + 0.08
4.13 = 0.07
6,926
7.711
3.83 + 0.09
4.05 = 0.07
7.384
6.806
FastSpeech 2 [38] (Mel PWG)
3.71 +: 0.08
3.29 = 0.09
11.86
2.805
Video-based Tacotron (Mel + PWG)
3.55 + 0.09
3.03 = 0.10
11.79
2.231
Neural Dubber (Mel + PWG)
3.74 + 0.08
3.91 = 0.07
7.212
7.037
Table I:
The evaluation results for the single-speaker AVD. The subjective metrics for audio quality and av sync are with 95% confidence intervals.
In addition, we compare our method with another baseline [17] which automatically stretches and compresses the audio signal to match the lip movement given an unaligned face sequence and speech audio. We use the speech generated by FastSpeech 2 (Mel + PWG) system, and then align the pre- rre-generated speech with the lip movement according to [17]. However, the quality and naturalness of its synthesized speech is much worse than the pre-generated speech due to challenging alignments. So this baseline not comparable to our Neural Dubber.
4.5.3 Multi-speaker AVD
Similar Section 4.5.2, we conduct human evaluation and quantitative evaluation on the LRS2 multi-speaker dataset to compare Neural Dubber with other systems in multi-speaker setting. Due to the failure of Video-based Tacotron in single-speaker AVD, we no longer compare our model with it. Note that we can not add a trivial speaker embedding module to FastSpeech 2, because the LRS2 dataset does not contain the speaker identity for each video. So directly train FastSpeech 2 on the LRS2 dataset without modifications. The results are shown in Table 2. We can see that Neural Dubber outperforms FastSpeech by a significant margin in terms of audio quality, exhibiting the effectiveness of ISE in multi-speake AVD. The qualitative and quantitative evaluations show that the speech synthesized by Neural Dubber is much better than that of FastSpeech 2 and is on par with the ground truth recordings in terms of synchronization. These results show that Neural Dubber can address the multi-speaker AVD which is more challenging than the single-speaker AVD.
In order to demonstrate that ISE enables Neural Dubber to control the timbre by the input face image, some audio clips are generated by Neural Dubber with the same phoneme sequence and mouth image sequence but different speaker face images as input. We select 12 males and 12 females from the test set of LRS2 dataset for this evaluation. For each person, we chose 10 face images with different head posture, illumination and facial makeup, etc.
We visualize the voice embedding of these audios in Figure 4 by using pre-trained speaker en- coder [50] from an open-souree repository We first use the speaker (voice) encoder to derive a high-level representation, i.e., 256-D embedding, from audio, which summarizes the character- istics the voice in the audio. Then we use t-SNE [47] to visualize the generated embedding. can be seen that the utterances generated from the images of the same speaker form a tight cluster, and that the cluster representing each speaker is separated from each other. In addition, there is distinctive discrepancy between the speech synthesized from the face images of different genders. concludes that Neural Dubber can use the face image to alter the timbre of the generated speech.
4.5.4 Comparing with the Lip-motion Based Speech Generation Method
Recently, some works have demonstrated the impressive ability to generate speech directly from the lip motion. However, the quality and intelligibility of the generated speech are relatively poor, and the word error rate (WER) is very high. In this section, we compare with a SOTA lip-motion based speech generation system Lip2Wav [35]. Because Lip2Wav can only generate word-level speech in the multi-speaker setting, we only compare Neural Dubber with Lip2Wav in the single-speaker
'nttps://githan http:p////gttuuu.ommm/emmem--aaa www.el/eeeeeeeeeeeeeeeeeeee

---

Method
Audio Quality
AV Syne
LSE-D 1
LSE-C1
GT
3.97 + 0.09
3.81 + 0.10
7.214
6.755
GT (Mel + PWG)
3.92 + 0.09
3.69 + 0.11
7.317
6.603
FastSpeech 2 [38] (Mel+ PWG)
3.15 + 0.14
3.33 + 0.10
||
10.17
3.714
Neural Dubber (Mel + PWG)
3.58 + 0.13
3.62 + 0.09
||
7.201
6.861
Table The evaluation results for the multi-speaker AVD. The subjective metrics for audio quality and sync with 95% confidence intervals.
setting still on the chem dataset. We use the official GitHub repository' to train Lip2Wav on our version of the chemistry lecture dataset. we mentioned in Section 4.1, the dataset is different from the original one in Lip2Wav. It only contains data of approximately 9 hours, which is much less than the original one (approximately 24 hours). In this experiment, the training and testing sets of Neural Dubber and Lip2Wav are identical, so the results can be compared directly. Following the Lip2War Way paper [35], we use STOI and ESTOI for estimating the intelligibility and PESQ for measuring the quality. In addition, using out-of-the-box ASR system, we evaluate the speech results using word error rates (WER). order to eliminate the influence of the ASR system, we also measure the WER for ground truth speech audio. All these metrics are computed on the test dataset.
Method
STOI t
ESTOI
PESQ
WER
Ground Truth
NA
NA
NA
7.57%
Lip2Wav
0.282
0.176
1.194
72.70%
Neural Dubber (ours)
0.467
0.308
1.250
18.01 %
Table 3: The comparison between Lip2Wav and Neural Dubber for the single-speaker setting on the chemistry lecture dataset.
As the comparison results in Table 3 show, Neural Dubber surpasses Lip2Wav by a big margin in terms of speech quality and intelligibility. Please note that STOI, ESTOI, and PESQ scores of Lip2Wav are lower than those in [35]. because the training data we used is much less than theirs. Most importantly, the WER of Neural Dubber 4x lower than that of Lip2Wav. It shows that Neural Dubber outperforms Lip2Wav significantly in pronunciation accuracy. WER of Lip2Wav is up to 72.70% indicating that it mispronounces a lot of content, which is unacceptable in the AVD task. Just like it is unacceptable for an actor to always mispronounce the lines. Please note that the WER of Lip2Wav we get is consistent with the results in [35] (see its Table 5). In summary, Neural Dubber far outperforms Wiv2 in terms of speech intelligibility, quality, and pronunciation accuracy (WER), and is much more suitable for the AVD task.
5 Limitations and Societal Impact
When the script is changed to be different from what the speaker is actually saying, our method only deal with the situation of modifying couple words. In addition, the lip movement of the modified text should be similar to the original lip movement in the video. The facial appearance may lead to timbre ambiguity due to the dataset bias. It might be offensive. Our method can dub videos automatically, which may be useful for filmmaking and video production.
6 Conclusion
In this work, we introduce a novel task, automatic video dubbing (AVD), which aims to synthesize human speech synchronized with the given video from text. To solve the AVD task, we propose Neural Dubber, a multi-modal TTS model, which can generate lip-synced mel-spectrograms in parallel. We design several key components including the video encoder, the text-video aligner and the ISE module for Neural Dubber to better solve the task. Our experimental results show that, in terms of speech quality, Neural Dubber is on par with FastSpeech on the chem dataset, even outperforms FastSpeech 2 on the LRS2 dataset due to ISE's help. More importantly, Neural Dubber can synthesize speech temporally synchronized with the video.
"https://githae htt.thhawb.oo/udrohh///lpplllllp
10

---

References
[1] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 2018. 2,
[2] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Deep lip reading: a comparison of models and an online application. In Interspeech, 2018.
[3] Sercan Ô Ank, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yong- guo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al. Deep voice: Real-time neural text-to-speech. In International Conference on Machine Learning, pages 195-204. PMLR, 2017. 7.7
[4] Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando De Freitas. Lipnet: End-to-end sentence-level lipreading. arXiv preprint arXiv:/611.01599 2016. 3
[5] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 67-74. IEEE, 2018. 6,
[6] Mingjian Chen, Xu Tan, Ren, Jin Xu, Hao Sun, Sheng Zhao, Tao Qin, and Tie- Yan Liu. Multispeech: Multi-speaker text to speech with transformer. arXiv preprint arXiv:2006.0m 04664, 2020.
[7] Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-baser models for speech recognition. arXiv preprint arXiv: 150.5:1500.07503,, 2015. 7
[8] Min Chu and Hu Peng. Objective measure for estimating mean opinion score of synthesized speech, April 4 2006. US Patent 7,024,362.
[9] Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? In British Machine Vision Conference, 2017.
[10] Joon Son Chung. Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In 20/7 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3444-3453. IEEE, 2017. 2,
[11] Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Asian conference on computer vision, pages 251-263. Springer, 2016.
[12] Joon Son Chung and Andrew Zisserman. Lip reading in profile. In British Machine Vision Conference, 2017.
[13] Soo-Whan Chung, Hong Goo Kang, and Joon Son Chung. Sceing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision. arXiv preprint arXiv:2004. 14326, 2020. 6
[14] Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. Jali: an animator-centric viseme model for expressive lip synchronization. ACM Transactions Graphics (TOG), 35 2016.
[15] Ariel Ephrat and Shmuel Peleg, Vid2speech: speech reconstruction from silent video. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5099- IEEE, 2017.
[16] Daniel Griffin and Jae Lim. Signal estimation from modified short-tim fourier transform. IEEE Transactions on acousties, speech, and signal processing, 32(2):236-22 22333, 1984. 2
[17] Tavi Halperin, Ariel Ephrat, and Shmuel Peleg. Dynamic temporal alignment of speech to lips. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3980-3984. IEEE, 2019.
11

---

[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 5,
[19] Andrew J Hunt and Alan W Black. Unit selection in a concatenative speech synthesis system using a large speech database. In 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings, pages 373-376. IEEE, 1996. 2
[20] Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang. Ignacio Lopez Moreno, et al. Transfer learning from speaker verifica- tion multispeaker text-to-speedt synthesis. In Advances in Neural Information Processing Systems, 2018. 2
[21] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics (TOG), 36( 2017.
[22] Christopher T Kello and David C Plaut. A neural network model of the articulatory-acousicc forward mapping trained on recordings of articulatory parameters. The Journal of the Acoustical Society of America, (16(4)22354-2366 2004. 3
[23] Changil Kim, Hijung Valentina Shin, Tae-Hyun Oh, Alexandre Kaspar, Mohamed Elgharib, and Wojeiech Matusik. On learning associations of faces and voices. In Asian Conference on Computer Vision, pages 276-292. Springer, 2018. 6
[24] Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for texe-th- via monotonic alignment search. arXiv preprint arXiv:2005. 2020.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 7
[26] Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. arXiv preprint 191.0:191,..6711, 2019. 2
[27] Yaman Kumar, Rohit Jain, Khwaja Mohd Salik, Rajiv Ratn Shah, Yifang Yin, and Roger Zimmermann. Lipper: Synthesizing thy speech using multi-view lipreading, In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2395, 2019.
[28) Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6706-6713, 2019. 5.7
[29] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Learnable pins: Cross-modal em- beddings for person identity. In Proceedings of the European Conference on Computer Vision (ECCV), pages 71-88, 2018. 6
[30] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: /609. 03499, 2016. 2
[31] Omkar M Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. 2015, 6
[32] Stavros Petridis, Themos Stafylakis, Pingehuan Ma. Feipeng Cai, Georgios Tzimiropoulos, and Maja Pantic. End-to-end audiovisual speech recognition. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6548-6552, IEEE, 2018. 4,
[33] Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-to-speech. In International Conference on Learning Representations, 2019. 2
[34] Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 2000-speaker neural text-to-speech. Proc. ICLR, pages 214-217, 2018. 2
12

---

[35] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. Learning individual speaking styles for accurate lip to speech synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15796-13803 2020. 2,3, 7,
8, 9, 10
[36] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, pages 484-492, 2020. 3,
[37] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617-3621. IEEF, 2019. 2
[38] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021. 2, 4, 8, 9, 10
[39] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Systems, 2019. 2, 4,7 7
[40] Alexander Richard, Michael Zollhoefer, Yandong Wen, Fernando de la Torre, and Yaser Sheikh. Meshtalk: 3d face animation from speech using cross-modality disentanglement. arXiv preprint arXiv:2104.08223, 2021. 3
[41] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural uts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779-4783. IEEE, 2018. 2, 5, 6, 7
[42] Themos Stafylakis and Georgios Tzimiropoulos. Combining residual networks with Istms for lipreading. arXiv preprint 17X3.01703.0405 2017. 4
[43] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-SShizeemeee Synthesizing obama: learning lip syne from audio. ACM Transactions on Graphics (ToG), 36(4): 2017.
3
[44] Hideyuki Tachibana, Katsuya Uenoyama, and Shunsuke Aihara. Efficiently trainable text-to- speech system based on deep convolutional networks with guided attention. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4784- 4788. IEEE, 2018. 6
[45] Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, and lain Matthews. A deep learning approach for generalized speech animation. ACM Transactions on Graphics (TOG), 36(4):1-11, 2017.
[46] Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. In Proceedings of the European conference on computer vision (ECCV), pages 716-731, 2020. 3
[47] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv:1176..000 03762 2017. 5.7
[49] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial animation with gans. International Journal of Computer Vision, 128(5):1398-1413 2020. 3
[50] Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4879-4883. IEEE, 2018. 9
13

---

[51] Yuxuan Wang, RJ Skenry-Ryan Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv: 1703. 10135, 2017. 2, 6,
[52] Olivia Wiles, A Koepke, and Andrew Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In Proceedings of the European conference on computer vision (ECCV), pages 670-686, 2018. 3
[53] Zhizheng Wu, Oliver Watts, and Simon King. Merlin: An open source neural network speech synthesis system. In SSW, pages 202-207, 2016. 2
[54] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-20000 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199 6203. IEEE, 2020. 2, 7,8
[55] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv: 02882, 2019.
[56] Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, and Stan Z Li. S3fd: Single shot scale-invariant face detector. In Proceedings of the IEEE international conference computer vision, pages 192-201, 2017. 2017. 7
[57] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by udversarially disentangled audio-visual representation. Proceedings of the AAAI Conference on Artificial Intelligence, pages 9299-9306, 2019. 3
(58) Yang Zhou, Zhan Xu, Chris Landreth, Evangelos Kalogerakis, Subhransu Maji, and Karan Singh. Visemenet: Audio-driven animator-centric speech animation. ACM Transactions on Graphics (TOG), 17- 2018. 3
14


Reviews:
[{"review_id": "zzs2vApHNKV", "reviewer": "Reviewer_vq7e", "summary": "This paper describes a novel task 'silent video dubbing' and proposes a method to solve it. The task involves generating realistic audio to accompany silent video, with the help of text. This is essentially similar to vid2speech [14], but with text guidance. The pipeline consists of several stages. (1) generating phoneme and video embeddings; (2) aligning text and video using the embeddings from (1) and attention network; (3) generating speaker embeddings to condition the decoder from images; (4) using (2) and (3) to generate the result voice. The authors provide example in supplementary materials. The experiments are performed on both single-speaker and multi-speaker SVD.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The task of dubbing silent videos using neural networks is novel to my knowledge. This is an interesting problem in film-making industry. \n\nThe pipeline well-engineered, and is well explained. The individual components in the pipeline are fairly standard, rather than novel contributions. However, credit can still be given to how these are put together.\n\nThe authors imply that ISE is a contribution of their own, but there are many previous works to learn joint speaker-face embeddings [1,2,3].\n\nI do not think that LSE-D is a fair metric, since this is too similar to the way in which text-video aligner is trained using attention alignment, therefore this unfairly biases towards the proposed method.\n\nOtherwise, qualitative and quantitative results appear to be good.\n\n\n[1] https://arxiv.org/abs/1805.00833\n[2] https://people.csail.mit.edu/changil/assets/face-voice-accv-2018.pdf\n[3] https://arxiv.org/abs/2004.14326", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not addressed this in any detail. They can mention the usual risks associated with deepfake-like research.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2hr"}, "extra_text": {}}, {"review_id": "hP0Z2oDYvyR", "reviewer": "Reviewer_RPCL", "summary": "The authors describe the problem of automatic dubbing of videos, ie generating speech from a script that matches the active speaking face in a video. To the best of this reviewer's knowledge, this is the first paper addressing this particular problem. The two main contributions to solve this problem is 1. a method to synchronize the generated TTS with the speaking face 2. sampling the video for a face image to infer a timbre for the speech to better match the face. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors have proposed what seems to be a novel problem and presented a solution which they have evaluated. The problem itself is closely related to a variety of work including [14, 31] where TTS is generated from lip video, without a text constraint (Vid2Speech, Lip2Wav). And Large-scale multilingual audio visual dubbing (https://arxiv.org/pdf/2011.03530.pdf) where the video is alternated to match the generated TTS (translated from the original script) [Yang20].\n\nThe evaluation is somewhat flawed in that videos with speech in English have the speech audio re-synthesized. In more practical situations, the dubbed speech is required in a language different to the spoken language in a video. The challenge is coming up with a good  synchronization when the phoneme sequences do not match due to the language mismatch. Still the results are fairly impressive (the demo clips in the appendix are much appreciated).\n\nOriginality\nThe work is original,  although somewhat contrived in that you rarely will have the case that the visemes will match the phonemes well; usually the language of viseme will be different than the phonemes in a new dub. It would be interesting to see how Neural Dubber works in this case.\n\nClarity\nOverall, the paper very nicely introduces the problem which they describe as Silent Video Dubbing.  Perhaps a more accurate term would be Automatic Video Dubbing, since the notion of dubbing already means that the audio track is being replaced with another within the video. SVD is already a common term within the machine learning community, and AVD follows from other similar terms like ASR for automatic speech recognition. The term \"silent video\" is also awkward; while video can mean media that has both video and audio in the common sense, technically it usually denotes just visual information (https://en.wikipedia.org/wiki/Video) as opposed to audio-visual data. If we don't qualify that video is only visual information, then terms like \"Video-based\" Tacotron, should be called \"Silent video-based\" Tacotron, which is strange. In section 4.1 only evaluation datasets are mentioned. Later, some mentions of training data are provided. It would be helpful to include them in section 4.1 to understand what actually needs to be trained and on what corpora.\n\nQuality\nThe paper clearly motivates this Automatic Video Dubbing problem, presents a solution to guiding the TTS generation to better match the speaking face, and provides a comprehensive analysis of the results with quantitate measurement using AV Sync and qualitative (human ratings). They attempt to provide a good competing baseline system, Video-base Tacotron (although result don't seem so good as the spectrograms don't synchronize well), but the reviewer appreciates the effort. There are results for single speaker and a variety of speakers. The ISE system, is helpful to generate voices that match the speaking faces gender or age, but perhaps can make unfair unbiases on what people should sound like based on their appearance. Still, the paper is comprehensive, and by providing the source code in the appendix, and operating on publicly available data sets, gives experimenters the best chance of being able to reproduce results.\n\nReferences\nThe author's have an odd referencing style. There is no need to reference multiple papers for a particular technique, only the first is required. e.g. if [35] is the first example of non-autoregressive TTS, then there is no need to cite [34 ] and [21] as well. Further, [34] is a follow on to [35] (FastSpeech), so unless the paper is citing some thing new in [34], it is better to only cite [35] for anything related to FastSpeech (many times both citations appear together). \n\nSignificance\nThe only downside to the paper, is how the Neural Dubber has been evaluated since re-generating English speech for a video of a person speaking English is a somewhat academic as mentioned earlier (if you already have the English speech, why do you need to re-generate it?). It would be interesting to see how the system would behave would the target phonemes mismatch the viseme language. Unfortunately, this is not answered in this paper. Thus work like in Yang20, is more relevant to the AVD task in practice. The capitalization on the references need to be fixed, e.g. t-sne->T-SNE, FastSpeech, tts -> TTS. Skerry is misspelled in line 449. Please go over the references to make sure they are accurately written.\n\nMinor comments:\n\nline 77: these are not the first papers on TTS. please update.\n\nFigure 1 seems to show that the SVD system doesn't use the green/blue face tracks since they bypass the box, but it fact it does (subtley the video is passed in via the black line). Maybe this could be clearer.\n\nFigure 2. would be helpful to add the labels for S_p, S_n, S_v etc.\n\nline 111: this set of references seemed biased/odd; this is not the complete set of papers evaluating lip reading. it makes more sense to use one that is the most recent state-of-the-art.\n\nline 126-128. the notation is little non-standard. typically lower case letter for scalar, bold lower case for vector and bold upper case for matrix. Thus the phone sequence is \\bm{p} = {p_1, p_2 ... and the subscript can mean an index instead of being also overloaded as a indicator of what it is in S_p.\n\nline 150: add that the FFT is simply a transformer with the dense layer replaced with conv. It's an odd/confusing name since the transformer is already feed-forward and FFT usually refers to fast Fourier transform.\n\nEquation (1). the dimensionality of H_vid and H_pho is different, how is this addressed?\n\nEquation (4), the terms are a mixture of quantities e.g. hop size, same rate and units e.g. frame per second. They should all be some quantity e.g. frame rate. \n\ntypo line 220\n\nadd units in line 239.\n\nPerhaps the Video-based Tacotron would preform better with a constrained alignment process.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors might discuss how using a face to influence what the speaking voice sounds like may have some negative impacts or introduce unwelcome bias. What would happen if a female speaker with masculine features were used: would producing a male-sounding voice be offensive? How would race play a role in the sound of the voice?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "hLqNsnXZ8F", "reviewer": "Reviewer_N6Lx", "summary": "In this paper, the authors propose to solve the problem of text to speech generation based on video. Thus, this is video-text guided speech generation task that they term as silent video dubbing. The method is based on a transformer architecture that combines text and visual lip motion representations as encoders and outputs the mel-spectrograms through the decoder of a transformer. \n\nThe method is compared with TTS systems and comparisons are done using human evaluation scores and audio-visual synchronisation scores. This is done over LRS2 and Lip2Wav for one out of 5 speakers. Comparisons are not done using standard PESQ, STOI or Word error rates. These are standard metrics used for TTS systems. Comparison is also not provided with single speaker Lip2Wav task or multi-speaker Lip2Wav on LRW dataset as done by Lip2Wav or Vid2Speech, or such methods that generate speech based on lip motion alone (not requiring text). ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method aims to solve the task of `silent video dubbing’ where given a video and a text, the aim is to generate the corresponding speech that has the same prosody as the video, thereby enabling the task of dubbing. \n\nThe method is based on a text-video alignment based transformer that additionally uses an image specific embedding with a variance adaptor (used in FastSpeech2) to match pitch and energy for the generated mel-spectrograms generated by the decoder. As is common in transformer architectures, positional encoding blocks are included.  The proposed architecture is modified from existing TTS systems to include lip video embeddings as an additional cue. \n\n\nPros:\nThe problem aims to address the generation of speech conditioned on text and video as compared to existing methods that aim to generate speech based only on text or generation of speech based only on lip motion. \n\nThe proposed method shows evaluation on LRS2 dataset and evaluation for a single speaker from a speaker-specific dataset. The evaluation is done using MOS scores, evaluation of a speaker specific speech generation visualisation and lip-synchronisation accuracy. The comparisons are done with TTS systems.\n\n\nCons:\n\nEvaluation limitations:\n\nThe main limitation of this work is that while, the novel task aims to combine the ability to generate speech from text and lip-motion, the comparisons are provided only with TTS systems and not with lip-motion based speech generation systems. Clearly, if the method uses more information as compared to lip-motion based speech generation systems, it would be relevant to compare their output with such systems\n\nThe second limitation of this work is that the evaluation measures are obtained through human evaluation and synchronisation, however, the standard metrics used for TTS systems such as PESQ, STOI are not evaluated. This is important as the quality of speech generation is very crucial for dubbing applications (as proposed in the paper). The overall qualitative comparison provided suggests that the actual audio quality is somewhat inferior to TTS systems, though the synchronisation appears to be better.\n\nSimilarly, when evaluating on single speaker setting, it would have been easy to provide a comparison with Lip2Wav as the setting is similar (only more information is added). The lip-motion based systems have also been compared using PESQ and STOI metrics appear to provide better audio as compared to the proposed system (quantitative comparison is not possible due to lack of results through the metrics).\n\nThis implies that though the proposed method uses more parameters (transformer architecture) and more information, the quality appears to be lesser as compared to lip-motion based generation and text-to-speech based systems. \n\n\nMinor writing issues:\n\nWhile, on the whole the paper is well written and clear, the paper has a few minor grammatical/spelling errors. The line numbers for the same are provided:\n38 “an one-to-many”\n73 “in a big margin”\n108  “quailty gains”\n127 “an phoneme sequence”\n154 “are suit for capturing”\n\nConclusion:\nTo conclude, while the paper proposes an interesting variant, due to the limitations in terms of evaluation, I tend to be negative about this work.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The submission can include possible societal concerns, though its application for creative industries can be useful.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {"ethical_concerns": "The authors acknowledge that the ethical issues are not fully addressed. Presently it uses only publicly available data sources. Thus, the issues are not different to existing similar systems. This can be acknowledged."}}, {"review_id": "RgorrOI46zY", "reviewer": "Reviewer_nxuo", "summary": "This paper presents \"Neural Dubber\", a method for dubbing silent videos based on given scripts. It essentially performs lip-synced TTS, with an added component (ISE) that ensures that the generated speech characteristics match the given face image. \n\nExperiments verify that the proposed TTS method does not hurt SOTA TTS performance, while controlling the prosody such that the aligned speech is much more synchronized than other methods compared to. The supplementary video qualitatively demonstrates some compelling results of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:**\n\nThe task of performing end-to-end lip-synced TTS is, as far as I am aware of, a novel one. The seemingly effective method used to solve this task is a well-motivated combination of components well-documented in the literature: per-modality feature extraction and processing using transformers --> attention-based alignment --> joint decoding. \n\nThe main weakness of this work is its conspicuously-missing reference to a very related work: \"Dynamic Temporal Alignment of Speech to Lips\", Halperin et al. (ICASSP'19), which, given an unaligned face sequence and audio track, automatically stretches and compresses the audio signal to match the lip movement. Using this method to provide a stronger baseline to compare Neural Dubbing against would help amplify the value of the proposed method, especially since the included baseline (video-based Tacotron) is  weak and uninformative.\n\n**Quality:** \n\nAs stated above, the method is well-motivated, and experiments verify that:\nThe proposed TTS method does not hurt SOTA TTS performance, as measured by listening study, while controlling the prosody such that the aligned speech is much more synchronized than other methods compared to, also subjectively measured. Objective SyncNet-based metrics also demonstrate superiority over baselines. The supplementary video qualitatively demonstrates some compelling results of the proposed method.\n\nThat said, a stronger baseline, as suggested above, would significantly strengthen these claims. Also, in the checklist the authors answered that they did describe the limitations of their work, however, I can't seem to find where this is mentioned. Another experiment that I'm curious about is: how well does this method work when the script is significantly different from what the person is actually saying, e.g., a sentence in another language, or just different words in the same language? \n\n**Clarity:**\n\nThe manuscript is clear, and seems to contain enough information to reproduce results. One correction I spotted:\nl.154 are suit --> are suitable\n\n**Significance:**\n\nThe significance of this work is in the fact that it presents and provides a solution for a novel problem, which is interesting from both a research perspective and from the real-world application viewpoint.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "IMO this aspect of the paper is missing. Discussing limitations will strengthen the manuscript, not weaken it. In addition, the proposed method could potentially be used for generating fake talking videos, but this depends on how well the method works on scripts different from what is actually being said by the speaker in the video (I assume this will not work well enough to deceive people).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The original submission was missing an essential comparison with Lip2Wav. This has been added in the rebuttal period, and all reviewers now recommend acceptance.

The authors should include the Lip2Wav comparison and the missing references highlighted by the reviewers in the final version. 
Decision: Accept (Poster)