Neural Dubber: Dubbing for Videos According to Scripts

1 Introduction . Dubbing is a post-production process of re-recording actors ’ dialogues in a controlled environment ( i.e. , a sound studio ) , which is extensively used in filmmaking and video production . There are two common application scenarios for dubbing . The first one is replacing previous dialogues because poor sound quality is very common for speech recorded on noise location or the scene itself is too challenging to record high-quality audio . The second one is replacing the actor ’ voices in foreignlanguage films with those of other performers speaking the audience ’ s language . For example , an English video needs to be dubbed into Chinese if it is shown in China . In this paper , we mainly focus on the first application scenario , also known as “ automated dialogue replacement ( ADR ) ” , in which the professional voice actor watches original performance in the pre-recorded video , and re-records each line to match the lip movement with proper prosody such as stress , intonation and rhythm , which allows their speech to be synchronized with the pre-recorded video . In this scenario , the lip motion ( viseme ) in the video is consistent with the given scripts ( phoneme ) , and the pre-recorded high-definition video can not modified during the ADR process . While dubbing is an impressive ability of professional voice actors , we aim to achieve this ability computationally . We name this novel task automatic video dubbing ( AVD ) : synthesizing human speech that is temporally synchronized with the given video according to the corresponding text . The main challenges of the task are two-fold : ( 1 ) temporal synchronization between synthesized speech ∗Corresponding to hangzhao @ mail.tsinghua.edu.cn 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and video , i.e. , the synthesized speech should be synchronized with the lip movement of the speaker in the given video ; ( 2 ) the content of the speech should be consistent with the input text . Text to speech ( TTS ) is a task closely related to dubbing , which aims at converting given texts into natural and intelligible speech . However , several limitations prevent TTS from being applied in the dubbing problem : 1 ) TTS is a one-to-many mapping problem ( i.e. , multiple speech variations can be spoken from the same text ) [ 38 ] , so it is hard to control the variations ( e.g. , prosody , pitch and duration ) in synthesized speech during generation ; 2 ) with only text as input , TTS can not utilize the visual information from the video to control speech synthesis , which greatly limits its applications in dubbing scenarios where the synthesized speech are required to be synchronized with the video . We introduce Neural Dubber , the first model to solve the AVD task . Neural Dubber is a multi-modal speech synthesis model , which generates high-quality and lip-synced speech from the given text and video . In order to control the duration and prosody of synthesized speech , Neural Dubber works in a non-autoregressive way following [ 38 ] . The problem of length mismatch between phoneme sequence and mel-spectrogram sequence in non-autoregressive TTS is usually solved by up-sampling the phoneme sequence according to the predicted phoneme duration . Meanwhile , a phoneme duration predictor is needed , where the ground truth is usually obtained from another model [ 39 , 38 ] or itself during training [ 24 ] . However , due to the natural correspondence between lip movement and text [ 10 ] , we do not need to get phoneme duration target in advance like previous methods [ 39 , 38 , 24 ] . Instead , we use the text-video aligner which adopts an attention module between the video frames and phonemes , and then upsample the text-video context sequence according to the length ratio of mel-spectrogram sequence and video frame sequence . The text-video aligner not only solves the length mismatch problem , but also allows the lip movement in the video to control the prosody of the generated speech explicitly by the attention between video frames and phonemes . In the real dubbing scenario , voice actors need to alter the timbre and tone according to different performers in the video . In order to better simulate the real case in the AVD task , we propose the image-based speaker embedding ( ISE ) module , which aims to synthesize speech with different timbres conditioning on the speakers ’ face in the multi-speaker setting . To the best of our knowledge , this is the first attempt to predict a speaker embedding from a face image with the goal of generating speech with a reasonable timbre that is consistent with the speaker ’ s facial features ( e.g. , gender and age ) . This is achieved by taking advantage of the natural co-occurrence of faces and speech in videos without the supervision of speaker identity . With ISE , Neural Dubber can synthesize speech with a reasonable timbre according to the speaker ’ s face . In other words , Neural Dubber can use different face images to control the timbre of the synthesized speech . We conduct experiments on the chemistry lecture dataset from Lip2Wav [ 35 ] for the single-speaker AVD , and the LRS2 [ 1 ] dataset for the multi-speaker AVD . The results of extensive quantitative and qualitative evaluations show that in terms of speech quality , Neural Dubber is on par with state-ofthe-art TTS models [ 51 , 41 , 38 ] . Furthermore , Neural Dubber can synthesize speech temporally synchronized with the lip movement in video . In the multi-speaker setting , we demonstrate that the ISE enables Neural Dubber to generate speech with reasonable timbre based on the speaker ’ s face , resulting in Neural Dubber outperforming FastSpeech 2 by a big margin in term of audio quality . We attach some audio files and video clips generated by our model in the supplementary materials . 2 Related Work . Text to Speech . Text to Speech ( TTS ) [ 3 , 41 , 51 , 39 ] , which aims to synthesize intelligible , natural and high-quality speech from the input text , has seen tremendous progress in recent years . Specifically , the prevalent methods have shifted from concatenative synthesis [ 19 ] , parametric synthesis [ 53 ] to end-to-end neural network-based synthesis [ 33 , 41 , 51 ] , where the quality of the synthesized speech is improved by a large margin and is close to that of the human counterpart . The general paradigm of end-to-end neural network-based methods usually first generate the acoustic feature ( e.g. , melspectrogram ) from text using encoder-decoder architecture autoregressively [ 51 , 41 , 34 , 28 ] or nonautoregressively [ 39 , 38 , 24 ] , then reconstruct the waveform signal using vocoder [ 16 , 30 , 37 , 26 , 54 ] . When it comes to the recent multi-speaker TTS system [ 20 ] , the speaker embedding is often extracted using speaker verification system , and is fed to the decoder of the TTS system in order to encourage the model to obtain a timbre inclination for the speaker of interest . Different from TTS , Neural Dubber is conditioned not only on texts but also on videos , intending to synthesize natural speech given both of them . Talking Face Generation . Talking face generation has a long history in computer vision , ranging from viseme-based models [ 14 , 58 ] to neural synthesis of 2D [ 43 , 46 , 52 ] or 3D [ 21 , 40 , 45 ] face . Recently neural synthesis approaches have been proposed to generate realistic 2D video of talking heads . Concretely speaking , Chung et al . [ 9 ] first generates lower face animation using cropped frontal images . After then Zhou et al . [ 57 ] further disentangles identity from speech using generative adversarial networks ( GANs ) . Wav2Lip [ 36 ] tries to explore the problem of visual dubbing , i.e. , lip-syncing a talking head video of an arbitrary person to match a target speech segment . From our perspective , however , such methods can not generate high-fidelity face and lip given speech , spawning the results are of low resolution and look uncanny sometimes . Besides , audios in most talking face pipelines need to be prepared in advance , thus , strictly speaking , this does not belong to dubbing ( re-recording ) 2 , but to the face synchronization while given audio . In contrast to the aforementioned works , Neural Dubber is not required to prepare audio beforehand and modify the lip motion , but generates speech audio synchronized with the video from scripts . Lip to Speech Synthesis . Given a video , the lip to speech task aims at synthesizing the corresponding speech audio by directly judging from the lip motion . While the conventional method [ 22 ] exploits the visual features extracted from active appearance models , recent end-to-end methods have also shed some light on it . In particular , Vid2Speech [ 15 ] and Lipper [ 27 ] generate low-dimensional linear predictive coding features to synthesize speech in the constrained scene . Vougioukas et al . [ 49 ] using the GANs-based method to exert for quality gains . Lip2Wav [ 35 ] has achieved promising results in real-life speaker-dependent scenarios , but it is still somewhat incongruous and prone to collapse in the multi-speaker setting . This is possibly because the word error rate in lip reading task [ 2 , 4 , 10 , 12 ] is still high , let alone the lip to speech synthesis . In Neural Dubber , the textual information is provided , allowing us to concentrate more on the alignment between the phoneme and lip motion in video , instead of decoding speech from lip motion directly . 3 Method . In this section , we first introduce the novel automatic video dubbing ( AVD ) task ; we then describe the overall architecture of our proposed Neural Dubber ; finally we detail the main components in Neural Dubber . 2https : //en.wikipedia.org/wiki/Dubbing_ ( filmmaking ) 3.1 Automatic Video Dubbing . Given a sentence T and a corresponding video clip ( without audio ) V , the goal of automatic video dubbing ( AVD ) is to synthesize natural and intelligible speech S whose content is consistent with T , and whose prosody is synchronized with the lip movement of the active speaker in the video V . Compared to the traditional speech synthesis task which only generates natural and intelligible speech S given the sentence T , AVD task is more difficult due to the synchronization requirement . 3.2 Neural Dubber . 3.2.1 Design Overview . Our Neural Dubber aims to solve the AVD task . Concretely , we formulate the problem as follows : given a phoneme sequence Sp = { P1 , P2 , . . . , PTp } and a video frame sequence Sv = { I1 , I2 , . . . , ITv } , we need to predict a target mel-spectrogram sequence Sm = { Y1 , Y2 , . . . , YTm } . The overall model architecture of Neural Dubber is shown in Figure 2 . First , we apply a phoneme encoder fp and a video encoder fv to process the phonemes and images respectively . Note that the images we feed to the video encoder only contain mouth region of the speaker following [ 10 , 32 , 42 ] . We use Smv to represent these images . After the encoding , raw phonemes turn into a sequence of hidden representationsHpho = fp ( Sp ) ∈ RTp×d while images of mouth region turn into a sequence of hidden representationsHvid = fv ( Smv ) ∈ RTv×d . Then we feedHpho andHvid into the text-video aligner ( which will be described in Section 3.2.3 ) and get the expanded sequence Hmel ∈ RTm×d with the same length as the target mel-spectrogram sequence Sm . Meanwhile , a face image randomly selected from the video frames is input into image-based speaker embedding ( ISE ) module ( which will be described in Section 3.2.4 ) to generate a image-based speaker embedding . We addHmel and ISE together and feed them into the variance adaptor to add some variance information ( e.g. , pitch and energy ) . Finally , we use the mel-spectrogram decoder to convert the adapted hidden sequence into mel-spectrogram sequence following [ 39 ] . Different from FastSpeech 2 [ 38 ] , our variance adaptor consists of pitch and energy predictors without duration predictor , because we solve the problem of length mismatch between the phoneme and mel-spectrogram sequence in the text-video aligner and the input of variance adaptor is as long as the mel-spectrogram sequence .

Summary: The authors describe the problem of automatic dubbing of videos, ie generating speech from a script that matches the active speaking face in a video. To the best of this reviewer's knowledge, this is the first paper addressing this particular problem. The two main contributions to solve this problem is 1. a method to synchronize the generated TTS with the speaking face 2. sampling the video for a face image to infer a timbre for the speech to better match the face.