Neural Dubber: Dubbing for Videos According to Scripts

1 Introduction . Dubbing is a post-production process of re-recording actors ’ dialogues in a controlled environment ( i.e. , a sound studio ) , which is extensively used in filmmaking and video production . There are two common application scenarios for dubbing . The first one is replacing previous dialogues because poor sound quality is very common for speech recorded on noise location or the scene itself is too challenging to record high-quality audio . The second one is replacing the actor ’ voices in foreignlanguage films with those of other performers speaking the audience ’ s language . For example , an English video needs to be dubbed into Chinese if it is shown in China . In this paper , we mainly focus on the first application scenario , also known as “ automated dialogue replacement ( ADR ) ” , in which the professional voice actor watches original performance in the pre-recorded video , and re-records each line to match the lip movement with proper prosody such as stress , intonation and rhythm , which allows their speech to be synchronized with the pre-recorded video . In this scenario , the lip motion ( viseme ) in the video is consistent with the given scripts ( phoneme ) , and the pre-recorded high-definition video can not modified during the ADR process . While dubbing is an impressive ability of professional voice actors , we aim to achieve this ability computationally . We name this novel task automatic video dubbing ( AVD ) : synthesizing human speech that is temporally synchronized with the given video according to the corresponding text . The main challenges of the task are two-fold : ( 1 ) temporal synchronization between synthesized speech ∗Corresponding to hangzhao @ mail.tsinghua.edu.cn 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and video , i.e. , the synthesized speech should be synchronized with the lip movement of the speaker in the given video ; ( 2 ) the content of the speech should be consistent with the input text . Text to speech ( TTS ) is a task closely related to dubbing , which aims at converting given texts into natural and intelligible speech . However , several limitations prevent TTS from being applied in the dubbing problem : 1 ) TTS is a one-to-many mapping problem ( i.e. , multiple speech variations can be spoken from the same text ) [ 38 ] , so it is hard to control the variations ( e.g. , prosody , pitch and duration ) in synthesized speech during generation ; 2 ) with only text as input , TTS can not utilize the visual information from the video to control speech synthesis , which greatly limits its applications in dubbing scenarios where the synthesized speech are required to be synchronized with the video . We introduce Neural Dubber , the first model to solve the AVD task . Neural Dubber is a multi-modal speech synthesis model , which generates high-quality and lip-synced speech from the given text and video . In order to control the duration and prosody of synthesized speech , Neural Dubber works in a non-autoregressive way following [ 38 ] . The problem of length mismatch between phoneme sequence and mel-spectrogram sequence in non-autoregressive TTS is usually solved by up-sampling the phoneme sequence according to the predicted phoneme duration . Meanwhile , a phoneme duration predictor is needed , where the ground truth is usually obtained from another model [ 39 , 38 ] or itself during training [ 24 ] . However , due to the natural correspondence between lip movement and text [ 10 ] , we do not need to get phoneme duration target in advance like previous methods [ 39 , 38 , 24 ] . Instead , we use the text-video aligner which adopts an attention module between the video frames and phonemes , and then upsample the text-video context sequence according to the length ratio of mel-spectrogram sequence and video frame sequence . The text-video aligner not only solves the length mismatch problem , but also allows the lip movement in the video to control the prosody of the generated speech explicitly by the attention between video frames and phonemes . In the real dubbing scenario , voice actors need to alter the timbre and tone according to different performers in the video . In order to better simulate the real case in the AVD task , we propose the image-based speaker embedding ( ISE ) module , which aims to synthesize speech with different timbres conditioning on the speakers ’ face in the multi-speaker setting . To the best of our knowledge , this is the first attempt to predict a speaker embedding from a face image with the goal of generating speech with a reasonable timbre that is consistent with the speaker ’ s facial features ( e.g. , gender and age ) . This is achieved by taking advantage of the natural co-occurrence of faces and speech in videos without the supervision of speaker identity . With ISE , Neural Dubber can synthesize speech with a reasonable timbre according to the speaker ’ s face . In other words , Neural Dubber can use different face images to control the timbre of the synthesized speech . We conduct experiments on the chemistry lecture dataset from Lip2Wav [ 35 ] for the single-speaker AVD , and the LRS2 [ 1 ] dataset for the multi-speaker AVD . The results of extensive quantitative and qualitative evaluations show that in terms of speech quality , Neural Dubber is on par with state-ofthe-art TTS models [ 51 , 41 , 38 ] . Furthermore , Neural Dubber can synthesize speech temporally synchronized with the lip movement in video . In the multi-speaker setting , we demonstrate that the ISE enables Neural Dubber to generate speech with reasonable timbre based on the speaker ’ s face , resulting in Neural Dubber outperforming FastSpeech 2 by a big margin in term of audio quality . We attach some audio files and video clips generated by our model in the supplementary materials . 2 Related Work . Text to Speech . Text to Speech ( TTS ) [ 3 , 41 , 51 , 39 ] , which aims to synthesize intelligible , natural and high-quality speech from the input text , has seen tremendous progress in recent years . Specifically , the prevalent methods have shifted from concatenative synthesis [ 19 ] , parametric synthesis [ 53 ] to end-to-end neural network-based synthesis [ 33 , 41 , 51 ] , where the quality of the synthesized speech is improved by a large margin and is close to that of the human counterpart . The general paradigm of end-to-end neural network-based methods usually first generate the acoustic feature ( e.g. , melspectrogram ) from text using encoder-decoder architecture autoregressively [ 51 , 41 , 34 , 28 ] or nonautoregressively [ 39 , 38 , 24 ] , then reconstruct the waveform signal using vocoder [ 16 , 30 , 37 , 26 , 54 ] . When it comes to the recent multi-speaker TTS system [ 20 ] , the speaker embedding is often extracted using speaker verification system , and is fed to the decoder of the TTS system in order to encourage the model to obtain a timbre inclination for the speaker of interest . Different from TTS , Neural Dubber is conditioned not only on texts but also on videos , intending to synthesize natural speech given both of them . Talking Face Generation . Talking face generation has a long history in computer vision , ranging from viseme-based models [ 14 , 58 ] to neural synthesis of 2D [ 43 , 46 , 52 ] or 3D [ 21 , 40 , 45 ] face . Recently neural synthesis approaches have been proposed to generate realistic 2D video of talking heads . Concretely speaking , Chung et al . [ 9 ] first generates lower face animation using cropped frontal images . After then Zhou et al . [ 57 ] further disentangles identity from speech using generative adversarial networks ( GANs ) . Wav2Lip [ 36 ] tries to explore the problem of visual dubbing , i.e. , lip-syncing a talking head video of an arbitrary person to match a target speech segment . From our perspective , however , such methods can not generate high-fidelity face and lip given speech , spawning the results are of low resolution and look uncanny sometimes . Besides , audios in most talking face pipelines need to be prepared in advance , thus , strictly speaking , this does not belong to dubbing ( re-recording ) 2 , but to the face synchronization while given audio . In contrast to the aforementioned works , Neural Dubber is not required to prepare audio beforehand and modify the lip motion , but generates speech audio synchronized with the video from scripts . Lip to Speech Synthesis . Given a video , the lip to speech task aims at synthesizing the corresponding speech audio by directly judging from the lip motion . While the conventional method [ 22 ] exploits the visual features extracted from active appearance models , recent end-to-end methods have also shed some light on it . In particular , Vid2Speech [ 15 ] and Lipper [ 27 ] generate low-dimensional linear predictive coding features to synthesize speech in the constrained scene . Vougioukas et al . [ 49 ] using the GANs-based method to exert for quality gains . Lip2Wav [ 35 ] has achieved promising results in real-life speaker-dependent scenarios , but it is still somewhat incongruous and prone to collapse in the multi-speaker setting . This is possibly because the word error rate in lip reading task [ 2 , 4 , 10 , 12 ] is still high , let alone the lip to speech synthesis . In Neural Dubber , the textual information is provided , allowing us to concentrate more on the alignment between the phoneme and lip motion in video , instead of decoding speech from lip motion directly . 3 Method . In this section , we first introduce the novel automatic video dubbing ( AVD ) task ; we then describe the overall architecture of our proposed Neural Dubber ; finally we detail the main components in Neural Dubber . 2https : //en.wikipedia.org/wiki/Dubbing_ ( filmmaking ) 3.1 Automatic Video Dubbing . Given a sentence T and a corresponding video clip ( without audio ) V , the goal of automatic video dubbing ( AVD ) is to synthesize natural and intelligible speech S whose content is consistent with T , and whose prosody is synchronized with the lip movement of the active speaker in the video V . Compared to the traditional speech synthesis task which only generates natural and intelligible speech S given the sentence T , AVD task is more difficult due to the synchronization requirement . 3.2 Neural Dubber . 3.2.1 Design Overview . Our Neural Dubber aims to solve the AVD task . Concretely , we formulate the problem as follows : given a phoneme sequence Sp = { P1 , P2 , . . . , PTp } and a video frame sequence Sv = { I1 , I2 , . . . , ITv } , we need to predict a target mel-spectrogram sequence Sm = { Y1 , Y2 , . . . , YTm } . The overall model architecture of Neural Dubber is shown in Figure 2 . First , we apply a phoneme encoder fp and a video encoder fv to process the phonemes and images respectively . Note that the images we feed to the video encoder only contain mouth region of the speaker following [ 10 , 32 , 42 ] . We use Smv to represent these images . After the encoding , raw phonemes turn into a sequence of hidden representationsHpho = fp ( Sp ) ∈ RTp×d while images of mouth region turn into a sequence of hidden representationsHvid = fv ( Smv ) ∈ RTv×d . Then we feedHpho andHvid into the text-video aligner ( which will be described in Section 3.2.3 ) and get the expanded sequence Hmel ∈ RTm×d with the same length as the target mel-spectrogram sequence Sm . Meanwhile , a face image randomly selected from the video frames is input into image-based speaker embedding ( ISE ) module ( which will be described in Section 3.2.4 ) to generate a image-based speaker embedding . We addHmel and ISE together and feed them into the variance adaptor to add some variance information ( e.g. , pitch and energy ) . Finally , we use the mel-spectrogram decoder to convert the adapted hidden sequence into mel-spectrogram sequence following [ 39 ] . Different from FastSpeech 2 [ 38 ] , our variance adaptor consists of pitch and energy predictors without duration predictor , because we solve the problem of length mismatch between the phoneme and mel-spectrogram sequence in the text-video aligner and the input of variance adaptor is as long as the mel-spectrogram sequence .

Summary: This paper presents "Neural Dubber", a method for dubbing silent videos based on given scripts. It essentially performs lip-synced TTS, with an added component (ISE) that ensures that the generated speech characteristics match the given face image.   Experiments verify that the proposed TTS method does not hurt SOTA TTS performance, while controlling the prosody such that the aligned speech is much more synchronized than other methods compared to. The supplementary video qualitatively demonstrates some compelling results of the proposed method.