Lip to Speech Synthesis with Visual Context Attentional GAN

1 Introduction . Lip to speech synthesis ( Lip2Speech ) is to predict an audio speech by watching a silent talking face video . While conventional visual speech recognition tasks require human annotations ( i.e. , text ) , Lip2Speech does not require additional annotations . Thus , it has drawn big attention as another form of lip reading . However , due to the ambiguity of homophenes that have similar lip movements and the voice characteristics varying from different identities , it is still considered as a challenging problem . Basically , synthesizing a speech from a silent lip movement video can be viewed as finding a mapping function of visemes into corresponding phonemes . However , only watching short clip-level ( i.e. , local ) lip movements could be challenging to distinguish the homophenes . Thus , global-level lip movements containing the visual context , hints for ambiguity of viseme-to-phoneme mapping , should also be considered along with local-level lip movements . Early deep learning-based works [ 1 , 2 , 3 ] predict each auditory feature ( e.g. , LPC , mel-spectrogram , spectrogram ) within a short video clip and extend the prediction to the entire speech by sliding a window over the whole video sequences . As they operate with clip-level videos of fixed length , they could fail on capturing the global context of the spoken speech . A recent work [ 4 ] brings Sequence-to-Sequence ( Seq2Seq ) architecture [ 5 , 6 ] that predicts the auditory feature conditioned on both the encoded visual context and the previous prediction and shows a promising performance . However , since they do not explicitly consider local visual features , they may produce out-of-sync speech to the input video . Moreover , due to the sequential nature of the Seq2Seq architecture , the method demands heavy inference time . Since the ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . output audio sequence length is determined when the input video is given for the Lip2Speech task , the time costs can be reduced by adopting different architectures that could predict the speech with one forward step , instead of using a Seq2Seq model which is originally designed to handle input and output with different varying sequence lengths . Lastly , all the above methods focus on handling speech synthesis of constrained speakers ( i.e. , 1 to 4 speakers ) , so they could fail to properly handle diverse speakers with one trained model . In this paper , we design a novel deep architecture , namely Visual Context Attentional GAN ( VCAGAN ) , that jointly models the local and global visual representations to synthesize accurate speech from silent talking face video . Concretely , the proposed VCA-GAN synthesizes the speech ( i.e. , melspectrogram ) based on the local visual features by finding a mapping function of viseme-to-phoneme , while the global visual context assists the generator for clarifying the ambiguity of the mapping . To this end , a visual context attention module is proposed where it extracts the global visual features from the local visual features and provides the global visual context to the generator . It is applied to the generator in multi-scale scheme so that the generator can refine the speech representation from coarse- to fine-level by jointly modelling both the local and the global visual context . Moreover , to guarantee the generated speech to be synced with the input lip movements , synchronization learning is performed that gives feedback to the generator whether the synthesized speech is synchronized or not with the input lip movement . The effectiveness of the proposed framework is evaluated on three public benchmark databases , GRID [ 7 ] , TCD-TIMIT [ 8 ] , and LRW [ 9 ] in both constrained-speaker setting and multi-speaker setting . The major contributions of this paper are as follows , 1 ) To the best of our knowledge , this is the first work to explicitly model the local and global lip movements for synthesizing detailed and accurate speech from silent talking face video . 2 ) We consider a mel-spectrogram as an image and solve the Lip2Speech problem efficiently using video-to-image translation . 3 ) This paper introduces synchronization learning which guides the generated mel-spectrogram to be in sync with the input lip video . 4 ) We show the proposed VCA-GAN can synthesize speech from diverse speakers without the prior knowledge of speaker information such as speaker embeddings . 2 Related Work . Lip to Speech Synthesis . There have been a number of researches and interests in visual-to-speech generation . Ephrat et al . [ 1 ] utilized CNN to predict acoustic features from silent talking videos . Then , they augmented the model to two-tower CNN-based encoder-decoder [ 2 ] where each tower encodes raw frames and optical flows , respectively . Akbari et al . [ 3 ] employed a deep autoencoder for reconstructing the speech features from the visual features encoded by a lip-reading network . Vougioukas et al . [ 10 ] directly synthesized the raw waveform from the video by using 1D GAN . Prajwal et al . [ 4 ] focused on learning lip sequence to speech mapping for a single speaker in an unconstrained , large vocabulary setting using a stack of 3D convolution and Seq2Seq architecture . Yadav et al . [ 11 ] used stochastic modelling approach with variational autoencoder . Michelsanti et al . [ 12 ] predicted vocoder features of [ 13 ] and synthesized speech using the vocoder . Different from the previous works , our approach explicitly models the local visual feature and global visual context to synthesize accurate speech . Moreover , we try to synthesize the speech from multi-speaker which has rarely been handled in the past . Visual Speech Recognition ( VSR ) . Parallel to the development of Lip2Speech , Visual Speech Recognition ( VSR ) have achieved a great advancement [ 14 , 15 , 16 , 17 , 18 , 19 ] . Slightly different from the Lip2Speech , VSR identifies spoken speech into text by watching a silent talking face video . Several works have recently showed state-of-the-art performances in word- and sentence-level classifications . Chung et al . [ 9 ] proposed a large-scale audio-visual dataset and set a baseline model for word-level VSR . Stafylakis et al . [ 20 ] proposed an architecture that is combined of residual network and LSTM , which became a popular architecture for word-level lip reading . Martinez et al . [ 21 ] replaced the RNN-based backend with Temporal Convolutional Network ( TCN ) . Kim et al . [ 19 , 22 ] proposed to utilize audio modal knowledge through memory network without audio inputs during inference for lip reading . Assael et al . [ 23 ] achieved end-to-end sentence-level lip reading network by adopting the CTC loss [ 24 ] . Different from the VSR methods , the Lip2Speech task does not require human annotations , thus is drawing big attention with its practical aspects . Attention Mechanism . The attention mechanism has affected many research fields , such as image captioning [ 25 , 26 , 27 ] , machine translation [ 28 , 29 ] , and speech recognition [ 30 , 31 , 32 ] . It can effectively focus on relative information and reduce the interference from less significant one . There have been several works that incorporate attention mechanism in GAN . Xu et al . [ 25 ] proposed a cross modal attention model to guide the generator to focus on different words when generating different image sub-regions . Qiao et al . [ 27 ] further developed it by proposing a global-local collaborative attentive module to leverage both local word attention and global sentence attention and to enhance the diversity and semantic consistency of the generated images . Li et al . [ 26 ] introduced channelwise attention-driven generator that can disentangle different visual attributes , considering the most relevant channels in the visual features to be fully exploited . In this paper , we design a cross-modal attention module working with video and audio modalities for context modelling during speech synthesis . By bringing the global visual context through the proposed visual context attention module , speech of high intelligibility can be synthesized . 3 Proposed Method . Since an audio speech and lip movements in a single video are supposed to be aligned in time , the speech can be synthesized to have the same duration as the input silent video . Let x ∈ RT×H×W×C be a lip video with T frames , height of H , width of W , and channel size of C. Then , our objective is to find a generative model that synthesizes a speech y ∈ RF×4T , where y is a target mel-spectrogram with F mel-spectral dimension and frame length of 4T . The frame length of mel-spectrogram is designed to be 4 times longer than that of video by adjusting the hop length during Short-Time Fourier Transform ( STFT ) . To generate elaborate speech representations , the proposed VCA-GAN ( Fig.1 ) refines the viseme-to-phoneme mapping with the global visual context obtained from a visual context attention module , and learns to produce a synchronized speech with given lip movements . Please note that we treat the mel-spectrogram as an image and train the model with 2D GAN [ 33 , 34 ] . 3.1 Visual context attentional GAN . Considering the entire context from the input lip movements , namely global visual context , can provide additional information that alleviates the ambiguity of homophenes besides the accurate temporal alignment of local visual representations . To achieve this , the generator synthesizes the speech from the local visual features while the global visual context is jointly considered at the intermediate layers of generator through the visual context attention module . Firstly , a local visual encoder φv encodes the video x into local visual features Fv = { f1v , f2v , · · · , fTv } ∈ RT×D , where D is the dimension of embedding . The local visual encoder φv is composed of combination of 3D and 2D convolutions . Due to the locality of the convolution operator , each local visual feature f tv contains short-clip level ( i.e. , local ) lip movements embedded through the 3D convolution . From the local visual features Fv , n generators ( ψ1 , ψ2 , . . . , ψn ) gradually generate and refine the speech representation from low to high resolution . The first generator ψ1 synthesizes coarse speech representation F 1a with the following equation , F 1a = ψ1 ( [ R ( Fv ) ; z ] ) , ( 1 ) where z is a noise drawn from a standard normal distribution , R : RT×D → RF4 ×T×D is a repeat operator that forms a 3D tensor of height F4 , width T , and channel D by repeating the input feature F 4 times , and [ ; ] represents concatenation . The objective of the subsequent generators is to refine the coarse speech representation by jointly modelling the local visual features and global visual context . To this end , a visual context attention module , composed of a global visual encoder and audio-visual attentions , is proposed . The global visual encoder φc derives the global visual features Cv ∈ RT×D by considering the relationships of the entire local visual features Fv through bi-RNN . Then , the audio-visual attention finds the complementary cues ( i.e. , global visual context ) for the speech synthesis , considering the importance of global visual features Cv accordingly with the speech representation F ia ∈ RFi×Ti×Di of i-th resolution . The audio-visual attention can be denoted as following equations , Qi = F ( F ia ) W iq , Ki = CvW ik , Vi = CvW iv , F ic = S ( softmax ( QiK > i√ d ) Vi ) , ( 2 ) where F ic represents the global visual context obtained , W i q ∈ RFiDi×d , W ik ∈ RD×d , and W iv ∈ RD× FiDi α represent query , key , and value embedding weights , respectively , F : RFi×Ti×Di → RTi×FiDi is a flatten operator that merges the spectral dimension and channel dimension of speech representation , S : RTi× FiDi α → RFi×Ti× Di α is a split operator that maps the 2D tensor into 3D tensor , and α is dimensionality reduction ratio . Note that the audio-visual attention is based on the scaled dot product attention [ 29 ] with multi-modal inputs . The visual context attention module and the audio-visual attention are illustrated in Fig.2 . The obtained global visual context F ic is concatenated with the coarse speech representation F i a , and the subsequent generators refine the speech representation iteratively with the following equation , F i+1a = ψi ( [ F i a ; F i c ] ) , for i = 1 , 2 , . . . , n− 1 . ( 3 ) Therefore , the following generators can get hints for synthesizing an accurate speech from the global visual context during finding the viseme-to-phoneme mapping using local visual features . In addition , to generate an image ( i.e. , mel-spectrogram ) with fine details , multi-discriminators are utilized following [ 25 ] . The multi-scale mel-spectrograms ( ŷ1 , ŷ2 , . . . , ŷn ) are synthesized from each generated speech representation ( F 1a , F 2 a , . . . , F n a ) through 1× 1 convolutions and passed into the multi-discriminators , as illustrated in Fig.1 .

Summary: This paper proposes a lip2speech generative framework based on the proposed Visual Context Attention (VCA) module. The VCA module follows the idea of self-attention and fuses the audio-visual information together to generate finer representations. The proposed pipeline stacks several generators together whose predictions are gradually refined by the VCA module. The global and local clues of input videos are also taken into consideration in this framework. The experiments are conducted on three datasets, while the authors have evaluated their method under different scenarios like constrained-speaker, unseen speaker, etc. The final performance is superior to other baselines.