Lip to Speech Synthesis with Visual Context Attentional GAN

1 Introduction . Lip to speech synthesis ( Lip2Speech ) is to predict an audio speech by watching a silent talking face video . While conventional visual speech recognition tasks require human annotations ( i.e. , text ) , Lip2Speech does not require additional annotations . Thus , it has drawn big attention as another form of lip reading . However , due to the ambiguity of homophenes that have similar lip movements and the voice characteristics varying from different identities , it is still considered as a challenging problem . Basically , synthesizing a speech from a silent lip movement video can be viewed as finding a mapping function of visemes into corresponding phonemes . However , only watching short clip-level ( i.e. , local ) lip movements could be challenging to distinguish the homophenes . Thus , global-level lip movements containing the visual context , hints for ambiguity of viseme-to-phoneme mapping , should also be considered along with local-level lip movements . Early deep learning-based works [ 1 , 2 , 3 ] predict each auditory feature ( e.g. , LPC , mel-spectrogram , spectrogram ) within a short video clip and extend the prediction to the entire speech by sliding a window over the whole video sequences . As they operate with clip-level videos of fixed length , they could fail on capturing the global context of the spoken speech . A recent work [ 4 ] brings Sequence-to-Sequence ( Seq2Seq ) architecture [ 5 , 6 ] that predicts the auditory feature conditioned on both the encoded visual context and the previous prediction and shows a promising performance . However , since they do not explicitly consider local visual features , they may produce out-of-sync speech to the input video . Moreover , due to the sequential nature of the Seq2Seq architecture , the method demands heavy inference time . Since the ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . output audio sequence length is determined when the input video is given for the Lip2Speech task , the time costs can be reduced by adopting different architectures that could predict the speech with one forward step , instead of using a Seq2Seq model which is originally designed to handle input and output with different varying sequence lengths . Lastly , all the above methods focus on handling speech synthesis of constrained speakers ( i.e. , 1 to 4 speakers ) , so they could fail to properly handle diverse speakers with one trained model . In this paper , we design a novel deep architecture , namely Visual Context Attentional GAN ( VCAGAN ) , that jointly models the local and global visual representations to synthesize accurate speech from silent talking face video . Concretely , the proposed VCA-GAN synthesizes the speech ( i.e. , melspectrogram ) based on the local visual features by finding a mapping function of viseme-to-phoneme , while the global visual context assists the generator for clarifying the ambiguity of the mapping . To this end , a visual context attention module is proposed where it extracts the global visual features from the local visual features and provides the global visual context to the generator . It is applied to the generator in multi-scale scheme so that the generator can refine the speech representation from coarse- to fine-level by jointly modelling both the local and the global visual context . Moreover , to guarantee the generated speech to be synced with the input lip movements , synchronization learning is performed that gives feedback to the generator whether the synthesized speech is synchronized or not with the input lip movement . The effectiveness of the proposed framework is evaluated on three public benchmark databases , GRID [ 7 ] , TCD-TIMIT [ 8 ] , and LRW [ 9 ] in both constrained-speaker setting and multi-speaker setting . The major contributions of this paper are as follows , 1 ) To the best of our knowledge , this is the first work to explicitly model the local and global lip movements for synthesizing detailed and accurate speech from silent talking face video . 2 ) We consider a mel-spectrogram as an image and solve the Lip2Speech problem efficiently using video-to-image translation . 3 ) This paper introduces synchronization learning which guides the generated mel-spectrogram to be in sync with the input lip video . 4 ) We show the proposed VCA-GAN can synthesize speech from diverse speakers without the prior knowledge of speaker information such as speaker embeddings . 2 Related Work . Lip to Speech Synthesis . There have been a number of researches and interests in visual-to-speech generation . Ephrat et al . [ 1 ] utilized CNN to predict acoustic features from silent talking videos . Then , they augmented the model to two-tower CNN-based encoder-decoder [ 2 ] where each tower encodes raw frames and optical flows , respectively . Akbari et al . [ 3 ] employed a deep autoencoder for reconstructing the speech features from the visual features encoded by a lip-reading network . Vougioukas et al . [ 10 ] directly synthesized the raw waveform from the video by using 1D GAN . Prajwal et al . [ 4 ] focused on learning lip sequence to speech mapping for a single speaker in an unconstrained , large vocabulary setting using a stack of 3D convolution and Seq2Seq architecture . Yadav et al . [ 11 ] used stochastic modelling approach with variational autoencoder . Michelsanti et al . [ 12 ] predicted vocoder features of [ 13 ] and synthesized speech using the vocoder . Different from the previous works , our approach explicitly models the local visual feature and global visual context to synthesize accurate speech . Moreover , we try to synthesize the speech from multi-speaker which has rarely been handled in the past . Visual Speech Recognition ( VSR ) . Parallel to the development of Lip2Speech , Visual Speech Recognition ( VSR ) have achieved a great advancement [ 14 , 15 , 16 , 17 , 18 , 19 ] . Slightly different from the Lip2Speech , VSR identifies spoken speech into text by watching a silent talking face video . Several works have recently showed state-of-the-art performances in word- and sentence-level classifications . Chung et al . [ 9 ] proposed a large-scale audio-visual dataset and set a baseline model for word-level VSR . Stafylakis et al . [ 20 ] proposed an architecture that is combined of residual network and LSTM , which became a popular architecture for word-level lip reading . Martinez et al . [ 21 ] replaced the RNN-based backend with Temporal Convolutional Network ( TCN ) . Kim et al . [ 19 , 22 ] proposed to utilize audio modal knowledge through memory network without audio inputs during inference for lip reading . Assael et al . [ 23 ] achieved end-to-end sentence-level lip reading network by adopting the CTC loss [ 24 ] . Different from the VSR methods , the Lip2Speech task does not require human annotations , thus is drawing big attention with its practical aspects . Attention Mechanism . The attention mechanism has affected many research fields , such as image captioning [ 25 , 26 , 27 ] , machine translation [ 28 , 29 ] , and speech recognition [ 30 , 31 , 32 ] . It can effectively focus on relative information and reduce the interference from less significant one . There have been several works that incorporate attention mechanism in GAN . Xu et al . [ 25 ] proposed a cross modal attention model to guide the generator to focus on different words when generating different image sub-regions . Qiao et al . [ 27 ] further developed it by proposing a global-local collaborative attentive module to leverage both local word attention and global sentence attention and to enhance the diversity and semantic consistency of the generated images . Li et al . [ 26 ] introduced channelwise attention-driven generator that can disentangle different visual attributes , considering the most relevant channels in the visual features to be fully exploited . In this paper , we design a cross-modal attention module working with video and audio modalities for context modelling during speech synthesis . By bringing the global visual context through the proposed visual context attention module , speech of high intelligibility can be synthesized . 3 Proposed Method . Since an audio speech and lip movements in a single video are supposed to be aligned in time , the speech can be synthesized to have the same duration as the input silent video . Let x ∈ RT×H×W×C be a lip video with T frames , height of H , width of W , and channel size of C. Then , our objective is to find a generative model that synthesizes a speech y ∈ RF×4T , where y is a target mel-spectrogram with F mel-spectral dimension and frame length of 4T . The frame length of mel-spectrogram is designed to be 4 times longer than that of video by adjusting the hop length during Short-Time Fourier Transform ( STFT ) . To generate elaborate speech representations , the proposed VCA-GAN ( Fig.1 ) refines the viseme-to-phoneme mapping with the global visual context obtained from a visual context attention module , and learns to produce a synchronized speech with given lip movements . Please note that we treat the mel-spectrogram as an image and train the model with 2D GAN [ 33 , 34 ] . 3.1 Visual context attentional GAN . Considering the entire context from the input lip movements , namely global visual context , can provide additional information that alleviates the ambiguity of homophenes besides the accurate temporal alignment of local visual representations . To achieve this , the generator synthesizes the speech from the local visual features while the global visual context is jointly considered at the intermediate layers of generator through the visual context attention module . Firstly , a local visual encoder φv encodes the video x into local visual features Fv = { f1v , f2v , · · · , fTv } ∈ RT×D , where D is the dimension of embedding . The local visual encoder φv is composed of combination of 3D and 2D convolutions . Due to the locality of the convolution operator , each local visual feature f tv contains short-clip level ( i.e. , local ) lip movements embedded through the 3D convolution . From the local visual features Fv , n generators ( ψ1 , ψ2 , . . . , ψn ) gradually generate and refine the speech representation from low to high resolution . The first generator ψ1 synthesizes coarse speech representation F 1a with the following equation , F 1a = ψ1 ( [ R ( Fv ) ; z ] ) , ( 1 ) where z is a noise drawn from a standard normal distribution , R : RT×D → RF4 ×T×D is a repeat operator that forms a 3D tensor of height F4 , width T , and channel D by repeating the input feature F 4 times , and [ ; ] represents concatenation . The objective of the subsequent generators is to refine the coarse speech representation by jointly modelling the local visual features and global visual context . To this end , a visual context attention module , composed of a global visual encoder and audio-visual attentions , is proposed . The global visual encoder φc derives the global visual features Cv ∈ RT×D by considering the relationships of the entire local visual features Fv through bi-RNN . Then , the audio-visual attention finds the complementary cues ( i.e. , global visual context ) for the speech synthesis , considering the importance of global visual features Cv accordingly with the speech representation F ia ∈ RFi×Ti×Di of i-th resolution . The audio-visual attention can be denoted as following equations , Qi = F ( F ia ) W iq , Ki = CvW ik , Vi = CvW iv , F ic = S ( softmax ( QiK > i√ d ) Vi ) , ( 2 ) where F ic represents the global visual context obtained , W i q ∈ RFiDi×d , W ik ∈ RD×d , and W iv ∈ RD× FiDi α represent query , key , and value embedding weights , respectively , F : RFi×Ti×Di → RTi×FiDi is a flatten operator that merges the spectral dimension and channel dimension of speech representation , S : RTi× FiDi α → RFi×Ti× Di α is a split operator that maps the 2D tensor into 3D tensor , and α is dimensionality reduction ratio . Note that the audio-visual attention is based on the scaled dot product attention [ 29 ] with multi-modal inputs . The visual context attention module and the audio-visual attention are illustrated in Fig.2 . The obtained global visual context F ic is concatenated with the coarse speech representation F i a , and the subsequent generators refine the speech representation iteratively with the following equation , F i+1a = ψi ( [ F i a ; F i c ] ) , for i = 1 , 2 , . . . , n− 1 . ( 3 ) Therefore , the following generators can get hints for synthesizing an accurate speech from the global visual context during finding the viseme-to-phoneme mapping using local visual features . In addition , to generate an image ( i.e. , mel-spectrogram ) with fine details , multi-discriminators are utilized following [ 25 ] . The multi-scale mel-spectrograms ( ŷ1 , ŷ2 , . . . , ŷn ) are synthesized from each generated speech representation ( F 1a , F 2 a , . . . , F n a ) through 1× 1 convolutions and passed into the multi-discriminators , as illustrated in Fig.1 .

Summary: This paper describes a synthesis approach that uses visual lip movement as input and generates speech as output.  They use a GAN style synthesizer. The most novel contribution is the synchronization technique to align visual and speech signals.