Lip to Speech Synthesis with Visual Context Attentional GAN
Year: 2022

Abstract:
In this paper, we propose a novel lip-to-speech generative adversarial network, Visual Context Attentional GAN (VCA-GAN), which can jointly model local and global lip movements during speech synthesis. Specifically, the proposed VCAGAN synthesizes the speech from local lip visual features by finding a mapping function of viseme-to-phoneme, while global visual context is embedded into the intermediate layers of the generator to clarify the ambiguity in the mapping induced by homophene. To achieve this, a visual context attention module is proposed where it encodes global representations from the local visual features, and provides the desired global visual context corresponding to the given coarse speech representation to the generator through audio-visual attention. In addition to the explicit modelling of local and global visual representations, synchronization learning is introduced as a form of contrastive learning that guides the generator to synthesize a speech in sync with the given input lip movements. Extensive experiments demonstrate that the proposed VCA-GAN outperforms existing stateof-the-art and is able to effectively synthesize the speech from multi-speaker that has been barely handled in the previous works.

heading: 1 Introduction
text: Lip to speech synthesis (Lip2Speech) is to predict an audio speech by watching a silent talking face video. While conventional visual speech recognition tasks require human annotations (i.e., text), Lip2Speech does not require additional annotations. Thus, it has drawn big attention as another form of lip reading. However, due to the ambiguity of homophenes that have similar lip movements and the voice characteristics varying from different identities, it is still considered as a challenging problem.
Basically, synthesizing a speech from a silent lip movement video can be viewed as finding a mapping function of visemes into corresponding phonemes. However, only watching short clip-level (i.e., local) lip movements could be challenging to distinguish the homophenes. Thus, global-level lip movements containing the visual context, hints for ambiguity of viseme-to-phoneme mapping, should also be considered along with local-level lip movements. Early deep learning-based works [1, 2, 3] predict each auditory feature (e.g., LPC, mel-spectrogram, spectrogram) within a short video clip and extend the prediction to the entire speech by sliding a window over the whole video sequences. As they operate with clip-level videos of fixed length, they could fail on capturing the global context of the spoken speech. A recent work [4] brings Sequence-to-Sequence (Seq2Seq) architecture [5, 6] that predicts the auditory feature conditioned on both the encoded visual context and the previous prediction and shows a promising performance. However, since they do not explicitly consider local visual features, they may produce out-of-sync speech to the input video. Moreover, due to the sequential nature of the Seq2Seq architecture, the method demands heavy inference time. Since the
∗Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
output audio sequence length is determined when the input video is given for the Lip2Speech task, the time costs can be reduced by adopting different architectures that could predict the speech with one forward step, instead of using a Seq2Seq model which is originally designed to handle input and output with different varying sequence lengths. Lastly, all the above methods focus on handling speech synthesis of constrained speakers (i.e., 1 to 4 speakers), so they could fail to properly handle diverse speakers with one trained model.
In this paper, we design a novel deep architecture, namely Visual Context Attentional GAN (VCAGAN), that jointly models the local and global visual representations to synthesize accurate speech from silent talking face video. Concretely, the proposed VCA-GAN synthesizes the speech (i.e., melspectrogram) based on the local visual features by finding a mapping function of viseme-to-phoneme, while the global visual context assists the generator for clarifying the ambiguity of the mapping. To this end, a visual context attention module is proposed where it extracts the global visual features from the local visual features and provides the global visual context to the generator. It is applied to the generator in multi-scale scheme so that the generator can refine the speech representation from coarse- to fine-level by jointly modelling both the local and the global visual context. Moreover, to guarantee the generated speech to be synced with the input lip movements, synchronization learning is performed that gives feedback to the generator whether the synthesized speech is synchronized or not with the input lip movement. The effectiveness of the proposed framework is evaluated on three public benchmark databases, GRID [7], TCD-TIMIT[8], and LRW[9] in both constrained-speaker setting and multi-speaker setting.
The major contributions of this paper are as follows, 1) To the best of our knowledge, this is the first work to explicitly model the local and global lip movements for synthesizing detailed and accurate speech from silent talking face video. 2) We consider a mel-spectrogram as an image and solve the Lip2Speech problem efficiently using video-to-image translation. 3) This paper introduces synchronization learning which guides the generated mel-spectrogram to be in sync with the input lip video. 4) We show the proposed VCA-GAN can synthesize speech from diverse speakers without the prior knowledge of speaker information such as speaker embeddings.

heading: 2 Related Work
text: Lip to Speech Synthesis. There have been a number of researches and interests in visual-to-speech generation. Ephrat et al.[1] utilized CNN to predict acoustic features from silent talking videos. Then, they augmented the model to two-tower CNN-based encoder-decoder [2] where each tower encodes raw frames and optical flows, respectively. Akbari et al.[3] employed a deep autoencoder for reconstructing the speech features from the visual features encoded by a lip-reading network. Vougioukas et al.[10] directly synthesized the raw waveform from the video by using 1D GAN. Prajwal et al.[4] focused on learning lip sequence to speech mapping for a single speaker in an unconstrained, large vocabulary setting using a stack of 3D convolution and Seq2Seq architecture. Yadav et al.[11] used stochastic modelling approach with variational autoencoder. Michelsanti et al.[12] predicted vocoder features of [13] and synthesized speech using the vocoder. Different from the previous works, our approach explicitly models the local visual feature and global visual context to synthesize accurate speech. Moreover, we try to synthesize the speech from multi-speaker which has rarely been handled in the past.
Visual Speech Recognition (VSR). Parallel to the development of Lip2Speech, Visual Speech Recognition (VSR) have achieved a great advancement [14, 15, 16, 17, 18, 19]. Slightly different from the Lip2Speech, VSR identifies spoken speech into text by watching a silent talking face video. Several works have recently showed state-of-the-art performances in word- and sentence-level classifications. Chung et al.[9] proposed a large-scale audio-visual dataset and set a baseline model for word-level VSR. Stafylakis et al.[20] proposed an architecture that is combined of residual network and LSTM, which became a popular architecture for word-level lip reading. Martinez et al.[21] replaced the RNN-based backend with Temporal Convolutional Network (TCN). Kim et al.[19, 22] proposed to utilize audio modal knowledge through memory network without audio inputs during inference for lip reading. Assael et al.[23] achieved end-to-end sentence-level lip reading network by adopting the CTC loss [24]. Different from the VSR methods, the Lip2Speech task does not require human annotations, thus is drawing big attention with its practical aspects.
Attention Mechanism. The attention mechanism has affected many research fields, such as image captioning [25, 26, 27], machine translation [28, 29], and speech recognition [30, 31, 32]. It can effectively focus on relative information and reduce the interference from less significant one. There have been several works that incorporate attention mechanism in GAN. Xu et al.[25] proposed a cross modal attention model to guide the generator to focus on different words when generating different image sub-regions. Qiao et al.[27] further developed it by proposing a global-local collaborative attentive module to leverage both local word attention and global sentence attention and to enhance the diversity and semantic consistency of the generated images. Li et al.[26] introduced channelwise attention-driven generator that can disentangle different visual attributes, considering the most relevant channels in the visual features to be fully exploited. In this paper, we design a cross-modal attention module working with video and audio modalities for context modelling during speech synthesis. By bringing the global visual context through the proposed visual context attention module, speech of high intelligibility can be synthesized.

heading: 3 Proposed Method
text: Since an audio speech and lip movements in a single video are supposed to be aligned in time, the speech can be synthesized to have the same duration as the input silent video. Let x ∈ RT×H×W×C be a lip video with T frames, height of H , width of W , and channel size of C. Then, our objective is to find a generative model that synthesizes a speech y ∈ RF×4T , where y is a target mel-spectrogram with F mel-spectral dimension and frame length of 4T . The frame length of mel-spectrogram is designed to be 4 times longer than that of video by adjusting the hop length during Short-Time Fourier Transform (STFT). To generate elaborate speech representations, the proposed VCA-GAN (Fig.1) refines the viseme-to-phoneme mapping with the global visual context obtained from a visual context attention module, and learns to produce a synchronized speech with given lip movements. Please note that we treat the mel-spectrogram as an image and train the model with 2D GAN [33, 34].

heading: 3.1 Visual context attentional GAN
text: Considering the entire context from the input lip movements, namely global visual context, can provide additional information that alleviates the ambiguity of homophenes besides the accurate temporal alignment of local visual representations. To achieve this, the generator synthesizes the speech from the local visual features while the global visual context is jointly considered at the intermediate layers of generator through the visual context attention module. Firstly, a local visual encoder φv encodes the video x into local visual features Fv = {f1v , f2v , · · · , fTv } ∈ RT×D, where D is the dimension of embedding. The local visual encoder φv is composed of combination of 3D and 2D convolutions. Due to the locality of the convolution operator, each local visual
feature f tv contains short-clip level (i.e., local) lip movements embedded through the 3D convolution. From the local visual features Fv, n generators (ψ1, ψ2, . . . , ψn) gradually generate and refine the speech representation from low to high resolution. The first generator ψ1 synthesizes coarse speech representation F 1a with the following equation,
F 1a = ψ1([R(Fv); z]), (1)
where z is a noise drawn from a standard normal distribution, R : RT×D → RF4 ×T×D is a repeat operator that forms a 3D tensor of height F4 , width T , and channel D by repeating the input feature F 4 times, and [ ; ] represents concatenation.
The objective of the subsequent generators is to refine the coarse speech representation by jointly modelling the local visual features and global visual context. To this end, a visual context attention module, composed of a global visual encoder and audio-visual attentions, is proposed. The global visual encoder φc derives the global visual features Cv ∈ RT×D by considering the relationships of the entire local visual features Fv through bi-RNN. Then, the audio-visual attention finds the complementary cues (i.e., global visual context) for the speech synthesis, considering the importance of global visual features Cv accordingly with the speech representation F ia ∈ RFi×Ti×Di of i-th resolution. The audio-visual attention can be denoted as following equations,
Qi = F(F ia)W iq , Ki = CvW ik, Vi = CvW iv,
F ic = S(softmax( QiK > i√ d )Vi), (2)
where F ic represents the global visual context obtained, W i q ∈ RFiDi×d, W ik ∈ RD×d, and W iv ∈ RD× FiDi α represent query, key, and value embedding weights, respectively, F : RFi×Ti×Di → RTi×FiDi is a flatten operator that merges the spectral dimension and channel dimension of speech representation, S : RTi× FiDi α → RFi×Ti× Di α is a split operator that maps the 2D tensor into 3D tensor, and α is dimensionality reduction ratio. Note that the audio-visual attention is based on the scaled dot product attention [29] with multi-modal inputs. The visual context attention module and the audio-visual attention are illustrated in Fig.2.
The obtained global visual context F ic is concatenated with the coarse speech representation F i a, and the subsequent generators refine the speech representation iteratively with the following equation,
F i+1a = ψi([F i a ;F i c ]), for i = 1, 2, . . . , n− 1. (3)
Therefore, the following generators can get hints for synthesizing an accurate speech from the global visual context during finding the viseme-to-phoneme mapping using local visual features.
In addition, to generate an image (i.e., mel-spectrogram) with fine details, multi-discriminators are utilized following [25]. The multi-scale mel-spectrograms (ŷ1, ŷ2, . . . , ŷn) are synthesized from each generated speech representation (F 1a , F 2 a , . . . , F n a ) through 1× 1 convolutions and passed into the multi-discriminators, as illustrated in Fig.1.

heading: 3.2 Synchronization
text: Synthesizing a synchronized speech with an input lip movement video is important since human is sensitive to the audio-visual misalignment. In order to generate a synchronized speech, we exploit two strategies: 1) synthesizing the speech by maintaining the temporal representations of input video, and 2) providing a guidance to the generator to focus on the synchronization.
As mentioned above, each visual representation f tv encoded from φv contains local-level lip movement information. We design the generator to synthesize the speech conditioned on the local visual features Fv without disturbing its temporal information, so that the output speech can be naturally synchronized through the mapping of viseme-to-phoneme.
Furthermore, to guarantee the synchronization, we adopt a modern deep synchronization concept [35] that learns not only synced audio-visual representations but also discriminative representations in a self-supervised manner. To this end, a local audio encoder φa is introduced that encodes local audio features, Fa = {f1a , f2a , . . . , fTa } ∈ RT×D, from the ground-truth mel-spectrogram y. With the encoded local audio features Fa and the local visual features Fv , a contrastive learning is performed to learn the synchronized representation by using the following InfoNCE loss [36, 37],
Lc(Fa, Fv) = −E [ log( exp(r(f ja , f j v )/τ)∑
n exp(r(f j a , fnv )/τ)
) ] , (4)
where r represents the cosine similarity metric, τ is temperature parameter, and fa and fv share the same temporal range. The loss function guides to assign high similarity to aligned pairs of audiovisual representations and low similarity to misaligned pairs. We can obtain the synchronization loss for the two encoders Le_sync = 12 (Lc(Fa, Fv) + Lc(Fv, Fa)), where the second term is formed in a symmetric way of Eq.(4) with negative audio samples. In addition, the audio features F̂na encoded from the last generated mel-spectrogram ŷn is also compared with the visual representations to guide the generator to synthesize the synchronized speech. It is guided with the loss function, Lg_sync = ||1− r(F̂na , Fv)||1, which maximizes the cosine similarity between the generated audio features and the given visual features leading the generated mel-spectrogram to be synchronized with the input video. Finally, the final loss for the synchronization is defined asLsync = Le_sync+Lg_sync.

heading: 3.3 Loss functions
text: To generate realistic mel-spectrogram, the objective function for the generator parts of the VCA-GAN is defined as
L = Lg + λreconLrecon + λsyncLsync, (5)
where λrecon and λsync are the balancing weights. Lg represents GAN loss that jointly models the conditional and unconditional distributions as follows,
Lg = − 1
2 Ei[logDi(ŷi) + logDi(ŷi,M(Cv))], (6)
where Di represents the i-th discriminator. The first term is unconditional GAN loss that makes the generated mel-spectrogram to be real, and the second term is conditional GAN loss that guides the generated mel-spectrogram should match with the abbreviated global visual contextM(Cv), where M(·) represents temporal average pooling. To complete the GAN training, the discriminator loss is defined as
Ld = − 1
2 Ei[logDi(yi) + log(1−Di(ŷi)) + logDi(yi,M(Cv)) + log(1−D(ŷi,M(Cv)))].
(7)
Finally, the reconstruction loss Lrecon is defined with the following L1 distance between generated and ground truth mel-spectrograms of i-th resolution,
Lrecon = Ei[||yi − ŷi||1]. (8)

heading: 3.4 Waveform conversion
text: The generated mel-spectrogram can be directly utilized for diverse applications, such as Automatic Speech Recognition (ASR) [38, 39], audio-visual speech recognition [40], and speech enhancement [41]. On the other hand, in order to hear the speech sound, the generated mel-spectrogram should be converted into a waveform. It can be achieved by bringing off-the-shelves vocoders [42, 43, 44, 45] that transform the audio spectrogram into waveform. For our experiments, we use Griffin-Lim [46] algorithm. Since the Griffin-Lim algorithm transforms linear spectrogram to waveform, we use additional postnet which learns to map the mel-spectrogram to linear spectrogram similar to [47, 2]. The postnet is trained using L1 reconstruction loss with the ground-truth linear spectrogram.

heading: 4 Experiments

heading: 4.1 Dataset
text: GRID corpus [7] dataset is composed of sentences following fixed grammar from 33 speakers. We evaluate our model in three different settings. 1) constrained-speaker setting: subject of 1, 2, 4, and 29 are used for training and evaluation. We follow the dataset split of the prior works [4, 2, 10, 3, 12]. 2) unseen-speaker setting: 15, 8, and 10 subjects are used for training, validation, and test, respectively. The dataset split from [10, 12] is used. 3) multi-speaker setting: all 33 subjects are used both training and evaluation. For the dataset split, we follow the well-known protocol in VSR of [23].
TCD-TIMIT dataset [8] is composed of uttering videos from 3 lip speakers and 59 volunteers. Following [4], the data of 3 lip speakers are used for the evaluation in constrained-speaker setting.
LRW [9] is a word-level English audio-visual dataset derived from BBC news. It is composed of up to 1,000 training videos for each of 500 words. Since the dataset was collected from the television show, it has a large variety of speakers and poses, presenting challenges on speech synthesis.

heading: 4.2 Implementation details
text: For the visual encoder, one 3D convolution layer and ResNet-18 [48], a popular architecture in lip reading [49], are utilized. Three generators are used (i.e., n=3) and 2× upsample layer is applied at the last two generators. Each generator is composed of 6, 3, and 3 Residual blocks, respectively. The global visual encoder is designed with 2 layer bi-GRU and one linear layer. For the audio encoder, 2 convolution layers with stride 2 and one Residual block are utilized. The postnet is composed of three 1D Residual blocks and two 1D convolution layers. Finally, the discriminators are basically composed of 2, 3, and 4 Residual blocks. Architectural details can be found in supplementary.
All the audio in the dataset is resampled to 16kHz, high-pass filtered with a 55Hz cutoff frequency, and transformed into mel-spectrogram using 80 mel-filter banks (i.e., F=80). For the dataset composed of 25 fps video (i.e., GRID and LRW), the audio is converted into mel-spectrogram by using window size of 640 and hop size of 160. For the 30 fps video (i.e., TCD-TIMIT), the window size of 532 and hop size of 133 are used. Thus, the resulting mel-spectrogram has four times the frame rate of the video. The images are cropped to the center of the lips and resized to the size of 112× 112. During training, the contiguous sequence is randomly sampled with the size of 40 and 50 for GRID and TCD-TIMIT, respectively. During inference, the network generates speech from arbitrary video frame length1. For the multi-scale ground-truth mel-spectrograms (i.e., y1 and y2), bilinear interpolation is applied to the ground-truth mel-spectrogram y (i.e., y3). We use Adam optimizer [50] with 0.0001 learning rate. The α, λrecon, and λsync are empirically set to 2, 50, and 0.5, respectively. The temperature parameter τ is set to 1. For the GAN loss, non-saturating adversarial loss [34] with R1 regularization [51] is used. Titan-RTX is utilized for the computing.

heading: 4.3 Experimental results
text: For the evaluation metrics, we use 4 objective metrics: STOI [52], ESTOI [53], PESQ [54], and Word Error Rate (WER). STOI and ESTOI are metrics for measuring the intelligibility of speech audio, and higher scores mean better intelligibility of speech audio. PESQ is a metric for measuring
1There begins a performance degradation from above about 10 times the length of those used during training.
perceptual quality of speech audio and a higher score implies the better perceptual quality of speech audio. WER measures how correct the predicted text from the generated speech is. A low error rate means better-generated audio containing accurate spoken content. ASR models (modified from [55]) that take the mel-spectrogram as input and are trained on each experimental setting are utilized to measure the WER.

heading: 4.3.1 Ablation study
text: We conduct an ablation study in order to confirm the effect of each module in the VCA-GAN. By adopting multi-speaker setting of GRID dataset, we build four variants of the proposed model by gradually adding each proposed module, shown in Table 1. For the WER measurement, a pre-trained ASR model on the same setting of GRID dataset is utilized. The Baseline is the model that is trained with Lrecon and GAN loss only without the guidance in multi-resolution. It achieves 0.726 STOI, 0.584 ESTOI, and 1.917 PESQ. When the proposed visual context attention module is added, the performances are improved in all the metrics, achieving 0.732, 0.596, 1.931, and 5.02% in STOI, ESTOI, PESQ, and WER, respectively. Please note the performance improvements are the largest when the proposed visual context attention is introduced. This results reflect that jointly modelling the local visual features and the global visual context during the speech synthesis is important for fine speech generation. Further, when the synchronization learning is performed, the intelligibility score, ESTOI, and WER are improved. Finally, by guiding the generation results in multi-resolution with multi-discriminators, we can synthesize the speech sound clearer with the improvement of PESQ.

heading: 4.3.2 Results in constrained-speaker setting
text: To compare the proposed VCA-GAN with state-of-the-art methods, we train and evaluate the VCAGAN in constrained-speaker setting on both GRID and TCD-TIMIT dataset. Table 2 indicates the comparison results on GRID. The proposed VCA-GAN achieves the best performance in all metrics with large margins except STOI, but it shows comparable performance with that of Lip2Wav [4]. In this experiment, WER is measured using Google ASR API for fair comparison, following [4]. The WER measured from the Google API is 12.25% which surpasses the previous state-of-the-art, Lip2Wav, by 1.83%. With the pre-trained ASR, the measured WER of VCA-GAN is 5.83% showing that the generated mel-spectrogram clearly contains the speech content of input lip movements. The comparison results on TCD-TIMIT dataset is shown in Table 3. Similar to the results on GRID dataset, the VCA-GAN shows the state-of-the-art performances in all three metrics. These results confirm that the VCA-GAN consistently synthesizes the speech from the lip movements with high intelligibility and clear sound regardless of the dataset.
Besides the speech quality metrics, we conduct Mean Opinion Score (MOS) tests. The subjects are asked to rate the 1) intelligibility , 2) naturalness, and 3) synchronization of the synthesized speech on a scale of 1 to 5. We ask 12 participants to evaluate samples from 4 different methods and ground-truth ones. The methods are all proceeded with the contrained-setting of GRID dataset and 20 samples per method are rated. As shown in Table 4, the proposed method achieves the best scores on all the three categories, consistent to the results on the above quality metrics. Specifically, the highest synchronization score verifies that the proposed VCA-GAN can generate well-aligned speech with the input lip movements through the proposed synchronization.

heading: 4.3.3 Results in unseen-speaker setting
text: Furthermore, we investigate the performance of the VCA-GAN on GRID dataset in unseen-speaker setting following [10, 12], shown in Table 5. We measure the WER using the pre-trained ASR model, trained in unseen-speaker setting of GRID dataset. Compared to the previous works [10, 12], the VCA-GAN outperforms in STOI, ESTOI, and PESQ. Since the model cannot access the voice characteristics of unseen speaker during training, the overall performance is lower than that of the constrained-speaker (i.e., seen-speaker) setting. Even in the challenging setting, the VCA-GAN achieves the best WER score with a large margin compared to the previous methods. The result indicates that the synthesized speech by using VCA-GAN contains the correct content, which is important in unseen-speaker speech synthesis.

heading: 4.3.4 Results in multi-speaker setting
text: For verifying the ability of the proposed VCA-GAN on multi-speaker speech synthesis, we train the model by using full data of GRID dataset. Since the voice characteristics varying from different identities, multi-speaker speech synthesis is considered as a challenging problem. Compared to the results of constrained-speaker setting on GRID in Table 2, three metric scores (i.e., STOI, ESTOI, and PESQ) shown in Table 6 do not show significant differences, meaning that the proposed VCA-GAN can synthesize multi-speaker speech without loss of the intelligibility and quality of the speech. Moreover, in Table 6, we compare the Character Error Rate (CER) and WER with a VSR method which predict a text from a given silent talking face video. Even though the VCA-GAN is not trained
with text supervision, the generated speech is intelligible enough to recognize the spoken speech, showing comparable results to the LipNet [23] which is a well-known VSR method.
Finally, we further extend our experiment using LRW dataset, shown in Table 7. While the previous Lip2Wav [4] method utilizes prior knowledge of speakers through speaker embedding [56] for training and inferring on the LRW dataset, the proposed VCA-GAN does not require any prior knowledge of the speakers. Without using additional speaker information, the VCA-GAN outperforms the previous method [4] in three speech quality metrics, STOI, ESTOI, and PESQ. For the WER, Lip2Wav is measured by using Google API while VCA-GAN is measured using the ASR model pre-trained on LRW dataset, so they cannot be directly comparable. However, it clearly shows that the synthesized speech correctly contains the right words even in the challenging environments by outperforming a VSR model [9].

heading: 4.3.5 Evaluation of audio-visual synchronization
text: In order to verify how well the generated speech achieves audio-visual synchronization, we adopt two metrics, LSE-D and LSE-C, proposed by [57]. They measure the degree of the synchronization between audio and video using pre-trained SyncNet [35]. The LSE-D measures the feature distance of two modalities, so less value means better synchronization. The LSE-C measures the confidence score, hence the higher score refers to the better audio-video correlation. Table 8 shows the comparison results in two synchronization metrics in constrained-speaker setting on GRID dataset. The proposed VCA-GAN achieves the best score on both LSE-D and LSE-C by surpassing the previous state-ofthe-art method [12]. This result shows that the proposed synchronization is effective for synthesizing the in-sync speech with the input lip movement video.

heading: 4.3.6 Qualitative results
text: We visualize the generated mel-spectrogram and the waveform of Vocoder-based method[12], the proposed VCA-GAN, and the ground-truth in Fig.3. The results are from the constrained-setting of GRID. We find that the generated mel-spectrograms of VCA-GAN are visually well matched
with the ground-truth. Moreover, by seeing the red dotted-line on mel-spectrogram that indicates the start of utterance of ground-truth, we can confirm that the results from the VCA-GAN are well synchronized while the results of [12] are slightly shifted to the right, compared to the ground-truth mel-spectrogram.

heading: 4.3.7 Computational cost
text: As the proposed VCA-GAN can synthesize the entire speech with one forward step, we can save the inference time than using a Seq2Seq-based method [4]. In order to examine the improved performance in terms of inference speed, we measure the inference time of a Seq2Seq-based method, Lip2Wav, and VCA-GAN including postnet when generating 3-sec speech. For the computing, Titan RTX is utilized. Table 9 shows the measured inference time of each method. The mean inference time of VCA-GAN for generating 3-sec speech takes 25.89ms and the Lip2Wav needs about 5 times more time than VCA-GAN.

heading: 5 Limitations and Societal Impacts
text: This work offers a powerful lip to speech synthesis method. However, as shown in Section 4.3.3, the speech synthesis performances on unseen speakers are still limited compare to the seen speakers. This is because of the unpredictable voice characteristics of unseen speaker that the model cannot properly synthesize. A possible direction for alleviating this limitation is to remove the identity factors of training samples and re-painting them on the generated speech.
With this work, several positive societal benefits can be derived such as assisting human conversations in crowd or silent environments and making it possible to communicate with the speech impaired. In contrast to the advantages, there are also some potential downsides. The lip reading can read the speech by only capturing the lip movement of a certain person, and the visual information can be more easily obtained than the high-quality audio information in a crowded environment or in a long-distance. Thus, the technology is possible of being misused in a surveillance system which can erode individual freedom and damages one’s privacy.

heading: 6 Conclusion
text: We have proposed a novel Lip2Speech framework, VCA-GAN, which generates the mel-spectrogram using 2D GAN by jointly modelling both local and global visual representations. Specifically, the visual context attention module provides the global visual context to the intermediate layers of the generator, so that the mapping of viseme-to-phoneme can be refined with the context information. Moreover, to guarantee the generated speech to be synchronized with the input lip video, synchronization learning is introduced. Extensive experimental results on three benchmark databases, GRID, TCD-TIMIT, and LRW, show that the proposed VCA-GAN outperforms existing state-of-the-art and effectively synthesizes the speech from multi-speaker.
Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers for their helpful comments to improve the paper. This work was partially supported by Genesis Lab under a research project (G01210424).

Review summaries:
The paper presents a visual context attentional GAN for synthesising speech from silent videos. It generates the mel-spectrogram by jointly modelling both local and global visual representations; furthermore a synchronisation technique is proposed. The proposed method is evaluated on three commonly used datasets and shows superior performance as compared with state-of-the-art methods in the literature.  

This paper describes a synthesis approach that uses visual lip movement as input and generates speech as output.  They use a GAN style synthesizer. The most novel contribution is the synchronization technique to align visual and speech signals.

In this work, the authors study the problem of synthesizing speech corresponding to a silent video. The main novelties proposed in this work are: a proposal to incorporate a “global visual context” which is hypothesized to help the model resolve ambiguity in the viseme-to-phoneme mapping; an explicit synchronization technique to ensure that the generated speech is time-aligned with the video; and the use of multiple discriminators to which allow the iterative generation of more fine-grained speech representations. Essentially, the authors propose to model the task as video to image translation, where the video corresponds to a log-mel spectrogram representation of the target speech. 

This paper proposes a lip2speech generative framework based on the proposed Visual Context Attention (VCA) module. The VCA module follows the idea of self-attention and fuses the audio-visual information together to generate finer representations. The proposed pipeline stacks several generators together whose predictions are gradually refined by the VCA module. The global and local clues of input videos are also taken into consideration in this framework. The experiments are conducted on three datasets, while the authors have evaluated their method under different scenarios like constrained-speaker, unseen speaker, etc. The final performance is superior to other baselines.