Lip to Speech Synthesis with Visual Context Attentional GAN
Authors: Minsu Kim
Joanna Hong
Yong Man Ro
Venue: NeurIPS 2021 Poster (2021)
Keywords: video driven speech synthesis
speech reconstruction from silent video
lip reading
audio-visual attention
context attentional gan

Abstract: In this paper, we propose a novel lip-to-speech generative adversarial network, Visual Context Attentional GAN (VCA-GAN), which can jointly model local and global lip movements during speech synthesis. Specifically, the proposed VCA-GAN synthesizes the speech from local lip visual features by finding a mapping function of viseme-to-phoneme, while global visual context is embedded into the intermediate layers of the generator to clarify the ambiguity in the mapping induced by homophene. To achieve this, a visual context attention module is proposed where it encodes global representations from the local visual features, and provides the desired global visual context corresponding to the given coarse speech representation to the generator through audio-visual attention. In addition to the explicit modelling of local and global visual representations, synchronization learning is introduced as a form of contrastive learning that guides the generator to synthesize a speech in sync with the given input lip movements. Extensive experiments demonstrate that the proposed VCA-GAN outperforms existing state-of-the-art and is able to effectively synthesize the speech from multi-speaker that has been barely handled in the previous works.

Lip to Speech Synthesis with Visual Context Attentional GAN
Minsu Kim, Joanna Hong, Yong Man Ro Image and Video Systems Lab KAIST {ms. k, joanna2587, ymro) Ckaist.aaat..r ac. kr
Abstract
In this paper, we propose a novel lip-to-speech generative adversarial network, Visual Context Attentional GAN (VCA-GAN), which can jointly model local and global lip movements during speech synthesis. Specifically, the proposed VCA- GAN synthesizes the speech from local lip visual features by finding a mapping function of viseme-to- tor-ttoo-ppheeee--- while global visual context is embedded into the intermediate layers of the generator to clarify the ambiguity in the mapping induced by homophene. To achieve this, visual context attention module is proposed where it encodes global representations from the local visual features, and provides the desired global visual context corresponding to the given coarse speech representation to the generator through audio-visie attention. In addition to the explicit modelling of local and global visual representations, synchronization learning introduced as a form of contrastive learning that guides the generator synthesize speech in sync with the given input lip movements. Extensive experiments demonstrate that the proposed VCA-GAN outperforms existing state- of-the-art and is able to effectively synthesize the speech from multi-speaker that has been barely handled in the previous works.
Introduction
Lip to speech synthesis (Lip2Speech) is to predict an audio speech by watching a silent talking face video. While conventional visual speech recognition tasks require human annotations (i.e., text), Lip2Speech does not require additional annotations. Thus, it has drawn big attention as another form of lip reading. However, due the ambiguity of homophenes that have similar lip movements and the voice characteristics varying from different identities, it is still considered as a challenging problem.
Basically, synthesizing speech from silent lip movement video can be viewed finding a mapping function of visemes into corresponding phonemes. However, only watching short clip-level (i.e., local) lip movements could be challenging to distinguish the homophenes. Thus, global-level lip movements containing the visual context, hints for ambiguity of viseme-to-phonene mapping, should also be considered along with local-level lip movements. Early deep learning-basse works [ 222 B predict each auditory feature (e.g., LPC, mee-spectroogram spectrogram) within a short video clip and extend the prediction to the entire speech by sliding a window over the whole video sequences. As they operate with clip-level videos of fixed length, they could fail on capturing the global context of (Seq2Seq) architecture [5] o
the spoken speech. recent work 4 brings Sequence-to-Sequee that predicts the auditory feature conditioned on both the encoded visual context and the previous prediction and shows promising performance. However, since they do not explicitly consider local visual features, they may produce out-of-syne speech to the input video. Moreover, due to the sequential nature of the Seq2Seq architecture, the method demands heavy inference time, Since the
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

output audio sequence length is determined when the input video is given for the Lip2Speech task, the time costs can be reduced by adopting different architectures that could predict the speech with one forward step, instead of using a Seq2Seq model which is originally designed to handle input and output with different varying sequence lengths. Lastly, all the above methods focus on handling speech synthesis of constrained speakers (i.e.. I to 4 speakers), so they could fail to properly handle diverse speakers with one trained model.
In this paper, we design a novel deep architecture, namely Visual Context Attentional GAN (VCA- GAN), that jointly models the local and global visual representations to synthesize accurate speech from silent talking face video. Concretely, the proposed VCA-GAN synthesizes the speech (i.e., mel- spectrogram) based on the local visual features by finding a mapping function of tiseme-to-hhoeee while the global visual context assists the generator for clarifying the ambiguity of the mapping. To this end, a visual context attention module is proposed where it extracts the global visual features from the local visual features and provides the global visual context to the generator. It is applied to the generator in multi-scale scheme so that the generator can refine the speech representation from coarse- to fine-level by jointly modelling both the local and the global visual context. Moreover, to guarantee the generated speech to be synced with the input lip movements, synchronization learning is performed that gives feedback to the generator whether the synthesized speech is synchronized or not with the input lip movement. The effectiveness of the proposed framework is evaluated on three public benchmark databases, GRID 77].. (CD-TIMITI and LRW in both constrained-speakeee setting and multi-speaker setting.
The major contributions of this paper are as follows, 1) To the best of our knowledge, this is the first work to explicitly model the local and global lip movements for synthesizing detailed and accurate speech from silent talking face video. 2) We consider a mel-spectrogram as image and solve the Lip2Speech problem efficiently using video-to-image translation. 3) This paper introduces synchronization learning which guides the generated mel-spectrogram to be in syne with the input lip video. 4) We show the proposed VCA-GAN can synthesize speech from diverse speakers without the prior knowledge of speaker information such speaker embeddings.
2 Related Work
Lip to Speech Synthesis. There have been a number of researches and interests in visual-to-speech generation. Ephrat et [l. utilized CNN to predict acoustic features from silent talking videos. Then, they augmented the model to two-tower CNN-based encoder-decoder [2] where each tower encodes raw frames and optical flows, respectively. Akbari al. 3 employed a deep autoencoder for reconstructing the speech features from the visual features encoded by a lip-reading network. Vougioukas et al. [10] directly synthesized the raw waveform from the video by using ID GAN. Prajwal et al.[4] focused on learning lip sequence to speech mapping for a single speaker in an unconstrained, large vocabulary setting using a stack of 3D convolution and Seq2Seq architecture. Yaday et al. used stochastic modelling approach with variational autoencoder. Michelsanti et al. [2] predicted vocoder features of [13 and synthesized speech using the vocoder. Different from the previous works, our approach explicitly models the local visual feature and global visual context to synthesize accurate speech. Moreover, we try to synthesize the speech from multi-speaker which has rarely been handled in the past.
Visual Speech Recognition (VSR). Parallel to the development of Lip2Speech, Visual Speech Recognition (VSR) have achieved a great advancement [14] 15 16
18 19). Slightly different from the Lip2Speech, VSR identifies spoken speech into text by watching silent talking face video. Several works have recently showed state-of-the-art performances in word- and sentence-level classifications. Chung et ( proposed a large-scale audio-visual dataset and set baseline model for werd-level VSR. Stafylakis et al. 20 proposed an architecture that is combined of residual network and LSTM, which became a popular architecture for word-level lip reading. Martinez er al. al. replaced the RNN-based backend with Temporal Convolutional Network (TCN). Kim er
17
al. [19 [22 proposed to utilize audio modal knowledge through memory network without audio inputs during inference for lip reading. Assael el al. 23 achieved end-to-end sentence-level lip reading
network by adopting the CTC loss (24]. Different from the VSR methods, the Lip2Speech task does not require human annotations, thus is drawing big attention with its practical aspects.

---

Viara' Cantrrr
anicroniztoon
Global Visual Encoder
Audio- Audio-Visual Attention
Audio-Misull Attention
Local Audio Encoder
2M Generator
g
xit Generator
Generator
4 spactrogram
Local
Ixmc
1x3 come
nolop
8/4
27
T xxx x.C Input video
Discriminators
Figure Overview of the VCA-GAN, Global visual context is provided through proposed visual context attention module to the generators to refine the speech representation from low- to high- resolution.
Attention Mechanism. The attention mechanism has affected many research fields, such image
captioning 255 26 277, machine translation [28 [29] and speech recognition [30 31
32]
It can
effectively focus on relative information and reduce the interference from less significant one. There have been several works that incorporate attention mechanism in GAN. Xu el al. 25 proposed cross
modal attention model guide the generator focus on different words when generating different image sub-regions. Qiao et al. al.277 further developed it by proposing a global-local collaborative attentive module to leverage both local word attention and global sentence attention and enhance the diversity and semantic consistency of the generated images. Li et al. (26] introduced channel- wise iitention-driven generator that can disentangle different visual attributes, considering the most relevant channels in the visual features to be fully exploited. In this paper, we design a cross-modal attention module working with video and audio modalities for context modelling during speech synthesis. By bringing the global visual context through the proposed visual context attention module, speech high intelligibility can synthesized.
Proposed Method
Since an audio speech and lip movements in a single video are supposed be aligned in time, the speech can be synthesized to have the same duration as the input silent video. Let x E xTxxxxxx be a lip video with 7 frames, height of H, width of W, and channel size of C. Then, our objective is to find a generative model that synthesizes speech y E R F44T where is target mel-spectrogram with F' mel-spectral dimension and frame length of 47'. The frame length of mel-spectroggam is designed to be 4 times longer than that of video by adjusting the hop length during Thme Fourier Transform (STFT). To generate elaborate speech representations, the proposed VCA-GAN (Fig refines the sseme-to-phoneme mapping with the global visual context obtained from visual context attention module, and learns to produce synchronized speech with given lip movements. Please note that we treat the mel-spectrogram as an image and train the model with 2D GAN B3 34
3.1 Visual context attentional GAN
Considering the entire context from the input lip movements, namely global visual context, can provide additional information that alleviates the ambiguity of homophenes besides the accurate temporal alignment of local visual representations. To achieve this, the generator synthesizes the speech from the local visual features while the global visual context is jointly considered at intermediate layers of generator through the visual context attention module. Firstly, a local visual encoder d, encodes the video into local visual features F, (ft. . where is the dimension of embedding. The local visual encoder & is composed of combination of 3D and 2D convolutions. Due to the locality of the convolution operator, each local visual
R
3

---

Visual Context Attention Module
Attention
#...)
global
features
Global visual features
attention score attended global visual features
Softmax
$0
x xxyva
flatten
Matmul
Input lip movements
visual features
spiit
W: 20
Speech representation
Global visual feeatures
Speech
Global visual context
representation
(a) Visual Context Attention module
(b) Audio-Visual Attention
Figure 2: Illustration of visual context modelling through the proposed visual context attention
module. (a) visual context module, and (b) audio-visual attention in detail.
feature f contains short-clip level (i.e., local) lip movements embedded through the 3D convolution. From the local visual features F., generators (1h1, 22,.... vvn) gradually generate and refine the speech representation from low to high resolution. The first generator 1/1 synthesizes coarse speech representation F! with the following equation,
Fd + (R)(F)):))
where = noise drawn from standard normal distribution, RR - R is a repeat 4
operator that forms 3D tensor of height F width T, and channel D) by repeating the input feature F' times, and [i] represents concatenation.
The objective the subsequent generators to refine the coarse speech representation by jointly modelling the local visual features and global visual context. To this end, visual context attention module, composed of a global visual encoder and visual attentions, is proposed. The global visual encoder d derives the global visual features C, RT'x by considering the relationships of the entire local visual features F, through bi-RNN. Then, the audio-visual attention finds the complementary
(i.e., global visual context) for the speech synthesis, considering the importance of global visual features C, accordingly with the speech representation FA E R x of i-th resolution. The visial attention can be denoted as following equations,
Q. = F(F:)W
K; = CCW
V = C.W.
F S(softmar Q.K.
VV.).
Vd.
where Fg represents the global visual context obtained, Wi E FLDddd W E Rxd and Wi E R Dx P.D; represent query, key, and value embedding weights, respectively. F F + RT is flatten operator that merges the spectral dimension and channel dimension speech representation, S R Fi - R Fxxxxx is split operator that maps the 2D tensor into 3D tensor, and dimensionality reduction ratio. Note that the audio-visual attention is based on the scaled dot product attention (29 with multi-modal inputs. The visual context attention module and audio-visual attention are illustrated Fig
The obtained global visual context Fi is concatenated with the coarse speech representation Fi and the subsequent generators refine the speech representation iteratively with the following equation,
Fi+1 wb
(b, ([F F:I), for 1,2.....
Therefore, the following generators can get hints for synthesizing an accurate speech from the global visual context during finding the diseme-to-phonee mapping using local visual features.
In addition, to generate an image (i.e., mel-spectrogramm) with fine details, multi-discriminators are utilized following 25 The multi-scale mel-spectrograms (i1, .2...., Fn) are synthesized from each generated speech representation (F! F2 F?...., .., FT) through 1 convolutions and passed into the
:.....::)
multi-discriminttee illustrated in [

---

3.2 Synchronization
Synthesizing a synchronized speech with an input lip movement video is important since human is sensitive to the audio-visual misalignment. In order to generate synchronized speech, exploit two strategies: 1) synthesizing the speech by maintaining the temporal representations of input video, and 2) providing guidance the generator focus on the synchronization.
As mentioned above, each visual representation f encoded from Φy contains local-level lip movement information. We design the generator synthesize the speech conditioned on the local visual features Fe without disturbing its temporal information, so that the output speech can be naturally synchronized through the mapping of viseme-to coenne-to-phooee phoneme.
Furthermore, to guarantee the synchronization, we adopt a modern deep synchronization concept 35 that learns not only synced audio-visual representations but also discriminative representations seff-supervised manner. To this end, a local audio encoder Φe introduced that encodes local audio features, Fg {Sa
E IRTx from the ground-truth mel-spectrogram With the encoded local audio features F' and the local visual features Fv, a contrastive learning is performed to learn the synchronized representation by using the following InfoNCE loss 336 37).
Fa, Fb) = -E log( exp(r(F2.F1/1 Σn (op(r(f..f://))
(4)
where r represents the cosine similarity metric, T temperature parameter, and fa and fo share the same temporal range. The loss function guides to assign high similarity to aligned pairs of audio- visual representations and low similarity to misaligned pairs. We can obtain the synchronization loss for the two encoders Le syme (Fa F.) + Le (F,, Fa)), where the second term is formed in a symmetric way of Eq.44) with negative audio samples. In addition, the audio features Fi encoded from the last generated mel-spectrogram Un, is also compared with the visual representations to guide the generator to synthesize the synchronized speech. It is guided with the loss function, Gg_enne = ||| r( r(f FI) which maximizes the cosine similarity between the generated audio features and the given visual features leading the generated mel-spectrogram to be synchronized with the input video. Finally, the final loss for the synchronization is defined as Layne = Le_eyme + Cygeyne
3.3 Loss functions
To generate realistic mel-spectrogram, the objective function for the generator parts of the VCA-GAN defined as
L == L.g + Xrecon Crecon + >sgnneesynes
where Arecon and Xsyne are the balancing weights. Cy represents GAN loss that jointly models the conditional and unconditional distributions as follows,
Cy 1:=--EE, (log D.( + log D.G.
(6)
where D, represents the i-th discriminator. The first term is unconditional GAN loss that makes the generated mee-sppectrogrmm to be real, and the second term is conditional GAN loss that guides the generated mel-spectrogram should match with the abbreviated global visual context (C.).. where M(-) represents temporal average pooling.
To complete the GAN training, the discriminator loss is defined
1 E, [log Di(y)) + log(1 D(().) + log Di(,, + log(1 M(C.)))
La - E.
Finally, the reconstruction loss Crecon is defined with the following I.1 distance between generated and ground truth mel- mel-sppetrograms of i-th resolution,
Lrecon E.llyy - [.ll
(8)
5

---

3,4 Waveform conversion
The generated mel-spectrogram can be directly utilized for diverse applications, such as Automatic
Speech Recognition (ASR) [8 [3), audio-visual speech recognition [40], and speech enhancement
[4T] On the other hand, in order to hear the speech sound, the generated mel-spectrogram should be converted into waveform. It can be achieved by bringing off- shfelle-sselles vocoders [42 43
44 45
that transform the audio spectrogram into waveform. For our experiments, we use Griffin-Lim [46 algorithm. Since the Griffin-Lim algorithm transforms linear spectrogram to waveform, we use additional postnet which learns to map the mel-spectrogram to linear spectrogram similar to [47 2
The postnet is trained using L1 reconstruction loss with the ground-truth linear spectrogram.
Experiments
4.1
Dataset
GRID corpus 7 dataset is composed of sentences following fixed grammar from 33 speakers. We
evaluate our model in three different settings. 1) constrained-speaker setting: subject of 1, 2, 4, and 29 are used for training and evaluation. We follow the dataset split of the prior works 2 TO, 12] 2) unscen-speaket setting: 15, 8, and 10 subjects are used for training, validation, and test, respectively.
The dataset split from [10] [12 is used. 3) multi-speaker setting: all 33 subjects are used both training
and evaluation. For the dataset split, we follow the well-known protocol in VSR of 223
TCD-TIMIT dataset X is composed of uttering videos from 3
lip speakers and 59 volunteers.
Following 4. the data of 3 lip speakers are used for the evaluation in constrained constrained-speaker setting.
LRW 99) is a word-level English audio-visual dataset derived from BBC news. It is composed of up to 1,000 training videos for each of 500 words. Since the dataset was collected from the television show, has a large variety of speakers and poses, presenting challenges on speech synthesis.
4.2 Implementation details
For the visual encoder, one 3D convolution layer and ResNet-18 [48), a popular architecture in lip reading 49). are utilized. Three generators are used (i.e., n=3) and 2 upsample layer is applied at the last two generators. Each generator is composed of 6, 3, and 3 Residual blocks, respectively. The global visual encoder is designed with 2 layer bi-GRU and one linear layer. For the audio encoder, 2 convolution layers with stride 2 and one Residual block are utilized. The postnet is composed of three 1D Residual blocks and two 1D convolution layers. Finally, the discriminators are basically composed of 2, 3, and 4 Residual blocks. Architectural details can be found in supplementary.
All the audio in the dataset is resampled to 16kHz, high-pass filtered with a 55Hz cutoff frequency, and transformed into -pe-stcecto using 80 mel-filter banks (i.e., F=80). For the dataset composed of 25 fps video (i.e., GRID and LRW), the audio is converted into mel-spectrogram by using window size of 640 and hop size of 160. For the 30 fps video (i.e., CCD-TIMIT), the window size of 532 and hop size of 133 used. Thus, the resulting mel-spectrogram has four times the frame rate of the video. The images are cropped to the center of the lips and resized to the size of 112 x 112. During training. the contiguous sequence is randomly sampled with the size of 40 and 50 for GRID and TCD-TIMIT, respectively. During inference, the network generates speech from arbitrary video frame For the multi-scale ground-truth mel-spectrograms (i.e., yı and 1/2). bilinear interpolation is applied to the ground-truth mel-spectrogram y (i.e., ya). We use Adam optimizer (50) with 0,0001 learning rate, The a, Arecon. and Asyme are empirically set to 50, and 0.5, respectively. The temperature parameter is set to 1. For the GAN loss, not-satura adversarial loss [34 with R1 regularization [51] is used.
is utilized for the computing.
4.3 Experimental results
For the evaluation metrics, we use 4 objective metrics: STOI 52). ESTOI (53, PESQ (54] and
Word Error Rate (WER). STOI and ESTOI are metrics for measuring the intelligibility of speech audio, and higher scores mean better intelligibility of speech audio. PESQ is metrie for measuring
There begins performance degradation from above about 10 times the length of those used during training.

---

Table 1: Ablation study in multi-speaker setting on GRID
Proposed Method
Visual context Synchro- nization
Multi
Baseline
STOI
ESTOI
PESQ
WER
attention
discriminators
X 0.726
0.584
1.917
5,43%
X
0.732
0.596
1.931
5.02%
0.733
0.601
1.935
4.91%
0.736
0.604
1.961
4.80%
Table 2: Performance comparison in constrained-speaker setting on GRID
Method
STOI
ESTOI
PESQ
WER
Vid2Speech
0,491
0.335
1.734
44.92%
Ephrat
0.659
0.376
1.825
27.83%
Lip2AudSped
0.513
0.352
1.673
32.51 %
ID GAN-based
10
0.564
0.361
1.684
26.64%
Lip2Wav 4
0.731
0.535
1.772
14.08%
VAE-based
0.724
0.540
1.932
Vocoder-based
12
0.648
0.455
1.900
23.33%
VCA-GAN 0.724
0.609
2.008
12.25%
perceptual quality of speech audio and a higher score implies the better perceptual quality of speech audio. WER measures how correct the predicted text from the generated speech is. A low error rate means
audio containing accurate spoken content. ASR models (modified from (55)) that take the mel-spectrogram as input and are trained each experimental setting are utilized to measure the WER.
4.3.1 Ablation study
We conduct an ablation study in order to confirm the effect of each module in the VCA-GAN. By adopting multi-speaker setting of GRID dataset, we build four variants of the proposed model by gradually adding each proposed module, shown in Table For the WER measurement, a pre-trained ASR model on the same setting of GRID dataset is utilized. The Baseline is the model that is trained with Crecon and GAN loss only without the guidance multi-resolltion. achieves 0.726 STOI, 0.584 ESTOI, and 1.917 PESQ. When the proposed visual context attention module is added, the performances are improved all the metrics, achieving 0.732, 0.596, 1.931, and 5.02% STOI, ESTOL, PESQ, and WER, respectively. Please note the performance improvements are the largest when the proposed visual context attention is introduced. This results reflect that jointly modelling the local visual features and the global visual context during the speech synthesis is important for fine speech generation. Further, when the synchronization learning is performed, the intelligibility score, ESTOI, and WER improved. Finally, by guiding the generation results in multi-resolution with multi-discriminators, we can synthesize the speech sound clearer with the improvement of PESQ.
4.3.2 Results in constrained-speaker setting
To compare the proposed VCA-GAN with state-of-the-art methods, we train and evaluate the VCA- GAN in constrained-speaker setting on both GRID and TCD-TIMIT dataset. Table indicates the comparison results on GRID. The proposed VCA-GAN achieves the best performance in all metries with large margins except STOI, but it shows comparable performance with that of Lip2Wav (4). In this experiment, WER measured using Google ASR API for fair comparison, following 44. The WER measured from the Google API is 12.25% which surpasses the previous state-of-the-an Lip2Wav, by 1.83%. With the pre-trained ASR, the measured WER VCA-GAN is 5.83% showing that the generated mel mel-spectrogram clearly contains the speech content of input lip movements. The comparison results on TCD-TIMIT dataset is shown Table 3 Similar to the results on GRID dataset, the VCA-GAN shows the state-of-the-an performances in all three metrics. These results confirm that the VCA-GAN consistently synthesizes the speech from the lip movements with high intelligibility and clear sound regardless of the dataset.
7

---

Table 3: Performance comparison in constrained-speaker setting on TCD-TIMI
Method
STOI
ESTO1
PESQ
Vid2Speech [
0.451
0.298
1.136
Ephrat et al.
0.487
0.310
1.231
Lip2AudSped B
0,450
0.316
1.254
ID GGNN-baed [T0
0,511
0.321
1.218
Lav2Wav 4
0.558
0.365
1.350
VCA-GAN
0,584
0.401
1,425
Table 4: MOS score comparison with 95% confidence interval computed from the t-distribution
Method
Intelligibility
Naturalness
Synchronization
ID GANGGNN-aar bused [00
2.74 1 0.61
2.28 +: 0.55
3.69 0.50
Wip2Wav 4
3.23 + 0.48
2.87 0.54
3.86 0.38
Vocoder-based
12
3.68 + 0.67
3,00 + 0.68
4.16 0.43
VCA-GAN
4,06 1 0.53
3.35 0.50
4.30 0.37
Actual Voice
4.94 + 0.04
4.94 0.09
4.87 0.12
Besides the speech quality metrics, we conduct Mean Opinion Score (MOS) tests. The subjects are asked to rate the 1) intelligibblity 2) naturalness, and 3) synchronization of the synthesized speech scale of to 5. We ask 12 participants to evaluate samples from 4 different methods and ground-tran tround-truth ones. The methods are all proceeded with the contrained-setting of GRID dataset and 20 samples per method rated. As shown in Table the proposed method achieves the best scores on all the three categories, consistent to the results on the above quality metrics. Specifically, the highest synchronization score verifies that the proposed VCA-GAN can generate well-aligned speech with the input lip movements through the proposed synchronization.
4.3.3 Results in unseen-speaker setting
Furthermore, we investigate the performance the VCA-GAN on GRID dataset unseen-speaker setting following [10 [2], shown in Table 5 We measure the WER using the pre-trained ASR model, trained in unseen-speaker setting of GRID dataset. Compared to the previous works 12) the VCA-GAN outperforms in STOI, ESTOI, and PESQ. Since the model cannot access the voice characteristics unseen speaker during training, the overall performance lower than that of the centtrained-speaaeeee (i.e., seen seen-speaker) setting. Even in the challenging setting, the VCA-GAN achieves the best WER score with large margin compared to the previous methods. The result indicates that the synthesized speech by using VCA-GAN contains the correct content, which is important in unseen-speaker speech synthesis.
4.3.4 Results in multi-speaker setting
For verifying the ability of the proposed VCA-GAN on multi-speaker speech synthesis, we train the model by using full data of GRID dataset. Since the voice characteristies varying from different identities, multi-speakerr speech synthesis is considered as a challenging problem. Compared to the results of constrained-speake setting GRID in Table 2 three metric scores (i.e., STOI, ESTOI, and PESQ) shown Table not show significant differences, meaning that the proposed VCA-GAN synthesize
speech without loss of the intelligibility and quality of the speech. Moreover, in Table we compare the Character Error Rate (CER) and WER with a VSR method which predict text from given silent talking face video. Even though the VCA-GAN is not trained
Table 5: Performance comparison in unseen-speaker setting on GRID
Method
STOI
ESTOI
PESQ
WER
GAN-based 110
0.445
0.188
1.240
38.51
Vocoder-based
[2
0.537
0.227
1.230
37.97%
VCA-GAN
0.569
0.336
1.372
23.45%

---

spectrogram
Waveform
Mci spectrogram
Waveform
Vocoder-based
[12]
VCA-GAN
Ground-Truth
Speaker (male): Green Five Now!
Speaker (female): 'Bin Red Five Please'
Figure 3: Qualitative results of generated mel-spectrogram and waveform.
Table 6: Performance multi-speaker setting on GRID
Method
STOI
ESTOI
PESQ
CER
WER
LipNet [23
2.0%
5.6%
VCA-GAN
0.736
0.604
1.961
1.8%
4.8%
with text supervision, the generated speech is intelligible enough to recognize the spoken speech, showing comparable results to the LipNet 23 which is well-known VSR method.
Finally, further extend experiment using LRW dataset, shown in Table 7 While the previous Lip2Wav [ method utilizes prior knowledge speakers through speaker embedding [56] for training and inferring the LRW dataset, the proposed VCA-GAN does not require any prior knowledge of the speakers. Without using additional speaker information, the VCA-GAN outperforms the previous method 4 in three speech quality metries, STOI, ESTOI, and PESQ. For the WER, Lip2Wav is measured by using Google API while VCA-GAN is measured using the ASR model pre-trained on LRW dataset, so they cannot directly comparable. However, it clearly shows that the synthesized speech correctly contains the right words even in the challenging environments by outperforming VSR model 99]
4.3.5
Evaluation of audio-visual synchronization
In order to verify how well the generated speech achieves audio-visual synchronization, adopt two metrics. LSE-D and LSE-C, proposed by [57] They measure the degree of the synchronization between audio and video using pre- mre-trained SyneNet 35 The LSE-D measures the feature distance of two modalities, so less value means better synchronization. The LSE-C measures the confidence score, hence the higher score refers the better audio-video correlation. Table shows the comparison results in two synchronization metrics in constrained-speake setting GRTD dataset. The proposed VCA-GAN achieves the best score on both LSE-D and LSE-C by surpassing the previous state-of- the-art method [12]. This result shows that the proposed synchronization is effective for synthesizing the in-syne speech with the input lip movement video.
4.3.6 Qualitative results
We visualize the generated mel-spectrogram and the waveform of Vocoder-based [2] the proposed VCA-GAN, and the ground-truth in Fig The results are from the constraine--setting of GRID. We find that the generated mel-spectrograms of VCA-GAN are visually well matched
Table 7: Performance comparison on LRW.
Reported by using Google API.
Method
STOI
ESTOI
PESQ
WER
Chung et al. 9
38.90%
Lip2Wav 4
0.543
0.344
1.197
134.20%
VCA-GAN
0,565
0.364
1.337
29.95%

---

Table 8: LSE-D and LSE-C comparisons for measuring synchronization
Method
LSE-D 4
LSE-C
ID GAN-bused
100
8,107
4.797
Vocoder-based 12
6.717
6.178
VCA-GAN
6.698
6.373
Table Comparison of inference speed with a Seq2Seq- method.
Method
Inference time
Lip2Wav A
141.73 ms
VCA-GAN
25.89 ms
with the ground-truth. Moreover, by seeing the red dotted-ling on mee-speecto that indicates the start of utterance of ground-truth we can confirm that the results from the VCA-GAN are well synchronized while the results of [12] are slightly shifted to the right, compared to the ground-truth mel-spectrogram.
4.3.7 Computational cost
As the proposed VCA-GAN can synthesize the entire speech with one forward step, we can save the inference time than using Seq2Seq-based method 44. order to examine the improved performance in terms of inference speed, we measure the inference time of Seq2Seq-baser method, Lip2Wav, and VCA-GAN including postnet when generating 3-sec speech. For the computing, Titan RTX is utilized. Table [ shows the measured inference time of each method. The mean inference time of VCA-GAN for generating 3-sec speech takes 25. 5..9mms and the Lip2Wav needs about 5 times more time than VCA-GAN.
5 Limitations and Societal Impacts
This work offers a powerful lip to speech synthesis method. However, as shown in Section 4.3.3 the speech synthesis performances unseen speakers are still limited compare to the seen speakers. This because the unpredictable voice characteristics of unseen speaker that the model cannot properly synthesize. A possible direction for alleviating this limitation is to remove the identity factors of training samples and re-painting them on the generated speech.
With this work, several positive societal benefits can be derived such as assisting human conversations in crowd or silent environments and making it possible communicate with the speech impaired. In contrast to the advantages, there are also some potential downsides. The lip reading can read the speech by only capturing the lip movement of a certain person, and the visual information can be more easily obtained than the high-quality audio information in a crowded environment or in long-distance. Thus, the technology is possible of being misused in a surveillance system which can erode individual freedom and damages one's privacy.
6 Conclusion
We have proposed a novel Lip2Speech framework, VCA-GAN, which generates the mpe-sppecroggra using 2D GAN by jointly modelling both local and global visual representations. Specifically, the visual context attention module provides the global visual context the intermediate layers of the generator, so that the mapping of Moreover, to guarantee the generated speech to be synchronized with the input lip video, synchro- nization learning is introduced. Extensive experimental results on three benchmark databases, GRID, TCD-TIMI, and LRW, show that the proposed VCA-GAN outperforms existing state-of-the-an and effectively synthesizes the speech from multi-speaker.
can be refined with the context information.
Acknowledgments and Disclosure of Funding
We would like to thank the anonymous reviewers for their helpful comments to improve the paper. This work was partially supported by Genesis Lab under research project (G01210424).
10

---

References
[1] Ariel Ephrat and Shmuel Peleg. Vid2speech: speech reconstruction from silent video. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5095-5099, IEEE, 2017.
[2] Ariel Ephrat, Tavi Halperin, and Shmuel Peleg. Improved speech reconstruction from silent video. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 455 462.-462222 2017.
[3] Hassan Akbari, Himani Arora, Liangliang Cao, and Nima Mesgarani. Lip2audspec: Speech reconstruction from silent lip movements video. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2516 2520. IEEE, 2018.
[4] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. Learning individual speaking styles for accurate lip to speech synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13796-13805, 2020.
[5] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks, arXiv preprint ar0.v:1409.322 15, 2014.
[6] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the propertics of neural machine translation: Encoder-decoder approaches. arXiv preprint (rXiv:1409.1255: 2014.
[7] Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. An audio- corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Sociery of America, 120(5):2421-2424, 2006.
[8] Naomi Harte and Eoin Gillen. Ted red-timit: An audio-visual corpus of continuous speech, IEEE Transactions on Multimedia, 17(5)6033 615, 2015.
[9] Joon Son Chung and Andrew Zisserman. Lip reading in the wild. In Asian Conference on Computer Vision, pages 87-103. Springer, 2016.
[10] Konstantinos Vougioukas, Pingchuan Ma, Stavros Petridis, and Maja Pantic. Video-driven speech recon- struction using generative adversarial networks. arXiv preprint arXiv: (eoe: ,,,00,63,,11111 2019.
[11] Ravindra Yadav, Ashish Sardana, Vinay P Namboodiri, and Rajesh M Hegde. Speech prediction in silent videos using variational autoencoders. arXiv preprint vrXii:2011.073:44 2020.
[12] Daniel Michelsanti, Olga Slizovskaia, Gloria Haro, Emilia Gómez, Zheng- Hua Tan, and Jesper Jensen. Vocoder-hased speech synthesis from silent videos. arXiv preprint arXiv:2004.0255 2020.
[13] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: a vocoder-basee high-quality speech syn- thesis system for real-time applications. IEICE TRANSACTIONS on Information and Systems, 99(7):1877 1884, 2016.
[14] Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence, 2018.
[15] Chenhao Wang. Multi-grained spatio-temporal modeling for lip-reading, arXiv preprint arXiv: arXiv:1908. 1618, 2019.
[16] Jingyun Xiao, Shuang Yang. Yuanhang Zhang, Shiguang Shan, and Xilin Chen. Deformation flow based two-stream network for lip reading. arXiv preprint arXiv:2003.0577 2020.
[17] X. Zhao, S. Yang, Shan, and X. Chen. Mutual information maximization for effective lip reading. In 2020 15th IEFE International Conference on Automatic Face and Gesture Recognition (FG 2020) (FG), pages 843-850, Los Alamitos, CA, USA, may 2020. IEFE Computer Society.
[18] Yuanhang Zhang, Shuang Yang, Jingyun Xiao, Shiguang Shan, and Xilin Chen. Can we read speech beyond the lips? rethinking roi selection for deep visual speech recognition. arXiv preprint arXiv:2003.033: 2020.
[19] Minsu Kim, Joanna Hong, Se Jin Park, and Yong Man Ro. Multi-modality associative bridging through memory: Speech sound recollected from face video. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 296-306. October 2021.
[20] Themos Stafylakis and Georgios Tzimiropoulos. Combining residual networks with Istms for lipreading. arXiv preprint 1rXiv:1703.04105 2017.
11

---

[21] Brais Martinez, Pingehuan Ma. Stavros Petridis, and Maja Pantic. Lipreading using temporal convolu- tional networks. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6319-6323. IECE, 2020.
[22] Minsu Kim, Joanna Hong, Se Jin Park, and Yong Man Ro. Cromm-vsr: Cross-modal memory augmented visual speech recognition. IEEE Transactions on Multimedia, pages 1-1, 2021.
[23] Yannis M Assael, Brendan Shillingford, Shimon Whiteson, and Nando De Freitas. Lipnet: End-to-end sentence-level lipreading. arXiv preprint arXiv: 161116111000 1.01599, 2016.
[24] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference Machine learning, pages 369-376, 2006.
[25] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1316-1322 1324, 2018.
[26] Bowen Li, Xiaojuan Qi. Thomas Lukasiewicz, and Philip HS Torr. Controllable text-to-image generation. arXiv preprint A..iv:99900008888 2019.
[27] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to-image generation by redescription. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1505- 1514, 2019.
[28] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint
2014.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, In Advances in neural information processing systems, pages 5998 0008-600,,200 2017.
[30] Kai Xu, Dawei Li, Nick Cassimatis, and Xiaolong Wang. Leanet: End-to-end lipreading with cascaded attention-cte. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 548-555 IEEE, 2018
[31] Stavros Petridis. Themos Stafylakis. Pingchuan Ma, Georgios Tzimiropoulos, and Maja Pantic. Audio- visual speech recognition with a hybrid etc/attention architecture. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 513-520. IEEE, 2018
[32] Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention- based models for speech recognition. arXiv preprint
2015.
[33] Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. arXiv preprint 1rXiv:1802.04200. 2018
[34] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farlcy, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv: 406.2661, 2014.
[35] Joon Son Chung and Andrew Zisserman. Out of time: automated lip sync in the wild. In Workshop on Multi-view Lip-reading, ACCV, 2016.
[36] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint
2018.
[37] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[38] Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin, Anmol Gulati, Ruoming Pang, and Yonghui Wu. Contexmet: Improving convolutional neural networks for automatic speech recognition with global context. arXiv preprint 23111 2020.
[39] Awni Hannun. Carl Case, Jared Casper. Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep speech: Scaling end-to-end speech recognition. arXiv preprint arXiv: 1412.5567 2014.
[40] Bo Xu, Cheng Lu, Yandong Guo, and Jacob Wang. Discriminative multi modlii-mddllity speech recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14433- 14442, 2020.
12

---

[41] Aviv Gabbay, Asaph Shamir, and Shmuel Peleg. Visual speech enhancement. arXiv preprint urXiv:1711.08789, 2017.
[42] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.0333 034999 2016.
[43] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6617-3221. IEEE, 2019.
[44] Kundan Kumar, Rithesh Kumar. Thibault de Boissiere. Lueas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron Courville. Melgan: Generative adversarial networks for conditional waveform synthesis, arXiv preprint 19Xiv:111006611, 2019.
[45] Jungil Kong, Jachyeon Kim, and Jackyoung Bae. gifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. arXiv preprint arXiv:2010.05646, 2020.
[46] Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. JEEE Transac- tions on acoustics, speech, and signal processing, -2(2)223-2244 1984.
[47] Yuxuan Wang, RJ Ekerry-Ryan, Daisy Stanton, Yonghui Wu, Ron Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao. Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint ar 1 7703. 10135, 2017.
[48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[49] Stavros Petridis, Themos Stafylakis, Pingehuan Ma, Feipeng Cai, Georgios Tzimiropoulos, and Maja Pantie. End-to-end audiovisual speech recognition. In 2018 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 6548 6552. IEEE, 2018.
[50] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint weXiiv:412..9999 2014.
[51] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 3481-3490. PMLR, 2018
[52] Cees II Taal, Richard C Hendriks, Richard Heusdens. and Jesper Jensen. A short-time objective intelli- gibility measure for time-frequent frequency weighted noisy speech. In 2010 IEEE international conference on acoustics, speech and signal processing, pages 4214-42144 4217. IEEE, 2010.
[53] Jesper Jensen and Cees Taal. An algorithm for predicting the intelligibility of speech masked by modu- lated noise maskers. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(11):2009- 2022, 2016.
[54] Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE International Conference Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 01CH37221), volume 2, pages 749-752. IEFF, 2001.
[55] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Erie Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech End-to-end speech recognition in english and mandarin. In International conference on machine learning, pages 173-182. PMLR, 2016.
[56] Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, et al. Transfer leamning from speaker verification to multispeaker text-to-speech synthesis. arXiv preprint arXiv: 1806.04558 2018
[57] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, pages 484 492, 2020.
13


Reviews:
[{"review_id": "s0amb7oyOxU", "reviewer": "Reviewer_Nvn1", "summary": "This paper describes a synthesis approach that uses visual lip movement as input and generates speech as output.  They use a GAN style synthesizer. The most novel contribution is the synchronization technique to align visual and speech signals.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is about GAN synthesis of speech based on lip images.  This is not the first paper to have done this: e.g. \"Video-Driven Speech Reconstruction using Generative Adversarial Networks\" Vougioukas1 et al. 2019 https://arxiv.org/pdf/1906.06301.pdf The architectural modification is the use of a hierarchical structure using local and global features.\n\n\n\"However, only watching short clip-level (i.e., local) lip movements could be challenging to distinguish the homophones.\" Define “homophones” here?  The most common usage of \"homophones\" is to refer to words with different written forms but identical spoken forms, like \"their\" and \"there\". it is not clear how global level information is helpful in distinguishing these.\n\nLine 111: \"...global visual context, can provide additional information that alleviates the ambiguity of homophones\"\tPlease explain how, perhaps with an example?\n\nLine 171 \", which maximizes the cosine similarity between the generated audio features and the given visual features leading  the generated mel-spectrogram to be synced with the input video.” In order to perform the synchronization between visual inputs and audio outputs, the two features mush share a common representation space.  Does this choice improve or impair quality?  \tPer Table 1 synchronization gives a very minor win, while the largest improvement comes from the use of visual attention.\n\nTable 2: The order of systems in Table2 is unclear.  Why this sorting?\n\nTable 1: It would be useful to compare the pretrained ASR (5.83%) to any other system.  Without that comparison, this number isn’t meaningful, and probably should be removed.\n\nLine 294: “In addition, for the WER, Lip2Wav is measured by using Google API while VCA-GAN is measured using the ASR model pre-trained on LRW dataset, so they cannot be directly comparable. . However, it clearly shows that the synthesized speech correctly contains the right words even in the challenging environments.”  It is impossible to draw comparisons here, this mismatched evaluation of intelligibility should be removed.  It would be better to either evaluate both using the same system, or leave the comparison out.\n\nIt would be helpful to have some subjective measure of the value of synchronized images and audio.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does include a section on Limitations and Societal Impact. While the potential abuse of visual based speech synthesis or recognition is acknowledged, there is no mention of mitigation factors or discussion of the balance between positive and negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"], "time_spent_reviewing": "2"}, "extra_text": {"ethical_concerns": "The Ethics Guidelines includes an example that covers this submission and similar work\n\n5. Develop or extend harmful forms of surveillance. For example: could it be used to collect or analyze bulk surveillance data to predict immigration status or other protected categories, or be used in any kind of criminal profiling?\n\nVisual information is much easier to collect at scale than high quality audio information.  E.g. in a crowded public place recording audio of a single speaker is very challenging, while collecting visual imaging of a single person is fairly straightforward.  Visual speech recognition and speech synthesis would extend the ability to surveil people's speech in areas where audio collection is impossible, but visual collection is possible.\n"}}, {"review_id": "mKsdjDhUU9", "reviewer": "Reviewer_uxUX", "summary": "In this work, the authors study the problem of synthesizing speech corresponding to a silent video. The main novelties proposed in this work are: a proposal to incorporate a “global visual context” which is hypothesized to help the model resolve ambiguity in the viseme-to-phoneme mapping; an explicit synchronization technique to ensure that the generated speech is time-aligned with the video; and the use of multiple discriminators to which allow the iterative generation of more fine-grained speech representations. Essentially, the authors propose to model the task as video to image translation, where the video corresponds to a log-mel spectrogram representation of the target speech.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis is a very well written paper (although it does have a number of minor typographical and grammatical errors; few are listed below) which is generally well motivated and easy to follow. The experimental section, in particular, is comprehensive and demonstrates that the proposed system outperforms previous works with larger gains in unseen and multi-speaker settings (which are more challenging to begin with). \n\nI had a few minor comments which if resolved would strengthen the paper further.\n\n- **Ablation results in Table 1 and Table 3**: It would be good to include WER numbers as well for the ablation study in Table 1 and the results in Table 3 to show fidelity in addition to intelligibility and quality, since it is also useful to see how well the reconstructed speech matches the ground-truth audio.\n\n- **Captured global context**: A point that is made in the paper, and in Table 1, is that the use of the global context helps resolve ambiguities. It would be interesting to present the results of some error analysis to show how the global context helps improve performance (WER results in Table 1, would help clarify this somewhat as well).\n\n- **Length of test videos**: In Section 4.2, the authors mention that during training the videos are segmented to smaller chunks, but during inference the authors evaluate the full videos. I was curious if the authors have performed any experiments to study the overall performance as a function of video length. Are the global features still useful for very long videos, for example? It would be useful to mention some statistics about the test video lengths for the various datasets (perhaps in a footnote if space is limited).\n\n- **Intelligibility and Quality Metrics**: Minor comment: In section 4.3, it might be useful to mention a couple of sentences about the STOI/ESTOI/PESQ metrics to help the reader understand what these capture, whether higher/lower are better and what level of differences are generally significant.\n\nTypographical and Grammatical Errors:\n1. Page 3, Line 98: “... contexture design … ”\n2. Page 3, Section 3: “... as the input silence video.” --> “... as the input silent video.”\n3. Page 3, Section 3: “... is designed to have 4 times longer ...” -->  “... is designed to be 4 times longer ...”\n4. Page 3, Section 3: “... dealing the mel-spectrogram … ” -->  “... treating the mel-spectrogram ...”\n5. Eqn 6: It might be better to use the notation L_{g}^{i} instead of L_g\n6. Page 6, Line 221: “... four times the frame of the ...” --> “... four times the frame rate of the ...”\n7. Page 8, Line 287: “Even the VCA-GAN is ...” --> “Even though the VCA-GAN is ...”\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One possible concern is that this work could be used for surveillance when only a video but no audio is available. The authors mention that this is a potential concern.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "_jGRzXgrGq", "reviewer": "Reviewer_35i8", "summary": "This paper proposes a lip2speech generative framework based on the proposed Visual Context Attention (VCA) module. The VCA module follows the idea of self-attention and fuses the audio-visual information together to generate finer representations. The proposed pipeline stacks several generators together whose predictions are gradually refined by the VCA module. The global and local clues of input videos are also taken into consideration in this framework. The experiments are conducted on three datasets, while the authors have evaluated their method under different scenarios like constrained-speaker, unseen speaker, etc. The final performance is superior to other baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTH\n+ The idea of adding an audio-visual attention module to iteratively refine the representation seems to be novel in the lip2speech field. The motivation for designing the VCA module seems reasonable to me.\n+ The performance of the proposed method looks promising. Different scenarios, including constrained-speaker, unseen speaker and multiple-speaker are evaluated, which demonstrates the generality of the proposed pipeline. The ablation study is also performed to illustrate the effectiveness of each component.\n+ The code is submitted for reproduction. \n+ The paper writing is well organised and easy to follow. \n\nWEAKNESS\n- In VCA module, why use speech representation as the query and use visual clues as the key and value?\n- For the pipeline design, why choose the number of generators (n) to 3? How will n affect the whole performance?\n- It is difficult for me to understand why the Audio-Visual Attention mechanism can generate better audio representations. Can the authors explain this more detailedly and intuitively?\n- The performance of audio-visual recognition models on the generated voice can also be a helpful evaluation metric.\n- I checked the demo in the supplemental material,  but the results of various methods (author's vs baselines) doesn't seem too much different to me.  \n\nOverall, the idea of this paper can be novel in the field of lip2speech, and the motivation and model design looks reasonable. The proposed method is evaluated extensively and the results are superior. I will vote for a marginally above. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, I think so.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "GsOda9XApO", "reviewer": "Reviewer_jtpu", "summary": "The paper presents a visual context attentional GAN for synthesising speech from silent videos. It generates the mel-spectrogram by jointly modelling both local and global visual representations; furthermore a synchronisation technique is proposed. The proposed method is evaluated on three commonly used datasets and shows superior performance as compared with state-of-the-art methods in the literature.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Table 1 presents an ablation study in multi-speaker setting on GRID. It is unclear what the difference between the last row of Table 1 and the last row of Table 6, both representing VCA-GAN but with minor performance difference - good to clarify. While the ablation study shows each component contributes to the performance improvement, but the improvement brought by each component is rather small. More specifically, all three components together improve STOI from 0.726 to 0.736 and PESQ from 1.917 to 1.961. This study makes it less clear why the proposed system works well on all three benchmark data sets. \n\nGood to synchronise audio and visual information. The three metrics used in the paper, also commonly used in the literature, are unable to show its whole merit since they are audio only metrics. Watching the demo video in supplementary material does not show the audio and visual sequences are synchronised well though. \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors investigate speech generation out of silent videos based on the so-called visual context attentional GAN.  The problem is interesting and challenging.  The authors propose to use global visual context to reduce the ambiguity when mapping visemes to phonemes and introduce a synchronization mechanism via contrastive learning to make speech and lip movements in sync.  Extensive experiments and evaluation are conducted to compare its performance with a variety of existing techniques.  The proposed VCA-GAN reports the state-of-the-art performance in the lip-to-speech domain.  The paper is well written and easy to follow.  The experiments are extensive yet controlled.  The rebuttal has clarifies most of the concerns in the review.  All reviewers are supportive on accepting the paper.  The authors should revise the submission to address the concerns (including the ethical concerns) raised by the reviewers.
Decision: Accept (Poster)