Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a visual context attentional GAN for synthesising speech from silent videos. It generates the mel-spectrogram by jointly modelling both local and global visual representations; furthermore a synchronisation technique is proposed. The proposed method is evaluated on three commonly used datasets and shows superior performance as compared with state-of-the-art methods in the literature.  
main_review: Table 1 presents an ablation study in multi-speaker setting on GRID. It is unclear what the difference between the last row of Table 1 and the last row of Table 6, both representing VCA-GAN but with minor performance difference - good to clarify. While the ablation study shows each component contributes to the performance improvement, but the improvement brought by each component is rather small. More specifically, all three components together improve STOI from 0.726 to 0.736 and PESQ from 1.917 to 1.961. This study makes it less clear why the proposed system works well on all three benchmark data sets. 

Good to synchronise audio and visual information. The three metrics used in the paper, also commonly used in the literature, are unable to show its whole merit since they are audio only metrics. Watching the demo video in supplementary material does not show the audio and visual sequences are synchronised well though. 
 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes a synthesis approach that uses visual lip movement as input and generates speech as output.  They use a GAN style synthesizer. The most novel contribution is the synchronization technique to align visual and speech signals.
main_review: This paper is about GAN synthesis of speech based on lip images.  This is not the first paper to have done this: e.g. "Video-Driven Speech Reconstruction using Generative Adversarial Networks" Vougioukas1 et al. 2019 https://arxiv.org/pdf/1906.06301.pdf The architectural modification is the use of a hierarchical structure using local and global features.


"However, only watching short clip-level (i.e., local) lip movements could be challenging to distinguish the homophones." Define “homophones” here?  The most common usage of "homophones" is to refer to words with different written forms but identical spoken forms, like "their" and "there". it is not clear how global level information is helpful in distinguishing these.

Line 111: "...global visual context, can provide additional information that alleviates the ambiguity of homophones"	Please explain how, perhaps with an example?

Line 171 ", which maximizes the cosine similarity between the generated audio features and the given visual features leading  the generated mel-spectrogram to be synced with the input video.” In order to perform the synchronization between visual inputs and audio outputs, the two features mush share a common representation space.  Does this choice improve or impair quality?  	Per Table 1 synchronization gives a very minor win, while the largest improvement comes from the use of visual attention.

Table 2: The order of systems in Table2 is unclear.  Why this sorting?

Table 1: It would be useful to compare the pretrained ASR (5.83%) to any other system.  Without that comparison, this number isn’t meaningful, and probably should be removed.

Line 294: “In addition, for the WER, Lip2Wav is measured by using Google API while VCA-GAN is measured using the ASR model pre-trained on LRW dataset, so they cannot be directly comparable. . However, it clearly shows that the synthesized speech correctly contains the right words even in the challenging environments.”  It is impossible to draw comparisons here, this mismatched evaluation of intelligibility should be removed.  It would be better to either evaluate both using the same system, or leave the comparison out.

It would be helpful to have some subjective measure of the value of synchronized images and audio.




limitations_and_societal_impact: The paper does include a section on Limitations and Societal Impact. While the potential abuse of visual based speech synthesis or recognition is acknowledged, there is no mention of mitigation factors or discussion of the balance between positive and negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this work, the authors study the problem of synthesizing speech corresponding to a silent video. The main novelties proposed in this work are: a proposal to incorporate a “global visual context” which is hypothesized to help the model resolve ambiguity in the viseme-to-phoneme mapping; an explicit synchronization technique to ensure that the generated speech is time-aligned with the video; and the use of multiple discriminators to which allow the iterative generation of more fine-grained speech representations. Essentially, the authors propose to model the task as video to image translation, where the video corresponds to a log-mel spectrogram representation of the target speech.

main_review: 
This is a very well written paper (although it does have a number of minor typographical and grammatical errors; few are listed below) which is generally well motivated and easy to follow. The experimental section, in particular, is comprehensive and demonstrates that the proposed system outperforms previous works with larger gains in unseen and multi-speaker settings (which are more challenging to begin with). 

I had a few minor comments which if resolved would strengthen the paper further.

- **Ablation results in Table 1 and Table 3**: It would be good to include WER numbers as well for the ablation study in Table 1 and the results in Table 3 to show fidelity in addition to intelligibility and quality, since it is also useful to see how well the reconstructed speech matches the ground-truth audio.

- **Captured global context**: A point that is made in the paper, and in Table 1, is that the use of the global context helps resolve ambiguities. It would be interesting to present the results of some error analysis to show how the global context helps improve performance (WER results in Table 1, would help clarify this somewhat as well).

- **Length of test videos**: In Section 4.2, the authors mention that during training the videos are segmented to smaller chunks, but during inference the authors evaluate the full videos. I was curious if the authors have performed any experiments to study the overall performance as a function of video length. Are the global features still useful for very long videos, for example? It would be useful to mention some statistics about the test video lengths for the various datasets (perhaps in a footnote if space is limited).

- **Intelligibility and Quality Metrics**: Minor comment: In section 4.3, it might be useful to mention a couple of sentences about the STOI/ESTOI/PESQ metrics to help the reader understand what these capture, whether higher/lower are better and what level of differences are generally significant.

Typographical and Grammatical Errors:
1. Page 3, Line 98: “... contexture design … ”
2. Page 3, Section 3: “... as the input silence video.” --> “... as the input silent video.”
3. Page 3, Section 3: “... is designed to have 4 times longer ...” -->  “... is designed to be 4 times longer ...”
4. Page 3, Section 3: “... dealing the mel-spectrogram … ” -->  “... treating the mel-spectrogram ...”
5. Eqn 6: It might be better to use the notation L_{g}^{i} instead of L_g
6. Page 6, Line 221: “... four times the frame of the ...” --> “... four times the frame rate of the ...”
7. Page 8, Line 287: “Even the VCA-GAN is ...” --> “Even though the VCA-GAN is ...”

limitations_and_societal_impact: One possible concern is that this work could be used for surveillance when only a video but no audio is available. The authors mention that this is a potential concern.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a lip2speech generative framework based on the proposed Visual Context Attention (VCA) module. The VCA module follows the idea of self-attention and fuses the audio-visual information together to generate finer representations. The proposed pipeline stacks several generators together whose predictions are gradually refined by the VCA module. The global and local clues of input videos are also taken into consideration in this framework. The experiments are conducted on three datasets, while the authors have evaluated their method under different scenarios like constrained-speaker, unseen speaker, etc. The final performance is superior to other baselines.
main_review: STRENGTH
+ The idea of adding an audio-visual attention module to iteratively refine the representation seems to be novel in the lip2speech field. The motivation for designing the VCA module seems reasonable to me.
+ The performance of the proposed method looks promising. Different scenarios, including constrained-speaker, unseen speaker and multiple-speaker are evaluated, which demonstrates the generality of the proposed pipeline. The ablation study is also performed to illustrate the effectiveness of each component.
+ The code is submitted for reproduction. 
+ The paper writing is well organised and easy to follow. 

WEAKNESS
- In VCA module, why use speech representation as the query and use visual clues as the key and value?
- For the pipeline design, why choose the number of generators (n) to 3? How will n affect the whole performance?
- It is difficult for me to understand why the Audio-Visual Attention mechanism can generate better audio representations. Can the authors explain this more detailedly and intuitively?
- The performance of audio-visual recognition models on the generated voice can also be a helpful evaluation metric.
- I checked the demo in the supplemental material,  but the results of various methods (author's vs baselines) doesn't seem too much different to me.  

Overall, the idea of this paper can be novel in the field of lip2speech, and the motivation and model design looks reasonable. The proposed method is evaluated extensively and the results are superior. I will vote for a marginally above. 
limitations_and_societal_impact: Yes, I think so.
review_title: null

Meta-review: The authors investigate speech generation out of silent videos based on the so-called visual context attentional GAN.  The problem is interesting and challenging.  The authors propose to use global visual context to reduce the ambiguity when mapping visemes to phonemes and introduce a synchronization mechanism via contrastive learning to make speech and lip movements in sync.  Extensive experiments and evaluation are conducted to compare its performance with a variety of existing techniques.  The proposed VCA-GAN reports the state-of-the-art performance in the lip-to-speech domain.  The paper is well written and easy to follow.  The experiments are extensive yet controlled.  The rebuttal has clarifies most of the concerns in the review.  All reviewers are supportive on accepting the paper.  The authors should revise the submission to address the concerns (including the ethical concerns) raised by the reviewers.
Decision: Accept (Poster)