Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose to solve the problem of text to speech generation based on video. Thus, this is video-text guided speech generation task that they term as silent video dubbing. The method is based on a transformer architecture that combines text and visual lip motion representations as encoders and outputs the mel-spectrograms through the decoder of a transformer. 

The method is compared with TTS systems and comparisons are done using human evaluation scores and audio-visual synchronisation scores. This is done over LRS2 and Lip2Wav for one out of 5 speakers. Comparisons are not done using standard PESQ, STOI or Word error rates. These are standard metrics used for TTS systems. Comparison is also not provided with single speaker Lip2Wav task or multi-speaker Lip2Wav on LRW dataset as done by Lip2Wav or Vid2Speech, or such methods that generate speech based on lip motion alone (not requiring text). 
main_review: The method aims to solve the task of `silent video dubbing’ where given a video and a text, the aim is to generate the corresponding speech that has the same prosody as the video, thereby enabling the task of dubbing. 

The method is based on a text-video alignment based transformer that additionally uses an image specific embedding with a variance adaptor (used in FastSpeech2) to match pitch and energy for the generated mel-spectrograms generated by the decoder. As is common in transformer architectures, positional encoding blocks are included.  The proposed architecture is modified from existing TTS systems to include lip video embeddings as an additional cue. 


Pros:
The problem aims to address the generation of speech conditioned on text and video as compared to existing methods that aim to generate speech based only on text or generation of speech based only on lip motion. 

The proposed method shows evaluation on LRS2 dataset and evaluation for a single speaker from a speaker-specific dataset. The evaluation is done using MOS scores, evaluation of a speaker specific speech generation visualisation and lip-synchronisation accuracy. The comparisons are done with TTS systems.


Cons:

Evaluation limitations:

The main limitation of this work is that while, the novel task aims to combine the ability to generate speech from text and lip-motion, the comparisons are provided only with TTS systems and not with lip-motion based speech generation systems. Clearly, if the method uses more information as compared to lip-motion based speech generation systems, it would be relevant to compare their output with such systems

The second limitation of this work is that the evaluation measures are obtained through human evaluation and synchronisation, however, the standard metrics used for TTS systems such as PESQ, STOI are not evaluated. This is important as the quality of speech generation is very crucial for dubbing applications (as proposed in the paper). The overall qualitative comparison provided suggests that the actual audio quality is somewhat inferior to TTS systems, though the synchronisation appears to be better.

Similarly, when evaluating on single speaker setting, it would have been easy to provide a comparison with Lip2Wav as the setting is similar (only more information is added). The lip-motion based systems have also been compared using PESQ and STOI metrics appear to provide better audio as compared to the proposed system (quantitative comparison is not possible due to lack of results through the metrics).

This implies that though the proposed method uses more parameters (transformer architecture) and more information, the quality appears to be lesser as compared to lip-motion based generation and text-to-speech based systems. 


Minor writing issues:

While, on the whole the paper is well written and clear, the paper has a few minor grammatical/spelling errors. The line numbers for the same are provided:
38 “an one-to-many”
73 “in a big margin”
108  “quailty gains”
127 “an phoneme sequence”
154 “are suit for capturing”

Conclusion:
To conclude, while the paper proposes an interesting variant, due to the limitations in terms of evaluation, I tend to be negative about this work.


limitations_and_societal_impact: The submission can include possible societal concerns, though its application for creative industries can be useful.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors describe the problem of automatic dubbing of videos, ie generating speech from a script that matches the active speaking face in a video. To the best of this reviewer's knowledge, this is the first paper addressing this particular problem. The two main contributions to solve this problem is 1. a method to synchronize the generated TTS with the speaking face 2. sampling the video for a face image to infer a timbre for the speech to better match the face. 

main_review: The authors have proposed what seems to be a novel problem and presented a solution which they have evaluated. The problem itself is closely related to a variety of work including [14, 31] where TTS is generated from lip video, without a text constraint (Vid2Speech, Lip2Wav). And Large-scale multilingual audio visual dubbing (https://arxiv.org/pdf/2011.03530.pdf) where the video is alternated to match the generated TTS (translated from the original script) [Yang20].

The evaluation is somewhat flawed in that videos with speech in English have the speech audio re-synthesized. In more practical situations, the dubbed speech is required in a language different to the spoken language in a video. The challenge is coming up with a good  synchronization when the phoneme sequences do not match due to the language mismatch. Still the results are fairly impressive (the demo clips in the appendix are much appreciated).

Originality
The work is original,  although somewhat contrived in that you rarely will have the case that the visemes will match the phonemes well; usually the language of viseme will be different than the phonemes in a new dub. It would be interesting to see how Neural Dubber works in this case.

Clarity
Overall, the paper very nicely introduces the problem which they describe as Silent Video Dubbing.  Perhaps a more accurate term would be Automatic Video Dubbing, since the notion of dubbing already means that the audio track is being replaced with another within the video. SVD is already a common term within the machine learning community, and AVD follows from other similar terms like ASR for automatic speech recognition. The term "silent video" is also awkward; while video can mean media that has both video and audio in the common sense, technically it usually denotes just visual information (https://en.wikipedia.org/wiki/Video) as opposed to audio-visual data. If we don't qualify that video is only visual information, then terms like "Video-based" Tacotron, should be called "Silent video-based" Tacotron, which is strange. In section 4.1 only evaluation datasets are mentioned. Later, some mentions of training data are provided. It would be helpful to include them in section 4.1 to understand what actually needs to be trained and on what corpora.

Quality
The paper clearly motivates this Automatic Video Dubbing problem, presents a solution to guiding the TTS generation to better match the speaking face, and provides a comprehensive analysis of the results with quantitate measurement using AV Sync and qualitative (human ratings). They attempt to provide a good competing baseline system, Video-base Tacotron (although result don't seem so good as the spectrograms don't synchronize well), but the reviewer appreciates the effort. There are results for single speaker and a variety of speakers. The ISE system, is helpful to generate voices that match the speaking faces gender or age, but perhaps can make unfair unbiases on what people should sound like based on their appearance. Still, the paper is comprehensive, and by providing the source code in the appendix, and operating on publicly available data sets, gives experimenters the best chance of being able to reproduce results.

References
The author's have an odd referencing style. There is no need to reference multiple papers for a particular technique, only the first is required. e.g. if [35] is the first example of non-autoregressive TTS, then there is no need to cite [34 ] and [21] as well. Further, [34] is a follow on to [35] (FastSpeech), so unless the paper is citing some thing new in [34], it is better to only cite [35] for anything related to FastSpeech (many times both citations appear together). 

Significance
The only downside to the paper, is how the Neural Dubber has been evaluated since re-generating English speech for a video of a person speaking English is a somewhat academic as mentioned earlier (if you already have the English speech, why do you need to re-generate it?). It would be interesting to see how the system would behave would the target phonemes mismatch the viseme language. Unfortunately, this is not answered in this paper. Thus work like in Yang20, is more relevant to the AVD task in practice. The capitalization on the references need to be fixed, e.g. t-sne->T-SNE, FastSpeech, tts -> TTS. Skerry is misspelled in line 449. Please go over the references to make sure they are accurately written.

Minor comments:

line 77: these are not the first papers on TTS. please update.

Figure 1 seems to show that the SVD system doesn't use the green/blue face tracks since they bypass the box, but it fact it does (subtley the video is passed in via the black line). Maybe this could be clearer.

Figure 2. would be helpful to add the labels for S_p, S_n, S_v etc.

line 111: this set of references seemed biased/odd; this is not the complete set of papers evaluating lip reading. it makes more sense to use one that is the most recent state-of-the-art.

line 126-128. the notation is little non-standard. typically lower case letter for scalar, bold lower case for vector and bold upper case for matrix. Thus the phone sequence is \bm{p} = {p_1, p_2 ... and the subscript can mean an index instead of being also overloaded as a indicator of what it is in S_p.

line 150: add that the FFT is simply a transformer with the dense layer replaced with conv. It's an odd/confusing name since the transformer is already feed-forward and FFT usually refers to fast Fourier transform.

Equation (1). the dimensionality of H_vid and H_pho is different, how is this addressed?

Equation (4), the terms are a mixture of quantities e.g. hop size, same rate and units e.g. frame per second. They should all be some quantity e.g. frame rate. 

typo line 220

add units in line 239.

Perhaps the Video-based Tacotron would preform better with a constrained alignment process.


limitations_and_societal_impact: The authors might discuss how using a face to influence what the speaking voice sounds like may have some negative impacts or introduce unwelcome bias. What would happen if a female speaker with masculine features were used: would producing a male-sounding voice be offensive? How would race play a role in the sound of the voice?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper describes a novel task 'silent video dubbing' and proposes a method to solve it. The task involves generating realistic audio to accompany silent video, with the help of text. This is essentially similar to vid2speech [14], but with text guidance. The pipeline consists of several stages. (1) generating phoneme and video embeddings; (2) aligning text and video using the embeddings from (1) and attention network; (3) generating speaker embeddings to condition the decoder from images; (4) using (2) and (3) to generate the result voice. The authors provide example in supplementary materials. The experiments are performed on both single-speaker and multi-speaker SVD.
main_review: The task of dubbing silent videos using neural networks is novel to my knowledge. This is an interesting problem in film-making industry. 

The pipeline well-engineered, and is well explained. The individual components in the pipeline are fairly standard, rather than novel contributions. However, credit can still be given to how these are put together.

The authors imply that ISE is a contribution of their own, but there are many previous works to learn joint speaker-face embeddings [1,2,3].

I do not think that LSE-D is a fair metric, since this is too similar to the way in which text-video aligner is trained using attention alignment, therefore this unfairly biases towards the proposed method.

Otherwise, qualitative and quantitative results appear to be good.


[1] https://arxiv.org/abs/1805.00833
[2] https://people.csail.mit.edu/changil/assets/face-voice-accv-2018.pdf
[3] https://arxiv.org/abs/2004.14326
limitations_and_societal_impact: The authors have not addressed this in any detail. They can mention the usual risks associated with deepfake-like research.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents "Neural Dubber", a method for dubbing silent videos based on given scripts. It essentially performs lip-synced TTS, with an added component (ISE) that ensures that the generated speech characteristics match the given face image. 

Experiments verify that the proposed TTS method does not hurt SOTA TTS performance, while controlling the prosody such that the aligned speech is much more synchronized than other methods compared to. The supplementary video qualitatively demonstrates some compelling results of the proposed method.
main_review: **Originality:**

The task of performing end-to-end lip-synced TTS is, as far as I am aware of, a novel one. The seemingly effective method used to solve this task is a well-motivated combination of components well-documented in the literature: per-modality feature extraction and processing using transformers --> attention-based alignment --> joint decoding. 

The main weakness of this work is its conspicuously-missing reference to a very related work: "Dynamic Temporal Alignment of Speech to Lips", Halperin et al. (ICASSP'19), which, given an unaligned face sequence and audio track, automatically stretches and compresses the audio signal to match the lip movement. Using this method to provide a stronger baseline to compare Neural Dubbing against would help amplify the value of the proposed method, especially since the included baseline (video-based Tacotron) is  weak and uninformative.

**Quality:** 

As stated above, the method is well-motivated, and experiments verify that:
The proposed TTS method does not hurt SOTA TTS performance, as measured by listening study, while controlling the prosody such that the aligned speech is much more synchronized than other methods compared to, also subjectively measured. Objective SyncNet-based metrics also demonstrate superiority over baselines. The supplementary video qualitatively demonstrates some compelling results of the proposed method.

That said, a stronger baseline, as suggested above, would significantly strengthen these claims. Also, in the checklist the authors answered that they did describe the limitations of their work, however, I can't seem to find where this is mentioned. Another experiment that I'm curious about is: how well does this method work when the script is significantly different from what the person is actually saying, e.g., a sentence in another language, or just different words in the same language? 

**Clarity:**

The manuscript is clear, and seems to contain enough information to reproduce results. One correction I spotted:
l.154 are suit --> are suitable

**Significance:**

The significance of this work is in the fact that it presents and provides a solution for a novel problem, which is interesting from both a research perspective and from the real-world application viewpoint.

limitations_and_societal_impact: IMO this aspect of the paper is missing. Discussing limitations will strengthen the manuscript, not weaken it. In addition, the proposed method could potentially be used for generating fake talking videos, but this depends on how well the method works on scripts different from what is actually being said by the speaker in the video (I assume this will not work well enough to deceive people).
review_title: null

Meta-review: The original submission was missing an essential comparison with Lip2Wav. This has been added in the rebuttal period, and all reviewers now recommend acceptance.

The authors should include the Lip2Wav comparison and the missing references highlighted by the reviewers in the final version. 
Decision: Accept (Poster)