LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading
Authors: Yochai Yemini
Aviv Shamsian
Lior Bracha
Sharon Gannot
Ethan Fetaya
Venue: ICLR 2024 poster (2024)
Keywords: lip-to-speech
lip-reading
diffusion models
Primary area: generative models
TL;DR: We present LipVoicer, a novel method that by incorporating the text modality generates high quality speech even for in-the-wild and rich datasets. Given a silent video, we first predict the spoken text using a pretrained lip-reading network.

Abstract: Lip-to-speech involves generating a natural-sounding speech synchronized with a soundless video of a person talking. Despite recent advances, current methods still cannot produce high-quality speech with high levels of intelligibility for challenging and realistic datasets such as LRS3. In this work, we present LipVoicer, a novel method that generates high-quality speech, even for in-the-wild and rich datasets, by incorporating the text modality. Given a silent video, we first predict the spoken text using a pre-trained lip-reading network. We then condition a diffusion model on the video and use the extracted text through a classifier-guidance mechanism where a pre-trained automatic speech recognition (ASR ) serves as the classifier. LipVoicer outperforms multiple lip-to-speech baselines on LRS2 and LRS3, which are in-the-wild datasets with hundreds of unique speakers in their test set and an unrestricted vocabulary. Moreover, our experiments show that the inclusion of the text modality plays a major role in the intelligibility of the produced speech, readily perceptible while listening, and is empirically reflected in the substantial reduction of the word error rate ( WER ) metric. We demonstrate the effectiveness of LipVoicer through human evaluation, which shows that it produces more natural and synchronized speech signals compared to competing methods. Finally, we created a demo showcasing LipVoicer’s superiority in producing natural, synchronized, and intelligible speech, providing additional evidence of its effectiveness. Project page and code: https://github.com/yochaiye/LipVoicer

Published as a conference paper at ICLR 2024
LIPVOIEER: GENERATING SPEECH FROM SILENT VIDEOS GUIDED BY LIP READING
Yochai Yemini, Aviv Shamsian, Lior Bracha, Sharon Gannot, Ethan Fetaya Faculty of Electrical Engineering Bar-Ilan University
Israel (yochai. yemini aviv. wwr..shhmmsaan hra.aa/i....rracla.sssaooo.oaano fetaya) (bi.aa)bbuuccco ac
ABSTRACT
Lip-to-speech involves generating a natural-sounding speech synchronized with a soundless video of person talking. Despite recent advances, current methods still cannot produce high-quality speech with high levels of intelligibility for challenging and realistic datasets such LRS3. In this work, present LipVoicer, a novel method that generates high-quality speech, even for in-the-wild and rich datasets, by incorporating the text modality. Given silent video, first predict the spoken text using a pre-trained lip-reading network. We then condition a diffusion model the video and use the extracted text through a classifier- Dlassifier-guidance guidance mechanism where pre-trained automatic speech recognition (ASR) serves as the classifier. Lip Voicer outperforms multiple lip-to-speech baselines on LRS2 and LRS3, which are in- in-the-wild datasets with hundreds of unique speakers in their test set and an unrestricted vocabulary. Moreover, our experiments show that the inclusion of the text modality plays major role in the intelligibility of the produced speech, readily perceptible while listening, and is empirically reflected in the substantial reduction of the word error rate (WER) metric. We demonstrate the effectiveness of Viiee through human evaluation, which shows that produces more natural and synchronized speech signals compared to competing methods. Finally, we created a demo showcasing Voice'sss's 's superiority and intelligible speech, providing additional evidence of its effectiveness. Project page and code: https:: /ttps://oe / ///////// liggithuus thub. httpuu/cooooooccaaeee//////u
producing natural, synchronized,
INTRODUCTION
In the lip-to-speech task, we are given a soundless video a person talking and are required to accurately and precisely generate the missing speech. Such task may occur, e.g.. when the speech signal is completely obfuscated due to background noises. This task poses a significant challenge as it requires the generated speech satisfy multiple criteria. This includes intelligibility, synchronization with lip motion, naturalness, and alignment with the speaker's 's characteristics such as age, gender, accent, and more. Another major hurdle for lip-to-speech techniques is the ambiguities inherent in lip motion, as several phonemes can be attributed to the same lip movement sequence. Resolving these ambiguities requires the analysis of lip motion in a broader context within the video.
Generating speech from a silent video has seen significant progress in recent years, partly due to advancements made in deep generative models. Specifically in applications such as rext-to-speech and poees-ppoogram-to-uueeee advancements, many lip-to-speech methods produce satisfying results only when applied to datasets with limited number of speakers, and constrained vocabularies, like GRID (Cooke et al.) (2066) and TCD-TIMIT (Harte & Gillen| 2015). Therefore, speech generation for silent videos tetthe-wild still lags behind. We found that these methods struggle to reliably generate natural speech with a high degree of intelligibility on more challenging datasets like LRS2 (Afouras et al. 2018a) and LRS3 (Afouras et al. 20188).
(neural vocoder) (Kong et al.) 2021 Kim et al. (022). Despite these
In this paper, we propose Voiver,,r, a novel approach for producing high-quality speech for silent videos. The first and crucial part of Voicer is leveraging a lip-reading model at inference time,

---

Published as a conference paper at ICLR 2024
Notse Prodiction
m2
Denoleed Nal-spactogam
Notey
X++1
X
ASR
Slient Lpp Video
Classifier
.n.iig hanper degnes and
Guidance
Lip Reader
wel rereet job
MelGen
ResNet-18
replicate
concatenate
.o.comy
DiffWave
Backbone
de
Cmg
Silent Up Video
(b)
Figure 1: An illustration of Viicer, a dual-stage framework for lip-to-speech. (a) To generate the speech from a given silent video at inference time, a pre-trained lip-reader provides additional guidance by predicting the text from the video. An ASR steers MelGen, which generates the mel- spectrogram, in the direction of the extracted text using classifier guidance, such that the generated mel-spectrogram reflects the spoken text. (b) MelGen, diffusion denoising model that generates mel-spectrograms conditioned on a face image and mouth region video extracted from the full-face video using classifier-free guidance. It is trained without the text guidance.
for extracting the transcription of the speech we wish to generate. Next, we train a diffusion model, conditioned only on the video, generate mel-spectrograms. The generation process at inference time is guided by both the video and the predicted transcription. Consequently, our model successfully intertwines the information conveyed by textual modality with the dynamics and characteristics of the speaker, captured by the diffusion model. Incorporating the inferred text has an additional benefit, the
it allows Lip Voicer alleviate the lip motion ambiguity to a great extent. Finally, DiffWave (Kong et al. 2021) neural vocoder to generate the raw audio. A diagram of with all of the components our approach is depicted in Fig. (except the vocoder). Some previous methods often use text to guide the generation process at train time. We, however, utilize it at inference time. The text, transcribed using a lip-reader, allows to utilize guidance (Dhariwal Nichol, 2021 Ho Salimans 2021) which ensures that the text of the generated audio corresponds to the target text.
We evaluate our Lip Voicer model on the challenging LRS2 and LRS3 datasets. These datasets are "in-the-wild videos, with hundreds of unique speakers and with an open vocabulary. We show that our proposed design leads to the best results on these datasets in both human evaluations as well as WER of an ASR system.
To the best of our knowledge, Lipvei is the first method to use text inferred by lip-reading to enhance lip-to-speech synthesis. The inclusion of the text modality in inference removes the uncertainty of deciphering which of the possible candidate phonemes correspond to the lip motion. Additionally, it helps the diffusion model to focus creating naturally synced speech. The speech generated by Lip Voicer is intelligible, well synchronized to the video, and sounds natural. Finally, Lip achieves state-of-the-art results for highly challenging in-the-win datasets.
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND
2.1 DENOISING DIFFUSION PROBABILISTIC MODELS (DDPM)
DDPM define a forward process that gradually turns the input into Gaussian noise, then learn the reverse process that tries to recover the input. Specifically, assume training data point is sampled from the data distribution we wish to model Xo Paata (x). The forward process is defined as ([x][xt-1) = N
- 3,I) where {B...... St.... 3T is pre-defined noise schedule. We can deduce from Gaussian properties that q(x, xn) = =N(x, Ially Vā, (1-ā,)] with ā₂ = I - Oa. and at = 1 3e. A sample of x2 can be obtained by sampling using the reparametrization trick, X = Vā + vāa Under mild conditions, the distribution at the final step (xx)) is approximately given by standard Gaussian distribution.
(O, and
In the reverse process proposed in Ho et al. (2020). Po [x-1 |xx) is then learned by a neural network that tries to approximate 9(x,-1 |x,, X In Ho et al. (2020) was also shown that in order to learn
p it is enough if our model's output E Ex t) is trained to recover the added noise e used to generate x from Xu- The loss function used to train the diffusion model is Ex,e. ||]c cx t) {) At inference time, given X and the inferred noise, we can sample from po po (x1-1 (2x) by taking
Xp-1
X
Vi-ai co (xe:)) + 3,z where z (O,
2.2 GUIDANCE
One key feature in many diffusion models is the use of guidance for conditional generation. Guidance, both with without classifier, enables us to "guide" iterative inference process to generate outputs that are more faithful to our conditioning information, e.g., in text-to-image, it helps enforce that the generated images match the prompt text.
Assume we wish to sample from (xx, cc) x, is our sample at the current iteration, c is some context, and p(c[x,) is pre-trained classifier. Our goal is generate x,- The idea of classifier guidance DDariiwl & Nichol (202) is to use the classifier to guide the diffusion process to generate outputs that have the right context c. Specifically, if the diffusion model returns eø (xx, t). the classifier guidance alters the noise term that will be used for the update to é eØ (X,, t) W wivII= - ā, Vx, log p(c(x)) where W1 a hyperparameter that controls the level of guidance.
In a later work (Ho & Salimans, 2021). classifier-fre free guidance that removes the dependence on an existing classifier is proposed. In classifier-free guidance, we make two noise predictions, one with the conditioning context information, 6o c, (). and one without it ea(x,, l). We then use = tØ (x,,, c, l) + (2(eo(X,, c, l) eo (xe, where the hyperparameter W2 controls the guidance strength. This allows us to enhance the update directions that correspond to the context c.
RELATED WORK
3.1 AUDIO GENERATION
Following their success in image generation, diffusion models took a leading role as a neural vocoder
and in text-to-speech. Examples include:
WaveGrad (Chen al. 2021), Diff-TTS (Jeong al.
2021), DiffWave (Kong et al. 2021). Grad-TTS (Popov et al. 2021), PriorGrad (Lee et al. 2021).
SaShiMi (Goel et al. [2022), Tango (Ghosal al. 2023), and Diffsound (Yang et al.) (023).
Other important audio generation models that do not rely on diffusion models include: WaveNet
(van den Oord et al.] 2016), HiFi-GAN (Su et al.) 2020), Tacotron (Wang et al.] 2017). VoiceLoop
TTigman et al. 2018). WaveRNN (Kalchbrenner et al. 2018). ClariNet (Ping al. 2018). GAN-TTS
(Binkowski et al. 2019), Flowtron (Valle et al. 2021). ), and AudioLM (Borsos et al. (022).
We note that despite the recent progress, unconditional speech generation quality is still unsatisfactory. Thus, some conditioning, e.g., text or mel-sppctrogram is required for high-quality speech generation. We also note that many audio generation models, and particularly tip-to-speech techniques, adopt sequential approach. Namely, first generating the mel-spectrogram and then using it to generate the raw audio using pre-trained vocoder.
3

---

Published as a conference paper at ICLR 2024
3.2 VISUAL SPEECH RECOGNITION
The field of lip-to-text, also referred to as lip-reading, has seen significant progress in recent years, and multiple techniques are now available (Ma et al., 2022: Shi et al. 2022 Ma al. 2023). Impressively, they can cope with adverse visual conditions, such as videos where the mouth of the speaker is only partly frontal, challenging lighting conditions and various languages. Use cases of visual speech recognition include resolving multi-talker simultaneous speech and transcribing archival silent films. In our context, using powerful lip-reading system can help guide the lip-to-speech process.
3.3 LIP-TO-SPEECH SYNTHESIS
The task in this research area is to reconstruct the underlying speech signal from a silent talking-fase face video. common approach to lip-to-speech includes a video encoder followed by a speech decoder that generates a mel-spectrogram, that is fed to neural vocoder to generate the final time-domain audio. These techniques have garnered significant attention due to their potential applications in cases where the speech is missing corrupted. This may occur, for example, due to strong background noise, or when the speech of a person located in the background of the video recording should be attended.
Preliminary studies such as Lip2Wav (Prajwal et al. 2020). End-to-end GAN (Mira et al., 2022).
VCA-GAN (Kim et al.] 2021) focused on datasetes with limited vocabularies and number of speakers. Subsequent works such as SVTS (de Mira et al. 2022) addressed more realistic in-the-wild datasets. Recent techniques tremendously improve performance by using AV-HuBERT (Shi et al. 2022) to represent speech by set of discrete tokens, which are predicted from the video. In RISI (Hsu et al. 2022), the authors use AV-HuBERT architecture to generate the audio from the tokens using Hi-i-GAN. Concurrently to our work, speech units are also used in Choi et al. (203bb) Also concurrent to Voicer, diffusion frassedio--bseed method is proposed in Choi et (2023a)
In Lip2Speech (Kim et al., 2023), the authors use the ground truth text and pre-trained ASR as an additional loss during training to try enforce that the generated speech will have the correct text. We, however, use predicted text at inference time.
VIPCER
This section details our proposed LipVoicer scheme for lip-to-speech generation. Given silent talking-face video V. Voicer generates mel-spectrogram that corresponds to a high likelihood underlying speech signal. The proposed method comprises three main components:
1. A mel-spectrogram generator (MelGen) that learns to create a mel-spectrogram image from V.
2. A pre-trained lip-reading network that infers the most likely text from the silent video.
3. An ASR system that anchors the mel-spectrogram recovered by MelGen to the text predicted by the lip-reader.
At first, we train MelGen, a conditional denoising diffusion probabilistic models (DDPM) model trained to generate a mel-spectrogram waveform conditioned on the video V without the text modality. We use classifier-free guidance to train MelGen (see Section 2.2 Similar to diffusion- based frameworks in text-to-speech, e.g. Jeong et al. (2021). we use a DiftWave (Kong al. 2021) residual backbone for MelGen. When considering the representation for V, we wish for our representation to encapsulate all the needed information to generate the mel-spectrogram, i.e. the content (spoken words) and dynamics (accent, intonation) of the underlying speech, the timing of each part of speech, as well as the identity of the speaker, e.g. gender, age, etc. However, we wish to remove all irrelevant information to help train and remove unnecessary computational costs. To this end, V is preprocessed by creating a cropped mouth region video V1 and randomly choosing a single full-face image Ip. Notably, V corresponds the content and dynamics, and Ip relates the speaker characteristics. The mouth cropping was implemented according to the procedure in Ma et al.) (2022).
To extract features from Vr, and Ir. we use an architecture similar to the one described in VisualVoice (Gao & Grauman 202T), audio-visual speech separation model. For Ir, the face embedding
4

---

Published as a conference paper at ICLR 2024
f E R, is computed using ResNet-18 18 (He et al. 2016) with the last two layers discarded. The lip video VL is encoded using a lip-reading architecture (Ma et al., 2021]. is composed of a 3D convolutional layer followed by ShuffleNet v2 (Ma et al., 2018) and then temporal convolutional network (TCN), resulting in the lip video embedding m E R where N and D. signify the number of frames and channels, respectively. In order merge the face and lip video embeddings, f is replicated N times and concatenated to m, yielding the video embedding E I xxxD, where D D Dm. Next, DDPM trained generate the mel-spectroggam with and without the conditioning on the video embedding following the classifier- frree mechanism (Ho & Salimans 2021
In order to make MelGen perform well in scenarios characterized by an unconstrained vocabulary, at inference time we use the text modality as additional source of guidance. In general, syllables uttered a silent talking-face video can be ambiguous, and may consequently lead to an incoherent reconstructed speech. It can therefore be beneficial to harness recent advances in lip-reading and ground the generated mel-spectrogram to the text predicted by a pretrained lip-reading network. The question is how to best include this textual information in our framework. One could simply add it as a global conditioning, similar to If; however, this ignores the temporal information in the text. One could also try align the text and the video, which complicated process that would introduce additional errors.
To circumvent the challenge of aligning text with video content, we employ text guidance by harnessing the classifier guidance approach (Dhariwal & Nichol 2021), similarly to Kim et al. (2022). Using a powerful ASR model, we can compute V. x log p(tER x) needed for guidance, where f R is the text predicted by lip-reader. The inferred noise é used in the inference update step of the
diffusion model is thus modified by both classifier guidance and classifier free guidance:
= cmg (X,, VL, II.(rr - W2 I - ā, Vx, log p(t..n(x,)
(1)
where is the mel-spectrogram at time step of the diffusion inference process, and 6mg (x,, Vz, IF, w1) (1 wi)co(X;, IF, w1cg(x;)
the estimated diffusion noise at the output of MelGen, and W1, w2 are hyperparameters. Note that we use ASR rather than audio-video ASR, namely x, is used as an input to the speech recognizer while the video signal is discarded in order to encourage the model to focus on audio generation.
In our experiments, noticed that during the generation process, Emg was much larger in magnitude compared to Vx, log ((t))(x)) which led to difficulties in correctly setting W2 and to unsatisfactory mel-spectrogram estimates. To remedy this, we followed Kim et al. (0022) by introducing gradient normalization factor, i.e. Eq. becomes
é Emg
W27VV ae Vx, logp(tzR|x.)
(3)
where
Emg
Tt. VI ā.
V logp(tLR|xx)ll
and || the Frobenius norm. The inference process is summarized in Algorithm in the Appendix.
Classifier guidance allows us to train MelGen that is solely conditioned on V, and use a pre-trained ASR to make the generated speech match ÈLR- As result, the ASR is responsible for the precise words in the estimated speech, and MelGen provides the voice characteristics, synchronization between V and and the continuity of the speech. One additional advantage of this approach is the modularity and ease of substituting both the lip-to-text and the ASR modules. If one wishes substitute these models with improved versions the future, the process can be accomplished effortlessly without requiring any re-training, Finally, a DiffWave vocoder (Kong et al., 202T is used transform the reconstructed mel-spectrogram to time-domain speech signal. Note that the vocoder does not appear in Fig.
5 EXPERIMENTS
In this section, we compare Voicer to various lip-to-speech approaches. We use multiple datasets and learning setups evaluate Lip Voicer. To encouraee future research and reproducibility. source
5

---

Published as a conference paper at ICLR 2024
code will be made publicly available. Additional ablation studies, experimental results GRID, and complete details are provided in the Appendix. We also created the following website https //lipvoicer tegggithhhu thub. i containing sample videos generated by all compared approaches. We believe this is the best way to fully understand the performance gain of Lip and highly encourage the readers visit.
Datasets LipVoicer is compared against the baselines on the highly challenging datasets LRS2 (Afouras et al. 2018a) and LRS3 (Afouras et al. (0T8b). LRS2 contains roughly 142,000 videos of British English in its train and pre-train splits, which amounts to 220 hours of speech by various speakers. In the test set, there are 1,243 videos. The train and pre-train sets of LRS3 comprise 9,000 different speakers, contributing 151,000 videos which are stretched across 430 hours of speech videos. There are 1,452 videos in the test split. The language spoken in LRS3 videos is English but with different accents, including non-native ones. We specifically select these in-the-wild wiid datasets, LRS2 and LRS3, for their diverse range of real-world scenarios with variations in lighting conditions, speaker characteristics, speaking styles, and speaker-camera alignment. We train LipVoicer using the pretrain+train splits of LRS2 and LRS3 on each dataset separately, and evaluation is carried out on the full unseen test data splits. Note that both datasets also include transcriptions, but we do not use them for either training or inference.
Implementation Details For predicting the text from the silent video at inference time, we use Ma et al. )223)) as our lip-reader for LRS2 and LRS3. It achieves a WER rate of 14.6% and 19.1% with respect to the ground truth text, respectively. The ASR deployed for applying classifier guidance was
Burchi & Timofte 2023). We modified its architecture by adding the diffusion time step embedding to the conformer blocks and then fine-tuned on LRS2 and LRS3. Finally, for the sake of fairness, we employ different ASR (Ma et al.) (023) evaluate the WER of the speech generated by our method and the baselines. We notice some variability in the choice of the ASRs used for benchmarking among the baselines, which make it difficult put the different methods on common ground. We believe that Ma et al. (2023) can serve as good candidate to be used in future studies, since it achieves state of the art results on LRS2 (1.5%) and LRS3 (1%) and a pre-trained model is publicly available. The rest of the implementation details for reproducing our experiments can be found in the Appendix.
Baselines We compare VipVoicer with recent lip-to-speedt synthesis baselines. The baseline methods include (1) SVTS (de Mira et al. 2022), a transformer-based based video to mel-spectrogram generator with pre-trained neural vocoder. (2) VCA-GAN (Kim et al. 2021), GAN model with visual context attention module that encodes global representations from local visual features. (3) Lip2Speech (Kim et al. 2023), a multi-task learning framework that incorporates ground truth text to form an additional loss to entorce the correspondence between the text predicted from the generated speech and the target text at train time. We trained VCA-GAN and Lip2Speech on LRS2 and LRS3, and used the LRS3 test files provided by the authors of SVTS to conduct the comparisons.
Unfortunately, we could not include ReVISE (Hsu et al. 2022 in our full comparisons since the test files are unavailable for public access and training the method is too computationally heavy (32 GPUs were reported in ReVISE). Nevertheless, we used the ASR utilized in Hsu et al, (2022) to compare it to Voicer Notably, in Hsu et al. (2022) it was reported that the ASR achieved WER score of 5.6% on the ground-trui -ruth test videos of LRS3, but we only managed to achieve 6.4% on the same data. While we are uncertain of the source of the degraded performance, we evaluated Lip Voicer using this ASR and achieved WER score of 33.2% on LRS3, whereas RISE reported 33.9%. Additionally,
generated with Lip the speech signals for the silent videos the project page of Re RIVE and incorporated the results in our demo website. We note that the speech units computed by AV-HuBERT are primarily learned in such a way that encourages speech continuity, and not any other explicit speech aspects (WER, intelligibility, etc.). We believe that this is the reason why ReV VISE still has relatively high WER, although much lower than its previous competitors.
Several metrics are used to evaluate the quality and intelligibility of our generated speech and compare it to the baseline methods, As the main goal of speech generation is to create a speech signal that sounds natural and intelligible to human listeners, our focus is on human evaluation measured by the mean opinion score (MOS). We also compare WER using a speech recognition model. Following Hsu et al. (2022): Hassid et al. (2021), we measure the synchronization between
Metries

---

Published as a conference paper at ICLR 2024
Intelligibility
Naturalness
Quality
Synchronization
4.33 + 0.04
4.43 + 0.04
4.34 0.04
4.39 0.04
LIP2SPEECH (Kim et al.
2023)
2.07 = 0.08
1.98 = 0.08
1.93 = 0.08
2.66 0.10
VCA-GAN Kim al.
2021
1.77 = 0.08
1.85 = 0.09
1.77 = 0.08
2.34 + 0.09
LIPVOICER (OuRS)
3.53 + 0.07
3.54 0.08
3.69 + 0.08
3.82 + 0.07
Table 1: LRS2 Human evaluation (MOS).
Intelligibility
Naturalness
Quality
Synchronization
GT
4.38 + 0.03
4.45 + 0.03
4.42 + 0.03
4.36 + 0.03
LIP2SPEECH (Kim et 2023
2.21 0.08
2.20 + 0.09
2.01 + 0.07
2.69 + 0.08
SVTS (de Mira et al. 2022
2.17 = 0.08
2.15 + 0.09
1.99 0.07
2.71 + 0.09
VCA-GAN Kim al. 2021
2.19 + 0.08
2.20 + 0.09
2.08 + 0.08
2.71 = 0.08
LIPVOICER (OurS)
3,44 0.07
3.52 + 0.07
3.42 + 0.08
3.56 + 0.07
Table LRS3 Human evaluation (MOS).
the generated speech and the matching video with SyncNet (Chung & Zisserman, 2016) scores. Specifically, we report the temporal distance between audio and video (LSE-D) and the confidence scores (LSE-C). We use the pre-trained SyncNet mode?"to calculate LSE-C and LSE-D.
To objectively quantify the quality and intelligibility of our scheme, we use DNSMOS (Reddy et al. 2022) and STOI-Net (Zezario et al.) 2020). It is worth mentioning that previous studies lip-to-speech synthesis have presented metrics that we believe are unsuitable for this task, and as result, refrained from including them in main report. Specifically, they short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ). Both metrics are intrusive, namely, they are based comparison with the clean raw audio signal. While they valuable for speech enhancement and speaker separation, in speech generation, it is not expected, even for perfect model, to recreate the original audio. This may stem, for instance, from variation of the pitch between the original speech and the reconstructed one (Sisman et al. 2020). Our goal is to generate speech signal that matches the video, and not the original speech. In Appendix [ we provide example that demonstrates the effectiveness of non-intrusive over intrusive metrics with regard to the lip-to-speech setup. In this sense, our problem more related to voice conversion. For completeness, the Appendix also includes the PESQ and STOI scores of Lip Voicer.
5.1 HUMAN EVALUATION RESULTS
Given 50 random samples from the test splits of LRS2 and LRS3 datasets (each), we used Amazon Mechanical Turk to rate the different approaches and the ground truth. The listeners were asked to rate the videos, on scale of 1-5, for Intelligibilitt, Naturalness, Quality, and Synchronization. We note that we observed nen-nggli amount of noise in the scores of the human evaluators. We ameliorated this effect by having a large number (16) distinct annotators per sample, and by having each annotator score all methods on specific sample. We also filtered out Turkers who rated the ground -ruth video with a score smaller than 4. This ensures that all methods compared are annotated by the exact same reliable set of annotators. We also note that for the SVTS (de Mira et al. 2022) baseline comparison, we used generated videos that were sent to us by the authors. As only LRS3 videos available, evaluate SVTS only LRS3.
Tables D and depict the MOS results on the LRS2 and LRS3 datasets. Voicer outperforms all three baseline methods: Lip2Speech, SVTS, and VCA-GAN. Not only that, is also remarkably close to the ground truth scores. The questionnaire, screenshots of the task, and other implementation details are given Appendix G
7

---

Published as a conference paper at ICLR 2024
WER +
S
DNSMOS
LSE-C
LSE-D J
GT
1.5%
0.91
3.14
6.840
7.194
LIP2SPEECH
51.4%
0.70
2.37
6,815
7.370
VCA-GAN
100.7%
0.51
2.26
3.369
10.703
LIPVOICER (OURS)
17.8% %
0.91
2.89
6.600
7.840
Table 3: Performance comparison between Lip Voicer and the baselines on LRS2.
WER +
STOI-Net t
DNSMOS 1
LSE-C t
LSE-D
GT
1.0%
0.93
3.30
6.880
7.638
LIP2SPRECH
57.4%
0.67
2.36
5.231
8.832
SVTS
82.4%
0.65
2.42
6.018
8,290
VCA-GAN
90.6%
0.63
2.27
5.255
8.913
LIPVOICER (OURS)
21.4%
0.92
3.11
6.239
8.266
Table 4: Performance comparison between Lip Voicer and the baselines on LRS3.
5.2 ORJECTIVE EVALUATION RESULTS
We further evaluated our method with the objective metrics. For SVTS, we report WER and synchronization metries only for LRS3, since the authors did not open-source their code and only released the generated test files for LRS3. From the WER scores, it is clear that method significantly improves over competing baselines. From the STOI-Net and DNSMOS metrics is apparent that Voicer generates much competitors. In addition to generating high-quality content, Lip Voicer demonstrates commendable synchronization scores, ensuring that the generated speech aligns seamlessly with the accompanying video. Qualitative results may be found in the Appendix.
intelligible and higher quality speech compared the
5.3 ABLATION STUDY
The architecture of LipVeicer requires several design choices: the values of Wi, network and the ASR used for guidance. We conducted ablation studies examining the influence of the hyperparameters and the modules and evaluated it on LRS3. Tables (oolesSaandjy show the performance of Lip Voicer for different values of wi and W2, respectively. Several conclusions
W1, W2, the lip reading
be drawn from the results. We observe (w) = -1) 1) that classifier-free guidance has a decisive role in terms of WER, speech quality, and synchronization. Decreasing the weight of the ASR increases the WER, as expected, while the generated speech still sounds natural and synchronized. It is also clear that ASR guidance vital, without it (w, 0) the WER plunges from 21.4% 86.2% on LRS3. Unsurprisingly, an excessive increase in the ASR weight only slightly degrades the WER, but is detrimental the speech quality and synchronization. Interestingly, setting W2 leads to better synchronization compared to w2 = 0 (no ASR guidance). We postulate that this occurs since the predicted text also helps align the generated speech with the video.
we 1.5
With regard to the lip-reader choice, we see in Table 7 that when the ground-truth text is used, the best results are achieved. less accurate lip-reader leads to worse WER but not speech quality, Synchronization is also impaired with worse lip-reading, probably due to the discrepancy between the lip motion and the predicted text. This ablation study means that by using a more powerful lip-readers, the performance of Lip Voicer can be further improved. Ablation studies for the ASR performance, vocoder choice, using the face embedding and the amount training data can be found in Appendix E
8

---

Published as a conference paper at ICLR 2024
W1
WER
STOI-Net
DNSMOS 1 LSE-C
LSE-D /
30.1%
0.92
2.96
2.543
11.729
1
22.7%
0.93
3.09
5.512
8.885
2
21.4%
0.92
3.11
6.239
8,266
3
22.4%
0.92
3.07
6.435
7.967
4
22.2%
0.92
3.03
4.608
9.636
Table 5: Ablation study for W1 evaluated LRS3. The row marked in bold indicates the value of used by Lip Voicer in the experimental study for LRS3.
U2
WER
STOI-Net
DNSMOS
LSE-C
LSE-D 1
0
86.2%
0.93
3.16
6.318
8.310
1
23.5%
0.93
3.12
6.556
7.870
1,5
21.4%
0.92
3.11
6,239
8.266
3
22.3%
0.91
2.87
5.189
9.156
4
24.4%
0.89
2.71
4.608
9.636
Table 6: Ablation study for we evaluated on LRS3. The marked in bold indicates the value of used by Lip Voicer in the experimental study for LRS3.
LR
LR WER
WER 4
STOI-Net t DNSMOS
LSE-C1
LSE-D
GT
0%
5.4%
0.92
3.10
6.257
8.220
Ma et al.
2023
19.1%
21.4%
0.92
3.11
6.239
8.266
Ma ct al.
(2022
32.3%
38.1%
0.92
3.09
6.053
8.362
Table 7: Ablation study for the choice of the lip reading accuracy, as evaluated on LRS3. LR signifies lip-reader.
LIMITATIONS AND SOCIAL IMPACTS
Loppoicer is powerful lip-to-speech method that has the potential to bring about social impacts. On the positive side, it can help restore missing or corrupt speech in important videos. However, there are also potential drawbacks consider. The generated speech may introduce the risk of misrepresentation or manipulation. While we use faithful lip-reading module, "bad-faith actors" can try to inject misleading text. Mitigating this potential risk an important challenge but beyond the scope of this work.
CONCLUSION
In this paper, we present Voicer, novel method that shows promising results in generating high- quality speech from silent videos. LipVoicer achieves this by utilizing text inferred from a lip-reading model to guide the generation of natural audio. We train and test Lip Voicer multiple challenging datasets comprised of in-the-wild videos. We empirically show that text guidance crucial to creating intelligible speech, measured by the word enror rate. Furthermore, human evaluation that LipVeicer faithfully recovers the ground truth speech and surpasses recent baselines in intelligibility, naturalness, quality, and synchronization. The impressive achievements of Voicer in lip-to-speech synthesis not only advance the current state-of- state-of-the-an but also pave the way for intriguing future research directions in this domain.
show through

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This project has received funding from the European Union's Horizon 2020 Research and Innovation Programme, Grant Agreement No. 871245, and was also supported in part by the Israeli Council for Higher Education, Data Science Program ("Audience" project) and Nvidia academic hardware grant.
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep mudio-visual speech recognition. In arXiv:/809.02/08, 2018a.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. LRS3-TED: a large-scale dataset for visual speech recognition. arXiv, abs/ 819800904999 2018b.
Mikolaj Binkkoski,, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. In International Conference on Learning Representations, 2019.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: a language modeling approach to audio generation. arXiv preprint arXiv:2209. 03 143, 2022.
Maxime Burchi and Radu Timofte. Audio-visual efficient conformer for robust speech recognition. In Proceedings of the IEEE/CVF Winter Conference Applications of Compuler Vision (WACV), pp. 2258-2267, January 2023.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021.
Jeongsoo Choi, Joanna Hong, and Yong Man Ro. Diffv2s: ceffusion-basede video-to- to-eeehh synthesis with vision- vision-guided speaker embedding. arXiv preprint arXiv:2308.00 077007877 2023a.
Jeongsoo Choi, Minsu Kim, and Yong Man Ro. Intelligible lip-to-speech synthesis with speech units, 2023b.
J. S. Chung and A. Zisserman. Out of time: automated lip sync in the wild. In Workshop Multi-view Lip-reading ACCV, 2016.
S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. INTERSPEECH. 2018.
Martin Cooke, Jon Barker, Stuart P. Cunningham, and Xu Shao. An audio-visual corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Society of America, 120 5 Pt 1:2421-4, 2006.
Rodrigo Schoburg Carrillo de Mira, Alexandros Haliassos. Stavros Petridis, Björ W. Schuller, and Maja Pantic. SVTS: scalable video-to-speech synthesis. In Interspeech, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems (NeurIPS). 2021.
Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross-modal
consistency. In CVPR, 2021.
Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio gener- ation using instruction-tuned Ilm and latent diffusion model. arXiv preprint arXiv:2304.13311, 2023.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! audio generation with state-space models. In International Conference on Machine Learning, 2022.
Naomi Harte and Eoin Gillen. Ted-timit: An audio-visual corpus of continuous speech. IEEE Transactions Multimedia, 17:603-615, 2015.
10

---

Published as a conference paper at ICLR 2024
Michael Hassid, Michelle Tadmor Ramanovich, Brendan Shillingford, Miaosen Wang. Ye Jia, and Tal Remez. More than words: In-the-wild visually-driven prosody for text- text-to-speech. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10577-10587, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recog- nition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 16. pp. 770-778. IEEE, June 2016.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, and Yossi Adi. Revise: Self-supervised speech resynthesis with visual imput for universal and generalized speech enhancement, arXiv e-prints, 2022.
Myconghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to-speech. INTERSPEECH, 2021.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, 2018.
Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-TTS, A diffusion model for text-to-speech via classifier guidance. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11119-11133. PMLR, 17-23 Jul 2022.
Minsu Kim, Joanna Hong, and Yong Man Ro. Lip to speech synthesis with visual context attentional gan. Advances in Neural Information Processing Systems, 34:2758-2777 2021.
Minsu Kim, Joanna Hong, and Yong Man Ro. Lip-to-speech synthesis in the wild with multi-task learning. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10(60): 1755-1758, 2009.
5m1r. owt.lr..rg/ppgg /aape//ppprrs././ / /0i/0/kigg9a... html
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference Learning Representations, 2021.
Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie- Liu. Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. In International Conference on Learning Representations, 2021.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Computer Vision ECCV 2018, pp. 122-138, 2018
Pingchuan Ma. Brais Martínez, Stavros Petridis, and Maja Pantic. Towards practical lipreading with distilled and efficient models. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021, Toronto, ON, Canada, June 6-1 I, 2021, pp. 7608-7612. IEEE, 2021.
Pingchuan Ma, Stavros Petridis, and Maja Pantic. Visual Speech Recognition for Multiple Languages in the Wild. Nature Machine Intelligence, 939, 2022.
Pingchuan Ma, Alexandros Haliassos, Adriana Foppa, Honglie Chen, Stavros Petridis, and Maja Pantic. Auto-avsr: Audio-visual speech recognition with automatic labels, 2023.
Rodrigo Mira, Konstantinos Vougioukas, Pingchuan Ma, Stavros Petridis, Bjöm W Schuller, and Maja Pantic. End-to-end sadeo-to-speech synthesis using generative adversarial networks. IEEE Transactions Cybernetics, 2022.
11

---

Published as a conference paper at ICLR 2024
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. Specaugment: A simple data augmentation method for automatic speech recognition, 2019. URL|attp://axxiv org/ abs 1 1904. .2500..88.0 C8779
Hyun Joon Park, Scok Woo Yang, Jin Sob Kim, Wooseok Shin, and Sung Won Han. Tri Triple adaptive attention normalization for any-to-any voice conversion. arXiv preprint arXiv:2303.09057. 2023.
Wei Ping. Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-to- speech. In International Conference on Learning Represenlutions, 2018.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In Internationall Conference on Machine Learning. 2021.
KR Prajwal, Rudrabha Mukhopadhyay, Vinay Namboodiri, and CV Jawahar. Learning individual speaking styles for accurate lip to speech synthesis. In Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Chandan K Reddy, Vishak Gopal, and Ross Cutler. Dnsmos p.835: non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 886-890, 2022.
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio- visual speech representation by masked multimodal cluster prediction. In International Con- ference on Learning Representations, 2022. URL http:htttt//ectt h net/ fortmfir 2101miluOH
Berrak Sisman, Junichi Yamagishi, Simon King, and Haizhou Li. An overview of voice conversion and its challenges: From statistical modeling to deep learning. JEEF/ACM Trans. Audio, Speech and Lang. Proc., 19:132-1555 nov 2020. ISSN 2329-9290.
Jiaqi Su, Zeyu Jin, and Adam Finkelstein. Hifi- gan: High-fidelity demoising and dereverberation based on speech deep features adversarial networks. Interspeech, 2020.
Yaniv Taigman, Lior Wolf, Adam Polyak, and Eliya Nachmani. Voiceloop: Voice fitting and synthesis via a phonological loop. In International Conference Learming Representations, 2018.
Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro. Flowtron: an autoregressive flow- based generative network for text-to-speed) synthesis. In International Conference on Learning Representations, 2021.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In 9th ISCA Speech Synthesis Workshop, 2016.
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. Interspeech, 2017.
Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEFF/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Ryandhimas E. Zezario, Szu-Wei Fu, Chiou-Shan Fuh. Yu Tsao, and Hsin-Min Wang. Stoi-net: A deep learning based non-intrusive speech intelligibility assessment model. 2020 Asia-Pacifh Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 482-486, 2020. URLhhtpss:/ api semant ooo
12

---

Published as a conference paper at ICLR 2024
A FULL IMPLEMENTATION DETAILS
In this section, we provide full details on the preprocessing, hyperparameters and architectures used for Voicer Our implementation was written in PyTorch, and we used 4 NVIDIA GeForce RTX 2080 Ti for our experiments.
A.1 VIDEO PREPROCESSING
The output of the preprocessing step of silent talking- face video is single face image and mouth crop video. The latter is used by the lip-reading network, while both components are fed into the MelGen module. We use 25 fps videos. The face image is randomly chosen from the full-face video frames. The image is resized to the resolution of 224 x 224. During training, brightness and color augmentations applied well. The mouth crop video is created by detecting and tracking 68 facial landmarks using dlib (King) 2009). The faces detected in the frames are then aligned to a mean reference face using a similarity transformation which removes rotation and scale differences. Next, 96 x 96 mouth region is cropped from each frame to form the mouth crop video. The last step is the conversion
gray-scale video. In the training stage, a random 88 x 88 crop is taken from the lip video and then horizontally flipped with probability 0.5. At inference time, we take a center 88 x 88 crop without flipping-
A.2 AUDIO PREPROCESSING
In Voicer, we need meesspectss of real speech signals for training MelGen and the vocoder. They were computed by applying the following steps. First, 16KHz speech signals were extracted from the training videos of the benchmark datasets and normalized to have a maximum magnitude of 0.9. Subsequently, the short-time Fourier transform (STFT) for all experiments was calculated with window size and Discrete Time Fourier Transform (DTFT) length of 640 samples. The hop size between two successive windows was 160 samples. The STFT magnitude was used to compute 80 mel frequencies with 20Hz and 8KHz being the lowest and highest frequencies, respectively. The result was clipped to have a minimum value of le - 5. The logarithm function was applied to compress the dynamic range. Finally, we linearly mapped the output of the previous step roughly be in [-1, 1] by using the global minimum and maximum values across the entire training split.
A.3 ARCHITECTURE
A.3.1 MELGEN
Backbone. The backbone architecture for MelGen is identical to DiffWave (Kong al. 2021 but we change several parameters. Using the notations from Kong et al. (202T). the diffusion process has = 400 steps. We set 31 0.0001 and 87 = 0.02, and use linear noise schedule, The number of input and output channels is fixed to 80, to match the number of mel-spectrogram frequencies. We use 12 residual layers with 512 residual channels, and the dilation factor was set to 1 for all residual layers. We base our backbone architecture on the publicly available implementation released by the authors of Goel et al. (202)).
P of DiffWave,
Conditioner. As mentioned, the video embedding serves the conditioner for MelGen. The face image analysis network takes in 224 x 224 image and comprises a ResNet-18 backbone with the last two layers scrapped, followed by fully- mulle-connected layer which outputs D, 128 dimensional embedding vector. The lip motion analysis network receives a 88 x 88 mouth crop of N consecutive 7 x kernel, followed by ShuffleNtt-V2
frames. It consists of 3D-convolutional layer with (Ma et al. 2018) network and then a TCN. Full details can be found in|Ma et al. (021). The output of the TCN takes the dimension x Dm,. where Dm 512. The face embedding is replicated N times and concatenated to the lip motion embedding, such that the ultimate video embedding v has the size of N x 640. Similarly to Kong et al. (2021), we upsample the time axis of the video embedding by applying two layers of 2D convolutions. As each mee-sspccroogrmm frame comprises 160 new samples, the audio sample rate is 16KHz and videos are sampled at 25 fps, we set the upsampling factor of each convolution layer to 2. Under this choice, the overall upsampling factor amounts to 4 which aligns the time resolution of the specttretr and the video.
https:: http://ggtttuhccce com bert fgu/d ffwave- sash (aahhim mi
13

---

Published as a conference paper at ICLR 2024
Training. During training, MelGen generates mel-spectrograass covering 1 second conditioned on the matching video with the equivalent number frames. It translates to 100 mel-spectrogram frames and hence the video comprises N 25 frames. Note that at inference time, since the DiffWave backbone is based on convolutional layers, we take a video of an arbitrary number of frames. In our experiments, we managed generate high-quality speech signals which were 18 seconds long. MelGen is trained with L1 loss on the diffusion noise prediction, without any additional loss terms. We trained on 1,000,000 mini-batches of 16 videos and used Adam optimizer with learning rate of 2e 4 without scheduling. For the classifier-free guidance mechanism, we follow Ho & Salimans (202T) by setting the dropout probability on the conditioning to 0.2. The null tokens for the face image and the lip motion video when the conditioning is dropped are randomly drawn in advance from a normal distribution.
A.4 ASR CLASSIFIER
As mentioned, Burchi & Timofte (023) was deployed as the ASR for guiding MelGen through the classifier guidance mechanism. We used the publicly available code and the pre-trained model released by the authors of Burchi & Timofte (2023), which was trained on the LRW, LRS2, and LRS3 datasets. In order to adapt their implementation to be compatible with LipVer diffusion time step embedding to the conformer blocks. The computation of the time step embedding the same positional encoding and two fully-connected
added
identical to Kong et al. (2021), i.e. layers which are shared across all conformer blocks. Each conformer applies an additional layer- specific fully-connected layer, and the result is summed with the input to the conformer block. In addition, since for Lip Voicer, the ASR classifier receives mel-spectrograms noised by the diffusion process, we remove the -peestcctr and SpecAugment (Park al.) 2019) layers from the original implementation of the ASR. We finetune our modified ASR on the train+pretrain splits of LRS2 and LRS3 using connectionist temporal classification (CTC) loss, where the input a mel-spectrogram from the training splits noised by the diffusion process and the target is the ground truth text. We used Adam optimizer with learning rate le 4 and akin Burrch & Timofte (2023), we used batch size Of 256 videos via mini-batches of size 64 and 4 accumulated steps.
B INFERENCE
We summarize the inference process in Algorithm
Algorithm Inference with Lip Voicer
1: Given silent video V
2: Vz + lip video
3: Ir <- randomly chosen face frame from V
4: Predict the text LR from Vz with a lip-reader
5: Initialize xx (O,
6: to T do
7:
Compute cmg with classifier-free guidance
8: Cmg (x,, IP,Ip = (1+w))ec VI,IF) wrc)(x))
9:
Compute è using classifier guidance ||Emgl]
10:
't- VI-a.-11Va, log px+],
11:
É= Emg - w2 5, V x, log lot.(())
ZT N 1) i else z 0
13:
X, vone X vT-a
81 8 2)
Biz
14: end for
return Xo
C GENERATING FULL DATASETS
In the course of our experiments, we noticed that different choices of hyperparameters influenced the quality of the reconstructed speech. In particular, the hyperparameters of Voicer the values
14

---

Published as a conference paper at ICLR 2024
of w1, w2 and fASR, where the latter denotes the diffusion time step of MelGen in which we start to apply the classifier guidance mechanism. When the ASR guidance was taken into account right from the start (t 400), it led to degraded results. By using hyperparameter grid search and listening tests, we eventually chose:
Wi = 2, w2 1.5, fASR = 270 for LRS3.
W1 1.8, W2 = 1.7, = 230 for LRS2.
W1 = 2.6, W2 = 0.7,4AS = 300 for GRID.
C.I VOCODER
We adopt DiffWave (Dhariwal & Nichol 2021)
vocoder (converting mel-spectrograms
speech signals). We use the conditional variant of DiffWave with 7 50 (DiffWavenAse) without changing the parameters of the architecture. We train the vocoder 1, 1,000,000 mini-batches with batch size 16 with a learning rate of 2e without scheduling on audio taken from the training split of LRS3 only.
D INTRUSIVE METRICS FOR LIP-TO-SPEECH
Following Sisman et al. 2020), we argue that using intrusive measures like PESQ and STOI in problem is fundamentally incorrect. To exemplify we show that if take recording of certain speaker, they cannot differentiate between the noisy version of the recording and the same recording with the change of speaker. To this end, we use the the following experiment: we took speech signal and created two versions of it. One is = x + where is a Gaussian noise with SNR=5dB. Second is 2, the output is In other words, is identical to with respect to the spoken words and their timing, but with a different yet fairly close voice. We computed the intrusive metrics PESQ, STOI, ESTOI and the non-intrusive ones DNSMOS and STOI-Net and received the results compiled in Table 8
voice conversion system (Park et al.) (023) where the input
The intrusive metries consider the voice converted version as equivalent to the highly noisy signal y, which is obviously wrong. For completeness, also bring here the PESQ and STOI scores for Voicer and the baselines for LRS2 (Table 9) and LRS3 (Tablc10).
E ADDITIONAL ABLATION STUDIES
E. ASR WER AND FACE EMBEDDING
Evaluating the performance of LipVoicer with respect to the WER of the guiding ASR is presented in Table [T] is clear that it is necessary for the ASR to be reliable. Table T2 provides objective metrics for the ablation study on using the face embedding, We implemented the ablated model by replacing the face embedding by the null embedding which is part of the massifier-free guidance mechanism. The full model is slightly better, possibly because the model was not trained when only the face image was replaced with the null embedding. The full model replaces the lip region video and the face image with null tokens simultaneously during training. In any case, the main aspect in which discarding of face embedding is manifested is the lack of personalized voice for each video.
PESQ
STOI
ESTOI
STOI-Net
DNSMOS
x (clean)
0.86
3.31
y (noisy)
1.38
0.74
0.52
0.74
2.45
2 (converted)
1.14
0.76
0.56
0.92
3.15
Table 8: Intrusive and non-intrusive metrics for the example showcasing why using the former in lip-to-speech should be avoided.
15

---

Published as a conference paper at ICLR 2024
PESQ
STOI
ESTOI
STOI-Net
DNSMOS
GT
0.91
3.14
LIPVOICER (OURS)
1.10
0.38
0.23
0.91
2.89
LIP2SPEECH
1.36
0,53
0.34
0.70
2.37
VCA-GAN
1.24
0.40
0.13
0.51
2.26
Table Intrusive and non-intrusive scores for LRS2.
PESQ
STOI
ESTOI
STOI-Net
DNSMOS
GT
0.93
3.30
LIPVOICER (OURS)
1.10
0.38
0.21
0.92
3.11
SVTS
1.26
0.53
0.31
0.65
2.42
LIP2SPEECI
1.31
0.50
0.27
0.67
2.36
VCA-GAN
1.24
0.47
0.21
0.63
2.27
Table 10: Intrusive and non-intrusive scores for LRS3
We have uploaded the demo page several examples which compare audio generated by the full and ablated models.
E.2 VOCABULARY SIZE
We also wish to test the influence of the amount of training data, which entails larger vocabulary. For LRS2 and LRS3, the vocabulary the pretrain split 2-3 times larger than the train split vocabulary. To this end, conduct MOS evaluation LRS3 that compares Lip Voicer which was trained on one of the following options: train+pretrain splits or the train split only. addition, we test Lip Voicer coupled with the following lip-reading networks and WER rates: (1) the ground-truth text (2) MA (Ma et al. 2023). achieving 19.1%. Comparing to the ground-truth text can better help understand the potential of proposed method. We only use here the MOS test, since it is important confirm that the audio quality is good before conducting objective tests which are covered in the main text.
For the ablation study, randomly select 50 samples from the LRS3 test split and assign three different annotators per sample. Participants are presented with all methods at once when the order of appearance is randomized between samples (see Fig. 4 The average scores are presented in Table 13, and seem to indicate that a larger vocabulary does indeed leads to better performance. When only the train split used for training, using the ground-truth or predicted text results in equivalent performance.
We also examined how the model's influence is affected by increasing the amount of training data even further. We thus trained Lip Voicer on LRS3 and videos from VoxCeleb2 (Chung et al. 2018) which contain English speech. The number of training videos amounted to roughly 830,000. can be seen from Table 14 that the enhanced training data leads to better results.
E.3 USING DIFFRRENT VOCODER
We compare here the metrics for LRS3 computed on speech signals generated using HiFi-GAN (Su et al., 2020) and DiffWave as the vocoders. Both vocoders used the exact same mel-spectrograms generated by Vipver Both vocoders used the exact same mel-spectrograms generated by Lip Voicer. Also note that the vocoders were trained on natural audio signals on not on mel-speccrograms generated VipVer We see in [5 that both options that use two different vocoders yield state-of-the- are results. This clearly indicates that the main and critical source of improvement the mel-spectrograms generated by method.
16

---

Published as a conference paper at ICLR 2024
ASR WER
WER L
STOI-Net 1
DNSMOS
LSE-C1
LSE-D +
3.6%
21.4%
0.92
3.11
6.239
8.266
6.1%
76.3%
0.92
2.98
7.118
7.171
Table 11: Ablation study for the ASR guidance evaluated on LRS3. Inferior ASR WER achieved by early stopping.
Face Embedding
WER
STOI-Net
DNSMOS 1
LSE-C1
LSE-D +
with 21.4%
0.92 0.92
3.11
6.239
8.266
w/o 22.9%
3.05
6.091
8.407
Table 12: Ablation study for using face embedding evaluated on LRS3.
E.4 QUALITATIVE RESULTS
We present qualitative comparison between the selespectosgrass generated by LipVoicer, the baselines, and the ground truth. The results, shown in Fig. demonstrate that even for a relatively long utterance, the mel-spectrogram generated by Voicer visibly resembles that of the original video. While all approaches manage to successfully detect the beginning and the end of speech segments, Lip Loicer is the only method that generates spectral content that precisely aligned with the ground truth mel-spectrogram. The baselines, on the other hand, struggle to achieve this level of fidelity. Particularly, they fail to generate pitch information of the speech, which results in an unnatural voice and impairs the naturalness of the speech signal. This comparison provides valuable insights into the capabilities of Lip Voicer to generate naturally looking spectrograms that synchronized with the original video's 's mel-spectrograms.
E.5 NATURALLNESS-IVTELLIIIN
TRADE-OFF
Another aspect of Lip Voicer that should be taken into consideration is the influence of the ASR classifier on the quality of lip-to-speede performance. Occasionally, the speech signal recovered by MelGen, namely without the classifier guidance provided by the ASR, sounds more natural than the one generated by the full scheme, i.e. Voicer In addition, the ASR guidance may lead to synchronization lapses when the predicted text does not significantly overlap with the ground truth text. However, as emerges from the results in T the inclusion of the ASR guidance has crucial impact on the intelligibility of the speech. In particular, it results in an invaluable gain on the WER metric. This trade-off is also depicted in Fig. [3] The mel-spectrogram produced without applying classifier guidance presents smooth transitions. this context, incorporating the ASR in the inference process degrades the mel-spectrogram, as can be seen by comparing Fig. 3bb and Fig. Bc However, the ASR guidance managed to reconstruct spectral content (right-hand side of the mel-spectrograms) more faithfully with respect to the setup where the ASR guidance is omitted. Consequently, while it may sometimes reduce the speech naturalness, in most cases the generated speech at the output of method is of high quality, sounds natural, and is intelligible.
F RESULTS FOR GRID DATASET
In this section, we evaluate Lip the GRID dataset (Cooke et al. 2006). GRID is audio- visual dataset comprising 33 speakers, where each speaker contributes 1,000 utterances. We adhere to the data split dictated by the lip-reading network (Ma et al.) 2022). Since the code implementation for SVTS (de Mira et al., 2022) is not publicly available and the implementation for Lip2Speech (Kim al. 2023) required significant modifications, compare only VCA-GAN (Kim et al.
17

---

Published as a conference paper at ICLR 2024
Source of text/ Lip-Reader
Training dataset split
Intelligibility
Naturalness
Quality
Synchronization
Ground-Trus Text
train
3.18 + 0.28
3.29 +: 0.28
3.07 0.25
3.36 = 0.26 4.14 +: 0.07
Ground- Text
train pretrain
4.14 1: 0.07
4.29 + 0.09
4.18 0.07
MA Ma al. 2023
train
3.21 1 0.24
3.30
1.24
3.43 0.25
3.46 0.22
train pretrain
3.68 1 0.20
3.64 0.19
3.79 1 0.21
3.43 = 0.21
Table 13: Ablation study the amount of training data, as evaluated on LRS3 dataset.
Training split
WER 4
STOI-Net
DNSMOS f LSE-C1
LSE-D 4
LRS3
21.4%
0.92
3.11
6.239
8.266
LRS3+VoXeelbb
20.6%
0.93
3.20
6.312
8.160
Table 14: Performance on LRS3 when training on LRS3+VoxCeleb2
2021 ). We evaluate in terms of MOS using the unseen spli where 29 speakers are used for training and the rest (subjects 1,2, 20, 22) are reserved for the test split.
We randomly selected a total of 50 samples from 4 diffferent unseen speakers. Each sample is ranked by 16 unique raters for quality, synchronization. intelligibility, and naturalness, on scale from 1 to 5. The MOS scores are presented in Table 16 While VCA-GAN managed to achieve good synchronization between the generated audio and the video, Voicer outperforms on all other achieve high level of naturalness and quality and the overall MOS benchmarks. In particular, performance is very close to the ground truth speech scores. Audio samples which were randomly chosen can be found at mtfxttss://iipvccee.e wwtprriieerrgttttte... gwthub..tttigggg 10/ swpp.ppp.ppp..oopp
G PROTOCOL FOR MOS EVALUATION
We used the Amazon Mechanical Turk (AMT) platform to compare the different baseline methods to ours. Participants were asked to rate the videos, on a scale of 1-5, for Intelligibility, Naturalness, Quality, and Synchronization. Fig. showw the task interface. where we simultancously display to single annotator the samples generated by (i) Voicer (ii) SVTS (iii) VCA-GAN (iv) Lip2 as well as the (v) ground-truth video for calibration. The order of the methods is randomized from sample to sample avoid bias. Each sample is evaluated by 16 distinct annotators. Having each annotator score all methods on a specific sample ensures that methods compared are annotated by the exact same set of annotators. We take 50 random samples from the test splits of each dataset, LRS3, and LRS2.
The instruction for the task read follows: Intelligibility evaluates how clear words in the synthetic speech sound. In other words, Can you understand the content easily? Naturalness evaluates how natural the speech is, compared to the actual human voice (e.g. synthetic speech might sound metallic or distorted). We say the sample is synced if there is no time lag between the video and the audio. Finally, Quality the overall quality score for the speech signal given by the rater.
Vocoder
WER
STOI-Net
DNSMOS 1 LSE-C
LSE-D J
DiffWave
21.4%
0.92
3.11
6.239
8.266
HiFi-GAN
21.9%
0.93
3.19
6.308
8.166
Table 15: Vocoder choice ablation
18

---

Published as a conference paper at ICLR 2024
[6) GT
(b) Voiver
(VTS
Lip2Speach
(c) VCA-GAN
Figure 2: The ground truth mel-spectrogram, and its reconstruction by Lip and the baselines.
(a) Ground Truth
(b) LipVoicer w/o ASR guidance
(c) Voiver with ASR guidance
Figure The influence of using an ASR for classifier guidance for lip-to-speech with Lip Voicer.
GRID
Intelligibility
Naturalness
Quality
Synchronization
GT
4.43 = 0.06
4.46 + 0.06
4.29 + 0.06
4.26 + 0.05
VCA-GAN Kim et al. 2021
3.29 0.15
3.23 = 0.17
2.98 - 0.14
3.68 + 0.11
LIPVOICER (OURS)
3.75 0.11
3.98 0.12
3.60 = 0.11
3,82 + 0.11
Table 16: GRID Human evaluation (MOS).
19

---

Published as a conference paper at ICLR 2024
Instuetera
X
stiees valnn mune toagg to Hight
and
the or rheit sparch sounds. other eeddy?
comperelie thrss aut mutatle dedurte))
Phee timallig salveen
Now GrmAd
Figure 4: MOS evaluations. Participants rank the samples generated by VCA-GAN (Kim et al. 202T), SVTS (de Mira et al. (022). LIP2SPEECH (Kim al. 2023). and LIPVOICER (OURS), and the ground-truth. The order of appearance randomized.
20


Reviews:
[{"review_id": "NOdpAbVWa1", "reviewer": "Reviewer_eBPV", "summary": "This work proposes to perform the lip-to-speech task by incorporating the predicted text to guide the diffusion model based learning process. Experiments on the large scale LRS2 and LRS3 show its superiority over others. The results are indeed appealing.", "questions": "(1) I am a little confused about the fig.1(a) as described above.\n(2) The modules and manners in the framework seems to be not new in the view of methodology, with the lipreading module, MelGen, text alignment manners already proposed by other works. Could the authors give a clarification of the contributions? Maybe I miss something?\n(3) The performance using the predicted text is already very appealing, but the involved lip reading model are almost the best two ones at present, with WER=19% and 26%. if the lip reading performance has been a much low value, e.g. WER=50%, what would be the performance here like?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The general structure is clear. The method is simple in general. It’s easy to follow. The performance is good, with a large margin over other methods. It’s also a nice try to include the predicted text into the learning process.", "weaknesses": "(1) I am a little confused with fig1.a. The output of the lipreading module is the predicted text. The output of the ASR modules is also the predicted text. There should be no connections from the output of the predicted text to the ASR module? The ASR module should take the output of MelGen as input? without the text predicted from LR module?\n(2) Lip2speech (Kim et al.(2023)) takes the ground-truth text as input to constrain the learning process and has shown the success of the role of text modality in this task. In this paper, the work uses the predicted text instead of the ground truth as Lip2Speech. But the manner is similar to Kim et al.(2022). So, besides using the predicted text with an existing method, is there some new contributions in the view of methodology?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oaeHGe3PqA", "reviewer": "Reviewer_x1vF", "summary": "This paper proposes LipVoicer, which incorporates the text modality by predicting the spoken text using a pre-trained lip-reading network and conditioning a diffusion model on both the video and the extracted text. To utilize the text modality into the diffusion model, the authors apply classifier-guidance mechanism, where a pre-trained automatic speech recognition (ASR ) serves as the classifier. The results demonstrate the effectiveness of LipVoicer in producing natural, synchronized, and intelligible speech.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. LipVoicer greatly improves the intelligibility of the generated speech and outperforms existing lip-to-speech baselines on challenging datasets, demonstrating its superior performance. \n\n2. The paper provides detailed implementation details, making it easier for others to reproduce and further improve upon the LipVoicer method. \n\n3. By introducing a pre-trained ASR model, this paper realizes a good application of classifier-guidance diffusion model in lip2speech task.", "weaknesses": "1. After listening to Demo page, it is found that the gap between different models is mainly in sound quality. The baselines are too weak in sound quality. However, the problem of sound quality can be solved by many existing generative models based on VAE/GAN/FLOW model. If the sound quality problem of baselines is solved, the advantage of the model proposed in this paper may not be so great.\n\n2. In previous studies, a very important motivation for lip2speech tasks was to dispense with text modality (otherwise, this task can be transformed into lipreading+TTS), because 80% of the world's languages have no written text. However, this paper still depends on the text modality, so it is difficult to give a high score to this article.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BsPSf71ZUb", "reviewer": "Reviewer_gU1H", "summary": "This paper proposes a method to generate a natural-sounding speech from silent video called LipVoicer. The method is different from previous work in a two key ways: (1) the proposed method uses a lip reading model during inference to generate guidance for the generation model, (2) the generative model is based on a diffusion model. The model is trained on LRS2 and LRS3 datasets, which contains challenging examples from near in-the-wild conditions. The proposed system significantly outperforms the baselines.\n\nThe two key ideas actually appeared in accepted recent/concurrent papers, and authors acknowledge these works. Lip reading-based text guidance is proposed in (Kim et al., ICASSP 2023), although it is not exactly the same in that this paper uses the text guidance during inference, whereas Kim et al. uses the guidance during training. The use of diffusion-based model for the lip-to-speech task has been proposed in (Choi et al, ICCV 2023a). Authors are not required to compare their own work to that paper under ICLR rules.", "questions": "- It is not clear why the addition of text guidance helps sync performance.\n- If lip reading networks are used, what is the advantage of the proposed system over a cascaded lip reading + TTS system apart from obtaining duration prediction from sync.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The key ideas are reasonable, and well-engineered combination of proven methods.\n- The use of pre-trained state-of-the-art lip reading model significantly lowers the WER significantly compared to existing methods. \n- The diffusion model generates natural-sounding output, according to the qualitative results reported.", "weaknesses": "- It is not clear if the performance improvement comes from the key improvements, or the replacement of the vocoder, which can be seen as a post-processing step rather than a key part of the algorithm. It is well known that DiffWave produces much more natural-sounding output compared to the Griffin-Lim algorithm used by the previous works.\n- The authors request subjective assessors to rate Intelligibility, Naturalness, Quality and Synchronisation, but it is not clear what the difference between Naturalness and Quality are. There is a screenshot of the evaluation page in the appendix, but it does not make it clear what 'quality' means. \n- The baseline models appear to be using pre-trained model weights. However, the models are not trained on the same data, so the results cannot be compared directly.\n- The method appears to apply Guided-TTS techniques to the problem of lip-to-speech. Although this is well engineered, in my opinion this work is better suited to speech or CV conference compared to ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xQgGylJjiv", "reviewer": "Reviewer_nskp", "summary": "This paper describes a new lip-to-speech method that leverages diffusion models via classifier and classifier-free guidance to reproduce accurate speech from silent videos. The base diffusion model is built upon the DiffWave architecture but generates mel-spectrograms instead. It receives a video of the speaker's mouth (encoded into feats via a lipreading backbone) and a still frame representing the speaker's identity (encoded via a simple ResNet) as the condition for generation. This condition is randomly removed during training to perform classifier-free guidance during inference, as proposed in many other diffusion models. After this is trained, the model is further guided via classifier guidance during inference so that the text extracted by a lip reading model from the video matches the text extracted via a speech recognition model from the generated audio. This model achieves SOTA performance on LRS2 and LRS3, and the design decisions are justified by a set of thorough ablations. Demos are also provided, which help contextualize these results empirically.", "questions": "The authors mention \" To encourage future research and reproducibility, our source code will be made publicly available\". Will this include training code, inference code, and pre-trained models? These would all be hugely helpful to the community in reproducing the authors' state-of-the-art results.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In general, I believe this paper is strong. It clearly sets a new state-of-the-art for lip-to-speech, which is a highly competitive field. \n\nI think the paper is well-written and the motivation for the task and each specific decision in the methodology is concise and meaningful. The methodology is clear and the discussion around the results is welcome. The model figure is adequate in my opinion, and the demos are also very welcome.\n\nThe method here is clearly novel - I don't think I've seen a lip-to-speech paper before with a similar architecture and that leverages classifier guidance from text in such an effective way. The choice for each model component makes sense and the training hyperparameters are described in detail to aid reproducibility. \n\nThe results are clearly strong and are compared with other works via subjective and objective metrics, which are very convincing. The ablations in tables 5, 6, and 7 are insightful and provide some further information about the importance of the weight of the classifier-free (w1) and classifier (w2) guidance, as well as the lip reading model that is used for the classifier guidance. The avoidance of intrusive measures such as PESQ or STOI is well-justified.\n\nThe limitations and social impacts are well addressed, and the conclusions are succinct and valid.", "weaknesses": "First and foremost, it is unfortunate that the authors do not compare directly with ReVISE in their tables, although this is fully justified by the lack of code, difficulty in reproducing their results from scratch, and the lack of samples for comparison. Therefore, I don't think it's fair for me or other reviewers to let this affect our judgment of the paper, as this is not the authors' fault. The presented model seems to compare favorably against ReVISE in the demos, which is encouraging.\n\nThe use of the DiffWave vocoder is reasonable, but it seems to be outperformed by HiFi-GAN and especially the recent BigVGAN. Would be interesting to see a comparison with these, or at least to justify why DiffWave was chosen, as HiFI-GAN is the typical choice in the majority of papers.\n\nIt would also be interesting to scale the model to larger datasets such as LRS3+VoxCeleb2 as was done in SVTS. This would help demonstrate the model's scalability to larger datasets, which is an important aspect of models in this field since audio-visual data is so plentifully available.\n\nI could not find any typos or substantial errors in the writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)