SyncTalklip: Highly Synchronized Lip-Readable Speaker Generation with Multi-Task Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 19:21:08

Abstract: Talking Face Generation (TFG) reconstructs facial motions concerning lips given speech input, which aims to generate high-quality, synchronized, and lip-readable videos. 
Previous efforts have achieved success in generating quality and synchronization, and recently, there has been an increasing focus on the importance of intelligibility. Despite these efforts, there remains a challenge in achieving a balance among quality, synchronization, and intelligibility, often resulting in trade-offs that compromise one aspect in favor of another. 
In light of this, we propose SyncTalklip, a novel dual-tower framework designed to overcome the challenges of synchronization while improving lip-reading performance. To enhance the performance of SyncTalklip in both synchronization and intelligibility, we design AV-SyncNet, a pre-trained multi-task model, aiming to achieve a dual-focus on synchronization and intelligibility. Moreover, we propose a novel cross-modal contrastive learning bringing audio and video closer to enhance synchronization.
Experimental results demonstrate that SyncTalklip achieves state-of-the-art performance in quality, intelligibility, and synchronization.
Furthermore, extensive experiments have demonstrated our model's generalizability across domains.
The code and demo is available at \url{https://sync-talklip.github.io }.

Main paper: 
 related work 2.1 talking face generation: 
 Initial work [18, 28] used deep learning to learn the mapping of a single speaker's speech to a lip, and then inputting that person's speech into the model to get the corresponding lip. As they are trained on only a specific speaker, they cannot synthesize new identities or voices. However, real-world applications require models that can easily handle generic identities, which means they can accurately generate the corresponding lip movements given any speech. This necessity has led to the development of speaker-independent models [13, 16] . As the need for synchronization and quality becomes increasingly critical, most of the subsequent work focuses on improving realism, audio-lip synchronization, etc., ignoring the intelligibility issue. [4] employs facial keypoints to bridge audio and video, and uses adversarial loss to make the generated video more realistic. [24] enhances synchronization by introducing an additional synchronization discriminator. [40] considers the video's speaking content, head gesture, and identification are considered separately, which contributes in gesture control. In addition, it applies contrast learning ideas to alignment constraints. In recent work, [34] uses a lip-reading expert to deal with the reading intelligibility problem of the TFG task, introduces the idea of contrast learning to enhance synchronization, and uses [5] , a state-of-the-art network in the field of audio-video alignment, for evaluation. However, it can not balance the individual properties well enough to achieve both good synchronization and lip-reading performance without compromising on metrics such as generation quality. So we propose SyncTalklip, which employs a new auxiliary lip-reading network to guarantee lip-reading performance while preserving synchronization in a more reasonable way.AV-HuBERT [26] is a framework for self-supervised representation learning [21] based on audio and video. In the pre-training phase, it can map paired audio-video embeddings close together. Forcing the model to capture temporal relations through a mask prediction task helps to learn the relationship between speech and lip movements, demonstrating that it can align the audio-video features semantically. The fine-tuned model performs well on the lip-reading task. Therefore, supervising the TFG task with AV-HuBERT as a discriminator can lead to an improvement in the intelligibility of the generated video. However, synchronization is not focused on in this self-supervised learning model.SyncNet [5] is a deep neural network specifically designed for visual synchronization of audio in video. The model trains two independent coding networks for temporal offset detection, which enables the network to recognize temporal deviations between audio and video streams.AV-HuBERT implements semantic-level alignment and Sync-Net implements temporal-level alignment. Although both are significant for audio-visual content, the focus on both components remains insufficient in existing work. Therefore, we designed AV-SyncNet, a pre-trained multi-task model aiming to integrate the strengths of both. 
method: 
 We propose a SyncTalklip network as shown in Fig. 2 . Given our focus on the mouth area, we mask and predict only the lower part of the image to enhance detail capture. AV-SyncNet is introduced as a lip-reading supervisor to obtain intelligibility improvement through lip-reading loss. A novel cross-modality contrastive learning loss is employed to improve synchronization. The model trains audio and video encoders and generators during gradient backpropagation.Although the AV-HuBERT [26] model is highly specialized for lipreading and the SyncNet [5] model excels in synchronization, neither model fully capitalizes on the combined benefits of these aspects. Previous TFG works [34] have considered the importance of intelligibility of the generated video but have not completely harnessed the potential of synchronization and intelligibility together. Therefore, we introduce SyncTalklip, which uses AV-SyncNet as a submodule and aims to integrate the strengths of these two areas. By finely tuning the balance between lip-reading accuracy and audio-video synchronization, SyncTalklip creates talking faces that are both high-quality, synchronized, and intelligible.Define 𝑓 𝑎 ∈ R 𝑇 ×𝐷 , 𝑓 𝑣 ∈ R 𝑇 ×𝐷 as the features generated by the audio and video encoders, respectively, where T is the sequence length and D is the embedding dimension. Previous work [34] has attempted to directly bring 𝑓 𝑎 and 𝑓 𝑣 closer using cosine similarity, which can be expressed as:EQUATIONBut there is no guarantee that the cosine similarity of the corresponding paired audio-video feature is the maximal. Specifically, denote (𝑓 𝑎 𝑖 , 𝑓 𝑣 𝑖 ) as the paired audio-video features, and {(𝑓 𝑎 𝑖 , 𝑓 𝑣 𝑗 )} 𝑗≠𝑖 as the unpaired audio-video features. There is no guarantee that: local-level and global-level. Note that the audio is pre-processed by ResNet [29] and the video is pre-processed by FFN [11] . Stage 2, the trained AV-SyncNet is applied to SyncTalklip, so that the generated video combines the performance of both synchronization and intelligibility.EQUATIONIn fact, there isn't even a guarantee that 𝑆𝑖𝑚(𝑓 𝑎 𝑖 , 𝑓 𝑣 𝑖 ) will be relatively large among {𝑆𝑖𝑚(𝑓 𝑎 𝑖 , 𝑓 𝑣 𝑗 )} 𝑇 𝑗=1 . Our AV-SyncNet model essentially customizes a more sensible distance metric for SyncTalklip, which can be seen as a customized distance abstract space. It ensures that the lip-reading performance degrades as little as possible while bringing the L1 distance of 𝑓 𝑎 and 𝑓 𝑣 closer during the training process. Through this metric, the distance of the paired audio-video features, denoted as 𝐷𝑖𝑠𝑡 (𝑓 𝑎 𝑖 , 𝑓 𝑣 𝑖 ), will be relatively small among {𝐷𝑖𝑠𝑡 (𝑓 𝑎 𝑖 , 𝑓 𝑣 𝑗 )} 𝑇 𝑗=1 . In this way, when training SyncTalklip, it becomes more effective to draw the audio-video features closer within the same distance abstraction space.AV-SyncNet merges the benefits of semantic-level alignment and temporal-alignment to avoid subtle lip synchronization problems while ensuring the accuracy of visual representation. SyncTalklip will map paired audio-video embeddings close together during pretraining, but downstream tasks will corrupt the alignment. Therefore, we constrain the audio-video embeddings through L1 loss, which can better converge to a local minimum than other losses [38] , such as L2 loss. Drawing closer in this abstract space may also give the model a better understanding of other attributes, such as speech habits and facial expression preferences [32] , which will be proved in Sec. 4. Knowledge learned with different goals interacts and reinforces each other, so the process of drawing closer is also a process of gaining a deeper understanding of semantics. In Fig. 2 (a), the encoder is pre-trained and fintune on the downstream task. Then it is used as an initialization of the audio encoder and a lip-reading expert in the SyncTalklip.Pretrain. The encoder is pre-trained using a self-supervised method, which is the same as [11] . Two steps are alternated during pretraining: feature clustering and mask prediction. The clustering phase uses a discrete latent variable model to form {𝑧 𝑡 } 𝑇 𝑡 =1 , and the model then performs mask prediction through the Transformer [30] architecture to learn a better representation of the audio and video in the semantic space 𝑓 𝑚 = {𝑓 𝑚 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑇 ×𝐷 , where T is the sequence length, D is the embedding dimension, and m means modality. Given the output probability {𝑝 𝑡 } 𝑇 𝑡 =1 , The pre-training loss is:EQUATIONwhere 𝑀 𝑎 and 𝑀 𝑣 denote the frames that are masked for the audio and video. 𝛼 controls the contribution of the unmasked regions in the overall objective.Finetune. The pre-trained model is fine-tuned for the lip-reading task, and the paired audio-video embedding is simultaneously aligned using Eq. 5, which effectively solves the fine-grained synchronization problem, and at the same time maintains the lipreading ability. As shown in Fig. 2 (a), with audio-only data as input, the data is processed by the fusion module and pre-trained encoder as𝑓 𝑎 = {𝑓 𝑎 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑇 ×𝐷 .Similarly, the video-only data is processed as 𝑓 𝑣 = {𝑓 𝑣 𝑡 } 𝑇 𝑡 =1 ∈ R 𝑇 ×𝐷 .} 𝑡 -1 𝑖=1 , 𝑓 𝑣 ), where 𝜔 𝑖 is the ground-truth transcription. Lip-reading Loss. The lip-reading loss is used to maintain the intelligibility of the model, which is a sequence-to-sequence loss. It is calculated after the decoder module using cross-entropy loss. Define 𝑆 as the length of the target text, and the lip-reading loss can be expressed as:EQUATIONAv-alignment Loss. Considering the problem mentioned in Sec. 3.1, this loss is used to create a customized distance metric for SyncTalklip, which is a crucial basis for Eq. 12. Define 𝜏 as a temperature ratio, which is a hyperparameter that weights the degree of synchronization. Then the 𝐿 𝑎𝑣 can be expressed as:EQUATIONDefine 𝐿 𝑙𝑖𝑝 as the loss of the lip-reading, and 𝐿 𝑎𝑣 as the loss of bringing embeddings closer. The AV-SyncNet loss is:EQUATIONwhere 𝜆 𝑎𝑣 and 𝜆 𝑙𝑖𝑝 scale the contributions of different loss items.As shown in Fig. 2 (b), the generated frames first undergo framelevel loss computation, after which the predicted frames are added back to the original sequence for stream-level loss computation. Denote the original video frame as 𝑣 𝑖 ∈ R 𝐶 ×𝐻 ×𝑊 , and the generated video frame as v𝑖 ∈ R 𝐶 ×𝐻 ×𝑊 , where 𝐶 denotes the channels. Take the prediction of frame 𝑖 as an example, the complete sequence is V = [𝑣 1 , ..., v𝑖 , ..., 𝑣 𝑇 ] ∈ R 𝑇 ×𝐶 ×𝐻 ×𝑊 . The encoded modality embeddings 𝑓 𝑎 and 𝑓 𝑣 are fed into the generator, which yields v𝑖 . Then compute the frame-level loss of (𝑣 𝑖 , v𝑖 ): reconstruction loss and adversarial loss. Subsequently, the lip-reading effect of the generated video stream V is evaluated with a frozen AV-SyncNet, from where the lip-reading loss is calculated. At the same time, it can also generate the video embedding 𝑓 𝑣 as a byproduct. The tunable audio encoder is init by the AV-SyncNet encoder, and generates the audio embedding 𝑓 𝑎 . Then 𝑓 𝑎 and 𝑓 𝑣 are brought closer through our novel cross-modality contrastive loss. Stream-level loss including contrastive loss and lipreading loss controls the generation ability, affecting the semantic and temporal alignment of the generated audio-visual outcome. In this process, all gradients are back-propagated to the generative system and the discriminator.Reconstruction Loss. Reconstruction loss is used to minimize the distance between the generated image and the real image. In this way, the model learns the pixel-level details of the generated image. The reconstruction loss is formulated as:EQUATION)Adversarial Loss. Adversarial Loss utilizes the framework of Generative Adversarial Networks (GAN) [6] , this loss function forces the generated images to be more visually realistic. Through adversarial training [14] , the generator learns to create realistic-looking images that are difficult to distinguish by the discriminator, thus enhancing the realism of the generated images. 𝑣 represents a sample drawn from a real data distribution, and v is the generated frames by our model. 𝐷 (𝑣) denotes the probability that the discriminator network evaluates 𝑣 to be true. The discriminator 𝐷 tries to output a high probability value for the true sample 𝑣 and a low probability value for the fake sample generated by SyncTalklip. The model minimizes 𝐿 𝑔𝑎𝑛 :EQUATION)EQUATIONEQUATIONLip-reading Loss. Lip-reading loss is generated by the lip-reading supervisor. This loss aims to improve the performance of the system in terms of intelligibility to ensure the correct delivery of linguistic content. It is calculated in the same way as Eq. 4. However, the AV-SyncNet here is frozen, and the gradient is used to update the generation system during backpropagation.Contrastive Loss. Utilizing the concept of contrastive learning, we propose a novel cross-modal contrastive learning strategy for audio-visual fine-grained alignment. Although it differs from traditional methods, the underlying idea remains similar. This strategy enhances the model's performance in synchronization and intelligibility by bringing paired embeddings closer together and increasing the distance between unpaired embeddings in L1 space, which corresponds to AV-SyncNet. Accordingly, the model minimizes the following loss:EQUATION)EQUATIONwhere the temperature coefficient, denoted by 𝜏, is used to control the degree of scaling, and 𝑓 𝑥 is the embedding of the modality frame 𝑥. By combining different loss functions, SyncTalklip employs a multi-task learning strategy [14] designed to optimize multiple objectives simultaneously. This strategy allows the model to not only perform well on a single task, but also to ensure synchronization of speech and lip movements as well as correct understanding of content while maintaining visual fidelity.The SyncTalklip loss is:𝐿 𝑡𝑜𝑡𝑎𝑙 = 𝜆 𝑐𝑎𝑣 • 𝐿 𝑐𝑎𝑣 + 𝜆 𝑔𝑎𝑛 • 𝐿 𝑔𝑎𝑛 + 𝜆 𝑟𝑒𝑐 • 𝐿 𝑟𝑒𝑐 + 𝜆 𝑙𝑖𝑝 • 𝐿 𝑙𝑖𝑝 (13)where 𝜆 𝑐𝑎𝑣 , 𝜆 𝑔𝑎𝑛 , 𝜆 𝑟𝑒𝑐 and 𝜆 𝑙𝑖𝑝 scale the contributions of different loss items. Performances of methods with * are collected from [23] and [34] . For [8] and [41] , we used the model given in the paper and did a partial test in LRS3.During the fine-tuning of AV-SyncNet, we adopted a novel strategy that incorporates a convolutional module before the alignment of audio-visual embeddings. This approach is designed to capture deep semantic relationships between neighboring frames, thereby shifting the model's focus from the local level to the global level [17] . However, it comes at the cost of sacrificing fine-grained supervision, in exchange for a deeper comprehension of the semantic information.To further improve the performance of the model, we freeze the AV-SyncNet audio encoder during the training of SyncTalklip. In this method, the output from the audio encoder serves as the input for computing contrast loss, yet it does not participate in the gradient updates. Instead, an additional audio encoder is employed to participate in the update process. The processing flow of the module is shown in the supplementary in detail. 
experiment 4.1 dataset: 
 The LRS3 dataset [2] consists of obtained from TED talks covering a large number of speakers and background noise environments. These videos contain sentences in the English language, which is beneficial for lip-reading and visual speech recognition research. The LRS3 dataset is widely used in research on lip-reading techniques due to its diversity and size. The LRS2 dataset [1] is from BBC, which consists of video, audio, and text for each sample, where the sample rates are 16 kHz for audio and 25 fps for video. The dataset contains more than 1,000 speakers, nearly 150,000 utterance instances, and nearly 63,000 different words, which makes the dataset extremely rich in data. The LRS2 dataset is particularly suitable for studying how to perform effective lip-reading in long video sequences, as it covers continuous natural conversations. The VoxCeleb2 dataset [22] contains over 1 million video clips from YouTube videos. These clips are from over 6,000 celebrities and cover multiple languages, accents, and background noise conditions. VoxCeleb2 is often used to train cross-modal recognition systems, especially those that combine visual and audio information. We only use the English portion of the VoxCeleb2.We detect faces in each image as the Region of Interest (ROI) and then crop the ROI to 96 × 96 pixels. Audio waveforms are preprocessed into mel-spectrograms. For AV-SyncNet, the inputs are lip ROIs for the visual stream and log filterbank energy features for the audio stream. For SyncTalklip, a randomly picked image within the video is used as the identity reference, and the pose reference is the same as the target face image, except that it masks the lower half of the face to prevent our SyncTalklip model from learning movements in the lip region. Based on the pre-trained model, we fine-tune the AV-SyncNet for 180K steps and train SyncTalklip for 60K steps on a single Nvidia 3090 GPU.AV-SyncNet is pre-trained on LRS3 (433h) and VoxCeleb2, and fine-tuned on LRS2 (224h). The model consists of 24 transformer blocks, 1024 embedding dimensions, 4096 feed-forward dimensions, and 16 attention heads. In the pre-training phase, five clusteringmask iterations are conducted as mentioned in Sec. 3.2. In the fine-tuning phase, the model is guided by a multi-task loss function, which includes lip-reading and local-level alignment tasks. For SyncTalklip (conv) model, a convolutional submodule is added for global-level alignment, which can better capture the temporal knowledge and add flexibility to the overall system.SyncTalklip is trained on the LRS2 dataset. The training process is conducted as follows: First, in the video part, the predicted video frames are masked and then fed into the audio and video encoders to generate the corresponding audio and video embeddings respectively. The Generation module generates the predicted frames and calculates the frame-level loss and stream-level loss. On the one Table 2 : Lip-reading effect of AV-SyncNet. AV-SyncNet(conv) denotes the convolution operation applied before calculating the audio and video loss. AV-SyncNet retains a good lipreading capability while aligning the audio and video, and the alignment quantization results are shown in the supplementary.Train AV-Align WER(%)↓AV-HuBERT LRS2(224h) × 25.0 AV-SyncNet LRS2(224h) ✓ 26.0 AV-SyncNet(conv) LRS2(224h) ✓ 26.1 Conformer LRS2+LRS3 × 40.9hand, the generated frames are compared with the corresponding frames of ground truth to compute the adversarial loss and reconstruction loss. On the other hand, it is inserted back into the original video sequence. The generated video stream is input to the trained expert system AV-SyncNet for lip reading task to compute the loss, while in the lip reading task a byproduct is generated, i.e., the corresponding video embedding. Contrastive loss is computed by combining the video embedding here with the audio embedding generated during the generation process. The detailed information of our method can be found in the supplementary.PSNR [31] and SSIM [19, 35] are used to evaluate the quality of the generated frames, LSE-D and LSE-C [24] are used to evaluate the synchronization of audio and video, and WER is used to evaluate the performance of the generated video in the lip reading task.Quality. PSNR and SSIM are used to evaluate the image quality, PSNR is based on the average of the squares of the errors and reflects the magnitude of the difference between the original image and the distorted image, the higher the PSNR the better the quality of the image, SSIM takes into account the structural information of the image, the brightness, the contrast, etc., and the closer its value is to 1 means the higher the quality.Synchronization. The LSE-D has been widely used as a measure of synchronization and employs the mechanism of the sliding window in a geometrical sense, which is a minimum of the Euclidean distance from the audio to the center of the corresponding video window [24] . LSE-C is widely used in previous works [24, 34] .Intelligibility. The word error rate (WER) is used as the evaluation index of speech recognition, which is defined as 𝑊 𝐸𝑅 = (𝑆 + 𝐾 + 𝐼 )/𝑁 , where 𝑆 denotes the number of words replaced, 𝐾 denotes the number of words deleted, 𝐼 denotes the number of words inserted, and 𝑁 denotes the total number of words in the reference text. 𝑊 𝐸𝑅 1 and 𝑊 𝐸𝑅 2 are scored by the AV-HuBERT (large) [26] , which are train on LRS2 and LRS3 seperately, while 𝑊 𝐸𝑅 3 is scored by the Conformer [20, 33] . We finetune the AV-SyncNet on LRS2, the performance of which is shown in Tab. 2. This result shows that it has good retention in lip-reading ability. The trained model is then used in SyncTalklip.In order to verify the reading intelligibility of the generated videos, the generated videos are put to test the lip-reading task on the LRS2 test set using AV-HuBERT to calculate the Word Error Rate (WER). As illustrated in Tab. 1, SyncTalklip demonstrates excellent performance across various key metrics, particularly in LSE-C, LSE-D, and WER. Specifically, SyncTalklip achieves leading LSE-C and LSE-D scores, which proves its superior synchronization capabilities. The PSNR and SSIM metrics are on par with Wav2Lip, which is well-known for its synchronization capabilities, and the scores of our model are closer to the ground truth. Moreover, SyncTalklip achieves the lowest WER among all competing methods when evaluated by AV-HuBERT, which achieves SOTA performance on the lip-reading task. Note that 𝑊 𝐸𝑅 1 and 𝑊 𝐸𝑅 2 for some methods exceed 100%. This is because the evaluator predicts a much longer sentence. To distinguish between the lipreading supervisor in the model and the evaluator used to assess the intelligibility of the generated images, we utilize the Conformer [33] [20] architecture in teacher-forcing mode [36] for assessment, and the results are shown in Tab. 4. These results underscore the ability of SyncTalklip to produce high-quality visual content without sacrificing synchronization. Overall, these results underscore the superiority of SyncTalklip in TFG.As shown in Tab. 3, a series of experiments are conducted to verify the usefulness of the lip-reading module, novel contrastive learning module, and AV-SyncNet. The module ensuring generation quality has been validated for its efficacy [24, 34] . Therefore, we will subsequently conduct ablation studies solely on the modules responsible for synchronization and intelligibility.Ablation on novel contrastive learning module. Using AV-SyncNet model as the supervisor, the contrastive learning module is removed, while the lip-reading module is retained. The experimental results reveal that the videos generated by the model show a significant dip in synchronization and a slight decrease in intelligibility. Furthermore, we still use AV-SyncNet model as the supervisor, but employ traditional contrastive learning instead, and the lip-reading module is retained. The experimental results reveal that the videos Ablation on lip-reading module. Using AV-SyncNet model as the supervisor, the lip-reading module is removed, while the novel contrastive learning module is retained. It shows that this configuration of the model achieves good results in maintaining synchronization and quality. However, it is also worth noting that the model exhibits significant shortcomings in terms of intelligibility.Ablation on AV-SyncNet. Using the AV-HuBERT model as the supervisor, the novel contrastive learning module and lip-reading module are retained. It shows that the videos generated by the model have a slight decrease in all attributes, which proves the significance of AV-SyncNet.These experimental results also demonstrate that relying solely on either module fails to yield the desired outcomes. This highlights the critical need for integrating these modules to produce video content that is high-quality, synchronized, and comprehensible.To trade off between synchronization and intelligibility, we conducted convolution and freeze exploration. As shown in Tab. 4, the models outperform in different aspects. We also performed zero-shot generalizability tests on the LRS3 dataset using the model trained on LRS2.Convolution. During the fine-tuning of AV-SyncNet, we explored a strategy that involves incorporating a convolutional module before computing the distance between vectors. After extensive training, our model has optimized the Word Error Rate (WER) by 1.2% compared to the baseline, as detailed in Tab. 4. However, this improvement in intelligibility comes at the expense of a slight decrease in synchronization and authenticity. The convolutional process shifts the model's focus from local to global features. A larger convolution kernel enables the model to capture more holistic semantic information, prioritizing a deeper understanding of the content over fine-grained details. Conversely, a smaller kernel preserves local nuances, enhancing the quality of individual frames. A larger window size offers profound insights into the overall semantics, but this can result in a lack of detailed one-to-one supervision and lower the single-frame quality. Fundamentally, this represents a balance between a global perspective and local precision within the architecture of the model.Freeze. To further improve the model's performance, we experimented with freezing the AV-SyncNet audio encoder during the training of SyncTalklip. The results are shown in Tab. 4. The experimental results demonstrate that the frozen SyncTalklip achieves a 1.2% improvement in word error rate (WER) compared to the base model, indicating an increase in content comprehensibility. However, there is a noticeable decrease in synchronization. This outcome can be attributed to the freezing operation, which preserves the lip-reading performance of AV-SyncNet, thereby enhancing its capability in this specific aspect. Nevertheless, this preservation also constrains the potential for synchronization improvements due to insufficient supervision of the audio-video synchronization. Additionally, after freezing, the model shifts a greater focus onto Generative Adversarial Networks (GANs), leading to further enhancements in the quality of the generated images. This strategy produces results that excel in terms of intelligibility and generation quality, making it meaningful for improving human-computer interaction experiences in dynamic real-world scenarios.Generalizability on the LRS3. In addition, we evaluated the generalization ability of the models on the LRS3 dataset, which is shown in Tab. 4. Compared to SyncTalklip (freeze), SyncTalklip and SyncTalklip (conv) are more capable of generalization, which means that using the SyncTalklip (freeze) model on zero-shot datasets should be done with caution. The freeze operation inherently limits the number of parameters that can be updated, thus reducing the ability of the model to absorb new information. This reduction in model complexity affects its flexibility, making it less adaptable than its unfrozen counterpart. Interestingly, the freeze operation seems to focus the model's learning on improving the quality of image generation.One hundred videos, randomly selected from real and generated datasets, are evaluated in our study. Each video is assessed by one participant, with a total of twenty participants involved. The scoring range is from one to five. The evaluation covered three main dimensions: video intelligibility, i.e., how clearly the viewer can understand the verbal content in the video; authenticity, which assesses how similar the generated video is to a real human video; and synchronization, i.e., how well the lip movements in the video During the evaluation process, we asked the evaluators to score the videos based on these three dimensions after watching each video sample, so as to comprehensively assess the quality of the videos generated by our method and its performance in different aspects. The experimental results show that our model is superior in perceptual evaluation, which is shown in Tab. 5 .In order to qualitatively evaluate the different approaches, we show two snapshots of the generated speech face videos in Fig. 3 . These snapshots visualize the generation quality of each model, and we encourage to go to the demo page to see a more detailed presentation. Specifically, the first row provides snapshots of the real videos, followed by images synthesized by various methods. Through the figures, we are able to see that the image frames generated by our proposed method are very similar to the real video and perform the best in terms of video quality and naturalness of mouth movements as well. 
conclusion: 
 In this paper, we analyze the challenges faced in the TFG task, and propose a novel SyncTalklip network to synthesize talking face videos with great quality, synchronization, and reading intelligibility. The Experimental results indicate that SyncTalklip surpasses existing methods across all the key performance metrics. A novel cross-modality contrastive learning approach is adopted in our model, which mainly contributes to the synchronization. Besides, AV-SyncNet customized a distance metric for SyncTalklip, whose effectiveness has been demonstrated. Additionally, we introduce variants such as global-level SyncTalklip and AV-SyncNet-frozen SyncTalklip, which excel at different attributes. Based on your needs, you can choose the appropriate model among the three. Moving forward, we aim to further explore audio-video synchronization and content generation technologies to realize more natural and enriched human-computer interaction experiences. Meanwhile, we anticipate that the community will engage in more extensive testing and application of SyncTalklip, thereby driving progress and fostering innovation in related technologies.  
: 
 solution strategy involves integrating an auxiliary network [5] to ensure audio-visual alignment. For generating quality, the strategy involves skip connections [25] and generative adversarial networks (GANs) [3, 9, 39] . The former helps to preserve the temporal dynamics of facial features, while the latter aids in simulating realistic visual output. However, merely achieving high image quality and lip-speech synchronization is not enough to ensure accurate content transmission, because a high-quality image may contain fine-grained lip errors, and precise synchronization may convey incorrect textual content. The McGurk effect [7] points out that when people simultaneously hear and see the semantically mismatched, yet synchronized, sequence of speech and lip movements, they may recognize a phoneme from either the audio or the video, or a fusion of the two. This demonstrates that the audio may mislead you into believing that the generated video is lip-readable. Therefore, to enhance the video's intelligibility, it is essential to detach the audio and separately consider the video's lip-reading performance. Recent work [34] introduces a lip-reading expert to understand content, but sacrifices synchronization and generation quality for intelligibility. Based on this background, our work is dedicated to improving intelligibility while preserving quality and synchronization. Synchronization can be seen as a temporal-level alignment, while intelligibility can be seen as a semantic-level alignment. To accomplish this dual-level alignment, we propose SyncTalklip, which synthesizes the interaction mechanism between visual and auditory information. This approach ensures that the visual representation is not only closely synchronized with the input speech but also more accurately reflects the true meaning of the linguistic content. To address the issue of audio and video features decoupled during fine-tuning for the Visual Speech Recognition (VSR) task [26] , we design AV-SyncNet. It can better encode semantic-aligned embeddings from audio and video modalities. Besides, We design a novel contrastive learning approach to enhance lip-speech synchronization, which shares a joint embedding space with AV-SyncNet.Our main contributions are as follows:• We propose AV-SyncNet to balance synchronization and lip-reading performance, which can better encode semanticaligned embeddings from audio and video modalities. • We introduce SyncTalklip, which applies AV-SyncNet as a supervisor and an initialization of the audio encoder. Additionally, SyncTalklip incorporates a novel cross-modal contrastive learning strategy for fine-grained alignment of audiovideo embeddings. • Experimental results demonstrate that SyncTalklip achieves state-of-the-art performance in quality, intelligibility, and synchronization, yielding speaking faces that highly synchronize audio contents while matching the linguistic message. • We explore a variety of avenues for optimizing the model and share the benchmark codes publicly, thus providing new directions of exploration and a basis for further research. 
 
 Appendix:  


Decision: The authors discuss lip-readable speaker generation using multi-task learning. However, the proposed AV-SyncNet is similar to TalkLip [34], which used a contrastive strategy to enhance lip-speech synchronization by attracting audio embeddings and their time-aligned visual context features while repelling the audio embeddings from different frames. Due to this similarity, the novelty of the proposed method may be diminished, resulting in limited contributions. We suggest the authors revise their paper according to the review feedback.

Reviews: review: paper_summary: paper_summary: The paper address the problem of talking-face generation. The major claim of the paper is to enhance the intelligibility of the model, which is a major drawback in most of the existing talking-face generation works. The idea of using AV-Hubert as an additional cue to understand the linguistic content is interesting and practical. The quantitative results indicate good performance in comparison to existing works on two standard talking-face datasets, however the qualitative results are limited in number and also on the variety of samples shown.
strength_weakness: strength_weakness: - The idea of using AV-Hubert as a supervisor for improving the intelligibity is interesting. The strategy of adding lip-reading loss is simple, yet effective and stronly imporves the performance of the network (in terms of WER).
- The paper is well written and easy to follow.
- Quantitative results are impressive, specifically WER. While the existing models achieve high quality and good synchornization, the ability to understand the underlying content is relatively lower, which is tackled in this work.
limitations: limitations: - Unfreezing AV-SyncNet for lip-reading loss: Authors mention that the AV-SyncNet is frozen to compute the lip-reading loss in SyncTalkLip. What happens if AV-SyncNet is unfrezzed? Does this lead to additional improvements in the performance or it only adds to the overhead in the training without significant gains?
- Qualitative results: 
    - The results presented in the demo page are limited. To better understand the capabilities of the model, it would be good to inlcude more qualitative samples. 
    - Longer samples needed - The current results are very short, now when the major claim of the paper is on intelligibility, it becomes important to judge how the model performs in generating longer sentences as opposed to short phrases like shown currently.
    - More results on real-world examples - The current results shown are from the datasets. However, the real applicability of the model lies in its ability tp perform on real, in-the-wild samples. It would be good to include more such results.
- Improvement in visual quality: The current results from the model are for very low-resolution, 96x96 pixels (can also be seen in qualitative results). However, for the model to be truely application in real settings, it need to perform well on sufficiently higher resolutions. It would be beneficial to experiment on some higher resolution videos and analyse the performance. 

- Minor comments:
    - Typos: 
        - Line 119 - Besides, We design a novel -> No need to capitalalize **W** in "We"
        - Line 289 -  fintune on the downstream ->  **fintuned** on the downstream
    - Improvement in Fig.2 - The cuurent diagram is a bit confusing and hard to understand, the authors can consider re-designing the diagram to make the flow easier.
paper_summary: paper_summary: This paper introduces a novel framework designed to address the challenges of generating high-quality, synchronized, and intelligible talking face videos. The authors highlight the difficulties in balancing quality, synchronization, and intelligibility in existing Talking Face Generation (TFG) systems, which often lead to trade-offs among these aspects.
strength_weakness: strength_weakness: The proposed SyncTalklip framework incorporates a dual-tower architecture that focuses on synchronization and intelligibility. It utilizes a pre-trained multi-task model called AV-SyncNet, which is designed to encode semantic-aligned embeddings from both audio and video modalities. A key innovation of SyncTalklip is the introduction of a novel cross-modal contrastive learning approach that enhances lip-speech synchronization.

The experiments and ablation studies are extensive and representative.
limitations: limitations: 1.The statement of “This approach ensures that the visual representation is not only closely synchronized with the input speech but also more accurately reflects the true meaning of the linguistic content.” is not true. When the audio signal is clean (without background noise, ASR is more accurate than VSR).
2.Talklip [34] also uses the contrastive strategy to enhance lip-speech synchronization by attracting audio embeddings and their time-aligned visual context features while repelling the audio embeddings from different frames. This functionality is very similar to the proposed AV-SyncNet. Thus, I think the authors should better clarify their difference to TalkLip and refine the first contribution point.
paper_summary: paper_summary: The paper introduces SyncTalklip, a novel framework designed to generate talking face videos that achieve a balance between visual quality, synchronization with audio, and lip readability. This is achieved by leveraging a dual-tower framework and a multi-task learning approach. The core innovation is the use of AV-SyncNet, a pre-trained model that facilitates both synchronization and intelligibility, along with a novel cross-modal contrastive learning strategy.

* AV-SyncNet: A pre-trained multi-task model that integrates audio and video synchronization with lip-reading performance.
* SyncTalklip Framework: Incorporates AV-SyncNet as a supervisor and an initialization of the audio encoder, enhancing synchronization and lip-reading accuracy.
* Cross-Modal Contrastive Learning: A novel strategy to bring audio and video embeddings closer, improving synchronization without sacrificing intelligibility.
* Experimental Validation: Demonstrates state-of-the-art performance in quality, intelligibility, and synchronization through extensive experiments.
strength_weakness: strength_weakness: * Novel Framework: The introduction of SyncTalklip addresses a significant gap in the balance between synchronization, quality, and intelligibility in talking face generation.
* Dual-Tower Design: Effective integration of AV-SyncNet ensures robust synchronization and intelligibility.
* Innovative Learning Strategy: The cross-modal contrastive learning approach is a novel contribution that enhances the model's performance in aligning audio and video embeddings.
* Comprehensive Evaluation: The model is extensively tested on multiple datasets (LRS2, LRS3, VoxCeleb2) and shows superior performance compared to existing methods.
limitations: limitations: * Complexity: The dual-tower framework and the multi-task learning approach add complexity to the model, which may make it challenging to implement and require significant computational resources.
* Generalization: While the model shows strong performance on specific datasets, its generalization to entirely new datasets or real-world scenarios may require further validation.
* Limited Focus on Fine-Grained Details: The shift towards global semantic information might result in the loss of some fine-grained details, affecting the overall quality in certain scenarios.
paper_summary: paper_summary: This paper proposes to use AV-SyncNet to enhance both synchronization and intelligibility and analyzes the challenges faced in the TFG task. The authors also propose a cross-modal contrastive learning bringing audio and video closer to enhance synchronization. The experiments are conducted on LRS2 and LRS3 to validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: The paper proposes an auxiliary model (AV-SyncNet) to improve TFG performance, refining a contrastive loss to minimize paired audio-visual feature distance.
limitations: limitations: This paper exhibits significant limitations in terms of problem definition, motivation, and experiments, which are outlined as follows:

1. This paper claims “Recent work [34] introduces a lip-reading expert to understand content, but sacrifices synchronization and generation quality for intelligibility.” in Line 103-105 and “Previous TFG works [34] …” in Line 210-212. However, the reported results from TalkLip [34] indicate that while they improved intelligibility, they also improved synchronization, which conflicts with the authors' description.

2. TalkLip employs AV-Hubert as an auxiliary model to improve intelligibility and synchronization, but it leads to a decline in quality. This paper utilizes a similar model AV-SyncNet and manages to enhance all three aspects simultaneously. In addition to explaining the improvements in intelligibility and synchronization, it should also clarify why quality is also enhanced.

3. In Line 220, the authors claim previous work [34] used cosine similarity to make the audio and video features closer. However, TalkLip [34] employs an infoNCE-based contrastive learning strategy to align audio and video features, rather than cosine similarity. 

4. The reported results in Table 1 of the paper are questionable, as the authors mention utilizing the models from the papers (FaceFormer [8] and MakeItTalk [41]) for testing. However, the results for FaceFormer and Wav2Lip on the LRS2 dataset seem simply copy from those reported in TalkLip [34]. 

5. The Table 1 states that TalkLip was trained on LRS2 (224h), but to my knowledge, TalkLip was trained on LRS2 (29h). Additionally, what does LES2 refer to in the table?

6. How is the dataset divided?

7. The author should include comparisons with commonly used person-generic TFG methods, such as VideoReTalking and IP-LAP.

8. In the paper and supplementary, I cannot find the visualization results for FaceFormer, which is inconsistent with the quantitative results.

9. The comparison in Figure 3 is unfair, for example, as can be seen in the last column of the right figure, there is no short white vertical bar in the bottom left corner of GT, TalkLip, and MakeItTalk, while it appears in wav2Lip and the proposed method.

10. The caption of Figure 1 aims to describe an overview of the TFG task, but the content of the figure does not reflect the process of the TFG task. Instead, it seems more like a description of performance evaluation metrics.

11. "talking face generation" might be a more suitable keyword than "talking head generation," as the latter term does not appear in the paper.

12. There are some issues with the expression of the formulas, such as inconsistencies between the symbols in Formula 3 and their corresponding explanations, lack of explanations for the symbols in Formula 5, and the use of the same symbol ($L_{total}$) in Formulas 6 and 13.

13. The English usage needs to be improved, for instance, the misspelled 'fintune' on Line 289.