Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes

Authors: weifeng  liu
tianyi  she
jiawei  liu
boheng  li
dongyu  yao
ziyou  liang
run  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 12:58:30

Abstract: In recent years, DeepFake technology has achieved unprecedented success in high-quality video synthesis, but these methods also pose potential and severe security threats to humanity. DeepFake can be bifurcated into entertainment applications like face swapping and illicit uses such as lip-syncing fraud. However, lip-forgery videos, which neither change identity nor have discernible visual artifacts, present a formidable challenge to existing DeepFake detection methods. Our preliminary experiments have shown that the effectiveness of the existing methods often drastically decrease or even fail when tackling lip-syncing videos.
In this paper, for the first time, we propose a novel approach dedicated to lip-forgery identification that exploits the inconsistency between lip movements and audio signals. We also mimic human natural cognition by capturing subtle biological links between lips and head regions to boost accuracy. To better illustrate the effectiveness and advances of our proposed method, we create a high-quality LipSync dataset, AVLips, by employing the state-of-the-art lip generators. We hope this high-quality and diverse dataset could be well served the further research on this challenging and interesting field. Experimental results show that our approach gives an average accuracy of more than 95.3% in spotting lip-syncing videos, significantly outperforming the baselines. Extensive experiments demonstrate the capability to tackle deepfakes and the robustness in surviving diverse input transformations. Our method achieves an accuracy of up to 90.2% in real-world scenarios (e.g., WeChat video call) and shows its powerful capabilities in real scenario deployment.
To facilitate the progress of this research community, we release all resources at https://github.com/AaronComo/LipFD.

Main paper: 
 introduction: 
 DeepFake refers to an AI-based technology for synthesizing fake media data [1] . The recent advancements in generative models, particularly the emergence of several GAN architectures [2, 3, 4] and the diffusion probabilistic models [5] , have enhanced the realism and quality of forged videos that can easily deceive humans. The prevalence of DeepFake poses potential security risks, e.g., political elections and identity verification, sparking public concerns [6] .DeepFake can be bifurcated into entertainment applications and illicit uses [7] . As illustrated in Fig. 1 , the popular DeepFake aims to bring fun to users by swapping faces to synthesize new content, such as gender swapping and age regression. Unfortunately, the severe DeepFake is utilized for illicit crimes, including manipulating political propaganda and fabricating pornographic content. The case is particularly alarming in LipSync fraud, where the audio drives the mouth movements on reconstructed video frames. These DeepFakes are generally exploited by malicious actors in real-world scenarios, such as the widely disseminated fabricated videos of Barack Obama saying things he never said on YouTube [8] , posing significant security threats. The escalating issue of real-time forgery necessitates an effective detector to identify videos generated through LipSync.Unlike popular DeepFake which manipulates facial attributes or replaces the entire face, LipSync does not tamper with identity and possesses subtle visual artifacts. More seriously, attackers can adaptively erase these visual artifacts through blurring. Since LipSync follows visual modification driven by audio modality, detecting LipSync forgeries naturally involves spotting the inconsistencies between lips and audio. Whereas the correlation between them is closely tied to individual talking styles, intensifying the challenges in developing a universal model to represent this correlation.Existing studies on DeepFake detection can be classified into unimodal-based and multi-modal-based methods, where the former relies on visual discrepancies arising from identity tampering to detect [9, 10, 11, 12] . However, unimodal detectors become less reliable when the forged videos are perturbed for targeted removal of LipSync artifacts. In recent years, several multi-modal-based methods have emerged [13, 14, 15] , including audio-visual fusion and audio-visual inconsistency. Fusion strategies may confound the learning of singular modality features and the performance postfusion is not necessarily enhanced [16] . [17] suggested training detectors to learn the inconsistencies between video frames and audio. However, as the arms race between DeepFake creation and detection intensifies, these inconsistencies are gradually reduced, making coarse-grained audio-visual alignment strategies less effective against advanced LipSync methods.Lip movements are discrete, while the audio spectrum is continuous, resulting in inherent inconsistencies in LipSync videos. As illustrated in Fig. 2 , we observe a temporal correlation between the energy variations in spectrum and lip movements. To the best of our knowledge, existing works naively align single-frame images with long-range audio clips, thus neglecting the temporal inconsistencies of audio-visual features [18, 19] . Our experimental evidence also indicates a marked decline in the efficacy of existing methods when confronting LipSync forged videos. Moreover, [20] demonstrated the mouth region's significance in facial appearance, surpassing even the eyes, due to its biological connections with other head regions. Humans naturally leverage the cues of local regions and head postures to discern facial semantics. While DeepFake technology has made strides in replicating overall facial dynamics, it often falls short of accurately simulating these subtle yet crucial biological interactions. Hence, we choose to exploit the biologically intrinsic correlation between lip movements and head postures as auxiliary information to detect deepfakes. This approach not only mimics the natural cognitive processes of humans but also capitalizes on the existing limitations of deepfakes. Over time, the energy gradually fades and shifts from middle to lower frequencies. (b) the first two frames show a highlighted high-frequency spectrum, contradicting the man not speaking. In the third frame, an unexpected lip opening appears at the darkest part of the spectrum. The mouth cannot change so drastically within a single frame, and this lip shape contradicts the spectrum information.In this paper, for the first time, we propose LipFD, a pioneering method that leverages the inconsistencies in audio-visual features for the Lip-syncing Forgery Detection. Specifically, our approach captures irregular lip movements that contradict the audio signal aligned with it in the temporal sequence of audio-visual features. We also devise a novel framework that dynamically adjusts the attention of LipFD to regions with different clipping ratios.To evaluate the effectiveness and generalization of our approach in detecting lip-syncing deepfakes, we utilize the state-of-the-art LipSync methods to generate massive high-quality lip forgery video dataset based on Lip Reading Sentences 3 (LRS3) [21] , Face Forensics++ (FF++) [22] , Deepfake Detection Challenge Dataset (DFDC) [1] . Experimental results show that our approach outperforms prior works by a notable margin, with an accuracy up to 96.93% for four types of lip forgery videos. Rigorous ablations of our design choices and comparisons with other detection methods demonstrate the superiority of our approach. Our main contributions can be summarized as follows:• We propose the first-of-its-kind approach dedicated to lip-syncing forgery detection that is often overlooked by existing studies. This method addresses the significant and growing threat of lip-syncing frauds, like those encountered in WeChat video calls.• In this work, we unveil a key insight that exploits the discrepancies between lip movements and audio signals for fine-grained forgery detection. Our approach introduces a dual-headed model architecture to enhance detection capabilities.• We construct the first large scale audio-visual LipSync dataset with up to 340,000 samples, and conducted comprehensive experiments on it alongside other DeepFake datasets. Our method demonstrated high efficacy and robustness, achieving around 95% average accuracy in LipSync detection, and up to 90.18% in real-world scenarios. 
related work: 
 Lip-syncing Generation. Lip-syncing facial manipulation, which forges a speaker's lip movements to match a given audio, is among the most threatening DeepFake applications due to its subtlety and difficulty to detect, typically falsifying the speaker's conveyed information. [23] disentangled the content and speaker information in the audio signal, allowing attackers to generate a forged video using just a single image and an audio segment. Still, it is weak in representing bilabial and fricative sounds due to the omission of short phoneme representations. [24] introduced a well-trained discriminator and a temporal consistency checker to address the loss of short-duration phoneme, enhancing the authenticity of generated videos. However, it exhibits weak temporal coordination in the lip movement of talking heads. [25] further focuses on the content of lip movements, making the forgeries challenging for both human eyes and machines to recognize.DeepFake Detection. The existing DeepFake detectors employ single-modal-based or multi-modalbased approaches to detect subtle differences between real and fake samples. Earlier single-modal detectors aspired to employ neural networks to automatically extract discriminative information [26, 27] , but they failed to detect unseen samples due to overfitting. To address this issue, some studies shift focus to frequency domain features [28, 29] or subtle forgery artifacts in more generalized datasets [30, 31] . Another line is to guide the network to focus on discriminative locations, such as automatically guiding the detector's positional attention through a double-stream network [32] , or manually cropping the lip region to extract artifacts formed by the inconsistent lip movements [33] . Although these works have achieved considerable performance on afore datasets, they are not sensitive when faced with advanced lip-syncing generators due to the absence of synchronized audio features. In the multi-modal-based detectors, noticing that the coordination of audio-visual modalities is an inherently challenging issue in any SOTA generator, [34] quantifies the disparity between audio and visual as the criterion for classification, but focusing too much on the background information in the video led to failure. In this context, [35] intentionally extracts talking head movements and establishes a correlation with audio for discrimination. These methods performed well in addressing audio-visual forgery, but are susceptible to the influence of noise or compression.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: A comprehensive discussion on the limitations is elaborated in E, where we reflect the underlying factors contributing to these limitations and propose potential directions for future improvements. 
lipsync forgery dataset: 
 To the best of our knowledge, the majority of public DeepFake datasets consist solely of videos or images, with no specialized one specifically dedicated to LipSync detection available. To fill this gap, we construct a high-quality Audio-Visual Lip-syncing Dataset, AVLips, which contains up to 340,000 audio-visual samples generated by several SOTA LipSync methods. The workflow is demonstrated in Fig. 3 .High quality. We employed a combination of static 'MakeItTalk' [23] and dynamic 'Wav2Lip' [24] , 'TalkLip' [25] , 'SadTalker' [36] generation methods to simulate realistic lip movements. These methods are widely recognized as high-quality work, capable of generating high-resolution videos while ensuring accurate lip movements. We applied a noise reduction algorithm to all audio samples before synthesis to reduce irrelevant background noise, ensuring the models can focus on speech content.Diversity. Our dataset encompasses a wide range of scenarios, covering not only well-known public datasets but also real-world data. Our aim is for this collection to act as a catalyst for advancing realtime forgery detection. To better simulate the nuances of real-world conditions, we have employed six perturbation techniques -saturation, contrast, compression, Gaussian noise, Gaussian blur, and pixelation -at various degrees, thus ensuring the dataset's realism and practical relevance. 
method: 
 In reality, the movements of a speaker's lip and head are closely intertwined with the spoken content, forming a natural and coherent unity. These physical movements naturally align with the timing and context of the speech. However, LipSync method, which solely relies on audio signals to generate lip movements frame by frame, only focuses on the precise alignment between lip shapes and speech at any given moment. It overlooks the broader temporal context and the overall coherence of lip and head movements during speech. Consequently, the generated outputs often exhibit inherent inconsistencies regarding temporal synchronization. These inconsistencies serve as valuable clues and insights for our detection efforts, highlighting the disparity between natural lip movements and artificially generated ones. Fig. 2 vividly exhibit the temporal features among those two classes.Extracting temporal inconsistencies between audio and video presents notable challenges due to the utilization of features from multiple modalities. To tackle this, we developed a dual-headed detection architecture presented in Fig. 4 . (1) The Global Feature encoder is dedicated to encoding temporal features, capturing the overarching correlation between audio and lip movements. (2) The Global-Region encoder aims to detect subtle visual forgery traces within regions of varying scales and integrate them with global features. (3) Moreover, we introduced an innovative Region Awareness module that dynamically adjusts the model's attention across different scales. We will demonstrate in Sec. 6.1 that this module stands as a cornerstone, harnessing features from regions of diverse sizes, thus empowering our model to effectively capture both the prominent changes in DeepFake and the subtle adjustments in LipSync.Based on the findings mentioned before, we need to extract features in the temporal domain. Inspired by the translation task in natural language processing, where transformers detect long-distance vocabulary correlations, we regard the inherent correlation between lip movements and spectral information as analogous to the relationship between 'vocabulary' in a 'sentence' sequence. To capture and encode this correlation, we employ a transformer model.To effectively carry out its task, the encoder necessitates extraordinary representational capacity, which can be attained through exposure to a vast number of images [37] . This capacity enables the encoder to accurately allocate attention to the relevant regions of interest. To satisfy this requirement, we choose a variant of vision transformer ViT:L/14 [38] , pre-trained on CLIP [39] . In our experiments, we use the final layer of CLIP: ViT-L/14's visual encoder for image embedding.Formulation. We denote the convolutional layer as Conv, which convolves images down to 224 × 224. We first crop source image I into 3 series as{c N h , c N f , c N l } i , i ∈ {0, ..., T -1},where N equals to batch size T notes the window size, c l is the lip region subject to modifications by LipSync, c f represents face area focused on by DeepFake, and c h encompasses the overall zone containing head posture and background information. The three-tiered cropping strategy emulate the human visual focus on key facial areas, spotlighting the lip and overall facial structure. Image I will be embedded into F G as global feature:EQUATIONEQUATIONThe encoder is constrained by L RA that is to be further described in the following section.LipSync tends to concentrate on the lower half of the face. Relying solely on coarse-grained global features is insufficient for representation. Hence, we use local features to better capture forgery traces.Formulation. For each crops c ∈ {c N h , c N f , c N l } i , region feature is defined as F R = E GR (c, F G , θ GR ).We hope this component can focus on the most informative parts of different cropped regions, i.e. lip for c l and head pose for c f , c h . Since lip forgery is often slightly manipulated only on the mouth, the unsupervised model may fail to learn proper representation. We further introduce a region awareness module that applies a modified fully connected layer followed by a sigmoid function, which takes both sub-regions within crops as well as pertinence between region features and their relevant global context into consideration, thus granting different weights to them. The weight is formulated as:EQUATIONwhere c i j denotes the i-th feature in c j and θ RA is the parameters of region awareness module RA(•). The final feature F is obtained by concatenating the global feature F G with three series of region features F R , which represent the relation between temporal features and region visual features:EQUATIONRegion Awareness Loss. We noticed that, regardless of the high-level patterns learned by the model, it is the lower part of the face that matters most [40, 41] . Other extracted information should be served as auxiliary. Hence, we designed L RA , encouraging the region awareness module to focus more on areas that are more frequently modified. Mathematically, the loss is defined as:EQUATIONwhere ω i max is the max weight in feature stacks, ω i h is the none-cropped region. k is a hyper-parameter used to adjust the steepness of the loss. With L RA , we hope the model can focus on areas with a higher probability of being modified, such as the face and lips.According to Eq. 4, the crop with the highest weight exerts dominance over the feature F , indicating that it encapsulates crucial discriminative information for the final detection.Classification. We implement a multi-layer perceptron [42] as our classifier optimized with a Binary Cross-Entropy loss:EQUATIONwhere y means the final predicted label. Finally, the objective is given by: Datasets. We trained our model on Wav2Lip-modified LRS3, a subset of our proposed AVLips. We evaluated our method performance on the following datasets: (1) FF++ [22] , which contains 2,000 samples. (2) DFDC [1] , which has 500 samples. (3) AVLips, our proposed dataset, which includes more than 20,000 samples. Since the baselines we compared against were primarily trained on the FF++ or DFDC datasets, to ensure fairness in the evaluation, we regenerated synthetic data for the first two datasets during the testing phase. This approach aims to maintain consistency and provide a level playing field for a fair comparison of the results.EQUATIONMetrics. Following existing works [43, 37, 44] , we adopt four popular metrics to get a comprehensive performance evaluation of LipFD. Specifically, we report ACC (accuracy), AP (average precision), FPR (false positive rate), and FNR (false negative rate). We use the AUC (area under the curve) as a metric to evaluate the performance in tackling various perturbation attacks.Baselines. We take the SOTA methods in general DeepFake detection and lip-based detection as baselines.(1) For image-based DeepFake detections, UniversalDetect [37] , DoubleStream [32] and SelfBlendedImages [31] are selected. (2) For video-based DeepFake detections, CViT [26] and RealForensics [35] are considered. (3) For the lip-based detection method, we employ the latest LipForensics [33] . Detailed information can be found at our Appendix. B. 
effectiveness evaluation: 
 In evaluating the performance of LipFD in detecting LipSync manipulation and the generation across different forgery techniques as well as obtaining a comprehensive performance evaluation, we use four different metrics to report the detection rate and false alarm rate.Table 1 shows the performance of LipFD and prior works. We take the most advanced general DeepFake detection methods SelfBlendedImages and UniversalFakeDetect, along with representative DoubleStream and CViT video stream detection models as the baseline for DeepFake detection. We also compared our method with the SOTA lip-based method, namely LipForensics, which guides facial judgment through lip pre-reading. In addition, we have also compared the SOTA multi-modal detection method, namely RealForensics. Experimental results demonstrate that LipFD outperforms all competitors to a significant extent with a high detection rate and low false alarm rate in detecting the three DeepFake datasets. Also, we find that LipFD attains a commendable precision, as evident from the AP metric. Furthermore, we observe some discernible patterns from Table 1 .First, advanced manipulations are hard to detect by general methods such as UniversalFakeDetect and SelfBlendedImages, indicating that single-frame-based detectors cannot capture dynamic forgeries.In addition, compared to the SOTA RealForensics method, our ACC exceeded it by 3.49%, 1.7%, and 2.73%, respectively. Similar improvements are reflected in the AP as well. This illustrates that concentrating on lip-syncing allows for the extraction of more potential discriminative features than solely observing lip-based movement.We observe some bad cases from Table 1 . For example, the AP score is 16.27% lower than LipForensics on the FF++ dataset. On the DFDC dataset, our method has an AP lower than LipForensics and RealForensics by 13.01% and 12.14%, respectively. As an explanation, these methods primarily aimed at detecting large-scale manipulations of faces in these types of forgeries. In contrast, LipFD focuses on subtle changes in lip inconsistency. Despite a subtle decrease in balance, LipFD still achieves optimal performance in terms of accuracy.A qualified detector should recognize fake videos generated by unseen methods. We analyze our model's generalizability using the protocol from [43, 37, 44] .Table 2 shows the results of LipFD on various types of methods. Surprisingly, our detector performs even better on data generated by the MakeItTalk method than on the training data itself. This is because MakeItTalk generates dynamic videos by transforming single static images, which inherently lack the coherence of real lip movements. When we use temporal audio-visual information for joint discrimination, it becomes easier to distinguish between real and fake videos. The weights assigned by Region Awareness. A higher value indicates that the corresponding region of the image has a more significant impact on the final feature vector.Robustness analysis aims to evaluate the capability of detectors to withstand common perturbation attacks, as corruptive manipulations on videos are prevalent in the wild, especially in the case of forged videos. Following the setup of RealForensics [35] , we train the model on AVLips without data augmentation and then discuss the robustness of the detectors by testing with unseen samples exposed to a set of perturbations. We investigate the performance of the detectors under six types of perturbation at five varying intensities. We use AUC score as evaluation metric, and the experimental results are presented in Fig. 5 . Evidently, our method outperforms the latest and the best DeepFake detectors RealForensics on most perturbation types.Our approach effectively against saturation and contrast perturbations which performing linear transformations in the HLS space. For compression, LipFD exhibits less corruption under varying levels of quality. Gaussian blur is applied with a fixed kernel size, adjusting the standard deviation for intensity. Both blurring and pixelation significantly degrade detector performance by disrupting high-frequency information.With the advancement of LipSync, certain forgery techniques have been employed for fraudulent purposes. To assess the practicality of our model in real-world scenarios, we conducted experiments across diverse network environments. Our model achieved up to 90.18% accuracy in a network with latency below 100ms which is the common situation of daily life [45, 46, 47] . Results are shown in Fig. 6 . For more details, please refer to our Appendix. D.6 Ablation StudiesQuestion: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the experimental results, as described in supplemental material? Answer: [Yes] .Justification: An anonymized repository link is provided, encompassing comprehensive, impartial, and reproducible code. Within the Sec. Sec. 5 and the Appendix. B, we elucidate settings and parameters to assure verifiability of our outcomes by researchers. Post-acceptance of this paper, the audio-visual dataset will be fully open-sourced. 
core modules: 
 Table 3 shows the overall situation of the experiment. Three significant components, Global feature encoder (E G ), Global-Region encoder (E GR ), and Region Awareness module, are ablated from the network separately, and their respective impact on the overall framework was reflected through changes in accuracy metric.Global-Region encoder. Global-Region encoder takes cropped images and a vector encoded by the Global feature encoder as input, merging them into latent codes representing the correlation between regional parts and temporal sequence. The encoder E GR plays a crucial role in the model. As shown in Table 3 , there is a significant drop in performance when E GR is ablated. In Fig. 7 we visualized the gradients from the last layer of it using Grad-Cam. In the third line with tag 'lip', the area near the lip has the deepest red color representing the highest gradient. The model focuses precisely on the shape of the entire lips. Meanwhile, in the above two lines, the encoder directs its attention to other features, specifically positional information, primarily on the bottom of the heads regardless of real or fake samples, for the reason that LipSync methods predominantly manipulate the bottom parts.Region awareness. The module assign different weights to the feature stack based on their contributions to discriminator. These features are then fused into the final feature, with higher weights EQUATIONFor the majority of forged video clips, the crops tagged as 'lip' are assigned significantly higher weights than other regions, indicating that these parts contain the most crucial contextual information for discrimination. On the contrary, our module leverages more information in larger-scale images ('face' and 'head') to form the latent code.In this section, we give a comprehensive view of the selection of the ViTs regarding different pretrained datasets and structures. Results are demonstrated in Table 4 .With the same architecture, the parameter count has a relatively small impact on final performance. Larger pretrained datasets and more challenging pretraining tasks lead to superior model performance and more balanced recognition capabilities (reflected in small differences in False Positive Rate and False Negative Rate). This aligns with our statement in the paper: 'To effectively carry out its task (capture temporal features), the encoder necessitates extraordinary representational capacity, which can be attained through exposure to a vast number of images'Under the same pretrained dataset, more advanced model architectures typically lead to better final performances. For example, Swin Transformer achieves better results than vanilla ViTs. This is possibly because the window-based approach employed by Swin Transformer is more suitable for capturing long-term dependencies in video data, assisting in better identification of temporal features.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: In the Sec. 5, we present the setup of training and evaluation, along with the details pertaining to the replication of baselines. Comprehensive information regarding hyperparameters, optimizers, and other pertinent details are encompassed within the appendices.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In this paper, we proposed LipFD, the first approach by exploiting temporal inconsistencies between audio and visual to detect lip forgery videos. LipFD demonstrates its efficacy in achieving high detection rates while exhibiting fabulous generalization to unseen data and robustness against various perturbations. We contribute AVLips, a high-quality audio-visual dataset for LipSync detection to the community, aiming to foster advancements in the domain of forged video detection. We hope our study encourages future research on lip-syncing DeepFake detection.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We have employed appropriate statistical tests to ascertain the significance of the differences observed between the methods under comparison. we have included a detailed discussion in the Appendix. B regarding the methodology for calculating these statistical measures, providing a full disclosure of our analytical approaches.Guidelines:• The answer NA means that the paper does not include experiments. Justification: In our manuscript, we detail the actual computational resources required for the experiments, as outlined in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
acknowledgement: 
 This  
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks, We use only standard datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA]Answer: [NA] Justification: Our study does not engage in crowdsourcing Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Figure 1 : A visualization comparison between common deepfakes and our studied lip-syncing deepfakes (LipSync). The former exhibits a substantial forgery area and identity manipulation, such as face or gender swapping, whereas the latter, relies on the synchronization of the minor lip region and given audio, without any alterations to the subject's identity. As illustrated in the comparison above, discerning the authenticity of an image sequence becomes arduous in the absence of labels.In recent years, DeepFake technology has achieved unprecedented success in highquality video synthesis, but these methods also pose potential and severe security threats to humanity. DeepFake can be bifurcated into entertainment applications like face swapping and illicit uses such as lip-syncing fraud. However, lip-forgery videos, which neither change identity nor have discernible visual artifacts, present a formidable challenge to existing DeepFake detection methods. Our preliminary experiments have shown that the effectiveness of the existing methods often drastically decrease or even fail when tackling lip-syncing videos. In this paper, for the first time, we propose a novel approach dedicated to lip-forgery identification that exploits the inconsistency between lip movements and audio signals. We also mimic human natural cognition by capturing subtle biological links between lips and head regions to boost accuracy. To better illustrate the effectiveness and advances of our proposed method, we create a high-quality LipSync dataset, AVLips, 
 
 Appendix: In this section, we will provide a detailed description the features. We have introduced the first audio-visual dataset specifically designed for LipSync detection. The goal of this dataset is to solve the issue of many existing DeepFake datasets lacking audio, while also providing foundational support for the field of lip forgery detection.Dynamic expansion. The raw dataset consists of video files in MP4 format and audio files in WAV format. The videos are manipulated using state-of-the-art LipSync generation methods to forge lip movements. The original dataset can be dynamically expanded up to 50 times its initial size using the provided preprocessing code. The expanded samples are represented in the format shown in Fig. 9 .The randomness algorithm ensures that each expansion generates unique data, ensuring data diversity and providing a convenient data processing approach for temporal detection methods.Real-world simulation. Since our ultimate goal is to achieve real-time detection in the real world, we employed seven perturbation methods listed in Table 1 , introducing various levels of perturbations to the images, to generate a substantial amount of robust training data. In addition, we have collected real-world samples from the internet, encompassing different scenes and varying levels of clarity to further enhance the diversity and realism of the dataset. To ensure a fair comparison, we have collected the only known method that employs lipreading from a high dimensional semantic viewpoint for enhancing DeepFake detection (LipForensics [33]), alongside top-performing recent DeepFake detectors. In addition, to fairly compare the performance of methods integrating both video and audio signals, we have collected the latest method that addresses video artifacts and audio-visual inconsistencies (RealForensics [35] ), which currently stands as the most effective audio-assisted detection method based on our research. The two methods above are both pretrained on FaceForensics++. We evaluated them simultaneously on LipSync and DeepFake detection tasks using well-known datasets FF++, DFDC, and our proposed AVLips.We ensured fair training settings across all baselines, with a batch size set to 32, the optimizer and the learning rate strictly adhering to the original paper's settings (Adam, 1e-3∼1e-6). Specifically: 1) Since LipForensics did not provide training scripts, we directly used the pre-trained checkpoints and performed inference under the same experimental settings. For other models, we fine-tuned pretrained weights on AVLips to meet the same baseline; 2) SelfBlendedImages, as an image-based DeepFake detector, was trained with random frames extracted from videos, following the original paper's settings; 3) UnivefsalFakeDetection, also an image DeepFake detection model, we preprocessed video data into single-frame images for training. Featuring a ResNet core with a Vision Transformer for temporal encoder, our model has approximate 310M parameters pretrained model size, which is comparable to the baselines listed in Table 1 .Due to the vulnerability of videos in the wild to varying degrees of corruptions, it is imperative for detectors not only to possess exceptional generalization capabilities but also to withstand common perturbations to accurately identifying fabricated videos. In this context, we investigate the performance of detectors under seven types of perturbations, each at five different intensity severity.Setup. In our work, conducted without data augmentation, we train on the LRS3, FF++, and DFDC datasets and then expose test samples to previously unseen perturbations to examine the robustness of our detector. These perturbations encompass block-wise distortion, variations in contrast and saturation, blurring, Gaussian noise, pixelation, and video compression. As illustrated in Table 1 , the block-wise changes the number of blocks, with a higher count indicating more severe distortion. Contrast and saturation are manipulated by altering the percentage of chrominance and luminance in video frames, where lower values correspond to greater corruptions. The blurring process entails adjustments to the size of the Gaussian kernel, and Gaussian white noise alters the variance of noise values. Video compression employs a constant rate factor to measure the ratio of video quality to size, with higher values denoting increased compression ratio.Results Analysis. In the absence of any perturbations, the state-of-the-art detector, RealForensics [35] , exhibits performance that is second only to our method. However, Figure 1 shows a significant decline in the performance of RealForensics across the majority of perturbation types, whereas our method remains efficacious under most corruptions. Perturbations involving contrast and saturation engage the percentage of chrominance and luminance in the HLS space, where our detector maintains high AUC values, suggesting an effective retention of detection capabilities in diminished visual quality. However, the detector encounters a moderate decline in performance under conditions of blurring, Gaussian noise, and pixelation, though it still surpasses the RealForensics. This indicates the noise and reduced resolution impact the detector's ability to accurately discern authenticity, potentially due to the refuction of high-frequency information. As for video compression, our approach exhibits remarkable resilience, achieving an average AUC of 0.886, which underscores the capacity of our detector to maintain high performance even when videos are subject to substantial compression, such as in real-world digital communications.To better demonstrate the effectiveness of our proposed method in tackling the real threat of LipSync which is prevalent in the video call or financial frauds, we design and carry out extensive experiments to illustrate its practicability.The quality of video calls and streaming clarity in the real world heavily relies on the quality of the network connection. Numerous applications employ algorithms such as ABR (Adaptive Bitrate) to dynamically adapt the audio and video bitrate and clarity, taking into account the user's network conditions. However, this adaptive process may inadvertently introduce visual blurring and noise to the video. Additionally, network latency results in network jitter and packet loss and disrupts the synchronization between audio and video [48, 49] , posing significant obstacles for accurate LipSync detection in real-world settings.In order to simulate real-world network environments, we conducted experiments using an Android device with root access. Using Android Traffic Control, we imposed strict network conditions on the devices, recording 1-minute English and 1-minute Chinese videos under network latencies of 100ms, 200ms, and 500ms, all at a resolution of 2340 × 1080. Each video was segmented into 5-second clips and dynamically expanded tenfold during preprocessing.Table 6 displays the accuracy of our model in two different real-world scenarios, aligning with the information presented in the line graph in the main text.It is evident that the accuracy of our model in WeChat video calls is generally lower compared to videos. This discrepancy can be attributed to our model's strong reliance on the inconsistency between audio and video. As mentioned in our earlier analysis, as the latency increases, the audio gradually lags behind the video, creating a natural time difference that impedes our model's performance. Conversely, in streaming videos, network latency primarily affects video bitrate and clarity, resulting in blurriness and noise reduction, while not significantly altering the synchronization between audio and video. Consequently, our model exhibits better overall performance in the task of streaming videos as opposed to video calls.Apart from network condition, language also associates with performance. Videos with Chinese language under normal network condition result in much lower accuracy. Chinese and English have distinct pronunciation characteristics. The syllable and phoneme structures in Chinese differ from English. Moreover, Chinese has a flatter intonation pattern, while English exhibits more pitch variation and prosodic contours. These phonetic and prosodic differences can impact linguistic patterns, making the correspondence between lip movements and audio more complex in Chinese videos, thereby reducing the accuracy of LipSync detection, as LipFD relies on the consistency between lip movements and audio spectrum.Our method not only achieved high accuracy on the LRS3, FF++, and DFDC datasets but also demonstrated its effectiveness in real-world evaluations under normal network conditions. However, the performance of our model decreases considerably when faced with Non-English-speaking videos as mentioned in Sec. D.2. So, it is crucial to incorporate multilingual training data to improve its accuracy and robustness in handling diverse linguistic contexts.With the significant advancements in instant communication and large vision models (LVMs), generative models have made progress in achieving real-time cross-language forgery [50, 51] . The necessity of deploying a real-time LipSync detection system has come into the spotlight. Two research directions hold great promise:• Multilingual LipSync Detection. Expanding LipSync detection to include multiple languages is an important area for future exploration. Investigating the challenges and differences in lip movements across various languages can contribute to developing more robust and accurate multilingual LipSync detection models. • Real-Time LipSync Detection. Enhancing LipSync detection algorithms to operate in real-time scenarios is another significant research direction. Real-time detection is crucial for applications such as live streaming and video conferencing. Developing efficient and accurate algorithms that can process and analyze audio and video in real-time will be essential for these applications.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Yes, the main claims articulated in the abstract and introduction accurately encapsulate the paper's seminal contributions and delineate its scope because they concisely summarize research questions addressed, methodologies applied, and key insight, ensuring reviewers understand the paper's significance from the outset.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate 'Limitations' section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory Assumptions and Proofs some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: We strictly adhere to the code of ethics set forth by NeurIPS in all aspects of our research.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: In our discussion, we examine the potential positive societal repercussions of the proposed detection methodology, along with avenues for future development.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper received positive-leaning reviews, ranging from "Borderline accept" to "Weak Accept". The reviewers noted the novel problem of lip forgery detection in DeepFakes, which this paper raises, the dataset proposed to provide foundations for further research, the novelty of the proposed method, and the paper's real-life applications. 

The reviewers were concerned with overfitting and generalization issues due to limited diversity in the fake videos used for training (K6BS). The proposed method's novelty was questioned (Qots). The reviewers found some explanations unclear (9Sa1, Ze8K) and a lack of comprehensive ablation studies was also mentioned (9Sa1, Ze8K). 

These limitations were not seen as reasons for the reviewers to recommend rejecting the paper, and I don't see them as such either.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel method, LipFD, dedicated to detecting lip-syncing forgeries by exploiting temporal inconsistencies between lip movements and audio signals. This unique approach addresses a significant gap in existing DeepFake detection methods. Experimental results demonstrate that LipFD achieves high accuracy across multiple datasets, showcasing its effectiveness and robustness.
strength_weakness: strength_weakness: - The diversity of fake videos in the training set is limited, as it only includes three methods: MakeitTalk, Wav2Lip, and TalkLip. This limitation can lead to overfitting, as the classifier may easily learn the distinct patterns of these methods. For example, Wav2Lip produces blurry lip images and shows obvious artifacts when fusing lip and facial images. To demonstrate generalizability, testing on additional state-of-the-art generation methods is encouraged.
- While the method performs well on the proposed LipSync dataset, there is some variability in performance across different datasets like FF++ and DFDC. This indicates potential limitations in generalizability across diverse datasets, possibly due to the limited variety of fake videos in the training set. A robust model should be capable of detecting both LipSync and general DeepFake videos effectively.
questions: questions: - There is a question about the spectrogram in Figure 2. How is the spectrogram obtained? From Figure 4, it seems to be the audio spectrogram. However, the audio of the fake video is real, so why are there unexpected characters like "the darkest part of the spectrum"?
- What is the meaning of "static" and "dynamic" in Line 122?
- There is a typo: LRS2 in Table 1 should be AVLips.
- Why use MakeitTalk? MakeitTalk generates the whole face instead of only the lip region, which does not align with the definition of LipSync as outlined in this paper.
limitations: limitations: No.
paper_summary: paper_summary: The proposed work introduces a pioneering method for detecting lip-syncing forgery, an often overlooked threat in current research. By leveraging discrepancies between lip movements and audio signals, a dual-headed detection architecture significantly enhances detection accuracy. This work also contributes to the first large-scale audio-visual LipSync dataset, comprising nearly one hundred thousand samples, and conducts extensive experiments that demonstrate our method's efficacy. Results show up to 94% average accuracy in LipSync detection, with robust performance in real-world scenarios.
strength_weakness: strength_weakness: the proposed algoritm, LipFD does not have a strong techincal novelty in learning region and global features from the multi-modal input.
questions: questions: N/A
limitations: limitations: This method might be under-optimized solution for the facial forgery detection.
paper_summary: paper_summary: This paper focuses on a new setting in Deepfake detection called lip-syncing fraud, which only contains fewer minor cues on the leap region. To tackle this issue, the authors provide a novel method called LipFD to obtain the features from both a global view and a regional view. Also, with the new AVLips dataset, this method shows a SOTA result compared to the recent methods.
strength_weakness: strength_weakness: Although the proposed method shows a good result, there are some confused expresses which may bring a hard understanding to readers:
1. For equation 3, what is $RA(\cdot)$ mean? What is $[F_G|\{F_R\}^i_j]$ means? There lack an explanation of these operations.
2. It will be better to have an ablation study on the selection of a vision transformer. Including the pretrain, the structure, etc.
3. It could be better to have more details about the dataset, including the number of samples, the visualization of samples with different methods, etc.
questions: questions: See weakness.
limitations: limitations: No, the author did not include. The authors should address limitations.
paper_summary: paper_summary: This paper tackle deepfake detector problem with audio-visual data focusing on lipsync fake which generally a higher quality fake data. For that, this paper propose a dataset and a method. The dataset (AVLips) is formed using available datasets and 3 methods for the lipsync methods. The method (LipFD) extracts global and local features. For global features, transformer is utilized to get the context from all regions. For local regions, the video is cropped to different face areas: face + background, face, lips areas; and extract feature from each area. Weighting is used to select which cropped areas are more important.
strength_weakness: strength_weakness: 1. Ablation removing one or two out of the 3 branches for local feature extraction is missing. Figure 8 is just showing the important weight extracted from the overall framework doesn't show exactly how much the performance drop if the branches are not included.

2. Details are not clear (see Questions).
questions: questions: 1. Line 267: is there statistics how often the latency below 100ms happen?

2. Does this work utilize audio data or not? Figure 4 bottom-left and Line 83-84 indicate audio data is used but I can't find anything related to audio input in the equations.

3. Eq. (5): Where is j index in the equation? And what is none-cropped region?

4. Line 185: \textit{notice} from where? 

5. Figure 8: why lip is not as important in real data? I assume synchronized lip is also sign of real.
limitations: limitations: I can't find the limitations section. In the checklist, there should be limitation section however the reference to the section is broken.

Comments: title: Thank You for Your Positive Feedback!
Review after Rebuttal
Thank You for Your Positive Feedback!
Review after rebuttal
Thanks to Reviewer 9Sa1
Thanks to Reviewer Qots
Thanks to Reviewer K6BS
Thank You for Your Positive Feedback!
comment: Dear Ethics Reviewer TeZm, we thank you for an insightful comments of our current ethics discussion. We sincerely hope our clarifications can address your concerns. 

**Q1:** The study must clarify its adherence to existing protocols, including IRB/human subjects accreditation, fair compensation, privacy, consent, and fair use/copyright, if collecting new data from human subjects.

**R1:** We recruited 30 university students and teachers as volunteers for a video call experiment in real-world scenarios, obtaining informed consent from all participants prior to the study. The participants were fully aware of the study's purpose, procedure, and potential risks, and explicitly consented to participate in building a real-world audio-visual dataset. Our institution's IRB approved the study, deeming it "minimal risk" and following a similar process to the exempt review in the United States. To ensure participant privacy and confidentiality, we anonymized and securely stored the collected data in accordance with ethical guidelines and relevant regulations, respecting the dataset license terms and providing fair compensation adhering to local regulations.

**Q2:** The study describes using existing datasets (FF++, DFDC) but does not explain its compliance with existing T&C, fair use, or copyright for those datasets -- this is necessary to describe in the paper.

**R2:** We built our AVLips dataset based on LRS2, FF++ and DFDC. 

- The LRS2 dataset is available to download for research purposes under a Creative Commons BY 4.0 license, which grant access for users to share and adapt (including redistribute, remix, transform, and build upon the material). 
- The FF++ dataset is released under the FaceForensics Terms of Use, and the code is released under the MIT license. Researcher shall use the Database for non-commercial research and educational purposes.
- The DFDC dataset follows a custom lisence, allowing researchers to use for non-commercial purposes.

**Q3:** This study describes the application of its methods on multiple languages -- the paper would be stronger and provide the reader more clarity on the attention towards diversity, breadth of application of the method, and performance of the method, if the languages it were tested on were clearly described in the paper.

**R3:** Thank you again for your thoughtful comments. We trained LipFD using AVLips, with the majority of the videos being in English, and a small portion in French, Spanish, and other languages. During testing, we observed that LipFD performed well on languages such as English, French, and Spanish, which belong to the Indo-European language family. However, its performance decreased on unseen Chinese data (which belongs to the Sino-Tibetan family). We speculate that this may be due to significant differences in phonemes and pronunciation rules between the two language families, preventing the model from generalizing effectively (Appendix D.2). To build a multilingual detector, we plan to introduce more diverse language samples in the future to enhance the model’s generalization capabilities.

**Q4:** The study describes having 'data diversity' in its training data -- An attention towards language, representation of facial types (age, sex, race, etc.), topic/format (video style) or origination of the video (location to account for cultural diversity), should be described of all three datasets in use.

**R4:** We would like to assure the reviewer that our training data does not contain or exacerbate bias against people of a certain gender, race, sexuality, or other protected characteristics. To alleviate the reviewer's concerns, we would like to provide additional information on the data diversity of the three datasets used in our study: LRS2, FF++, and DFDC.

- **LRS2 Dataset.**  The LRS2 dataset is the largest publicly available lip reading sentence dataset, comprising facial videos of hundreds of speakers with diverse ages, races, and genders. The majority of the videos are in English, with a small portion in French, Spanish, and other languages. The dataset covers a range of topics, including news and talk shows from BBC programs.

- **FF++ Dataset.** The FF++ dataset is a large-scale face forgery detection dataset, consisting of 977 YouTube videos featuring frontal faces of diverse ages, genders, and races. While the primary language of the dataset is English, it also includes videos in other languages, which helps ensure cultural diversity.

- **DFDC Dataset.** The DFDC dataset, designed by Meta for deepfake research, features 3,500 actors with diverse genders, skin tones, races, ages, and other characteristics. The dataset covers a wide range of topics and includes videos with English, Chinese, Spanish, and Arabic audio.

Our training datasets, LRS2, FF++, and DFDC, provide a diverse range of languages, facial types, and video formats, which helps to ensure that our models are robust and generalizable to different cultural contexts.
Dear Ethics Reviewer Q9nT, we sincerely thank you for your valuable time and comments. We sincerely hope our clarifications and new experiments can address your concerns. 

**Q1:** It appears unclear to what extent this experiment involved collecting new data on human subjects (WeChat calls), and how the experiment was set up: Did the authors conduct the calls? Did they hire subjects? Did they transmit videos? I recommend the authors clarify the setup of the real scenario experiments.

**R1:** Thank you for your valuable time and meaningful ethical review. We hope the following responses can address your concerns.

To verify the effectiveness of our detection method in real-world scenarios, we conducted experiments on WeChat video calls and streaming media detection. We recruited 30 volunteers, including students and teachers aged 20 to 33 from 5 universities worldwide, with an equal gender ratio and fluency in both Chinese and English.

- **Video Call Experiment:** With the volunteers' consent, we asked them to initiate video calls with us. The call videos were then LipSync-forged to obtain data samples. Using Android Traffic Control, we imposed strict network conditions on the devices, recording 1-minute English and 1-minute Chinese videos under network latencies of 100ms, 200ms, and 500ms, all at a resolution of 2340x1080. Each video was segmented into 5-second clips and dynamically expanded tenfold during preprocessing, resulting in $30*2*1*12*10=7200$ samples.
- **Streaming Media Experiment:** Volunteers recorded 1-minute videos speaking in Chinese and English and uploaded them to Bilibili, where the platform automatically adjusted the resolution based on network conditions. We then played these videos under network latencies of 100ms, 200ms, and 500ms while recording the screen and audio to obtain videos with three different resolutions. These were preprocessed as before, yielding 7,200 samples for each resolution. All uploaded videos were deleted from Bilibili afterward to protect the volunteers' privacy.

The entire study received IRB approval from our institution, with procedures similar to an exempt review in the U.S., as it was deemed "minimal risk" by IRB staff. To maximize privacy protection, we will seek volunteers' consent before including their valuable video samples in our AVLips dataset. If any volunteer prefers not to make their data public, their data will be withheld.
Thank you so much for your positive feedback! It encourages us a lot!
Thanks for your nice explanation of my concerns. I think this is good work and I will keep my rate as weakly accept.
Thank you so much for your positive feedback! It encourages us a lot!
Thanks for the authors' detailed feedback. After reviewing the rebuttal, I find that the authors have addressed most of my concerns. Consequently, I have decided to maintain my initial rating.
Dear Reviewer 9Sa1:

Please allow us to sincerely thank you again for reviewing our paper and the valuable feedback, and in particular for recognizing the strengths of our paper in terms of new setting, novel method, new dataset, and high effectiveness.

Please kindly let us know if our response and the new experiments have properly addressed your concerns. We are more than happy to answer any additional questions during the discussion period. Your feedback will be greatly appreciated!

Best,

Paper6542 Authors
Dear Reviewer Qots:

Please allow us to sincerely thank you again for reviewing our paper and the valuable feedback, and in particular for recognizing the strengths of our paper in terms of pioneering method, novel, meaningful and useful research problem, convincing results, and impressive application scenarios.

Please kindly let us know if our response and the clarifications have properly addressed your concerns. We are more than happy to answer any additional questions during the discussion period. Your feedback will be greatly appreciated!

Best,

Paper6542 Authors
Dear Reviewer K6BS:

Please allow us to sincerely thank you again for reviewing our paper and the valuable feedback, and in particular for recognizing the strengths of our paper in terms of novel problem, novel and unique approach, great significance on filling the gap, good writing, and high effectiveness and robustness.

Please kindly let us know if our response and the new experiments have properly addressed your concerns. We are more than happy to answer any additional questions during the discussion period. Your feedback will be greatly appreciated!

Best,

Paper6542 Authors
Thank you so much for your positive feedback! We will ensure all your suggested modifications and additional experiments are properly included in our revision. Thank you again for the precious time and positive feedback. It encourages us a lot!
Thank you authors for the clear rebuttal. I am increasing my rating to weak accept and I hope the paper is modified based on the rebuttal for more clarity to the readers.