Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes
Authors: Weifeng Liu
Tianyi She
Jiawei Liu
Boheng Li
Dongyu Yao
Ziyou Liang
Run Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: DeepFake Detection
LipSync Detection
AI security
Primary area: safety_in_machine_learning

Abstract: In recent years, DeepFake technology has achieved unprecedented success in high-quality video synthesis, but these methods also pose potential and severe security threats to humanity. DeepFake can be bifurcated into entertainment applications like face swapping and illicit uses such as lip-syncing fraud. However, lip-forgery videos, which neither change identity nor have discernible visual artifacts, present a formidable challenge to existing DeepFake detection methods. Our preliminary experiments have shown that the effectiveness of the existing methods often drastically decrease or even fail when tackling lip-syncing videos.
In this paper, for the first time, we propose a novel approach dedicated to lip-forgery identification that exploits the inconsistency between lip movements and audio signals. We also mimic human natural cognition by capturing subtle biological links between lips and head regions to boost accuracy. To better illustrate the effectiveness and advances of our proposed method, we create a high-quality LipSync dataset, AVLips, by employing the state-of-the-art lip generators. We hope this high-quality and diverse dataset could be well served the further research on this challenging and interesting field. Experimental results show that our approach gives an average accuracy of more than 95.3% in spotting lip-syncing videos, significantly outperforming the baselines. Extensive experiments demonstrate the capability to tackle deepfakes and the robustness in surviving diverse input transformations. Our method achieves an accuracy of up to 90.2% in real-world scenarios (e.g., WeChat video call) and shows its powerful capabilities in real scenario deployment.
To facilitate the progress of this research community, we release all resources at https://github.com/AaronComo/LipFD.

Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes
Weifeng Liu'", Tianyi She Jiawei Liu', Boheng Li,, Dongyu Yao", Ziyou Liang', Run Wang" 1.t
Key Laboratory of Aerospace Information Security and Trusted Computing.
Ministry of Education, School of Cyher Science and Engineering, Wuhan University, China Nanyang Technological University
3 Carnegie Mellon University
(weifengliu, tianyishe, jiaweiliu, wangrun) Owhuu edu. cn boheng001 0e ntu. edu.
raindy@cmu.ed edu
Facial-Editing DeepFakee
Face Swapples
Gender Seapping
Age Retresion
"Hello." Fake Vidco
Lip-Syncing Faep
"See it."
Real Video
Figure visualization comparison between common deepfakes and our studied lip-syncing deepfakes (LipSync). The former exhibits substantial forgery and identity manipulation, such as face or gender swapping, whereas the latter, relies on the synchronization of the minor lip region and given audio, without any alterations the subject's identity. As illustrated in the comparison above, discerning the authenticity of an image sequence becomes arduous in the absence of labels.
Abstract
In recent years, DeepFake technology has achieved unprecedented success high- quality video synthesis, but these methods also pose potential and severe security threats to humanity. DeepFake can be bifurcated into entertainment applications like face swapping and illicit uses such as lip-syncing fraud. However, lip-forgery videos, which neither change identity nor have discernible visual artifacts, present formidable challenge existing DeepFake detection methods. Our preliminary experiments have shown that the effectiveness of the existing methods often dras- tically decrease even fail when tackling lip-syneing videos. In this paper, for the first time, we propose novel approach dedicated to lip-forgery identification that exploits the inconsistency between lip movements and audio signals. We also mimic human natural cognition by capturing subtle biological links between lips and head regions boost accuracy. To better illustrate the effectiveness and ad- vances of our proposed method, we create a high-quality LipSyne dataset, AVLips,
"Equal contribution.
t "oorrsppnniing author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

by employing the state-of-the tfr----rttt-tttt- lip generators. We hope this high-quality and diverse dataset could be well served the further research on this challenging and interesting field. Experimental results show that our approach gives an average accuracy of more than 95.3% in spotting lip-syncing videos, significantly out- performing the baselines. Extensive experiments demonstrate the capability to tackle deepfakes and the robustness in surviving diverse input transformations. Our method achieves an accuracy of up to 90.2% in real-world scenarios (e.g., WeChat video call) and shows its powerful capabilities in real scenario deployment. To facilitate the progress of this research community, release all resources at https: //t::://tttbbb com/ AaronComo/LippD
1 Introduction
DeepFake refers to an Al-based technology for synthesizing fake media data 1. The recent advancements in generative models, particularly the emergence of several GAN architectures (2 and the diffusion probabilistic models [5], have enhanced the realism and quality of forged videos that can easily deceive humans. The prevalence of DeepFake poses potential security risks, e.g., political elections and identity verification, sparking public concerns [6).
DeepFake can be bifurcated into entertainment applications and illicit uses 77. As illustrated in Fig. Il the popular DeepFake aims to bring fun to users by swapping faces to synthesize new content, such as gender swapping and age regression. Unfortunately, the severe DeepFake utilized for illicit crimes, including manipulating political propaganda and fabricating pornographic content. The case is particularly alarming in LipSyne fraud, where the audio drives the mouth movements on reconstructed video frames. These DeepFakes are generally exploited by malicious actors in real-world scenarios, such as the widely disseminated fabricated videos of Barack Obama saying things he never said on YouTube [8), posing significant security threats. The escalating issue of real-time forgery necessitates an effective detector to identify videos generated through LipSyne.
Unlike popular DeepFake which manipulates facial attributes or replaces the entire face, LipSync does not tamper with identity and possesses subtle visual artifacts. More seriously, attackers can adaptively erase these visual artifacts through blurring. Since LipSync follows visual modification driven by audio modality, detecting LipSync forgeries naturally involves spotting the inconsistencies between lips and audio. Whereas the correlation between them is closely tied to individual talking styles, intensifying the challenges in developing a universal model to represent this correlation.
Existing studies on DeepFake detection can be classified into unimodal- and multi-modal-baat -based methods, where the former relies on visual discrepancies arising from identity tampering detect
9 10 12]. However, unimodal detectors become less reliable when the forged videos are perturbed for targeted removal of LipSync artifacts. In recent years, several multi-modal-beeee based methods have emerged [13] [1] [5), including audio-visual fusion and audio-visual inconsistency. Fusion strategies may confound the learning of singular modality features and the performance post- fusion is not necessarily enhanced [16. [77 suggested training detectors to learn the inconsistencies between video frames and audio. However, as the arms race between DeepFake creation and detection intensifies, these inconsistencies are gradually reduced, making coarse- -raiied audio-visure alignment strategies less effective against advanced LipSyne methods.
Lip movements are discrete, while the audio spectrum is continuous, resulting in inherent inconsisten- cies in LipSyne videos. As illustrated in Fig. 2 we observe temporal correlation between the energy variations in spectrum and lip movements. To the best of our knowledge, existing works naively align single-frame images with long-range audio clips. thus neglecting the temporal inconsistencies of audio-visual features [18] 19). Our experimental evidence also indicates a marked decline in the
efficacy of existing methods when confronting LipSyne forged videos. Moreover, (20 demonstrated the mouth region's 's significance in facial appearance, surpassing even the eyes, due to its biological connections with other head regions. Humans naturally leverage the cues of local regions and head postures to discern facial semantics. While DeepFake technology has made strides in replicating overall facial dynamics, often falls short of accurately simulating these subtle yet crucial biological interactions. Hence, we choose to exploit the biologically intrinsic correlation between lip movements and head postures as auxiliary information to detect deepfakes. This approach not only mimics the natural cognitive processes of humans but also capitalizes on the existing limitations of deepfakes.
2

---

(3) Real pallern
(b) Fake palien
Figure (a) shows the correlation between lip movements and corresponding spectrogram genuine pattern. When the woman starts talking, the middle and high frequencies in the spectrum lighted. Over time, the energy gradually fades and shifts from middle to lower frequencies. (b) the first two frames show a highlighted high-frequency spectrum, contradicting the man not speaking. In the third frame, an unexpected lip opening appears at the darkest part of the spectrum. The mouth cannot change drastically within a single frame, and this lip shape contradicts the spectrum information.
In this paper, for the first time, propose LipFD, pioneering method that leverages the inconsis- tencies in audio-visual features for the Lip-syncing Forgery Detection. Specifically, our approach captures irregular lip movements that contradict the audio signal aligned with it in the temporal sequence of audio-visual features. We also devise a novel framework that dynamically adjusts the attention of LipFD regions with different clipping ratios.
To evaluate the effectiveness and generalization of our approach in detecting lip-syncing deepfakes, we utilize the state-of-the-art LipSync methods to generate massive high-quality lip forgery video dataset based on Lip Reading Sentences 3 (LRS3) [2I, Face Forensics++ (FF++) [22]. Deepfake
Detection Challenge Dataset (DFDC) [[]. Experimental results show that approach outperforms prior works by a notable margin, with an accuracy up to 96.939 93.93% for four types of lip forgery vidcos. Rigorous ablations of our design choices and comparisons with other detection methods demonstrate the superiority of our approach. Our main contributions can be summarized as follows:
We propose the first-of-its-kind approach dedicated to lip-syncing forgery detection that is often overlooked by existing studies. This method addresses the significant and growing threat of lip-syncing frauds, like those encountered in WeChat video calls.
In this work, we unveil a key insight that exploits the discrepancies between lip movements and audio signals for finegrained forgery detection. Our approach introduces a sual-headed model architecture to enhance detection capabilities.
We construct the first large scale audio-visue visual LipSync dataset with up to 340,000 samples, and conducted comprehensive experiments on it alongside other DeepFake datasets. Our method demonstrated high efficacy and robustness, achieving around 95% average accuracy in LipSync detection, and up to 90. 18% in real-world scenarios.
2 Related Work
Lip-syneing Generation. Lip-syncing facial manipulation, which forges speaker's lip movements to match given audio, is among the most threatening DeepFake applications due to its subtlety and difficulty to detect, typically falsifying the speaker's conveyed information. 23 disentangled the content and speaker information in the audio signal, allowing attackers to generate a forged video using just a single image and an audio segment. Still, it is weak in representing bilabial and fricative sounds due to the omission of short phoneme representations. (24) introduced well-trained discriminator and a temporal consistency checker to address the loss of short-duration phoneme, enhancing the authenticity of generated videos. However, it exhibits weak temporal coordination in the lip movement of talking heads. (25 further focuses on the content of lip movements, making the forgeries challenging for both human eyes and machines to recognize.
DeepFake Detection. The existing DeepFake detectors employ single-modal- -based or multi-modal- based approaches to detect subtle differences between real and fake samples. Earlier single-modal detectors aspired to employ neural networks to automatically extract discriminative information [26] 27], but they failed to detect unseen samples due to overfitting. To address this issue, some
3

---

World Samples
Data Samples
WAV
WAV
MP4
WAV
Target
Audio
Andia
MakeliTalk
TalkLip
Forgery Video Generation
SOTA LipSyne
Data Samples
Figure 3: AVLips dataset construction. Utilizing static and dynamic methods, we generated high- quality videos with realistic lip movements. The diverse dataset includes various real-world scenarios. Perturbations were applied for robust model training.
studies shift focus to frequency domain features 28 29 or subtle forgery artifacts in more generalized datasets [30 31. Another line is to guide the network to focus discriminative locations, such as automatically guiding the detector's positional attention through a double-stream network (32), or manually cropping the lip region extract artifacts formed by the inconsistent lip movements 331. Although these works have achieved considerable performance on afore datasets, they are not sensitive when faced with advanced lip-syncing generators due the absence of synchronized audio features. the multi-modal based detectors, noticing that the coordination of audio-visual modalities an inherently challenging issue in any SOTA generator, [4 quantifies the disparity between audio and visual as the criterion for classification, but focusing too much on the background information in the video led to failure. In this context, (35) intentionally extracts talking head movements and establishes a correlation with audio for discrimination. These methods performed well addressing audio-visual forgery, but are susceptible to the influence of noise or compression.
3 LipSync Forgery Dataset
To the best of our knowledge, the majority of public DeepFake datasets consist solely of videos or images, with specialized one specifically dedicated to LipSync detection available. To fill this gap, we construct high-quality Audio-Visse Lip-syncing Dataset, AVLips, which contains up to 340,000 audio-visual samples generated by several SOTA LipSyne methods. The workflow is demonstrated in Fig. 3
High quality. We employed combination of static 'MakeltTalk' 23 and dynamic 'Wav2Lip' 24), 'TalkLip' (25), 'SadTalker' (36) generation methods to simulate realistic lip movements. These methods widely recognized as high-quality work, capable of generating high-resolution videos while ensuring accurate lip movements. We applied a noise reduction algorithm to all audio samples before synthesis to reduce irrelevant background noise, ensuring the models can focus speech content.
Diversity, Our dataset encompasses a wide range of scenarios, covering not only well-known public datasets but also real-world data. Our aim for this collection to act as a catalyst for advancing real- time forgery detection. To better simulate the nuances of real-world conditions. we have employed six perturbation techniques saturation, contrast, compression, Gaussian noise, Gaussian blur, and pixelation at various degrees, thus ensuring the dataset's 's realism and practical relevance.
Method
In reality, the movements of speaker's lip and head are closely intertwined with the spoken content, forming natural and coherent unity. These physical movements naturally align with the timing and

---

Feaver Stakk
Face Detection & Frame Crop
EGR
Tac
Region
desiiter
Global Feature Encoder
Real
Fake
Figure 4: Overview of LipFD framework. Blue components represent our main modules in LipFD. The input image was generated by pre-processing, which consists of 'I' frames in the target video and their audio spectrogram. (a) The aim of Global Feature Encoder, a self-attention model, is to extract long-term information between video frames and audio, finding unreasonable correspondences between lip movements and audio. (b) ΣGR encodes three series of crops. focusing on different parts for each region, and concatenates them with global feature FG. (c) The Region Awareness module assigns corresponding weights to the features based on their importance. (d) All features are fused together into unified representation F based on their respective weights for final inference.
context of the speech. However, LipSync method, which solely relies audio signals generate lip movements frame by frame, only focuses on the precise alignment between lip shapes and speech any given moment. overlooks the broader temporal context and the overall coherence of lip and head movements during speech. Consequently, the generated outputs often exhibit inherent inconsistencies regarding temporal synchronization. These inconsistencies serve valuable clues and insights for our detection efforts, highlighting the disparity between natural lip movements and artificially generated ones. Fig. vividly exhibit the temporal features among those two classes.
Extracting temporal inconsistencics between audio and video presents notable challenges due to the utilization of features from multiple modalities. To tackle this, developed dual-headed detection architecture presented in Fig. (1) The Global Feature encoder is dedicated to encoding temporal features, capturing the overarching correlation between audio and lip movements. (2) The Global-Region encoder aims to detect subtle visual forgery traces within regions of varying scales and integrate them with global features. (3) Moreover, introduced innovative Region Awareness module that dynamically adjusts the model's attention across different scales. We will demonstrate in Sec. T. that this module stands cornerstone, harnessing features from regions of diverse sizes, thus empowering our model to effectively capture both the prominent changes in DeepFake and the subtle adjustments in LipSyne.
4.1 Global Feature Encoding
Based on the findings mentioned before, we need to extract features in the temporal domain. Inspired by the translation task in natural language processing, where transformers detect long-distance vocabulary correlations, we regard the inherent correlation between lip movements and spectral information as analogous the relationship between 'vocabulary' in 'sentence" sequence. To capture and encode this correlation, we employ a transformer model.
To effectively carry out its task, the encoder necessitates extraordinary representational capacity, which can be attained through exposure to a vast number of images [37]. This capacity enables the encoder accurately allocate attention to the relevant regions of interest. To satisfy this requirement, we choose a variant of vision transformer VIT:L/14 (38), pre-trained on CLIP [39]. In our experiments, use the final layer of CLIP: ViT-L/14's visual encoder for image embedding.
Formulation. denote the convolutional layer as Conw, which convolves images down to 224 x 224. We first crop source image into 3 series {ej ca. ee J., i {0, .O..T-T- 1}. where N
5

---

equals to batch size T' notes the window size, Ci is the lip region subject to modifications by LipSyne, of represents face area focused on by DeepFake, and cb encompasses the overall zone containing head posture and background information. The three-tiered cropping strategy emulate the human visual focus on key facial areas, spotlighting the lip and overall facial structure. Image will be embedded into FG as global feature:
FG ViT(Conw(()
(1)
( e .. Crop(T, (1.0, 0.44)),)), i E {0, 1,
The encoder is constrained by CRA that be further described the following section.
4.2 Region Awareness
LipSync tends to concentrate the lower half of the face. Relying solely on coarse-grained global features is insufficient for representation. Hence, we use local features to better capture forgery traces.
region feature is defined Fn EaR(C, Pa, 0GR). We hope this component can focus on the most informative parts of differ- ent cropped regions, i.e. lip for e and head pose for cf, Ch- Since lip forgery often slightly manipulated only on the mouth, the unsupervised model may fail to learn proper representation. We further introduce a region awareness module that applies a modified fully connected layer followed by sigmoid function, which takes both sub-regions within crops as well pertinence between region features and their relevant global context into consideration, thus granting different weights to them. The weight is formulated as:
Formulation. For each crops E fch,eh ,
of
Wey RA((F(((F)))) Oj E (ch: c1)
where of denotes the i-th feature in ej and ORA is the parameters of region awareness module RA(-). The final feature F is obtained by concatenating the global feature Pa with three series of region features FR. which represent the relation between temporal features and region visual features:
F [uj(we [Fcl(FR)j)
Woj
Region Awareness Loss. We noticed that, regardless of the high-level patterns learned by the model, it is the lower part of the face that matters most [40 41). Other extracted information should be served as auxiliary. Hence, we designed CRA, encouraging the region awareness module to focus more areas that are more frequently modified. Mathematically, the loss is defined as:
k
LRA
(5)
exp(
( maz (w))))
where whan is the max weight feature stacks, w is the none-cropped region. l: is hyper-parameter used to adjust the steepness of the loss. With CRA, we hope the model can focus on areas with higher probability of being modified, such as the face and lips.
4.3 Lip Forgery Detection
According to Eq. 4 the crop with the highest weight exerts dominance over the feature F, indicating that it encapsulates crucial discriminative information for the final detection.
Classification. We implement multi-layer perceptron Cross-Entropy loss:
42 our classifier optimized with Binary
L.s (F)))(F) + v) log(1 -F))
(6)
where means the final predicted label. Finally, the objective is given by:
min w CRA(GRR, ORA) + Cote (0ds)
(7)
0 Oete BOR

---

Table I: Cross-datasets validation. Results on AVLips, FF++ and DFDC are reported, including acc. ap.Jpr fpr, and fnr. The best result is highlighted in bold, while the second-ranking one is underscored. Throughout the entire experiment, the threshold for the AP metric was set 0.5.
AVLs
Ff++
DFDC
Method
ACC
AP
ACC
AP
ACC
AP
CVT
65.54
56.68
0.07
0.61
62.86
54.17
0.24
0.50
70.99
58.06
0.06
0.50
DoubleStream
0.36
0.21
0.24
UniverealFakeDeeitt
50.03
50.02
0.99
0.01
50.43
50.16
0.99
0.01
49.86
49.94
0.98
0.01
SelFlledeeImages
49.99
52.13
64.59
57.93
49.06
RealForrnnics
91.78
90.14
0.02
0.14
93.57
91.32
0.03
0.10
92.54
91.62
0.00
0.15
LipForensics
85.13
81,56
0.18
0.10
94.03
93.25
0.04
0.08
90.75
87.32
0.08
0.11
LiptDD (Ours)
95.27
93.08
0.04
0.04
95.10
76.98
0.06
0.05
94.53
78.61
0.08
0.04
Table 2:
generalisa- tion, Evaluation scores when videos are ex- posed to various unseen forgery algorithms.
Table 3: Overall ablation results regarding core modules. We evaluated our model's performance after removing components listed in the left column.
Companent
ACC
AP
FPR
FNR
AUC
Method
ACC
FPR
FNR
AUC
Global Encoder
95.07
91.81
0.02
0.07
95.09
Wav2Lip (Dynamic)
95.27
0.04
0.04
95.27
Global Region Encoder
72.52
64.38
72.50
MakelTTalk (Static)
96.93
0.02
0.03
96.89
Region Awareness
76.45
72.65
0.38
0.09
76.32
Talkf (Dynamic)
79.33
0.34
0.04
80.36
Foll model
95.27
93.08
0.04
0.04
95.27
5 Experiment
5.1 Setup
Datasets. We trained our model on Wav2Lip-modified LRS3, a subset of our proposed AVLips. evaluated our method performance the following datasets: (1) FF++ [22), which contains 2,000 samples. (2) DFDC [0, which has 500 samples. (3) AVLips, our proposed dataset, which includes more than 20,000 samples. Since the baselines
compared against primarily trained on the
FF++ or DFDC datasets, to ensure fairness the evaluation, we regenerated synthetic data for the first datasets during the testing phase. This approach aims maintain consistency and provide level playing field for a fair comparison of the results.
Metries. Following existing works [3 44) we adopt four popular metrics to get comprehensive performance evaluation of LipFD. Specifically, we report ACC (accuracy), ) AP (average precision), FPR (false positive rate), and FNR (false negative rate). We use the AUC (area under the curve) as metric to evaluate the performance in tackling various perturbation attacks.
Baselines. We take the SOTA methods in general DeepFake detection and lip-based detection as baselines. (1) For image-based DeepFake detections, UniversalDetect (37]. DoubleStream (32) and
SelfBlendedImages (31 are selected. (2) For video-based DeepFake detections, CViT (26 and
RealForensies 355 are considered. (3) For the lip-based detection method, we employ latest
LipForensics
33 Detailed information can be found at our Appendix. B
5.2 Effectiveness Evaluation
In evaluating the performance of LipFD in detecting LipSync manipulation and the generation across different forgery techniques as well as obtaining comprehensive performance evaluation, use four different metrics to report the detection rate and false alarm rate.
Table shows the performance of LipFD and prior works. We take the most advanced general DeepFake detection methods SelfBlendedImages and UniversalFakeDettee along with representative DoubleStream and CViT video stream detection models as the baseline for DeepFake detection. We also compared our method with the SOTA lip-based method, namely LipForensics, which guides facial judgment through lip pre-reading. In addition, we have also compared the SOTA multi-modal detection method, namely RealForensics. Experimental results demonstrate that LipED outperforms all competitors significant extent with high detection rate and low false alarm rate in detecting
7

---

Contrast
Saturation
Blur
8.9
03
8.6
0.5
0.5
93
033
Noise
Pixelation
JPEG Compress
83
0.3
87
0.3
86
84
04
Figure 5: Robustness against various unseen corruptions. Average AUC scores across five intensity levels for various corruptions. For detailed analysis, please refer to the appendix.
WeChat Calls
Stresming
100
90.18
90
81.67
70
71.34
60.98
< 100 MS
-,000 MS
5.000 MS
< 100 MS
200 MS
500 MS
1000 MS
200 MS
560 MS
Figure 6: Performance in real scenarios. The x-axis represents network delay time, where a higher delay indicates degradation in image transmission quality and clarity. Consequently, this degradation adversely impacts the audio-video synchronization in WeChat video calls.
the three DeepFake datasets. Also, we find that LipFD attains a commendable precision, as evident from the AP metric. Furthermore, observe some discernible patterns from Table
First, advanced manipulations are hard to detect by general methods such as UniversalFakeDerrs and SelfBlendedlmagee indicating that single-frame-base detectors cannot capture dynamic forgeries. In addition, compared to the SOTA RealForenniis method, our ACC exceeded it by 3.49%, 1.7%, and 2.73%, respectively. Similar improvements reflected in the AP as well. This illustrates that concentrating on lip-syncing allows for the extraction of more potential discriminative features than solely observing tip-based movement.
We observe some bad cases from Table For example, the AP score 16.27% lower than LipForen- sics on the FF++ dataset. On the DFDC dataset, our method has an AP lower than LipForensics and RealForensies by 13.01% and 12.14% respectively. As an explanation, these methods primarily aimed at detecting large-scale manipulations of faces in these types of forgeries. In contrast, LipFD focuses on subtle changes in lip inconsistency. Despite subtle decrease balance, LipFD still achieves optimal performance in terms of accuracy.
5.3 Generalizability to Unseen Forgery
A qualified detector should recognize fake videos generated by unseen methods. We analyze model's generalizability using the protocol from 44 37 44
Table shows the results of LipFD various types of methods. Surprisingly, our detector performs even better on data generated by the MakeltTalk method than on the training data itself. This because MakeltTalk generates dynamic videos by transforming single static images, which inherently lack the coherence real lip movements. When we use temporal audio-visus information for joint discrimination, becomes easier to distinguish between real and fake videos.

---

100
LII
CDI
1:68
Eam
Content sensitivity. Left is real, right fake. Visualization of the gradients from the last layer of the Global-Region encoder, which reflects the regions LipFD relies on.
Figure 7:
Figure 8: The weights assigned by Region Awareness. A higher value indicates that the corresponding region of the image has more significant impact on the final feature vector.
5.4 Robustness Evaluation
Robustness analysis aims to evaluate the capability of detectors to withstand common perturbation videos prevalent in the wild, especially in the case of
attacks, corruptive manipulations forged videos. Following the setup of RealForensics [35. we train the model on AVL.ps without data augmentation and then discuss the robustness of the detectors by testing with unseen samples exposed to a set of perturbations. We investigate the performance of the detectors under six types of perturbation at five varying intensities. We use AUC score as evaluation metric, and the experimental results are presented in Fig, 5 Evidently, our method outperforms the latest and the best DeepFake detectors RealForensics on most perturbation types.
Our approach effectively against saturation and contrast perturbations which performing linear transformations in the HLS space. For compression, LipFD exhibits less corruption under varying levels of quality. Gaussian blur is applied with fixed kernel size, adjusting the standard deviation for intensity. Both blurring and pixelation significantly degrade detector performance by disrupting high-frequency information.
5.5 Performance in Real Scenarios
With the advancement of LipSync, certain forgery techniques have been employed for fraudulent purposes. To assess the practicality of our model in real-world scenarios, we conducted experiments across diverse network environments. Our model achieved up to 90.18% accuracy in a network with latency below 100ms which is the common situation of daily life 45 46 47]. Results are shown in Fig. For more details, please refer to our Appendix.
6
Ablation Studies
6.1 Core Modules
Table shhows the overall situation of the experiment. Three significant components, Global feature encoder (Eg), Global-Region encoder (EGR), and Region Awareness module, are ablated from the network separately, and their respective impact the overall framework was reflected through changes in accuracy metric.
Global-Region encoder. Global-Region encoder takes cropped images and a vector encoded by the Global feature encoder as input, merging them into latent codes representing the correlation between regional parts and temporal sequence. The encoder EGR plays crucial role in the model. As shown in Table 3] there is a significant drop in performance when ËGR is ablated. In Fig. 7 we visualized the gradients from the last layer of it using Grad-Cam. In the third line with tag 'lip the area near the lip has the deepest red color representing the highest gradient. The model focuses precisely on the shape of the entire lips. Meanwhile, in the above two lines, the encoder directs its attention to other features, specifically positional information, primarily on the bottom of the heads regardless real or fake samples, for the reason that LipSyne methods predominantly manipulate the bottom parts.
Region awareness. The module assign different weights to the feature stack based on their contributions to discriminator. These features are then fused into the final feature, with higher weights

---

Table Performance under different ViT structures. We selected six popular vision transformers
as Global Feature Encoder and tested the final performance of our model.
ViTs
ACC
AP
HPR
FNR
CITP:
95.27
93.08
0.04
0.04
CITB:
95.00
92.05
0.03
0.07
ImageNet 16T//L1
93.28
91.13
0.09
0.04
ImageNet: V1T/B16 ViT//B1
93.27
91.13
0.09
0.04
SmageNet:Swin-
94.66
92.53
0.06
0.04
ImageNet:Swin-S
94.59
90.71
0.02
0.09
indicating greater influence. Representative weights are shown in Fig. normalized as follows:
wi Wh + Wy + wi
Wi E {Wh, w1)
(8)
For the majority of forged video clips, the crops tagged 'lip' are assigned significantly higher weights than other regions, indicating that these parts contain the most crucial contextual information for discrimination. On the contrary, module leverages more information in larger-scale images ('face' and "head") to form the latent code.
6.2 Selection of the Vision Transformers
In this section, give comprehensive view of the selection of the ViTs regarding different pretrained datasets and structures. Results are demonstrated Table 4
With the same architecture, the parameter count has relatively small impact on final performance. Larger pretrained datasets and more challenging pretraining tasks lead to superior model performance and more balanced recognition capabilities (reflected in small differences in False Positive Rate and False Negative Rate). This aligns with our statement in the paper: 'To effectively carry out its task (capture temporal features), the encoder necessitates extraordinary representational capacity, which can be attained through exposure to a vast number of images'
Under the same pretrained dataset, more advanced model architectures typically lead to better final performances. For example, Swin Transformer achieves better results than vanilla ViTs. This is possibly because the window-based approach employed by Swin Transformer is more suitable for capturing long-term dependencies video data, assisting in better identification of temporal features.
Conclusion
In this paper, proposed LipFD, the first approach by exploiting temporal inconsistencies between audio and visual to detect lip forgery videos. LipFD demonstrates its efficacy achieving high detection rates while exhibiting fabulous generalization unseen data and robustness against various perturbations. We contribute AVLips, a high-quality audio-visual dataset for LipSync detection to the community, aiming to foster advancements in the domain of forged video detection. We hope study encourages future research on lip-syncing DeepFake detection.
8 Acknowledgement
This research was supported in part by the National Key Research and Development Program of China under No.2021 YFB3100700 the National Natural Science Foundation of China (NSFC) under Grants No. 62202340. 62372334, the CCF-NSFOCUS 'Kunpeng' Research Fund under No. CCF-NSFOCUS 2023005, the Open Foundation of Henan Key Laboratory of Cyberspace Situation Awareness under No. HNTS2022004, Wuhan Knowledge Innovation Program under No. 2022010801020121 the Fundamental Research Funds for the Central Universities under No. 2042023kf0121.
10

---

References
[1] Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer. The deepfake detection challenge (dfdc) dataset. arXiv preprint arXiv:2006.07397, 2020.
[2] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[3] Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, and Shilei Wen. Stgan: A unified selective transfer network for arbitrary image attribute editing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2019.
[4] Yunjey Choi, Minje Choi, Munyoung Kim. Jung- Woo Ha. Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-imagee translation. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 8789-8797, 2018.
[5] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances nearal information processing systems, 33:6840-6851, 2020.
[6] Run Wang, Ziheng Huang. Zhikai Chen, Li Liu, Jing Chen, and Lina Wang. Anti-forgery: Towards a stealthy and robust deepfake disruption attack via adversarial perceptual- perturbations. arXiv preprint arXiv:2206. 00477, 2022.
[7] Felix Juefei-Xu, Run Wang. Yihao Huang, Qing Guo. Lei Ma, and Yang Liu. Countering malicious deepfakes: Survey, battleground, and horizon. International journal of computer vision, 130(7)1678-1134, 2022.
[8] Supasorn Suwajanakorn, Steven M Seitz, and Ira mamnllmachrr-Shhizermnn Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics (ToG), 36(4):1-13, 2017.
[9] Kevin Lutz and Robert Bassett. Deepfake detection with inconsistent head poses: Reproducibil- ity and analysis. arXiv preprint arXiv: 108. 12715, 2021.
[10] Yiru Zhao, Wanfeng Ge, Wenxin Li, Run Wang, Lei Zhao, and Jiang Ming. Capturing the persistence of facial expression features for deepfake video detection. In Information and Communications Security: 21st International Conference, ICICS 2019, Beijing, China, December 15-17, 2019, Revised Selected Papers 21, pages 630-645. Springer, 2020.
[11] Ziyou Liang, Run Wang, Weifeng Liu, Yuyang Zhang, Wenyuan Yang, Lina Wang, and Xingkai Wang. Let real images be as a judger, spotting fake images synthesized with generative models. arXiv preprint arXiv:2403.1 16513. 2024.
[12] Felix Juefei-Xu, Run Wang, Yihao Huang, Qing Guo, Lei Ma, and Yang Liu. Countering malicious DeepFakes: Survey, battleground, and horizon. International Journal of Computer Vision, 130(7):1678-1733 July 2022.
[13] Davide Cozzolino, Alessandro
Pianese, Matthias Nießner, and Luisa Verdoliva. Audio-visun person-of ofaoon-of-nngg intenterest deepfake detection. In Proceedings the the IEEESCVF Conference on Computer Vision and Pattern Recognition, pages 943 903,952,222 2023.
[14] Wenyuan Yang, Xiaoyu Zhou, Zhikai Chen, Bofei Guo, Zhongjie Ba, Zhihua Xia, Xiaochun Cao, and Kui Ren. Avoid-df: Audio joint learning for detecting deepfake. IEEE Transactions on Information Forensics and Security, 18:2015-2029, 2023.
[15] Ammarah Hashmi, Sahibzada Adil Shahzad, Wasim Ahmad, Chia Wen Lin, Yu Tsao, and Hsin-Min Wang. Multimodal forgery detection using ensemble learning. In 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1532. IEEE, 2022.

---

[16] Hasam Khalid, Shahroz Tariq, Minha Kim, and Simon S Woo. Fakeavceleb: A novel audio- video multimodal deepfake dataset. arXiv preprint arXiv:2108.05088 2021.
[17] Sneha Muppalla, Shan Jia, and Siwei Lyu. Integrating audio-visual features for multimodal deepfake detection. arXiv preprint arXiv:23 200.::233000000 0302277, 2023.
[18] Shruti Agarwal, Hany Farid, Ohad Fried, and Mancesh Agrawala. Detecting deep-fake videos from phoneme-viseme mismatches. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2814-2822, 2020.
[19] Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. Emotions don't lie: An audio-visual deepfake detection method using affective cues. In Proceedings of the 28th ACM international conference on multimedia,, pages 2823-2832, 2020.
[20] Irene Kotsia, Ioan Buciu, and Ioannis Pitas. An analysis of facial expression recognition under partial facial image occlusion. Image and Vision Computing, 26(7):1052-100 1067, 2008.
[21] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lrs3-ted: a large-scale dataset for visual speech recognition. arXiv preprint arXiv: /809,00400 2018.
[22] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics+:: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1-11, 2019.
[23] Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. ACM Transactions On Graphics (TOG), 39(6);1-15, 2020.
[24] KR Prajwal, Rudrabha Mukhopadhyay. Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference multimedia, pages 484-492, 2020.
[25] Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T Tan, and Haizhou Li. Seeing what you said: Talking face generation guided by a lip reading expert. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14653-14662, 2023.
[26] Deressa Wodajo and Solomon Atnafu. Deepfake video detection using convolutional vision transformer. arXiv preprint 1rXi::2102. 2021.
[27] Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi-attentional deepfake detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2185-2194, 2021.
[28] Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu. Spatial-phase shallow learning: rethinking face forgery detection in frequency domain. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 772-781, 2021.
[29] Mengjie Wu, Jingui Ma, Run Wang, Sidan Zhang, Ziyou Liang, Boheng Li, Chenhao Lin, Liming Fang, and Lina Wang. Traceevader: Making deepfakes more untraceable via evading the forgery model attribution. Proceedings of the AAAI Conference on Artificial Intelligence. Mar. 2024.
[30] Shen Chen, Taiping Yao, Yang Chen, Shouhong Ding, Jilin Li, and Rongrong Ji. Local relation learning for face forgery detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 1081-1088 2021.
[31] Kaede Shiohara and Toshihiko Yamasaki. Detecting deepfakes with self-blended images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18729, 2022.
[32] Chao Shuai, Jieming Zhong, Shuang Wu, Feng Lin, Zhibo Wang, Zhongjie Ba, Zhenguang Liu, Lorenzo Cavallaro, and Kui Ren. Locate and verify: A two-stream network for improved deepfake detection. In Proceedings of the 31 ACM International Conference on Multimedia, pages 7131-7142, 2023.
12

---

[33] Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips don't t lie: generalisable and robust upproach to face forgery detection. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 5039-5049, 2021.
[34] Komal Chugh, Parul Gupta, Abhinav Dhall, and Ramanathan Subramanian. Not made for each aiter--udio-visua dissonance-based deepfake detection and localization. In Proceedings of the 28th ACM international conference mullimedia, pages 439-447, 2020.
[35] Alexandros Haliassos, Rodrigo Mira, Stavros Petridis, and Maja Pantic. Leveraging real talking faces via supf-supervision for robust forgery detection. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 14950-14960 2022.
[36] Wenxuan Zhang, Xiaodong Cun, Xuan Wang. Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8652-8661, 2023.
[37] Utkarsh Ojha, Yuheng Li, and Yong Jac Lee. Towards universal fake image detectors that generalize across generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[38] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint oxXiv22010.11922 2020.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[40] Jiazhi Guan, Zhanwang Zhang, Hang Zhou, Tianshu Hu, Kaisiyuan Wang, Dongliang He, Haocheng Feng, Jingtuo Liu, Errui Ding, Ziwei Liu, et al. Stylesync: High-fidelity generalized and personalized lip sync in style-based generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1505-1515, 2023.
[41] Tackyung Ki and Dongchan Min. Stylelipsyne: Style-based personalized lip-sync video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22841-22850, 2023.
[42] Hind Taud and JF Mas. Multilayer perceptron (mlp). Geomatic approaches for modeling land change scenarios, pages 451-455, 2018
[43] Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola. What makes fake images detectable? understanding properties that generalize. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt XXVI 16, pages 103-120. Springer, 2020.
[44] Shichao Dong, Jin Wang, Renhe Ji, Jiajun Liang, Haoqiang Fan, and Zheng Ge. Towards a robust deepfake detector: Common artifact deepfake detection model. arXiv preprint arXiv:2210. 14457, 2022.
[45] Zoom Support. Accessing meeting and phone statistics, 2024.
[46] Microsoft. Media quality and network connectivity performance in microsoft teams, 2021.
[47] Ookla. Speedtest global index fedex-intee index-internet speed around the world speedtest global index, 2024.
[48] Nanditha Rao, A Maleki, F Chen, Wenjun Chen, C Zhang, Navneet Kaur, and Anwar Haque. Analysis of the effect of qos on video conferencing qoe. In 2019 15th International Wireless Communications & Mobile Computing Conference (/WCMC), pages 1267-1272 IEEE, 2019.
[49] Asif Ali Laghari and Mureed Ali Laghari. Quality of experience assessment of calling services in social network. ICT Express, ((2):158-161. 2021.
13

---

[50] Zhenhui Ye, Jinzheng He, Ziyue Jiang, Rongjie Huang. Jiawei Huang, Jinglin Liu, Yi Ren, Xiang Yin, Zejun Ma, and Zhou Zhao. Geneface++: Generalized and stable real-time audio-driven 3d talking face generation. arXiv preprini Xiix:23335007877 2023.
[51] Yongyuan Li, Xiuyuan Qin, Chao Liang, and Mingqiang Wei. Hdtr-net: A real-time high- definition teeth restoration network for arbitrary talking face generation methods. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 89-103. Springer, 2023.
14

---

A AVLips Dataset
In this section, will provide detailed description of the features. We have introduced the first audio-visual dataset specifically designed for LipSync detection. The goal of this dataset is to solve the issue of many existing DeepFake datasets lacking audio, while also providing foundational support for the field of lip forgery detection.
A.1 Features
Dynamic expansion. The raw dataset consists of video files in MP4 format and audio files in WAV format. The videos are manipulated using state-of-the-art LipSync generation methods to forge lip movements. The original dataset can be dynamically expanded up to 50 times its initial size using the provided preprocessing code. The expanded samples are represented the format shown in Fig. 2 The randomness algorithm ensures that each expansion generates unique data, ensuring data diversity and providing a convenient data processing approach for temporal detection methods.
Real-world simulation. Since our ultimate goal achieve real-time detection in the real world, we employed seven perturbation methods listed in Table introducing various levels of perturbations the images, to generate substantial amount of robust training data. addition, we have collected real-world samples from the internet, encompassing different scenes and varying levels of clarity to further enhance the diversity and realism of the dataset.
FF++
DFDC
LRS3
Real-World
Figure 9: Expended data samples. Each sample consists T frames of video images and their corresponding audio spectra, serving as temporal representation of the audio-visual context.
15

---

Block Wise
Contrast
Saturation
Gaussian Blur
Gmussian Noise
Pixelation
Compression
Real
Fake
Ours
0.962
0,831
0,908
0,539
0.587
0.578
0,886
R.F.
0.625
0.563
0.624
0.504
0.454
0.621
0.622
Figure 10: Perturbed samples and average results. Real Fake videos are corrupted using common perturbation methods at intensity level 3, followed by the extraction of video frames obtain samples. Average AUC the evaluation metric, indicating better robustness detectors with higher values. R.F. stands for RealForensics detection.
B Experiment Setup
To ensure a fair comparison, we have collected the only known method that employs lipreading from a high dimensional semantic viewpoint for enhancing DeepFake detection (LipForensics (3)). alongside top-performing recent DeepFake detectors. addition, to fairly compare the performance of methods integrating both video and audio signals, we have collected the latest method that addresses video artifacts and audio-visual inconsistencies (RealForensics most effective audio-assisted detection method based on research. The two methods above are both pretrained on FaceForensics++. We evaluated them simultancously on LipSync and DeepFake detection tasks using well-known datasets FF++, DFDC, and our proposed AVLips
35 which currently stands as the
We ensured fair training settings across all baselines, with a batch size set to 32, the optimizer and the learning rate strictly adhering to the original paper's 's settings (Adam, le-3~1e-6), Specifically: 1) Since LipForensies did not provide training scripts, we directly used the pre-trained checkpoints and performed inference under the same experimental settings. For other models, we fine-tuned pretrained weights on AVL.ps to meet the same baseline; 2) SelfBlendedlmages, as an image-based DeepFake detector, was trained with random frames extracted from videos, following the original paper's settings; 3) UniversalPakeDDttttton. also an image DeepFake detection model, we preprocessed video data into single-fran frame images for training. Featuring ResNet core with a Vision Transformer for temporal encoder, our model has approximate 310M parameters pretrained model size, which is comparable to the baselines listed in Table
C Robustness Evaluation
Due to the vulnerability of videos in the wild to varying degrees of corruptions, it is imperative for detectors not only to possess exceptional generalization capabilities but also withstand com- mon perturbations to accurately identifying fabricated videos. In this context, we investigate the performance of detectors under seven types perturbations, each at five different intensity severity.
Setup. In our work, conducted without data augmentation, train on the LRS3, FF++, and DFDC datasets and then expose test samples to previously unseen perturbations to examine the robustness of our detector. These perturbations encompass block-wise distortion, variations in contrast and saturation, blurring, Gaussian noise, pixelation, and video compression. As illustrated in Table 1, the block-wise changes the number of blocks, with higher count indicating more severe distortion. Contrast and saturation are manipulated by altering the percentage of chrominance and luminance in video frames, where lower values correspond to greater corruptions. The blurring process entails adjustments to the size of the Gaussian kernel, and Gaussian white noise alters the variance of noise
16

---

Table Robustness experiment parameters. Each perturbation method employs five unique sets of hyperparameter values, modifying them solely during the video preprocessing phase.
Severity
Type
Hyperparameter
2
3
4
5
Block-wise
Block number
16
32
48
64
80
Color Contrast
Pixel value
0.85
0.725
0.6
0.475
0.35
Color Saturation
YChCr channel
0.4
0.3
0.2
0.1
0.0
Gaussian Blur
Gaussian kernel size
13
Gaussian Noise
Noise variance
0.001
0.002
0.005
0.01
0.05
Pixelation
Pixelation Level
2
3
4
5
6
Compression
Constant Rate Factor
30
32
35
38
40
Table 6: Evaluation of Real-world scenarios. Detection accuracy under various network delays and languages. CH stands for Chinese, and EN is in short for English.
Latency
100ms
200ms
500ms
Language
CH
EN
EN
EN
WeChat video calls
72.53
81.67
71.34
52.89
Streaming media
74.41
90.18
82.24
60.98
values. Video compression employs constant rate factor to measure the ratio of video quality to size, with higher values denoting increased compression ratio.
Results Analysis. the absence of any perturbations, the state-of-the-ard detector, RealForensics (35, exhibits performance that is second only to our method. However, Figure shows significant decline in the performance of RealForenssis across the majority perturbation types, whereas our method remains efficacious under most corruptions. Perturbations involving contrast and saturation engage the percentage chrominance and luminance in the HLS space, where our detector maintains high AUC values, suggesting effective retention of detection capabiliiies in diminished visual quality. However, the detector encounters a moderate decline in performance under conditions of blurring, Gaussian noise, and pixelation. though it still surpasses the RealForensies. This indicates the noise and reduced resolution impact the detector's 's ability to accurately discern authenticity, potentially due to the refuction of high-frequency information. As for video compression, approach exhibits remarkable resilience, achieving an average AUC 0.886, which underscores the capacity of our detector maintain high performance when videos are subject to substantial compression, such as in real-world digital communications.
D Real-world Scenario
To better demonstrate the effectiveness of our proposed method in tackling the real threat of LipSync which is prevalent in the video call or financial frauds, we design and carry out extensive experiments to illustrate its practicability.
The quality of video calls and streaming clarity the real world heavily relies on the quality of the network connection. Numerous applications employ algorithms such as ABR (Adaptive Bitrate) to dynamically adapt the audio and video bitrate and clarity, taking into account the user's network conditions. However, this adaptive process may inadvertently introduce visual blurring and noise to the video. Additionally, network latency results in network jitter and packet loss and disrupts synchronization between audio and video 48 49), posing significant obstacles for accurate LipSync detection in real-world settings.
D.1 Setup
In order to simulate real-world network environments, we conducted experiments using an Android device with root access. Using Android Traffic Control, we imposed strict network conditions on the
17

---

devices, recording 1-minute English and I-minute Chinese videos under network latencies of 100ms, 200ms, and 500ms, all resolution of 2340 x 1080. Each video was segmented into 5-second clips and dynamically expanded tenfold during preprocessing.
D.2 Performance
displays displays the accuracy of our model in two different real-world scenarios, aligning with the information presented in the line graph in the main text.
It is evident that the accuracy of model in WeChat video calls is generally lower compared to streaming videos. This discrepancy can be attributed to our model's strong reliance on the inconsistency between audio and video. As mentioned in earlier analysis, the latency increases, the audio gradually lags behind the video, creating a natural time difference that impedes our model's performance. Conversely, in streaming videos, network latency primarily affects video bitrate and clarity, resulting in blurriness and noise reduction, while not significantly altering the synchronization between audio and video. Consequently, our model exhibits better overall performance in the task of streaming videos as opposed to video calls.
Apart from network condition, language also associates with performance. Videos with Chinese language under normal network condition result in much lower accuracy. Chinese and English have distinct pronunciation characteristics. The syllable and phoneme structures in Chinese differ from English. Moreover, Chinese has flatter intonation pattern, while English exhibits more pitch variation and prosodic contours. These phonetic and prosodic differences patterns, making the correspondence between lip movements and audio more complex in Chinese videos, thereby reducing the accuracy of LipSyne detection, LipFD relies the consistency between lip movements and audio spectrum.
impact linguistic
E
Discussion and Future Work
Our method not only achieved high accuracy on the LRS3, FF++, and DFDC datasets but also demonstrated its effectiveness in real-world evaluations under normal network conditions. However, the performance our model decreases considerably when faced with Non- Mon-English- English-speaking videos as mentioned in Sec. D.2 So, it is crucial to incorporate multilingual training data to improve its accuracy and robustness handling diverse linguistic contexts.
With the significant advancmments in instant communication and large vision models (LVMs), generative models have made progress achieving real-time cross-language forgery (50, (1). The necessity of deploying a real-time LipSync detection system has come into the spotlight. Two research directions hold great promise:
Multilingual LipSyne Detection. Expanding LipSync detection include multiple languages an important area for future exploration. Investigating the challenges and differences in lip movements across various languages can contribute to developing more robust and accurate multilingual LipSync detection models.
Real-Time LipSyne Detection. Enhancing LipSyne detection algorithms to operate real-tima scenarios is another significant research direction. Real-time detection is crucial for applications such as live streaming and video conferencing. Developing efficient and accurate algorithms that process and analyze audio and video in real-time will be essential for these applications.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Yes, the main claims articulated in the abstract and introduction accu- rately encapsulate the paper's seminal contributions and delineate its scope because they concisely summarize the research questions addressed, methodologies applied, and key insight, ensuring reviewers understand the paper's significance from the outset.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: A comprehensive discussion on the limitations is elaborated in where we reflect the underlying factors contributing these limitations and propose potential directions for future improvements.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate 'Limitations' section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The theoretical derivations presented within the manuscript are substanti- ated with detailed proofs in the Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The manuscript meticulously details the experimental procedures, provid- ing all necessary settings, data, and code critical for the replication of the experiments, thus ensuring the reproducibility of the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
20

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: An anonymized repository link is provided, encompassing comprehensive, impartial, and reproducible code. Within the Sec. Sec. and the Appendix. [B] we elucidate settings and parameters to assure verifiability of our outcomes by researchers. Post-acceptance of this paper, the audio-visual dataset will be fully open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
sublic/guides hateai/ggiiiee//oetttiiisssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In the Sec. 5 we present the setup of training and evaluation, along with the details pertaining to the replication of baselines. Comprehensive information regarding hyperparameters, optimizers, and other pertinent details are encompassed within the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
21

---

Justification: We have employed appropriate statistical tests to ascertain the significance of the differences observed between the methods under comparison. we have included a detailed discussion in the Appendix, B regarding the methodology for calculating these statistical measures, providing a full disclosure of our analytical approaches.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In our manuscript, we detail the actual computational resources required for the experiments, outlined in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that NeurlPS Code of Ethics /ttee//trrre//orrrp...cp/////// htt.c//ppb../tii.c....teeeeeeeee
didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: We strictly adhere to the code of ethics set forth by NeurIPS in all aspects of our research.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
22

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In our discussion, we examine the potential positive societal repercussions of the proposed detection methodology, along with avenues for future development.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeen of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks, We use only standard datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
23

---

Justification:
paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssii thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We furnish the data, pre-trained models, and code through an anonymous repository for accessibility.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Our study does not engage in crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

---

Answer:
[NA]
Justification:
Our study does not engage in crowdsourcing
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "qz2PyAMKkZ", "reviewer": "Reviewer_K6BS", "summary": "The paper introduces a novel method, LipFD, dedicated to detecting lip-syncing forgeries by exploiting temporal inconsistencies between lip movements and audio signals. This unique approach addresses a significant gap in existing DeepFake detection methods. Experimental results demonstrate that LipFD achieves high accuracy across multiple datasets, showcasing its effectiveness and robustness.", "questions": "- There is a question about the spectrogram in Figure 2. How is the spectrogram obtained? From Figure 4, it seems to be the audio spectrogram. However, the audio of the fake video is real, so why are there unexpected characters like \"the darkest part of the spectrum\"?\n- What is the meaning of \"static\" and \"dynamic\" in Line 122?\n- There is a typo: LRS2 in Table 1 should be AVLips.\n- Why use MakeitTalk? MakeitTalk generates the whole face instead of only the lip region, which does not align with the definition of LipSync as outlined in this paper.", "limitations": "No.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper addresses a novel problem by focusing on specific DeepFake types that are challenging to detect with current DeepFake detection algorithms but perform quite well in state-of-the-art models.\n- The paper is well-written and easy to follow. Experimental results indicate the effectiveness of the proposed method.\n- The proposed dataset provides a solid foundation for further research in this field.", "weaknesses": "- The diversity of fake videos in the training set is limited, as it only includes three methods: MakeitTalk, Wav2Lip, and TalkLip. This limitation can lead to overfitting, as the classifier may easily learn the distinct patterns of these methods. For example, Wav2Lip produces blurry lip images and shows obvious artifacts when fusing lip and facial images. To demonstrate generalizability, testing on additional state-of-the-art generation methods is encouraged.\n- While the method performs well on the proposed LipSync dataset, there is some variability in performance across different datasets like FF++ and DFDC. This indicates potential limitations in generalizability across diverse datasets, possibly due to the limited variety of fake videos in the training set. A robust model should be capable of detecting both LipSync and general DeepFake videos effectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer K6BS, we sincerely thank you for your valuable time and comments. We are encouraged by your positive comments on our novel task, interesting idea, good writing and high effectiveness. We sincerely hope our clarifications and new experiments can address your concerns. We are happy to answer more questions and conduct more experiments if needed.\n\n---\n\n**W1:** The diversity of fake videos in the training set is limited which can lead to overfitting. Testing on additional SOTA methods is encouraged.\n\n**R1:** Thank you for the constructive comment! We totally understand your concerns about the diversity of fake videos in our datasets. We hope the following explanations can address your concerns:\n\n- We chose MakeitTalk, Wav2Lip, and TalkLip for evaluation because they are the most representative and influential LipSync methods, covering audio-driven single image talking face generation [1, 2], contrastive learning-based approaches [3, 4], and audio-driven video lip-syncing [5, 6]. Our method, **trained only on Wav2Lip forged data**, performed well on these methods (Table 2).\n- Regarding generalizability, we **retrained LipFD on MakeitTalk, Wav2Lip, and TalkLip**, then evaluated it on the recent SOTA LipSync method, SadTalker. The results suggest our model maintains high transferability to unseen forgery methods. We will evaluate LipFD on more SOTA methods in our revision.\n\n|AUC|AP|FPR|FNR|\n|-|-|-|-|\n|94.53|99.00|0.09|0.01|\n\n[1] Zhang, Wenxuan, et al. \"Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation.\" CVPR. 2023.\n\n[2] Wang et al. Audio2head: Audio-driven one-shot talking-head generation with natural head motion. IJCAI. 2021.\n\n[3] Zhou et al. Pose-controllable talking face generation by implicitly modularized audio-visual representation. CVPR. 2021.\n\n[4] Wang et al. \"Seeing what you said: Talking face generation guided by a lip reading expert.\" CVPR. 2023.\n\n[5] Guan et al. \"Stylesync: High-fidelity generalized and personalized lip sync in style-based generator.\" CVPR. 2023.\n\n[6] Peng et al. \"Synctalk: The devil is in the synchronization for talking head synthesis.\" CVPR. 2024.\n\n---\n\n**W2:** LipFD has some variability in performance across FF++ and DFDC.\n\n**R2:** Thanks for your insightful comment! We hope our following explanations can address your concerns:\n\n- We admit that our model does not achieve the best performance on FF++ and DFDC. This is mainly because (1) Since our method is primarily targeted towards LipSync detection, we only include videos that involve LipSync in our AVLips dataset; (2) our dataset was constructed based on LRS2, which only includes head data, while the fake videos in FF++ and DFDC typically involve the whole body as well as backgrounds. Due to these domain gaps, solely training on our AVLips may lead to suboptimal performances on general DeepFakes.\n- The response to reviewer Ze8K W1 shows that features contained in face and lip regions are both utilized by LipFD, so we believe our model can detect general DeepFakes by simply enriching the training set. To verify this, we fine-tuned LipFD using 400 DFDC samples and tested it on unseen DFDC data. The result shows that LipFD achieves better performance with only a little bit more DeepFake data.\n\n|AUC|AP|FPR|FNR|\n|-|-|-|-|\n|92.50|89.31|0.07|0.07|\n\n---\n\n**Q1:** Fig. 2: how is the spectrum obtained? Why are there unexpected characters like \"the darkest part of the spectrum\"?\n\n**R3:** Thank you for the insightful question. We are deeply sorry that our submission may lead you to some misunderstandings. We hope the following clarifications can address your question:\n- The spectrogram is extracted using the librosa library. It is a standard library for audio processing that is widely utilized by previous works.\n- Please kindly note that we do not mean that the audio of synced video is fake. Typically, if the spectrum of sometime is dark, it means there is a silence or very small volume at the moment [1] (i.e., the person is not talking). However, the LipSync forgery may misallocate lip opening frames (Figure 2b, frame 3) that is contradictory to the spectrum information, leading to temporal inconsistency between audio and video. Our LipFD leverages this inconsistency as a feature to detect LipSync forgeries.\n\n[1] Ilyas, Hafsa, Ali Javed, and Khalid Mahmood Malik. \"AVFakeNet: A unified end-to-end Dense Swin Transformer deep learning model for audio–visual​deepfakes detection.\" Applied Soft Computing 136 (2023): 110124.\n\n---\n\n**Q2:** What is the meaning of \"static\" and \"dynamic\"?\n\n**R4:** Thank you for the insightful question. Dynamic methods refer to those that take a video as input and generate a forged video (e.g., Wav2Lip, TalkLip). Static methods, on the other hand, use a single image as input to generate a video through their models (e.g., MakeItTalk, Wav2Lip with the `--static` flag enabled).\n\n---\n\n**Q3:** There is a typo: LRS2 in Table 1 should be AVLips.\n\n**R5:** Thanks for the helpful comments. We will modify them accordingly and conduct a careful proofreading to avoid other typos in our revision.\n\n---\n\n**Q4:** Why use MakeitTalk?\n\n**R6:** Thanks for the insightful question! We categorize LipSync methods into static (generate from a single image) and dynamic (generate from a video). MakeitTalk, a leading static method, generates realistic lip movements, expressions, and head movements. It is defined as a talking face generation method that includes LipSync [1] and is considered a LipSync method in follow-up papers [2, 3]. Therefore, we include MakeitTalk as a LipSync method in our work.\n\n[1] Zhou et al. \"Makelttalk: speaker-aware talking-head animation.\" TOG. 2020.\n\n[2] Zhang et al. \"Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation.\" CVPR. 2023.\n\n[3] Guan et al. \"Stylesync: High-fidelity generalized and personalized lip sync in style-based generator.\" CVPR. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CnxwfWw7qB", "reviewer": "Reviewer_Qots", "summary": "The proposed work introduces a pioneering method for detecting lip-syncing forgery, an often overlooked threat in current research. By leveraging discrepancies between lip movements and audio signals, a dual-headed detection architecture significantly enhances detection accuracy. This work also contributes to the first large-scale audio-visual LipSync dataset, comprising nearly one hundred thousand samples, and conducts extensive experiments that demonstrate our method's efficacy. Results show up to 94% average accuracy in LipSync detection, with robust performance in real-world scenarios.", "questions": "N/A", "limitations": "This method might be under-optimized solution for the facial forgery detection.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. this work proposes a new research problem -- lip forgery detection, which is meaningful and useful. A dataset for this research problem is also proposed.\n\n2. The anonymous github makes this work very convincing.\n\n3. The real-life applications shown in Fig. 6 is very impressive.", "weaknesses": "the proposed algoritm, LipFD does not have a strong techincal novelty in learning region and global features from the multi-modal input.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Qots, we sincerely appreciate your precious time and valuable comments. Your positive comments of our interesting and relevant topic, clear and simple presentation, novel ideas, convincing experimental evaluation, and impressive application are very encouraging to us. We sincerely hope our following clarifications can address your concerns.\n\n---\n\n**W1:** the proposed algoritm, LipFD does not have a strong techincal novelty in learning region and global features from the multi-modal input.\n\n**R1:** Thanks for your valuable comments. We recognize that we are based on widely used global features and multi-modal inputs. However, our method differs in the following aspects:\n\n- First, we focus on the LipSync forgery detection problem, which is different from what previous literature have explored. While we totally agree that learning region and global features from multi-modal input is a widely used strategy for different tasks [3, 4, 5, 6], we arrived at this method in a motivated, principled way, driven by our novel observations on both global temporal inconsistencies and forgery traces in local regions in LipSync forgeries. These fundamental differences set our method apart from existing techniques, despite we finally converged to similar feature extraction strategies.\n- Regarding local features, previous methods for identifying deepfakes mainly rely on single-frame images, using a single encoder to extract detailed features [1, 2]. We innovatively introduce a novel Region Awareness module to dynamically weight different region features, allowing the model to automatically focus on the more informative parts. Experimental results demonstrate that our strategy achieves excellent results.\n- Regarding global features, previous methods mostly utilize a dual-headed architecture, using separate video and audio encoders to extract features from both modalities. However, this separation of audio and video inherently introduces alignment issues between audio and video features, making it challenging for encoders to accurately align specific video segments with audio segments. We align audio and images at the fine-grained level during the preprocessing stage for the first time and use ViT's patching technique to capture the inconsistencies between the temporal lip movements and audio spectra, achieving good results. \n- We also want to respectfully mention that, beyond LipFD, we also introduced the first large scale audio-visual LipSync dataset into the community, and offered an in-depth investigation on the unique properties of LipSync forgeries. We provide these findings for the first time, and we hope our datasets and analysis can help the community better understand the unique properties and challenges associated with LipSync forgeries. We will add more discussions on this aspect in our revision. \n\nThank you again for the helpful comments.\n\n\n[1] Haliassos, Alexandros, et al. \"Lips don't lie: A generalisable and robust approach to face forgery detection.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[2] Haliassos, Alexandros, et al. \"Leveraging real talking faces via self-supervision for robust forgery detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Oorloff, Trevine, et al. \"AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Wang, Rui, et al. \"AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategies.\" arXiv preprint arXiv:2403.14974 (2024).\n\n[5] Wang, Kai, et al. \"Region attention networks for pose and occlusion robust facial expression recognition.\" IEEE Transactions on Image Processing 29 (2020): 4057-4069.\n\n[6] Peng, Ziqiao, et al. \"Synctalk: The devil is in the synchronization for talking head synthesis.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PX9NQIE1dn", "reviewer": "Reviewer_9Sa1", "summary": "This paper focuses on a new setting in Deepfake detection called lip-syncing fraud, which only contains fewer minor cues on the leap region. To tackle this issue, the authors provide a novel method called LipFD to obtain the features from both a global view and a regional view. Also, with the new AVLips dataset, this method shows a SOTA result compared to the recent methods.", "questions": "See weakness.", "limitations": "No, the author did not include. The authors should address limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This work provides a new setting on Deepfake called AVLips with a large number of high-quality samples.\n2. The method mainly focuses on generating the features from the lips region which is novel.", "weaknesses": "Although the proposed method shows a good result, there are some confused expresses which may bring a hard understanding to readers:\n1. For equation 3, what is $RA(\\cdot)$ mean? What is $[F_G|\\{F_R\\}^i_j]$ means? There lack an explanation of these operations.\n2. It will be better to have an ablation study on the selection of a vision transformer. Including the pretrain, the structure, etc.\n3. It could be better to have more details about the dataset, including the number of samples, the visualization of samples with different methods, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9Sa1, we sincerely thank you for your valuable time and feedback. We are encouraged by your positive comments on our novel explorations, insightgul investigations, extensive experiments, and good motivation. We sincerely hope our following clarifications and new experiments can address your concerns.\n\n---\n\n**W1:** For equation 3, what is $RA(\\cdot)$ mean? What is $[F_G| \\\\{ F_R \\\\} {_j^i}]$ means? There lack an explanation of these operations.\n\n**R1:** Thank you for your valuable comments. $RA(\\cdot)$ is our Region Awareness Module (line 183) that takes $[F_G| \\\\{ F_R \\\\} {_j^i}]$ as input and computes weights based on their content (lines 178-183). $[F_G| \\\\{ F_R \\\\} {_j^i}]$ denotes the global feature ($F_G$, line 169) concatenated with three series of region features ($F_R$, line 175) to represent the relationship between temporal features and region visual features (Fig. 4b). We will add more details in our revision.\n\n---\n\n\n**W2:** It will be better to have an ablation study on the selection of a vision transformer. Including the pretrain, the structure, etc.\n\n**R2:** Thank you for your valuable comments. Following your helpful suggestions, we conducted an experiment regarding different pertrained datasets and structures. \n\n- With the same architecture, the parameter count has a relatively small impact on final performance. Larger pretrained datasets and more challenging pretraining tasks lead to superior model performance and more balanced recognition capabilities (reflected in small differences in False Positive Rate and False Negative Rate). **This aligns with our statement in the paper: \"To effectively carry out its task (capture temporal features), the encoder necessitates extraordinary representational capacity, which can be attained through exposure to a vast number of images\" (lines 158-159).**\n- Under the same pretrained dataset, more advanced model architectures typically lead to better final performances. For example, Swin Transformer achieves better results than vanilla ViTs. This is possibly because the window-based approach employed by Swin Transformer is more suitable for capturing long-term dependencies in video data, assisting in better identification of temporal features. We will further explore more effective model achitectures in our future work.\n\n| ViTs             | ACC   | AP    | FPR  | FNR  |\n| ---------------- | ----- | ----- | ---- | ---- |\n| CLIP:ViT/L14     | 95.27 | 93.08 | 0.04 | 0.04 |\n| CLIP:ViT/B16     | 95.00 | 92.05 | 0.03 | 0.07 |\n| ImageNet:ViT/L16 | 93.28 | 91.13 | 0.09 | 0.04 |\n| ImageNet:ViT/B16 | 93.27 | 91.13 | 0.09 | 0.04 |\n| ImageNet:Swin-B  | 94.66 | 92.53 | 0.06 | 0.04 |\n| ImageNet:Swin-S  | 94.59 | 90.71 | 0.02 | 0.09 |\n\n---\n\n**W3:** It could be better to have more details about the dataset, including the number of samples, the visualization of samples with different methods, etc.\n\n**R3:** Thank you for your valuable feedback. We have provided a detailed description and visualization of the dataset in Appendix. A. Here, we continue to supplement with additional information:\n\n- Data volume:\n    - Videos: 12,000\n    - Audios: 12,000\n- Number of samples: 600,000 (up to a fifty-fold expansion ratio)\n- Forgery methods:\n    - Dynamic: Wav2Lip, TalkLip\n    - Static: MakeItTalk\n- Perturbation methods: blockwise, contrast adjustment, saturation adjustment, Gaussian blur, pixelation, compression\n- Covered scenes:\n    - Common datasets: LRS2, FF++, DFDC\n    - Real-world scenarios: video calls, streaming media\n\nFurther data visualization and statistical analyses will be included in the the revision.\n\n---\n\n**L1:** The authors should address limitations.\n\n**R4:** Thank you for this pointing out! We are deeply sorry for our formatting oversight. We have discussed our limitations in Appendix. E. Here are some more detailed explanations:\n\n- During real-world evaluation of WeChat video calls, we found that LipFD performs slightly worse on Chinese samples compared to English. We believe this is due to pronunciation differences between Chinese and English, as well as our model being trained on a purely English dataset. In future work, we plan to incorporate more Chinese corpus for training to build a general LipSync detector.\n- Our model did not achieve optimal performance on FF++ and DFDC primarily due to being trained solely on the LipSync dataset, which exhibits a significant domain gap with DeepFake. To enhance our performance, we plan to incorporate DeepFake data into our training set to develop a more comprehensive detector.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4yIi2bI104", "reviewer": "Reviewer_Ze8K", "summary": "This paper tackle deepfake detector problem with audio-visual data focusing on lipsync fake which generally a higher quality fake data. For that, this paper propose a dataset and a method. The dataset (AVLips) is formed using available datasets and 3 methods for the lipsync methods. The method (LipFD) extracts global and local features. For global features, transformer is utilized to get the context from all regions. For local regions, the video is cropped to different face areas: face + background, face, lips areas; and extract feature from each area. Weighting is used to select which cropped areas are more important.", "questions": "1. Line 267: is there statistics how often the latency below 100ms happen?\n\n2. Does this work utilize audio data or not? Figure 4 bottom-left and Line 83-84 indicate audio data is used but I can't find anything related to audio input in the equations.\n\n3. Eq. (5): Where is j index in the equation? And what is none-cropped region?\n\n4. Line 185: \\textit{notice} from where? \n\n5. Figure 8: why lip is not as important in real data? I assume synchronized lip is also sign of real.", "limitations": "I can't find the limitations section. In the checklist, there should be limitation section however the reference to the section is broken.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 4, "strengths": "1. Contributions (dataset and method) in addressing lipsync based deepfake look sufficient.\n\n2. Fine-grained features are considered. \n\n3. Analysis in real scenarios is interesting.", "weaknesses": "1. Ablation removing one or two out of the 3 branches for local feature extraction is missing. Figure 8 is just showing the important weight extracted from the overall framework doesn't show exactly how much the performance drop if the branches are not included.\n\n2. Details are not clear (see Questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Ze8K, we are genuinely grateful for your valuable feedback. We sincerely hope our clarifications below can address your concerns. \n\n---\n\n**W1:** Ablation removing one or two out of the 3 branches for local feature extraction is missing.\n\n**R1:** Thank you for your suggestion. Following your comment, we have conducted ablation experiments for the three branch structures.\n\n|Removed|ACC|AP|FPR|FNR|\n|-|-|-|-|-|\n|None|95.27|93.08|0.04|0.04|\n|Head|90.63|84.15|0.00|0.18|\n|Face|86.32|84.12|0.23|0.04|\n|Lip|89.22|87.61|0.19|0.03|\n|Face + Lip|64.15|62.11|0.69|0.03|\n|Head + Lip|82.97|82.39|0.33|0.01|\n|Head + Face|74.05|65.78|0.03|0.47|\n\n- **Head:** Contains features present in real videos. Retaining only this branch increases the FPR as the model classifies more samples as real. This region covers a larger area and contains more coarse-grained features, leading to lower model accuracy.\n- **Face:** Contains features from both real and forged videos, with more real features. It provides rich facial (e.g., expressions) and lip movement information. The model's accuracy is lower when retaining only this branch compared to retaining two branches, showing the need for additional information from other regions.\n- **Lip:** Contains features specific to forged videos. A near-zero FPR shows the model captures LipSync synthesis features well. However, lip features alone don't cover all lipsync dynamics (e.g., rorrelation between lip movements and head poses), and video cropping blurriness can also increase the FNR.\n\nThese three branches significantly impact the model's performance. Experiments show retaining more branches increases accuracy. The Region Awareness dynamically weights three regions, adjusting the model's attention to effectively utilize features from real and fake samples, achieving a balanced performance.\n\n---\n\n**Q1:** Is there statistics how often the latency below 100ms happen?\n\n**R2:** \nThanks for the insightful question. According to statistical data from Akamai and Ookla, average global internet latency for broadband connections often ranges from 30 ms to 60 ms [1]. Practical observations in video conferencing support this:\n\n- Microsoft Teams: Reports network latency around 50 ms to 100 ms in optimal conditions. [2]\n- Zoom: Reports network latency often around 100 ms, with a recommended latency of 150 ms or less to avoid noticeable delays between video and audio. [3]\n\n[1] Ookla. \"Speedtest Global Index – Internet Speed around the world – Speedtest Global Index.\" Speedtest. 2024.\n\n[2] Microsoft. \"Media Quality and Network Connectivity Performance in Microsoft Teams.\" Microsoft. 2021.\n\n[3] Zoom Support. \"Accessing meeting and phone statistics.\" 2024.\n\n---\n\n**Q2:** Does this work utilize audio data or not? \n\n**R3:** Thank you for the insightful question! We followed Yang et al. [1] for data preprocessing to encode both image and audio spectrum. Here's an outline of our process:\n\n- Convert audio to spectrum using librosa.\n- Convert the power spectrum (amplitude squared) to dB units.\n- Calculate the magnification ratio $r$ for spectral alignment based on the number of video frames $N$, pixel length per frame $W$, and spectrum length $L$. Formula: $r=\\frac{N\\cdot W}{L}$.\n- Resize the spectrum and align 5 frames with the corresponding audio spectrum.\n\nAs a result, the input samples ($I$ in Eq. (1) and Image in Fig. 4 bottom-left) contain both video and audio information. We will modify Eq. (1) to make it clearer in our revision.\n\n[1] W. Yang et al., \"AVoiD-DF: Audio-Visual Joint Learning for Detecting Deepfake.\" TIFS. 2023\n\n---\n\n**Q3:** Where is j index in Eq. (5)? What is none-cropped region?\n\n**R4:** Thanks for pointing it out. Here is the revised formula:\n\n\\begin{equation}L_{RA}(\\theta_{GR},\\theta_{RA})=\\sum_{j=1}^N\\sum_{i=1}^{T}\\frac{k}{\\exp([\\omega_j^i]_{max}-[\\omega_j^i]_h)}\\end{equation}\n\nIndex $i$ denotes the frame number in one frame, and index $j$ indicates the sample currently being processed in the batch. We accumulate the losses of the $N$ samples in a batch from $1$ to $N$. \n\nNone-cropped region is equal to \"Head\" region. (i.e., images in Fig 4. head series.)\n\n---\n\n**Q4:** Line 185: notice from where?\n\n**R5:** Our model is primarily designed for detecting LipSync forgery. Since LipSync precisely synthesizes the shape and movement of a person's lips based on a given audio, the alterations are primarily focused on the lower part of the face [1, 2]. Therefore, we artificially constrain the model's focus, directing it to concentrate more on the regions where LipSync makes high-frequency modifications, namely the face and lip regions.\n\n[1] Guan et al. \"Stylesync: High-fidelity generalized and personalized lip sync in style-based generator.\" CVPR. 2023.\n\n[2] Ki et al. \"StyleLipSync: Style-based personalized lip-sync video generation.\" CVPR. 2023.\n\n---\n\n**Q5:** Figure 8: why lip is not as important in real data?\n\n**R6:** \nAs mentioned in response to your W1, both the Head and Face regions contain features necessary for determining real videos. For real samples, the lip region remains important, but its weight decreases due to the increased importance of the facial and head regions. The Region Awareness module assigns higher weights to the Head and Face regions to preserve features specific to real videos. Lip region features are maintained at a similar weight to the Face region to retain traces of forgery.\n\n---\n\n**L1:** I can't find the limitations section.\n\n**R1:** Thank you for pointing this out! We apologize for our formatting oversight. We have discussed our limitations in Appendix E. Here are more details:\n\n- LipFD performs slightly worse on Chinese samples compared to English, likely due to pronunciation differences and training on an English dataset. We plan to include more Chinese data in future work.\n- The model underperformed on FF++ and DFDC due to being trained solely on the LipSync dataset. We plan to add DeepFake data to improve general detection capabilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received positive-leaning reviews, ranging from "Borderline accept" to "Weak Accept". The reviewers noted the novel problem of lip forgery detection in DeepFakes, which this paper raises, the dataset proposed to provide foundations for further research, the novelty of the proposed method, and the paper's real-life applications. 

The reviewers were concerned with overfitting and generalization issues due to limited diversity in the fake videos used for training (K6BS). The proposed method's novelty was questioned (Qots). The reviewers found some explanations unclear (9Sa1, Ze8K) and a lack of comprehensive ablation studies was also mentioned (9Sa1, Ze8K). 

These limitations were not seen as reasons for the reviewers to recommend rejecting the paper, and I don't see them as such either.
Decision: Accept (poster)