X2-DFD: A framework for explainable and extendable Deepfake Detection
Authors: Yize Chen
Zhiyuan Yan
Guangliang Cheng
Kangran Zhao
Siwei Lyu
Baoyuan Wu
Venue: NeurIPS 2025 poster ()
Keywords: Deepfake Detection
Forgery Detection
Fake Image Detection
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose an explainable and extendable framework to enhance deepfake detection via multimodal large-language models.

Abstract: This paper proposes **$\mathcal{X}^2$-DFD**, an **e$\mathcal{X}$plainable** and **e$\mathcal{X}$tendable** framework based on multimodal large-language models (MLLMs) for deepfake detection, consisting of three key stages. 
The first stage, *Model Feature Assessment*, systematically evaluates the detectability of forgery-related features for the MLLM, generating a prioritized ranking of features based on their intrinsic importance to the model.
The second stage, *Explainable Dataset Construction*, consists of two key modules: *Strong Feature Strengthening*, which is designed to enhance the model’s existing detection and explanation capabilities by reinforcing its well-learned features, and *Weak Feature Supplementing*, which addresses gaps by integrating specific feature detectors (e.g., low-level artifact analyzers) to compensate for the MLLM’s limitations.
The third stage, Fine-tuning and Inference, involves fine-tuning the MLLM on the constructed dataset and deploying it for final detection and explanation.
By integrating these three stages, our approach enhances the MLLM's strengths while supplementing its weaknesses, ultimately improving both the detectability and explainability.
Extensive experiments and ablations, followed by a comprehensive human study, validate the improved performance of our approach compared to the original MLLMs.
More encouragingly, our framework is designed to be plug-and-play, allowing it to seamlessly integrate with future more advanced MLLMs and specific feature detectors, leading to continual improvement and extension to face the challenges of rapidly evolving deepfakes. Code can be found on https://github.com/chenyize111/X2DFD.

X2-DFD: A framework for eA eX'plainable and eA'tendable Deepfake Detection
Yize Chen Zhiyuan Yan Guangliang Cheng', Kangran Zhao Siwei Lyu', Baoyuan Wu
1 The Chinese University of Hong Kong, Shenzhen Shenzhen Loop Area Institute School of Electronic and Computer Engineering, Peking University, P.R. China
Department of Computer Science, University of Liverpool, Liverpool, L69 7ZX, UK DDepartment of Computer Science and Engineering, University at Buffalo, State University of New York, Buffalo, NY, USA
Abstract
This paper proposes -DF, an e. e-''lainable and eA'ttndable framework based multimodal large-language models (MLLMs) for deepfake detection, consisting of three key stages (see Figure 1). The first stage, Model Feature Assessment, systematically evaluates the detectability of forgery-related features for the MILM,, generating a prioritized ranking of features based on their intrinsic importance to the model. The second stage, Explainable Dataset Construction, consists of two key modules: Strong Feature Strengthening, which designed to enhance the model's s existing detection and explanation capabilities by reinforcing its well-learned features, and Weak Feature Supplementing. which addresses gaps by integrating specific feature detectors (e.g., low-level artifact analyzers) compensate for the MLLM's limitations. The third stage, Fine-tuning and Inference, involves fine- tuning the MILM the constructed dataset and deploying it for final detection and explanation. By integrating these three stages, our approach enhances the MILM's strengths while supplementing its weaknesses, ultimately improving both the detectability and explainability. Extensive experiments and ablations, followed by comprehensive human study, validate the improved performance of our approach compared to the original MLLMs. More encouragingly, our framework designed be plug-and-play, allowing it to seamlessly integrate with future more advanced MLLMs and specific feature detectors, leading to continual improvement and extension face the challenges of rapidly evolving deepfakes. Code can be found on https: / /ggtttu.... chenyize ihenyizel11/X2DFD.
1 Introduction
Current generative AI technologies have enabled easy manipulation of facial identities, with many applications such as filmmaking and entertainment [52]. However, these technologies can also be misused to create deepfakes" for malicious purposes, including violating personal privacy, spreading misinformation, and eroding trust digital media. Therefore, there is a pressing need to establish reliable and robust system for detecting deepfakes. In recent years, numerous deepfake detection methods have been proposed [36, 45, 88, 32, 6, 60, 77, 73], with the majority focusing on addressing the generalization issue when the manipulation methods between training and testing vary. However,
"Equal contribution, "Corresponding author. "The "deepfake" used here refers explicitly face forgery images videos. Full (natural) image synthesis not strictly within our scope.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Stage-1: Model Feature Assessment
Stage-2: Explainable Dataset Construction
Stage-3: Fine-tuning and Inference
Pergurs Fenture List
Capabilliy Amrsmment
Gencrating Explasssion
Duttact C Corstraciio
Model Fine-taring
SFS
Streeg Facker
MFA
Peplxattc
traggs
KILM
X-DED
WFS
Week Equlaatic
Dyploraiin
X-DDD
Figure 1: High-level overview of our framework, consisting of three key stages: (1) Model Feature Assessment (MFA) evaluates and ranks the forgery-related features (e.g.. blending artifacts) to generate a feature set, (2) Strong Feature Strengthening (SFS) enhances the model's 's strong features for improved detection and explanation, while Weak Feature Supplementing (WFS) leverages Specific Feature Detector (SFD) to compensate the model's weak features, and eventually resulting in an explainable dataset, and (3) The MLLM is fine-tuned using the dataset and then used for inference.
these methods typically only output probability indicating whether given input is Al-generated [8, 82], without providing intuitive and convincing explanations behind the prediction.
Multimodal Large Language Models (MLLMs) have shown remarkable potential in many vision tasks
[70, 76, 52]. Given their strong rision-langual reasoning capabilities, MLLMs offer promising avenue for addressing the explainability gap in visual forgery detection. Recent studies [28, 59, 31, 83] have explored this direction by prompting human annotators or LLMs to describe forgery cues from multiple dimensions, which the MLLMs are then trained to detect. However, these approaches often overlook a key challenge: the reliability of the generated explanations. Due MLLMs' well-documented tendency to hallucinate, especially under uncertain conditions [4], it is crucial to ensure that the models rely on their "familiar" forgery cues with strong discrimination for detection. Intuitively, not all forgery features are equally useful- useful-some some can be effectively leveraged for detection, while others are weakly utilized or ignored altogether.
To investigate this, we conduct a comprehen- sive analysis of how well pre-frained MLLMs can utilize various forgery-related cues. As shown in Figure 2, certain cues exhibit strong de- tection performance (e.g., facial structures and skin tone), whereas others offer limited discrimi- native value (e.g., blending artifacts and lighting inconsistencies). When cue is unfamiliar or ineffective for the model, explanations based on become unreliable. In contrast, cues that align well with the model's capabilities produce more robust and trustworthy explanations. Therefore, to ensure reliable explanations, it is essential to explicitly identify and promote cues that the MLLM can reliably understand and leverage.
Semontle
Capabiliit
Signal
Feature Index
Skin Kon0
Conteut
Lagont
Lighting
Blending
Shadow
Figure 2: The diagram shows that pretrained mod- els (e.g.. LLaVa) effectively distinguish real from fake content using semantic features (e.g., Skin tone, Contour), but perform poorly with signal features (c.g., Blending, Lighting).
Inspired by the above investigations, we propose 2DFDD a novel framework that utilizes MILMS for deepfake detection. The key idea of our approach is to enhance the strengths and supplement the weaknesses of the original MLLMs. Our framework operates through three core stages. First, the Model Feature Assessment (MFA) assesses the intrinsic capability of the original MLLMs in deepfake detection. This stage quantifies the discriminative capability of each forgery- related feature, producing a prioritized ranking based on its importance to the model. Second, the Strong Feature Strengthening (SFS) and Weak Feature Supplementing (WFS) reinforce strong features and compensate for weak ones, resulting in a more explainable dataset. Third, we use the created dataset from the second stage to fine-tune the MILM and then use for improved detection and explanation. This integration enables us to leverage the strengths of both MI.LMs and Specific Feature Detectors (SFDs) effectively and fuse the large and small models adaptively. Encouragingly, the modular-base based design of the proposed DDD framework enables seamless integration with future MLLMs and SFDs as their capabilitiees evolve.
Our main contributions are threefold.
We systematically assess the intrinsic capabilities of MILMSs for deepfake detection: To our knowledge, we are the first work to provide an in-depth analysis of MLLMs' inherent ability in

---

deepfake detection. Our findings reveal that MLLMs exhibit varying discrimination capabilities across different forgery features.
We enhance MLLMs' explainability by reinforcing their strong features: Building on their strengths, we fine-tune MLLMs to generate explanations based on their most "familiar" forgery features, improving both detection accuracy and explainability.
We further integrate Specific Feature Detectors to supplement the model's weakness, For forgery features where MLLMs struggle, we incorporate SFDs to complement their limitations, creating a more robust detection system.
2 Related Work
Conventional Deepfake Detection Early detection methods typically focus on performing feature engineering to mine manual feature such as eye blinking frequency [35], warping artifacts [36]. headpose [8 and etc. Recent conventional deepfake detectors mainly focus on dealing with the issue of generalization [78], where the distribution of training and testing data varies. Until now, they have developed novel solutions from different directions: constructing pseudo-fake samples to capture the blending clues [36, 32, 61, 89], learning firtiilly anomalies [23,, 45, 47, 53], focusing on the ID inconsistency clues between fake and corresponding real [17], performing disentanglement learning to learn the forgery-related features [77, 80, 22], performing reconstruction learning to learn the general forgery clues [5, 66), locating the peaporlltempora inconsistency [24, 68, 90, 79, 85], and etc. However, these methods can only provide real or fake predictions without providing detailed explanations. The lack of convincing and compe-compeeeensibll explanations might confuse users
about why the predictions deemed fake.
Deepfake Detection via Multimodal Large Language Model Vision and language are the two important signals for human perception, and visual- languae multimodal learning has thus drawn lot of attention in the AI community. Recently, the LLaVA series [44, 43, 42] have explored simple and effective approach for visual-language multimodal modeling. In the field of deepfake detection, [28, 59] have investigated the potential of prompt engineering in face forgery analysis and proposed that existing MILMS show better explainability than previous conventional deepfake detectors. In addition, [34, 31] probed different MLLMs for explainable fake image detection and [34, 69] by presenting a labeled multimodal database for fine-tuning. In parallel, VIPGuard [40] explores explainable deepfake detection by leveraging identity information through an MLLM. More recently, [87] proposed using pairs of human-generated visual questions answering (VQA) construct the fine-tuning dataset, but manually creating detailed annotations can be very costly. Addressing this limitation, [27] recently introduced an automated pipeline using GPT-4o [1] generate VQA pairs for dataset construction and MLLM training. However. new critical question was then raised: Can MLLMs (e.g., LLaVa) fully comprehend the fake clues identified by GPT-4o? We argue that there could remain a "capability gap" between different MILMs, particularly between "annotation generators" (GPT-4o) and "consumer models" (LLaVA). This gap exposes two unresolved challenges: (1) systematically analyzing the limitations of MLLMM-asse detectors in understanding all synthetic forgery clues (e.g.. identifying specific detection capabilities they lack) and (2) developing methods to enhance their existing strengths (e.g.. semantic consistency analysis) while compensating for weaknesses (e.g., fine-grained artifact recognition). To our knowledge, most existing works fail to adequately address the two key challenges, leaving critical void in building more robust and explainable deepfake detection systems.
3 Method
In this work, we propose a general explainable and extendable multimodal framework for deepfake detection, which consists of three key stages: (1) Model Feature Assessment (MFA) evaluates and ranks the forgery-related features, (2) Strong Feature Strengthening (SFS) enhances the model's strong features and Weak Feature Supplementing (WFS) leverages Specific Feature Detector (SFD) to compensate the model's weak features, and eventually resulting in an explainable dataset, and (3) The MI.LM fine-tuned on this dataset and then used for inference for enhanced deepfake detection and explanations. In the following content, will introduce the technical details of these stages.
3

---

F Festure-rsie related
Model Feature Assessment
Question Generation
Feature
(MFA)
F6
Strong Features
Stage
Genert: ye-tr-pe questane siutify decpfike.
Image blarr?? 12: Olending arillcts?
MLLM
LLM
Strony Feature Strengthening
Dutaset Construction
Strong Festures
Why Gibe?
Stage 2
Why
MILM
Iebated Exphersiimn
Image
Weak Feature Supplementing
Weak Feaure Rebate Explession
Explanation
Dataset
Ouput Scace
Specill Peataar Dotecter
Model Fine-tuning
Madel Inference
Stage 3
MLLM
this image fake??
This image fnke becouse facial fostures ard muoriisteni, f--) The facial contours and proportiens not ralund, and skin tone and
The scure by specilic
festure detestor k1S52
X
X7-DFD
artifaats.
Figure 3: A comprehensive breakdown of the three-stage methodology for 22-DDD In Stage 1, an automated procedure for forgery-related feature generation, evaluation, and ranking is implemented within the MFA (Model Feature Assessment) module. Stage incorporates the SFS (Strong Feature Strengthening) module, which automates the generation of explanatory annotations for fine-tuning dataset consisting of real and fake images, leveraging strong features, alongside the WFS (Weak Feature Supplementing) module, which employs specific feature detector to produce explanations for weak features. Stage 3 entails model fine-tuning and inference, empowering the model excel in detection performance and provide precise explanations, utilizing both its proficient strong features and less proficient weak features for improved detection and explanation.
3.1 Model Feature Assessment (MFA)
As depicted in the top row of Figure 3, the MFA module consists of three sequential stages: feature- related question generation, assessment, and ranking. Each stage plays crucial role identifying and prioritizing forgery-related features.
Step 1: Feature-related Question Generation. For each candidate forgery-related feature, corre- sponding question is formulated to assess its presence in an image. Given that these features are not predefined, Large Language Model (LLM), such GPT-4o, is leveraged to generate diverse set of N2 questions, denoted as F, These questions are designed to probe key forgery indicators, in- cluding facial inconsistencies, unnatural color, and texture mismatches elements critical for deepfake detection.
Step 2: Model Feature Assessment. Each generated question is paired with image from the assess- ment dataset, forming structured prompts for model inference. The Multi-Modal Large Language Model (MLLM) then responds with binary outputs (yes or no). which are aggregated into confusion matrix to quantify the reliability of each feature. Specifically, for an image xy and question F,, where i represents the index of forgery feature-related question, and j represents index of an image, the MLLM produces:
Ri, = MBuse F,)
(1)
where Raj E (yes, no), representing the model's response. This step ensures that the generated questions effectively capture forgery-related discrepancies.
Step 3: Feature Ranking, prioritize the most discriminative features, questions are ranked based on their Balanced Accuracy (BA):
TP:
TN,
BA 2
(2)
TP: + FN;
TN, + FP;

---

where TP,, TN,, FP,, FN; denote True Positives, True Negatives, False Positives, and False Negatives, respectively. Due to potential class imbalance between real and fake samples in the dataset, we use the simple and widely adopted Balanced Accuracy (BA) metric. This fairly evaluates both classes, aiding effective feature ranking. The ranking identifies the most reliable forgery-related features for discrimination.
Following the automated ranking, human verification is further conducted to ensure the reliability of the identified fake features. This step mitigates potential biases or misinterpretations by the LLM, refining the final selection of discriminative features. Additionally, irrelevant or con-discriminative features are filtered out, with minimal instances of erroneous or unrelated outputs.
3.2 Strong Feature Strengthening (SFS)
The SFS module constructs datasets by leveraging the strong feature capabilities identified as high- performing by the MFA module. This process comprises two key steps.
Step 1: Real/Fake Prompts Generation. Leveraging the strong features from the MFA module, we generate specialized prompts to guide the MILM''s focus during the fine-tuning phase. Specifically, we first utilize GPT-4o to summarize these strong features and construct two distinct prompts: one tailored for real images (Praa) and another for fake images (aake). These prompts are formulated as: Preal ((Frea)) Pake f(Frake), where F and Fike denote the sets of strong features relevant to real and fake images, respectively. Also, I represents any LLMs. Here, we employ GPT-4o for implementation.
Step 2: Fine-tuning Dataset Construction. A fine-tuning dataset Djc comprising VQA-style (vi- sual question answering) pairs, which is constructed by pairing each image with the correspond- ing (real or fake) prompt. Each image annotated with the specific features exhibits, and the standardized prompt Prized is defined as: Prixed = "Is this image real or fake?" The model's re- sponse is structured to begin with definitive "Taie image is real/fake followed by explanation based the identified features. Formally, the final answer is represented as: Arinat = "This image is real/fake" + Areal/taxe- Consequently, each VQA-style pair the fine-tuning dataset Dy: is formalized as: VQA = -Imagee Prixed, Arinal).
3.3 Weak Feature Supplementing (WFS)
The WFS module construct datasets by integrating specific feature detectors, which specialized in detecting features where the MILM shows weakness. This module follows two steps:
Step 1: Specific Feature Detector Invocation. For features that the MLLM identifies weak,
deploy an specific deepfake detector (e.g., a blending-based detector [41]). This specific feature detector processes the input image and generates prediction. Note that we also employ other SFDs for implementation, and we provide an in-depth analysis for this in the Appendix G. Specifically, when utilizing blending detector o (llendDetecto(zz)) where x denotes the input image, and a denotes the sigmoid function that transforms the logits output of the BlendDetector into the 0-1 range.
instance of SFD, blending score is produced:
Step 2: Integrution of Specific Feature Detection Results into the Fine-tuning Dataset.
The blending score obtained from the specific detector is incorporated into the fine-tuning dataset by appending
the existing prompts. This is done by adding a statement such as: "And the bl blending feature score of content is: s" Additionally, based on the score, correspond- ing response aligned with the probability is included, specifically in the Fine-tuning Dataset Con- struction section of the SFS. This integration ensures that the MLLM benefits from both its intrinsic detection capabilities and the specialized insights provided by the SFD.
3.4 Model Finetune and Inference
After obtaining the constructed dataset, the following steps involve fine-tuning and inference. The stage following two steps:
Step MI.MM Fine-tuning. The initial MILM is fine-tuned using the dataset Dyt- This process adjusts the projector to accurately link image artifacts with corresponding fake labels. Additionally,
5

---

Low-Rank Adaptation (LoRA) [25] is applied to selectively update a subset of the model's parameters, enhancing its focus on deepfake features while preserving overall model integrity. This fine-tuning can be expressed as:
Dre
Mbssc
M Mino-tuned:
where M tune-tumed represents the enhanced MLLM with superior deepfake detection capabilities.
Step 2: Integration of Specific Feature Detection into Inference Prompts. Generally, during the inference, the SFD detector's blending score s incorporated into the MLLM's prompt-based reasoning process. The final output of the model is structured, to begin with definitive state- ment: "This image is real/fake", followed by reasoning based on identified visual features. Based on the blending score s, the model appends a descriptive statement: Afinal = "This image is real/fake" (binary results) + AreaL/fakee (explanations) abvious/minima blending artifacts" (clues from SFD). The model acquires this response pattern through training. This approach ensures that the MLLM effectively leverages SFDs to enhance its detection performance, particularly for features where it initially demonstrated weakness.
"And this image contains
4 Experiment
4.1 Experimental Setup
Datasets. We evaluate our proposed method on a diverse set of widely-used deepfake detection
datasets, including the Deepfake Detection Challenge (DFDC) [15], its preview version (DFDCP) [16].
DeepfakeDetection (DFD) [13], Celeb-DF-v2 (CDF-v2) [37], FaceForensics++ (FF++) [57] (c23
version for training), DFo [29], WildDeepfake (WDF) [95], FFIW [92], and the newly released DF40 dataset [75]. which incorporates state-of-the-arr forgery techniques such as Facedancer [56], FSGAN [50], inSwap [58], e4s [33], Simswap [7], and Uniface [91]. In line with the standard deepfake benchmark [78],
the c23 version of FF++ for training and other datasets for testing in the main table. Additionally, we evaluated a broader range of facial forgery types using the DiFF [9] dataset, a comprehensive collection of diffusion-genett our method on a wider spectrum of forgery techniques.
facial images, which allowed us to test
Evaluation Metrics. We assess the performance of our model in terms of both detection perfor- mance and explanation quality. For detection, we adopt the Area Under the Curve (AUC) as the primary metric to evaluate the model's ability distinguish real from fake content across entire datasets, reporting both frame-level and video-level AUC scores. Additional metries, including Accuracy (Acc.), Equal Error Rate (EER), and Average Precision (AP), are also provided for comprehensive analysis. For explanation, we follow [87] by using auman-annotated data to measure text similarity between model-generated explanations and human-labeled ground truth, employing standard metrics such as BLEU [51], CIDEr [65], RO ROUGE- [38], METEOR [14], and SPICE [2]. Beyond text similarity, we engage human evaluators and GPT-44o to assess the quality of explanations regarding forgery content, following prior studies [72, 21J. Evaluators rate the explanations on a scale from 0 (very poor) to (excellent), ensuring robust qualitative evaluation.
Implementation Details. We initialize our model with the LLaVa- weights and fine-tune the LLaVA model [44] using its official implementation codebase. For the specific feature detectors (SFD), we adopt blending- based approach as proposed in [41]. Training is performed on a single NVIDIA 4090 GPU for 3 epochs, with a learning rate of 2 x 10-5 5 in two layer mip projector and 2 10 in others, a rank of 16, and an alpha value set conventionally to twice the rank at 32. We use a batch size of 4, a gradient accumulation step of 1, and a warmup ratio of 0.03 to stabilize training.
4.2 Generalizability Evaluation
Following the common settings of previous works [73, 10], we first compare our method with 33 SOTA detectors (including both conventional and multimodal-based detectors) via cross-dataset evaluations (see Table 1.) The results of other compared baselines are mainly cited from their original papers. Our approach excels across both frame-level and video-level evaluations, maintaining superior results when compared to other methods. The table clearly highlights our method's capability to generalize and consistently achieve higher detection performance at the frame level and video level, respectively.

---

Table I: Cross-dataset evaluations with 33 existing detectors. The top two results highlighted, with the hest in bold and the second-best underlined. **' indicates reproductions based on the pre-trained checkpoints released by the authors, and refers to MLLM-baser detectors, which can also output explanations. f: The LAA-Net is trained on the high-quality FF++ (raw) data, whereas our method is trained on the compressed (c23) version.
Frame-L.eve AUC
Viveo-Level AUC
Method
CDF DFDCP DFDC DFD Avg.
Method
CDF DFDCP DFDC
DFD Avg.
Xception [12]
73.7
73.7
70.8
81.6
75.0
Xception [12]
81.6
74.2
73.2
89.6 79.7
FWA [36]
66,8
63.7
61.3
74.0
66.5
PCL+12G [89]
90.0
74,4
67.5
Efficient-bl [64]
74.9
72.8
69.6
81.5
74.7
LipForensics [24]
82.4
73.5
Face X-ray [32]
67.9
69.4
63.3
76.7
69.3
FTCN [90]
86.9
74.0
71.0
94.4
81.6
F3-Net [53]
77.0
77.2
72.8
82.3
77.3
VTT-B (CIIP) [54]
88.4
82.5
76.1
90.0
84.3
SPSL [45]
76.5
74.1
70.1
81.2
75.5
CORE [49]
80.9
72.0
72.1
88.2
78.3
SRM [47]
75.5
74.1
70.0
81.2
75.2
SBI* [60]
90.6
87.7
75.2
88.2
85.4
ViT-B (IN21K) [19]
75.0
75.6
73.4
86.4
77.6
UIA-VIT [94]
82.4
75.8
94.7
ViT-B (CLIP) [54]
81.7
80.2
73.5
86.6
80.5
SLADD* [6]
79.7
77.2
RECCE [5]
73.2
74.2
71.3
81.8
75.1
DCL [63]
76.9
75.0
92.1
83.1
IID [26]
83.8
81.2
SeeABLE [30]
87.3
86.3
75.9
ICT [18]
85.7
84.1
CFM [46]
89.7
80.2
70.6
95.2
83.9
LSDA [73]
83.0
81.5
73.6
88.0
81.5
UCF [77]
83.7
74.2
77.0
86.7
80.4
VLFFD) [62]
83.2
83.2
94.8
NACO [85]
89.5
76.7
FTAAÍ [27]
74.0
92.0
AllFreeziig [68]
89.5
RepDFDf [39]
80.0
90.6
77.3
TALL-Swin [71]
90.8
76.8
MFCLIP t [86]
83.5
86.1
StyleDFD [11]
89.0
96.1
KFD-VLM f [84]
89.9
86.7
92.3
LAA-Net [48]
95.4
86.9
98.4
DFD (7B)
90.4
87.3
83.7
92.3
88.4
PDDD (7B)
95.4
89.3
86.0
95.8
91.6
.2DDFD (13B)
91.3
90.3
83.4
92.5
89.4
2DDDF (13B)
95.7
91.0
85.7
96.1
92.1
4.3
Explainability Evaluation
Annotated Explainability Evaluation. We assess the performance of our model using the DD- VQA [87] test dataset, which incorporates human-annotated data from FF++ [57]. The evaluation employs suite of metrics, including BLEU [51], CIDEr [65], ROUGE-L [38], METEOR [14],
and SPICE [2], to quantify the alignment between our model's responses and numan-annotann ground truth. The MLLMs assessed for explanation quality include LLavA [43], Llama3.2V [20], Qwen2VL. [67] and GPT4o [1]. The models use the same prompt to generate explainable outputs. To ensure fair comparison, particularly given GPT-4o's 's tendency to refuse responses with the same prompt, adopt prompting strategy from [28]. This leads GPT-4o to generate shorter responses, resulting in lower scores. The evaluation results are summarized in Table 2 (Annotated Explainability Evaluation). Due to the DD-VQA only annotating the artifact in some specific parts (e.g., nose, eyes), GPT-4o and human experts are required to evaluate both annotated and unannotated scenarios, ensuring thorough assessment of model explainability across different scenarios.
Table 2: Explainability evaluation across annotated and unannotated settings, comparing five scores for annotated explainability, alongside human and GPT-4o evaluations (scored 0-5) for unannotated explainability. The best result per metric is highlighted in bold.
Annotated Explainability
Unannotated Explainability
Model
BLEU
CIDEr
ROUGE-L METEOR
SPICE
Avg.
Human-Eval
GPT4-Eval
Avg.
I.LVA-78 [44]
0.183
0.021
0.139
0.110
0.085
0.108
2.368
1.542
1.955
Llama3.2 Vision [20]
0.131
0.009
0.131
0.081
0.116
0.093
2.265
1.667
1.966
Qwen2.5-VL-77 [67]
0.140
0.012
0.143
0.081
0.150
0.105
2.034
1,383
1.709
GPT-40 [1]
0.123
0.011
0.082
0.051
0.072
0.068
2.559
2.055
2.307
Ours
0.203
0.027
0.155
0.148
0.155
0.138
3.572
2.668
3.120
Unannotated Explainability Evaluation. To evaluate unannotated explainability, we build on insights from prior work [72, 21,93) 93] and utilize both human evaluators and GPT-4o to assess model performance across three key dimensions: (1) detection ability, (2) reasonableness of explanations, and (3) level of detail. Each dimension is scored on scale from 0 to 5. The evaluation results are
7

---

summarized in Table 2 (Unannotated Explainability Evaluation). The results demonstrate that our model achieves strong performance both human-eval and GPT-eval. Additional details include the experiment of the setting of human study, Graphical User Interface (GUT) of human study, and the evaluation prompt of GPT4o can be found in Appendix.
4.4 Computational Complexity Evaluation
We performed a simple evaluation of the time complexity. Compared to other pretrained MLLMs, the model does not require much additional inference time in the same inference framework, seen in Table 3, making the time acceptable. However, when compared to conventional Mon-MLLM meth- ods, the model, despite gaining interpretability and richer feature space, requires more inference time. Nevertheless, believe that MILMM inference frameworks advance, along with pruning and quantization lightweight methods, MLLMs will benefit from these improvements when applied to Deepfake detection.
Table 3: Comparison of average inference time per image for different models.
Model
Seconds
Non-MLLMs- Xception
-0.03
Mon-MLLMs- CDFA
-0.05
Llama-3.2-11
-3.3
Qwen2.5 VL-IInstrct 7B
-1.6
LLaVa-7B
-1.2
Ours (X2DFD w/ SFDs)
-1.3
5 Ablation Study and Analysis
Here, we address several key research questions through ablation studies and in-depth analysis.
Question 1:
Why is fine-tuning MLLMs with their strong features more effective than using all?
To enhance the reasoning and detection capabilities of MLLMs, we introduce the Strong Feature Strengthening (SFS) module. In this module, we focus selectively amplifying the most discrimi- native forgery-related Teatures-referree to as strong features. These strong features are identified through Model Feature Attribution (MFA), which ranks features based their importance scores across different modalities and samples. Fine-tuning on these "strong features" is more effective because it focuses the learning process highly discriminative, reliable forgery-relater related cues while avoiding the disruptive noise introduced by weak features.
This design leads us to critical question: Are these strong fea- tures truly more effective for im- proving model performance than using the full set of features? To answer this, we compare two strategies: (1) enhancing all fea- tures in the LLM -.MM---gerrteed fea- ture list and (2) enhancing only the strong features selected via MFA. As shown in Table 4, the latter consistently outperforms the former across all datasets. These results confirm that selectively strengthening the most discriminative features not only improves the model's performance but also yields more reliable model explanations.
Table 4: Comparison model detection performance and interpretation between using all features and strong features, both enhanced by the Strong Feature Strengthening (SFS) module. The evaluation metrics include AUC and GPT--4o evaluation. Additionally, results for the Top-K feature selection strategy and the Feature Explanation are provided for comparison.
Model Configuration
CDF
Uniface
HPS-Diff
Avg
GPT4o Eval
No Feature Explanation
80.3
81.9
84.7
82.3
X2DFD (Top-K=25)
83.0
84.3
87.1
84.8
2,91
X2DFD (Top-K=50)
83.2
84.5
88.7
85.5
3.02
X2DFD (Top-K=75)
81.9
2.77
X2DFD (Top-K=100)
79.0
2.63
Question 2: How to ensure SFS module works when should, and stays silent when shouldn' 't?
To further improve model generalization and in- terpretability, we extend our framework with two new components: the Specific Feature Detection (SFD) module and the Weak Feature Supplement- ing (WFS) module. While the earlier SFS module focuses on enhancing strong, highly discrimina- tive features, it may overlook subtle patterns crit- ical for certain forgery types. To address this, WFS is designed to teach the model how to lever-
Table 5: Effect of excluding supplementary features during training (WFS) and including them at inference (SFD infer) on model performance.
Varient
CCFF DFD DFDC DFDCP Avg.
WFS x SFD infer X
83.2 91.4
79.2
82.0
84.0
WFS X SED infer /
81.7
90.6
79.1
81.3
83.2
WFS / SFD infer
90.4 92.3
83.7
88.4
8

---

age weak features provided by SFD features
that are otherwise hard for MLLMs to interpret directly.
To investigate whether this combination yields synergistic benefits, we compare three variants: (1) baseline without WFS and SFD inference, (2) enabling SFD only during inference (without WFS), and (3) enabling both WFS and SFD. As shown in Table 5, the model achieves the best performance when WFS is present, demonstrating that SFD's weak signals become more useful once the model has learned how to utilize them through WFS, Without WFS, simply adding SFD at inference may not help- and can even lead to degradation indicating that "1 "1+1" only becomes greater than 2 when weak features are integrated structurally during training.
Beyond synergy, it is crucial to ensure that the introduction of SFD does not interfere in sce- narios where its cues are irrel- evant. For example, blending- based detectors may provide lim- ited value on datasets like SRI, which contain no blending traces. We test this by introducing SRI as training set and comparing performance. As shown in Table 6, the model not only maintains its effectiveness but even improves, suggesting that when SFD signals are weak or absent, the model naturally downplays them. This demonstrates that our design is adaptive-SFD helps when it can, and steps aside when it should.
Table 6: Comparison of AUC performance for models trained on FF++ alone versus FF++ with SRI, evaluated other datasets.
Train Data
CDF DFDCP DFDC DFD Uniface Esgan Inswap Simswap Avg.
FF++ SRI x
90.4
87.3
83.7
92.3
85.5
91.1
81.2
85.1
87.1
FF++ SRI / 91.5
89.3
83,9
92.7
87.4
89.9
81.0
86.1
87.7
Overall, our framework achieves both synergistic improvement and non-intrusive integration: WFS enables the model to benefit from weak features without forcing reliance, and SFD contributes only when its signals are relevant.
Question 3: How can we generate the most suitable set of N forgery-related questions in MFA?
The Model Feature Assessment (MFA) module evaluates the model's discriminative ability by asking it to answer curated set of N forgery-related questions. A key challenge here is: how to generate the most suitable questions that effectively probe the model's 's understanding of diverse forgery cues. To explore this, we compare different question-generation generation strategies: (1) human written features based on expert knowledge [87]. (2) features automatically generated by large language models (LLMs),
including Claude3.5-Sonnet [3] and GPT--4 [1].
As shown in Table 7, questions
Table 7: Comparison between LLMs and human annotators for generating N forgery-related questions for MFA.
generated by LLMs slightly out- perform those crafted by hu- mans in terms of detection perfor- mance across multiple datasets. This suggests that LLMs can cap- ture a broader and potentially more nuanced range of forgery-
Variant
CDF DFDCP DFDC DFD Uniface Fsgan Simswap Avg
Human Writing [87]
89.1
89.7
83.6
92.5
82.3
89.1
87.0
87.6
Claude3.5 5-Sonett [3]
90.1
88.5
83.5
93.0
84.9
90.0
85.6
87.9
GPT40 [1]
90.3
89.7
83.5
92.5
85.2
89.9
84.9
87.8
related features, possibly including cues overlooked by human experts. However, LLM questions are not always they may sometimes be generie, redundant, or irrelevant (e.g., mistakenly treating "Photoshop traces" as core deepfake features). On the other hand, although human-designed features may narrower in scope, they offer higher precision and domain relevance, leading to robust results. To balance these strengths and weaknesses, we adopt a hybrid strategy: (1) Use an LLM to generate a diverse pool of forgery-related questions. (2) Rank them by relevance scores. (3) Apply human verification to filter out irrelevant or low-quality questions.
Question 4: Can framework benefit from extending multiple SFDs?
With the continuous development of generative technologies. diffusion- generative methods and techniques have been emerging rapidly. To test the scalability of our framework, we extended it to another SFD, AlignedForensies [55], and incorporated a new training dataset, consisting of fake faces generated through diffusion techniques from 1000 external images, which form a subset of the face-swar swapping data in DiFF [9]. We then tested not only the sice-swapping scope, which was previously the focus of this work (e.g.. CDF and Simswap), but also expanded the evaluation to include face editing, image-to-image editing, and text-to-image generation using the DiFF dataset, which also focuses on faces.

---

As shown in Table 8, we found that the X2DFD framework not only effectively extended the CDFA's blending-based SFD but also enhanced the performance when extending both the CDFA and AlignedForensies (diffusion-bail based SFD). This extension allowed the framework to achieve strong results across various forgery types and methods.
Table AUC Performance Evaluation for SFD Integra-
tion X2DFD-Sig (CDFA only) vs. X2DFD-Mul (CDFA & AliineeFooensics))
Model
CDF Simswap Diff-FE Dift-121 Diff-T21
CDFA
87.9
76.1
74.6
81.7
87.3
X2DFD-Sig
90.4
85.1
82.1
81.7
88.3
X2DFD-Mul
90.3
88.5
92.1
88.6
92.2
6 Conclusion
In this paper, we propose -DDDFD, a unified multimodal framework for explainable and extend- able deepfake detection. For the first time, we systematically evaluate the intrinsic capabilities of the pre-trained MLLMs, revealing their varying effectiveness across different forgery-related features. Inspired by this, implement targeted fine-tuning strategy, which has largely improved the explainability of the MILMMs, specifically capitalizing on their strengths. Furthermore, by integrating specific feature detectors (SFD), design adaptive fusion module combine the complementary advantages of both MILMs and conventional detectors for improved detection.
Limitations and Future Work. While our framework demonstrates strong performance in detecting identity-specific facial forgeries, it has certain limitations. First, multimodal large language models (MLLMs) operate in much larger parameter space, which leads to richer visual feature space related to Deepfake detection. However, this comes at the cost of slower inference speeds. Future advancements the field could help accelerate inference, benefiting from the rapid development of the domain. Second, our current implementation focuses solely on static image detection. However, real-world applications increasingly involve multimodal forgeries across video and audio streams. Extending our method to handle videos and audio- visual deepfakes is a critical next step for building a comprehensive and practical detection system. Third, As forgery technologies advance and realistic deepfakes exhibit fewer detectable artifacts that can be captured by natural language descriptions, our MLLM-bbased method may become less effective in interpreting semantic artifacts, thereby requiring the SFD module to play more crucial role in the framework.
Broader Impacts. This research advances machine learning with a new framework to detect and explain Deepfake images, effectively identifying deepfakes and reducing misuse of generative models for significant societal benefit. However, it risks being used to improve deepfake realism. counter this, following previous works [74, 75] implement access controls. We will urge researchers to minimize harms while maximizing the positive impact of this work.
Ethics & Reproducibility statements. All facial images used are from publicly available datasets with proper citations, ensuring no violation of personal privacy. And the human study has received the IRB approval.
Content Structure of the Appendix. Due to page constraints, we include additional analyses and experiments in the Appendix. Specifically, the Appendix contains the following sections: Overview of Appendix, Experiment Setting Details, Additional Experimental Results, Human Study and GPT4 Evaluation, Additional Analysis of Model Feature Assessment, Additional Analysis of Strong Feature Strengthening, Additional Analysis of Weak Feature Supplementing, Additional Analysis of Ablation Study, Sample Showing. For further details, please refer to the Appendix.
Acknowledgments
Baoyuan Wu was supported by Guangdong Basic and Applied Basic Research Foundation (No. 10241151502000 550200955, Guangdong Provincial Program (No. 2023TQ07A352), Shenzhen Science and Technology Program (No. (CYJ2022001311136000 1), and Longgang District Key Laboratory of Intelligent Digital Economy Security.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777 2023.
[2] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCC 2016: 14th European Conference, Amsterdam, The Netherlands, October 1114, 2016, Proceedings, Part 14, pages 382-398. Springer, 2016.
[3] Anthropic. Introducing the next generation of claude. https://wwws http:://www.atthrpi....om www.rapic..o.om.o....o cownn//s//cmm////////e/ claude 3- family, 2024. 2024.03.
[4] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint vrXiv:2404.18955 2024.
[5] Junyi Cao, Chao Ma. Taiping Yao, Shen Chen, Shouhong Ding, and Xiaokang Yang. End-to-end aeconstructoon-caasfffctttt learning for face forgery detection. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 41 13--4122, 2022.
[6] Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang. Self-supervise learning of adversarial example: Towards good generalizations for deepfake detection. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 18710-18719, 2022.
[7] Renwang Chen, Xuanhong Chen, Bingbing Ni, and Yanhao Ge. Simswap: An efficient framework for high fidelity face swapping. In Proceedings the 28th ACM international conference on multimedia, pages 2003-2011. 2020.
[8] Ruoxin Chen, Junwei Xi, Zhiyuan Yan, Yue Zhang, Shuang Wu, Jingyi Xie, Xu Chen, Lei Xu, Isabel Guan, Taiping Yao, et al. Dual data alignment makes ai-generated image detector easier generalizable. arXiv preprint arXiv:2505. 14359, 2025.
[9] Harry Cheng, Yangyang Guo, Tianyi Wang, Liqiang Nie, and Mohan Kankanhalli. Diffusion facial forgery detection. In Proceedings of the 32nd ACM international conference on multimedia, pages 5939-5948, 5948, 2024.
[10] Jikang Cheng. Zhiyuan Yan, Ying Zhang, Yuhao Luo. Zhongyuan Wang. and Chen Li. Can leave deepfake data behind training deepfake detector? arXiv preprint arXiv:2408. 17052, 2024.
[11] Jongwook Choi, Tachoon Kim, Yonghyun Jeong, Seungryul Back, and Jongwon Choi. Exploiting style latent flows for generalizing deepfake video detection. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1133-1143, 2024.
[12] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251 1258,, 2017.
[13] Deepfakedetection,
https: htt://aaggeee wwt.glgooooooooooooooooooo contributing data- to-ta-to-deee deepf axeeefeatittiioonn html Accessed 2021-11-13.
2021.
[14] Michael Denkowski and Alon I Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376-380, 2014.
[15] Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer. The deepfake detection challenge (dfde) dataset. arXiv preprint arXh:20006007399, 2020.
[16] Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton Ferrer. The deepfake detection challenge (dfde) preview dataset. arXiv preprint
2019.
[17] Shichao Dong, Jin Wang, Renhe Ji, Jiajun Liang, Haoqiang Fan, and Zheng Ge. Implicit identity leakage: The stumbling block to improving deepfake detection generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3994-4004, 2023.
[18] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Ting Zhang, Weiming Zhang, Nenghai Yu, Dong Chen, Fang Wen, and Baining Guo. Protecting celebrities from deepfake with identity consistency transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9468-9478, 2022.
[19] Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXi:22010.11929 2020.

---

[20] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman. Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llaa 3 herd of models. arXiv preprint urXiv:2407.21788 2024.
[21] Niki M Foteinopoulon, Enjie Ghorbel, and Djamila Aouada. A hitchhiker's guide to fine-grained face forgery detection using common sense reasoning. Advances Neural Information Processing Systems, 17:2943-2976, 2025.
[22] Xinghe Fu, Zhiyuan Yan, Taiping Yao, Shen Chen, and Xi Li. Exploring unbiased deepfake detection via token-level shuffling and mixing. arXiv preprint arXiv:2501.43776 2025.
[23] Qiqi Gu, Shen Chen, Taiping Yao, Yang Chen, Shouhong Ding, and Ran Yi. Exploiting fine- ine-grained face forgery clues via progressive enhancement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages
2022.
[24] Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips don't lie: generalisable and robust approach to face forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 5039-5049, 2021.
[25] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen, Lora: Low-rank adaptation large language models, arXiv preprint arXiv: (06::iv:::66000 09685, 2021.
[26] Baojin Huang, Zhongyuan Wang, Jifan Yang. Jiaxin Ai, Qin Zou, Qian Wang, and Dengpan Ye. Implicit identity driven deepfake face swapping detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4490-444904 4499, 2023.
[27] Zhengchao Huang, Bin Xia, Zicheng Lin, Zhun Mou, and Wenming Yang. Ffax: Multimodal large language model based explainable open- open-world face forgery analysis assistant. arXiv preprint arXiv:2408, 10072, 2024.
[28] Shan Jia, Reilin Lyu, Kangran Zhao, Yize Chen, Zhiyuan Yan, Yan Ju, Chuanbo Hu, Xin Li, Baoyuan Wu, and Siwei Lyu. Can chatgpt detect deepfakes? a study of using multimodal large language models for media forensics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 4324 4333, 2024.
[29] Liming Jiang, Ren Li, Wayne Wu, Chen Qian, and Chen Change Loy. Deeperforemsies.1.0 .:: A large-scale dataset for real-world face forgery detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
[30] Nicolas Larue, Ngac-Son Vu, Vitomir Struc, Peter Peer, and Vassilis Christophides. Sceable: Soft discrepancies and bounded contrastive learning for exposing deepfakes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21011-21021, 2023.
[31] Jiawei Li, Fanrui Zhang, Jiaying Zhu, Esther Sun, Qiang Zhang, and Zheng-Jun Zha. Forgerygpt: Multimodal large language model for explainable image forgery detection and localization. arXiv preprint arXiv:2410. 10238, 2024.
[32] Lingzhi Li. Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. Face x-ray for more general face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020.
[33] Maomao Li, Ge Yuan, Cairong Wang, Zhian I Liu, Yong Zhang, Yongwei Nie, Jue Wang, and Dong Xu. EAs: Fine-grained face swapping via editing with regional gan inversion. arXiv preprint arXiv:2310. 150 2023.
[34] Yixuan Li, Xuelin Liu, Xiaoyang Wang, Shiqi Wang, and Weisi Lin. Fakebench: Uncover the achilles' heels of fake images with large multimodal models. arXiv preprint arXiv:2404. 13306, 2024.
[35] Yuezun Li, Ming-Ching Chang, and Siwci Lyu. In ictu oculi: Exposing ai created fake videos by detecting eye blinking. In 2018 IEEE International workshop on information forensics and security (WIFS), pages 1-7. Icce, 2018
[36] Yuezun Li and Siwci Lyu. Exposing deepfake videos by detecting face warping artifacts. arXiv preprint arXiv: 11111111100000000 2018
[37] Yuezun Li, Xin Yang. Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-df: A new dataset for deepfake forensics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
12

---

[38] Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summurization branches oud, pages 74-81, 2004.
[39] Kaiqing Lin. Yuzhen Lin, Weixiang Li. Taiping Yao, and Bin Li. Standing on the shoulders of giants: Reprogramming visual-language model for general deepfake detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 5262-5270, 2025.
[40] Kaiqing Lin, Zhiyuan Yan, Ke-Yue Zhang, Li Hao, Yue Zhou, Yuzhen Lin, Weixiang Li, Taiping Yao, Shouhong Ding, and Bin Li. Guard me if you know me: Protecting specific face-identity from deepfakes. arXiv preprint arXiv:2505. 19582, 2025.
[41] Yuzhen Lin, Wentang Song, Bin Li, Yuezun Li, Jiangqun Ni, Han Chen, and Qiushi Li. Fake it till you make it: Curricular dynamic forgery augmentations towards general deepfake detection. In European Conference on Computer Vision, pages 104- 122. Springer, 2024.
[42] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[43] Haotian Liu. Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024.
[44] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[45] Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue. Weiming Zhang, and Nenghai Yu. spateal-phase shallow learning: rethinking face forgery detection in frequency domain. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 772-781, 2021.
[46] Anwei Luo, Chenqi Kong, Jiwu Huang, Yongjian Hu, Xiangui Kang, and Alex C Kot. Beyond the prior forgery knowledge: Mining critical clues for general face forgery detection. IEEE Transactions on Information Forensics and Security, 19: 168-1182. 2023.
[47] Yuchen Luo, Yong Zhang, Junchi Yan, and Wei Liu. Generalizing face forgery detection with high- frequency features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16317-16326, 2021.
[48] Dat Nguyen, Nesryne Mejri, Inder Pal Singh, Polina Kuleshova, Marcella Astrid, Anis Kacem, Enjic Ghorbel, and Djamila Aouada. Laa-net: Localized artifact attention network for quality-agnostic and generalizable deepfake detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17395-17405, 2024.
[49] Yunsheng Ni, Depu Meng, Changqian Yu, Chengbin Quan. Dongehun Ren, and Youjian Zhao. Core: Consistent representation learning for face forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12-22 21, 2022.
[50] Yuval Nirkin, Yosi Keller, and Tal Hassner. Fsgan: Subject agnostic face swapping and reenactment. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7184-7193, 2019.
[51] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[52] Gan Pei, Jiangning Zhang, Menghan Hu, Zhenyu Zhang, Chengjie Wang, Yunsheng Wu, Guangtao Zhai, Jian Yang, Chunhua Shen, and Dacheng Tao. Deepfake generation and detection: A benchmark and survey. arXiv preprint 2xXiv:2403.17888 2024.
[53] Yuyang Qian, Guojun Yin. Lu Sheng, Zixuan Chen, and Jing Shao. Thinking in frequency: Face forgery detection by mining frequency-aware clues. Eurupean conference on computer vision, pages 86-103. Springer, 2020.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[55] Anirudh Sundara Rajan, Utkarsh Ojha, Jedidiah Schloesser, and Yong Jae Lee. Aligned datasets improve detection of latent diffusion-generated images. In The Thirteenth International Conference on Learning Representations, 2025.
13

---

[56] Felix Rosberg, Eren Erdal Aksoy, Fernando Alonso-Fernandez, and Cristofer Englund. Facedancer: Pose-and occlusion-aware high fidelity face swapping. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3454-3463, 2023.
[57] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias NieBner. Faceforensics**: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF Confer- ence on International Conference on Computer Vision, 2019.
[58] Somdev Sangwan. Roop.
www/ com/smmd3v/roop 2020. [GitHub repository].
[59] Yichen Shi, Yuhao Gao, Yingxin Lai, Hongyang Wang. Jun Feng, Lei He, Jun Wan, Changsheng Chen, Zitong Yu, and Xiaochun Cao. Shield: An evaluation benchmark for face spoofing and forgery detection with multimodal large language models, arXiv preprint arXiv:2402.00 2tti:24022004778, 2024.
[60] Kaede Shiohara and Toshihiko Yamasaki. Detecting deepfakes with self-blended images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18720-18729, 2022.
[61] Kaede Shiohara, Xingchao Yang, and Takafumi Taketomi. Blendface: Re-designing identity encoders for face-swapping. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 7634-7644, 2023.
[62] Ke Sun, Shen Chen, Taiping Yao, Ziyin Zhou, Jiayi Ji, Xiaoshuai Sun, Chia-Wen Lin, and Rongrong
Ji. Towards general lesuul-iiguuisicc face forgery detection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 99576-19586 2025.
[63] Ke Sun, Taiping Yao, Shen Chen, Shouhong Ding. Jilin Li, and Rongrong Ji. Dual contrastive learning for general face forgery detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2316-22111 2624222222222 2022.
[64] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the International Conference on Machine Learning. pages 6105 61. PMLR, 2019.
[65] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[66] Chengrui Wang and Weihong Deng, Representative forgery mining for fake face detection. I Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14923-14932, 2021.
[67] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing I Liu, Jialin Wang. Wenbin Ge. et al. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024.
[68] Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, and Houqiang Li. Altfreezing for more general video face forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4129-4138, 4138, 2023.
[69] Siwei Wen, Junyan Ye, Peilin Feng, Hengrui Kang, Zichen Wen, Yize Chen, Jiang Wu, Wenjun Wu, Conghui He, and Weijia Li. Spot the fake: Large multimodal model-based synthetic image detection with artifact explanation. arXiv preprint
Atl::2503.4905 2025.
[70] Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu. Multimodal large language models: A survey. In 2023 IEEE International Conference Big Duta (BigData), pages 2247-2256. IEEE, 2023.
[71] Yuting Xu, Jian Liang, Gengyun Jia, Ziming Yang, Yanhao Zhang, and Ran He. Tall: Thumbnail layout for deepfake video detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 22658 2266226668222 2023.
[72] Zhipei Xu, Xuanyu Zhang, Runyi Li, Zecheng Tang, Qing Huang, and Jian Zhang. Fakeshield: Explain- able image forgery detection and localization via multi-modal large language models. arXiv preprint arXiv:2410.02766 2024.
[73] Zhiyuan Yan, Yuhao Luo. Siwei Lyu, Qingshan Liu, and Baoyuan Wu. Transcending forgery specificity with latent space augmentation for generalizable deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8984-8994, 2024.
[74] Zhiyuan Yan. Jiangming Wang, Zhendong Wang, Peng Jin, Ke- Yue Zhang, Shen Chen, Taiping Yao, Shouhong Ding, Baoyuan Wu, and Li Yuan. Effort: Efficient orthogonal modeling for generalizable ai di-generated image detection. arXiv preprint arXiv:241115533,p0 2024.
14

---

[75] Zhiyuan Yan. Taiping Yao, Shen Chen, Yandan Zhao, Xinghe Fu, Junwei Zhu. Donghao Luo, Chengjie Wang. Shouhong Ding. Yunsheng Wu, al. Df40: Toward next-generation deepfake detection. arXiv preprint arXiv:2406. 13495, 2024.
[76] Zhiyuan Yan, Junyan Ye, Weijia Ii, Zilong Huang, Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, and Li Yuan. Gpt-imgeval: A comprehensive benehmark for diagnosing gpt4o in image generation. arXiv preprint arXiv:2504.02782, 2025.
[77] Zhiyuan Yan, Yong Zhang, Yanbo Fan, and Baoyuan Wu. Ucf: Uncovering common features for generalizable deepfake detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22412-22423, 2023.
[78] Zhiyuan Yan, Yong Zhang, Xinhang Yuan. Siwei Lyu, and Baoyuan Wu. Deepfakebench: A comprehensive benchmark of deepfake detection. arXiv preprint arXiv:2307.01426 2023.
[79] Zhiyuan Yan, Yandan Zhao, Shen Chen, Mingyi Guo, Xinghe Fu, Taiping Yao, Shouhong Ding, and Li Yuan. Generalizing deepfake video detection with plug-and-play: Video-level blending and spatiotemporal adapter tuning. arXiv preprint arXiv:2408. I 17065, 2024.
[80] Tianyun Yang, Juan Cao, Qiang Sheng. Lci Li, Jiaqi Xirong Li, and Sheng Tang. Leaming to disentangle gan fingerprint for fake image attribution. arXiv preprint arXiv: 1ee:22106007799 2021.
[81] Xin Yang, Yuezun Li, and Siwei Lyu. Exposing deep fakes using inconsistent head poses. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8261-8265 IREE, 2019.
[82] Zheng Yang. Ruoxin Chen, Zhiyuan Yan, Yue Zhang, Xinghe Fu, Shuang Wu, Xiujun Shu, Taiping Yao, Shouhong Ding, and Xi Li. All patches matter, more patches better: Enhance ai-generated image detection via panoptic patch learning. arXiv preprint arXiv:2504.0139 2025.
[83] Junyan Ye, Baichuan Zhou, Zilong Huang, Junan Zhang, Tianyi Bai, Hengrui Kang, Jun Hc, Honglin Lin, Zihao Wang, Tong Wa, ct al. Loki: A comprehensive synthctic data detection benchmark using large multimodal models. arXiv preprint arXiv:2410.09732. 2024.
[84] Peipeng Yu, Jianwei Fei, Hui Gao, Xuan Feng, Zhihua Xia, and Chip Hong Chang. Unlocking the capahilities of lsionnllnguuge models for generalizahle and explainable deepfake detection. arXiv preprint arXiv:2503. 14853, 2025.
[85] Daichi Zhang, Zihao Xiao, Shikun Li, Fanzhao Lin, Jianmin Li, and Shiming Ge. consistency representation for face forgery video detection. In European Conference on Computer Vision, pages 407-424. Springer, 2024.
Learning natural
[86] Yaning Zhang. Tianyi Wang. Zitong Yu, 7an Gao, Linlin Shen, and Shengyong Chen. Mfelip: Multi-modal fine-grained clip for generalizable diffuusion face forgery detection. IEEE Transactions on Information Forensics and Security, 2025.
[87] Yue Zhang, Ben Colman, Xiao Guo, Ali Shahriyari, and Gaurav Bharaj. Common sense reasoning for deepfake detection. European Conference on Computer Vision, pages 399-415. Springer, 2024.
[88] Hanqing Zhao. Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi- attentional deepfake detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2185-221 2194, 2021.
[89] Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuanjun Xiong, and Wei Xia. Learning self-consistency for deepfake detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15023-15033, 2021.
[90] Yinglin Zheng, Jianmin Bao, Dong Chen, Ming Zeng, and Fang Wen. Exploring temporal coherence for more general video face forgery detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15044-15054, 2021.
[91] Jiancan Zhou, Xi Jia, Qiufu Li, Linlin Shen, and Jinming Duan. Uniface: Unified cross-entropy loss for deep face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20730 2023.
[92] Tianfei Zhou, Wenguan Wang, Zhiyuan Liang, and Jianbing Shen. Face forensics in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2021.
15

---

[93] Ziyin Zhou, Yumpeng Luo, Yuanchen Wu, Ke Sun, Jiayi Ji, Ke Yan, Shouhong Ding, Xiaoshuai Sun, Yunsheng Wu, and Rongrong Ji. Aigi-holmes: Towards explainable and generalizable ai-generated image detection via multimodal large language models. arXiv preprint vrXiv:2507.02660 2025.
[94] Wanyi Zhuang. Qi Chu, Zhentao Tan, Qiankun Liu, Haojie Yuan, Changtao Miao, Zixiang Luo, and Nenghai Yu. Uia-vit: Unsupervised inconsistency-awe awwre method based on vision transformer for face forgery detection. In European conference on computer vision, pages 391-407. Springer, 2022.
[95] Bojia Minghao Chang. Jingjing Chen, Xingjun Ma, and Yu Gang Jiang. Wilddcepfake: A challenging real-world dataset for deepfake detection. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2382 2390, 2020.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims presented in the abstract and introduction offer a precise overview of what accomplished and investigated in this research, providing accurate reflection of the paper's 's scope and contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It fine include aspirational goals motivation as long as it clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations the study, See "Limitations and Future Work" in the conclusion section of the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g.. the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should diseuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material. Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed information on the experimental setup for the reproduction of experimental results in Appendix B.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NP does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibilityy which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have released the training and testing code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
public/gu (eellic/guide////e Luddes/Coeeuumii ssionPolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
// //nips.coo cc/ ctppb.ccc httppppulllluleeeeeeee http:///guuiee//eeeebiiiiiiiee
Cooe/Subieeeeeeeeeee SondeeSummsssiinPP.e swiiisssiccyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and basclines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper outline all Details in section implement details, including data splits, hyper-parameters, how they were chosen, type of optimizer, and other relevant factors, ensuring thorough understanding of the results. For more details, see Section (Implementation Details) and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Although do not report explicit error bars, have considered the random- ness and run the experiments multi times for main results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the emoo bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should careful not to show tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were caleulated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have clearly reported the resources cost used for experiment in section "Implementation Details".
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te://nneuus/eeeeee cc/public//eicccccccccc www.eehhiiiiilenneeeeeeeeeeeaeee
Answer: [Yes]
Justification:
The research presented in the paper fully adheres to the NeurlPS Code of Ethics in all aspects. Additionally, all experiments involving human studies have received Institutional Review Board (IRB) approval.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special considera- tion due laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed it in "Broader Impacts" in section conclusion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance). faimess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
the other hand, not needed to point out that a
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our model is designed for detection and defense. While widely available detectors could theoretically be used to improve generators, we believe the benefit for safety outweighs the risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Yes, we have properly credited the creators and original owners of the assets used in the paper, and we have explicitly mentioned and respected their licenses and terms of use.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e-g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release the training and testing code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Yes, the paper includes the details of the experiment instructions and relevant screenshots of the human study; see the "Human Study" section.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: Yes, we obtained Institutional Review Board (IRB) approvals. For more details refer to "Human Study"
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We utilized LLMs both component of our methodology and for editing the manuscript.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: //tprrip/////pp//e/// cc/Conferences /2025/ (2025/LLM) for what should or should not be described.
23

---

A Overview
Due to space constraints, we have included additional important content in the supplementary materials. Below is brief outline of the supplementary content to facilitate readers easily locate the relevant sections:
Appendix B: Experiment Setting Details
Appendix B.1: Details of Datasets
Appendix B.2: Implementation Details
Appendix C: Additional Experimental Results
Appendix C.I: Results of Robustness Against Unseen Perturbations
Appendix C.2: Results of Training and In-domain Testing on FF++
Appendix C.3: Results of monsppaanii Evaluation on DF40
Appendix C.4: Results of Experiments on Different L.Ms/MMLMM
Appendix C.5: Results of Experiments on GenAl Images
Appendix D: Human Study and GPT4 Evaluation
Appendix D.I: Human Study Details
Appendix D.2: GPT-4 Evaluation Details
Appendix E: Additional Analysis of Model Feature Assessment
Appendix E.1: Evaluation Setup MFA
-Appendix E.2: Evaluation of the Overall Detection Performance Details
Appendix E.3: Deeply Investigation of Individual Feature's Discrimination
Appendix F: Additional Analysis of Strong Feature Strengthening
Appendix G: Additional Analysis of Weak Feature Supplementing
Appendix H: Additional Analysis of Ablation Study
Appendix I: Sample Showing
B Experiment Setting Details
B.1 Details of Datasets
VQA Datasets Construction Details In this part, we detail the data construction process for our deepfake detection framework, 2DDDDD which integrates multiple stages: Multi-Feature Analysis (MFA), Strong Feature Selection (SFS), Weak Feature Selection (WFS), and Model Inference (MI). The process, illustrated in Figure 4, leverages multimodal large language models (MLLMs) and specific feature detectors (SFD) to construct a robust dataset for training and fine-tuning.
The data construction pipeline begins with the MFA stage, where an MLLM generates yes-or-no questions (e.g., "Is the image blurry?" or "Are there blending artifacts?") identify deepfake characteristics in given dataset. These questions are evaluated by the MLLM, producing a feature score list that ranks features by their relevance to deepfake detection. In the SFS stage, the top-K features are selected to generate real and fake prompts, such as providing reasoning for why an image might be deepfake or real. These prompts are paired with the deepfake dataset to construct fine-tuning dataset, where the MILM answers questions like "Is this image real or fake?" to generate labeled data. The WFS stage further refines this dataset by incorporating an External Dedicated Detector (SFD), which assigns confidence scores (e.g., SFD score 0.901 for fake samples) to filter out unreliable samples, ensuring the dataset's quality for fine-tuning. Finally, in the MI stage, the fine-tuned MILM, now strengthened -DFD, performs inference
samples, producing accurate deepfake detection results (e.g.. "This image is fake" with detailed reasoning about blending artifacts and omitted features). This multi-stage stage pipeline ensures that the constructed dataset is both comprehensive and reliable, enabling the 2DDDDD model to achieve robust performance across various deepfake detection scenarios.
Training and Testing Image Datasets We trained our model using the FF++ dataset [57]. For preprocessing and cropping, we adopted the methods from DeepfakeBench [78]. We utilized frames per video for training and 32 frames per video for testing.

---

Feature Generation
Feature Evaluation
Feature Ranking
Op la the imsa harrs?
Yes
Ope 1.st
Geserne N yeu-dren guotiime skanify
Qp image bloddng wtifees?
Eake Teal
649
MILMM
0.540
deyylkk chancetssiss Dxial images.
Oy iihht
Unsiaate Every Feature
316
463
Oy 0,52
Scree
Teatume Genemition Prompt
Fosture 1i
Deepfikke Donsee
Feature
Fesuro
Rant Last
Prompt Generation
Fine-tuning
Contruction
Model Fine-tuning
Frompt
Eatts
Real
Gescral Rale
Pronue
Pronge
Aralyaz the for
Provide shatt ieaso wayy imsgee
MLLM & Strengthened MILM
SPS
Q
pozze:
Fin--aring Ulsbact
MLLM
LLM
2. Enzziine othes-emito)
Genera Rale
Provido clor
Quastion
Answncts
Tap-k
General Bale for Strong Hioturee
irrgge night desplike.
Ourget
Reel Prosupt
External Dedicated Detector (EDD)
Fine-tuning Dataset Contruction
Model Fine-tuning
Despãão Durast
WES
abennostion cnteraal
MILM
mB
seate
Strengthened
Answes
Finetumed
EDD
Model
Freiuning
Darae
despfakz maats Sesture, eyes, and soout, aue cot conablest with otler. The facial contours tecture ratural, dxepitkke the image contain the Hending ariifucte
This takee
External
image or faie?! Bly the oberemation ecternal
Finetuned MLLM
Dedicated
Detector
Figure 4: Overview of the data construction pipeline for the 2FD framework. The pipeline
consists of four stages: (1) Multi-Feature Analysis (MFA) for generating yes-or-no questions to identify deepfake characteristics, resulting in a feature list; (2) Strong Feature Selection (SFS) for creating real and fake prompts based on top-K features, followed by dataset construction using MLLM-generated answers; (3) Weak Feature Selection (WFS) for fine-tuning the dataset with SFD scores, filtering samples based on reliability; and (4) Model Inference (MI) for final evaluation.
B.2 Implementation Details
The data splits and preprocessing follow the DeepfakeBench [78]. We train with a learning rate of 2 x 10-5 in two layer mlp projector and x 10-4 in others which borrow from the official implementation of LLava [43], rank of 16, and an alpha value set conventionally to twice the rank at 32. We training for three epochs, for each epoch on NVIDIA 4090 (Driver Version: 535.247.00 7001; CUDA Version: 12.2), AMD 32-Corecost for 4 hours by training on FF++ [57], each video we take frames for training. AUC is calculated by directly obtaining the token probabilities. Previous AUC calculations for large models mostly relied on averaging methods, such as [28], but this approach is not very accurate because: (1) multiple samplings are needed to approximate the true probability distribution, and (2) large models inherently perform inference with default temperature, which itself involves sampling over probabilities. Averaging over multiple samples effectively results in second layer of sampling, making the evaluation less accurate. Therefore, this paper, we calculate AUC by directly obtaining token probabilities.
25

---

Additional Experimental Results
C.1 Results of Robustness Against Unseen Perturbations
To evaluate the robustness our model to random perturbations, we follow the methodology outlined in previous studies [24, 90], which examines four types of degradation: Gaussian blur, block-wise distortion, contrast changes, and JPEG compression. Each perturbation is applied at five different levels to assess the model's performance under varying degrees of distortion.
To highlight the advantages of our approach over conventional detectors like FWA [36], SBI [60], and X-ray [32], we conducted multiple evaluations. As illustrated in Figure 5, which shows the video-level AUC results for these unseen perturbations using a model trained on FF+ c23, our method consistently demonstrates superior robustness compared other RGB-basee methods.
MIC PN
MAX PHU
12
Level
Figure 5: Robustness evaluation. We adopt four types of degradation for examining the robustness of our model: Gaussian blur, block-wise distortion, contrast changes, and JPEG compression. Our model shows superior robustness over other compared models.
C.2 Results of Training and In-domain Testing on FF++
In manuscript, mainly focus on the cross-domain evaluation assess the generalization performance of different models. Here, we conduct the in-domain evaluation on the FF++ dataset and compare our approach with the other four SOTA methods: FWA, Face X-ray, SRM, and CDFA. Following DeepfakeBench [78], we train all models on FF++ (c23) and test them on FF++ (c23), FF++ (c40), FF-DF, FF-F2F, FF-FS, and FF-NT. As shown in Table 9, the in-domain results demonstrate that our framework achieves the best performance, outperforming all other methods.
Table 9: In-domain results in the FF++ dataset (AUC)
Detector
FF++223
FF++c40
FF-DF
FF-F2F
FF-FS
FF-NT
AVG
FWA [36]
87.7
73.6
92.1
90.0
88.4
81.2
85.5
Face X-ray [32]
95.9
79.3
97.9
98.7
98.7
92.9
93.9
SRM [47]
95.8
81.1
97.3
97.0
97.4
93.0
93.6
CDFA [41]
90.2
69.0
99.9
86.9
93.3
80.7
90.2
Ours
96.6
82.6
99.9
97.2
98.1
91.0
94.2
C.3 Results of Cross-manipulation Evaluation on DF40
Evaluating model's performance mrossmmanpplattion tasks helps assess whether it handle previously unseen fake types. We use the recently released DF40 dataset [75] for evaluation. Our method generally outperforms other models average, particularly the e4s, Inswap, and SimSwap methods (see Table 10). This shows that our method effectively learns more generalizable features for detection, even against the latest techniques.
C.4 Results of Experiments Different LLMs/MLMMs
We conducted experiments using various models, including GPT4o [1], Claude3.5-Somm [3], and LLa Va [44], to evaluate the adaptability and robustness of our framework.
26

---

Table 10: Cross-manipulation evaluations within the FF++ domain (frame-level AUC only). We leverage the DF40 dataset [75] and select six representative face-swapping methods generated within the FF++ domain, keeping the data domain unchanged. The top two results are highlighted, with the best result in bold and the second-best underlined.
Method
Venues
uniface
e4s
Facedancer
Esgan
inswap
simswap
Avg.
RECCB [S]
CVPR 2022
84.2
65.2
78.3
88.4
79.5
73.0
78.1
SBI [60]
CVPR 2022
CORE [49]
CVPRW 2022
81.7
63.4
71.7
91.1
79.4
69.3
76.1
II [26]
CVPR 2023
79.5
71.0
79.0
86.4
74.4
64.0
75.7
UCF [77]
ICCV 2023
78.7
69.2
80.0
88.1
76.8
77.5
LSDA [73]
CVPR 2024
85.4
68.4
75.9
83,2
81.0
72.7
77.8
CDFA [41]
2024
76.5
72.0
ProDet [10]
NIPS 2024
84.5
71.0
73.6
86.5
78.8
77.8
78.7
Ours
85.5
90.3
82.6
91.1
81.2
85.1
85.9
Different LLMS to Generate Questions in MFA. In the MFA stage, we employed different L.Mss, such as GPT4o and Claude 3.5-Sonnet, to generate forgery-related questions and test the adaptability of our framework. The results, shown in GPT4o LLaVa-7B and Claude 3.5-Sonne' + LLaVa-76 demonstrate consistent performance regardless of the LLM used. Questions from Claude 3.5-Sonnet were also effective (see Table 21 and 22).
Table 11: Experiments different LLMs/MLLMs conducted to evaluate their performance under various conditions. The evaluation metric used for these experiments is the Area Under the Curve (AUC)
Variant
CDF
DFDCP
DFDC
DFD
Uniface
cds
Facediaceer
FSGAN
Inswap
Simswap
Human Writinn +.LaV--78 [43]
89.1
89.7
86.1
89.1
78.7
GPT LLLNa-7B [13]
90.3
89.7
83.5
92.5
85.2
91.2
83.8
89.9
78.5
81.9
$7.0
GPT4a]]] Qwen2VL-78 [67]
89.8
90.3
82.9
93.3
84.9
91.2
81.6
90.1
79.9
84.0
86.8
Claude3. Chadee..-Someel 5.5.--5e LLNNa-78
GPT LLaNa-13B
91.3
90.3
83.1
92.5
86.0
92.5
84.5
91.0
80.6
85.1
87.8
Different MILMs for Fine-tuning in SFS and WFS. the SFS and WFS stages, we investigate the impact of using different sizes of MLLMs with the same architecture during fine-tuning. For instance, we compare GPT4o + LLaVa-7B and GPT4o + LLaVa- / 1BB. The results indicate that as the size model increases, the performance of the framework in Table 11 improves proportionally, benefiting from the enhanced capabilities of the larger MLLMs. Additionally, we tested Qwen2VL-7B, which yielded performance comparable to LLaVa-7B.
Comparison of Human-generated and Model-generatee questions.
We also compared the effectiveness of human-written questions with those generated by advanced LLMs, such GPT4o and Claude 3.5 5.5-Sonnet, in the MFA stage. The results on Table 11, exemplified by Human Writing + LLaVa-7B (Avg AUC 86.2) versus GPT4o + LLaVa-77 (Avg AUC 87.0) and Claude 3.5-Sonnet LLLVVa7BB (Avg AUC 87.0), reveal that human-generater generated questions perform comparably to model-generated ones. Notably, questions produced by LLMs often exhibit greater competitiveness, leveraging their ability to generate diverse and nuanced forgery-related prompts, which further enhances the framework's is adaptability and performance.
Summary of Findings. These experiments collectively highlight the robustness of our framework. It not dependent on specific LLMs or MLLMs, making it adaptable to a wide range of models. Fur- thermore, the performance and size of the underlying models improve, framework effectively leverages these advancements to achieve enhanced results.
C.5 Results of Experiments GenAl Images
In addition to face swap forgery, we further explored the effectiveness of our framework on GenAl Image. Please refer to the supplementary material for more details and examples.
27

---

D
Human Study and GPT4 Evaluation
D.1 Human Study Details
The human study was designed to evaluate the explainability of deepfake detection model by involving human evaluators in assessing three key dimensions: detection ability, reasonableness of explanations, and level of detail. Below are the key details of the experimental setup:
Participant Recruitment. We recruited 15 well-educated participants and provided them with detailed guideline ensure clear understanding of the experimental task. Participants were aged between 20 and 40.
Task Description. Participants were presented with set of 100 samples, each consisting of deepfake image, the model's detection output (X²-DDD), and associated explanation generated by Multimodal Large Language Model (MLLM). They were tasked with scoring the model's performance across three dimensions on scale from to
Detection Ability: How accurately does the model identify the deepfake? (0 completely incorrect, 5 perfectly accurate)
Reasonableness of Explanations: How logical and understandable is the explanation? (0 completely unreasonable, 5 = highly reasonable)
Level of Detail: How detailed and specific is the explanation? (0= no detail, 5 very detailed)
Study Procedure. The study was conducted using custom-built GUI. Participants completed an initial 10-minute training session to familiarize themselves with the task and scoring criteria. Each participant evaluated random 50 samples, ensuring diverse coverage of the dataset. The study took approximately 20 minutes per participant, and participants volunteered their time without compensation.
Dataset. All samples were sourced from the testing datasets of DD-VQA. Each sample included both the raw image and the model-geneee gederrted explanation, with focus on unannotated regions.
Evaluation Metrics. Scores for each dimension were averaged across all participants. Additionally, participants provided a final overall score for each sample on scale from 0 to 5, categorized as follows: 0 (very poor), (poor), (fair), 3 (good), (very good), and 5 (excellent).
GUI of Human Study. he human study was conducted using custom-designed Graphical User Interface (GUT) to facilitate efficient and user-friendly evaluation. The GUI was developed using Python with the flask library and hosted on server. Below are the details of the GUI design: The GUI consisted of three main panels (see Figures 6 to for screenshots):
Image Panel: Displayed the deepfake image on the left side of the screen, with zoom and pan functionality for detailed inspection.
Explanation Panel: Presented the model's detection output (x22DDD) and other tested MLLLM's explanations on the right side.
Scoring Panel: Provided sliders for each answer allowing participants to select a score from 0 to 5.
User Interaction. Participants could navigate between samples using "Next" and "Previous" buttons. A progress bar indicated the number of samples completed. The GUI automatically saved scores database after each submission.
D.2 GPT-4 Evaluation Details
In addition to the human study, we conducted evaluations using GPT-4o to comprehensively assess the unannotated explainability of our deepfake detection model. This subsection provides the
28

---

Answer Evaluation
Background: Deepfwlas syrthelic media, tysicelly videoe created advenoed artficial inteligenoe techniquee like learning to menipulille generala reatiaic-kooking contents often suparimpoeing persor''s face onto another's rilaing concern sbout misinformaiin and privacy.
Ground Truth: fake
Description: mage manppulates dighally woman nose large
appearance imagn
manipulation leads unnturai
specific exaggerate features, the woman's
evaluate model's
dimensions:
Defection Accuracy: Explanations: sufficinnt
well-ounnded
explanations?
Based the above criteria, please provide your final assesement of the model's response.
Ploc
Cood
Dusiieal
Figure 6: Human evaluation
Answer Evaluation
Background: Decofakes are synthctic media, typically videos or mages, created using advanced artficial intelligence tachniques like deep learning manipulate generate realssicclooking content, often by superimposing one person's face ento another's, s. raising concerss about misiformaiion and privacy.
Ground Truth: real
Description:
shows man mustache, wearing
making a surprised
syea, appeer imrgee overall composition image suggest genuine, roal-fff momant captured
evaluate model's based dimensions::
Detestion Accuracy:
Reasonableness Explanations: and
explanations? appropritt anelysia??
Level Detail:
Based on the above oriteria, please provide your final assessment of the model' response.
Findl Soore:
Next
Figure 7: Human evaluation
implementation details of the automated evaluation process using GPT-4o, including the code structure, prompt formatting, and result processing.
The evaluation process was implemented in Python, leveraging the OpenAl API to interact with GPT-4o. The script performs the following key steps:
Data Loading. The script loads a JSON file containing metadata for the evaluation samples, including image IDs, ground truth labels (e.g.. "real" or "fake"), and model-generated descriptions.

---

Ground Truth: take
Description: image
distorted image Addttionlly, eyeis descrited heving altered, fake artifiial
lint,
factors suggest image digially manipulated
model's resporse
kay dimansions:
1. Delection Accuracy:
answer same Ground explanations?
Reasonableness Explanations: plausible
Does provide and appropriate analysis?
Based the criteria, please provide final assessment of model's response.
Final Soore:
Proviuus
Next
Figure 8: Human evaluation
Background: Decptaiios are synthede typically imaess oreated using advanoodd tochniques deep leamning manis genorate personts
about miliformation
Dosoription: The image is real es shows woman with gi glasses en to tolevision, The woman is wooring glasseo and appcars satement. presence ielovision background furthrr supports authentiily
Please evelcate the model's response based on three key dimensions::
Detection Acsurncy:
model Ressonablenee of Explanations: How plausiie wel founted modelfs explanat
3. Level of Detail: Dows the model piovide suffiient and appropiate dotal its analyels?
Based above criteria, please provide final assessment the model's
Final Scom:
Reuaeen
Provicus
Nost
Figure 9: Human evaluation
Prompt Formatting. For each sample, the script formats an evaluation prompt that includes the ground truth label and the model's description (the blue guideline shown in Figure 6). The prompt instructs GPT-4o to evaluate the model's response across three dimensions detection accuracy, reasonableness of explanations, and level of detail-and provide scores 0-5 scale (0 Very Poor, 5 = Excellent). Additionally, final overall score is requested.
Image Encoding. The script encodes each image into Base64 format enable GPT-4o to process both the textual prompt and the visual content. Images are sourced from the directory
Result Processing and Storage. The script processes all 100 samples, storing the results in list. Each result entry includes the image ID, image path, ground truth, prompt, and GPT-4o's response. The final output ISON file provides a comprehensive record of GPT-4o's evaluations, which are later aggregated and compared with human scores.
30

---

Model Comparison Dashboard
Model
Fales Dov. satura snnurthencs, mocs eyes, novesioe shading
Thes mmppr uilil fymmu unfe altero lbend wemen foos so exposstes mmgge quilitg wnll dnceeaed divegeats Fauld naansent Ovanal,
mase appaess digitall alandd manipulitte The sua scoggendtad fastoee unaara ightiig aggent photogapph nather udied
VL-2 Insoust
shardd weaippllieed.. The aoagenated fattaees and unaaaaal tititnne
nathr
image
ouss
iaaeets shons naman blo pirc lpetkt worse appsars be mana repersec fecial emers destrtioes indicaaa
consiiter
Figure 10: Answer from different model
Deseriptions
Lieer 3.2 118 Warke
This phots man mass anchor polttician blury, which saggents than cropped imasge thene bacipeound baired
photsgrph, The image appeare asill from brasdicsts
screnshet from than
The imsae appears sereesshoc from telvisiinn shiw era aidoo, lho bookgooed and light sugges taken from bnoolcast er recerding
Owendh Mt--7i-tstrac
oues
losges because bewn digiualy rend aulated aneate as sesatua and anticcaal appearssio faoe boon altendd artiiial uneauunt, phosigoph These ebersente suppest the image has digitdly raa auuddddd uners Fall affind arperration being panaine
baskgound
Figure 11: Answer from different model
31

---

Goond Teatcreal
Model Descriptions
GP744
Unee 3.2
The image's aiarry poaniyy and the age appeare streping, ougpaste that mos loe csuse clumbiburreee Additoraily Oventl image eppeera a curd somove aleppin..
high coult saben quality carera device
Iasa-n' 5-75
The image appear bo real The abeinas siges digital nan voudatio or socrios would aaggectt take
The image appear to real
There sre obvibus of dightal mantoulete dattr or motitin that would suggeett
becosme showe fere,
redia fes
and distrried nossing
oheeed photo, other focalfeut withe supenta Blatd preuie oxprnse mallers of mank apprrciiso relfeer guldte lered
Figure 12: Answer from different model
Jacs, unveturl and the skin eyed.
Uame 2.2 115 Walos
apprets have maniana aand, and im not saso wiee. The wartants moath oppaare to soo agoi agpear to sse spart The imaage
Bove
inage appon be roal shows penoon with halk bur cantren and deganandi are out rlear mouugh doteurabe alocniiity of laage
tereal it with lang but fiv conkat beciganns nut oher esegt deternim authvetaily the
oues
suthe srtaedicee of Imags. The womans fic features are perficcityy symmssticaa and there are a df diseriion mani thet would indicate
Figure 13: Answer from different model
32

---

Project Sumarry:
Fith the rapid cevelopaent of Deeptake technslogy, the generati oe of take trages and videss has bocowo inercasingly realistic, postig signifian cant shroats society and individml privacy. Deepi aos Fiy various deep learning models for detecting Deepfake content, there are still notable shortzosttus in. difrernnces dirietly iapset humm trust understanding during the dotection process.
expleimbiiity and transparenyy. These
This project aimw mallunte
bettoee model"
explatncbillss lity in leeptake dotection and tuman intul tion tarough horaz eva fon study. &e will compure our model with current mainstrous Doopfake detection models the esplanations provided. The acudy will involve parcicipatts evalunting the detect ion resul and explenstioon lons fron 2ereens molels, le, neseesing the effectivenees veness wodel real-world app lostiioonn..
the ivenes, accarscy, oser trast
Figure 14: Human study material part
Project Significance:
This research will reveal the strengths and waaknesses of different Deepfake detection models is toras explaindbilit, esperially al famont with human intuition. By comparing our model with other minstreas sodels, we hopo to deconstrate that our model is not only conpetitive in detection sccurscy
schieves higher expleimability. The project outcoses will offer nee insights and directions for the future design and upplication fon of foopfokn de detection models, promoting development conhnoling ogl es.
autisfactio
tronsparent and reliable detection
Potential Side Iffects, liaaards, Energeney Plens:
Side Effects:
During the project, participants may experionce confusion or reduced trust real innges due expouur a large anount of Deep fake content, Leading ditficulty in distinguishing between inngos.
Take
Emergency Plan: will inform participents that there curvently sarfous offective fake de detection solut solutions iscluding the developing, which cun effectively counter Deepfake attacks to some estent. Through education and explaration, will help partislpants understand the progress of the technology and the available protective aeasures, echsncing their trast in real tmages.
are
Figure 15: Human study material part2
Potestial Ethical Issues and Countermatures (including Informed Consent, Benefit Distribution):
Privacy Protection, Physical
Informed Consent: Fartieipents nned to fully understond the resmarch iee will ensure that all participants
purpose, and potestial
volustaril fly sian Informed consent forms Counteraasure: Provi de detailed research explanat lons and Q&A cessions to ensure participants fully understand agree participate.
Privacy Prosection:
All partielpart information and data involved in the
resourch will be keot atrictly confidentinl and will not be used for any purposes
research. Countermessure: [mpl emple incryttin Lon and monteiionton to ensure partleipunt privacy is not
coeproatsed.
Physicel Harm: Although this study mannly involves perchological and cogni assessannss,
nintaino posential poychologioal inpacts parsicipents and avoid any form of porchologiesl stress or hare.
Counteracasure: Conduct real-tine sonitoring during the stody parsicipant mental heilth, and provide part perticipnnts with the right kithdrow From the study at any cime.
unfairly dering the study and that the outcomes of the research do not dispropotionately benefit dissdvantage individull Countermcasure:
Distribution:
parsicipants
reasonable distribution snsuring opennees and iransparency. The research resul ts will primurtly focus acadenie and social contribation, not for persona] or commecial pain.
research sutcomes,
Figure 16: Human study material part3
33

---

E Additional Analysis of Model Feature Assessment
E.1
Evaluation Setup of MFA
Model. We choose the mainstream MILM,, i.e., LLaVA [44] as the implementation instance of the pre-trained MILMMs Additionally, we choose classical detector, Xception [12], baseline model for comparison.
Dataset. We evaluate the models on several widely-used deepfake datasets, including the Deepfake
Detection Challenge (DFDC) [15], the preview version of DFDC (DFDCP) [16], DeepfakeDetection (DFD) [13]. Celeb-DF-v2 (CDF-v2) [37]. DF40 dataset [75], which incorporates state-of of-the-art forgery techniques such as Facedancer [56]. Fsgan [50]. Inswap [58], e4s [33]. Simswap [7], and Uniface [91]. providing comprehensive foundation for evaluating overall detection performance.
Evaluation Metrics. We use the Area Under the Curve (AUC) as the primary evaluation metric, enabling to assess the model's ability distinguish between real and fake images across the whole dataset. In this section, we use the frame-level AUC for evaluation. For individual feature discrimination, we focus forgery-related features such "Is the face layout unnatural?" with responses of either "yes" or "no." The proportions of "yyes" and "no" answers for real and fake images are calculated as follows, with the ranking score S14) defined based on the balanced accuracy of the responses:
Y-(0)
(=
5(9)
fake
Y- real
N real
(( fake
fake
Here, y-(3) of "no" answers for real and fake, respectively. This formulation ensures that both true positive and true negative rates are considered, providing a balanced measure of feature discrimination.
real and y-1 denote the number of "yes" answers, while N-61) fake
real and N2(0) take represent the number
F,2 Evaluation the Overall Detection Performance
The comparison between LLaVA [44] and Xception [12] highlights a notable performance gap. Results Figure 17 (left) indicate that the average AUC for LLaVA is 63.7%, while Xception achieves 75.8%, showing notable gap of 12. % points. This suggests that, while the LLaVA has certain zero-shot capabilities other tasks such as (general) image classification, it is still not as strong the traditional detector in detecting deepfakes.
However, LLaVA shows strong detection abilities in specific methods (eg., e4s), sometimes even surpassing Xception (see Figure 17 (left)). This motivates us to further investigate its intrinsic detection capabilities, and understand the model's "strengths and weaknesses" in deepfake detection. Below, we provide a detailed investigation of the discrimination of each forgery-relatee feature.
Features
610
197
2100
ArB
Feature Index
Figure 17: (Left) AUC comparison between (zero-shot) LLaVA (blue) and Xception (red) for deepfake detection across different datasets; (Right) Balance accuracy score for individual feature discrimination, with Strong features in the top-left corner and Weak features in the bothom-right corner based on discrimination scores. Full questions/features are provided in the Table.
E.3
Deeply Investigation of Individual Feature's Discrimination
Step 1: Question Generation. For each candidate forgery-related feature, we formulate corre- sponding interrogative statement. For instance, the feature "blurry" is transformed into the question "Is the image blurry?". Recognizing that the candidate features are not pre-specified by developers, we employ Large Language Model (LLM), i.e., GPT-4o, to automatically generate a comprehensive list of F; questions. These questions target key forgery indicators, including but not limited to lighting anomalies, unnatural facial expressions, and mismatched textures, which are critical for identifying deepfakes.
34

---

Step 2: Question Evaluation. Each generated question is paired with an image from the assessment dataset form a prompt for constructing the fine-tuning dataset. The model responds with binary output ("yes" or "no") based on its interpretation of the image in relation to the question. These responses are aggregated into a confusion matrix for each question, thereby quantifying the detection capability of the associated forgery-related features. Mathematically, for each question F; and image Zy the MLLM produces:
R Muuse R),
where Ri,, (yes, no), representing the model's 's response for each image-question pair.
Step 3: Question Ranking, According to the accuracy of all candidate questions, we obtain descending ranking of questions, i.e., the ranking of forgery-related features. This ranking allows us to quantify how well each feature contributes to distinguishing between real and fake images. Specifically, the accuracy of each question is computed by evaluating the proportion of correct responses across the dataset. Specifically, for each question Q We calculate the true positive rate (TPR) and true negative rate (TNR), then take their average to obtain the Balanced Accuracy, as follows:
TP: TP; + FN;
TN: TN; + FP,
Balanced Accuracy,
where: TP, denotes True Positives for question F., TN; the True Negatives for question F,, FP; the False Positives for question F,, and FN; the False Negatives for question F..
Subsequently, questions are ranked in descending order based on their balance accuracy scores, thereby prioritizing forgery features that effectively discriminate between real and fake images.
Strong Features. Strong features typically involve semantic-level facial structural or appearance anomalies. As shown in the strong feature section of Fig. 17 (right), which primarily includes facial irregularities such as unusual facial layouts (eg.. Rank 9, 11, 17) or distorted facial features (eg.. Rank 3, 4, 14), eg.. the nose, eyes, or mouth. Since the pre-trained MLLM is good at extracting and utilizing these features for detection, it can provide more reliable and accurate explanation.
Weak Features. Weak features typically involve fine-grained, low-level lextures, such blending anomalies. As shown in Fig. 17 (right), these weak features are primarily subtle details related to texture, reflection, shadow, and blending. Examples of texture issues include rough or overly smooth surfaces (eg., Rank 68, 77, 83). Furthermore, inconsistencies in lighting and shadows (eg., Rank 85, 86, 90, 96) and blending artifacts on the face (eg., Rank 54, 84, 88) are also prominent. Since these signal-level anomalies are challenging for pre-trained MLLMs to detect, the pre-trained MLLM is likely to struggle reliably distinguishing between real and manipulated content when relying on these weak features for detection and explanation.
Feature Score. The scores for different forgery-related features are presented, where Table 19 highlights the top 50 strong features, and Table 20 shows the 50 weakest features based on their scores.
Does the model know these features are related to deepfake?
We used a series of questions to query the model, applying simple prompt augmentation with the feature-related questions mentioned above. "yes" indicates the model knows these features are related to deepfake detection, while a "no" indicates the model does not. Detailed results are shown in Table 17 and Table 18.
F Additional analysis of Strong Feature Strengthening
Following the Model Feature Assessment (MFA) module, we observed significant performance improvements in forgery detection after applying the Strong Feature Strengthening (SFS) module, The generalization performance improved by 20% AUC (see Table 14) on average compared to the pretrain model.
we conducted a detailed analysis by re-evaluating the model using the Model Feature Assessment (MFA) module to compare the discriminative capability of forgery-rclated related features before and after
35

---

applying the SFS module. As illustrated in Figure 18, over 60% of the features exhibited enhanced discriminative power post-SFS, with particularly notable improvements strong features.
Feature Capabilities Ablation
Balance Average
Safer
80
Figure 18: Comparison of feature capability before and after SFS. After adding the external detector to supplement the MLLM, the model's feature capabilities (almost all) can be further improved.
G Additional Analysis of Weak Feature Supplementing
In addition to the used blending model [41], we also try other instances to implement the SFDs in the WFS module of our framework, each targeting specific types of artifacts, where SRI focusing on the generative artifacts by deep nets, F3-Net [53] focusing on the frequency-level anomalies, and SBI and CFDA focusing on the blending boundaries. Based on these empirical attempts, We summarize the general criteria under which conditions the selected SFD instance can be used in our framework. Specifically, the integrated SFD instance should meet the following criteria:
Criteria-l: Each SFD instance should focus on only one type of feature that is positively correlated with fake;
Criteria-2: The score given by the SFD instance can accurately reflect the characteristics of the corresponding feature;
Criteria-3: The data distribution of this feature in the dataset is relatively uniform.
Below, we show a detailed illustration of using other SFD instances for implementation one by one.
AIGC Expert Integration. We first consider implementing an AIGC expert to learn the deep genera- tive artifacts. For implementation, we introduce the SRI model, based on self-reconstruetion images generated by Simswap [7] and train on the Xception model, designed to capture self-reconstruction generative features. However, from Figure 19, integrating this model into our framework results in only a minor performance improvement of 0.3%. Further analysis reveals a negative correlation between the model's 's features and fake labels in the training set (do NOT meet the Criteria-1), indicating that these artifacts are poorly represented in the training data. Consequently, the model struggles to leverage the expert-provided features effectively, offering limited benefits over not using the expert model.
20.--
4.0
1.0
250-
145-
13-
50
04
50
72-
23-
60
43
C4
12
02
50
23
24
3.4
1.
1.
Figure 19: The probability distributions of different expert models on the FF++ training dataset. From left to right, the models are SRI, F3-Net, SBI, and CDFA, corresponding to experts in capturing relftreconttructionn, frequency anomalies, and self- self-blending artifacts, respectively. The blending here directly uses the trained weights.
Frequency Expert Integration. We then integrate a frequency- based model F3-Ner and train on the FF++ dataset [57] to capture frequency anomalies. However, from Figure 19, the overall model's 's performance identical to that of the expert, with no improvement. Although the expert features are positively correlated with fake labels, the frequency-based scores are overfitted to the training set and do not accurately reflect the true feature quantity, with only near-1 (1 for fake) and near-O (0 for real) predictions (Not satisfy the Criteria-2) This leads to a shorteut, where the model relies solely on the
36

---

expert's output without learning from the feature information, thus limiting the extendability of the integrated model.
Table 12: Comparison of methods across datasets with values rounded to two decimal places, where the evaluation metric is AUC, The "Diff" column shows the difference from the milms average.
Variant
CDF
DFDCP
DFDC
DFD
Uniface
cds
Facedanser
ESGAN
Inswap
Simswap
Avg
Dift
MLLM
83.3
82.0
79.2
91.4
84.5
94.1
79.9
88.0
77.2
83.3
84.3
0.0
SRI
42.9
97.3
69.1
SRI+MLLM
83.2
82.5
77.6
88.8
85.6
95.8
81.9
88.5
77.9
84.6
84.6
+0.3
77.0
77.2
82.3
87.5
75.4
77.2
79.4
F3NeL-MLLM
76.8
77.8
73.1
83.3
88.4
75.5
76.6
89.8
84.6
78.4
80.4
-3.9
82.1
85.5
83.4
87.7
SBI+MLLM
88.6
85.5
75.6
90.8
58.7
93.7
77.6
88.
61.6
91.1
86.2
+1.9
CDFA
87.9
86.6
90.9
76.5
76.1
CDFA+MLLM
90.3
89.7
83.5
92.5
85.2
91.2
83.8
89.9
78.5
84.9
87.0
+2.7
SBI and CFDA Models Integration. We also integrate another blending-basee expert model, SBI [60], which specializes in detecting blending artifacts. From Figure 19, we can see that trained using self-blending techniques on real images to prevent overfitting, the SBI model's expert features show strong correlation with fake labels, and scoring effectively quantifies the extent of blending artifacts. Similarly, the incorporation of the CFDA model [.1], enhanced version of the SBI model, results in an additional performance boost, indicating that as the expert model's ability capture blending features improved, the overall model's generalization capability also increases.
To explain criteria-3, conducted additional experiments using non-uniform data distribution. Specifically, we created an extremely imbalanced dataset by removing large portion of fake samples that do not contain the blending feature. As the imbalance increased, the model's performance degraded, and in extreme cases, it began to rely on shortcut solutions. In the remove 95 and remove 99.5 cases, we removed 95% and 99.5% of samples close the real distribution, respectively, resulting in highly imbalanced datasets with mostly fake samples remaining.
Table 13: Performance Comparison of Different Models Various Datasets. The remove 95 and remove 99.5 scenarios represent extreme cases of data imbalance by removing 95% and 99.5% of the samples near the real distribution, respectively.
Varient
Celeb-DF-v2
DFDCP
E4S
Facedancer
Fsgan
Inswap
Simswap
Average
remove 99.5
75.6
79.0
63.6
67.2
80.2
63.0
70.6
remove 95
79.3
81.4
68.9
69.7
82.1
65.7
70.3
73.9
CDFA+MLLM
90.3
89.6
91.2
83.8
89.9
78.5
84.9
86.9
H
Additional Analysis of Ablation Study
Effects of Strong Feature Strengthening Module. We observed substantial performance boost the model after fine-tuning with dataset constructed using strong features, as evidenced by the leap from Variant-1 to Variant-5. Remarkably, this significant improvement occurred even without the aid of specific feature detectors (WFS), underscoring the potency strong feature utilization. This finding prompted us to investigate the underlying causes. We reexamined the feature capabilities of the pre- pre-trained model (Variant-1) and compared them to those of the model enhanced with strong features (Variant-5). As illustrated in Figure 18, the majority of feature capabilities exhibited marked enhancement following the application of strong feature strengthening. Intriguingly, even some initially weaker features demonstrated noticeable improvement post-enhancement.
Effect of Model Feature Assessment Module. Without Model Feature Assessment (MFA), identi- fying the model's strong features becomes impossible, and relying on weaker features undermines both the reliability of the model's explanations and its overall performance. To explore this, we allowed the model to construct the dataset using the entire list of forgery-related features rather than prioritizing strong ones. In Variant-3, which adopts this approach without enforcing strong feature use, performance lags significantly behind Variant-5, where MFA pinpoints and leverages strong features for fine-tuning, as evidenced in Table 14. To further investigate the absence of
37

---

Table 14: Ablation study regarding the effectiveness of each proposed module via cross-datased evaluations. All models are trained on the FF++ c23 dataset and evaluated with metrics in the order of AUC || AP EER (frame-level). The results show incremental benefit in each module. We use / to indicate the presence of a module and X to indicate its absence.
Ours
CDF
DFDC
Simswap
Unilace
Avg.
MFA
SPS
WFS
AUC
AP
EEK
AUC
AP
EEK
AUC
AP
EER
AUC
AP
EER
AUC
AP
EER
AUC
AP
EER
1
x
x
X
52.1
08.2
48.7
69.8
95.2
16.4
57.8
59.9
44.6
64.0
64.1
40.4
65.5
65.6
39,0
61.8
l
70.6
41.8
2
x
52.1
4
49.4
75.0
96.0
31.5
63.5
66.0
39.9
59.3
59.6
43.7
57.
58.5
44.1
61.5
69.5
|1.8
79.0
88.3
998
18.0
77.8
81.9
84.0
25.9
82.3
84.8
82.0
87.3
2556
87.9
17.6
85.6
5
X
83.2
90.5
99.0
158
79.2
83.3
85.0
74.8
84.5
96-
22.4
23.0
5
X
J
88.1
93.6
20.4
91.1
98.6
16.8
82.0
84.5
26.1
78.0
76.5
28.8
78.7
77.1
28.6
83.6
86
24.1
7
/
X
/
87.3
93.2
21.0
90.2
98.7
18.0
82.0
82.1
26.6
76.7
75.0
29.5
71.2
75.8
29.0
82.6
8499
24.8
8
V
V
90.4
94,9
17.7
92.3
99.1
15.5
83.7
86.1
24.8
85.1
85.7
23.2
85.5
86.4
22.4
87.4
90.4
20.7
MFA, we introduced Variant-6, which integrates specific feature detectors (WFS) without MFA, and compared it to Variant-8, the full model with MFA, SFS, and WFS. The results clearly show Variant-6 underperforming Variant-8, highlighting a notable disadvantage when model assessment is omitted. These experiments Variant-3 versus Variant-5 and Variant-6 versus Variant-8 collectively demonstrate that assessing and utilizing strong features via MFA is critical for optimizing model effectiveness in deepfake detection. Strikingly, despite the pre-trained Variant-1's limited detection ability, Variant-2 reveals the model's feature capability by using a single MFA-identified strong feature (e.g.. distortion ilstortion--resentt for fake, absent for real) for direct judgment without training. This underscores the model's strong feature-identification capability and the potential of tapping these features, which combinations Variants and further amplify.
We evaluate the generalization performance of our model in cross-dataset evaluation scenarios through an ablation study involving several variants to systematically assess the contributions of different components. The variants include: Variant-1, pre-trained MLLM Lla Va-...5-7B[44] as baseline, without any feature strengthening or supplementation; Variant-2, which utilizes the single strongest feature identified by Model Feature Assessment (MFA) for deepfake detection; Variant-3, Assumes all features are strong and uses them to construct the dataset without prior evaluation or ranking, followed by fine-tuning the model; Variant-4, simple test of the specific feature detectors ability, for this model can only output a single probability, as proposed in [41]; Variant-5, only use strong features to construct datasets after feature assessment, and use this dataset to fine-tune; Variant-6, Uses all strong features without feature assessment, integrating the specific,faauue feature detectors (SFD) construct datasets, followed by fine-tuning the model.; and Variant-7, Constructs datasets solely using the specific feature detectors (SFD), masking visual information to prevent strong feature leakage. followed by fine-tuning the model. Variant-8. Full model, integrating MFA, SFS, and WFS. The results, presented in Table 14, demonstrate incremental improvements across various datasets. For instance, Variant-8 achieves the highest average AUC (87.4%), AP (90.3%), and lowest EER (20.7%), highlighting the synergy of MFA, SFS, and WFS in optimizing deepfake detection performance.
Feature Capability. feature strengthening.
We also conduct comparative study of feature capabilities before and after
Effect of inconsistent use of supplementary features in training and inference. The model performs best when supplementary features are used consistently during both training and inference (average auc: 87.8), indicating that these features significantly enhance performance. When supple- mentary features are omitted entirely from both stages, the performance drops (average auc: 0.83.3), though it remains better than when features used inconsistently. Specifically, when features are used during training but not inference, the performance suffers greatly (average auc: 76.6), suggesting the model relies on these features and struggles without them at inference time. On the other hand, when features are introduced at inference but not used during training, the model achieves slightly better results (average auc: 82.5), but it cannot fully leverage unseen features, showing the importance of using supplementary features consistently across both phases.
Extension in new datasets. Our model, trained on a mixture of datasets including FF++, showed improved overall performance when we added new dataset without blending artifacts to the training process. This demonstrates that incorporating diverse datasets with supplementary features, even
38

---

Table 15: Impact of omitting supplementary features during Training and Adding Them During Inference, on Model Performance
Variant
Celeb-DF-v2
DFD
DFDC
DFDCP
DFR
WDF
FFIW
Avg.
no train + no infer
83.2
91.4
79.2
82.0
93.7
76.8
76.6
83.3
train no infer
76.5
84.7
71.0
72.0
89.5
71.3
71.3
76.6
no train + infer
81.7
90.6
79.1
81.3
92.6
74.7
77.3
82.5
train infer
90.4
92.3
83.7
87.3
97.6
81.4
81.7
87.8
from different domains, as shown in Table 16 enhances the model's generalization and comprehensive performance.
Table 16: Comparison of Model Performance When Trained on FF++ Alone vs. FF++ and SRI Across Different Datasets
Variant
CDF
DFDCP
DFDC
DFD
Uniface
Facedancer
Fsgan
Inswap
Simswap
Truin IT++
87.3
83.7
92.3
91.2 93.0
85.1
Train FF++ and 91.5
89.3
83,9
92.7
87.4
84.6
89.9
81.0
86.1
I Sample Showing
Here we present some failure cases of pre-trained MILMss and then compare them with our results. See the supplementary material for more examples and details.
an image forensic expert Deepfake Check image artifact attribute listed the following list and ONLY return the attribute number in this image. The artifact list is (1-asymmetric eye iris; 2-irregular glasses shape or reflection: 3-irregular teeth shape or texture: irregular ears or earrings; 5-strange hair texture; 6-inconsistent skin texture: 7-inconsistent lighting and shading: 8-strange background; 9-weird hands; 10-unnatural edges).
you
Label: real
artifacts, reviowed image and found displays the following Stribbut::-strange hair texture. The texture of the hair appears somewhat artificial, common characteristic of Al-generated images.
Attribute number: Based on the provided
Detection
USER: This image real, tell the
The image provided appears be real person. possibly from television interview public appcarance. The reason it seems real is due to the natural lighting, human facial expressions, and other small details such
GPT:
Explanation
churacteristics authentic photograph person live setting. However, based this image alone, cannot confirm its exact context.
positioning,
this image deepfake, tell me reason.
Detection& Explanation
public real is due to the natural lighling, human facial expressions, and other small details such as skin texture and natural positioning, which are characteristics of an autbentic photograph of a porson in setting. However, based image alone,
image you provided appears uppearance. The
person, possibly
exact source context.
Figure 20: Diagram showing how the GPT4o generates results for detection, explanation, and detection with explanation.
For LLa Va, the same prompts GPT-4 ensure fairness in the evaluation process. The robustness of Llava in these tasks is illustrated in Figure 21.
39

---

Table 17: Relationship between various facial features and deepfake detection (Part I)
No.
Question
Pretrain
the face color related to deepfake detection?
No
2
Are the eyes related to deepfake detection?
No
3
Are the facial features related to deepfake detection?
No
4
Is the nose contour related to deepfake detection?
No
5
Is face blurriness related to deepfake detection?
No
6
Is the skin tone related to deepfake detection?
No
Are the cheeks related to deepfake detection?
No
8
Is the skin tone pattern related deepfake detection?
No
Is the placement of facial features related to deepfake detection?
No
10
Are the lips related to deepfake detection?
Yes
11
Is facial symmetry related to deepfake detection?
No
12
Is the lighting the cheeks related deepfake detection?
Yes
13
Is the facial lighting related to deepfake detection?
No
14
Are the shapes of facial features related deepfake detection?
No
15
Is facial evenness related to deepfake detection?
Yes
16
Are the cheekbones related deepfake detection?
No
17
Is the face layout related to deepfake detection?
No
18
Are the lip edges related to deepfake detection?
No
19
Is facial detail related to deepfake detection?
Yes
20
Is check smoothness related deepfake detection?
No
21
Is the forehead shape related to deepfake detection?
No
22
bace-background blending related to deepfake detection?
Yes
23
Is skin texture related to deepfake detection?
No
24
Are the eyelashes related to deepfake detection?
No
25
Are facial lines related to deepfake detection?
No
26
facial expression related to deepfake detection?
No
27
Is the nose shape related to deepfake detection?
No
28
Are color changes on the face related to deepfake detection?
No
29
Is the mouth shape related deepfake detection?
No
30
Are the face edges related to deepfake detection?
No
31
Is facial rigidity related to deepfake detection?
No
32
Are sharp facial lines related to deepfake detection?
No
33
Is skin perfection related deepfake detection?
No
34
Is forehead shininess related to deepfake detection?
Yes
35
Are sharp face edges related to deepfake detection?
Yes
36
Is skin smoothness related to deepfake detection?
No
37
Are eye details related deepfake detection?
No
38
Are smooth facial lines related to deepfake detection?
No
39
Is lip texture related deepfake detection?
Yes
40
Is forehead shine evenness related to deepfake detection?
No
41
Are the eyebrows related to deepfake detection?
No
42
Are unusual eye appearances related to deepfake detection?
No
43
Are facial transitions related to deepfake detection?
Yes
44
Is face color related to deepfake detection?
No
45
facial emotion exaggeration related to deepfake detection?
No
46
Is unusual face layout related to deepfake detection?
No
47
Are eye reflections related to deepfake detection?
No
48
Is skin texture roughness related deepfake detection?
No
49
Is the jawline related to deepfake detection?
No
50
Is facial expression stiffness related deepfake detection?
Yes
40

---

Table 18: Relationship between various facial features and deepfake detection (Part 2)
No.
Question
Pretrain
51
nose texture related to deepfake detection?
No
52
Is skin shininess under the nose related to deepfake detection?
No
53
uneven facial sharpness related to deepfake detection?
Yes
54
Is facial blending related to deepfake detection?
No
55
Is facial lighting evenness related to deepfake detection?
No
56
Is nose bridge smoothness related to deepfake detection?
No
57
Is the hairline related to deepfake detection?
No
58
Is skin texture evenness related to deepfake detection?
No
59
Is facial feature balance related to deepfake detection?
No
60
Is facial symmetry related to deepfake detection?
No
61
Is forced facial expression related to deepfake detection?
No
62
Are the nostrils related to deepfake detection?
No
63
Are unnatural lip appearances related to deepfake detection?
No
64
Is partial skin smoothness related deepfake detection?
No
65
Is lip texture related to deepfake detection?
No
66
Is lighting around the nose related deepfake detection?
Yes
67
Are facial feature proportions related to deepfake detection?
Yes
68
Is skin smoothness around the nose related deepfake detection?
No
69
Are soft facial creases related to deepfake detection?
No
70
Are teeth appearances related deepfake detection?
No
71
Is neck-face transition related to deepfake detection?
No
72
Is skin tone variation related to deepfake detection?
No
73
Is face edge sharpness related to deepfake detection?
No
74
Is chin outline visibility related to deepfake detection?
No
75
Is facial lighting evenness related to deepfake detection?
Yes
76
Are ear details related to deepfake detection?
No
77
Is chin smoothness related to deepfake detection?
No
78
Are bright facial areas related to deepfake detection?
No
79
Is skin brightness near the mouth related to deepfake detection?
No
80
Are nostril appearances related to deepfake detection?
No
81
Are dimples related to deepfake detection?
Yes
82
Is jawline prominence related to deepfake detection?
No
83
Is under-eye texture related to deepfake detection?
No
84
Is facial blending related to deepfake detection?
Yes
85
Is chin shadow related deepfake detection?
No
86
Are forehead shadows related to deepfake detection?
No
87
Is nose light reflection related to deepfake detection?
No
88
Is aice-background transition related to deepfake detection?
No
89
Is forehead light reflection related deepfake detection?
No
90
Are nose shadows related to deepfake detection?
No
91
Is lighting around the mouth related to deepfake detection?
No
92
Is neck smoothness related to deepfake detection?
No
93
Are face outlines related deepfake detection?
No
94
Are face edges related to deepfake detection?
No
95
Are skin details related deepfake detection?
No
96
Are under-eye shadows related to deepfake detection?
No
97
Are check shadows related to dcepfake detection?
No
98
Are cheekbone appearances related to deepfake detection?
No
99
Is facial lighting related to deepfake detection?
No
100
Are facial wrinkle details related to deepfake detection?
No
41

---

Table 19: Top Strong 50 Features
Rank
Question
Pretrain
Strengthened
Is the face color unusual?
0.6340
0.7486
Is there something wrong with the eyes? Do the facial features look oddly shaped? Is the contour of the nose incorrect? Is part of the face blurry? Does the skin tone make the face look fake?
0.6309
0.6320
0.6292
0.6636
0.6278
0,5817
0.6231
0.7643
6
0.6165
0.6479
Is there something wrong with the cheek? Are there strange patterns in the skin tone? Are the face parts out of place?
0.6144
0.8082
0,6144
0.8075
0.6130
0.7919
10
Do the lips seem out of place strangely shaped?
0,6127
0.7408
Is one side of the face uneven with the other?
0.6123
0.7622
12
Are there strange lighting spots the cheeks?
0.6111
0.8029
13
Does the lighting change strangely on the face?
0.6092
0.8006
14
Are the shapes of the eyes, nose, or mouth unnatural?
0.6054
0.6714
15
Does the face look uneven or off?
0.6048
0.6732
16
Does the cheekbone appear too flat? Does the face layout look wrong?
0.6014
0.7406
17
0.5986
0.5422
18
Are the edges of the lips too smooth?
0.5979
0.6264
19
Is part of the face lacking detail?
0.5942
0.6843
20
Are the cheeks too smooth?
0.5934
0.6728
21
Does the forehead look odd in shape?
0.5911
0.7493
22
Does the face mix poorly with the background?
0.5902
0.6382
23
Is the skin texture uneven?
0.5861
0.7158
Are the eyelashes missing or blurred?
0.5857
0.6733
25
Are the face lines uneven or changing in different areas?
0.5826
0.6546
Does the face lack expression?
0.5822
0.6679
27
Does the nose shape look odd?
0,5812
0.5306
Are the color changes on the face and skin sudden?
0.5807
0.6643
29
Does the mouth appear too flat?
0,5775
0,6542
30
Are the edges of the face too sharp? rigid?
0.5774
0,8188
31
Does the face appear
0.5770
0.7446
Are the face lines too sharp?
0.5761
0.7724
33
Does the skin look too perfect, like it was edited?
0.5755
0.5749
34
Is the forehead too shiny?
0.5737
0.8168
35
Are the face edges sharp?
0.5720
0.8162
36
Does the face skin look too smooth?
0.5640
0.5306
37
Are the eyes blurry or lacking detail?
0.5636
0.5362
38
Are the face lines too smooth?
0.5549
0.5927
39
Are the lips too smooth or lacking texture?
0.5537
0.5475
40
Is the forehead's 's shine uneven?
0,5515
0.7208
41
Are the eyebrows too dark or too light?
0.5454
0.5075
42
Do the eyes look odd?
0.5433
0.5389
43
Are transitions on the face poorly blended?
0.5410
0.5854
44
Do the face colors look strange?
0.5382
0.6163
Does the face show emotions that seem exaggerated?
0.5355
0.6337
46
Does the face layout look unusual?
0.5344
0.5377
47
Do the eyes have unnatural reflections?
0.5323
0.6417
48
Does the face have rough uneven skin texture?
0.5292
0.7973
49
Does the jawline appear too sharp or unclear?
0.5292
0,5017
50
Does the facial expression look stiff?
0.5289
0.5346
42

---

Table 20: Bottom 50 Weak Features
Rank
Question
Pretrained
Strengthened
51
Does the nose lack texture?
0.5231
0.5111
52
Is the skin too shiny under the nose?
0.5223
0.7458
53
the sharpness of the face uneven in parts?
0.5214
0.5701
54
Does the blending on the face look unnatural or uneven?
0.5212
0.5151
55
the lighting on the face strange or uneven?
0.5200
0.5968
56
Does the nose bridge appear too smooth?
0.5172
0.5395
57
Does the hairline seem unnatural?
0.5148
0.5495
58
Does the face skin texture look uneven?
0.5144
0.5489
59
Do the face parts look out of balance?
0.5137
0,5887
60
Are the facial features too symmetrical?
0.5130
0.7300
61
Does the facial expression look forced? Are the nostrils hard to see?
0.5116
0,5562
62
0.5115
0.6535
63
Do the lips look unnatural?
0.5110
0.5555
64
Does the face skin look smooth in some areas?
0.5089
0.5287
65
Do the lips lack natural texture?
0.5083
0,5855
66
Is the lighting around the inconsistent?
0,5080
0.7257
67
Do the sizes of the eyes, nose, and mouth seem off?
0.5038
0.5275
68
Does the skin around the nose look unnaturally smooth?
0.5030
0,5309
69
Are the facial creases too soft?
0.5028
0.7943
70
Do the teeth appear blurry or unrealistic?
0.5028
0.5210
71
Is the transition between the neck and the face not smooth?
0.5026
0.5330
72
the skin tone different in parts of the face?
0.5023
0.5749
73
Does the face lack sharpness around the edges?
0.5021
0.5311
74
the chin outline hard to see?
0.5021
0.6160
75
Is the lighting uneven the face?
0.5012
0.6351
76
Are the details around the ears unclear?
0.5010
0.6751
77
Is the chin too smooth compared to the rest of the face?
0.5010
0.5664
78
Do the bright areas on the face seem odd?
0.5007
0.5196
79
Is the skin near the mouth unnaturally bright?
0.5007
0.5930
80
Are the nostrils blurry or unclear?
0.5007
0.5125
81
Are the dimples missing poorly defined?
0.5005
0.5000
82
the jawline too pronounced or too faint?
0.5000
0.5003
83
Is the area under the eyes missing natural texture?
0.5000
0.5111
84
Is there blending on the face that looks edited?
0.5000
0.5014
85
Does the shadow under the chin seem unnatural?
0.5000
0.5090
86
Is the forehead missing natural shadows?
0.5000
0,5000
87
Does the light reflection on the nose look strange?
0.5000
0.5049
88
Are the transitions between the face and the background poorly blended?
0,5000
0.5447
89
Does the light reflection on the forehead look artificial?
0.5000
0.5007
90
Are there missing shadows around the nose?
0.5000
0.5217
91
Does the lighting around the mouth look unusual?
0.5000
0.5301
92
Does the neck look unnaturally smooth compared to the face?
0.5000
0.6259
93
Do the face outlines look off?
0.5000
0.5247
94
Do the edges around the face look unnatural?
0.5000
0.5299
95
Are the fine details on the skin missing?
0.5000
0.5165
96
Are the shadows under the eyes missing?
0.5000
0.5000
97
Are the checks lacking shadows?
0,5000
0.5000
98
Do the checkbones appear unnaturally smooth?
0.5000
0.5709
99
Does the face appear overly lit in certain areas?
0.5000
0.6758
100
Are the wrinkles on the face lacking detail?
0.5000
0.5014
43

---

Table 21: Questions list generated by Clande3.5-Sonnet (partl)
No.
Question
1
Are there noticeable inconsistencies in facial symmetry? Return yes
2
Does the skin texture appear artificially smooth or lacking natural details? Return me yes or no Are the eyes misaligned or disproportionate? Return me yes or
3
4
Is there unnatural blending between facial features and background? Return me yes or no
5
Do shadows and lighting appear inconsistent across the face? Return me yes or no
6
Are facial expressions unnatural or mechanically rigid? Return me yes or no Does the hairline show signs of artificial blending? Return me yes or no Are there visible artifacts or glitches in the image? Return me yes or no Do reflections the eyes match the environment? Return me yes or no Is there proper alignment of facial features? Return me yes or no Does the skin show natural imperfections and pores? Return me yes or no Are teeth shapes and alignment realistic? Return yes
7
8
9
10
11
12
13
Is there consistent image quality across all facial areas? Return me yes or no Do facial proportions follow natural human anatomy? Return me yes no
14
15
Are shadows cast appropriately based on lighting? Return me yes or no Does facial hair follow natural growth patterns? Return yes
16
17
there proper depth and dimension to facial features? Return me yes or no Are color tones consistent throughout the face? Return yes
18
19
Do glasses and accessories appear properly attached? Return me yes or no
20
Is there natural variation in skin texture? Return me yes
21
Are facial contours anatomically correct? Return me yes or no Does the head size match body proportions? Return me yes or no Is there appropriate detail in fine features? Return me yes or no Are transitions between features naturally blended? Return me yes or no
22
23
24
25
Do facial movements appear fluid and natural? Return me yes or no Are ear shapes and positions symmetrical? Return me yes or no
26
27
Do eyebrows have natural hair patterns? Return me yes or no
28
Is there consistent resolution between face and background? Return me yes or no Are nose contours anatomically accurate? Return me yes or
29
30
Does makeup application appear natural? Return me yes or no Are facial wrinkles and lines age-appropriate? Return me yes no
31
32
Do eyelashes appear realistic and properly attached? Return me yes or no
33
Is there natural skin coloration variation? Return me yes or no Are facial highlights consistent with lighting? Return me yes or no
34
35
Do lips have natural texture and color? Return yes or there proper depth in eye sockets? Return me yes or no Are facial moles and marks naturally placed? Return me yes no Do teeth have individual characteristies? Return me yes or no
36
37
38
39
Is there natural asymmetry in facial features? Return me yes Are skin pores visible where expected? Return me yes or no
40
41
Do facial muscles move naturally? Return yes
42
Is there consistent focus across the image? Return me yes or no Are shadows under facial features natural? Return me yes or
43
44
Do earrings and jewelry sit naturally? Return me yes or no
45
Is there proper skin subsurface scattering? Return me yes or no Are facial proportions consistent in different angles? Return me yes or no
46
47
Do eye corners have natural creases? Return me yes or no
48
Is there natural variation in lip texture? Return me yes or no Are facial hair shadows realistic? Return me yes or no
49
50
Do glasses cast appropriate shadows? Return me yes no
44

---

Table 22: Questions list generated by Claude3.5-Sonnet (part2)
No.
Question
51
there natural skin translucency? Return me yes or no Are facial expressions emotionally consistent? Return me yes or no Do neck muscles align naturally? Return me yes or no
52
53
54
Is there proper depth in smile lines? Return me yes no
55
Are eye reflections consistent with scene lighting? Return me yes or no Do facial features maintain proportion when moving? Return me yes or no there natural skin aging present? Return me yes or no Are hair strands individually visible? Return yes
56
57
58
59
Do facial veins appear natural where visible? Return me yes or no
60
Is there consistent skin tone across transitions? Return me yes no Are nostril shapes symmetrical? Return me yes or no
61
62
Do ears have natural internal structure? Return yes there proper depth in nasolabial folds? Return me yes or no Are eye bags and circles age-appropriate? Return me yes or Do facial piercings sit naturally? Return me yes or no there natural variation in beard density? Return me yes no Are lip lines naturally defined? Return me yes or no Do cheekbones have natural contours? Return me yes or no
63
64
65
66
67
68
69
Is there proper temple definition? Return me yes or no Are eye whites naturally textured? Return me yes or no
70
71
Do facial scars appear authentic? Return me yes or no there natural jaw definition? Return me yes or no Are facial dimples naturally placed? Return me yes no
72
73
74
Do eyebrow hairs have direction variation? Return me yes or no
75
Is there proper chin definition? Return me yes
76
Are facial freckles naturally distributed? Return me yes or no
77
Do eyelids have natural creases? Return yes there consistent skin shininess? Return me yes or no Are facial tattoos properly embedded? Return me yes
78
79
80
Do smile lines appear natural? Return me yes or no
81
Is there proper forehead texture? Return me yes or Are eye corners naturally aged? Return me yes or no
82
83
Do facial muscles show proper definition? Return me yes
84
Is there natural lip symmetry? Return me yes or no Are ear lobes naturally shaped? Return yes
85
86
Do facial shadows have color variation? Return me yes or no there proper nose bridge definition? Return me yes or no Are facial pores consistently sized? Return me yes or no
87
88
89
Do eyebrows have natural thickness variation? Return me yes or
90
Is there natural skin clasticity? Return me yes no
91
Are facial creases novement-appropriaaee Return me yes or no
92
Do teeth have natural translucency? Return me yes or no there proper cheek coloring? Return me yes or no Are eye bags naturally shadowed? Return me yes or no
93
94
95
Do facial features maintain proper scale? Return me yes or no
96
Is there natural skin undertone? Return me yes
97
Are facial expressions muscle-consistent? Return me yes or no
98
Do wrinkles have proper depth? Return me yes no there natural facial bone structure? Return me yes or no Are skin textures consistently detailed? Return yes
99
100
45

---

USER: I want to work an image forensic expert for Deepfake faces. Check the image has the artifact attribute listed in the following list and ONLY return the attribute number in this image. The artifact (1-asymmetric eye iris; 2-Irreuular glasses shape or reflection; 3-iregulaar teeth shape texture; irregular earring:: 5-strange texture: 6-inconsistent skin texture; 7-inconsistent lighting and shading; 8-strange background; 9-weird hands; 10-unnatural edges].
Label: real
Detection
LLaVa:
image real, tell reason.
The for the imuge real related the subject maiter depicted the image, which with a beard and mustache. The man appcars to be the famous actor, John Wayne, known for his distinctive facial appearance movies. The image a close-up, and John Wayne's facial features are clearly visible and rocognizable. It's photo-shoppad digitally altered image. The context of the image John Wayne talking someane, which is also consistent with his well known roles.
LLaVa:
Explanation
this image
deepfake,
reason.
Detection& & Explanation
LLaVa: The image is real, as it is an autheatic photo of John Wayne, a famous actor known distinctive board and mustache. The close-up view of John Wayne's face in the image allows for clear recognition of his facial features, making a genuine represenlation of him.
Figure 21: Diagram showing how the LLaVa generates results for detection, explanation, and detection with explanation.
USER:Is image
OURS:
This image is real. The image appears be real, the facial features, such his nosc, eyes, mouth, are well-defined and not distorted. The skin tone appears natural, and there are signs of blurriness unusual lighting. The man's facial expressions and overall appcarance seem genuine, making likely that the image man.
Label: real
Detection
true representation
image does not any blending artifacts.
USER:
This image real, tell the
The image the man's face appears to real, it displnys well-defined structure with clear skin, prominent eyes, and defined, and there is no noticeable blurriness or distortion in the image. The overall appearance of the man's face suggests that a gemuine representation the individual. And this image does not vontain any blending artifacts.
OURS:
neither overly smooth
flat checkbones. The man's facial features well-
Explanation
USER: this image real fake?
This image is real. The image appears to be real, as the man's facial features, such as his nose, eyes, and mouth, are well-defined unusual lighting. The man's facial expressions and ovenll appearance genuine, making likely that the image is a true representation of the man, And this image does not contain any blending artifacts.
OURSS:
Detection&
not distorted. skin appears natural,
are signs of blurriness
Explanation
Figure 22:
Diagram showing how the ours model generates results for detection, explanation, and detection with explanation.
46

---

image
reason
MLLM:
The image is roal, featuring a bald man wearing a blue shirt and a white collar. He is talking and has mustache. The image not photoshopped. The man appears be reporter presenter, and his expression suggests that he is engaged in discussion or interview. The image geouine representation of man und
Fake
imago, resson
MILM:
The image is fake or manipulated picture a bald man with gostee, likely created for entertainment artistic purposes. The reason this manipulation
humorous unconventional visual effect, make statement, generate curiosity and discussion. The image have been created using photoshup sollware, resulting unusual combination of a bald man with goalev. This manipulation gees beyond typical represcutation beld mon, adds distinctive element image that not commonly seen in real life.
Fake
Figure 23: Bad sample of Pre-trained model (part1)
image or
detail.
The image is real, shows a woman with blue eyes and rod lips. The image also features TV in the background, which is an indication that the setting is likely TV studio or live broadcast. The the TV and the woman's makeup sugyest that this real, live situation, rather than fake manipulated image.
MLLM:
USER: Why this image fake?
The image is fake because the person's 's fuce uppears to be photoshogped, This can be inferred from the fact that her eyes do appear looking the camers, and the lighting and background consistent. Additionnlly,, the woman's face distosted, and she has fake eyelashes and blurry face. The lace and appearance, making clear image not genuine
MLLM:
manipulation creates an photograph.
Fake
Figure 24: Bad sample of Pre-trained model (part2)
What the feature fake in this image?
In this image, the photoshop feuture being used is the "face swap' technique. This technique involve replacing person's face with another person's is face, creating blend of the individunls. In this case, man's face has been swapped with another man's face, resulting in unique and unconvventional appearance. you can the the other without, creating mix of their facial features.
image. image collage two different people, one with heard and
Fake
Ql: Is this image photoshoppod? Retur
Q2: the "lace swap" used in the image! Return yes
Q3: Does the imagu show bland of face? Retum
A2: Yes, swop
Yes, image
fuses.
Was man's face moonventional appearance? Retum yos
Q5: there any blur the image?
Q6: Does the image mix ficial feature?? yes no.
Return yes
Fake
A4: Yes.
A8: Yes, the image is blurty.
Afe Yes.
Q!: : this image photoshoppol? Retur yes
Q2: the "fice soup" techniqqe noed in this image? Return yes no.
Qt: the image show bland two face?? Retum yes
A2: swap
Q4: Was man's face mnsorvattioall appearance? Retum
QS: there any blur the image?
Qúi Does the image mix facial festures?? Retum yes no.
Return yes
Real
No.
AS: Yos, the image Murry..
Figure 25: Bad sample of Pre-trained model (part3)
47

---

USER: Is this image fake? Tell the reason detail?
This image appears real. shows woman speaking into camera with her mouth wide open, possibly in an interview setting. Deepfake images often involve more discrepancies in facial features, lighting, and overall content when compared to regular image with a genuine, real-world socnario. image seems capture
Pretrained MLLM;
moment, so does not suggest any deepfake content possibility.
USER:
This image is fake, tell me the reasons.
The inage is fake because manipulated photo of a woman with different background. The woman's face placed different body, creating unnatural and unrealistic appearance. This manipulation commonly done entertainment, artistic puppses,, or create humorous effect. The image does not depict a real-life secnario, and the altered elements make it a fake representation of persun.
Pretrained MILM:
USER: image real or fake?
DDFFD : This image is fake, The image appears to be deepfake us the woman's 's face appears to be distorted, with an unusual layout unnatural tone. The ficiial features seem to mismatched, and overall appearance with real person. And image show the obvious blending artifact.
consistent
Figure 26: Comparison between Pre-trained LLaVa and ours
Q: What is the feature of fake this image?
A: this image, the photoshop feature being used the "face swap' technique. This technique involve replacing one person's 's face with another person's face, creating blend of the two individuals. In this case, man's face been swapped And you notice the blur in image. The image is collage of two different people, with beard and the other without, creating mix of their facial features.
man's face, resulting unique unconventional appeurance.
Fake
Q1: Is this image photoshopped??
Q2: the "lace swap" used in the image? Retam
Q3: Does the image show blend of faee? Return
Retum
A1: Yes
A2: Yes, face swap is used.
A3: Yes, image shows blend of two Bloos.
Q4: man's inconvantional appearance? Renum no.
any blur image?
Q6: Dees image
Return yes or no.
feutures? ? Return yes no.
Fake
A4: Yes.
A5: Yes, the image blurry.
QI: is this image photoshuppod? Retum or
Q2: the "Grve swap' tochmique used in this image? Retura yes no.
Q3: Does the imager show blend of two face? Return or
Al:
A2: No, face swop not used.
A3: No.
Q4: man's unconventt unconventional appearance? Retum yes oc no.
Q5: there blur in the image?
Q6: Does the image mix facial fentures? Return yes no.
Return yes or no.
Real
image blurry.
No.
Figure 27: Feature related questions
48


Reviews:
[{"review_id": "06oVQbKSar", "reviewer": "Reviewer_uoAb", "summary": "This paper presents a detailed exploration of the potential of MLLM for deepfake detection. It focuses on one of the main challenges in this area — how to build a good image-text dataset. The author raise two central questions — whether MLLM can interpret forgery clue descriptions generated by humans or GPT, and which types of clues are more effective — and provide preliminary experimental validation. They further propose enhancing the model's strong features while supplementing weak ones using external deepfake detectors. A concrete framework is introduced and validated through extensive experiments.", "questions": "1. Why is fine-tuning MLLM only with strong features more effective than using all available features? The paper demonstrates this via experiments, but there’s not much explanation behind it.\n2. As noted, the proposed method still heavily relies on the capabilities of specific detectors. Could the authors analyze the deeper reason behind this? Is it because the visual encoder of the MLLM fails to capture fine-grained forgery cues?\n3. For deepfake images/videos without obvious artifacts, how can one obtain accurate and reliable textual descriptions to effectively utilize the capabilities of MLLM?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The paper suggests a good way to generate text descriptions of forgery features with little manual effort, which helps solve a key problem when using MLLM for deepfake detection.\n2. It presents a well-designed integration framework that leverages the complementary strengths of MLLM and Specific Feature Detector.\n3. The paper conducts thorough ablation studies to demonstrate the significance and effectiveness of feature selection and other individual components.\n4. The paper is well written, easy to follow, and the ideas are clearly explained.\n\n### Weaknesses\n1. The overall performance boost comes from using the SFD module. It’s not very clear if MLLM alone can really beat traditional deepfake detection models.\n2. The method works well on datasets with clear fake artifacts, but it may not be strong enough for high-quality fakes, like those made by diffusion models. This is understandable, but still a limitation.", "was_revised": true, "final_justification": "The authors have patiently and thoroughly addressed the concerns I raised, providing clear explanations and additional clarifications where necessary. While I appreciate their efforts and responsiveness, my overall assessment of the paper remains unchanged, and I am therefore maintaining my original rating.", "rebuttal": "Dear Reviewer uoAb，\n\nWe sincerely appreciate your time and effort in reviewing our work. We are extremely encouraged by your positive assessment and your recognition of our work's key strengths, including its **efficient data generation method, a well-designed integration framework, detailed ablation studies, and clear writing**. In the following, we would like to respond to your valuable comments separately.\n\n---\n\n>**Q1.** The overall performance boost comes from using the SFD module. It’s not very clear if MLLM alone can really beat traditional deepfake detection models.\n\n**R1:** Thank you for your great observation. To verify the contribution with MLLM alone, we conduct an additional ablation (see **Table 1**), comparing the generalization performance among SFD, MLLM alone, MLLM + SFD (ours).\n\nWe summarize the key observations below:\n\n- **The MLLM alone is also competitive and can beat traditional models on certain testing datasets.** As shown in the \"Average\" column, MLLM only, trained using our proposed strategy, achieves an average AUC of **82.6%**, demonstrating competitive—and even superior—average performance compared to the SOTA traditional detector CDFA **(81.0%)**, a SFD specifically designed for capturing blending artifacts.\n- **SFDs typically perform well on specific types of fakes but fail to maintain generalization across all types.** As shown in **Table 1**, CDFA (a blending-based SFD) performs very well on blending-based deepfakes such as CDF and DFDC (where additional blending operations are used in face-swapping, leaving strong blending artifacts), but struggles on non-blending face-swapping methods (where entire images are generated), achieving only 76.1% and 76.5% on SimSwap and UniFace, respectively. **In contrast, *MLLM only* surpasses CDFA by nearly 10% on these cases.**\n    - This finding validates our core finding: **the MLLM is a strong generalist detector, yet it exhibits specific low-level \"weaknesses\"** (not as effective as SFDs specifically designed to capture certain fine-grained artifacts). This is precisely why our \"Weak Feature Supplementing\" (WFS) module is necessary, and why the full X2DFD framework achieves the best overall performance.\n\n\n**Table 1: Ablation study comparing the performance of SFD, MLLM only, and ours (measured by AUC\\%)**\n| | CDF | DFDC | Simswap | Uniface | **Average** |\n| :--- | :--: | :--: | :--: | :--: | :---: |\n| **SFD (CDFA, ECCV 2024)** | 87.9 | 83.5 | 76.1 | 76.5 | 81.0 |\n| **MLLM only** | 83.2 | 79.2 | 83.3 | 84.5 | 82.6 |\n| **SFD + MLLM (ours)** | 90.4 | 83.7 | 85.1 | 85.5 | **86.2** |\n\n\n\n\n---\n\n>**Q2.** As noted, the proposed method still heavily relies on the capabilities of specific detectors. Could the authors analyze the deeper reason behind this? Is it because the visual encoder of the MLLM fails to capture fine-grained forgery cues?\n\n\n\n\n**R2:** Thank you for the question. We would like to clarify that our proposed framework does **not heavily rely on specific feature detectors (SFDs) in a dependent or restrictive manner**. Instead, it is designed as an adaptive fusion approach that **leverages two complementary components**: the MLLM and the SFD.\n\n- Specifically, the MLLM excels at identifying general, high-level inconsistencies and semantic anomalies in images, while the SFD is specialized in detecting fine-grained, localized artifacts—such as blending boundaries or texture irregularities—that are often indicative of image manipulation. **By adaptively combining these two complementary signals, our framework achieves a synergistic effect**, where the whole is greater than the sum of its parts (\"1+1 > 2\") in terms of detection performance.\n- This design choice is **grounded in evidence from recent studies**, such as [1,2], which demonstrate that MLLMs and traditional detectors capture different types of forgery cues and thus offer complementary strengths. Therefore, rather than depending solely on SFDs, our method strategically integrates their fine-grained sensitivity with the broader contextual understanding of MLLMs, leading to more robust and comprehensive detection.\n\n\n\n---\n\n>**Q3.** The method works well on datasets with clear fake artifacts, but it may not be strong enough for high-quality fakes, like those made by **diffusion models**. This is understandable, but still a limitation.\n\n\n**R3:** Thanks for your very insightful comment! Following your suggestion, we add an additional experiment for testing our model on high-quality diffusion-based fakes, suggested by the reviewer. Specifically: \n- We test our models on the **DiFF dataset** [3], a **diffusion-based** face forgery dataset. We then incorporate a **new SFD specialized in detecting diffusion artifacts** [4].\n\n- **Specifically, our framework maintains effective performance on diffusion-based forgeries.** Results in **Table 2** demonstrate that the MLLM trained by our pipeline achieves **significant improvement** over the traditional baseline (Xception), \n    - Furthermore, we see that **adding an additional SFD (a diffusion-based expert) can further enhance** the detection performance. This highlights the advantage of our framework's extendability, as integrating a domain-specific priors like SFD yields substantial improvements.\n\n- Additionally, it is worth noting that **MLLM alone can also learn some *generalizable* discriminative features for detection.**  Although not all these features are explainable for human. \n    - Results in **Table 3** show that human performance (leveraging human-understandable features for detection) is substantially lower than that of the MLLM. This indicates the MLLM can capture certain discriminative patterns beyond human perception—features that are generalizable even if not fully explainable to humans.\n\n\n**Table 2 Practical extension and validation for diffusion-based forgery, evaluation metric is AUC**\n\n| | CDF | Simswap | Diff-FE | Diff-FS | Diff-I2I | Diff-T2I |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Xception** |73.7 |77.6 | 58.6 | 85.9 | 56.8 | 62.4 |\n| **MLLM alone** | 83.2 | 83.3 | 80.1 | 90.1 | 80.3 | 84.2 |\n| **X2DFD(w/ SFD)** | **90.3** |**88.5** |**92.1** |**97.2** |**88.6** | **92.2**|\n\n\n\n**Table 3 Experiments on without obvious artifacts, evaluation metric is fake ACC**\n\n| | FreeDoM_T | CoDiff |\n| :--- | :---: | :---: |\n| **Human performance (ACC)** | 25.4 | 27.8 |\n| **MLLM only** | 59.8 | 53.4 |\n| **X2DFD (W/ SFD)** | **79.9** | **70.5** |\n\n---\n\n>**Q4.** For deepfake images/videos **without obvious artifacts**, how can one obtain accurate and **reliable textual descriptions** to effectively utilize the capabilities of MLLM?\n\n\n\n**R4:** Thank you very much for this meaningful quesiton. However, please note that obtaining a completely accurate and reliable caption for realistic deepfakes without obvious artifacts is highly challenging and remains an open problem for the entire field. \n\nGenerally, we think there are two main solutions to measure the reliability of MLLM's explanations: \n- For those **semantically understandable features**, **human** can act as a reasonable judger (see the **human study of our manuscript**), since human (experts) are capable of understanding most high-level semantic concepts.\n- For those **low-level detection signals**, it is very difficult for humans to interpret (but might be captured by machine). This is because **not all discriminative features are human-explainable**. In such cases, we argue that an **SFD (as a domain expert) can offer a more reliable prediction than a human**, especially when its prediction is made with high confidence. This is precisely why we design an extendable framework that allows integration of any SFD to enhance the overall reliability of the final decision.\n\nIn our framework's implementation, guided by these two principles, we first instill human semantic knowledge into the MLLM during training, enabling it to generate human-understandable explanations. Then, for more realistic fakes where low-level artifacts dominate, we feed the SFD's prediction into the MLLM, allowing it to produce a **final explanation that includes both the human's understanding and SFD's domain-specific prior**, thereby improving overall reliability.\n\n\nTo summarize, evaluating explanation reliability—especially for highly realistic fakes—remains an open challenge. When features are semantically interpretable, humans can act as judges; otherwise, we should rely on domain experts (i.e., SFDs). This dichotomy **directly motivates the design of our extendable framework.**\n\n---\n\n\n\n>**Q5.** Why is fine-tuning MLLM only with **strong features more effective than using all available features**? The paper demonstrates this via experiments, but there’s not much explanation behind it.\n\n**R5:** Thank you for this question. Fine-tuning on \"strong features\" is more effective because it focuses the learning process on **a set of highly discriminative features**, whereas including \"weak features\" introduces disruptive noise. Specifically,\n\n* **Focusing on highly discriminative features**: Fine-tuning only on \"strong features\" concentrates the learning process on **reliable, relatively high-quality forgery features** that the MLLM \"familiar with\" and can effectively utilize for detection.\n* **Avoiding disruptive noise**: Including weak features in the training can introduce disruptive noise. Our preliminary investigation has demonstrated that the MLLM struggles to use these weak features for detection (resulting in lower detection accuracy).\n\n\n---\n\n**References:**\n\n[1] Q-bench: A benchmark for general-purpose foundation models on low-level vision. ICLR, 2024\n\n[2] Can chatgpt detect deepfakes? a study of using multimodal large language models for media forensics. CVPR Workshop, 2024\n\n[3] Diffusion Facial Forgery Detection. ACMMM, 2024\n\n[4] Aligned Datasets Improve Detection of Latent Diffusion-Generated Images. ICLR, 2025"}, {"review_id": "h2kI4dSjVb", "reviewer": "Reviewer_M9Ae", "summary": "This paper introduces a multi-stage approach for leveraging MLLMs for deefpake detection.\n \nFirst, a frozen MLLM is asked to generate a list of questions for assessing whether some deepfake characteristics are present in an image (ie skin color consistency, mouth blurriness...). Then each of these questions is passed to a frozen along with deepfake data MLLM to observe whether the MLLMs is capable to detect the characteristics described in these questions (blurriness, consistency... the authors call them artifacts or sometimes features, which confusing with respect to actual features as in a feature space).\nThis assessment helps defining what questions/artifacts the MLLM is capable of detecting in a zeroshot way. The questions then are ranked and a top-k are selected to stage 2.\nStage 2 consists in using the questions of stage 1 to generate textual prompts (real and fake) to augment some deepfake dataset.\nThis helps building a VQA-style dataset that can be used to finetune an MLLM and making better what it already is. The intuition of finetuning is since the MLLMs is already good in zeroshot on these question, finetuning, would make it better.\nSince the MLLM cannot pick up all possible artifact, the authors incorporate the predictions of a blending-based detector (the fake-it-till-you-make-it one) to compensate for the lack of performance of the MLLM for some questions/artifacts. The output of this specialized detector is also used as an annotation.\n The final model is finetuned using the VQA style built dataset which contains the initial textual prompts from stage 2 and the score of the specialized detecto", "questions": "Other/questions:\nIt would be nice if Tab2 indicated an arrow regarding the metrics to indicated whether higher/lower is better \nWhat would be the results if the model used LAA-Net as a dedicated detector (although this still remains within the assumption that MLLMs are only performing badly on blending-based artifacts)\nWhy is \"Photoshop traces\" irrelevant aren't Adobe using AI in their software nowadays?\nWhy is the title including \\chi as X, it sounds phonetically wrong.\nWhy is the work does not generalize to other types of deepfakes, doesn't this mean that the generated questions are not actually indicative enough of deepfake artifacts, or that maybe they are biased towards faceswap?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Extensive implementation details.\n- Several datasets are considered, including DF40\n\nWeaknesses:\n- Unclear added scientific value/novelty, although the work introduces a sophisticated pipeline for explainable forgery detection, it remains incremental and mostly seem like an engineering solution rather than a research one\n- In the related works, the limitations of existing benchmarks on forgery detection using VQA i.e. [33,19,29] are unclear.\n- Several typos: Fig 1 caption: \"improvsed\", L38 \"relies\", L82 \"there\", L170, L173, \\mathcal{M} is undefined.\n- The writing style is often vague, for examples the fact of referring to the questions/artifacts learned by the model as \"features\", or statements like \"learning to learn\" etc, so, overall the writing could benefit from more clarity.\n- The reliance on a blending-based detector as the dedicated detectors assumes that MLLMs are only bad in blending-based forgeries, whereas they can be bad in on other types of artifacts.\n- The performance on CDF, DFD and DFDC is extremely close LAA-Net which is a simpler blending-based approach. This makes the proposed very complex in comparison to dedicated detector, which only requires generating self-blended images on the go.\n- Too many acronyms making the paper hard to read\n- It is unclear whether the generated question are reliable, MLLMs are can often give different answers making the annotations and mostly random. Was there some process that ensured that the questions being ranked are actually reliable, like some sort of asking the same question 100 times and average over the answers?\n- The claims on extendability are not convincing and seem overstated, if by being extendable the authors mean using off-the-shelf existing approaches to generate pseudo-labels, build a dataset and train an existing architecture, then previous methods such as Face-Xray and sbi are also extendable.\n- The complexity of the proposed approach is unclear.", "was_revised": true, "final_justification": "The authors have provided a solid rebuttal. I decided, therefore, to increase my rating. It is recommended to include all the new elements discussed in the rebuttal in  the final version of the paper.", "rebuttal": "Dear Reviewer M9Ae，\n\nThank you for your thoughtful comments. We hope the following responses alleviate your concerns.\n\n---\n\n>**Q1.** Unclear added scientific value/novelty, mostly seem like an engineering solution rather than a research one.\n\n**R1:** We thank the reviewer for the feedback. We would like to clarify that our core contribution is not merely an engineering pipeline, but a **novel framework for adapting MLLMs to deepfake detection**.\n\nSpecifically, our unique scientific contributions include:\n\n* **We are the first work to systematically assess the intrinsic capabilities of MLLMs for deepfake detection**, identifying its 'strong' and 'weak' features and then prioritizing features the model can reliably detect (strong features), which is new in the whole field.\n* **We are the first work that proposes an adaptive framework to effectively fuse and leverage the complementary capabilities of MLLM and SFD for comprehensive detection.** In this manner, we fully leverage the MLLM's strengths while supplementing its weaknesses by involving specific domain priors from SFDs.\n\nOverall, we believe this framework represents a significant research contribution and scientific value to the whole community.\n\n---\n\n>**Q2.** The reliance on a blending-based detector as the dedicated detectors assumes that MLLMs are only bad in blending-based forgeries, whereas they can be bad in on other types of artifacts.\n\n**R2:** We appreciate the reviewer for this valuable point. We would like to clarify two key points:\n- **We do NOT consider that MLLMs perform poorly only in detecting blending-based forgeries**—*blending traces are merely a prominent example used for illustration*.\n    - The weaknesses of MLLMs, as identified by our MFA and supported by previous studies [1, 2], mainly lie in detecting fine-grained, low-level clues, and blending artifacts are just one notable instance of such clues. To address this limitation, our framework is designed to adaptively integrate and leverage the strengths of both MLLMs and specialized detectors (SFDs), regardless of the type of artifacts involved.\n- **Our proposed framework is NOT limited to supporting blending-based SFDs**; instead, it is designed to be *fully extendable to and compatible with any dedicated detector*.\n    - To demonstrate the flexibility and effectiveness of our framework in handling diverse artifact types, **we have integrated a new SFD specifically trained on diffusion-based artifacts**.\n    - Results in **Table 1** confirm that our framework is **not restricted to blending artifacts but is broadly effective across different types of forgeries**.\n\n**Table 1: Comprehensive performance evaluation across diverse forgery artifacts, evaluation metric is AUC**\n|Model|CDF|Simswap-DF40|Diff-FE|Diff-I2I|Diff-T2I|\n:-|:-:|:-:|:-:|:-:|:-:\n**CDFA (ECCV2025)**|87.9|76.1|74.6|81.7|87.3\n**X2DFD(Ble-CDFA)**|90.4|85.1|82.1|81.7|88.3\n**X2DFD(Ble+Diff)**|90.3|88.5|92.1|88.6|92.2\n\n---\n\n>**Q3.** The performance on CDF, DFD and DFDC is extremely close LAA-Net which is a simpler blending-based approach. This makes the proposed method very complex compare to dedicated detector, which only requires generating self-blended images on the go.\n\n**R3** Thanks for the great question and your concern. Although the MLLM is larger than traditional detectors, the extra capacity enables it to learn richer, more diverse cues for real-vs-fake discrimination. As **Table 2** shows, when trained with our strategy the MLLM alone already outperforms the blending-based baseline, confirming that the additional parameters are effectively utilized for a more general detection, not merely overhead.\n\nAdditionally, we must claim that **evaluating *only* on the three datasets CDF, DFD, and DFD, is not fair to compare** the performance between MLLM and those blending-based methods. Here is the reason:\n- **The three benchmarks are all \"blending-based\" deepfakes**. CDF, DFD, and DFDCP, mentioned by the reviewer, are created with face-swapping pipelines that explicitly include a final \"blending\" step to merge the donor face into the target image.\n- **Modern deepfakes might no longer need blending**. Recent generators such as SimSwap, InSwapper can directly synthesize the entire swapped face without any explicit blending stage.\n    - The DF40 benchmark (NeurIPS 2024) already demonstrates that blending artifacts are largely absent in these new forgeries.\n    - Therefore, performance on CDF/DFD/DFDCP alone does not reveal how well a detector generalizes to current, non-blending deepfakes.\n    - To provide a fair comparison, we re-evaluated both LAA-Net (blending-based) and our model X2DFD(Ble+Diff) on different forgery types in **Table 2**.\n- **Our MLLM-based model can learn more general patterns across different forgery types, even for those latest ones without blending:**\n    - **LAA-Net drops sharply on non-blending forgeries**, confirming its reliance on blending artifacts.\n    - **Our MLLM-based X2DFD remains stable**, showing stronger generalization beyond specific artifacts.\n\n**Table 2 Experiments comprehensive datasets compare to LAA-Net, evaluation metric is AUC.**\n\n|Model|DFDC|Simswap-DF40|Uniface-DF40|SDXL-Diff|HPS-Diff|**AVG**|\n:-|:-:|:-:|:-:|:-:|:-:|:-:\n**LAA-Net(CVPR2024)**|78.5|80.9|80.7|78.8|80.8|79.9\n**MllM only**|79.2|83.3|84.5|83.2|88.7|83.8\n**X2DFD(Ble+Diff)**|**83.2**|**88.5**|**90.1**|**89.5**|**95.4**|**89.3**\n\n---\n\n>**Q4.** What would be the results if the model used LAA-Net as a dedicated detector (Although the model is still blending-base)\n\n**R4:** Following your suggestion, we conduct new experiments **integrating** different blending-based SFDs **LAA-Net**, and compared their performance with our original **CDFA**-based version. The results are presented in **Table 3**, which clearly indicate that our **X2DFD framework demonstrates strong and consistent performance regardless of the specific blending-based detector used**.\n\n**Table 3: Performance with different blending-based SFDs, evaluation metric is AUC**\n|Model|CDF|Uniface|Diff-FE|Diff-I2I|\n:-|:-:|:-:|:-:|:-:\nX2DFD(Ble-LAANet)|89.8|86.3|80.5|81.2\nX2DFD(Ble-CDFA)|90.4|85.5|82.1|81.7\n\n---\n\n>**Q5.** It is unclear whether the generated questions are reliable, MLLMs can often give random answers.\n\n**R5:** Thank you for this important question on reliability. We address this with a two-part strategy to ensure our generated questions are reliable **(detailed in lines 320-336 in the manuscript)**:\n\n* **Human-in-the-Loop Filtering:** We use a hybrid process where an MLLM first generates a candidate pool of questions. Our annotators then manually **filter this pool to remove any ambiguous or low-quality questions**. This crucial step ensures only reliable questions are used for ranking.\n\n* **Deterministic Ranking:** To eliminate randomness during the ranking process itself, we use a low temperature setting (**T=0.2**). This standard practice [1,2] makes the MLLM's output highly consistent and effectively deterministic for any given input.\n\n---\n\n>**Q6.** The claims on extendability are not convincing and seem overstated, if by being extendable the authors mean using off-the-shelf existing approaches to generate pseudo-labels, build a dataset and train an existing architecture, then previous methods such as Face-Xray and sbi are also extendable.\n\n**R6:** We sincerely thank the reviewer for this insightful comment, which allows us to further clarify the novel and distinctive nature of \"extendability\" in our framework.\n\n- Our notion of \"extendability\" is **fundamentally different from prior works** like Face-Xray or SBI, which focus on training detectors using pseudo-fakes. Instead, we leverage the MLLM as an intelligent \"agent\"—capable of reasoning and tool use—to **dynamically integrate diverse expert detectors (e.g., blending- or diffusion-based) as complementary sources of forensic priors**.\n    - The MLLM’s large capacity and proven ability to call and interpret tools allow it to adaptively fuse expert outputs, achieving synergistic performance (1+1 > 2). In contrast, **small models alone struggle to learn and combine such diverse patterns due to limited capacity and potential conflicts.**\n- **Prior methods are not competitors, but potential expert modules within our framework.** Our contribution is the architecture—the \"how\" and \"who\"—that enables flexible, plug-and-play integration of specialized \"what\" modules. This agent-expert paradigm represents a novel, extensible approach to forgery detection, distinct from traditional ensemble or training-based methods.\n\n---\n\n>**Q7.** Why is the work does not generalize to **other types of deepfakes.**\n\nThank you for raising this important question. Following your suggestion, we have now validated across a much broader range of deepfakes. From our results in **Table 2 of reviewer uoAb**, we can see that **our framework demonstrates strong generalization across diverse forgery types.**\n\n---\n\n>**Q8.** The complexity of the proposed approach is unclear.\n\n**R8:** Thank you for asking about computational complexity. We measure this by calculating the average inference time per image on a single NVIDIA 4090 GPU. The results are shown in **Table 4**.\n\n**Table 4: Comparison of average inference time per image.**\n|Model|Average Inference Time (s/image)|\n:-|:-:\n**Llama-3.2-11B**|3.3\n**Qwen2.5 VL-Instruct 7B**|1.6\n**LLaVa-7B**|1.2\n**Ours (X2DFD w/ two SFDs)**|1.3\n\n---\n\n>**Q9.** Minor Clarifications and Presentation.\n\n**R9:** Thank you very much for the detailed feedback and valuable suggestions! We will revise the manuscript to address all the points raised.\n\n---\n\n**Reference:**\n\n[1] A learning algorithm for Boltzmann machines. Cognitive Science, 1985\n\n[2] Distilling the knowledge in a neural network. arXiv, 2015"}, {"review_id": "WInUmm6wYB", "reviewer": "Reviewer_6sJc", "summary": "This paper introduces X2-DFD, a novel framework for explainable and extendable deepfake detection that leverages Multimodal Large-Language Models (MLLMs). The core idea is to strategically utilize MLLM strengths while compensating for their inherent weaknesses, significantly enhancing both the detectability and explainability of deepfake images. The framework's modular design allows for seamless integration with future, more advanced MLLMs and Specific Feature Detectors (SFDs), ensuring its continuous improvement and scalability in the face of rapidly evolving deepfake challenges.", "questions": "（1）How will the training dataset be expanded to include newer deepfake types (e.g., from diffusion models) to test the framework's real-world applicability? (Refers to Weakness 1)\n\n（2）What are the plans for simultaneously integrating multiple SFDs, and what are the expected benefits or challenges? (Refers to Weakness 2)\n\nIf both questions are addressed well, I will consider increasing the score.", "limitations": "X2-DFD mainly focuses on face-swapping deepfakes; its effectiveness with full image generation (e.g., diffusion models) is less explored. Like the author mentioned implementation focuses solely on static image detection, the framework could expand to other data types like audio or audio-visual forgery. However, for face forgery images discussed in paper, the framework is very comprehensive.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n(1)Seamless SFD Integration: X2-DFD's modular design allows for the seamless integration of Specific Feature Detectors (SFDs). This modularity is crucial in the rapidly developing field of generative AI, ensuring the framework's continuous improvement and adaptability to new deepfake techniques.\n\n(2)Comprehensive Experiments and Clear Explanations: The paper presents comprehensive experiments covering both deepfake detection and explanation.\n\n(3)Novel and potential Core Idea: The methodology behind X2-DFD is highly reasonable and innovative. The approach of leveraging MLLM strengths while compensating for their weaknesses offers significant potential for future framework expansion and maybe a fundamental solution for both deepfake detection and explanation.\n \nWeaknesses\n\n(1)Limited Scope for Forgeries: The current framework primarily demonstrates strong performance in face-swapping. However, the exploration of more advanced techniques like Entire Face Synthesis (EFS) forgeries, particularly those generated by diffusion models, is relatively limited. Expanding the training dataset to include these latest forgery types would be beneficial.\n\n(2)Underexplored Simultaneous Use of Multiple SFDs: While the authors comprehensively discuss SFD integration, the paper does not explore or discuss scenarios involving the simultaneous use of multiple SFDs. Integrating multiple \"expert\" SFDs concurrently could offer a more robust solution for detecting and explaining diverse types of fakes.\n\n(3)Minor Writing Issue: In Section 5, \"Ablation Study and Analysis,\" the phrasing of the question \"How to ensure WFS module\" should be rephrased for clarity.", "was_revised": true, "final_justification": "After reading the rebuttal, I believe the authors have clarified some of my concerns. I think the paper offers some valuable insights for the community.", "rebuttal": "Dear Reviewer 6sJc,\n\nWe sincerely appreciate your constructive feedback and are encouraged by your recognition of our work's strengths, including its **seamless SFD integration, comprehensive experiments, and novel core idea**. We will now address your comments in detail.\n\n---\n\n\n>**Q1** How will the training dataset be expanded to include newer deepfake types (e.g., from diffusion models) to test the framework's real-world applicability?\n\n\n**R1:** Thank you for this critical question regarding the framework's generalization to new deepfake types. To directly investigate this, we conduct an experiment focused on expanding the training data.\n\n- We first establish a baseline model trained **only on the FF++ dataset**. We then expand this training set by **adding 1,000 images from two different modern, diffusion-based datasets** to assess the impact on generalization:\n    - **FF++ & XDXL:** adding entire face synthesis (EFS) images generated by the SDXL diffusion model.\n    - **FF++ & Diffface:** adding images generated by a diffusion-based face-swapping method.\n\n\n- The results in **Table 1** demonstrate that simply adding a small number of new examples to the training data yields only limited improvement in generalization. We expand our latest tests cover the four **most common types of facial forgeries: Face-Swap (FS), Image-to-Image (I2I), Text-to-Image (T2I), and Face-Editing (FE)**. The average performance only increased from 87.3% to 87.7%.\n\n\n**Table 1: Evaluating generalization by extending training datasets, evaluation metric is AUC.**\n| Training Set| CDF | Simswap | Diff-FE | Diff-FS | Diff-I2I | Diff-T2I |**Average** |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |:---:|\n| FF++ (Baseline) | **90.4** | 85.1 |82.1 |**96.0** |81.7 | 88.3| 87.3 |\n| FF++ & XDXL |88.4 |83.8 | **85.3** | 92.4 | **84.7** | 87.5 | 87.0 |\n| FF++ & Diffface | 89.3 | **85.3** |84.3 |95.4 |83.3 | **88.6**| **87.7** |\n\n\n\n---\n\n\n>**Q2.** Underexplored Simultaneous Use of Multiple SFDs: While the authors comprehensively discuss SFD integration, the paper does not explore or discuss scenarios involving the simultaneous use of multiple SFDs. Integrating multiple \"expert\" SFDs concurrently could offer a more robust solution for detecting and explaining diverse types of fakes. \n\n**R2:** Thank you for this excellent and insightful question. You have identified the core principle of extendability in our framework, and we conduct experiments to validate this exact direction **integrating multiple SFDs**. \n\n- To test this, we build directly on our previous experiments, using the FF++ & Diffface dataset as our training set. We then compare two models:\n    - Single SFD: X2DFD using only the blending-based SFD.\n\n    - Multiple SFDs: X2DFD integrating two SFDs simultaneously—the **original blending-based SFD and a new, diffusion-based feature detector.**\n\n    - The results, shown in **Table 2** below, clearly highlight the superiority of the multi-SFD approach. By integrating multiple specialized SFDs, the model's average performance boosts **from 87.7% to 91.5%**. This experiment demonstrates two advantages of framework extensibility：\n        - Our framework can **simultaneously expand multiple SFDs.**\n        - **By extending the corresponding SFD model, supplementing the framework's weakness**, and the overall performance of the framework has been improved.\n\n**Table 2: Performance on integrating multiple SFDs, evaluation metric is AUC.**\n| Model | CDF | Simswap | Diff-FE | Diff-FS | Diff-I2I | Diff-T2I |**Average**|\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |:---:|\n| **Xception** |73.7 |77.6 | 58.6 | 85.9 | 56.8 | 62.4 |69.2\n| single SFD  | 89.3 | 85.3 |84.3 |95.4 |83.3 | 88.6| 87.7 |\n| **ble&Diff (w/ New SFD)** | **90.3** | **88.5** | **92.1** | **97.2**| **88.6** | **92.2** | **91.5** |\n\n---\n\n>**Q3.** What are the plans for simultaneously integrating multiple SFDs, and what are the expected benefits or challenges? \n\n**R3:** Thank you for this excellent question, which touches upon the core vision for our framework's extendability and future evolution. Our response covers our current achievements, the long-term benefits, the primary challenges, and our plans to address them.\n\n- **Current achievement: successful integration of multiple SFDs.** \n    - We are pleased to report that we've already **successfully implemented and validated the simultaneous integration of multiple Specialized Forgery Detectors (SFDs)**. \n    - As shown in our experimental results in **Table 2**, combining detectors designed for different artifact types (e.g., blending-based and diffusion-based forgeries) **leads to a significant performance improvement**. \n    - This confirms the immediate benefit of our multi-SFD approach, proving that a synergistic combination of specialized \"experts\" can indeed elevate overall detection capabilities. This initial success forms the robust foundation for our future advancements.\n\n\n- **Future plan: automated and high-Performance integration**\n    - Our plan is to evolve the current integration process into a **more automated and higher-performance** intelligent system. We aim to develop a sophisticated automated framework that can not only rapidly evaluate and select new SFDs as they emerge but also **dynamically optimize their combinations**. \n    - This means moving beyond a static aggregation of detectors to a system that intelligently understands how different SFDs interact and can configure them for optimal performance against evolving forgery techniques. This intelligent automation will be crucial for maintaining state-of-the-art detection capabilities in a rapidly changing threat landscape.\n\n- **Core benefit: tackling diverse and complex forgeries in real-world Scenarios**\n\n    - A key benefit of integrating multiple SFDs is our enhanced ability to **address the increasingly diverse and complex types of forgeries encountered in real-world scenarios**. Forgery techniques are not monolithic; they vary widely from subtle pixel manipulations to sophisticated AI-generated deepfakes, often combining multiple methods. By leveraging SFDs that specialize in detecting different artifact signatures.\n\n- **Core challenge: feature conflict and coupling in a multi-expert system**\n\n    - The core challenge we anticipate as the number of integrated SFDs increases relates to **navigating potential feature conflict and interdependencies among these \"experts.\"**\n    - **Adapting SFDs for greater specialization.** Another significant challenge might involve modifying individual SFDs to become even more vertical and specialized. \n        - As we integrate more detectors, we may find that some SFDs are not sufficiently granular in their focus. Reworking these existing SFDs to detect even finer, more specific types of forgery artifacts—perhaps by refining their training data or architectural biases—could be a challenge. \n        - It requires a deep understanding of each forgery type's unique fingerprint and the ability to surgically enhance SFDs without compromising their overall performance or introducing new biases.\n\n\n---\n\n>**Q4.** Minor Writing Issue.\n\n**R4:** Thank you for your valuable comments. We will update the corresponding position in the revised manuscript."}, {"review_id": "HiFOKdm82i", "reviewer": "Reviewer_1fa1", "summary": "This paper tackles the problem of deepfake detection with the help of an MLLM by funetuning it with a self-constructed new dataset. The dataset constructing process consists of 2 stages. To construct the dataset, for a given sample, it first evaluates and ranks forgery-related features (as strong and weak features)  based on their detectability. Based on the ranking results, they further suggest directly using the strong feature as classification prompts, while for the weak features, it includes an additional prompts related to its blending factor. After finetuning the MLLM on the newly contructed dataset via LoRA, this method shows leading performance in different datasets.", "questions": "See weakness", "limitations": "The dataset construction process in the proposed framework relies on human verification to ensure the reliability of identified weak features, which can be labor-intensive and time-consuming for large datasets. This dependency on human intervention may introduce scalability challenges and could potentially limit the framework’s performance due to human subjectivity or errors in feature validation.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros:\n \n1. This method focuses on generating reliable explanations by prioritizing features the model can reliably detect, which is relatively new in the field;\n\n2. The manuscript is overall well-written, the pipeline figures clear and easy to understand.\n\nCons:\n\n1. The context don't elaborate on how to group features as strong or weak, is there a selected threshold for the computed balance accuracy? If yes, there should be ablation study regarding this matter.\n\n2. The current weak feature supplementing module relies on the blending factor as additional classification clue, why specifically use this? There are also other low-level clues in previous works that can be used for classification (lightening or shadow as shown in Fig. 2). Would it be better to use others or combine them? More justifications are required in here.\n\n3. How about the results of using MLLMs finetuning only with the original dataset for comparison?\n\n4. It is suggested to put the zero-shot acc. of the MLLM in the manuscript instead of the supplementary material.", "was_revised": true, "final_justification": "Most of my concerns are addressed. After reading comments from other reviewers, I change my rating to positive.", "rebuttal": "Dear Reviewer 1fa1，\n\nWe are sincerely grateful for the reviewer's feedback. We are encouraged by your recognition of the **novelty of our approach to generating reliable explanations** and for their positive feedback on the clarity of the manuscript and the figures.\n\n---\n\n>**Q1. Methodological Question 1 (Threshold for Features):** The context doesn't elaborate on how to group features as strong or weak, is there a selected threshold for the computed balance accuracy? If yes, there should be ablation study regarding this matter.  \n\n\n**R1:** Thanks for your insightful question! In this work, we do not use a fixed accuracy threshold to group features. Instead, our approach is based on **relative ranking**. Specifically,\n\n- We rank all potential features by their balance accuracy performance and designate the **top-K as 'strong'** and the remainder as 'weak'. We believe this is a more robust method, as the concept of a 'strong' feature is relative to the model's capabilities.\n\n- Following your suggestion, we conducted an ablation study on the number of strong features, $K$, to analyze its impact on both generalization (AUC) and explanation quality (evaluated by GPT-4o, see Appendix D.2).\n\n- As shown in **Table 1**, our analysis reveals a clear trade-off:\n    * A **small $K$** (e.g., $K=25$) can be suboptimal because the feature set is not comprehensive enough.\n    * A **large $K$** (e.g., $K=100$), which includes many weak features, is also detrimental. These less distinguishable features act as **noise**, distracting the model's attention and causing it to generate unreliable explanations. This ultimately degrades detection performance.\n\nThe optimal performance is achieved at **$K=50$**, which strikes the most effective balance between feature comprehensiveness and reliability.\n\n**Table 1: MLLM ablation study on the number of strong features ($K$). The evaluation metric is AUC.**\n| **Model Configuration** | **CDF** | **Uniface** | **HPS-Diff** | **Avg** | **GPT4o Eval** |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| X2DFD (Top-K=25) | 83.0 | 84.3 | 87.1 | 84.8 | 2.91 |\n| **X2DFD (Top-K=50)** | **83.2** | **84.5** | **88.7** | **85.5** | **3.02** |\n| X2DFD (Top-K=75) | 81.9 | 83.2 | 84.6 | 83.2 | 2.77 |\n| X2DFD (Top-K=100) | 79.0 | 82.3 | 83.6 | 81.6 | 2.63 |\n\n---\n\n>**Q2.** How about the results of using MLLMs finetuning only with the original dataset for comparison?\n\n**R2:** Thank you for your insightful question, follw your suggestion we add the experiment for comparison:\n\n* **MLLM Alone (Tuning on Prioritizing Features):** Fine-tuning the MLLM using our curated set of high-quality, discriminative features achieves an average AUC of **85.5%**. This approach provides the model with meaningful data that improves its detection capabilities.\n\n* **No explanation (Tuning on Origin):** Fine-tuning the MLLM on only the original dataset's binary **real/fake** labels results in a lower average AUC of **82.3%**.\n\nThis comparison clearly demonstrates that our strategy of strategically selecting and using high-quality features for fine-tuning leads to a substantial improvement in performance over the standard baseline approach.\n\n\n**Table 2: Comparison of Fine-tuning Methods. The evaluation metric is AUC.**\n| **Model Configuration** | **CDF** | **Uniface** | **HPS-Diff** | **Avg** |\n| :--- | :---: | :---: | :---: | :---: |\n| **MLLM Alone** | **83.2** | **84.5** | **88.7** | **85.5** |\n| **No explanation**| 80.3 | 81.9 | 84.7 | 82.3 |\n\n\n\n---\n\n\n>**Q3.** The current weak feature supplementing module relies on the blending factor as additional classification clue, why specifically use this? There are also other low-level clues in previous works that can be used for classification (lightening or shadow as shown in Fig. 2). Would it be better to use others or combine them? More justifications are required in here.\n\n\n\n**R3:** Thank you for this insightful question. Our use of **blending clues** in the manuscript was intended as a **primary and clear-cut example** to illustrate our core thesis: MLLMs are strong generalists that benefit from specialist detectors (SFDs) for fine-grained cues they often miss. Actually, the SFD used in our framework can be implemented by any specific expert. \n\nTo empirically validate the flexibility and power of our framework beyond just blending artifacts, and following the excellent suggestions from you and other reviewers, we have integrated an additional SFD that targets a completely different, yet highly relevant, low-level artifact: **diffusion traces**.\n\nThis new experiment demonstrates two key points:\n- Our framework's performance is not tied to any single artifact; it is **truly extendable**.\n- Combining multiple SFDs targeting different artifacts results in a **synergistic effect**, leading to superior and more comprehensive detection capabilities.\n\n\nAs the results in **Table 4** clearly show, integrating a diffusion-specialized SFD alongside the original blending-based SFD leads to a massive leap in overall performance, boosting the average AUC from 84.8% to **89.9%**.\n\n\n\nIn summary, the **choice of blending was illustrative, not restrictive**. The core contribution is our **pluggable architecture**, which is artifact-agnostic and designed to leverage any number of specialist detectors. We thank you for prompting this clarification and will include these compelling new results in our revised manuscript.\n\n\n\n**Table 3: Performance with multiple SFDs, evaluation metric is AUC.**\n\n| Model | CDF (Blending) | SimSwap (No Blend) | Diff-FE (Diffusion) | Diff-I2I (Diffusion) | **Average** |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| **X2DFD (w/ Blending SFD)** | **90.4** | 85.1 | 82.1 | 81.7 | 84.8 |\n| **X2DFD (w/ Ble & Diff SFDs)** | 90.3 | **88.5** | **92.1** | **88.6** | **89.9** |\n\n\n---\n\n\n\n\n>**Q4.** The dataset construction process in the proposed framework relies on human verification to ensure the reliability of identified weak features, which can be labor-intensive and time-consuming for large datasets. This dependency on human intervention may introduce scalability challenges and could potentially limit the framework’s performance due to human subjectivity or errors in feature validation.\n\n**R4:** Thank you for raising this important question! Indeed, there is a trade-off between reliability and efficiency in human verification during the finetuned dataset construction process. While human annotation helps reduce hallucinations in LLM outputs and improves dataset quality, it can be costly and time-consuming—especially at scale. This challenge has widely been recognized across the LLM community.\n\nTo address this, we plan to train a preference-based MLLM to emulate human judgment, enabling automatic verification without ongoing human intervention. This approach aligns with common practice in scalable dataset curation and offers a practical path toward efficient, high-quality data generation.\n\n\n\n---\n\n\n\n>**Q5. Presentation Issue:** The zero-shot accuracy of the MLLM should be moved from the supplementary material to the main manuscript for better visibility.\n\n**R5:** We appreciate your suggestion and will incorporate it into the revised version of the manuscript."}]

Author rebuttal:

Meta-review: This paper proposes an explainable and extendable framework for deepfake detection leveraging MLLMs. Reviewers found the idea novel, the framework well-designed, and experiments comprehensive, with strengths in modularity and clarity. However, concerns were raised about reliance on specific detectors, incremental novelty, complexity, and generalization to high-quality fakes. The rebuttal addressed most concerns with additional experiments and clarifications, improving confidence in the contributions. Overall, the submission presents a promising and timely direction with strong empirical support, though some limitations remain.
Decision: Accept (poster)