Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection
Authors: Peipeng Yu
Jianwei Fei
Hui Gao
Xuan Feng
Zhihua Xia
Chip Hong Chang
Venue: ICML 2025 poster (2025)
Keywords: Deepfake Detection
Large Vision-Language Models
Prompt Tuning
Primary area: applications->computer_vision
TL;DR: use LVLM for deepfake detection

Abstract: Current Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in understanding multimodal data, but their potential remains underexplored for deepfake detection due to the misalignment of their knowledge and forensics patterns. To this end, we present a novel framework that unlocks LVLMs' potential capabilities for deepfake detection. Our framework includes a Knowledge-guided Forgery Detector (KFD), a Forgery Prompt Learner (FPL), and a Large Language Model (LLM). The KFD is used to calculate correlations between image features and pristine/deepfake image description embeddings, enabling forgery classification and localization. The outputs of the KFD are subsequently processed by the Forgery Prompt Learner to construct fine-grained forgery prompt embeddings. These embeddings, along with visual and question prompt embeddings, are fed into the LLM to generate textual detection responses. Extensive experiments on multiple benchmarks, including FF++, CDF2, DFD, DFDCP, DFDC, and DF40, demonstrate that our scheme surpasses state-of-the-art methods in generalization performance, while also supporting multi-turn dialogue capabilities.

Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection
Peipeng Yu! Jianwei Fei? Hui Gao
Xuan Feng
Zhihua Xia
Chip-Hong Chang?
Abstract
cations in the entertainment and art fields, such as Stable Diffusion (Esser et al., 2024) and DALL-E (Ramesh et al., 2021), their misuse poses critical security risks to society (Wang et al., 2024b). These tools allow users to synthesize realistic but nonexistent content by merely inputting care- fully designed prompts, making deepfake generation more accessible and potentially dangerous than ever before.
Current Large Vision- Language Models (LVLMs) have demonstrated remarkable capabilities in un- derstanding multimodal data, but their potential remains underexplored for deepfake detection due to the misalignment of their knowledge and foren- sics patterns. To this end, we present novel framework that unlocks LVLMs' potential capa- bilities for deepfake detection. Our framework includes showledge-guided Forgery Detector (KFD), a Forgery Prompt Learner (FPL), and a Large Language Model (LLM). The KFD is used to calculate correlations between image features and pristine/deepfake image description embed- dings, enabling forgery classification and local- ization. The outputs of the KFD subsequently processed by the Forgery Prompt Learner to con- struct fine-grained forgery prompt embeddings. These embeddings, along with visual and ques- tion prompt embeddings, are fed into the LLM to generate textual detection responses. Exten- sive experiments on multiple benchmarks, includ- ing FF++, CDF2, DFD, DFDCP, DFDC, and DF40, demonstrate that scheme surpasses state-of-the-ar methods in generalization perfor- mance, while also supporting multi-turn dialogue capabilities. Our code is available at https: / //github. wttp:cttooboooooiooooooooc LIVLeLLLLMDDDDE
Large Vision-Language Models (LVLMs) present promis- ing leverage to this issue. Pre-trained on extensive and diverse datasets, LVLMs capture vast amounts of knowl- edge about natural objects, providing significant potential to improve generalization in identifying manipulated content. An LVLM typically uses an image encoder extract image features, which are then combined with text prompts and input into the Large Language Model (LLM) to generate responses. For instance, inputting image along with a prompt like "This is a facial image designed for deepfake detection, and it should not exhibit any localized color dis- crepancies or evident signs of splicing. Is this a deepfake image?" allows the LVLM assess potential manipula- tions. However, existing LVLMs are primarily optimized for general image understanding tasks and may not effectively capture the fine-grained features required for deepfake de- tection. Directly performing fine-tuning presents challenges, as the LVLM may struggle to interpret specialized terms like "color discrepancies" or "visual artifacts" as intended in the context of forgery detection. Therefore, it is crucial to design fine- fine-grained prompt embeddings to facilitate LVLM training.
In this paper, we aim to unlock the capabilities of Large Vision-Language Models for deepfake detection tasks. Hu- mans naturally use specific descriptors, such as subtle vi- sual artifacts, localized lighting inconsistencies, and overly smoothed textures, to characterize manipulated content. However, these features are difficult accurately replicate by data simulation or feature augmentation alone, limiting the ability of current methods to fully interpret manipulated content (Zhang et al., 2024). To address this limitation, we propose to explore the correction between images and tex- tual descriptions to assist deepfake detection. As shown in Figure we propose novel LVLM-base deepfake detec- tion framework under the guidance of pretrained knowledge
1. Introduction
The rapid advancement of generative artificial intelligence has significantly accelerated the development of deepfake technology, facilitating realistic facial manipulation and reenactment. While these technologies have notable appli-
'College of Cyber Security, Jinan University University of Macau "School of Electrical and Electronic Engineering, Nanyang Technology University, Correspondence Zhihua Xia <xia. 163.com>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Unlocking the Capabilities of Large Vision-Languege Models for Generalizable and Explainable Deepfake Detection
Tear Isaration
Extensive experiments on multiple benchmarks, includ- ing FF++, CDF1, CDF2, DFD, DFDCP, DFDC, and DF40, demonstrate that our scheme outperforms exist- ing methods in generalization performance, with the added capability of supporting multi-turn dialogue.
Detactor
Fecasty Sore
Ia lenoskepphciin
Tou The lenor dhseco
Locatoe
LVEM
lepat Enage
Feraayy taak
lqpal hrage
Desectice
Harus Guitanee
Desction
Herue Guidbason
Mali-am inkcncikk
Klatirum Iraercction
(n) Conventionall Deepöke
Ditection Methools
(bj LVI MI hear Dertatiio Methoos
2. Related Works
Haros Plice
Lights secoaaale
dá skeptikk image?
Deepfake Detection Methods: Conventional classification architectures have achieved significant success in detecting forgery clues in early deepfake content. Various strategies, such as data augmentation (Li et al., 2020a; Shiohara & Yamasaki, 2022; Nguyen et al., 2024), feature consistency analysis (Zhao et al., 2021b; Yan et al., 2023), and frequency domain anomaly detection (Jeong et al., 2022; Liu et al., 2021; Wang et al., 2023a), have been explored in recent years to enhance the generalization of deepfake detection models. While these methods achieve a high detection accu- racy, they primarily rely on data augmentation or enhanced feature extraction (Yan et al., 2024), and often neglect the integration of external human knowledge. However, many deepfake characteristies are embedded in human knowledge, which is challenging to capture through data or feature aug- mentation alone. This limitation significantly constrains the generalization capabilities of existing algorithms. In this pa- per, we propose an WVLM-based deepfake detection frame- work that aligns image features with real/fake descriptions to enhance the model's capacity to detect unseen deepfakes.
Foom
LL
Kanvliida-gaskii Ditection
The
00 Libeaa
Dilectie Reills
Perpary Dataction
Huzus Gasibaee
Maliitan Enenetice
approsch
Figure 1. Comparison of existing deepfake detection methods. Existing approaches can perform localization and classification but ignore integrating external knowledge. Our scheme introduces a Knowledge-guidee detection module to enhance generalization capabilities, and enables multi-turn turn dialogues.
to perform generalizable and explainable detection (Lester et al., 2021; Liu et al., 2022). We first integrate external knowledge train a forgery detector, and then incorporate its features into LLM to generate responses. Specifically, the framework comprises two key stages: Knowledge-guides Forgery Detector Training and L.M Prompt Tuning. In the Knowledge-guide guided Forgery Detector Training phase, we aim to train a high-precision deepfake detector. Leveraging pre-trained multimodal encoders, extract image features from images and textual features from descriptions of pris- tine and forged images. By calculating the correlation be- tween these image features and description text embeddings, we generate consistency maps that represent the alignment between visual content and textual descriptions. These maps are subsequently processed by the Forgery Locator and Clas- sifier to produce a forgery segmentation map and forgery score. Subsequently, in the LLM Prompt Tuning phase, we incorporate deepfake detection knowledge into the LLM to generate forgery detection descriptions. To ensure accurate deepfake detection, we train the LVLM using simulated forgery image-text pairs specifically tailored for this task. Our contributions are summarized as follows:
Large Vision-Language Models: Recent advancements in Large Vision-Language Models (LVLMss) have showcased their potential in multimodal tasks (Gunjal al., 2024; Leng et al., 2024; Gu et al., 2024). A typical LVLM ar- chitecture comprises an image encoder, a projector, and a LLM. The image encoder extracts visual features from input images, which are then transformed by the projector into visual prompt embeddings. These visual embeddings, combined with textual prompt embeddings, are fed into the LLM to generate responses. Building this architecture, models such as BLIP-2 (Li et al., 2023), LLaVA (Liu et al., 2024a), and MiniGPT-4 (Zhu et al., 2024) have achieved notable advancements in language instruction following (Su et al., 2023; Yang et al., 2024) and visual reasoning (Chen et al., 2024) for natural scenes. Some studies have also explored the application of LVLMs in forgery detec- tion. FakeShield (Xu et al., 2024) constructed a large-scale image-text dataset and introduced an LVLM- frame- work specifically designed for forgery detection. FKA-OWl (Liu et al., 2024b) proposed a novel fake news detection framework that leverages forgery-specific knowledge to aug- ment LVLMs, enabling them to reason about manipulations. Similarly, FFAA (Huang et al., 2024) proposed a multi- modal LVLM approach for explainable, open-world face forgery analysis, highlighting the potential of LVLMs in
We propose a novel LVLM-based framework for deep- fake detection that integrates fine- mne-grained forgery prompt embeddings through prompt tuning, which sig- nificantly enhances model generalization and explain- ability.
We introduce a Knowledge-guided Forgery Detector to generate forgery consistency maps to align image fea- tures with description text embeddings of both pristine and deepfake images for enhanced generalization.
2

---

Unlocking the Capabilities of Large Vision- Vision-Language Models for Generalizable and Explainable Deepfake Detection
Knowlolge guided Devection:
Additionally, user queries are encoded as question prompt embeddings Pqquestion- To train the model for forgery detec- tion, we employ two-stage process. In the first stage, we train
Shs feee deignal Se deptia shold cny boilind cake dberppackks spking. bs the deptlic leagh?
LVLM Text Encoder
LVLM Iuagee Enceder
Visual-text lignment
hewwledge-guider Forgery Detector (KFD) to per- form forgery detection and localization by calculating the correlation between image content features and descriptions of pristine/deeffae images. This stage ensures that the KFD can effectively classify and localize forgery artifacts by learning fine- aineggriined visual-text correlations. In the sec- ond stage, we perform LLM Prompt Tuning to integrate the KFI) knowledge into the LVLM framework. Specifically, we design a Forgery Prompt Learner to convert forgery- related features into forgery prompt embeddings. These embeddings, along with the question and visual prompt em- beddings, are then fed into the LLM to generate a textual detection result. To further enhance the interpretability of our model, we adopt an alternating training strategy using both deepfake datasets and general Visual Question Answer- ing (VQA) datasets. This enables the model to accurately detect deepfakes while retaining multi-turn dialogue capa- bilities.
Quary Iaage
Locator
Classifier
Projector
Forgery Prompt Leamer
Large Language Model
Viraal Pronge Eiwholiding
This deepfikke image, aniihees the center face treas.
Furgery Pronpt Emilodllig
Quertion Pronge Emnheiding
Figure 2. Overview of the proposed framework. The frame- work consists of three main components: (I) a Knowledge-guided Forgery Detection module, structured as multi task learning framework comprising three parts: a Visual-text alignment mod- ulc, a forgery classifier, and a forgery locator. (2) a Forgery Prompt Learner, which is designed to fuse forgery detection outputs and generate forgery prompt embeddings. (3) Large Language Model (LLM), which utilizes the extracted question, visual, and forgery prompt embeddings to generate responses.
3.1. Knowledge-guided Forgery Detector
Forgery Visual-text Alignment: To acquire forgery detection-related knowledge, inspired by (Jeong et al., 2023), we align image content features with predefined text description features to obtain fine-grained forgery fea- tures. This process is illustrated in Figure 3. Specifically, this process involves a pretrained image encoder and pre- trained text encoder. Both the image and text encoders are sourced from ImageBind (Girdhar et al., 2023), a large-scale multimodal pre-trained model with extensive cross- mooal knowledge. We first define real and fake image descrip- tions (Dreat and Dakee) and utilize a text encoder to extract their semantic features. These features are concatenated with a learnable embedding to obtain the task-specific text embedding Fiezt E 222x where Ctezt denotes the di- mensionality of the text embedding. For the visual features, we select layers from the image encoder and obtain the intermediate features extracted by each selected layer. The extracted features are then processed by linear layers to generate visual features F E XgH.,w..C...... where i in- dicates the i-th layer. The similarity maps between visual features and textual features are calculated and concate- nated as consistency maps, The formula for computing the consistency maps is as follows:
forgery detection tasks.
Despite these advancements, current LVLMs primarily fo- cus on general language processing and visual understand- ing, often missing the fine-grained details that are essential for deepfake detection tasks. This limitation restricts their effectiveness in forgery localization and classification. To bridge this gap, we develop novel deepfake detection framework based on LVLMs that constructs fine-grained forgery prompt embeddings guide the LLM in detecting subtle manipulations. By integrating rich external knowl- edge in the pretrained LVLM, our scheme could enhance generalization across diverse forgery types while retaining the model's original dialogue capabilities.
3. Proposed Method
Our objective is to enable LVLMs to accurately distinguish between real and fake faces. Although LVLMs are trained on large-scale datasets, they are primarily trained for gen- eral image understanding tasks and often lack the sensitivity for detecting forgery details. To address this limitation, we propose a novel basdd deepfake detection frame- work that enhances sensitivity to deepfake artifacts through constructing fine-grained forgery prompts. As shown in Figure 2, our scheme builds upon conventional LVL.M framework, which comprises an image encoder, a projector, and an LLM The image encoder extracts content features from input images, which are subsequently transformed into visual prompt embeddings Fvisual by the projector.
Ptezt = (Fi FT
(I)
Additionally, to optimize the extracted image features, we compute the cosine similarity between the features of a reference image (pristine image) Férez and the input image features F This similarity optimization enhances the robustness of the features extracted by the image encoder.
3

---

Unlocking the Capabiliites of Large Vision-Lani LiiiunLanguugge Models for Generalizable and Explainable Deepfake Detection
where the Forgery Locator, and gi is the ground truth mask. Dice loss optimizes the overlap between the predicted segmentation and the ground truth. À is the loss weight for balancing these two losses.
LVLM Text Encoder
Dreat
Drabe
Furgury Cheaifier
Fingery Lucker
Additionally, we use binary cross-entropy loss to optimize the performance of the forgery classification task. The for- mula is follows:
LVLM Image Encoder
Letts - (cclog(() + -c) log(1 -)),
(4)
Input Image
Powzt
where è is the predicted forgery score that the image is fake, and the ground truth label for real and 1 for fake).
Befoesce-keodd Opimioatio
3.2. Forgery Prompt Learner and LLM
LVLM Image Encoder
Forgery Prompt Learner: To effectively convert forgery- related features into inputs for the L.M,, we propose a Forgery Prompt Learner to transform the forgery segmen- tation map, the forgery score, and the consistency maps into forgery prompt embeddings. At the same time, we add a learnable prompt embeddings for the forgery prompt learner to incorporate extra information for the deepfake detection task. The Forgery Prompt Learner consists of two convolutional neural networks, one fully connected layer, and the learnable prompt embeddings Enase IR C.s. where Comb represents the dimensionality of the embed- ding vectors. Specifically, the two convolutional networks transform the forgery segmentation map and consistency maps into vector representations, Five E RD2xCan and Econs E IR x respectively. The forgery score is ex- panded into E,ss E R These embeddings are con- catenated and fed into the convolution layer to generate the forgery prompt embeddings Fjjogery E R Finally, the forgery prompt embeddings, visual prompt embeddings, and question prompt embeddings are input into the LLM.
X
Real Reference Image (for traininn only)
Peof
Ferer
Figure 3. Overview of the Knowledge-guided Forgery Detector. It computes correlation between image and text modalities to assist forgery classification and localization. The Reference based Opti- mization process is applied exclusively during the training phase to enhance the robustness of extracted features.
Note that reference images are used only during training and are not employed at inference time. The similarity is calculated as:
Pref [Cos(PPree) ).
(2)
To enhance the model's sensitivity to deepfake content, we introduce Forgery Lo- cator and a Forgery Classifier to locate forgery areas and classify pristine and deepfake images. The Forgery Locator consists of three branches. Each branch performs down- sampling and up-sampling operations on the corresponding consistency maps, followed by interpolation, concatenation, and convolutional layers to generate the segmentation map. The Forgery Classifier also consists of three branches. The three feature maps are first processed by convolution and pooling operations, and then concatenated to form a unified feature representation. After that, we use two fully con- nected layers to calculate the probability of the image being Dice Loss improve the accuracy
Forgery Locator and Classifier:
The LLM processes the prompt embeddings to in- terpret the context and accurately identify forged regions, By integrating visual details (from Eejogerry and Evisuat) with user queries (from Equestion.. the LLM produces responses that provide forgery detection judgments and precisely lo- calize manipulated regions (e.g., eyes, mouth). Here, we employ prompt tuning and LoRA to fine-tune the LLM using simulated image-text pairs specifically tailored for deepfake detection tasks. To ensure the LLM generates ac- curate responses, we use cross-entropy loss to quantify the discrepancy between the predicted response and the target label. The formula is defined follows:
LLM:
real fake. Here, of forgery segmentation. We further enhance the robustness of extracted forgery features by optimizing the matching degree between the textual consistency map (Prezt) and the reference consistency map (Pref). Both maps are expected to accurately localize the forged regions. The localization loss is formulated follows:
Litm = - Σ ys log(()))
(5)
where y/ represents the predicted probability for the j-th token, and y, denotes the corresponding ground truth label.
Lioe Dice((((rreat) g() (DDec((((rer)) (3)

---

Unlocking the Capabilities of Large Vision- Vision-Language Models for Generalizable and Explainable Deepfake Detection
3.3. Data for LLM Prompt Tuning
provides textual description of deepfake detection task.
Forgery Data Simulation: We intend for the LLM to iden- tify pristine and deepfake images, while also locating the forged regions. This requires training image-text pairs specifically depicting manipulated areas, which are currently unavailable. To address this gap, we draw on techniques from SBI (Shiohara& & Yamasaki, 2022) to construct image- text pairs using existing real images. First, we generate facial landmarks from a real image reat, then randomly se- lect 1 regions (e.g., the nose, mouth, eyes) target forgery areas. We apply slight affine transformation to the real image, resulting in an affine-transfome transformed image Tofinee The original real image is used as the background (target face), and the tranfored image serves as the fore- ground (source face). Following the approach in (Nguyen et al., 2024), we apply Poisson blending to combine the foreground and background images. The blending process is defined as follows:
4. Experiments
4.1. Experimental Settings
The FaceForensics+* (FF++) (Rossler et al., 2019) dataset includes 1,000 real videos and 5,000 forgery videos across five deepfake categories, which is one of the most widely-used datasets for deepfake detection. DFD (Dufour & Gully, 2019), CDF1, CDF2 (Li et al., 2020b), DFDCP (Dolhansky, 2019), DFDC (Dolhansky et al., 2020), and DF40 (Yan et al.) are commonly used datasets for evalu- ating generalization performance in deepfake detection. The images are all cropped using Dlib and RetinaFace. We train only on real data from the FF++ dataset.
Datasets.
Evaluation Metries. Following existing approaches (Sh- iohara & Yamasaki, 2022; Nguyen et al., 2024), we the video-level Area Under the Receiver Operating Char- acteristic Curve (AUC) and Average Precision (AP) as our evaluation metric. Additionally, we assess the LLM's per- formance by evaluating the binary classification (Yes or No) of authenticity in its textual output, allowing us to calculate a corresponding video-level AUC for the L.M'' responses.
IM = M O fiff + (1 - M) @ Ireat:
(6)
where M is the Convex Hull mask constructed based on the selected forgery region, with values ranging from 0 to 1. The symbol @ denotes element-wiss wise multiplication.
Question and Answer Content: Training an LVLM re- quires a large number of visual question-answer pairs. Therefore, we construct corresponding text queries for each image. To ensure compatibility with the deepfake detec- tion task, we first include a background description in each query, for example: "This is a facial image designed for deepfake detection, and it should not exhibit any localized color discrepancies or evident signs of splicing.", ", which can be regarded kind of human prior knowledge. Ad- ditionally, we incorporate the prediction results from the Knowledge-guided Forgery Detector (KFD) into the prompt, such as "According to KFD prediction, the forgery score is 0.93." We then ask a question related to the content of the image, such as "Is this a deepfake image?" The LVLM's response states whether a forgery is present in the image and where the forgery area is. For instance. "Yes. This a deepfake image, and the artifact is at the center face of the image." Here, the forgery regions are defined according to those selected during forgery data simulation. By defining both queries and responses, we can train the LVLM to dis- tinguish between pristine and deepfake images. The prompt format input to the LVLM follows this format:
Compared Methods. We evaluate our approach against sev- eral state-of ofrtth--rt deepfake detection algorithms (Rossler et al., 2019; Li et al., 2020a; Qian et al., 2020; Zhao et al., 2021a; Liu et al., 2021; Zhao et al., 2021b; Cao et al., 2022; Shiohara & Yamasaki, 2022: Wang et al., 2023a;b; Dong et al., 2023; Yan et al., 2023; Xu et al., 2023; Yan et al., 2024; Nguyen et al., 2024; Cheng et al., 2024; Lin et al., 2025; Luo et al., 2024; Ba al., 2024; Fu et al., 2025) and WVLM-bseed Methods (Khan & Dang-Nguyen, 2024; Su et al., 2023; Liu et al., 2024b; Wang et al., 2024a)
Implementation Details. Our approach leverages the PandaGPT architecture, which incorporates the ImageBind- Huge model as the image and text encoder. We extract features from the 16th, 24th, and 32nd layers of the encoder to compute consistency maps with the text features, which are then passed to the Vicuna-7B model for inference. For multi-turn dialogue capability, we alternate training between the deepfake dataset and the PandaGPT dataset. The forgery region number n is set as 3. All the images are cropped to 224 224. Training is conducted on two Nvidia RTX 4090 GPUs over 50 epochs, using the Adam optimizer with learning rate of le-4 and weight decay of le-5. The loss parameter À is set as 1.
# #uman: < <Img> <Ins> < Img> Img>>Eoogerry Eorgery [Task description] [KFD result] Is this deepfake image? #*
4.2. Comparison with SOTA Detection Methods
where Evisual represents visual prompt embeddings, Eforgery denotes the forgery prompt embeddings learned by the Forgery Prompt Learner, KFD result indicates the forgery classification results, and Task description
We first compare our approach with several -tate-of-the-art deepfake detection methods (Li et al., 2020a; Shiohara & Yamasaki, 2022; Cao et al., 2022; Huang et al., 2023; Yan
5

---

Unlocking the Capabilities of Large Vision-Lann Language Models for Generalizable and Explainable Deepfake Detection
Intra-datasct
Cross-datasct
Methods
FF++
CDF2
DFD
DFDC
CDFI
DFDCP
AUC
AP
AUC
AP
AUC
AP
AUC
AP
AUC
AP
AUC
AP
Xception (Rossler et al., 2019)
CVPR'1S 19
97.23
96.44
81.65
88.74
89.75
80.98
90.00
69.90
81.95
FBceXRay+BII (Li et al., 2020a)
CVPR'20
79.50
95.40
93.34
80.58
73.33
80,92
72.65
F3Net (Qian al., 2020)
ECCV'20
98.20
96.17
78.88
86.23
93.33
97.70
71.77
71.99
81.11
89.61
73.50
79.30
Multiattentioaal (Zhao et al., 2021a)
CVPR'21
68.26
92.95
63.02
SPSL (Liu et ul., 2021)
CVPR'21 21
96.91
89.37
79.86
87.83
96.12
98.20
66.16
71.13
85.02
92.17
75.86
82.64
PCL+12G (Zhao al., 2021b)
CVPR'21
99.11
90.03
99.07
74.27
RECCE (Cao et al,, 2022)
CVPR'22
99.32
97.25
82.31
88.26
98.26
98.40
69.58
69.98
81.49
89.04
71.49
77.04
SBI (Shiohara & Yamasaki, 2022)
CVPR'22 22
99.15
99.15
93.82
92.99
96.32
96.13
74.47
74.09
93.44
93.77
90.95
85.98
SFDG (Wang al., 2023a)
CVPR'23
75.83
88.00
73.63
AliFreezing (Wang et al., 2023b)
CVPR'23
93.81
98.74
89,50
87.41
98.50
94.59
64,75
67.52
88.48
92.79
64.05
76.22
CADDM (Dong et al., 2023)
CVPR'23 23
99.70
93.88
99.03
73.85
85.68
74.19
UCF (Yan et al., 2023)
CVPR'23
98.69
97.99
83.73
90.10
94.50
98.04 08.04
75.11
74.76
86.08
91.78
80.50
77.16
TALL (Xu et al., 2023)
ICCV'23
99.87
90.79
76.78
LSDA (Yan et al., 2024)
CVPR'24 24
91.10
77.00
LAANet (Nguyen
2024)
CVPR'24
99.96
95.40
97.64
99.51
99.40
86.94
97.70
CFM (Tuo et al., 2024)
TIRS'24
89.65
80.22
ED (Ba et al., 2024)
AAAT'24
93.6
75.4
90.2
ProDet (Cheng al., 2024)
NIPS'24
92.62
96.05
71.52
72.8
94.48
96.6
82.83
88.89
RepDFD (Lin et al., 2025)
AAAI'25
89.94
80.99
95.03
UDD (Fu et al., 2025)
AAAI'25
93.13
81.21
88.11
Ours
99.53 99.54
94.71
93.59
99.64
99.68
79.12
77.69
97.62
97.67
91.81
88.26
Table 1. Intra-dataset and cross-dataset evaluation of KFD compared with existing deepfake detection methods. The highest AUC AP scores among all methods are highlighted bold, while the second- sooud-h scores are underlined. All experimental results are sourced from the original papers or reproduced using publicly available code repositories.
al., 2024; Tan et al., 2024).
DF40-PS
UniFace
InS
FSGAN FaceDancer
c4s
SBI
89.02
88.52
89.62
78.18
86.36
Intra-Datased Evaluation. Following the catra-datastt pro- tocol outlined in previous works (Yan et al., 2024; Nguyen et al., 2024), we compare our approach with existing state- of of-the-art deepfake detection methods based on the outputs of KFD. As shown in Table our method achieves com- petitive results, reaching a detection AUC of 99.53% on the FF++ dataset.
CADDM
86.86
78.65
88.86
76.54
87.92
Ours
90.61
87.64
93.75
82.97
94,68
DF40-FR
PIRender
HyparReenact
FOMM
FS, vS_vid2vid
LIA
SBI
81.81
65.31
88.05
83.72
89.22
CADDM
77.37
69.26
84.77
72.86
69.67
88.29
81.55
93.34
71.56
StyleGAN2 StyleGAN-XL
DDIM
DiT-XL/2
Pix,rt-oo
DF40-
SBI
97.91
23.26
99,56
79.04
98.78
DotsseDttsett Evaluation. Following previous works (Li et al., 2020a; Shiohara & Yamasaki, 2022; Nguyen et al.,
CADDM
100.00
98.69
98.10
79.90
99.74
Ours
100.00
100.00
99.93
94.18
100.00
2024), we further perform a cross-dataset evaluation. The models are trained on real data from the FF++ dataset, and the detection performance evaluated on CDF1, CDF2, DFD, DFDCP, and DFDC datasets. We report video-level AUC and AP scores across these datasets in Table 1. As summarized in Table 1, our method achieves superior video- level AUC and AP scores across these diverse datasets, out- performing offte--f-theaan baselines.
Table 2. Cross Crass-manipulatioon evaluation on DF40 dataset. All methods are evaluated on various types of manipulated subsets of DF40. The best-performing values highlighted in bold.
ing a diverse range of manipulation methods. Our mod- els are trained exclusively on real images from FF++ and subsequently tested on unseen manipulation types within DF40. As presented in Table 2, approach consis- tently outperforms existing methods across multiple forgery types, achieving state-of-the-ane detection results. Notably, our method demonstrates robust generalization capabilities when confronted with advanced deepfake generation tech- niques, including FSGAN, E4S, LIA, StyleGAN, DDIM,
Crass-Maniuulation Evaluation of KFD. To evaluate the robustness of our algorithm against unknown forgery tech- niques, we assess its detection performance data gen- erated by unseen manipulation methods within the DF40 dataset. DF40 comprises synthetic deepfake samples gen- erated from real images in the FF++ dataset, encompass-
6

---

Unlocking the Capabilities of Large Models for Generalizable and Explainable Deepfake Detection
CDF2
DFD
DFDC
Image
SBI- -BI-EAIBB
Ours
Figure 4. GradCAM visualizations on CDF2, DFD, and DFDC datasets.
Methods
FF++
CDF2
DFD
CDF1
DFDCP
Avg
et al., 2024a; Liu et al., 2024b). In these evaluations, both the image and the corresponding query are provided as in- puts the LVLM, and the model is required determine the authenticity of the image (real fake). For classifica- tion task, our Knowledge-guided Forgery Detector (KFD) demonstrates significantly superior detection performance compared to existing WVLM-based classification methods. For VQA models such as PandaGPT, Qwen2-VI. and FAK- Owl, we utilize the LLM's output ("Yes" or "No") to classify authenticity and subsequently compute the AUC values for evaluation. As summarized Table 3, our method consis- tently outperforms existing LVLM-based VQA methods, achieving better performance the FF++, CDF2, DFD, CDFI, and DFDCP datasets.
Clipping
91.32
79.97
88.74
81.45
69.26
82.15
RepDFD
89.94
95.03
92.49
UDD
93.13
88.11
90.62
Ours-KFD
99.53
94.71
99.64
97.62
91.81
96.66
PandaGPT
63.42
62.53
64.56
55,01
46.06
58.32
Qwen2-VL.
50.50
48.88
42.38
46.46
51.71
47.99
FAK-Owl 67.54
60.85
67.99
69.84
70.72
67.39
Ours-LVLM
97.11
89.90
92.26
95.97
86.70
92.39
Table Comparison of scheme with existing LVLM- methods. Clipping (Khan & Dang-Nguyen, 2024) and PandaGPT (Su al., 2023) are fine-tuned on the FF++ dataset. FAK-Owl (Liu et al., 2024b) is trained based on original image-text pairs, and Qwen2-VI. (Wang et al., 2024a) evaluated with its original pre-trained weights. The AUC scores for Ours-LVLM are derived from textual output
Dialogues Visualization.
Unlike prior detection methods, our approach not only supports deepfake detection but also facilitates multi-turn dialogues, enabling users to further in- quire about the image content. Figure presents some example dialogues under intra-dataset and cross-dataset evaluations. It shows that our proposed scheme accurately identifies forged regions within the images, while the LLM provides precise and contextually relevant judgments. Ad- ditional multi-turn dialogue examples are provided in the supplementary material.
L.M,, and the AUC scores for Ours- KFD derived from the KFD output. The best and second-best results are highlighted in bold and underlined, respectively.
and PixArt-a.
GradCAM Visualization. To further interpret the decision process of our model, we use GradCAM (Selvaraju et al., 2017) to visualize attention when encountering unknown datasets. As shown Figure 4, we compare our method with SBI (Shiohara & Yamasaki, 2022) across the CDF2, DFD, and DFDC datasets. While SBI can detect forged regions, it may misidentify these areas and tends to highlight irreevant regions when encountering unknown forgeries. In contrast, our scheme, guided by external knowledge, effectively identifies forgery regions.
4.4. Analysis
Number of Training Images. Obtaining large-scale face images in real-world scenarios often infeasible. Hence, we assess our model's performance with varying numbers of training images. Specifically, 100, 200, and 500 real images from FF++ to create corre- sponding fake image-text pairs for training. The training steps are fixed as 500. The video-level AUC is calculated based on the responses of LLM. As shown in Table 4, our approach achieves state-of-the-art performance using only 500 training images. Although reducing the training set size
randomly sample 50,
4.3. Comparison with LVLM-based Methods
Detection Performance of LVLM. We benchmark our framework against state-of-the-aan LVLM-basee classifica- tion methods (Khan & Dang-Nguyen, 2024; Lin et al., 2025; Fu et al., 2025) and VQA methods (Su al., 2023; Wang
7

---

Unlocking the Capabilities of Large Vision-Languen Models for Generalizable and Explainable Deepfake Detection
DeepPake
Face2Face
FaceShifter
CDF2
DFDCP
DFDC
Real
Yeu This decpfikke image, the antfact the center fice of ingee
Yes. This d doasliike intage antinct at the leht eye. lell jaw, right jaw image.
Yes. This deepdiic imegt. the overospredd.
No. This is priitize imgee
No, Bii sot deeplike image. The image real photograph of mon with beant
No. this is decplace image: The image appeass be aecuise photogrmph.
FaceSwap
NeuralTextture
Pristine
Fake
Yes This is desplik imagn. dee annas on antiact. The antifit vishie redemooryy. fice. right jos image
This doepdice mage. the aeridic center the image.
No. This is not a deeplike image.
is dasptikk imags.
center
Yes. Thii deepbioo image the carter
Yes deepdiiz imege antiizi Bex image.
image.
image.
Intra-datasee Evaluations
(b) Cross-datass Evaluations
Figure 5. Forgery Localization and LLM Responses under Intra-dataset and Cross-dataset Evaluations. Each example includes the original image (top-left), the corresponding forgery segmentation map (top-right), and the textual detection result generated by the L.L (bottom).
Test set AUC
FF++
CDF2
DFDC
Number
FPL. LLM Lora
CDF1
CDF2
DFD
DFDC
DFDCP
AUC
AP
AUC
AP
AUC
AP
77.78
82.64
82.25
70.49
84.70
50.00
50.00
50.28
65.56
50.00
49.93
100
82.59
81.74
86.34
72.72
85.06
63.42
61.22
62.53
59.02
55.11
53.47
200
84.53
80.68
88.14
74.12
83.48
96.59
96.13
89.25
94.69
67.65
64.07
500
84.17
83.31
88.17
74.23
84.65
97.11
96.97
89.90
94.51
68.77
64.57
Table 4. Generalization evaluation across different numbers of training images.
Table 5. Ablation study results on FF++, CDF2, and DFDC. The v in the FPL column indicates the inclusion of forgery prompt embeddings in the framework. The / in the LLM column signifies the use of an LLM inference, while the / in the LoRA column denotes whether LoRA is used to fine-tune the LLM.
leads slight degradation in accuracy, our method still maintains competitive results with as few 100 training images. This underscores the robustness of our framework, particularly in scenarios where data are scarce.
Training Set
Test Set AUC (%)
Dataset
#Real
FF++
CDF2
DFDCP
DFDC
Avg
FF++ CDF2
720
99.54
94.32
92.39
77.03
90.82
Effect of Prompt Tuning. The prompt tuning process is designed to convert forgery detection knowledge into the input of LLM to facilitate accurate detection. This pro- cess involves the Forgery Prompt Learner (FPL), the LLM, and the LoRA strategy. To evaluate the effectiveness of this component, conduct ablation experiments FF++, experinne
622
97.95
95.79
86.36
70.27
87.59
DFDCP
737
99.15
85.17
93.74
69.64
86.93
Table 6. Gemeralization evaluation across various training datasets.
CDF2, and DFDC datasets. For each configuration, the AUC is calculated based on the LLM's output in determin- ing real versus fake images. As shown in Table 5, models equipped with the Forgery Prompt Learner demonstrate higher AUC values, indicating improved effectiveness for deepfake detection tasks. Furthermore, the integration of LoRA further enhances performance, achieving superior results across multiple datasets compared to configurations without LoRA.
our approach across different datasets, we train the model on various training sets and conducted cross-dataset eval- uation on FF++, CDF2, DFDCP, and DFDC datasets. We calculate detection AUC values based on the LLM's re- sponses, as shown in Table 6. The results indicate that our method exhibits strong robustness across diverse datasets and demonstrates its ability to generalize across different types of forgeries using varied real data.
Generalizability to Training Datasets. The generalizabil- ity of deepfake detection is closely related to the diversity of the training data used. To verify the effectiveness of
Effects of different LLM architectures. The deepfake detection performance is also related to the specific LLM architecture used, different LLMS exhibit unique charac-
8

---

Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection
Test set AUC (%)
enabling accurate forgery detection responses. Extensive evaluations across multiple benchmarks, including FF++, CDFI, CDF2, DFD, DFDCP, and DFDC, demonstrate that our scheme outperforms existing methods in generaliza- tion performance. Notably, our framework also supports multi-turn dialogue, providing interactive and explainable detection results. These findings underscore the potential of LVLM- approaches in advancing deepfake detection methodologies.
LLM Architecture
FF++
CDF1
DFD
DFDC
Avg
Llama-3.2-1B
96.49
95.64
82.74
65.66
83.13
L.ama-3.2-3B
97.08
95.30
82.80
66.
85.32
Vicuna-7B
97.11
95.97
84.26
67.65
86.25
Table Generalization evaluation across different LLM archi- tectures. The AUC is computed based on the LLM output.
Test set AUC (%)
Methods
A Acknowledgements
CDF1
CDF2
DFDC
DFDCP
Avg
This work is supported in part by the National Key R&D Program of China under Grant number 2022YFB3103100. in part by the National Natural Science Foundation of China under grant numbers, U23B2023 and 62472199, Guang- dong Key Laboratory of Data Security and Privacy Pre- serving under Grant 2023B 12 212060036, in part by Engi- neering Research Center of Trustworthy AI, Ministry of Education, in part by the Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier 2 Award No. MOET2EP50220-0003.
SBI
93.44
93.82
74.47
90.95
88.17
w/o ROP
96.18
92.77
79.31
91.73
90.00
w/ ROP
97.62
94.71
79.12
91.81
90.82
Table Ablation study on the Reference-based Optimization Process. The highest AUC scores among all methods are high- lighted bold.
teristics. To examine detection performance across various LLM architectures, we evaluate three models: Llama-3.2 1B, Llama-3.2-3B, and Vicuna-7B. The evaluation is con- ducted on the FF++, CDF1, DFD, and DFDC datasets. As shown in Table 7, we observe that larger-scale LLM archi- tectures consistently yield superior detection performance. This trend suggests that models with increased parameter capacity are more adept at capturing fine-grained forgery artifacts.
Impact Statement
This research aims to improve generalizability and explain- ability for deepfake detection. However, the reliance on large-scale datasets for training raises potential privacy con- cerns, and biases in the training data could lead to unequal performance across different demographic groups. Never- theless, developing algorithms capable of detecting deep- fake data is essential, and we advocate for robust security measures and legal frameworks to govern the use of such technology.
Ablation Study on the Reference-based Optimization. To validate the effectiveness of the reference based optimiza- tion process, we train the model under two configurations: with and without Reference- based Optimization Process (ROP). We then evaluate the generalization capability of our approach across various datasets. Table 8 summarizes the generalization performance on the CDFI, CDF2, DFDC, and DFDCP datasets. The results demonstrate that even without similarity optimization, the proposed framework outperforms existing methods. Furthermore, introducing the similarity optimization process yields additional perfor- mance gains, underscoring its effectiveness in enhancing the generalization capability of deepfake detection models.
References
Ba, Z., Liu, Q., Liu, Z., Wu, S., Lin, F., Lu, L., and Ren, K. Exposing the deception: Uncovering more forgery clues for deepfake detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 719- 728, 2024.
Cao, J., Ma, C., Yao. T., Chen, S., Ding, S., and Yang, X. End-to-end reconstruction-clasiffiee face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4113-4122, 2022.
learning for
5. Conclusion
In this work, we introduce a novel deepfake detection frame- work that leverages LVLMs enhance generalization and explainability. By integrating Knowledge-guidd guided Forgery Detector, we effectively align image features with textual descriptions of pristine and deepfake images to facilitate forgery classification and localization. Furthermore, we incorporate a Forgery Prompt Learner capable of transform- ing fine-grained forgery features into inputs for the L.M,,
Chen, L., Li, B., Shen, S., Yang, J., Li, C., Keutzer, K., Darrell, T., and Liu, Z. Large language models are visual reasoning coordinators. Advances in Neural Information Processing Systems, 36. 2024.
Cheng, J., Yan, Z., Zhang, Y., Luo, Y., Wang, Z.,, and Li,

---

Unlocking the Capabilities of Large Vision- Vision-Language Models for Generalizable and Explainable Deepfake Detection
Can we leave deepfake data behind in training deepfake detector? volume 37, pp. 11979-21998, 2024.
Jeong, J., Zou, Y., Kim, T., Zhang, D., Ravichandran, A., and Dabeer, O. Winclip: Zero-/few-shot anomaly classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19606-19616, June 2023.
Dolhansky, B. The dee pfake detection challenge (dfdc) pre view dataset. arXiv preprint arXiv:1910.0860 191000884, 2019.
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., and Ferrer, C. C. The deepfake detection chal- lenge (dfdc) dataset, arXiv preprint arXiv:2006.07397, 2020.
Jeong, Y., Kim, D., Ro, Y., and Choi, J. Frepgan: Robust deepfake detection using frequency-level perturbations. In Proceedings of the AAAI conference on artificial intel- ligence, volume 36, pp. 1060-1068, 2022.
Dong, S., Wang, J., Ji, R., Liang. J., Fan, H., and Ge, Z. Im- plicit identity leakage: The stumbling block to improving deepfake detection generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1994-4004, 2023.
Khan, S. A. and Dang-Nguyen, D.-T. Clipping the de- ception: Adapting vision-language models for universal deepfake detection. In Proceedings of the 2024 Inter- national Conference on Multimedia Retrieval, pp. 1006- 1015, 2024.
Leng, S., Zhang, H., Chen, G., Li. X., Lu, S., Miao, C., and Bing. L.. Mitigating object hallucinations in large vision- language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872-13882, 2024.
Dufour, N. and Gully, A. Deepfakes detection dataset., 2019.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high- righuresolution image synthesis. Forty-first International Conference on Machine Learning, 2024.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efffiiinn prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, 2021.
Fu, X., Yan, Z., Yao, T., Chen, S., and Li, X. Exploring unbiased deepfake detection via token-level shuffling and mixing. 39(3):3040-300 3048, 2025.
Li, J., Li, D., Savarese, S., and Hoi, Blip-2: Bootstrapping language-iman image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19930-197423 PMLR, 2023.
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Imagebind: One embedding space to bind them all. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180- 15190, 2023.
Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F,, and Guo, B. Face x-ray for more general face forgery detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5001500-1550000000000000000000 5001-5010, 2020a.
Gu, Z., Zhu, B., Zhu, G., Chen, Y., Tang, M., and Wang, J. Anomalygpt: Detecting industrial anomalies using large vision-language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38,
Li, Y., Yang, X., Sun, P., Qi, H., and Lyu, S. Celeb-df: large-scale challenging dataset for deepfake forensies. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3207-3216, 2020b.
pp. 1940- 2024.
Gunjal, A., Yin, J., and Bas, E. Detecting and prevent- ing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelii gence, volume 38, pp. 18135-18143, 2024.
Lin, K., Lin, Y., Li, W., Yao, T., and Li, B. Standing on the shoulders of giants: Reprogramming visual-language model for general deepfake detection. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 39, pp. 5262-5270, 2025.
Huang, B., Wang, Z., Yang, J., Ai. Zou, Q., Wang. Q., and Ye, D. Implicit identity driven deepfake face swapping detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1490-4499, 2023.
Liu, H., Li, X., Zhou, W., Chen, Y., He, Y., Xue, H., Zhang, W., and Yu, N. Spatial-phase shallow learning: rethinking face forgery detection in frequency domain. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 772-781, 2021.
Huang, Z., Xia, B., Lin, Z., Mou, Z., and Yang, W. Ffaa: Multimodal large language model based explainable open-world face forgery analysis assistant. arXiv preprint arXiv:2408. /0072, 2024.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024a.
10

---

Unlocking the Capabilities of Large Vision- Vision-Language Models for Generalizable and Explainable Deepfake Detection
Liu, X., Ji, K., Fu, Y., Tam, W., Du, Z.,, Yang, Z., and Tang, J. P-tuning: Prompt tuning can be comparable fine- tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, 2022.
Tan, C., Zhao, Y., Wei, S., Gu, G., Liu, P., and Wei, Y. Rethinking the up-sampling operations in enn-based gen- erative network for generalizable deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 281338333 30--21139, 2024.
Liu, X., Li, P., Huang, H., Li. Z., Cui, X., Liang, J.. Qin, L., Deng, W., and He, Z. Fka-owl: Advancing multimodal fake news detection through knowledge -uggmgeee Ivlms. In Proceedings of the 32nd ACM International Confer- ence on Multimedia, pp. 10154-10163, 2024b.
Wang. P., Bai, S., Tan, S., Wang. S., Fan, Z.., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vI: Enhancing Aision-languag model's perception of the world at any resolution. arXiv preprint
2024a.
Wang, T., Liao, X., Chow, K. P., Lin, X., and Wang, Y. Deepfake detection: A comprehensive survey from the reliability perspective. ACM Computing Surveys, 2024b.
Luo, A., Kong, C., Huang, J., Hu, Y., Kang, X., and Kot, A. C. Beyond the prior forgery knowledge: Mining criti- cal clues for general face forgery detection. IEEE Trans- actions on Information Forensics and Security, 19:1 168- 1182, 2024. doi:
Wang, Y., Yu, K., Chen, C., Hu, X., and Peng, S. Dynamic graph learning with content-guided spatial-frequnncy re- lation reasoning for deepfake detection. In Proceedings of the IEEE/CVV Conference on Computer Vision and Pattern Recognition, pp. 1278-7287 2023a.
Nguyen, D., Mejri, N., Singh, I. P., Kuleshova, P., Astrid, M., Kacem, A., Ghorbel, E., and Aouada, D. Laa-net: Localized artifact attention network for quality and generalizable deeplake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 17395-17405, 2024.
Wang. Z.. Bao, J., Zhou, W., Wang. W., and Li, H. Alt- freezing for more general video face forgery detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 41294411444 4138,, 2023b.
Qian, Y. Yin, G., Sheng, L., Chen, Z., and Shao, Thinking in frequency: Face forgery detection by mining frequency- aware clues. In European conference on computer vision, pp. 103. Springer. 2020.
Xu, Y., Liang, J., Jia, G., Yang. Z., Zhang. Y., and He, R. Tall: Thumbnail layout for deepfake video detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 22658-226666 2023.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831. Pmlr, 2021.
Xu, Z., Zhang, X., Li, R., Tang, Z., Huang, Q. and Zhang. J. Fakeshield: Explainable image forgery detection and lo- calization via multi-modal large language models. arXiv preprint arXiv:2410.0276. 2024.
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., and Nießner, M. Faceforensics*#:: Learning to de- tect manipulated facial images. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1-11, 2019.
Yan, Z., Yao, T., Chen, S., Zhao, Y., Fu, X., Zhu, J., Luo, D., Wang, C., Ding, S., Wu, Y., et al. Df40: Toward next- next-generation deepfake detection. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explana- localization.
Yan, Z., Zhang, Y., Fan, Y., and Wu, B. Ucf: Uncovering common features for generalizable deepfake detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22412-22423, 2023.
tions from deep networks via gradient- In Proceedings of the IEEE international conference on computer vision, pp. 618 6626, 2017.
Shiohara, K. and Yamasaki, T. Detecting deepfakes with self-blended images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 185720-18722 2022.
Yan, Z.,, Luo, Y., Lyu, S., Liu, Q., and Wu, B. Transcending forgery specificity with latent space augmentation for generalizable deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8984-8994, 2024.
Su. Y., Lan, T., Li, H., Xu, J., Wang, Y., and Cai, D. Pandagpt: One model to instructiio-fflllow them all. In Proceedings of the 1st Workshop on Taming Large Lan- guage Models: Controllability in the era of Interactive Assistants!, pp. 11-23, 2023.
Yang, R., Song, L., Li, Y., Zhao S., Ge, Y., Li, X., and Shan, Y. Gpt4tools: Teaching large language model to use tools via self iestrinnstttttion. Advances in Neural Information Processing Systems, 36, 2024.

---

Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection
Zhang. Y., Yu, Z.,, Wang, T., Huang, X., Shen, L.., Gao, Z and Ren, J. Genface: A large-scale fine-grained face forgery benchmark and cross appearance-edge learning. IEEE Transactions on Information Forensics and Security, 2024.
Zhao, H., Zhou, W., Chen, D., Wei, T., Zhang, W., and Yu, N. Multi-attentional deepfake detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2185-2194, 2021a.
Zhao, T, Xu, X., Xu, M., Ding, H., Xiong, Y., and Xia, W. Learning self-consistency for deepfake detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 15023-15033, 2021b.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4: Enhancing lision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2024.
12

---

Unlocking the Capabilities of Large Vision-Langue Models for Generalizable and Explainable Deepfake Detection
A. Appendix
A.I. Implementation Details
Knowledge-guided Forgery Detector (KFD); To incorporate external knowledge for precise deepfake detection, we leverage the image and text encoders of the ImageBind-huge model to extract image content features and textual description features. For the visual features, we select layers 16, 24, and 32 from the image encoder extract corresponding features. Since the dimensionality of the text features is fixed at 4096, while the dimensionality of the image features varies across layers, we employ linear layers to project the image content features to match the dimensionality of the text features. By calculating the correlation between the text features and each set of image content features, we generate three consistency maps. These maps are subsequently passed to the Forgery Locator and Classifier for forgery localization and classification. During training, the batch size is set to 16.
Real and fake descriptions: Textual descriptions are generated via GPT-4, validated by human annotators. Examples include "Inconsistent head poses" or "Mismatched skin texture". Detailed examples are provided follows:
deeefaae_descrittione
I'unnatural blending edges", "inconsisten" head poses", noticeable blending artifacts", "mismatched skin texture", "unnatural reflections", unnatural looking hairlines", "misaligned facial features", "digital artifacts visible "strange light flares around the lips", "jawline appears overly smooth", "distorted shadows", sover-sharpened exaggerated facial features", "distorted edges around the face", "fake")
real www.ffeeddscripttiiooc
["natural light and shadow transitions", "natural eye reflections "hatural appearance", "natural skin tores and color variations", "realisth reflections in their eyes", "naturally textured skin", "fine hair near the eyebrows and forehead", "realistic texture", "natural sparkle in their eyes", "natural highlights on the bridge of the nose", "natural skin texture", "natural skin folds around the eyes", "natural light sources", "smooth yet textured skin the forehead", "skin texture is detailed and realistic", "real"]
Due to computational constraints, we employ Vicuna-7B as the LLM to process visual, forgery, and question prompt embeddings. For LoRA adaptation, we add low-rank matrices with rank of to the q-proj. k_proj, e_proj, and o-proj modules. The lora aapha parameter is set 32, and lora.dropout is set to 0.1. During LLM training, the batch size is set to 1.
About LLM:
The video-level AUC is computed by aggregating frame-level binary outputs. For each video, we sample 32 frames uniformly and calculate the ratio of "yes" responses (indicating "fake"). This ratio indicates the video's s probability score to be a fake. To extract "yes" response from model output, we implemente a deterministic rule-based parsing strategy for extracting binary labels ("yes"""no") from model outputs. If the output contains "yes" or "is deepfake", the frame is labeled fake. If the response contains "no" or "not deepfake", the frame is labeled real. If none of the keywords are detected, the response is default labeled real.
Video-level AUC Calculation for LLM:
A.2. Robustness Perturbations
Given that deepfake content frequently shared on social media, is often subject to various perturbations such as noise, compression, and image enhancement. To assess robustness under these conditions, we apply a range of perturbations the FF++ dataset and then evaluate detection AUC and text description accuracy. As shown in Table 9, we follow the settings from (Nguyen et al., 2024) to benchmark approach against existing methods. The results demonstrate that our method achieves better robustness compared to prior approach (Shiohara & Yamasaki, 2022), maintaining high AUC values even under challenging conditions.
Methods
Saturation
Constrast
Block
JPEG
Noise
Blur
SBI 99.13
98.37
99.15
77.88
58.10
67.12
Ours
99.20
98.12
99.54
87.96
62.69
81.37
Table 9. Robustness evaluation on FF++. The highest AUC values among all methods are highlighted in bold.
13

---

Unlocking the Capabiliites of Large Vision- LasgunLLa Models for Generalizable and Explainable Deepfake Detection
A.3. Inference Time
In this section, we compare the inference time of our approach with existing approaches using a single NVIDIA RTX 4090 GPU. The inference time, measured in seconds per frame, is reported alongside the corresponding AUC scores for reference. As shown in Table 10, the variant employing only KFD exhibits slightly longer inference time than CADDM, yet achieves substantially higher precision. Similarly, our LVLM incurs greater inference time than FAK-Owl but also attains significantly superior precision. Moreover, unlike FAK-Owl which is constrained to binary (Yes/No) responses, our approach supports multi-turn dialogues providing improved interpretability and enhanced generalization. Overall, despite a higher inference time, our approach provides more precise detection and comprehensive interpretability.
Table 10. Inference time AUC on CDF2 dataset.
Inference time per frame(s)
AUC
CADDM
0.026
85.68
Ours-KFD
0.059
97.62
FAK-Owl 0.642
69.84
Ours-LVLM
1.211
95.97
A.4. Failure Cases and Limitations
Our approach faces limitations in the training strategy. The alternating training strategy for multi-turn dialogue introduces domain gaps: general-purpose VQA datasets prioritize object-centric reasoning, whereas fine-grained forgery detection requires localized artifact analysis. This misaligmment occasionally results in decrease in forgery detection performance (see Table 3). The Failure cases are shown in Figure 6. To address these limitations, we will construct domain-specinn forgery QA datasets with spatially grounded annotations in future works.
Image Type
Real
Fake
Location
Correct
Correct
LLM judgement
Wrong
Wrong
Q: this a despfake image?
Q: this deepfake image?
Yes, this deepfake image. image been generated using combination of machine learning slgorithms and human medifiie include text "Slovenly" "Slevenly" has been added the image.
No, not deepfake image. The image with a digital camera. The photo was edited with Adobe Photoshop Lightroo.
Q& A
The has been
Figure 6. Failure cases of our method. While the model precisely locate forgery regions, the L.M fails to generate accurate textual descriptions corresponding identified content.
14

---

Unlocking the Capabilities of Large
Models for Generalizable and Explainable Deepfake Detection
A.5. Conversations under various datasets
In this section, present additional dialogue results. We apply our method across five subsets of the FF++ dataset and obtain the corresponding responses, shown in Figure 7. It can be observed that the responses accurately determine authenticity. Furthermore, as illustrated in Figure 8, proposed approach continues to effectively identify forgeries in cross-dataset conversations.
DeepFake
Face2Face
FaceShifter
FaceSwap
NeuralTexture
Pristine
despfiic image.
imaps.
éorplki
desplko
the
deeplikk
Yes taage aniber. The setikes steble undomaniayy artihet haman faxs right image
You This derplkk aetiect right loft right the
Yos This despllk unidue the of
left
Yos Thes doepóko the anifect at of
fizz
unitee locmtd durp. and the arribet vastle and high contme:
This deepdake
Yeu. This doapfáce innae not focts, blumal
Yeu. This dezpláke image. the artifict the eye, lafi pis, jightjen jou ohe tragge
Yeu. This decplike imaas. the anillet the lell left chadk the imagee
Yes. This desplike the anillat is right right eye, left of image.
Yea. This deepláke imags. urifcct letl rightaye ohe irage.
No. This real image.
Figure 7. Forgery localization results and LLM responses on various forgery types from the FF++ dataset.
15

---

Unlocking the Capabilities of Large VisionLLanguan Models for Generalizable and Explainable Deepfake Detection
CDF2
DFDCP
DFDC
DFD
No, this not a doepfake image. The image appears be genuine photograph of person with visible nose und mouth.
No. this is not deepfake image. It is a real photograph of person's face
No. This not deepfake image. The image is genuine photograph of person.
No. This is pristine image.
No. this not a deepfake image. The image real photograph of with beard.
No, not deepfake image. image appeurs genuine photograph.
pristine image.
No. pristine image.
Yes. This is deepfake image, the artifict is at the center ficee of the image.
Yes. This is deepfake image, the artifact at the center face of the image.
Yes. This is deepfáke image. the artifact is at the center face of the image.
Yes. This is deepfake image. the artifict is at the center face of the image.
Yes. This is deepfake image, the artifact is at the nose ridgs. left chock, right check of the image.
Yes. This is deepfake image, the artifact nose ridge, left check image.
Yes. This a deepfikke image, artifisct the full fice image.
Yes. This deepfake image, artifact the full face of the image.
Figure Forgery localization results and LLM responses on various forgery types from the CDF2, DFDCP, DFDC, and DFD datasets.
A.6. Multi-turn Conversations
Unlike traditional deepfake detection algorithms, our proposed framework not only achieves accurate detection but also supports multi-ture dialogue capabilities. For instance, when a user queries, "Is this deepfake image?", "one???",ar the model provides precise responses, identifying whether the image is manipulated and offering relevant details. Furthermore, the framework enables users inquire about additional information, such as forgery scores specific aspects of the image content. As illustrated in Figures 9-26, we present examples from the FF++, CDFI, CDF2, and DFDC datasets. These results demonstrate the model's ability generate accurate and contextually relevant responses to user queries, underscoring its potential for real-world applications.
16

---

Unlocking the Capabilities of Large Vision-Langueee Models for Generalizable and Explainable Deepfake Detection
Figure 9. Conversations with real images from the FaceForen- sics++ dataset.
Figure 12. Conversations with real images from the FaceForen- sics++ dataset.
Figure Conversations with real images from the FaceForen- sics++ dataset.
Figure 13. Conversations with DeepFake images from the Face- Forensics++ dataset.
Figure 11. Conversations with real images from the FaceForen-
Figure 14. Conversations with DeepFake images from the Face- Forensics+ dataset.
sics++ dataset.
17

---

Unlocking the Capabilities of Large Vision-Langueee Models for Generalizable and Explainable Deepfake Detection
land hagg
Figure 15. Conversations with Face2Face images from the Face- Forensicst+ dataset.
Figure 18. Conversations with FaceSwap images from the Face- Forensics++ dataset.
Figure Conversations with Face2Face images from Face- Forensics++ dataset.
Figure 19. Conversations with NeuralTexture images from FaceForensics+* dataset.
Figure 17. Conversations with FaceSwap images from the Face-
Figure 20. Conversations with NeuralTexture images from the FaceForensics** dataset.
Forensies++ dataset.
18

---

Unlocking the Capabilities of Large Vision-Languune Models for Generalizable and Explainable Deepfake Detection
Figure 21. Conversations with fake images from the CDF1 dataset.
Figure 24. Conversations with fake images from the CDF2 dataset.
Figure 22. Conversations with fake images from CDF1 dataset.
Figure 25. Conversations with fake images from the DFDC dataset.
Figure 23. Conversations with fake images from the CDF2 datasct.
Figure 26. Conversations with fake images from the DFDC dataset.
19


Reviews:
[{"review_id": "1E1VeBNb6M", "reviewer": "Reviewer_KySN", "summary": "This papers introduces a method based on large vision language models (LVLMs) for the task of deepfake detection. To this end, the authors proposed a number of modules to enhance LVLMs performance on deepfake detection, including a knowledge-guided forgery adaptation module (KFD), a multi-modal prompt tuning framework and an iterative refinement strategy. As for data, the authors prepared their multimodal training data based on the real videos in FF++. In experiments, the authors conducted comprehensive experiments, including intra-dataset evaluation, cross-dataset evaluation, cross-manipulation evaluation of KFD, GradCAM visualization, etc. Experimental results demonstrate the effectiveness of the proposed method.\n\n## update after rebuttal\nThe authors have partially addressed my concerns in the rebuttal phase. However, based on the authors' responses, I believe that a significant portion of the main paper content requires substantial revision, such as the missing demonstrations for multi-turn dialogue capabilities and details for metric calculations. I feel uncertain whether such changes can be properly reflected in the revised version of the paper. Therefore,  I will maintain my original rating and recommend that the authors consider resubmitting the revised paper to a future conference.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1) Multi-turn Dialogue Capabilities:** We appreciate your feedback. Following the strategy in AnomalyGPT, the alternating training strategy (Section 3.3, Implementation Details) inherently preserves Vicuna-7B’s multi-turn dialogue capabilities. While Figure 5 illustrates single-turn examples for clarity, the supplementary material (available on https://anonymous.4open.science/r/DFDGPT-8E5C) includes multi-turn dialogue screenshots. We apologize for the submission negligence and confirm full accessibility of all examples post-publication.\n\n**2) Video-level AUC Calculation for LLM:** The video-level AUC is computed by aggregating frame-level binary outputs. For each video, we sample 32 frames uniformly and calculate the ratio of “yes” responses (indicating “fake”). This ratio indicates the video’s probability score to be a fake. To extract “yes” response from model output, we implemented a deterministic rule-based parsing strategy for extracting binary labels (\"yes\"/\"no\") from model outputs. If the output contains \"yes\" or \"is deepfake\", the frame is labeled fake. If the response contains \"no\" or \"not deepfake\", the frame is labeled real. If none of the keywords are detected, the response is default labeled real. It is the common strategy to SOTA LLM-based methods and we use the same sampling strategy (32 frames/video) and aggregation (mean pooling) for fair comparison. We will add this clarification in Section 4.1.\n\n**3) CADDM Data Simulation:** Thank you for identifying this mistake. Yes, CADDM used fake data for training. We corrected it in Table 2.\n\n**4) Inference Time:** We add the evaluation of inference time on the CDF2 dataset. The inference time in our method is mainly consumed by the LLM we used, as shown in Table A. Our method with VLM model only takes more time for inference than CADDM, but get much better precision. Our method incorporating both the VLM and LLM takes more time than FAK-Owl, but also have much better precision. In addition, FAK-Owl can only provide the binary (Yes/No) responses and lacks multi-turn dialogue capabilities. As a whole, despite taking more inference time, our method has better precision and enables explainability and generalization.\n\n||Inference time per frame(s)|AUC|\n| :-: | :-: | :-: |\n|CADDM|0\\.026|85\\.68|\n|Ours-VLM|0\\.059|97\\.62|\n|FAK-Owl|0\\.642|69\\.84|\n|Ours-LLM|1\\.211|95\\.97|\n\n**5) Iterative Refinement Strategy:** The iterative refinement strategy refers to the alternating training between the deepfake detection task and general visual dialogue task, as described in Section 4.1 (Implementation Details). This strategy enables our model to retain multi-turn reasoning capabilities by optimizing forgery detection loss and dialogue loss cyclically (see the supplementary materials in https://anonymous.4open.science/r/DFDGPT-8E5C).", "claims_and_evidence": "Some of the claims are not fully supported by experimental results.\nFor example, the authors claimed that their method not only supports deepfake detection but also facilitates multi-turn dialogues in Section 4.3. \nHowever, the results in Figure 5 seems only cover the results for single-turn dialogues.", "methods_and_evaluation_criteria": "In this paper, the authors mainly used the criteria of video-level Area Under the Receiver Operating Characteristic Curve (AUC), namely video-level AUC. \nBased on my understanding, the video-level AUC requires a probability score for each video, and such a probability score is usually calculated by averaging the probability score of sampled frames in each video.\nHowever, in this paper, the authors' method can only provide 'yes' or 'no' for each sampled frame, namely a one-hot vector instead of a probability score. How do the authors calculate the video-level AUC then, by considering the fraction of 'yes' responses in all responses of sampled frames?\nIf so:\n- How do the authors extract 'yes' from models' responses, with chatgpt or manual rules?\n- Is it still fair to compare with other SOTA-methods, where they calculate video-level AUC based the mean probability score of sampled frames?\n- How many sampled frames are used per video?\n\nI believe such discussions are required.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Most of the experiments are sound and valid.\nHowever, I have some concerns over the results in Cross-Manipulation Evaluation of KFD, namely Table 2.\nBased on my understanding, the training of CADDM typically required fake data. How to the authors trained CADDM with only real data in this section?", "relation_to_broader_scientific_literature": "The task of this paper is to perform generalized deepfake detection based on LVLMs.\nFirstly, this could expand the applicability of LVLMs to the area of deepfake detection, further fostering the development of AGI.\nBesides, this could also boost the development of deepfake detection, by presenting models of better performance.", "essential_references_not_discussed": "The authors have conducted comprehensive literature review.", "other_strengths_and_weaknesses": "- One major weakness of the proposed method is its inference time.\nUsing the Vicuna-7B model for inference could significantly slow down the inference speed compared with other SOTA methods, which typically contain less than 1B parameters.\nIt is recommended that authors should discuss about the inference time and the overall throughput.\n- I did not find the specific details for \"An iterative refinement strategy enabling multi-turn dialog for evidence-based reasoning\", which is mentioned in the abstract. I feel like this paper is still unfinished.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tlDsEpnT86", "reviewer": "Reviewer_MUQX", "summary": "This paper proposes leveraging LLM and VLM to improve the model generalization and explainability. It is achieved by a two-stage pipeline: A Knowledge-guided Detection using humans prior to generating feature embedding; leveraging these embedding for LLM to output detection results. The experimental results show that it successfully incorporates the capacity of LLM into deepfake detection.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1)AUC Calculation by LLM Output:** To ensure rigorous and reproducible evaluation of text-level AUC, we implemented a deterministic rule-based parsing strategy for extracting binary labels (\"yes\"/\"no\") from model output. If the output contains \"yes\" or \"is deepfake\", the frame is labeled fake. If the response contains \"no\" or \"not deepfake\", the frame is labeled real. If none of the keywords are detected, the response is default labeled real. The experimental results in Table 3,4,5,7 prove the effectiveness of our scheme.\n\n**2)Cross-dataset evaluation of LLM&VLM:** We appreciate the suggestion. A figure illustrating the cross-dataset evaluation has been added to the anonymous GitHub repository and is available at https://anonymous.4open.science/status/DFDGPT-8E5C. We will incorporate this figure into Section 4.3.\n\n**3) Comparison with Recent SOTA:**  Similar to Table 2, we add more comparisons against recent SOTA detection methods across various test sets.  The new results are listed in Table A below and will be incorporated into our manuscript. ProDet is evaluated using publicly available code, while the results for other approaches are obtained from their original publications.\n\nTable A. Generalization Performance across various datasets.\n\n|Methods|Venue|CDF2||DFDC||CDF1||DFDCP||\n|-|-|-|-|-|-|-|-|-|-|\n|||AUC|AP|AUC|AP|AUC|AP|AUC|AP|\n|ProDet|NIPS’24|92.62|96.05|71.52|72.8|94.48|96.66|82.83|88.89|\n|RepDFD|AAAI’25|89.94|-|80.99|-|-|-|95.03|-|\n|CFM|TIFS’24|89.65|-|80.22|-|-|-|-|-|\n|ED|AAAI’24|93.6|-|75.4|-|-|-|90.2|-|\n|UDD|AAAI’25|93.13|-|81.21|-|-|-|88.11|-|\n|Ours|-|94.71|93.59|79.12|77.69|97.62|97.67|91.81|88.26|\n\n**4) More Datasets:** As suggested, we have used the DF40 dataset to further evaluate the generalization capability of our approach. The DF40 dataset comprises several synthetic deepfake datasets that are generated using real images in FF++. We use these datasets to evaluate our method’s cross-manipulation detection ability. Notably, our approach continues to exhibit robust detection performance against state-of-the-art deepfake generation techniques (e.g., FSGAN, E4S, LIA, StyleGAN, DDIM, PixArt-α, etc.). \n\nTable B. Generalizaiton performance across various deepfake methods. The models are all trained on the Faceforensic++ dataset and then evaluated on unseen types of deepfakes.\n|Face-swapping||||||||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||UniFace|SimSwap|InSwapper|FSGAN|FaceDancer|BlendFace|e4s|FaceSwap|\n|SBI|89.02|93.22|88.52|89.62|78.18|95.09|86.36|94.37|\n|CADDM|86.86|90.41|78.65|88.86|76.54|90.75|87.92|97.96|\n|Ours|90.61|90.97|87.64|93.75|82.97|92.10|94.68|93.34|\n|Face-reenactment|||||||||\n||PIRender|OneShot|HyperReenact|FOMM|FS_vid2vid|TPSMM|MCNet|LIA|\n|SBI|81.81|87.54|65.31|88.05|83.72|82.13|83.47|89.22|\n|CADDM|77.37|85.05|69.26|84.77|72.86|71.35|73.40|69.67|\n|Ours|88.29|90.45|81.55|93.34|71.56|78.14|81.48|99.99|\n|EntireFaceSynthesis|||||||||\n||VQGAN|StyleGAN2|StyleGAN3|StyleGAN-XL|DDIM|DiT-XL/2|PixArt-α|RDDM|\n|SBI|91.50|97.91|97.91|23.26|99.56|79.04|98.78|53.66|\n|CADDM|99.99|100.00|100.00|98.69|98.10|79.90|99.74|98.59|\n|Ours|99.99|100.00|100.00|100.00|99.93|94.18|100.00|86.59|\n\n**5) Limitations about Simulation:** Yes. The SBI-based synthetic forgery pipeline is primarily designed to simulate face-blending artifacts by focusing on modeling boundary inconsistencies. Although it is effective in detecting certain forgeries, its performance degrades when applied to fully synthesized images generated by models such as StyleGAN-XL and RDDM, as evidenced by the experimental results in our response to Q3. While our approach is built on SBI, the integration of pre-trained knowledge from LVLM has notably enhanced its generalized detection ability across various types of forgeries (see Table B). In future work, we will incorporate model-related artifacts into the forgery generation process to further improve detection performance. We will include a dedicated discussion section to further elaborate on these limitations and outline potential directions for future work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes, it makes sense.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "It is partially reasonable. I have a few suggestions.\n1. Fig. 5 is a crucial experiment to validate one of the most important advantages of using LLM&VLM, that is, its explainability. Hence, it should be conducted on a larger scale, at least not limited to the in-dataset scenarios. In addition, you should have appropriate strategies for evaluating the accuracy of the generated text, otherwise, how do you know if the output text is appropriate?\n2. More recent SoTA should be compared. For example, [1] [2] [3]. Among them, [1] also discusses the usage of SBI, which is deployed in this paper.\n3. More datasets are recommended. The applied datasets in Tab. 1 are indeed only three types, i.e., CDF, DFDC, and DFD. You may include more datasets like DF40 for better illustrations.\n\n\n\n\n\n[1] Can We Leave Deepfake Data Behind in Training Deepfake Detector?// NIPS24\n\n[2] Exploring Unbiased Deepfake Detection via Token-Level Shuffling and Mixing // AAAI25\n\n[3] DiffusionFake: Enhancing generalization in deepfake detection via guided stable diffusion //NIPS24", "relation_to_broader_scientific_literature": "It is relevant to the generalizable deepfake detection.", "essential_references_not_discussed": "Please refer to Experimental Designs Or Analyses.", "other_strengths_and_weaknesses": "The method employed is rather simple, essentially following a contrastive learning approach. However, the problem it addresses is intriguing and holds practical significance. My primary concern is that the experiments may be somewhat insufficiently conducted， please refer to Experimental Designs Or Analyses.", "other_comments_or_suggestions": "**update after rebuttal**\nThe authors have partially addressed my concern, therefore I retain my original rating.", "questions_for_authors": "Notably, SBI can only simulate the face-blending artifacts. Therefore, it cannot provide forgery clues about generative artifacts. How can your method deal with the fake image generated by Entire-face Synthesis without blending?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ky4jInvyb", "reviewer": "Reviewer_Led7", "summary": "This paper proposes a novel framework that leverages Vision-Language Models (VLMs) for deepfake detection, addressing their current limitations in forensic analysis. The core innovation is a three-component approach: (1) a knowledge-guided forgery adaptation module that aligns VLM semantic space with forensic features through contrastive learning, (2) a multi-modal prompt tuning framework that optimizes visual-textual embeddings for improved localization and explainability, and (3) an iterative refinement strategy enabling evidence-based reasoning through multi-turn dialog. The architecture integrates a Knowledge-guided Forgery Detector with a Large Language Model, allowing the system to not only detect and localize deepfakes across diverse manipulation types but also provide natural language explanations about the detected forgeries, significantly advancing both generalizability and interpretability in deepfake detection.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1) Multi-turn Dialogue Capabilities:** See response to KySN Q1.\n\n**2) Learnable Context:** Textual descriptions are generated via GPT-4, validated by human annotators. Examples include “Inconsistent head poses” or “Mismatched skin texture”. These annotations are available in https://anonymous.4open.science/r/DFDGPT-8E5C.\n\n**3) Quantitative Evaluation of Localization Accuracy and Explanation Quality:** We thank the reviewer for the valuable suggestion. As suggested, we have conducted a quantitative evaluation for both the localization and explanation capabilities of our approach using two metrics: Text Localization Accuracy (TLA) and Cosine Semantic Similarity (CSS). TLA measures the consistency between the tampered region descriptions produced by our LLM and the ground-truth localization annotations, by using the Dice coefficient. To objectively assess the explanation quality, following FakeShield (ICLR’25), we calculate the CSS by computing the cosine similarity between the high-dimensional semantic vector representations of the generated explanation text and the corresponding ground-truth text.\nFor this evaluation, we trained both our approach and a fine-tuned version of PandaGPT on our synthetic forgery dataset. The results, summarized in the table below, indicate that our method achieves significantly higher localization accuracy and explanation quality than PandaGPT. We appreciate the reviewer's suggestion, and we will incorporate these quantitative evaluation metrics and the corresponding results into the revised manuscript.\n\n|| CDF1     |         | CDF2    |         | DFDC    |         | DFDCP   |         |\n|:--------:|:--------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n|          | TLA     | CSS     | TLA     | CSS     | TLA     | CSS     | TLA     | CSS     |\n| PandaGPT | 0.6239  | 0.7666  | 0.6241  | 0.7717  | 0.6389  | 0.7606  |  0.6220 | 0.7846  |\n| Ours     | 0.7762  | 0.8532  | 0.7755  | 0.8498  | 0.7662  | 0.8235  | 0.7842  | 0.8370  |\n\n**4) Component Contribution:** We appreciate the opportunity to clarify the contributions of each module in our framework. In Table 5, we present ablation experiments on our three primary modules, which validate their individual effectiveness. The Reference-based Optimization Process (ROP) is specifically designed to enhance the training stability and feature discriminability of the Knowledge-guided Forgery Detector (KFD). To isolate its contribution, we compare its performance with and without ROP. The ROP is not directly connected to other components (LLM and LoRA), and its benefits are implicitly propagated through KFD’s refined features during training. We will explicitly clarify ROP’s role in Section 3.1 to avoid ambiguity.\n\n**5) Simulation Limitations:** See response to MUQX Q4.\n\n**6) AUC Calculation by LLM Output:** See response to MUQX Q1.\n\n**7) Implementation Fairness about Qwen:** We note that Qwen-VL is a general-purpose visual question answering model and is not specifically designed for deepfake detection, which leads to its lower performance. To ensure fairness, all methods are evaluated under identical pre-processing conditions (32 frames per video, 224×224 resolution) and consistent evaluation protocols. Furthermore, we will release our code to facilitate reproducibility.\n\n**8) Inference Time:** See response to KySN Q5.\n\n**9) Failure Cases and Limitations:**  Our approach faces limitations in the training strategy. The alternating training strategy for multi-turn dialogue introduces domain gaps: general-purpose VQA datasets prioritize object-centric reasoning, whereas fine-grained forgery detection requires localized artifact analysis. This misalignment occasionally results in a decrease in forgery detection performance (see Table 2). The Failure cases are available in https://anonymous.4open.science/r/DFDGPT-8E5C. To address these limitations, we will construct domain-specific forgery QA datasets with spatially grounded annotations. We will add a discussion section to elaborate limitations and future works.  \n\n**10) SOTA Generators Tested:** We evaluated various deepfakes generated by state-of-the-art (SOTA) models, including StyleGAN-3, DiT-XL/2, Stable Diffusion, etc. For experimental results, refer to the response to MUQX Q3.\n\nWe thank all reviewers for their constructive feedback. Revisions will address every point rigorously.", "claims_and_evidence": "The paper's claims are partially supported by evidence, with several limitations:\n\n1. The multi-turn dialogue capability claim is prominently featured in the abstract and conclusion but receives minimal validation in the main paper. Figure 5 shows only single-turn examples, and the paper notes that \"Additional multi-turn dialogue examples are provided in the supplementary material.\" However, no appendix or supplementary material is found. Without these examples, this key claimed contribution lacks sufficient evidence.\n\n2. The \"knowledge-guided forgery adaptation module\" lacks clear documentation of what specific external manipulation knowledge is incorporated. While Section 3.1 describes the mechanism, it doesn't detail the nature of the textual descriptions ($D _ {real}$ and $D _ {fake}$) that serve as the knowledge source.\n\n3. The forgery localization capability is demonstrated qualitatively through GradCAM visualizations in Figure 4, but no quantitative evaluation of localization accuracy is provided, making it difficult to objectively assess this capability.\n\n4. The ablation study in Table 8 evaluates the Reference-based Optimization Process but doesn't fully isolate the contribution of each of the three claimed novel components, particularly the multi-modal prompt tuning framework's specific impact.\n\n5. The data simulation process described in Section 3.3 is used for training, but there's insufficient evaluation of how well this simulated data represents real-world deepfakes, which could affect generalization claims.", "methods_and_evaluation_criteria": "The proposed methods generally align with the deepfake detection problem, but several evaluation aspects raise concerns:\n\n1. For evaluation datasets, the authors appropriately use standard benchmarks (FF++, CDF2, DFD, DFDCP, DFDC) and conventional metrics (AUC and AP), which is reasonable for comparative assessment.\n\n2. The cross-dataset and cross-manipulation evaluations are particularly appropriate for testing generalization capabilities, which is a critical challenge in deepfake detection.\n\n3. However, for a framework claiming explainability as a key contribution, there's no quantitative evaluation of explanation quality. While textual outputs are shown in Figure 5, no metrics assess whether these explanations accurately identify the manipulation technique or forgery characteristics.\n\n4. For the claimed localization capability, only qualitative GradCAM visualizations are provided without quantitative localization accuracy metrics, making it difficult to objectively compare with other localization approaches.\n\n5. The paper emphasizes multi-turn dialogue as a key capability. However, it neither establishes a standardized evaluation protocol to measure its effectiveness in deepfake analysis nor provides any examples of such interactions.\n\n6. The forgery data simulation approach for training relies on Poisson blending of affine-transformed real images, but there's limited analysis of whether this adequately represents the artifacts found in modern AI-generated deepfakes, potentially limiting real-world applicability.", "theoretical_claims": "This paper does not present significant theoretical claims requiring formal mathematical proofs. The work is primarily empirical, focusing on the design and evaluation of a VLM-based framework for deepfake detection.\n\nThe mathematical formulations presented in the paper (Equations 1-6) employ standard techniques commonly used in machine learning:\n- Equations 1-2 describe similarity computations between visual and textual features\n- Equation 3 uses the established Dice loss for segmentation\n- Equation 4 employs standard binary cross-entropy loss for classification\n- Equation 5 uses cross-entropy loss for the LLM\n- Equation 6 describes a basic blending process for generating training data\n\nThese formulations appear correctly applied for their intended purposes, but they don't constitute novel theoretical contributions requiring verification of mathematical proofs.", "experimental_designs_or_analyses": "I examined several aspects of the experimental methodology, finding both strengths and limitations:\n\n1. The dataset selection and evaluation protocols using standard benchmarks (FF++, CDF2, DFD, DFDCP, DFDC) with established metrics (AUC, AP) follow sound practices in the field.\n\n2. The cross-dataset evaluation appropriately tests generalization capability, training on FF++ real data and testing across multiple datasets.\n\n3. However, the LLM evaluation process lacks methodological clarity. The paper states they \"utilize the LLM's output ('Yes' or 'No') to classify authenticity\" but doesn't explain how they handle cases where LLM outputs may be nuanced or ambiguous rather than strictly binary.\n\n4. The LVLM-based method comparison in Table 3 shows baseline performances that are surprisingly low (e.g., Qwen2-VL at ~48% AUC on some datasets, near random chance), raising questions about implementation fairness or model configuration.\n\n5. The training data simulation using Poisson blending of affine-transformed images (Section 3.3) may not adequately represent artifacts found in modern AI-generated deepfakes, yet this potential limitation isn't acknowledged.\n\n6. The ablation studies, while informative, don't systematically isolate the contribution of each of the three claimed core components, particularly for the multi-modal prompt tuning framework.", "relation_to_broader_scientific_literature": "This paper's contributions relate to several research streams:\n\n1. **Deepfake detection methods**: Traditional approaches (Li et al., 2020; Shiohara & Yamasaki, 2022; Nguyen et al., 2024) focused on data augmentation, feature consistency, and frequency domain analysis. This work acknowledges their limitations in capturing human knowledge about forgery characteristics and proposes VLMs as a solution.\n\n2. **Vision-Language Models**: Builds upon recent LVLM architectures like BLIP-2 (Li et al., 2023), LLaVA (Liu et al., 2024), and MiniGPT-4 (Zhu et al., 2024), but adapts them specifically for forensic analysis --- a departure from their general image understanding focus.\n\n3. **Multimodal forensics**: Extends work like FakeShield (Xu et al., 2024) and FKA-Owl (Liu et al., 2024b), which also applied LVLMs to forgery detection, by introducing the knowledge-guided forgery detector and incorporating detailed localization capabilities.\n\n4. **Prompt tuning literature**: The forgery prompt learning approach builds upon prompt tuning techniques (Lester et al., 2021; Liu et al., 2022) but adapts them for the multimodal forensic context.\n\n5. **Explainable AI**: While works like FFAA (Huang et al., 2024) also explored explainable forgery analysis, this paper's integration of multi-turn dialogue capabilities represents an evolution in interactive forensic analysis.", "essential_references_not_discussed": "The paper adequately cites relevant related works, with no major omissions.", "other_strengths_and_weaknesses": "**Strengths:**\n1. The integration of VLMs with deepfake detection addresses a genuine need for improved generalization and explainability in forensic analysis.\n2. The knowledge-guided approach acknowledges an important gap in current methods: the difficulty of capturing human forensic knowledge through data augmentation alone.\n3. The reference-based optimization process shows promising results for enhancing feature robustness.\n4. The paper demonstrates versatility by evaluating both cross-dataset and cross-manipulation scenarios, which is crucial for real-world applications.\n\n**Weaknesses:**\n1. The performance improvements, while consistent, are relatively modest (average 1.34% AUC improvement) given the complexity of the proposed framework.\n2. The technical description of the knowledge acquisition process is vague --- specifically how the \"learnable context\" is integrated with predefined real/fake descriptions.\n3. The framework's complexity (multiple interconnected components) may hinder practical deployment compared to simpler approaches.\n4. The paper doesn't adequately analyze failure cases or where the approach struggles, which would provide valuable insight into its limitations.\n5. The training data simulation approach may not capture the sophisticated artifacts produced by state-of-the-art deepfake generators, potentially limiting real-world effectiveness.\n6. The lack of quantitative metrics for both localization accuracy and explanation quality makes it difficult to objectively assess two of the framework's key claimed capabilities.", "other_comments_or_suggestions": "1. **Typos and clarifications needed:**\n   - Abstract: \"due to the misaligned of their knowledge\" should be \"misalignment\"\n   - Section 3.2: \"$E _ {forgery} \\in \\mathbb{R}^{{n _ f} \\times C _ {emb}}$\" - $n _ f$ is not properly defined\n\n2. **Technical clarifications needed:**\n   - The \"learnable context\" mentioned in Section 3.1 needs more detailed explanation\n   - The specific format of prompts used during LLM evaluation should be explicitly shown\n   - The process for converting LLM outputs to binary decisions for AUC calculation requires clarification\n\n3. **Evaluation suggestions:**\n   - Include quantitative metrics for localization accuracy\n   - Provide more examples of multi-turn dialogues in the main paper\n   - Compare computational efficiency and inference time with existing methods\n   - Consider human evaluation for explanation quality\n\n4. **Missing discussion:**\n   - Analysis of potential failure cases would strengthen the paper", "questions_for_authors": "1. Could you provide quantitative evaluation metrics for the localization accuracy of your approach? The current paper only shows qualitative visualization without objective metrics to compare against other localization-capable methods.\n\n2. What specific \"external manipulation knowledge\" is incorporated into your framework? Section 3.1 mentions \"real and fake image descriptions\" but doesn't detail their content or source, making it difficult to assess this key component.\n\n3. The multi-turn dialogue capability is prominently claimed but not demonstrated. Could you explain how you evaluate dialogue quality, and how your approach specifically enables multi-turn reasoning beyond what existing LVLM methods provide?\n\n4. In Table 3, several baseline LVLM methods (e.g., Qwen2-VL at 47.99% average AUC) perform near random chance. What explanation do you have for these unexpectedly low baselines?\n\n5. How does your forgery data simulation approach using Poisson blending of affine-transformed images capture the artifacts produced by modern AI-generated deepfakes? This seems critical for real-world generalization.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by 3 experts in the field who provided detailed suggestions. The reviewers’ recommendations were divergent, with reviewer MUQX recommending Weak Accept, and reviewers KySN and Led7 recommending Weak Reject.

All three reviewers noted the paper's strengths in addressing the important problem of generalizable and explainable deepfake detection by leveraging VLMs. Their concerns, however, included insufficient validation and detail for the claimed multi-turn dialogue capabilities (KySN, Led7), computational cost and inference time due to the large models used (KySN), the lack of quantitative metrics for evaluating the claimed localization and explanation quality (MUQX, Led7), and insufficient comparison with the most recent state-of-the-art methods (MUQX). Methodological clarity regarding AUC calculation from binary outputs and the representativeness of the simulated training data were also questioned (all three reviewers).

I read the reviews and rebuttal and recognize the potential merits in the paper. Though I share the reviewers' concerns, these seem to focus more on how the method was evaluated (and where the results were to be found in the supplemental) rather than the essence of the proposed contribution. These concerns, while valid, do not rise in my opinion to the level of a reject recommendation and so recommend accordingly.
Decision: Accept (poster)