Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes
Authors: Kaiqing Lin
Zhiyuan Yan
Ke-Yue Zhang
Li Hao
Yue Zhou
Yuzhen Lin
Weixiang Li
Taiping Yao
Shouhong Ding
Bin Li
Venue: NeurIPS 2025 poster ()
Keywords: Personalized Deepfake Detection
Multi-media Forensic
Face Recognition
MLLMs
Primary area: applications

Abstract: Securing personal identity against deepfake attacks is increasingly critical in the digital age, especially for celebrities and political figures whose faces are easily accessible and frequently targeted.
Most existing deepfake detection methods focus on general-purpose scenarios and often ignore the valuable prior knowledge of known facial identities, e.g., "VIP individuals" whose authentic facial data are already available. 
In this paper, we propose **VIPGuard**, a unified multimodal framework designed to capture fine-grained and comprehensive facial representations of a given identity, compare them against potentially fake or similar-looking faces, and reason over these comparisons to make accurate and explainable predictions.
Specifically, our framework consists of three main stages. First, we fine-tune a multimodal large language model (MLLM) to learn detailed and structural facial attributes. 
Second, we perform identity-level discriminative learning to enable the model to distinguish subtle differences between highly similar faces, including real and fake variations. Finally, we introduce user-specific customization, where we model the unique characteristics of the target face identity and perform semantic reasoning via MLLM to enable personalized and explainable deepfake detection.
Our framework shows clear advantages over previous detection works, where traditional detectors mainly rely on low-level visual cues and provide no human-understandable explanations, while other MLLM-based models often lack a detailed understanding of specific face identities.
To facilitate the evaluation of our method, we build a comprehensive identity-aware benchmark called **VIPBench** for personalized deepfake detection, involving the latest 7 face-swapping and 7 entire face synthesis techniques for generation. 
Extensive experiments show that our model outperforms existing methods in both detection and explanation.
The code is available at https://github.com/KQL11/VIPGuard .

Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes
Kaiqing Lin Zhiyuan Yan Ke-Yue Zhang', Li Hao Yue Zhou', Yuzhen Lin', Weixiang Li', Taiping Yao"i, Shouhong Ding', Bin Li
*Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen Key Laboratory of Media Security, and SZU-AFS Joint Innovation Center for AI Technology. Shenzhen University School of Electronic and Computer Engineering, Peking Univerisity. 'Tencent Youtu Lab
Abstract
Securing personal identity against deepfake attacks is increasingly critical in the digital age, especially for celebrities and political figures whose faces are easily accessible and frequently targeted. Most existing deepfake detection methods focus on general-purpose scenarios and often ignore the valuable prior knowledge known facial identities, e.g., "VIP individuals" whose authentic facial data already available. In this paper, we propose VIPGuard, a unified multimodal framework designed capture ane-grained and comprehensive facial representa- tions of a given identity, compare them against potentially fake or similar-looking faces, and reason these comparisons make accurate and explainable predic- tions. Specifically, our framework consists of three main stages. First, we fine-tune multimodal large language model (MLLM) learn detailed and structural facial attributes. Second, we perform identity-level discriminative learning to enable the model distinguish subtle differences between highly similar faces, includ- ing real and fake variations. Finally, we introduce user-specific customization, where we model the unique characteristics of the target face identity and perform semantic reasoning via MILM to enable personalized and explainable deepfake detection. Our framework shows clear advantages over previous detection works, where traditional detectors mainly rely on low-level visual cues and provide no underttaaddt explanations, while other MLLL-bassed models often lack a detailed understanding of specific face identities. To facilitate the evaluation of our method, we build comprehensive identity-aware benchmark called VIPBench for personalized deepfake detection, involving the latest 7 face-swapping and 7 entire face synthesis techniques for generation. Extensive experiments show that our model outperforms existing methods in both detection and explanation. The code is available at https: sstps://githut //gttuu..... com Com//QQL11VVIPGual
1 Introduction
The rapid advancement of generative Al techniques [51, 59, 68, 75, 52, 13] has led to the widespread creation and dissemination of deepfake content synthetic media where person's 's identity is manip- ulated or replaced, often without consent. These techniques severely threaten personal reputation and
Equal contribution, Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Image
Image (Natural)
(Natural)
Heavier under-ep -aye pouches In the generated image.
Clearer naselabiel folds VIP Guard in the generated image.
VIP Guard
Image (Generated)
Image (Generated)
Rider eye-to-brow distance An the generatas 1950e
Broader nose the generated zhope
faoe
face
Guard
Figure 1: An illustrative comparison between natural images (from LAION-Face [87]) and images generated by GPT-4o [48], showing localized inconsistencies in facial attributes, such as eye pouches and facial shapes.
public trust, especially for high-profile individuals such as celebrities, political figures, and public officials [2]. To this end, protecting one's identity from such manipulations has become more than just personal privacy issue it is pressing societal concern.
existing methods designed for general- purpose use [50, 36, 56, 76, 72, 44, 88, 89, 82, 83, 84). They aim to classify any face image or video real fake, without considering whose face is being targeted. In the real world, many scenarios provide access to prior knowledge about the target identity such as in the case of public figures or known individuals [16]. This opens new avenue for detection: Can we leverage the known facial identity to improve both detection and explainability in personalized deepfake detection? Rather than treating all faces the same, identity- aware detection methods [54, 19, 17, 16, 81] focus on the semantic alignment between the input image and the authentic identity. By doing so, they provide more personalized and context- aware predictions, potentially improving both detection performance and explainability.
Although deepfake detection has gained attention,
However, existing identity-aware detectors [16, 81] fail to fully utilize the detailed identity-specific information. They primarily rely on global facial features while neglecting fne-graimed semantic details-sude such as eye shape, facial contours, or other attributes. As shown in Figure 1, an image generated by the latest GPT-4o [48] can appear highly realistic at first glance, but still contains subtle inconsistencies local facial regions-for example, unusually pronounced eye bags. When the target identity is known and all facial details are available, leveraging these fine- tne-grained discrepancies for detection becomes especially promising.
To this end, in this paper, we propose VIPGuard, a unified multimodal framework for detecting and explaining deepfakes targeting specific users. VIPGuard addresses identity-aware deepfake detection by explicitly incorporating known facial identity priors, including both global identity information and detailed structural facial attributes from the VIPs. To achieve this, this paper, for the first time, reformulates forgery detection as a fine- ffine-grained face recognition task, targeted at VIP identities. To leverage both global and local facial information, we use pre-trained face models [62, 14, 11] to extract global facial priors (face similarity scores) and local facial priors (facial attributes). Leveraging these priors, we aim to enable an MLLM perform forgery detection through semantic comparisons between suspect and authentic faces across visual and textual inputs. We train VIPGuard in three stages: (1) fine-tuning MLLM with a variety of facial attribute data to enhance facial understanding; (2) Performing discriminative learning to distinguish some arbitrary identities from manipulated or similar-looking faces by reasoning;
(3) Supporting personalized detection by learning unique and lightweight VIP token to represent each target identity for customized reasoning.
In this work, refer such individuals as "VIPs" whose identities aim protect.

---

Data Collection
VIP-Guard
Evaluation
Facial Atriibutes
Pre-trained MLLM
VIPEval
Deveription Dataset
Testt
Stage
ID Discrimination Datuser
Stage 2,3
Train
Are they the somte person?
Noi
CraudD
Fine-gruined Facial Comparison
Guard
Figure 2: Overview of the data collection, VIP-Guard, and Evaluation. VIPGuard's training and infer- ence pipeline for facial attribute understanding, identity discrimination, and VIP user customization.
To enable robust evaluation in personalized deepfake detection, we additionally present VIPBench comprehensive and identity-centric benchmark that differs fundamentally from conventional deepfake benchmarks [77, 74]. While existing benchmarks typically treat faces generically and overlook whose identity is being manipulated, VIPBench focuses explicitly on identity-aware scenarios, where prior knowledge of the target individual is available. VIPBench includes 22 specific target identities and total of 80,080 images, covering both real and forged samples. These forgeries are generated using 14 state-of-the-- thrrtttt methods, spanning 7 face-swapping (FS) and 7 entire-face synthesis (EFS) techniques, providing diverse manipulation types and realistic evaluation settings. By centering evaluation around known identities and incorporating
annotations, VIPBench allows models to be assessed not only on detection accuracy but also on their ability to leverage identity-specific cues.
Our main contributions are summarized as follows:
We introduce new formulation for personalized deepfake detection that targets specific individuals, casting it as fine-grained face recognition problem based on both global identity features and detailed facial attributes. This formulation requires only a small number of authentic reference images per VIP user, making it practical for real-world personalized protection scenarios.
We propose VIPGuard, unified multimodal framework for identity-aware deepfake detec- tion and explanation. VIPGuard incorporates pre-trained facial prior models and multimodal large language models, and follows a three-stage pipeline that extracts identity features, performs visual-text reasoning, and supports personalized detection through lightweight identity tokens.
We introduce VIPBench, a comprehensive benchmark for evaluating identity-awan deep- fake detection. It consists of 22 real-world target identities and 80,080 images generated by 14 state-of-the-art manipulation methods, enabling fine-grained and realistic assessment of personalized detection performance.
2 Related Works
2.1 General Deepfake Detection
Conventional Deepfake Detection Current deepfake detection faces significant challenges in generalization. To illustrate, researchers have explored a range of approaches, including data aug- mentation [38, 36. 56, 4, 6]. trequency-basee cues [50, 43, 88]. identity-awan learning [15, 27], disentanglement [78, 76], reconstruction [61, 79], and custom network designs [12]. Data augmenta- tion has proven especially effective for improving generalization for example, FWA [38] simulates warping artifacts, Face X-ray and SBI [36, 56] target blending boundaries, and SLADD [4] uses adversarial examples to challenge models. Despite these advances, most traditional detectors still offer only binary outputs without uuman-understandabble explanations, leaving users unclear about why face is classified as fake- fake-liniting trust and explainability.
Deepfake Detection via Multimodal Large Language Model Vision and language are two core modalities in human perception, driving growing interest in visual- multimodal learning.
3

---

In deepfake detection, several studies [31, 55, 80] have explored prompt-based strategies for face forgery analysis, showing that multimodal large language models (MLLMs) offer better explainability than traditional detectors. Others [20, 70, 71, 35, 66, 33, 86, 25] have investigated different MI.LMS for explainable detection, while [37] introduced laheled multimodal dataset to support fine-tuning. Moreover, 2DDDDD [7] further investigates hybrid frameworks that integrate conventional visual models with MLLMs. However, most of these methods are designed for general-purpose purpose detection and overlook the valuable identity- information available in many real-world scenarios.
2.2 Personalized Deepfake Detection for Specific Identity Protection
To detect identity inconsistencies in forged faces, prior works [54, 19, 17, 16, 10, 46] use reference images for personalized deepfake detection. For example, ICT-Ref [16] employs a transformer to detect mismatches between inner- and outer-face regions, while DifIID [81] uses reconstruction- based identity distances to identify fakes. However, these methods mainly rely on global identity features and overlook deeper, user-specific information. This limits their robustness to distribution shifts and prevents them from offering lumanunndersaandable explanations.
3 VIPBench: A New Benchmark for Personalized Deepfake Detection
To promote the training and evaluation of personalized deepfake detection, build a comprehensive identity-aware benchmark called VIPBench for personalized deepfake detection. The training set of VIPBench progressively fine-tunes MILMM,, advancing from basic facial attribute recognition to fine-grained identity inconsistency detection. Moreover, we introduce a new evaluation dataset for identity-aware deepfake detection, setting that currently lacks sufficient evaluation resources, to assess the effectiveness of different methods. We obtain all facial images from open-sourced datasets, including LAION-Face [87], CrossFaceID [64], and FaceID-6M [63], subjected some preprocessing (details in supplementary materials). The main idea of the dataset construction pipeline described below, while complete details are provided in the supplementary materials.
Facial Attributes Description Dataset We present the Facial Attribute Description Dataset-a multimodal dataset composed of high-resolution facial images paired with rich facial attribute descriptions. The dataset (Dr is collected about 30,000 high-resolution (more than 1024 x 1024) images from LAION-Face to facilitate foundational facial understanding in MLLMs. In addition, an MLLM specialized in facial analysis can act as a captioner to generate descriptive facial attribute information. As illustrated in Figure 3, we obtained detailed attributes (e.g.. face shape, skin condition) via Meg'ii's official API and subsequently refined them by human experts. Figure 3 (d) provides some examples of these facial attributes, which were then transformed into diverse VQA formats (multiple-chotie choice, short/long answer). The process can be formulated as Eq.
DFA {4, VQA (12,,,,,,,.... ..2. (.kk)) (a.....,, a.k) E A i i= 1, I].
where A= A=F(I) denotes the set of extracted facial attributes, F is the API, and is the facial image. The dataset A comprises VQA instances generated from all k-tuples of attributes drawn from A. This dataset provides rich supervision for training an MLLM to understand fundamental facial characteristics. The other details about DAA A are shown in the supplementary material.
Identity Discrimination Dataset By reformulating personalized deepfake detection as a target- face-centric and fine-graines face recognition problem, we constructed the Identity Discrimination Dataset DIp. This dataset, comprising Deeneral and Drip includes facial images and corresponding annotations intended for reasoning about fine- een-ggriied identity discrimination. Specifically, DID comprises facial image pairs: positive (same-identity real-real) and negative (all others). As shown in the 'Data Collection' part of Figure 1, we built facial pairs centered VIP users in Diip while facial pairs in Dje Dgeneral were built using arbitrary identities. Given the high resemblance of forged faces to genuine ones, which is challenge for conventional face recognition, augmented negative samples using SimSwap [5] (face swapping) and Are2Face [49] (entire face synthesis). These negative pairs real-real pairs and aal-versuu-forgen pairs, with these two categories balanced roughly 1:1. As demonstrated in Figure 3 (b), for each image pair, we first generated a
include different-identte
Punps:/www..eeeellppuusocomcm

---

Facial Attribute Description Dataser
P4
Choase the mest accurate option for the serice Croe's fert. Bule
Face Inslyais
Chotce
(a)
Stort Answer
Deseribe
the
Answer
lhe autn raturo) there freecias mcles there foreheed wrinklas cree fast.
Attribute List A
ID Discrimination Dataset
Train
deterntre worther Please pe persst fooe similuraty intly thos Lmu fooe 18/382.
fociat
D
[Pe] The be irages are of the same periee,
Coptiosar
Attrisbes
compura the twe focial imass and decemmie thos the
ctall
the face te $2/100 Juagment isaed du differant fenuel astraberes proviss pou
You noed to sots chip chap-by-ste by-strp
API
cepertan fand very
(bay, hene is the coperiion besed the pros ded Jwaes
[b] The tan jocpes are fffereet peses.
Paze Scare
VIP Eval
Open- Source ce
Commercial API
CelebDi
(Reul)
Constaterezz
Klirg
Krx loo
EFS
Targei Mirlloy
(c)
Ours
(Real)
FS
7
Quit3
Elandfaca
Wififar
(Fukc)
Generator
Statisties
Open Seen-Sourcee
SimSwap
InSwap
PuLID
Jar Type
Nouth Type
Eyebrow Type
Ris
O Hornal
Sharp
Thin D Nemnll Thick Setle
G tech Roise right This Bady Road
Than D Round
Spuare
Seall
Flui
(d)
Light Feur
thart C Netunal
Frectle
Eye FindLines 2 ras O
Crow
bork
2 ras O Ko
e Yex 1o
Figure 3: Illustration of the proposed VIPBench, which includes three personalized datasets, (a) Facial Attributes Description Dataset AF (b) Identity Discrimination Dataset DID, and (c) VIPEval DEvel- (d) Some examples of the facial attributes used in the DFA are also illustrated here, while the full set is available in the supplementary material. The real images shown in (c) are from CelebDF (87) and VIPBench, while the fake ones are generated using multiple models. All images in
(d) are sourced from LAION-Face [87].
facial attribute list using our fine-tuned Qwen2.5VLL7B model [1] G, trained on DFA. Then, we acquired the facial similarity scores by pre-trained facial recognition model f. Finally, we utilized a commercial model (Gemini 2.5 Pro [21] was selected in this paper), denoted as API in Figure (b), to form the training data. Specifically, the API was then strictly required reason identity discrepancies between image pairs based solely on the provided facial attributes G(I,) and G(L;) and similarity scores fl,, 12). ensuring that the analysis was grounded in real features and not influenced by hallucinated content. The above process can be formulated as
Prompt API (Q, f (I,, 12), 4,),, G(4),
(2)
DID = (1, I,. VQA (I,, I,, Prompt) T,, T..
where I; and I, are facial images, Q is the prompt inputted into Gemini, and J denotes a pre- constructed pool of image-name pairs (see supplemental material for details).
VIPEval (User-Specific Evaluation Dataset): We introduce user-specific evaluation dataset called VIPEval for assessing user-specific forgery detection performance. Conventional datasets [39,
GGemini API version in use: 2.5 S--pro-exx-0--111
5

---

53] typically lack a sufficient number of high-resolution real images per individual, along with corresponding variations across different face forgery methods, which makes it challenging to evaluate personalized forgery detection approaches. The dataset includes images from rich sources and various resolutions with diverse generation methods, making more representative of real-worls conditions. To address this limitation, we carefully selected 22 unique identities from the previously described image name pool J, ensuring overlap with the identities in the Dipperal For each identity, 40-60 real images were collected, 20 reserved for testing in this benchmark D DEval and the remainder used to construct DiD. The benchmark includes reserved real images and a comprehensive set of forged counterparts. For each of the 22 test identities, 420 images are generated per method using seven distinct face-swapping (FS) techniques [5. 57, 30, 67, 69, 65, 18]. Additionally, three open-source entire-face synthesis (EFS) methods [49, 28, 24] generate 10 images per real test image by varying random seeds or prompts, resulting in 200 images per identity. Furthermore, four commercial API-based EFS methods [48, 32, 60, 34] produce 20 images per identity. In total, the dataset comprises 80,080 images.
4
VIPGuard: A Multimodal Framework for Personalized Deepfake Detection
4.1 Problem Formulation and Comparison with Prior Works
We consider a personalized deepfake detection scenario in which the detector has access to several authentic images of the target user (c.g., a celebrity, hereafter referred to as a VIP), along with a collection of real facial images from other unrelated individuals. These authentic images serve as prior knowledge for the detector to better recognize and protect the target individual. The objective is to model user- user-specific facial characteristics identify suspicious images and protect the VIP's identity from forgery attacks. For forgeries, such as face-swapped images, we impose a realistic constraint that the source identities used for manipulation (denoted as DDowree are unseen by the detector during training. This assumption is closer to the real-world scenario, where attackers can use arbitrary faces that are not available to the defender. This setting contrasts with identity- aware detection methods [16, 19, 46), which assume that both the target identity (ID"s) and source identities (ID)surres) are known during training-ice all relevant faces are included in the reference set {IDvi, 1 Dsource Dothers) J. Such a closed-world assumption simplifies the problem but rarely realistic in practical applications. In our formulation, the detector only has access to (ID"i, ID Dothers during training, while I D remains unknown. This difference introduces a more challenging yet practical problem setting, emphasizing the need for generalization to unseen source identities and better alignment with real-world deployment scenarios.
4.2 Method Overview
To address the challenge, we propose a framework, VIPGuard, which develops an MLLM capable of identifying suspicious images of a specific user, while providing uuman-understandall explanations based on the user's unique facial attributes. We reformulate personalized deepfake detection as fine- grained face recognition problem centered on the protected target, where forgeries are detected through the MLLM's reasoning over global identity features and detailed facial attributes. To equip the model with this capability, we start from pre-trained MLLM (Qwen-2.5-VL-7) [1]) and progressively fine-tune it through a three-stage process: Face Attribute Learning, Identity Discrimination, and User-Specific Customization. We describe each stage in detail below.
4.3 Three-Stages Training of VIPGuard
Stage 1: Face Attributes Learning It is crucial first to enhance the model's capability to recognize and utilize fine-grained facial attributes, as naive MILMs inherently lack a sufficient understanding of human facial features for effective VIP identity protection. To this end, fine-tune the pre-trained MILM on DFA. which contains a large number of samples for facial attribute recognition (see Section 3). We integrate LoRA [26] modules into the pre-trained MLLM and perform supervised fine-tuning using an autoregressive loss, as defined in Eq. 3:
N
L(
Σ log [po Iei (v)),,
where O denotes the parameters of both the inserted LoRA modules and the original MILLM, N is the length of the output prompt, 2; is the i-th token to be predicted while Izi are the previous tokens,

---

Stuge
Siage
Facial Auribute Learning
Idousity Diserimiaation Leurning
UurrSpociffe Castamization
VIP Token f
Instract
Ieatruct
Iquery
Jim
Instrnct
Fim
Tokenizer
Visual
Tokenizer
Visual Encoder
Tokenizer
Visual Encoder
00
Gross-Attn
Gross-Athh
PD
LLM
LLM
LLM
Traindble
Frozen
Textuol Loken
Image Loken
VIP Token
Figure 4: Illustration of the three stages training the proposed VIPGuard framework.
Ey represents the visual encoder of the MLLM, and / is the input image. After training, the MLLM can understand and then recognize facial attributes.
Stage Identity Discrimination Learning As illustrated Figure 1, although the fake face appears realistic at first glance, subtle discrepancies remain in its detailed facial attributes. Therefore, we reformulate personalized deepfake detection as fine- tne-grained face recognition problem centered on the protected target, where forgeries are detected through the MILM's reasoning over global identity features and detailed facial attributes. this stage, we further fine-tune the MI.IM on DID for fine-grained facial recognition across face pairs-each consisting of an arbitrary query face Iquery and arbitrary input face Iimg- thereby equipping it with foundational ability to identity reason. Degnenn contains large number of positive (same-identity) and negative (including both different-identity and forgery) face pairs, each annotated with VQA-style reasoning questions and answers. These annotations support fine-grained identity discrimination by reasoning over both global and local facial prior. Here, we employ face recognition models [62, 14, 11] to provide face similarity scores, serving as a global facial prior, while the local facial prior-i.e.... knowledge of facial attributes has already been incorporated into the fine-tuned MLLM during Stage 1. Specifically, as shown in Figure 4, the query Iquery and the input image [mg are first inputted into the visual encoder Ey of our MLLM to obtain the visual token sequences fquery and frmg. Pre-trained MLLMs typically lack task-specific optimization for facial recognition, rendering the vanilla visual features fquery and fing suboptimal for facial identity discrimination. Hence, we introduce Cross-Attention (Cross-Attn) module [40] to capture fine-grained differences in visual tokens between distinct facial images, which is formulated as follows
QK
g = softmax
V,
where Q = Squery, K = V= ftng-
(4)
Vdx
We optimize the MILM by an autoregressive loss shown as Eq. 5 below.
L( Σ log (po Eeis fquery fimg. 9)]
(5)
After training, the model can perform fine-grained facial recognition between any two faces, while also providing detailed explanations of their differences.
Stage 3: User-Specific Customization Upon completing Stage 2, the MLLM acquires a basic capability for fine-grained facial identity comparison with reasoning. In Stage 3, further introduce a user-specific customization to enable personalized forgery detection for a given VIP user. To this end, we construct dataset D ID with the same structure as Digneral centered on the VIP's identity, allowing the model identify suspicious images by reasoning based on the VIP user's facial prior. Motivated by the Yo'LLaVA [47], to facilitate lightweight deployment, we incorporate a learnable VIP token trained on D'Ip denoted as μ., which encodes ceentit-speccffe features of the VIP user. During this stage, the parameters of the MILM from Stage 2 are frozen, and only the VIP token is trained on Drip thereby refining the model's 's ability to perceive and distinguish the target VIP user. As illustrated in Figure the learnable VIP token μ,, which is vector of size 32 x d, is employed to
7

---

substitute the visual feature representation fquery of the query image query. Formally, the prediction procedure can be described as follows
QK vdk
g softmax
where Q -e K= fimg. V,
(6)
L(u)
Σ
Me Squery. g)]
Notably, the query images in DID are not used: only the input image [mmg and the corresponding reasoning annotation are fed into the MLLM. After training, the MLLM can perform personalized forgery detection for the VIP user by determining whether the identity of the input image Img belongs to the target individual and providing user-centric explanation. Furthermore, during inference, we can substitute the corresponding VIP token μe for each user, enabling efficient and accurate detection of malicious facial forgeries without requiring any model retraining. This design supports lightweight and scalable deployment.
5 Experiments
In this section, present comprehensive experiments evaluate the effectiveness of our method. For general deepfake detectors, we used models trained on standard deepfake datasets. For ID-aware detectors, assumed access 20-30 real images of each VIP user. Detailed experimental settings are provided in the supplementary material.
For evaluation, report three standard metrics: Area Under the Curve (AUC), Equal Error Rate (EER), and Accuracy (ACC). AUC measures the model's ability to distinguish between positive and negative classes across all thresholds. EER represents the point where the false acceptance rate is equal to the false rejection rate. ACC denotes the proportion of correct predictions. We AUC and EER compare our method following prior work [74, 77]. Moreover, ACC is used to compare with API-based commercial generators (e.g., GPT-4o* [48]).
Evaluation Metrics
Comparison with Deepfake Detection Methods VIPEval We evaluated general deepfake detectors, ID-aware detectors, and our proposed method on VIPEval, reporting results for face swapping (Table 1) detection and entire faces synthesis detection (Table 2). Our method significantly improves the detection of all facial forgery types. General deepfake detectors (e.g.. Effort (73]) excel at face swapping detection but fail detect more realistic forgeries from commercial APIs due to unseen artifacts. ID>aware detectors, unlike general detectors reliant on low-level cues, leverage adentity-relateds semantic consistency for robust detection across diverse generation techniques and data. However, existing ID-aware methods (c.g., DiffID, ICT-Ref) struggle with fully synthesized faces due their reliance on global facial features, a limitation amplified by advancing forgery techniques. Conversely, VIP-Guard captures identity-specifi cues by jointly leveraging global facial representations and local attribute analysis, enabling robust protection of VIP users with few real photos across diverse forgery methods.
Table Evaluation of generalization performance (AUC (%) / EER (%) for face swapping detection on VIPEval.
BleadFaace [57]
Ghost [18]
HifiFaac (65)
InSwap [30]
MobileSwap [69]
SinSSap
Usiliaae [67]
Xception [9]
64.79/38.
59.91
EllicienNee (58)
/ 60.91
43.52/51.32
61.22/42
78.54
89.67/ /6,455
56.55 45.00
36.53 06.33/50.33 55.223
UCH
64,31/ / 38.18
65/92/35.58 33.68
65.62/ 36.82
66.02 /3,332
87.33 / /21.00
63.11/36.82
07:25 /34.32
ProDet [8]
59.84 42,95
53.18 447.73
56.48/ /6.36
73.01 /34.32
56.27/ /5.68
29.34 /5.00
RECCE [3]
56.11/
61.53
19.32
CDBA [42]
59.86 45.25
70.50 36.82
85.43/ 24.09
'00522 36.59
98.75 5.00
71,22/350
73.30 34.77
RepDED [41]
70.34/
78.76 28.18
80.09/ / 27 27.50
71.66 74.36/34,32 34,32
95.76 / 10.45
79.40/27.
82.36 / 25.45
Effort
37.61
9X.23
97.16/8.522
DimhD 181]
03.35/24.47
66.02 338.88
81.25 20.33
66 12.22.1
75.01
72.03 33.91
83:30 24.40
JCT [16]
13.37
86.52 15.51
14.05
17.87
$7.45
$6.57/
VIPGuand
99.45/0.11
97.97
/8.24
99.43/
*GPT4 API version in use: GPT-44-20224085
8

---

Table 2: Evaluation of generalization performance (AUC (%) EER (%)) for entire face synthesis detection on VIPEval.
Method
Con [S]
Open-Suurce Arc3Faace [49]
Pul.ID [24]
CPT-4o [48]
Timeng [10]
Commercial-AP
TongYi (60)
Kling [14]
Xception
42,02/ 54-77
48.86
99.23/44.09
58. 18.13/44 13//4635
57.37 /44.55
34.56/ 65.066.005
44.
BtticienNNet
47.19/50.23
75.35
/45.00
41.15/55.00
UCF [76]
62. 94099
56.62 46.36
$4.16/ 46. 36
59. 19.06/ /42 73
71 71.08/32.76
$2.58/ 22:73
63.31 /40
ProDer [8]
63.68/ 37-95
59.62.1 40,91
67,022 36,82
59.23 00.25/445 /40.91
59.71 /42.73
89.80 18.41
72,53/ 32.73
RECCE
57.79
83.63/
69.00
97.00/
CDFA [42]
77.62/
67.09/ 79.09
67.93/
73.46/
71.98 344,09
90.52./ 17.95
77.47/ 30.00
ReptiFD [41]
83.52/24.35
61.67 44.59
74.65 1..65::32..7 1.22.7
73.62/ 00022/522222
62.78 440.91
93.80/ 14.10
60. /43.41
Edfon (75)
38.68/47.35
57.03 /4.89
56.31 /44.89
49. 30
63.60 /40.34
82.93 23.86
56.84 /4.889
DiEID [82]
75.85/32.01
78.47 (88986
70.36 7 235.22
45.26 55.997
04.29/339.500
$4.66 /23.58
69.51 35.56
[16]
63.15/39984
70.27 32.84
72.36 331.99
65.36 /35.73
/3.44
50.05 /4..0//45.00
VIPGiaard
99.69/0.45
98.05 4.80
98.96/1.90
89.03/ 16.14
97.04/5.36
99.76/ 0.23
/9.27/1.25
Comparison with Other LLM-based Methods on VIPEval This experiment evaluated the detec- tion capabilities of Multimodal Large Language Models (MLLMs) on VIPEval by comparing various MLLM-based methods, including FFAA [29], an MLLM specialized for face forgery detection, alongside naive MLLMs. Detection performance was measured using Accuracy (ACC) due to the API's binary (Real/Fake) output. As presented in Table 3, our method consistently outperforms other approaches across all forged image types, demonstrating the effectiveness of ddentity-speciffe semantic detection.
Table 3: Comparison (ACC (%)) of our method and other MILMs on the VIPEval.
Method
BlendFace
HifiFace
MobileSwap
UniFace
Consisten:ID
Arc2Face
GPT-46-2024408-066EM
71.85
84,36
94.35
70.72
46.70
50.90
Gemini-2. 1omini-2.5-pro--xp-- 0--- [21]
80.87
82.68
91.75
83.58
69.85
79.98
Owen2.5VL. [I]
50.05
49,86
49.99
50.02
49.51
49.69
LLaMA3.2Vsson 1IB [22]
44.76
43.93
45.71
54.09
38.56
43.61
FFAA [29]
88.60
89.95
91.36
89.30
63.53
59.89
VIPGuard
95.51
95.82
96.71
95.88
95.91
89.71
One-shot Performance on other Deepfake detection datasets This experiment evaluates our method against several existing ID-aware approaches established benchmark datasets. The evaluation specifically employed the challenging CelebDF [39] dataset and a CelebDF-related subset of DF40 [74]. Owing limited real sample diversity in these datasets, our method was evaluated in one-shot setting, using a single real image per user. To mitigate similarity from shared video sources, reference and test images were sampled from different videos. As shown in Table 4, method still achieves competitive performance compared to other approaches. Notably, due to limited real image availability, these results were obtained using VIP-Guard at Stage 2, omitting Stage 3. The results also verify the effectiveness of Stage 2 of our method.
Table 4: Evaluation of frame-level performance (AUC (%) EER (%) in CelebDF [39] and DF40 [74] under one-shot setting. For each identity, only a single real image is available.
Method
CelebDF
DF40
BlendFaaee
EAS
FaceDameer
ESGAN
SimSwap
UniFaae
InSwap
Dit--r>
6666/21.77 /21.79 21.79
76.83/ 28.00 99
84.79 /4
81.04/27.02
82.88 23.77
64.87/ 38
84.04 22.53
50 62 /43.19
ICT--Ref
$1.86/2..18 26.18
76.47/ 30.63
83.33 24.93 24.93
91.57 16.33
73,62 32.95
82.41 /25.22
80,34 27.07
69.91 33.58
VIPGuard
87.96 20.84
81.87 23.77 26.23
89.74/ 19.03
86. /2.66
86.44 20.87
W3.05 /24.92
36,70/ 22.57
70997 /44.15
Visual Examples of Our Model Explanations Figure presents two representative examples illustrating VIPGuard's ability to detect anomalous local facial attributes in deepfake images generated by entire face synthesis (EFS, left) and face swapping (FS, right) techniques. Despite the high global facial similarity between the real and fake images, VIPGuard accurately identifies subtle but critical differences in localized regions such as the eyes, lips, and skin textures. For instance, discrepancies in eye size, lip thickness, and specific skin features (e.g., glabella wrinkles, eye pouches, moles, and crow's feet) are effectively captured by our model. These examples highlight two key strengths of VIPGuard: (1) its ability to detect forgery-induced induced anomalies in local facial regions where current

---

generation methods often fail, and (2) its capacity to reason over these attributes for reliable identity verification, even when global facial appearance is highly similar.
Real
Here the compatiion based the provided images and attniuutes:
Real
Deay,
based procided images artr butes:
Image 1 has normal-sized sixed eyes compared to Image 2's bigger eyes
There are noticeable differences; Image 1 has
normal-sized syea compared to image 2's
Image 1 displays thinner lips compared
larger eyes.
While both show smiles, the lip thickness differs; Image has thinner compared the fuller in
nifferences euist skin (natural
wheat) presence specific
2.
wrinkles like glabella wrinktas eye pouches, which are noted only in the second Image.
Differences exist skin details; Image mole speciffic wrinkle patterns (forehead, crow's wtss feet), which are absent in Image 2.
Fake
Fake
Overall Conclusion: Despita some shared butes relatively high similertt (83) (83/200), the distinct differences klentified
Overall Conclusione Des some shared astributes and a high similaity score (87/100), the distinct differences key facial regians such the eyes, mouth, and skin features strongly indicate
skin regions confirm that images different Individuals.
depict differen
Individuals.
Figure 5: Visual illustration of the analysis of VIP-Guard detecting anomalous local facial attributes for EFS (left) and FS (right). The two real images are sourced from LAION-Face [87], while the fake images in the left and right subfigures were generated by GPT-4o [48] and HifiFace [65], respectively,
5.1 Effectiveness in Annotation- FFee Scenarios
To validate VIPGuard's practicality, we tested its Stage 3 performance when trained using only VIP images, foregoing textual amnotations. As shown in Table 5, the model's 's performance remains strong, with the average AUC decreasing only slightly from 98. 98% to 95.86%. This robustness stems directly from the design of our framework. Stage 2 is responsible for internalizing a vast repository of discriminative knowledge by learning to distinguish between arbitrary image pairs with the aid of textual supervision. This foundational pre-training is so effective that the User-specific fine-tuning in Stage 3 requires only visual information to achieve a high degree of accuracy. Consequently, this experiment validates VIPGuard as a highly practical and readily deployable framework for real-world applications, including annotation free scenarios. The model's ability to achieve such high efficacy under these constraints is direct testament to the comprehensive and foundational knowledge provided by Stage 2.
Table 5: Performance (AUC (%)) of VIPGuard under different training configurations in Stage 3. Images- + Annotation denotes training with both images and textual descriptions, while Only Images uses visual inputs only.
Variant
BlendFace
InSwap
Arc2Face
Pul.ID
Average
Only Images
98.45
92.91 99.43
94.35 98.05
97.72 98.96
y5 86
Images + Annotatioo
99.48
98.98
Conclusion
This paper proposes VIPGuard, addressing a critical gap in deepfake detection by leveraging known facial identities to enable personalized, accurate, and explainable detection. Unlike traditional detectors that mainly rely on low-level visual artifacts or ceneral-purposs MLLMs lacking identity awareness, VIPGuard integrates fine-grained attribute learning, identity-level discriminative training, and user-specific customization through a unified multimodal framework. Combined with our newly proposed VIPBench, which enables rigorous evaluation across diverse and advanced forgery types, our approach demonstrates clear superiority in both detection performance and explainability, offering robust and scalable solution for safeguarding high-risk individuals against identity- -based deepfakes.
Content Structure of the Appendix. Due to page constraints, we include additional analyses and experiments in the Appendix, containing comprehensive ablation studies (Appendix D.2), robustness evaluation (Appendix D.4), adaptive choice of VIP Token (Appendix D.3), more visual examples of model explanations (Appendix D.5), details of dataset construction (Appendix E), an ethical statement (Appendix A), and limitations and future work (Appendix C). For further details, please refer to the Appendix.
10

---

Acknowledgments and Disclosure of Funding
Dr Bin Li was supported in part by NSFC and in part by Shenzhen R&D Program (Grant JCYJ2025060418 19000041111110111 XYSPG20241221174322000 Dr Weixiang Li was supported in part by NSFC (Grant 62202310, 62572328) and in part by Guangdong Basic and Applied Basic Research Foundation (Grant 2025A1515010292),
References
[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 250.:22502.19933, 2025.
[2] Matyas Bohacek and Hany Farid. Protecting world leaders against deep fakes using fa- cial, gestural, and vocal mannerisms. Proceedings of the National Academy of Sciences, 119(48))e211033511 19, 2022.
[3] Junyi Cao, Chao Ma, Taiping Yao, Shen Chen, Shouhong Ding, and Xiaokang Yang. End- to-end wwwpssrrcctionccassfcctttt learning for face forgery detection. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 41 4113-4122, 2022.
[4] Liang Chen, Yong Zhang, Yibing Song. Lingqiao Liu, and Jue Wang. Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18710-18719, 2022.
[5] Renwang Chen, Xuanhong Chen, Bingbing Ni, and Yanhao Ge. Simswap: An efficient framework for high fidelity face swapping. In Proceedings of the 28th ACM international conference on multimedia,, pages 2003-2011, 2020.
[6] Ruoxin Chen, Junwei Xi, Zhiyuan Yan, Ke-Yue Zhang, Shuang Wu, Jingyi Xie, Xu Chen, Lei Xu, Isabel Guan, Taiping Yao, et al. Dual data alignment makes ai-generated image detector easier generalizable. arXiv preprint arXiv:2505. 14t:29 2025.
[7] Yize Chen, Zhiyuan Yan, Siwei Lyu, and Baoyuan Wu. xd-dfd: A framework for explainable and extendable deepfake detection. arXiv preprint
2024.
[8] Jikang Cheng, Zhiyuan Yan, Ying Zhang, Yuhao Luo, Zhongyuan Wang, and Chen Li. Can leave deepfake data behind in training deepfake detector? arXiv preprint arXiv:2408. /7052, 2024.
[9] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Pro- ceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1251-1258. 2017.
[10] Davide Cozzolino, Andreas Rössler, Justus Thies, Matthias Nießner, and Luisa Verdoliva. Id- reveal: Identity-aware deepfake video detection. In Proceedings of the IEEF/CVF international conference on computer vision, pages 15108-1511 15117, 2021.
[11] Jun Dan, Yang Liu, Haoyu Xie, Jiankang Deng, Haoran Xie, Xuansong Xie, and Baigui Sun. Transface: Calibrating transformer training for face recognition from data-centric perspective. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages October 2023.
[12] Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K Jain. On the detection of digital face manipulation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2020.
[13] DeepFacelab. https: htt.//g..ttttttt////ege/.g... com//pperco/DDepaccll....
[14] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 4690-4699, 2019.

---

[15] Shichao Dong, Jin Wang. Renhe Ji, Jiajun Liang, Haoqiang Fan, and Zheng Ge. Implicit identity leakage: The stumbling block improving deepfake detection generalization. In Proceedings of the IEEF/CVE Conference Computer Vision and Patlern Recognition, pages -4044, 2023.
[16] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Ting Zhang. Weiming Zhang, Nenghai Yu, Dong Chen, Fang Wen, and Baining Guo. Protecting celebrities from deepfake with identity consistency transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 1978, June 2022.
[17] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Dong Chen, Fang Wen, and Baining Guo. Identity-driven deepfake detection. arXiv preprint 0rXiv:20/2.039300 2020.
[18] FaceFusion 3.0.
2025.
[19] Mingqi Fang, Lingyun Yu, Hongtao Xie, Qingfeng Tan, Zhiyuan Tan, Amir Hussain, Zezheng Wang, Jiahong Li, and Zhihong Tian. Stidnet: Identity-awan face forgery detection with spatiotemporal knowledge distillation. IEEE Transactions on Computational Social Systems, 11(4):3354-336, 2024.
[20] Niki M Foteinopoulou, Enjie Ghorbel, and Djamila Aouada. A hitchhiker's guide to fine- grained face forgery detection using common sense reasoning. Advances in Neural Information Processing Systems, 37:2943-2976 2025.
[21] Google DeepMind. Gemini 2.5: Our most intelligent ai model. https://blog.googlee updaaes--mrcch-aeee....
technology/ wowllogggggoooeeeeeeee google-deepmindd gemini-model-thiieeee 2025.
[22] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint M4xXx:2407.217880 2024.
[23] Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Li. Towards fast, accurate and stable 3d dense face alignment. In Computer Vision ECCV 2020, pages 152-168, 2020.
[24] Zinan Guo, Yanze Wu, Chen Zhuowei, Peng Zhang, Qian He, et al. Pulid: Pure and lightning id customization via contrastive alignment. Advances in neural information processing systems, 37:36777-36804, 2024.
[25] Xinan He, Yue Zhou, Bing Fan, Bin Li, Guopu Zhu, and Feng Ding. Viforgery face triad: Detection, localization and attribution via multimodal large language models. arXiv preprint MXXi::5503.06122 2025.
[26] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022.
[27] Baojin Huang, Zhongyuan Wang, Jifan Yang. Jiaxin Ai, Qin Zou, Qian Wang, and Dengpan Ye. Implicit identity driven deepfake face swapping detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4490-4499, 2023.
[28] Jiehui Huang, Xiao Dong, Wenhui Song. Zheng Chong, Zhenchao Tang, Jun Zhou, Yuhao Cheng, Long Chen, Hanhui Li, Yiqiang Yan, et al. Consistentid: Portrait generation with multimodal fine- fine-grained identity preserving. arXiv preprint arXiv:240..16771, 2024.
[29] Zhengchao Huang, Bin Xia, Zicheng Lin, Zhun Mou, and Wenming Yang. Ffaa: Multimodal large language model based explainable open-worke face forgery analysis assistant. arXiv preprint arXiv: 10072 2024.
[30] inswapper. https: setps://giis //:::://gtttt http:/.c./.aooaaaag/innnaaetaa
12

---

[31] Shan Jia, Reilin Lyu, Kangran Zhao, Yize Chen, Zhiyuan Yan, Yan Ju, Chuanbo Hu, Xin Li, Baoyuan Wu, and Siwei Lyu. Can chatgpt detect deepfakes? study of using multimodal large language models for media forensics. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 4324 4333, 2024.
[32]
JiMeng-Al.
org/ 2025.
[33] Hengrui Kang, Siwei Wen, Zichen Wen, Junyan Ye, Weijia Li, Peilin Feng, Baichuan Zhou, Bin Wang, Dahua Lin, Linfeng Zhang, et al. Legion: Learning to ground and explain for synthetic image detection. arXiv preprint arXhv:2503. 2025.
[34] Kling AI. https: ///www klingainiiii www.ingaa...omgg//////// wot.:/ooao.g.//000000
[35] Jiawei Li, Fanrui Zhang, Jiaying Zhu, Esther Sun, Qiang Zhang, and Zheng-Jun Zha. Forgerygpt: Multimodal large language model for explainable image forgery detection and localization. arXiv preprint arXiv:2410. 102388 2024.
[36] Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. Face x-ray for more general face forgery detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[37] Yixuan Li, Xuelin Liu, Xiaoyang Wang. Shiqi Wang, and Weisi Lin. Fakebench: Uncover the achilles' heels of fake images with large multimodal models. arXiv preprint urXiv:2404.13300 2024.
[38] Yuezun Li and Siwei Lyu. Exposing deepfake videos by detecting face warping artifacts. arXiv preprint 1XX1::8111000650 2018.
[39] Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-df: A new dataset for deepfake forensics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[40] Hezheng Lin, Xing Cheng, Xiangyu Wu, and Dong Shen. Cat: Cross attention in vision transformer. In 2022 IEEE international conference on multimedia and expo (ICME), pages 1-6. IEEE, 2022.
[41] Kaiqing Lin, Yuzhen Lin, Weixiang Li, Taiping Yao, and Bin Li. Standing on the shoulders of giants: Reprogramming visual-language model for general deepfake detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 5262-5270, 2025.
[42] Yuzhen Lin, Wentang Song, Bin Li, Yuezun Li, Jiangqun Ni, Han Chen, and Qiushi Li. Fake it till you make it: Curricular dynamic forgery augmentations towards general deepfake detection. arXiv preprint arXiv:2409. 100099..44. 14444, 2024.
[43] Yuchen Luo, Yong Zhang, Junchi Yan, and Wei Liu. Generalizing face forgery detection with high-frequency features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[44] Long Ma. Zhiyuan Yan, Yize Chen, Jin Xu, Qinglang Guo, Hu Huang, Yong Liao, and Hui Lin. From specificity to generality: Revisiting generalizable artifacts in detecting face deepfakes. arXiv preprint arXiv:2504.048277 2025.
[45] Meta AI. Introducing llama 3.2: Vision, edge, and mobile models. https: //ai meta. com/ blog/1lamn 3- 2- 2--conectt2200 vision-eeee wwwssiionnnee--eeee wetee--oobll-deveieeeee 2024.
[46] Liyue Ming, Peisong He, Haoliang Li, Shiqi Wang, and Xinghao Jiang, Critical contour prior-guided graph learning with pose calibration for identity-aware deepfake detection. IEEE Transactions on Multimedia, 2025.
[47] Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. Yo" do'llava: Your personalized language and vision assistant. arXiv preprint 2rXiv:2406.09400 2024.
[48] OpenAl. GPT-4o. http:////pppaaa..oomo//eeeeeella -at---pt------ 2025.
13

---

[49] Foivos Paraperas Papantoniou, Alexandros Lattas, Stylianos Moschoglou, Jiankang Deng, Bernhard Kainz, and Stefanos Zafeiriou. Arc2face: foundation model for id-consistent human faces. In European Conference on Computer Vision, pages 241-261. Springer, 2024.
[50] Yuyang Qian, Guojun Yin, Lu Sheng. Zixuan Chen, and Jing Shao. Thinking in frequency: Face forgery detection by mining frequency-aware clues. In European Conference Computer Vision, pages 86-103. Springer, 2020.
[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[52]
Roop.
[53] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the
IEEE/CVF Conference International Conference Computer Vision, pages 1-11, 2019.
[54] Dongyao Shen, Youjian Zhao, and Chengbin Quan. Identity-referenced deepfake detection with contrastive learning. In Proceedings of the 2022 ACM Workshop Information Hiding and Multimedia Security, pages 27-32, 2022.
[55] Yichen Shi, Yuhao Gao, Yingxin Lai, Hongyang Wang, Jun Feng, Lei He, Jun Wan, Changsheng Chen, Zitong Yu, and Xiaochun Cao. Shield: An evaluation benchmark for face spoofing and forgery detection with multimodal large language models. arXiv preprint arXiv:2402.06 04178, 2024.
[56] Kaede Shiohara and Toshihiko Yamasaki. Detecting deepfakes with self-blended images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18720-1877 18729, 2022.
[57] Kaede Shiohara, Xingchao Yang, and Takafumi Taketomi. Blendface: Re-designing identity encoders for face-swapping. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 7634-7644, 2023.
[58] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114. PMLR, 2019.
[59] Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait givveggenrrating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402. 17485, 2024.
[60] Tong Yinn. https: tttpss:/ttnyy// //tng//ongggg////// al aliyun.cc.m.om www/wanxiang///.omomm.m.mo 2025.
[61] Chengrui Wang and Weihong Deng. Representative forgery mining for fake face detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 14923-14932, 2021.
[62] Hao Wang, Yitong Wang. Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5265-5274, 2018.
[63] Shuhe Wang. Xiaoya Li, Jiwei Li, Guoyin Wang, Xiaofei Sun, Bob Zhu, Han Qiu, Mo Yu, Shengjie Shen, Tianwei Zhang, et al. Faceid-6m: A large-scale, open-source faceid customiza- tion dataset. arXiv preprint arXiv:2503.07091 2025.
[64] Shuhe Wang. Xiaoya Li, Xiaofei Sun, Guoyin Wang, Tianwei Zhang, Jiwei Li, and Eduard Hovy. Turn that frown upside down: Faceid customization via cross-training data. arXiv preprint A50i 15407, 2025.
[65] Yuhan Wang, Xu Chen, Junwei Zhu, Wenqing Chu, Ying Tai, Chengjie Wang, Jilin Li, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Hififace: 3d shape and semantic prior guided high fidelity face swapping. arXiv preprint arXiv:2106 09960, 2021.
14

---

[66] Siwei Wen, Junyan Ye, Peilin Feng. Hengrui Kang, Zichen Wen, Yize Chen, Jiang Wu, Wenjun Wu, Conghui He, and Weijia Li. Spot the fake: Large multimodal model-based synthetic image detection with artifact explanation. arXiv preprint arXiv:2503. /4905, 2025.
[67] Chao Xu, Jiangning Zhang, Yue Han, Guanzhong Tian, Xianfang Zeng, Ying Tai, Yabiao Wang, Chengjie Wang, and Yong Liu. Designing one unified framework for high-fidelity face reenactment and swapping. In European conference on computer vision, pages 54-71. Springer, 2022.
[68] Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: Lifelike audio-driven talking faces generated in real time. arXiv preprint arXiv:2404. 10667, 2024.
[69] Zhiliang Xu, Zhibin Hong. Changxing Ding, Zhen Zhu, Junyu Han, Jingtuo Liu, and Errui Ding. Mobilefaceswap: A lightweight framework for video face swapping. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2973-2981, 2022.
[70] Zhipei Xu, Xuanyu Zhang, Runyi Li, Zecheng Tang, Qing Huang, and Jian Zhang. Fakeshield: Explainable image forgery detection and localization via multi- motil large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[71] Zhipei Xu, Xuanyu Zhang, Xing Zhou, and Jian Zhang. Avatarshield: Visual reinforcement learning for human-centric video forgery detection. arXiv preprint 1rXXi:2505..5173 2025.
[72] Zhiyuan Yan, Yuhao Luo, Siwei Lyu, Qingshan Liu, and Baoyuan Wu. Transcending forgery specificity with latent space augmentation for generalizable deepfake detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8984-8994, 2024.
[73] Zhiyuan Yan, Jiangming Wang, Zhendong Wang, Peng Jin, Ku-Yue Zhang, Shen Chen, Taiping Yao, Shouhong Ding, Baoyuan Wu, and Li Yuan. Effort: Efficient orthogonal modeling for generalizable ai-generated image detection. arXiv preprint arXiv:2411.15633 2024.
[74] Zhiyuan Yan, Taiping Yao, Shen Chen, Yandan Zhao, Xinghe Fu, Junwei Zhu, Donghao Luo, Chengjie Wang, Shouhong Ding, Yunsheng Wu, et al. DF40: Toward next-generation deepfake detection. arXiv preprint arXiv:2406. /3495, 2024.
[75] Zhiyuan Yan, Junyan Ye, Weijia Li, Zilong Huang, Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, and Li Yuan. Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation. arXiv preprint arXiv:2504.0277 2025.
[76] Zhiyuan Yan, Yong Zhang, Yanbo Fan, and Baoyuan Wu. Ucf: Uncovering common features for generalizable deepfake detection. arXiv preprint arXiv:2304.13945 2023.
[77] Zhiyuan Yan, Yong Zhang, Xinhang Yuan, Siwei Lyu, and Baoyuan Wu. Deepfakehench:
A comprehensive benchmark of deepfake detection. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 4534-4565. Curran Associates, Inc., 2023.
[78] Tianyun Yang, Juan Cao. Qiang Sheng, Lei Li, Jiaqi Ji, Xirong Li, and Sheng Tang. Learning to disentangle gan fingerprint for fake image attribution. arXiv preprint 2XXi:2106.087444 2021.
[79] Zheng Yang, Ruoxin Chen, Zhiyuan Yan, YueYue Zhang, Xinghe Fu, Shuang Wu, Xiujun Shu, Taiping Yao, Shouhong Ding, and Xi Li. All patches matter, more patches better: Enhance ai- ni-generated image detection via panoptic patch learning. arXiv preprint orXiv:250..0136 2025.
[80] Junyan Ye, Baichuan Zhou, Zilong Huang, Junan Zhang, Tianyi Bai, Hengrui Kang, Jun He, Honglin Lin, Zihao Wang, Tong Wu, al. Loki: A comprehensive synthetic data detection benchmark using large multimodal models. arXiv preprint arXiv:2410.09733. 2024.
[81] Chuer Yu, Xuhong Zhang, Yuxuan Duan, Senbo Yan, Zonghui Wang, Yang Xiang, Shouling Ji, and Wenzhi Chen. Diff-id: An explainable identity difference quantification framework for deepfake detection. IEEE Transactions on Dependable and Secure Computing, 21 50059 5045, 2024.
15

---

[82] Zeqin Yu, Bin Li, Yuzhen Lin, Jinhua Zeng, and Jishen Zeng. Learning to locate the text forgery smartphone screenshots. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[83] Zeqin Yu, Jiangqun Ni, Yuzhen Lin, Haoyi Deng, and Bin Li. Diffforensics: Leveraging diffusion prior to image forgery detection and localization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12765-12774, 2024.
[84] Zeqin Yu, Jiangqun Ni, Jian Zhang, Haoyi Deng, and Yuzhen Lin. Reinforced multi-teacher knowledge distillation for efficient general image forgery detection and localization. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 995-1003, 2025.
[85] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. JEEE SPL, 23(10):1499-13 1503, 2016.
[86] Yue Zhang, Ben Colman, Xiao Guo, Ali Shahriyari, and Gaurav Bharaj Common sense reasoning for deepfake detection, In European Conference on Computer Vision, pages 415- Springer, 2024.
[87] Yinglin Zheng, Hao Yang. Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learning in a visual- linguistic manner. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18697-18709, 2022.
[88] Yue Zhou, Bing Fan, Pradeep K. Atrey, and Feng Ding. Exposing deepfakes using dual-channel network with multi-axis attention and frequency analysis. In Proceedings of the 2023 ACM Workshop on Information Hiding and Multimedia Security, pages 169-174, 2023.
[89] Yue Zhou, Xinan He, KaiQing Lin, Bin Fan, Feng Ding, and Bin Li. Breaking latent prior bias in detectors for generalizable aige image detection. arXiv preprint arXiv:2506.0083 00874, 2025.
16

---

Appendix
A
Ethics Statement
This study involves facial forgery detection and may raise privacy and ethical concerns related to the use of human facial data. We affirm that all source images used in our datasets are obtained from publicly available and legally compliant open-source datasets [87, 63, 64], which are intended for academic research purposes. Our research is dedicated advancing facial privacy protection and deepfake detection. The developed models and datasets are intended solely for ethical, academic use and will not be released for commercial or malicious purposes. This study follows the ethical guidelines provided by NeurIPS Code of Ethics, and does not involve any personally identifiable information collected directly by the authors. Our VIPBench dataset is released under the Creative Commons Crribution-NonCommeee
(CC BY-NC) license (more details can be seen in https: //creativecommont org/ hrg/11z sicessssbbyy www.byy...0.....oo Furthermore, access to the dataset will be managed through request form (hosted on HuggingFace) to monitor and control its usage. All interested parties are required to complete the form, and each request will be manually reviewed to help prevent potential misuse.
B Boarder Impact
The research presented in this paper introduces VIPGuard, multimodal framework for personalized deepfake detection, and VIPBench, a benchmark dataset for evaluating identity-aware detectors. This work addresses a critical and growing societal challenge: the malicious misuse of Al-generated media. particularly in the form of deepfakes targeting specific individuals such as celebrities. Our method promotes positive societal impact in several key areas:
Mitigating Disinformation and Identity Theft: Deepfakes pose significant threat by enabling the creation of highly realistic fake media, which can be weaponized to harm repu- tations, manipulate public perception, or conduct fraud. VIPGuard offers a targeted defense mechanism. enhancing the security and privacy of individuals by providing personalized and explainable detection tools.
Promoting Safer AI Ecosystems:
By releasing VIPBench, we aim to catalyze further research in robust, personalized detection. Public benchmarks encourage accountability, replication, and the development of defenses that are grounded in real-world threats. Fur- thermore, we will implement a review-based distribution process for the dataset to ensure its lawful and responsible use.
C Limitation and Future Works
VIPGuard is designed to protect the identities of target users against deepfakes. While the method demonstrates strong generalization capabilities, it does not yet fully exploit additional modalities such as audio, 3D facial models, temporal consistency, and other complementary cues. For instance, these modalities could significantly improve the detection of audio-driven manipulation techniques, which rely heavily on temporal dynamics and inter-frame consistency. Beyond facial images, incorporating such prior information could further enhance detection performance. As part of future work, we plan integrate these complementary modalities to strengthen VIPGuard's 's effectiiveness. recognize two key limitations observed during evaluation: (1) Significant variations in head pitch and yaw angles, such noticeable rotations exceeding 50% can affect facial features and identity information. However, this limitation is shared by most face- based systems, including standard face recognition models. (2) Significant age differences between training and test data can lead to discrepancies in performance. For instance, training may rely on younger facial images while testing may involve older ones. This issue, arising from identity shifts across age, can alleviated by expanding and diversifying the dataset. Inspired by these observations, we plan to address these potential failure cases in future work by incorporating 3D facial information, learning identity-related temporal cues from vidcos, and exploring other complementary sources of information.
addition, we
17

---

Table 6: An ablation study the effectiveness (AUC (%)) of different components in VIPGuard
Variant
Description
BlendFace
InSwap
Arc2Face
PuLID
Baseline
Qwen-2.5-VI-77
51.00
49.79
49.85
50.30
Stage 3
Only performing User-Specific Cus-
71.96
59.10
62.12
68.18
tomization
Stage 3,
Enhancing facial understanding and then performing Eser-Specific Cus- tomization
96.50
87.38
88.95
94.11
Stage 1,2
Enhancing facial understanding and general identity discrimination be- tween arbitrary face pairs
90.82
73.44
77.25
71.08
Stage 2,
Enhancing facial understanding. general identity discrimination, and User-Specific Customization
99.48
96.40
98.05
98.96
D Experiments
D.1 Implementation Details
This paper adopted the pre-trained Qwen-2.5-VL-77 model [1] as the backbone, adhering to its default pre-pree pre-processing settings. Input images were resized to 448 x 448 when larger than this size. The model was optimized using the Adam optimizer with cosine learning rate decay schedule, starting from an initial learning rate 3e-5. To accommodate GPU memory limitations, the equivalent batch size was maintained at 72 for both Stage I and Stage 2 by applying gradient accumulation. In Stage 3, the effective batch size was reduced performed using
and the initial learning rate was set All training was computation within the open-source Swift framework. The model was trained for epochs in Stage I, and for epoch each in Stage and Stage 3. For the evaluation in the main paper, we obtained the AUC and EER metrics by computing the normalized probability of output logits in MLLM's prediction head for the words 'Yes' and 'No'. During evaluation, VIPGuard was only required to output the final prediction (i.e., 'Yes' or 'No') without providing any explanatory content.
D.2 Ablation Study
In this section, we conduct a series of ablation studies to comprehensively evaluate the effectiveness of different components and design choices our proposed VIPGuard framework. The AUC (%) metric was selected to exhibit the performance in detecting multiple facial forgeries.
Exploration for the Composition in VIPGuard In this paper, we propose three-stage learning framework VIPGuard to respectively improve MLLM's face understanding. fine- aineggrained discrim- ination between arbitrary face pairs, and fine-grat mne-grained discrimination for VIPs. To evaluate the contribution of each stage, we conduct ablation study using different combinations of the three stages, as demonstrated in Table 6. The base model, Qwen-2.5-VL--77 demonstrates near-random performance across all datasets, indicating its limited native capability in detecting forged faces. Adding only Stage 3, which introduces user-specific tuning, moderately improves performance (e.g., +20.96 on BlendFace and +17.88 PuLID), but results remain suboptimal due the model's lack of foundational face understanding. Introducing Stage I alongside Stage 3 yields substantial gains across all benchmarks, confirming that basic face comprehension is essential for effective user- specific customization. Combining Stages and 2, without user-specific tuning, enhances general discrimination (c.g.. +45.50 on BlendFace compared to the base model), but still inferior the full configuration. Notably. the detection for VIP users was conducted under one-shot setting, where the MLLM compares a single reference image of the target user with the queried image. Without user-specific customization, the MLLM cannot accurately capture the nuanced identity traits of the
18

---

100
Sample Num
90
AUC (%)
O 5
85
- 10
80
-O- 20
BlendFace
InSwap
Arc2Face
PuLID
Figure 6: Ablation study on VIPGuard performance (AUC (%) with varying numbers of available authentic images of target users. 'A11' refers to using all available authentic images, typically ranging from 20 to 40 images for each identity in VIPBench.
VIP user, resulting in limited detection performance. The complete VIPGuard pipeline (Stages 2, and 3) achieves the highest performance across all datasets, with detection rates exceeding 96% in most cases (e.g., 99.48 on BlendFace, 98.96 on PuLID), highlighting the critical synergy among the three stages. These results demonstrate that VIPGuard's effectiveness hinges the sequential integration of facial understanding, general identity discrimination, and user-specific adaptation. Each component contributes uniquely, and omitting any stage leads measurable performance degradation.
Impact of the Number of Available Authentic Images for Target Users (VIPs) While abundant negative samples can be obtained from real images of different identities, the number of accessible authentic images for target user limits the positive samples available for Stage training, thereby varied the
directly impacting the model's user- spersspecffic detection capability. In this experiment, quantity of available authentic images for a specific user and explored its effect on VIPGuard's performance (Figure 6). When only authentic image was accessible, the scarcity of positive samples precluded Stage training: consequently, the Stage trained model was used directly for detection without User-Specific Customization. The results indicate that VIPGuard's s detection capability improves as more authentic images of the specific user become available. Compared to directly using the Stage model, employing Stage 3 with even just three authentic images (positive samples) yielded a substantial improvement in detection performance. Furthermore, as the number of accessible user samples increased, detection performance across various forgery types showed consistent improvement.
Impact of the Size of VIP tokens in Stage3 In Stage 3, the VIP Token is of size n x d, where denotes the number of VIP Tokens and d represents the feature dimensionality. We investigated the impact of varying n on performance. As shown in Figure 7, increasing n up to 32 improved performance, reaching an optimum n 32. However, further increasing n 64 128 resulted in a performance decline, potentially due to overfitting on the limited training samples. Therefore, we finally set to 32.
Performance with Different Backbones To further validate the universality of VIPGuard, we evaluated its performance using different multimodal large language model (MLLM) backbones. As shown in Table 7, we replaced the backbone with LLaMA-3.2 Vision- [45], Qwen-2.5-VL. [1], and Qwen-2.5-VL. 5-VL-BB [1], respectively. Across all configurations, VIPGuard consistently achieved strong and stable performance improvements
the baseline model, demonstrating its adaptability to diverse MILMM architectures and its great detection performance across different generators.
Impact of the Face Recognition Models utilized to provide the global facial information and output similarity score. To determine which reco-ninonn backbone most suitable for computing similarity VIPGuard, we conducted comparative evaluation of several popular models, including CosFace [62], ArcFace [14], and
the proposed method, the face recognition model
19

---

100
0
8
98
AUC (%)
32
64
128
94
BlendFace
InSwap
Arc2Face
PuLID
Figure 7: Ablation study on VIPGuard performance (AUC (%) with varying the size n of the VIP token.
Table 7: Performance Comparison (AUC (%) of VIPGuard with Different MLLMs.
Method
BlendFace
InSwap
Arc2Face
PuLID
Average
Baseline
Qwen-2.5-VL-7B (Baseline)
51.00
49.79
49.85
50.30
50.24
VIPGuard
LLaMA-3.2 3.2i--iii
96.27
88.97
99.00
99.52
95.94
Qwen-2.5-VL--3 Qwen-2.5-VL-7B
90.71
87.85
87.82
97.94
91.08
99.48
96.40
98.05
98.96
98.23
TransFace [11]. Each model was integrated into the VIPGuard framework to perform identity similarity estimation, and the performance was measured by AUC (%) across four representative forgery datasets: BlendFace [57], InSwap [30], Arc2Face [49], and PuL.D [24]. As summarized in Table 8, TransFace consistently achieved the highest accuracy among the tested models, demonstrating its superior capability capturing discriminative and fine mne-grained identity representations. Therefore, we adopt TransFace as the default face-recognition model in our experiments.
Table 8: Performance (AUC (%)) of VIPGuard using different face-recognition models as similarity components. TransFace exhibits the best overall performance and thus used by default in framework.
Face Model
BlendFace
InSwap
Arc2Face
PuLID
Average
CosFace
95.12
83.32
83.31
91.74
88.37
ArcFace
98.14
83.17
94.41
97.88
93.40
TransFace
99.48
96.40
98.05
98.96
98.23
D.3 Adaptive VIP Token Selection
In practical scenarios, it is common to encounter multiple VIP users requiring protection, which poses the challenge of automatically identifying and selecting the appropriate VIP token without manual effort. To address this, we develop an adaptive variant, Adaptive VIPGuard, that autonomously selects the most relevant VIP token, thereby improving the scalability and usability of the proposed framework in real-world applications. Specifically. we first leverage face recognition model to identify the VIP user whose facial features are most similar to those in the input image. Subsequently, Adaptive forgery detection. As
VIPGuard utilizes the corresponding VIP token to perform identity shown Table 9, the comparable performance between Adaptive VIPGuard and the original VIPGuard demonstrates the robustness and practicality of our framework in real-world applications.
20

---

Table 9: Comparison of AUC (%) for adaptive VIP token selection in VIPEval, where Adaptive VIPGuard refers to the automatic selection of the VIP token without manual intervention.
Method
BlendFace
InSwap
Arc2Face
PuLII
Average
VIPGuard
99.48
96.40
98.05 97.63
98.96 98.95
98.23
Adaptive VIPGuard
99.31
96.14
98.01
D.4
Robustness Evaluation
We conducted experiments under common image degradations, including Gaussian noise, Gaussian blurring, and JPEG compression, to evaluate the robustness of VIPGuard. As shown in Table 10, VIPGuard remains highly effective even under severe degradation levels. Because the detector primarily relies on high-level semantic cues such as facial structure and identity-related features, inherently less sensitive to low-level pixel distortions. Table 10 presents the average AUC performance of VIPGuard in detecting BlendFace [57], InSwap [30], Arc2Face [49], and PuLID [24] forgeries under different degradation intensities. The results demonstrate minimal performance decline across all settings, confirming the model's robustness and strong generalization capacity. The detailed degradation configurations are provided Table 11. These results confirm that VIPGuard maintains stable detection performance demonstrating its robustness and practicality for real-world deployment.
when subjected considerable image degradation,
Table 10: VIPGuard's performance (AUC (%)) under different image degradations. Higher levels indicate stronger degradation. The results represent the average AUC across BlendFace, InSwap, Arc2Face, and PuLID.
Level
Gaussian Noise (Color)
Gaussian Blurring
JPEG Compression
None
98.23
98.23
98.23
97.07
98.17
98.03
96.53
98.16
98.10
3
94.12
98.05
97.78
Table 11:
Degradation configurations for robustness evaluation. Gaussian noise (O(0, o²)) was applied in YCbCr space: Gaussian blurring was defined by kernel size (K) and standard deviation (o); JPEG compression was applied using different quality factors (QF).
Level
Gaussian Noise (Color)
Gaussian Blurring
JPEG Compression
22) N(0, 11
K (7,7),
QF 90
K (13, 13), a =
QF 60
3
((0, 182)
K (21,21)
QF 30
D.5
Explanation Examples
We additionally supplement some explanation results, which are exhibited in Figures 8, 9, 10, and 11.
21

---

Image 1 (Real)
Image 2 (Real)
Can you describe the face's structure, skin condition, and overall apperrace?
Attr
Attr 2:
VIP Guard
Please compare face similarity of these two face 93/100. The face tokens shown follows, <|face_pad|- The facial attributes for the first person is: (Attr_ The facial attributes for second person (Attr_ 2] You need to make step-by-step jyddsmeee judgment based on different facial attributes and provide your conclusion.
facial images determine whether they belong the person. The
Okay, here is the comparison based on provided images and attributes:
consistent face shape with jaw. The proportions regarding temporal, zygomatic, and jaw widths appear identical.
Outer face region: Both images
Eye region: While both have bushy eyebrows, Image describes 'normal' eyes with angulus oculi medialis of 53, whereas Image 2 describes 'thin' eyes with an angulus ocull medialis of 45. This difference in classification and angle measurement is noticeable.
VIP Guard
Nose region: Image identifies the nose 'thin', while Image identifies as 'normal.. represents difference the attribute description.
Mouth region: Both images describe mouth having 'smile lips'. Visually, the shape. fullness, and curvature appear consistent between the two images. Skin region: Although both share light skin tone, freckles/males, and fine lines around the eyes, Image notes the presence of nasolabial folds, which are absent in Image 2 according to the descriptions.
Summary: The comparison reveals strong similarities in fundamental structure outer mouth, eyebrow regions. However, discrepancies eye type, type, presence nasolabial folds, and the expressed emotion. These differences might stem from variations in lighting, expression, angle, or subjective interpretation during attribute annotation.
specific classifications of
Despite the noted differences some attribute classifications (eyes, nose, skin details), core facial structure appears highly consistent across both images. Considering high face similarity score 93/100 and visual evidence strong resemblance, the analysis supports the statement that both images belong to the same likely represent minor differences in image conditions annotation nuances rather than identity differences.
Overall Conclusion:
The variations
[Yes] The two images are of the same person.
Figure 8: An example VIPGuard in detecting the real faces. The image and image are from LAION-Face [87].
22

---

Image 1 (Real)
Image (Arc2Face)
Can you describe the face's structure, skin condition, and overall appearance?
User
Attr
Attr 2:
VIP Guard
Please compare the facial images determine whether they belong the person. The face similarity of these two face 91/100. The face tokens shown follows, <[face_ pad|- The facial attributes for the first person is: [Attr_1/0n The facial attributes for the second person is: [Attr 2] You need to make step-by-step judgment based on different facial attributes and provide your conclusion.
Okay, here is the comparison based the provided images and attributes:
Outer face region: [No] Both images show individuals with similar oval face shape and flat jawline. The proportions regarding temporal, zygomatic, and jaw widths appear consistent between the two faces.
Eye region: There are noticeable differences in eye region. Image has normal-sized eyes with wider angulus ocull medialls (71). Although both have bushy eyebrows, the eye size and spacing differ significantly.
VIP
angulus medialis (53), while Image features larger eyes with
Nose reglon: A difference exists in the nose structure. Image 1 displays thinner nose, whereas Image has a nose described as normal, which appears slightly broader, particularly the bridge and tip. Mouth region:
are described having smile lips, the actual expression differs (smile neutral). More importantly, lip fullness the specific curvature lips appear distinct between the two images.
Skin region: Differences are noted in the skin. Image 1 lacks mole, whereas Image 2 has one mole present. Additionally, the skin texture and pore appearance seem slightly different upon visual inspection.
While some general facial structures like shape and jaw type share similarities, significant differences observed the eye size spacing, shape, characteristics, and skin details.
Summary:
Overall Conclusion: Despite the high face similarity score of 91/100 and shared attributes like face shape and jaw type, the distinct differences identified in the eye region. nose, mouth, skin, and overall expression confirm that the two images feature different individuals.
[No] images different people.
Figure 9: An example of VIPGuard in detecting the forgery generated by Arc2Face [49]. The image 1 is from LAION-Face [87] and the image 2 was generated by Arc2Face.
23

---

Image 1 (Real)
Image (GPT4o)
Can you describe the face's structure, skin condition, and overall appearance?
User
Attr
Attr 2:
VIP Guard
Please compare the facial images determine whether they belong the person. The face similarity of these two face 93/100. The face tokens shown follows, <[face_paat pad|- The facial attributes for the first person is: [Attr_1/0 The facial attributes for the second person is: [Attr 2] You need to make step-by-step judgment based on different facial attributes and provide your conclusion.
Okay, here is the comparison based on the provided images and attributes:
Outer face region:
Both images show a long face shape with a square jaw. The described proportions between temporal, zygomatic, and jaw widths consistent in both attribute sets and appear visually similar.
VIP
Eye region: Although both have big eyes and bushy eyebrows, there is a difference in the eye in Image
angulus medialis measurement (63 71). Additionally, occluded, whereas both eyes are open in Image 1.
Nose region: Both images feature thick and the visual appearance the shape and size consistent between two images.
Mouth region: Both individuals display smile lip type. The general shape, curvature, and teeth exposure during the smile appear very similar.
Skin region: While both share natural skin tone, crow's feet, fine lines, nasolabial folds, and eye pouches, Image
distinct mole the cheek, which absent Image
Summary:
The comparison reveals strong similarities outer face shape, and mouth. However, noticeable differences exist in the eye region (specifically the angulus oculi medialis measurement and eye occlusion) and the skin region (presencelabsence of distinct mole).
Overall Conclusion: Despite numerous similarities and high face similarity score (93/100), specific differences observed in eye skin regions confirm the two images depict different Individuals.
[No] The two images are of different people.
Figure 10: An example of VIPGuard in detecting the forgery generated by GPT-4o [48]. The image from CrossFaceID [64] and the image 2 was generated by GPT-4o.
24

---

Image (Real)
Image (BlendFace)
Can you describe the face's structure, skin condition, and overall overall appearance?
Attr 1:
Attr_ 2:
User
VIP
Please compare the two facial Images and determine whether they belong to the same person. The face similarity of these two face is 68/100. The face tokens shown as follows, iffac__pad|> The attri autri for the first person is: [Attr_1]] The facial attributes for the second person is: [Attr 2] You need to make step step-by--tep judgment based on different facial attributes and provide your conclusion.
Okay, here is the comparison based on the provided images and attributes:
Outer face region: There is a noticeable difference in the face shape (square vs. oval) and jaw type (square vs. flat).
Eye region: Differences exist in eye size (thin vs. normal), the inner comer angle (angulus ocull medialis), and the apparent iris-to-eye ratio.
VIP Guard
Nose reglon: The nose types are distinctly different (thick vs. thin).
Mouth region: The lip types differ significantly (normal vs. smile lips), and the expressions are different (neutral vs. smiling).
Skin region: Differences are observed skin tone (natural vs. light) and the presence of specific wrinkles forehead wrinkles and crow's feet.
Summary:
The analysis reveals substantial differences major facial regions, including face shape, jaw type, eye characteristics, nose type, mouth appearance, skin attributes, and overall descriptors like gender.
Based on the numerous significant dissimilarities identified in the facial attributes across multiple regions, and supported by low similarity score 68/100, concluded that the two images show faces belonging to different individuals.
Overall Conclusion:
[No] The two images are of different people.
An example of VIPGuard in detecting the forgery generated by BlendFace [57]. The image 1 is from FacelID66 [63] and the image 2 was generated by BlendFace.
Figure 11:
25

---

Eyes Type
Type
Type
Eyebrow Type
Big
Nornal:
Sharp
Thin Normol
Arch
Raise
Thin
Round
Thick
Eight
Seot1
Flat
Smile
Bushy Round
Type
Type
G Long
Oval
x Thick
Crow's
feet
Forehead Wrinkle
Normol
Normal
Pointed
Thin
M
X Yes +
Glabella
Eye Pouch
Wrinkles
Molle
Fold
S Yes
I Yes
No
+ Yes
No
Yes
ho
Skin Tone
G Whoot O Natural
Crow
Light Fair
Freckle
Finelines
Feet
Bark
G Yes
Yes
No
Yes
Figure 12: Visualization of different facial attributes. All the images are from LAION Face [87]
E Dataset Construction
We introduce a new dataset, VIPBench, designed to provide a more comprehensive evaluation of Identity-aware Deepfake Detection methods [54, 19, 17, 16, 81]. In this section, detailed describe the construction pipeline of VIPBench, which contain three part-Facial Attributes Description Dataset DFA, Identity Diserimination Dataset DID, and VIPEval D The detailed process encompasses data
and the construction of visual question answering (VQA) components, which are shown below.
E.1 Facial Attributes Description Dataset
To improve the ability facial understanding MILMs, we proposed Facial Attributes Description Dataset D a multimodal dataset composed of high-resolution facial images paired with rich facial attribute descriptions.
Data Collection& Coolection&PPrprocesssii Preprocessing High- reghutreso facial images essential for effective facial understanding, as they preserve rich and detailed information. To this end, we collected a large number of high-quality facial images (with resolution more than 1024 x 1024) from LAION-Fac [87], filtering out those that were blurry or occluded. Specifically, we employed MTCNN [85] to crop faces from the raw images. We retained only those samples where facial landmarks could be reliably detected to ensure clarity. Finally, we selected faces with yaw and pitch angles below 15" to ensure frontal and unobstructed views 3DDFA [23].
Construction of VQA data We adopt commercial facial analysis tool, Meg official API obtain detailed facial attributes. The facial attributes are shown in Figure 12. Subsequently. this structural information was used to build a VQA dataset that textually describes facial images. Specifically, constructed three types of VQA data: multiple choice, short answer, and long answer. For the multiple-choice and short-answer formats, we directly constructed the VQA data using the extracted facial attributes, as illustrated in Figure 13. addition, to generate long-am oong-answer VQA data, we employed GPT-4o [48] to reorganize the structured facial attribute information into coherent and natural language descriptions, as demonstrated Figure 14. Figure 15 illustrates an example the long answer format in AA The naive MILM will gain improved facial understanding abilities after training on the proposed facial VQA datasets.
htapss/www.fecclluspuss.comm
26

---

VQA: Multi-Choices
Questia tion
Which of the following best describes the person's Skin tone? Light Wheat Fair D. Natural E. Dark
Answer
VQA: Short Answer
Question
Describe the observed Forehead wrinkle of the person the Input Image.
Answer
No.
Figure 13: Examples of our VQA data formats, including multiple-choice question and short- answer question based on facial attributes. The two images are from LAION-Face [87]
Prompt to GPT4o
You helpful assistant responsible for paraphrasing restructuring the input text. Your goal is to improve its clarity, style, tone without altering the original meaning. Do not fabricate introduce any information that is not already present the input. You must not change any attribute names or values. For each major category, organize all content into a coherent paragraph, and include clear subheading for each category. You must only output the rewritten text-do not include any additional or unrelated responses.
-Tee attributes in skin region of face as following:
Skin tone: light
Mole: yes
Forehead wrinkle: no
Crows feet: no
Eye yes
Glabella wrinkle:
Nasolabial fold: no
Eye pouch: yes
Response of GPT4o
The characteristics of the facial skin region are follows: the skin tone is light, there freckles, mole present, there forehead wrinkles, crow's feet, fine lines visible around the eyes, there are no glabella wrinkles, no nasolabial folds, and eye pouches present.
An example of using GPT-4o to generate a coherent textual description by reorganizing structured facial attribute data. This case focuses on attributes related to the skin region as representative example.
Figure 14:
27

---

VQA: Long Answer
Question
you describe the face's structure, skin condition, and overall appearance?
Outer Face Region (attribute list: [face type,jaw type, comparison of temporal width and zygomatic width, comparison jaw width and zygomatic width, comparison of temporal width and jaw width)) Eyes Region (attribute list: [eye type, angulus oculi medialis, eyebrow type)), and Nose Region type))
(attribute list:
Mouth Region (attribute list: [mouth type]).
Skin Region (attribute list: (skin tone, freckle, mole, forehead wrinkle, crows feet, eye finelines, glabella wrinkle, iasolabial fold, eye pouch]).
Overall facial description (attribute [gender, smile, glass, head position (up/down), head position (left/right), eye visibility and glasses detection, nose visibility, chin visibility, face visibility, emotion, face qualityl)
Answer
Outer face reglon: In the outer face region, the attributes include an oval face type flat jaw type. the comparison temporal width zygomatic width shows them similar, while jaw width zygomatic width, additionally, the temporal width wider in compárison to the jaw width.
compared the
Eye region:
For the eyes region, the face features big eyes, with an angulus oculi medialis measurement 60, the eyebrows are described bushy.
Nose region: The nose region is characterized by thick nose type.
Mouth region: the mouth region, the type is categorized smill lip.
Skin region: The skin attributes include wheat skin tone, with freckles moles present. the forehead wrinkle- free, while crows feet and fine lines around the eyes are noticeable. there are no glabella wrinkles, and nasolabial folds are present, but there is no eye pouch.
Overall facial description: The overall description of the face includes the following characteristics: gender female, exhibiting smile, and there are glasses worn. the head position is normal when facing forward. both eyes are visible, open, and not wearing glasses. the nose and chin are clear, and both the left and right sides of the face are visible. the emotion displayed is happiness, and the face quality is described as very clear, making suitable for face verification.
Figure 15: An example of long-answer VQA data. The images come from LAION-Face [87]
28

---

E.2 ID Discrimination Dataset
In this work, we formulate the Deepfake detection task as fine aineggrined face recognition problem tailored to specific users. To support this identity verification task, we introduce two datasets: Dip and DID The former consists of facial image pairs drawn from arbitrary identities, whereas the latter comprises facial image pairs constructed specifically around a particular user.
Data Collection& Preprocessepoor To construct these datasets, first require a large volume of facial images with known identities. To this end, we source relevant images from existing open source datascts, including LAION-Face [87], FaceID-6M [63], and CrossFacelD [64]. As illustrated in Figure 16, we utilized the image-caption pairs from these open source datasets and employed Deepseek to extract names from the captions, thereby generating pool J of image-name pairs. We retrieved multiple images for each identity based on the extracted names and applied the same image pre-processing techniques as described in Section E.1. Subsequently, illustrated in Figure 17, we constructed facial image pairs for both DD Deeneral and DID. The collected images were organized into a large number of pairs, each comprising a reference image !" and a test image 1 For Dip positive pairs comprise two real images of the same identity (denoted as 1 -/ag Trop: Same ID), while negative pairs include either two real images from different identities (Treat- real Diff ID) or real image paired with its corresponding fake counterpart (Tyeat-Tiake Taa-Joakk-aameee
-Same ID). To generate fake images, we employed SimSwap [5] and Are2Face [49]. Notably, when using SimSwap, we replaced the identity vector typically required for face swapping with a random noise vector o ~ N (O, 1), enabling the substitution of the inner face while preserving the outer facial features. In addition, constructed partially swapped images by applying different masks to the eyes, nose, mouth, and inner face regions. The ratio of samples in Dipne is
(Fraat-iin ID):
Diff (Frea-T Same ID) :2:1:1
which ensures a balanced number of positive and negative samples.
Dieneral the construction pipeline for DID "ip followed nearly identical procedure, with the key distinction that all reference images I" exclusively belonged to VIP users. Due to the few number of the real images of VIP users, the ratio was adjusted to
Similar to DD
(Fea ID): (Frea-tfea Diff ID): (Tjeat-ljaake Saae ID) = :1:5:5
where the number of negative samples ten times that of the positive samples.
Raw-Data
"Extrocting name"
User
User
Name Retrieve
Caption
Name
LAION-Face
Pool T
Deepseck-V3 V3
CrossFaceID
FMceID-6M
Mmage-Caption Pair
Image-Name Pair
ID
VIP
Figure 16: Pipeline for constructing the image-nama pair pool J.
Construction of VQA data After constructing the facial image pair dataset, we first annotate facial attributes using a captioning model (Qwen-2.5-VL--7)) which was fine-tuned on the DFA dataset. These attribute annotations, together with facial similarity scores, then provided input to Gemini' [21], which is tasked with analyzing the similarities and differences between the two faces. Figure 18 illustrates example of the prompt used during this process. The prompts adjusted based on the sample type. For positive samples, Gemini is explicitly instructed with the note: "Note that these two images show the same person. For negative samples, the prompt "Note that these two images show different persons." " Furthermore, in cases
states:
Gemini API version in use: 2.5-pro-xxp-034 03-25.
29

---

D Doeneral
D
ID
ID
Positive Samples
Negative Samples
Positive Samples Negative Samples
1
r
I
1
DD
Fuke
VIP
VIP
VIP
Fake
ID
ID
ID
Figure 17: The composition of facial pairs in DDD
Doeneral and DID
where the negative sample involves a fake image, we provide more nuanced prompt: "Although the faces may appear similar, they are not the same person. Importantly, Gemini was constrained base its reasoning strictly on the provided attribute annotations, thereby mitigating hallucinations. Finally, the VQA data and the corresponding facial image pairs constituted our dataset Deeneral and Diip
E.3
VIPEval: User-Specific Evaluation Dataset
To better evaluate the effectiveness of user-specific protection methods and ID-aware detectors, constructed a multi-resolution dataset that includes seven types of face swapping and seven types of fully synthetic face forgeries. In this section, present representative samples from the dataset. In Figure 19, we present a comparison between authentic images from CelebDF [39) and our VIPEval dataset. VIPEval includes large number of images with various resolutions for each identity that capture significantly more facial detail.
Data Collection& Preprocessing The data collection and pre-processing procedure in VIPEval follows the same steps as described in Section 1. Building on this, we collected images for 22 target identities, obtaining approximately 40-60 real images per identity. For each identity, 20 real images were selected and fixed as the real samples in the test set, as well as the target images for generating fake content. The remaining 20-40 images were used as training data for that identity, and were used construct D D following the method described Section E.2. The training D is individually customized for each identity.
30

---

Prompt to Gemini
You will receive two input images with same identity:
Image The of the primary (reference) user. Image 2: The compared.
The face similarity
between Image and Image
Note that these two images show the same person.
Now, the region information of facial attribute of these two images is provided as follows: For Image 1: fattr 1) For image 2: (attr 2]
According to the input images, you can supplement additional facial attribute information (such as the lips in mouth and comparisons based your
ratio between the size and whole eye in the eye region, the curvature
region, the variations in pore size and appearance, and understanding the specified facial regions above. For each region, first provide conclusion by stating either [Yes] if there is noticeable difference, or [No] if there is no significant difference, Then explain the specific similarities or differences based on the facial attributes for that region. If the conclusion is [No], you should give more detailed explanation. If the conclusion is [Yes], you must provide simpler explanation. After completing conclusion. You may also refer to the face similarity score supporting evidence your analysis.
region-by-Fee be-region analysis, write short summary overall
Note: There may be more than one region that shows dissimilarity.
Note; You do not need to identify the specific person-your task is to analyze and compare based solely on facial attributes.
Note: You must only output relevant responses without any unrelated information.
Respond using conversational only-do not return JSON code format.
Note: You should provide precise
short possible.
Note: You should only provide the comparison process but not list the facial attributes.
Response of Gemini
Okay, here is the comparison based on the provided images and attributes:
1. Outer face region:
[Yes] While the jaw type and relative width comparisons heympoal/aygomaaicjjaa/ consistent, the described face type differs (long square). Visually, the overall shape appears very similar, suggesting the difference might due angle variations or detection nuances.
2. Eye region: (Yes] Although the eye type (big) and eyebrow type (bushy) match, there is noted difference in the angulus oculi medialis measurement (63 vs. 71). Visually, the eyes, iris size relative to the eye, and eyebrows appear highly consistent shape and placement.
The type is consistently identified thick both descriptions, Visually, the shape, bridge structure, and nostril appearance very similar across both images.
3. Nose region: [No]
4. Mouth region: [No] The mouth type is described as smile lip in both instances. The shape, curvature, and thickness of the lips appear consistent in the images.
5. Skin region: [No] Both descriptions indicate natural skin tone without freckles or moles, but with fine lines around the eyes, nasolabial folds, and eye pouches. These features, along with the general skin texture, appear consistent between the images, accounting for minor lighting variations.
Overall facial description: [No] aspects match: male gender, neutral expression, no glasses, normal head pose, clear visibility of features, very clear face quality suitable verification.
The comparison reveals consistency across most facial regions, including the nose, mouth, skin characteristics, and overall description. Minor discrepancies were noted in the outer face type description and the eye angle measurement, though visual inspection shows strong similarities in these areas as well.
Summary:
Despite the minor noted differences specific attribute descriptions (face type, eye angle), the overwhelming similarity in major facial features (eyes, nose, mouth shape), skin details, and overall appearance, strongly supported by the high face similarity score of 91/100, indicates that the two images belong to the same person.
Overall Conclusion:
Figure 18: An instance of using Gemini to generate a textual description for facial discrimination.
face score is the face similarity score calculated by face recognition models. attr 1 and attr formatted as shown in the answer in Figure 15, should be filled with attribute information generated by the captioning model.
31

---

VIPBench VIPEval)
Ao
Ehe
Celeb DF
fnr jog Stoedd fuidstaenigg den Sade of mag selv.
IRs Ces
inrs
Kos Aey
rtes Col
Figure 19: Comparasion of VIPBench (Ours) and CelebDF [39]. The quality of authentic images is higher in VIPBench. The images in VIPBench are from LAION-Face [87], FaceID-6M [63], and CrosI [64].
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose method named VIPGuard and dataset named VIPBench. It is exactly matches the experiment this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitations in Supplementary material ("Limitations").
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Our work is purely empirical and does not include any formal theoretical results or proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We will provide comprehensive experimental details in the supplemental material. All experiments in this paper are reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
34

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release the dataset and code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. cctpublic/geeeeeccccccccccccccce wwwlic/guis////eee (CodeSubmissionPoll
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We append all the experimental details and dataset setting in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We report metrics averaged over multiple independent support robust- ness, but have omitted explicit error bars in this version due to space constraints; we will include full variability analysis with standard deviations in the supplemental material and public code release.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
35

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Detailed compute specifications (hardware type, memory usage) are not included in this submission.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https htap:::///euuree http///eerrppp.eee
Answer:
[Yes]
Justification:
We confirm that our research fully adheres the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The impacts will be discussed in the supplementary material
Guidelines:
The answer NA means that there is no societal impact of the work performed.
36

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
This paper proposes a dataset involving facial images. To strictly prevent misuse, the dataset will be managed through request form (hosted HuggingFace) to monitor and control its usage.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We reference all the open-source models including Qwen2.5VL, LLaMA 3.2
Vision, and including their paper and original sources.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
37

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcom thcode. com/coo/////oooo conddataeet has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce a New Deepfake dataset and provide its detailed schema, usage instructions, and metadata in Appendix; full documentation will accompany the public release.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve any crowdsourcing experiments or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve any human subjects or user studies.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
38

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We used Qwen2.5VL. as our foundation model.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: httpps////euripae for what should or should not be described.
39


Reviews:
[{"review_id": "ifmnYQlzZi", "reviewer": "Reviewer_AVpP", "summary": "This paper proposes VIPGuard, a multimodal large language model (MLLM) designed for personalized and explainable deepfake detection, specifically for known individuals. It reformulates deepfake detection as a fine-grained face recognition task based on detailed facial attributes. The authors also present VIPBench, a comprehensive identity-aware deepfake benchmark to evaluate personalized deepfake detection models. Based on this benchmark, VIPGuard shows superior deepfake detection performance.", "questions": "- Fine-grained facial attributes, such as wrinkle and skin tone, are compared between the first image and the second image in the explanation. These attributes can vary when the conditions, such as lighting, cosmetics, etc., changed. Therefore, proposed method seems to be sensitive to the first image input. Would you suggest analysis about it?\n- VQA data for VIPBench, which is used to train VIPGuard, is generated by GPT-4o and Gemini 2.5 Pro. If those models are already able to explain visual attributes well, why are their deepfake detection capabilities lower than VIPGuard? Is it possible to get better results from those models by prompt engineering?", "limitations": "- It would be inefficient to utilize MLLM, which has parameter size of 7~8B.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n- VIPGuard demonstrates superior deepfake detection capabilities, outperforming existing methods.\n- VIPGuard offers human-understandable explanations for its deepfake detections.\n- This paper introduces VIPBench, a comprehensive identity-aware benchmark built to evaluate personalized deepfake detection.\n\n### Weaknesses:\n- The novelty of the proposed method appears limited. VIPGuard follows standard training process of MLLMs.\n- The experimental results seem insufficient. It would be more compelling if the VIPGuard were applied to other MLLMs, such as LLaMA-3.2-Vision, Qwen2.5-VL-3B, etc.\n- The ablation studies provided in D.3 seem insufficient to fully explain the individual contribution of each stage. Additional results would be beneficial. (for Stage 2, Stage 3, and Stage 2, 3)", "was_revised": true, "final_justification": "Although the rebuttal addresses most questions, the method’s novelty still seems limited to a standard MLLM fine-tuning procedure. Moreover, the use of VIP tokens also raises practicality concerns: if multiple tokens are trained for different VIPs, VIPGuard cannot select the correct one when a VIP image is given. The original paper seems to assume that the user should 1) determine whether the input belongs to previously-seen VIP or a new one and 2) set an appropriate VIP token manually or decide to train new VIP token. Such assumptions do not reflect practical scenarios.\n\nRegarding the remaining issues, the rating remains unchanged at borderline reject.", "rebuttal": "We sincerely appreciate the reviewer's detailed feedback and are encouraged by the positive assessment of **the SOTA detection performance and explainability of our method, and the novelty of the proposed personalized deepfake detection benchmark**. \n\nBelow, we provide point-by-point responses to the concerns raised by the reviewer.\n\n---\n\n> **Q1**:\n>\n>\n> VIPGuard follows standard training process of MLLMs, making the novelty limited.\n\n\n**R1**:\n\n\nWe sincerely appreciate the reviewer’s feedback. However, we would like to clarify that, our approach is **fundamentally not a naive application of standard MLLM training pipelines, but rather a purpose-built framework specifically designed for the unique challenges of our task**, i.e., personalized deepfake detection.\n\nWhile MLLMs provide a foundational architecture, directly applying standard MLLM training paradigms to this task leads to poor performance, as shown in our ablation studies (**Table A1**). This is because **personalized deepfake detection requires fine-grained understanding of facial identity priors—such as subtle anatomical structures and local attributes**—that vanilla MLLMs, trained on general vision-language tasks, inherently lack.\n\nTo overcome this limitation, we propose VIPGuard, a novel and comprehensive framework that **goes far beyond standard MLLM training in both design and execution**. Our key innovations include:\n\n1. **Task-Specific Formulation:** \n\t- **We reformulate deepfake detection as a face comparison task guided by identity priors—a paradigm not explored in prior MLLM-based detection methods**. This enables the model to detect subtle forgery artifacts by comparing a query image against the known facial characteristics of a specific individual (\"VIP\"). This shift in perspective is itself a conceptual innovation.\n\t\n2. **Comprehensive and Custom Data Construction Pipeline:** \n\t- Unlike existing methods that rely on off-the-shelf LLMs or generic annotations, **we introduce a dedicated facial annotation model trained to extract precise identity, attribute, and similarity signals from faces.** This allows us to generate high-quality, identity-centric training data that explicitly teaches the MLLM to reason about facial consistency—a critical capability missing in standard training setups.\n\t\n3. **Progressive Learning Strategy**:\n\t- We design a three-stage progressive learning framework—facial attribute learning → general face comparison → VIP-specific customization—that gradually equips the MLLM with the fine-grained facial understanding necessary for personalized detection. This structured curriculum addresses the MLLM’s initial lack of facial expertise and ensures effective adaptation to the target task. \n\t- As demonstrated in **Table A1**, naive end-to-end training (i.e., standard MLLM fine-tuning) performs significantly worse, confirming that our improvements are not due to scale or data alone, but to our tailored methodology.\n\n\nIn summary, our work does NOT naively apply MLLMs to the task. Instead, **we identify the limitations of standard MLLM training in the context of personalized forgery detection and introduce a series of task-driven innovations**—*from data construction to model training—to overcome them*. The substantial performance gains in Table A1 and the ablation studies strongly support the necessity and effectiveness of our proposed framework.\n\n**TABLE A1:** Comparison (AUC %) between VIPGuard and Naive MLLM Training across four forgery methods.\n\n| **Method**                | **BlendFace** | **InSwap** | **Arc2Face** | **PuLID** | **Average** | **Improvement (+)** |\n| :------------------------ | :-----------: | :--------: | :----------: | :-------: | :---------: | :-----------------: |\n| Qwen-2.5-VL-7B (Baseline) |     51.00     |   49.79    |    49.85     |   50.30   |    50.24    |          -          |\n| Naive MLLM Training       |     71.96     |   59.10    |    62.12     |   68.18   |    65.34    |       + 15.10       |\n| **VIPGuard (Ours)**       |   **99.48**   | **96.40**  |  **98.05**   | **98.96** |  **98.23**  |     **+ 47.99**     |\n\n---\n\n> **Q2:**\n> \n> \n> It would be more compelling if the VIPGuard could be applied to other MLLMs, such as LLaMA-3.2-Vision, Qwen2.5-VL-3B, etc.\n\n\n**R2:**\n\n\nWe appreciate your valuable comment. Following your suggestion, we conducted an additional ablation study of altering different MLLMs in the VIPGuard. \n\nTable A2 shows that our method consistently achieves strong detection performance on personalized deepfake detection across various MLLMs. These results will be incorporated into the revised manuscript to strengthen its overall quality.\n\n**TABLE A2:** Performance Comparison (AUC (%)) of Our Method with Different MLLMs.\n\n| **MLLM**                  | **BlendFace** | **InSwap** | **Arc2Face** | **PuLID** | **Average** |\n| - | :-: | :-: | :-: | :-: | :-: |\n| Qwen-2.5-VL-7B (Baseline) |     51.00     |   49.79    |    49.85     |   50.30   |    50.24    |\n| LLaMA-3.2-Vision-11B      |     96.27     |   88.97    |    99.00     |   99.52   |    95.94    |\n| Qwen-2.5-VL-3B            |     90.71     |   87.85    |    87.82     |   97.94   |    91.08    |\n| Qwen-2.5-VL-7B            |     99.48     |   96.40    |    98.05     |   98.96   |    98.23    |\n\n---\n> **Q3:**\n> \n> \n> More comprehensive ablation studies would be beneficial. (for Stage 2, Stage 3, and Stage 2, 3)\n\n\n**R3:**\n\n\nThank you for your valuable comment. We have added additional results in Table A3 to demonstrate the effectiveness of Stage 2, Stage 3, and Stages 2,3. Due to MLLMs’ limited understanding of faces, directly applying Stage 3 results in poor performance. In contrast, enhancing the model’s basic perception through Stage 2 significantly improves performance, further demonstrating the effectiveness of the proposed progressive framework. These results will be included in the revised version of the paper, and we believe they will strengthen our work.\n\n**Table A3.** An ablation study on the performance (AUC (%)) of different components in VIPGuard\n\n| **Variant** | **BlendFace** | **InSwap** | **Arc2Face** | **PuLID** |\n| - | :-: | :-: | :-: | :-: |\n| Baseline    |     51.00     | 49.79      |    49.85     |   50.30   |\n| +Stage 2    |     95.29     | 73.57      |    97.24     |   94.33   |\n| +Stage 3    |     71.96     | 59.10      |    62.12     |   68.18   |\n| +Stage 2,3  |     99.63     | 93.99      |    98.10     |   98.76   |\n\n---\n> **Q4:**\n> \n> \n> The proposed method seems to be sensitive to the first image input. Would you suggest analysis about it?\n\n\n**R4:**\n\n\nThank you for your comment. The first input image serves as the reference ($I_{ref}$ in Figure 4 of our paper), representing the facial image of the protected user. In our method, **we replace it with a VIP token to mitigate the sensitivity issue.**\n- Challenges may arise when detection relies solely on a single reference image without training the VIP token $\\mu$, a setting referred to as *OneShot*. Under this setting, changing conditions may alter the image attributes, making detection to variations in the initial input. \n- In contrast, our method replaces the first image with a learned VIP token obtained through training. **This token encodes the user’s identity from multiple images by training, resulting in a more robust and stable representation**. Table A4 shows that VIPGuard consistently outperforms the OneShot setting, confirming its robustness and effectiveness.\n- Figure 1 in the appendix shows that performance improves as more authentic images are encoded into VIP tokens, highlighting their effectiveness in mitigating sensitivity.\n\n**TABLE A4.**  Performance (AUC (%)) comparison between the *OneShot* and VIPGuard. VIPGuard, which utilizes a learned identity token trained from multiple images, \n\n| **Method** | **BlendFace** | **InSwap** | **Arc2Face** | **PuLID** |\n| - | :-: | :-: | :-: | :-: |\n| One Shot   |     90.82     | 73.45      | 77.25        | 71.08     |\n| VIPGuard   |   **99.48**   | **96.40**  | **98.05**    | **98.96** |\n\n---\n> **Q5:**\n> \n> \n> VQA data for VIPBench, which is used to train VIPGuard, is generated by GPT-4o and Gemini 2.5 Pro. \n> 1. If those models are already able to explain visual attributes well, why are their deepfake detection capabilities lower than VIPGuard's? \n> 2. Is it possible to get better results from those models by prompt engineering?\n\n\n**R5:**\n\n\n1. **The key difference lies in the information available to the model when functioning as a _Captioner_ versus as a _Detector_.** This distinction stems from the differing roles played by Gemini 2.5 Pro (and similar models) in each case:\n\t- *Captioner:* Generates textual annotations for MLLM training. During this process, we explicitly **provide rich prior information (e.g., category labels)**, and the Captioner formats it into the desired textual structure.\n\t- *Detector:* Performs detection based on the input image and question, but is **not allowed to access any prior information at inference time**.\n\n\tDuring annotation generation, Gemini 2.5 Pro, acting as a Captioner, was given full prior information—including category labels, as shown in the following equation (also Eq. (2) in our paper):\n\n\t$Prompt = Gemini(Q, f(I_i, I_j), I_i, I_j, G(I_i), G(I_j))$ \n        \nHere, $Q$, $f(I_i, I_j)$, and $G(I_i), G(I_j)$ denote priors such as category, facial similarity, and facial attributes. Gemini 2.5 Pro and other commercial models merely format this information into text; **they do not perform any detection themselves.** \n\n2. These models cannot detect effectively during testing—when such priors are still unavailable—even with prompt engineering. In contrast, **VIPGuard is trained to perceive this information and perform detection independently, without relying on external priors.**"}, {"review_id": "k4fUBqgJfh", "reviewer": "Reviewer_ZrBG", "summary": "This paper presents VIPGuard, a framework designed to detect and explain Deepfakes when the user’s identity is known. The authors detail a comprehensive data preparation pipeline and a three-stage training framework that employs MLLM. This framework integrates crucial aspects of facial recognition, comparisons between arbitrary pairs of faces, and user-specific customization. Additionally, a novel evaluation benchmark is proposed to assess identity-aware Deepfake detection methods. The findings suggest promising potential for real-world applications, providing an effective solution for Deepfake detection in scenarios where the individual’s identity is known. Overall, this work introduces sufficient novelty and contributes to the advancement of identity-aware Deepfake detection technologies.", "questions": "1.\tHow does the number of available authentic images of each VIP user affect the final performance?\n\n2.\tIt is suggested that the authors place the problem formulation in the appendix to the main content, which could help readers better understand the specific setting of this work.\n\n3.\tIt is suggested that the authors detail the inference process of the method in the manuscript and also report the inference time.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n1.\tThis paper introduces a novel framework, VIPGuard, designed to detect deepfake images by leveraging identity information. It effectively uses an MLLM (Qwen-2.5-VL-7B) to analyze facial semantic features and assess global similarities among different face models.\n\n2.\tIt presents a well-structured three-stage training pipeline that gradually adapts the MLLM for detailed facial analysis. This progression from general facial recognition to specific individual identification is both logical and technically sound.\n\n3.\tA new dataset, VIPBench, for ID-aware deepfake detection is proposed. Diverse manipulation approaches are considered in collecting the dataset, including recently released commercial methods, which better align with real-world complexity.\n\n4.\tExperimental results demonstrate that the proposed framework achieves competitive performance compared to existing approaches.\n\n\n**Weaknesses**\n\n1. Stage 3 of VIPGuard still relies on VIP user annotation via Gemini or similar APIs, which may hinder practical deployment. An experiment using only image inputs would enhance the evaluation.\n\n2. Some errors in Eq. (5) and Eq. (6) lead to inconsistencies with Fig. 4. In Eq. (5), the symbol $g$ represents the features produced by the Cross-Attention module; however, its definition is missing in the equation. Moreover, the symbol $g$ should also be included in Eq. (6).", "was_revised": true, "final_justification": "The authors' response addresses my initial concern. I have no further questions.", "rebuttal": "We sincerely appreciate your positive feedback on **the novelty and logic of our method, the proposed benchmark, and the overall performance**. Below, we provide detailed responses to the concerns you raised.\n\n\n\n---\n> **Q1:**\n> \n> \n> Assessing VIPGuard without relying on textual annotations during training is helpful for understanding its real-world applicability.\n\n\n**R1:** \n\n\nThank you for your valuable comment. As shown in the newly added Table B1, **VIPGuard performs well in Stage 3 using only image inputs**. Despite being trained without textual annotations, it shows only a slight performance drop, demonstrating its practicality for real-world use. We will include this experiment in the revised paper to improve it further.\n\n**TABLE B1:** Performance (AUC(%)) of VIPGuard under different settings in the Stage 3. *Images + Annotation* indicates the joint use of both images and their corresponding textual annotations during training. *Only Image* refers to training with images alone, without accompanying annotations.\n\n| **Variant**           | **BlendFace** | **InSwap** | **Arc2Face** | **PuLID** | **Average** |\n| --------------------- | :-----------: | :--------: | :----------: | :-------: | ----------- |\n| *Only Images*         |     98.45     |   92.91    |    94.35     |   97.72   | 95.86       |\n| *Images + Annotation* |   **99.48**   | **99.43**  |  **98.05**   | **98.96** | **98.98**   |\n\n\n\n---  \n> **Q2:**\n> \n> \n> There are some symbol errors in Eq. (5) and Eq. (6) of the paper, which should be fixed.\n\n\n**R2:**\n\n\nThank you for your comment. We apologize for the errors in Eq. (5) and Eq. (6). They will be been corrected in the revised version as follows:\n\n*Eq. (5)*: \n\n$L(\\theta)=-\\sum^N_{i=1}log[p_\\boldsymbol{\\theta}(x_i|x_{i-1}, f_{ref}, f_{query},g)]$\n\n*Eq. (6)*:\n\n$L(\\theta)=-\\sum^N_{i=1}log[p_\\theta(x_i|x_{i-1}, \\boldsymbol{\\mu}, f_{query},g)]$\n\nMoreover, we provide the equation that describes the computation of the feature representation $g$ from the Cross-Attention module, as follows:\n\n$g = \\text{softmax}\\left(\\frac{QK^\\top}{\\sqrt{d_k}}\\right)V$\n\nwhere $K$ and $V$ are the visual features of the reference image $I_{\\text{ref}}$ in Eq. (5) or the VIP token $\\mu$ in Eq. (6), after the linear projection layer; and $Q$ is the visual feature of the query image $I_{\\text{query}}$, also after the linear projection layer.\n\n\n\n---  \n> **Q3:**\n> \n> \n> How does the number of available authentic images of each VIP user affect the final performance?\n\n\n**R3:**\n\n\nWe appreciate your comment. We would like to clarify that this experiment was already included in Figure 1 of our appendix. In this experiment, using just three authentic images per VIP user leads to a significant improvement compared to the one-shot setting (i.e., one authentic image per user, equivalent to using only the Stage 1+2 model). Our results show a progressive performance gain as the number of available authentic images increases from 3 to 20. **In conclusion, providing more authentic images per VIP user enhances the model’s ability to detect fake images.**\n\n\n\n---\n> **Q4:**\n> \n> \n> It is suggested that the authors detail the inference process of the method in the manuscript and also report the inference time.\n\n\n**R4:**\n\n\nThank you for your comment. To perform detection, VIPGuard will use the following pre-defined template. In this template, *FACE_SCORE* is the facial similarity score and 
 is the placeholder of the output of the cross attention module.\n\n```\n\nPlease determine whether the person in the input image is VIP user. The face similarity of these two face is {FACE_SCORE}/100. The face tokens are shown as follows, 
 . You should directly answer 'yes' or 'no' without any explanation.\n\n```\n\nFor each VIP user, we input a set $\\mathcal{X}$ of authentic images into the face model $E_{face}$ and calculate a centric vector $f_{center}$ by the following formulation.\n\n$f_{center}=\\frac{1}{N}\\sum_{x\\in{\\mathcal{X}}} E_{face}(x)$\n\nwhere $x$ is the input image and $N$ is the size of the set $\\mathcal{X}$.\n\nIn inference, we will calculate the facial similarity score between the feature $f$ of the current input image $x$ and $f_{center}$, as shown following\n\n$f = E_{face}(x)$\n\n$sim = 100 \\Biggl[\n        \\frac{1}{2}\n        \\Bigl(\n          \\frac{f \\cdot f_{\\text{center}}}\n               {\\lVert f\\rVert_{2} \\cdot \\lVert f_{\\text{center}}\\rVert_{2}}\n          + 1\n        \\Bigr)\n      \\Biggr]$\n\nFinally, the input image, facial similarity score, VIP token, and textual template are fed into the MLLM to produce the final output. Moreover, when Qwen-2.5-VL-7B is selected as the backbone, the inference time is about 2.09 seconds per sample without any acceleration techniques.\n\n\n---\n> **Q5:**\n> \n> \n> It is suggested that the authors place the problem formulation in the appendix to the main content, which could help readers better understand the specific setting of this work.\n\n\n**R5:**\n\n\nThank you for your valuable suggestion. We will revise the manuscript accordingly."}, {"review_id": "ymbX9XhAZR", "reviewer": "Reviewer_Zyjs", "summary": "This paper proposes a new deepfake detection framework, called VIPGuard. Unlike existing methods that typically focus on general-purpose detection, VIPGuard emphasizes individualization. Several modules are thus designed, including a fine-tuned multimodal large language model (MLLM), an identity-level discriminative learner, and a user-specific customization step. Experimental evaluations show clear advantages of the proposed framework over existing baselines.", "questions": "Below, I elaborate on my comments regarding weaknesses:\n\n1. In Step 2 of the proposed framework, the identity discrimination function is learned using a large dataset of positive and negative face pairs, which essentially assumes knowing the forgery scheme that will be employed by adversaries. In my opinion, this is a strong assumption. How well does your method perform against a new forgery scheme (or an unknown deepfake generation method)?\n\n2. The paper focuses on 22 unique identities in the dataset and evaluation. How can your method be applied to a new identity beyond the 22 considered identities? Do we need to retrain the whole model? If the identity-specific characteristics are different, how can you make sure the learned detection scheme generalizes to unseen identities?\n\n3. The proposed framework consists of several modules adapted from existing literature. It is unclear how novel the proposed method is. Please clarify your work's key technical contributions.\n\n4. It is unclear whether the proposed detection method can still perform well when there are natural image transformations (i.e., blurring, Gaussian noise, compression, etc.). Please clarify.\n\n5. Are there any failure modes you observed that are worth mentioning? Please discuss potential adaptive forgery strategies that may render the proposed detection method less effective.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+ Developing effective deepfake detection techniques is a timely and important topic.\n\n+ The paper tackles the problem from an individualized perspective, which is new.\n\n+ The paper makes good efforts in constructing a comprehensive benchmark for identity-aware deepfake detection, which can be of independent interest for future research.\n\n+ Experimental results show impressive improvement over existing deep fake detection methods.\n\nWeaknesses:\n\n- It is unclear how well the proposed VIPGuard method generalizes to unseen forgery schemes.\n\n- It is unclear how the proposed framework can be used for a new identity.\n\n- The technical novelty of the proposed method seems limited.\n\n- The paper does not discuss how robust the detection method is when there are image transformations involved.\n\n- There is a lack of discussion about the failure cases.", "was_revised": true, "final_justification": "The authors' rebuttal addressed most of my primary concerns, so I will increase my evaluation from \"borderline reject\" to \"borderline accept\". The authors are suggested to include the additional experiments and discussions in the revised version of their work.", "rebuttal": "Thank you for your valuable feedback. We appreciate **your recognition of the novelty of our problem and the uniqueness of our approach to addressing it, as well as the strength of our method, and the contribution of our benchmark**. Below, we respond to your comments point by point:\n \n---\n\n> **Q1:**\n> \n> \n> 1. This paper assumes knowing the forgery scheme that will be employed by adversaries, which is a strong assumption. \n> 2. How does the proposed method perform against an unknown forgery method?\n\n\n\n**R1:**\n\n\nThank you for your comment. We list our response as follows:\n- We have already presented **the experiments involving *unknown* forgery detection in our paper (generalization)**, with no leakage of identity information or prior knowledge of the forgery methods. Specifically, we have evaluated the performance of our method on the following **12 unknown forgery methods**: *BlendFace, Ghost3, HifiFace, InSwap, MobileSwap, UniFace, ConsistentID, PuLID, GPT-4o, Jimeng AI, Tongyi, and Kling AI*. We just utilized the SimSwap and Arc2Face as the training fake images in our method. \n- Based on the proposed method, we enable MLLM to learn facial priors and reason with fine-grained facial information. Tables 1–4 in our paper show that our method **consistently achieves the best generalization performance across all evaluated forgery methods**. \n\n---\n> **Q2:**\n> \n> \n> The paper focuses on 22 unique identities in the dataset and evaluation.\n>1. How can your method be applied to a new identity beyond the 22 considered identities?\n>2. Do we need to retrain the whole model?\n>3. If the identity-specific characteristics are different, how can you make sure the learned detection scheme generalizes to unseen identities?\n\n\n**R2:**\n\n\nThank you for your comment. We respond to your question as follows:\n\n1. To apply VIPGuard to a new user, we need to train a user-specific VIP token $\\mu$ using a few of his/her authentic images. We generate positive and negative face pairs from these images to train the VIP token $\\mu$, while keeping the rest of the model frozen.\n2. **No, the whole model remains frozen.** Only the lightweight VIP token (0.11M parameters) is trained for each new identity, highlighting excellent adaptability across different identities.\n3. Our paper focuses on ID-aware deepfake detection, which is defined as **detecting facial forgeries by effectively utilizing known identity information, aligning [1,2]**. Therefore, handling unseen identity conflicts with this topic. For VIPGuard, at least one authentic image of the user is required for forgery detection. As the number of available images increases, the model’s detection performance improves (see Figure 1 in the appendix), demonstrating the scalability of our method.\n\n[1] Diff-id, TDSC, 2024.\n\n[2] ICT-Ref, CVPR, 2022.\n\n---\n> **Q3:**\n> \n> \n> The proposed framework consists of several modules adapted from existing literature. It is unclear how novel the proposed method is. Please clarify your work's key technical contributions.\n\n\n**R3:**\n\n\nThank you for your comment. We provide a detailed explanation to clarify our novelty as follows:\n\n\n- Firstly, to better address your concern, we sincerely ask if you could provide **more specific details** about the mentioned existing literature, such as relevant references or method names.\n- Secondly, we would like to re-emphasize that our work introduces novel contributions that have not been proposed in previous research. The key aspects of our novelty are as follows: \n\n\t1. **Task-Specific Formulation:** \n\t\t- **We reformulate deepfake detection as a face comparison task guided by identity priors—a paradigm not explored in prior MLLM-based detection methods**. This enables the model to detect subtle forgery artifacts by comparing a query image against the known facial characteristics of a specific individual (\"VIP\"). This shift in perspective is itself a conceptual innovation.\n\t\t\n\t2. **Comprehensive and Custom Data Construction Pipeline:** \n\t\t- Unlike existing methods that rely on off-the-shelf LLMs or generic annotations, **we introduce a dedicated facial annotation model trained to extract precise identity, attribute, and similarity signals from faces.** This allows us to generate high-quality, identity-centric training data that explicitly teaches the MLLM to reason about facial consistency—a critical capability missing in standard training setups.\n\t\t\n\t3. **Progressive Learning Strategy**:\n\t\t- We design a three-stage progressive learning framework—facial attribute learning → general face comparison → VIP-specific customization—that gradually equips the MLLM with the fine-grained facial understanding necessary for personalized detection. This structured curriculum addresses the MLLM’s initial lack of facial expertise and ensures effective adaptation to the target task. \n\t\t- As demonstrated in **Table C3**, naive end-to-end training (i.e., standard MLLM fine-tuning) performs significantly worse, confirming that our improvements are not due to scale or data alone, but to our tailored methodology.\n\n- Finally, while our method is partially inspired by Cross Attention [3, 4] and Yo’LLaVA [5] in the design of certain modules, there is no overlap with these works in terms of target tasks or pipeline. We clarify as follows:\n\t1.\t**Cross Attention [3, 4]**: We use cross-attention to compare two facial images. However, this doesn’t lessen our method’s novelty, **since cross-attention is just a standard deep-learning building block**.\n\t2.\t**Yo’LLaVA [5]**: We propose the use of a VIP token to capture specific user identity information, which is used to discriminate between the subtle identity shifts, AI-generated or real. In contrast, Yo’LLaVA focuses on enhancing model performance for conversational tasks (such as certain personalized conversational tasks). To further clarify the distinction, we provide experimental results in Table C3, where Qwen-2.5-VL-7B serves as the baseline. The significant improvement we observe strongly verifies the novelty and effectiveness of our approach.\n\n**TABLE C3:** A comparison (AUC (%)) between our method, naively training MLLM, and Yo’LLaVA. Our method shows the best results of detection.\n\n| **Method (AUC (%))**      | **BlendFace** | **InSwap** | **Arc2Face** | **PuLID** | **Average** | **Improvement (+)** |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: |\n| Qwen-2.5-VL-7B (Baseline) |     51.00     |   49.79    |    49.85     |   50.30   | 50.24       | + 0.0               |\n| Naive Training MLLM       |     71.96     |   59.10    |    62.12     |   68.18   | 65.34       | + 15.10             |\n| Yo'LLaVA [5]              |     65.32     |   56.56    |    62.85     |   66.61   | 62.84       | + 12.60             |\n| **VIPGuard (Ours)**       |   **99.48**   | **96.40**  |  **98.05**   | **98.96** | **98.23**   | **+ 47.99**         |\n\n[3] Transformer, NIPS 2017.\n\n[4] CAT, ICME 2022.\n\n[5] Yo'llava. NIPS 2024.\n\n---\n> **Q4:**\n> \n> \n> Does our detector stay robust after typical image degradations—blur, Gaussian noise, and compression?\n\n\n**R4:**\n\n\nThank you for the insightful comment. We have added a robustness study (Table C4-1) showing that VIPGuard remains highly effective under common degradations. ** Because the detector relies on high-level semantic features such as facial information, it is inherently less sensitive to these low-level distortions.** The degradation settings are listed in Table C4-2.\n\n\n**TABLE C4-1:** VIPGuard's performance  (AUC %) under image degradations. Higher levels indicate stronger degradation. The results represent the average performance of VIPGuard in detecting BlendFace, InSwap, Arc2Face, and PuLID.\n\n| **Level** | **Gaussian Noise Color** | **Gaussian Blurring** | **JPEG Compression** |\n| - | :-: | :-: | :-: |\n| None      |          98.23           |         98.23         |        98.23         |\n| 1         |          97.07           |         98.17         |        98.03         |\n| 2         |          96.53           |         98.16         |        98.10         |\n| 3         |          94.12           |         98.05         |        97.78         |\n\n**TABLE C4-2:** Degradation settings included: Gaussian noise ( $\\mathcal{N}(0,\\sigma^{2})$) applied in YCbCr space; Gaussian blur defined by kernel ($K$) and standard deviation ($\\sigma$); and JPEG compression with quality factor (QF).\n\n| **Level** | **Gaussian Noise Color** | **Gaussian Blurring** | **JPEG Compression** |\n| - | :-: | :-: | :-: |\n| 1         |       $\\mathcal{N}(0, 8^2)$        |       $K=(7,7), \\sigma=1$       |        QF 90         |\n| 2         |       $\\mathcal{N}(0, 11^2)$       |      $K=(13,13), \\sigma=2$      |        QF 60         |\n| 3         |       $\\mathcal{N}(0, 18^2)$       |      $K=(21,21), \\sigma=3$      |        QF 30         |\n\n---\n> **Q5:**\n> \n> \n> 1. Are there any failure modes you observed that are worth mentioning?\n> \n> \n> 2. Please discuss potential adaptive forgery strategies that may render the proposed detection method less effective.\n\n\n**R5:**\n\n\nThank you for your valuable comment. We provide a brief explanation below. \n1. **Significant variations** in head pitch and yaw angles, such as noticeable rotation of more than 50°, affect facial features and identity information. However, this limitation is shared by all face-based methods, including face recognition systems.\n2. **Significant age differences** between training and test data. For example, training is based on younger facial images, but testing involves older facial images. However, this issue, stemming from notable identity shifts, can be effectively mitigated by expanding the dataset.\n\n**Inspired by your comment, we plan to address these potential failure cases in future work by incorporating 3D facial information, learning identity-related temporal cues from videos, and exploring other complementary sources of information.** We will include additional analysis related to these aspects in the revised version."}, {"review_id": "FdjREHZySn", "reviewer": "Reviewer_jxQd", "summary": "The paper proposes a novel and highly relevant approach which redefines personalized deepfake detection using fine-grained detection mechanisms along with MLLMs. The authors also introduce a new benchmark VIP-Bench as a tool for evaluating these types of identity-informed deepfakes, with ablation studies of multiple types of deepfake generation.\n\nThe paper redefines deepfake detection as a fine-grained face recognition task specifically tailored for \"VIP individuals\" whose authentic facial data is known. This contrasts with most existing general-purpose deepfake detectors that do not leverage identity-specific prior knowledge.\n\nVIPGuard Framework: The authors also introduce VIPGuard, a unified multimodal framework designed for personalized deepfake detection and explanation. It incorporates both global identity information and detailed structural facial attributes of the VIPs.\n\nThree-Stage Training Pipeline: VIPGuard operates through a progressive fine-tuning process:\n\n1. Facial Attribute Learning: An MLLM (specifically Qwen-2.5-VL-7B) is fine-tuned on a Facial Attribute Description Dataset (DFA) to enhance its understanding of detailed and structural facial attributes.\n\n2. Identity Discrimination Learning: The model undergoes further fine-tuning using an Identity Discrimination Dataset (DID) to learn to distinguish subtle differences between highly similar faces, including real and fake variations, by reasoning over global and local facial priors.\n\n3. User-Specific Customization: A lightweight, learnable VIP token is introduced to model the unique characteristics of a target VIP's face identity, enabling personalized and explainable deepfake detection. This stage refines the model's ability to perceive and distinguish the target VIP user.\n\nVIPBench Benchmark: To rigorously evaluate the method, the paper introduces VIPBench, a comprehensive identity-aware benchmark. This dataset fundamentally differs from conventional deepfake benchmarks by explicitly focusing on scenarios where prior knowledge of the target individual is available. VIPBench includes 22 unique target identities and a total of 80,080 images generated by 14 state-of-the-art manipulation techniques (7 face-swapping and 7 entire-face synthesis methods), providing a diverse and realistic evaluation setting.", "questions": "1. Computational Resource Reporting:\n\nQuestion/Suggestion: The NeurIPS Paper Checklist indicates \"No\" for providing sufficient information on computer resources needed to reproduce experiments, with the justification that \"Detailed compute specifications (hardware type, memory usage) are not included in this submission\" due to space considerations. To enhance reproducibility, please provide comprehensive details on the computational resources used for training and evaluation.  It is worth curating current included content for the sake of providing this detail, in order to facilitate reproducibility.\n\nProposal for improvement: This should include the specific type of GPUs (e.g., NVIDIA A100, V100, number of cards), total GPU memory used, and an estimate of the training time per stage (in hours/days) and total inference time for the VIPBench dataset. This level of detail is crucial for other researchers to gauge the feasibility of reproducing your work.\n\n2. Statistical Significance and Variability Analysis:\n\nQuestion/Suggestion: The paper's checklist states \"No\" for reporting error bars, noting that full variability analysis with standard deviations will be in the supplementary material. However, the included appendix does not appear to contain these standard deviations or error bars for the ablation studies or main performance tables. Please include standard deviations or appropriately defined error bars for all key experimental results, especially in Table 1 (ablation study) and Tables 1-3 in the main paper.\n\nProposal for improvement: Clearly state the number of independent runs used to calculate these statistics and how the error bars are defined (e.g., standard deviation or standard error). This will provide a more robust assessment of the method's consistency and reliability.\n\n3. Discrepancy in Safeguards Documentation:\n\nQuestion/Suggestion: The NeurIPS Paper Checklist indicates \"NA\" for safeguards, stating that the work \"does not release any pretrained models or scraped datasets with high misuse risk\". However, Section A of the appendix explicitly details safeguards for the VIPBench dataset, including a request form and manual review to prevent misuse. This presents a contradiction between the checklist and the appendix's content. Please clarify this inconsistency and update the checklist to accurately reflect the responsible data release mechanism you have implemented for VIPBench.\n\nProposal for improvement: If safeguards are in place for the dataset, the answer to the checklist question \"Does the paper describe safeguards...?\" should be \"Yes\", referencing the detailed explanation in the appendix.\n\n4. Usage and Selection of Facial Recognition Models:\n\nQuestion/Suggestion: In Stage 2 (Identity Discrimination Learning), the paper mentions employing \"face recognition models to provide face similarity scores\". It's unclear from the text whether all three referenced models (CosFace, ArcFace, TransFace) are used simultaneously, or if a specific one was chosen, or how they were combined if multiple were utilized. Could you please specify which facial recognition model(s) were ultimately used to generate the \"face similarity scores\" and provide a brief rationale for this choice or combination? Clarifying the specific model and its integration will provide better insight into the technical implementation of Stage 2 and its reliance on external facial priors.\n\n5. Novelty of \"Fine-Grained Face Recognition\" Formulation:\n\nQuestion/Suggestion: The paper states it \"reforms forgery detection as a fine-grained face recognition task\" for the first time, and also mentions existing \"identity-aware detection methods\", including ICT-Ref, which it critiques for \"fail[ing] to fully utilize the detailed identity-specific information\". Could you further elaborate on the specific aspects that make your reformulation of deepfake detection as a fine-grained face recognition task fundamentally novel and distinct from previous \"identity-aware\" approaches?\n\nProposal for improvement: Explain precisely how your \"fine-grained\" approach differs in its conceptualization or technical execution from prior identity-aware methods that also aimed to leverage identity information. This will help clarify the unique contribution of your core problem formulation beyond simply utilizing more detailed facial attributes.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths---\n\nResearch Quality:\n\nNovelty and Originality: The paper introduces a novel approach to deepfake detection by focusing on \"VIP individuals\" and leveraging prior knowledge of their authentic facial identities. This is a significant departure from most existing general-purpose deepfake detection methods that overlook this valuable identity-specific information. It uniquely reforms forgery detection as a fine-grained face recognition task.\n\nTechnical Soundness: \n\nThe proposed VIPGuard framework employs a well-structured three-stage pipeline:\nStage 1: Facial Attribute Learning fine-tunes a Multimodal Large Language Model (MLLM) (specifically Qwen-2.5-VL-7B) to understand detailed facial attributes.\nStage 2: Identity Discrimination Learning enables the model to distinguish subtle differences between highly similar faces, including real and fake variations, by reasoning over global and local facial priors.\nStage 3: User-Specific Customization models unique characteristics of the target face identity using a lightweight VIP token, facilitating personalized and explainable deepfake detection. This progressive fine-tuning approach appears logically sound and addresses a clear gap identified in existing methods.\n\nStrong Experimental Results:\n\nThe paper presents extensive experiments demonstrating that VIPGuard significantly outperforms existing general deepfake detectors and other ID-aware methods.\n\nIt shows superior detection performance for both face-swapping and entire face synthesis techniques on the newly introduced VIPBench dataset, achieving high AUC and low EER values.\n\nThe proposed framework is shown to consistently outperform other MLLM-based methods in accuracy.\n\nThe ablation studies confirm the effectiveness and necessity of each of the three training stages, highlighting their synergistic contribution to VIPGuard's high performance.\n\nThe model also demonstrates explainability, providing human-understandable reasoning for its predictions.\nReproducibility: The authors commit to providing comprehensive experimental details in the supplementary material. The appendix indeed includes detailed implementation settings such as the backbone model, input image size, optimizer, learning rate, batch sizes, gradient accumulation, training epochs, and the use of mixed-precision computation within the Swift framework.\n\nNew Dataset (VIPBench): \n\nThe creation of VIPBench is a significant contribution, providing a comprehensive, identity-aware benchmark specifically designed for personalized deepfake detection. The appendix thoroughly details its construction, including the Facial Attributes Description Dataset (DFA), Identity Discrimination Dataset (DID), and VIPEval, along with data sources, preprocessing steps, VQA data construction, and the diverse range of 14 state-of-the-art generation techniques used\n\nWeaknesses---\n\nStatistical Significance: The submitted paper checklist indicates \"No\" for statistical significance (error bars) and justifies it by space constraints, promising full variability analysis in the supplementary material and public code release. However, as noted in our conversation, the provided appendix does not yet contain these standard deviations or error bars for ablation studies. \n\nSafeguards Checklist vs. Content: The checklist surprisingly marks \"NA\" for safeguards, implying no high-risk assets requiring them. This is directly contradicted by the included appendix which details explicit safeguards for the VIPBench dataset, including a request form and manual review to prevent misuse. This inconsistency between the checklist answer and the appendix's content should be reconciled.\n\nCompute Resources Information: In the paper's checklist, in response to the question \"For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?\", the authors marked \"No\".\n\nThe NeurIPS guidelines recommend that papers indicate the type of compute workers (CPU or GPU), internal cluster or cloud provider, relevant memory and storage, and the amount of compute required for individual experimental runs and total compute. This information is currently lacking in the submission", "was_revised": true, "final_justification": "The paper's clarity issues were addressed by the authors satisfactorily.  However, the main concern with this paper, which led to the initial score, was not sufficiently addressed: novelty.  The authors contend that simply combining an identity aware technique and adding a semantic layer from MLLM use constitute a major leap forward, but the empirical results don't support that conclusion.  \n\nAdditionally, if the use of generative workflows is the major intervention to existing technique they should center the paper on that discovery rather than an ensemble.", "rebuttal": "We sincerely thank the reviewer for **the positive evaluation of our work, including its novelty, technical soundness, strong experimental results, and the contribution of the VIPBench dataset**. Below, we provide detailed responses to address the concerns raised and clarify any remaining questions.\n\n---\n\n> **Q1:**\n> \n> \n> Which face-recognition model(s) did the authors actually use to compute similarity, and why were they chosen?\n\n\n**R1:**\n\n\nThank you for your insightful comment. Table D1 reports the comparative results across the tested face-recognition models. Our ablation study shows that **TransFace** gives the best performance, so we used it empirically. We will add this explanation to the revised manuscript.\n\n**TABLE D1:** Performance (AUC %) of VIPGuard using different face models as components.\n\n| **Face Model** | **BlendFace** | **InSwap** | **Arc2Face** | **PuLID** | **Average** |\n| :------------- | :-----------: | :--------: | :----------: | :-------: | :---------: |\n| CosFace        |     95.12     |   83.32    |    83.31     |   91.74   |    88.37    |\n| ArcFace        |     98.14     |   83.17    |    94.41     |   97.88   |    93.40    |\n| TransFace      |   **99.48**   | **96.40**  |  **98.05**   | **98.96** |  **98.23**  |\n\n---\n> **Q2:**\n> \n> \n> What is truly novel about casting deepfake detection as a fine-grained face-recognition task, and how does this differ from earlier identity-aware methods such as ICT-Ref?\n\n\n**R2:**\n\n\nThank you very much for your insightful comment. Our method differs from previous identity-aware detection approaches in the following key aspects:\n1. **User-specific identity modeling:**\n    Our method uniquely models the protected user’s face by directly leveraging full facial information for detection. In contrast, **prior methods focus on inconsistencies introduced by face-swapping operations rather than on the specific identity itself**. For example, ICT-Ref detects discrepancies between inner and outer face identity embeddings, while Diff-ID compares differences before and after self-swapping. However, fully generative forgery methods often preserve identity consistency, causing these inconsistency-based methods to degrade significantly. As shown in Table 2 of our paper, our method maintains strong performance even under such conditions, highlighting the strong generalization ability of our method across different forgery types.\n    \n2. **Use of local facial attributes:**\n    Unlike previous approaches that treat the face as a whole and rely mostly on global features, our method explicitly learns and utilizes local facial attributes. This enables it to capture fine-grained details that forgery methods struggle to replicate, leading to improved detection performance through the combination of global and local cues. \n    \n3. **Leveraging MLLMs for semantic reasoning:**\n    By leveraging MLLMs to model high-level semantic facial attributes, our method effectively captures identity-related features and benefits from the strong generalization ability of large models. Moreover, the textual description of facial attributes enhances our model’s interpretability, which previous methods do not achieve.\n\n---\n> **Q3:**\n> \n> \n> Could the authors provide the compute-resource details for reproducibility?\n\n\n**R3:**\n\n\nThank you for highlighting this point. In the revision, we will add a table describing our hardware (multi-GPU workstation, multi-core CPU, and system memory) and software environment (recent Linux distribution, mainstream deep-learning framework with the corresponding CUDA/cuDNN stack). \n\n\n---\n> **Q4:**\n> \n> \n> The appendix still lacks error bars or standard deviations for the experiments.\n\n\n**R4:**\n\n\nThank you for your comment. In the revision, we will append error bars or standard deviation values to experiments, and briefly describe the procedure used to compute them. This will make the reported results more transparent and statistically sound.\n\n\n\n---\n> **Q5:**\n> \n> \n> The checklist lists safeguards as “N/A,” yet the appendix outlines explicit safeguards for VIPBench, creating a contradiction that needs clarification.\n\n\n**R5:**\n\n\nThank you for catching this inconsistency. We will update the checklist to “Yes” and ensure its description matches the safeguards already detailed in the appendix in the revised manuscript."}]

Author rebuttal:

Meta-review: This paper proposes a personalised and explainable deepfake detection framework called VIPGuard tailored for known individuals. The proposed method leverages a multimodal large language model to capture fine-grained facial attributes, perform identity-level discriminative learning, and incorporate user-specific customisation for robust detection. To evaluate performance, the paper also proposes VIPBench, which is a benchmark for identity-aware deepfake detection covering multiple generation techniques. Experiments demonstrate that the proposed method outperforms existing detectors.

Four expert reviewers initially raised several concerns including limited novelty, insufficient experimental results and ablation studies, heavy reliance on VIP user annotation via Gemini or similar APIs, lack of clarity on whether the proposed method generalizes to unseen forgery schemes and new identities, and the lack of failure case analysis. The rebuttal addressed most of these concerns resulting in two borderline accept, one accept and one borderline reject. However, the main concern about novelty has not been fully addressed and two out of four reviewers still consider the contribution to have limited novelty. Despite this, considering the positive aspects of the paper (the introduction of a new benchmark for personalized deepfake detection and demonstrated improvements over existing methods) the AC believes that the strengths outweigh the weaknesses and recommends acceptance.
Decision: Accept (poster)