# Integrating perceptual quality analysis and caption-based features for robust deepfake video detection

Hessen Bougueffa Eutamene, Wassim Hamidouche, Mamadou Keita, Abdelmalik Taleb-Ahmed, Abdenour Hadid

<figure><img src="image_1.png" /></figure>

## To cite this version:

Hessen Bougueffa Eutamene, Wassim Hamidouche, Mamadou Keita, Abdelmalik Taleb-Ahmed, Abdenour Hadid. Integrating perceptual quality analysis and caption-based features for robust deepfake video detection. Computers and Electrical Engineering, 2025, 128 (Part A), pp.110699. 10.1016/j.compeleceng.2025.110699. hal-05254097

HAL Id: hal-05254097

https://hal.science/hal-05254097v1

Submitted on 10 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

# Integrating perceptual quality analysis and caption-based features for robust deepfake video detection☆

Hessen Bougueffa Eutamene<sup>a ID,*</sup>, Wassim Hamidouche<sup>b ID</sup>, Mamadou Keita<sup>a ID</sup>, Abdelmalik Taleb-Ahmed<sup>a ID</sup>, Abdenour Hadid<sup>c ID</sup>

## ARTICLE INFO

### Keywords:

- Deepfake video detection

- Perceptual quality analysis

- Vision-language models (VLMs)

- LoRA fine-tuning

- AI-generated video forensics

- Transformer-based deepfake detection

## ABSTRACT

Inspired by the benefits of both linguistic representations as well as visual quality analysis, we propose a multimodal framework for deepfake detection with reliable accuracy. Our method acquires low-level perceptual features from frames of video, including contrast, brightness, and sharpness, and computes artifact scores for artificial anomaly detection. In parallel, our method produces descriptive frame-level captions, summed up for generating video-level summaries for capturing contextual coherence. For training the model, our method leverages the FaceForensics++ dataset, comprising several techniques of deepFake manipulation, including DeepFake, Face2Face, FaceSwap, as well as NeuralTextures. Metadata comprised of quality measurements, the artifact scores, as well as textual captions, is tokenized, processed by a DeepSeek V2 Lite model, fine-tuned by the Low-Rank Adaptation (LoRA) procedure, as the backbone for classification. We test the proposed model on both intra-dataset as well as cross-dataset settings, demonstrating good accuracy as well as impressive generalization ability. Our method achieves a 96.51% accuracy for classifying, by integrating low-level perceptual as well as high-level semantic reason, effectively marking manipulated content as well as upholding the integrity of the medium in a world of increasingly evolving generative methods. The employed source code as well as dataset preprocessing scripts for our work are made available for public access on: https://github.com/HessenUPHF/perceptual-caption-detection.

## 1. Introduction

The rapid evolution of generative models has transformed video synthesis, enabling the effortless creation of remarkably realistic content. While recent advancements in generative models have enabled innovation across entertainment, education, and communication, they also raise serious concerns about media integrity and trustworthiness. Powerful techniques such as GANs [1], DeepFake frameworks [2], and Diffusion Models [3] can now convincingly mimic authentic video content, deceiving both human observers and traditional detection systems.

The significance of synthetic video extends well beyond scholarly curiosity, prompting serious issues across journalism, national security, and digital forensic communities. Studies demonstrate that individuals find it difficult to distinguish between real, authentic

E-mail addresses: Hessen.BougueffaEutamene@uphf.fr (H.B. Eutamene), whamidouche@gmail.com (W. Hamidouche), mamadou.keita@uphf.fr (M. Keita), abdelmalik.taleb-ahmed@uphf.fr (A. Taleb-Ahmed), abdenour.hadid@ieee.org (A. Hadid).

https://doi.org/10.1016/j.compeleceng.2025.110699

Received 18 April 2025; Received in revised form 24 June 2025; Accepted 1 September 2025

Available online 11 September 2025

0045-7906/© 2025 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

<footnote><sup>a</sup> Laboratory of IEMN, CNRS, Centrale Lille, UMR 8520, Univ. Polytechnique Hauts-de-France, Valenciennes, 59300, France</footnote>

<footnote><sup>b</sup> Khalifa University, Abu Dhabi, United Arab Emirates</footnote>

<footnote><sup>c</sup> Sorbonne Center for Artificial Intelligence, Sorbonne University Abu Dhabi, Abu Dhabi, United Arab Emirates</footnote>

<footnote>☆ This article is part of a Special issue entitled: 'idagi' published in Computers and Electrical Engineering.</footnote>

<footnote>* Corresponding author.</footnote>
video footage and artificially generated video, their accuracy being only marginally higher than randomness. With the rapid development of generative models, however, most current methods prove insufficient—they frequently rely on superficial pixel patterns or hand-tuned features easily bypassed by powerful deepfake systems [4,5]. This increasing challenge highlights the acute necessity for more stringent, adaptable detection tools able to match the changing dynamics of synthetic media.

Recent advances in GAN-based architectures have significantly improved the quality of deepfake videos, making detection increasingly challenging. Several face manipulation frameworks, such as AttGAN [6], DeepFaceLab [7], Faceswap [8], STGAN [9], and StarGAN [10], have enabled high-resolution facial synthesis, face-swapping, and attribute-based editing. These models have been widely used to create misleading digital content, including the infamous deepfake of former U.S. President Barack Obama [11], where an AI-generated version of his face and voice was used to spread fabricated statements. Such incidents highlight the potential misuse of deepfake technology in disinformation campaigns, political propaganda, and digital fraud.

The traditional deepfake detection methods, such as Convolutional Neural Network (CNN) and Vision Transformer (ViT) [12], mostly concentrate on low-level visual artifacts, not necessarily identifying the inconsistency of meaning or scene reasoning. But with the advent of large-scale Vision-Language Model (VLM), such as Contrastive Language-Image Pre-training (CLIP) [13], Bootstrapped Language-Image Pre-training 2 (BLIP2) [14], and SimVLM [15], there is a new paradigm of deepfake detection, aligning visual perception with linguistic representation. Since they can combine visual and textual information, offering a richer semantic representation of manipulated content, these models are becoming increasingly popular for applications like image captioning, video summarization, and deepfake detection. Transformer-based models have also demonstrated great promise for deepfake detection using contextual reasoning [12].

Aside from semantic and deep learning-based methods, investigators have also considered perceptual quality metrics to identify subtle artifacts imposed by generative models. No-reference image quality assessment techniques, including BRISQUE [16] and NIQE [17], measure distortions of contrast, sharpness, and noise patterns that commonly disclose synthetic content. It has recently been shown that incorporating such perceptual cues into their respective detection models can enhance insensitivity to compressed or adversarially amplified deepfakes. For instance, DeepFidelity [18] assesses perceptual forgery fidelity over several quality domains in order to distinguish between authentic and manipulated content. These methods prove that low-level visual artifacts provide excellent complements for semantic-based reasoning for detecting manipulated content.

To address these challenges in detecting deepfakes, we propose a novel framework that integrates both visual quality analysis and linguistic representation. This combination is designed to enhance both the accuracy and general robustness of the detection system. Our main contributions can be summarized as follows:

* We propose a unified framework that combines visual quality analysis and linguistic representation to enhance the detection of manipulated videos in a multimodal manner, addressing the limitations of single-modality approaches.

* We employ a dual-level captioning strategy, where frame-level captions capture fine-grained details of individual frames, allowing for precise identification of visual artifacts and inconsistencies, while video-level captions provide a broader contextual understanding of the entire video, enabling the detection of logical inconsistencies across frames.

* We conduct a comprehensive quantitative analysis of low-level quality attributes (sharpness, contrast, brightness, and artifact score) often overlooked in deepfake detection. By systematically evaluating these attributes, our model captures subtle distortions and statistical anomalies that distinguish manipulated videos.

* To validate our approach, we compare it with recent state-of-the-art methods, achieving a detection accuracy of 96.51%.

* Our framework is modular and extensible, allowing for seamless integration of future advancements in generative video technologies and detection methods, making it a scalable solution for evolving deepfake detection challenges.

By integrating these components, our framework addresses the limitations of traditional detection systems, offering both high accuracy and interpretability. Experimental results demonstrate the robustness of our approach across diverse datasets and generative models, paving the way for reliable AI-driven solutions in combating synthetic media.

The remainder of this paper is organized as follows. Section 2 reviews related work on deepfake detection, covering both spatial and frequency-based approaches, as well as recent multimodal detection techniques. Section 3 describes our proposed methodology, detailing the integration of visual quality assessment, vision-language modeling, and deep feature extraction. Section 4 summarizes the experimental results, evaluates the performance of our model against state-of-the-art baselines, and analyzes classification metrics. Section 5 provides a detailed analysis of our results, including confidence score evaluations and cross-dataset generalization. Section 6 discusses the ablation study, highlighting the impact of removing different feature components on detection accuracy. Finally, Section 7 concludes the paper and outlines directions for future research.

## 2. Related work

This section explores the various methodologies and advancements in deepfake detection, emphasizing key developments and diverse approaches used to identify manipulated videos. It provides a foundational understanding of existing research efforts and their implications for improving video authenticity verification.
## 2.1. Deepfake creation techniques

The creation of highly realistic and coherent video sequences has been made possible by the rapid growth of generative models, significantly advancing the field of video synthesis. A variety of architectures, including Transformer, Diffusion Models, Variational Autoencoder (VAE), and Generative Adversarial Networks (GAN), have played a key role in these developments. The quality, realism, and controllability of AI-generated videos have all improved as a result of these strategies.

By implementing a three-dimensional sparse attention mechanism, GAN-based models, such as Godiva [19], have made significant improvements in open-domain text-to-video creation. More recent transformer-based models, such as CogVideo [20], extend text-to-image synthesis models to video creation by using a hierarchical frame-rate training technique to enhance motion coherence.

Diffusion models have recently emerged as a powerful alternative to GANs for generating high-quality videos. Methods such as NUWA [21] and NUWA-Infinity leverage 3D transformer architectures to synthesize coherent video clips while maintaining semantic and spatial consistency. These models allow for generating arbitrarily long videos with fine-grained control over motion and content.

Recent advancements include Latte [22], a latent diffusion transformer designed for high-fidelity long video generation, and VideoCrafter2 [23], which overcomes data limitations by integrating open diffusion models for scalable video generation. These models demonstrate improved efficiency in generating motion-consistent and high-resolution video sequences.

Another notable development is the release of VideoGPT [24], which uses transformers combined with a Vector Quantized Variational Autoencoder (VQ-VAE) (Vector Quantized Variational Autoencoder) framework to create video sequences from acquired discrete latent representations. High-fidelity video outputs can now be produced more accurately by modeling long-range temporal relationships, as demonstrated by models such as Video LDM [25].

Additionally, latent diffusion-based models such as ModelScope [26] Text-to-Video have demonstrated superior performance in scalable video generation by leveraging large-scale web-collected video datasets. These models employ text-conditioned diffusion processes to create videos that accurately reflect input textual prompts, offering new capabilities for controllable video generation.

As video generation models continue to evolve, their increasing sophistication poses challenges for deepfake detection systems. The enhanced realism, improved temporal consistency, and refined motion synthesis capabilities of these generative models necessitate the development of advanced detection frameworks capable of distinguishing AI-generated content from authentic videos. Traditional classifiers trained on older deepfake generation techniques may struggle to generalize to newer, high-fidelity manipulation methods.

## 2.2. Deepfake detection techniques

Deepfake detection techniques have evolved significantly, leveraging both traditional and deep learning-based approaches. Early methods primarily focused on detecting inconsistencies in facial features, unnatural expressions, and frame-by-frame analysis. CNN and ViT have been widely employed to identify visual artifacts introduced by generative models. Notable architectures, such as Xception [27] and Efficientnet [28], have demonstrated strong performance in identifying manipulated content by analyzing spatial and temporal inconsistencies. Other works, such as MesoNet [29] further contributed to establishing standard baselines and datasets for deepfake detection.

Recent advancements in the field have proposed alternative architectural strategies for deepfake detection. One method utilizes facial geometry — specifically golden ratio priors — to guide frame selection and combines CNN-based backbones such as VGG19 and EfficientNet with capsule networks, achieving competitive results on Celeb-DF and DFDC datasets [30]. Another study proposes a dual-stream CNN that processes both spatial and frequency representations to detect generative artifacts in manipulated content [31]. These contributions reflect ongoing efforts to improve robustness through diverse modeling approaches.

However, these methods often struggle with generalization to unseen deepfake techniques as generative models continue to advance. To overcome these limitations, researchers have explored hybrid approaches that incorporate additional contextual and statistical features, such as quality metrics and metadata, to improve classification accuracy. These enhancements help detect manipulations that might otherwise go unnoticed in purely visual analyses.

Recent studies have introduced advanced models, including transformers and diffusion-based architectures, to enhance deepfake video detection. Zhang et al. [32] proposed a Spatiotemporal Dropout Transformer (STD-Transformer), which integrates random patch dropout across both spatial and temporal dimensions. This approach effectively captures subtle temporal inconsistencies and spatial artifacts in manipulated videos, significantly improving the model's robustness and generalization across diverse deepfake datasets.

Other works, such as [33], utilized CNN-based methods trained on mixed datasets, showing high accuracy in detecting deepfake content. Similarly, MesoNet, proposed by Afchar et al. [29], is a compact CNN-based model designed to identify facial manipulation using deep learning.

Hybrid models combining Recurrent Neural Networks (RNNs) with CNNs have been proposed to enhance sequential data processing for deepfake detection. For example, Liu et al. [34] introduced a lightweight 3D CNN with a Channel transformation (CT) module, which significantly improved learning efficiency and deepfake detection accuracy.

Another promising direction includes transfer learning-based approaches. Ranjan et al. [35] explored the use of transfer learning to enhance deepfake detection performance across various manipulation techniques.

Moreover, research has shown that generative models often struggle to maintain consistent perceptual quality across frames, making temporal and spatial inconsistencies valuable indicators of manipulated content. Liang et al. [36] proposed a two-stream
deepfake detection framework with hierarchical supervision, which enables robust detection by learning spatial and temporal artifact patterns across frames.

Additionally, Khalid et al. [37] proposed OC-FakeDect, which balances deepfake detection across different manipulation styles by learning from real-world examples.

AIGVDet, proposed by Bai et al. [38], is a spatio-temporal CNN that captures motion discrepancies between real and AI-generated content by analyzing both RGB frames and optical flow maps.

Similarly, Ji et al. [39] introduced the Dual-Branch 3D Transformer (DuB3D), which enhances detection accuracy by integrating motion information with spatial-temporal features.

Wang et al. [40] proposed an adaptive fusion strategy that integrates spatial and frequency-based features to enhance deepfake detection accuracy. Their method employs a multi-branch architecture that extracts deepfake-related artifacts at different feature levels, improving its robustness across various manipulation techniques.

Naskar et al. [41] introduced a deepfake detection framework based on deep feature stacking and meta-learning. Their model demonstrated high accuracy on Celeb-DF (V2) and FaceForensics++ datasets, showing that stacking multiple feature representations enhances deepfake detection robustness. Feature selection is performed using a ranking-based method, and classification is handled by a multi-layer perceptron (MLP) meta-learner.

Transformer-based detection techniques have gained significant focus, improving generalization across different datasets. DeMamba, a plug-and-play transformer-based module, explores both local and global spatial-temporal inconsistencies to improve classification accuracy [42].

Liu et al. [43] introduced Diffusion-generated Video Detection (DIVID), a method that detects diffusion-generated videos by analyzing temporal and spatial artifacts through reconstruction errors, thereby improving robustness in forensic detection.

*Vision-language and multimodal approaches.* With the increasing sophistication of deepfake generation techniques, researchers have turned to VLM to enhance detection capabilities. Vision-language models, such as CLIP [13] and BLIP2 [14], bridge the gap between image and textual representations, enabling deeper semantic analysis of manipulated content.

Recent works have leveraged multi-modal learning to analyze both visual artifacts and textual inconsistencies in AI-generated media. Khan et al. [44] investigated the adaptation of pre-trained CLIP models for deepfake detection, showing that prompt tuning strategies significantly improve detection accuracy. Their method demonstrated strong generalization across different manipulation types, highlighting the potential of leveraging vision-language contrastive learning for forensic analysis.

Recent studies have introduced vision-language forensic models specifically tailored for video deepfake detection. Lin et al. [45] explored reprogramming VLMs, such as CLIP, for general deepfake detection by using learnable visual perturbations and adaptive text prompts. Their method achieved superior cross-dataset and cross-manipulation performance with minimal trainable parameters.

Similarly, Jia et al. [46] investigated whether Multimodal Large Language Model (MLLM) such as ChatGPT-4 Vision could be adapted for deepfake detection. Their experiments demonstrated that MLLMs can effectively analyze inconsistencies between visual and textual content when guided by carefully engineered prompts, offering a new perspective on AI-driven media forensics.

While VLMs show promise in detecting multimodal inconsistencies, their general-purpose training limits sensitivity to subtle deepfake artifacts. Most VLMs are trained for general-purpose tasks and may lack the sensitivity needed to capture subtle semantic manipulations in deepfakes. Fine-tuning these models for forensic applications remains a crucial area of ongoing research, particularly to address adversarial robustness and video-specific artifacts.

Overall, despite significant progress across CNN-based, transformer-based, and multimodal approaches, deepfake detection remains a rapidly evolving challenge. As generative models become more sophisticated and harder to detect, future detection frameworks must incorporate semantic reasoning, spatio-temporal analysis, and cross-modal consistency to ensure robust and generalizable performance.

## 2.3. Quality metric-based detection

Beyond deep learning-based techniques, researchers have explored the use of low-level quality metrics to detect deepfake manipulations. These methods analyze sharpness, contrast, brightness, noise levels, and artifacts to identify inconsistencies introduced during the generation process. Quality assessment models such as BRISQUE [16] and NIQE [17] have been widely used to evaluate the perceptual quality of images and videos.

Recent research has highlighted the importance of combining perceptual quality analysis with deep learning to improve deepfake detection. Peng et al. [18] proposed DeepFidelity, a perceptual forgery assessment framework that measures localized visual degradation to differentiate real and fake faces, demonstrating high accuracy across multiple datasets. In parallel, Anan et al. [47] proposed CAE-Net, a convolution- and attention-based ensemble model that jointly exploits spatial and frequency domain features for robust deepfake image detection.

Moreover, research has shown that generative models struggle to maintain consistent perceptual quality across frames, making temporal inconsistencies a valuable indicator of manipulated content. Concas et al. [48] introduced a quality-based artifact modeling approach that analyzes temporal artifacts in facial regions, such as the mouth and eyes, by computing quality measures for each frame and comparing subsequent frames. This method effectively captures subtle inconsistencies introduced by deepfake generation techniques, enhancing detection robustness against highly realistic manipulations.

Despite the effectiveness of quality metrics, these methods have limitations, such as sensitivity to compression artifacts and variations in dataset quality. However, when combined with deep feature representations, these quality-based cues significantly
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** Illustration of our model architecture for deepfake video detection. Metadata is generated via perceptual quality analysis (BRISQUE + ResNet50), while BLIP2 generates frame-level and video-level captions. These features are combined and processed by a LoRA-tuned DeepSeek V2 Lite model for classification.</figcaption></figure>

enhance the detection of manipulated media. Our proposed framework builds upon this insight by uniquely integrating low-level perceptual metrics with high-level semantic information derived from frame-level and video-level captions. Unlike prior methods that focus solely on visual artifacts or spatio-temporal features [32,38,42], or those leveraging vision-language models for classification [44,46], our method unifies both perceptual and semantic reasoning within a lightweight, LoRA-tuned LLM architecture. This dual-level integration offers improved robustness, interpretability, and generalization across datasets.

## 3. Proposed approach

Fig. 1 illustrates the architecture of our proposed deepfake detection framework. First, each video frame is analyzed using BRISQUE and ResNet-50 to extract perceptual quality metrics, including artifact score, sharpness, contrast, and brightness. Simultaneously, BLIP2 generates frame-level captions, which are aggregated into a video-level semantic summary. These multimodal features — visual and textual — are concatenated and passed to a LoRA-tuned DeepSeek V2 Lite model for binary classification. The entire pipeline combines frozen encoders with efficient low-rank adaptation for scalable and robust detection. Key components of the architecture include:

* **Visual Quality Analysis:** Pre-trained models, such as Brisque and Resnet-50, are used to evaluate frame-level quality metrics, including artifact presence, sharpness, brightness, and contrast. These low-level cues provide early signals of manipulation and degradation.

* **Semantic Caption Generation:** We employ BLIP2 to extract frame-level captions using a structured prompt. These captions reflect the semantic and contextual details of each frame. The frame-level captions are then aggregated into a video-level caption, offering a comprehensive summary of video content and aiding the detection of logical inconsistencies.

* **Multimodal Feature Fusion:** Both the extracted visual quality metadata and video-level caption are concatenated to form a unified multimodal representation, combining both low-level and high-level descriptors.

* **Classification with LoRA-Tuned LLM:** The concatenated features are tokenized and passed to DeepSeek V2 Lite, a vision-language model fine-tuned using Low-Rank Adaptation (LoRA). A binary classification layer is appended to determine whether the input video is real or fake.

### 3.1. Perceptual metadata generation

The first step involves generating metadata that captures low-level visual quality signals. These signals are extracted from individual video frames using BRISQUE and ResNet-50, the four low-level quality metrics — artifact score, sharpness, contrast, and brightness — are calculated as follows (see Table 1 for a summary):
<table><caption>**Table 1**

Quality metrics & artifact score calculation methods and normalization ranges.</caption><thead><tr><th>Metric</th><th>Calculation</th><th>Normalization</th><th>Tool</th></tr></thead><tbody><tr><td>Artifact score</td><td>BRISQUE (NSS-based)</td><td>[0, 100]</td><td>OpenCV</td></tr><tr><td>Sharpness</td><td>Laplacian variance</td><td>None</td><td>OpenCV</td></tr><tr><td>Contrast</td><td>Std. dev. of pixel intensities/255</td><td>[0, 1]</td><td>OpenCV</td></tr><tr><td>Brightness</td><td>Mean HSV (V)/255</td><td>[0, 1]</td><td>OpenCV</td></tr></tbody></table>

**Artifact score (α).** Computed using the BRISQUE (Blind/Referenceless Image Spatial Quality Evaluator) algorithm [16], which quantifies distortions (e.g., blur, noise) via natural scene statistics (NSS) in the spatial domain.

For each frame, we use OpenCV's QualityBRISQUE module with pre-trained models (brisque_model_lIVE.yml and brisque_range_lIVE.yml) to generate a normalized score ∈ [0, 100], where higher values indicate stronger artifacts.

**Sharpness (σ).** Measured as the variance of the Laplacian-filtered image edges, where higher values indicate sharper frames.

```
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
sharpness = cv2.Laplacian gray, cv2.CV_64F).var()
```

**Contrast (γ).** Defined as the normalized standard deviation of pixel intensities (∈ [0, 1]):

```
contrast = gray.std() / 255.0
```

**Brightness (β).** Computed as the mean value of the HSV Value channel, normalized to [0, 1]:

```
brightness = np.mean(hsv[: : , 2]) / 255.0
```

**Aggregation.** For a video with n frames, each metric is averaged across all frames to form the final metadata vector:

$$M = \left\{ \frac{1}{n} \sum_{i=1}^{n} \alpha_i, \frac{1}{n} \sum_{i=1}^{n} \beta_i, \frac{1}{n} \sum_{i=1}^{n} \gamma_i, \frac{1}{n} \sum_{i=1}^{n} \sigma_i \right\}$$

We specifically selected ResNet50 and BRISQUE for their complementary strengths and proven effectiveness in quality-based deepfake detection. ResNet50, a widely-used lightweight CNN, efficiently identifies subtle structural and texture anomalies common in manipulated videos, while its computational efficiency supports practical deployment. BRISQUE, a reference-free perceptual quality model, robustly quantifies unnatural artifacts like excessive blur, sharpness distortions, and irregular textures typically introduced by generative methods. Together, these models form a powerful combination capable of capturing both subtle visual anomalies and perceptual distortions, significantly enhancing the robustness of our deepfake detection framework.

## 3.2. Caption-based semantic understanding

To capture high-level semantic content, we generate frame-level captions Cᵢ using BLIP2, guided by the following structured prompt:

> "Provide a detailed analysis of the image quality, identifying any signs of manipulation or unnatural features".

These captions are aggregated by simple textual concatenation, preserving their chronological order to form a video-level caption.

$$C_{\text{video}} = \text{Aggregate}(C_1, C_2, \dots, C_n) \qquad (1)$$

This video-level caption offers a comprehensive linguistic perspective of the visual content.

## 3.3. Multimodal feature concatenation

We integrate both the metadata and the semantic caption representation into a single input sequence:

$$F = \text{Concat}(M, C_{\text{video}}) \qquad (2)$$

This multimodal representation captures both perceptual inconsistencies and contextual contradictions.
<table><caption>**Table 2**
Model parameter distribution: Trainable vs. Frozen.</caption><thead><tr><td>Category</td><td>Count</td></tr></thead><tbody><tr><td>Trainable parameters</td><td>2,195,458</td></tr><tr><td>Frozen parameters</td><td>8,064,724,480</td></tr><tr><td>Total parameters</td><td>8,066,919,938</td></tr><tr><td>Percentage trainable</td><td>0.03%</td></tr></tbody></table>

### 3.4. Parameter-efficient classification via LoRA

To adapt DeepSeek V2 Lite for binary classification, we employ LoRA, which introduces trainable low-rank matrices in the attention mechanism of the model. The original projections are:

$$Q = XW_Q, \quad K = XW_K, \quad V = XW_V \qquad (3)$$

LoRA updates the weight matrices via:

$$\Delta W_Q = B_Q A_Q, \quad \Delta W_K = B_K A_K, \quad \Delta W_V = B_V A_V \qquad (4)$$

The updated self-attention computations become:

$$Q' = X(W_Q + \Delta W_Q), \quad K' = X(W_K + \Delta W_K), \quad V' = X(W_V + \Delta W_V) \qquad (5)$$

Here, $B_Q$, $B_K$, $B_V$ and $A_Q$, $A_K$, $A_V$ are low-rank matrices with rank $r = 32$, and $\Delta W_*$ represents the trainable low-rank adaptation to the frozen weights. Only these LoRA parameters in the final transformer block are fine-tuned, while the rest of DeepSeek V2 Lite remains frozen.

### 3.5. Classification layer and output

The final step involves passing the tokenized multimodal features through DeepSeek V2 Lite and the appended classification layer:

$$T = Tokenizer(F), \quad y = DeepSeek(T) \qquad (6)$$

where $y \in \{\text{Real, Fake}\}$ represents the predicted label.

To perform binary classification, we formulate the input as a natural language instruction prepended to the multimodal features. Specifically, we use the prompt:

> "Analyze the following metadata and captions. Determine if the video is FAKE or REAL".

This prompt is followed by a combination of perceptual metadata (artifact score, brightness, contrast, and sharpness), multiple frame-level captions, and the aggregated video-level caption. These elements are concatenated and tokenized as input to the DeepSeek V2 Lite model for classification.

### 3.6. Model efficiency

The efficiency of our parameter tuning is illustrated in Table 2, which shows that only 0.03% of the parameters are trainable, making the framework computationally efficient and scalable. Fig. 2 illustrates the integration of LoRA into DeepSeek V2 Lite. It shows how trainable low-rank matrices are inserted into the Q, K, and V projections, modifying attention computations while preserving the frozen backbone.

## 4. Experimental setup

This section presents the experimental setup used to evaluate the proposed deep-fake detection framework. It describes the datasets and preprocessing pipeline, outlines the evaluation metrics, details the implementation configuration, and concludes with a description of the baseline models selected for comparison.
<figure><img src="image_5.png" /><figcaption>**Fig. 2.** LoRA fine-tuning in DeepSeek V2 Lite. The pre-trained transformer model remains frozen while LoRA introduces trainable low-rank matrices (Q, K, V) to adapt the model for deepfake detection. The final classification layer determines whether the input is real or fake.</figcaption></figure>

## 4.1. Datasets

In this work, we utilize the FaceForensics++ [33] dataset for training and validating our model. FaceForensics++ is a large-scale benchmark for deepfake detection, containing multiple manipulation techniques, including DeepFake, Face2Face, FaceSwap, and NeuralTextures. This dataset provides high-quality synthetic videos with varying levels of compression, making it a suitable choice for developing robust detection models.

For training, we selected 720 videos, comprising 360 real and 360 fake videos. The validation set consists of 140 videos, while the test set includes 140 videos. This controlled dataset split ensures balanced learning and evaluation. This dataset split aligns with standard practices established in prior research [40] and is chosen specifically to ensure fair comparison with existing state-of-the-art methods.

To assess the generalizability of our model, we perform cross-dataset testing on two external data sets:

* **DFDC [49]:** A large-scale dataset containing a mix of deepfake manipulation techniques. We select 140 videos for testing.

* **Celeb-DF v2 [4]:** A dataset featuring high-quality deepfake videos that are visually difficult to distinguish from real footage. We select 140 videos for testing.

Additionally, we evaluate performance on FaceForensics++ test subsets, which include DeepFake, Face2Face, FaceSwap, and NeuralTextures, providing a comprehensive assessment across multiple manipulation types.

**Pre-processing Pipeline:** To prepare the datasets for training and evaluation, we apply a systematic pre-processing pipeline to ensure consistency across all video samples. Given the complexity of deepfake manipulations, accurate face detection and cropping are essential for improving the robustness of detection models. The key steps include:

* **Face Detection and Cropping:** We employ Multi-task Cascaded Convolutional Network (MTCNN) [50] to detect and extract facial regions from each video frame. MTCNN is chosen for its robustness in handling variations in pose, lighting, and occlusion. Detected faces are cropped and resized to 224 × 224 pixels.

* **Frame Sampling:** To reduce computational load while preserving temporal information, 10 evenly spaced frames are sampled per video. This strikes a balance between efficiency and content representativeness.

* **Data Organization:** All extracted frames are categorized into real or fake classes, following a consistent naming convention that facilitates efficient training and evaluation.

This pre-processing pipeline ensures a standardized and well-structured dataset, thereby enhancing the reliability of the deepfake detection process.

An overview of the dataset split, including the number of videos and extracted frames, is shown in Table 3.
<table><caption>**Table 3**
Dataset split: Number of videos and corresponding frames (Real = Re, Fake = Fa).</caption><thead><tr><th rowspan="2">Dataset</th><th colspan="4">#Videos</th><th colspan="4">#Frames</th></tr><tr><th>Train (Re/Fa)</th><th>Val (Re/Fa)</th><th>Test (Re)</th><th>Test (Fa)</th><th>Train (Re/Fa)</th><th>Val (Re/Fa)</th><th>Test (Re)</th><th>Test (Fa)</th></tr></thead><tbody><tr><td>FF++</td><td>360/360</td><td>70/70</td><td>70</td><td>70</td><td>3600/3600</td><td>700/700</td><td>700</td><td>700</td></tr><tr><td>Celeb-DF v2</td><td>-</td><td>-</td><td>70</td><td>70</td><td>-</td><td>-</td><td>700</td><td>700</td></tr><tr><td>DFDC</td><td>-</td><td>-</td><td>70</td><td>70</td><td>-</td><td>-</td><td>700</td><td>700</td></tr></tbody></table>

## 4.2. Evaluation metrics

We evaluate our model using two widely recognized metrics: Accuracy and F1-Score. Accuracy provides an indication of overall performance by measuring the percentage of correct predictions. The F1-score comprehensively assesses the model's capability, particularly in dataset imbalance scenarios, by balancing precision and recall. Together, these metrics ensure reliable performance evaluation across various datasets.

## 4.3. Implementation details

The experiments were conducted using the PyTorch deep learning framework on a Windows machine equipped with an NVIDIA Quadro RTX 5000 GPU with 16 GB of memory. For our study, we used pre-trained models such as BLIP2 and implemented LoRA fine-tuning with the following configuration: rank ($r$) of 32, scaling factor ($\alpha$) of 16, and a dropout rate of 0.05. The training was performed using the AdamW [51] optimizer with a learning rate of 0.0002 and a cosine learning rate scheduler. The model was trained for 5 epochs with a micro-batch size of 1, utilizing gradient accumulation with 4 steps to simulate a larger effective batch size. For perceptual metadata extraction, we use OpenCV (v4.5) for BRISQUE and quality metrics, and PyTorch (v1.12) for ResNet-50 feature extraction. All frames are processed at 224 × 224 resolution. BRISQUE scores are computed using pre-trained models (brisque model live.yml), and ResNet-50 weights are frozen to avoid overfitting.

To ensure a fair and consistent comparison, we retrained all baseline models, including XceptionNet, EfficientNet, MesoNet, and Deepfake AFSFF [40], Deepfake Detection [41], using our subset of the data set and the standardized pre-processing pipeline. Each model was trained with the same train, validation, and test splits as our proposed method. We perform a grid search for key hyperparameters, such as learning rate, batch size, and number of epochs, and select the configuration that achieves the best validation performance. This uniform training procedure ensures that all reported results are directly comparable and reflect performance under identical experimental conditions.

## 4.4. Baselines

To establish benchmarks, we selected three widely adopted architectures for deepfake detection: Xception [27], EfficientNet [28], and MesoInception-4 [29]. These models were chosen for their proven effectiveness in media forensics and represent distinct approaches, from depthwise separable convolutions (Xception) to compound scaling (EfficientNet) and lightweight mesoscopic analysis (MesoInception-4). All baselines were initialized with ImageNet-pre-trained weights and adapted for binary classification via a single fully connected layer, ensuring fair comparison with our multimodal framework while maintaining their original architectural strengths.

# 5. Results and discussion

This section presents the evaluation results of our proposed model and provides a comprehensive analysis of its performance across different datasets. We compare our approach to various baseline models, highlight key factors contributing to its effectiveness, and analyze cases where it excels or encounters challenges.

## 5.1. Performance evaluation and analysis of model effectiveness

Table 4 presents the performance comparison of our proposed model against several state-of-the-art deepfake detection methods across DeepFake Detection Challenge (DFDC), Celebrity DeepFake (Celeb-DF), and FaceForensics++ subdatasets, including Deep-Fake, Face2Face, FaceSwap, and NeuralTextures. Among baseline models, XceptionNet achieved an average accuracy of 85.28%, performing well on DeepFake with 97.3% accuracy but struggling with NeuralTextures, where it reached only 72.0%, highlighting difficulties in handling texture-based manipulations. EfficientNetB7 slightly improved performance, reaching an overall accuracy of 82.64%, yet struggled to generalize effectively, particularly on NeuralTextures, where it also achieved only 72.0%. Deepfake AFSFF demonstrated competitive results, achieving an average accuracy of 93.11%, performing exceptionally on Face2Face with 99.16% accuracy but experiencing minor misclassification errors on FaceSwap, where it achieved 98.94%. Our proposed model outperformed all baseline methods, achieving an overall accuracy of 96.51%. Specifically, it reached 90.50% accuracy on DFDC and 93.89% on Celeb-DF v2, demonstrating improved generalization across datasets. On FaceForensics++, the model excelled across multiple deepfake categories, achieving 98.99% accuracy on DeepFake, 99.23% on FaceSwap, and 97.32% on NeuralTextures, confirming
<caption>**Table 4**</caption>

Performance comparison (Accuracy↑/F1-Score↑) on DFDC, Celeb-DF v2, and FaceForensics++ (FF++) subsets (DF, F2F, FS, NT). All methods trained on FF++.

<table><caption>**Table 5**</caption><thead><tr><th rowspan="2">Method (Year)</th><th rowspan="2">DFDC↑</th><th rowspan="2">Celeb-DF v2↑</th><th colspan="4">FF++↑</th><th rowspan="2">Avg↑</th></tr><tr><th>DF</th><th>F2F</th><th>FS</th><th>NT</th></tr></thead><tbody><tr><td>XceptionNet [27] (2017)</td><td>80.12/80.12</td><td>79.54/79.54</td><td>97.30/97.30</td><td>90.50/90.50</td><td>92.20/92.20</td><td>72.00/72.00</td><td>85.28/85.28</td></tr><tr><td>EfficientNet [28] (2019)</td><td>87.36/87.36</td><td>87.06/87.06</td><td>95.50/95.50</td><td>79.60/79.60</td><td>82.50/82.50</td><td>82.70/82.70</td><td>85.79/85.79</td></tr><tr><td>EfficientNetB7 (2020) [28] (2019)</td><td>89.50/89.50</td><td>93.44/93.44</td><td>97.30/97.30</td><td>90.50/90.50</td><td>92.20/92.20</td><td>72.00/72.00</td><td>82.64/82.64</td></tr><tr><td>MesolInception-4 [29] (2018)</td><td>70.46/70.46</td><td>65.64/65.64</td><td>87.30/87.30</td><td>56.20/56.20</td><td>61.20/61.20</td><td>40.70/40.70</td><td>63.62/63.62</td></tr><tr><td>Deepfake AFSFF [40] (2024)</td><td>83.55/83.55</td><td>82.70/82.70</td><td>98.80/98.80</td><td><b>99.16/99.16</b></td><td>98.94/98.94</td><td>96.96/96.96</td><td>93.11/93.11</td></tr><tr><td>Deepfake detection [41] (2024)</td><td>71.22/71.22</td><td>65.83/65.83</td><td>98.32/90.75</td><td>90.75/92.33</td><td>92.33/92.33</td><td>82.62/82.62</td><td>83.51/83.51</td></tr><tr><td><b>Our model(2024)</b></td><td><b>90.50/90.50</b></td><td><b>93.89/93.89</b></td><td><b>98.99/98.99</b></td><td>99.10/99.10</td><td><b>99.23/99.23</b></td><td><b>97.32/97.32</b></td><td><b>96.51/96.51</b></td></tr></tbody></table>

Performance comparison of models on C40 dataset: (Accuracy ↑/F1-Score ↑) Across different manipulation types.

<table><thead><tr><th>Method (Year)</th><th>DeepFake ↑</th><th>Face2Face ↑</th><th>FaceSwap ↑</th><th>NeuralTextures ↑</th><th>Average ↑</th></tr></thead><tbody><tr><td>XceptionNet [27] (2017)</td><td>80.27/80.27</td><td>69.53/69.53</td><td>82.41/82.41</td><td>69.34/69.34</td><td>75.39/75.39</td></tr><tr><td>EfficientNet [28] (2019)</td><td>78.79/78.79</td><td>61.15/61.15</td><td>73.74/73.74</td><td>79.65/79.65</td><td>73.33/73.33</td></tr><tr><td>EfficientNetB7 [28] (2020)</td><td>80.27/80.27</td><td>69.53/69.53</td><td>82.41/82.41</td><td>69.34/69.34</td><td>75.39/75.39</td></tr><tr><td>MesolInception-4 [29] (2018)</td><td>72.02/72.02</td><td>43.18/43.18</td><td>54.70/54.70</td><td>39.20/39.20</td><td>52.27/52.27</td></tr><tr><td>Deepfake AFSFF [40] (2024)</td><td>81.51/81.51</td><td>76.18/76.18</td><td><b>88.43/88.43</b></td><td>93.38/93.38</td><td>84.88/84.88</td></tr><tr><td>Deepfake detection [41] (2024)</td><td>80.47/80.47</td><td>75.88/75.88</td><td>86.30/86.30</td><td>89.12/89.12</td><td>84.88/84.88</td></tr><tr><td><b>Our model (2024)</b></td><td><b>81.68/81.68</b></td><td><b>76.43/76.43</b></td><td>87.21/87.21</td><td><b>95.56/95.56</b></td><td><b>85.22/85.22</b></td></tr></tbody></table>

the effectiveness of integrating quality assessment metrics and multimodal representations in deepfake detection. The strong performance of our model can be attributed to its ability to leverage multiple sources of information beyond pixel-based analysis. By integrating visual quality metrics such as artifact score, brightness, sharpness, and contrast alongside linguistic representations from frame and video captions, our approach provides a richer semantic understanding of deepfake inconsistencies. The frame-level and video-level captioning process using BLIP2 captures subtle temporal inconsistencies across frames, enhancing the model's ability to detect deepfake-generated anomalies that would otherwise be overlooked in single-frame detection approaches. Additionally, incorporating ResNet-50 and BRISQUE allows the detection of fine-grained distortions introduced by generative models, enabling a more robust classification of manipulated videos.

Despite its strong performance, some challenges remain. The slightly lower accuracy on DFDC (90.50%) compared to other datasets can be attributed to the complexity of the DFDC dataset's synthetic videos, which exhibit higher realism and greater diversity in manipulation techniques. This makes it more difficult for pixel-based and texture-based analyses to capture detectable inconsistencies. Similarly, while our model achieves 97.32% accuracy on NeuralTextures, certain fine-grained texture-based deepfakes remain difficult to distinguish due to their high fidelity, reducing pixel-level anomalies and making detection more challenging. Another limitation observed is the misclassification of some real videos as fake, particularly those with high compression artifacts, as compression-induced distortions sometimes mimic deepfake artifacts.

To further assess the robustness of our model under different compression levels, we evaluated it on the FaceForensics++ C40 dataset, which represents a lower compression setting compared to the previously analyzed datasets. Table 5 presents the accuracy and F1 score of various deepfake detection models in C40. The results show that our model maintains strong performance in the C40 compression setting, achieving an overall accuracy of 85.22%, outperforming all baseline models. Although Deepfake AFSFF [41] achieves the highest accuracy on FaceSwap (88.43%), our model consistently outperforms on Face2Face, DeepFake, and NeuralTextures. In particular, the model achieves an accuracy of 95.56% on NeuralTextures, the highest among all evaluated models, indicating its effectiveness in detecting subtle texture-based manipulations even under low compression conditions. The improvements in our model's performance on C40 compared to previous results on FaceForensics++ (HQ) can be attributed to its ability to leverage multimodal analysis, combining low-level quality metrics with semantic captioning techniques. The stronger separation of fake and real distributions in NeuralTextures, observed in previous perceptual quality analysis, aligns with this improved detection capability. However, the results also highlight that FaceSwap remains a challenging subset across models, emphasizing the need for further enhancements in temporal consistency analysis to capture long-range dependencies in manipulated content.

Fig. 3 illustrates the confusion matrices in different datasets, providing deeper insights into the effectiveness of the classification. The confusion matrix for FaceForensics++ shows that the model correctly classified 68 real and 68 fake videos, with only 2 misclassifications, demonstrating near-perfect accuracy. For Celeb-DF v2, the model correctly classified 66 real and 66 fake videos, with 4 misclassified instances, suggesting strong adaptation to high-quality deepfake videos. On DFDC, the model correctly classified 63 real and 63 fake videos, with 7 misclassifications per class, reflecting the challenges posed by the diverse and highly realistic synthetic techniques used in DFDC. Overall, our model demonstrates state-of-the-art accuracy, strong generalization across datasets, and robustness in deepfake detection. The integration of multimodal feature analysis with visual quality assessment and linguistic understanding significantly enhances detection performance, setting a new benchmark in deepfake analysis and reinforcing the importance of leveraging multiple modalities for reliable deepfake detection.
<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Confusion matrices for the proposed model on FaceForensics++, Celeb-DF, and DFDC datasets. The results highlight the model's classification effectiveness in distinguishing real and fake videos.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 4.** Examples of correct and misclassified frames. Misclassified real videos (a–b) suffered from compression artifacts, while misclassified fakes (c–d) had subtle generative flaws. Correct classifications (e–h) aligned with clear quality metrics.</figcaption></figure>

**Fig. 4** contextualizes our quantitative results through representative examples. Frames (a) and (b) demonstrate real videos misclassified as fake, where the BRISQUE model misinterpreted compression blur (scores: 0.82, 0.79) as artificial artifacts. Frames (c) and (d) show undetected fakes where BLIP2 generated misleading captions (e.g., 'a man face with natural lighting, without blur). In contrast, frames (e)–(h) highlight correct classifications: (e) and (f) exhibit unambiguous natural textures, while (g) and (h) display clear generative anomalies such as inconsistent jawline shadows (*artifactscores* > 0.9). These cases align with the confusion matrices in Fig. 3, which illustrates the interaction between low-level quality metrics and high-level semantic analysis.

To visually assess the discriminative capacity of different input modalities, we employ t-SNE to project the input feature embeddings into a 2D space. Fig. 5 compares the distribution of real and fake videos using metadata-only features (Fig. 5(a)), caption-only features (Fig. 5(b)), and combined metadata + caption inputs (Fig. 5(c)). The metadata-only plot shows limited separation, as low-level perceptual cues alone are often insufficient to capture complex manipulations. Captions, while more semantically informative, still present overlap between real and fake clusters. In contrast, the combined representation constructed via simple textual concatenation exhibits clear and well-defined clusters, indicating stronger discriminative power. This visualization supports our quantitative results by highlighting how early-stage multimodal fusion, even when implemented via concatenation, enhances real/fake separability in feature space.
<figure><img src="image_8.png" /><figcaption>Fig. 5. t-SNE visualizations of input feature embeddings. (a) Metadata-only features. (b) Caption-only features. (c) Fused metadata + caption features. Real videos are shown in green, fake videos in red. Clearer cluster separation is observed in the fused representation, validating the effectiveness of multimodal fusion.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 6. Confidence score distribution for real vs. fake predictions.</figcaption></figure>

## 5.2. Confidence score analysis

To further evaluate the reliability of our model's predictions, we analyze the confidence scores assigned to real and fake video classifications, derived directly from the model's softmax outputs. Fig. 6 illustrates the confidence score distribution across the test dataset, highlighting a clear separation and high certainty in predictions.
<table><caption>**Table 6**</caption><caption>Ablation study results: Performance comparison of the full model vs. Variants without specific features.</caption><thead><tr><th rowspan="2">Method (Year)</th><th rowspan="2">DFDC ↑</th><th rowspan="2">Celeb-DF v2 ↑</th><th colspan="4">FF++ ↑</th><th rowspan="2">Avg ↑</th></tr><tr><th>DF</th><th>F2F</th><th>FS</th><th>NT</th></tr></thead><tbody><tr><td>Our model WQM (2024)</td><td>71.55/71.55</td><td>80.33/80.33</td><td>75.13/75.13</td><td>80.01/80.01</td><td>70.89/70.89</td><td>69.72/69.72</td><td>74.60/74.60</td></tr><tr><td>Our model WAS (2024)</td><td>60.33/60.33</td><td>63.12/63.12</td><td>70.63/70.63</td><td>74.56/74.56</td><td>68.68/68.68</td><td>76.88/76.88</td><td>69.03/69.03</td></tr><tr><td>Our model WM (2024)</td><td>59.56/59.56</td><td>60.32/60.32</td><td>62.45/62.45</td><td>70.18/70.18</td><td>60.72/60.72</td><td>71.44/71.44</td><td>64.11/64.11</td></tr><tr><td><strong>Our model (2024)</strong></td><td><strong>90.50/90.50</strong></td><td><strong>93.89/93.89</strong></td><td><strong>98.99/98.99</strong></td><td>99.10/99.10</td><td><strong>99.23/99.23</strong></td><td><strong>97.32/97.32</strong></td><td><strong>96.51/96.51</strong></td></tr></tbody></table>

Our analysis shows high mean confidence scores: 95.97% for real videos and slightly higher, 97.17%, for fake videos, indicating the model's greater certainty when identifying fake content. The median confidence scores are notably high as well, at 99.77% for real videos and 99.66% for fake videos, demonstrating the robustness and consistency of predictions. Additionally, low standard deviation values (99.16% for real videos and 71.8% for fake videos) further confirm the stability and reliability of the model's performance.

These results underline the effectiveness of our multimodal approach, integrating visual quality analysis and linguistic cues, and emphasize the model's strong generalization capabilities across different datasets.

### 5.3. Analysis of perceptual quality features

To further analyze the impact of perceptual quality assessment in deepfake detection, we examined the distribution of key visual quality metrics, including artifact score, brightness, sharpness, and contrast. Fig. 7 illustrates the distributions of these quality attributes across the training (Fig. 7(a)), validation (Fig. 7(b)), and test datasets (Fig. 7(c)), respectively.

Our findings reveal the following patterns:

*   **Artifact Scores:** Fake videos exhibit higher artifact scores, indicating visible inconsistencies introduced by generative models. This separation between real and fake distributions suggests that artifact analysis is a strong indicator of synthetic content.

*   **Sharpness scores:** Fake videos tend to have lower sharpness scores, often due to blurring effects present in AI-generated content. This highlights that sharpness degradation is a significant factor in identifying manipulated videos.

*   **Brightness and contrast:** Although real and fake videos show overlapping brightness distributions, contrast variations are noticeable, particularly in certain datasets where generative models struggle to maintain consistent lighting conditions.

These insights validate the importance of integrating quality-based features alongside multimodal representations, enhancing the robustness of deepfake detection frameworks.

## 6. Ablation study

To assess the contribution of individual components to our model's performance, we conducted an ablation study by systematically removing key features and observing the resulting impact on accuracy and F1-score. Specifically, we evaluated three variations of our model:

*   **WQM (Without Quality Metrics):** The model is trained without incorporating sharpness, contrast, and brightness metrics.

*   **WAS (Without Artifact Score):** The artifact score, which captures deepfake-related distortions, is excluded from the metadata.

*   **WM (Without Metadata):** All metadata features, including quality metrics and artifact scores, are removed, leaving only textual and visual features.

Table 6 reports the accuracy of our model in different ablation settings, quantifying the individual impact of perceptual quality metrics and artifact scores on the deepfake detection performance. Removing quality metrics (WQM) led to a significant drop in performance, with the average accuracy decreasing from 96.51% to 74.60%, demonstrating the essential role of these low-level perceptual features in deepfake detection. The exclusion of the artifact score (WAS) further reduced accuracy to 69.03%, emphasizing the importance of capturing compression artifacts and synthetic distortions in the detection process.

The most substantial performance degradation was observed in the WM setting, where all metadata features were removed, resulting in an average accuracy drop to 64.11%. This confirms that the integration of visual quality metrics and artifact analysis significantly improves the model's ability to differentiate real from fake videos. The impact of removing these components is most pronounced in datasets like DFDC, where deepfake artifacts are more subtle, making detection considerably more difficult without metadata-based analysis.

These findings highlight the importance of including metadata within the pipeline of classification. By combining quality measurement and artifact score, our model is able to capture subtle discrepancies that would otherwise go unnoticed by depending exclusively on visual and text features. The ablation study confirms that excluding these integral parts leads to tangible degradation, affirming our suggested approach.

Further, the inter-correlation between quality metrics is evidenced by the correlation heatmap (Fig. 8(b)). Artifact score is negatively correlated with the indicators of sharpness and brightness, reflecting that processed videos contain compression artifacts
<figure><img src="image_10.png" /><figcaption>(a) Training Set Metadata Distributions</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(b) Validation Set Metadata Distributions</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(c) Test Set Metadata Distributions</figcaption></figure>

**Fig. 7.** Distributions of extracted quality metrics (artifact score, brightness, sharpness, contrast) across the training, validation, and test datasets. These distributions highlight the statistical differences between real and fake videos, supporting the effectiveness of perceptual features in deepfake detection.
<figure><img src="image_13.png" /><figcaption>**Fig. 8.** Analysis of quality metrics in fake video classification. (a) Importance of artifact score, sharpness, brightness, and contrast in classification. (b) Correlation between quality metrics, highlighting their relationships and significance in detecting manipulated videos.</figcaption></figure>

as well as missing fine detail along with unstable illumination. These are further indications of the need for the usage of multiple quality features for an accurate deepfake detection.

To support such findings, we further examined feature importance of our quality metrics within our classifier pipeline. Fig. 8(a) indicates that the most powerful factor for detecting manipulated content is artifact score, followed by that of sharpness, brightness, and contrast. This is corroborated by our ablation study findings, which demonstrate that by using a combination of several quality properties, one can build a better quality-detecting model.

* The elimination of quality metrics (WQM) caused the biggest accuracy decline for all the datasets, affirming that contrast, brightness, as well as sharpness, have an essential role in identifying real and manipulated videos.

* Omitting the artifact score (WAS) further caused a reduction of performance, reaffirming that compression artifacts as well as synthetic distortions are essential deepfake indicators.

* The removal of all metadata (WM) revealed the lowest accuracy, indicating that both artefact detection as well as perceptual quality measurements are needed for effective deepfake identification.

* The correlation heatmap supports such findings by demonstrating how such quality measurements interact with one another and feed into the overall decision-making process of the model.

t-SNE analysis Fig. 5 indicates: Although perceptual metadata (i.e., artifact scores) and captions are not individually strong enough for strong detection, their combination forms a synergistic feature space where real and forged videos can get linearly separated. This is confirmed by our ablation study Table 6, where metadata removal induced a 32.4% accuracy decrease, emphasizing its essential contribution for the detection of distortions not covered by semantic captions. There is further scope for improving separability by including dynamic quality measurements (i.e. frame-to-frame consistency of artifacts) or adversarial caption augmentations to overcome challenges left by residuals such as DFDC samples of high quality.

## 7. Conclusion and future work

In this study, we propose a new multimodal deepfake framework that combines vision-language modeling with visual quality assessment for improving the credibility of deepfake classification. By integrating low-level subjective features including sharpness, contrast, brightness, and artifacts with high-level semantic reasoning based on captions per frame as well as per video, our method effectively recognizes both the synthesized anomalies as well as contextual inconsistencies.

Experimental evaluation of our method on FaceForensics++, DFDC, and Celeb-DF conveys that our method is state-of-art, performing better than both CNN- as well as transformer-based deepfakes detection methods. Confidence score analysis further reflects the stability of our method, with a clear differentiation between real as well as forged classifications.

Our findings highlight the crucial role of metadata-based analysis in deepfake detection. Ablation studies confirm the significant contribution of artifact scores and quality metrics to overall classification performance. Cross-dataset evaluations on FaceForensics++ (C40) validate the generalizability of our model across different compression levels and unseen deepfake manipulation techniques.

Despite these strong results, the proposed framework has a few limitations. First, frame-level captioning may occasionally lead to hallucinated or overly neutral descriptions, particularly in cases of subtle manipulations. Second, our fixed sampling strategy of
10 frames per video may miss localized or short-lived artifacts, especially in long or complex scenes. Finally, synthetic videos with minimal visual degradation can challenge quality-based detection, as they lack strong perceptual cues. These limitations point to areas for further enhancement.

However, our analysis also reveals several challenges, especially with respect to handling FaceSwap manipulations and DFDC videos of high quality, where deepfake artifacts are subtle and not easily identifiable. In our future work, we will improve our framework by including temporal consistency analysis for detecting long-distance dependencies among manipulated videos. Moreover, self-supervised learning methods could enhance detection on low-resource and unseen deepfake datasets. Another potentially fruitful direction is investigating adversarial robustness techniques — such as contrastive learning and data augmentation — to further enhance generalization capabilities with respect to sophisticated deepfake generation techniques. Finally, adapting our approach for audio-visual deepfake detection through analysis of speech inconsistencies as well as lip-synchronization errors could provide for a fuller forensic framework. In summary, our findings show that multimodal combination of visual quality analysis as well as linguistic representation substantially improves deepfake accuracy, opening the way for more understandable, adaptable solutions for countering AI-generated disinformation.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgments

This work has been partially funded by the project PCI2022-134990-2 (MARTINI) of the CHISTERA IV Cofund 2021 program. Abdenour Hadid is funded by TotalEnergies collaboration agreement with Sorbonne University Abu Dhabi. Project funded by ANR for CHIST-ERA (MARTINI : 23-Z6-15-849).

## Data availability

My code available, you will find my code for github in the abstract.

## References

[1] Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. In: Advances in neural information processing systems. vol. 27, 2014.

[2] Korshunov P, Marcel S. DeepFakes: A new threat to face recognition? Assessment and detection. 2019, arXiv preprint arXiv:1812.08685.

[3] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. In: Advances in neural information processing systems. vol. 33, 2020, p. 6840-51.

[4] Li Y, Yang X, Sun P, Qi H, Lyu S. Celeb-DF: A large-scale challenging dataset for DeepFake forensics. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. CVPR, 2020, p. 3207-16.

[5] Yang X, Li Y, Lyu S. Exposing deep fakes using inconsistent head poses. In: CASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing. ICASSP, 2019, p. 8261-5.

[6] He Z, Zuo W, Kan M, Shan S, Chen X. AttGAN: Facial attribute editing by only changing what you want. IEEE Trans Image Process 2019;28(11):5464-78.

[7] Perov I, Gao D, Chervoniy N, Liu K, Marangonda S, Umé C, et al. DeepFaceLab: Integrated, flexible and extensible face-swapping framework. 2020, arXiv preprint arXiv:2005.05535.

[8] Perarnau G, van de Weijer J, Raducanu B, Álvarez JM. Invertible conditional GANs for image editing. In: NIPS workshop on adversarial training. 2016.

[9] Liu M, Ding Y, Xia M, Liu X, Ding E, Zuo W, Wen S. STGAN: A unified selective transfer network for arbitrary image attribute editing. In: Proceedings of the IEEE conference on computer vision and pattern recognition. CVPR, 2019, p. 3673-82.

[10] Choi Y, Choi M, Kim M, Ha J-W, Kim S, Choo J. StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. CVPR, 2018, p. 8789-97.

[11] Li Y, Chang M-C, Lyu S. Exposing DeepFake videos by detecting face warping artifacts. 2018, arXiv preprint arXiv:1811.00656.

[12] Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Houlsby N. An image is worth 16x16 words: Transformers for image recognition at scale. Int Conf Learn Represent (ICLR) 2021.

[13] Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin J, Clark P, et al. Learning transferable visual models from natural language supervision. In: International conference on machine learning. 2021.

[14] Li J, Li D, Savarese S, Hoi S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: International conference on machine learning. PMLR; 2023, p. 19730-42.

[15] Wang Z, Bhosale S, Zhang D, Hou L, Zettlemoyer L, Cox D. SimVLM: Simple visual language model pretraining with weak supervision. 2021, arXiv preprint arXiv:2108.10904.

[16] Mittal A, Moorthy AK, Bovik AC. No-reference image quality assessment in the spatial domain. IEEE Trans Image Process 2012;21(12):4695-708.

[17] Mittal A, Soundararajan R, Bovik AC. Making a "completely blind" image quality analyzer. IEEE Signal Process Lett 2013;20(3):209-12.

[18] Peng B, Lin L, Li X, et al. DeepFidelity: Perceptual forgery fidelity assessment for Deepfake detection. 2023, arXiv preprint arXiv:2312.04961.

[19] Wu C, Huang L, Zhang Q, Li B, Ji L, Yang F, Sapiro G, Duan N. Godiva: Generating open-domain videos from natural descriptions. 2021, arXiv preprint arXiv:2104.14806.

[20] Hong S, Zhou Y, Wang J, Shi B, Yuan L. CogVideo: Large-scale pretraining for text-to-video generation with Hierarchical training. 2022, arXiv preprint arXiv:2205.15868.

[21] Wu C, Liang J, Ji L, Yang F, Fang Y, Jiang D, Duan N. Nüwa: Visual synthesis pre-training for neural visual world creation. In: European conference on computer vision. Springer; 2022, p. 720-36.

[22] Ma X, Wang Y, Jia G, Chen X, Liu Z, Li Y-F, Chen C, Qiao Y. Latte: Latent diffusion transformer for video generation. 2024, arXiv preprint arXiv:2401.03048.
[23] Chen H, Zhang Y, Cun X, Xia M, Wang X, Weng C, Shan Y. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024, p. 7310–20.

[24] Yan W, Gupta A, Wang J, Zhu X, Wu J. VideoGPT: Video generation using discrete representations. Advances in neural information processing systems. In: Advances in neural information processing systems. 2021.

[25] Ho J, Salimans T, Gritsenko A, Chan W, Norouzi M, Poole B. Video diffusion models. 2022, arXiv preprint arXiv:2204.03458.

[26] Zhai S, Lin T, Zhang H, Liu C, Wang J, Zhou Y. ModelScope T2V: A large-scale pretrained model for text-to-video generation. 2023, arXiv preprint arXiv:2303.13007.

[27] Chollet F. Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. CVPR, 2017, p. 1251–8. http://dx.doi.org/10.1109/CVPR.2017.195.

[28] Tan M, Le Q. EfficientNet: Rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th international conference on machine learning. ICML, 2019, p. 6105–14, arXiv:1905.11946.

[29] Afchar D, Nozick V, Yamagishi J, Echizen I. MesoNet: A compact facial video forgery detection network. In: 2018 IEEE international workshop on information forensics and security. WIFS, IEEE; 2018, p. 1–7.

[30] Dinçer R, Kose U, Arslan M. Golden ratio based deep fake video detection system with fusion of capsule networks. Comput Electr Eng 2024;117:108476. http://dx.doi.org/10.1016/j.compeleceng.2024.108476.

[31] Rao M, Gupta D, Kumari S, Singh A. Deep learning model to detect deceptive generative adversarial network generated images using multimedia forensic. Comput Electr Eng 2023;113:108264. http://dx.doi.org/10.1016/j.compeleceng.2023.108264.

[32] Zhang D, Lin F, Hua Y, Wang P, Zeng D, Ge S. Deepfake video detection with spatiotemporal dropout transformer. In: Proceedings of the 30th ACM international conference on multimedia. ACM; 2022, p. 1234–42. http://dx.doi.org/10.1145/3503161.3547913.

[33] Rossler A, Cozzolino D, Verdoliva L, Riess C, Thies J, Nießner M. Faceforensics++: Learning to detect manipulated facial images. In: Proceedings of the IEEE/CVF international conference on computer vision. 2019, p. 1–11.

[34] Liu X, Zhao J, Wang Y. A lightweight 3D CNN with channel transformation for Deepfake detection. Pattern Recognit Lett 2021;150:57–64.

[35] Ranjan R, Sankaranarayanan S, Castillo C, Chellappa R. Detecting and mitigating adversarial attacks on deepfake detection models using transfer learning. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. WACV, 2020.

[36] Liang Y, Wang M, Jin Y, Pan S, Liu Y. Hierarchical supervisions with two-stream network for Deepfake detection. Pattern Recognit Lett 2023;172:121–7.

[37] Khalid R, Woo S-S. OC-FakeDect: Classifying deepfakes using one-class variational autoencoder. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. CVPRW, 2020, p. 2794–803.

[38] Bai J, Lin M, Cao G, Lou Z. AI-generated video detection via spatial-temporal anomaly learning. In: Chinese conference on pattern recognition and computer vision. PRCV, Springer; 2024, p. 460–70.

[39] Ji L, Lin Y, Huang Z, Han Y, Xu X, Wu J, Wang C, Liu Z. Distinguish any fake videos: Unleashing the power of large-scale data and motion features. 2024, arXiv preprint arXiv:2405.15343.

[40] Wang F, Chen Q, Jing B, Tang Y, Song Z, Wang B. Deepfake detection based on the adaptive fusion of spatial-frequency features. Int J Intell Syst 2024;2024(1):7578036.

[41] Naskar G, Mohiuddin S, Malakar S, Cuevas E, Sarkar R. Deepfake detection using deep feature stacking and meta-learning. Heliyon 2024;10(4).

[42] Chen H, Hong Y, Huang Z, Xu Z, Gu Z, Li Y, Lan J, Zhu H, Zhang J, Wang W, et al. Demamba: AI-generated video detection on million-scale genvideo benchmark. 2024, arXiv preprint arXiv:2405.19707.

[43] Liu Q, Shi P, Tsai Y-Y, Mao C, Yang J. Turns out i'm not real: Towards robust detection of AI-generated videos. 2024, arXiv preprint arXiv:2406.09601. URL: https://arxiv.org/abs/2406.09601.

[44] Khan M, Dang-Nguyen D-T. CLIPping the deception: Adapting vision-language models for universal deepfake detection. 2024, arXiv preprint arXiv: 2402.12927. URL: https://arxiv.org/abs/2402.12927.

[45] Lin W, Wu Z, Zhang Y, Wang Y, Keutzer K. Standing on the shoulders of giants: Reprogramming visual-language models for general deepfake detection. 2024, arXiv preprint arXiv:2409.02664. URL: https://arxiv.org/abs/2409.02664.

[46] Jia R, Zhang J, Liu S, Zhou A, Wang J, Zhang Y. Can ChatGPT detect DeepFakes? A study of using multimodal large language models for media forensics. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. CVPRW, 2024, URL: https://openaccess.thecvf.com/content/CVPR2024W/WMF/papers/Jia_Can_ChatGPT_Detect_DeepFakes_A_Study_of_Using_Multimodal_Large_CVPRW_2024_paper.pdf.

[47] Anan K, Bhattacharjee A, Intesher A, Islam K, Assaeem Fuad A, Saha U, Imtiaz H. CAE-Net: generalized deepfake image detection using convolution and attention mechanisms with spatial and frequency domain features. arXiv e-prints 2025. http://dx.doi.org/10.48550/arXiv.2502.10682, arXiv:2502.10682.

[48] Concas S, La Cava SM, Casula R, Orru G, Puglisi G, Marcialis GL. Quality-based artifact modeling for facial deepfake detection in videos. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024, p. 3845–54.

[49] Dolhansky B, Howes R, Pflaum M, Baram N, Ferrer CC. The deepfake detection challenge dataset. 2020, arXiv:2006.07397.

[50] Zhang K, Zhang Z, Li Z, Qiao Y. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 2016;23(10):1499–503.

[51] Loshchilov I, Hutter F. Decoupled weight decay regularization. 2019, arXiv preprint arXiv:1711.05101.