On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detection
Authors: Xiufeng Song
Xiao Guo
Jiache Zhang
Qirui Li
LEI BAI
Xiaoming Liu
Guangtao Zhai
Xiaohong Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Forensics，Multi-Modal Large Language Model
Primary area: machine_vision
TL;DR: We propose a powerful detector for diffusion video forensics.

Abstract: Large numbers of synthesized videos from diffusion models pose threats to information security and authenticity, leading to an increasing demand for generated content detection. However, existing video-level detection algorithms primarily focus on detecting facial forgeries and often fail to identify diffusion-generated content with a diverse range of semantics. To advance the field of video forensics, we propose an innovative algorithm named Multi-Modal Detection(MM-Det) for detecting diffusion-generated videos. MM-Det utilizes the profound perceptual and comprehensive abilities of Large Multi-modal Models (LMMs) by generating a Multi-Modal Forgery Representation (MMFR) from LMM's multi-modal space, enhancing its ability to detect unseen forgery content. Besides, MM-Det leverages an In-and-Across Frame Attention (IAFA)  mechanism for feature augmentation in the spatio-temporal domain. A dynamic fusion strategy helps refine forgery representations for the fusion. Moreover, we construct a comprehensive diffusion video dataset, called Diffusion Video Forensics (DVF), across a wide range of forgery videos. MM-Det achieves state-of-the-art performance in DVF, demonstrating the effectiveness of our algorithm. Both source code and DVF are available at https://github.com/SparkleXFantasy/MM-Det.

On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detection
Xiufeng Song', Xiao Guo?, Jiache Zhang', Qirui Li', Lei Bai Xiaoming Liu", Guangtao Zhai', Xiaohong Liu
Shanghai Jiao Tong University MMichigan State University SShanhaii Artificial Intelligence Laboratory {akikaze, zjc he, iapple1, zhaiguangtao, xiaohongl poohongliuu@ss juj@sjtuu. edw.euuu.uuuu............m. cn {guoxiall, 1isx msu. edu baisanshi @gmail. com "orresponding Author
Abstract
Large numbers of synthesized videos from diffusion models pose threats to infor- mation security and authenticity, leading to an increasing demand for generated content detection. However, existing video-level detection algorithms primarily focus on detecting facial forgeries and often fail to identify diffusion--generated content with diverse range of semantics. To advance the field of video forensics, we propose an innovative algorithm named Multi-Modal Detection(MNN (MM-Det) for detecting cettusionnngneertte videos. MM-Det utilizes the profound perceptual and comprehensive abilities of Large Multi-modal Models (LMMs) by generating space,
Multi Modill Forgery Representation (MMFR) from LMM's enhancing its ability to detect unseen forgery content. Besides, MM-Do leverages In-and-Across Frame Attention (IAFA) mechanism for feature augmentation the
domain. A dynamic fusion strategy helps refine forgery repre- sentations for the fusion. Moreover, we construct a comprehensive diffusion video dataset, called Diffusion Video Forensics (DVF), across wide range of forgery videos. MM-Det achieves state-of-the-art performance in DVE, demonstrating the effectiveness of our algorithm. Both source code and DVF are available at link.
Introduction
Recent years have witnessed significant advance- ments in diffusion generative methods, which have led to the creation of extraordinarily visually pelling content in video generation [5, 4, 61]. AI- though the latest generated videos impress society with their versatility and stability, synthetic media also poses risk of malicious attacks, such coun- terfeit faces created by deepfakes [49] and falsifi cations in business, raising public concerns about information security and privacy. In response to such issues, researchers have made significant progress forgery detection, addressing problems on image edit- ing manipulation [60, 30, 17] and CNN-synthesized images [53, 40, 55, 36, 18). To enhance the trustwor- thiness and reliability of current detectors the face of evolving generative video methods, aim to de- velop a generalizable detection method for diffusion- based generative videos.
Diff. Fused
Zscope
LMM
06 0J 0.8
a.pika
Sora
OSora
St. V. D.
Figure Multi-Modal Detection (MM-Det) leverages features from spatiotemporal (ST) information ), CLIP encoder [39] (D), and LMM The Fused feature achieves saate-of-thear performance in our Diffusion Video Forensics (DVF) dataset.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Real Images
Fake Images
Briefly deseribe and inspect the image for forgery detection.
Vision
perception
Domain
is of the sun over the sea with over-saturated color and lighting.
Muuti-Modal Forgery Representation
Conceptual Text
Semantic Vision
Give concrete reasons why is real or fake.
reasoning
Real
Text
Fake
Domain
is fake, because the unrealistic color is a typical sign of forgery.
Consistent Shape
Unrcalistic Color
LMMs detect visual artifacts and anomalies, offering detailed textual reasoning that explains whether the image is generated using Artificial Intelligence (Al) techniques. The powerful representation in the visual domain enables L.Ms to understand complex contexts within frames. Furthermore, their advanced language reasoning capability implicitly reveals image authenticity and provenance. For instance, the term like "coosiistntt shape" refers to common features in authentic content, while "unrealistic color" signifies typical artifacts in forged content. This linguistic proficiency stems from the superior perception and comprehension abilities of LMMs, contributing to generalizable multimodal feature space. By leveraging the visual understanding and textual reasoning abilities of LMMs, we construct a Multi-Modal Forgery Representation (MMFR).
Figure 2:
Previously, the video forensics community emphasized more on developing facial forgery detection algorithms [57, 73, 10, 62, 70], which may struggle to address recent fraudulent videos (e.g., sora, pika, etc.). Compared to facial forgery, diffusion-based generated content contains more diverse semantics, making it more challenging to distinguish diffusion forgery contents from real ones. Towards these challenges, a new thread of research on CNN-generated image detection has emerged [53, 40, 55, 36, 18]. These works aim to learn common generation traces in image-level content, but do not design specific mechanisms to capture temporal inconsistencies in videos.
Therefore, previous defensive efforts might not be able to provide a video-level detection algorithm for newly emerged generated videos with diverse manipulation artifacts and visual contexts. Meanwhile, Large Multi-modal Models (LMMs) show unparalleled problem-solving ability [2, 24, 23, 75, 69, 28), thanks to its powerful multi-modal representations. However, such representations are barely studied in the video forensics task.
Motivated by the limitation of previous work and the unprecedented understanding ability of LMMs. we propose a video-level detection algorithm, named Multi-Modal Detection (MM-Det), to capture forgery traces based perception and reasoning capabilities of LMMs to learn a generalizable forgery feature, as depicted in Fig. 2. To the best of our knowledge, we are the first to use LMMs for video forensic work.
LMM- multi-modal representation. MM-Det takes advantage of the
Aside from multi-modal representations, two common sources of generative errors that can be leveraged for video discrimination are spatial artifacts and temporal inconsistencies. Our approach aims to effectively identify these two types of errors as an auxiliary feature in forgery detection. Inspired by the previous work [55, 33, 41] that shows the effectiveness of reconstruction for detecting diffusion images, we extend this idea into the video domain, amplifying diffusion artifacts both in spatial and temporal information. To capture such artifacts efficiently, we leverage a Vector Quantised- Variational AutoEncoder (VQVAE) [51] for a fast reconstruction process, as detailed in Fig. 3. Moreover, we design novel an-and-Acrron Frame Attention (IAFA) into a Transformer- based network, which balances frame-level forgery traces with information flow across frames, thus aggregating local and global features.
Although diffusion methods demonstrate strong capabilities in video generation, the lack of public datascts on diffusion videos hinders research efforts in the video forensic community. In light of this,

---

we have established comprehensive dataset for diffusion generated videos, named Diffusion Video Forensics (DVF). DVF includes generated content from a variety of diffusion models, featuring rich semantics and high quality, serving as general benchmark for open-world video forensics tasks. The main contributions of this paper are as fol- lows:
diffusion-generated
Rool
Reokito VIgVAK Original
We propose detection method called MM- Det that leverages a Multi-Modal Forgery Rep- resentation from LMMs effectively detect diffusion-generrane videos with strong gener- alization capability.
o A powerful and innovative In-and-Across Frame Attention (IAFA) mechanism is intro- duced to aggregate global and local patterns within forged videos, enhancing the detection of spatial artifacts and temporal inconsistencies.
Figure 3:
The residual difference between VQ- VAE [51] reconstructed images and real ones. Given an encoder & and a decoder D of a VQ-VAE and taking the input video v, the reconstructed video v' is obtained as v' = Dv)(v)). The VQ- VAE reconstruction of real images exhibits obvi- ous edges and visible traces, whereas diffusion- generated ones are reconstructed more effectively, offering residual difference images with fewer vis- ible traces.
We introduce large-scale dataset, named the Diffusion Video Forensics (DVF) dataset, com- prising high-quality forged videos generated us- ing 8 diffusion- methods. The DVF dataset contains diverse forgery types across videos of varying resolutions and durations, effectively serving as benchmark for forgery detection in real-world scenarios.
Our MM-Det achieves state-of-the-art detection performance the DVF dataset. Also, detailed analysis provided to showcase the effectiveness of multi-modal representations in detecting forgeries, paving the way for compelling opportunities for using LMMs in future multi-media forensic research.
2 Related Works
Early work [53, 22, 13, 47, 65, 16] observed that forgery traces exist in images generated by AI techniques, and such traces are commonly used as evidence to distinguish diffusion- generated content [40, 6, 7] and attribute if two images are generated by the same method [66, 37]. However, identifying unseen and diverse frequency- based clues in real-world scenarios is challenging. For that, existing frame-level forgery detectors concentrate on improving the generalization ability. For example, some works [36, 17, 26, 29] introduced features from pre- rre-traieed CLIP [39] encoders for the forensic task to help reduce the overfitting issue on specific forgery types and increase the robustness towards detection [36, 8] and localization [17]. [64] and [12] proposed proactive methods to protect images from manipulation based on image watermarking and steganography. Also, reconstruction errors through the inversion process of DDIM [44] are studied by prior works [55, 33, 41, 32] for diffusion generative content detection. Moreover, the previous work [63, 46, 34, 18] develops specific techniques that increase the generalization unseen forgeries. For example, HiFi- Net [18] proposes a tree structure to model the inherent hierarchical correlation among different forgery methods, NPR [46] devises a special representation as generative artifacts, and the training set diversity can also contribute to generalization ability [34]. Unlike prior works, our MM-Det leverages multi-modal reasoning to achieve a high level generalization ability.
Frame-level Detector
Video-level Detector Early video-level methods primarily focused on detecting facial forgery. For
example, [25] learned the boundary artifacts between the original background and manipulated
faces. [19] discriminated fake videos from the inconsistency of mouth motion.
[73] designed
multi-attentional detector to capture deepfake features and artifacts. F3Net [38] captured global and
local forgery traces in the frequency domain. [10. 15, 74, 56] explored temporal information and inconsistency from fake videos. Most recently. DD-VQL [70] formulates deepfake detection as
sentence- sentence-generation problem, largely improving the interpretation of deepfake detection. However, these studies are restricted to facial forgery methods, which are insufficient for the current defensive systems that address diverse content produced by diffusion models. Therefore, we develop MM-Det to detect diffusion video content, pushing forward the frontier of forgery video detection.
3

---

Inst. T. Briefly describe the picture and analyze the authenticity.
Input X
Embedding
Visual Encoder
x
Reconstruction
VQ-VAR
Projection W
Large Language Model
*
CNN Encoder
Color
Object
FC- Tokens
Spatiotemporal Artifacts
Fake
Real
IAFA
Texture
Semantics
N
LMM Branch
FM
ST
ST Branch
Trainable
Frozen
Dynamic Fusion Strategy
Real
Fake
Figure 4: Multi-Modal Detection network (MM-Det) architecture. Given an input video X. the Large Multi-modal Model (LMM) branch takes the frame and instructions to generate Multi-Modal Forgery Representation (MMFR). Hidden states from the visual encoder and large language model are extracted to form the MMFR, denoted F ass which helps capture the forgery traces among different diffusion-genedn
videos. In the Temporal (ST) branch, videos are first reconstructed via VQ-VAE and then fed into a CNN encoder, followed by In-and-Across Frame Attention (IAFA) modules detailed in Sec. 3.2. IAFA is introduced to capture features based on spatial artifacts and temporal inconsistencies, termed as FST. At last, a dynamic fusion strategy combines F M and ST for the final forgery prediction.
Large Multi-modal Models (LMMs) LMMs possess generalizable problem-solving abilities in real-world tasks, including object detection[14], semantic segmentation[67] and visual question answering[590 [59]. [2, 23, 27, 28] studied feature alignment schemes to bridge visual and textual domains for LMMs. [68, 72, 71] extended the boundaries of LMMs to multi-modal downstream tasks. [45] aligns multi-moda features for capabilities on cross-domain behaviors. [58] developed
Large Language Model (LLM)-based) feature extractor for cheap-fake detection. Inspired by these studies, we stimulate the powerful perceptual and reasoning ability of an LMM by introducing the multi-modal feature space in video forgery detection.
Methods
In this section, we introduce the Multi-Modal Detection (MM-Det) framework for diffusion video detection, as depicted in Fig. 4. More formally, Sec. 3.1 details Large Multi-modal Model (LMM) branch that learns a Multi-Modal Forgery Representation (MMFR). Then Sec. 3.2 reports a Spatio- Temporal (ST) branch that utilizes In-and-Across Frame Attention (IAFA) to capture spatial artifacts and temporal inconsistencies forged videos. Lastly, dynamic fusion technique reported in Sec. 3.3 adaptively combines outputs from the LMM branch and ST branch.
3.1 Multi-Modal Forgery Representation
We propose a novel Multi-Modal Forgery Representation (MMFR) from the LMMs in LMM branch. This representation utilizes the powerful perceptual and reasoning abilities of LMMs in the form of instruction conversations.
space of
Specifically, LMM branch is built on the top of LLaVA [28], one representative LMM, which has two key components: visual encoder (e.g., D.). instantiated by visual encoders from the Contrastive Language- Image Pre- Trainini (CLIP) [39], and the large language model 'D (i.e., Llama 2 [50]). Let us denote the input video as X Nx
that contains N frames, where each frame represented as x E x xC First, x is fed to D, to obtain the corresponding visual representation
Fy E 2 Such Fy not only contains rich semantics also shows impressive generalization ability and robustness in the forgery detection task [36, 43, 9]. Then, textual instruction T is sampled from

---

P-Tokens
FC- Tokens
Self-Atm
Esr E Nxn
Wsy
Esy E MXD
Self-
Attn
Attn
Alln
Fu E AxD
Channel-wise
WM R
F ERD
Attention
(a) Mechanism of In-and-Across Frame Attention
(b) Dynamic Fusion Strategy
Figure 5: (a) In-and-Across Frame Attention (IAFA): Each input frame (or its feature map) is divided into patches that are transformed into tokens, termed P-tokens ). We introduce additional frame- centric tokens FC-tokens each transformer layer, self-attention is applied alternately among all P-tokens from video frames as well as among the same frame's P-tokens and its FC-tokens. (b) The dynamic fusion strategy captures that takes FT and F FM inputs and output channel-wise dependencies, which help refine forgery representations for the fusion.
encapsulating the global forgery information of the video frame. In
pre-defined templates Q to guide the LMM on forgery detection reasoning, Both visual representation Fv and instruction T are fed Dr. which generates enhanced visual representations (e.g. Fr.) We convert Fv into a sequence of visual tokens [28] (i.e., H, (he,m 1M m== E R RxxxD), and T is transformed into textual tokens H, [ ROXD Both H, and H, are taken as the input to DL generating Fz E Sx 1 that can be tokenized into the language response providing reasoning (Fig. about the authenticity of the input x. This procedure is formulated as
F,. = (H, H.) = (T,
(1)
where T guides the pre-trained Dr, in comprehending visual content (i.e., Fv), discerning the subset information from Fy. This instruction T enables LMM branch to obtain the multi-modal representation that leverages the generalization ability from the pre-trained large language model Llama 2 (i.e., Dr), being different to prior work [36, 9] that only relies on Fg.
Lastly, we retrieve the final MMFR, denoted as F E Mx2, by concatenating Fv and F,, after linear layer(i.e., PROJ), as
F
(RO
3.2 Capturing Spatial-Tempore Forgery Traces
Targeting capturing spatiotemporal artifacts in video tasks, introduce Spatial-Temport (ST) branch that learns effective diffusion forgery representation at the video level. Through a reconstruc- tion procedure, we amplify the diffusion traces in the frequency domain, which is then captured by In-and-Across Frame Attention (IAFA) to form an effective video-level feature.
Amplification of Diffusion Traces Similar to prior studies [55, 33] that discovered specific generative traces of diffusion models through reconstruction diffusion-genetti generated images, utilize an Autoencoder to amplify diffusion traces in videos. The reconstruction procedure can be expressed as follows.
More formally, denote the input video as X NxHxWXC that contains N frames, in which each frame is represented
x E H xC We leverage a VQ-VAE [51] to obtain the reconstructed version xf x. which is denoted as & I #X The difference between x and & makes an effective indicator of showing if the input is generated by diffusion models, depicted in Fig. 3. Therefore, we jointly proceed x and to the following proposed modules for learning an effective representation of discerning forgeries.
is worth mentioning that the prior approach [55] also adopts the idea of using the residual difference between original and reconstructed inputs to help forgery detection, but the reconstruction method requires multiple time-step denoising operations, which are computationally infeasible to reconstruct all frames from X. In contrast, our VQ-Veebbased reconstruction method only requires one single forward propagation to obtain reconstructed frame x, meanwhile preserving the effectiveness in indicating the discrepancy between real and fake inputs.
5

---

Time Time
Time
Time
Time
Time
Time
Time Time
VC1 OSora Real
SLVVD. Sora Zeerope
St.DiF Pile S..
Figure 6: Sampled videos from DVF dataset. DVF contains video generation methods, including
7 text-to-video methods and 1 image-to-video method. Real videos are selected from Internvid-
10M [54] and Youtube-8M [1]. (Key: OSora: OpenSora: VC1: Videocrafter1 [5]: Zscope: Zeroscope;
St. V. D.: Stable Video Diffusion [4]; St.Diff.: Stable Diffusion [42]; St. V:Stable Video]
Integration of Spatial and Temporal Information Same as the previous work [3, 35] that utilizes VIT to learn video-level information, transform each imput video frame (i.e., x) into L tokens, and propose IAFA for information aggregation, as depicted in Fig. 5a. Let us denote frame-level tokens as Patch-wise tokens (P-tokens), as they represent local information of one patch from More formally,
the ith frame, x; is divided into L patches, and all patches are projected into T; {t. (t 1 LxD, where represents the dimension of each P-token. Also, to capture the global forgery information for each video frame, introduce additional tokens called trame-Centric tokens (FC-tokens). The FC-token is denoted as p: E RD for frame x; and attends other tokens within the same frame X..
During the forward propagation, conduct IAFA based Transformer, with each block containing two self-attentions and consecutively modeling the local and global forgeries at cach video frame. Specifically, the first self-attention captures dependencies among P-tokens that restore local forgery clues. This is formulated by Eq. 3 that t/ RD attends the token to E RD that represents pth token from gth frame x.. Consequently, given the ith frame i.e., X,, the second self-attention is conducted among the FC-token (i.e., {p.}) and P-tokens (i.e., {t/, t/... t)- from the same frame, which encapsulates patch-wise forgery information into the global one for learning the more robust representation. We formulate this procedure in Eq. 4.
Σ ATTN(t', t2) i,j E [1, [1, P Σ ATTM(P,, t2) j1
(3)
where ATTN refers to the self-attention operation.
3.3 Dynamic Fusion
We devise the dynamic fusion strategy (i.e., Dy) that combines spatiotemporal information from ST branch and MMFR (i.e., F, and F Fm.) for the final prediction, by adjusting their contributions based forgeries from the input. More formally, D, (Fig. 5b) learns channel-wise dependencies among forgery representations (e.g., FST and Fx) via the attention mechanism, generating w E N+ as the output. This procedure can be expressed w (COCCCCAT(FST. F))).. Also, contains WST R and Wa RM representing learned channel-wise weights for Est and FAr respectively. Such channel-wise weights are important in the fusion purpose, as they help emphasize useful information we use WSr and Was to refine forgery representations as 'S = F FSTWST and F' = FM Lastly, F F's and F's, are concatenated into the fused representation Fg (M which is used for the final scalar prediction via the average pooling (i.e., AVG) and linear layers(i.e. PROJ):
(ROJ(AV PROD((AG(CONNCATFF ST: F'M)).
(5)
Diffusion Video Forensics (DVF) Dataset
We construct large-scale dataset for the video forensic task named Diffusion Video Forensics (DVF), as shown in Fig. 6. DVF contains diffusion generative methods, including Stable Diffusion [42], VideoCrafter1 [5], Zeroscope, Sora, Pika, OpenSora, Stable Video, and Stable Video Diffusion(4].
To efficiently streamline the collection, we construct an effective automated pipeline that generates forgery videos based real videos and prompts. Specifically, start from two real video datasets,

---

Line
Tixe2Video
Duration(s)
Namber
109
A motor race
VCI
Zscope
Ddft.
Text
OSoru
10P
Pika
Intertind- 1084
YouTubs-MM
St.Dift.
Zécope
Real
1k
10c
Image
USora
VCI
St.
V.
10
Vinoo2Vidoo
St.
St. V.D.
360pp
540p
720pp
1000p
(a) Generation pipeline
(b) Durations and resolutions
(c) Statistics DVF
Figure 7: The overview of DVF dataset: (a) The procedure of forged video generation and collection.
Real frames and captions sampled from Internvid- [54] and Youtube-8M [1] for text-to-video
and mogee-ooviddoo generation. (b) DVF contains videos various resolutions and durations. (c)
The statistics of DVF, measured by the numbers of frames and videos. [Key: VCI: Videocrafterl [5];
Zscope: Zeroscope; OSora: OpenSora; St.Diff.: Stable Diffusion [42]; St. V.:Stable Video; St. V. D.:
Stable Video Diffusion [4]]
Internvid-10M: [54] and Youtube-SM [1]. Real videos sampled for rich semantic content, with their frames and captions used for generation. Fig. 7 a introduces the generation process of DVE. For open-sourced generation methods, prompt is fed to a text-to-video method(i.e. VideoCrafterl, Zeroscope, OpenSora), or a frame is provided to an image-to-video method((Sabll Video Diffusion) to generate the corresponding fake video. For commercial and clureeduuredd datasets(i.e. Stable Diffusion, Stable Video, Sora, Pika), forgery videos are collected from official websites and social media. In total, collect 3, 938 fake videos and 2, 750 real vidcos in DVE. As shown in Fig. 7 b and Fig. c, our dataset contains multiple resolutions and durations. The video number each dataset varies from 0.1k to 2.8k, with the corresponding frame numbers from 4.2k to 784k. More details about DVF are provided in Appendix 8.3.1.
Training Strategy
This section details our two-stage training strategy, in which we first finetune the LMM branch via instruction tuning [28] and then optimize the entire framework in an end-to-end manner.
LMM Branch Instruction Tuning We first adapt LLaVA [28] to the forgery detection downstream task based on instruction tuning, an empirically effective way for various downstream tasks, which leverages LoRA [21] to improve the reasoning ability of Large Language Models(IL.Ms). For that, construct a large image-text paired dataset, named Rich Forgery Reasoning Dataset. Please refer to Appendix 8.3.2 for more details. We use multi-turn conversations fine-tune the LMM, enhancing its ability to identify and judge the authenticity of input images. Following the instruction tuning strategy of LLaVA [28], we only fine-tune the projection layers and LLM in LL.VA. More formally, we formulate the objective function as the loss for an auto-regressive model, which is based on answer tokens from the LLM, as:
T
((),) -- - log(po, (8"1s²<())
(6)
I-1
where sf refers to the tth prediction token, T refers the length of total prediction tokens, and 0 refers to the trainable parameters in the LMM.
End-to-End Training After fine-tuning LLaVA, use this model form LMM branch of MM- Det, and then the entire model is trained in an end-to end-to-end manner. Please note that all parameters in LMM branch are frozen ensure the optimal multi-modal representation formally, we denote MM-Det's final prediction scalar and the ground truth as s and respectively, and the model is optimized by the cross-entropy loss L follows:
be obtained. More
L(02) (y log D(v) (1 y) log(1 D(v))
where 0g refers trainable parameters in both ST branch and dynamic fusion modules.
7

---

Table 1: Video forgery detection performance on the DVF dataset measured by AUC (%). [Key:
Best; Second Best; Stable Diff.: Stable Diffusion: Avg.: Average]
Video-
Zero-
Open- Sora
Stable Diff.
Stable Video
Method
Crafterl
scope
Pika
Avg.
CNNDet [53]
85
88.2 + 1.5
78.0 1.8
6.5
77.3
2..5
7..
78.2 L.5
DIRE [55]
25
61.8
1.
60.5
65.8
6.6
29
62.
Raising [8]
63.84116
60.7 1.9
64.1
68.8
1.
78.2
62.8418
67.04.0
Uni-FD [36]
3..0
71.2 234
76.6
2.4
80.2 1.9
66.7
2
74.1 1:2
F3Net [88]
8
80.5 2.2
69.3
6.9
2:3
2.
85.1
81.3
ViVIT [3]
39.1
7..4
77.7 2.3
69.4
83.1
8..1
79.1 +
TALL [62]
76.0
65.9 1.99:..
62 1.3
61.3
72.3 :
65.8
79.5 2
69.5
TS2-Net [31]
61.8439
2.8
3.4
78.0
78.2 2.8
6..1
78.6 4:30
72.1 1.2.8
DEKFAKE [43]
1.77
68.2
35.8
64.1
8..
85.4
70.6 1.9
72.1
Hiti-Nei Net [8]
30.2
89.7 2:99
80.1 2.2
70.1
87.8 :1 2.9
89.2
83.1 2:2.
84.3
MM-Det (Ours)
93.5 1:36
94.0
88.8 AM
86.2 1:8
95.9 1:2..
95.7 =
89.9 2.0
92.0
6 Experiments
6.1 Setup
In the experiment, we use the proposed DVF for the evaluation. In training, 1, 000 videos from YouTube and 973 fake videos generated by Stable Video Diffusion serve as the training set, in which 90% are used for training and the remaining 10% for validation. Real videos from Internvid- 10M [54] and fake videos from 6 generative methods are used as testing samples. More details on training and testing are provided in Appendix 8.4.
For a fair comparison, we choose the following 10 recent detection methods as baselines. CN- NDet [53] applies a ResNet [20] as the backbone for forgery detection. F3Net [38] utilizes frequency traces left in forgery content. HiFi-Net [18] devise a specific hierarchical fine-grained learning scheme to learn wide range of forgery traces. Clip-Raising [9], Uni-FD [36] takes advantage of
a pre-trained CLIP [39] as training-free free feature space. DIRE [55] detects diffusion images based on reconstruction process of DDIM [44]. ViViT [3], TALL [62]. and TS2-Net [31] take advantage of spatiotemporal information in various visual tasks. DE-FAKE [43] adopts visual and textual representations based on a CLIP encoder for image forgery detection. For the measurement, we metric. We computer means and deviations of the
choose AUC since it is a wreeshold-indepeneee performance across 5 runs different random seeds.
6.2 Video Forgery Detection Performance
In Tab. 1, our proposed MM-Det achieves SoTA performance in detecting diffusion video, surpassing the second-best method, i.e., Het,,Net, by 7.7% the average of AUC scores. Specifically, for prior methods that are based pre-trained CLIP features, such as Raising [8] and Universal FD [36], they remain effective on certain types of diffusion content (i.e. Stable Diffusion), but fail on most others. Simple structures like CNN [53] exceed these based methods after being fine-tuned on our proposed DVF, reaching an average AUC score of 78.2%. which proves the necessity of such datasets. As for method, MMtDet outperforms other methods in most datasets. Compared with frequency-based forgery methods, e.g., HiFi-Net [38] and CLIP-based methods [8, 36), our method improves the performance from (VideoCraftterr) to mentioning that HiFi-Net makes the second-best performer in our DVF dataset, achieving 84.3% AUC scores. We believe this indicates the multi-branel feature extractor used in HiFi-Net carries versatile forgery traces at multiple resolutions, enhancing the learning of the forgery invariant. The failure of frequency traces and CLIP features raises the need for more effective features. As for spatiotemporal baselines [3, 62, 31], J. we outperform them by + 12.9%((iviiV)) + +.2.5%%(AL... and + (TS2% demonstrating the effective features of MMFR and IAFA. At last, our detector improves by 19.9% to another multi-modal detector [57]. which utilizes visual information and corresponding captions for feature enhancement. generalizable than a simple combination of visual features and text descriptions in that the powerful perceptual and reasoning abilities of LMMs play a crucial role in discriminating between real and fake content.
Diffusion). It is worth
shown that the introduction MMFR is more
8

---

Table 2: Ablation analysis measured by AUC (%). [Key: Best; Avg.: Average: Rec.: Diffusion
Reconstruction Procedure; Fus. Dynamic Fusion Strategyl.
VIT
Modules
MMFR
Crafter1
Viden-
scope
Zero
Open Sora
Sora
Pika
Stable Diff.
Stabie Vidzo
Avg.
6..2
80.1+22
61.=21
59.6
79.2
89.2
87.4
75.5=
1.
17.6=18
06.2+22
8..2+1.
77.2=1.
90.2
20.1
86
82.0
93.9
91.9
85.7
18.5
89.7
93.1+82
81.0=a
83.2=20
910+22
8.0
86.2+28
88.7=28
92.1
93
26.5
83.2+1.
90.1
90.2
87.6
89.1=2.
93.5 1336
94.0
88.8
86.2
95.9 152.91288
95.7
89.9
92.0
Original
ViVIT
IAFA
Figure 8: Visualization of artifacts captured from our IAFA and ViVIT [3]. We use activation maps to highlight spatial weights within each frame. All content is generated by VideoCrafterl. Features from the last layer of transformers extracted for visualization.
6.3 Ablation Study
Tab. 2 shows the impact of individual modules proposed in MM-Det. Specifically, we use the Hybrid ViT [11] as the base model and incorporate with the reconstruction procedure for diffusion trace amplification, which enhances the detection performance by +1.7% AUC score. Such module raises the performance in OpenSora and Sora, revealing that frequency-based information benefits forgery detection on these methods. Detection performance is further increased by using IAFA, which strengthens the learning between in-frame and cross-frame information, increasing + +3.00% AUC score to the base model. The rise in performance indicates such temporal information benefits most types of forgery video detection. After that, Tab. 2's line indicates the effectiveness of MMFR: detector purely based on such representation higher than the base model. In addition, by merging MMFR (i.e., LMM branch) and ST branch, the performance rises by +0.4%. Finally, with the dynamie fusion strategy, method receives impressive 92.0% AUC score for all generative methods, higher than every single feature. These experiments highlight the necessities of each module in our framework. Moreover, an ablation study on LLMs is detailed in Appendix 8.5 to prove the effectiveness of various LLMs in MM-Det.
receive 88. 88.7% performance in AUC, +13.2%
6.4 Spatio Temporal Information Anaylsis
In the analysis of IAFA, we visualize feature activation maps from the last layer of VIT in the ST branch based on the 1.2-norm. We compare our feature maps with another spatiotemporal baseline, ViVIT [3], depicted in Fig. 8. While attention maps of ViVIT are sparse and irregular, the ones of our IAFA have tendency to concentrate on the segmentation objects, indicating that IAFA captures typical spatial forgery regions in frames. The attention mainly focuses on common forgery traces, such as blurred generative patterns and defective parts of objects, signaling that diffusion models might find it difficult generate delicate content. The concentration of activation on certain informative objects discloses both spatial artifacts of existing generative methods, demonstrating the effectiveness of our proposed ST branch.
diffusion -eeneratd

---

Pre-trained Fine tuned
0500
Pio)
0.6
20. X D.
5t X
Pale
20
LMM Dacoder Layar
(a)
(b)
(c)
Figure 9: (a): Clustering accuracy using features from different layers in LMM branch showcases
MMFR's ability to discern forgeries. (b)(c): t-SNE [52] visualization of features from ST and LMM branches. For each dataset, 100 videos are sampled and clustered for good visibility. Both features demonstrate boundaries between real and forgery videos. (b) Features from the ST branch. (c):
Features from the LMM branch.
6.5 Multi-modal Forgery Representation Analysis
Fig. 9 details the effectiveness of MMFR in LMM branch. First, as depicted in Fig. 9 a, we quantify the detection ability of features from each Transformer-based decoder layer in the LLM. Specifically, we evaluate both pre-trained LLa VA [28] and its fine-tuned version on the task of distinguishing real and fake frames. These frames randomly sampled from 1, 000 real videos and equivalent fake ones from Stable Video Diffusion [42]. Laser-wise outputs from the large language model in LLa VA (i.e., Vicuna [50]) are obtained for i th layer in the LLM, we denote its output features f?, i E (3). f.. Empirically, we observe that features extracted from fine-tuned LLaVA show promising classification accuracy in the last few layers, e.g., 22-nd or later layers. This phenomenon indicates that specific layers in LLMs indeed generate features that can be used for image forensic tasks. Such features are utilized in MMFR to exhibit high generalization ability towards diverse and unseen forgeries. Secondly, the comparison between pre-trained and fine-tuned LLaVA highlights the importance of downstream lusk-oriented instruction tuning for LMMs. This conclusion is consistent with the findings of prior works [27, 28, 69, 68].
. The K-Means clustering algorithm is adopted to evaluate the classification accuracy based
In addition, shown in Fig. and 9 c, we analyze features from ST branch and LMM branch through t-SNE [52]. Both features achieve superior performance in separating real and forgery videos. In Fig. b, spatiotemporal information forms a rough boundary between real and fake videos. This feature is effective for VideoCrafterl [5], Zeroscope, Stable Video, and Pika, whose durations and resolutions are similar to the training set. However, the detection performance might decrease on Sora and OpenSora with overlay in the clusters. We suppose that various resolutions and durations may compromise the generalization ability, magnifying the importance of a comprehensive dataset for these videos. Fig. 9 demonstrates the more powerful feature from LMM branch. Samples from Zeroscope, Sora, and Pika are compacted into denser area, indicating the ability of LLMs to conduct generalizable reasoning. Such features provide new insights for detection when spatial and temporal artifacts are not obvious among the latest forgery videos.
7 Conclusion
In this work, we develop an effective video-level algorithm termed Multi-Modal Detection (MM-Det) for diffusiion-eenrrated video detection. MM-Det leverages a novel generalizable Multi-Modal Forgery Representation (MMFR) that is obtained from multi-modal spaces in LMMs. Specifically, the proposed MM-Det has two major branches: the LMM branch, which incorporates vision and text features from the fine-tuned foundation model, and the ST branch, which concentrates on modeling spatial-temporal information aggregated through an-and-Across Frame Attention. Extensive experiments demonstrate the effectiveness of our proposed detector. In addition, establish comprehensive dataset for various diffusion generative vidcos, which we hope will serve as general benchmark for real-world video forensic tasks.
Acknowledgement This work was supported in part by the National Natural Science Foundation of China under Grant 62301310 and 62225112, 12, and in part by Sichuan Science and Technology Program under Grant 2024NSFSC1426.
10

---

References
[1] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Patsev,, George Toderici, Balakrishnan Varadarajan. and Sudheendra Vijaynnaraiimha. Youtube-8m: large-scale video classification benchmark. arXiv preprint arXiv: 1609,08675, 2016.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-2372 2022.
[3] Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision,
pages 6836-6846, 2021.
[4] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Sealing latent video diffusion models to large datasets, arXiv preprini arXiv:231..15127, 2023.
[5] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun. Shaoshu Yang, Jinbo Xing. Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.1999 2023.
[6] Riccardo Corvi, Davide Cozzolino, Giovanni Poggi. Koki Nagano, and Luisa Verdoliva. Intriguing properties synthetic images: from generative adversarial networks to diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 973-982, 2023.
[7] Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Giovanni Poggi, Koki Nagano, and Luisa Verdo- liva. On the detection of synthetic images generated by diffusion models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[8] Davide Cozzolino, Giovanni Poggi, Riccardo Corvi, Mathhis NieBner, and Luisa Verdoliva. Raising the bar of ai-generated image detection with clip. arXiv preprint arXiv:2312.00/99) 2023.
[9] Davide Cozzolino, Giovanni Poggi, Riccardo Corvi, Matthias NieBner, and Luisa Verdoliva. Raising the bar of ai-generated image detection with clip. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages
2024.
[10] Davide Cozzolino, Andreas Rössler, Justus Thies, Mathiis NieBner, and Luisa Verdoliva. Id-reveal: Identity-aware deepfake video detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15108-15117, 2021.
[11] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. / /929, 2020.
[12] Kang Fu, Xiaohong Liu, Jun Jia, Zicheng Zhang, Yicong Peng. and Jia Wang. Rawiw: Raw image watermarking robust isp pipeline. Displays, 82:102637, 2024.
[13] Diego Gragnaniello, Davide Cozzolino, Francesco Marra, Giovanni Poggi, and Luisa Verdoliva. Are gan generated images easy detect? a critical analysis of the of In 2021 IEEE international conference on multimedia and expo (ICME), pages 1-6. IEEE, 2021.
[14] Xiuye Gu, Tsung-Y Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint aixiv:2004. 13921, 2021.
[15] Zhihao Gu, Yang Chen, Taiping Yao, Shouhong Ding, Jilin Li, Feiyue Huang, and Lizhuang Ma. Spatiotem- poral inconsistency learning for deepfake video detection. In Proceedings of the 29th ACM international conference on multimedia, pages 3473-348 2021.
[16] Xiao Guo, Vishal Asnani, Sijia Liu, and Xiaoming Liu. Tracing hyperparameter dependencies for model parsing via learnable graph pooling network. In Proceeding of Thirty-eighth Conference on Neural Information Processing Systems, Vancouver, Canada, December 2024.
[17] Xiao Guo, Xiaohong Liu, lacopo Masi, and Xiaoming Liu. tanguage-guider hierarchical tie-ggrined image forgery detection and localization. In International Journal of Computer Vision, December 2024.
[18] Xiao Guo, Xiaohong Liu, Zhiyuan Ren, Steven Grosz, lacopo Masi, and Xiaoming Liu. Hierarchical fine- iie-grained image forgery detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
11

---

[19] Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips don't lie: A generalisable and robust approach to face lorgery detection. In Proceedings the IEEE/CVF conference on computer vision and puttern recognition, pages 5039-5049, 2021.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900 2021.
[22] Yonghyun Jeong, Doyeon Kim, Seungjai Min, Seongho Joe, Youngjune Gwon, and Jongwon Choi, Bihpf: Bilateral high- pass filters for robust deepfake detection. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 48 57, 2022.
[23] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730- 19742. PMLR, 2023.
[24] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- pre-training for unified
understanding and generation. In International conference on machine learning, pages 12888 12900. PMLR, 2022.
[25] Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang. Dong Chen, Fang Wen, and Baining Guo. Face x-ray for more general face forgery detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5001-5010, 2020.
[26] Ajian Liu, Shuai Xue, Jianwen Gan, Jun Wan, Yanyan Liang, Jiankang Deng. Sergio Escalera, and Zhen Lei. Cipl-fas: Class free prompt learning for generalizable face anti-spoofing, arXiv preprint arXiv:2403. 14333, 2024.
[27] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[29] Huan Liu, Zichang Tan, Chuangchuang Tan, Yunchao Wei, Jingdong Wang, and Yao Zhao. Forgery- aware adaptive transformer for generalizable synthetic image detection. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10770-10788 2024.
[30] Xiaohong Liu, Yaojie Liu, Jun Chen, and Xiaoming Liu. Pscc-net: Progressive spatio-channel correlation network image manipulation detection and localization. IEEE Trunsactions on Circuits and Systems for Video Technology, 32(1 1):7505-755 -5100, 2022.
[31] Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. Ts2-net: Token shift and selection transformer for tixt-video retrieval. European conference on computer vision, pages 319-335. Springer. 2022.
[32] Yunpeng Luo, Junlong Du, Ke Yan, and Shouhong Ding. Lare" 2: Latent reconstruction error based method diffusion- image detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17015, 2024.
[33] Ruipeng Ma. Jinhao Duan, Fei Kong, Xiaoshuang Shi, and Kaidi Xu. Exposing the fake: Effective diffusion -ennerate images detection. arXiv preprint arXiv:2307. 06272, 2023.
[34] Sara Mandelli, Nicolò Bonettini, Paolo Bestagini, and Stefano Tubaro. Detecting gan generated images by orthogonal training of multiple cnns. In 2022 IEEE International Conference Image Processing (ICIP), pages 309100001300 -0995 IEEE, 2022.
[35] Daniel Neimark, Omri Bar. Maya Zohar, and Dotan Asselmann. Video transformer network. In P Proceedings of the IEEF/CVF international conference on computer vision, pages 3163-3172, 2021.
[36] Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24480-24489, 2023.
[37] Yongyang Pan. Xiaohong Liu, Siqi Luo, Yi Xin, Xiao Guo, Xiaoming Liu, Xiongkuo Min, and Guangtao Zhai. Towards effective user attribution for latent diffusion models via watermark edtermerk-informed blending. arXiv preprint arXiv:2409.10955 2024.
12

---

[38] Yuyang Qian, Guojun Yin. Lu Sheng, Zixuan Chen, and Jing Shao. Thinking in frequency: Face forgery detection by mining frequency-aware clues. In European conference computer vision. pages 86-103. Springer, 2020.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR. 2021.
[40] Jonas Ricker, Simon Damm, Thorsten Holz, and Asja Fischer. Towards the detection of diffusion model deepfakes. arXiv preprint wrXiv22210.1577, 2022.
[41] Jonas Ricker. Denis Lukovnikov, and Asja Fischer. Aeroblade: Training- free detection of latent diffusion images using autoencoder reconstruction error. arXiv preprint arXiv:2401. 17870. 2024.
[42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684 10695, 2022.
[43] Zeyang Sha, Zheng Li, Ning Yu, and Yang Zhang. De-fake: Detection and attribution of fake images generated by text-to-image generation models. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages
2023.
[44] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.088 2020200 2020.
[45] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction- follow them all. arXiv preprint arXiv:2305. 1635, 2023.
[46] Chuangehuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. Rethinking the up- sampling operations cnn-based generative network for generalizable deepfake detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28130-28 28139, 2024.
[47] Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, and Yunchao Wei. Learning on gradients: Generalized artifacts representation for gan-generated images detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12105-12114 12114, 2023.
[48] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family highly capable multimodal models. arXiv preprint arXiv:2312. 18055 2023.
[49] Justus Thies, Michael Zollhofer. Marc Stamminger. Christian Theobalt, and Matthias NieBner. Face2face: Real-time face capture and reenactment of rgb videos. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2387-2395 2016.
[50] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arxiv preprint wixi:2307.002880 2023.
[51] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[52] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
[53] Sheng- Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cun-generated images are surprisingly easy to spot... for now. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages (695-8704, 2020.
[54] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint 27X1v:2307.06942 2023.
[55] Zhendong Wang. Jianmin Bao, Wengang Zhou, Weilun Wang, Hezhen Hu, Hong Chen, and Houqiang Li. Dire for diffusion- generated image detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22445-22453 2023.
[56] Zhendong Wang. Jianmin Bao, Wengang Zhou, Weilun Wang, and Houqiang Li. Altfreezing for more general video face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4139-4138,.. 2023.
13

---

[57] Deressa Wodajo and Solomon Atnafu. Deepfake video detection using convolutional vision transformer. arXiv preprint arXiv:2102.1112, 2021.
[58] Guangyang Wu, Weijie Wu, Xiaohong Liu, Kele Xu, Tianjian Wan, and Wenyi Wang. Cheap-fake detection with Im using prompt engincering. In 2023 IEEF International Conference on Multimedia and Expo Workshops (ICMEW), pages 105-109. TEEE, 2023.
[59] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat- tat-Seng Chua. Next-gpt: Any-to-am any multimodal Ilm. arXiv preprint arXiv:2309.cmm 05519, 2023.
[60] Yue Wu, Wael Imagedd, and Premkumar Natarajan. Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9543-9552, 2019.
[61] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dy- namicrafter: Animating open-domain images with video diffusion priors. arXiv preprint ar 2iv:23300 12100, 2023.
[62] Yuting Xu. Jian Liang. Gengyun Jia, Ziming Yang. Yanhao Zhang, and Ran He. Tall: Thumbnail layout for deepfake video detection. In Proceedings of the IEEE/CVF international conference on computer vision. pages 22658- 226688, 2023.
[63] Zhiyuan Yan, Yuhao Luo, Siwei Lyu, Qingshan Liu, and Baoyuan Wu. Transcending forgery specificity with latent space augmentation for generalizable deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8984-8994, 2024.
[64] Yiwei Yang, Zheyuan Liu, Jun Jia, Zhongpai Gao, Yunhao Li, Wei Sun, Xiaohong Liu, and Guangtao Zhai. Diffstega: Towards universal training-free coverless image steganography with diffusion models. arXiv preprint or Xiv:2407. 10459, 2024.
[65] Yuguang Yao, Xiao Guo, Vishal Asnani. Yifan Gong, Jiancheng Liu, Xue Lin, Xiaoming Liu, Sijia Liu, et al. Reverse engineering deceptions on machine-and human-centric attacks. Foundations and Trends@ in Privacy and Security, 6(2):5--152, 2024.
[66] Ning Yu, Larry S Davis, and Mario Fritz. Attributing fake images to gans: Learning and analyzing gan fingerprints. In Proceedings of the IEFF/CVF international conference on computer vision, pages 7556-7566, 2019.
[67] Ao Zhang. Liming Zhao, Chen- Wei Xie, Yun Zheng, Wei Ji, and Tat- Tat-Seng Chua. Next-chat: An Imm for chat, detection and segmentation. arXiv preprint arXiv:2311 100498,, 2023.
[68] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction- tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.00888. 2023.
[69] Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024.
[70] Yue Zhang, Ben Colman, Xiao Guo, Ali Shahriyari, and Gaurav Bharaj. Common sense reasoning for deepfake detection. European Conference on Computer Vision, 2025.
[71] Yue Zhang. Ziqiao Ma. Jialu Li, Yanyuan Qiao, Zun Wang. Joyce Chai, Qi Wu, Mohit Bansal, and Parisa Kordjamshidi. Vision-and-language navigation today and tomorrow: A survey in the era of foundation models. arXiv preprint WwX.::24470000.355 2024.
[72] Yue Zhang, Zhiyang Xu, Ying Shen, Parisa Kordjamshidi, and Lifu Huang. Spartun3d: Situated spatial understanding of 3d world in large language models. arXiv preprint arXiv:2410.03877 2024.
[73] Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi- attentional deepfake detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2185-2194, 2021.
[74] Yinglin Zheng, Jianmin Bao. Dong Chen, Ming Zeng, and Fang Wen. Exploring temporal coherence for more general video face forgery detection. Proceedings of the IEEE/CVE international conference on computer vision, pages 15044-15054, 2021.
[75] Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.1059) 2023.
14

---

8 Appendix Supplemental Material
8.1 Limitations
Although our proposed MM-Det advances in detecting fake videos, further issues are left to handle. First, the landscape of video manipulation technology evolves, new techniques and tools will outpace existing detection methods. The gap between training data and real-world applications can lead to misleading results. Currently, our method is limited to fully synthesized diffusion videos, lacking the generalization to more delicate forgeries like partial manipulation. A possible reason is that small forgery traces disappear after multiple downsampling operations in the deep network of LMMs. Besides, the integration of a large language model into detection costs huge computational complexity, which is not an optimal choice for an environment with limited resources.
In conclusion, while our algorithm makes a critical step forward in detecting fake videos, it faces significant challenges due the rapid advancement of video manipulation technologies. Addressing these limitations requires further research to keep pace with the evolving techniques in digital content manipulation.
8.2 Broader Impacts
In this work, our team has developed an effective algorithm to detect fake videos, breakthrough that promises to fortify the authenticity of online media. In real-world social media where misinformation can spread rapidly, our method acts as crucial safeguard by empowering platforms to flag and remove deceptive videos before they can mislead users. However, methods may fail in extreme situations, such as blurred images or noisy images. The algorithm should be carefully treated to avoid misleading results. The long-term influence of work protects public trust by ensuring the authenticity of digital content. Meanwhile, precaution is needed for a fair application.
8.3 Datasets
8.3.1
Diffusion Video Forensics
We propose comprehensive dataset, named Diffusion Video Forensics (DVF), for diffusion video forensics, as shown in Tab. S1. DVF consists of fake videos generated from 8 different generation methods, covering text-to-video and image-to-video generative methods. In total, We make collection of 2, 788 real videos and 4, 111 fake videos. Real videos are from YouTube and Internvid- 10M [54].
We formally introduce the generation pipeline for video collection. Generation methods in DVF are divided into closed-sourced methods and open-sourced methods. For closed-sourced methods(Sora, Pika, Stable Diffusion [42] and Stable Video), we collect video samples from official websites and social media like TikTok to form the forgery video datasets. For open-sourced methods, the generation pipelines are divided into text-to-vided (OpenSora, VideoCrafter [5] and Zeroscope) and image-to- video (Stable Video Diffusion [4]). For a text-to-video generative method, real data derives from a text-image paired video dataset, Internvid- 10M. Specifically, we fetch paired real videos R
(r1, r2,..., rN) and corresponding captions C {c1, C2, CN We directly apply the
Table S1: Diffusion Video Forensics Composition [Key: T2V: Text-to- 12V: Image-to-Vided
Dataset
Source
Video Number
Resolution
Real Stable Video Diffusion
Youtube & Internvid-10M
2, 750
1, 280 x 720
I2V
1, 1,800
1, 024 x 576
VideoCrafterl
T2V
450
1, 024 x 576
Zeroscope
T2V
800
1,024 x 576
Sora
Social Media
153
1, 280 x 720
Pika
T2V, 12V
122
1, 1,280 X 720
OpenSora Stable Diffusion
T2V
500
512 x 512
I2V
12
1, 080 x 1, 920
Stable Vidco
T2V
101
1, 02024 x 576, 1, .20 080
15

---

Template
Instruction
Evaluate if the image real fake. Imppec the image to judge its validity
Inspect the image to judge validity and give the reasons...
Unnatural color
Rich Forgery
Reasoning
Lack of depth
Dataset
Focus on untsaturall elementts Tused objocts/odd backerounds artifacts
Gemini
Frame
Image
Text
Give concrcte reasons why it realffake.
The overview of Rich Forgery Reasoning Dataset. To obtain text-image paired data forgery reasoning, we take advantage of the powerful reasoning ability of Gemini [48] to generate ground truth for each image. First, an instruction I' is sampled from template, along with frame f xxxxxxx XxC fed into Gemini for forgery analysis and detection. The response r contains detailed judgment and reasoning the content and authenticity of f(e.g., analyses color and depth). Finally, and r are collected as the text-image paired data to form Rich Forgery Reasoning Dataset.
Figure S1:
captions as prompts to generate fake video datasets F. such that F [f, ff..f. = (e()),, [1, Nj, where Ii denotes a text-to-video method. Both R and F are included DVF as real and fake datasets. For image-to toaeeettoviddeo methods, real videos come from Youtube-SM [1], which are denoted as {ri, r2, ..., IN r r xxxxxxXxx For each video Tú a real frame I, E R "xxxxxx Wxx is randomly sampled from and serves as the conditional input for generation. The fake datasets are obtained as {fi; f,,,,. f; Jim (r:), i [1, NJ. where Jim denotes an image-to-video method.
8.3.2 Rich Forgery Reasoning Dataset
We construct text-image paired dataset, called Rich Forgery Reasoning Dataset (RFRD), to support instruction-tuning LMMs on the forgery detection task, as shown in Fig. S1. We start from the YouTube and Stable Video Diffusion dataset in DVF, where fake videos. Depending on the powerful reasoning ability of Gemini [48] v1.5 5 Pro, we follow the scheme in Fig. S1 to generate ground truth for frames. In total, 921 real frames and 3, 579 fake frames are sampled to generate 5, 500 image-text paired textual descriptions. These descriptions are then cleaned and converted into 38k multi-turn conversations for fine-tuning LLaVA [28] LMM branch of MM-Det.
select 1, 000 real videos and 1, 800
8.4 Implementation Details
Hyperparameters of MM-Det We introduce the implementation of our MM-Det. In ST branch, we employ Hybrid-Ve ViT [11] to build the video-level feature encoder. We choose Hybrid-ViTL
with a ViT-B/16 on top of a ResNet-50 backbone, the patch size 14 x 14, and the hidden size 768. We employ IAFA in each attention block of the ViT, with FC-tokens initialized as the class token of VIT. In the embedding stage, learnable spatial and temporal embeddings are introduced in the form of addition. All patches at the same position within frames share the same spatial embedding and patches with the same timestep share the same time embedding. A pre-trained VQ- VAE is applied to reconstruct videos, with hidden size 256 and the codebook size K = 512. We train the VQ-VAr VAE on 50, 000 images from ImageNet. In LMM branch, utilize LLa [28] v1.5, with CLIP [39] encoder of LLPP-Vi----ptcchl LTTT---pttch4-3366 14-336 and a large language model of Vicuna-7b for reasoning. Our proposed MMFR is composed of the pooler output Fv IR 1024 from the last layer of & and the embedding of output Fy from the last layer of D. To balance effectiveness and efficiency, we fix the length of output tokens into 64 and obtain Fz 64046006
Training and Inference As for the experimental resources in training and inference, conduct all experiments using single NVIDIA RTX 3090 GPU and a maximum of 200G memory.
The training strategy of MM-Det is in two-stage. We first conduct instruction tuning for LLaVA in the LMM branch based on LoRA [21]. We start from a pre-trained LLaVe VA vl.5 and train on our collected Rich Forgery Reasoning Dataset detailed in Sec. 8.3.2. We use Adam optimizer with the learning rate set as 2e -5 for 10 epochs. After that, we integrate V. into MM-Det and conduct the overall training. The training set is split into 9: 1 for training and validation data. For each video,
16

---

Table S2: Evaluation on different Large Language Models measured by AUC(%). [Key: Best; Avg.:
Average].
Video- Crafterl
Stable Diffusion
Stable Video
LLM
Zeroscope
OpenSora
Sora
Pika
Avg.
N/A
2.
90.1
8.
22
93.9
85.7 26
88.5
Vicuna-7b
a.6
94.0
88.8
86.2 1.
96.9
96.7 95.9 2.3
89.9
92.0 +.60
Vicuna-133 13b
95.5 2.5
93.2 +.2
89.2
23
96. 1.9
90.8
92.2 = 2.4
Mistral-7b
92.612.
92.1 8..
83.2122
94.6
93.61220
86.2
89.8 2.6
Table S3: Perfomance of MM-Det common post- processncss operations measured by AUC(%).
N/A
Blur a = 3 JPEG Q= = 50 Resize 0.7 Rotate 90 Mixed
92.0
86.2
91.1
89.9
90.1 88.6
successive 10 frames randomly sampled and cropped into 224 224 the input. We use Adam optimizer with the learning rate set as 1e-4 for training until the model converges.
For inference, we evaluate all models at the video level. For frame-level baselines, the score of an entire video is obtained as average score of all frames. For video-level methods, successive clips are fed according to the corresponding window size, and the entire score is obtained as the average score of all clips. addition, to leverage the efficiency for inference, MM-Det first caches MMFR for each video by conducting reasoning at the interval of frame every 6 seconds. During inference, each video clip directly applies MMFR from the nearest cached frame as an approximation to reduce the huge computational cost. For each baseline, conduct evaluations on different seeds (1, 100, 999, 1234, 9999) and choose the average score.
8.5 Ablation Study on LLMs
We adopt alternative LLMs in our MM-Deet to evaluate different choices of language backbones. Performance reported Tab. $2. More formally, for fair comparison, when using different L.LMs, maintain other components e.g., CLIP, the reconstruction procedure, IAFA, and the dynamic fusion the original MM-Det remained. Specifically, the introduction of a combined vision and text space from Vicuna-7b in LLaVA improves the performance by +-355%. As for the choice of LMMs, Vicuna-77 achieves an average AUC score 92.0%. +2.2% higher than Mistral-7b. We suppose this result may be attributed to different attention mechanisms in Vicuna and Mistral. Vicuna-13b gains a further improvement by +0.2% due to incremental parameters in capturing more effective multi-modal feature spaces. These results prove that our MMFR is effective and extensible to other language models.
8.6 Robustness Analysis
To analyze the robustness of our method, we conduct additional evaluation of MM-Det based on common post-processing operations. We choose Gaussian blur with a JPBG compression with quality Q 90, resize with ratio of 0.7, rotation with an angle of 90, and a mixture of all operations as unseen perturbations in real-world scenarios. Testing samples are selected from DVF to form total of 500 real videos and 500 fake videos. As reported in Tab. S3, MM-Det meets degradation of 0.9%(JEEG Compression) to G.asGannnian blur), with all performance above 86%. The results indicate the effectiveness of our method under these operations.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims fit with the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

For each theoretical result, the paper provides the full set of assumptions and complete and correct proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Although no link is available for our work now, we are strongly intended to share our work in public when the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbccccccceecce cerollccggiideee CodeeSbbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provide sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpprs////eriisss/ cc httppp/pllbtiiiceeeeeeeeeeeeeeet csGuidel ines?
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discusses both positive and negative social impacts in Boarder
Impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
The paper has no risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators and owners of assets are properly credited, and the license and terms of use explicitly mentioned and properly are respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper well documented, with the documenta- tion provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
The paper does not involve study participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "PznJolN35h", "reviewer": "Reviewer_X8oW", "summary": "This paper proposed a video-level detection algorithm, named Multi-Modal Detection (MM-Det) for video forensics. MM-Det consists of Multi-Modal Forgery Representation (MMFR) that discriminates fake videos from real ones, In-and-Across Frame Attention (IAFA) that balances frame-level forgery traces with information flow across frames, and Dynamic Fusion Strategy for augmentation in features of high correlation and suppression for helpless ones integrates perception and reasoning abilities for video forensic work. In addition, this paper establishes a high-quality dataset including videos generated from various diffusion-based algorithms. Evaluation of several benchmarks confirms the effectiveness of MM-Det on general content from unseen diffusion models.", "questions": "1.\tWhich part in Figure 3 denotes LLM?\n2.\tAnalysis and experiment to illustrate the functionality of IAFA. \n3.\tAblation study on LLM.", "limitations": "The author has fully discussed the limitations of this work and pointed out the direction of future work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1.\tThe proposed multimodal representation fusion, In-and-Across Frame Attention, and Dynamic Fusion Strategy appear valid and reasonable.\n2.\tA new data set is proposed which is valuable for video forensics research.", "weaknesses": "1.\tAs for LLM, it isn't easy to see which module represents LLM from the paper and Figure 3. If LLM is decoupled and divided into several modules, the article should give explanations. \n2.\tThe author declares that IAFA conducts global and local attention alternately. However, the paper lacks adequate analysis and experiments to illustrate the functionality of IAFA. \n3.\tBesides, LLM plays an important role in MM-Det. I think conducting an ablation study on LLM is necessary to see if different LLMs influence detection performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude to the reviewer for the insightful feedback. We appreciate the constructive positive comments on the novelty, soundness, and valuable contribution of our paper. In response, we present additional experiments and address the weaknesses and questions highlighted in the review.\n> **Q1** As for LLM, it isn't easy to see which module represents LLM from the paper and Figure 3.\n\nWe provide a more concrete explanation here. In our MM-Det framework, we apply a Large Multimodal Model to capture multimodal forgery representation, whose modules and outputs are denoted in dark blue in Figure 3 of our paper. The composition of the LMM branch is a typical structure of a multimodal large language model, including a tokenizer and an embedding layer for text input, a visual encoder for image input, and a transformer decoder layer as the language backbone. \n\n> **Q2** If LLM is decoupled and divided into several modules, the article should give explanations.\n\nOur proposed method obtains multimodal forgery representations (MMFR) by taking advantage of pretrained visual encoder and language decoder backbone from a LMM. Specifically, given an LMM composed of a vision encoder denoted as $\\\\mathcal{E}_v$, a text encoder denoted as $\\\\mathcal{E}_t$, and a language decoder backbone denoted as $\\\\mathcal{D}_t$. The input is a video sequence $\\\\{\\\\mathbf{x}\\\\}^N$ of $N$ frames and an instruction prompt $\\\\mathbf{p}$ from our forgery templates. \nWe capture informative multimodal features $\\\\mathbf{F}_m$ by conducting text generation and extracting hidden states from the last layer of $\\\\mathcal{E}_v$ and $\\\\mathcal{D}_t$, which are feature representations in both visual and textual embedding space. This process can be expressed as: \n\\begin{equation}\\\\mathbf{F}_v =\\\\mathcal{E}_v(\\\\mathbf{x})\\\\end{equation}\n\\\\begin{equation}\\\\mathbf{F}_t =\\\\mathcal{D}_t(\\\\mathbf{F}_v, \\\\mathcal{E}_t(\\\\mathbf{p}))\\\\end{equation}\n\\\\begin{equation}\\\\mathbf{F}_m=\\\\{\\\\mathbf{F}_v, \\\\mathbf{F}_t\\\\}\\\\end{equation}\nwhere $\\\\mathbf{F}_v$ and $\\\\mathbf{F}_t$ denote features in the visual and textual embedding space, respectively.\n\n> **Q3** Illustration of the functionality of IAFA\n\nWe provide further analysis and demonstrate additional experiments in Q1 of the overall response. \n\n> **Q4** An ablation study on LLMs\n\nWe provide an extensive ablation study on the influence of LLMs in Q3 of the overall response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zVpiCO3Do9", "reviewer": "Reviewer_f8wk", "summary": "This paper proposes a method for detecting diffusion-generated fake videos using a multi-modal approach. Key contributions include:\nA Multi-Modal Forgery Representation leveraging vision and language capabilities of large multimodal models \nAn In-and-Across frame attention to capture spatial-temporal forgery traces\nA fusion strategy to combine multi-modal features\nA new dataset of diffusion-generated videos for benchmarking\nThe method outperforms existing approaches on cross-dataset evaluation.", "questions": "How does the computational cost and inference speed compare to existing methods?\nHow well does the method generalize to detecting non-diffusion generated fake videos?\nCould the authors discuss the transferability of the proposed MMFR to other forms of synthetic media detection, such as audio or text?", "limitations": "Potential biases in the dataset\nScalability to longer videos or different resolutions\nAdaptability to rapidly evolving diffusion models\nPotential for overfitting to current generation artifacts", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Creation of a new dataset to address lack of benchmarks in this area\nUse of LMMs for video forgery detection, leveraging their visual and language understanding Combination of frame-level and video-level features \nEvaluation on multiple diffusion video generation methods", "weaknesses": "The use of LMMs and complex Transformer architectures might impose high computational demands, which could limit practical deployment in resource-constrained environments.\nDiscussion on Failure Cases: The paper could benefit from a detailed discussion on scenarios where the proposed method fails or underperforms, which could guide future improvements and research.\nLimited discussion of computational requirements and inference speed\nLimited exploration of how the method generalizes to non-diffusion generated videos\nit is unclear how the system performs under different operational conditions or with lower-quality videos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable and insightful feedback. We appreciate all constructive comments on the novelty, soundness, and clarity of the paper. Here, we demonstrate additional experiments and answer the weaknesses and questions mentioned in the review.\n\n> **Q1**: Computational analysis\n\nAs for computational requirements, our method is implemented on a single 4090 GPU with 24G memory for both training and inference. We provide further computational analysis and inference speed for our methods and other baselines in Table F.\n\n**Table F**: Comparison of computational cost and inference speed\n|Model|GFLOPs|Params|FPS|\n|--|--|--|--|\n|ViViT[1]|$84.1$|$26.4M$|$1201$|\n|TALL[2]|$30.4$|$86.6M$|$1445$|\n|UniversalFD[3]|$1556.4$|$304.5M$|$93$|\n|MM-Det(Ours)|$9345.2$|$6919M$|$40$|\n\nCompared with other baselines, our method reaches 6x GFLOPs and 23x Params of UniversalFD[3], which also uses a CLIP encoder for detection. As for inference speed, we conduct video-level inference at 40 fps. Regarding the computational cost and efficiency, we argue that the bottleneck of our method lies in the integration of the large multimodal model, which makes up 94% FLOPs in inference and 98% Params in total. With the improvement of LMM techniques and the appearance of more computational-friendly LMMs, such limitations will be solved. In addition, LMM inference on cloud service is available in practical deployment. Therefore, we hold the opinion that the computational cost of our method will be relieved to a large extent in the future. Further discussion is beyond the scope of our paper.\n\n> **Q2**: Generalization ability\n\nWe report the performance of our method on multiple GAN-based methods in Q2 of the overall response, where we provide extensive experiments and results on \n\n1) Generalization ability on non-diffusion videos,\n\n2) Generalization ability on evolving diffusion videos,\n\n3) Scalability to multiple durations and resolutions.\n\nWith all promising performance, we prove that our method is effective in most cases.\n\n> **Q3**: Robustness analysis\n\nWe provide additional experiments for robustness analysis in Table G.\n\n**Table G**: Performance of MM-Det on multiple post-processing in AUC\n|N/A|Blur $\\\\sigma=3$|JPEG $Q=50$|Resize $0.7$|Rotate $90$|Mixed|\n|:--:|:--:|:--:|:--:|:--:|:--:|\n|$95.5$|$89.2$|$93.2$|$91.7$|$92.1$|$91.9$|\n\nOur method remains effective under all such post-processing conditions.\n\n> **Q4**: Transferability to other modalities\n\nOur proposed MMFR takes advantage of feature spaces from multimodal encoders and text decoders in LMMs to form an effective representation for forgery detection tasks, which is a transferable method to other media. As long as the LMM is adapted to downstream tasks in audio or text, features from the encoder and decoder stand for a strong representation of unseen synthetic types. Therefore, our method is generalizable to other modalities.\n\n> **Q5**: Discussion on failure cases\n\nWhile our paper makes significant strides in addressing diffusion video detection, our exploration does not extensively cover partially manipulated content. For diffusion manipulations that only happen in small areas, our method may fail to capture informative features. A possible reason is that small forgery traces disappear after multiple downsampling operations in the deep network of LMMs. Future research endeavors may benefit from investigating the limitation to tackle the challenge of minor forgery detection.\n\n**References**\n\n[1] Arnab A, Dehghani M, Heigold G, et al. Vivit: A video vision transformer\n\n[2] Xu Y, Liang J, Jia G, et al. Tall: Thumbnail layout for deepfake video detection\n\n[3] Ojha U, Li Y, Lee Y J. Towards universal fake image detectors that generalize across generative models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gFx8F40wJk", "reviewer": "Reviewer_H1bu", "summary": "This manuscript presents a new approach to detect fake videos generated with diffusion models. This approach is based on multimodality analysis and reports promising results on a database introduced by the author(s). These results are based on both frame and video levels. This work is well written and organized.", "questions": "How can you add the method to a general method to detect all or most forgery video types?\n\nWhat do you think will happen if you have created a video using other methods such as GAN methods?", "limitations": "These limitations are addressed by the author(s).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This poropsed approach sounds like a good novelty and shows its effectiveness with a good ablation study. As the type of generators increases, the methods can protect the users from the generated videos with fraudulent aspects in the real world. The users can verify the videos with the help of the methods.", "weaknesses": "A clear justification for the complex approach is not provided, and readers expect more details.\n\nI think that the method is limited in a global application to the recognition of videos generated with diffusion models and that we should use other methods for other types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude to the reviewer for the insightful and valuable feedback.  We appreciate all constructive comments on the novelty, effectiveness, and clarity of the paper. We provide additional experiments and answer the weaknesses and questions mentioned in the review as follows.\n\n> **Q1**: A clear justification on MM-Det\n\nIn our paper, We propose Multi-Modal Detection (MM-Det) for forgery detection. Briefly, the method integrates Multi-Modal Forgery Representations (MMFR) in the LMM branch with highly effective spatiotemporal features in the ST branch. In addition, A dynamic fusion strategy is applied to aggregate these features and adjust feature weights. Our proposed In-and-Across Frame Attention (IAFA) benefits our proposed method in effectively capturing forgery features. Besides, our method maintains a strong generalization ability on unseen forgery types with the help of large multimodal models. \n\nWe provide detailed analyses and experiments for the functionality and effectiveness of our proposed MMFR and IAFA in the overall response. Specially, we provide\n\n1) the functionality of IAFA compared with other baselines in Q1,\n\n2) the generalization of our method on GAN and latest diffusion methods in Q2,\n\n3) the performance of our method on different video durations and resolutions in Q2,\n\n4) the ablation study on different LLMs for our method in Q3.\n\n> **Q2**: Generalization ability\n\nWe provide additional performance results on GAN and latest diffusion methods in Q2 of the overall response to prove the generalization ability of our method on other types of forgery methods.\n\n> **Q3**: Improvements on more forgery content\n\nAdditional performance results on GAN and latest diffusion methods in Q2 of the overall response prove the effectiveness of our method, showing that our finetuned large multimodal model specializes in forgery detection and demonstrates generalization ability on most forgery types. For adaptation to more forgery content,  enlarging the number and types of datasets and fine-tuning large multimodal models contributes to more promising performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hP3yvrK1Rt", "reviewer": "Reviewer_LBha", "summary": "This work focuses on diffusion model detection, a core and popular research topic recently. It identifies limitations in previous studies that concentrate on fake face and image-level detection and explores the idea of using recent LMM to detect forgery. The proposed Multi-modal Forgery Representation (MMFR) leverages existing LMM and introduces a new in-and-across frame attention mechanism. Additionally, a new dataset is proposed, and empirical results demonstrate the effectiveness of the proposed methods.", "questions": "Please refer to the weakness section.\n\n1. The overall work is novel and demonstrates strong empirical performance, but I have minor concerns regarding its contributions, specifically the in-and-across attention mechanism and the dataset. \n\n2. Additionally, more related work should be discussed and compared to provide a comprehensive context.", "limitations": "Yes, the authors adequately discussed the limitations and societal impact of their work in Appendix.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The motivation behind this work is clear, and the idea of using LMMs for image forensics is novel. This idea potential pioneers a new research direction.\n\n2. Constructing a large-scale diffusion video dataset for the community is a reasonable endeavor and can benefit the entire research community.\n\n3. I personally appreciate the analysis in section 4.4, which explains how the authors designed the MMFR and utilized outputs from specific layers of LLama.\n\n4. The main result suggests the proposed method achieves SoTA video-level performance and can be used as the baseline method for the future works.", "weaknesses": "1. To me, the contribution of in-and-across frame attention seems limited. Providing a comparison to other video-level ViTs and more details about its differences from other works regarding ViT would make this contribution more convincing.\n\n2. The DE-FAKE [R1] method also adopts multi-modal representations, but this work is neither discussed nor compared in the paper. \n\n [R1: DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Models]\n\n3. The proposed dataset is a significant contribution of the paper. Therefore, instead of individual frames, it would be more convincing to include sample video frames either in the supplementary material or on the project page.\n\n4. The main performance table needs more details and a clarification on why it contains both frame and video comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all insightful comments and suggestions. We appreciate the appraisal of the contribution, soundness, and clarity of the paper. We provide additional experiments and answer the weaknesses and questions in the review.\n\n> **Q1**: Comparison with other spatiotemporal networks\n\nWe provide a comparison of IAFA with other ViT baselines and analyze our advantages in Q1 of the overall response.\n\n> **Q2**: Discussion and comparison with other multimodal baselines\n\nIn previous studies on forgery detection, DEFAKE[1] adopts multimodal representation for forgery detection by using semantic descriptive prompts as an augmentation for visual features. However, semantic descriptions do not have strong correlations with the authenticity of images, which is ineffective in forgery tasks. As a comparison, our method utilizes the powerful reasoning ability of finetuned Large Multimodal Models (LMMs) to provide more accurate features in the text space. Therefore, our method is more effective in forgery detection.\n\nWe provide a comparison of DEFAKE[1] with our method in Table H. Each model is trained on the same training dataset SVD, and evaluated on other diffusion datasets in our proposed dataset DVF.\n\n**Table H**: Comparison between DEFAKE and our method in AUC\n|Model|Videocrafter|Zeroscope|Sora|Pika|Opensora|Stable Diffusion|Stable Video|Average|\n|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|DEFAKE(SIGSAC2023)|$72.3$|$70.3$|$67.3$|$88.4$|$53.6$|$86.0$|$74.1$|$73.1$|\n|Ours|$99.2$|$98.4$|$97.4$|$95.5$|$99.4$|$98.0$|$98.4$|$98.0$|\n\nAs is demonstrated, our proposed multimodal representation is effective in most diffusion forgery content.\n\n> **Q3**: Visualization of DVF\n\nWe will release sample video frames from DVF instead of individual frames for better visualization.\n\n> **Q4**: A clarification of the main table\n\nTo justify the effectiveness of our method both on the image-level and video-level detection, the main performance table contains both frame and video comparisons. For frame-level detection, frames from each video are treated individually for training and testing to demonstrate the generalization ability of our proposed Multi-Modal Forgery Representation(MMFR) without spatio-temporal information. For video-level detection, continuous video clips are forwarded during training and evaluation. In this case, both multimodal representation and spatiotemporal information take effect. We use these experiment settings to validate the effectiveness of MMFR and IAFA.\n\n> **Q5**: Discussion and comparison with more works\n\nWe provide additional analysis and experiments regarding more related works.\n\nIn the overall response, we provide\n\n1) the functionality of IAFA compared with other baselines in Q1,\n\n2) the generalization of our method on GAN and latest diffusion methods in Q2,\n\n3) the performance of our method on different video durations and resolutions in Q2,\n\n4) the ablation study on different LLMs for our method in Q3.\n\nIn the PDF material, we provide\n\n1) Figure A:  Visualization on attention heatmaps from IAFA.\n\n**References**\n\n[1] Sha Z, Li Z, Yu N, et al. De-fake: Detection and attribution of fake images generated by text-to-image generation models\n\n[2] Arnab A, Dehghani M, Heigold G, et al. Vivit: A video vision transformer", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F5trGL235H", "reviewer": "Reviewer_Niir", "summary": "To detect video forensics, the authors propose an innovative Multi-Modal Forgery Representation (MMFR) to discriminate fake videos from real ones. Besides, the authors establish a high-quality dataset including videos generated from various diffusion-based algorithems to evaluate the effectiness of the proposed detector.", "questions": "From the visualization results, the colors of the fake image are more colorful, is the author's method still valid if the brightness is adjusted to make them similar to the real image?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "（1）The authors propose a video-level detection algorithm, named MM-Det, to capture forgery traces based on an LMM-based representation for the generalizable fake video detection. MM-Det utilizes perception and reasoning capabilities from LMMs through multi-turn conversations to learn a generalizable forgery feature.  \n（2）The authors extent reconstruction representation for detecting diffusion images into the video domain to amplify diffusion artifacts both in spatial and temporal information.  \n（3）The authors are the first to establish a comprehensive dataset for diffusion-generated videos, named DVF. The proposed detector achieves promising detection performance on a wide spectrum of forgery videos.", "weaknesses": "(1) I didn't understand how the authors use VQVAE to amplify diffusion features, and it seems that this reconstruction branch is not shown in Figure 3.\n(2) There have been some works [1,2] that incorporate temporal token in self-attention, is this different from IAFA? What are the advantages of the proposed method?  \n[1] Wang J, Yang X, Li H, et al. Efficient video transformers with spatial-temporal token selection[C]//European Conference on Computer Vision. Cham. Springer Nature Switzerland, 2022: 69-86.  \n[2] Liu Y, Xiong P, Xu L, et al. Ts2-net: Token shift and selection transformer for text-video retrieval[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 319-335.  \n(3) As shown in Section 4.2, the authors collected a training set and a test set to evaluate the performance of different methods. What is the relationship between these data and DVF? The article does not seem to describe the benefits of the DVF, e.g., how much performance improvement does the DVF bring to the authors' method, and how much performance improvement does the DVF bring to other methods?  \n(4) Forgery video detection is challenging task, however, the authors' experimental content is too small and simple, only 2 tables of experimental content is difficult to fully verify the effectiveness of the proposed method. The authors need to do some other types of experiments to further validate the performance of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful and helpful feedback. We appreciate all constructive comments on the novelty, clarity, and valuable contribution of the paper. We demonstrate additional experiments and answer all weaknesses and questions mentioned in the review. \n> **Q1** The usage of VQVAE\n\nAs for diffusion detection, a series of previous works[1,2] found that the reconstruction process of autoencoders extracts discriminative features for diffusion. Following [2], our work introduces a pretrained vector quantized variational autoencoder (VQVAE) for an augmentation to diffusion forgery features. We display the contribution of VQVAE reconstruction in Figure B of our PDF material.\n\nFor brevity and legibility, we do not visualize an explicit reconstruction process in Figure 3 of our paper, as it is not the main contribution of our method. We have provided a detailed explanation of this process in Section 3.2, Lines 147-158 of our paper.\n\n> **Q2** Analysis of IAFA\n\nWe provide analysis and experiments in Q1 of the overall response.\n\n> **Q3**: The relationship between experiment datasets and our proposed DVF\n\nBoth training and testing datasets are subsets of DVF. They do not overlap with each other and form the entire DVF together.\n\nSpecifically, the dataset DVF includes real videos from Internvid-10M and fake videos from 8 diffusion methods. In experiments, we choose one diffusion method, Stable Video Diffusion, as the training sets, and evaluate all baselines on the other 7 diffusion methods.\n\n> **Q4**: Contribution of DVF\n\nWe provide additional experiments to prove the contribution of DVF on performance in Table I. For our method, we emphasize the contribution of DVF in finetuning LMMs. For other methods, DVF provides diffusion traces for forgery detection. We prove the effectiveness of DVF by comparing pretrained baselines with ones fine-tuned on DVF, for which we choose UnviersalFD[5] and F3Net[6], and comparing our method between pretrained and fine-tuned LMMs.\n\n**Table I**: Performance of pretrained and fine-tuned detectors on DVF in terms of AUC\n|Model|Videocrafter|Zeroscope|Sora|Pika|Opensora|Stable Diffusion|Stable Video|Average|\n|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|UniversalFD(Pretrained)|$59.2$|$69.4$|$65.6$|$65.2$|$61.7|$81.2$|$59.2$|$65.9$|\n|UniversalFD(Finetuned)|$93.6$|$90.1$|$85.4$|$93.0$|$83.9$|$81.5$|$87.9$|$87.9$|\n|F3Net(Pretrained)|$60.1$|$65.2$|$60.7$|$63.6$|$59.2$|$65.8$|$61.3$|$62.3$|\n|F3Net(Finetuned)|$96.1$|$91.8$|$66.0$|$95.6$|$85.9$|$86.3$|$96.0$|$88.2$|\n|Ours(w/Pretrained LMM)|$95.3$|$95.2$|$89.6$|$94.3$|$90.2$|$86.6$|$96.7$|$92.6$|\n|Ours(w/Finetuned LMM)|$99.2$|$98.4$|$97.4$|$95.5$|$99.4$|$98.0$|$98.4$|$98.0$|\n\nAs is shown, DVF benefits other baselines in $+22.0\\\\%$ and $+25.9\\\\%$ in AUC. For our method, the LMM finetuned on DVF also outperforms pretrained one by $+5.4\\\\%$ in AUC.\n\n> **Q5**: More experiments on validation of our method\n\nWe provide further analysis and experiments to validate our method. In the overall response, we provide\n\n1) the functionality of IAFA compared with other baselines in Q1,\n\n2) the generalization of our method on GAN and latest diffusion methods in Q2,\n\n3) the performance of our method on different video durations and resolutions in Q2,\n\n4) the ablation study on different LLMs in Q3.\n\nIn our PDF material, we provide\n\n1) Figure A: attention heatmaps of our proposed IAFA compared with other spatiotemporal networks.\n\n> **Q6**: From the visualization results, the colors of the fake image are more colorful, is the author's method still valid if the brightness is adjusted to make them similar to the real image?\n\nOur method is effective after adjusting the brightness. We provide more visualization results of our LMMs on both real and fake content with similar brightness in Figure C of our PDF material.\n\n**References**\n\n[1] Wang Z, Bao J, Zhou W, et al. Dire for diffusion-generated image detection\n\n[2] Ricker J, Lukovnikov D, Fischer A. AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error\n\n[3] Wang J, Yang X, Li H, et al. Efficient video transformers with spatial-temporal token selection\n\n[4] Liu Y, Xiong P, Xu L, et al. Ts2-net: Token shift and selection transformer for text-video retrieval\n\n[5] Ojha U, Li Y, Lee Y J. Towards universal fake image detectors that generalize across generative models\n\n[6] Qian Y, Yin G, Sheng L, et al. Thinking in frequency: Face forgery detection by mining frequency-aware clues", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers for their valuable comments and suggestions. We are delighted to see (a) all reviewers give positive feedback, (b) all reviewers recognize our proposed MMFR's novelty, (c) our MMDet achieves promising and generalizable performance on diffusion forgery detection (f8wk, H1bu, LBha, Niir), (d) a comprehensive dataset is proposed for future work (X8oW, f8wk, LBha, Niir) and (e) the insightful ablation study has been appreciated (H1bu). Moreover, we receive concerns about (a) illustration of the functionality of IAFA (X8oW, LBha, Niir), (b) MM-Det's generalization ability to more forgeries (f8wk, H1bu, LBha, Niir), and (c) additional justification on our MM-Det  (X8oW, H1bu, LBha, Niir). Therefore, we provide precise answers to all questions, which mainly include 1) additional analysis and experiments on IAFA, (2) experiments on the generalization ability of MM-Det, and (3) an ablation study on LLMs.
Additionally, please check our PDF attachment for more figures.

> **Q1**. Analysis and experiments of In-and-Across-Frame Attention

Spatio-temporal networks[1-3] have been discussed in the field of video-level tasks, such as video understanding and retrieval. These works utilize spatial and temporal attention to capture video-level information and represent the global attributes of videos. However, It is worth noticing that AI-generated videos often contain inconsistent frames, giving rise to forgery artifacts that happen on small periods or even single frames. This property makes it difficult for conventional video methods to capture local artifacts. To address this problem, we propose our effective attention mechanism IAFA, which specializes in local information aggregation.

The advantage of our IAFA over other baselines lies in that IAFA preserves local information at the frame level when conducting spatiotemporal attention, thus being effective in learning forgery features. Specifically, an additional temporal token is introduced in each frame for aggregation of frame-level forgery features. During forward propagation, our designed IAFA conducts in-and-across-frame attention to model both local and global forgeries at each frame consecutively. 

To prove the effectiveness of IAFA, we compare the detection performance of IAFA with other spatiotemporal baselines in Table A. Our IAFA is based on a Hybrid ViT[4], and we select TS2-Net[1], ViViT[2], TALL[3], and Hybrid ViT[4] for comparison. Each model is trained on the same training dataset SVD from our proposed DVF, and evaluated on the rest diffusion datasets. 

**Table A**: Comparison between IAFA and spatiotemporal baselines in AUC
|Model|Videocrafter|Zeroscope|Sora|Pika|Opensora|Stable Diffusion|Stable Video|Average|
|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|Hybrid ViT|$72.3$|$70.3$|$67.3$|$88.4$|$53.6$|$86.0$|$74.1$|$73.1$|
|ViViT(CVPR2021)|$89.2$|$88.0$|$81.6$|$92.7$|$85.2$|$88.1$|$92.1$|$88.1$|
|TS2-Net(ECCV2022)|$60.7$|$72.0$|$81.0$|$80.2$|$74.3$|$60.2$|$80.2$|$72.7$|
|TALL(CVPR2023) |$76.5$|$61.8$|$62.3$|$79.9$|$69.8$|$85.9$|$64.8$|$71.6$|
|Hybrid ViT w/IAFA(Ours)|$94.4$|$94.2$|$82.0$|$95.4$|$82.0$|$92.8$|$93.9$|$90.6$|

Overall, our proposed IAFA achieves the best performance in the evaluation of unseen forgeries.

To further validate the functionality of IAFA on the preservation of local information, we provide the comparison of attention heat maps between IAFA and ViViT[2] in Figure A of our PDF material.

> **Q2**: Generalization ability

We provide an extensive performance of MM-Det on more generated videos in Table B. We choose 4 GAN-based methods[5-8] and one diffusion tool Kling (released on July 8th) for comparison. It is worthwhile noting that Kling was released after our submission.

**Table B**: Performance on GAN and diffusion videos in AUC
|StyleGAN-V[5]|StyleSV[6]|StyleSV-MTM[7]|TATS[8]|Kling| Average|
|:--:|:--:|:--:|:--:|:--:|:--:|
|$97.2$|$95.6$|$99.8$|$99.9$|$99.8$|$98.5$|

Our method gains competitive results, proving the generalization ability on unseen forgery types.

In addition, we report the performance of our method regarding video lengths and resolutions in Tables C and D.

**Table C**: Scalability on resolution in AUC
|1920x1080|1280x720|1024x576|512x512|
|:--:|:--:|:--:|:--:|
|$97.0$|$96.8$|$94.4$|$97.2$|

**Table D**: Scalability on duration in AUC
|[0, 2)s|[2, 10)s|[10, 20)s|>20s|
|:--:|:--:|:--:|:--:|
|$94.1$|$96.6$|$97.5$|$96.8$|

Our method is generalizable in multiple durations and resolutions.

> **Q3**: Ablation study on LLMs

We conduct an extensive ablation study on the influence of LLMs in Table E. We choose Vicuna-7b, Vicuna-13b and Mistral-7b as the LLM backbone.

**Table E**: Ablation study on LLMs in AUC
|LLM|Videocrafter|Zeroscope|Sora|Pika|Opensora|Stable Diffusion|Stable Video|Average|
|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|N/A|$94.4$|$94.2$|$82.0$|$95.4$|$82.0$|$92.8$|$93.9$|$90.6$|
|Vicuna-7b|$99.2$|$98.4$|$97.4$|$95.5$|$99.4$|$98.0$|$98.4$|$98.0$|
|Vicuna-13b|$98.4$|$98.9$|$97.4$|$95.4$|$99.5$|$97.6$|$98.8$|$98.0$|
|Mistral-7b|$98.1$|$98.5$|$95.6$|$96.6$|$99.4$|$96.3$|$98.9$|$97.6$|

Overall, our proposed method is effective in different LLMs.

**References**

[1] Liu Y, Xiong P, Xu L, et al. Ts2-net: Token shift and selection transformer for text-video retrieval

[2] Arnab A, Dehghani M, Heigold G, et al. Vivit: A video vision transformer

[3] Xu Y, Liang J, Jia G, et al. Tall: Thumbnail layout for deepfake video detection

[4] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale

[5] Skorokhodov I, Tulyakov S, Elhoseiny M. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2

[6] Zhang Q, Yang C, Shen Y, et al. Towards smooth video composition

[7] Yang C, Zhang Q, Xu Y, et al. Learning modulated transformation in GANs

[8] Ge S, Hayes T, Yang H, et al. Long video generation with time-agnostic vqgan and time-sensitive transformer

Meta-review: This manuscript studied AI-Generated video detection based on LLM and learning multi-model forgery representation. A new dataset is also presented to address the lack of benchmarks in AIGC detection.
The four reviewers generally appreciate the contributions of this paper and give positive comments.
As for the weaknesses posed in the review process, the reviewers also are satisfied with the responses from the authors.
Hence, the AC suggests to accept this manuscript.
Decision: Accept (poster)