The Coherence Trap: When MLLM-Crafted Narratives Exploit Manipulated Visual Contexts

## Abstract

Abstract The detection and grounding of multimedia manipulation has emerged as a critical challenge in combating AI-generated disinformation. While existing methods have made progress in recent years, we identify two fundamental limitations in current approaches: (1) Underestimation of MLLM-driven deception risk: prevailing techniques primarily address rule-based text manipulations, yet fail to account for sophisticated misinformation synthesized by multimodal large language models (MLLMs) that can dynamically generate semantically coherent, contextually plausible yet deceptive narratives conditioned on manipulated images; (2) Unrealistic misalignment artifacts: currently focused scenarios rely on artificially misaligned content that lacks semantic coherence, rendering them easily detectable. To address these gaps holistically, we propose a new adversarial pipeline that leverages MLLMs to generate high-risk disinformation. Our approach begins with constructing the MLLM-Driven Synthetic Multimodal (MDSM) dataset, where images are first altered using state-of-the-art editing techniques and then paired with MLLM-generated deceptive texts that maintain semantic consistency with the visual manipulations. Building upon this foundation, we present the A rtifact-aware M anipulation D iagnosis via MLLM (AMD) framework featuring two key innovations: Artifact Pre-perception Encoding strategy and Manipulation-Oriented Reasoning, to tame MLLMs for the MDSM problem. Comprehensive experiments validate our framework’s superior generalization capabilities as a unified architecture for detecting MLLM-powered multimodal deceptions.

## 1 Introduction

[Uncaptioned image]: x1.png

Generative techniques have become a transformative force in artificial intelligence[44, 48, 31, 10, 4], demonstrating remarkable adaptability across various domains and modalities.
These advancements, while enriching multimedia content, also pose significant challenges to information security. In the media industry in particular, maliciously fake content manipulated by such models can profoundly mislead audiences [51, 37].
The unchecked spread of fake media has already negatively affected political, financial, and other sectors[6, 32, 35], gradually becoming a major social issue[27].

To confront the above issues, many attempts have been made to detect fake multimedia and ground fine-grained forgery components[21, 38, 52, 49]. While various fake news scenarios have been explored, including Luo *et al*.’s[24] investigation of out-of-context social image-text pairs and Shao *et al*.’s[38] work on detecting randomly tampered regions or words, our analysis reveals two critical limitations in existing research: 1) Neglect of emerging risks from MLLMs: Current paradigms predominantly address rule-based text manipulation, overlooking the sophisticated linguistic capabilities of modern multi-modal large language models (MLLMs). MLLM-generated text exhibits superior fluency and contextual coherence, significantly increasing deception potential and public susceptibility. 2) Semantic misaligned artifacts. Most methodologies independently manipulate visual and textual elements, producing semantically discordant multimedia outputs. This misalignment not only renders manufactured disinformation too simplistic to effectively deceive the public, but also fails to replicate real-world adversarial behavior, as sophisticated attackers typically maintain meticulous visual-textual consistency to maximize manipulative impact. These limitations render the multi-modal disinformation scenarios considered in existing works insufficiently realistic.

To address these weaknesses, we take MLLM into consideration and focus on detecting the semantic-aligned manipulation. We first construct the MLLM-Driven Synthetic Multi-modal (MDSM) dataset, which manipulates image and text in a coordinated fashion using MLLM. For the image manipulation, we consider the typical Face Swap and Face Attribute editing. For text, We innovatively guide MLLM to generate modality-aligned yet misleading fake narratives using image editing metadata. As shown in Fig. [1](https://arxiv.org/html/2505.17476v1#S1)(b), after replacing Donald Trump’s face with Micheál Martin’s, we use the swapped name, Micheál Martin, to guide MLLM in generating text, ensuring that the named entity in the text aligns with the image. Following this strategy, we construct over 441k sample pairs.

The alignment of modalities and the authentic texts from MLLMs pose significant challenges for the detection of manipulated media. First, the strategy of perceiving inconsistencies between images and text through contrastive learning, as employed by prior works[52, 38], is ineffective in MDSM where images and text are well-matched already. Merely observing aligned image-text pairs is inadequate for reliable detection. Consequently, external clues and contextual knowledge are essential. Second, existing architectures like ASAP[52] and HAMMER[38], which feature multiple detection and grounding heads, are complex and lack generalizability to unseen media sources. To address these challenges, we propose Artifact-aware Manipulation Diagnosis via MLLM (AMD), which leverages MLLMs’ comprehensive understanding of real-world multimedia and their ability to provide unified textual outputs. And AMD generates detection and grounding results in a coherent, text-based format, offering a more intuitive and generalized solution.

In summary, we highlight our contributions of this paper as follows:

- •
We make an early exploration to detect and ground the MLLM-driven manipulation in multimedia and establish an MLLM-Driven Synthetic Multimodal (MDSM) dataset, which defines a more challenging and practical problem for misinformation detecting.
- •
We propose an Artifact-aware Manipulation Diagnosis framework (AMD) for the MDSM problem that synergizes artifact pre-perception encoding and manipulation-oriented reasoning to effectively adapt MLLMs for precise manipulation analysis.
- •
Comprehensive evaluations validate AMD’s effectiveness and generalization capability, outperforming existing methods while maintaining parameter efficiency.

## 2 Related Work

Deepfake Detection.
With the rapid advancement of generative models and the explosive growth of generated data, Deepfake detection technology has also progressed swiftly, enabling it to address the information risks posed by counterfeit data. Historically, Deepfake detection has been categorized into unimodal and multimodal detection approaches. Unimodal detection includes both image-based [53, 19] and text-based [40, 12] methods, all of which have achieved strong intra-domain performance.
Recently, with the rise of Multimodal Large Language Models (MLLMs), multimodal Deepfake detection has garnered increasing attention [21, 38, 22]. Among them, HAMMER [38] combines contrastive learning
to build a robust detection model that not only classifies manipulation types but also grounds manipulation locations. However, HAMMER does not address the enhancement of cross-domain performance. FKA-Owl [22], which is based on MLLM, incorporates more world knowledge to improve the model’s cross-domain performance. Despite this, FKA-Owl lacks grounding functionality and, due to the incorporation of MLLM, has become particularly cumbersome.

Multi-Modal Large Language Model.
In recent years, Multi-Modal Large Language Models (MLLMs) have emerged as a crucial technology for understanding and reasoning across multiple modalities, particularly text and images. By extending the capabilities of Large Language Models (LLMs) to incorporate visual inputs, these models have demonstrated outstanding performance in tasks such as image captioning and visual question answering. CLIP [33] and ALIGN [15] leveraged contrastive learning to align visual and textual representations, enabling efficient zero-shot vision-language understanding. Subsequently, models such as Flamingo [1] and BLIP-2 [18] have introduced vision-language transformers, integrating pre-trained LLMs with vision encoders to enhance cross-modal reasoning and generative capabilities. More recently, GPT-4V [28] and Florence-2 [4] have significantly enhanced the potential of MLLMs in tackling complex multi-modal tasks by leveraging a more efficient framework and larger-scale pre-training data.
A key advantage of MLLMs is their ability to acquire extensive world knowledge during large-scale pretraining, which significantly enhances their reasoning capabilities in downstream tasks.

In this work, we fully leverage the intrinsic world knowledge of MLLMs to enhance the robustness of multi-modal manipulation detection and improve inference capabilities in unknown scenarios.

## 3 MDSM Dataset Construction

Figure: Figure 1: Construction pipeline of MDSM. (a) Face Swap and Text Fabrication steps, where Trump’s face is swapped with Musk’s. (b) Face Attribute and Text Fabrication steps, where Hammond’s expression is edited to anger. (c) Leveraging MLLM to analyze manipulated facial identities in the image and craft tailored prompts to guide it in generating forgery texts.
Refer to caption: x2.png

Fig. [1](https://arxiv.org/html/2505.17476v1#S3.F1) illustrates the MDSM construction pipeline. Multi-modal news samples are first collected from diverse sources, alongside a Celebrity Head-shot reference dataset. These resources then drive a two-stage synthesis process: 1) advanced image editing models generate visual manipulations, 2) MLLMs produce semantically aligned textual narratives. We elaborate on these components below.

### 3.1 Multi-Modal Media Source Collection

We use the GoodNews [5], VisualNews [11], and N24News [54] datasets as the original datasets $\mathcal{D}_{O}$,
which consist of image-text pairs sourced from various real-world news outlets. Given the significant influence of human-centric news among various forms of multi-modal media, we focus on human-centric data for MDSM. $\mathcal{D}_{O}$ is filed by detecting faces in images with Dlib [17] and identifying person names in texts with BERT [8]. Only pairs containing both faces and named entities are kept in the source pool $\mathcal{D}_{S}=\{p_{s}|p_{s}=(I_{s},T_{s})\}$ for manipulation.

### 3.2 Celebrity Head-shot Dataset

Before detailing the construction of MDSM, we first collect the Celebrity Head-shot Dataset $\mathcal{D}_{C}$, which contains over 29k pairs of head-shot images and corresponding names to facilitate the aligned manipulation of multi-modal media.

### 3.3 Multi-Modal Media Manipulation

In the image modality, two main attacks, Face Swap (FS) and Face Attribute (FA), are employed. For the text modality, we utilize advanced MLLMs to generate semantic-aligned texts for the images.

$\triangleright$ Face Swap Manipulation. Face swap is a critical tool for attackers to forge images of public figures and politicians, posing threats to societal security (Fig. [1](https://arxiv.org/html/2505.17476v1#S3.F1)a). We use two representative face swap methods, SimSwap [7] and e4s [23], to perform such manipulations. We prioritize modifying larger faces to target the primary subject in the image (Fig. [2](https://arxiv.org/html/2505.17476v1#S3.F2)c). Given a source image $I_{s}$, we randomly choose one of the two methods to replace the largest face $F_{s}$ in $I_{s}$ with a face $F_{c}$ from $\mathcal{D}_{C}$, generating a manipulated face swap sample $I_{fs}$. The bounding box $y_{box}=\{x_{1},y_{1},x_{2},y_{2}\}$ of the swapped face is stored as an annotation, and the name $N_{c}$ of $F_{c}$ is recorded for text fabrication.

**Table 1: Comparison of the proposed MDSM with existing misinformation datasets, where MM Det., Text Det., Man. Type Det., and Im. GD stand for Multi-media Detection, Text Detection, Manipulation Type Detection and Image Grounding.**
| Datasets | Samples | Modality | Tasks | Semantic | MLLM |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Text | Image | MM Det. | Text Det. | Man. Type Det. | Im. GD | Alignment | Inclusion |
| LIAR [47] | 13K | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| DFIM-HQ [25] | 140K | ✗ | ✓ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ |
| MEIR [36] | 139k | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| MiRAGeNews [13] | 15k | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ |
| COSMOS [41] | 453k | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| $\text{DGM}^{4}$ [38] | 230k | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ |
| MDSM (Ours) | 441k | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

Figure: Figure 2: MDSM statistics. (a) Heat-map of the distribution of different media sources and manipulation categories. (b) Types of face attributes. (c) The distribution of the manipulated image area as a proportion of the entire image in the image modality.
Refer to caption: x3.png

$\triangleright$ Text Fabrication for Face Swap.
We use Qwen2-VL [45] to generate consistent narratives matching the manipulated image $I_{fs}$. This requires knowing: 1) the inserted person’s name $N_{c}$, and 2) the names of people remaining in $I_{fs}$. We extract the full name list from the original text $T_{s}$ using BERT [8], then input these names and the face $F_{s}$ into Qwen2-VL to identify the name matching $F_{s}$. We replace this name with $N_{c}$ to form the final name set $N_{A}$. Finally, we input $I_{fs}$ and a prompt (as shown in Fig. [1](https://arxiv.org/html/2505.17476v1#S3.F1)c) into Qwen2-VL to generate aligned text.

$\triangleright$ Face Attribute Manipulation.
Face emotion editing, is also considered in our dataset (Fig. [1](https://arxiv.org/html/2505.17476v1#S3.F1)b). Our pipeline uses StyleCLIP [30] and HFGI [46] for attribute manipulations. For image $I_{s}\in D_{s}$, we analyze facial expressions using an emotion classifier [26] to determine emotional attributes (positive or negative). We randomly select a method to manipulate the primary face $F_{s}$’s attributes inversely to the classification outcome, producing $I_{fa}$. To ensure diversity, we control manipulation intensity with varied emotion prompts and introduce age modifications. The distribution of face attribute prompts is shown in Fig. [2](https://arxiv.org/html/2505.17476v1#S3.F2)(b), with $y_{box}$ stored as annotation.

$\triangleright$ Text Fabrication for Face Attribute.
Similar to face swapping, text forgery for face attribute editing is also generated by Qwen2-VL but with distinct prompts. Specifically, we instruct the MLLM to focus primarily on facial expressions while downplaying unchanged elements. The full name list is initially extracted from the source text $T_{s}$ and incorporated into the prompts (Fig. [1](https://arxiv.org/html/2505.17476v1#S3.F1)c) in a manner analogous to face swapping.

### 3.4 Dataset Statistics

With the above steps, we finally harvest our MDSM dataset $\mathcal{D}_{M}$, a large scale, semantic-aligned multi-modal benchmark with high-fidelity texts from MLLM. The detailed distribution of MDSM data is shown in Fig. [2](https://arxiv.org/html/2505.17476v1#S3.F2)(a). Compared with the existing manipulation detection benchmarks in Tab. [1](https://arxiv.org/html/2505.17476v1#S3.T1), MDSM has the following advantages: 1)Risk Consideration of MLLM. MDSM acknowledges the emerging challenges posed by MLLMs and utilizes multi-modal to create semantically coherent and contextually plausible narratives for manipulated images. This scenario, though under-explored, is highly significant for the detection and grounding of manipulated multi-modal media. 2)Semantic Alignment. MDSM is an aligned image-text multi-modal media manipulation benchmark, which is a significant and more practical scenario for multi-modal manipulation detection. 3)Large Scale. Our MDSM comprises 441,423 samples and is the largest benchmark for detecting and grounding multi-modal manipulation. 4)Diverse Multi-media Sources. The multi-modal media of MDSM sources from diverse media sites, including The Guardian, The New York Times, The Washington Post, USA Today, and the BBC. Consequently, the generality of methods can be assessed via cross-domain evaluation.

Our proposed MDSM defines three tasks: 1)Fake Multi-modal Media Detection. True for the manipulated media and False for the original ones. 2)Manipulation Type Detection, recognizing Face Swap (FS), Face Attribute (FA), and Text Fabrication (TF). 3)Image Grounding, locating the bounding box of the manipulated region in image.

## 4 Methodology

Figure: Figure 3: Illustration of our proposed AMD. Given a manipulated image-text pair, (a) maps the image and prompts into a unified dimension and concatenates them with the artifact token, (b) extracts perceptual clues into the artifact token, (c) integrates multi-modal features to generate text-based detection results, and (d) outputs and visualization.
Refer to caption: x4.png

Fig. [3](https://arxiv.org/html/2505.17476v1#S4.F3) illustrates the architecture of our Artifact-aware Manipulation Diagnosis framework (AMD). Built upon Florence-2 [4] to leverage real-world knowledge, AMD follows a sequence-to-sequence framework for joint textual detection and grounding. Multimodal inputs are processed through three stages, as shown in Fig. [3](https://arxiv.org/html/2505.17476v1#S4.F3)a-c, and outputs localized predictions with textual explanations.

### 4.1 Multi-modal Input Embedding.

Prompt construction. To adapt the MLLM for the MDSM task while preserving its inherent knowledge, we develop heuristic question(human)-answer(assistant) prompts where the image-question pair serves as input and the text response constitutes AMD’s output:

###Human:$<\textit{Task}>$$<\textit{Options}>$$<\textit{Grounding}>$.

###Assistant:$<\textit{Response}>$[$<\textit{Coordinates}>$].

In this prompt paradigm,

- •
$<\textit{Task}>$: Specifies the manipulation detection objective and pairs the input image-text.
- •
$<\textit{Options}>$: Lists all candidate answers for MDSM task.
- •
$<\textit{Grounding}>$: Conditionally triggers region localization (via coordinates in brackets) only when image manipulation is detected.
- •
$<\textit{Response}>$: Encapsulates the correct answers.
- •
$[<\textit{Coordinates}>]$: Optionally encloses tampered region coordinates if the image is altered.

Artifact Token Embeddings. To effectively adapt the MLLM into MDSM context while preserving its pretrained knowledge, we introduce a learnable Artifact Token that explicitly encodes artifacts from heterogeneous inputs. Formally, let the artifact token embeddings be denoted as ${E}_{a}\in\mathbb{R}^{n_{a}\times d}$, where $n_{a}$ indicates the token count and $d$ the embedding dimension. The textual input is processed through an embedding layer to obtain text embeddings ${E}_{t}\in\mathbb{R}^{n_{t}\times d}$ , while the visual input is encoded via a vision backbone $\mathcal{E}_{v}$ followed by a LayerNorm-augmented linear projection, yielding image embeddings ${E}_{v}\in\mathbb{R}^{n_{v}\times d}$. The above embeddings are concatenated to construct the input sequence: ${S}_{\text{inp}}=[{E}_{v};{E}_{a};{E}_{t}]$, where $[\cdot;\cdot]$ means concatenating along the token dimension.

### 4.2 Artifact Pre-perception Encoding

This stage aims to perceive manipulation artifacts within input data and condense these forensic clues into the artifact token. Specifically, the input sequence $S$ undergoes processing through the multimodal encoder $\mathcal{E}^{p}_{m}$, yielding $\hat{S}=[\hat{E}_{v};\hat{E}_{a};\hat{E}_{t}]$. To inject artifact-aware clues into the artifact token embedding $\hat{E}_{a}$, we pick $\hat{E}_{a}$ from $\hat{S}$ and feed it into an artifact-aware classification head. As illustrated in Fig. [3](https://arxiv.org/html/2505.17476v1#S4.F3), this classification head is optimized via a manipulation detection objective to explicitly encode artifact-related patterns into $\hat{E}_{a}$.

Particularly, the embedding $\hat{E}_{a}$ is encoded into a global representation via weighted pooling. Firstly, the token scores $\mathcal{W}\in\mathbb{R}^{1\times n_{a}}$ are calculated as:

$$ $\mathcal{W}=m^{\top}\text{ReLU}(\mathcal{M}\hat{E}_{a}^{\top}+b),$ (1) $$

where $\mathcal{M}\in\mathbb{R}^{h\times d}$, $m\in\mathbb{R}^{h}$, and $b\in\mathbb{R}^{h}$, with $h$ as the hidden dimension. After normalizing $\mathcal{W}$ via softmax, the artifact representation $\bar{E}_{a}$ is derived as a weighted sum, $\text{softmax}(\mathcal{W})\cdot\hat{E}_{a}$.

Then we equip a binary classifier $C_{a}$ to determine whether artifact traces are present:

$$ $\mathcal{L}_{APE}=\mathbb{E}_{(I,T)\sim\mathcal{D}_{M}}\mathbf{CE}(C_{a}(\bar{ E}_{a}),y_{\text{fd}}),$ (2) $$

where $\mathbf{CE}$ means cross-entropy loss, $y_{\text{fd}}$ is the label of fake multimodal media detection task.

Task Adaption & Knowledge Preservation. To effectively inject the artifact clues into $\hat{E}_{a}$ without distorting the original real-world knowledge of MLLM, two strategies are adopted, we 1) freeze the parameters of $\mathcal{E}_{m}^{p}$ during artifact perception loss optimization (Eq. [2](https://arxiv.org/html/2505.17476v1#S4.E2)), such that allowing more artifact clues can be accumulated into the artifact token as well as preserving the raw MLLM knowledge; 2) replace the text and image embeddings in $\hat{S}$ with the original ones to preserve the original MLLM knowledge, *i.e.,* feeding $S_{a}=[{E}_{v};\hat{E}_{a};{E}_{t}]$ to the subsequent modules, as shown in Fig. [3](https://arxiv.org/html/2505.17476v1#S4.F3).

### 4.3 Manipulation-Oriented Reasoning

Manipulation-Oriented Reasoning (MOR) is in charge of generating the textual answer in response to the question prompt. MOR first encodes the sequence ${S}_{a}$, subsequently, the language $Decoder$ takes the resultant features to generate the answers. To acquire an accurate response, we augment the network optimization in MOR with two guiding tasks: visual Artifact Capture via Grounding and Manipulation-focused Guidance.

Visual Artifact Capture via Grounding.
The sequence $S_{a}$ is fed into multimodal encoder $\mathcal{E}_{m}$, resulting in a new sequence $S_{m}=[\hat{E}_{v}^{m};\hat{E}_{a}^{m};\hat{E}_{t}^{m}]$. Given that visual embeddings contain rich local spatial information related to artifact traces, we propose a Visual Artifact Aggregation (VAA) module to aggregate spatial information in $\hat{E_{v}^{m}}$ to perform manipulation bbox grounding. Firstly, the $\hat{E_{a}^{m}}$ is transformed into a query token $q_{a}\in\mathbb{R}^{1\times d}$ using the attention-based weighted pooling (Eq. [1](https://arxiv.org/html/2505.17476v1#S4.E1)). Then, $q_{a}$ collects visual manipulation clues from image features $\hat{E_{v}^{m}}$ via cross attention:

$$ $u_{agg}=\text{Attention}(q_{a},\hat{E}_{v}^{m},\hat{E}_{v}^{m}).$ (3) $$

Subsequently, the $u_{agg}$ is sent to the bbox detector to generate artifact coordinates. We follow [34] to construct the image manipulation grounding loss using L1 loss $\mathcal{L}_{1}$ and GIoU loss $\mathcal{L}_{IoU}$:

$$ $\mathcal{L}_{IMG}=\mathbb{E}_{(I,T)\sim\mathcal{D}_{M}}(\mathcal{L}_{1}+ \mathcal{L}_{IoU}).$ (4) $$

Manipulation-focused Guidance
further highlight whether the multimodal input is manipulated or not, tuning the MLLM to be sufficiently sensitive to the fake multi-modal media. To fully capture manipulation-related information embedded within different modalities, we propose a Dual-Branch Manipulation guidance strategy. Specifically, each modality feature in the encoder output sequence $S_{m}$ is treated as a query $Q$ and undergoes interaction for binary classification. Given that artifact traces predominantly appear in the image modality, the sequence composed of $\hat{E}_{a}^{m}$ and $\hat{E}_{v}^{m}$ is regarded as the image modality feature. The interaction process is formulated as:

$$ $u_{v}=\text{Attention}(\hat{E}_{v+a}^{m},\hat{E}_{t}^{m},\hat{E}_{t}^{m}), \quad u_{t}=\text{Attention}(\hat{E}_{t}^{m},\hat{E}_{v+a}^{m},\hat{E}_{v+a}^{ m}),$ (5) $$

where $\hat{E}_{v+a}^{m}$ represents the concatenation of $\hat{E}_{v}^{m}$ and $\hat{E}_{a}^{m}$, while $\hat{E}_{t}^{m}$ corresponds to the textual sequence. The cross-modal interaction outputs, $u_{v}$ and $u_{t}$, are respectively processed by a binary classifier $C_{m}$ to distinguish between manipulated and original multimodal media. Thus the Dual-Branch Manipulation guidance loss can be calculated as:

$$ $\mathcal{L}_{DBM}=\mathbb{E}_{(I,T)\sim\mathcal{D}_{M}}\sum_{x\in\{v,t\}} \mathbf{CE}(C_{m}(u_{x}),y_{fd}).$ (6) $$

Language modeling.
The input sequence $S_{a}$ is processed through an encoder-decoder architecture, ultimately generating a pure text output that includes choices and coordinates (Fig. [3](https://arxiv.org/html/2505.17476v1#S4.F3)d) as specified in the prompts. In this stage, an autoregressive approach is adopted, where the decoder generates the target sequence $y$ conditioned on $S_{m}$. The language modeling loss $\mathcal{L}_{LM}$ [4] is used to supervise the decoded text outputs.

### 4.4 Token Redundancy Penalty

To suppress the redundancy and increase the information density among tokens in $E_{a}$, we design a Token Redundancy Penalty (TRP) optimization term.
Specifically, we first encourage the columns of $E_{a}$ to be as orthogonal as possible by introducing a loss term $\mathcal{L}_{\mathrm{orth}}$, which increases the matrix rank. We construct Gram matrix of $E_{a}$, $G=E_{a}E_{a}^{\top}\in\mathbb{R}^{n_{a}\times n_{a}}$, and the orthogonality of the columns can be measured by the off-diagonal elements of the Gram matrix. Ideally, if the columns are orthogonal, the off-diagonal entries of $G$ should be zero. Therefore, we define:

$$ $\mathcal{L}_{\mathrm{orth}}=\left\|G-\mathrm{Diag}\left(\mathrm{diag}(G)\right )\right\|_{F}^{2},$ (7) $$

where $\mathrm{Diag}(G)$ denotes a diagonal matrix retaining only the diagonal elements of $G$, and $\left\|\cdot\right\|_{F}$ denotes the Frobenius norm used to aggregate the differentiable loss.

To avoid a potential checkerboard pattern in $E_{a}$ under the constraint of $\mathcal{L}_{\mathrm{orth}}$—which could lead to loss of information—we further introduce a modulation constraint $\mathcal{L}_{\mathrm{mod}}$ based on the Kullback–Leibler (KL) divergence. Particularly, we first normalize the components to form a distribution: $p_{t,i}=\frac{{E_{a}}_{t,i}^{2}}{\sum_{i=1}^{d}{E_{a}}_{t,i}^{2}}$. While the target distribution is set as the even distribution ($\frac{1}{d}$), thereby encouraging each component to contain information evenly with following constrain:

$$ $\mathcal{L}_{\mathrm{mod}}=\frac{1}{n_{a}}\sum_{t=1}^{n_{a}}\left(\sum_{i=1}^{ d}p_{t,i}\log p_{t,i}+\log d\right),$ (8) $$

Finally, the overall Token Redundancy Penalty is defined as the combination of both terms:

$$ $\mathcal{L}_{{TRP}}=\mathcal{L}_{\mathrm{orth}}+\mathcal{L}_{\mathrm{mod}},$ (9) $$

$\mathcal{L}_{TRP}$ is imposed on the $S_{inp}$ sequence during the APE stage.

### 4.5 Training and Inference

Training.
All guiding losses above and the language modeling loss are incorporated into the training process, forming a unified optimization framework as follows:

$$ $\mathcal{L}=\mathcal{L}_{APE}+\mathcal{L}_{DBM}+\mathcal{L}_{IMG}+\mathcal{L}_ {TRP}+\mathcal{L}_{LM},$ (10) $$

Inference. All auxiliary heads for $\mathcal{L}_{APE},\mathcal{L}_{DBM},\mathcal{L}_{IMG}$, and $\mathcal{L}_{TRP}$ are discarded during inference.
For a piece of multimodal media, the image and the question (text & prompts) follow the same steps shown in Fig. [3](https://arxiv.org/html/2505.17476v1#S4.F3) and generate the textual detection and grounding results.

## 5 Experiments

**Table 2: Comparison of multi-modal learning methods for MDSM, where the background gray indicates the intra-domain performance. The better results in each group are in boldface. AVG refers to the average performance across the five news domains.**
| Setting | Method | Test Domain |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NYT | Guardian | USA | Wash. | BBC | AVG |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  |  | ACC | mAP | mIoU | ACC | mAP | mIoU | ACC | mAP | mIoU | ACC | mAP | mIoU | ACC | mAP | mIoU | ACC | mAP | mIoU |
| Zero-Shot | Qwen2.5-VL [3] | 47.74 | 29.24 | 0.00 | 35.18 | 25.70 | 0.00 | 24.66 | 40.60 | 0.00 | 25.11 | 40.29 | 0.28 | 35.89 | 31.51 | 0.00 | 33.72 | 33.47 | 0.06 |
| Qwen3-a22b [43] | 45.29 | 25.01 | 0.71 | 38.12 | 27.41 | 1.19 | 22.87 | 39.17 | 0.10 | 22.87 | 40.77 | 1.34 | 37.22 | 31.60 | 0.98 | 33.27 | 33.69 | 0.86 |  |
| GPT-4o [14] | 48.48 | 27.90 | 0.82 | 35.68 | 29.49 | 1.23 | 24.62 | 39.88 | 1.37 | 23.62 | 38.89 | 1.22 | 37.19 | 30.48 | 1.20 | 33.92 | 33.33 | 1.17 |  |
| Gemini-2.0 [42] | 56.05 | 33.16 | 1.44 | 41.26 | 24.37 | 1.12 | 29.60 | 38.29 | 1.40 | 29.15 | 35.20 | 2.42 | 38.12 | 29.13 | 2.25 | 38.83 | 32.03 | 1.72 |  |
| Tr. on NYT | ViLT [16] | 83.27 | 64.27 | 22.73 | 72.18 | 31.76 | 20.21 | 70.34 | 36.45 | 21.48 | 65.71 | 36.23 | 17.56 | 74.33 | 36.10 | 19.36 | 73.17 | 40.96 | 20.27 |
| HAMMER [38] | 79.20 | 55.86 | 51.34 | 68.23 | 40.10 | 21.56 | 71.52 | 41.17 | 13.74 | 68.50 | 41.47 | 13.92 | 67.37 | 42.23 | 16.12 | 70.96 | 44.16 | 23.34 |  |
| HAMMER++ [39] | 79.61 | 57.06 | 54.44 | 66.99 | 38.07 | 17.34 | 67.18 | 37.58 | 10.76 | 66.28 | 37.97 | 10.88 | 66.12 | 37.82 | 13.68 | 69.23 | 41.70 | 21.42 |  |
| FKA-Owl [22] | 94.67 | 78.18 | 55.81 | 77.20 | 46.88 | 43.67 | 78.00 | 44.45 | 50.73 | 75.49 | 50.83 | 43.53 | 84.65 | 60.73 | 43.28 | 81.60 | 56.77 | 46.23 |  |
| AMD(Ours) | 92.24 | 84.47 | 72.94 | 80.21 | 64.00 | 62.51 | 78.56 | 68.49 | 55.17 | 82.64 | 69.41 | 56.66 | 86.14 | 60.58 | 70.54 | 83.96 | 69.39 | 63.56 |  |
| Tr. on Guardian | ViLT [16] | 68.80 | 43.99 | 21.77 | 85.29 | 67.34 | 41.80 | 70.34 | 46.24 | 37.68 | 78.61 | 47.17 | 38.13 | 80.00 | 44.79 | 38.97 | 76.61 | 49.90 | 35.67 |
| HAMMER [38] | 61.89 | 37.98 | 18.84 | 78.50 | 52.40 | 51.53 | 74.78 | 50.76 | 43.40 | 75.11 | 50.34 | 46.36 | 81.32 | 50.15 | 56.03 | 74.32 | 48.33 | 43.23 |  |
| HAMMER++ [39] | 62.75 | 36.45 | 23.76 | 80.95 | 59.92 | 64.67 | 75.36 | 48.77 | 47.13 | 76.30 | 49.56 | 48.91 | 80.12 | 50.36 | 57.97 | 75.10 | 49.01 | 48.49 |  |
| FKA-Owl [22] | 80.60 | 40.44 | 26.33 | 92.60 | 78.24 | 71.04 | 80.90 | 51.80 | 50.93 | 78.88 | 51.62 | 50.88 | 87.61 | 68.57 | 61.82 | 84.12 | 58.13 | 52.20 |  |
| AMD (Ours) | 84.29 | 48.54 | 52.38 | 91.43 | 80.85 | 85.09 | 88.80 | 53.05 | 52.51 | 86.64 | 54.07 | 53.27 | 89.74 | 64.75 | 61.82 | 88.18 | 60.25 | 61.02 |  |

**Table 3: Comparison of multi-modal learning methods for $\text{DGM}^{4}$, where the guardian domain with background gray is intra-domain. $\text{P}_{tok}$ is Precision of fake token grounding.**
| Method | Test Domain |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Guardian | USA | Wash. | BBC | AVG |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | ACC | mAP | $\text{P}_{tok}$ | mIoU | ACC | mIoU | $\text{P}_{tok}$ | mIoU | ACC | mAP | $\text{P}_{tok}$ | mIoU | ACC | mAP | $\text{P}_{tok}$ | mIoU | ACC | mAP | $\text{P}_{tok}$ | mIoU |
| ViLT [16] | 68.27 | 42.29 | 69.87 | 43.19 | 52.79 | 31.28 | 62.11 | 33.78 | 55.76 | 33.26 | 57.17 | 31.10 | 44.14 | 39.68 | 59.06 | 21.96 | 55.24 | 36.63 | 62.05 | 32.49 |
| HAMMER [38] | 78.34 | 66.79 | 78.27 | 61.09 | 64.97 | 40.49 | 73.76 | 40.51 | 63.54 | 40.26 | 76.13 | 38.53 | 54.97 | 40.84 | 81.48 | 43.74 | 65.45 | 47.10 | 77.41 | 45.97 |
| HAMMER++ [39] | 79.13 | 67.11 | 78.24 | 62.15 | 65.25 | 40.74 | 73.24 | 41.14 | 63.83 | 40.34 | 76.17 | 38.21 | 54.24 | 41.25 | 81.73 | 43.23 | 65.61 | 47.36 | 77.34 | 46.19 |
| FKA-Owl [22] | 82.97 | 53.86 | 87.70 | 65.69 | 67.57 | 38.97 | 79.44 | 32.57 | 67.05 | 37.70 | 81.55 | 31.86 | 70.26 | 40.20 | 84.54 | 46.48 | 71.96 | 42.68 | 83.31 | 44.15 |
| AMD(Ours) | 84.61 | 68.50 | 82.78 | 81.24 | 70.62 | 43.20 | 75.73 | 41.99 | 70.28 | 43.36 | 77.76 | 39.05 | 72.37 | 56.57 | 83.76 | 45.20 | 74.47 | 52.91 | 80.01 | 51.87 |

Experimental Setup.
DaViT [9] serves as the image encoder $\mathcal{E}_{v}$. The multimodal encoder $\mathcal{E}_{m}$ and decoder are based on Florence-2[4], and $\mathcal{E}_{m}^{p}$ is kept frozen during training. The length of Artifact Token is set to 32. Code will be released for reproducibility.

### 5.1 Quantitative Results

Effectiveness & Generalization. We assess AMD’s in-domain and cross-domain performance against four SOTA methods on the MDSM and $\text{DGM}^{4}$ dataset.
For MDSM, models are trained on two larger subsets, The Guardian and The New York Times (NYT), and tested on the remaining ones (Tab. [2](https://arxiv.org/html/2505.17476v1#S5.T2)).
For $\text{DGM}^{4}$, we use the largest subset, The Guardian, as the training domain (Tab. [3](https://arxiv.org/html/2505.17476v1#S5.T3)).
Additionally, Tab. [2](https://arxiv.org/html/2505.17476v1#S5.T2) presents zero-shot results of leading general-purpose models on MDSM. Evaluation metrics are following [38]. Our key observations are as follows:

1)
Forgery-trace-based methods, ViLT [16] and HAMMER [38], show limited performance within and across domains. In contrast, MLLM-based approaches, including FKA-Owl [22] and AMD, achieve significantly stronger results. For example, trained on MDSM-NYT (Tab. [2](https://arxiv.org/html/2505.17476v1#S5.T2)), AMD attains an intra-domain mAP of 84.47 and cross-domain scores above 60, while HAMMER has an mAP of 55.86 and cross-domain performance below 45. This affirms that MLLMs’ open-world knowledge enhances manipulation detection in complex scenarios.

**Table 4: Ablation on each components (a) and discussion regarding performance on test set of difference MLLMs & differnece manipulation type (b).**
| Components | NYT | Guardian |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LM | APE | IMG | DBM | TRP | ACC | mAP | mIoU | ACC | mAP | mIoU |
| ✓ |  |  |  |  | 76.92 | 46.38 | 58.77 | 82.33 | 53.57 | 53.55 |
| ✓ | ✓ |  |  |  | 82.93 | 47.12 | 60.13 | 87.68 | 56.51 | 57.91 |
| ✓ | ✓ | ✓ |  |  | 82.97 | 47.18 | 61.78 | 87.99 | 56.43 | 60.53 |
| ✓ | ✓ | ✓ | ✓ |  | 83.42 | 66.47 | 62.14 | 87.88 | 60.26 | 60.97 |
| ✓ | ✓ | ✓ | ✓ | ✓ | 83.96 | 69.39 | 63.56 | 88.18 | 60.25 | 61.02 |

2)
On both MDSM and $\text{DGM}^{4}$, AMD consistently achieves strong grounding performance. In tab. [2](https://arxiv.org/html/2505.17476v1#S5.T2), when trained on NYT and Guardian, AMD achieves average mIoU of 63.56 and 61.02, respectively, while FKA-Owl attains only 46.23 and 52.20.
Although we designed more detailed prompts for general-purpose models, their grounding performance remains extremely poor, with mIoU scores not exceeding 3. This superiority of AMD stems from our carefully crafted question-answer heuristic prompts and the unified decoding framework of the MOR module. Unlike other methods that produce grounding results for every sample, AMD omits coordinate outputs when no manipulation is detected, thereby avoiding unnecessary errors and alleviating performance bottlenecks.

3)
On $\text{DGM}^{4}$ (Tab. [3](https://arxiv.org/html/2505.17476v1#S5.T3)), MLLM-based approaches also demonstrate superior performance. AMD achieves 74.47 ACC, 52.91 mAP, 80.01 $\text{P}_{tok}$, and 51.87 mIoU, outperforming the HAMMER series.
Despite trailing FKA-Owl in new text localization task metric $\text{P}_{tok}$, AMD surpasses it in the other three tasks, especially mAP and mIoU. These results highlight the strong generalization ability of AMD when facing novel datasets and manipulation tasks.

Generalization Assessment across MLLMs.
To assess AMD’s generalization to different MLLM-generated narratives, we use four popular MLLMs, including Qwen-VL [2], X-InstructBLIP [29], LLaVA [20], and mPLUG-Owl [50], to regenerate test set narratives. AMD, trained on NYT, is then evaluated on the new test set, recording the TF category AP for both intra-domain (NYT) and cross-domain settings. As shown in Tab. [4](https://arxiv.org/html/2505.17476v1#S5.T4)b, bar chart I, AMD achieves intra-domain and cross-domain APs of over 76 and 53, respectively, demonstrating its robustness across diverse MLLMs.

Details of manipulation type detection.
Using AMD trained on the NYT domain as an example, the bar chart II in Tab. [4](https://arxiv.org/html/2505.17476v1#S5.T4)b shows that text-modal (TF) manipulations are harder to detect than image-modal ones. FA achieves intra-domain AP of 88.45 and cross-domain AP of 71.37, while TF reaches 79.84 and 57.53, respectively. This highlights the deceptive nature of MLLM-generated narratives.

### 5.2 Ablation Study

Tab. [4](https://arxiv.org/html/2505.17476v1#S5.T4)a presents the results for each component considered in our study. We use a fine-tuned Florence-2 with our designed prompts as the baseline for comparison. As shown, incorporating Artifact Pre-perception Encoding (APE) improves all three task metrics, especially binary classification accuracy, which increases from 76.92 to 82.93 on NYT and from 82.33 to 87.68 on Guardian. This demonstrates that pre-perception of manipulation traces is vital for aiding MLLMs in multi-media manipulation detection. Adding auxiliary tasks, such as Dual-Branch Manipulation (DBM) and Image Manipulation Grounding (IMG), enhances fake news classification and grounding performance, while also slightly improving binary classification. Notably, DBM significantly boosts AMD’s mAP, increasing from 47.18 to 66.47 on NYT and from 56.43 to 60.26 on Guardian. Furthermore, the incorporation of the Token Redundancy Penalty (TRP) yields comprehensive performance gains, especially exhibiting stable improvements in ACC and mIoU across both domains.

## 6 Conclusion

This study discloses two critical limitations in current multimedia manipulation detection: underestimation of dynamic semantic deception risks posed by MLLMs and the unrealistic, semantically incoherent misalignment artifacts among existing benchmarks. To address these challenges, we construct the MLLM-Driven Synthetic Multimodal (MDSM) dataset and the Artifact-aware Manipulation Diagnosis (AMD) framework to address this new and challenging problem. AMD integrates Artifact Pre-perception Encoding and Manipulation-Oriented Reasoning to enhance detection of MLLM-generated multimodal disinformation. Comprehensive experiments demonstrate the framework’s superior generalization capabilities, validating its effectiveness as a unified solution for combating advanced MLLM-driven deception.

Broader impact and limitation.
Our MDSM dataset effectively simulates real-world attacks and aids in detecting fake news. However, it currently focuses only on high-risk manipulations of celebrity faces in multimodal news, with the full generated image left for future exploration.

## References

- Alayrac et al. [2022]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikoł aj Bińkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan.
Flamingo: a visual language model for few-shot learning.
In *NeurIPS*, 2022.
- Bai et al. [2023]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.
Qwen technical report.
*arXiv preprint arXiv:2309.16609*, 2023.
- Bai et al. [2025]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Bin et al. [2023]
Xiao Bin, Wu Haiping, Xu Weijian, Dai Xiyang, Hu Houdong, Lu Yumao, Zeng Michael, Liu Ce, and Yuan Lu.
Florence-2: Advancing a unified representation for a variety of vision tasks.
In *CVPR*, 2023.
- Biten et al. [2019]
Ali Furkan Biten, Lluis Gomez, Marçal Rusiñol, and Dimosthenis Karatzas.
Good news, everyone! context driven entity-aware captioning for news images.
In *CVPR*, 2019.
- Cantarella et al. [2023]
Michele Cantarella, Nicolò Fraccaroli, and Roberto Volpe.
Does fake news affect voting behaviour?
*Research Policy*, 52(1):104628, 2023.
- Chen et al. [2020]
Renwang Chen, Xuanhong Chen, Bingbing Ni, and Yanhao Ge.
Simswap: An efficient framework for high fidelity face swapping.
In *ACM MM*, 2020.
- Devlin et al. [2018]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language understanding.
*arXiv preprint arXiv:1810.04805*, 2018.
- Ding et al. [2022]
Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan.
Davit: Dual attention vision transformers.
In *ECCV*, 2022.
- Fu et al. [2025]
Jiale Fu, Yuchu Jiang, Junkai Chen, Jiaming Fan, Xin Geng, and Xu Yang.
Speculative ensemble: Fast large language model ensemble via speculation.
*arXiv preprint arXiv:2502.01662*, 2025.
- Fuxiao et al. [2020]
Liu Fuxiao, Wang Yinghan, Wang Tianlu, and Ordonez Vicente.
Visual news: Benchmark and challenges in news image captioning.
In *EMNLP*, 2020.
- Huang et al. [2023]
Kung-Hsiang Huang, McKeown Kathleen, Nakov Preslav, Choi Yejin, and Ji Heng.
Faking fake news for real fake news detection: Propaganda-loaded training data generation.
In *Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics*, 2023.
- Huang et al. [2024]
Runsheng Huang, Liam Dugan, Yue Yang, and Chris Callison-Burch.
MiRAGeNews: Multimodal realistic AI-generated news detection.
In *EMNLP*, 2024.
- Hurst et al. [2024]
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Jia et al. [2021]
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.
Scaling up visual and vision-language representation learning with noisy text supervision.
In *ICML*, 2021.
- Kim et al. [2021]
Wonjae Kim, Bokyung Son, and Ildoo Kim.
Vilt: Vision-and-language transformer without convolution or region supervision.
In *ICML*, 2021.
- King [2009]
Davis E. King.
Dlib-ml: A machine learning toolkit.
*Journal of Machine Learning Research*, 10:1755–1758, 2009.
- Li et al. [2023]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models.
In *ICML*, 2023.
- Li et al. [2020]
Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo.
Face x-ray for more general face forgery detection.
In *CVPR*, 2020.
- Liu et al. [2023a]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
In *NeurIPS*, 2023a.
- Liu et al. [2025]
Huan Liu, Zichang Tan, Qiang Chen, Yunchao Wei, Yao Zhao, and Jingdong Wang.
Unified frequency-assisted transformer framework for detecting and grounding multi-modal manipulation.
*International Journal of Computer Vision*, 133:1392–1409, 2025.
- Liu et al. [2024]
Xuannan Liu, Peipei Li, Huaibo Huang, Zekun Li, Xing Cui, Jiahao Liang, Lixiong Qin, Weihong Deng, and Zhaofeng He.
Fka-owl: Advancing multimodal fake news detection through knowledge-augmented lvlms.
In *ACM MM*, 2024.
- Liu et al. [2023b]
Zhian Liu, Maomao Li, Yong Zhang, Cairong Wang, Qi Zhang, Jue Wang, and Yongwei Nie.
Fine-grained face swapping via regional gan inversion.
*CVPR*, 2023b.
- Luo et al. [2021]
Grace Luo, Trevor Darrell, and Anna Rohrbach.
NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media.
In *EMNLP*, 2021.
- Mathews et al. [2023]
Sherin Mathews, Shivangee Trivedi, Amanda House, Steve Povolny, and Celeste Fralick.
An explainable deepfake detection framework on a novel unconstrained dataset.
*Complex & Intelligent Systems*, 9(4):4425–4437, 2023.
- Octavio et al. [2017]
Arriaga Octavio, Valdenegro-Toro Matias, and Plöger Paul-Gerhard.
Real-time convolutional neural networks for emotion and gender classification.
*ArXiv*, abs/1710.07557, 2017.
- Olan et al. [2024]
Femi Olan, Uchitha Jayawickrama, Emmanuel Ogiemwonyi Arakpogun, Jana Suklan, and Shaofeng Liu.
Fake news on social media: the impact on society.
*Information Systems Frontiers*, 26(2):443–458, 2024.
- OpenAI [2023]
OpenAI.
Gpt-4 technical report.
*CoRR*, abs/2303.08774, 2023.
- Panagopoulou et al. [2024]
Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles.
X-instructblip: A framework for aligning image, 3d, audio, video to llms and its emergent cross-modal reasoning.
In *ECCV*, 2024.
- Patashnik et al. [2021]
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.
Styleclip: Text-driven manipulation of stylegan imagery.
In *ICCV*, 2021.
- Peng et al. [2025]
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang.
Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl.
*arXiv preprint arXiv:2503.07536*, 2025.
- Petratos [2021]
Pythagoras N. Petratos.
Misinformation, disinformation, and fake news: Cyber risks to business.
*Business Horizons*, 64(6):763–774, 2021.
- Radford et al. [2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision.
In *ICML*, 2021.
- Rezatofighi et al. [2019]
Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.
Generalized intersection over union: A metric and a loss for bounding box regression.
In *CVPR*, 2019.
- Rocha et al. [2021]
Yasmim Mendes Rocha, Gabriel Acácio De Moura, Gabriel Alves Desidério, Carlos Henrique De Oliveira, Francisco Dantas Lourenço, and Larissa Deadame de Figueiredo Nicolete.
The impact of fake news on social media and its influence on health during the covid-19 pandemic: A systematic review.
*Journal of Public Health*, pages 1–10, 2021.
- Sabir et al. [2018]
Ekraam Sabir, Wael AbdAlmageed, Yue Wu, and Prem Natarajan.
Deep multimodal image-repurposing detection.
In *ACM MM*, 2018.
- Shao et al. [2022]
Rui Shao, Tianxing Wu, and Ziwei Liu.
Detecting and recovering sequential deepfake manipulation.
In *ECCV*, 2022.
- Shao et al. [2023]
Rui Shao, Tianxing Wu, and Ziwei Liu.
Detecting and grounding multi-modal media manipulation.
In *CVPR*, 2023.
- Shao et al. [2024]
Rui Shao, Tianxing Wu, Jianlong Wu, Liqiang Nie, and Ziwei Liu.
Detecting and grounding multi-modal media manipulation and beyond.
*TPAMI*, 46(8):5556–5574, 2024.
- Sheng et al. [2022]
Qiang Sheng, Juan Cao, Xueyao Zhang, Rundong Li, Danding Wang, and Yongchun Zhu.
Zoom out and observe: News environment perception for fake news detection.
In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics*. Association for Computational Linguistics, 2022.
- Shivangi et al. [2023]
Aneja Shivangi, Bregler Chris, and Nießner Matthias.
Cosmos: Catching out-of-context image misuse using self-supervised learning.
In *AAAI*, 2023.
- Team et al. [2023]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.
Gemini: a family of highly capable multimodal models.
*arXiv preprint arXiv:2312.11805*, 2023.
- Team [2025]
Qwen Team.
Qwen3: Think deeper, act faster, 2025.
Accessed: 2025-02-07.
- Tong et al. [2024]
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, Xichen Pan, Rob Fergus, Yann LeCun, and Saining Xie.
Cambrian-1: A fully open, vision-centric exploration of multimodal llms.
In *NeurIPS*, 2024.
- Wang et al. [2024]
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024.
- Wang et al. [2022]
Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen.
High-fidelity gan inversion for image attribute editing.
In *CVPR*, 2022.
- Wang [2017]
William Yang Wang.
“liar, liar pants on fire”: A new benchmark dataset for fake news detection.
In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)*, 2017.
- Wu et al. [2024]
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al.
Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.
*arXiv preprint arXiv:2412.10302*, 2024.
- Yang et al. [2023]
Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng.
Exploring diverse in-context configurations for image captioning.
In *NeurIPS*, 2023.
- Ye et al. [2024]
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou.
mplug-owl3: Towards long image-sequence understanding in multi-modal large language models.
*arXiv preprint arXiv:2408.04840*, 2024.
- Zellers et al. [2019]
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi.
Defending against neural fake news.
In *NeurIPS*, 2019.
- Zhang et al. [2024]
Zhenxing Zhang, Yaxiong Wang, Lechao Cheng, Zhun Zhong, Dan Guo, and Meng Wang.
Asap: Advancing semantic alignment promotes multi-modal manipulation detecting and grounding.
*arXiv preprint arXiv:2412.12718*, 2024.
- Zhao et al. [2021]
Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuanjun Xiong, and Wei Xia.
Learning self-consistency for deepfake detection.
In *ICCV*, 2021.
- Zhen et al. [2021]
Wang Zhen, Shan Xu, Zhang Xiangxie, and Yang Jie.
N24news: A new dataset for multimodal news classification.
In *International Conference on Language Resources and Evaluation*, 2021.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 2
presentation: 3
questions: 1.Could the authors elaborate on the choice to focus specifically on facial modifications and replacements? Have considerations been made regarding the potential extension of the framework to other types of visual or contextual manipulations, such as scene-level or object-level edits?
2.Would the authors be able to provide additional details about the dataset used to train the artifact-aware classification head? In particular, it would be helpful to know the scale of the data and the relative proportions of different manipulation types included during training.
soundness: 2
strengths: 1.The paper clearly highlights the risks posed by semantically coherent forgeries generated by modern Multimodal Large Language Models (MLLMs).
2.It presents a large-scale, semantically aligned multimodal dataset, effectively filling a crucial gap in resources for studying MLLM-driven misinformation.
3.The proposed framework integrates Artifact Pre-perception Encoding (APE) and Manipulation-Oriented Reasoning (MOR), leveraging the reasoning capabilities of MLLMs to collaboratively analyze image-text manipulations. By synergizing APE and MOR, it effectively adapts MLLMs for precise manipulation analysis.
4.The authors conduct comprehensive experiments on both the MDSM and DGM4 datasets, achieving state-of-the-art average cross-domain generalization performance.
summary: This paper introduces a novel forgery detection framework that targets semantically coherent multimodal content generated by modern Multimodal Large Language Models (MLLMs). The proposed framework leverages Artifact Pre-perception Encoding (APE) and Manipulation-Oriented Reasoning (MOR) to collaboratively analyze image-text manipulations through the reasoning capability of MLLMs. In addition, the authors construct an MLLM-Driven Synthetic Multimodal (MDSM) dataset, where image and text modifications are jointly guided by MLLMs to ensure semantic alignment.
weaknesses: 1.The paper aims to tackle the challenge of forgery detection in semantically aligned image-text scenarios. However, its model design primarily focuses on visual forgeries while largely overlooking textual forgeries.
2.Limited interpretability: although some visualizations are provided, the paper does not clearly demonstrate which specific forgery cues the model captures, nor does it offer a human-understandable reasoning process behind its decisions.
3.The organization of Section 2.2 could be improved — the relationships among its subsections are not clearly articulated, making the overall flow somewhat difficult to follow.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 2
questions: Please see the question above.
soundness: 3
strengths: 1. The motivation is clear. The paper explicitly argues that prior work assumes crude cross-modal inconsistency, which makes detection too easy because the text and image obviously disagree. By contrast, MDSM uses an MLLM to generate fluent, contextually aligned fake narratives that match the manipulated visual identity. This is a meaningful direction.

2. AMD outputs manipulation decisions and the tampered region coordinates as a single textual answer instead of separate detection heads. This makes the downstream application easier.

3. They employ a cross-domain setting for evaluation, emphasizing the generalization ability.
summary: The paper tackles multimodal misinformation detection under the claimed more realistic threat model, generating images that have been locally manipulated (via face swap or face attribute editing), and paired with text generated by a multimodal LLM (MLLM) that is semantically consistent with the manipulation. The authors propose MDSM, a 441k-sample dataset where faces in news images are edited and paired with coherent deceptive narratives. Then introduce AMD, a Florence-2–based model that can handle both (i) binary “fake vs real,” (ii) manipulation type, and (iii) manipulated-region bounding boxes tasks in text form.
weaknesses: 1. The novelty claimed based on the flaw of previous works is not strong enough. Prior work like FKA-Owl is already an MLLM-style system that “incorporates more world knowledge to improve the model’s cross-domain performance,” explicitly targeting multimodal fake news scenarios. The paper acknowledges this but still claims current approaches “fail to account for sophisticated misinformation synthesized by MLLMs,” which is not persuasive enough. 

2. On the model side, AMD is essentially Florence-2 + DaViT with several augmented function modules. This is solid engineering, but conceptually similar to known frameworks in this area. 

3. The paper repeatedly stresses that MDSM “defines a more challenging and practical problem,” “high-risk disinformation,” and “semantically coherent and contextually plausible narratives,” and that previous benchmarks are “too simplistic to effectively deceive the public.” Are there any quantitative and empirical demonstrations of the claimed risk?

4. MDSM only keeps “human-centric” news with faces and named entities, then applies two manipulation types in the image domain, including Face Swap and Face Attribute editing. This is not a general-purpose “multimodal misinformation” benchmark. A multimodal celebrity face tampering benchmark is more appropriate.

5. In the cross-domain training setting, how could the data leakage risk be avoided? Would there be the same celebrity headshot in training and testing sets? Besides, the distribution of manipulations and the linguistic style of fabricated text are almost identical across “domains.” It looks closer to the same attacker pipeline, different news source name.

6. In the tables, you compare single mAP values across models: how exactly is multi-label type detection scored as AP? Is it per manipulation type and averaged? Macro/micro? The paper does not say.

7. Which parameters are actually trainable in APE? Only the Artifact Token + classifier Ca? Is there a two-stage schedule or joint multitask training with loss L? The description is not clear enough for training details.

8. In ablations, why do the authors never isolate TRP alone?

9. If the AMD decoder is effectively trained to emit the answer at the end of that rigid QA format, how robust is it to prompt perturbations? Could an adversary just reformulate the question?

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. While the "coherence trap" is well-motivated, similar aligned fakes have been studied in prior work (e.g., MMFakeBench). How does this work differ in problem formulation or threat model beyond dataset scale?

2. The AMD framework builds directly on Florence-2 with minimal architectural changes. To what extent do the gains come from the model design versus the strong pre-trained backbone?

3. What are the main failure modes of AMD? A deeper analysis of false positives/negatives or challenging manipulation types (e.g., subtle edits, coherent text-only fakes) would strengthen the paper.

4. How generalizable is AMD beyond news domains? The reliance on Florence-2’s world knowledge may limit performance on out-of-distribution content (more sophisticated, MLLM-generated manipulations or content generated by different large models).
soundness: 2
strengths: 1. The paper identifies and formalizes the "coherence trap," a highly relevant and critical issue in the era of advanced generative models, where the very coherence that makes AI-generated content useful also makes it dangerously deceptive.

2. The construction of MDSM is a significant contribution. Its scale, diversity of sources, and alignment between modalities make it a valuable resource for the research community.

3. The paper presents thorough experiments, including ablation studies and cross-domain evaluations, demonstrating the superior performance of AMD over state-of-the-art baselines on the MDSM benchmark.
summary: This paper focuses on the "coherence trap" phenomenon in multimodal misinformation detection. The authors first construct MDSM, a large-scale, diverse, and aligned benchmark for multimodal manipulation detection, comprising challenging samples from reputable news sources. To address this challenge, the authors propose AMD (Artifact-aware Manipulation Diagnosis), a novel framework built on Florence-2 that integrates artifact pre-perception encoding and manipulation-oriented reasoning. Experiments show that AMD outperforms existing methods on MDSM, demonstrating improved robustness in detecting coherently generated fake news.
weaknesses: 1. While the paper formally defines and highlights the "coherence trap" as a critical challenge in multimodal misinformation detection, the underlying concept of detecting semantically aligned fake content is not entirely novel. Prior works, such as MMFakeBench, have already explored scenarios involving coherent image-text manipulations. 

2. The proposed AMD framework is built upon the powerful Florence-2 model and leverages its strong pre-trained multimodal understanding. The architectural innovation of AMD itself appears limited. 

3. While the paper shows AMD's overall success, a deeper analysis of when and why AMD fails (e.g., specific types of manipulations it struggles with, examples of false positives/negatives) would strengthen the work and provide more insight for future research.

[1]Liu X, Li Z, Li P P, et al. MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs. ICLR 2024

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Did the authors use GPT-4o and Gemini-2.0 on all the images for the test set? Seems like quite a large test set for a paid model. How large is the test set?

* Why didn’t the authors include more Open-Source models in Table 2, currently there is basically one 1, since only variants of QWEN is included.
soundness: 3
strengths: * I believe this paper is relevant and has a few strengths to it. There are some interesting experiments that are done with both a zero-shot setting and training models on the MDSM dataset.

* Authors spent time trying to do LoRA finetuning on their dataset which I think was an important experiment the have included.

* Showcasing how other models like HAMMER and FKA-Owl which are prevalent in multi-modal manipulations was good to have and some discussion of the analysis

* Including a human evaluation of some of the manipulated images/text was good to include for this work to showcase how humans can be fooled by multi-modal models
summary: This paper looks at investigating how MLLM driven misinformation in two modalities, namely text and image. They construct a dataset MDSM that simulates realistic multimodal manipulations where images are first manipulated and paired with MLLM-generated deceptive texts that maintain semantic consistency with the visual manipulations. Additionally this paper proposes a detection and grounding framework that outputs both coordinates of the manipulation and explanations (AMD).
weaknesses: * I believe that the authors should try and include more Open-Source multimodal models, for zero-shot evaluation in Table 2, currently the only model present is Qwen and no other popular models like Deepseek, LLaVa, Yi-VL.

[1] Deepseek llm: Scaling open-source language models with longtermism.
[2] Visual instruction tuning, Neurips 2023
[3] Yi: Open foundation models by 01.ai, 2024.