MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs

## Abstract

Abstract Current multimodal misinformation detection (MMD) methods often assume a single source and type of forgery for each sample, which is insufficient for real-world scenarios where multiple forgery sources coexist.
The lack of a benchmark for mixed-source misinformation has hindered progress in this field. To address this, we introduce MMFakeBench, the first comprehensive benchmark for mixed-source MMD. MMFakeBench includes 3 critical sources: textual veracity distortion, visual veracity distortion, and cross-modal consistency distortion, along with 12 sub-categories of misinformation forgery types. We further conduct an extensive evaluation of 6 prevalent detection methods and 15 large vision-language models (LVLMs) on MMFakeBench under a zero-shot setting.
The results indicate that current methods struggle under this challenging and realistic mixed-source MMD setting. Additionally, we propose an innovative unified framework, which integrates rationales, actions, and tool-use capabilities of LVLM agents, significantly enhancing accuracy and generalization.
We believe this study will catalyze future research into more realistic mixed-source multimodal misinformation and provide a fair evaluation of misinformation detection methods.

## 1 Introduction

Recent advances in generative models for texts [, ] and images [, ] have significantly lowered the barrier to producing diverse multimodal misinformation, posing threats to politics, finance, and public health. For instance, the misinformation “COVID-19 vaccine causes brain damage”, shown in Fig. 1, accompanied by a highly convincing image, can lead to public distrust in medical treatments and vaccine refusal. Therefore, identifying multimodal misinformation on social media is urgent.

Most current multimodal misinformation detection (MMD) methods [, , , , , ] typically assume that each sample has a single, known forgery source, as depicted in Fig. 1 Top. This assumption is overly simplistic and fails to capture the complexity of real-world scenarios, where misinformation often originates from multiple, random sources. To address this mixed-source MMD problem, two key challenges need to be solved.
First, existing datasets primarily consist of single-source misinformation, lacking misinformation from multiple sources. This limitation prevents comprehensive evaluation of MMD methods. Second, there is a lack of general detectors capable of effectively handling mixed-source misinformation. Hence, we present MMFakeBench, encompassing the mixed-source MMD benchmark, evaluations, and framework.

Benchmark: We introduce MMFakeBench, the first comprehensive benchmark for evaluating mixed-source MMD. As shown in Fig. 1 Bottom, leveraging advanced AI tools, such as diffusion generators and ChatGPT, MMFakeBench provides 12 forgery types with 11,000 data pairs from three primary sources: textual veracity distortion, visual veracity distortion, and cross-modal consistency distortion. Textual veracity distortion encompasses three types of rumors: natural, artificial, and GPT-generated rumors. Unlike [, , , ] that focus solely on single-source, text-only rumors, MMFakeBench incorporates text-image rumors using highly relevant real or AI-generated images.
Visual veracity distortion filters existing PS-edited images [, ] according to misinformation standards and incorporates high-quality AI-generated images.
Cross-modal consistency distortion integrates inconsistencies from both edited and repurposed perspectives into five distinct sub-categories.

Evaluations: To access the current advancements in mixed-source MMD, we conduct a comprehensive evaluation of 6 state-of-the-art detection methods and 15 large vision-language models (LVLMs) on MMFakeBench. Specifically, we evaluate 6 detection methods in a single-source setting and assess their combined performance (text, image, and cross-modal inconsistency detectors) in the mixed-source setting. Additionally, we evaluate 15 large vision-language models (LVLMs), including proprietary models such as GPT-4V []. The results indicate that existing detection methods exhibit poor generalization. Although LVLMs show robust generalization capabilities, their overall performance still requires improvement.

Framework: Based on our analysis, we propose a simple yet effective LVLM-based framework called MMD-Agent, which enhances detection performance and can serve as a new baseline for future research. MMD-Agent decomposes mixed-source detection into three stages: textual veracity check, visual veracity check, and cross-modal consistency reason. This decomposition ensures methodical and thorough reasoning. At each stage, MMD-Agent instructs LVLMs to generate multi-perspective rationales, integrating model actions for coherent decisions. Additionally, the models interact with external environments (e.g., Wikipedia) to incorporate supplementary information into their reasoning.

In summary, the main contributions are:
(1) We introduce mixed-source multimodal misinformation detection (MMD), a challenging setting for detecting misinformation from diverse and uncertain sources, breaking free from single-source constraints, and advancing practical misinformation detection tasks.
(2) We develop MMFakeBench, the first benchmark dataset for evaluating mixed-source MMD. The dataset contains 3 critical categories (textual veracity distortion, visual veracity distortion, and consistency reasoning) and 12 sub-categories of forgery types.
(3) Using the newly collected dataset, we benchmark mixed-source MMD by evaluating 6 prevalent detection methods and 15 LVLMs.
(4) We propose MMD-Agent, a simple yet effective LVLM-based framework. It outperforms previous methods and LVLMs on the MMFakeBench benchmark, highlighting the potential of mixed-source MMD and providing a new baseline for future research.

**Table 1: Comparison of misinformation datasets. (k) denotes the number of rumor types.**
| Dataset | Textual Veracity Distortion | Visual Veracity Distortion | Cross-modal Consistency Distortion |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Text<br>(Rumor) | Supporting Image | Text<br>(Veracity) | Fact-conflicting Image | Image/Text<br>Repurposing | Image/Text<br>Editing |  |  |  |
| Repurposed | AI-generated | PS-edited | AI-generated |  |  |  |  |  |
| FEVER [] | ✔ (1) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Politifact [] | ✔ (1) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Gossipcop [] | ✔ (1) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Snopes [] | ✔ (1) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| MOCHEG [] | ✔ (1) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| LLMFake [] | ✔ (1) | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| EMU [] | ✗ | ✗ | ✗ | ✗ | ✔ | ✗ | ✗ | ✗ |
| Fakeddit [] | ✗ | ✗ | ✗ | ✔ | ✔ | ✗ | ✗ | ✗ |
| MAIM [] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✔ | ✗ |
| MEIR [] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✔ |
| NewsCLIPpings [] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✔ | ✗ |
| COSMOS [] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✔ | ✗ |
| DGM4 [] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✔ |
| MMFakeBench | ✔ (3) | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |

## 2 Related Work

##### Misinformation Benchmarks.

One group of misinformation datasets primarily focuses on distorting textual veracity. The Fever [] dataset is constructed by manipulated Wikipedia sentences with manual annotation. Unlike these artificial rumors, other datasets, such as Snopes [], FakeNewsNet [], and MOCHEG [], collect natural rumors from fact-checking websites. Recently, the LLMFake [] dataset is curated by instructing large language models (LLMs) to generate diverse misinformation. Apart from textual veracity distortion, the EMU [] and Fakeddit [] datasets collect Photoshop-edited images from the Reddit platform. Another group of misinformation datasets focuses on disrupting cross-modal consistency. The MAIM  [] and MEIR [] datasets employ caption replacement and entity swapping, respectively. The NewsCLIPpings [] dataset retrieves out-of-context images to support certain narratives. COSMOS [] is created by linking images with various captions. The recent dataset DGM^4 [] introduces global and local manipulation to alter semantics and sentiment. Different from these works containing only single-source misinformation, we propose the first benchmark dataset for evaluating mixed-source MMD, involving textual veracity distortion, visual veracity distortion, and cross-modal consistency distortion, shown in Table 1.

##### Misinformation Detection.

Current misinformation detection approaches are mainly divided into two categories. The first is to check textural veracity. Earlier works focus on better-constructing features based on writing style [], sentiment [], user feedback []. Recent works utilize pre-trained language models [, , ] and their reasoning ability to assess the claim’s veracity. The second is to incorporate cross-modal features to detect semantic inconsistencies using contextual attention [], multi-view [], causal reasoning [], instruction tuning [, ]. However, those work each target at a single-source problem, and evaluations are conducted in constrained scenarios. Our work is the first to introduce a comprehensive benchmark for mixed-source multimodal misinformation detection.

##### Large Vision-Language Models.

Large language models (LLMs) such as GPT-3 [] and Vicuna [] have demonstrated remarkable performance on various linguistic tasks. Inspired by LLMs, models like LLaVA [] and MiniGPT-4 [] facilitate image-text feature alignment by leveraging visual instruction data for fine-tuning. More recently, the evolution of LVLMs has driven advancements in creating diverse and high-quality multimodal instruction datasets. Models such as InstructBLIP [], mPLUG-Owl [, ], VILA [], LLaVA-1.5 [] exemplify these developments. In this paper, we explore the reasoning capabilities [, ] of LVLMs to address the challenge of mixed-source multimodal misinformation by integrating rationales, actions, and tool-use capabilities.

Figure: Figure 2: (a) Statistics of the MMFakeBench Benchmark. (b-d) Illustrations of using collaborative generative models and AI tools to generate different sources of misinformation.
Refer to caption: x2.png

## 3 MMFakeBench Benchmark

In MMFakeBench, we address three critical multimodal misinformation sources: textual veracity distortion, visual veracity distortion, and cross-modal consistency distortion. For textual veracity distortion, it includes three types of text-based rumors paired with two types of supporting images to mimic real-world multimodal scenarios. For visual veracity distortion, we employ two visual techniques to introduce fact-conflicting information in images while the texts remain veracious. For cross-modal consistency distortion, both texts and images are veracious, yet cross-modal inconsistencies exist between them with three repurposed methods and two edited methods.

### 3.1 Three Misinformation Sources

#### 3.1.1 Textual Veracity Distortion

Textual veracity distortion is a critical misinformation category. The dataset in this category comprises 3,300 samples. Previous works [, , , ] focus on single-source and single-modal textual rumors. However, the types of real-world rumors are diverse, and those accompanied by images can have a significantly greater impact. To address this, MMFakeBench introduces a broader range of rumor types and augments them with highly relevant supporting images to enhance perceived credibility.

##### Textual Rumor.

As shown in Table 1, unlike previous methods that consider only one type of textual rumor, we consider three distinct types: (1) Natural Rumor. We select natural rumors from Politifact [] and Gossipcop [], which provide political news and entertainment stories derived from fact-checking websites. (2) Artificial Rumor. We collect artificial rumors from the FEVER dataset [], which is curated by manually modifying Wikipedia sentences. (3) GPT-Generated Rumor. We instruct ChatGPT(^1^11gpt-3.5-turbo: [https://platform.openai.com/docs/models/gpt-3-5](https://platform.openai.com/docs/models/gpt-3-5)) to produce GPT-generated rumors via three prompt approaches proposed by []: arbitrary generation, rewriting generation, and information manipulation. Arbitrary generation is utilized to generate misinformation in specific domains. Rewriting generation addresses the concise and synthetic traces of artificial rumors, while information manipulation involves altering factual information in real claims from Politifact and Gossipcop.

##### Supporting Image.

We use either AI-generated images or carefully selected real images to support the content presented in the rumor text: (1) AI-Generated image.
For artificial rumors and their derived GPT-generated rumors, as well as some less harmful gossip, we utilize generative models to create supporting images.
We utilize three advanced models: Stable Diffusion XL [], DALL-E3 [], and Midjourney V6 [] to enhance the diversity of the generated images.
For each rumor, we randomly select a generative model to produce a corresponding supporting image. As many rumors are highly abstract and lack concrete descriptions of objects and scenes, directly using these texts as conditions often yields images that are neither realistic nor relevant. To address this, as shown in Fig. 2 (b), we instruct ChatGPT to enrich the rumors with more detailed descriptions. These enriched contexts are then used as input for generative models, ensuring alignment with the textual rumors. (2) Repurposed image. To avoid creating new high-risk images, especially for sensitive topics like politics and gossip, we use repurposed images from the VisualNews [] dataset, which contains numerous image-text pairs from real-world news sources. We select images with high semantic relevance to the textual rumors based on text-image CLIP similarity and text-text CLIP similarity. Images with similarity scores above a certain threshold are chosen as supporting images.

#### 3.1.2 Visual Veracity Distortion

The visual veracity distortion dataset comprises 1,100 samples where the text is real and the misinformation exists in the image. Previous datasets [, ] focus solely on PS-edited (Photoshop-edited) images, containing both misleading and non-misleading content. In this study, we manually select the misleading ones and include them in MMFakeBench. Besides, we incorporate AI-generated images with veracity distortion, which is increasingly harmful due to advancements in diffusion generators.

##### PS-Edited Image.

The PS-edited images are derived from the "manipulated content" samples in the Fakeddit dataset [], which is designed for multimodal fake news detection. These samples originate from the "Photoshop battles comments" on Reddit. However, not all PS edits result in misinformation by our benchmark standards, as some are minor aesthetic modifications that do not compromise visual veracity. Thus, to build our visual veracity distortion dataset, we manually select 550 PS-edited images containing fact-conflicting content from the 7,693 samples of the "manipulated content" set.

##### AI-generated Image.

We propose an automated pipeline that generates fact-conflicting descriptions from text captions and then creates high-quality images. Specifically, we first collect image-text pairs from the MS-COCO [] and VisualNews datasets []. Based on the original text captions, we use ChatGPT to generate corresponding fact-conflicting descriptions, which are manually verified, as depicted in Fig. 2 (c). For example, from the caption “Scenic boats daily travel the Thames in England”, we generate the description “The Queen’s face appears in the Thames”.
These descriptions, combined with the original captions, are used as prompts in the Midjourney V6 model [] to create corresponding images.
The resulting text-image pairs contain original factual text and generated images with additional fact-conflicting information.

#### 3.1.3 Cross-modal Consistency Distortion

In cross-modal consistency distortion, both the text and image with veracity, but either the text or image is replaced/manipulated to disrupt their overall consistency.
Previous datasets [, , ] focus on inconsistencies from a single source, either edit-based or repurposed-based. In contrast, our MMFakeBench integrates inconsistencies from both edited and repurposed perspectives into five distinct sub-categories, a total of 3,300 image-text pairs.

Figure: Figure 3: Comparison of standard prompt and proposed MMD-Agent methods, (a) Three examples of multimodal misinformation from distinct sources. (b) LVLMs with standard prompt methods fail to make correct judgments. (c) MMD-Agent instructs LVLMs to decompose complex mixed-source detection into smaller subtasks. Each subtask is addressed by integrating multi-perspective rationales.
Refer to caption: x3.png

##### Repurposed Inconsistency.

Our dataset encompasses three types of repurposed inconsistency, curated directly from the NewsCLIPpings [] dataset: semantic query, person query, and scene query. (1) Semantic query retrieves repurposed images based on specific semantic content. (2) Person query ensures the individual mentioned in the caption appears in the mismatched image. (3) Scene Query relies on spatial similarity to retrieve comparable scene information from repurposed images.

##### Edited Inconsistency.

Our dataset contains two types of edited inconsistency: text editing and image editing.
For text editing, we select samples from the DGM^4 [] dataset, which modifies sentiment words with their antonyms.
Notably, some samples in DGM^4 contain fact-conflicting content post-editing.
To avoid redundancy with textual veracity distortion, we filter out these samples. For the remaining text editing and all image editing inconsistencies, we build upon the COCO-Counterfactuals [] dataset.
COCO-Counterfactuals dataset encompasses original image-text pairs $\left(text_{ori},img_{ori}\right)$ and edited image-text pairs $\left(text_{edit},img_{edit}\right)$ which are obtained via Instruct-Pix2Pix model []. As illustrated in Fig. 2 (d), we separately extract word disparities between $text_{ori}$ and $text_{edit}$ and select samples with significant semantic differences using CLIP similarity. Then, we reassemble the two pairs and obtain $\left(text_{edit},img_{ori}\right)$ as text-edited consistency distortion samples and $\left(text_{ori},img_{edit}\right)$ as image-edited consistency distortion samples.

### 3.2 Real Data Collection

In addition to the misinformation data, we collect 3,300 real data pairs, ensuring both textual and visual veracity and exhibiting strong image-text consistency.
Given that our synthetic data is derived from multiple datasets, we construct the real dataset from the same corresponding sources, including MS-COCO [], VisualNews [], and real image-text pairs from Fakeddit []. We further divide VisualNews into four distinct news sources: The Guardian, BBC, USA TODAY, and The Washington Post. Finally, we build the real dataset by
equally selecting from six distinct sources.

### 3.3 MMFakeBench Analysis

MMFakeBench consists of 11,000 image-text pairs, which are divided into a validation set and a test set following []. The validation set, comprising 1,000 image-text pairs, is intended for hyperparameter selection, while the test set contains 10,000 pairs.
MMFakeBench encompasses one real category and three misinformation categories.
Detailed statistics are shown in Figure 2 (a).
MMFakeBench is partitioned into 30% for textual veracity distortion, 10% for visual veracity distortion,30% for cross-modal consistency distortion, and 30% for real data.
The three misinformation categories can be further subdivided into 12 detailed subcategories based on the sources of the text and images, as shown in Figure 2 (a). Such a comprehensive benchmark highlights the challenges of intertwining mixed-source and multiple-types multimodal misinformation in the real world.

## 4 Our MMD-Agent Framework

We present a simple yet effective framework called MMD-Agent, which integrates the rationales, actions, and tool-use capabilities of LVLM agents. As shown in Fig. 3, MMD-Agent involves two main processes: (1) Hierarchical decomposition and (2) Integration of multi-perspective rationales.

Specifically, we instruct LVLMs $\mathcal{M}$ to decompose the task of mixed-source multimodal misinformation detection into three smaller subtasks: textual veracity check, visual veracity check, and cross-modal consistency reasoning. During the intermediary phase, each subtask $t$ is addressed by generating multi-perspective rationales $\mathcal{R}_{t}$ and integrating them with model actions $a_{t}$ to facilitate decision-making $d_{t}$:

$$ $d_{t}=\mathcal{M}\left(a_{t},\mathcal{R}_{t}\right).$ (1) $$

The rationales aim to provide valuable insights by reasoning within the current context, addressing the requirement of detecting diverse clues in misinformation. As depicted in Fig. 3 (c), these rationales can encompass multiple perspectives, such as textual key entity (Rationale 1), injecting retrieval knowledge (Rationale 2),
factual analysis (Rationale 3), and commonsense analysis ( Rationale 5 and Rationale 7). The model is guided to generate reasoning paths and make decisions for each subtask individually, leveraging its inherent understanding and capabilities.

## 5 Experiments

We select 6 state-of-the-art misinformation detection methods and 15 large vision-language models (LVLMs) for preliminary benchmarking using the MMFakeBench dataset, aiming to explore their applicability in the mixed-source MMD setting. Additionally, we evaluate the performance of our proposed framework, MMD-Agent. All evaluations are conducted under a zero-shot setting on our benchmark. All experiments are performed on NVIDIA GeForce 3090 GPUs with PyTorch.

### 5.1 Experimental Setting

##### Baseline Models.

We select LVLMs of varying sizes as baseline models. (i) Language model with 7B parameter including Otter [], MiniGPT-4 [], InstructBLIP [], Qwen-VL [], VILA [], PandaGPT [], mPLUG-Owl2 [], BLIP-2 [] and LLaVA-1.6 []. (ii) Language model with 13B parameter including VILA, InstructBLIP, BLIP-2, and LLaVA-1.6. (iii) Language model with 34B parameter including LLaVA-1.6. (iv) Closed-source model including GPT-4V [].

##### Evaluation Metrics.

We evaluate the performance of different baselines using multi-class classification. Consistent with [, , ], we adopt the widely-used macro-F1 metric, which balances precision and recall through a harmonic mean. Beyond the F1 score, we also include macro-precision, macro-recall, and macro-accuracy as complementary evaluation metrics. Specifically, we construct robust regular expressions that are employed to extract key phrases from the long responses for accurate answer matching. If a model’s response lacks a valid answer, we classify it as a pseudo choice “Z” and consider the response incorrect, following [].

**Table 2: Overall results (%) of different models on the MMFakeBench validation and test set with the comparison of standard prompt (SP) and proposed MMD-Agent framework.**
|  |  |  | Validation (1000) | Test (10000) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model<br>Name | Language<br>Model | Prompt<br>Method | F1$\uparrow$ | Precision$\uparrow$ | Recall$\uparrow$ | ACC$\uparrow$ | F1$\uparrow$ | Precision$\uparrow$ | Recall$\uparrow$ | ACC$\uparrow$ |
| Language Model with 7B Parameter |  |  |  |  |  |  |  |  |  |  |
| Otter-Image [] | MPT-7B | SP | 5.2 | 10.5 | 3.4 | 4.1 | 4.9 | 9.3 | 3.3 | 4.0 |
| MiniGPT4 [] | Vicuna-7B | SP | 5.2 | 5.2 | 21.2 | 9.0 | 5.3 | 6.9 | 21.0 | 9.1 |
| InstructBLIP [] | Vicuna-7B | SP | 7.1 | 7.9 | 6.5 | 7.8 | 8.1 | 16.4 | 7.2 | 8.5 |
| Qwen-VL [] | Qwen-7B | SP | 7.5 | 10.3 | 24.3 | 11.0 | 8.0 | 35.9 | 25.5 | 11.6 |
| VILA [] | LLaMA2-7B | SP | 11.5 | 7.5 | 25.0 | 30.0 | 11.5 | 7.5 | 25.0 | 30.0 |
| PandaGPT [] | Vicuna-7B | SP | 11.8 | 9.8 | 25.0 | 30.0 | 11.6 | 8.6 | 25.0 | 30.0 |
| mPLUG-Owl2 [] | LLaMA2-7B | SP | 14.5 | 22.2 | 25.9 | 31.1 | 15.1 | 25.2 | 26.3 | 31.5 |
| BLIP2 [] | FlanT5-XL | SP | 16.4 | 20.1 | 27.5 | 33.0 | 16.7 | 17.3 | 27.7 | 33.2 |
| LLaVA-1.6 [] | Vicuna-7B | SP | 17.4 | 14.8 | 25.7 | 30.8 | 19.0 | 16.5 | 26.9 | 32.3 |
| Language Model with 13B Parameter |  |  |  |  |  |  |  |  |  |  |
|  |  | SP | 11.5 | 7.5 | 25.0 | 30.0 | 11.6 | 32.5 | 25.0 | 30.0 |
| VILA [] | LLaMA2-13B | MMD-Agent | 22.7 | 27.3 | 24.4 | 28.7 | 24.0 | 30.4 | 25.5 | 29.4 |
|  |  | SP | 13.7 | 13.2 | 24.0 | 28.8 | 13.9 | 25.5 | 24.3 | 29.1 |
| InstructBLIP [] | Vicuna-13B | MMD-Agent | 26.0 | 33.3 | 30.1 | 29.5 | 24.5 | 32.1 | 28.8 | 27.3 |
|  |  | SP | 16.7 | 34.9 | 27.3 | 32.8 | 16.3 | 34.6 | 27.3 | 32.8 |
| BLIP2 [] | FlanT5-XXL | MMD-Agent | 31.6 | 39.8 | 32.2 | 34.4 | 28.8 | 39.0 | 30.4 | 32.1 |
|  |  | SP | 12.0 | 22.5 | 25.0 | 30.0 | 14.4 | 35.7 | 26.0 | 31.2 |
| LLaVA-1.6 [] | Vicuna-13B | MMD-Agent | 38.0 | 44.5 | 41.0 | 40.6 | 34.5 | 42.7 | 37.5 | 37.4 |
| Language Model with 34B Parameter |  |  |  |  |  |  |  |  |  |  |
|  |  | SP | 25.7 | 44.5 | 33.7 | 40.4 | 25.4 | 44.1 | 33.8 | 40.5 |
| LLaVA-1.6 [] | Nous-Hermes-2<br>-Yi-34B | MMD-Agent | 49.9 | 54.4 | 52.9 | 48.7 | 47.7 | 52.1 | 49.6 | 46.6 |
| Proprietary LVLMs |  |  |  |  |  |  |  |  |  |  |
|  |  | SP | 51.0 | 66.8 | 49.7 | 54.0 | 48.8 | 63.0 | 48.7 | 54.2 |
| GPT-4V [] | ChatGPT | MMD-Agent | 61.6 | 67.8 | 59.3 | 62.1 | 61.5 | 67.7 | 59.1 | 61.0 |

Table: Table 3: (a) Comparison with single-source detectors for MMFakeBench. (b) Ablation studies on hierarchical (Hier.) decomposition and rationales $\mathcal{R}_{t}$ of each sub-task $t$. TVD, VVD, and CCD denote textual veracity distortion, visual veracity distortion, and cross-modal consistency distortion, respectively. Corpus denotes the general corpus in LVLMs that is not tailored for misinformation detection. * denotes each powerful single-source detector that is used for mixed detection.

### 5.2 Main Results

##### Comparison of Different LVLMs.

We present a comprehensive comparison of different LVLMs using the MMFakeBench, detailed in Table 2. Our key findings are summarized as follows:

1) Challenges of MMFakeBench: The benchmark poses substantial challenges to current models. Notably, GPT-4V, despite its advancement, achieves an F1-score of only 51.0% with the standard prompt. This indicates considerable room for improvement and highlights the rigorous standards of this benchmark.

2) Disparity between Open-source Models and GPT-4V: Although LLaVA-1.6-34b is the leading open-source model, it achieves an F1-score of just 25.7% with the standard prompt, significantly lower than GPT-4V. This highlights a pronounced disparity in detection capabilities between open-source and proprietary models.

3) Impact of Parameter Quantity: Comparing models within the same series, such as LLaVA-1.6-Vicuna-7b and LLaVA-1.6-34b, we observe that models with larger parameter counts exhibit better performance. Smaller LVLMs face constraints in instruction-following and high predicted consistency, as detailed in the Appendix. These results indicate that 7B parameter models lack sufficient multimodal understanding to effectively combat misinformation.

4) Effectiveness of MMD-Agent: Given the limitations inherent in small-scale
models, we selected moderately sized open-source and proprietary models as baselines to compare the proposed MMD-Agent with the standard prompt. MMD-Agent significantly improves the F1-score performance of both open-source models and GPT-4V. Notably, the LLaVA-1.6-34B using MMD-Agent achieves an F1-score of 49.9%, approaching the 51% score of GPT-4V with the standard prompt.
This suggests that MMD-Agent could serve as a general framework for future research on the MMFakeBench benchmark.

##### Comparison with Single-source Detectors.

We compare LLaVA-1.6-34B and GPT-4V utilized in MMD-Agent, with several competitive single-source detectors [, , , , , ], which represent the SOTA methods for each single-source scenarios. The details of each detector are presented in the Appendix. For a fair comparison, in addition to the single-source misinformation detection via existing detectors, we integrate the three most powerful detectors in distinct sources (i.e., FakeingFakeNews [], LNP [] and HAMMER []) to assess the capability of mixed detection. Mixed detection utilizes our proposed hierarchical framework by replacing LVLMs with relevant detectors. The results in Table 3 (a) show that LLaVA-1.6-34B and GPT-4V perform better than single-source detectors for both binary and multiclass classification by a large margin. This demonstrates that LVLMs trained with a large general corpus achieve promising generalization performance in mixed-source MMD and can serve as potential baseline models for future study.

Figure: Table 4: Performance (F1 score (%)) of models on different sources of misinformation.

Figure: Figure 4: Performance (detection success rate) of models on different types of rumor.
Refer to caption: x4.png

### 5.3 Experimental Analysis

Ablation Study on Hierarchical Decomposition and Rationales. We first investigate the effects of instructing LVLMs only with hierarchical decomposition in comparison with standard prompts. In Table 3 (b), the decomposition method performs better for solving multi-task interference. Additionally, we conduct an ablation study by sequentially generating multi-perspective rationales for individual sub-tasks. Results in Table 3 (b) show that augmenting decisions with rationales outperform its ablation part, especially for textual veracity distortion and visual veracity distortion.

Analysis of Misinformation Sources and Types. We compare the F1 scores of various LVLMs across misinformation sources in Table 5.2. Across all sources, the majority of open-source models perform worse than GPT-4V by a huge margin, particularly in terms of textual veracity distortion. This indicates that open-source models are considerably challenged by textual veracity distortion. Within textual veracity distortion, we further report the detection accuracy of selected models across three types of textual rumors. Fig. 5.2 shows that all open-source models performed significantly worse in the “natural rumor” and “GPT-generated rumor” sub-categories. This indicates that GPT-generated rumors are challenging and closer to real-world rumors.

Figure: Figure 5: One of the most harmful examples involves a GPT-generated rumor supported by an AI-generated image, challenging for LLaVA-1.6-34B. More examples can be found in the Appendix.
Refer to caption: x5.png

Error Analysis. A fundamental root cause of textual veracity checking errors in the LLaVA-1.6-34B is the lack of useful external knowledge. This deficiency is exemplified in Fig.5, where the knowledge contains only commonsense information but fails to provide relevant events. Moreover, this AI-generated image in the image-text pair exhibits high fidelity and strong coherence, thus evading detection in visual veracity and cross-modal consistency. These instances underscore the dangers of using collaborative generative modes to automatically generate multimodal misinformation.

## 6 Conclusion

In this paper, we introduce MMFakeBench, the first comprehensive benchmark for detecting mixed-source multimodal misinformation. MMFakeBench contains three primary misinformation categories along with 12 sub-categories of forgery types. We conduct comprehensive evaluations of 6 prevalent detection methods and 15 LVLMs on the MMFakeBench dataset. Furthermore, we propose an innovative unified framework and perform extensive experiments to demonstrate its effectiveness.

## 7 Appendix

This appendix contains additional details for the paper, titled “MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs”. The appendix is organized as follows:

- •
§7.1 Additional Experimental Results.
- •
§7.2 Existing Single-source Detectors Details.
- •
§7.3 Instruct Prompts for ChatGPT.
- •
§7.4 Example Analysis.
- •
§7.5 Broader Impact.
- •
§7.6 Limitations.

### 7.1 Additional Experimental Results

##### Instruction Following and Prediction Consistency.

We evaluated the instruction following capabilities and prediction consistency to further study the multimodal understanding of LVLMs on mixed-source multimodal misinformation detection (MMD). We report the success rate in heuristic matching (Match) with regular expressions and prediction consistency rate (Consist.). The results are shown in the Table 5. Among all LVLMs, small-scale models like Otter-Image and InstructBLIP, achieve the lower matching success rate. While there exist small-scale LVLMs that perfectly follow the format of the regular expressions and achieve high success rates (>99%) in matching, most small-scale models exhibit high predicted consistency rates. This indicates small-scale models may prefer to predict a certain category answer among all given choices. Additionally, the leading open-source model, LLaVA-1.6-34B, and the proprietary model GPT-4V demonstrate superior instruction-following capabilities and lower prediction consistency. This indicates their significant potential in addressing mixed-source MMD, positioning them as valuable baseline models.

**Table 5: Statistics of Instruction following capabilities and predicted consistency tendency of LVLMs.**
| Model | Match | Consist. | Model | Match | Consist. | Model | Match | Consist. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Language Model with 7B Parameter |  |  |  |  |  |  |  |  |
| Otter-Image [] | 9.8 | 100 | Qwen-VL [] | 88.6 | 92.9 | mPLUG-Owl2 [] | 100 | 96.6 |
| MiniGPT4 [] | 100 | 88.6 | VILA [] | 100 | 100 | BLIP2 [] | 100 | 93.6 |
| InstructBLIP [] | 25.7 | 96.11 | PandaGPT [] | 100 | 98.9 | LLaVA-1.6 [] | 100 | 76.9 |
| Language model with 13B Parameter |  |  |  |  |  |  |  |  |
| VILA [] | 100 | 100 | InstructBLIP [] | 99.9 | 91.0 | BLIP2-FlanT5-XXL [] | 100 | 49.4 |
| Language model with 34B Parameter |  |  |  |  |  |  |  |  |
| LLaVA-1.6 [] | 100 | 52.6 | - | - | - | - | - | - |
| Proprietary LVLMs |  |  |  |  |  |  |  |  |
| GPT-4V [] | 99.9 | 36.5 | - | - | - | - | - | - |

##### Evaluation Results for Binary Classification.

In addition to multi-class classification, we also provide binary classification performance to assess the overall detection capability of baseline models in mixed-source MMD. Based on the 4 categories in the mixed-source MMD settings, we develop binary evaluation metrics via mapping techniques. Specifically, we standardize the assignment of labels denoting “textual veracity distortion”, “visual veracity distortion”, and “cross-modal consistency distortion” to the classification of “Fake” while reserving the label “True” to denote real data. Similar to the multi-classification evaluation, we adopt the widely used F1 score. In addition to the F1 score, we also use precision, recall, and accuracy as supplementary evaluation metrics. The specific evaluation results are shown in Table 6. From the results, we make the following observations:

- •
Current models including open-source models and GPT-4V are challenged by the MMFakeBench dataset in binary classification detection. Despite being an advanced modal, GPT-4V attains a mere F1-score of 72.3% using the standard prompt on the validation set.
- •
The proposed framework MMD-Agent yields substantial improvement on recent LVLMs, especially on open-source models. For instance, for the BLIP2-FlanT5-XXL model, MMD-Agent achieves a 19.9% increase in F1 score on the validation set. This may be credited to the effective integration of rationales, actions, and tool use in enhancing multimodal understanding in mixed-source MMD.

**Table 6: Binary overall results of different models on the MM-FakeBench validation and test set with the comparison of standard prompt (SP) and proposed MMD-Agent framework.**
|  |  |  | Validation (1000) | Test (10000) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model<br>Name | Language<br>Model | Prompt<br>Method | F1 | Precision | Recall | ACC | F1 | Precision | Recall | ACC |
| Language Model with 7B Parameter |  |  |  |  |  |  |  |  |  |  |
| Otter-Image[] | MPT-7B | SP | 7.9 | 4.1 | 4.5 | 7.9 | 8.6 | 32.4 | 5.0 | 8.6 |
| MiniGPT4 [] | Vicuna-7B | SP | 40.4 | 38.2 | 45.7 | 63.1 | 41.7 | 41.0 | 47.4 | 65.2 |
| InstructBLIP [] | Vicuna-7B | SP | 14.7 | 30.8 | 13.2 | 8.1 | 16.1 | 40.5 | 14.2 | 8.8 |
| Qwen-VL [] | Qwen-7B | SP | 43.6 | 50.6 | 44.9 | 60.3 | 44.0 | 51.6 | 45.2 | 60.5 |
| VILA [] | LLaMA2-7B | SP | 41.2 | 35.0 | 50.0 | 70.0 | 41.2 | 35.0 | 50.0 | 70.0 |
| PandaGPT [] | Vicuna-7B | SP | 24.6 | 60.6 | 50.5 | 30.9 | 24.1 | 61.7 | 50.4 | 30.6 |
| mPLUG-Owl2 [] | LLaMA2-7B | SP | 47.2 | 64.9 | 52.3 | 70.6 | 48.7 | 71.1 | 53.3 | 71.4 |
| BLIP2 [] | FlanT5-XL | SP | 41.2 | 35.0 | 50.0 | 70.0 | 41.2 | 35.0 | 50.0 | 70.0 |
| LLaVA-1.6 [] | Vicuna-7B | SP | 48.1 | 48.2 | 48.5 | 59.5 | 52.5 | 53.0 | 52.6 | 62.5 |
| Language Model with 13B Parameter |  |  |  |  |  |  |  |  |  |  |
|  |  | SP | 41.1 | 35.0 | 50.0 | 70.0 | 41.1 | 35.0 | 50.0 | 70.0 |
| VILA [] | LLaMA2-13B | MMD-Agent | 56.5 | 62.2 | 56.9 | 70.3 | 56.6 | 64.3 | 57.2 | 71.2 |
|  |  | SP | 41.1 | 35.0 | 49.9 | 69.9 | 41.1 | 35.0 | 49.9 | 69.8 |
| InstructBLIP [] | Vicuna-13B | MMD-Agent | 51.3 | 53.4 | 54.0 | 53.1 | 47.9 | 50.1 | 50.1 | 49.9 |
|  |  | SP | 31.6 | 63.4 | 53.6 | 35.5 | 30.6 | 64.9 | 53.4 | 34.9 |
| BLIP2 [] | FlanT5-XXL | MMD-Agent | 51.5 | 53.4 | 54.0 | 53.6 | 51.8 | 54.0 | 54.7 | 53.5 |
|  |  | SP | 41.1 | 35.0 | 50.0 | 69.7 | 42.3 | 57.3 | 50.1 | 69.5 |
| LLaVA-1.6 [] | Vicuna-13B | MMD-Agent | 51.8 | 66.7 | 54.6 | 71.4 | 50.2 | 67.3 | 53.9 | 71.3 |
| Language Model with 34B Parameter |  |  |  |  |  |  |  |  |  |  |
|  |  | SP | 62.9 | 67.1 | 70.0 | 63.4 | 64.3 | 68.8 | 71.7 | 64.8 |
| LLaVA-1.6 [] | Nous-Hermes-2<br>-Yi-34B | MMD-Agent | 67.2 | 70.4 | 66.0 | 75.1 | 68.1 | 71.1 | 67.0 | 75.6 |
| Proprietary LVLMs |  |  |  |  |  |  |  |  |  |  |
|  |  | SP | 72.3 | 72.1 | 72.8 | 75.6 | 74.2 | 73.5 | 76.9 | 76.4 |
| GPT-4V [] | ChatGPT | MMD-Agent | 74.0 | 73.4 | 75.5 | 76.8 | 72.8 | 72.4 | 75.4 | 75.0 |

##### Analysis of More Misinformation Types.

Within visual veracity distortion, we report the detection accuracy of selected models across two types of fact-conflicting images. Fig.7.1 shows the challenging nature of both types of such fact-conflicting images for existing models. Notably, even the advanced GPT-4V achieves a detection success rate of less than 50% on both types. Additionally, we present an analysis of the detection accuracy of selected models across different types of image-text inconsistency. As shown in Fig.7.1, edited inconsistency emerges as a more substantial challenge compared to repurposed inconsistency. This finding suggests that editing methods introduce minor alterations to images or text, necessitating enhanced multimodal reasoning capabilities.

Figure: Figure 6: Performance (detection success rate) of models on different types of fact-conflicting images.
Refer to caption: x6.png

Figure: Figure 7: Performance (detection success rate) of models on different types of inconsistent image-text pairs.
Refer to caption: x7.png

### 7.2 Existing Single-source Detectors Details

##### FakeingFakeNews.

The FakingFakeNews [] is designed for the detection of textual fake news, particularly for those natural human-written misinformation. It proposes an innovative approach for generating training instances, leveraging established styles and strategies commonly employed in human-authored propaganda. FakingFakeNews employs the ROBERTA [] model as the backbone and trains it on its own proposed PROPANEWS dataset. In our experiments, we utilized the default configuration of the ROBERTA detector provided within the FakingFakeNews framework, retaining its default hyperparameters.

##### CNNSpot.

CNNSpot [] is an artificial image detector designed specifically for identifying images produced by generative models. It employs the ResNet-50 model as the classifier backbone. Notably, CNNSpot recognizes that data augmentation, including JPEG compression and Gaussian blur, can enhance the generalization capabilities of the detector. In our study, we utilize the pre-trained CNNSpot model with default hyperparameters to perform the detection of visual veracity distortion.

##### UnivFD.

UvinFD [] is a general-purpose fake image detector that uses a feature space not explicitly trained to distinguish between real from fake images. When given access to the feature space of a pre-trained vision-language model, UvinFD employs the nearest neighbor to identify fake images originating from various sources. The utilization of the large pre-trained model results in a smooth decision boundary, thereby enhancing the generalization capability of the detector. In our work, we use the pre-trained detector of UnivFD with default hyperparameters to conduct the visual veracity distortion detection task.

##### LNP.

LNP [] utilizes a well-trained denoising model to extract noise patterns from spatial images. Subsequently, it discerns fake images by analyzing the frequency domain of these noise patterns. Additionally, LNP employs the ResNet-50 model as the classifier backbone. In our study, we utilize the pre-trained LNP detector with default hyperparameters to conduct the visual veracity distortion detection task.

##### HAMMER.

HAMMER [] is a multimodal detector designed to identify multimedia manipulation. It is built upon the pre-trained vision-language model, ALBEF [] which comprises two unimodal encoders and a multimodal Aggregator. To accomplish the multimodal manipulation detection task, HAMMER employs hierarchical manipulation reasoning consisting of shallow and deep manipulation reasoning. Shallow manipulation reasoning involves semantic alignment between image and text embeddings, while deep manipulation reasoning performs deep cross-modal fusion for forgery detection. In our paper, we employ the off-the-shelf HAMMER detector with default hyperparameters to detect cross-modal consistency distortion.

##### FakeNewsGPT4.

FakeNewsGPT4 [] is developed based on large vision language models (LVLMs) to detect multimodal fake news. It identifies two types of forgery-specific knowledge: semantic correlation and artifact tract, and augments LVLMs with these two knowledge. Specifically, it extracts semantic correlations using a multi-level cross-modal reasoning module and comprehends unimodal localized details through a dual-branch fine-grained verification model. In our study, we employed the off-the-shelf FakeNewsGPT4 with default hyperparameters to accomplish the cross-modal consistency distortion detection task.

##### Mixed Detection

We combine the three most powerful models on each single-source detection task (i.e., FakingFakeNews for textual veracity distortion, LNP for visual veracity distortion, and HAMMER for cross-modal consistency distortion) to perform mixed detection. Specifically, utilizing our proposed hierarchical decomposition framework, we sequentially assess textual veracity, visual veracity, and cross-modal consistency with three single-source detectors and assign corresponding multi-class labels.

### 7.3 Instruct Prompt for ChatGPT

The construction of MMFakeBench employs the advanced ChatGPT to assist us in generating textual rumors, expanding detailed descriptions, and generating fact-conflicting descriptions. The specific prompts provided in this work are summarized as follows.

##### Instruct Prompts to Ask ChatGPT to Generate Textual Rumors.

Fig. 8 illustrates the prompt utilized for asking ChatGPT to generate textual rumors with different prompt methods. These methods include arbitrary generation, rewriting generation, and information manipulation.

Figure: Figure 8: Prompt used to ask ChatGPT to generate textual rumors with arbitrary generation, rewriting generation, and information manipulation methods.
Refer to caption: x8.png

Figure: Figure 9: Prompt used to ask ChatGPT to expand detailed descriptions of rumors.
Refer to caption: x9.png

Figure: Figure 10: Prompt used to ask ChatGPT to generate fact-conflicting descriptions.
Refer to caption: x10.png

##### Instruct Prompts to ask ChatGPT to Expand Detailed Descriptions.

Fig. 9 illustrates the prompt utilized for asking ChatGPT to expand detailed descriptions for textual rumors. Based on the responses, we prompt the stage-of-art diffusion generators to generate realistic and relevant supporting images.

##### Instruct Prompts to ask ChatGPT to Generate Fact-conflicting Descriptions.

Fig. 10 illustrates the prompt utilized for asking ChatGPT to generate fact-conflicting descriptions. Then, we combine these descriptions with original captions as prompts in the Midjourney V6 model [] to create corresponding images with additional fact-conflicting information.

Figure: Figure 11: An error example of the textual veracity distortion with the comparison between LLaVA-1.6-34B and GPT-4V.
Refer to caption: x11.png

### 7.4 Example Analysis

In this section, we conduct a comparative analysis of the leading open-source model LaVA-1.6-34B and the proprietary GPT-4V performance across different mixed-source multimodal misinformation examples. This analysis is accompanied by an examination of particular error instances to explore underlying causes.

Figure: Figure 12: An error example of the visual veracity distortion with the comparison between LLaVA-1.6-34B and GPT-4V.
Refer to caption: x12.png

Figure: Figure 13: An error example of the cross-modal consistency distortion with the comparison between LLaVA-1.6-34B and GPT-4V.) .
Refer to caption: x13.png

##### Analysis of Textual Veracity Distortion.

Fig. 11 exemplifies a typical example pertaining to the textual veracity distortion within the mixed-source multimodal misinformation. It underscores the discernible proficiency of different models in leveraging external knowledge. Specifically, in Fig. 11, GPT-4V effectively utilizes external knowledge, as denoted by the assertion “John Burke is an American actor and filmmaker” to deduce the inaccuracy of the claim regarding “John Burke” being an astronaut. In contrast, LLaVA-1.6-34B exhibits an incapacity to achieve comparable inference in this regard. This reflects the GPT-4V’s superiority in integrating external knowledge and multimodal understanding.

##### Analysis of Visual Veracity Distortion.

Fig. 11 exemplifies a typical example pertaining to the visual veracity distortion within the mixed-source multimodal misinformation. We find that LLaVA-1.6-34B exhibits superior acumen in visual perception. As shown in figure 12, LLaVA-1.6-34B successfully discerns the substitution of a person’s head with that of a cat. Conversely, GPT-4V offers a rudimentary depiction of the image content, lacking the nuanced perception demonstrated by LLaVA-1.6-34B. This discrepancy underscores the subtle perceptual capabilities in addressing the visual veracity distortion challenge.

##### Analysis of Cross-modal Consistency Distortion.

Fig. 11 exemplifies a typical example pertaining to the cross-modal consistency distortion within the mixed-source multimodal misinformation. It reveals that GPT-4V exhibits robust cross-modal comprehension and reasoning abilities. Specifically, GPT-4V can not only extract semantic elements such as a “battlebus” from the textual context and a “van” from the paired image but also discern the likelihood of the image depicting a protest rather than “Nick Clegg at work” as indicated in the text. In contrast, LLaVA-1.6-34B solely provides descriptions of the content present in the image and text, without showcasing adequate reasoning capabilities. This discrepancy emphasizes the importance of advanced reasoning capability in addressing cross-modal consistency distortion challenges.

### 7.5 Broader Impact

This paper contains examples of harmful texts or images, raising concerns about the manipulation of public opinion and the potential defamation of individuals. To reduce its social impact, we avoid creating new high-risk images, especially for sensitive topics like politics and gossip. Additionally, addressing these concerns necessitates the establishment of robust regulatory frameworks and ethical guidelines to ensure the technology is used responsibly and for the benefit of society. Therefore, we are committed to incorporating safeguards when releasing our benchmark. Specifically, we will require users to adhere to usage guidelines.

Despite the challenges, the impact of our MMFakeBench is profound. This study is expected to stimulate future research focusing on addressing more realistic instances of mixed-source multimodal misinformation. Additionally, it endeavors to furnish a comprehensive and impartial evaluation platform for the diverse array of misinformation detection methodologies. Thus, MMFakeBench can foster advancements in the field and contribute to the development of more robust and effective techniques for combating the proliferation of mixed-source multimodal misinformation in the real world.

### 7.6 Limitations

While our MMFakeBench marks a critical advancement in mixed-source multimodal misinformation detection, it is important to recognize certain limitations. Our proposed framework utilizes external knowledge retrieved from the Wikipedia API. While the integration of such external knowledge has enhanced the performance of our baseline models, it may not always provide useful information for particularly challenging natural rumors and GPT-generated rumors. Future research should explore the use of a more advanced retrieval augmentation generation (RAG), which could lead to further performance improvements.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: * Can the authors possibly give further examples of evaluation type datasets that look at evaluation in a zero-shot setting for misinformation detection.
soundness: 3
strengths: * Utilizing a human evaluation helped strengthen the claim that currently LVLMs models have room for improvement when it comes to tackling the tasks proposed with the MMFakeBench dataset.
* Table 2 was a well thought out table that coverages a wide range of LVLMs
* The proposed dataset is unique as it proposes different tasks related to mixed-source multimodal misinformation detection, as it shows in Table 1 the difference between its dataset and previous multimodal misinformation datasets.
summary: This paper looks at creating a multimodal misinformation dataset called MMFakeBench that can be used to evaluate detection methods and Large-Vision Models (LVLMs) under a zero-shot setting. For the dataset a validation set is created for hyperparameter selection and also used to conduct a human evaluation of the proposed MMFakeBench dataset. Utilizing the MMFakeBench dataset, 15 LVLMs are evaluated on the test set of the dataset and highlights the shortcomings of these LVLMs when compared to the human evaluation results.
weaknesses: * Currently the dataset size seems somewhat low when comparing it to other datasets mentioned in the paper. However I do recognize that this is specifically meant to be an evaluation dataset.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is the precise scope of this work? For example, does deepfake detection fall within its scope? If so, how does the work address multi-modal detection challenges, such as face reenactment or face swapping?

2. How do Vision-Language Models (VLMs), which incorporate both language and vision modules, perform within the proposed benchmark?

More questions can be seen in the "Weakness" part.
soundness: 3
strengths: 1. It is important to create such a benchmark and baseline for the detection of multi-source fakes.

2. The proposed three types based on the sources of falsified content are new to the field and technical sound.

3. The designed MMD-Agent is well-designed and novel.
summary: This work proposes the first benchmark in mixed-source multimodal misinformation detection (MMD). The authors provide comprehensive evaluations and analysis under the proposed evaluation settings, providing a new baseline for future research.
weaknesses: 1. Lack of comparison with related detection methods: This work includes two detectors within the visual veracity distortion (VDD) field, both relying solely on visual modules for detection. Why not include comparisons with recent methods that integrate both visual and language modules, which are more directly relevant to this research?

2. Insufficient comparison with deepfake detection methods: If "misinformation" encompasses any fake content, it would be beneficial for the authors to compare their method with established deepfake detectors. Given the high profile and potential danger of deepfakes, adding such comparisons would strengthen the paper’s impact and comprehensiveness.

3. Lack of intuitive visualizations for the proposed method: It is recommended that the authors use visualization tools like GradCAM to provide clearer, more intuitive insights into their detection results. Additionally, the current t-SNE visualizations could be improved to more clearly convey their underlying meaning.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. For detecting outputs from generative models, a CNN often achieves better detection performance than an LVLM. Consider including performance results using foundation models paired with a fine-tuned network.

2. Some details in Table 1 are unclear. For instance, why do MEIR and DGM4, which involve text editing, lack Textual Veracity Distortion? Edited texts are essentially a type of rumour.

Based on the clarification regarding the proposed dataset as an evaluation resource and the additional explanations provided, I believe you have addressed most of my concerns and I have decided to raise my rating to 6. However, as MMFakeBench is mix-sourced, compared to NewsCLIPpings, which focuses on OOC detection, more data is needed to achieve sufficient diversity and ensure the validity of experimental results. I look forward to your future work.
soundness: 2
strengths: The paper offers relatively comprehensive coverage of misinformation types, utilizes a diverse set of detection methods, and includes clear figures explaining the dataset and workflow.
summary: The authors introduce MMFakeBench, presented as a benchmark for evaluating mixed-source multimodal misinformation detection (MMD). They develop a multi-class evaluation metric and conduct evaluations of six state-of-the-art detection methods and 15 large vision-language models on MMFakeBench. Additionally, the authors propose an LVLM-based framework, MMD-Agent, which is intended to enhance detection performance in mixed-source MMD tasks.
weaknesses: 1. The contributions of this work are relative limited. Compared to prior benchmarks like DGM4, the primary difference lies in the use of large models to generate data, without introducing novel, highly deceptive misinformation generation methods. Additionally, the proposed MMD-Agent framework contributes minimally to the field, relying heavily on the use of LVLMs.

2. The dataset size is relatively small. While recent benchmarks like DGM4 contain over 200k samples, MMFakeBench includes only about 10k. This raises concerns about whether each misinformation type is represented with enough diversity to support model generalization.

3. It is unclear whether the misinformation types included are meaningful. For example, the Cross-modal Consistency Distortion example in Figure 2 — “An old man reading a book on a park bench” versus “An old man reading a newspaper on a park bench” — lacks practical relevance. Emphasis should be placed on edits involving named entities, such as celebrities, to ensure sufficient impact and deception. Similarly, the Visual Veracity Distortion example with “Veracious Text & AI-generated Image” lacks sufficient misleading potential; the reader’s first impression might be that the image is obviously edited, which fails to show it can lead viewers into perceiving falsified content as authentic.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1.	What’s the distinction between Natural Rumor and Artificial Rumor? 
2.	Do the definitions of the three critical sources apply to other publicly available datasets?
3.	The generation style of the visual veracity distortion data is relatively uniform. Does this meet the benchmark's intended purpose?
4.	Whether the accuracy of all GPT-generated rumors has been manually verified is necessary.
5.	Has the impact of dataset imbalance (between real and fake, as well as imbalances among different subclasses) been validated? This makes it difficult to reuse the dataset.
6.	The authors should provide further analysis of the experimental results in Figure 5, comparing their performance differences to better illustrate the distinctions between different categories(Natural Rumor, Artificial Rumor and GPT-generated Rumor).
7.	I believe that mixing human-generated and LLM-generated misinformation is very meaningful; however, other datasets, such as Twitter and GossipCop, also contain three critical categories. How are the limitations of the different datasets in Table 1 assessed?
soundness: 3
strengths: 1.	From a mixed-source perspective, this paper proposes a new benchmark for multimodal misinformation detection, addressing some weaknesses in existing datasets.
2.	The authors proposed a novel framework named MMd-Agent, which achieved impressive performance.
3.	The paper is written in an easily understandable manner, and the experimental evaluations are thorough.
summary: This paper proposed a new benchmark of multimodal misinformation detection, and the authors constructed a multimodal misinformation data generation pipeline by integrating characteristics from previous multimodal misinformation benchmarks. Additionally, the authors proposed a novel framework named MMd-Agent, which decomposes the multimodal misinformation detection task into three sub-tasks based on the three critical sources (textual veracity distortion,visual veracity distortion, and cross-modal consistency distortion) identified in this paper and processes them in a step-by-step manner, this method finally achieved good results.
weaknesses: 1.	This paper did not present the existing high-impact multimodal misinformation detection benchmarks or compare them to the proposed benchmark concerning the three critical sources.. Could the authors list some relevant multimodal misinformation benchmarks, multimodal misinformation benchmarks detection methods, and SOTA？
2.	This paper did not present a summary of mainstream methods for detecting multimodal misinformation and summarize the classifications of these methods？
3.	The experimental analysis lacks depth; for example, it only addresses some performance limitations of LLaVA-1.6-34B and GPT (as mentioned in the Error Analysis), while other models are evaluated solely on their performance. I hope the authors can conduct a more thorough analysis of the experimental section and consider analyzing the shortcomings of larger models as well.
4.	The novelty of the proposed MMd-Agent is somewhat limited.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. External knowledge sources: Have you considered other external knowledge bases besides Wikipedia to enhance the detection of complex rumors?

2. Model tuning: Have you attempted to fine tune LVLMs on a subset of MMFakeBench to observe their performance improvement?

3. Practical application: How does MMD Agent perform in real-time detection scenarios? What is its computational requirement?

4. Scalability: Can your method be extended to other modalities, such as audio or video, to address more complex types of false information?
soundness: 3
strengths: 1. Originality: The introduction of MMFakeBench addresses a previously unexplored field of multimodal false information detection, making it more representative of real-world scenarios.

2. Quality: The benchmark is carefully constructed, covering 12 types of forgery under three main categories, providing a robust evaluation platform.

3. Clarity: The structure of the paper is reasonable, and the categories included in the benchmark, the evaluation process, and the explanation of the proposed framework are clear and concise.

4. Significance: By revealing the limitations of current detection methods and proposing new frameworks, the paper has the potential to drive future research and improvement in this field.
summary: This paper proposed MMFakeBench, a benchmark for multimodal disinformation detection from mixed sources. Unlike existing benchmarks focusing on single-source forgery, MMFakeBench approximates the real-world by encompassing three key sources: textual truthfulness distortions, visual truthfulness distortions, and cross-modal consistency distortions. complex disinformation. The benchmark contains 12 sub-categories of forgery types, totaling 3300 samples. The authors evaluate 6 current state-of-the-art detection methods and 15 LVLMs in a zero-shot setting, revealing the challenges faced by current methods in this mixed-source environment. To improve the detection accuracy, they propose MMD-Agent, which is based on the framework of LVLMs and divides the detection process into 2 phases, i.e., Hierarchical decomposition and Integration of internal and external knowledge.
weaknesses: 1. Limited exploration of external knowledge sources: The paper acknowledges that reliance on sources such as Wikipedia is a limitation but can further explore its impact on detection performance and possible solutions.

2. Evaluation depth: Although the paper evaluated multiple models, it mainly focused on the zero-shot setting. If fine-tuning models or more ablation experiments are added, the experimental results may be more convincing.

3. The handling of rumors generated by GPT: The paper points out the challenges of detecting rumors generated by GPT, but further analysis or solutions can be proposed. There is no analysis of why GPT-generated Rumor is closer to Natural Rumor, or in other words, why is GPT-generated Rumor about as difficult to detect as Natural Rumor? After all, Artificial Rumor is also written by humans, so it should be about the same difficulty as Natural Rumor, but the experimental result is that Natural Rumor is the easiest to detect.

4. It is suggested to discuss and compare more related works such as [1] in this paper.

[1] Detecting and Grounding Multi-Modal Media Manipulation and Beyond. TPAMI 2024.

Meta-review: This work introduces a mixed-source multimodal misinformation detection dataset (MMD). The authors provide comprehensive evaluations on the proposed evaluation settings, and introduce a new baseline for this dataset. Most of the reviewers have agreed that the authors have successfully addressed their concerns and raised their initial ratings. Overall, the comments of the reviewers are positive, and based on the positive comments and the increased ratings after rebuttal, AC recommends to accept this work.

Decision: Accept (Poster)