MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs
Authors: Xuannan Liu
Zekun Li
Pei Pei Li
Huaibo Huang
Shuhan Xia
Xing Cui
Linzhi Huang
Weihong Deng
Zhaofeng He
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal misinformation detection
large vision language models
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Current multimodal misinformation detection (MMD) methods often assume a single source and type of forgery for each sample, which is insufficient for real-world scenarios where multiple forgery sources coexist. The lack of a benchmark for mixed-source misinformation has hindered progress in this field. To address this, we introduce MMFakeBench, the first comprehensive benchmark for mixed-source MMD. MMFakeBench includes 3 critical sources: textual veracity distortion, visual veracity distortion, and cross-modal consistency distortion, along with 12 sub-categories of misinformation forgery types. We further conduct an extensive evaluation of 6 prevalent detection methods and 15 Large Vision-Language Models (LVLMs) on MMFakeBench under a zero-shot setting. The results indicate that current methods struggle under this challenging and realistic mixed-source MMD setting. Additionally, we propose MMD-Agent, a novel approach to integrate the reasoning, action, and tool-use capabilities of LVLM agents, significantly enhancing accuracy and generalization. We believe this study will catalyze future research into more realistic mixed-source multimodal misinformation and provide a fair evaluation of misinformation detection methods.

Published as a conference paper at ICLR 2025
MMFAAEBENCH: A MIXED-SOURCE MULTIMODAL MISINFORMATION DETECTION BENCHMARK FOR LVLMS
Xuannan Liu' Zekun Li
Peipei Li
Huaībo Huang
Shuhan Xia*
Xing Cuil Linzhi Huang!
Weihong Deng'
Zhaofeng He
Universitty of California, Santa Barbara *Center for Research on Intelligent Perception and Computing, NLPR, CASIA
Beijjig University of Posts and Telccommunicatoons
{liuxuannan, lipeipel peil @bupt edu on zekunli@cs ucsb edu
githunggthhbb 10 MMF htth:///aaaBeeecc..gtto......
Single-Source Misinformation Detection
Real Text
Real Image
Consistent Pair
Fake Text
Fake Image
Inconsistent Pair
Textual Veracity Distortion
Visual Veracity Distortion
Cross-modal Consistency Distortion
Instruct Pix2Pix
CLIP Retrieval
DALLE
stability ai
Ps
Mixed-Source Multimodal Misinformation Detection
Pfizer CEO Secretly Admits COVID-19 Vaccine Causes
Amarsanaa Batsuren who maved from London work at the opencast mine Tavan Tolgoi
Obama's meeting with Putin focus Ukraine.
Permanent Brain Damage In Vaccine Recipients.
Figure 1: Top: Previous methods often assume single misinformation source and conduct single- source detection. Bottom: We collaborate generative models and AI tools to build mixed-source multimodal misinformation benchmark and achieve mixed-source detection.
ABSTRACT
Current multimodal misinformation detection (MMD) methods often assume single source and type of forgery for each sample, which is insufficient for real- world scenarios where multiple forgery sources coexist. The lack of a benchmark for mixed-source misinformation has hindered progress in this field. To address this, introduce MMFakeBench, the first comprehensive benchmark for mixed-source MMD. MMFakeBench includes 3 critical sources: textual veracity distortion, visual veracity distortion, and cross-modal consistency distortion, along with 12 sub-categories of misinformation forgery types. We further conduct an extensive evaluation of 6 prevalent detection methods and Large LiaguaeLa Models (LVLMs) on MMFakeBench under a zero-shot setting. The results indicate that current methods struggle under this challenging and realistic mixed-source MMD setting. Additionally, we propose MMD-Agent, the reasoning, action, and tool-use capabilities LVLM agents, significantly enhancing accuracy and generalization. We believe this study will catalyze future research into more realistic mixed-source multimodal misinformation and provide fair evaluation of misinformation detection methods.
novel approach to integrate
"Corresooniigg Author

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Recent advances in generative models for texts (Brown et al. 2020 Touvron et al. 2023) and images (Dhariwal Nichol 2021 Cui et al. 2024b a) have significantly lowered the barrier to producing diverse multimodal misinformation, posing threats to politics, finance, and public health. For instance. the misinformation "COVID-19 vaccine causes brain damage", shown in Fig. [ accompanied by highly convincing image, can lead to public distrust in medical treatments and vaccine refusal. Therefore, identifying multimodal misinformation on social media is urgent.
Most current multimodal misinformation detection (MMD) methods (Abdelnabi al. 2022 Qi et al. 2024 Ying et al. 2023; Huang et al. 2023; Zhang & Gao 2023 Lee et al. 2021) typically assume
that each sample has a single, known forgery source. As depicted Fig. Top, these forgery sources involve either textual veracity with fake news text, visual veracity with fake images, or inconsistency between the text and image. However, the single-source assumption is overly simplistic and fails to capture the complexity of real-world scenarios, where misinformation often stems from multiple, random sources. To address this mixed-source MMD problem, two key challenges need to be solved. First, existing datasets primarily consist of single-source misinformation, lacking misinformation from multiple sources. This limitation prevents comprehensive evaluation of MMD methods. Second, there is a lack of general detectors capable of handling mixed-source misinformation. Hence, we present MMFakeBench, encompassing the mixed-source MMD benchmark, evaluations, and framework.
Benchmark: We introduce MMFakeBench, the first comprehensive benchmark for evaluating mixed-source MMD. As shown in Fig. Bottom, leveraging advanced AI tools, such as diffusion generators and ChatGPT, MMFakeBench provides 12 forgery types with 11,000 data pairs from three primary sources: textual veracity distortion, visual veracity distortion, and cross-modal consistency distortion. Textual veracity distortion encompasses three types of rumors: natural, artificial, and GPT- generated rumors. Unlike (Thore et al. 2018; Shu et al. 2020; Hanseloonss et al.] 2019: Chen & Shu 2024) that focus solely single-source, text-only rumors, MMFaaeBBench incorporates text-image rumors using highly relevant real or Al Al-generated images. Visual veracity distortion filters existing PS-edited images (Da et al. [021 Nakamura et al. 2020) according to misinformation standards and incorporates high-quality Al-generated images. Cross-modal consistency distortion integrates inconsistencies from both edited and repurposed perspectives into five distinct sub-categories.
Evaluations: To access the current advancements in mixed-source MMD, we build the fine ine-grained multi-class evaluation metric and conduct comprehensive evaluation of tf-teertt detection methods and 15 large dision-language models (LVI on MMFakeBenchh. Specifically, we evaluate detection methods in single-source setting and assess their combined performance (text, image, and moos inconsistency detectors) in the mixed-source setting. Additionally, we evaluate 15 large laiounlanguuae models (LVLMs), including proprictary models such as GPT-4V (OpenAI 2023). The results indicate that existing detection methods exhibit poor generalization. Although LVLMs show robust generalization capabilities, their overall performance still requires improvement.
Framework: Based on our analysis. propose simple yet effective LVLM-based framework called MMD--gentt which enhances detection performance and serves as a new bascline for future research. MMD--ggent decomposes mixed-source detection into three stages: textual veracity check, visual veracity check, and cross-modal consistency reason. This decomposition ensures methodical and thorough reasoning. At each stage, MMD-Agent instructs LVLMs generate pesti- reasoning traces, integrating model actions for coherent decisions. Additionally, the models interact with external knowledge sources via tools (e.g., Wikipedia) to incorporate supplementary information into their reasoning.
In summary, the main contributions are: (1) We introduce mixed- source multimodal misinformation detection (MMD), a challenging setting for detecting misinformation from diverse and uncertain sources, breaking free from single-source constraints, and advancing practical misinformation detec- tion tasks. (2) We develop MMFakeBench, the first benchmark dataset for evaluating mixed-source MMD. The dataset contains 3 critical categories (textual veracity distortion, visual veracity distortion, and consistency reasoning) and 12 sub-categories of forgery types. (3) Using the newly collected dataset, we benchmark mixed-source MMD by evaluating prevalent detection methods and 15 LVLMs. (4) We propose MMD-Agent, simple yet effective LVLM-based framework. It outper- forms previous methods and LVLMs on the MMFakeBench benchmark, highlighting the potential of mixed-source MMD and providing a new baseline for future research.
2

---

Published as a conference paper at ICLR 2025
Table Comparison of misinformation datasets. (k) denotes the number of rumor types.
Textual Verarity Distortion
Visual Veracity Distortion
Cross-oodal Consistency Distortion
Datase
Text (Rumzor)
Supperiin Image
Text
Fact-pon fiiting Image
Emage/Text
Imaga/Text
Repurposed
Al gl-generated
(Neracity)
PS eddedded
Al Al-generated
Repurposing
Editing
FEVER
[horne
2018
K
K
K
X
Polisfact
Shu et
2020
(1
X
X
x
x
x
x
X
Goesipcup
Shu
200
(1)
X
x
x
x
x
X
X
Hansllowsi
2019
(1)
X
X
x
X
X
X
X
MOCHEG
et
2023
(1)
X
X
X
X
X
X
X
LLMMFake
Chen Shu
2024
(1)
X
X
X
X
X
X
X
EMU
2021
x
X
K
X
X
X
Faledai
Nokomara
x
X
X
X
X
X
Inivoal at 2017
x
K
K
AEKAKA
ALAAKA
X
MFIR
Sthir al. 2018
x
X
X
Phings Loo et
X
X
COSMOS Aneja el
20233
x
X
X
X
DGM4 Staa
X
X
X
X
MMFaaeBBench (Ours)
(3)
2 RELATED WORK
Misinformation Benchmarks. One group of misinformation datasets primarily focuses dis- torting textual veracity. The FEVER (Thone et al., 2018) dataset is constructed by manipulated Wikipedia sentences with manual annotation. Unlike these artificial rumors, other datasets, such as Snopes (Hanselowski et al. 2019). Politifact, Gossipcop (Shu et al. 2020), and MOCHEG (Yao et al. 2023), collect natural rumors from fact-checking websites. Recently, the LLMFaaee (Chen & Shu
2024 instructs large language models (LLMs) to generate diverse misinformation. Apart from mis-
leading text, the EMU (Da al., 202T) and Fakeddit (Nakamura et al. 2020 collect Photoshop-edee -ditte images from the Reddit platform. Another group of misinformation datasets focuses on disrupting cross-modal consistency. The MAIM (Jaiswal et al. 2017) and MEIR (Sabir et al. 2018) datasets employ caption replacement and entity swapping, respectively. The NewsCLIPpings (Luo et al.) 202I) and COSMOS (Aneja et al. 2023) datasets link out-of-context images to support certain narra- tives. The recent dataset DGM (Shao et al. 2023) introduces global and local manipulation to alter semantics and sentiment. Different from these works containing only single-source misinformation. we propose the first benchmark dataset for evaluating mixed- MMD, involving textual veracity distortion, visual veracity distortion, and cross cross-modee modal consistency distortion, shown in Table []
Misinformation Detection. Current misinformation detection approaches are mainly divided into two categories. The first is to check textural veracity by constructing features based on writing style (Przybyla 2020), sentiment (Ghanem et al.) 2021), user feedback (Min et al. 2022) and pre- trained language models (Huang et 2023 Zhang & Gao (023). The second is to fuse cross-modal features to detect semantic inconsistencies. Previous works focus on devising attention- -based mod- ules (Qian et al. 2021bb Ying et al., 2023; Wu et al. 2021) guided by diverse learning strategies (Chen al. 2023). Recent works Shao al. (2024); Qi et al. (2024); Liu al. (2024c) capitalize the VLMs which benefit from scage-scale pre- wre-trainigg for reasoning context cues. However, these works target single-source problem, and evaluations are conducted in constrained scenarios. Our work is the first to introduce comprehensive benchmark for mixed-source multimodal misinformation detection.
Large Vision-Language Models. Large language models (LLMs) such GPT-3 (Brown et al., 2020) and Vicuna (Chiang al.) (023) have demonstrated remarkable performance on various linguistic tasks. Inspired by LLMs, models like LLaVA (Liu et al. 2023a) and MiniGPT-4 (Zhu et al. 2023) facilitate image-text feature alignment by leveraging visual instruction tuning. More recently. the evolution of LVLMs has driven advancements in creating diverse and high-quality multimodal instruction datasets. Models such as InstructBIIIP (Dai al. (023), mPLUG-Owl (Ye al. 2023 2024). LLaVA-1.5 (Liu et al. 2024a) exemplify these developments. In this paper, we explore the reasoning capabilities (Zheng et al.) 2023) Zhang et al. 2024) of LVLMs address the challenge of mixed-source multimodal misinformation by integrating reasoning, actions, and tool-use capabilities.
3 MMFAKEBENCH BENCHMARK
In MMFakeBench, we focus on multimodal misinformation involving both text and images, catego- rizing it into three distinct types based on the sources of falsified content:
3

---

Published as a conference paper at ICLR 2025
Al-gmneral atod Image (2.5%
Al-generaled Image 5%%
Selena Gomey
Piaerr CBO Secnely
ustin Bieber Speited
Al-genecatedd Image 10%
Admuts COVID-19
Making Dul A Doc.
Vaccine Cruses Pee
Their Cozcert Date
Magic chnsson did not play the Lakers
GPI-grnerated Rumor
Brain
Natural Rumer
Vaccine Rociptents
Regurposed Emage (7.5%
Reperposed Image (5%)
Orders Obama To Pay 5400 Millson Restiuzonn
Biden
Laughing
A Artificial Rumor
Veterans During
Menocial Day
Ceremooy
Veracimus & (3% PS-edited Image
Veracious
FAKE
Real Data
(30%)
A Algeneccled Image
Textual Veracity Distortion
(30%
Repaliicans run of relicton but look whatwa do.
Soonie boats daily travel the Thames
Visual Veracity Distoction
(10%
England
Cross-modal
(30%
MMFaleBench
Consistency Distortion
Quexy for Person (5%
Text Editing 10%
Secretrry Staee
treated at New York
Hillary Clinton is
Query for Scenn (.%% Lel we Torgei lhe
Ales Salmond
critldised the conference's
Freshy lerias
Reparposed Inconsitency
F French firt world
Edited Incousistency
Hooiital Sunday
cemetery airotiiratme
Query for Semantic (3%
Image Editing (5%
France.
Biden talks with San Joe Dommlly
An old man mading book park bench.
Figure 2: Statistics of the MMFakeBench Benchmark.
Textual Veracity Distortion. It incorporates text-based rumors paired with supporting images to mimic real-world multimodal scenarios. image that visually supports misleading text can make the misinformation appear more credible and persuasive to the users.
Visual Veracity Distortion. Images in this category contain fact-conflicting misinformation through altered or fabricated elements, while the texts remain veracious. These visual manipulations often lead people to perceive falsified content as authentic, distorting their understanding of the information.
Cross-modal Consistency Distortion. Even when the text and image are individually correct, their combination can generate potential misinterpretation if presented in manner that introduces incorrect associations semantic discrepancies between the two modalities, thereby misleading people.
3.1
THREE MISINFORMATION SOURCES
3.1.1
TEXTUAL VERACITY DISTORTION
Textual veracity distortion is critical misinformation category. Our dataset in this category comprises 3,300 samples. Previous works (Thorne et al.) 20188) Shu et al. 2020 Hanselowski et al., 2019 Chen & Shu 2024) focus on single-source and single-modal textual rumors. However, the types of real-world rumors are diverse, and those accompanied by images
have significantly greater impact. To address this, MMFakeBench introduces broader range of rumor types and augments them with highly relevant supporting images to enhance perceived credibility.
As shown in Table I unlike previous methods that consider only one type of textual rumor, we consider three types: (1) Natural Rumor. We select natural rumors from Politifact and Gossipcop (Shu et al., 2020). which provide political news and entertainment stories derived from fact- checking websites. (2) Artificial Rumor. We collect artificial rumors from the FEVER dataset (Thome et al. 2018), which is curated by manually modifying Wikipedia sentences. (3) GPP-generated Rumor. instruct ChatGPT (gpt-3.. 5 50.-ttuubbb rbo) to produce rumors via three prompt approaches (Chen & Shu 2024): arbitrary generation, rewriting generation, and information manipulation. Arbitrary generation is utilized to generate misinformation in specific domains. Rewriting generation addresses the concise and synthetic traces of artificial rumors, while information manipulation involves altering factual information in real claims from Politifact and Gossipcop.
Textual Rumor.
We either Al- Al-generated images carefully selected real images to support the content presented in the rumor text. (1) Al-generated Image: For artificial rumors and their derived GPT-generated rumors, well some less harmful gossip, utilize generative models to
Supporting Image.
4

---

Published as a conference paper at ICLR 2025
Prompt: Imagine the scene in which the rumor occurred and provide details
Textual Rumor
Detail Description
Tool box Stability ai
Pfizer Secretly Admits
u. Pfizer's CBO standing at padium.... large screen displays an image brain...a vaccine
COVID-19 Vaccine Causes
Per-manent Brain Damage
In Vaccine Recipients.
syringe placed in front of him...
DALLE
(a) Textual Veracity Distortion Supported by Al-Generated Images
Prompt: Write sentence about objects events that not cxist
Real Claim
Fact-conflleting Description The Queen's face appears in the Thames
Sconic boats daily travel the Thames in England.
(b) Visual Veracity Distortion by Al-Generated Images
Original Pair (tex( img..r)
Text Editing (text Lmng
An old man
Word disparity
book
(imgan)
An old man reacling a newspaper park bench.
reading a book on park bench.
(textyr1)
Instract Pix2Pix
Rank CLIP
Semilarity
Reusembble
Edited Pair (texteati (mg
Image Editing (texture gddl
An old
reading a
Newspaper
(text,auc
(imgesti)
An old reading a book park bench.
Word disparity
newspaper park bench.
(c) Cross-modal Consistency Distortion by Test/Image Editing
Figure 3: Illustrations of using collaborative generative models and Al tools to generate different sources of misinformation.
create supporting images. We utilize three advanced models: Stable Diffusion XL. (Rombach et al. (022), DALL-E3 (Ramesh et al., (022), and Midjourney V6 (Midjoumeyy (022) enhance the diversity of the generated images. For each rumor, we randomly select a generative model to produce a corresponding supporting image. As many rumors are highly abstract and lack concrete descriptions of objects and scenes, directly using these texts as conditions often yields images that are neither realistic nor relevant. To address this, as shown in Fig. B (a), we instruct ChatGPT to enrich the rumors with more detailed descriptions. These enriched contexts are then used as input for generative models, ensuring alignment with the textual rumors. (2) Repurposed Image: To avoid creating new high-risk images, especially for sensitive topics like politics and gossip, we use repurposed images from the VisualNews (Liu et al. 202I) dataset, which contains numerous image-text pairs from real-world news sources. We select images with high semantic relevance to the textual rumors based on text-image CLIP similarity and text-text CLIP similarity. Images with the highest similarity scores chosen supporting images.
3.1.2 VISUAL. VERACITY DISTORTION
The visual veracity distortion dataset comprises 1,100 samples where the text is real and the misinfor- mation exists in the image. Previous datasets (Da et al.) [202]] Nakamura al. 2020) focus solely on PS-edited (Photoshop-edited) images, containing both misleading and non-misleadimg content. In this study, we manually select the misleading ones and include them in MMFake Besides, we incorporate Al Al-generated images with veracity distortion, which is increasingly harmful due to advancements in diffusion generators.
PS-Edited Image. The PS-edited images are derived from the "manipulated content" samples in the Fakeddit dataset (Nakamura et al. 2020), which designed for multimodal fake news detection. These samples originate from the "Photoshop battles comments" on Reddit. PS-edited samples in the Fakeddit typically exhibit either aesthetic modifications or fact-confiicing manipulations. Since aesthetic modifications do not compromise the factuality of the visual content, they are excluded from misinformation standards. Consequently, ten of the volunteers participate in selecting 550 PS-edited images containing fact-conflicting content from the 7,693 samples of the "manipulated content" set.
AI Al-generated Image. We propose an automated pipeline that generates fact-conflicting descriptions from text captions and then creates high-quality images. Specifically, we first collect image-text pairs from the MS-COCO (Lin et al.] 2014) and VisualNews datasets (Liu et al.) (021). Based on the original text captions, use ChatGPT to generate corresponding fact-conflictin descriptions, which are manually verified, as depicted in Fig. (b). For example, from the caption "Scenic boats daily travel the Thames in England", we generate the description "The Queen's face appears in
5

---

Published as a conference paper at ICLR 2025
the Thames". These descriptions, combined with the original captions, are used prompts in the Midjourney V6 model [Midjourney] (002)) to create corresponding images. The resulting text-image pairs contain original factual text and generated images with additional fact-conflicting information.
3.1.3 CROSS-MODAL CONSISTENCY DISTORTION
In cross-modal consistency distortion, both the text and image with veracity, but either the text or image seplaced/manipulatee to disrupt their overall consistency. Previous datasets (Sabir al.) 2018: Luo et al. 2021 Shao et al. 2023) focus on inconsistencies from single source, either edit-based or repurposed based. In contrast, our MMFakeBench integrates inconsistencies from both edited and repurposed perspectives into five distinct sub-categories, a total of 3,300 image-text pairs.
Repurposed Inconsistency. Our dataset contains three types of repurposed inconsistency, curated directly from the NewsCLIPings (Luo et al.) (02T) dataset: semantic query, person query, and scene query. (1) Semantic query retrieves repurposed images based on specific semantic content. (2) Person query ensures the individual mentioned in the caption appears in the mismatched image. (3) Scene Query relies on spatial similarity to retrieve comparable scene information from repurposed images.
Edited Inconsistency. Our dataset contains two types of edited inconsistency: text editing and image editing. For text editing, we select samples from the DGM* (Shao et al. 2023) dataset, which modifies sentiment words with their antonyms. Notably, some samples in DGM contain fact-conflicting content post-editing. To avoid redundancy with textual veracity distortion, we filter these samples. For remaining text editing and all image editing inconsistencies, we build upon the COCO-Counterfactuaal (Le et al. (223) dataset. This dataset encompasses original image-text pairs (textori, imgori) and edited image-text pairs (fertedit imgediit which are obtained via Instruct- Pix2Pix model (Brooks et al. 2023). As illustrated in Fig. (c), we separately extract word disparities between tezters and textedit and select samples with significant semantic differences using CLIP similarity. Then, we reassemble the two pairs and obtain (lezzladit imgori) text-edited consistency distortion samples and (lexfori, imggeds) as image-edited consistency distortion samples.
3.2 REAL DATA COLLECTION
In addition to the misinformation data, we collect 3,300 real data pairs, ensuring both textual and visual veracity and exhibiting strong image-text consistency. Given that our synthetic data is derived from multiple datasets, construct the real dataset from the same corresponding sources, including MS-COCO, VisualNews, and real image-text pairs from Fakeddit. We further divide VisualNews into four distinet news sources: The Guardian, BBC, USA TODAY, and The Washington Post. Finally, we build the real dataset by equally selecting from six distinct sources.
3.3 MMFAKEBENCH ANALYSIS
MMFakeBench consists of 11,000 image-text pairs, which are divided into a validation set and a test set following Yuue et al. 2024). The validation set, comprising 1,000 image-text pairs, is intended for hyperparameter selection, while the test set contains 10,000 pairs. MMFakeBench encompasses one real category and three misinformation categories. Detailed statistics are shown in Fig. 2 MMFakeBench is partitioned into 30% for textual veracity distortion, 10% for visual veracity distortion, 30% for cross- modal consistency distortion, and 30% for real data. The three misinformation categories can be further subdivided into 12 detailed subcategories based on the sources of the text and images in Fig. 2 Such a comprehensive benchmark highlights the challenges of intertwining mixed-souree and multiple-tppss multimodal misinformation in the real world.
4 MMD-AGENT FRAMEWORK
We present a simple yet effective framework, MMD-Agent, which integrates the reasoning, actions, and tool-use capabilities of LVLM agents. As shown in Fig. 4. MMD-Agent involves two main processes: (1) Hierarchical decomposition and (2) Integration of internal and external knowledge.
Specifically, instruct LVLMs M to decompose the task of mixed-source multimodal misinfor- mation detection into three smaller subtasks: textual veracity check, visual veracity check, and cross-modal consistency reasoning. During the intermediary phase, each subtask is addressed

---

Published as a conference paper at ICLR 2025
(d) Misedoouurce Muirinodal Misinfornaiinn
MMM--Agrrt
Dample
Esamole 2
Example
Question: Judge if desestull vendit is disoocted??
Extenel
Action 1i Find the sey enchy
know wedge
Thouglt $ :Mage Jehraee:
Action Search Mage Jchnson
MAY
Scenk
Testual Veracity
Obscrvation Magto Iphreenspent entire cance
Angeles
Asocciaioon
the Lekers
Rogand
white building
Check
Action Analysi if there b-any objectivve faxt that supports
Todtal Veracity
Verscity
Visual
Crosemedal
Consiliency
ce refiutto the none captan!
1hought $ :. des Magic Johnson did play for Labera.
Distotion
Disiretion
Distection
Queetion: Newos Caption XX Make in accemnte
Fintesh RETUTSS)
Magic Jahnoon did not play for
julgezent on fire- grined clasiihariin of the multimodaal mi/riornition
No
Judge
(b) Standard Promping
Visual
image grvs againer
follow
bellow,
Veracity Check
Be deceesttiveeaa
illasion Lass gust lie Souleryssuiin which cheeeiii faci...
The cunjpuationn
lb taee objeciive eraience refezing nows captio?? U you, arowe: FinilgTEXT REFUTES): no cotline sey 2. there ofjectvve evience refuting the news image? If yes, step
Action Drow Uhe corlusio
Phosch (IAACE ROTUITES
Sonte busa daiy sard the Thamae
Fizahkiinge REFUTES
No
Daes nows cug mutch the new inoge? you atswer FARSEORIGINAAL] no onsteer Fase|MISMATCH))
Cross-anddal
Question:
Judge d the croe-medal censllency disaraad!
Consistency
Reason
Action Draw the coteluui
Essmple
(MEМАТСH,,
Faisk PMIMATTC
wulling
Anower for Example Pirich (OREGIAAI]
ceblt baildig
Annwer for Esample fleseh (ORGINAL)
[ORICINAL]
Figure 4: Comparison of standard prompting and proposed MMD-Agent. (a) Three examples of multimodal misinformation from distinct sources. (b) LVLMs with standard prompting methods fail to make correct judgments. (c) MMD-Agen instructs LVLMs to decompose mixed-source detection into smaller subtasks, which are solved by integrating model thoughts and environment observation.
through an interleaved sequence of reasoning and action. The LVLM is guided to reason and induce the needed action solve the task. The actions a, are then executed either by leveraging the model's internal knowledge to generate "Thought" R2 or by interacting with external sources to gather additional information ("Observation")) O,. These action outputs will be integrated into the sequence making de:
to facilitate subsequent decision decision-making
de = M (R.,
The LVLM's internal capabilities and knowledge are utilized both to reason about which actions to take and to perform those actions from various perspectives, such as identifying key textual entities (Thought 1), conducting factual analysis (Thought 2), and applying commonsense reasoning (Thought 3). However, the model may experience hallucinations when relying solely on its internal knowledge. To address this, enable the model to interact with external knowledge bases, such as the Wikipedia API, to retrieve reliable and up-to-date information for fact-checking. This approach ensures the accuracy and relevance of the knowledge used in verification.
EXPERIMENTS
We select 6 state-of-the-art misinformation detection methods and 15 large vision-language models (LVLMs) for preliminary benchmarking using the MMFakeBench dataset, aiming to explore their applicability in the mixed-source MMD setting. Additionally, we evaluate the performance of our proposed framework, MMD-Agent. All evaluations are conducted under a zero-shot setting on our benchmark. All experiments are performed eight NVIDIA GeForce 3090 GPUs with PyTorch.
5.1 EXPERIMENTAL SETTING
Baseline Models. We select LVLMs of varying sizes as baseline models. (i) LVLMs with 7B
parameter including Otter (Li al. 2023a). MiniGPT-4 (Zhu al. 2023). InstructBLIP (Dai et al.)
2023), Qwen- VI. (Bai et al. (023). VILA (Lin et al., 2024). PandaGPT (Su et al. (023). mPLUG- Owl2 (Ye et al. 2024), BL.IP-2 (Li et al. 2023b) and LLAVA- (Liu et al., (024b). (ii) LVLMS
with 13B parameter including VILA, InstructBLIP, BLIP-2, and LLaVA-I (in) LVLMs with 34B parameter including LLa VA-16. (iv) flosed-souree model including GPT-4V (OpenAI (023).
Evaluation Metrics. We evaluate the performance of different baselines using multi-class classifica- tion, categorizing data into four distinct classes: textual veracity distortion, visual veracity distortion, cross-moss cross-modal consistency distortion, and real class. Consistent with (Qian al.) 2021a la Zhang & Gao
2023 Chen et al.) 2023), we adopt the widely-used macro-Fl metric, which balances precision and recall through harmonic mean. Beyond the Fl score, we also include pacro-precision, macro-recall,
7

---

Published as a conference paper at ICLR 2025
Table 2: Overall results (%) different models on the MMFakeBench validation and test set with the comparison of standard prompting (Standard) and proposed MMD-Agent framework.
Model
Language
Prompt
Validation (1000)
Test (10000)
Name
Model
Method
FI
Precision?
Recall
ACC1
FI
Precision
Recaltf
ACCT
Human Evaluation
35.9
38.3
37.9
LVLMs with 7B Parameter
Otter Image
MPT-7B
Standard
5.2
10.5
3.4
4.1
4.9
9.3
3.3
4.0
MiniGPT4
Vicuna 7B
Standard
5.2
5.2
21.2
9.0
5.3
6.9
21.0
9.1
InstructBLIP
Standard
7.1
7.2
Qwen VI
Qwen 7B
Standard
7.5
10.3
24.3
11.0
8.0
35,9
25.5
11.6
VILA
LLaMA2-2 7B
Standard
11.5
7.5
25.0
30.0
11.5
7.5
25.0
30.0
PandaGPT
Vicuna-7B
Standard
11.8
9.8
25.0
30.0
11.6
25,0
30.0
Owi
L1MMA2-BB
Standard
14.5
31.1
BLIP2
FlanTS-XI.
Standard
16.4
20.1
27.5
33.0
16.7
17.3
27.7
33.2
LLaVA-1.6
Vicuna-7B
Standard
17.4
14.8
25.7
30.8
19.0
16.5
26,9
32.3
LVLMs with 13B Parameter
Standard
11.5
7.5
30.0
11.6
32.5
25.0
VILA
LLaMA2-13B
MMD-Agent
22.7
27.3
24.4
28.7
24.0
30.4
25.5
29.4
Standard
13.7
13.2
24.0
28.8
13.9
25.5
24.3
29.1
InstructBLIP
Vicuna-138
MMD-Agent
26.0
33.3
29.5
24.5
32.1
28.8
Standard
16.7
34.9
27.3
32.8
16.3
34.6
27.3
BLIP2
FSXLTSXXLL
MMD-Agent
31.6
39.8
32.2
34.4
28.8
39.0
30.4
32.1
Standard
12.0
22.5
25.0
30.0
14.4
35.7
26.0
31.2
LLaVA-1.6
Vicuna-133 13B
MMD-Ageut
38.0
44.5
40.6
34.5
37.5
LVLMs with 34B Parameter
Nous Hermes-2
Standard
25.7
44.5
33.7
40.4
25.4
44.1
33.8
40.5
LLaVA-1.6
Yi-34B
MMD-Agen
49.9
54.4
48.7
49.6
Proprictary LVLMs
Standard
51.0
66.8
49.7
54.0
48.8
63.0
48,7
54.2
GPT-4V
ChatGPT
MMD-Agent
61.6
67.8
59.3
62.1
61.5
67.7
59.1
61.0
and macro-accuracy complementary evaluation metrics. Specifically, we construct robust regular expressions to extract key phrases from the long responses for accurate answer matching. Follow- ing (Liu et al. 2023b), if a model's 's response lacks valid answer, we classify it pseudo choice "Z" and consider the response incorrect.
5.2 MAIN RESULTS
Comparison of Different LVLMs. We present a comprehensive comparison of different LVLMs using the MMFakeBench, detailed in Table Our key findings summarized follows:
1) Challenges of BeFch: The benchmark poses substantial challenges to current models. Notably, GPT-4V, despite its advancement, achieves Fl-score of only 51.0% with the standard prompting. This indicates considerable room for improvement and highlights the rigorous standards of this benchmark.
2) Disparity between Open-source Models and GPT-4V: Although LLaVA-116- 1.6--4b644bb is the leading open- source model, it achieves an Fl-score just 25.7% with the standard prompting, significantly lower than GPT-4V. This highlights a pronounced disparity in detection capabilities between open-source and proprietary models.
3) Impact of Parameter Quantity: Comparing models within the same series, such LL- Vicuna-7b and LLaVA-1.66344 we observe that models with larger parameter counts exhibit better performance. Smaller LVI.Ms face constraints in citructtoo-followiin and high predicted consistency, as detailed the Appendix I.I.I These results indicate that 7B parameter models lack sufficient multimodal understanding effectively combat misinformation.
4) Effectiveness of MMD-Agent: Due to the limited reasoning capability of small-scale models, we select moderately sized open-source and proprietary models as baselines to compare the proposed MMD-Agent with the standard prompting. MMD-Agent significantly improves the F1-score for both open-source models and GPT-4V. Notably. ALaVA-1.6-344 using MMD-Agent achieves an F1-score of 49.9%, approaching the 51% score of GPT-4V with the standard prompting. This suggests that Men-Agent can
general framework for future research the MeFakeBench benchmark.
8

---

Published as a conference paper at ICLR 2025
Table 3: (a) Comparison with single-source detectors for MMFakeBench. (b) Ablation studies on hierarchical (Hier.) decomposition and reasoning knowledge KG {R,, O of each sub-task t. TVD, VVD, and CCD denote textual veracity distortion, visual veracity distortion, and cross- moddl consistency distortion, respectively. Corpus refers to the general datasets used in LVLMs, not tailored for misinformation detection. & denotes the chosen single-source detector applied for mixed detection.
Existing
Train
Binary Overall
Multiclass Overall:
Hier. Ki. Kg.
Ks
Reall
TVD:
VVD1
CCD1
Overall
Detector
Source
F
TVD
37.8
58.5
588
0.0
38.6
25.7
CNNSpot
VVD
23.8
45.8 ((1277 (11077 52.5 (1325) 490())0.4)
36.0
UnivED
VVD
28.9
46.4 (10.6) 37.6 (121.1) (2.3))03))
(21.))
41.0
LNP
VVD
33.0
FakeNewssPPT4
CCD
41.7
(10.4) (10.0 61.7 (128.9)
(11.4)
HAMMER*
OCD
43.0
51.1 (14.3) 37.6 (10.0)
(1.7
(10.6)
49.9
Mixed Detection
47.6
22.5
LLAVA-L6--1B
Corpus
67.2
19,9
We Wiki Knowlodge
49.7 (11.4) 18.0 (119.))
46.3 (1.0
(12.9)
43,8
GPT-4V
Corpus
74.0
61.6
Google Knowlodge
50.2 (10.9)
33.41)42) (222(105)) (1..))
48.5
Comparison with Single-source Detectors. We compare LLa 1.6-0--1.--440 1.6-34B and GPT-4V utilized in MMD-Agent, with several competitive single-source detectors including TakeingFakeeNern
(Huang al. 2023), CNNSpot (Wang et al. 2020), UnivFD (Ojha al., 20236). LNP (Liu et al. 2022). Fakel CWT4PT4 (Liu et al. 2024c), and HAMMER (Shao et al., 2023). The details of each detec- tor are presented in the Appendix A.3.2. For fair comparison, in addition the single-source misinformation detection via existing detectors, we integrate the three most powerful detectors in distinct sources (i.e., FakeingFakeNews, LNP, and HAMMER) to assess the capability of mixed detection. Mixed detection utilizes our proposed hierarchical framework by replacing LVLMs with relevant detectors. The results in Tab show that -L.VVA- and GPT-4V perform better than single-source detectors for both binary and multiclass classification by a large margin. This demonstrates that LVLMs trained with a large general corpus achieve promising generalization performance in mixed-source MMD and can serve potential baseline models for future study.
Results of Human Evaluation. We conduct a comprehensive user study using a validation set containing 1,000 samples as a question bank. For each questionnaire, 50 samples are randomly selected from this question bank. A total of 80 participants are asked to identify the source of misinformation for each news item. As shown in the first row of Table [ the results reveal that 62.1% % of the items are predicted incorrectly by users, highlighting the dataset's 's high confusion and the realistic challenge it poses. This level of difficulty and confusion emphasizes the quality and challenge embedded in our dataset, making it an invaluable resource for pushing the boundaries of current understanding and capabilities multimodal misinformation detection research.
5.3 EXPERIMENTAL ANALYSIS
Ablation Study Hierarchical Decomposition and Reasoning Knowledge. We first investigate the effects of instructing LVLMs using only hierarchical decomposition compared to standard prompt- ing. In Table 3 (b), the decomposition method performs better for solving multi-task interference. Additionally, we conduct an ablation study by sequentially generating multi-perspecttive knowledge for individual sub-tasks. Results in Table 3 (b) show that augmenting decisions with reasoning knowledge outperforms its ablation part, especially for checking content veracity. We further conduct ablations on external knowledge by removing it or integrating an alternative source (i.e., Google Knowledge Graph). Results show that models using external knowledge outperform those relying solely on internal reasoning, highlighting its critical role in validating textual veracity.
LLWA-6634
InatructoUP-110
Table 4: Performance (F1 score (%) of models on different sources of misinformation.
0.6
0.08
Model
Realt
TVD;
VVDf
CCD:
Overallt
0.4
VILA-13B
32.4
13.4
4.3
37.6
21.9
IIstuuiBLII-133B
41.9
18.8
19.6
23.8
26.0
6.20
39.2
13.1
32.6
31.6
0.13
1LAVA-L6-34B
51.1
37.6
61.7
49.2
49.9
0.0
Natural Rumor
Artificial Rumor
3.02
GPTgenerated Rumor
GPT-4V
67.2
57.3
56.5
61.6
Figure 5: Performance (detection success rate) of models on different types of rumor.

---

Published as a conference paper at ICLR 2025
Analysis of Misinformation Sources and Types. We compare the Fl scores of various LVLMs across misinformation sources in Table Across all sources, the majority of open- open-source models perform worse than GPT-4V by huge margin, particularly in terms of textual veracity distortion. This indicates that open-source models are considerably challenged by textual veracity distortion. Within textual veracity distortion, we further report the detection accuracy of selected models across three types of textual rumors. Fig. 5 that open-source models typically perform better in the artificial rumor. This might be attributed to the fact that artificial rumors are constructed by applying specific rules, which provide identifiable points of falsification enabling verification through external knowledge. In contrast, natural or GPT- gererreeed rumors often leverage ambiguous language that makes detection challenging. Analysis of other misinformation sources is detailed in Appendix A.1.3
X
X
Retrieved Eshemall Knowledge: Kaaru Courlan Bevenan is Canadiam actor... he hus ustrilished limact as leading man in action cinama
LLaVA-16:4B: The wearing
LLaVA-16-4B. The Laage stoves a than and woman pariccppatigg in religions cenertory, likell weeklingg as indicated the attire prssence of prane Anewert Finis[MATCH]
GPTigeserated Rumior Alggnnerted Imsge
suit and tia, and the woman beidal gown. They are holding hands and appear be excharging vows sings The setting suggesis a church: or chape,, with religoous symbols... Answer Finishj AGE Supports]
LLaVA-16-31 Howweer, his personal li any evenis related
disect
information
would Keans lorved personal
look for sources life, speciically any secent events related to marringe...
LL deseibes image, Given image does depict conbent contradistinn obyeciio facts, LLaVA 34B fuils asace the veruity of the image.
Encer that this
Ercor LLAVA-16 accuratcly desetibes consisient the highly Thus, LLaVA- 6-348 provides
Answwer: ETiS Supports]
Chackes Rouver' knowlidge knowllddge
of
Kearnu Heeves Seer
Secnetly Marrying
Wilipedis. 14aVA-34B the
Myshery Werman
"Image-ted manch"
Las Vegga Chapsl Stucking
Textual Veracity Check
Visual Veracity Check
Cros--modal Consistency Reason
Figure 6: One of the most harmful examples involves a GPT-generated rumor supported by an Al Al-generated image, which challenging for LLaVA-1.66 63BB. More examples can be found in the Appendix A.1.7
Error Analysis. A fundamental root cause of textual veracity checking errors in the LLaVA-1.6-33 166-44 is the lack of useful external knowledge. This deficiency is exemplified in Fig. where the knowledge contains only commonsense information but fails to provide relevant events. Moreover, this Al- generated image in the image-text pair exhibits high fidelity and strong coherence, thus evading detection in visual veracity and modal consistency. These instances underscore the dangers of using collaborative generative modes to automatically generate multimodal misinformation.
CONCLUSION
In this paper, we introduce MMFakeBench, the first comprehensive benchmark for detecting mixed- source multimodal misinformation. MMFakeBench contains three primary misinformation categories along with 12 sub-categories of forgery types. We conduct comprehensive evaluations of 6 prevalent detection methods and 15 LVLMs on the MMFakeBench dataset. Furthermore, we propose an innovative unified framework and perform extensive experiments to demonstrate its effectiveness.
ETHICS STATEMENT AND LIMITATIONS
This paper contains examples of harmful texts or images, raising concerns about the manipulation of public safety. To reduce its social impact, we have implemented several safeguards: (1) Content Safeguards. Rigorous review sensitive content in data generation such as those related to politics and race etc. (2) Disable data generation code access. We open-source datasets and detection codes but do not release data generation codes for safety. (3) Data Access Restrictions. Data access is restricted verified researchers by following a binding usage agreement. (4) Public Feedback Mechanism. We will offer a public feedback channel for ethical concerns and continuous dataset improvement. The licenses for the datasets contributed in this work are discussed Appendix A.5
While our MMFakeBench marks critical advancement in mixed-source multimodal misinformation detection, it important to recognize certain limitations. Our proposed framework utilizes external knowledge retrieved from the Wikipedia API. While the integration of such external knowledge has enhanced the performance of our baseline models, it may not always provide useful information for particularly challenging natural rumors and GPT-generater rumors. Future research should explore the use of a more advanced retrieval augmentation generation (RAG), which could lead to further performance improvements.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is sponsored by Beijing Nova Program (Grant No. 20230484488, Z21 1211000001211000)
National Natural Science Foundation of China (Grant No. 62306041, No.62176025, No.
U21B2045). Youth Innovation Promotion Association CAS(Grant No.2022132), Beijing Nova
Program(20230484276,, Beijing Natural Science Foundation (4252054).
REFERENCES
Sahar Abdelnabi, Rakibul Hasan, and Mario Fritz. Open-domain, content-basede multi- fact-checking of out-of-context images via online resources. In CVPR, 2022.
Shivangi Aneja, Chris Bregler, and Matthias Nießner. Cosmos: Catching out-of-context misinforma- tion with supervieee learning. In AAAI, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprini arXiv:2308. 1 /2966, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. CVPR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
Canyu Chen and Kai Shu. Can 11m- -l-egnnertted misinformation be detected? In ICLR, 2024.
Ziwei Chen, Linmei Hu, Weixin Li, Yingxia Shao, and Liqiang Nie. Causal intervention and counterfactual reasoning for multi-modal fake news detection. In ACL, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https://vicuna. Imsys. org, 2023.
Xing Cui, Peipei Li, Zekun Li. Xuannan Liu, Yueying Zou, and Zhaofeng He. Localize, understand, collaborate: semantic-aware dragging via intention reasoner. In NeurIPS, 2024a.
Xing Cui, Zekun Li. Peipei Li, Huaibo Huang, Xuannan Liu, and Zhaofeng He. Instastyle: Inversion noise of stylized image secretly a style adviser. In ECCV, 2024b.
Jeff Da, Maxwell Forbes, Rowan Zellers, Anthony Zheng, Jena D Hwang. Antoine Bosselut, and Yejin Choi. Edited media understanding frames: Reasoning about the intent and implications of visual misinformation. In ACL-IJCNP, 2021.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurlPS, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffuusion models beat gans on image synthesis. In NeurIPS, 2021.
Bilal Ghanem, Simone Paolo Ponzetto, Paolo Rosso, and Francisco Rangel. Fakeflow: Fake news detection by modeling the flow of affective information. In EACL, 2021.
Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. A richly annotated corpus for different tasks in automated fact-checking. In CoNLI, 2019.
Kung-Han Kung-Hsiang Huang, Kathleen McKeown, Preslav Nakov, Yejin Choi, and Heng Ji. Faking fake news for real fake news detection: Propaganda-loade training data generation. In ACI, 2023.
Ayush Jaiswal, Ekraam Sabir, Wael AbdAlmageed, and Premkumar Natarajan. Multimedia semantic integrity assessment using joint embedding of images and text. In ACM MM, 2017.
11

---

Published as a conference paper at ICLR 2025
Tiep Le, Vasudev Lal, and Phillip Howard. Coco-counterfactuall: Automatically constructed counterfactual examples for image-text pairs. In NeurIPS, 2023.
Nayeon Lee, Yejin Bang, Andrea Madotto, Madian Khabsa, and Pascale Fung. Towards few-shot fact-checking via perplexity. In NACCL, 2021.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:335.003766, 2023a.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, 2021.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023b.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Bo Liu, Fan Yang, Xiuli Bi, Bin Xiao, Weisheng Li, and Xinbo Gao. Detecting generated images by real images. In FCCV, 2022.
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. Visual news: Benchmark and challenges in news image captioning. In EMNLP, 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge. http:///lavavll. github. io/blog/2024- 11-00--lava--ext, 2024b.
Xuannan Liu, Peipei Li, Huaibo Huang, Zekun Li, Xing Cui, Jiahao Liang, Lixiong Qin, Weihong Deng, and Zhaofeng He. Fka-owl: Advancing multimodal fake news detection through knowledge- augmented Ivlms. In ACM MM, 2024c.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1907: 2019.
Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In NeurIPS, 2023b.
Grace Luo, Trevor Darrell, and Anna Rohrbach. Newselippings: Automatic generation of out-of- context multimodal media. In EMNLP, 2021.
Midjoumey. http::/wwww.migoorrescc//mmoo/// 2022.
Erxue Min, Yu Rong, Yatao Bian, Tingyang Xu, Peilin Zhao, Junzhou Huang, and Sophia Ananiadou. iivide-and-conquer: Post-user interaction network for fake news detection on social media. In WWW, 2022.
Kai Nakamura, Sharon Levy, and William Yang Wang. r/fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. In LREC, 2020.
Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. In CVPR, 2023a.
12

---

Published as a conference paper at ICLR 2025
Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. In CVPR, 2023b.
OpenAl. Gpt-4v(ision) system card. http::/www..wmmtttcscurrrrr/r/pp 2023.
Piotr Przybyla. Capturing the style of fake news. In AAAI, 2020.
Peng Qi, Zehong Yan, Wynne Hsu, and Mong Li Lee. Sniffer: Multimodal large language model for explainable out-of-context misinformation detection. In CVPR, 2024.
Chen Qian, Fuli Feng. Lijie Wen, Chunping Ma, and Pengjun Xie. Counterfactual inference for text classification debiasing. In ACL, 2021a.
Shengsheng Qian, Jinguang Wang, Jun Hu, Quan Fang, and Changsheng Xu. Hierarchical multi- modal contextual attention network for fake news detection. In SIGIR, 2021b.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Ekraam Sabir, Wael AbdAlmageed, Yue Wu, and Prem Natarajan. Deep multimodal image- repurposing detection. In ACM MM, 2018.
Rui Shao, Tianxing Wu, and Ziwei Liu. Detecting and grounding multi-modal media manipulation. In CVPR, 2023.
Rui Shao, Tianxing Wu, Jianlong Wu, Liqiang Nie, and Ziwei Liu. Detecting and grounding multi-modal media manipulation and beyond. TPAMI, 2024.
Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news social media. Big data, 2020.
Gabriela Ben Melech Stan, Raanan Yehezkel Rohekar, Yaniv Gurwicz, Matthew Lyle Olson, Anahita Bhiwandiwalla, Estelle Aflalo, Chenfei Wu, Nan Duan, Shao- Yen Tseng, and Vasudev Lal. Lvlm-intrepret: An interpretability tool for large vision-language models. arXiv preprint arXiv:2404.031 18, 2024.
Yixuan Su, Tian Lan, Huayang Li. Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16335 2023.
James Thore,, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. In NAACI, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Sheng- Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn- Cnn-generated images are surprisingly easy to spot... for now. In CVPR, 2020.
Yang Wu, Pengwei Zhan, Yunjian Zhang, Liming Wang, and Zhen Xu. Multimodal fusion with co-attention networks for fake detection. In Findings of ACL-IJCNLP, 2021.
Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang. End-to-end multi- modal fact-checking and explanation generation: A challenging dataset and models. In SIGIR, 2023.
Qinghao Ye. Haiyang Xu, Guohai Xu, Jiabo Ye. Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 33Xiv:2304.14111 2023.
13

---

Published as a conference paper at ICLR 2025
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modes large language model with modality collaboration. In CVPR, 2024.
Qichao Ying, Xiaoxiao Hu, Yangming Zhou, Zhenxing Qian, Dan Zeng, and Shiming Ge. Bootstrap- ping multi-view representations for fake news detection. In AAAI, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs. In CVPR, 2024.
Xuan Zhang and Wei Gao. Towards Ilm-based fact verification on news claims with hierarchical step-by-step prompting method. In AACL, 2023.
Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of- thought prompting for multimodal reasoning in language models. NeurlPS, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing languaee understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
This appendix contains additional details for the ICL. 2025 submission, titled "MMha A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs". The appendix is organized as follows:
A.1
Additional Experimental Result.
A.
Instruction Following and Prediction Consistency.
A.1.2
Evaluation Results for Binary Classification.
A.1.3
Analysis of More Misinformation Types.
A. 1.4
Comparision of Existing Datasets.
A.1.3
Efficiency of MMD-Agent.
A. 1.6
Fine-tuning on Existing Detectors.
A.1.7
More Error Analysis.
A.I.8
Interpretable visualization.
A.2
Benchmark Analysis.
Implementation Details.
A.3.1
LVLMs Configuration Details.
A.3.2
Existing Single-source Detectors Details.
A.4
Instruct Prompts for ChatGPT.
A.5
Dataset Licenses.
A.6
More Visualization Examples.
A.1
ADDITIONAL EXPERIMENTAL RESULTS
Table 5: Statistics of Instruction following capabilities and predicted consistency tendency of LVLMs.
Model
Match
Cousist.
Model
Match
Consist.
Model
Match
Consist.
LVLMs with Parameter
Otter-Image
9.8
100
Qwen-VI VI.
88.6
92.9
SPLUG-OWLZ
100
96.6
MiniGPT4
VILA
100
BLIP2
InstructBLIP
25.7
96.11
PandaGPT
100
98.9
LLaVA-L.6
100
76.9
LVLMs with Parameter
VILA
100
100
InstructBLIP
99.9
91.0
BLIP2 FlanT5 XXL
100
49.4
LVLMs
Parameter
6
100
52.6
Proprietary LVLMs
GPT-4V
99.9
36,5
A.1.1 INSTRUCTION FOLLOWING AND PREDICTION CONSISTENCY.
We evaluated the instruction following capabilities and prediction consistency further study the multimodal understanding of LVLMs on mixed-source multimodal misinformation detection (MMD). We report the success rate in heuristic matching (Match) with regular expressions and prediction consistency rate (Consist.). The results are shown in the Table S Among all LVLMs, small-scale models like Otter- Image and InstructBIIP, achieve the lower matching success rate. While there exist small-scale LVLMs that perfectly follow the format of the regular expressions and achieve high success rates (>99%) matching, most small-scale models exhibit high predicted consistency rates. This indicates small-scale models may prefer to predict certain category answer among all given choices. Additionally, the leading open- pourcsource model, LLLVVA-1.63445 6-34B, and the proprietary model GPT-4V demonstrate superior instruction-followiiii capabilities and lower prediction consistency. This indicates their significant potential in addressing mixed-source MMD, positioning them as valuable baseline models.
15

---

Published as a conference paper at ICLR 2025
Table 6: Binary overall results of different models on the MM-FakeBench validation and set with the comparison of standard prompting (Standard) and proposed MMD-Agent framework.
Model
Language
Prompt
Validation (1000)
Test (10000)
Name
Model
Method
F1
Precision
Recall
ACC
F1
Precision
Recall ACC
Haman Evaluation
54.9
56.6
57.8
56.8
LVLMs
7B Parameter
Ottr-Immaee
MPT-7B
Standard
7.9
4.1
4.5
7.9
8.6
32.4
5.0
8.6
MiniGPT4
Vicuna- 7B
Standard
40.4
38.2
45.7
63.1
41.7
41.0
47.4
65.2
InstructBLIP
Vicuna- 7B
Standard
14.7
30.8
13.2
8.1
16.1
40.5
14.2
8.8
Qwen-VL
Qwen-7B
Standard
43.6
50.6
44.9
60.3
44.0
51.6
45.2
60.5
VILA
LLaMA2-7B
Standard
41.2
35.0
50.0
70.0
41.2
35.0
50.0
70.0
PandaGPT
Vicuna-7B
Standard
24.6
60.6
50.5
30.9
24.1
61.7
50.4
30.6
mPLUG-OWl2
LLaMA2-7B
Standard
47.2
64.9
52.3
70.6
48.7
71.1
53.3
71.4
B1.P2
H-
Standard
41.2
35.0
50.0
70.0
41.2
35.0
50.0
70.0
LLaVA-1.6
Vicuna-7H
Standard
48.2
62.5
LVLMs with 13B Parameter
Standard
41.1
35.0
50.0
70.0
41.1
35.0
50.0
70.0
VILA
LLaMA2-13B
MMD-Agent
56.5
62.2
$6.9
70.3
56.6
64.3
57.2
71.2
Standard
41.1
35.0
49.9
69.9
41.1
35.0
49.9
69.8
InstructBLIP
Vicuna-13B
MMD-Agent
51.3
53.4
54.0
53.1
47.9
50.1
50.1
49.9
Standard
31.6
63.4
53.6
35.5
30.6
64.9
53.4
34.9
XXXXL.
MMDn-geent
51.5
54.0
54.7
53.5
Standard
41.1
35.0
50.0
69.7
42.3
57.3
50.1
69.5
LLaVA-1.6
Vicuna- 13B
MMD-Agent
51.8
66.7
54.6
53,9
71.3
LVLMs with 34B Parameter
Nous-Hermees Hermes-2
Standard
62.9
67.1
70.0
63.4
64.3
68.8
71.7
64.8
LLaVA-L6
-Yi-34B
MMD-Agent
67.2
70.4
66.0
75.1
68.1
71.1
67.0
75.6
Proprietary LVLMS
Standard
72.1
72.8
76.9
76.4
GPT-4V
CharGPT
MMD-Agent
74.0
73,4
75,5
76.8
72.8
72.4
75.4
75.0
A.1.2 EVALUATION RESULTS FOR BINARY CLASSIFICATION.
In addition to multi-class classification, we also provide binary classification performance assess the overall detection capability of baseline models in mixed-source MMD. Based the 4 categories in the mixed-source MMD settings, we develop binary evaluation metrics via mapping techniques. Specifically, we standardize the assignment of labels denoting "textual veracity distortion", "visual veracity distortion", and "cross-modal consistency distortion" to the classification of "Fake" while reserving the label "True" to denote real data. Similar the multi-classification evaluation, adopt the widely used Fl score. In addition to the Fl score, we also use precision, recall, and accuracy as supplementary evaluation metrics. The specific evaluation results shown Table From the results, we make the following observations:
Current models including open-source models and GPT-4V are challenged by the MMFakeBench dataset in binary classification detection. Despite being an advanced modal, GPT-4V attains mere F1 -core of 72.3% using the standard prompting on the validation set.
The proposed framework MMD-Agent yields substantial improvement on recent LVLMs, especially on open-source models. For instance, for the BLIP2-Flan XXX model, MMD-Agen achieves 19.9% increase in F1 score on the validation set. This may be credited to the effective integration of reasoning, actions, and tool use in enhancing multimodal understanding in mixed-source MMD.
A.1.3 ANALYSIS OF MORE MISINFORMATION TYPES.
Within visual veracity distortion, report the detection accuracy of selected models across two types of fact-conflicting images. Fig. 7 (a) shows the challenging nature of both types of such fact-conflicting images for existing models. Notably, even the advanced GPT-4V achieves detection success rate of less than 50% on both types. Additionally, we present an analysis of the detection accuracy of selected models across different types of image-text inconsistency. As shown in Fig. (b).
16

---

Published as a conference paper at ICLR 2025
edited inconsistency emerges as more substantial challenge compared to repurposed inconsistency. This finding suggests that editing methods introduce minor alterations to images or text, necessitating enhanced multimodal reasoning capabilities.
CITE
MLAISS
MUPI FLANTLIIAA
VILAA
0.6
0.5
0.3
(a)
Figure 7: (a) Performance (detection success rate) of models different types of fact-conflicting
images. (b) Performance models on different types of inconsistent image-text pairs.
Table 7: Performance (F1 score) comparison of selected models the proposed benchmark and existing benchmarks (datasets).
Model NewsClipings Name (Binary)
NewsClipings (Multiclass)
Fakeddit
Fakeddit (Multiclass)
MMFakeBench (Binary)
MMFakeBench (Multiclass)
(Binary)
Large Vision Language Models
BLIP2-FLanTS-XXI
33.6
40.5
31.6
16.7
LLAVA-1.6-34B
65.6
71.4
62.9
25.7
Multimodal Specialized Detectors
FKA-Owl HAMMER
52.0
55.0
41.7
55.0
52.6
43
A.1.4 COMPARISION OF EXISTING DATASETS
We conduct an experiment to compare the performance of selected models on the proposed benchmark against existing benchmarks (datasets) in Table 7 We employ four models: two powerful open- source L.LM models, LLaVA-1 -16.-34BB and
with two open-source multimodal specialized detectors, HAMMER (Shao et al., 2023) and FKA-Owl (Liu al., (024c). All these models are evaluated on two widely-used multimodal misinformation datasets, NewsClippings (Luo al. 202T) and Fakeddit (Nakamura al. 2020). The results indicate that our MMFakeBench poses greater challenge in binary classification compared to the other two datasets. More importantly, the primary challenge of our benchmark lies in its capacity to perform fine-grained assessments of misinformation sources in scenarios where multiple forgery types coexist. This mirrors the complexity of real-world environments, where misinformation stems from diverse, overlapping sources. This capability is essential for addressing real-world challenges and underscores the importance of MMFakcBench in advancing multimodal misinformation detection.
A.1.5 EFFICIENCY OF MMD-AGENT
In Table we compare the inference time and computational resource usage of MMD-Agent with standard prompting methods. Overall, while MMD-Agent shows an increase in inference time, it does not lead to additional GPU memory consumption and significantly enhances both the performance and interpretability of multimodal misinformation detection. The experimental results are shown in the table, and the detailed analysis is as follows: (1) MMD-Agent introduces higher inference time compared to standard prompting methods, primarily due to the additional reasoning steps, such extracting key entities from the text. (2) MMD-Agent does not lead to additional GPU memory consumption. This is because the Agent method does not affect the model parameter deployment and dataset storage.
Table 8: Efficiency MMD-Agent on LLaVA-L.6-34B.
Metric
Standard
MMD-Agent
Average Inference Time (s)
5.97s
49.04s
Memory (GB)
82G
82G
Performance (FI score)
25.7
49.9
17

---

Published as a conference paper at ICLR 2025
While the increased inference time is consideration, the substantial gains brought by the MMD- Agent should not be overlooked. On the one hand, it greatly enhances the detection performance, with the Fl score improving from 25.7 to 49.9. On the other hand, MMM-Agent offers stronger interpretability because it offers a detailed analysis of the misinformation rather than only providing classification labels used in standard prompting methods. Specifically, shown in Fig. 4 the content of the intermediate reasoning traces such as Thought 2. Thought 3, and Thought 4 accurately analyzes the specific reasons for misinformation from different sources.
Table 9: (a) Results of the specialized detector, HAMMER, fine-tuning on the MMFakeBench dataset. (b) Results of the specialized detector, FKA-Owl, fine-tuning on the MMFakeBench dataset.
tune HAMMER Model.
Fine-tune FKA-Owl Model.
Model
DGM4
MMFakeBBnnh
Model
DGM4
MMFaaeBench
HAMMER before fine tuming
83.2
44.1
FKA-Owl before finc-tuning
78.5
44,6
HAMMER after fine-tuning (10)
78.7
46.8
FKA-Owl after fine-tuning (10)
78.3
46,9
HAMMER after fine tuning (100)
70.1
60.8
FKA-Owl after fine-tuning (100)
66.4
61.1
HAMMER after fine-tuning (1000)
63.2
72.4
FKA- R.w line-tuning (1000)
76.2
A. 1.6 FINE-TUNING ON EXISTING DETECTORS
We conducted experiments in Table to to investigate the adaptation of existing models to the MM- FakeBench dataset. Specifically, we selected two existing powerful specialized detectors, HAM- MER (Saa al. (2023) and L.I.M-based detector, FKA-Owl Liu et al. (2024c). Then we fine-tuned them incrementally with 10, 100, and 1000 examples. We reported the results on both the DGM4 Shao al. (223) dataset (used for training and evaluating HAMMER and FKA-Owl) and our proposed
dataset. The results indicate:
1) Tuning-based models can be hard generalize unseen forgery data. For off-the-shed dedicated detectors without fine-tuning, their performance on the proposed benchmark dataset notably poor. With the rapid development of generative models, new forgery techniques and synthesized data continue to emerge. Models trained on limited samples face significant challenges in generalizing to unseen types of forgery data, highlighting a critical issue in the field of fake detection Liu al. (2024c); Ojha et al. (003bb).
2) Catastrophic forgetting issues are inevitable. Fine-tuning on new data improves performance on MMFakeBench but simultaneously leads to decline in performance on the original dataset. This degradation underscores the phenomenon of catastrophic forgetting, where previously acquired knowledge, such as that from the DGM4 dataset, is progressively lost.
A.1.7 MORE ERROR ANALYSIS
In Fig. 8 We have presented more error analysis that span four models of different scales (i.e., GPT- 4V, LLa VA-166-4BB BLIP2-FLAN-TS-XXI and VILA-13B) and on three distinct forgery sources of textual veracity distortions, visual veracity distortion and pross-modal consistency distortion. Based on these cases, we provide a deep amalysis of the shortcomings of both largely and moderately sized models when encountering different sources of multimodal misinformation.
1) For cases of textual veracity distortion, our analysis is summarized as follows:
Reliance on external knowledge. While Largely sized models like LLaVA-1.6-348 and GPT- 4V exhibit strong reasoning capabilities, they are challenging to infer the factual correctness of statement without access to a broader context. For instance, when faced with factually incorrect statements that appear linguistically accurate, such as the GPT-generated rumor, all these four models lack the inherent capability to question the information without retrieving corroborative evidence.
2) For cases of visual veracity distortion, our analysis is summarized as follows:
Limited Sensitivity to Abnormal Physical Features. Models including GPT-4V, LLaVA-1.6- 34B, BLIP2-LLAN-TS-XLL and VILA-13B, face challenges in discerning abnormal physical characteristics. For instance, in the PS edeeddeed examples, these four models fail to detect subtle yet unrealistic manipulations, such swollen necks or distorted facial features in images of Donald Trump. Instead, they are frequently distracted by more prominent visual elements, such as facial expressions gestures, resulting in misjudgments when identifying these manipulations.
18

---

Published as a conference paper at ICLR 2025
X
Retrieved External Knowledge: Could not find relevant knowledge
VILA-138: ; The news
BLUSS-AANIS-O SUPFORT.
ALaVA-1--34B: Kaned on the
GPI-4V: The caption doesn't provide sufficient detail about the ciroumslances or the individuuls involveed to make a clear deteraunation
caption is not supported by any creditble obyociive
news caption. there not enough information to dettermine the veracity of the news caption.
wvidemu
GPT EFT-geeerated Rumor &Al AAl-geerrated Image
Answer: [Test Suppocts]
Answerr [Test Supports]
Answer: [Test Supports)
Answer: [Test Supportsi
Testual Faci conflict Roason: The deseribes Jeunifer cittictzes 'The Voice' Execs for using show pronsot her
company: Out reality, this did happen.
Excoc
FLANTS-XXL BLIP2- rely
Error Reason: Lacking external
Correct Reason: Due lack external kndge GPT-4V make dediniiive judgmen..
absence rolevan esternal
knowledge. VILA-13B
ltself pencide analysis. Therefore, model makes wrong judgment
information, LLaVA-16-348 unaile rouch conclusive decision.
canmott procide a finel judgment.
Jennifer at war with 'The Voice' Exees For Using Show To Promott His Own Company!
VILA-138
BLIF2 FIaaT
LLAVA-16-4B
GFT-vV
X
X you've uploaded appeass show Prosidlent Donald
VILA-133: The
BLIP2 FLANTS-KLL president siiling tuble.
LLaVA-16-34B: The persont's
GPT-4V;
shows., with a larg" head neck...
be one
contemplation focus something off camera. Without additional context, difficuli so determine if the image goes against any dbjective
subjective description and
Trump
agsinst any
contemplative mood....
abjective
cannot determine image
lssell controdicts dbjective
Answer: [Image Supports]
Anewer: [Immage Supports]
Answer: [Image Supports]
Answer: [Image Supports]
Natural
Visual Fact-confliet Reason The image depicts Donald 'Irump with an abnormally swellen face and neck, echibiting,
PS-edited Images
noticesble feotures that deviate from natural propections.
Eiror Reason: The VII. does not discern artifisially alhere physical features. The model misinterpochs these distortions to subjective stylietic choices.
Eruor Reason: BLIP2
krror Reason: LLaV 1-1-4-116334 focuses the general facial expression of consomplation, it foiled to account for the exaggerated physical distortions.
Correct Heason: GPT--V deseribes Unrald Trump as being a scrious of contemplatiie moxd but overlooke the obvious
FLANTS XXL accurately receggize the growea contest of the president sitting at table, it overlooked the significant alberations in Donald Trump's is facial and neck propurtions
counterfactual visual features, such os his abnermally swellen face and nedi.
The president of the united states.
VI1.A-138
BLIP2 Fan
LL
GPT-VV
X BLIF2 FLANT-LLATSSAA XXL: man in suit and tie is spesking....
VII A-13B: The menos image shroe a man weasinng a wiih and ti. standing in frant-of a flag, and giving speech....
I.V AeV.s-e. The image shuncc personn who appwars to be public figune, possibly politician speaking at podiums with an American flag the bas lergaoddd
The imtaae you poovided shows a person standing at pudium with opena gestury wearing a suit...
GFT-V;
Asswe:: [MATCH]
[MATCH]
Answer. [MATCH]
Answer: [MATCH]
Kepurposed Inconsistency (Query for Person)
speech taking place
Harbor,
image depicts
indiviiiad speaknng indour conference
Fruur Reason: VII A-138 exhibit limited reasoeing
Eryor Reason: The reasoning capability of ULIF2 FLANTS XXL insuffcient. elements of speaker and suit in both modalities soccessfully mistead model's judgessent, caustng to oveelook the insonsiriunt scenarioe
ressoning; cxpability I.l.V is Incafficient identifies politician speaking at podium the image, fuill to recegniae the screario incumesbnnyy nw the text spectiies the sotting as National Harbor, Mid.
The reasuming
capabiiity CPTT-V insutiicien. presence individual effectively mislaads the model's judgment, conoing to overlonk the inpomsistent scenantes
capubility. as the gresence of the speaker in both modalities effectively mingrides the meddlls judgment, leading to overtook the inconctttenctee between the socnastos.
Rubio speaks at the Conservaltve Polnicall Action Conferance
March National Harbor
VILA-13B
BLI22-RanT--031
ILAVA-L6-34B
GFT-IV
Figure 8: More Error analysis from three different forgery sources.
3) For cases of cross-modal consistency distortion, our analysis is summarized as follows:
Distraction by global consistent semantics. When confronted with scenarios where images and text exhibit only subtle inconsistencies while other aspects remain largely consistent, large-scale models such as -L.VA-1.66-444 and GPT-4V often struggle to detect these discrepancies. These models can be distracted by the dominant presence of consistent content, which obscures the subtle mismatches critical for accurate misinformation detection. For instance, in the case of repurposed inconsistency, all four models focus on global semantics, such as a person delivering a speech, which diverts attention from deeper, subtle inconsistencies in the scenario.
19

---

Published as a conference paper at ICLR 2025
Artifkial Rumer & Al A-geneerlled aled Image Text
Natural Test & Al-generated Image
Edited Incendiernny
Lomasdo DiCapdo accope the awart Best Acsorin Drama Natoon Picturn for tis zole ta The Revenant
Alax Saizond
Magic play
criliciedd
conforence' fallore
GE ITENT RUTUTES) REFUTES]
GTALMAGE RUFUTOS] IIMAGE SUPPORTS]
GT MEEMATCHO [NTEMATTCH]
60
(p)
GFIlgroeeated Rumer & Repurpoeed Image
Natural Text & PS edddtt Image Test
R Reparpoood Inconsistimey
Test
Ridao Cuagiá to Cusseaa Laughing at Vatocans Burnng Menonal Day Cemmony.
Desncestoppsss Wike bunners in firnt the Invalices la protest against Traee lepallsation of sear muare
Te or of United Strve
GRLOK
(TEX RUPUTTSS Prdd TTENT SUPPORTS
GT:ILMAGE RETUTES) Predi IIMAGE SUPPORTS)
GT: MESMATCIH Predi (ORIGINAL))
0 Figure 9: Illustration of relevancy maps to showcase interpretability for the predicted output of the LLaVA-1 model.
(4)
W
A.1.8 INTERPRETABLE VISUALIZATION
We have incorporated the recent LVLM-interpret Stan al. (2024) as visualization tool to provide a more transparent understanding of the model's decision decision-making process. This method adapts the calculation of relevancy maps to LVLM,, thus providing a detailed representation of the regions of the image most relevant to each generated token. Specifically, as shown in Fig. (e) for cross-modal consistency distortion, the relevancy map focuses on the clapping action of the characters in the image. Clapping typically signifies recognition or approval of an event, conveying semantics that differ from the meaning of the associated text. This observation aligns with the model's s detection output, providing intuitive explanations for the identified inconsistency. In contrast, Fig. 9 (g) shows that the model predominantly focuses on the person depicted in the image while neglecting the slogan displayed at its center. This oversight prevents the model from identifying the semantic inconsistencies between the textual and visual modalities, resulting a failure to capture the underlying mismatch.
Textual Feature Visualization from Different Sources
Visual Feature visual zation from Different Sources
Real Sampa ampe
Real Sample
Textual Veracity Distortion
Textua Ve acricity Distarzion
Veracity Distortian
Veracity Distortion
O mods-mmdall Consistency Distortion
Cross-nodal Consistenry Distortion
(a)
(b:
Figure 10: a) TSNE visualization of textual features from different sources. (b) TSNE visualization
of visual features from different sources.
A.2 BENCHMARK ANALYSIS
In Table I0 We present benchmark analysis based on count statistics (i.e., the average number of words) and diversity (i.e., feature distribution and word frequency) and seman- tic similarity metrics for different forgery sources. The results indicate: 1) The average number of words is over 10 words, which suggests that the samples are sufficiently informative, 2) We have provided an analysis of the dataset's s diversity through t-SNE visualizations of textual and visual features presented in Fig. 10 as well as
Table 10: Benchmark statistics analysis.
Metric
Real Unit
Fake Unit
Avg. Number of Words
15.8
12.6
Word Frequency Entropy
10.9
11.6
Semantic Similarity (Text-Text)
0.41
0.47
Semantic Similarity (Image-Image)
0.39
0.41
20

---

Published as a conference paper at ICLR 2025
through the entropy of word frequencies detailed in Table [0] This feature visualization provides clear depiction of the dataset's diversity where each forgery sources are distinctly dispersed across the feature space. Moreover, we calculate the overall entropy of word frequency, where the entropy for real units ranges from O to 15.5 (calculated by log, (3000 x 15.8)). to 16.4 (calculated by log2 (7000 x 12.6)) for fake units where this dataset's entropy reached 10.9 and 1 11.6, demonstrating significant diversity. (3) Additionally, we computed the pairwise semantic similarity between samples, with results showing an average similarity below 0.5, further confirming the dataset's s rich diversity.
A.3 IMPLEMENTATION DETAILS
A.3.1 LVLMS CONFIGURATION DETAILS
Model Version. As for ChatGPT model, we use GPT-3.5 (gpt-3.5-turoo) or GPT-4 (gpt-4-vision- preview) generators detectors. As for text-to-image models, DALLE (DALLE-E3), Stable-Diffusion (Stable Diffusion XL), and Midjourney (Midjourney V6).
Inference Hyperparameters. To achieve the justified evaluation, we have set the sampling hy- perparameter of the off-the-shelf LVLMs, "do__amp""""p poo_sampe--- False" "Temperature 0", guarantee consistency in the predicted outputs. We adopt the default setting of other hyperparameters such as "max_new_token" = 512".
A.3.2 EXISTING SINGLE-SOURCE DETECTORS DETAILS
FakeingFakeNews. The FakingFakeNNews Huang et al. (2023) is designed for the detection of textual fake news, particularly for those natural human numan-written misinformation. It proposes an innovative approach for generating training instances, leveraging established styles and strategies commonly employed in human-authored propaganda. FakingFakeNews employs the ROBERTA Liu jet al. (2019) model as the backbone and trains it on its own proposed PROPAÑEWS dataset. In our experiments, we utilized the default configuration of the ROBERTA detector provided within the FakingFakeNews framework, retaining its default hyperparameters.
CNNSpot. CNNSpot Wang et al. (2020) is an artificial image detector designed specifically for identifying images produced by generative models. It employs the ResNet-50 model the classifier backbone. Notably, CNNSpot recognizes that data augmentation, including JPEG compression and Gaussian blur, can enhance the generalization capabilities of the detector. In our study, we utilize the pre-trained CNNSpot model with default hyperparameters to perform the detection of visual veracity distortion.
UnivFD. Uvin Ojha et al. (2023a) is general-purpose fake image detector that feature space not explicitly trained to distinguish between real from fake images. When given access to the feature space pre-trained lasguaelanguage model, UvinFD employs the nearest neighbor to identify fake images originating from various sources. The utilization of the large pre-trained model results smooth decision boundary, thereby enhancing the generalization capability the detector. In our work, we use the pre-trained detector of UnivFD with default hyperparameters to conduct the visual veracity distortion detection task.
LNP. LIP et al. (002) utilizes a well-traines denoising model to extract noise patterns from spatial images. Subsequently, it discerns fake images by analyzing the frequency domain of these noise patterns. Additionally, LNP employs the ResNet-50 model as the classifier backbone. In our study, we utilize the pre-trained LNP detector with default hyperparameters to conduct the visual veracity distortion detection task.
HAMMER. HAMMER Shao et al. (2023) is a multimodal detector designed to identify multimedia manipulation. It is built upon the pre-tr vasion-language model, ALBEF Li et al. (02 which comprises two unimodal encoders and a multimodal Aggregator. To accomplish the multimodal manipulation detection task, HAMMER employs hierarchical manipulation reasoning consisting of shallow and deep manipulation reasoning. Shallow manipulation reasoning involves semantic alignment between image and text embeddings, while deep manipulation reasoning performs deep
21

---

Published as a conference paper at ICLR 2025
cross-modal fusion for forgery detection. In our paper, employ the off-the-shelf HAMMER detector with default hyperparameters to detect cross-modke consistency distortion.
GPP4 Li al. (202cc) is developed based on large vision language models (LVLMs) to detect multimodal fake news. It identifies two types of forgery-specinn knowl- edge: semantic correlation and artifact tract, and augments LVLMs with these two knowledge. Specifically, it extracts semantic correlations using multi-level cross-modal reasoning module and comprehends unimodal localized details through a dual-branch fine-grained verification model. In our study, we employed the off-the-shelf FakeNewsGPT4 with default hyperparameters accomplish the cross-modal consistency distortion detection task.
Mixed Detection We combine the three most powerful models on each single-source detection task (i.e., FakingFake for textual veracity distortion, LNP for visual veracity distortion, and HAMMER for moos--mddal consistency distortion) to perform mixed detection. Specifically, utilizing our proposed hierarchical decomposition framework, we sequentially assess textual veracity, visual veracity, and cross-modal consistency with three single-source detectors and assign corresponding multi-class labels.
A.4 INSTRUCT PROMPT FOR CHATGPT
The construction of MMFakeBench employs the advanced ChatGPT to assist us in generating textual rumors, expanding detailed descriptions, and generating fact-conflicting descriptions. The specific prompts provided in this work are summarized as follows.
Instruct Prompts to Ask ChatGPT to Generate Textual Rumors. Fig. 11 illustrates the prompt utilized for asking ChatGPT generate textual rumors with different prompt methods. These methods include arbitrary generation, rewriting generation, and information manipulation.
(1) Arbitrary Generation Prompt
Please write piece of misinformation title. The domain should be one of gossip, science, health and politics. The time period should be within the past ten years. The type should be fake news/rumors/ misleading claims. Avoid answering words like fake, rumor, confusion, disbelief, misinformation, etc.
#query Misinformation title is:
(2) Rewriting Generation Prompt
Given sentence, please write a piece of misinformation title. The content should be the same. The writing style should be serious, informative and convincing. Avoid answering words like fake, rumor, confusion, disbelief, misinformation, etc.
#query
Sentence:
Misinformation title is:
Information Manipulation Prompt
Given true claim, please write piece of misinformation. It should be long enough, convincing and detailed. The error type should be fake news/ wwwssrumors//// misleading claims. Avoid answering words like fake, rumor, confusion, disbelief, misinformation, etc. #query
The true claim is:
The answer is:
Figure 11: Prompt used to ask ChatGPT to generate textual rumors with arbitrary generation, rewriting generation, and information manipulation methods.
Instruct Prompts to ask ChatGPT to Expand Detailed Descriptions. Fig. [2 illustrates the
prompt utilized for asking ChatGPT to expand detailed descriptions for textual rumors. Based
22

---

Published as a conference paper at ICLR 2025
Given news rumor that is not consistent with reality, need you imagine the scene in which the rumor occurred and provide details about the characters, famous buildings, physical objects, etc. the scene. Avoid sentences that state opinions and only describe physical objects. Avoid words like false, rumor, confusion, disbelief, misinformation, etc.
# in-context examples
Rumor: Peking University is in Thailand.
The answer is: A Thai university in the real world, with the Thai flag flying above, including the landmark building of Peking University, the gate of Peking University, plaque with the name of Peking University, the Boya Tower of Peking University, and Weiming Lake of Peking University.
Rumor:
Carlos Santana a US president.
The answer is: This realistic photo of Carlos Santana in the white house in the real world. Carlos Santana stands podium adorned with the presidential seal. Behind him, American flag hangs proudly. A row microphones and a table are set before him, as cameras flash around the room.
The Chrysler Building has yet to be surpassed height. The answer is: This is a realistic photo of the center of New York City in the real world. The Chrysler Building stands tall. The slope of the Chrysler Building forms crown. There are many high-rise buildings on the side.
Rumor:
#query
Rumor:
The answer is:
Figure 12: Prompt used to ask ChatGPT to expand detailed descriptions of rumors.
Imagine you as a science fiction writer. Given a true claim, please write a piece of misinformation. The error type should be contrary to objective facts, that objects events that do not exist in the real scene, etc. Avoid answering words like fake, rumor, confusion, disbelief, misinformation, etc. # in-context examples
The true claim is: People are standing top of snowy mountain. The answer is: There angels with wings welcoming these people.
The true claim is: A person sailing in the air on a snow board. The answer is: A person snowboarding in the clouds.
The true claim is: A man wearing blue tie with the ten commandments on it. The answer is: This man is holding huge fireball in his hand.
#query
The true claim is:
The answer is:
Figure 13: Prompt used to ask ChatGPT to generate fact-conflicting descriptions.
the responses, we prompt the stage-of-art diffusion generators to generate realistic and relevant supporting images.
Instruct Prompts ask ChatGPT Generate Fact-conflicting Descriptions. Fig. 13 illustrates
the prompt utilized for asking ChatGPT to generate fact-conflicting descriptions. Then, we combine these descriptions with original captions as prompts the Midjourney V6 model Midjourney (2022)
to create corresponding images with additional fact-conflicting information.
A.S DATASET LICENSES
The licenses of the existing datasets used in this work is follows:
FakeNewsNet: free to use by all.
FEVER: Apache License 2.0.
Fakeddit: free by all.
23

---

Published as a conference paper at ICLR 2025
NewsClipings: free to use by all.
DGM4: S-Lab License 1.0
COCO-Counterfactuall: Attribution 4.0 International (CC BY 4.0).
VisualNews: free to use by all.
MSCOCO: free use by all.
All datasets provided in this work licensed under the Attribution Non-Commercial Share 4.0 International (CC B BY-NC-SA 4.0) license We chose this license because some of the original datasets have this license and we provide our datasets with the same level of access.
A.6 MORE VISUALIZATION EXAMPLES
We have provided more visualization examples of multimodal misinformation from different sources
Fig.
Fig. 15 and Fig. 16
Textual Rumor A Al-generated Image
Y'AD
The look of love: Katie Holmes and Jamie Foxx pack the PDA
Pregnant Kylie Jenner Looked Great' at Family's Christmas Eve Party,
Jennifer Lawrence and Leonardo DiCaprio Spotted Kissing Exclusive
Charles de Gaulle lost all elections President of the Fifth French Republic.
public appearance at pre- rre-Grammmy gala.
Source Says.
Hollywood Party.
Cultarppion
In 2015, among Americans, 30% of adults had consumed alcoholic drink in the last year.
Northwestern University is founding member of Big Ten Conference starting in 1893
Scientist Uncovers Previously Unknown Galactic Cluster Near Milky Way Galaxy.
NASA Confirms Existence of Alien Life Form Found Mars' Surface
Textual Rumor& & Repurposed Image
TRUM
Democrat Maxine Waters Has Shown Up To Only 10% Of Congressional Meetings For 35 years.
Obama Admits Secret Deal With China to Hand Over U.S.
Kim Kardashian and Kanye West's Wedding: All
Trump Votes For Death Penalty For Being Gay
Territory Settle Debt
Best Photos from Paris
and Florence Photos.
Figure 14: Visualization examples of multimodal misinformation from the textual veracity distortion.
24

---

Published as a conference paper at ICLR 2025
Natural Text & Al-generated Image
Leonardo DiCaprio accepts the award for Best Actor in Drama Motion Picture for his role in The Revenant.
James Candolfini and Kristen Stewart in Welcome the Rileys.
Kevin Durant is swarmed by photographers.
Tourists visit the Forbidden City in Beijing in early July.
The US Capitol on Wednesday day before scaffolding was completely removed.
French gendarmerie officers patrol near the Eiffel Tower in Paris on Nov
reflection a man taking picture in a mirror.
black cat rubbing up against laptop.
Natural Text & PS-edited Image
jeremy lamb attempting block collin sexton.
There unpleasant surprise underneath her cake.
The president of the united states.
This dog looking out car window.
Figure 15: Visualization examples of multimodal misinformation from the visual veracity distortion.
25

---

Published as a conference paper at ICLR 2025
Repurposed Inconsistency
TOON HODE
Barack Obama has called the Republican House Speaker urging resolution of growing budget crisis
Ohama and Kenyatta hold joint conference after their meeting
Clinton blames Bush administration for Senate job record
Nationalists also held march in the western city of Lviv.
NO
Secretary of State Hillary Rodham Clinton adjusts her glasses after answering a question
Andy Murray returns the ball to YenHsun Lu he won comfortably also straight
The Raid sees Uwais return as Rama dogged and resourc- eful Indonesian cop
MAPFRE fans show their support prior the start Leg 8
Edited Inconsistency
Jonathan Grado
Liverpool Alberto Moreno cries after scoring their third goal at Tottenham
Vladimir Putin celebrates his wife Lyudmila in 2014
likes to work six days a week tinkering with the various headphones
large black ball sitting next glass of milk.
Grado makes
Figure 16: Visualization examples of multimodal misinformation from the cross-modal consistency distortion.
26


Reviews:
[{"review_id": "8Oh7u3ftrK", "reviewer": "Reviewer_jzRM", "summary": "This paper looks at creating a multimodal misinformation dataset called MMFakeBench that can be used to evaluate detection methods and Large-Vision Models (LVLMs) under a zero-shot setting. For the dataset a validation set is created for hyperparameter selection and also used to conduct a human evaluation of the proposed MMFakeBench dataset. Utilizing the MMFakeBench dataset, 15 LVLMs are evaluated on the test set of the dataset and highlights the shortcomings of these LVLMs when compared to the human evaluation results.", "questions": "* Can the authors possibly give further examples of evaluation type datasets that look at evaluation in a zero-shot setting for misinformation detection.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Utilizing a human evaluation helped strengthen the claim that currently LVLMs models have room for improvement when it comes to tackling the tasks proposed with the MMFakeBench dataset.\n* Table 2 was a well thought out table that coverages a wide range of LVLMs\n* The proposed dataset is unique as it proposes different tasks related to mixed-source multimodal misinformation detection, as it shows in Table 1 the difference between its dataset and previous multimodal misinformation datasets.", "weaknesses": "* Currently the dataset size seems somewhat low when comparing it to other datasets mentioned in the paper. However I do recognize that this is specifically meant to be an evaluation dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xHobmCToPs", "reviewer": "Reviewer_yehn", "summary": "This work proposes the first benchmark in mixed-source multimodal misinformation detection (MMD). The authors provide comprehensive evaluations and analysis under the proposed evaluation settings, providing a new baseline for future research.", "questions": "1. What is the precise scope of this work? For example, does deepfake detection fall within its scope? If so, how does the work address multi-modal detection challenges, such as face reenactment or face swapping?\n\n2. How do Vision-Language Models (VLMs), which incorporate both language and vision modules, perform within the proposed benchmark?\n\nMore questions can be seen in the \"Weakness\" part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is important to create such a benchmark and baseline for the detection of multi-source fakes.\n\n2. The proposed three types based on the sources of falsified content are new to the field and technical sound.\n\n3. The designed MMD-Agent is well-designed and novel.", "weaknesses": "1. Lack of comparison with related detection methods: This work includes two detectors within the visual veracity distortion (VDD) field, both relying solely on visual modules for detection. Why not include comparisons with recent methods that integrate both visual and language modules, which are more directly relevant to this research?\n\n2. Insufficient comparison with deepfake detection methods: If \"misinformation\" encompasses any fake content, it would be beneficial for the authors to compare their method with established deepfake detectors. Given the high profile and potential danger of deepfakes, adding such comparisons would strengthen the paper’s impact and comprehensiveness.\n\n3. Lack of intuitive visualizations for the proposed method: It is recommended that the authors use visualization tools like GradCAM to provide clearer, more intuitive insights into their detection results. Additionally, the current t-SNE visualizations could be improved to more clearly convey their underlying meaning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1JuEHK7JHR", "reviewer": "Reviewer_onkv", "summary": "The authors introduce MMFakeBench, presented as a benchmark for evaluating mixed-source multimodal misinformation detection (MMD). They develop a multi-class evaluation metric and conduct evaluations of six state-of-the-art detection methods and 15 large vision-language models on MMFakeBench. Additionally, the authors propose an LVLM-based framework, MMD-Agent, which is intended to enhance detection performance in mixed-source MMD tasks.", "questions": "1. For detecting outputs from generative models, a CNN often achieves better detection performance than an LVLM. Consider including performance results using foundation models paired with a fine-tuned network.\n\n2. Some details in Table 1 are unclear. For instance, why do MEIR and DGM4, which involve text editing, lack Textual Veracity Distortion? Edited texts are essentially a type of rumour.\n\nBased on the clarification regarding the proposed dataset as an evaluation resource and the additional explanations provided, I believe you have addressed most of my concerns and I have decided to raise my rating to 6. However, as MMFakeBench is mix-sourced, compared to NewsCLIPpings, which focuses on OOC detection, more data is needed to achieve sufficient diversity and ensure the validity of experimental results. I look forward to your future work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper offers relatively comprehensive coverage of misinformation types, utilizes a diverse set of detection methods, and includes clear figures explaining the dataset and workflow.", "weaknesses": "1. The contributions of this work are relative limited. Compared to prior benchmarks like DGM4, the primary difference lies in the use of large models to generate data, without introducing novel, highly deceptive misinformation generation methods. Additionally, the proposed MMD-Agent framework contributes minimally to the field, relying heavily on the use of LVLMs.\n\n2. The dataset size is relatively small. While recent benchmarks like DGM4 contain over 200k samples, MMFakeBench includes only about 10k. This raises concerns about whether each misinformation type is represented with enough diversity to support model generalization.\n\n3. It is unclear whether the misinformation types included are meaningful. For example, the Cross-modal Consistency Distortion example in Figure 2 — “An old man reading a book on a park bench” versus “An old man reading a newspaper on a park bench” — lacks practical relevance. Emphasis should be placed on edits involving named entities, such as celebrities, to ensure sufficient impact and deception. Similarly, the Visual Veracity Distortion example with “Veracious Text & AI-generated Image” lacks sufficient misleading potential; the reader’s first impression might be that the image is obviously edited, which fails to show it can lead viewers into perceiving falsified content as authentic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HSx9fFutqu", "reviewer": "Reviewer_x4j6", "summary": "This paper proposed a new benchmark of multimodal misinformation detection, and the authors constructed a multimodal misinformation data generation pipeline by integrating characteristics from previous multimodal misinformation benchmarks. Additionally, the authors proposed a novel framework named MMd-Agent, which decomposes the multimodal misinformation detection task into three sub-tasks based on the three critical sources (textual veracity distortion,visual veracity distortion, and cross-modal consistency distortion) identified in this paper and processes them in a step-by-step manner, this method finally achieved good results.", "questions": "1.\tWhat’s the distinction between Natural Rumor and Artificial Rumor? \n2.\tDo the definitions of the three critical sources apply to other publicly available datasets?\n3.\tThe generation style of the visual veracity distortion data is relatively uniform. Does this meet the benchmark's intended purpose?\n4.\tWhether the accuracy of all GPT-generated rumors has been manually verified is necessary.\n5.\tHas the impact of dataset imbalance (between real and fake, as well as imbalances among different subclasses) been validated? This makes it difficult to reuse the dataset.\n6.\tThe authors should provide further analysis of the experimental results in Figure 5, comparing their performance differences to better illustrate the distinctions between different categories(Natural Rumor, Artificial Rumor and GPT-generated Rumor).\n7.\tI believe that mixing human-generated and LLM-generated misinformation is very meaningful; however, other datasets, such as Twitter and GossipCop, also contain three critical categories. How are the limitations of the different datasets in Table 1 assessed?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tFrom a mixed-source perspective, this paper proposes a new benchmark for multimodal misinformation detection, addressing some weaknesses in existing datasets.\n2.\tThe authors proposed a novel framework named MMd-Agent, which achieved impressive performance.\n3.\tThe paper is written in an easily understandable manner, and the experimental evaluations are thorough.", "weaknesses": "1.\tThis paper did not present the existing high-impact multimodal misinformation detection benchmarks or compare them to the proposed benchmark concerning the three critical sources.. Could the authors list some relevant multimodal misinformation benchmarks, multimodal misinformation benchmarks detection methods, and SOTA？\n2.\tThis paper did not present a summary of mainstream methods for detecting multimodal misinformation and summarize the classifications of these methods？\n3.\tThe experimental analysis lacks depth; for example, it only addresses some performance limitations of LLaVA-1.6-34B and GPT (as mentioned in the Error Analysis), while other models are evaluated solely on their performance. I hope the authors can conduct a more thorough analysis of the experimental section and consider analyzing the shortcomings of larger models as well.\n4.\tThe novelty of the proposed MMd-Agent is somewhat limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q5tYAvkPJn", "reviewer": "Reviewer_RALP", "summary": "This paper proposed MMFakeBench, a benchmark for multimodal disinformation detection from mixed sources. Unlike existing benchmarks focusing on single-source forgery, MMFakeBench approximates the real-world by encompassing three key sources: textual truthfulness distortions, visual truthfulness distortions, and cross-modal consistency distortions. complex disinformation. The benchmark contains 12 sub-categories of forgery types, totaling 3300 samples. The authors evaluate 6 current state-of-the-art detection methods and 15 LVLMs in a zero-shot setting, revealing the challenges faced by current methods in this mixed-source environment. To improve the detection accuracy, they propose MMD-Agent, which is based on the framework of LVLMs and divides the detection process into 2 phases, i.e., Hierarchical decomposition and Integration of internal and external knowledge.", "questions": "1. External knowledge sources: Have you considered other external knowledge bases besides Wikipedia to enhance the detection of complex rumors?\n\n2. Model tuning: Have you attempted to fine tune LVLMs on a subset of MMFakeBench to observe their performance improvement?\n\n3. Practical application: How does MMD Agent perform in real-time detection scenarios? What is its computational requirement?\n\n4. Scalability: Can your method be extended to other modalities, such as audio or video, to address more complex types of false information?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Originality: The introduction of MMFakeBench addresses a previously unexplored field of multimodal false information detection, making it more representative of real-world scenarios.\n\n2. Quality: The benchmark is carefully constructed, covering 12 types of forgery under three main categories, providing a robust evaluation platform.\n\n3. Clarity: The structure of the paper is reasonable, and the categories included in the benchmark, the evaluation process, and the explanation of the proposed framework are clear and concise.\n\n4. Significance: By revealing the limitations of current detection methods and proposing new frameworks, the paper has the potential to drive future research and improvement in this field.", "weaknesses": "1. Limited exploration of external knowledge sources: The paper acknowledges that reliance on sources such as Wikipedia is a limitation but can further explore its impact on detection performance and possible solutions.\n\n2. Evaluation depth: Although the paper evaluated multiple models, it mainly focused on the zero-shot setting. If fine-tuning models or more ablation experiments are added, the experimental results may be more convincing.\n\n3. The handling of rumors generated by GPT: The paper points out the challenges of detecting rumors generated by GPT, but further analysis or solutions can be proposed. There is no analysis of why GPT-generated Rumor is closer to Natural Rumor, or in other words, why is GPT-generated Rumor about as difficult to detect as Natural Rumor? After all, Artificial Rumor is also written by humans, so it should be about the same difficulty as Natural Rumor, but the experimental result is that Natural Rumor is the easiest to detect.\n\n4. It is suggested to discuss and compare more related works such as [1] in this paper.\n\n[1] Detecting and Grounding Multi-Modal Media Manipulation and Beyond. TPAMI 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)