ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection
Authors: Zhihao Sun
Haoran Jiang
Haoran Chen
Yixin Cao
Xipeng Qiu
Zuxuan Wu
Yu-Gang Jiang
Venue: NeurIPS 2025 poster ()
Keywords: Image Manipulation Detection Assistant
Chain-of-Clues
Multimodal Large Language Model
Primary area: applications

Abstract: Multimodal large language models have unlocked new possibilities for various multimodal tasks. However, their potential in image manipulation detection remains unexplored. When directly applied to the IMD task, M-LLMs often produce reasoning texts that suffer from hallucinations and overthinking. To address this, we propose ForgerySleuth, which leverages M-LLMs to perform comprehensive clue fusion and generate segmentation outputs indicating specific regions that are tampered with. Moreover, we construct the ForgeryAnalysis dataset through the Chain-of-Clues prompt, which includes analysis and reasoning text to upgrade the image manipulation detection task. A data engine is also introduced to build a larger-scale dataset for the pre-training phase. Our extensive experiments demonstrate the effectiveness of ForgeryAnalysis and show that ForgerySleuth significantly outperforms existing methods in generalization, robustness, and explainability.

ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection
Zhihao Sun Haoran Jiang".2, 1.2, HHooaa Chen Yixin Cao!.2, Xipeng Qiul?²,
Zuxuan Wul.2t, Yu-Gang Jiang
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University Shanghai Collaborative Innovation Center of Intelligent Visual Computing
https//githbb.com/uuuzhhaaa SloggeyySS
Abstract
Multimodal large language models have unlocked new possibilities for various multimodal tasks. However, their potential in image manipulation detection remains unexplored. When directly applied to the IMD task, M-LLMs often produce reasoning texts that suffer from hallucinations and overthinking. To address this, we propose ForgerySleuth, which leverages M-LLMs to perform comprehensive clue fusion and generate segmentation outputs indicating specific regions that are tampered with. Moreover, we construct the Forgery Analysis dataset through the Chain-of of-Clues prompt, which includes analysis and reasoning text to upgrade the image manipulation detection task. A data engine is also introduced to build a larger- scale dataset for the pre-training phase. Our extensive experiments demonstrate the effectiveness of PorgeryAnalysiee and show that ForgerySleuth significantly outperforms existing methods in generalization, robustness, and explainability.
Introduction
Recent advancements in multimodal large language models (M-LLMs) [1] have unlocked new possibilities across a variety tasks, propelling advancements numerous traditional research fields [2, 3, 4]. Despite these advancements, few attempts have been made to enhance the ability of multimodal large language models on the image manipulation detection task (IMD). In fact, with the rise of advanced image generation and editing tools, both creative and malicious alterations to visual content have become increasingly difficult to detect. As a result, identifying manipulated images has become essential for maintaining the integrity of digital media [5, 6, 7].
While appealing, directly applying M-LLMs for IMD produces results in text, which falls short even compared with traditional IMD methods that are able to generate segmentation masks to highlight tampered regions. In addition, M-LLMs are often prone to hallucinations and overthinking, making them unreliable for precise manipulation detection. For instance, as shown in Figure 1 (a). M-LLMs may incorrectly identify non-tampered regions, such as the "fish" area, as altered. This highlights the need to enhance the reasoning capabilities of M-LLMs for tampered region detection. Furthermore, M-LLMs, pre-trained on vast datasets, excel in recognizing authentic image patterns and world knowledge, yet lack the ability to combine low-level statistical features to pinpoint manipulation evidence, which has proven crucial for IMD [8, 9, 10, 1 11].
In this paper, we explore how to unleash the power of M-LLMs for IMD tasks. We introduce carefully designed modules that empower an M-LLM as an IMD expert, named ForgerySleuth. ForgerySleuth
t: corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Upon the analysis, the position and lighting of the fish seem bit awkward and may have been tampered with. addition, the head the human body extremely unressonable, which makes the authenticity of this image highly doubeful.
LLaMA:
Overal
GPT-40: Yes, the image appears to have been digitally altered. The
the have their swapped creating humorous effect. The size the fish elso seems abnormal, but further enalysis is needed to confirm whether has been manipulated.
that
dog's,
Could you detect alterations in this image?
ForgerySleuthh (Ours): These clues confirm that the Image is <FAKE>, and manipulation detected in [SEG]. have identified following clues, levels clues are evidence.
Relevanox
Consenness
high-level anomalies are doubts,
lower-
11S-7B-vttexxllnnattry
High-Level Semantic Anomalies: 1. Content Contrary to Common
[SEG]
Sense: The Image shows person's body with a dog's face
QWen-VL-pplus
2. Misleading Content:
depict reality, biologically, human body cannot... boundary around the reglon shows irregulnities and unnatural edges where the dag's face meets the human neck.
QWen2 7B-intuttt
Middle-Level Visual Defects: 1. Boundary Artifacts:
GPT-4o
2. Perspective: The size and angle of the face do not align correctly with the body... Low-Level Pixel Statisties: CoCol Inconsistency.... Textural and Sharpness
ForgerySleuth 78 (Ours)
(a) Performance M-LLMs Image Manipulation Detection Task
(b) Comparison on ForgeryAnalysis
Performance and comparison of existing M-LLMs the image manipulation detection task. Our ForgerySleuth assistant provides explanatory analysis with Chain-of-Clund and demonstrates the best forgery analysis capabilities.
Figure 1:
is expected provide textual explanation of detected clues with the reasoning process, along with a segmentation mask to highlight tampered regions. Inspired by the fact that existing IMD methods rely low-level features, such noise patterns, to localize precise tampered regions, aim to capture similar low-level features through ForgerySleuth. To do so, ForgerySleuth integrates M-LLMs with trace encoder, enabling the model to leverage world knowledge to detect high-level semantic anomalies while still capturing low-level forgery traces. Additionally, inspired by LISA [12], we introduce a vision decoder with fusion mechanism that uses attention to combine high-level anomalies in the LLM tokens and low-level traces in the trace embedding, ensuring the generation of accurate segmentation masks.
To further enhance the reasoning capability of M-LLMMs on manipulation detection, we propose a supervised fine-tuning (SFT) dataset ForgeryAnalysi specifically tailored for the IMD task. Each entry in Forgery Analysis is initially generated by GPT-4o using a novel Chain-of- prompt. Specifically, we ask GPT-4o produce a detailed thought process and provide reasoning for why a particular region is tampered with, including high-level semantic anomalies (e.g. content that contradicts common sense), mid-level visual defects (such lighting inconsistencies), and low-level pixel information (such as color, texture, etc.). The generated entries are then reviewed and refined by experts. Additionally, we build data engine based on this dataset automate forgery analysis. enabling us to create larger-scale Forgery Analysis-PT dataset for pre-training.
Extensive experiments popular benchmarks demonstrate the success of ForgerySleuth in pixel- level manipulation localization and text-based forgery analysis. Specifically, our approach outper- forms the current SoTA method by up 24.7% 24.7% in pixel-level localization tasks. Moreover, in the Forgery Analysis-Evar comprehensive scoring, our method surpasses the best available model, GPT-4o, achieving an improvement of 35.8%. In summary, our main contributions include:
Novel Exploration. We explored the role of M-LLMs in image manipulation detection, upgrading the manipulation detection task by incorporating clues analysis and reasoning.
Valuable Dataset. We constructed Analysy dataset, providing instructions for analysis and reasoning through Chain-of-Clues prompting. Additionally, we developed data engine to automate forgery analysis, enabling the creation of large-scale dataset.
Practical Framework. We introduced ForgerySSeuth assistant framework, which integrates M-LLMS with a trace encoder to leverage multi-level clues. The vision decoder with fusion mechanism enables comprehensive clues fusion and segmentation outputs.

---

2 Related Work
2.1 LLMs and Multimodal LLMs
The success of large language models in various natural language processing tasks has led researchers to explore their integration with vision modalities, resulting in the development of M-LLMs. BLIP-2 [13] introduces a visual encoder to process image features. LLaVA [14] aligns image and text features to achieve comprehensive visual and language understanding. Researchers also utilize prompt engincering to connect independent vision and language modules via API calls without end-to-end training [15, 16, 17]. However, while these approaches enable M-LLMs to perceive, the intersection with vision-centrie tasks, such as segmentation, remains underexplored. Additionally, VisionLL.M [18] and LISA [12] effectively integrate segmentation capabilities into M-LLMs, making them support vision-centric tasks, such as segmentation.
With advancements in fundamental reasoning and multimodal information processing, M-LLMs have demonstrated impressive proficiency across a diverse range tasks, including image captioning and video understanding [1]. Moreover, M-LLMs have been developed to address more complex real-world tasks in robotics, such as embodied agents [19, 20] and autonomous driving [4]. However, integrating M-LLMs into the field of image manipulation detection remains unexplored. While M- LLMs possess valuable world knowledge and can potentially detect high-level anomalics, M-LLMs often prone to hallucinations and overthinking, making them unreliable for precise manipulation detection. Furthermore, there is no existing IMD dataset with analysis instructions for supervised fine-tuning, which further restricts their capabilities.
2.2 Image Manipulation Detection
Image manipulation detection is a critical task in digital image forensics. The task has evolved beyond merely determining whether an image is authentic [21, 22]; it involves localizing tampered regions and providing segmentation masks [23, 24], which leads to more intuitive results. Early attempts [25, 26, 27, 28, 29] identified anomalies and designed corresponding hand-engineered features. These efforts systematically use various tampering clues, laying a solid foundation for the field. However, such hund-engineered features are specific to certain tampering types, which limits their applicability in real-world scenarios.
Recent approaches have shifted to more general capability of identifying complex and unknown manipulations. comaanicc-anostiie features, less dependent on specific content, are thought to provide better generalization [10, 11]. Common strategies include incorporating filters or extractors to capture low-level noise features [8. 9, 30, 31] and segh-frequency features [32], and using content features extracted from the image view as supplement to detect manipulation traces [10, 30. 31J. However, many of these features are learned implicitly by the network, which limits their explainability. Studies also [11, 32] detect anomalies by comparing patch-level or object-level features. However, capturing high-level semantic anomalies, such as content that conflicts with common sense or physical laws, is still challenging. In this work, extend the task by presenting reasoning process with multiple levels of clues expressed in natural language, making the detection results more comprehensible. Our proposed ForgerySleuth framework leverages M-LLMs to address this challenge, effectively leveraging world knowledge to detect high-level semantic anomalies while still capturing low-level forgery traces using trace encoder. FakeShield [33] is concurrent work that proposes multimodal large model for image manipulation detection.
3 ForgeryAnalysie Dataset
Our goal is to leverage existing M-LLMS to construct a high-quality dataset for IMD. We first describe how M-LLMs are utilized to generate initial clue analyses from various types of manipulated image sources in Section 3.1. These analyses are then meticulously refined by experts to create a high-quality dataset of 2,370 samples, which are used for the supervised fine-tuning (SFT) phase and the evaluation of M-LLMs. Section 3.2 describes our proposed data engine, which expands the 2k high-quality analysis instructions from Section 3.1 to 50k. This expansion supports the pre-training phase while maintaining quality standards. Detailed statistics of our dataset, along with examples of tampering analysis instructions, are provided in Section A.
3

---

(a) SFT Data
(b) Chain-of-Clues (CoC)
Prompt (tampered type, reasoning example)
(Doubts): high-level semantic anomalies
incrinn content
miskcading content
contcarry common settc
Filter vague response
(Evidence): middle-leve visual defects
Image
GPT-4o API
Delete vague ballucinations
boundary trace
perspective
lighting
physiczi constraint
(Evidence): low-level pixel statistics
Add Chain-offCCles
textural
sharpness
Allganeration fingerpeint
Reference
Forgery
Analysis
Image with tampered highlight
(2k)
Data
Engine
LoRA
Engine
Prompt Engine (tampered type, clues levels, CoC guidanee)
Forgery-
Forgery-
Analyst
Analyst
ForgeryAnalysis
.PT (50k)
Data Engine and PT Data
Prompt
GPT-4o
Experts
Data Engine
Figure 2: KorgeryAnalysee Dataset Construction Pipeline. Our pipeline begins with (a) GPT-4o generating initial analyses for manipulated images with annotated regions, followed by human expert review. The refined analyses are organized into (b) the Chain-of-Clues format. This human-curated ForgeryAnalysis (2k) dataset is used to train data engine. Finally, (c) this data engine generates ForgeryAnallsis-PT, a larger-scale dataset for model pre-training.
3.1 Forgery Analysis Data
Due to the absence of instruction-follon following datasets that support clue analysis, we introduce the AnageryAnalysis dataset to support the supervised fine-tuning (SFT) and evaluation of M-LLMs. As illustrated in Figure 2 (a), for each image selected from various data sources, we utilize the advanced M-LLM, GPT-4o, to generate initial clue analyses. Experts then carefully revise these analyses to eliminate hallucinations and ensure high quality. Furthermore, we have integrated a Chain-of-Clues structure to enhance reasoning capabilities.
Data Source. To ensure diverse dataset that covers various types of manipulation, collect 4,000 tampered images from existing IMD datasets, including MIML [34], CASIA2 [35], DEFACTO [36], and AutoSplice [37]. Each image is accompanied by annotations indicating the tampered regions. These sources include common real-world manipulation techniques such as splicing, copy-man sopy-move, object removal, Al generation, and Photoshop edits.
Instruction Design. We aim to provide highly specific and content- driven descriptions of clues for each image, rather than vague general analysis. We refer the clues and evidence widely used in digital image forensics and manipulation detection [38, 39] to design a more reliable and effective instruction structure. The key clues include but are not limited 1) low-level pixel statisties (e.g., noise, color, texture, sharpness, and Al-generation fingerprints). 2) middle-level visual defects (e.g., traces of tampered boundaries, lighting inconsistencies, and perspective relationships), and 3) high-level semantic anomalies (e.g., content that contradicts common sense, incites, or misleads).
The prompt instructs GPT-4o to assume it has detected manipulation in the highlighted region of the reference image and to analyze its detection based on clues from various levels and aspects. We provide the tampering type to help GPT-4o focus relevant clues, along with reasoning example to guide the content and format of the output. We also instruct GPT-4o to incorporate corresponding world knowledge, such well-known individuals or landmarks.
The responses generated by GPT-4o are subsequently revised by experts to ensure quality. Responses lacking detailed content analysis are filtered out, retaining only high-quality responses as draft data. The experts carefully review vague and incorrect statements that may arise from hallucinations, removing irrelevant clues and evidence. Inspired by works such as [40, 41] that introduce Chain-of- the step-by-step reasoning capabilities L.LMs, we propose Chain-of- (CoC) prompting approach, illustrated in Figure 2 (b). The reasoning process begins with "unveiling doubts" using
Chain-of-Clues.
(CoT) prompting and demonstrate its effectiveness in enhancing

---

Text
h
Embedding
LoRA
Anomaly Trace
Clues Image
Xp
LLM
Language
Fusion
bsgg
Fusion
Head
Darg
CLIP
Self- seffAAttnntion
Self- Arll-tttenion
bseg
Anomaly
Clues Image Attention
Trace Attention
Trace Encoder
Anoomaly Trace
Clues Image
Fusion
Fusion
Vision
MLP
MLP
Xang
Vision
Decoder
Anomaly Attention
Trace
Image to Clues Attention
Backbone
M
(a) Framework of ForgerySleuth
(b) Fusion Mechanism
Figure 3: Framework of ForgerySleuth. Given an image Ximg and prompt query Xp, the M-LLM Fm detects high-level semantic anomalies and generates a textual output T. The trace encoder Fr captures low-level, semantic-agnostic features. The vision decoder D fuses vision embeddings with the prompt embedding corresponding [SEG] token generate the segmentation mask M. LoRA is utilized in trainable modules for fine-tuning.
high-level clues, followed by "pinpointing evidence" through middle-level and low-level features. We organize the clues according to this structure, creating coherent chain clues, which results in 2,370 manually revised samples. To ensure accuracy, we conduct additional cross-validation with more than two experts, selecting 618 samples for evaluation. The remaining data are used for supervised fine-tuning
3.2 ForgeryAnalyss Engine
The pre-training phase requires larger-scale dataset compared the supervised fine-tuning phase. However, the high cost of GPT-4o and the need for expert revisions make expanding the dataset a challenge. To overcome this limitation and create a pre-training dataset, we introduce an additional data engine [42, 43, 44, 45] that scales up the training set. Specifically, we fine-tune LLaVA-v1.5-13B using LoRA [46] on the Forgery ForgeryAnalysis SFF dataset, aiming replicate the ability of GPT-4o to analyze clues while having experts eliminate any hallucinations. The resulting model is used as our data engine that able annotate data automatically. In particular, we select 50k images from existing public datasets. The data engine receives input that includes explicit information about tampered region, aiming to generate more precise and comprehensive clue analyses. It outputs manipulation analyses organized in the CoC format, illustrated in Figure (c). For more detailed information on the data engine, including the specific prompts used, please refer to Section A.2, and Figure 17 for output examples. The entire analysis generation process takes approximately 16 A800 GPU days. We refer to this dataset as Forgery Analysis-PT.
4 ForgerySleuth
Given input image for detection and prompt specifying the detection request, goal is to output binary segmentation mask M of the tampered regions, well text T that includes reasoning and evidence. This poses a challenge as the model needs to bridge vision and language modalities and capture tampering features across multiple levels. To address this, ForgerySleuth uses M-LLMs to connect vision and language modalities, enabling the description of detected clues and evidence in textual form. Additionally, we incorporate vision decoder into the multimodal large language models to perform dense prediction, generating the tampered region mask for this specific task. The pipeline of our framework is illustrated in Figure 3 (a).
More formally, we extend the original LLM vocabulary with new tokens, including [SEG]. which requests segmentation output, and <REAL> and <FAKE>, which indicate the image classification results. Given the input image Xing to be detected along with the prompt Xy. we first feed them into the M-LLM Fm, which outputs the hidden embedding at the last layer of LLM. We then extract the embedding hagg corresponding the [SEG] token from the hidden embedding h and apply an
5

---

MLP projection layer y to obtain hgg. The process can be formulated as
h = Fm (Xamg: Xp),
(1)
bug = (bh.e)
(2)
Fm, denotes M-LLM without the language head. This embedding represents the suspicious anomalous regions the M-LLMM detects and serves as the input to guide the subsequent vision decoder.
Inspired by existing IMD methods, we propose an independent trace encoder F& to focus on low-level features, which complements the relatively high-level vision and semantic features discovered by the M-LLM and provides more reliable tampering evidence, helping to minimize hallucinations from the LLM. Specifically, constrained convolutions [47] are employed with residual connections [48] the front part of the encoder to suppress the image content and learn manipulation features adaptively. The imput image Xing also fed into encoder F,, producing dense manipulation trace features f, which can be expressed by
fr (Xing)
The vision backbone Fe. simultaneously extracts dense vision content feature f. to support more precise segmentation. Finally, fo, and hagg are fed the vision decoder generate the segmentation mask M which indicates the tampered region. The language modeling head H processes and outputs Yia, containing an analysis of the reasoning and evidence. It can be formulated as
f Fo (XKngg)
M D(fe, fú, hseg):
T-H(h)
(5)
Fusion Mechanism. To integrate the image content embedding fo, trace embedding f,, and the LLM output tokens hagg obtained from the M-LLM, we take inspiration from Transformer segmentation models [49, 50, 42] and design vision decoder with a fusion attention mechanism, as illustrated in Figure (b). Here, the first layer of the module computes attention between anomalies the LLM output tokens and traces in the trace embeddings, facilitating the organization and pinpointing of clues. The subsequent layers focus on attention between refined clues in the upgraded tokens and content in the image embeddings, enabling more precise segmentation of the tampered regions.
Learning Objective. Our framework trained end-to-end using both the reasoning text loss Lix and the tampered region mask loss Lmask. The final learning objective L is formulated weighted sum of these losses, with weight Apxt and Amask as
L = Anxhret + Amask Lmask.
(6)
Specifically, Lxs is the auto-regressive cross-entropy (CE) loss for reasoning text generation, guiding the model to collect multi-level clues, while Lmust is the mask loss, which encourages the model produce precise segmentation results. Lmuuk is implemented by the combination of binary cross- entropy (BCE) loss and DICE loss, with respective weights Aoce and Adiee Given the ground-truth targets T and M, the final loss functions are defined as
L CE(T,T)) T),
Cmakk Abce (M, M) + MM)
(8)
It is noteworthy that no separate loss is required for the classification result, as the supervision for <REAL> and <FAKE> tokens are integrated into the text loss Lis-
4.1 Training Strategy
Trainable Parameter. To preserve the world knowledge and the normal patterns of authentic images learned by the pre-trained M-LLM Fm. we adopt LoRA [46] for efficient fine-tuning. The vision backbone F. is entirely frozen to retain the capacity for modeling image content features, which are crucial for accurate segmentation. Both the trace encoder J, and the vision decoder are fully trainable and fine-tuned. Additionally, the token embeddings of the L.M,, the language modeling head H. and the projection layer y are updated during training. Despite the large model scale, only 5.47% of the parameters trainable, making end-to-end training more efficient.

---

Table I: Manipulation localization results comparing ForgerySleuth with SoTA methods.
Optimal Threshold
Fixed Threshold (0.5)
Method
Columbia
Coverage
CASIA1
NIST16 COcoGlide Columbia
Coverage
CASIAI
NIST16 COCOGlide
Mantra-Net [8]
0.650
0,486
0.320
0.225
0.673
0.317
0.180
0.172
0.516
SPAN [9]
0.873
0.428
0.169
0.363
0.350
0.759
0.235
0.112
0.228
0.298
NetS-Net [10]
0.781
0.659
0.650
0.372
0.642
0.729
0.514
0.528
0.320
0.486
PSCC-Net [53]
0.760
0.615
0.670
0.210
0.685
0.604
0.473
0,520
0.113
0,515
CAT-Net2 [54]
0,923
0.582
0.852
0,417
0.603
0.859
0,381
0.752
0.308
0,434
TruFor [30]
0,914
0.735
0.822
0,470
0,720
0,859
0.600
0.737
0.399
0,523
Unionl'or. [31]
0.925
0.720
0.863
0.489
0.742
0.861
0.592
0,760
0.413
0,536
SorutrySleuth
0.792
0.870
0.610
0.751
0.925
0.804
0.518
0.562
Data Formulation. The training process involves two phases: 1) Pre-training Phase: (a) In the first stage, we aim to align the framework modules and ensure that the model can perform basic segmentation and visual reasoning tasks. For foundational segmentation abilities, we use semantic segmentation datasets such as ADE20k [51] and COCO- Stuff [52], transforming these datasets into visual dvestion-answer pairs using class names as questions. Additionally, we incorporate ReasonSeg dataset focuses on the IMD task, including analysis instructions that enable the model to recognize tampering traces and identify clues. We also utilize public IMD datasets, including MIMI. [34] and CASIA2 [35], with prompts randomly selected from simple or vague responses. 2) Supervised Fine-tuning Phase: The Forgery Analysis-SFT dataset, meticulously revised by experts to ensure the accuracy of reasoning and analysis, is used for final supervised fine-tuning standardize the analysis and output.
[12] to strengthen visual reasoning capabilities. (b) The AargeryAnalysii
Experiment
5.1 Experimental Setting
Testing Dataset. We utilize six publicly accessible test datasets, which are Columbia [55], Coverage [56], CASIA1 [35], NISTI6 [57]. IMD20 [58], and COCOGlide [30], to evaluate and compare our method with state-of-the-an methods thoroughly. To effectively evaluate the model's generalization capability, these datasets are ensured to be disjoint from the training data. Additionally, we use our AnaayisAnalysis-Fvii dataset assess the reasoning and analysis capabilities of the methods.
Evaluation Metrics. Localizing the tampered regions at the pixel level is crucial in image manipula- tion detection. We follow established practices [31] by using optimal threshold and fixed threshold F1 scores and the threshold-independenn Area Under the Curve (AUC) metric. To ensure fairness and precision in our comparative analysis, we refer to some results for other methods from [31, 32].
Evaluating the novel reasoning analysis outputs presents a unique challenge, as it involves assessing the comprehension, reasoning, and correctness in generating text explanations. To measure the similarity between the generated text and the ground-truth text, thereby reflecting its accuracy, we incorporate Semantic Textual Similarity (STS) metric. We follow SBERT [59] and use STS to measure the similarity in our Forgery Analysis-Eii Eval, where various models are utilized to calculate the similarity. Additionally, inspired by previous work [60, 14], we use GPT-4 as an automated evaluator to assess the reasoning performance of different models, which enables a more holistic evaluation of the generated analysis text, Figure 18 and Section D.3 provide the prompt format and the evaluation criteria. We conduct evaluations twice and report the average performance.
5.2 Manipulation Detection Results
The results in Table and Table 2 demonstrate the performance of our ForgerySleuth and comparisons with SoTA methods for image manipulation localization, using pixel-level Fl scores and AUC metrics, respectively. Our method consistently achieves the highest or second-highed AUC. Regarding F1 scores, our approach surpasses other methods across all datasets, showcasing reliability under both fixed and optimal thresholds. On challenging datasets like NIST16 and IMD20, Forgery out- performs UnionFormer by margins of 0. 105 and 0.121 for fixed and optimal thresholds, respectively, which we believe is significant given its challenging nature. Furthermore, on the COCOGlide dataset, which features novel diffusion- based manipulations, our model also exceeds the UnionFormer. These significant improvements can be attributed to the capability to effectively capture both low-level
7

---

Table 2: Manipulation localization results of ForgerySleuth and SoTA methods, using pixel-level AUC as the evaluation metric.
Method
Columbia
Coverage
CASIA1
NIST16
IMD220
Mantra-Nel [8]
0.824
0.819
0.817
0.795
0.748
SPAN [9]
0.936
0,922
0.797
0.840
0,750
PSCC-Net
0.982
0.847
0.829
0.855
0,806
ObjectForner [32]
0.955
0,928
0.843
0.872
0.821
TruFor [30]
0.947
0.925
0.957
0.877
UnionFormer [31]
0.989
0.945
0.972
0.881
0.860
VorgerySleute
0.992
0.962
0.969
0.911
Table 3: Robust evaluation results of Forgery-Sleuth and existing methods using pixel-level AUC.
Distortion
SPAN
PSCC-Net
ObjectFor.
UnionFor.
Ours
w/o distortion
0.8395
0.8547
0.8718
0.8813
0.8982
Resize (0.78x)
0.8324
0.8529
0.8717
0.8726
0.8962
Resize (0.25x)
0,8032
0.8501
0.8633
0.8719
0.8792
GSBr (k
0.8310
0.8538
0.8597
0.8651
0.8863
GSBr (k 15)
0.7915
0.7993
0.8026
0.8430
0.8658
GSN (a 3)
0.7517
0.7842
0.7958
0.8285
0.8452
GSN (a = 155)
0.6728
0.7665
0.7815
0,8057
0.8139
JPEG (q 100)
0.8359
0.8540
0.8637
0.8802
0.8974
JPFG (q 50)
0.8068
0.8537
0.8624
0.8797
0.8839
trace features and high-level semantic inconsistencies, enabling it to detect even subtle generative manipulations. Overall, the results emphasize the generalization ability of ForgerySleuth.
Robustness Evaluation. We apply several image distortions to the NIST16 dataset following [31, 32] to evaluate the robustness of our method and compare the results with other methods. The distortions included 1) resizing images to different scales, 2) applying Gaussian blur with different kernel sizes k, 3) adding Gaussian noise with various standard deviation a,, and 4) compressing images using JPEG with different quality factors q. The results in Table 3 show that our model consistently outperforms other methods across all types of distortions. This improvement in robustness stems from the ability to identify level semantic anomalies rather than relying solely low-level statistical features that are more susceptible to distortions.
5.3 Forgery Analysis Results
We compare our forgery analysis results with several M-LLMs, including GPT-4o, Qwen2-V [61], and LISA [12]. Additionally, we perform LoRA fine-tuning on the LISA using the ForgeryAnalysis. SFT to provide more comprehensive comparison. We incorporate STS to measure the similarity between the generated text and the ground-truth text in Forgery Analysis-Eval. The results, shown in Table 4, confirm that our ForgerySleuth consistently outperforms other methods.
We also leverage GPT-4 as an evaluator based on the Forgery Analysis- dataset to assess the quality of text analysis and reasoning. We collect answers from each M-LLM, and GPT-4 assigns score from I to 10 for each response. Beyond scoring, GPT-4 provides detailed explanations for its ratings. We also report the recall rate to directly reflect the ability identify tampered images.
Figure 1 (b) presents the scores different models without additional fine-tuning across various evaluation dimensions. The existing models exhibit low recall rates, and the overall evaluation suggests that they struggle to identify manipulations and provide accurate analyses. Figure 5 shows the scores of LISA and ForgerySleuth after SFT, along with the versions without SFT. ForgerySleuth shows an improvement of 5.05 in the overall score compared to LISA, further demonstrating the effectiveness of our design targeted specifically for the IMD task. Furthermore, using the SFT dataset results in performance gains for both methods, indicating the quality of the ForgeryAnalysia dataset.
5.4
Ablation Study
We conduct an extensive ablation study to analyze the effect of our ForgeryAnalysis dataset and each component and setting within ForucrySleuth framework. We report the pixel-level localization

---

Tampered (Mask) MVSS-Net
PSCC-Ne Net
CAT-Net2
Ours
Coverage
Overall
Detai
Recall
NIST16
IMD20
Relevince
Correctness
-exppatrt-
w/o.
LISA: -ex1t-elaaaa
CocOGlide
SFT
ForgerySleuth-78
ForgerySleuth-78 (Ours)
w. SFT
Figure 4: Visualization results comparing ForgerySleuth with exist- Figure 5: Forgery analysis re-
ing methods. The examples are taken from various datasets.
sults of the ablation study.
Table 4: Forgery analysis results of ForgerySleuth Table 5: Ablation study on different parts of and SoTA methods, evaluated using STS.
Forgery Analysis.
Model
Dim Ours
GPT-4o QWen2
QWen LISA
NIST16 IMD20 COCOGlide Forgery
Minil M.16-v
0.926
Forgery SFF
MiniLM-LL12-2
384
0.919
0.645
0.505
0.475
0.350
Po-geryA.PPT
0,516
0.705
0,571
8.71
mpoct-base-22
768
0,961
0.724
0.635
0.546
0.401
ForgeryA.-PTRee PT&SSFT
0.518
0.710
0.562
9.45
performance IMD datasets using the Fl score. We use ForgeryAnalysion Analysis-Eve to evaluate the quality the reasoning text, following the scoring criteria described Section 5.3.
Contribution of ForgeryAnalyste The experimental results in Table 5 show the performance drop when ForgeryAnalysis- data excluded during the pre-training phase, demonstrating the importance of large-scale data. Using Forgery AnalyssSA.. the model achieves strong performance on the IMD task, even without the final stage of supervised fine-tuning with EnngeryAnalysis... data. However, this final fine-tuning further enhances the quality of the tampering analysis text.
Effeet of Designed Components. As shown Table 6, removing the trace encoder Fe significantly degrades performance across all datasets, confirming its critical contribution in capturing low-level trace features and mitigating hallucinations in the M-LLM. At the same time, the performance drop observed when the fusion mechanism is excluded highlights the effectiveness of our fusion strategy integrating multimodal and trace-based information.
Impact of Training Strategies. In our comparison of various training strategies for the visual encoder Fu. including trainable and LoRA fine-tuned, we discover that the best performance is achieved by keeping the encoder completely frozen. This could be attributed to the fact that trainable and LoRA fine-tuned strategies slightly diminish the generalization ability of the original SAM.
5.5 Qualitative Results
Figure 4 showcases pixel-level localization results across different datasets, comparing our framework with other SoTA methods. The masks are displayed without binarization to provide more detailed view of the localization capability. Our method consistently delivers more precise tampered region detection with higher confidence across various types manipulation. More detailed examples of clues and analysis provided by ForgerySleuth can be found in Section A.4. Across different types of manipulation, relevant clues and high-quality analysis demonstrate the effectiveness of M-LLM in capturing high-level semantic anomalies.

---

Table 6: Ablation study on different modules and settings, using pixel-level Fl with fixed threshold 0.5 as the evaluation metric.
Setting
CASIA1
NIST16
IM1200
COCOGlide
w/o, Trace Enc. Ft
0.637
0.323
0.622
0.395
2. w/. Pusion Mechanism
0.628
0.463
0.649
0.513
w. F. Trainable
0.755
0.451
0.716
0.526
W. F. LoRA-ft
0.766
0.493
0.731
0.547
ForgerySleuth
0.804
0.518
0.710
0.562
Conclusion
In this work, we explored the potential of multimodal large language models in the image manipulation detection task. The proposed ForgerySleuth integrates M-LLMs with trace encoder, allowing the model to utilize world knowledge to detect high-level semantic anomalies while effectively capturing low-level forgery traces. Additionally, we introduced a vision decoder with fusion mechanism to integrate different features, ultimately producing precise segmentation masks. We also proposed supervised fine-tuning dataset, ForgeryAnalysie
specifically designed for the IMD task. Each entry was initially generated by GPT-4o using a novel Chain-of- of-Clues prompt and then reviewed and refined by experts. Furthermore, we developed data engine based on this dataset automate forgery analysis, facilitating the creation of a larger-scale purposes. discussion of limitations can be found in Section E. We have already made the resources publicly available, including the data, code, and weights, to provide resources for advancing the field.
EnggeryAnalysisse
dataset for pre- pre-trainin training
Acknowledgment. This work was supported part by the National Natural Science Foundation of China (Grant 62472098). We would like to express our gratitude to Ruirui Tu and Xu Han for their valuable assistance with data annotation, which significantly contributed the development of the ForgeryAnalysi. dataset.
References
[1] S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, "A survey on multimodal large language models," arXiv preprint arXiv:2306. 13549, 2023.
[2] M. Moor, Q. Huang, S.
Wu, M. Yasunaga, Y. Dalmia, J. Leskovec, C. Zakka, E. P. Reis, and Rajpurkar, "Med--lamingo: a multimodal medical few-shot learner," in Machine Learning for Health, 2023, pp. 353-367.
[3] A. Hu, Y. Shi, H. Xu, J. Ye, Q Ye, M. Yan, C. Ci, Q. Qian, J. Zhang, and Huang, "mplug- paperowl: Scientific diagram analysis with the multimodal large language model," in Proceed- ings of the 32nd ACM International Conference on Multimedia, 2024, pp. 6929-6938.
[4] C. Cui, Y. Ma, X. Cao, W. Ye, Y. Zhou, K. Liang, Chen, J. Lu, Z. Yang, K.-D. Liao al., "A survey on multimodal large language models for autonomous driving," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 958-979.
[5] V. Schetinger, M. M. Oliveira, R. da Silva, and T. J. Carvalho, "Humans are easily fooled by digital images," " Computers Graphics, vol. 68. pp. 142-151, 2017.
[6] S. J. Nightingale, K. A. Wade, and D. G. Watson, "Can people identify original and manipulated photos of real-worll scenes?" Cognitive Research: Principles and Implications, vol. pp. 1-21, 2017.
[7] Z. Lu, Huang. L. Bai, Qu, C. Wu, X. Liu, and W. Ouyang, "Seeing is not always believing: benchmarking human and model perception of ai-generated images," Advances in Neural Information Processing Systems, vol. 36, 2024.
[8] Y. Wu, W. AbdAlmageed, and P. Natarajan, "Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features," " in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2019, pp. 9543-9552.
10

---

[9] X. Hu, Z. Zhang, Z. Jiang, S. Chaudhuri, Z. Yang, and R. Nevatia, "Span: Spatial pyramid atten- tion network for image manipulation localization," " in Proceedings of the European Conference on Computer Vision, 2020, pp. 312-328.
[10] X. Chen, C. Dong, J. Ji, J. Cao, and X. Li, "Image manipulation detection by multi-view multi- scale supervision," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 185-14 193.
(11) Z. Sun, H. Jiang, D. Wang, X. Li, and J. Cao, "Safl-net: Semantic-agnostic feature learning net- work with auxiliary plugins for image manipulation detection," in Proceedings of the IEEE/CVF International Conference Computer Vision, 2023, pp. 22 4244-22443.
[12] X. Lai, Z. Tian, Y. Chen, Li, Y. Yuan, S. Liu, and J. Jia, "Lisa: Reasoning segmentation via large language model," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 9589.
[13] J. Li, D. Li, S. Savarese, and S. Hoi, "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," in International conference on machine learning, 2023, pp. 19 133-19 742.
[14] H. Liu, C. Li. Q. Wu, and Y. J. Lee, "Visual instruction tuning," Advances in neural information processing systems, vol. 36, 2024.
[15] Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, "Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face," Advances in Neural Information Processing Systems, vol. 36, 2024.
[16] C. Wu, S. Yin, W. Qi, X. Wang. Z. Tang, and N. Duan, "Visual chatgpt: Talking, drawing and editing with visual foundation models," arXiv preprint arXiv:2303.0400 2023.
[17] R. Yang, L. Song, Y. Li, S. Zhao, Y. Ge, X. Li, and Y. Shan, "Gpt4tools: Teaching large language model to use tools via seff-instruction," Advances in Neural Information Processing Systems, vol. 36, 2024.
[18] W. Wang, Z. Chen, X. Chen, J. Wu, X. Zhu, G. Zeng, P. Luo, T. Lu, J. Zhou, Y. Qiao al., "Visionllm: Large language model is also an open-ended decoder for vision-centric tasks," Advances in Neural Information Processing Systems, vol. 36, 2024.
[19] Huang, S. Yong, X. Ma, X. Linghu, P. Li, Y. Wang, Q. Li. S.-C. Zhu, B. Jia, and S. Huang, "An embodied generalist agent in 3d world," in Proceedings of the 41st International Conference on Machine Learning, 2024, pp. 403---04
[20] Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei, "Kosmos-2: Grounding multimodal large language models to the world," arXiv preprint arXiv:2306. 14824, 2023.
[21] J. Chen, X. Kang, Y. Liu, and Z. J. Wang, "Median filtering forensics based on convolutional neural networks," " IEEE Signal Processing Letters, vol. 22. no. 11, pp. 1849-1853, 2015.
[22] B. Bayar and M. C. Stamm, "A deep learning approach to universal image manipulation detection using a new convolutional layer," in Proceedings of the 4th ACM workshop on information hiding and multimedia security, 2016, pp. 5-10.
[23] R. Salloum, Ren, and C.-C. J. Kuo, "Image splicing localization using multi-task fully convolutional network (mfcn)," Journal of Visual Communication and Image Representation, vol. 51, pp. 201-209 2018
[24] Bappy, M. Mohammed, L. Nataraj, A. Flenner, S. Chandrasekaran, A. Roy-Chowdhury, J. H. Lawrence Peterson et al., "Detection and localization of image forgeries using resampling features and deep learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 69-77.
[25] P. Ferrara, T. Bianchi, A. De Rosa, and A. Piva, "Image forgery localization via fine-graing ine-grained analysis of cfa artifacts," IEEE Trunsactions on Information Forensics and Security, vol. 7, no. 5. pp. 1566-1577, 2012.
[26] B. Mahdian and S. Saic, "Using noise inconsistencies for blind image forensics." Image and vision computing, vol. 27, no. 10, pp. 1497-1503, 2009.
[27] M. K. Johnson and H. Farid, "Exposing digital forgeries by detecting inconsistencies in lighting." " in Proceedings of the 7th workshop Multimedia and security, 2005, pp. 1-10.
11

---

[28] Q. Liu, X. Cao, C. Deng, and X. Guo, "Identifying image composites through shadow matte consisteney." IEEE Transactions on Information Forensies and Security. vol. 6, no, 3, pp. 1111-1122, 2011.
[29] E. Wengrowski, Z. H. Sun, and A. Hoogs, "Reflection correspondence for exposing photograph manipulation," in IEEE International Conference on Image Processing, 2017, pp. 4317-4321.
[30] F. Guillaro, D. Cozzolino, A. Sud, N. Dufour, and L. Verdoliva, "Trufor: Leveraging all-round clues for trustworthy image forgery detection and localization," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20 1000-20000 615.
[3I] Li, W. Ma, J. Guo, S. Xu, B. Li, and X. Zhang, "Unionformer: Uniffed-leamiing transformer with multi-view representation for image manipulation detection and localization," in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 12533.
[32] Wang, Z. Wu, Chen, X. Han, A. Shrivastava, S.-N. Lim, and Y.-G. Jiang. "Objectformer for image manipulation detection and localization." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2364-2373.
[33] 7. Xu, X. Zhang, R. Li, Z. Tang, Q. Huang, and J. Zhang, "Fakeshield: Explainable image forgery detection and localization via multi-modal large language models," in International Conference on Learning Representations, 2025.
[34] C. Qu, Y. Zhong, C. Liu, G. Xu, D. Peng, F. Guo, and L. Jin, "Towards modern image manipulation localization: A large-scale dataset and novel methods." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 10 781-10 790.
[35] J. Dong, W. Wang, and T. Tan, "Caisa image tampering detection evaluation database," in IEEE China Summil and International Conference on Signal and Information Processing, 2013, pp. 422-426.
[36] G. Mahfoudi, B. Tajini, F. Retraint, F. Morain-Nicolis J. L. Dugelay, and P. Marc, "Defacto: Image and face manipulation dataset," in 27th European Signal Processing Conference, 2019, pp. 1-5.
[37] S. Jia, M. Huang, 7. Zhou, Y. Ju, J. Cai, and S. Lyu, "Autosplice: A text-prompt manipulated image dataset for media forensics." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 893-903.
[38] W. Wang, J. Dong, and T. Tan, "A survey of passive image tampering detection," in Digital Watermarking: 8th International Workshop, 2009, pp. 308-322.
[39] L. Zheng, Y. Zhang, and V. L. Thing, "A survey on image tampering and its detection in real-world photos," Journal of Visual Communication and Image Representation, vol. 58, pp. 380-399, 2019.
[40] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., "Chain-of- thought prompting elicits reasoning in large language models," Advances in neural information processing systems, vol. 35, pp. 24 24 837, 2022.
[41] S. Yao, D. Yu, Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, "Tree of thoughts Deliberate problem solving with large language models," Advances in Neural Information Processing Systems, vol. 36, 2024.
[42] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., "Segment anything." in Proceedings of the IEEE/CVF International Conference Computer Vision, 2023, pp. 4025-4026.
[43] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson et al., "Sam 2: Segment anything in images and videos," arXiv preprint arXiv:2408.00714 2024.
[44] L. Yang, B. Kang. Z. Huang, X. Xu, J. Feng, and H. Zhao, "Depth anything: Unleashing the power of large-scale unlabeled data," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024, pp. 10077110381
[45] I.. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, Zhao, and D. Lin, "Sharegpt4v: Improving large multi--odal models with better captions," arXiv preprint arXiv:231 2023.
12

---

[46] E. J. Hu, P. Wallis, Z. Allen- Zhu, Y. Li, S. Wang, L. Wang, W. Chen et al., "Lora: Low-rank adaptation of large language models," in International Conference on Learning Representations. 2022.
[47] B. Bayar and M. C. Stamm, "Constrained convolutional neural networks: A new approach towards general purpose image manipulation detection," IEFE Transactions on Information Forensics and Security, vol. 13, no. 11, pp. 2691-2706, 2018
[48] K. He, X. Zhang, S. Ren, and Sun, "Deep residual learning for image recognition," " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, 1, pp. 770-778.
[49] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," in European conference on computer vision, 2020, pp. 213-229.
[50] B. Cheng, A. Schwing, and A. Kirillov, "Per-pixel classification is not all you need for semantic segmentation," Advances in neural information processing systems, vol. 34, pp. 17 864-178775 875, 2021.
[51] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba, "Semantic understanding of scenes through the ade20k dataset," International Journal of Computer Vision, vol. 127, pp. 302-321, 2019.
[52] H. Caesar, Uijlings, and V. Ferrari, "Coco-stuff: Thing and stuff classes in context," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018,
pp. 1209-1218.
[53] X. Liu, Y. Liu, J. Chen, and X. Liu, "Pscc-net: Progressive spatio-channel correlation network for image manipulation detection and localization." IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 11, pp. 7505-7517, 2022.
[54] M.-J. Kwon, S.-H. Nam, I.-J. Yu, H.-K. Lee, and C. Kim, "Learning jpeg compression artifacts for image manipulation detection and localization," International Journal of Computer Vision, vol. 130, no. 8. pp. 1875-1892 2022.
[55] Y.-F. Hsu and S.-F. Chang, "Detecting image splicing using geometry invariants and camera characteristics consistency," in IEEE International Conference on Multimedia and Expo, 2006,
pp. 549-559522 552.
[56] B. Wen, Y. Zhu, R. Subramanian, T.-T. Ng. X. Shen, and S. Winkler, "Coverage- a novel database for copy-move forgery detection," in IEEE international conference image process- ing, 2016, pp. 161-165.
[57] H. Guan, M. Kozak, E. Robertson, Y. Lee, A. N. Yates, A. Delgado, D. Zhou, T. Kheyrkhah, J. Smith, and J. Fiscus, "Mfc datasets: Large-scale benchmark datasets for media forensic challenge evaluation," " in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, 2019, pp. 63-72.
[58] A. Novozamsky, B. Mahdian, and S. Saic, "Imd2020: A large-scale annotated dataset tailored for detecting manipulated images," " in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, 2020, pp. 71-80.
[59] N. Reimers and Gurevych, "Sentence-bert: Sentence embeddings using siamese bert- networks," in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
[60] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez. I. Stoica, and E. P. Xing, "Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality." March 2023. [Online]. Available:
[61] P. Wang, S. Bai, S. Tan, S. Wang. Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge et al., "Qwen2-vl: Enhancing languaaee model's perception of the world at any resolution," arXiv preprini arXiv:2409. 12191, 2024.
[62] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He, "Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters," in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp- 3505-3506.
13

---

[63] I. Loshchilov, "Decoupled weight decay regularization," arXiv preprint arXiv: 171100100,
2017.
14

---

A AnalsryAnalysis Dataset
A.1 ForgeryAnalyee Data
We utilize the advanced M-LLM, GPT-4o, to generate the initial clue analyses, carefully designing prompts to ensure GPT-4o provides accurate and detailed responses. First, we inform GPT-4o of its role as an assistant, clearly outlining the levels of clues, along with specific examples, and specifying the task it needs to complete. The detailed prompt is as follows:
[ROLE] You are a rigorous and responsible image tampering (altering) detection expert. You can detect whether image has been tampered with, localize the exact tampered region, and analyze your detection decision according to tampering clues of different levels. These clues include but are not limited low-level pixel statistics (such as noise, color, textural, sharpness, and Al-generation fingerprint), middle-level visual defects (such as traces of tampered region or boundary, lighting inconsistency, perspective relationships, and physical constraints), and high-level semanticc anomalies (such as content contrary to common sense, inciting and misleading content), etc. Altering operations could be divided into types, including "splice", "copy-move", "remove". and "Al-generate", leaving different clues that you should consider.
[TASK] Now, your task is provide analysis. Please note that in real detection scenarios, you cannot know in advance whether an image has been tampered with and the specific tampered region. However, now will tell you this information to help you conduct more rigorous and accurate analysis based on this. There is no need to include all aspects and views in your analysis, give some of your most confident points.
The details of each conversation round are illustrated in Figure 12. In each round, we provide two images: the tampered image to be analyzed and reference image with the tampered region highlighted. The prompt includes the <FAKE> token to indicate that the image is manipulated, specific tampering types to help the model focus on relevant clues, and structured clue analysis format. Based on the response of GPT-4o, experts then conduct further revisions. The experts carefully review vague and incorrect statements that may arise from hallucinations, removing irrelevant clues and evidence. They also reorganize the clues into the Chain-of-Clues structure, which guides the reasoning process. This begins with "unveiling doubts" using high-level clues and continues with "pinpointing evidence" using middle-level and low-level features. The experts also check special tokens, such as <FAKE> and [SEG], to meet the requirements for subsequent model training.
A.2 Forgery Analyst Engine
It is worth noting that the M-LLMs used in the data engine ForgeryAnalyss and the detection frame- work ForgerySleuth are different and independent. Although both M-LLMs are designed analyze forgery and produce text-based clue analyses, their inputs and tasks are distinct. ForgeryAAnalya receives input that includes explicit information about the tampered region (highlighted to indicate tampering) with the goal of generating more precise and comprehensive clue analyses to construct pre-training data. In contrast, ForgerySleuth takes an image to be analyzed, aiming for the M-LLM to identify high-level semantic anomalies for detecting tampered regions. Furthermore, we design different prompts tailored to these two specific tasks. our experiments, Forgery Analyst employs LLaVA-v1.5-133. while the MLLM in ForgerySleuth uses LLaVA-7B-vi balancing performance and efficiency.
We also designed a dedicated prompt for the data engine. In addition to indicating the image type as <FAKE> and specifying the tampering type [MANIPULATION-TYPS we provide detailed instructions and examples of the Chain-of-Cluc (CoC), as well as the required output data format. The specific prompt format as follows:
You are rigorous and responsible image tampering (altering) detection expert. You can localize the exact tampered region and analyze your detection decision according to tampering clues at different levels. Assuming that you have detected this is <FAKE> image and the manipulation type [MANIPULATIONTTPE)] the exact tampered region boundary is highlighted with color in this image (and your detection IS correct).
15

---

Table Statistic our ForgeryAnalyse Dataset.
Dataset
Split
Count
Sum
ForgeryAnalyste Analysis-Eval
618
PorgeryAnalysia
PorgeryAnnlysis-SFT
1752
2,370
FurgeryAnalysis- An
50K
50k
Table 8: Overview of public dataset utilized in the construction of ForgeryAnalyse and for evaluation.
Dataset
Usage
Authentic
Tampered
Manipulation Types
Source
MIML [34]
Train. Construct
11,142
123.150
Manual Editing
PSBattles
CASTA2 [35]
Train, Construct
7,491
5,063
Splice, Copy-move
Corel
DEFACTO [36]
Train, Construct
149,587
Splice, Copy-move, AIGC (GAN)
MS COCO
AutoSplice [37]
Train, Construct
2,273
3,621
AIGC
Visual News
Columbia [55]
Eval
183
180
Splice
Coverage [56]
Eval
100
100
Copy-move
CASIA1 [35]
Eval
800
920
Splice, Copy-move
Corel
NIST16 [57]
Eval
564
Splice, Copy-move
IMD20 [58]
Eval
414
2,010
Manual Editing
COCOGlide [30]
Eval
512
AIGC (Diffusion)
Please provide chain-of-clues supporting your detection decision the following style: high- level semantic anomalies (such content contrary to common sense, inciting and misleading content), middle-level visual defects (such as traces tampered region or boundary, lighting inconsistency, perspective relationships, and physical constraints) and # low-level pixel statistics (such as noise, color, textural, sharpness, and Al-generation fingerprint), where the high-level anomalies are significant doubts worth attention, and the middle-level and low-level findings are reliable evidence.
A.3 Statistics
Table presents the data statistics of the ForgeryAnalysi dataset. ForgeryAnalysis-Eval and are initially generated by GPT-4o and fully revised by experts. They are used for evaluating the quality of manipulation analysis generated by the M-LLMs and for the final supervised fine-tuning, respectively. proposed data engine, ForgeryAnalyss maintaining consistency in data format with the other subsets. Table 8 provides detailed statisties the scale, source, and manipulation types from each public dataset utilized in the construction of Forgery Analysis and for evaluation.
is automatically constructed by our
A.4 Cases
Figures 13, 14, 15 and 16 present examples of analysis texts from Forgery Analysis-Fval for four different tampering types. These diverse cases highlight the variations in detectable clues across different tampering types and illustrate how varying levels of clues support manipulation detection. The analyses in Forgery Analysi--vall are cross-checked by multiple experts to ensure comprehensive and accurate clues. Figure 17 shows analysis texts in Anallsisnn PT. Although the subset is automatically generated by the data engine, it also provides precise descriptions and analyses of the tampered regions.
B ForgerySleuth
B.1 Framework Details
Trace Encoder. Considering that the vision backbone (ViT-H SAM backbone) pre-trained on tasks highly correlated with semantics and remains frozen during the fine-tuning of our framework, the semantic-agnostic features widely used IMD tasks [8, 9, 10, 11] cannot be effectively leveraged. We propose independent trace encoder Ft, equipped with a noise enhancement module to focus on low-level features and provide more reliable tampering evidence. Specifically, the noise enhancement module, positioned at the front of the encoder, uses constrained convolutions [47] to compute local
16

---

Table 9: Manipulation localization results comparing ForgerySleuth with SoTA methods.
Optimal Threshold
Fixed Threshold F1 (0.5)
Method
Columbia
Coverage
CASIA1
NIST16 COcoGlide
Columbia
Coverage
CASIAI
NIST16 COCOGlide
Mantra-Net [8]
0.650
0,486
0.320
0.225
0.673
0.508
0.317
0.180
0.172
0.516
SPAN [9]
0.873
0.428
0.169
0.363
0.350
0.759
0.235
0.112
0.228
0.298
MVSS-Net [10]
0.781
0,659
0,650
0.372
0.642
0.729
0.514
0.528
0.320
0.486
PSCC-Net [53]
0.760
0.615
0.670
0.210
0.685
0.604
0.473
0.520
0.113
0.515
CAT-Ner22 [54]
0.923
0,582
0.852
0.417
0,603
0.859
0.381
0.752
0.308
0.434
TruFor (30]
0.914
0.735
0.822
0.470
0.720
0.859
0.600
0.737
0.399
0.523
UnionFor. [31]
0.925
0.720
0.863
0.489
0.742
0.861
0.592
0.760
0.413
0.536
FakeShield [33]
0.306
0,085
0.620
0.119
0.659
0.285
0.052
0.566
0.099
0.536
onlyy TP samples
0.874
0.470
0.696
0.514
0.659
0.813
0.289
0.635
0.431
0.536
ForgerySleuth
0,931
0.792
0.870
0.610
0.751
0.925
0.684
0.804
0.518
0.562
Table 10: Manipulation localization results of ForgerySleuth and SoTA methods, using pixel-level AUC as the evaluation metric.
Method
Columbia
Coverage
CASIA1
NIST16
IMD22
Mantra-Net [8]
0.824
0.819
0.817
0.795
0.748
SPAN
0.936
0,922
0.797
0.840
0.750
PSCC-Nel [53]
0,982
0.847
0.829
0.855
0.806
ObjectFor, [32]
0,955
0.928
0.843
0.872
0.821
TruFFor [30]
0.947
0.925
0.957
0.877
UnionFor. [31]
0.989
0.945
0.972
0.881
0.860
FakeShield [33]
0.787
0.185
0.780
only TP samples
0.924
0.761
0.883
0.801
0.868
ForgerySleuth
0.992
0.962
0.969
0.898
0.911
differences, extract noise features, and suppress image content. The convolution kernel constraints defined as follows:
((.0) = 1, ((m,)) = 0,
(9)
(m,n)
where n) denotes the spatial index of the values within the convolution kernel, with (0, 0) positioned at the center. The constrained convolutions are still trainable, allowing them to learn manipulation features more adaptively than fully fixed- perd-paraetter noise extractors. The extracted noise features further enhance the original features by residual connections. The encoder employs ViT-B architecture and all parameters are fine-tuned during training.
In summary, we employ the trace encoder F, to capture low-level manipulation features, leveraging constrained convolutions within the noise enhancement module achieve this. Meanwhile, the vision backbone F. utilizes pre-trained parameters from SAM, which is widely recognized for its ability capture dense visual content features, thereby supporting more precise segmentation.
Fusion Mechanism. To integrate the image content embedding for trace embedding f,, and the L.M output tokens hagg obtained from the M-LLM, we take inspiration from Transformer segmentation models [49, 50, 42] and design a vision decoder with fusion attention mechanism, illustrated in Figure (b). The mechanism consists of three layers, with each layer performing four steps: self-attention on the LLM output tokens upgraded tokens, gross-atteniin to the trace or image embeddings, point-wise MLP, and cross-are attos-tnttntion from the trace or image embeddings (as queries) back to the tokens. The first layer of the module computes attention between anomalies the L.M output tokens and traces the trace embeddings, facilitating the organization and pinpointing of clues. The subsequent layers focus on attention between refined clues in the upgraded tokens and content in the image embeddings, enabling more precise segmentation of the tampered regions.
from tokens (as queries)
17

---

C
Comparison with FakeShield
C.1
Details
FakeShield [33] is a concurrent work with the same motivation ours. proposes multimodal large model for image manipulation detection. This further reinforces the significance of providing reasonable forgery analysis, alleviating the explainability issues in existing image manipulation detection methods. However, there are some fundamental differences between FakeShield and our approach.
From a data perspective, FakeShield relies on GPT-4o to generate tampering analysis texts, construct- ing managg-mask-deserrin
triplets for training and evaluation. However, this process presents certain challenges: First, the generated texts lack structured reasoning, as FakeShield does not impose explicit guidance how the detected clues is organized and analysis data is formulated. Second, reliance on GPT-4o alone makes data prone to hallucinations, potentially introducing unreliable or misleading explanations.
In contrast, we propose a three-level Chain-of-Clues (CoC) structure, which enables a more structured and interpretable analysis. Our design is based two key insights: 1) Previous research [38, 39] suggests that forensic clues are naturally organized hierarchically. 2) Studies on Chain-of- Thought (CoT) prompting [40, 41] have demonstrated its effectiveness in improving reasoning within large language models. Inspired by these findings, we structure the tampering analysis data into a progressive reasoning process, encouraging the model identify, refine, and interpret forensic clues at different levels. Additionally, to reduce hallucinations and improve reliability, we manually verify the generated analysis texts and expand the dataset using additional data engine.
From a model perspective, FakeShield employs a multimodal forgery localization module (MFLM) that leverages the segment anything model (SAM) and large language model to localize tampered regions based on their general vision and language features. However, recent studies [10. 54, 11] highlight the crucial role of low-level noise features in image manipulation detection, while pre- trained models for general vision tasks primarily capture high-level semantic features. To address this challenge, introduce a trace encoder with noise enhancement to compensate for the limitations of the base vision model. In addition, we design a dedicated fusion mechanism to better integrate multiple features. Ablation studies in Table 6 further validate the effectiveness of these components.
In the following experiments, we comprehensively compare FakeShield and our method, ForgerySleuth, demonstrating that ForgerySleuth consistently achieves superior performance in various evaluation aspects.
C.2 Evaluation
Experimental Setting. We reproduce the FakeShield model using its publicly released official code repository and pre-trained weights. The following comparative experiments adopt the same experimental settings and evaluation metrics as described in Section 5.1.
We follow established practices [31] by using optimal threshold and fixed threshold Fl scores and the cheshhold-nndependent Area Under the Curve (AUC) metric. Inspired by prior work [60, 14], we use GPT-4 as an automated evaluator to assess the reasoning performance of different models. To address concerns regarding potential limitations of GPT-4, we incorporate an additional metric, semantic textual similarity (STS). Specifically, following SBERT [59]. we use STS to measure the similarity between the generated analysis and the ground-trum -tuth text in Forgery Analysis-
Manipulation Detection Results. Tables 9 and 10 present the performance of ForgerySleuthh FakeShield, and other SoTA methods in the manipulation localization task, reporting Fl scores and AUC metrics, respectively. It essential to note that the official implementation of FakeShield follows a two-stage pipeline. First, it determines whether an image has been manipulated. Only if an image is classified fake does it proceed to generate tampering mask. In contrast, ForgerySleuth and other detection methods output mask for all images while also providing a real/fake classification result.
We adopt the following strategy for FakeShield to maintain consistency with other methods and ensure a fair comparison. FakeShield classifies an image as real, we assume that the predicted mask is all-zero matrix (i.e., indicating no tampered region). The corresponding results
18

---

Table 11: Forgery analysis results of ForgerySleuth and SoTA methods, evaluated using GPT-4 on AorgeryAnallysis-Evall dataset.
Method
Recall
Correctness
Relevance
Detail
Overall
LISA [12]
2.27
2.51
1.12
1.92
QWen
43.9%
3.22
2.92
3,10
3.41
QWen2 [61]
29.6%%
3.70
4.14
2.57
3.34
GPT-40
68.8%
7.14
7.97
5.84
6.96
FakeShicld [33]
52.8%
5.91
6.35
4.67
5.55
ForgerySleuth
92.2%
9.10
9.60
9.89
9.45
Table 12: Forgery analysis results of ForgerySleuth and SoTA methods, evaluated using STS.
Model
Ours
FakeShield
GPT-4o
QWen2
QWen
LISA
all-MiniLM--L--1
0.926
0.662
0.725
0.595
0.551
0.313
All-MiniLM-LI2----
0.919
0.514
0.645
0.505
0,475
0.350
all-mppett-usee-22
0.961
0.695
0.724
0.635
0.546
0.401
shown in the FakeShield row. The notably lower accuracy stems from the low recall rates, where FakeShield fails to detect manipulations, leading to significant missed detections. To further analyze FakeShield's localization performance, also report its performance detected manipulated images (i.e., True Positive samples). As indicated in the row for only true positive samples, its performance improves significantly under this setting, but it still falls short compared to ForgerySleuth. However, since manipulation detection is performed without prior knowledge of whether image manipulated, the first setting (which considers all images) provides a more realistic and fair evaluation, while the second setting serves only as a supplementary analysis.
only the successfully
Forgery Analysis Results. We employ GPT-4 evaluator on the Anallss-n dataset to assess the quality of textual analysis and reasoning. The evaluation considers several key dimensions: correctness, relevance, and detail. The results presented in Table 11 (corresponding Figure 1 (b)). FakeShield exhibits low recall rates, further corroborating the previously discussed issue of missed detections. Additionally, its analysis falls behind Forgery Sleuth in both accuracy and level of detail.
However, since GPT-4 itself is susceptible to hallucinations, this approach does not guarantee a fully objective and quantitative evaluation. To address this limitation, we incorporate Semantic Textual Similarity (STS) to measure the similarity between the generated text and the ground-truth text in AnaggeryAnalyiis-Baaa Eval. The results, shown in Table 12, further confirm that our ForgerySleuth consistently outperforms FakeShield.
Qualitative Results. We present the performance of ForgerySleuth and FakeShield in manipulation localization and forgery analysis. shown in Figure 8. The results demonstrate that ForgerySleuth not only achieves higher accuracy in localizing manipulated regions, but also generates more detailed and precise analysis.
D Experiment
D.1 Experimental Implementation Details
Implementation Details. We employ LLaVA-7B-vI-1I [14] the base multimodal LLM (F,,,) and use the ViT-H SAM [42] backbone for the vision encoder (F.)) For training, we utilize 2 NVIDIA 80GB A800 GPUs, with training scripts optimized by DeepSpeed [62], which helps reduce memory usage and accelerate training. We use the AdamW W [63] optimizer, setting the learning rate to 0.0002 with no weight decay. The learning rate scheduled using WarmupDecayL.R with 100 warmup iterations. The weights for the text generation loss Amx and mask loss Amnk are both set to 1.0, while the BCE loss Aree and DICE loss Amice are weighted at 1.0 and 0.2, respectively. The batch size per device is 4, with gradient accumulation steps set to 4.
19

---

Tampered
image manipulation region is [SEG] The interaction betwean the tiger and the bird is highly unusual. natural setting, tigers apex predators...
Sure, this <REAL> image with modifications detected. [SEG] used output
Authentic
Image without tampered region. [SEG] is used to output mask.
Figure 6: Predictions of ForgerySleuth on authentic and tampered images.
Table 13:
Human evaluation results for the forgery analysis task.
Method Samples
Recall
Correctness
Relevance
Detail
Overall
QWen2
139
35.3%
4.95
5.10
2.85
4.11
GPT-4o
65.7%
8.06
8.32
7.18
Ours 147
91.2%
9.48
9.53
9.61
9.44
D.2 Manipulation Detection Results
Modeling the natural laws of real samples and distinguishing them from tampered images crucial. We utilized two special tokens, <REAL> and <FAKE>, which indicate the detection results. In addition, the output mask provides valuable information for verifying image authenticity. Figure 6 illustrates the ForgerySleuth detection results for authentic and manipulated images, including the predicted tampering masks and textual analysis. Our method achieves real/fake classification accuracies of 0.989 on Columbia and 0.910 CASIAI, demonstrating its effectiveness in manipulation detection.
D.3 Forgery Analysis Results
We leverage GPT-4 for evaluation based on the Forgery Analysis-Ayssseeve dataset to assess the quality of text reasoning and explanations. Ratings are based on several dimensions, including the correctness of tampered objects, the relevance of clues manipulation, and the detail of analysis, reflecting capability of comprehension, reasoning, and correctness. We collect responses from each M- LLM, and GPT-4 assigns scores from 1 to 10 for each response, with higher scores indicating better performance. Beyond scoring, GPT-4 provides explanations for its ratings, ensuring transparency and consistency in the evaluation process. Figure 18 illustrates the prompt structure used for the evaluation and the response GPT-4. To ensure consistent and fair scoring, the evaluation prompt includes clear scoring criteria for assessment aspects, including correctness, relevance, and details. GPT-4 assigns a score for each evaluation dimension and provides detailed comments to justify the rating.
To directly assess the ability to detect manipulated images, we explicitly instruct the evaluated M-LLMs through prompts to additionally output <FAKE> <REAL> to indicate their detection results. models, except SSS--7B-v--explnnnte
can provide the required response. We use the recall rate reflect the ability to correctly identify tampered images.
Human Evaluations. also conduct human evaluations under the same experimental setup as evaluation using GPT-4 described above. A total of 14 volunteers participated, each scoring for 30 random samples. The results are summarized in Table 13.
D.4 Qualitative Results
We present additional results of our ForgerySleuth in both forgery analysis and manipulation localiza- tion tasks. We also provide forgery analysis results from existing M-LLMs (GPT-4o and Qwen-VL)
20

---

This image is <FAKE>... The presence of small animals, specifically cats, wearing helmets and space uncommonn.. highly unrealistic tampered objects lighting (cats) is inconsistent with the..
Tampered
This Image is <FAKE>... The altered section features individual holding a distinctly shaped or sized onject that does not align with scene's context expected object's atypical...
setting.
Figure 7: Failure cases in predictions of ForgerySleuth.
and segmentation masks from traditional image manipulation detection (IMD) methods, illustrating the advantages of our proposed IMD assistant in terms of accuracy and explainability. These cases are from public datasets IMD20 and NIST16, demonstrating the generalization capabilities of our method.
In the example in Figure 9, both M-LLMs classify the image as real without detecting manipulation. Similarly, most IMD methods, except CAT-Net2, fail to localize the manipulated regions accurately, However, ForgerySleuth identifies the tampered regions and provided detailed tampering analysis. Figure 10 presents a case with more apparent tampering traces. GPT-4o and Qwen-VL both exhibit varying degrees of overthinking, leading to inaccurate analysis. Our method demonstrates higher localization precision and analytical accuracy. In the case of Figure 11, ForgerySleuth precisely localizes the tampered regions, even including the shadow of the person. Compared the textual outputs of M-LLMs, the mask generated by ForgerySleuth provides a more intuitive and accurate detection result.
E
Limitation and Future Work
E.1
Failure Cases
We analyzed Forgery 's detection failures and identified two representative error patterns.
Semantic Description Inaccuracies. In these cases, our model can accurately localize manipulated regions, but its textual analysis contains description errors. For instance, in the first example in Figure 7, the model misidentifies dogs as cats. Importantly, this does not impact the detection logic or localization accuracy. These errors often arise when specific semantic concepts in images are challenging to detect clearly. Future work could improve this by enhancing the visual understanding capabilities of MILM and LLM backbone models.
Failures with Small Manipulations in Complex Scenes. In complex image content with small manipulated regions, the model's localization precision can degrade, sometimes leading to incorrect textual analysis and, consequently, a detection failure. For example, in the second instance in Figure where a spliced cartoon character present, the model only localized a partial manipulation (e.g., the guitar in hand) while overlooking the character itself, primarily due to the small proportion of the manipulation and the image's overall complexity. Detecting small region manipulations remains significant challenge, and MLLMs tend to focus on global semantic information rather than fine- grained local details. Therefore, developing methods to effectively detect anomalies in small regions crucial research direction.
E.2 Constraints in Evaluation Methods
Our paper evaluates analysis accuracy using GPT-4o and STS methods on the ForgeryAnalysis-eva dataset. However, our evaluation methods are subject to specific constraints: our assessment relies on GPT-40, a large external model that may exhibit inherent biases or hallucinations, and STS only reflects semantic similarity between two sentences. Thus, exploring more comprehensive and objective evaluation methods, including the quantitative assessment of model hallucinations, an important direction for future work.
21

---

E.3 Model Scale and Inference Costs
While leveraging LLMs to complement traditional image manipulation detection methods, which often primarily focus on low-level traces, this approach increases the model's scale and computational cost of generating detailed explanations. In the future, we plan to investigate model light- weighthigg solutions. This includes developing version where generating such detailed output is optional, which would significantly reduce inference time. Alternatively, we could explore distilling L.M capabilities into smaller models, retaining only the essential knowledge for manipulation detection.
22

---

1. Whether the picture has been tampered with Description the tempered area: The picture has not been tempered with. It appears to be genuine photograph taken directly by camera.
Judoment basis: The lighting and shadows in the image are cansistent, with the light source appearing to come from above, casting natural shadows under the fish and on the person's arm, The edges of the objects, such as the fish, the person's hand, and the kayak, are smooth and show no signs of unnatural pixal distribution or editing. The resalution across the image is uniform, without any areas of alurring or jaggedness that would auggest manipulztion. The perspactive and scale relationships within the image are realistic; the fish is progortionete to the person's hand and the kayak, end the background elements, such as the trees and water, align with the expected depth of field. The reflections on the water are appropriate for the lighting conditions and the ripples in the water suggest movement consistent with the kayak's artivity. There are no signs of unnatural text or lngos, and the content of the picture does not violate any physical lows. The image's elements interact with each other in physically plausible menner.
This <FAXE> Image with modifications detected in the [SEG). We have identiited the tollowing clues, where high-level anomallns are doubts, and lewer-levels clues are evidence.
(a)
1. Content Contrary to Common Sense: The image shows person's bady with a dog's facc, which is anatomicalyy impossible. This unnetural combination lacks logical coherence real-world scenarios, confirming manipulation.
High-LLvel Semantic Anomalies:
2. Mistrading Content: evident that this does not depict reality, as hiologically, human body cannot have dog's head. The content misleacing deception.
unnsturrll
intended for humer
[SEG]
Middle-Level Visual Defects
1. Boundery Artifacts: The boundary around the region shows irreguenities and unnatural edges where the dog's face meets the human neck. These artifacts result from poor blending techniques in image menipuletion.
Perspective: There is diserepency in the perspective and preportions of the dog's face compared to the human body. The size and angle of the face do not align correctly with the body, further confirming tampering.
Loweeevel Pixel Statistics
1. Color Inconeistency: The color tone and balance of the dog's face do not match the sión tone and lighting an the human body and
arm, indicating manipulation.
2. Textural and Sharpness: The texture on the dog's face lacks the same sharpness and detail level as seen on the person's body. The dog's face appoars smoother or grainier than the human body, suggesting that the face was pasted onto the original image.
1. Whether the picture been lampered with Description of the tampered area: The picture has been tampered with. The tampered area is ocated on the lower right side of the image, near the bettom edge. and is edjacent to the person's right side. appears to be dog lying on the ground.
(b)
2. Judgment basis: The shedows cest by the dog do not metch the lighting in the rest the image. The doe's s shadow is inconsistent with the direction and quality of the shadows seen elsewhere in the photo, suggesting that the dog may have been adced into the scene, Additionally, the edges around the dog are slighdy blurred and do not blend seamlessly with the ground, indicating a possible compesite. The dog's fur also scoms to have a different resolution compared to the rest of the image, appearing slightly more pixclated which could be a sign of tampering.
[SEG]
This image is <FAKE>, and the tampered region is [SEG). Through our analysis, we have discovered several key indicators of tampering. High level somantic anamolies raise serious concems, while middle- level and low level findings provide solid evicence. High-Level Semantic Anomalies:
(b)
The combination of the hat, hand-heid object, and the entire setting creates deceptive messages, implying attempt manipulate the viewer or change the context of the original picture.
a Misleading or Inciting Content:
of specific ieems, such as the uniquely designed het and the object resembling gun, incongruous otherwise ordinery digital atteration.
# Content Contrary to Common Sense: The
[SEG]
suggesting
Middlee-Lvel Visual
the object Lighting Inconsistency: components hermonice
competible exhibit lighting lighting direction.
rest of scene.
shedows reflections
# Perspective Relationships: temm appear hurriedly inserted, disregarding altored areas, For example, the edges the hat the object being held unnaturally crisp and fall to bland smoothly with their surroundings.
plecement and dimensions scene's natural depth and
het and hand-held object not align with scene's
perspective.
#2 Traces Tampered Region: Discemible lines
Low-Level Statistics
Noise Inconsistency: The sections with the gun-like object, the har, and the item near the dog reveall noise pattern discrepancies
when compared adjacent regions, indicating tampering.
Discrepancy: shades attered areas slightly differ from their suroonndings, particularty evident the and item held by the individuel.
Figure 8: Comparison of results between ForgerySleuth and FakeShield
23

---

The image appears to bo real, doakting a scone from roleer darby event. The participents are wearing protective gear such as helmets, knoe pads, and elbow pads, which is typical for the sport, The expressions and body language of the individua's suggest they are actively engaged in the game, There are na obvious signs of digtal manipulstion, such as incondistent lighting, shacows, or distorted proportions that would indicate the image has been altered. Additionally the background shows spectators watching the event, adding to the authenticity of the scene.
several throughout Image. Shadows highlights align assumed light rughting Shadows The lighting sources.
image appears authentic
**Proportions Perspective*:: indicate manipulation. inconslstencles
subjects' proportions perspective
satting.
notiorable
angle
"Blending and Edges** **; The edges of the subjects are cleen end well-defired with no visible signs cut-and-peste operetions, such blurred overly sharp ouilines.
EEvirrnmenn Interaction The people the scene interact naturally with the roller sicetes' contect with the floor and the interection between individuals are logical and cohesive. Overall, the elements the image work together harmoniousiy, suggesting it is not manipulated.
surroundings. Detail
This <FAKE> image with modifications detected in the [SEG). We have found several dlues, where high-level anomalies create substantial doubts, and other-level findings contribute reliable evidence.
High-Level Semantic Anomalies
Conteent Contrary Common Sense**:
of
clothes amidst of roller derby
unusuel, which
the middle the action with
Middie-LLval Visual Defacts
[SEG]
**Lghting Inconeistency**
align lighting conditions surrouncinng environment. shadows highlights on amateur player not expected lighting direction and intensity osserved
lighting tempered figure
derby participents.
**Perspecive Relationships**:
player alightly misaligned perspective match naturally with the sumounding players, indicating figure was not amateur player
group. posittoning relative
original scone.
**Physical Constraints"
other players awkward. referee the surrounding skaters, suggesting that the amateur player was added without considering the physical dynamics the scene.
amateur player's interaction
natural contact or engagement between
Low--Lvvl Pixel Statistics
**Nois Inconsistency**
The tampered region shows different noise pattern compered to the surrounding areas. The ameteur player has smoother texture that does not match the netural noise present in the beckground. **Color Discrependee**
There are subtle differences in color grading between the tampored figure and the rest the image. The amatour player has sión tones and lighting that do not pertectly match the other elements in the scene.
**hharpness Variedions**
The edges of the amatour player are unnaturally sharp compared to the rast of the image. This disparity in sharpress indicates that the rigure was cut and pasted from another source.
Manbra: Net
OSN
MVSS-Net
PSCC-Net
CAT-Ne2
Figure 9: Comparison of results between our ForgerySleuth, existing multimodal large language models, and traditional image manipulation detection methods. The image is c93gg4y_00 0.ppg from the IMD20 dataset.
24

---

The image appears to be manipulated due to several incorsistencies:
1. **Cal's Position The cat is positioned in an unneturel way, with its legs extended upwards as if it were floating or being
thrown into the air. Cats typically do not react like this when placed in water.
2. +*atter Splach+e; The splash of water around the tube does not seem consistent with the position of the boy and the cat
3. Lighhtnngg and Shadows* The lighting end shadows on the cat do not metch these the buy end the tube. This suggests that
the cat may have been edded to the image after the original photo was taken.
4. "Eges and Blurring * There are some areas where the edges the cat appear slightly blurred or out of facus compared to the
rest of the imge, which can be common indicator digital manipulstion.
These factors combined suggest that the image has been altered, most Ikeely by adding the cat to the sceno.
Analyyii of Manipuistion Detection: Semeentic Inconsistency ++:
slide, with the in extremely unneturel posture (especially with its leg raised high), This is highly unikeely real scenerio, particularly given that cats generally avoid water.
The soene depicts child and together
While the infistable tube's yellow surfece blends reasonably well the background, slightly out
lighting characteristics still
suggesting potentall manipuletion. **Lghhtng Physical Consistency*** *+;
*+tat*:; The
particulary ralsed
bright enviranment the
slide. cat lacks
reflective highlights that visible other
"Tnflatable
softened, and sharpness compared barkground edge lighting appears slightly
suggests potantial blanding
">ddge Transition**:
Examining the
tube blending artifacts, perticularly
tube. edges appear overly smoothed, hinting post-pracessing.
Similarly
edge
signs digital manipulation, such inconsistent
reflections mismatched
**Behevioral Logic*e;
participate activity shoving discomfort. demeenor this eoove observations, uttimately judged as ccomposte manipulated (FAKE))
highly implausible
Condusion: pase
the
signs manipuletion. The image
with modifications detected in the (SEG). After thorough examinetion, identified the following clues. High-level enomelies significant doubts requiring attention, while micdie-level low findings provide evidence.
This <FAKE>
High-Level Semantic Anomalles
=*Common Sense Content**; The scanario
highly uniikaly. calmly sitting flotation unmolistic behavior, indicating manipulation.
device
*MMissading Content" image
Intended
amusement,
inherently misleeding.
[SEG]
MivelieLLvel
tampered region does not allgn Ligghting Incondistency*** The lighting lighting direction and interslty.
image. Reflections
water and boy's body inconsistent positioning perspective
"Perppcctvv "elationships**
fletatton device not naturally allgn
human occupant.
cat appears unnaturally positioned, lacking the expected interaction with water surroundings. *hysiccl Constraints** The 628 and position
cal relative to the boy indicate tampering. The cat unnaturally large compared to the boy and pleced an awkword position that defies physical realism.
Low-Leve Pixel Statistics
*Nosee Incomsistency*** **; The noise pettern the tampered region differs from the surrounding arees. The texture the weter and the fictation device eround the cat exhibit difeeentt levels of araininess and noise, indicating manipulation.
"Coolor Discrepency **; The color tones in the tampered region, perticulanty the cat, do not blend seamlessly with the surrounding weter and the boy's skin. This suggests that the cet was added from a different source with different lighting conditions. **sharaness Vartotions* The adges of the tampered region, particularly around the cat's body, appear unnaturally sharp compared to the rest of the image, indicating cut aa-and-ia paste operation.
Mantra-Net
OSN
MVSS-Net
PSCC-Net
CAT-Net2
Figure 10: Comparison of results between our ForgerySleuth. existing multimodal large language models, and traditional image manipulation detection methods. The image c8zfOhg_0, from the IMD20 dataset.
25

---

The image
be manipuleted for several
"Frrregrundd Subject The holding baseball not naturally integrated do not
The lighting, shadows, bottom, someane be playing
overall quelity digitally.
background.
have
artificially. baseball. This mismatch context setting like track with lighting the
> BBackrrunnd Context"
** Lightin and Shadews ": The by the <*image Quality person's image
shadows
the resolution clarity objects noticrable the siightly background.
the position
background.
detalled
factors collectively
image manipulated.
<FAKE>
Analysis:
I. L*Lighting and Shadowst*; The shadow of the person holding the bot does not align well with the other shadows the scone,
indicating possible manipuibtion. The shadow is also very sharp and dark compared to the rest of the scone, which suggests inconsistency.
2. Aligmment and P The penson with the bat does not appear to be integrated smoothly into the environment. The she and angic the figure soem off compared to the surrounding elements, lkk the cars.
3. *Ittegrstion with Beckground**: The person holding the bet appears to be layered on tpp of the existing beckground. The edge
around the Figure looks tou cleen and sharp, lacking the naturel blending seen in real-world images.
*onntes Clarit=*:: + person in baseball uniform standing in car racing environment seems out of context, adding to the suspicion of image manipulation.
This <FAKE> image with modifications deterted in the [SEG]. We have found several clues, where high-level anomelles create substantel doubts, and other-level findings contribute reliable evidence. High-Leve Semantic Anomalies
1. *Conttntt Contrary to Common Senserr; The presence of professional baseball player in full unitorm, standing improbable. suggesting that the image was staged or the player was placed into this environment unnaturally.
parking lot next to sports cars, is contextually ocd. Such a scenario
The image may give the Impression that the athicte is involved in supports particular brand, which could be considered advertorial content without dear discicsure. Middlee-Lave Level Visual Defects
*MMilcading Contenter:
[SEG]
1. LLighting Inconsisteney**> **; The lighting on the athlete does not align with the lighting direction and intensity of the cars and beckground. Shadows and highlights the ethlete not indicating memipuattion.
naturally the sunlight and shadows cast
<*Bonncary Traces"*; Noticeable
athiete the background indicate blending.
cut-and-paste manipulstion.
These boundary
neturally within the scene's perspective, The sucgesting that barkground, Incicating that Image. athicto
** Perspeciivee Reletionships*** The sthiete standing athicte's
angle aligned Low-Level Statistics
inserted from
**Noise Inconsistency**. The
region differs from
background. athlete's
section different grain
seturation suggesting manipulation.
lighting Discrepency** uniform
scenee incicating tampering. overall lighting tampered region background. color correspond difference sharpness edges athiete are sharp compared rest image, suggesting manipuletion.
surrounding environment.
**
Mantra-Net
OSN
MVSS-Net
PSCC-Net
CAT-Net2
Figure 1I:
Comparison of results between our ForgerySSeethh existing multimodal large language models, and traditional image manipulation detection methods. The image is 6C2016 2_3312__pppp from the NISTI6 dataset.
26

---

Assuming you have detected highlighted with color in another reference image (and your detection correct). Please analyze your detection decision according clues of different levels and aspects including not limited low-leval pivel statisties (such noise, color, textural, sharpness, and Al generation fingerprint,, middle level visual defects (such as traces tampered region or boundary, lighting inconsistency. perspactive relatonships, and physical constraints), and high-level semantic anomalles (such as content contrary to common sense, incting and misleading content), etc. There is no need to include al espects and views in your analysis, give some of your most confident points.
<FAKE> image with "AI-generate" manipulation, and the exact tampered region
image manipulated using Al-genereled content, specifically tergeting bus the center the image, hiahliahted the reference. Below breakdewn of detection dues acress various levels:
Low-Level Statistics:
overall scene, which depicts pest-disester landscape, contains natural graininess, subtle noise, raugh textures consistent with debris and rubbic. However, the bus in the midcle shows different tharacteristics, appears smoother locks the same level noise noise discrepancy a common marker Al-genersted content.
Noise Texture Discrepancles:
sumounndins, indicating that was generated inserted separately. This
Color and Sharpness Anomalles: tones of the devastated environment, The sharpness of the ale does match sightly blurred background, Implying that wes not part of the original scene, This sherpness inconsistency indicates different image source or digital generetion. Middle-Leen Level Visual Defects:
bus unnaturally vivid colors (yollow and black), which stand out in contrast muted
Boundary Traces: The edges of bus appear unnaturally smooth, especially around the rear end and where blends into background. This smoothness in stark contrest to the jagged debri and uneven structures in the environment. The lack of blending or transition between the bus and its surroundings, especially along its bese, highlights its insertion into the scere. Lighting Incorsistency: The overall lighting in the image is eifiused and soft, typical of overcast pust-uisaster conditions, The bus, however, shows inconsistent lighting, with reflective surfaces that suggest stranger light sources that do not exst in the surrounding scene. The shadows on the bus are less pronounced and con't metch the difuueed light in the image, further indicating maniuuatton Perspective and Physical Constraints: The bus sppeers flosting or elevated, which defies the physical laws of gravity and common sense. There is no dear indication of support or ground contact for the bus, suggesting It was artricially placed in this environment. Morcover, the size and angle of the bus do not match the perspective of the background clements, further reinforcing the hypothesis that it was generated and inserted unnaturally.
High-Leve evel Semantic Anomalies
Contrary to Common Sense: From high level contextual perspactive, the bus baing suspended in suth chaotic, disaster- stricken environiennt is illogical. Buses do not floet mid-air, and there are no physical cues in the scene (such as debris lifting or supporting the bus) to explein it's position. This e direct indication of content memiulatioon
Inappropriate Object Placement: In disaster scenario like this, objects like vehicles are usually scattered, damaged, or buried within the debris. The bus, however, appears undamaged and disproportonaaelyy out of place. The placement of the bus in this specific location within the scene is highly unnaturaa and lacks coherence with the overall nerrative of desiruction.
Condusion:
This image has been manipuisted using Al Al-generated content, with the tampered reglon being the floating bus In the middle. The analysis of naise patterns, boundary traces, lighting inconsistencies, and semantic Illogic diearly indicate that the bus was artifiiially Inserted into the scene. This manipulstion disrupts the overall realism of the disaster environment and highlights the telitale signs of Al generated imagery.
tampered region [SEG]. identified the fallowing cluns, <FAKE>, attention, the middle-level low-level findings reliable evidence. signfficant
high- lovel anomalias
Ligh-Level Semantic Anomalies Contrary stricken environment illogical. Buses high-leval contextual perspective, the being suspended such chaotic, disaster- are physical
lifting
supporting the explain Its position.
Ineperogriete Object Placement: disaster scenerio
objects vehicles usuelly scattered, dameged, or buried bus, however, appears undamaged and disproportonnatlly Middla-Lave Visual
place.
Lighting Inconsistency:
overall lighting
image is diffused and soft, typical overcast oost-disester conditions. The bus, sugpest stronger light sources
however, shows incorsistent lighting, with reflective
not exist the surrourding image.
scene. shadows
bus less pronounced and con't match dirtused light bus appears floating or elevated, which defies the physical of gravity common artifiiially placed environment.
Perspective and Physical Constraints:
sense.
dear indication support or ground contact for bus, suggesing Morcover, the anglo the does not match the perspective of background clements. Statistics
Low-Level
Noise and Texture Discrepandles: bus the middle shows different characteristies. It appears smoother and lacks the same level of noise as its surroundings. This noise discrepancy a common marker Al-generated contint.
overall scone contains natural graininess, subtle noise, and rough textures. However,
Color Anomalies: The bus has unnaturally vivid colors, which stand out contrast to muted tones of the devastoted environment.
The sharpness the bus also does not match the slightly blurred background, implying that it was not part of the original scene. This sherpness inconsistency indicetes a different image source or digial generation.
Sharpresss Anomalies:
Figure 12: Illustration of the ForgeryAnalysis dataset creation process. The data is initially generated by GPT-4o, then revised by experts to ensure the accuracy of the analysis, with clues organized in the Chain-of- (CoC) format.
27

---

High-Level Semantic Anomalies:
*Content Contrary Commio Sensete; in the sign semantically unusual unlikely The phrase displayed context. Typically, age restrictions are human years, highllghting the takeness the atteretion.
TOBASAO
Middie-Level Visual Defects:
Tampered Regions Boundaries tampered lotters
siightly abrupt against the background, revealing the typical telltale signa of cut paste operations. odges around the tamporod appear mismatched and inconsistent.
horders
2. L*ighhing Incomsistency
The lighting on the tampered area of the sign window's lower right corner ("UNDER DOG YEARS")
F
does reflect lighting conditions the surrounding environment. natural reflections and shadows do not match the rest of the image.
3. **Perpeccivee Reletionships**
There is slight perspective incompistercy the elignment the tempered letters "R" and "K", The letters do not perfectly follow the perspective lines of the original text's plane.
Low-Leval Pixel Statistics:
1. NNooe and Texture Inconsistency
The tampered text "TOBARKO" shows inconsistencies in noise patterns and textural details when compared with the rest of the image. The non-edited parts have finer texture, while the "R" and "K" characters in "TORARKO" show sighhty different graininess.
2. **Color Dstrependes**:
There subile mismeich in the color tores of the tampered letters compared to the originel cheracters, The tampered letters display a less natural integration with the background color gradient.
3. **Sharpness and Blurring**;
The tampered arees, especially the letters "R" and "k", exhibit different sharpness levels compered to their aumoundings. This incicates they have bean copled and pasted from different source.
CASE I. Semantically abnormal text.
High-Lovel Semantic Anomalies:*
Uncommon Content:* e* The presence of a cartoon character interacting with real-world elements is semantically odd and unlikely, alerting viewers to potential maripulation.
**nntradcction to Physical Realit.** The cartoon character, by nature, does not adhere to real-world physical tonstraints. Its existence and Interaction within the scene defy common logical and physical expectations, such as shedow cesting and 3D interaction with the environimntt.
**Middie-Leve Level Visual Defects:*x
Boundary Traces: x> The boundaries around the cartoon character are not seamless. Upon clse inspaction, there can be visible traces of tampering work, such edge halaing pioel misalignment. Lighhtnng Inconsistency::" The lighting on the cartoon character does not match the lighting conditions of the actual scene. The cartoon's illumination seems uniform, while the real-world elements show natural lighting variations and shadows.
Persppctvvve Isques:** *x The cartoon character is visually inconsistent in perspective alignment with the rest of and depth not coherent with surrounding reel objects, making evident that
the scene, The it digitally inseried.
**Loww LLow-Level Pixel Statistics:**
Noici inconsistenoy:"* The tampered regian exhibits different noise pattern
the character's has smoother textures, the surrounding Image noise granulerit.
character and the world objects incansistances. Specificall,, transition boundary smooth surrounding polette.
Color Anomalies: The blending around intersection
*Sharpness character exhibits differances edge sharpneis. neture
boundary between
image elements inserted edges unnaturally sharp compared organic
scene.
CASE 2. Presence of elements with different styles.
Figure 13: Examples from the Forgery Analysis-AaysssEEva dataset. The data is initially generated by GPT-4o and then cross-revised by multiple experts. The manipulation type for these images is "splice".
28

---

High-Level
same position, having Identical poses damages highly unnatural contrary real-world circumstances. resemblance typically indicator image cloning copy-pasta tampering.
**Duplicated Content: presence two figures sitting
wearing same
depicted breaks general nues common where identiccll individuels appeering as zombie end mirroring eath other's positions highly improbable, confirming suspicions of artificial manipulation
Common Violation
realism,
# Middle-Level Visual Defects:
1. LLLighting incondizzendls:** There is notiosable difeerecce in shadow and lighting. The shadows and highlights on the manipulated person (an the right side) refiect slight inconslstencles compared to the image's rest. 2. speesveevvv Discrepanty:" The perspective of the manipulated figure does not align perfectly with natural physks.
Low-Lovel Pixel Statistics:
1. **Noise Inconsistancy::1 Upon dose inspection, the noise pattern the tampered region slightly different from the rest of the imago.
CASE 3. Duplicuted content.
Contrary Common Sense**:
historical historically inaccurate. anomaly contradicts common understanding hertage
Sphinx, knowledge.
Sphinxes side-by-side
only Great Sphinx exists, making duplicated
anomaly.
Middie-Level
L*ghhtnnn Inconsistencies*
right Sphinx reveals differences shadow lighting compared copied region, hinting inconsisiency.
2. *Perspective Relationships** The proportions and alignment between the pyramid and the tampered Sphinx area exhibit perspective mismetches.
# Low-Leval Pixel Statistics
1. **Noise Petternp**:
Upan dose inspaction, the tampered region exhibits different noise level or pattern compared to the rast the Image.
2. TeTexural Mismatches*+;
Fine details such as erosion patterns on the Sphinx show unnaturally replication.
CASE 4. Contradict common historical knowledge.
Figure 14: Examples from the ForgeryAnallsis--n dataset. The data is initially generated by GPT-4o and then cross-revised by multiple experts. The manipulation type for these images is "copy-move".
29

---

High-Level Semantic Anomalies
1. *CCnttestual Ireeglaritess" A noticeable gep in the snow where tracks should have been continuous or expected ooject that now missing disrupts the logical rlow of the scene. This suggests something unusual and petentially manipuleted.
Middie-Lovel Visual Defects
1. "Boundary Traces""; Subde but identriable bouncaries are present around the tampered area, where the poorly executed removal of onject person.
snow's texture and color slightly mismatch. This
2. **Lgghting Inconsistency"
The lighting and shadows within the tampered region the
align with natural light
evident in the 3. **Perspeciive and Physical Constreints The perspective of the trecks around the tempered that
highighting artificeel modificaioon
appears unnatural. It
the tracks has been removed modified, disrupting the physical inconsistency
flow
envirenment.
Low-Level Pixel Statistics
**Notse Inconsistency**: tampered
noticeably smoother nolse compared balance tampored region appears slight
surrounding lacking the netural rendomness
the
**Calor
compared that
match adjacent
**Sherpness Clarity**; Edges
region image, suggesting integration.
sharpness gradient untouched
CASE Noise inconsistency.
High-Level Semantic Anomalles
**onteent
shadow suggests presence common indicetes manipulation.
dog) should
missing,
Middie-Level
Region//oundery***
unnatural edges, indicating manipulation. traces suggest object removed without seamlessly blending surrounding paxels.
boundary tampered
*ughhtnng Incomsistency
lighting manipulated region, including reflection, does not direction intensity scene, revealing tamporing.
light
*ePrspecttvve "clationships"*:
The reflection in the water suggests object was there, but the size shape the reflection not align with surronnding reflections, indicating image editing.
# Low-Level Pixe! Statistics
1. **Noise Distribution"*
The noise pattern in the tampered region inconsistent with the rest of the image, appearing more smoothed, which indicates manipulstion.
2. *>Color Discontruutt"::
There is color mismatch between the tampered regian and its surroundings, with the tones and hues the reflection not perfectly matching the water and beach.
3. **Sharpness and Bluriness*:;
The boundaries of the tampored ragion are either unnaturally sharp compared to the rest of the image, suggesting poor blending during the editing process.
CASE 6. Unusual shadows and reflections.
Figure 15: Examples from the forgeryAnalysiss dataset. The data is initially generated by GPT-4o and then cross-revised by multiple experts. The manipulation type for these images is "remove".
30

---

High-Level Semantic Anomalies
**Content Sense**:
The flower's young child.
positioning slightly the natural posture and handling
Middie-Level Visual Defects
>-Traces Tampered Region Boundary boundary of the manipulated
distinct and not merge eberretions along edges
tontextuel
original **Ughhing Incomsistency"*
image.
tampered
photo.
lighting the manipulated
neturel lighting child's intensity shadowing tampered inconsisentt conform
scene, suggesting light sources the original Image.
addition
**Physical Constraints
appears unnatural, slight misalignment, further confirming that flowor been digitally manipulated into scene.
inspection, way
fingers
Low-Level Statistics
**Noise Inconsistencies**
While the beckground and the rest of the child's festures have consistent noise distribution, the tempered part of the rose exhibits smoothr texture, indiceting noie reduction typical in composite images.
2. **Color end Textural Mismalchees*
The colar gradient and shading on the tampered part of the rose do not blend seamlessly with the surrounding
real elements.
3. **Sharpmess Discrepencies**:
The boundary of the tampered region Is unusually sharp compared to its surroundings. The difference In
sharpness is a strong low-level cue that suppests an overtay.
CASE 7. Unnatural interaction.
Nigh-Level Semantic Anomelies:
1. *Conntent Plousbilit*** While somewhat plausible, the suspended plane in this context unusual, Most cranes are not designed such delicate operations, and the scene might evoke skepticism regarding its realness, raising suspicion manipulation.
2. *hhysical tonstrainst": The suspension points and the way the plane is held appear thysics-defylng The ropes and balance don't align with the expected real-wordd weight distribution and grovitational effects on such an object.
Middle-Lovee Visual Defects:
1. *Lighting Inconsistency**, The lighting on the plane doeen't match the lighting on other objects and people in the scone. Spacifically, look the shadows on the plane the marked region they appear unnatural compared to how light interacts with the surrounding environment.
2. **Boundary Traces**; There are subile boundary artifacts around the tampered plane region. The boundaries of the plane show a certain level of pixel interpolation which often occurs during oopy-pasie tampering, leading to visible seams upon clese Inspection.
3. Persppective Reletionships** There slight perspective mismatch between the plane and the surrounding elaments, including the rope and the ship. The alignment and proportions appear off, which indicates manipulation.
Lov-Leval Pixel Statistics:
The nais the tampered region (the plane] is Inconsistent with other parts of the Image. Typically, authentic images uniform noise distribution, while Al-genereted manipulations often have differing noise characteristics in tampered
1. **Nole *attems?":
2. **Color and Sharpness**; The colors in the tampered region are slightly more saturated, and the edges are the
sharper
Authentic objects tend blend neturally with the ambient softness and
corsistency the picture.
CASE Physies defying.
Figure 16: Examples from the GPT-4o and then cross-revised by multiple experts. The manipulation type for these images is "Al-generate".
dataset. The data is initially generated by
31

---

High-Level Semantic Anomalles
*Contont Contrary to Common Sonse
The presence praminent, well-known Individual on trash fire is highly unusual and unilkely real-world contexts, raising immediate suspision about the image's authenticity.
2. **Inciting and Mideading Contant*>;
The juxtaposition of a public figure with suth an expiosive and dostructive background can be misleading and potentially inciting. This unusual context strongly suggests an intent to provoke reaction or convey felse nerretive.
Middie-Level Visual Defects
1. >BBunnaayy Traces
The edges around the tampered region are Irreguier and show unnetural transitions. Evidence this Includes sight color and texture mismatches along the perimeter the face.
2. *Lighting Inconsistency**s
The lighting on the face does not match the rest of the scone. The smoke and flames aurrounding the face should exhibit the same lighting effects (reffection, shadow, glow); nowever, the tampered region shows inconsistent lighting, which is
indicator of manipulation.
3. -<Perspective halationships=*
The perspective the face does not align well with the of object trash and flames It the been pasted onto the without considering the overall
making it look place.
Low-Level Pixel Statistics
**Noise Texture**
The face
while different
compared smoke
indicating
manibulstion.
**Color Sharpness**:
The sunounding elaments appear siightly blumeed
match the of fire.
The well defined, while
Nigh-Level Semantic Anomelies
*Cnntent Contrary to Common Sense**:
The presence person in roller coaster seat without any visible meens of support is contrary to common emusemeen park rides, where riders are typically seated enciosed cars or restrained some mennee. The absence of such restraints, es well es the leck seat belt or berrier in the vehicle, makes scenario implsusible.
Middle- Level Visual Defacts
1. ><Lgghting Inconsistency"ee:
The lighting on the person in the tampered region does not match the lighting conditions of the rest of the scene.
For instánce, the rollar coaster lacks shadows or reflections that should correspond to the person if they were. genuinely there.
2. **Traces of Tampered Region Boundary**:
Upon close inspection, the boundaries of the tampered area reveal subtle artifacts such as unnatural edges or blending issues, suggesting it was superimposed onto the scone.
Low-Level Pixel Statistics
1. *Nolse Amalysis
The tampered region shows different noise pettem compared to the rest of the Image. The natural noise the roller coaster seat should be consistent, but the tampered area displays imeguiar noise levels that differ from the background.
2. C>Color Consistency**:
There slight color mismatches between the person and the background. The tampered region has different
hue and saturation, which suggests that the person was added from different source image with different lighting conditions.
3. **Sherpness** *
The region and edges around the unneturally compared the of the indicating
onto
Examples from the ForgeryAnalysis-PP dataset. The data is automatically generated by our data engine Analety
Figure 17:
32

---

Given the tampered image, the reference forgery analysis text, and the generated anelysis text, assess the quality of the genereted enalyss based on the following criteri.. Pleese note! TAMPERED OBJECTS refer to "dog." "lece, "gress, etc,, while "boundary artifacts" or "lighting incondistencies' serve CLUES.
1. Correctness of Tempered Object (NOT Clues) Detection
8-10 points: ALL cluss accurately idantify major tampered oojects and describe alterations but may miss subtle detalis.
4-7 points: ALL dues reveal the tampered regions but lack carty miss same reglans.
1-3 points: Score this range ANY clues fall identity tampered objects, ANY incorrect detection is present.
Relevance of Clues Tampered Object
8-10 points: ALL or MCST cluns directly relate tampered elements, logically supporting the analysis.
4-7 points: SOME clues loosely related
clear connection the tampered content. 1-3 ALL Clues irreevant misleeding, contributing tampening anelysis.
Detail Clues and Analysis
9-10 points:
Provides comprehensive and well-explatned dues thorough reasoning. depth Contains detail clues indesquately explained. explonations.
3-5 Limited detail with
Minimal
insufficient
to each criterion, with
reflecting better performance. Provide the "Scorel: (score)/10 Commentl: (text) (score)/10
Comment2: (text) Score3: (score)/10 Comment3: (text)"
Reference Analysis Text:
lower-levels High-Level Semanttc Anomalies: Content Contrary Common Sense: dog's person's
have identified following
high-level anomalies
contrery common sense. unnatural combination logical coherence real-world scenarios, confirming manipulation. misleading presents unnetural intended
deception...
Generated Analysis
image appears digitally manibulated. hands, element out and arteicial. Additionally, the human's face no signs of tampering at the pivel
obvious
being held person's onto
fish, further suggests that the image was altered post pedd-productio. level, overall composition elements present the image indicate that edited.
Score1: 1/10
Comment1: The generated analysis fails to correctly identify tampered objects the reference anelysis. Insteed recognizing
bady, incormeclly discusses rish
mentianed the reference
Score2: 1/10
Comment2: provided
mention misleading not related the tampored cantont, which involves dog's the reference body.
2/10
the generated analysis minimal aligned with tampering described the referance enalysis depth, perticulerly explaining clues, incorrectly focuses detecting non-existent elements, providing useful insight into actuel mempuletion.
Figure 18: Illustration of the prompt and GPT- response of text analysis quality evaluation. The evaluation prompt includes clear scoring criteria for the assessment aspects, ensuring consistent and fair scoring.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately state our main contributions, the scope of our method, and these are well-supported by the results presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in the Appendices due to page con- straints.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
34

---

Answer: [NA]
Justification: This paper focuses on novel method and new dataset for image manipulation detection and does not include theoretical results like theorems or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of method, relevant training config- urations, and evaluation protocols. For our novel dataset, we thoroughly describe its construction, composition, and splits. These details are sufficient for reproducing our main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
35

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they 5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code, data, and model weights are already open-sourced. To facilitate reproducibility while ensuring reviewer anonymity, provide access to the code and data as described in our supplementary material. While the pre-trained model weights are also open-sourced, due their significant size, they are not directly included in the current supplementary package. We are committed to providing public links for resources in the abstract of the camera-ready paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.c cc/ public/guider htall//gguides///eeuuuiiin hte////oeSuuuiiseeeeee cassssooolllyll icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https: //ippsccoo cctppblcc//aapptpccccccccccccccc guwdee/ddeeeeee deedddee/oooouurrn wommiissiooooiiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Comprehensive details, including hyperparameters, optimizer types, learning rates, batch sizes, and other critical training and testing specifics, are thoroughly introduced in the Appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: To ensure fair comparisons, our experimental setup and evaluation protocols are consistent with established state-of-the-art methods the field. While comprehensive statistical significance tests are not uniformly standard practice in this research domain, we address potential experimental variability. For experiments involving stochastic elements, conduct evaluations twice and report the average performance.
36

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details on the computational resources used for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :
htcp/pbllie/thhcsciieeeeieeeeeee ines?
Answer: [Yes]
Justification: The research presented in this paper has been conducted with careful consider- ation of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
37

---

Answer: [Yes]
This paper discusses the broader societal impacts of our work. A significant positive impact is the potential to enhance the authenticity of visual content in media, which can help combat the spread of misinformation. However, we also acknowledge potential negative impacts. One key concern is the possibility of inaccurate detections, which could lead to the mislabeling of legitimate content.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We have implemented several safeguards. The model weights are hosted on an authoritative platform. Moreover, access to the data is controlled via an application process, which allows us carefully review and approve user requests.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
38

---

Justification:
For every asset used in this paper, we have properly cited the corresponding original paper or source. We have also ensured that our fully complies with the respective licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcom thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All assets introduced in this paper are well documented, and this documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmdel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
The instructions given to participants are provided in the appendices.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
39

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The potential risks involved in user study, such time expenditure, were disclosed to the participants within the annotation system and user experiment system. The experiments, conducted under limited risk conditions, received approval from our laboratory's s review process.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Large Language Models (LLMs) constitute an important component of the core methodology in this research. The relevant aspects of their usage are described in detail within the main body of the paper.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https :ttps:///nuuee /tt///uuris..// wwwwrrpp.../oooeeeeeeeeeeeeeeeee // for what should or should not be described.
40


Reviews:
[{"review_id": "36aj5iG9dv", "reviewer": "Reviewer_i8FQ", "summary": "In this work, the authors explore the image manipulation detection tasks using multimodal large language models. They propose ForgerySleuth, which integrates MLLMs with a trace encoder to use the world knowledge to capture high-level semantic anomalies as well as low-level traces. They also introduced a supervised fine-tuning dataset, ForgeryAnalysis, designed to pre-training and fine-tuning for IMD task, as well as a data engine to facilitate the creation of a larger-scale ForgeryAnalysis-PT dataset for pre-training. The author conduct experimentations on 6 different datasets, including Columbia, Coverage, CASIA1, NIST16, IMD20, and COCOGlide, and demonstrate superior performance. Code is also publicly available.", "questions": "1. Need more details on dataset curation and construction. For example, how did you edit the image? what are the process that leads to modification of a specific part of the image? Do you also have positive examples in your dataset?\nAlso please add more explanation to the caption of figure 2.\n2. While the method is well-motivated for real-world usage, the paper lacks practical demonstrations. Could the authors provide a proof-of-concept demo or case study (e.g., detecting social media tampering or forensic usage) to better contextualize the real-world relevance?\n3. The supervised fine-tuning and evaluation datasets seem limited and potentially overlapping. Can the authors clarify how they ensured no data leakage between fine-tuning and evaluation sets? Have they tested generalization to out-of-domain manipulations or unseen manipulation types?\n4. Can you provide more details on the data engine you use?", "limitations": "Discussion very limited. Should expand it other than stating \"the model is too big\" and \"create a lighter version and faster inference\". There is no insight. Please discuss your limitations. What are some problems or challenges when face during the data generation? Any problem with the quality of the data? What are some patterns of failure you found in your proposed method?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. Paper is well-written and easy to understand in most part. The topic and task of the paper is well motivated and important.\n2. The author curate dataset and data curation pipeline, which is useful for future research and manipulation detection applications.\n3. Thorough experimentation and ablations. Thorough analysis on different types of manipulations.\n\n\nWeakness:\n1. Need more details on dataset curation and construction. Currently, this section is really unclear. Including the caption of figure 2.\n2. It would be nice to discuss and demonstrate some practical usage of your work, or demo.\n3. The SFT and eval dataset split is very limited, which is a concern for generalization. \n4. poorly-written limitation section.", "was_revised": true, "final_justification": "Based on the rebuttal, I'm satisfied with the response. I decide to keep my rating of \"Borderline accept\".", "rebuttal": "We sincerely appreciate your feedback and positive comments on our work. We have addressed each weakness and question raised, and the paper has been updated and expanded accordingly. Below is a detailed response to your points.\n\n**Q1. More details on dataset construction.**\n\nThank you for requesting more details on our dataset construction. To clarify, our pipeline does not include the image editing process itself. Instead, we use several public image manipulation detection datasets as our data sources. These datasets already contain real images, manipulated images, and their corresponding manipulation region annotations. A detailed introduction to these sources is provided in **Lines 124-128** of our paper. \n\nWe acknowledge that the data production process may require further clarification. While **Appendix Figure 14** already illustrates a complete data production overview, we have added more explicit indicators in the main text to guide readers to this figure and its detailed explanation in the Appendix. Furthermore, as suggested, we have revised the caption for **Figure 2** to enhance clarity. The updated caption reads:\n\n*Figure 2: ForgeryAnalysis Dataset Construction Pipeline. Our pipeline begins with (a) GPT-4o generating initial analyses for manipulated images with annotated regions, followed by human expert review. The refined analyses are organized into (b) the CoC format. This human-curated ForgeryAnalysis (2k) dataset is used to train a data engine. Finally, (c) this data engine generates ForgeryAnalysis-PT, a larger-scale dataset for model pre-training.*\n\n**Q2. Real-world application.**\n\nWe agree on the importance of real-world application. We've enhanced the experimental analysis section (**Section 5.2**) to emphasize the following points, which illustrate ForgerySleuth's potential in social media and forensic analysis:\n\nFirst, a portion of ForgeryAnalysis's source data (e.g., the MIML [34] dataset collected from PhotoshopBattles social platform) comprises images edited by real users for various purposes, commonly found on social media. This represents a typical real-world scenario, indicating our ForgeryAnalysis-eval already encompasses this type of data.\n\nSecond, we have further augmented our evaluation with experiments on the GRE [a] dataset. This dataset contains real-world black-box edited data sourced from X (formerly Twitter) and Weibo. On the manipulation region detection task, our method achieved an F1 score of 70.3%. These results further demonstrate ForgerySleuth's generalization in complex real-world scenarios.\n\n*[a] Sun, Z., Fang, H., Cao, J., Zhao, X., & Wang, D. (2024, October). Rethinking image editing detection in the era of generative AI revolution. In Proceedings of the 32nd ACM International Conference on Multimedia (pp. 3538-3547).*\n\n**Q3. The eval split in ForgeryAnalysis.**\n\nWe understand your concern regarding the potentially limited size of our ForgeryAnalysis supervised fine-tuning and evaluation datasets. This limitation results from the cost of ensuring high-quality human expert review and cross-validation for accuracy. However, our evaluation extends beyond this single dataset, also including tests on the manipulation region localization task, ensuring comprehensive validation.\n\nWe ensured no data leakage between our fine-tuning and evaluation sets by implementing a strict split based on the original image source. Specifically, images derived from the same base image (e.g., different manipulations of the same original photograph) are allocated exclusively to either the training/fine-tuning set or the evaluation set, but never both.\n\nRegarding generalization to out-of-domain or unseen manipulation types, our evaluation includes performance on six distinct benchmark datasets (**Lines 246-250**). These datasets are entirely separate from our training data source (**Lines 124-128**) and feature different manipulation types. This cross-dataset evaluation approach, employing distinct datasets with various manipulation types, is a widely accepted practice in the image manipulation detection field to demonstrate generalization capabilities.\n\nTo provide further clarity, we have now included a new table to provide detailed statistics on the scale, source, and manipulation types from each public dataset utilized in the construction of ForgeryAnalysis and for evaluation.\n\n| Dataset | Usage | Authentic Samples | Tampered Samples | Manipulation Types | Source |\n| --- | --- | --- | --- | --- | --- |\n| MIML [34] | Train, Construct ForgerAnalysis | 11,142 (PSBattles) | 123,150 | Manual Editing | PhotoshopBattles |\n| CASIA2 [35] | Train, Construct ForgerAnalysis | 7,491 | 5,063 | Splice, Copy-move | Corel |\n| DEFACTO [36] | Train, Construct ForgerAnalysis | - | 149,587 | Splice, Copy-move, AIGC (GAN) | MS COCO |\n| AutoSplice [37] | Train, Construct ForgerAnalysis | 2,273 | 3,621 | AIGC | Visual News |\n| Columbia [55] | Eval | 183 | 180 | Splice | - |\n| Coverage [56] | Eval | 100 | 100 | Copy-move | - |\n| CASIA1 [35] | Eval | 800 | 920 | Splice, Copy-move | Corel |\n| NIST16 [57] | Eval | - | 564 | Splice, Copy-move | - |\n| IMD20 [58] | Eval | 414 | 2010 | Manual Editing | - |\n| COCOGlide [30] | Eval | 512 | 512 | AIGC (Diffusion) | - |\n\n**Q4. More details on the data engine.**\n\nWe regret any lack of clarity regarding our data engine. As detailed in **Appendix Section A.2**, we provided a more comprehensive description of our data engine, including its architecture and prompt design. Additionally, **Appendix Figure 19** already offers specific examples that illustrate the data engine's outputs for better understanding.\n\nBased on your valuable feedback, we recognize that these details might not have been sufficiently emphasized in the main text. Therefore, we have added explicit guidance and incorporated the following content in **Section 3.2** of the main paper:\n\n*The data engine receives input that includes explicit information about the tampered region (highlighted to indicate tampering), aiming to generate more precise and comprehensive clue analyses. It outputs manipulation analyses organized in the CoC format, as illustrated in Figure 2(c). For more detailed information on the data engine, including the specific prompts used, please refer to Appendix Section A.2, and Appendix Figure 19 for output examples.*\n\n**Q5. Limitations.**\n\nWe are highly appreciative of your feedback on our limitations section. Your suggestions prompted us to reconsider our work's current limitations and future improvement directions. We have revised and expanded **Appendix Section D.6 (Limitation and Future Work)** in the paper, integrating the failure case analysis (**Appendix Section D.5**) for a more comprehensive discussion. Our updated limitations section is as follows:\n\n***1. Challenges in Detection Performance.** We analyzed ForgerySleuth's detection failures and identified two representative error patterns:*\n\n*a. Semantic Description Inaccuracies: In these cases, our model can accurately localize manipulated regions, but its textual analysis contains description errors. For instance, in the first example in Appendix Figure 9, the model misidentifies dogs as cats. Importantly, this does not impact the detection logic or localization accuracy. These errors often arise when specific semantic concepts in images are challenging to detect clearly. Future work could improve this by enhancing the visual understanding capabilities of MLLM and LLM backbone models.*\n\n*b. Failures with Small Manipulations in Complex Scenes: In complex image content with small manipulated regions, the model's localization precision can degrade, sometimes leading to incorrect textual analysis and, consequently, a detection failure. For example, in the second instance in Appendix Figure 9, where a spliced cartoon character is present, the model only localized a partial manipulation (e.g., the guitar in hand) while overlooking the character itself, primarily due to the small proportion of the manipulation and the image's overall complexity. Detecting small region manipulations remains a significant challenge, and MLLMs tend to focus on global semantic information rather than fine-grained local details. Therefore, developing methods to effectively detect anomalies in small regions is a crucial research direction.*\n\n***2. Constraints in Evaluation Methods.** Our paper evaluates analysis accuracy using GPT-4o and STS methods on the ForgeryAnalysis-eval dataset. However, our evaluation methods are subject to specific constraints: our assessment relies on GPT-4o, a large external model that may exhibit inherent biases or hallucinations, and STS only reflects semantic similarity between two sentences. Thus, exploring more comprehensive and objective evaluation methods, including the quantitative assessment of model hallucinations, is an important direction for future work.*\n\n***3. Model Scale and Inference Costs.** While leveraging LLMs to complement traditional image manipulation detection methods, which often primarily focus on low-level traces, this approach increases the model's scale and computational cost of generating detailed explanations. In the future, we plan to investigate model light-weighting solutions. This includes developing a version where generating such detailed output is optional, which would significantly reduce inference time. Alternatively, we could explore distilling LLM capabilities into smaller models, retaining only the essential knowledge for manipulation detection.*"}, {"review_id": "SJPO3lmCzE", "reviewer": "Reviewer_sU1j", "summary": "The paper presents ForgerySleuth, a framework that uses multimodal large language models (M-LLMs) to detect and explain image manipulations. It combines high-level reasoning from the M-LLM with low-level visual features from a trace encoder to produce both segmentation masks and textual analysis. The authors also built a dataset, ForgeryAnalysis, using a Chain-of-Clues prompting method and expert refinement. Experiments show improved performance over existing methods in both detection accuracy and explanation quality.", "questions": "1. The main paper does not clearly specify the dimensions and spatial shapes of key features such as $h_{seg}$, $f_t$, and $f_c$. It would improve clarity if the authors could explicitly provide this information in the main text.\n\n1. While the Chain-of-Clues prompting is central to the method, it is unclear how much it contributes compared to simpler reasoning instructions. Is it possible to provide ablation results or analysis comparing with non-CoC-based annotations?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The paper explores a novel application of M-LLMs in image manipulation detection, a relatively underexplored area.\n- The proposed system effectively combines high-level semantic reasoning with low-level trace features, leading to strong performance in both localization and explanation.\n- The Chain-of-Clues structure improves interpretability and aligns well with the needs of forensic analysis.\n- The ForgeryAnalysis dataset is carefully constructed, with expert refinement and scalable pretraining data, supporting both accuracy and explainability.\n- Experimental results are comprehensive, covering multiple benchmarks and including both quantitative and qualitative evaluation.\n\n## Weaknesses\n- The paper lacks detail on key implementation aspects, such as the dimensions and alignment of feature representations across modules.\n- Most components are built on existing methods (e.g., trace encoder, LoRA, attention fusion), and the novelty mainly lies in their combination rather than in any single technical innovation.\n- The discussion on limitations is minimal in the main paper, and some design choices (e.g., freezing the vision encoder) are not well justified.", "was_revised": true, "final_justification": "The rebuttal and follow-up responses addressed my main concerns. The authors clarified key implementation details (e.g., feature dimensions and alignment), provided new ablation results demonstrating the value of Chain-of-Clues prompting, and offered a more comprehensive discussion of limitations and failure cases. These updates improve the clarity and completeness of the paper.\n\nThat said, I still find the technical novelty to be somewhat limited, as the approach mainly builds on existing components. Overall, I see this as a solid application-driven contribution with strong experimental support, and I maintain my rating accordingly.", "rebuttal": "We sincerely appreciate your thorough review and positive assessment of our work. We have carefully considered all your questions and suggestions, and our revisions are detailed below.\n\n**Q1. Dimensions and alignment of features**\n\nWe appreciate your suggestion for clearer specifications regarding feature dimensions and their alignment. We have revised **Section 4**, specifically **Lines 181-205,** to provide this information.\n\n**Semantic Clue Embedding $h_{seg}$.** Given an input image $x_{img} \\in \\mathbb{R}^{B \\times H \\times W \\times 3}$ (where $B$ is batch size, and $H$ and $W$ are the height and width of the image respectively) and prompt $x_p$, we first feed them into the M-LLM $F_m$. Fm outputs a hidden embedding $\\tilde{h} \\in \\mathbb{R}^{B×L×\\tilde{D_{m}}}$ from its last layer (where $L$ is sequence length, and $\\tilde{D_{m}}$ is the hidden layers' dimension of the LLM). We then extract the embedding $\\tilde{h_{seg}} \\in \\mathbb{R}^{B×\\tilde{D_{m}}}$ corresponding to the [SEG] token from $\\tilde{h}$ and apply an MLP projection layer $\\gamma$ to obtain $h_{seg} \\in \\mathbb{R}^{B×D_{m}}$. This $h_{seg}$ serves as the semantic clue embedding that guides mask generation.\n\n**Trace Feature $f_t$.** Simultaneously, the input image $x_{img}$ is processed by the trace encoder $F_t$. This encoder extracts low-level forensic traces, yielding manipulation trace features $f_t \\in \\mathbb{R}^{B \\times H_t \\times W_t \\times D_t}$, where $H_t = H / \\text{patch.size}$ and $W_t = W / \\text{patch.size}$.\n\n**Content Feature $f_c$.** The vision backbone $F_v$ processes $x_{img}$ to produce a dense vision content feature $f_c \\in \\mathbb{R}^{B \\times H_c \\times W_c \\times D_c}$, where $H_c = H / \\text{patch.size}$ and $W_c = W / \\text{patch.size}$.\n\nFor effective multimodal fusion, the dimensions $D_m$, $D_t$, and $D_c$ are projected to a common embedding dimension $D$ before they are fed into the fusion mechanism.\n\n**Q2. Technical innovation.**\n\nWe acknowledge that ForgerySleuth integrates existing methods such as LLMs, LoRA, and attention fusion. While such combinations are common in application-oriented solutions, our core innovation lies in the motivation and precise design of how the LLM addresses high-level semantic anomalies while the trace encoder captures low-level statistical features. Experiments further demonstrate the effectiveness of these components, including our method and dataset.\n\n**Q3.  Ablation study of CoC prompting.**\n\nWe appreciate your suggestion for an ablation study on CoC prompting, which effectively clarifies its contribution. To demonstrate how much CoC contributes compared to simpler reasoning instructions, we have conducted new ablation experiments using non-CoC-based annotations. We explored two experimental settings:\n\n**Setting 1 (Only Detection Result):** We removed all clues, providing only simple text about detection results (e.g., \"This is a <FAKE> image with modifications detected in the [SEG].\" or \"This is a <REAL> image without modifications detected, and the [SEG] is meaningless.\").\n\n**Setting 2 (Unstructured Clues):** We provided summarized text and clues without organizing them into the CoC structure. The order of different clues was randomized.\n\nWe employed the same evaluation setup as described in **Table 5** in the main text, with *all-mpnet-base-v2* for STS calculation. The results are presented in the table below:\n\n| Setting | NIST16 | IMD20 | COCOGlide | ForgeryAnalysis-Eval | STS |\n| --- | --- | --- | --- | --- | --- |\n| ForgerySleuth (with CoC) | 0.518 | 0.710 | 0.562 | 9.45 | 0.961 |\n| Setting 1 (Only Detection Result) | 0.448 | 0.526 | 0.533 | - | - |\n| Setting 2 (Unstructured Clues) | 0.505 | 0.681 | 0.564 | 7.94 | 0.785 |\n\nThese results demonstrate the effectiveness of the CoC structure in guiding the model towards more precise localization and coherent, structured explanations.\n\n**Q4. Justification for freezing the vision encoder.**\n\nWe appreciate your question regarding the justification for freezing the vision encoder. The primary reason for freezing the vision backbone $F_v$ is to retain its robust capacity for modeling image content features, which are crucial for accurate segmentation. This design choice is detailed in **Lines 227-229**.\n\nFurthermore, we have validated this decision through ablation experiments, as presented in **Lines 328-331** and **Table 6**. Our comparison of various training strategies for the vision backbone $F_v$, including trainable and LoRA fine-tuning, consistently demonstrated that keeping the encoder completely frozen achieves the best performance. This can be attributed to the observation that trainable and LoRA fine-tuned strategies slightly diminish the generalization ability of the original SAM backbone.\n\n**Q5. Limitations.**\n\nWe appreciate you highlighting the limited scope of our initial limitations discussion. We have accordingly revised and expanded **Appendix Section D.6 (Limitation and Future Work)** in the paper, integrating the failure case analysis (**Appendix Section D.5**) for a more comprehensive discussion. Our updated limitations are as follows:\n\n***1. Challenges in Detection Performance.** We analyzed ForgerySleuth's detection failures and identified two representative error patterns:*\n\n*a. Semantic Description Inaccuracies: In these cases, our model can accurately localize manipulated regions, but its textual analysis contains description errors. For instance, in the first example in Appendix Figure 9, the model misidentifies dogs as cats. Importantly, this does not impact the detection logic or localization accuracy. These errors often arise when specific semantic concepts in images are challenging to detect clearly. Future work could improve this by enhancing the visual understanding capabilities of MLLM and LLM backbone models.*\n\n*b. Failures with Small Manipulations in Complex Scenes: In complex image content with small manipulated regions, the model's localization precision can degrade, sometimes leading to incorrect textual analysis and, consequently, a detection failure. For example, in the second instance in Appendix Figure 9, where a spliced cartoon character is present, the model only localized a partial manipulation (e.g., the guitar in hand) while overlooking the character itself, primarily due to the small proportion of the manipulation and the image's overall complexity. Detecting small region manipulations remains a significant challenge, and MLLMs tend to focus on global semantic information rather than fine-grained local details. Therefore, developing methods to effectively detect anomalies in small regions is a crucial research direction.*\n\n***2. Constraints in Evaluation Methods.** Our paper evaluates analysis accuracy using GPT-4o and STS methods on the ForgeryAnalysis-eval dataset. However, our evaluation methods are subject to specific constraints: our assessment relies on GPT-4o, a large external model that may exhibit inherent biases or hallucinations, and STS only reflects semantic similarity between two sentences. Thus, exploring more comprehensive and objective evaluation methods, including the quantitative assessment of model hallucinations, is an important direction for future work.*\n\n***3. Model Scale and Inference Costs.** While leveraging LLMs to complement traditional image manipulation detection methods, which often primarily focus on low-level traces, this approach increases the model's scale and computational cost of generating detailed explanations. In the future, we plan to investigate model light-weighting solutions. This includes developing a version where generating such detailed output is optional, which would significantly reduce inference time. Alternatively, we could explore distilling LLM capabilities into smaller models, retaining only the essential knowledge for manipulation detection.*"}, {"review_id": "HIF7wnJFym", "reviewer": "Reviewer_w2m9", "summary": "This paper proposes ForgerySleuth, an image manipulation detection method that leverages the capabilities of VLMs to achieve detection, localization, and explanation of manipulated content. To enable explanatory capabilities for tampered areas, the authors construct a specialized dataset using GPT-4o for both pretraining and evaluation. Experimental results demonstrate that the proposed method achieves effective performance across detection, localization, and explanation tasks.", "questions": "1.  The main concerns are shown in the weakness section. Please address the questions about the dataset and novelty.\n\n2. Did you test the contributions of each semantic level? I am curious about the individual contributions of low, middle, and high levels to ForgerySleuth's performance.\n\n3. Did ForgerySleuth undergo testing on cross-domain tasks, like image deepfake detection and localization (face and AIGC)? Experiments in these areas might improve the effectiveness and generalizability of the proposed work.\n\n4. I am concerned about the implementation details. For example, why are λbce and λdice set to 1.0 and 0.2, respectively? Any insights or ablation experiments could support these parameter choices.\n\n5. I advise the authors to include GT masks in the examples (Appendix) to better demonstrate the differences between ForgerySleuth and other methods.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper proposes an interesting and important topic: providing explanations for manipulation detection. This is particularly relevant as visual models become more and more powerful.\n- The experiments demonstrate that the proposed method achieves superior results compared to traditional IFDL methods and Large Language Models (as shown in the Appendix).\n\n**Weaknesses**\n\n- Novelty of Method: The method used in ForgerySleuth lacks complete novelty, as demonstrated by similar approaches in [1,2] and FakeShield, which is discussed in the Appendix. All these methods utilize LLMs to provide explanations for deepfake detection and localization using similar frameworks (e.g., employing SEG tokens to generate masks for manipulation areas, following LISA). The differences between ForgerySleuth and these existing methods are not clearly articulated. A more comprehensive literature review and detailed comparison with existing methods would be helpful to better understand the proposed method's unique advantages.\n\n- Dataset: I'm concern about the progress of building the dataset.\n  1. The details regarding expert involvement in the GPT-4o data review process are not provided. This process is quite important since it serves as the foundation for generating the larger ForgeryAnalysis-PT dataset.\n\n  2. In Figure 2(c), the ground truth (GT) masks are not input to the Forgery-Analyst. Instead, the authors choose to use \"images with tampered regions highlighted.\" What does \"highlighted\" mean in this context, and why are GT masks not used instead? What are the differences between these two approaches? I did not find a detailed explanation of this design choice in the paper.\n\n  3. There are some unclear elements in Figure 2. The figure suggests that Forgery-Analyst is also GPT-4o (though it is not), which creates confusion about the model architecture.\n  4. Since the ForgeryAnalysis Dataset utilizes public data, the authors should provide detailed information about the source datasets used, presented in a table or figure. This is necessary for fair comparison with other methods evaluated on public datasets.\n\n[1] Kang, Hengrui et al. “LEGION: Learning to Ground and Explain for Synthetic Image Detection.” ArXiv abs/2503.15264 (2025): n. pag.\n\n[2]  Huang, Zhenglin et al. “SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model.” ArXiv abs/2412.04292 (2024): n. pag.", "was_revised": true, "final_justification": "The authors solve most of my questions so I intend to accept.", "rebuttal": "We sincerely appreciate your review and constructive suggestions. We apologize for any lack of clarity in the initial submission. We have carefully addressed each of your concerns and questions, and we hope our responses resolve the issues you raised.\n\n**Q1. Novelty of Method.**\n\nWe appreciate you pointing out these relevant works. Based on your feedback, we have expanded **Section 2** to better articulate ForgerySleuth's novelties and distinctions.\n\n(1) **Manipulation Type:** While LEGION and SIDA are designed for deepfake detection, focusing on traces left by generative models. Our work and FakeShield address local manipulation, where only a part of the image is edited. These edits can include splicing, manual alterations, or regional generation. Therefore, our work and FakeShield are more relevant in terms of task scope, which is why we focused on comparing with it in the Appendix.\n\n(2) **Model Architecture:** While the overall framework integrating an LLM with a vision segmentation head appears similar (as this design naturally aligns with the task), our core innovation is the motivated and precise design of the LLM addressing high-level semantic anomalies, and the trace encoder capturing low-level statistical features. Furthermore, our fusion mechanism allows these distinct features to complement each other. The effectiveness of these components is further demonstrated by ablation studies in Section 4.x and Q3 below.\n\n(3) **Structured CoC**: Unlike other methods that simply list clues in their datasets, we organized clues into a hierarchical CoC structure during data construction. This not only offers users more coherent analysis but also guides the model to focus on distinct clue levels during training, thereby serving as more effective supervision. The effectiveness of CoC is further demonstrated by new ablation studies in Q3 below.\n\nIn summary, our work goes beyond merely integrating an LLM with a segmentation head. We specifically designed our data and model components to leverage distinct forensic clues for image manipulation detection. Furthermore, it is important to note that **our paper was publicly released before LEGION and SIDA, and our model/code before FakeShield.**\n\n**Q2. More details on dataset construction.**\n\nWe sincerely appreciate your detailed questions and constructive suggestions regarding our dataset construction. We acknowledge that, due to space constraints, many dataset building details were primarily placed in **Appendix Section A**, leading to less clear representation in the main text. We've carefully considered each of your points and have made the following responses and revisions:\n\n**Q2.1.** Details on expert involvement and review process.\n\nAs mentioned in **Lines 141-147**, the experts' review process primarily involved three steps: filtering vague responses, deleting ambiguous clues, and organizing the analysis into the Chain-of-Clues format. We have provided a complete example of this editing process in **Appendix Figure 14**, with more detailed descriptions of the review guidelines available in **Appendix Lines 556-565**. We have now added explicit instructions in the main text to guide readers to this example for better understanding.\n\n**Q2.2.** Explanation of “images with tampered regions highlighted”.\n\nWe apologize for the oversight regarding the detailed explanation of this design choice. We have added the following clarification in **Appendix Section A.2**:\n\n*We chose to indicate manipulation by highlighting tampered regions. Here, \"highlighting\" means outlining the boundaries of the tampered region with a selected color (red, green, or blue, selected based on the least dominant channel in the current image's RGB to ensure visual prominence and distinction from content). The boundary is expanded outwards by 11 pixels to prevent the highlight line from obscuring subtle tampering clues.*\n\n*We did not provide the image and mask as two separate inputs because our experiments showed that MLLMs (like GPT-4o and LLaVA in our method) struggled to accurately correlate corresponding positions between two input images, while annotating directly on a single image is more accurate. **Appendix Figure 19** further illustrates our data engine's input and output examples, including this highlighting method.*\n\n**Q2.3.** Unclear elements in Figure 2.\n\nWe apologize for the confusion caused by the ambiguous elements in **Figure 2**. It seems the consistent use of green for both the GPT-4o logo and our data engine might have led to misinterpretations. We've updated **Figure 2(c)** to directly label the model architecture as *LLaVA-13B* rather than *Data Engine*, which should avoid any misunderstanding regarding the model structure.\n\n**Q2.4.** Source datasets used.\n\nWe appreciate you raising this point, as clarity on source datasets is crucial for fair comparison. The “existing public datasets” mentioned in **Line 161** refer to the source datasets introduced in **Lines 124-128**. These source datasets are completely independent from our evaluation benchmark datasets (**Lines 246-250**), ensuring fair assessment. We acknowledge that this brief mention was not clear. We have now revised **Line 161** and included a new table to provide detailed statistics on the scale, source, and manipulation types from each public dataset used to construct ForgeryAnalysis and evaluation.\n\n| Dataset | Usage | Authentic Samples | Tampered Samples | Manipulation Types | Source |\n| --- | --- | --- | --- | --- | --- |\n| MIML [34] | Train, Construct ForgerAnalysis | 11,142 (PSBattles) | 123,150 | Manual Editing | PhotoshopBattles |\n| CASIA2 [35] | Train, Construct ForgerAnalysis | 7,491 | 5,063 | Splice, Copy-move | Corel |\n| DEFACTO [36] | Train, Construct ForgerAnalysis | - | 149,587 | Splice, Copy-move, AIGC (GAN) | MS COCO |\n| AutoSplice [37] | Train, Construct ForgerAnalysis | 2,273 | 3,621 | AIGC | Visual News |\n| Columbia [55] | Eval | 183 | 180 | Splice | - |\n| Coverage [56] | Eval | 100 | 100 | Copy-move | - |\n| CASIA1 [35] | Eval | 800 | 920 | Splice, Copy-move | Corel |\n| NIST16 [57] | Eval | - | 564 | Splice, Copy-move | - |\n| IMD20 [58] | Eval | 414 | 2010 | Manual Editing | - |\n| COCOGlide [30] | Eval | 512 | 512 | AIGC (Diffusion) | - |\n\n**Q3. Contributions of each semantic level.**\n\nWe appreciate this insightful suggestion, which prompted us to conduct additional ablation experiments to further illustrate the contributions of different semantic levels to ForgerySleuth's performance. Given the extensive data processing and model training required, we have completed the following experiments. \n\nWe modified the ForgeryAnalysis-PT and SFT datasets for these specific tests, while the base pre-training process and evaluation datasets remained unchanged. Our experimental settings include:\n\n**Setting 1 (*w/o* High-Level Clues):** We only removed high-level clues from the textual analysis.\n\n**Setting 2 (*w/o* Low-Level Clues):** We only removed low-level clues from the textual analysis.\n\n**Setting 3 (*w/o* Low-Level Clues and Trace Encoder):** We removed low-level clues from the textual analysis and also removed the Trace Encoder.\n\nThe evaluation metrics are consistent with **Table 5** in the main text, with the addition of STS calculated using the *all-mpnet-base-v2* model.\n\n| Setting | NIST16 | IMD20 | COCOGlide | ForgeryAnalysis-Eval | STS |\n| --- | --- | --- | --- | --- | --- |\n| ForgerySleuth | 0.518 | 0.710 | 0.562 | 9.45 | 0.961 |\n| Setting 1 (*w/o* High-Level Clues) | 0.493 | 0.551 | 0.548 | 6.31 | 0.764 |\n| Setting 2 (*w/o* Low-Level Clues) | 0.513 | 0.685 | 0.557 | 7.12 | 0.824 |\n| Setting 3 (*w/o* Low-Level Clues and Trace Encoder) | 0.336 | 0.620 | 0.409 | 5.88 | 0.576 |\n\nFrom these experiments, it can be observed that high-level features contribute relatively more to the manipulation analysis. Removing only low-level features from the textual analysis while retaining the Trace Encoder impacts the comprehensiveness of the manipulation analysis, but has a smaller effect on localization accuracy. In contrast, removing both low-level features from the text and the Trace Encoder significantly degrades localization performance. This further validates the effectiveness of the Trace Encoder in the manipulation detection task.\n\n**Q4. Evaluation on cross-domain tasks.**\n\nWe appreciate this important question regarding cross-domain generalization, particularly for deepfake and AIGC content.\n\nWhile our primary focus is on local image manipulation detection (distinct from full-image deepfake generation as explored by some methods), our evaluation inherently includes relevant cross-domain elements. Our existing evaluation dataset on COCOGlide already incorporates images edited using AIGC methods, and this dataset contains examples of partial face deepfake manipulations (as shown in **Appendix Figure 18**).\n\n**Q5. Implementation details.**\n\nWe appreciate your question regarding the implementation details, specifically the weights for $\\lambda_{bce}$ and $\\lambda_{dice}$. Our initial hyperparameter choices were informed by best practices from relevant works in the field, such as LISA, and LLaVA. We then conducted further experimentation to refine these parameters.\n\nFor the two segmentation loss weights ($\\lambda_{bce}$ and $\\lambda_{dice}$), BCE loss typically serves as the primary loss, while Dice loss is used to enhance performance on imbalanced classes. Therefore, $\\lambda_{bce}$ was set to 1.0. We then experimented with various values for $\\lambda_{dice}$ (0.1, 0.2, 0.3, 0.5, 1.0) and selected 0.2 based on its optimal performance in the segmentation task.\n\n**Q6. Include GT masks.**\n\nThank you for this valuable suggestion. We agree that including GT masks will enhance the clarity of our qualitative examples. We have updated figures in the appendix to include GT masks alongside our predictions and those of other methods, providing a clearer visual comparison of localization accuracy."}, {"review_id": "zJfXLUAP5s", "reviewer": "Reviewer_oWxz", "summary": "The paper is introducing a dataset (ForgeryAnalysis) and a method (ForgerySleuth) for image manipulation detection. Both leverage MLLMs to generate text captions in the dataset and then train an MLLM for the classification and forgery mask prediction.", "questions": "There are several unclear areas in both the method and the experimental results presented, extensively analysed above. The main issues that would need to be addressed are clarity in the method and more detailed ablations.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper addresses the need for task specific datasets and methods leveraging the common sense reasoning of MLLMs. Specifically, the introduction of a dataset with captions should encourage research in the task. Moreover, the results obtained by ForgerySleuth are good compared to previous works on the detection task.\n\nSeveral aspects of the proposed method are unclear and are not properly explained in the paper.\n- What prompt is used for GPT4? This should be shown in the text and figure 2\n- Does GPT4 calculate the low, mid and high level \"clues\" or simply rephrase based on a pre-computed set of features?\n- What kind of responses did GPT4 produce? Are there any examples of good vs hallucinated responses?\n- For the CoC, it is stated that human experts curated the GPT4 responses. how many experts per sample and how is consistency ensured?\n- No statistics of the introduced dataset are provided to better understand the composition of the dataset\n- In terms of the ForgerySleuth training, it is unclear what the prompt x_p is.\n- It is also unclear whether the CoC produced by GPT4 is used to train the model autoregressively or whether it is simply T\n- Lines 182-183 mention enriching the LLM vocabulary with [SEG], <REAL> and <FAKE> tokens. Are these new embeddings or trainable tokens similar to the [CLS]? The phrasing and notation is ambiguous, as tin Fig. 3 it appears that embeddings are added but then a trainable token it taken for h_seg.\n- Fusion mechanism seems like a key part of the method, but is left to the appendix.\n\nIn terms of the results, the localisation results seems convincing but more details on the detection and the qualitative output of the text should be included in the paper:\n- Table 4: Llava is finetuned to produce part of the dataset. How does that perform on the STS?\n- Classification (real/fake) performance should also be included as STS is not necessarily capturing this\n- The ablation is incomplete: How does the classification/mask prediction perform without the CoC reasoning? how does the vision decoder perform without the LLM output?\n- Are there any failure cases? Is there a pattern in the failure cases? The appendix includes 2 failure cases, this is not enough to draw conclusions.", "was_revised": true, "final_justification": "Thank you for the very detailed responses to all reviews. Most of the concerns are addressed by the rebuttal; however, the discussion lacked transparency. Specifically, the questions asked originally were not answered directly or were evaded, raising questions about the understanding and interpretation of the paper. As such, I have raised the score, but I am less confident about it post-rebuttal.", "rebuttal": "We sincerely appreciate your review and constructive suggestions. We acknowledge that some crucial details were initially placed in the Appendix, which might have led to confusion. Your valuable feedback has prompted us to integrate more key information directly into the main text.\n\n**Q1. More details on the dataset.**\n\n**Q1.1.** GPT-4o prompt details.\n\nThe prompt of GPT-4o consists of two parts: first, the role and task instructions (detailed in **Appendix Lines 542-555**), and second, the specific per-sample input prompt (illustrated in **Appendix Figure 14**). Due to text length constraints, we can not place the full prompt in Figure 2, but we have added clear indicators in the main text to ensure readers can easily find these details in the Appendix.\n\n**Q1.2.** GPT-4o clues generation.\n\nWe appreciate you highlighting this point. We have clarified in **Line 136** of the main text that: \n\n*We utilize the advanced M-LLM, GPT-4o, to generate the initial clue analyses based on its visual understanding of the input image (with highlighted regions) and its inherent world knowledge. For more detailed content and examples of GPT-4o response, please refer to Appendix Figure 14.*\n\n**Q1.3.** Examples of GPT-4o response.\n\n**Appendix Figure 14** and **Lines 556-565** illustrate the initial clues generated by GPT-4o and detail how human experts correct ambiguous or hallucinated parts. We've now explicitly labeled cases of both high-quality responses and instances that required correction due to hallucination or vagueness.\n\n**Q1.4.** Human expert review.\n\nA total of five experts reviewed and refined the generated entries, with each expert revising between 450-800 entries. Consistency was effectively ensured because the expert editing process primarily involved deleting vague content and reorganizing existing information into the defined Chain-of-Clues format, rather than introducing new information. We also provided strict guidelines on the CoC format. Furthermore, for accuracy, 618 entries selected for evaluation were cross-validated by more than two experts.\n\n**Q1.5.** Dataset statistics.\n\nWe appreciate your suggestion about the statistics of our dataset. While **Appendix Table 7** already provides statistics on the number of samples in different splits, we agree that the composition was not detailed. For our ForgeryAnalysis-SFT and Eval datasets, samples are selected proportionally from MIML, CASIA2, DEFACTO, and AutoSplice datasets. The larger ForgeryAnalysis-PT dataset includes 30k, 4k, 13k, and 3k samples, respectively, from these sources. We have now supplemented this by including a statistics table for the source dataset of ForgeryAnalysis in **Appendix A.3**.\n\n| Dataset | Authentic Samples | Tampered Samples | Manipulation Types | Source |\n| --- | --- | --- | --- | --- |\n| MIML [34] | 11,142 (PSBattles) | 123,150 | Manual Editing | PhotoshopBattles |\n| CASIA2 [35] | 7,491 | 5,063 | Splice, Copy-move | Corel |\n| DEFACTO [36] | - | 149,587 | Splice, Copy-move, AIGC (GAN) | MS COCO |\n| AutoSplice [37] | 2,273 | 3,621 | AIGC | Visual News |\n\n**Q2. More details on the method.**\n\n**Q2.1.** Training Prompt $x_p$.\n\nWe apologize for initially overlooking the inclusion of the training prompt $x_p$. This prompt is the instruction given to guide the model during training. We have now added a clear explanation in **Appendix Section B**, including an example of the prompt we use:\n\n*Is this image authentic, or has it undergone manipulation? If manipulated, show the precise segmentation mask of the manipulation. Please provide the chain-of-clues supporting your detection decision in the following style: # high-level semantic anomalies (such as content contrary to common sense, inciting and misleading content), # middle-level visual defects (such as traces of tampered region or boundary, lighting inconsistency, perspective relationships, and physical constraints) and # low-level pixel statistics (such as noise, color, textural, sharpness, and AI-generation fingerprint), where the high-level anomalies are significant doubts worth attention, and the middle-level and low-level findings are reliable evidence.*\n\n**Q2.2.** CoC usage for training.\n\nThe CoC generated by GPT-4o and refined by human experts is used to train the model auto-regressively. This corresponds to the \"text $\\hat{T}$ that includes reasoning and evidence\" mentioned in **Line 168**. We have stated this in the revised text to avoid any confusion.\n\n**Q2.3.** Special tokens.\n\nRegarding [SEG], < REAL >, and < FAKE > tokens mentioned in **Lines 182-183**, these are new and trainable special tokens added to the LLM's vocabulary. When these tokens are added, trainable embeddings are created for them in the LLM's embedding layer. Specifically, $\\tilde{h_{seg}}$ is extracted from the LLM's last-layer embedding corresponding to the [SEG] token. This approach is common in visual language model architectures for grounding tasks, such as LISA [12].\n\n**Q2.4.** Fusion mechanism.\n\nThank you for your suggestion. We have moved its illustrative figure to the main text and will retain detailed explanations in the Appendix with clear guidance.\n\n**Q3. Evaluation and ablation study.**\n\n**Q3.1.** The qualitative output of the text.\n\nWe provided various qualitative outputs of different manipulation types, along with comparisons to existing methods, in **Appendix Figures 11-13**. We have added explicit indications in the relevant sections of the main text to ensure readers are directed to these figures.\n\n**Q3.2.** The performance of fine-tuned LLaVA.\n\nWhen LLaVA (as our data engine) processes images for analysis, its input includes highlighted tampered regions to facilitate the generation of more accurate manipulation analyses. This task, while related, is distinct from the primary manipulation detection task and relatively simpler. We evaluated the STS performance of our data engine (the fine-tuned LLaVA). The results, evaluated using the same setup as **Table 4**, show STS scores of 0.991, 0.976, and 0.990 across three models, indicating the reliability of our data engine.\n\n**Q3.3.** Classification (real/fake) performance.\n\nWe agree that including classification (real/fake) performance is important, as STS primarily assesses analysis text quality. We integrated classification as an evaluation dimension (*Recall*) within our ForgeryAnalysis-Eval benchmark, using output special tokens < REAL > and < FAKE >. The relevant results are presented in **Figure 1(b)** and **Figure 4**. Additionally, our method achieves classification accuracies of 0.989 on Columbia and 0.910 on CASIA1, further demonstrating its robust detection capability.\n\n**Q3.4.** More ablation experiments.\n\nWe appreciate your suggestion for further ablation. In our architecture, the decoder requires the LLM's outputted [SEG] token (specifically its $h_{seg}$ embedding) to guide segmentation, preventing us from completely removing the LLM's supervision. However, to achieve a similar validation purpose, we removed all clues, providing only simple text about detection results and [SEG] token. We explored two experimental settings:\n\n**Setting 1 (Only Detection Result):** We removed all clues, providing simple text about detection results (e.g., \"This is a < FAKE > image with modifications detected in the [SEG].\" or \"This is a < REAL > image without modifications detected, and the [SEG] is meaningless.\").\n\n**Setting 2 (Unstructured Clues):** We provided summarized text and clues without organizing them into the CoC structure. The order of different clues was randomized.\n\nWe used the same evaluation setup as described in **Table 5** in the main text, with *all-mpnet-base-v2* for STS calculation.\n\n| Setting | NIST16 | IMD20 | COCOGlide | ForgeryAnalysis-Eval | STS |\n| --- | --- | --- | --- | --- | --- |\n| ForgerySleuth | 0.518 | 0.710 | 0.562 | 9.45 | 0.961 |\n| Setting 1 | 0.448 | 0.526 | 0.533 | - | - |\n| Setting 2 | 0.505 | 0.681 | 0.564 | 7.94 | 0.785 |\n\nThese results demonstrate the effectiveness of the CoC structure in guiding the model towards more precise localization and explanations. Furthermore, if the LLM's output cues are absent, the segmentation performance also declines.\n\n**Q3.5. Failure cases analysis.**\n\nWe appreciate your suggestion for an expanded failure case analysis. While the original paper showcased two representative failure cases, we have expanded the failure case analysis in **Appendix Section D.5** as follows, with additional examples for each primary error pattern.\n\n*We analyzed ForgerySleuth's detection failures and identified two representative error patterns:*\n\n*a. Semantic Description Inaccuracies: In these cases, our model can accurately localize manipulated regions, but its textual analysis contains description errors. For instance, in the first example in Appendix Figure 9, the model misidentifies dogs as cats. Crucially, this does not impact the detection logic or localization accuracy. These errors often arise when specific semantic concepts in images are hard to detect clearly. Future work could improve this by enhancing the visual understanding capabilities of MLLM and LLM backbone models.*\n\n*b. Failures with Small Manipulations in Complex Scenes: In complex image content with small manipulated regions, the model's localization precision can drop, sometimes leading to incorrect textual analysis and, consequently, a detection failure. For example, in the second instance in Appendix Figure 9, a spliced cartoon character is present. Due to its small proportion and the image's overall complexity, the model only localized a part of the manipulation (like the guitar in hand) while overlooking the cartoon character. Detecting small region manipulations remains a significant challenge, and MLLMs tend to focus on global semantic information rather than fine-grained local details. Therefore, developing methods to effectively detect anomalies in small regions is a crucial research direction.*"}]

Author rebuttal:

Meta-review: This paper proposes ForgerySleuth, a framework that leverages multimodal large language models to detect, localize, and explain image manipulations, along with a new dataset (ForgeryAnalysis). Reviewers appreciated the paper's clear motivation, thorough experimentation, and the structured approach to interpretability. Reviewers i8FQ and sU1j in particular were excited about the proposed dataset and the effectiveness of the ForgerySleuth method. The chain-of-clues structure was noted as compelling for improving interpretability.

Initial concerns, such as unclear dataset construction, lack of practical examples, and limited discussion of limitations, were largely addressed in the rebuttal through clarifications and additional analysis. While some questions remain about reliance on AI-generated annotations and the degree of novelty, the work is technically solid and contributes meaningfully to an underexplored problem. The paper ended the discussion with four borderline accept ratings, and the AC recommends acceptance.
Decision: Accept (poster)