Innovative Image Fraud Detection with Cross-Sample Anomaly Analysis: The Power of LLMs
Authors: QiWen Wang, Zhenghao Lin, Chen Lin, Junqi Yang, Zhenzhe Ying, Weiqiang Wang
Accepted at: acl25

Innovative Image Fraud Detection with Cross-Sample Anomaly Analysis: The Power of LLMs

Qiwen Wang1, Junqi Yang3, Zhenghao Lin2, Zhenzhe Ying3, Weiqiang Wang3, Chen Lin2*
1Institute of Artificial Intelligence, Xiamen University, China
2School of Informatics, Xiamen University, China
3Ant Group, China
chenlin@xmu.edu.cn

Abstract
The financial industry faces a substantial workload in verifying document images. Existing methods based on visual features struggle to identify fraudulent document images due to the lack of visual clues on the tampering region. This paper proposes CSiAD (Cross-Sample Image Anomaly Detection) by leveraging LLMs to identify logical inconsistencies in similar images. This novel framework accurately detects forged images with slight tampering traces and explains anomaly detection results. Furthermore, we introduce CrossCred, a new benchmark of real-world fraudulent images with fine-grained manual annotations. Experiments demonstrate that CSiAD outperforms state-of-the-art image fraud detection methods by 79.6% (\( F1 \)) on CrossCred and deployed industrial solutions by 21.7% (\( F1 \)) on business data. The benchmark is available at https://github.com/XMUD/CSIAD.

1 Introduction
In recent years, almost all users have submitted image versions or screenshots of common documents to financial institutions to authenticate identity, ownership, transaction records, etc. With the advancement of image editing techniques (Chen et al., 2023; Tuo et al., 2024; Zhao et al., 2021; Zeng et al., 2024), malicious parties can generate highly realistic counterfeit materials at minimal cost, posing a serious threat in the financial sector. Automatic *image fraud detection* is critical to enhance security and operational efficiency.

Existing image fraud detection methods predominantly rely on visual features (Zhu et al., 2024; Yu et al., 2024; Guo et al., 2023). However, unlike natural images with complex textures and colors, images encountered in the financial sector are *document images* (Luo et al., 2023; Cloud, 2024), which share a simple background (e.g., typically white), highly structured layouts, and consistent graphical elements (e.g., uniform text fonts). The tampered regions hardly display any visual clues on the edge or surface. As illustrated in Figure 1(a), the SOTA image fraud detection method (Qu et al., 2023) cannot accurately locate the tampered region. Furthermore, it lacks explanations for its decision to speed up human verification.

Although image fraud detection is limited when based on *single-image* visual features, reviewers can detect anomalous images based on *cross-sample* logical inconsistencies in manual verification processes. As illustrated in Figure 1(b), the tampered region includes the account and address information because the forged payment record copies the account number from other images and alters the account name and address.

Since manually cross-sample reasoning is time-consuming, we propose CSiAD, which leverages LLMs for accurate and interpretable image fraud detection. CSiAD operates through two stages: (I) *Retrieval Ensemble*, which discovers potential image associations by aggregating visual and textual retrieval results, and (II) *Fact-Driven Cross-Sample Inference*, which detects and concludes anomaly from associated images. Specifically, in stage II, CSiAD first adopts an LLM to generate *logical rules* that an authentic image should follow, then refines the rules using unsupervised factual evidence, effectively mitigating LLM’s hallucinations while improving detection accuracy. Next, CSiAD detects, verifies, and explains the anomalies based on the self-generated rules.

To better address the gaps in image fraud detection in the financial sector, we present a new benchmark CrossCred, different from existing datasets (Dong et al., 2024; Qu et al., 2023; Wang et al., 2022; Qu et al., 2024) in the following aspects. (1) CrossCred is entirely from *real-world* data in complex financial fraud scenarios, where images are meticulously forged to deceive current
Figure 1: Comparison between TTD (Tampered Text Detection) and CSIAD (Cross-Sample Image Anomaly Detection). The forged image is a mobile payment record. Left: Existing TTD relies on visual features of a single sample, which struggles with subtle tampering traces. Right: CSIAD incorporates cross-sample analysis based on textual information, effectively increasing the accuracy and explainability of image fraud detection.

detection technology. On the contrary, most existing datasets are synthesized with random modifications. (2) CrossCred contains detailed human annotations, including tampered regions and explanations. (3) CrossCred supports image fraud detection via cross-sample inference (i.e., each fraud case is associated with a few similar images), while previous datasets only contain a single sample for each case.

Experiments on CrossCred show that CSIAD improves the image fraud detection performance, achieving an \( F1 \) of 81.9%, outperforming the SOTA image tampering detection method by 79.6%. Further evaluation on a real-world business dataset reveals that our model achieves an \( F1 \) of 85.7%, surpassing deployed industrial solutions by 21.7%.

In conclusion, our study makes several noteworthy contributions:

• Novel methodology. We present CSIAD: the first model that detects counterfeit document images by leveraging LLMs to uncover logical inconsistencies through cross-sample inference. In addition to enhancing detection accuracy, CSIAD provides fine-grained explanations that enable real-time monitoring and analysis of anomalous traffic.

• New benchmark. We release CrossCred: a real-world dataset that covers various types of fraud images in the financial sector with detailed human annotations to evaluate image fraud detection methods.

• Superior performance in academic and business scenarios. CSIAD achieves accurate and interpretable fine-grained anomaly detection on both CrossCred and industrial datasets, showing significant improvements over academic baselines and sophisticated business solutions.

2 Related Work

Document Image Tampering Datasets. Recent datasets like RIFLC (Cloud, 2022), TTI (Cloud, 2023), TextTamper (Dong et al., 2024), DocTamper (Qu et al., 2023), Tampered-IC13 (Wang et al., 2022), and OSTF (Qu et al., 2024) have been developed for tampering detection. However, they focus on single-sample analysis, lack fine-grained manual annotations, and fail to replicate real-world fraud scenarios. In contrast, our proposed CrossCred features fine-grained manual annotations and supports cross-sample detection.

Document Image Fraud Detection Methods. Existing methods for document image fraud detection focus on visual element anomalies at the individual sample level (Qu et al., 2023; Sun et al., 2024; Wang et al., 2022; Dong et al., 2024; Xu et al., 2022; Qu et al., 2024; Sun et al., 2019). However, they rely on obvious visual cues and struggle with subtle tampering traces, revealing their limitations. In this study, we propose CSIAD, leveraging LLMs for cross-sample analysis to complement existing approaches.

Self-Reflection Ability of LLMs. Recent research on self-reflection introduces advanced prompting strategies, enabling LLMs to refine responses via feedback (Shinn et al., 2023; Bai et al.,
2022; Paul et al., 2024; Madaan et al., 2023). Feedback can come from external sources (e.g., models, tools, knowledge bases) (Gou et al., 2024; Olausson et al., 2024; Chen et al., 2024; Nathani et al., 2023; Kim et al., 2023) or internal self-assessment (Madaan et al., 2023; Zhang et al., 2024; Yao et al., 2024; Lightman et al., 2024). However, studies (Huang et al., 2024; Zhang et al., 2024; Stechly et al., 2023; Liang et al.) show LLMs’ correction capabilities are unreliable without external feedback. In this study, we propose a fact-driven verification mechanism, guiding LLMs to autonomously select reflection paths based on objective factual data, ensuring reliable outputs.

3 Methodology

3.1 Framework Overview

To determine whether a document image is fraudulent, a natural workflow is to “consult” relevant images to identify the potential anomalies. As shown in Figure 2, CSiAD contains four modules: the Retrieval Ensemble module aims to recall as many relevant images given the query image \( I_q \) to form a batch of suspicious images \( S_q \); the Sample-Specific Rule Generation module generates a set of rules \( \mathcal{R}_{ini} \) that normal images must adhere to; the Fact-Driven Verification module further validates and filters the rules to obtain \( \mathcal{R}_{ver} \); and finally, the Rule-based Anomaly Detection module detects and interprets the anomalies in \( S_q \).

3.2 Retrieval Ensemble

We combine visual and textual retrieval to maximize the recall of relevant images. Adopting visual retrieval alone can lead to increased computational complexity by analyzing irrelevant images, e.g., certificates may share similar visual appearances yet contain different information (Figure 3(a)). It can also reduce detection accuracy by missing relevant images, e.g., images differ visually while conveying related content (Figure 3(b)).

Furthermore, we combine dense retrieval and sparse retrieval. The former captures document-level semantic relationships, while the latter excels at nuanced keyword connections, benefiting highly structured documents.

Specifically, the visual features are extracted by CN-CLIP (ViT-B/16) (Yang et al., 2023), dense textual vectors by CN-CLIP (RoBERTa-wwm-Base) (Yang et al., 2023). We use an LLM to extract structured texts \( \mathcal{T} \) and a HashingVectorizer to extract sparse textual vectors.

The similarity score between the given image \( I_q \) and any image in the database \( I_i \) is computed by:

\[
s(I_q, I_i) = \alpha \times \cos(\mathbf{v}_q, \mathbf{v}_i) \\
+ \beta \times \cos(\mathbf{e}_q, \mathbf{e}_i) \\
+ \gamma \times \cos(\mathbf{t}_q, \mathbf{t}_i),
\]

where \( \mathbf{v}_q, \mathbf{e}_q, \mathbf{t}_q \) are the visual feature, dense textual vector, and sparse textual vector, respectively. \( \cos(\cdot, \cdot) \) denotes the cosine similarity. The weighting coefficients \( \alpha, \beta, \) and \( \gamma \) are meticulously calibrated through preliminary experiments.

Finally, the retrieved images associated with \( I_q \) are collected to form \( S_q \):

\[
S_q = \{I_i \mid \forall s(I_q, I_i) \geq \delta\},
\]

where \( \delta \) is the correlation truncation score.

3.3 Sample-Specific Rule Generation

The retrieved batch of images \( S_q \) shows high similarity, suggesting potential anomalies. Intuitively, we can use an LLM \( \mathcal{M} \) to infer abnormal images based on a set of rules. The rules can be predefined, e.g., by human experts. Unfortunately, since new types of documents emerge, e.g., payment via a newly launched app, it is impractical to enumerate all possible rules manually. An LLM can explicitly or implicitly generate the rules. However, if no groundings are provided, the LLM often generates trivial rules, e.g., *the payment amount should be negative*.

Consequently, we encourage \( \mathcal{M} \) to generate sample-specific rules. As shown in Figure 2(b), we provide \( \mathcal{M} \) with structured text \( \mathcal{T} \) (details in the last subsection) and few-shot business rule examples \( \mathcal{F}_{rule} \). Given the potentially lengthy context of \( \mathcal{T} \), the first step in the prompt \( p_{gen\_rule} \) for Sample-Specific Rule Generation is to focus on key fields within \( \mathcal{T} \). Leveraging these key fields and \( \mathcal{F}_{rule} \), we then facilitate the generation of initial rules \( \mathcal{R}_{ini} \):

\[
\mathcal{R}_{ini} = \mathcal{M}(p_{gen\_rule}(\mathcal{T}, \mathcal{F}_{rule}))
\]

To ensure that edge anomaly cases such as unconventional or emerging anomaly patterns are also detected, we incorporate a temperature sampling strategy (Chang et al., 2023), which enables \( \mathcal{M} \) to generate more creative rules during decoding by utilizing a higher sampling temperature.

3.4 Fact-Driven Rule Verification

We implement a fact-based verification mechanism to tackle potential errors or hallucinations in \( \mathcal{R}_{ini} \),
Figure 2: Framework for CSIAD. The diagram outlines the key modules: Retrieval Ensemble, Sample-Specific Rule Generation, Fact-Driven Verification, and Rule-Based Anomaly Analysis.

e.g., "Identical taxpayer identification number → Same tax paid". The assumption is that if we hold a fact repository that covers a sufficient number of scenarios, we can detect and remove the erroneous or hallucinatory rules.

Specifically, we construct an anomaly-free fact repository \( \mathcal{G} \). Our key insight is that the quality of rule verification improves as the size of \( \mathcal{G} \) increases, as a larger repository covers a broader range of scenarios described by the rules, thus enhancing the comprehensiveness of the verification. \( \mathcal{G} \) consists of historical images submitted in the system. Firstly, real-world business databases are typically vast, with an inherently low anomaly rate (approximately one in a thousand). Secondly, the anomaly rate can be further reduced by employing basic business anomaly detection methods to filter out obvious outliers, making \( \mathcal{G} \) reliable for ground truth fact verification.

We then use \( \mathcal{G} \) to validate the rules in \( \mathcal{R}_{ini} \). Formally, we first retrieve factual images from \( \mathcal{G} \) that match \( I_q \):
\[
\mathcal{G}_q = \{ I_q^{(i)} \mid \forall I_q^{(i)} \in \mathcal{G},\ s(I_q, I_q^{(i)}) \geq \delta \}
\]
Then, for \( r_{ini}^{(i)} \in \mathcal{R}_{ini} \) to be validated, we employ a prompt \( p_{ver\_rule} \) designed for fact-driven rule verification, guiding \( \mathcal{M} \) to examine \( \mathcal{G}_q \):
\[
\mathcal{R}_{ver} = \{ r_{ini}^{(i)} \mid \forall r_{ini}^{(i)} \in \mathcal{R}_{ini}, \\
\quad \mathcal{M}(p_{ver\_rule}(r_{ini}^{(i)}, \mathcal{G}_q)) = \text{True} \}
\]
If \( \mathcal{M} \) searches for any instances where the conditions specified by \( r_{ini}^{(i)} \) are violated in \( \mathcal{G}_q \), it signifies that \( r_{ini}^{(i)} \) is not universally applicable to all valid images and is subsequently pruned.
(a) Visually similar & Textual content different

![Two visually similar financial document images with entirely different textual content.](page_186_72_495_180.png)

(b) Visually dissimilar & Textual content highly correlated

![Two visually dissimilar financial document images with highly related textual content.](page_186_256_495_180.png)

Figure 3: Visual and Textual Discrepancies in Financial Document Images. (a) Two visually similar financial document images with entirely different textual content. (b) Two visually dissimilar financial document images with highly related textual content.

3.5 Anomaly Detection and Verification

Once a set of fact-grounded rules is established, the next step is to leverage \( \mathcal{R}_{ver} \) to guide the anomaly detection phase(Figure 2(d)).

Rule-Based Anomaly Detection. We let \( \mathcal{M} \) to sequentially verify each rule in \( \mathcal{R}_{ver} \), i.e., whether the given image group \( \mathcal{S}_q \) exhibits rule violations through prompt \( p_{gen\_con} \). We denote the initial anomaly conclusions as:

\[
C_{ini} = \{\mathcal{M}(p_{gen\_con}(\mathcal{T}, r^{(i)}_{ver})) | \forall r^{(i)}_{ver} \in \mathcal{R}_{ver}\}
\]

each anomaly conclusion \( c^{(i)}_{ini} \in C_{ini} \) includes the relevant image IDs and the explanation that details the cause of the anomaly and its location.

Fact-Driven Anomaly Verification. Given the lengthy and complex contexts within the structured text of multiple images, there is an elevated risk of factual errors in \( C_{ini} \). To ensure reliable anomaly detection, it is essential to verify \( C_{ini} \). During this verification phase, the fact repository denotes the structured textual \( \mathcal{T} \) furnished to \( \mathcal{M} \). We assess the validity of each conclusion to determine whether it is a degenerated case due to insufficient supporting evidence in \( \mathcal{T} \) through a specifically designed prompt \( p_{ver\_con} \). The verified anomaly conclusions are denoted as \( C_{ver} \).

\[
C_{ver} = \{c^{(i)}_{ini} | \forall c^{(i)}_{ini} \in C_{ini}, \\
\mathcal{M}(p_{ver\_con}(\mathcal{T}, c^{(i)}_{ini})) = \text{True}\}.
\]

Table 1: Comparison between CrossCred and existing image tampering detection datasets (DocTamper (Qu et al., 2023), TextTamper (Dong et al., 2024)).

<table>
  <tr>
    <th>Aspect</th>
    <th>DocTamper</th>
    <th>TextTamper</th>
    <th>CrossCred</th>
  </tr>
  <tr>
    <td>Data Source</td>
    <td>Simulation</td>
    <td>Simulation</td>
    <td>Real-world</td>
  </tr>
  <tr>
    <td>Tamper</td>
    <td>Random</td>
    <td>Random</td>
    <td>Targeted</td>
  </tr>
  <tr>
    <td># Doc Type</td>
    <td>N/A</td>
    <td>N/A</td>
    <td>61</td>
  </tr>
  <tr>
    <td>Annotation</td>
    <td>Coarse</td>
    <td>Coarse</td>
    <td>Fine-grained</td>
  </tr>
  <tr>
    <td>Detection</td>
    <td>Single-Sample</td>
    <td>Single-Sample</td>
    <td>Multi-Sample</td>
  </tr>
</table>

For the concrete prompts utilized in each module, please refer to Appendix D.

4 Benchmark

As shown in Table 1, existing document image anomaly detection datasets synthesize images by tampering with a random region, e.g., copy-move, splicing, etc. They fail to simulate real-world targeted image fraud, such as altering critical fields like amounts, dates, and identity information. They lack fine-grained manual annotations. Additionally, they only contain one sample for an abnormal image, forcing the detection method to rely purely on the visual features of a single sample. While forged images are produced in batches nowadays, single-sample detection is suboptimal.

To address these shortcomings, we propose a benchmark consisting of real-world fraud images, where the tampering is proficiently targeted for financial theft and crimes. The benchmark\footnote{Data strictly desensitized and encrypted with comprehensive protection measures, containing no Personally Identifiable Information.} contains fine-grained, manually annotated anomalies, and supports multi-sample detection.

4.1 Benchmark Construction

The benchmark construction process consists of four steps, as illustrated in Figure 4.

Step 1: Image Grouping. In the financial service industry, diverse document types are utilized, such as tax invoices, financial statements, identity documents, contracts, etc. Each document type can be associated with various formats, e.g., mobile payment records by different apps. Images of the same document type and format are grouped together to form a candidate *case* for analysis.
Step 2: Pre-Annotation. It is infeasible to check all images manually, given the large volume. Thus, we leverage an LLM to pre-annotate the image groups formed in Step 1. If anomalous behaviors are detected among the images within a group, we classify them as an anomalous case and retain them for further analysis. Specifically, from a collection of 400K business images, we identified 4,063 potential anomalous cases.

Step 3: Human Annotation. Our analysis indicates that pre-annotated anomalous cases contain a notable proportion of false positives. To enhance reliability, we manually verify these cases through expert evaluation, involving (1) logical consistency checks against financial rules and (2) commonsense validity assessments. Each case undergoes ≥2-minute expert review, resulting in 558 validated anomalous cases. Each case includes multiple images of the same document type and format. Detailed annotations identify the (possibly multiple) anomalous points in each image, specifying their locations and interpretations.

Our data annotation work was carried out by a professional annotation team within the company. This team consists of 10 experienced annotators and follows a strict three-level quality control process: 1) Preliminary annotation by annotators. 2) Double-checking by a quality inspection team leader and the annotation project leader. 3) Final confirmation by the data requester. For further details on the annotation process, see Appendix G.

Step 4: Deduplication and Balancing. We observed frequent resubmissions of images (e.g., through different URLs) and imbalanced distributions of document types. To address this, we performed deduplication to eliminate redundant images and balanced the document types to ensure uniform distribution, resulting in the retention of 109 anomalous cases.

4.2 Benchmark Statistics
Our benchmark comprises 109 cross-sample anomaly cases (396 total samples, average of 3.63 samples per case) and 109 randomly selected anomaly-free images, totaling 505 images, spanning 61 document types. The complete type distribution of CrossCred is detailed in Appendix F. We plan to release the CrossCred benchmark upon acceptance.

5 Experiments

5.1 Implementation details

Settings. For the experiments, we use two LLMs: Qwen2-VL (Wang et al., 2024) for structured text extraction and Qwen2.5-72B-Instruct (Qwen et al., 2025) for the rest of the tasks. Except for the 1.0 temperature used when sampling the creative rules, we use a temperature of 0.0 when calling the API. The retrieval module adopts weights \( \alpha = 0.5, \beta = 0.1, \gamma = 0.4 \), which achieve superior relevant sample recall performance (Figure ??). The correlation truncation score \( \delta \) is set to 0.88.

Evaluation Set. For evaluation set construction, we adopt a sampling approach: one image per anomaly case is allocated to the evaluation set as \( I_q \), and the remaining images in the case are included in the database for retrieval. This results in a balanced evaluation set containing 218 images (109 normal / 109 anomalous), while the reference database comprises 98,758 images.

Evaluation Metrics. Given that each anomaly case may contain multiple anomalies, we implement dual-level evaluation metrics (coarse/fine-grained) to assess the performance.

Coarse-grained metrics. The primary objective is to determine whether an image is correctly identified as anomalous. We treat anomaly detection as a binary classification task, evaluated through standard measures, including Accuracy (\( Acc \)), Recall (\( Rec \)), Precision (\( Pre \)), and F1 score (\( F1 \)).

Fine-grained metrics. These are designed to verify whether multiple anomalies are correctly located and explained in each case. Specifically, an LLM is called to compare \( C_{ver} \) with ground-truth annotations. Fine-grained accuracy is calculated based on whether the localization and explanation
Table 2: Evaluation results on CrossCred dataset. Retriever → Single refers to retrieving relevant images using only the image modality, while Retriever → Ensemble refers to retrieving images using ensemble retrieval. Methods marked with * indicate the use of BizRule for reflection.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="4">Coarse-grained</th>
    <th colspan="3">Fine-grained</th>
  </tr>
  <tr>
    <th>Acc</th>
    <th>Rec</th>
    <th>Pre</th>
    <th>F1</th>
    <th>Rec<sup>†</sup></th>
    <th>Pre<sup>†</sup></th>
    <th>F1<sup>†</sup></th>
  </tr>
  <tr>
    <td>DTD</td>
    <td>48.6</td>
    <td>43.1</td>
    <td>48.5</td>
    <td>45.6</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Tifdm</td>
    <td>49.1</td>
    <td>34.9</td>
    <td>48.7</td>
    <td>40.6</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <th colspan="8">Retriever → Single</th>
  </tr>
  <tr>
    <td>CoT Prompt</td>
    <td>29.8</td>
    <td>35.8</td>
    <td>32.0</td>
    <td>33.8</td>
    <td>10.2</td>
    <td>31.3</td>
    <td>15.4</td>
  </tr>
  <tr>
    <td>Self-Consistency*</td>
    <td>76.6</td>
    <td>68.8</td>
    <td>81.5</td>
    <td>74.6</td>
    <td>28.2</td>
    <td>45.7</td>
    <td>34.9</td>
  </tr>
  <tr>
    <td>Self-Check*</td>
    <td>74.3</td>
    <td>72.5</td>
    <td>75.2</td>
    <td>73.6</td>
    <td>29.9</td>
    <td>47.9</td>
    <td>36.8</td>
  </tr>
  <tr>
    <td>Self-Reflection*</td>
    <td>74.8</td>
    <td>74.3</td>
    <td>75.0</td>
    <td>74.7</td>
    <td>29.6</td>
    <td>47.9</td>
    <td>36.6</td>
  </tr>
  <tr>
    <td>CSIAD</td>
    <td>78.0</td>
    <td>75.2</td>
    <td>79.6</td>
    <td>77.4</td>
    <td>30.2</td>
    <td>50.5</td>
    <td>37.8</td>
  </tr>
  <tr>
    <th colspan="8">Retriever → Ensemble</th>
  </tr>
  <tr>
    <td>CoT Prompt</td>
    <td>31.7</td>
    <td>36.7</td>
    <td>33.3</td>
    <td>34.9</td>
    <td>9.8</td>
    <td>35.8</td>
    <td>15.4</td>
  </tr>
  <tr>
    <td>Self-Consistency*</td>
    <td>77.1</td>
    <td>69.7</td>
    <td>81.7</td>
    <td>75.3</td>
    <td>28.6</td>
    <td>48.2</td>
    <td>35.9</td>
  </tr>
  <tr>
    <td>Self-Check*</td>
    <td>72.5</td>
    <td>73.4</td>
    <td>72.1</td>
    <td>72.7</td>
    <td>29.9</td>
    <td>49.5</td>
    <td>37.3</td>
  </tr>
  <tr>
    <td>Self-Reflection*</td>
    <td>74.3</td>
    <td>77.1</td>
    <td>73.0</td>
    <td>75.0</td>
    <td>30.6</td>
    <td>53.4</td>
    <td>38.9</td>
  </tr>
  <tr>
    <td>CSIAD</td>
    <td><b>82.1</b></td>
    <td><b>80.7</b></td>
    <td><b>83.0</b></td>
    <td><b>81.9</b></td>
    <td><b>32.1</b></td>
    <td><b>55.1</b></td>
    <td><b>40.6</b></td>
  </tr>
</table>

of \( c_{i}^{(i)} \) match the annotations. The following aspects are emphasized: Recall (\( Rec^† \)), Precision (\( Pre^† \)), and F1 score (\( F1^† \)). For a more detailed calculation procedure of the fine-grained metrics, please refer to Appendix B.

5.2 Baseline
We conducted a comparative analysis with state-of-the-art document image tampering detection approaches in the field of computer vision, including DTD (Qu et al., 2023) and Tifdm (Dong et al., 2024).

We also evaluate CSIAD against several baselines by applying advanced LLM inference strategies on CSIAD’s retrieved relevant images. These baselines include the Standard CoT Prompt (Kojima et al., 2022), Self-Consistency (Wang et al., 2023), Self-Reflection (Shinn et al., 2023), and Self-Check (Miao et al., 2024). Since methods incorporating reflection processes are not consistently reliable without external knowledge (Huang et al., 2024; Zhang et al., 2024), we offer a manually curated rule library (BizRule) from the production system to serve as a reference for reflection.

5.3 Main Results
Table 2 summarizes the experimental results on CrossCred, demonstrating CSIAD’s superiority across all baselines. Due to the lack of publicly available real-world datasets from similar scenarios, this paper does not include a comparison across different open benchmark datasets.

Compared with traditional image tampering detection methods, their performance in scenarios lacking evident tampering traces is suboptimal (Table 2). In contrast, CSIAD significantly outperforms these methods, achieving average \( F1 \) improvements of 80.2% and 90.1% in single-modal retrieval and ensemble retrieval, respectively. This highlights the effectiveness of CSIAD in addressing challenging cases where traditional methods fail.

Table 2 also shows that using the ensemble retriever significantly improves \( Rec \) across all evaluated methods and boosts all metrics for CSIAD. To further explore the impact of retrieval ensemble, we conducted a comparative analysis between the single-modal retriever and the ensemble retriever in retrieving relevant images. We annotate the relevant images and report the recall rates in Table 3. The ensemble retriever notably increases \( Recall@1 \), \( Recall@3 \), and \( Recall@5 \). Thus, fusing visual and textual retrieval results can obtain more relevant images, and improve the recall rate of anomaly detection in Table 2.

At the same time, the improved recall of relevant images leads to decreased \( Acc \) and \( Pre \) for some baselines because it is more challenging to identify image fraud in a larger set. On the contrary, CSIAD leverages a fact-driven verification mechanism that dynamically allows the LLM to adapt its verification path based on specific queries. This adaptability significantly improves both \( Acc \) and \( Pre \), with gains of 5.3% and 4.3%, respectively.

Compared with other baselines using LLM inference strategies, our method achieves optimal fine-grained performance, attaining an \( F1^† \) of 40.6%, a 4.4% improvement over the best baseline. This gain stems from the fact-driven verification mechanism, which grounds generated rules and anomaly conclusions with objective factual data rather than subjective model inferences, effectively reducing fine-grained misjudgments.

Table 3: Comparison of the recall rates between Retriever → Single and Retriever → Ensemble in retrieving relevant images.

<table>
  <tr>
    <th>Method</th>
    <th>Recall@1</th>
    <th>Recall@3</th>
    <th>Recall@5</th>
  </tr>
  <tr>
    <td>Retriever → Single</td>
    <td>28.2</td>
    <td>62.4</td>
    <td>77.7</td>
  </tr>
  <tr>
    <td>Retriever → Ensemble</td>
    <td><b>33.8</b></td>
    <td><b>73.2</b></td>
    <td><b>90.6</b></td>
  </tr>
</table>
Table 4: The Effect of the Different Modules. We conduct comparisons across four modules: Temperature Sampling(TS), Rule Generation(RG),Rule Verification(RV),and Anomaly Verification(AV)

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">Coarse-grained</th>
    <th colspan="3">Fine-grained</th>
  </tr>
  <tr>
    <th>Acc</th>
    <th>Rec</th>
    <th>Pre</th>
    <th>F1</th>
    <th>Rec<sup>e</sup></th>
    <th>Pre<sup>e</sup></th>
    <th>F1<sup>e</sup></th>
  </tr>
  <tr>
    <td>RG RV AV</td>
    <td>31.7</td>
    <td>36.7</td>
    <td>33.3</td>
    <td>34.9</td>
    <td>9.8</td>
    <td>35.8</td>
    <td>15.4</td>
  </tr>
  <tr>
    <td>w/o TS</td>
    <td>61.9</td>
    <td>79.8</td>
    <td>58.8</td>
    <td>67.7</td>
    <td>33.7</td>
    <td>44.7</td>
    <td>38.4</td>
  </tr>
  <tr>
    <td>✓ ✓ ✓</td>
    <td>79.4</td>
    <td>77.1</td>
    <td>80.8</td>
    <td>78.9</td>
    <td>31.9</td>
    <td>51.0</td>
    <td>39.3</td>
  </tr>
  <tr>
    <td>✓ ✓ ✓</td>
    <td>61.9</td>
    <td>76.2</td>
    <td>60.3</td>
    <td>66.1</td>
    <td>31.9</td>
    <td>46.1</td>
    <td>37.7</td>
  </tr>
  <tr>
    <td>✓ ✓ ✓</td>
    <td>79.8</td>
    <td>75.2</td>
    <td>82.8</td>
    <td>78.9</td>
    <td>30.2</td>
    <td>53.7</td>
    <td>38.7</td>
  </tr>
  <tr>
    <td>w/ TS</td>
    <td>60.6</td>
    <td>82.6</td>
    <td>57.3</td>
    <td>67.7</td>
    <td>34.6</td>
    <td>45.2</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td>✓ ✓ ✓</td>
    <td>81.2</td>
    <td>81.7</td>
    <td>80.9</td>
    <td>81.3</td>
    <td>33.0</td>
    <td>52.0</td>
    <td>40.4</td>
  </tr>
  <tr>
    <td>✓ ✓ ✓</td>
    <td>62.4</td>
    <td>82.6</td>
    <td>58.8</td>
    <td>68.7</td>
    <td>34.6</td>
    <td>47.6</td>
    <td>40.1</td>
  </tr>
  <tr>
    <td>✓ ✓ ✓</td>
    <td>82.1</td>
    <td>80.7</td>
    <td>83.0</td>
    <td>81.9</td>
    <td>32.1</td>
    <td>55.1</td>
    <td>40.6</td>
  </tr>
</table>

5.4 Ablation Analysis

Effectiveness of Temperature Sampling. We introduced a temperature sampling strategy during the rule generation phase, utilizing a high temperature (\( T = 1.0 \)). Table 4 shows that \( \mathcal{M} \)'s performance with TS outperforms that without TS (\( T = 0.0 \)). In particular, \( Rec \) increased by 7.31%, reaching 80.7%. This phenomenon arises because \( T = 0.0 \) produces deterministic rules, which are insufficient for the probabilistic nature of real-world image diversity. In contrast, higher-temperature sampling (\( T > 0.0 \)) expands \( \mathcal{M} \)'s probability space, enabling the generation of diversified rules that capture edge anomalies more effectively than fixed-pattern ones.

Effectiveness of Rule Verification Modules. Table 4 also shows that removing the rule verification process—where \( \mathcal{M} \) reflects on anomalies based solely on initial rules—improves recall but significantly reduces precision, with \( Pre \) dropping by 14.2% (with TS) and 13.6% (without TS). We conclude that well-defined rules provide \( \mathcal{M} \) with clear analytical guidance, enabling it to focus more effectively on identifying and interpreting critical information.

Quality of CSID's Reflection. We further analyzed the quality of rule reflection results in Table 5. Based on the correctness of pre- and post-verification results, all reflective processes were categorized into three types: (1) Invalid Reflection (\( \times \Rightarrow \times \)): both pre- and post-reflection results are incorrect; (2) Toxic Reflection (\( \checkmark \Rightarrow \times \)): an initially correct response becomes incorrect after reflection; (3) Valid Reflection (\( \times \Rightarrow \checkmark \)): an initially incorrect conclusion is corrected.

The statistics for each reflection category across different methods are presented in Table 5. Our analysis demonstrates that CSID optimizes rule verification in two aspects: boosting Valid Reflection rates (correcting errors while preserving valid rules) and reducing Toxic Reflection rates (erroneously filtering out valid rules). Maintaining accurate rules is crucial in anomaly detection, and CSID strikes an optimal balance between correction and retention. These results highlight how CSID utilizes reflection to improve both accuracy (by fixing errors) and reliability (by reducing over-correction).

Table 5: Comparison of reflection outcomes across different LLM-based reasoning strategies. We categorize each case based on the correctness of the pre- and post-reflection outputs: Invalid, Toxic, and Valid.

<table>
  <tr>
    <th>Method</th>
    <th>Invalid</th>
    <th>Toxic</th>
    <th>Valid†</th>
  </tr>
  <tr>
    <td>Self-Consistency</td>
    <td>40.4%</td>
    <td>5.2%</td>
    <td>54.4%</td>
  </tr>
  <tr>
    <td>Self-Check</td>
    <td>42.6%</td>
    <td>3.7%</td>
    <td>53.7%</td>
  </tr>
  <tr>
    <td>Self-Reflection</td>
    <td>44.1%</td>
    <td>4.00%</td>
    <td>51.9%</td>
  </tr>
  <tr>
    <td>CSIAD</td>
    <td>43.4%</td>
    <td>2.1%</td>
    <td>54.5%</td>
  </tr>
</table>

Table 6: Comparison between the deployed industrial solution and CSID on both the proposed benchmark (CrossCred) and real-world business data (Biz.Data).

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th rowspan="2">Method</th>
    <th colspan="4">Coarse-grained</th>
  </tr>
  <tr>
    <th>Acc</th>
    <th>Rec</th>
    <th>Pre</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>CrossCred</td>
    <td>Deployed</td>
    <td>59.6</td>
    <td>28.4</td>
    <td>75.6</td>
    <td>41.3</td>
  </tr>
  <tr>
    <td></td>
    <td>CSIAD</td>
    <td>82.1</td>
    <td>80.7</td>
    <td>83.0</td>
    <td>81.9</td>
  </tr>
  <tr>
    <td>Biz.Data</td>
    <td>Deployed</td>
    <td>84.3</td>
    <td>67.9</td>
    <td>73.1</td>
    <td>70.4</td>
  </tr>
  <tr>
    <td></td>
    <td>CSIAD</td>
    <td>92.0</td>
    <td>85.7</td>
    <td>85.7</td>
    <td>85.7</td>
  </tr>
</table>

5.5 Comparative Analysis with Existing Industrial Solution

To compare CSID with industrial solutions\(^2\), we evaluate the latter on CrossCred and validate CSID using industry-confirmed anomalous cases. Experimental results are presented in Table 6.

Performance of Industrial Solution on CrossCred. We conducted a comparative evaluation of CSID against the solution deployed in business systems. The experimental results, as shown in Table 6, demonstrate that CSID significantly outperforms the existing industrial solution

\(^2\)Due to security protocols, the methodology behind business solutions remains confidential and is not publicly disclosed.
on CrossCred, achieving a 98.3% improvement in \( F1 \). This underscores the superior effectiveness of CSIAD in handling the CSIAD task.

Performance of CSIAD on Existing Business Data. CrossCred is specifically designed for the CSIAD task. To provide a more comprehensive assessment of CSIAD’s performance in real-world business scenarios, we collected a batch of real-world cases confirmed as anomalous by industrial practices for further evaluation. The experimental results reveal that CSIAD achieves significant performance in these real-world scenarios, with a 21.7% improvement in \( F1 \), further demonstrating its effectiveness in handling complex, real-world cases.

6 Conclusion

In this work, we introduce CSIAD, the first method to leverage LLM reasoning for cross-sample logical anomaly detection in image fraud detection, showcasing the potential of using textual features to uncover image anomalies. We propose an innovative Fact-Driven Verification mechanism as a post-generation refinement method to address potential errors or hallucinations by LLMs, enabling LLM reflections to autonomously select reasoning paths and focus on objective factual data, effectively reducing misjudgments. Experimental results demonstrate that CSIAD significantly outperforms traditional image tampering detection methods and existing industrial solutions. Given these advantages, we believe our framework can benefit various real-world applications.

7 Acknowledgments

Chen Lin is the corresponding author. Chen Lin is sponsored by the Natural Science Foundation of China (No.62372390,62432011) and CAAI-Ant Group Research Fund.

8 Limitations

Our research primarily focuses on images of documents encountered in financial business scenarios. Despite the promising results demonstrated in this study, our method has several limitations. Firstly, we only have access to images within a limited time frame. Although we have comprehensively mined and showcased the existing anomaly types and image categories in the CrossCred benchmark, our research scope remains constrained by the available data. Additionally, due to privacy and security considerations, the scale of our benchmark is restricted. If future access to a broader image database becomes available, we will be able to acquire a more diverse and research-worthy set of anomaly cases. Secondly, one potential limitation of CSIAD is its reliance on the instruction-following capabilities of LLMs. While CSIAD is designed to leverage advanced LLMs for cross-sample analysis, the use of smaller or less capable LLMs may result in suboptimal performance, as their ability to interpret and execute complex instructions is comparatively weaker. Solutions such as enhancing the instruction-following capabilities of smaller LLMs through fine-tuning or leveraging knowledge distillation techniques to transfer the capabilities of larger models to smaller ones could be explored.

References

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022. Constitutional ai: Harmlessness from ai feedback. Preprint, arXiv:2212.08073.

Chung-Ching Chang, David Reitter, Renat Akstitov, and Yun-Hsuan Sung. 2023. KL-divergence guided temperature sampling. Preprint, arXiv:2306.01286.

Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. 2023. Textdiffuser: Diffusion models as text painters. In Advances in Neural Information Processing Systems, volume 36, pages 9353–9387. Curran Associates, Inc.

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations.

Alibaba Cloud. 2022. Real-world image forgery localization challenge.

Alibaba Cloud. 2023. Detecting tampered text in images tianchi competition.

Alibaba Cloud. 2024. Ai identity verification - financial certificate tampering detection.
Li Dong, Weipeng Liang, and Rangding Wang. 2024. Robust text image tampering localization via forgery traces enhancement and multiscale attention. IEEE Transactions on Consumer Electronics, 70(1):3495–3507.

Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. CRITIC: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations.

Xiao Guo, Xiaohong Liu, Zhiyuan Ren, Steven Grosz, Iacopo Masi, and Xiaoming Liu. 2023. Hierarchical fine-grained image forgery detection and localization. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3155–3165.

Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations.

Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. In Advances in Neural Information Processing Systems, volume 36, pages 39648–39677. Curran Associates, Inc.

Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.

Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889–17904. Association for Computational Linguistics.

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let’s verify step by step. In The Twelfth International Conference on Learning Representations.

Dongliang Luo, Yu Zhou, Rui Yang, Yuliang Liu, Xianjin Liu, Jishen Zeng, Enming Zhang, Biao Yang, Ziming Huang, Lianwen Jin, and Xiang Bai. 2023. Icdar 2023 competition on detecting tampered text in images. In Document Analysis and Recognition - IC-DAR 2023, pages 587–600, Cham. Springer Nature Switzerland.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems, volume 36, pages 46534–46594. Curran Associates, Inc.

Ning Miao, Yee Whye Teh, and Tom Rainforth. 2024. Selfcheck: Using LLMs to zero-shot check their own step-by-step reasoning. In The Twelfth International Conference on Learning Representations.

Deepak Nathani, David Wang, Liangming Pan, and William Wang. 2023. MAF: Multi-aspect feedback for improving reasoning in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6591–6616.

Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024. Is self-repair a silver bullet for code generation? Preprint, arXiv:2306.09896.

Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2024. REFINER: Reasoning feedback on intermediate representations. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1100–1126.

Chenfan Qu, Chongyu Liu, Yuliang Liu, Xinhong Chen, Dezhi Peng, Fengjun Guo, and Lianwen Jin. 2023. Towards robust tampered text detection in document image: New dataset and new solution. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5937–5946.

Chenfan Qu, Yiwu Zhong, Fengjun Guo, and Lianwen Jin. 2024. Generalized tampered scene text detection in the era of generative ai. arXiv preprint arXiv:2407.21422.

Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Preprint, arXiv:2303.11366.

Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. 2023. Gpt-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems. Preprint, arXiv:2310.12397.
Hao Sun, Jie Cao, Zhida Zhang, Tao Wu, Kai Zhou, and Huaibo Huang. 2024. Learning fine-grained and semantically aware mamba representations for tampered text detection in images. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 44–57. Springer.

Keyu Sun, Gang Cao, Qian Zhao, and Jianglong Zhang. 2019. Differential abnormality-based tampering detection in digital document images. In 2019 IEEE/ACIS 18th International Conference on Computer and Information Science (ICIS), pages 145–149.

Yuxiang Tuo, Wangmeng Xiang, Jun-Yan He, Yifeng Geng, and Xuansong Xie. 2024. Anytext: Multilingual visual text generation and editing. In The Twelfth International Conference on Learning Representations.

Peng Wang, Shuai Bai, Siman Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. Preprint, arXiv:2409.12191.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.

Yuxin Wang, Hongtao Xie, Mengting Xing, Jing Wang, Shenggao Zhu, and Yongdong Zhang. 2022. Detecting tampered scene text in the wild. In European Conference on Computer Vision, pages 215–232. Springer.

Wenbo Xu, Junwei Luo, Chuntao Zhu, Wei Lu, Jinhua Zeng, Shaopei Shi, and Cong Lin. 2022. Document images forgery localization using a two-stream network. International Journal of Intelligent Systems, 37(8):5272–5289.

An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou. 2023. Chinese clip: Contrastive vision-language pretraining in chinese. Preprint, arXiv:2211.01335.

Yuxuan Yao, Han Wu, Zhijiang Guo, Biyan Zhou, Jiuhui Gao, Sichun Luo, Hanxu Hou, Xiaojin Fu, and Linqi Song. 2024. Learning from correctness without prompting makes llm efficient reasoner. Preprint, arXiv:2403.19094.

Zeqin Yu, Jiangqun Ni, Yuzhen Lin, Haoyi Deng, and Bin Li. 2024. Diffforensics: Leveraging diffusion prior to image forgery detection and localization. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12765–12774.

Weichao Zeng, Yan Shu, Zhenhang Li, Dongbao Yang, and Yu Zhou. 2024. Textctrl: Diffusion-based scene text editing with prior guidance control. Preprint, arXiv:2410.10133.

Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. 2024. Self-contrast: Better reflection through inconsistent solving perspectives. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3602–3622. Association for Computational Linguistics.

Lin Zhao, Changsheng Chen, and Jiwu Huang. 2021. Deep learning-based forgery attack on document images. IEEE Transactions on Image Processing, 30:7964–7979.

Xuekang Zhu, Xiaochen Ma, Lei Su, Zhuohang Jiang, Bo Du, Xiwen Wang, Zeyu Lei, Wentao Feng, Chi-Man Pun, and Jizhe Zhou. 2024. Mesoscopic insights: Orchestrating multi-scale & hybrid architecture for image manipulation localization. Preprint, arXiv:2412.13753.
A Distribution of Multi-Feature Fusion Recall Scores.

We constructed a dedicated validation set to optimize recall parameters. Using a grid search approach, we explored the optimal weight coefficients assigned to three features: image features, dense textual features, and sparse textual features. The comprehensive findings from this experiment are illustrated in Figure ??.

Observations reveals that the optimal weight combination was determined to be [0.5, 0.1, 0.4], corresponding to weight coefficients of \( \alpha = 0.5, \beta = 0.1, \gamma = 0.4 \). This result highlights that sparse textual features play a crucial role in improving recall accuracy, with importance comparable to image features. This aligns with our analysis: in scenarios with a high density of similar certificate images, keywords or key terms within certificates are critical for differentiation.

B Regarding Fine-grained Metric Calculation

Necessity for fine-grained metrics. A set of financial documents to be analyzed may contain multiple anomalies (e.g., in the set of ID cards, there could be three independent anomalies: *same ID number but different names / different birth dates / different addresses*). Therefore, we have conducted fine-grained individual annotations for each anomaly point in every anomalous case within the benchmark, rather than simply labeling as *anomalous*.

Calculation of fine-grained metrics. For a given anomaly case, the ground truth annotation is \( L = \{l_1, l_2, \ldots, l_k\} \), where \( l_i \) represents the \( i \)-th anomaly point. Let the CSIA D’s predicted anomalies be \( P = \{p_1, p_2, \ldots, p_m\} \).

To determine semantic equivalence between predicted and ground-truth anomalies, we define a binary matching function:

• Match(\( p, l \)) = 1 if the predicted anomaly \( p \) and the ground-truth anomaly \( l \) are semantically equivalent;

• Match(\( p, l \)) = 0 otherwise.

For each predicted anomaly \( p_j \) in the prediction set \( P \), we compare it against all ground-truth anomalies \( l_i \) in the annotation set \( L \).

• If there exists at least one ground-truth anomaly \( l_i \) such that Match(\( p_j, l_i \)) = 1, then \( p_j \) is considered a valid prediction, meaning that the LLM has successfully identified an anomaly that is semantically equivalent to a labeled ground truth.

• If no such \( l_i \) is found, then \( p_j \) is counted as a false positive.

To quantitatively evaluate fine-grained semantic correctness, we define the following metrics.

\[
\text{Pre}^t = \frac{\sum_{p_j \in P} \max_{l_i \in L} \text{Match}(p_j, l_i)}{|P|}
\] (8)

\[
\text{Rec}^t = \frac{\sum_{l_i \in L} \max_{p_j \in P} \text{Match}(p_j, l_i)}{|L|}
\] (9)

To make the computation process more intuitive, we illustrate the metrics with a concrete example.

Suppose the ground truth annotations are \( L = \{l_1, l_2, l_3\} \) and the model predictions are \( P = \{p_1, p_2\} \), where the semantic equivalence matrix is given by:

<table>
  <tr>
    <th></th>
    <th>l_1</th>
    <th>l_2</th>
    <th>l_3</th>
  </tr>
  <tr>
    <td>p_1</td>
    <td>1</td>
    <td>1</td>
    <td>0</td>
  </tr>
  <tr>
    <td>p_2</td>
    <td>0</td>
    <td>0</td>
    <td>1</td>
  </tr>
</table>

Then, for this sample case, we compute:

\[
\text{Pre}^t = \frac{\max(1, 0, 0) + \max(0, 1, 0)}{2} = \frac{1 + 1}{2} = 1,
\]

\[
\text{Rec}^t = \frac{\max(1, 0) + \max(1, 0) + \max(0, 1)}{3} = \frac{1 + 1 + 1}{3} \approx 0.67
\] (10)

C Error Rate Analysis

We evaluated 109 anomalous cases within the benchmark, of which 21 were misclassified. A detailed quantitative analysis of the error rates at each stage of the CSIA D pipeline was performed for these misclassified cases, as presented in the Table 7.

The failure patterns associated with each step are as follows:

• Retrieval Ensemble: Failed to retrieve associated images.

• Structured Text Extraction: Incorrect text information extraction.
Table 7: Error statistics at each stage of the CSIAD.

<table>
  <tr>
    <th>Step</th>
    <th>Error Statistics</th>
    <th>Error Distribution (%)</th>
  </tr>
  <tr>
    <td>1. Retrieval Ensemble</td>
    <td>3</td>
    <td>14.29</td>
  </tr>
  <tr>
    <td>2. Structured Text Extraction</td>
    <td>11</td>
    <td>52.38</td>
  </tr>
  <tr>
    <td>3. Sample-Specific Rule Generation</td>
    <td>4</td>
    <td>19.05</td>
  </tr>
  <tr>
    <td>4. Fact-Driven Rule Verification</td>
    <td>0</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>5. Rule-Based Anomaly Detection</td>
    <td>2</td>
    <td>9.52</td>
  </tr>
  <tr>
    <td>6. Fact-Driven Anomaly Verification</td>
    <td>1</td>
    <td>4.76</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>21</th>
    <th>100.00</th>
  </tr>
</table>

• Sample-Specific Rule Generation: Failed to generate actionable rules.

• Fact-Driven Rule Verification: Misclassification of valid rules as erroneous.

• Rule-Based Anomaly Detection: Logical reasoning errors.

• Fact-Driven Anomaly Verification: Misjudgment of correctly analyzed anomalies.

We found that the most error-prone stage is Structured Text Extraction. Multimodal LLMs may fail to accurately extract textual information due to complex image content (e.g., blurring, deformation, etc.). Incomplete or incorrectly structured text can directly lead to deviations in subsequent steps.

D Our Prompt

D.1 Prompt for Rule Generation

Your task is to analyze the provided structured text of images and identify the rules that need to be followed. Please complete the task according to the steps below.

Step 1. Sample Type Identification
Analyze the sample content to identify:
1. Category (e.g., invoice, order, tax certificate, etc.);
2. All fields included in the sample;
3. Key identifiers involved in the fields (Key identifiers refer to fields that uniquely identify an object or sample, such as ID numbers, contract numbers, or user IDs).

Step 2. Rule Generation
Based on the <sample type>, <all fields >, and provided <rule examples> obtained in Step 1, explore as many relevant rules as possible.
Requirements:
1. Rules must be directly related to the sample category. For example, invoice rules should relate to invoice content, and contract rules should relate to contract content.
2. Each rule should use strong constraint language (e.g., "must," "should be identical"), avoiding ambiguous terms like "can" or "usually."
3. Each rule should focus on "information consistency" or "logical conflict resolution."

Output Format:
<Rule_Generation>
1.Rule xxx
2.Rule yyy
</Rule_Generation>

---
The provided rule examples are as follows:
[rule_list]

The structured text information of the certificate is as follows:
[text_KVs]]

D.2 Prompt for Fact-Driven Rule Verification

You are required to analyze the given structured text step by step to verify whether the provided rule is violated within the structured data. The analysis process must be fully documented before drawing a conclusion.

Step 1: Decompose the rule and extract the relevant fields.

Step 2: Focus on the structured text and extract the fields identified in Step 1.

Step 3: Analyze whether the structured text extracted in Step 2 aligns with the described rule.
- If the data conforms to the rule, the rule should be retained.
- Otherwise, the rule should be discarded.

Step 4: Output based on the analysis process.
Use the following format:
<Rule_Verify>
Retain/Discard
</Rule_Verify>
---
The reference structured text is as follows:
[[fact_case]]

The rule to be verified is as follows:
[[rule]]

D.3 Prompt for Anomaly Conclusion Generation

You are required to perform cross-sample anomaly analysis on multiple transaction samples and identify potential anomalies. Please strictly follow the requirements below to complete the task.

Step 1: Analyze the fields mentioned in the provided rule.

Step 2: Analyze the structured text of the sample group and extract the content related to the fields mentioned in Step 1.

Step 3: Verify whether there are any sub-sample groups violate the given rule. If violations are found, indicate the affected record IDs and analyze the cause of the anomaly.

Step 4: Output the result in the following format:
- For anomalies: If anomalies exist, output in the following format:
<Anomaly>
Violation of rule<Rule Description>,
Affected records: [[<Record ID1>, <Record ID2>, ...],
Explanation: <Anomaly Explanation>
</Anomaly>

- For no anomalies: If no anomalies are found, output the following:
<Anomaly>
No anomaly
</Anomaly>

---
The rules to be verified are as follows:
[[rule_list]]

The structured text information of the sample group to be analyzed is as follows:
[[text_KVs]]

D.4 Prompt for Fact-Driven Anomaly Verification

Judge if the anomaly conclusion is consistent with the structured text of the certificate image:

Step1: Extract from Explanation
Extract the <fields> and <field content> mentioned in the explanation from the Anomaly Conclusion;

Step2: Extract from RawText
Extract the field content related to the mentioned fields in Step1 from the Structured Texts.

Step3: Combine the Field Content,
Consolidate the content extracted in Step 1 and Step 2 for the same fields within the same record ID into a unified list.

---
The given anomaly conclusion:
[[anomaly_conclusion]]
Structured text information of the image group: [[text_KVs]]

After LLM integration, use a regular expression to check whether the field content mentioned in the anomaly conclusion is consistent with the original text.

Judge if the anomaly conclusion aligns with the rule description:

Your task is to proofread the cross-sample anomaly conclusions for multiple transaction samples and verify the authenticity of potential anomalies. Please analyze step by step.

Step 1: Anomaly Confirmation
Analyze the following content:
- Mentioned anomaly: Violation of the rule <Rule Description>
- Affected records: <Record ID1>, <Record ID2>
- Conclusion 1: Does it satisfy the rule 's premise: Yes/No
- Conclusion 2: Does it violate the rule: Yes/No
- Decision result: Anomaly confirmed/Anomaly not confirmed
// The anomaly is considered confirmed if both conclusions are "Yes." If at least one is "No," the anomaly is considered not confirmed.

Step 2: Anomaly Conclusion Revision
Based on the analysis results in Step 1, revise the anomaly conclusion and output the revised conclusion in the following format:
<Conclusion_Verify>
Anomaly confirmed/not confirmed
</Conclusion_Verify>

---
The given anomaly conclusion:
[[anomaly_conclusion]]
E  Case Study

E.1  Complete Workflow of Cross-Sample Analysis.

Question: Is there any anomaly present in this financial document image?

(a) Retrieval Ensemble

![Retrieval Ensemble showing query image and recalled images](page_246_312_1057_384.png)

(i) Structured Texts

<table>
  <tr>
    <th>Type: Payment Details</th>
    <th>Text: KV:</th>
    <th>1. 公司名称: 国网德州供电公司，金额: -300.00，状态: 已到账，支付成功时间: 2024-08-13 10:03:27，账号: 8123531476000，户名: 张华，地址: 内蒙古自治区通辽市——1.</th>
    <th>2. 公司名称: 国网德州供电公司，金额: -300.00，状态: 已到账，支付成功时间: 2024-08-13 10:03:27，账号: 8123531476000，户名: 张华，地址: 广东省梅州市——1.</th>
    <th>3. 公司名称: 国网德州供电公司，金额: -300.00，状态: 已到账，支付成功时间: 2024-08-13 10:03:27，账号: 8123531476000，户名: 张华，地址: 山东省济宁市——1.</th>
    <th>4. 公司名称: 国网四川省电力公司(南川)，金额: -100.00，状态: 已到账，支付成功时间: 2024-10-04 08:45:47，账号: 8429574631023，户名: 韦宝，地址: 四川省南充市——3.</th>
  </tr>
</table>

(b) Sample-Specific Rule Generation

Rule Generated:
1. Records sharing the same account number must have identical addresses.
2. The account holder name must be consistent across records with the same account number.
3. The transaction amount must be identical for records with the same account number.
4. Records associated with the same account number must maintain uniform status.
5. Payment success times must be identical for all records linked to the same account number.
6. While company names may vary, payment success times and processing times must be perfectly aligned for records with the same account number.
7. Processing times must remain consistent among records associated with the same account number.
8. The company name must be uniform across all records sharing the same account number.

(c) Fact-Driven Rule Verification

Verifying Rule 1. Records sharing the same account number must have identical addresses.
Cur_type: Payment Details

Verification Process:
Step 1: Decompose the rule and extract the fields mentioned in the rule:
Account Number, Address
Step 2: Focus on the structured text extracted from the Fact Repository and extract the fields mentioned in Step 1:

records:
<record_id: 2>
  payment_account_number: 37952846
  address_information: No. 456, Meili Road, Huaiyin District, Jinan, Shandong
</record_id>
<record_id: 2>
  payment_account_number: 37952846
  address_information: No. 456, Meili Road, Huaiyin District, Jinan, Shandong
</record_id>
<record_id: 3>
  payment_account_number: 672703124
  address_information: No. 88, Quancheng Road, Lixia District, Jinan, Shandong
</record_id>
<record_id: 3>
  payment_account_number: 672703124
  address_information: No. 88, Quancheng Road, Lixia District, Jinan, Shandong
</record_id>

Step 3: Analyze whether the extracted structured text matches the content described in the rule:
Analysis:
- Record ID 1 and Record ID 2 have the same payment account number <37952846> and the same address <No.456, Meili Road, Huaiyin District, Jinan, Shandong>.
- Record ID 3 and Record ID 4 have the same payment account number <672703124> and the same address <No.88, Quancheng Road, Lixia District, Jinan, Shandong>.
After analysis, all records with the same payment account number also have the same address information. The field relationships in the structured text match the rule as described. Therefore, the rule is valid and should be retained.

Step 4: Based on the analysis, the output is as follows:
Conclusion Summary: Retain

Figure 5: Complete Workflow of Cross-Sample Analysis (1/3).
Verifying Rule: 2. The account holder name must be consistent across records with the same account number
Cur_type: Payment Details

Verification Process:
Step 1: Decompose the rule and extract the fields mentioned in the rule:
Account Number, Account Holder Name

Step 2: Focus on the structured text extracted from the Fact Repository and

records:
- record_id: 1
  account_holder_name: Jinan Huayin East and Hardware Store
  payment_account_number: 37592846
- record_id: 2
  account_holder_name: Jinan Huayin East and Hardware Store
  payment_account_number: 37592846
- record_id: 3
  account_holder_name: Jinan Huayin Youxi Dry Cleaning Store
  payment_account_number: 672703124
- record_id: 4
  account_holder_name: Jinan Huayin Youxi Dry Cleaning Store
  payment_account_number: 672703124

Step 3: Analyze whether the extracted structured text matches the content described in the rule:
Analysis:
• Record ID 1 and Record ID 2 share the same payment account number (37592846), and the account holder name is Jinan Huayin East and Hardware Store for both.
• Record ID 3 and Record ID 4 share the same payment account number (672703124), and the account holder name is Jinan Huayin Youxi Dry Cleaning Store for both.
After analysis, all records with the same account number also have consistent account holder names. The relationships between fields in the structured text match the rule as described. Therefore, the rule is valid and should be retained.

Step 4: Based on the analysis, the output is as follows:
Conclusion Summary: Retain

Verifying Rule: 3. The transaction amount must be identical for records with the same account number
Cur_type: Payment Details

Verification Process:
Step 1: Decompose the rule and extract the fields mentioned in the rule:
Account Number, Amount

Step 2: Focus on the structured text extracted from the Fact Repository and

records:
- record_id: 1
  payment_account_number: 37592846
  amount: 18.80
- record_id: 2
  payment_account_number: 37592846
  amount: 18.80
- record_id: 3
  payment_account_number: 672703124
  amount: 23.80
- record_id: 4
  payment_account_number: 672703124
  amount: 23.80

Step 3: Analyze whether the extracted structured text matches the content described in the rule:
Analysis:
• For Payment Account Number 37592846, both Record ID 1 and Record ID 2 have the same amount of 18.00.
• For Payment Account Number 672703124, Record ID 3 has an amount of 20.00, while Record ID 4 has an amount of 23.00.
After analysis, the samples with Payment Account Number <37592846> have consistent amounts, but the samples with Payment Account Number <672703124> have inconsistent amounts. Therefore, the relationships between the fields in the structured text do not align with the rule.

Step 4: Based on the analysis, the output is as follows:
Conclusion Summary: Discard

Rules 4-8 follow the same rule verification steps. After verification by the fact repository, the rules that are consistent with the patterns observed in normal samples. The following are all the rules that have been retained:

Rule Verified:
1. Records sharing the same account number must have identical addresses.
2. The account holder name must be consistent across records with the same account number.
4. Records associated with the same account number must maintain uniform status.
8. The company name must be uniform across all records sharing the same account number.

(d) Rule-Based Anomaly Detection

Analysing Rule: 1. Records sharing the same account number must have identical addresses

Analysis Process:
Step 1: Rule Analysis
The rule states that if multiple electricity bill records share the same account number, the addresses for these samples must be identical. The relevant fields are <Account Number> and <Address>.

Step 2: Analysis of given Structured Texts
The content relevant to the above fields is extracted as follows:

records:
- record_id: 1
  account_number: 8123531476000
  address: "No. 88, Jiangou Road, Chaoyang District, Inner Mongolia"
- record_id: 2
  account_number: 8123531476000
  address: "No. 88, Shiji Avenue, Meizhou City, Guangdong Province"
- record_id: 3
  account_number: 8123531476000
  address: "Building T3, Southern Section, Science and Technology Park, Nanhan District"
- record_id: 4
  account_number: 8123531476000
  address: "No. 4, Floor 323, Block A, KejiHuaYuan, No. 8 Yuejin Road, Fucheng District"

Step 3: Verification of Rule Violation
Records 1, 2 and 3 share the same account number, <8123531476000>, but the addresses differ.

Step 4: Analysis Result Output
• Conclusion: Violation of rule "Records sharing the same account number must have identical addresses".
• Affected records: [1, 2, 3]
• Explanation: All account numbers are <8123531476000>, but the addresses are <No.88 Jiangou Road...>, <No.100 Shiji Avenue...>, <Building T3, Southern Section...>, respectively.
Conclusion Summary: Anomaly

(e) Fact-Driven Anomaly Verification

Verifying Anomaly: Violation of the rule <if multiple electricity bill records share the same account number, the addresses for these samples must be identical>. Affected records: [1, 2, 3]. Explanation: All account numbers are <8123531476000>, but the addresses are <No.88 Jiangou Road...>, <No.100 Shiji Avenue...>, <Building T3, Southern Section...>, respectively.

Verification Process:
Step 1: Confirm whether the fields are faithful to the original text
Fields by explanation:

records:
- record_id: 1
  account_number: 8123531476000
  address: "No. 88, Jiangou Road, Chaoyang District, Inner Mongolia"
- record_id: 2
  account_number: 8123531476000
  address: "No. 88, Shiji Avenue, Meizhou City, Guangdong Province"
- record_id: 3
  account_number: 8123531476000
  address: "Building T3, Southern Section, Science and Technology Park, Nanhan District"
Fields by raw text:

records:
- record_id: 1
  account_number: 8123531476000
  address: "No. 88, Jiangsu Road, ChaoYang District, Inner Mongolia"
- record_id: 2
  account_number: 8123531476000
  address: "No. 108, Shiji Avenue, Meizhou City, Guangdong Province"
- record_id: 3
  account_number: 8123531476000
  address: "Building T3, Southern Section, Science and Technology Park, Nanshan District"

Fields combined:

fields:
- record_id: 1
  account_number: "8123531476000"
  address: "No. 88, Jiangsu Road, ChaoYang District, Inner Mongolia"
- record_id: 2
  account_number: "8123531476000"
  address: "No. 108, Shiji Avenue, Meizhou City, Guangdong Province"
- record_id: 3
  account_number: "8123531476000"
  address: "Building T3, Southern Section, Science and Technology Park, Nanshan District"

Field Confirmation Conclusion: The fields match the original text

Step 2: Confirm whether the mentioned anomaly matches the rule

• Conclusion 1: Does the rule’s premise hold?
The premise of the rule is “if multiple electricity bill records share the same account number”, and the “account number” field in all affected records is <8123531476000>, which is consistent across all records. Therefore, the premise is met, and Conclusion 1 is “Yes”.

• Conclusion 2: Does the rule’s condition hold?
The conclusion states that “the addresses must be identical”, but the addresses for the affected records are clearly different, so they do not satisfy the rule’s conclusion, resulting in a violation. Conclusion 2 is “Yes”.

Anomaly Confirmation Conclusion: The mentioned anomaly matches the rule

Decision: Anomaly confirmed

The anomaly analysis processes for Rules 2, 4, and 8 are the same as the anomaly analysis process for Rule 1 mentioned above, and are therefore omitted here. The final conclusion is that the financial document images to be inspected exhibit multiple anomalies, as detailed below:

For Rule 2: Violation of rule “The account holder name must be consistent across records with the same account number”.
• Affected records: [1,2,3]
• Explanation: All account numbers are <8123531476000>, but the account holder names are <“Zhang Wei”>, <“Wang Fang”>, <“Li”>.

For Rule 4: No violations were found for the rule: “Records associated with the same account number must maintain a consistent status”.
• Affected records: None
• Explanation: None

For Rule 8: Violation of rule “The company name must be uniform across all records sharing the same account number”.
• Affected records: [1,2,3]
• Explanation: All account numbers are <8123531476000>, but the company names are <“State Grid Chifeng Power Supply Company”>, <“State Grid Meishou Power Supply Company”>, <“State Grid Liangshan Power Supply Company”>.
E.2 Comparison Case

Verifying Rule: (Should be retained) If the Taobao store operation certificates share the same store address, their business license numbers must be identical.

Cur_type: Taobao Store Operation Certificate

Verification Process:
Step 1: Decompose the rule and extract the fields mentioned in the rule:
Business License Number, Store Address
Step 2: Focus on the structured text extracted from the Fact Repository and extract the fields mentioned in Step 1:

records:
- record_id: 1
  business_license_number: 91310108MA3Z9R1XTX
  store_address: https://shop34568572.taobao.com
- record_id: 2
  business_license_number: 91310108MA3Z9R1XTX
  store_address: https://shop34568572.taobao.com
- record_id: 3
  business_license_number: 91310108MA3Z9R1XTX
  store_address: https://shop34568572.taobao.com
- record_id: 4
  business_license_number: 91310108MAA8B2S8Y
  store_address: https://shop133457349.taobao.com
- record_id: 5
  business_license_number: 91310108MA3Z9R1XTX
  store_address: https://shop34568572.taobao.com

Step 3: Analyze whether the extracted structured text matches the content described in the rule:
Analysis:
• Record ID 1 and Record ID 3 have the same store address:
https://shop34568572.taobao.com;
• Record ID 1 and Record ID 3 share the same business license number: <91310108MA3Z9R1XTX>;
After analysis, samples with the same store address also have identical business license numbers. The relationships between the structured text fields are consistent with the rule description. Therefore, the rule is valid and retained.

Step 4: Based on the analysis, the output is as follows:
Conclusion Summary: Retain (√)

Verifying Rule: (Should be discard) If multiple tax certificates share the same taxpayer identification number, the actual tax paid in these samples must be identical.

Fact-Driven Rule Verification

Verifying Rule: If the Taobao store operation certificates share the same store address, their business license numbers must be identical.
Cur_type: Taobao Store Operation Certificate

Verification Process:
Step 1: Decompose the rule and extract the fields mentioned in the rule:
Taxpayer Identification Number, Actual Tax Paid
Step 2: Focus on the structured text extracted from the Fact Repository and extract the fields mentioned in Step 1:

records:
- record_id: 1
  taxpayer_identification_number: 91330104MAA8B2S8Y
  actual_tax_paid: 106220.71
- record_id: 2
  taxpayer_identification_number: 91330104MAA8B2S8Y
  actual_tax_paid: 9240.00
- record_id: 3
  taxpayer_identification_number: 91330104MAA8B2S8Y
  actual_tax_paid: 6340.00
- record_id: 4
  taxpayer_identification_number: 91330104MAA8B2S8Y
  actual_tax_paid: 106220.71
- record_id: 5
  taxpayer_identification_number: 91440108MA27D9Q4M
  actual_tax_paid: 2750.00

Step 3: Analyze whether the extracted structured text matches the content described in the rule:
Analysis:
• Record ID 1 and Record ID 4 share the same taxpayer identification number <91330104MAA8B2S8Y> and have identical actual tax paid <106220.71>, which complies with the rule.
• Record ID 2 and Record ID 3 share the same taxpayer identification number <91330104MAA8B2S8Y> but have different actual tax paid values <9240.00> and <6340.00>, which violates the rule.
After analysis, samples with the same taxpayer identification number have discrepancies in the actual tax paid. Therefore, the rule is invalid and is discarded.

Step 4: Based on the analysis, the output is as follows:
Conclusion Summary: Discard (√)

LLM Self-Reflection-Based Rule Verification

Verifying Rule: If the Taobao store operation certificates share the same store address, their business license numbers must be identical.
Cur_type: Taobao Store Operation Certificate

Verification Process:
Step1: Verification of necessity.
• A shared store address does not necessarily mean that all stores must operate under the same business license.
For example, a single physical location may host multiple distinct businesses, each with its own business license.
• Therefore, enforcing the requirement that stores with the same address must have identical business license numbers is unnecessary. Conclusion on necessity: “No”.

Step2: Verification of Reasonableness.
• This contradicts the given common-sense principle that “If multiple Taobao store operation certificates share the same store address, their business license numbers should be identical”.
• Conclusion on reasonableness: “No”.

Step3: Based on the analysis, the output is as follows:
Conclusion Summary: Discard (√)

LLM Self-Reflection-Based Rule Verification

Verifying Rule: If the Taobao store operation certificates share the same store address, their business license numbers must be identical.
Cur_type: Taobao Store Operation Certificate

Verification Process:
Step1: Verification of necessity.
• The Taxpayer Identification Number is a key identifier in tax certificates, uniquely determining the taxpayer entity. If this information is the same, then the tax certificates should correspond to the same transaction or the same tax period.
• Therefore, requiring other information, such as tax authority, tax type, tax period, payment date, actual tax paid, and certification date, to be completely consistent helps ensure data accuracy and consistency, preventing potential errors and errors.
• Conclusion on necessity: “Yes”.

Step2: Verification of Reasonableness.
• Tax-related details (such as tax type, tax period, payment date, and actual tax paid) hold high uniqueness and certainty in tax administration. Therefore, requiring consistency in this information across multiple tax certificates is reasonable.
• Conclusion on reasonableness: “Yes”.

Step3: Based on the analysis, the output is as follows:
Conclusion Summary: Retain (√)

Figure 8: A comparison case for Fact-Based Rule Verification and LLM Self-Reflection-Based Rule Verification. The Left column illustrates the verification process of a rule that should be retained, while the Right column depicts the process for a rule that should be discarded. As shown, the conclusions drawn from Fact-Based Rule Verification are accurate in both cases, whereas the conclusions produced by the LLM Self-Reflection-Based approach are incorrect.
F  Dataset Composition

Table 8: The distribution of anomalous certificate images in the evaluation data. The Type includes the financial document image type and the description of the anomaly type. Group Num. refers to the number of financial document image groups of this type, while Image Num. refers to the total number of financial document images contained within these groups.

<table>
  <tr>
    <th>Type</th>
    <th>Group Num.</th>
    <th>Image Num.</th>
  </tr>
  <tr><td>Tax Payment Certificate_same certificate number_different internal information_2</td><td>8</td><td>26</td></tr>
  <tr><td>Payment Details_same user ID_different other user information_1</td><td>7</td><td>21</td></tr>
  <tr><td>Tax Payment Certificate_same certificate number_different internal information_1</td><td>7</td><td>22</td></tr>
  <tr><td>Tax Payment Certificate_same certificate number_different internal information_3</td><td>6</td><td>28</td></tr>
  <tr><td>Mini Program Info_same Mini Program ID_different Mini Program name_1</td><td>6</td><td>20</td></tr>
  <tr><td>No Tax Arrears Certificate_same certificate number_different internal information_1</td><td>6</td><td>19</td></tr>
  <tr><td>Handheld Commitment Letter_same identity information_multiple registered email accounts_1</td><td>5</td><td>23</td></tr>
  <tr><td>Taobao Store Operation Certificate_same store address_different internal information_1</td><td>5</td><td>28</td></tr>
  <tr><td>No Tax Arrears Certificate_same certificate number_different internal information_2</td><td>3</td><td>9</td></tr>
  <tr><td>Payment Details_same user ID_different other user information_3</td><td>3</td><td>14</td></tr>
  <tr><td>Life Payment Voucher_same household number_different address information_1</td><td>3</td><td>11</td></tr>
  <tr><td>Tax Payment Certificate_same certificate number_different internal information_4</td><td>2</td><td>11</td></tr>
  <tr><td>Tax Certificate_same certificate number_different internal information_4</td><td>2</td><td>12</td></tr>
  <tr><td>Tax Payment Certificate_same taxpayer identification number_different taxpayer name_1</td><td>2</td><td>8</td></tr>
  <tr><td>Tax Payment Certificate_same verification code_different internal information_1</td><td>2</td><td>4</td></tr>
  <tr><td>No Tax Arrears Certificate_same certificate number_different internal information_3</td><td>2</td><td>7</td></tr>
  <tr><td>Sales Contract_same contract number_different internal information_1</td><td>2</td><td>6</td></tr>
  <tr><td>Permanent Resident Registration Card_same ID number_different name_1</td><td>2</td><td>4</td></tr>
  <tr><td>Tax Certificate_same certificate number_different internal information_2</td><td>1</td><td>3</td></tr>
  <tr><td>VAT Electronic General Invoice_same invoice number_different internal information_1</td><td>1</td><td>5</td></tr>
  <tr><td>Authorization Letter_same authorization dealer contract number_different authorized person_1</td><td>1</td><td>5</td></tr>
  <tr><td>Marriage Certificate_same marriage certificate number_different internal information_1</td><td>1</td><td>3</td></tr>
  <tr><td>House Lease Contract_same lessee (Party B) ID number_different name_1</td><td>1</td><td>2</td></tr>
  <tr><td>Payment Details_same user ID_different other user information_2</td><td>1</td><td>6</td></tr>
  <tr><td>House Lease Contract_same lessor (Party A) ID number_different name_1</td><td>1</td><td>2</td></tr>
  <tr><td>Sales Contract_same contract number_different internal information_2</td><td>1</td><td>4</td></tr>
  <tr><td>Commercial Office Lease Contract_same lessee (Party A) ID number_different name_1</td><td>1</td><td>4</td></tr>
  <tr><td>ID Card Front_same ID number_different name_1</td><td>1</td><td>2</td></tr>
  <tr><td>Payment Details_same user ID_different other user information_4</td><td>1</td><td>6</td></tr>
  <tr><td>Sales Contract_same contract number_different agent_1</td><td>1</td><td>2</td></tr>
  <tr><td>Business License_same unified social credit code_different business owner_1</td><td>1</td><td>2</td></tr>
  <tr><td>Purchase and Sales Contract_same order number_different internal information_1</td><td>1</td><td>2</td></tr>
  <tr><td>Tax Certificate_same certificate number_different internal information_3</td><td>1</td><td>3</td></tr>
  <tr><td>JD Order Details_same order number_different internal information_1</td><td>1</td><td>2</td></tr>
  <tr><td>Tax Certificate_same certificate number_different internal information_1</td><td>1</td><td>2</td></tr>
  <tr><td>Tax Payment Certificate_same certificate number_different taxpayer name_1</td><td>1</td><td>2</td></tr>
  <tr><td>Promissory Note_same lender's ID number_different name_1</td><td>1</td><td>4</td></tr>
  <tr><td>Promissory Note_same lender's ID number_different name_2</td><td>1</td><td>6</td></tr>
  <tr><td>Promissory Note_same lender's ID number_different name_3</td><td>1</td><td>2</td></tr>
  <tr><td>Outbound Order_same order number_different internal information_1</td><td>1</td><td>2</td></tr>
  <tr><td>Proof of Employment_same employee ID number_different name_1</td><td>1</td><td>4</td></tr>
  <tr><td>VAT Electronic General Invoice_same invoice number_different internal information_1</td><td>1</td><td>2</td></tr>
  <tr><td>Authorization Letter_same authorization dealer contract number_different authorized person_1</td><td>1</td><td>3</td></tr>
  <tr><td>Income Tax Details_same taxpayer identification number_different taxpayer name_1</td><td>1</td><td>6</td></tr>
  <tr><td>No Tax Arrears Certificate_same taxpayer identification number_different issuing unit_1</td><td>1</td><td>3</td></tr>
  <tr><td>Taobao Store Operation Certificate_same store address_different internal information_2</td><td>1</td><td>6</td></tr>
  <tr><td>Taobao Store Operation Certificate_same store address_different internal information_3</td><td>1</td><td>3</td></tr>
  <tr><td>Tax Payment Certificate_same taxpayer identification number_different taxpayer name_2</td><td>1</td><td>2</td></tr>
  <tr><td>Promissory Note_same borrower ID number_different name_1</td><td>1</td><td>2</td></tr>
  <tr><td>Tax Payment Certificate_same certificate number_different internal information_5</td><td>1</td><td>6</td></tr>
  <tr><td>Tax Payment Certificate_same certificate number_different internal information_6</td><td>1</td><td>4</td></tr>
  <tr><td>Tax Payment Certificate_same certificate number_different internal information_7</td><td>1</td><td>3</td></tr>
  <tr><td>Tax Payment Certificate_same certificate number_different issuance date_1</td><td>1</td><td>2</td></tr>
  <tr><td>Tax Certificate_same certificate number_different internal information_4</td><td>1</td><td>6</td></tr>
  <tr><th>Total</th><th>109</th><th>396</th></tr>
</table>
Table 9: The distribution of non-anomalous financial document images in the evaluation data. The Type includes only the description of the financial document image type. Group Num. refers to the number of financial document image groups of this type, while Image Num. refers to the total number of financial document images within these groups. The Group Num. and Image Num. are the same because non-anomalous images do not belong to associated image groups, and each non-anomalous image is treated as an individual.

<table>
  <tr>
    <th>Type</th>
    <th>Group Num.</th>
    <th>Image Num.</th>
  </tr>
  <tr>
    <td>National Emblem Side of ID Card</td>
    <td>14</td>
    <td>14</td>
  </tr>
  <tr>
    <td>Account Details</td>
    <td>9</td>
    <td>9</td>
  </tr>
  <tr>
    <td>Business License</td>
    <td>7</td>
    <td>7</td>
  </tr>
  <tr>
    <td>Bill Details</td>
    <td>7</td>
    <td>7</td>
  </tr>
  <tr>
    <td>ID Card Front Side</td>
    <td>6</td>
    <td>6</td>
  </tr>
  <tr>
    <td>Income and Expenditure Record</td>
    <td>6</td>
    <td>6</td>
  </tr>
  <tr>
    <td>Taobao Store Operation Certificate</td>
    <td>5</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Detail Inquiry</td>
    <td>5</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Transaction Details</td>
    <td>4</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Payment Details</td>
    <td>4</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Bank Card</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>No Tax Arrears Certificate</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Passport</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Courier Receipt</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Foreign Card Details</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Income and Expenditure Details</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Income and Expenditure Information</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Permanent Resident Registration Card</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Tax Payment Certificate</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Order Details</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Transfer Record</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Account Details</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Repayment Details</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Pass</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Wallet Records</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Transaction Details</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Contact Person</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Logistics Details</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Repayment Inquiry Details</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Inquiry Details</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Transaction Flow</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Record Inquiry</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Medical Form</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Sale Listing</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Promissory Note</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Balance Details</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Detail Information</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>109</th>
    <th>109</th>
  </tr>
</table>
G  Human Annotation Cases

a) Annotation Process Case 1

LLM Pre-Annotation

Conclusion
Anomaly Description: Violation of the rule <Records share the same transaction ID, their other information must be identical>
Involved Records: [1, 3, 4]
Explanation: These records share the same transaction ID <489000456798307> but differ in other fields (payment account, address information).

Involved Certificate Images and Corresponding IDs
[1, 2, 3, 4]

Human Annotation

Conclusion
1. True Positive (TP): Actual-Anomaly, Predicted-Anomaly (Correct)
2. False Positive (FP-A)*: Actual-Anomaly, Predicted-No Anomaly (Incorrect)
3. False Negative (FN)*: Actual-Anomaly, Predicted-No Anomaly
4. False Positive (FP-N): Actual-No Anomaly, Predicted-Anomaly
5. True Negative (TN): Actual-No Anomaly, Predicted-No Anomaly

Select Images Containing Anomalous Elements
[1 2 3 4 5 6]

Revised Anomaly Conclusion
Anomaly1: [Description]: Violation of the rule <Same transaction ID = Identical payment accounts>. Explanation: These records share the same transaction ID <489000456798307> but have different payment accounts.
Anomaly2: [Description]: Violation of the rule <Same transaction ID = Identical payment accounts>. Explanation:

b) Annotation Process Case 2

LLM Pre-Annotation

Conclusion
Anomaly Description: Violation of the rule <Records with different proof numbers, other information may or may not be related>
Involved Records: [1, 2]
Explanation: These records have different proof numbers<ShanShui202412345> but share identical display name, taxpayer ID, document types, document numbers, expiration dates, issuing units, and issue dates, violating the rule.

Involved Certificate Images and Corresponding IDs
[1, 2, 3, 4]

Human Annotation

Conclusion
1. True Positive (TP): Actual-Anomaly, Predicted-Anomaly (Correct)
2. False Positive (FP-A)*: Actual-Anomaly, Predicted-No Anomaly (Incorrect)
3. False Negative (FN)*: Actual-Anomaly, Predicted-No Anomaly
4. False Positive (FP-N): Actual-No Anomaly, Predicted-Anomaly
5. True Negative (TN): Actual-No Anomaly, Predicted-No Anomaly

Select Images Containing Anomalous Elements
[1 2 3 4 5 6]

Revised Anomaly Conclusion
Anomaly1: [Description]: Violation of the rule <Same Certificate ID = identical taxpayer ID>. Explanation: These records share the same Certificate ID <ShanShui202412345> but have different Taxpayer IDs.

c) Annotation Process Case 3

LLM Pre-Annotation

Conclusion
Anomaly Description: Violation of the rule <Records share the same business license number, their certified entity, store name, store address, certification date, and issuing unit must be identical>
Involved Records: [1, 5]
Explanation: The records share the same business license number<91370033MA1Y546PSU> but differ in store name and store address, violating the rule.

Involved Certificate Images and Corresponding IDs
[1, 2, 3, 4, 5]

Human Annotation

Conclusion
1. True Positive (TP): Actual-Anomaly, Predicted-Anomaly (Correct)
2. False Positive (FP-A)*: Actual-Anomaly, Predicted-No Anomaly (Incorrect)
3. False Negative (FN)*: Actual-Anomaly, Predicted-No Anomaly
4. False Positive (FP-N): Actual-No Anomaly, Predicted-Anomaly
5. True Negative (TN): Actual-No Anomaly, Predicted-No Anomaly

Select Images Containing Anomalous Elements
[1 2 3 4 5 6]

Revised Anomaly Conclusion
None

Figure 9: Examples of the Human Annotation Process. Given the LLM’s pre-annotated conclusions, human annotators need to determine whether anomalies exist in the actual images and whether the LLM’s pre-annotated conclusions align with the ground truth. They then select one of five predefined labels (TP, FP-A, FN, FP-N, TN) to evaluate the LLM’s pre-annotated results. If the selected label is FP-A or FN (marked with *), it indicates that the LLM failed to identify the correct anomalous elements, and annotators are required to correct or supplement the anomalous elements along with the associated images. Figures (a), (b), and (c) demonstrate examples of LLM pre-annotated results as TP, FP-A, and FP-N, respectively.

Reviewer paper summaries: This paper proposes an anomaly detection method to tackle the problems in the real financial world by leveraging multimodal LLM to unravel logical inconsistencies among samples. The proposed method provides a cross-sample detection methodology to detect anomalies in the image by converting images to structured data and uses LLMs to verify the consistency between the well-structured data and a pre-verified fact repository based on a set of rules generated by LLMs.
Summary of strengths: ● Innovatively using cross-sample inference to find logical inconsistencies among document images, breaking the limits that previous one-sample feature-based detection methods have.

● Providing fine-grained interpretations when detection anomalies, improving the explanability to anomalies and detection processes.

● Many experiments are conducted, comparing multiple baselines and illustrating the effectiveness of each module proposed in detail. Also, a comparison was made with current industrial solutions, improving the possibility that the proposed method will be used in real-world applications.
Summary of weaknesses: ● Writing flaws：e.g. spaces between left bracket and word in line 007. line 103 "CSIDAprovides".

● The evaluation dataset is relatively small, with only 109 normal and anomalous pairs.

● The necessity of using LLM in the modules is not fully illustrated. Take Figure 2(b) as an example: an LLM is used to generate $R_{ini}$, but in real life, the rules such as "Identical account numbers -> Same address", without doubt, are well pre-defined by professional institutes or strictly protected. Why should one use LLM to generate rules and do fact verification? As a regulator or checker,  one can simply query the records in the database to identify the truthfulness of the image. Also, the fact repository $G$ can also be constructed by those institutes and the rules should be static, as mandatory enforcement standards are applied to those transactions scenarios.

● Is the computation efficiency of the framework evaluated? Cross-sample verification seems costly when generating rules and retrieving fact cases
Comments, suggestions and typos: 1. Illustrate the whole process more clearly.

2. Clarify the scenario this submission considers and the real problems this solution faces in reality.
Overall assessment scores: 3.5
Metareview score: 3.5
Metareview: This paper proposes an anomaly detection method to tackle the problems in the real financial world by leveraging multimodal LLMs to unravel logical inconsistencies among samples. The proposed method is evaluated using a newly curated dataset comprising manipulated images across diverse domains. The authors claim improvements over existing techniques in terms of detection accuracy, robustness, and scalability.
Summary of reasons to publish: - The proposed method uses LLMs and multimodal feature retrieval ensembles, which are well-suited for tasks involving text and image analysis.
- The method tries to address the various known problems of LLMs through additional steps. For example, the method verifies the generated set of rules against a set of normal samples before using it.
- The reported results seem promising.
- An ablation study for the components of the framework is included as well, demonstrating the benefits of each module. 
- The benchmark introduced also represents a strong asset of the paper.