CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning

## Abstract

Abstract Recent agentic Multi-Modal Large Language Models (MLLMs) such as GPT-o3 have acheived near-ceiling scores on various existing benchmarks, motivating a demand for more challenging test tasks.
These MLLMs have been reported to excel in a few expert-level tasks for humans, e.g., GeoGuesser, reflecting their potential as a detective
who can notice minuscule cues in an image and weave them into coherent, situational explanations, leading to a reliable answer.
But can they match the performance of excellent human detectives? To answer this question, we investigate some hard scenarios where GPT-o3 can still handle, and find a common scenario where o3’s performance drops to nearly zero, which we name CaughtCheating .
It is inspired by the social media requests that ask others to detect suspicious clues from photos shared by the poster’s partner.
We conduct extensive experiments and analysis to understand why existing MLLMs lack sufficient capability to solve this kind of task. CaughtCheating provides a class of challenging visual perception and reasoning tasks with great value and practical usage. Success in these tasks paves the way for MLLMs to acquire human-level detective perception and reasoning capabilities.
The data and code are available at https://github.com/mingliiii/CaughtCheating .

## 1 Introduction

Figure: Figure 1: An example from CaughtCheating. Query: “My boyfriend said he’s dining alone at the restaurant and sent me this photo. Do you notice anything suspicious in this image that contradicts his claim?” Suspicious Clue: “There are other people, including a girl with long hair, visible in the spoon’s reflection.” In this example, most human participants, and the strong o3 are not capable of identifying this clue when not given any hints, indicating the demand of superior detective-level capabilities.
Refer to caption: extracted/6565317/images/intro.jpeg

Recently advanced Multi-Modal Large Language Models (MLLMs) or corresponding Agents, such as GPT-o3 (OpenAI, 2025) and Gemini-2.5 Pro (DeepMind, 2025b), have demonstrated extraordinary visual perception and reasoning capabilities (Yue et al., 2024b; Zhang et al., 2024a; Wang et al., 2024c; Chen et al., 2024a, b).

Recent studies have demonstrated that MLLMs are even capable of addressing far more demanding challenges, e.g., GeoGuesser, estimating an image’s geographic location (Luo et al., 2025; Huang et al., 2025a).
These kinds of tasks represent scenarios that even humans cannot accomplish easily, which require detective-level capabilities.
These findings raise an important question: Do recent MLLMs truly acquire detective-level perception and reasoning capabilities? If so, what is the boundary of their competence?

Motivated by Human’s Last Exam (Phan et al., 2025), we aim to explore and evaluate the boundary of the detective-level ability (Gu et al., 2023; Yuan et al., 2025; de Lima et al., 2025) of MLLMs on visual perception and reasoning tasks.
We investigate a number of hard scenarios where GPT-o3 can solve the queries even though they are challenging for humans.
Then we discover a common scenario where o3’s performance drops dramatically to almost the random guess level.
This scenario is inspired by the social media requests that ask others to detect potential suspicious clues from photos shared by the poster’s partner, which go against the partner’s claims.
Figure [1](https://arxiv.org/html/2507.00045v1#S1.F1) shows an example, in which the user query is: “My boyfriend said he’s dining alone at the restaurant and sent me this photo. Do you notice anything suspicious in this image that contradicts his claim?”
This image itself seems an ordinary food-sharing image, while in the reflection of the spoon, there are other people, including a girl with long hair can be visible, which is suspicious and violates the claim of being alone.
For this kind of task, we find that most humans, and the strong MLLMs like o3, are not able to identify the clues, indicating the superior detective-level capabilities required.

Thus, to explore the boundary of the visual perception and reasoning capabilities of current MLLMs (Johnson et al., 2017; Zellers et al., 2019; Chen et al., 2024a, b), we collect these images and construct the CaughtCheating benchmark.
This benchmark consists of $100$ images in total(^1^11This kind of data is intrinsically scarce. We have manually screened and verified almost all the existing related posts on public social media to construct this benchmark.), sourced from publicly posted photographs on social media.
The dataset is nearly evenly split into a Clued category and a Unclued category, and primarily features scenes from hotels and dining locations.
Annotations for each image include a primary question about potential violation of the original claims, corresponding deterministic and non-deterministic clues, and a series of decomposed questions to analyze the visual reasoning process of MLLMs.

CaughtCheating is more challenging than the previous tasks because the targets to be identified are not directly defined in the query, and thus can not be solved by an exhaustive grid search.
For example, when o3 tries to solve the query in Figure [1](https://arxiv.org/html/2507.00045v1#S1.F1), it conducts the exhaustive grid search by focusing on one part of the figure at a time.
However, even if it has tried focusing its attention on the area with the spoons, it still can not find this clue(^2^22o3’s visual reasoning traces are presented in Appendix [E](https://arxiv.org/html/2507.00045v1#A5).).
To theoretically analyze the difficulty dependencies between CaughtCheating and existing challenging tasks and understand the reasons behind the failures of o3, we introduce the Guided Search theory from cognitive science (Wolfe et al., 1989; Itti and Koch, 2001; Itti et al., 2002; Duncan and Humphreys, 1989) and the factors that guide attention in visual search.
According to the theory, CaughtCheating has low bottom-up salience, lacks top-down feature guidance, and contains blurry scene structure and meaning.

Extensive evaluation results demonstrate that current MLLMs perform poorly on our detection-level benchmark of CaughtCheating. Notably, even the best-performing model (o3) achieved only $26.0\%$ accuracy in detecting the deterministic clues hidden in the images and $17.2\%$ IoU (the intersection over union).
Moreover, the accuracy of justifying the absence of suspicious clues (Unclued Acc) is only $8.0\%$, resulting in the overall F1 score is only $23.9\%$.
Through investigation, we find that the current advanced MLLMs, e.g., o3 and Gemini-2.5-pro, not only fail to identify the deterministic clues, but also tend to hallucinate and accuse everything by generating lots of so-called suspicious clues, even for innocent images, which is not preferred.
Taken together, these results show the significance of CaughtCheating, which reveals that recent MLLMs still lack detective-level capabilities, and further exposes the current boundary of their visual perception and reasoning capabilities.
Our contributions can be summarized as:

Figure: Figure 2: Demonstration of GPT-o3’s multimodal visual-reasoning breadth. (a) *Visual search*: locating Waldo in a densely populated illustration. (b) *Visual search for camouflage*: spotting a nearly invisible copperhead snake hidden among dry leaves. (c) *GeoGuessr*: identifying the upper terminal of Chair 1 at New Mexico, and estimating its latitude/longitude from a single image. (d) *TimeGuesser*: dating the photograph by matching architectural signage and period vehicles to museum and heritage records. These examples highlight o3’s strong visual perception and reasoning capacity across various visual tasks that most humans can not accomplish.
Refer to caption: extracted/6565317/images/example_llm_can_do.png

- •
We systematically evaluate the limits of current MLLMs in visual perception and reasoning, analyzing how they solve various complex tasks via sophisticated reasoning strategies, and identify a common scenario where even advanced models like o3’s performance drops to nearly zero.
- •
We present CaughtCheating, the first benchmark specifically designed to assess the ability to actively search and detect subtle, context-dependent suspicious clues in real-world images. Most human annotators and state-of-the-art agentic MLLMs struggle to succeed on CaughtCheating tasks, highlighting the lack of detective-level exploration skills.
- •
We analyze why even the most advanced agentic MLLMs fail on CaughtCheating. Inspired by the Guided Search theory, we find that these models often lack awareness of what to search for and how to relate observed details to the query. Our findings offer insights into both the construction of more challenging benchmarks and the limitations of existing MLLMs.

## 2 Exploring the Boundary of Visual Perception and Reasoning

### 2.1 Reasoning Trace Analysis of o3

As shown in the Figure [2](https://arxiv.org/html/2507.00045v1#S1.F2), $4$ representative task scenarios are selected for our qualitative analysis towards the boundary of MLLM visual perception and reasoning capabilities.
These tasks have been shown can be solved by the powerful agentic MLLM, GPT-o3, even if most of them can not be solved by individual humans(^3^33All the screenshots of o3 reasoning traces for solving these examples are provided in the Appendix [E](https://arxiv.org/html/2507.00045v1#A5).).

When solving (a), o3 systematically sweeps the image from broad overviews to focused zooms, homing in on red-and-white horizontal stripes of the character “Waldo”. After eliminating false matches quadrant by quadrant, it confirms Waldo’s outfit and hat, then translates his pixel coordinates into an easy landmark description.
When solving (b), the o3 methodically zooms into different areas of the leaf-litter image, from the center, lower left, and lower right, to searching for irregular shapes or patterns. Spotting rounded tan-brown coils with dark hourglass bands just left of center, then it recognize the tell-tale camouflage of a venomous pit viper (likely a copperhead).
When solving (c), o3 compares visual clues in the photo, red chairs on blue lift towers, the wide west, facing vista over Albuquerque’s grid, and the tree-line/elevation typical of Sandia Crest, with known features of Sandia Peak Ski Area. Cross-checking those details against published coordinates confirms the match.
When solving (d), o3 cross-checks catalog records for Henry King’s glass-plate negatives with heritage reports that caption this very view “c. 1890s.” Then it matches visual clues, horse buses and a Sydney Municipal, dense telegraph wires but no electric-tram overhead, and the original Anthony Hordern’s “Palace Emporium” sign that vanished after the $1901$ fire, to pin the scene to the year.

According to the above analysis, we find that the o3 model approaches these tasks with a methodical, exhaustive grid search, inspecting each region or object one by one until all plausible candidates are ruled in or out.
However, the effectiveness of this exhaustive approach will be largely negatively affected if the target object is easily overlooked.
Figure [1](https://arxiv.org/html/2507.00045v1#S1.F1) presents an example: When trying to solve the given query, o3 zooms in on the areas including pizza to confirm if slices were missing, the spoon and glass reflections to spot another diner, and the wing plate and surrounding dishes to gauge portion sizes and leftover clues.
However, it fails to notice that there are multiple people visible in the spoon’s reflection.
Compared with other objects, the spoon is so negligible that o3 does not pay much attention to it, thus leading to the failure.
Moreover, even occasionally, o3 coincidentally pays more attention to the spoon, it can not successfully perceive the content in the reflection.
To conclude, we find that even though o3 is able to accomplish some complex tasks, it mainly relies on an exhaustive grid search, which indicates a lack of detective-level visual perception and reasoning capabilities.

### 2.2 Guided Search Theory

To theoretically analyze the differences between the existing visual tasks and CaughtCheating, we introduce the Guided Search theory (Wolfe et al., 1989) and the corresponding factors (Wolfe and Horowitz, 2017) that guide attention in visual search in the area of cognitive science.
In its theory, searching involves directing attention to objects that might be the target.
This process is guided to the most promising items and locations by five factors discussed in the theory: bottom-up salience, top-down feature guidance, scene structure and meaning, the previous history of search, and the relative value of the targets and distractors.
Through investigation on the reasoning traces of o3, we find this theory, though initially proposed in the area of cognitive science, is still applicable to the current MLLMs.
We argue that CaughtCheating is significantly more challenging than many existing visual reasoning tasks, including those depicted in Figure [2](https://arxiv.org/html/2507.00045v1#S1.F2), due to the interplay of these factors.

Bottom-Up Salience refers to the extent to which an item “pops out” from its surroundings due to its intrinsic visual properties (e.g., color, orientation, contrast). This aspect represents the easiest strategy to make visual search hard.
In examples like Figure [2](https://arxiv.org/html/2507.00045v1#S1.F2) (a) and (b), both the targeting objects have low bottom-up salience, making them hard to find and requiring exhaustive searches. Similarly, suspicious cues in CaughtCheating also have extremely low bottom-up salience, like a reflection in a spoon, a partially obscured object, or a subtle item in the background, and are easily overlooked.

Top-Down Feature Guidance involves using knowledge about the target’s properties to guide search. Previous tasks benefit significantly from top-down guidance. For Waldo, the model searches for specific red-and-white stripes as a distinct character. For the snake, the query about “danger” might guide the model to look for threatening patterns. GeoGuesser and TimeGuesser rely on identifying specific architectural styles, vegetation, or period-specific artifacts. However, this is where CaughtCheating poses a major hurdle. The “target”, i.e., the suspicious clue, is often not a predefined object but an anomaly whose significance is context-dependent.
Lacking the top-down feature guidance, the model does not know what to look for because the clue could be almost anything (an extra glass, a reflection, an out-of-place item).
As observed, even if o3 occasionally focuses on the correct object (like the spoon), it may still fail to perceive the detail within it or infer its implication.

For Scene Structure and Meaning, the understanding of typical scene layouts and the relationships between objects helps guide attention to likely target locations. For previous tasks, o3 leverages scene context effectively. In GeoGuesser, it compares visual clues with known features of geographical locations. In TimeGuesser, it matches visual clues like vehicles and signage to historical records.
However, for CaughtCheating, the image itself might seem like an ordinary food picture or a hotel picture. Allocating the critical visual clues for the task does not merely require object recognition; it also needs to interpret subtle social cues and deviations from a presumed norm (e.g., “dining alone”). Current MLLMs struggle with this divergent reasoning over subtle, context-dependent cues, often focusing on non-deterministic details rather than decisive evidence.

In summary, CaughtCheating is more complex due to the extremely low bottom-up salience of crucial cues, the profound lack of specific top-down feature guidance, and the need to interpret subtle social context rather than just recognizing objects or well-defined patterns. While current agentic MLLMs can methodically search and identify objects through a process of elimination and feature matching, CaughtCheating demands a more nuanced “detective-level” ability to identify initially inconspicuous details and infer their significance within a specific social claim.

## 3 Benchmark Construction

Figure: Figure 3: An example of the annotation for the "Clued" category. Each image is annotated with a general question assessing overall suspicion and decomposed questions focused on a deterministic clue (here, the feminine bow hair accessory). Decomposed questions include perception-based inquiries (clue identification) and reasoning-based inquiries (social implications and contradictions), all annotated with the expected answer "yes".
Refer to caption: extracted/6565317/images/example_annotation.png

### 3.1 Image Collection

We collect images from publicly posted photographs on social media, focusing on those posts that request others to detect potential suspicious clues that violate their partners’ claims from the photos.
We only collect images that either clearly contain or lack subtle clues related to potential violation of the claim.
Each image is manually reviewed to ensure sufficient resolution quality for identifying such clues.
Due to the limited availability of images with naturally occurring subtle clues, we apply minimal cropping to some images that originally show multiple people, transforming them into single-person photos while preserving subtle indicators of another person’s presence.
This approach allows us to create challenging cases where the clues are interpretable for humans but not immediately obvious.
To ensure practical relevance, we exclude any synthetic images generated by image generation models.
After careful selection and verification, we construct a dataset of 100 images, split into Clued and Unclued categories, with all personal information removed.
A detailed version of Benchmark Construction, including the image examples, is provided in the Appendix [B](https://arxiv.org/html/2507.00045v1#A2).

### 3.2 Annotation

After constructing the image set, we annotate each image with a set of questions and corresponding ground-truth answers.
A detailed annotated example is shown in Figure [3](https://arxiv.org/html/2507.00045v1#S3.F3).
For images in Clued category, we annotate each one using a question instantiated from the template: “My [girlfriend/boyfriend] said [she/he] is [in a certain scenario] and sent me this photo. Do you notice anything suspicious in the image that contradicts [her/his] claim?”
Among the potential clues, the one that deterministically shows the violation of the providing claim (a clearly identifiable, contextually inappropriate element) will be selected as the Deterministic Clue, e.g., a pair of slippers is being worn by someone in Figure [3](https://arxiv.org/html/2507.00045v1#S3.F3).
The remaining clues are labeled as Non-deterministic Clues (weaker or more ambiguous signals), e.g., the rose bouquet, the TV shows, and the far-reached drinks.
These non-deterministic clues might be suspicious, but apparently not enough to infer the potential claim violation.
The reason we provide these clues is to avoid punishing models when they mention these clues.

Furthermore, we construct a series of decomposed questions designed to analyze the visual reasoning process of MLLMs, shown in the right part of Figure [3](https://arxiv.org/html/2507.00045v1#S3.F3).
This series includes:
(1) Decomposed Perception Question, which assesses whether the MLLMs can identify the deterministic clue when we explicitly mention the clue and position.
(2) Decomposed Reasoning Question, which assesses whether MLLMs can understand the social implications of the clue, or whether MLLMs can imply the relation between the clue and the potential cheating.
The correct answer to each of these decomposed questions is annotated as “yes”.
These decomposed questions can be utilized for in-depth analysis on why MLLMs can not solve the question.

We annotate each image in the Unclued category using the same initial question template, with “There is no clear evidence.” as the ground-truth answer.

### 3.3 Data Distribution

Our dataset comprises 100 samples evenly distributed between Clued (50) and Unclued (50) categories. The images are captured in three common scene settings: hotels (69), dining venues (29), and karaoke bars (2). The gender distribution of photographers is balanced, with 55 male and 45 female photographers. This distribution reflects realistic patterns of photos with potential suspicious clues while maintaining a balanced representation across different categories and scenes.

### 3.4 Evaluation Metrics

We employ several evaluation metrics to comprehensively assess MLLMs’ performance in detecting potential claim violations from images.
Clued Accuracy (Clued Acc) measures whether MLLMs can successfully identify the key deterministic clues in images from the Clued category.
Intersection over Union (Clued IoU) evaluates how well MLLMs identify all relevant non-deterministic clues while avoiding unrelated elements in the Clued category.
Unclued Accuracy (Unclued Acc) assesses whether MLLMs can correctly determine the absence of suspicious clues in images from the Unclued category.
In addition to the above three metrics, we also report the accuracy of MLLMs on the decomposed questions in the analysis, including Decomposed Perception Accuracy (Dec. P Acc), Decomposed Reasoning Accuracy (Dec. R Acc), and Decomposed Overall Accuracy (Dec. Acc) for in-depth analysis.
These metrics together provide a comprehensive evaluation framework that captures both the accuracy of clue detection and the quality of reasoning in different scenarios.

To compute these metrics, we need to parse the key points from MLLMs’ open-ended responses and compare them with the ground-truth answers. Given the complexity of this task and the diversity of the responses, we recommend using human evaluators as the primary judges for the most accurate assessment. However, to enable fair and automated comparison across different models, we also develop an automatic evaluation approach using GPT-4.1 to parse and compare the model response. To validate the reliability of our automatic evaluation method, we calculate the inter-rater agreement between human evaluators and GPT-4.1 using Cohen’s Kappa coefficient. The resulting kappa scores of $0.82$ for Clued Acc and $0.943$ for Unclued Acc demonstrate strong alignment between human and automatic evaluation, indicating the reliability of our automated assessment approach.

Except for the provided accuracies on the two categories for an intuitive understanding of the discrepancies, we also provide the Precision, Recall, and F1 score for each model for a more comprehensive evaluation.
The value of the Recall is the same as the Clued Acc, and the value of F1 serves as an overall metric to evaluate the performance of the model.
A detailed version of Evaluation Metrics, including the calculation and transformation between metrics, is provided in the Appendix [C](https://arxiv.org/html/2507.00045v1#A3).

**Table 1: The accuracies, IoU on the Clued category, and the accuracy on the Unclued category, and the overall precision, recall, and F1 score. Models are grouped by parameter size and type (open-source vs. proprietary). Clued Acc and IoU represent the capability of MLLMs to identify the suspicious clues, which directly reflects the MLLMs’ visual perception and reasoning abilities. Even the best performing model, GPT-o3, only achieves 26.0% accuracy and 17.2% IoU, indicating the current boundary of MLLMs’ capabilities. Unclued Acc represents the capability of MLLMs to not generate any suspicious clues if the image is unclued. F1 score shows the overall capability of MLLMs on CaughtCheating, where GPT-o3, achieves only 23.9%. The highest F1 score is 23.9%, which is much lower than the human performance, indicating the current boundary of MLLMs’ capabilities.**
|  | Clued | Unclued | Overall |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Acc $\uparrow$ | IoU $\uparrow$ | Acc $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | F1 $\uparrow$ |
| InternVL2-1B (Chen et al., 2024e) | 0.0 | 0.0 | 82.0 | 0.0 | 0.0 | 0.0 |
| LLaVA-OV-1B (Li et al., 2024a) | 0.0 | 0.0 | 86.0 | 0.0 | 0.0 | 0.0 |
| InternVL2.5-1B (Chen et al., 2024e) | 0.0 | 0.0 | 94.0 | 0.0 | 0.0 | 0.0 |
| InternVL2-2B (Chen et al., 2024e) | 0.0 | 0.0 | 76.0 | 0.0 | 0.0 | 0.0 |
| InternVL2.5-2B (Chen et al., 2024e) | 0.0 | 0.0 | 68.0 | 0.0 | 0.0 | 0.0 |
| Qwen2.5-VL-3B (Bai et al., 2025) | 2.0 | 0.0 | 50.0 | 3.8 | 2.0 | 2.6 |
| LLaVA-v1.6-Mistral-7B (Li et al., 2024a) | 0.0 | 0.0 | 82.0 | 0.0 | 0.0 | 0.0 |
| LLaVA-OV-7B (Li et al., 2024a) | 2.0 | 0.0 | 52.0 | 4.0 | 2.0 | 2.7 |
| Qwen2.5-VL-7B (Bai et al., 2025) | 2.0 | 3.9 | 66.0 | 5.6 | 2.0 | 2.9 |
| InternVL2-8B (Chen et al., 2024e) | 0.0 | 0.0 | 76.0 | 0.0 | 0.0 | 0.0 |
| InternVL2.5-8B (Chen et al., 2024e) | 0.0 | 0.0 | 72.0 | 0.0 | 0.0 | 0.0 |
| LLaVA-1.6-Vicuna-13B (Li et al., 2024a) | 0.0 | 0.0 | 72.0 | 0.0 | 0.0 | 0.0 |
| InternVL2-26B (Chen et al., 2024e) | 2.0 | 1.8 | 10.0 | 2.2 | 2.0 | 2.1 |
| InternVL2.5-26B (Chen et al., 2024e) | 0.0 | 0.0 | 80.0 | 0.0 | 0.0 | 0.0 |
| InternVL2.5-38B  (Chen et al., 2024e) | 2.0 | 0.0 | 76.0 | 7.7 | 2.0 | 3.2 |
| InternVL2-40B (Chen et al., 2024e) | 4.0 | 0.7 | 12.0 | 4.4 | 4.0 | 4.2 |
| InternVL2-72B (Chen et al., 2024e) | 4.0 | 0.8 | 16.0 | 4.5 | 4.0 | 4.3 |
| InternVL2.5-72B (Chen et al., 2024e) | 2.0 | 0.8 | 80.0 | 9.1 | 2.0 | 3.3 |
| LLaVA-OV-72B (Li et al., 2024a) | 0.0 | 1.3 | 72.0 | 0.0 | 0.0 | 0.0 |
| GPT-4o (OpenAI et al., 2024) | 4.0 | 1.0 | 54.0 | 8.0 | 4.0 | 5.3 |
| Gemini-2-flash (DeepMind, 2025a) | 10.0 | 0.0 | 6.0 | 9.6 | 10.0 | 9.8 |
| Gemini-2.5-flash (DeepMind, 2025b) | 18.0 | 5.1 | 22.0 | 18.8 | 18.0 | 18.4 |
| Gemini-2.5-pro (DeepMind, 2025b) | 20.0 | 15.1 | 22.0 | 20.4 | 20.0 | 20.2 |
| GPT-o3 (OpenAI, 2025) | 26.0 | 17.2 | 8.0 | 22.0 | 26.0 | 23.9 |
| Human | 56.0 | / | 68.0 | 63.6 | 56.0 | 59.6 |

**Table 2: Performance on decomposed questions. Dec. P and Dec. R is the Decomposed Perception Accuracy and Decomposed Reasoning Accuracy of the model on the decomposed questions. Dec. is the Decomposed Accuracy which represents the proportion of the model correctly answering all the decomposed questions.**
|  | Dec. P | Dec. R | Dec. | Clued $\uparrow$ |
| --- | --- | --- | --- | --- |
| GPT-4o | 52.0 | 12.8 | 2.0 | 4.0 |
| Gemini-2-flash | 74.0 | 69.6 | 38.0 | 10.0 |
| Gemini-2.5-flash | 72.0 | 39.2 | 20.0 | 18.0 |
| Gemini-2.5-pro | 80.0 | 52.9 | 34.0 | 20.0 |
| GPT-o3 | 62.0 | 24.5 | 2.0 | 26.0 |
| Human | 82.0 | 97.8 | 80.0 | 56.0 |

Figure: Figure 4: Case studies of the models’ performance on the CaughtCheating examples. 3 representative models are selected, including GPT-o3, GPT-4o and InternVL2.5-1B, and 3 images are selected: (a) A difficult Clued image, (b) An easy Clued image, and (c) An Unclued image. The models’ responses are truncated for better visualization.
Refer to caption: extracted/6565317/images/example_answers.png

## 4 Experimental Results

### 4.1 Main Results

The main results are shown in Table [1](https://arxiv.org/html/2507.00045v1#S3.T1).
We report the accuracy and IoU on the Clued category and accuracy on the Unclued category.
Models are grouped by parameter size and type (open-source vs. proprietary).
For open-source models, we evaluate the LLaVA-OV (Li et al., 2024a), InternVL2 (Chen et al., 2024e),
InternVL2.5 (Chen et al., 2024e), and Qwen2.5-VL (Bai et al., 2025) families.
For proprietary models, we evaluate the GPT-4o (OpenAI et al., 2024), Gemini-2-flash (DeepMind, 2025a), Gemini-2.5-flash (DeepMind, 2025b), Gemini-2.5-pro (DeepMind, 2025b), and GPT-o3 (OpenAI, 2025) models.
Human performance is also reported for reference.

Clued Acc and Clued IoU represent the capability of MLLMs to identify the suspicious clues, which directly reflects the MLLMs’ visual perception and reasoning abilities.
For previous open-source models, the performance is almost negligible, as no models can reach an accuracy above 5%, indicating their inferior capabilities on visual perception, reasoning, or even instruction following.
As for proprietary models before the reasoning era, GPT-4o achieves 4.0% accuracy and 1.0% IoU, and Gemini-2-flash achieves 10.0% accuracy and 0.0% IoU. The performances are slightly better, indicating their better capabilities in instruction understanding and following, but still they can not reach accuracies above 10%.

Only for the recent strong large reasoning models, like Gemini-2.5-pro and GPT-o3, the performances can reach above 20% accuracy and 10% IoU, indicating their strong capabilities on visual perception and reasoning.
But still, even the best performing model, GPT-o3, only achieves 26.0% accuracy and 17.2% IoU, indicating the current boundary of MLLMs’ capabilities.
Considering that even human participants can only reach approximately 50% accuracy, we believe this benchmark is challenging enough and shows the current boundary of their visual perception and reasoning capabilities.

In the meantime, we also report the Unclued Acc to evaluate the capability of MLLMs to not generate any suspicious clues if the image is unclued. This is also important for the real-world application, as we do not prefer MLLMs to suspect and accuse anything if the image providers are innocent.
As shown in the table, most of the models reach high accuracies on this category; however, this performance is due to their inability to generate any suspicious clues.
On the contrary, the advanced agentic models, Gemini-2.5-pro and GPT-o3, achieve low accuracies on this category, indicating their hallucination of nonexistent suspicious clues even on unclued images.
These low accuracies reveal their lack of strong reasoning abilities to identify if something is suspicious or not.

Finally, the F1 scores represent the overall performance of the model, which is the harmonic mean of the Precision and Recall.
The highest F1 score is 23.9%, which is much lower than the human performance, indicating the current boundary of MLLMs’ capabilities.

### 4.2 Decomposition Analysis

To better understand why the current advanced MLLMs can not perform well in this task, we design a set of decomposed questions for each image in the Clued category. These questions are divided into two types: perception questions, which test whether the model can accurately identify the key deterministic clue when it is explicitly mentioned, and reasoning questions, which assess whether the model can correctly infer the implications or contradictions associated.
By evaluating model performance on these questions, we can disentangle errors caused by failures in visual perception from those arising in higher-level reasoning. This fine-grained analysis helps reveal whether a model’s failure is due to not seeing the clue at all, or seeing it but not understanding its significance, thus providing deeper insight into the limitations.

As shown in Table [2](https://arxiv.org/html/2507.00045v1#S3.T2), the Dec. P is far higher than the Clued Acc, indicating that the models can identify the key deterministic clue when it is explicitly mentioned.
Just like how humans do during the investigation process:
When the human participants are given the image, it’s hard for them to identify the suspicious clues at the first place, e.g. the refelction in Figure [1](https://arxiv.org/html/2507.00045v1#S1.F1) and the femine bow hair in Figure [3](https://arxiv.org/html/2507.00045v1#S3.F3), but once they are explicitly mentioned or pointed out, they will admit the presense of the items. This human behavior leads to the relatively high Dec. P for humans.

For the Dec. R, the performances are all relatively lower, especially for GPT-4o and GPT-o3. We find that it is because these two models refuse to answer any gender-related questions due to the safety alignment. However, even if 4o and o3 refuse to directly answer some of these questions related to genders, they might still use the gender-related information as suspicious clues. As for the Gemini families, the Dec. R accuracies are all relatively higher, while still lower than the Dec. P.

These results together indicate that current advanced MLLMs can identify the key subtle items in the image if they are explicitly mentioned. However, in CaughtCheating, when being asked to identify the suspicious clues without being given any hints, they tend to do an exhaustive search and generate lots of clues without really judging if the clues are suspicious or not, and at the same time, ignore the key but subtle deterministic clues. These behaviors are similar to humans and verify the hypothesis based on Guided Search theory.

## 5 Case Studies

In this section, we provide some examples to show how exactly different models perform on CaughtCheating, shown in Figure [4](https://arxiv.org/html/2507.00045v1#S3.F4).
In the figure, 3 representative models are selected, including GPT-o3, GPT-4o and InternVL2.5-1B, and 3 images are selected: (a) A difficult Clued image, (b) An easy Clued image, and (c) An Unclued image.

In (a), there is an elbow, and fingers are visible at the left edge of the photo, clearly indicating the presence of another person. However, all the models fail to identify this subtle but deterministic clue and focus on the reflection of the television, even though there are no visible clues in the reflection, as another person is sitting by the table.
What’s worse, o3 and 4o keep mentioning the two bottles or cups, which are obviously provided by the hotel and can not be the suspicious clues. On the contrary, InternVL2.5-1B can not provide any clues by saying this is a normal hotel image.
In (b), the reflection in TV clearly shows there are two people on the bed, thus all the selected models can identify this clue.
These 2 examples, (a) and (b), show that:
(1) models are able to see through reflections, and
(2) Reflection does not always contain suspicious clues, which further verifies that CaughtCheating is challenging since there are no fixed rules for the suspicious clues.

(c) shows an Uncled image, which is merely an ordinary food-sharing image. However, o3 still tries to generate a lot of so-called suspicious clues, including the amount of food, the place settings, and etc. This behaviour is not expected since we only want models to generate clues really suspecious, rather than accusing everything, which further indicates the values of CaughtCheating. Similarly to the above examples, InternVL2.5-1B can not provide any clues by saying this is a normal food-sharing image, that’s why it reaches the highest on the Unclued Acc.

## 6 Conclusion

In this work, we present CaughtCheating, a novel benchmark designed to evaluate the capabilities of MLLMs in detecting subtle, context-dependent suspicious clues in real-world images.
Our experiments reveal that even state-of-the-art models, such as o3, consistently fail to identify the hidden clues within these photos, suggesting the current boundary of the detective-level ability of MLLMs on visual perception and reasoning.

## Ethical Concern and Limitation

Because our benchmark relies exclusively on publicly available, annotatable social-media photographs, the source pool overwhelmingly features cisgender, heterosexual couples; the scarcity of labeled images depicting LGBTQ+ or non-monogamous relationships, therefore, forced us to center this demographic. The same data constraint limited us to a handful of commonplace settings, such as hotels, restaurants, cafés, and vacation scenes, so contexts such as nightlife, workplaces, or culturally specific environments remain undersampled. Finally, the benchmark targets one complex form of visual reasoning rooted in a particular social norm, detecting suspected infidelity, without extending to the wider spectrum of complex social reasoning inferences people could draw from images. These constraints arise from limited public data, and our future benchmarks will build on more diverse, richly annotated datasets that broaden demographic coverage, scenario variety, and the range of socially grounded visual-reasoning tasks.

## References

- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025.
[Qwen2.5-vl technical report](https://arxiv.org/abs/2502.13923).
*Preprint*, arXiv:2502.13923.
- Brown et al. (2020)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020.
Language models are few-shot learners.
*Advances in neural information processing systems*, 33:1877–1901.
- Chen et al. (2024a)
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and 1 others. 2024a.
Are we on the right way for evaluating large vision-language models?
*arXiv preprint arXiv:2403.20330*.
- Chen et al. (2024b)
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. 2024b.
M3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought.
*arXiv preprint arXiv:2405.16473*.
- Chen et al. (2025)
Ruxiao Chen, Chenguang Wang, Yuran Sun, Xilei Zhao, and Susu Xu. 2025.
From perceptions to decisions: Wildfire evacuation decision prediction with behavioral theory-informed llms.
*arXiv preprint arXiv:2502.17701*.
- Chen et al. (2024c)
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, and 1 others. 2024c.
Do not think that much for 2+ 3=? on the overthinking of o1-like llms.
*arXiv preprint arXiv:2412.21187*.
- Chen et al. (2024d)
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024d.
Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.
*arXiv preprint arXiv:2412.05271*.
- Chen et al. (2024e)
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024e.
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 24185–24198.
- Chowdhery et al. (2023)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2023.
Palm: Scaling language modeling with pathways.
*Journal of Machine Learning Research*, 24(240):1–113.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*.
- de Lima et al. (2025)
Edirlei Soares de Lima, Marco A Casanova, Bruno Feijó, and Antonio L Furtado. 2025.
Characterizing the investigative methods of fictional detectives with large language models.
*arXiv preprint arXiv:2505.07601*.
- DeepMind (2025a)
Google DeepMind. 2025a.
[Gemini 2.0 flash](https://deepmind.google/technologies/gemini/flash/).
- DeepMind (2025b)
Google DeepMind. 2025b.
[Gemini 2.5 pro preview model card](https://storage.googleapis.com/model-cards/documents/gemini-2.5-pro-preview.pdf).
Technical report, Google DeepMind.
- Deng et al. (2025a)
Huilin Deng, Ding Zou, Rui Ma, Hongchen Luo, Yang Cao, and Yu Kang. 2025a.
Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning.
*arXiv preprint arXiv:2503.07065*.
- Deng et al. (2025b)
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. 2025b.
Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement.
*arXiv preprint arXiv:2503.17352*.
- Deng et al. (2024)
Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, Quanquan Gu, James Y Zou, Kai-Wei Chang, and Wei Wang. 2024.
Enhancing large vision language models with self-training on image comprehension.
*Advances in Neural Information Processing Systems*, 37:131369–131397.
- Duncan and Humphreys (1989)
John Duncan and Glyn W Humphreys. 1989.
Visual search and stimulus similarity.
*Psychological review*, 96(3):433.
- Fan et al. (2025)
Chenrui Fan, Ming Li, Lichao Sun, and Tianyi Zhou. 2025.
Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill?
*arXiv preprint arXiv:2504.06514*.
- Fu et al. (2025)
Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Florencio, and Cha Zhang. 2025.
Refocus: Visual editing as a chain of thought for structured image understanding.
*arXiv preprint arXiv:2501.05452*.
- Gandhi et al. (2023)
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. 2023.
Understanding social reasoning in language models with language models.
*Advances in Neural Information Processing Systems*, 36:13518–13529.
- Gu et al. (2023)
Zhouhong Gu, Lin Zhang, Jiangjie Chen, Haoning Ye, Xiaoxuan Zhu, Zihan Li, Zheyu Ye, Yan Gao, Yao Hu, Yanghua Xiao, and 1 others. 2023.
Piecing together clues: A benchmark for evaluating the detective skills of large language models.
*arXiv preprint arXiv:2307.05113*.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*.
- Hao et al. (2025)
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. 2025.
Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark.
*arXiv preprint arXiv:2501.05444*.
- Huang et al. (2025a)
Jingyuan Huang, Jen-tse Huang, Ziyi Liu, Xiaoyuan Liu, Wenxuan Wang, and Jieyu Zhao. 2025a.
Vlms as geoguessr masters: Exceptional performance, hidden biases, and privacy risks.
*arXiv preprint arXiv:2502.11163*.
- Huang et al. (2025b)
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. 2025b.
Vision-r1: Incentivizing reasoning capability in multimodal large language models.
*arXiv preprint arXiv:2503.06749*.
- Itti and Koch (2001)
Laurent Itti and Christof Koch. 2001.
Computational modelling of visual attention.
*Nature reviews neuroscience*, 2(3):194–203.
- Itti et al. (2002)
Laurent Itti, Christof Koch, and Ernst Niebur. 2002.
A model of saliency-based visual attention for rapid scene analysis.
*IEEE Transactions on pattern analysis and machine intelligence*, 20(11):1254–1259.
- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*.
- Jin et al. (2024)
Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, and 1 others. 2024.
Graph chain-of-thought: Augmenting large language models by reasoning on graphs.
*arXiv preprint arXiv:2404.07103*.
- Jin et al. (2025)
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025.
Search-r1: Training llms to reason and leverage search engines with reinforcement learning.
*arXiv preprint arXiv:2503.09516*.
- Johnson et al. (2017)
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017.
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 2901–2910.
- Kojima et al. (2022)
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.
Large language models are zero-shot reasoners.
*Advances in neural information processing systems*, 35:22199–22213.
- Lee et al. (2024)
Seungpil Lee, Woochang Sim, Donghyeon Shin, Wongyu Seo, Jiwon Park, Seokki Lee, Sanha Hwang, Sejin Kim, and Sundong Kim. 2024.
Reasoning abilities of large language models: In-depth analysis on the abstraction and reasoning corpus.
*ACM Transactions on Intelligent Systems and Technology*.
- Li et al. (2024a)
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024a.
[Llava-onevision: Easy visual task transfer](https://arxiv.org/abs/2408.03326).
*Preprint*, arXiv:2408.03326.
- Li et al. (2024b)
Ming Li, Han Chen, Chenguang Wang, Dang Nguyen, Dianqi Li, and Tianyi Zhou. 2024b.
Ruler: Improving llm controllability by rule-based data recycling.
*arXiv preprint arXiv:2406.15938*.
- Li et al. (2024c)
Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. 2024c.
Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning.
In *Findings of the Association for Computational Linguistics ACL 2024*, pages 16189–16211.
- Li et al. (2024d)
Ming Li, Pei Chen, Chenguang Wang, Hongyu Zhao, Yijun Liang, Yupeng Hou, Fuxiao Liu, and Tianyi Zhou. 2024d.
Mosaic-it: Free compositional data augmentation improves instruction tuning.
*arXiv preprint arXiv:2405.13326*.
- Li et al. (2025a)
Ming Li, Yanhong Li, Ziyue Li, and Tianyi Zhou. 2025a.
How instruction and reasoning data shape post-training: Data quality through the lens of layer-wise gradients.
*arXiv preprint arXiv:2504.10766*.
- Li et al. (2024e)
Ming Li, Yanhong Li, and Tianyi Zhou. 2024e.
What happened in llms layers when trained for fast vs. slow thinking: A gradient perspective.
*arXiv preprint arXiv:2410.23743*.
- Li et al. (2025b)
Ming Li, Zhengyuan Yang, Xiyao Wang, Dianqi Li, Kevin Lin, Tianyi Zhou, and Lijuan Wang. 2025b.
What makes reasoning models different? follow the reasoning leader for efficient decoding.
*arXiv preprint arXiv:2506.06998*.
- Li et al. (2025c)
Ming Li, Ruiyi Zhang, Jian Chen, Jiuxiang Gu, Yufan Zhou, Franck Dernoncourt, Wanrong Zhu, Tianyi Zhou, and Tong Sun. 2025c.
Towards visual text grounding of multimodal large language model.
*arXiv preprint arXiv:2504.04974*.
- Liang et al. (2025)
Yijun Liang, Ming Li, Chenrui Fan, Ziyue Li, Dang Nguyen, Kwesi Cobbina, Shweta Bhardwaj, Jiuhai Chen, Fuxiao Liu, and Tianyi Zhou. 2025.
Colorbench: Can vlms see and understand the colorful world? a comprehensive benchmark for color perception, reasoning, and robustness.
*arXiv preprint arXiv:2504.10514*.
- Lightman et al. (2023)
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023.
Let’s verify step by step.
In *The Twelfth International Conference on Learning Representations*.
- Liu et al. (2024)
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024.
Improved baselines with visual instruction tuning.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 26296–26306.
- Liu et al. (2023)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023.
Visual instruction tuning.
*Advances in neural information processing systems*, 36:34892–34916.
- Lu et al. (2023)
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023.
Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.
*arXiv preprint arXiv:2310.02255*.
- Lu et al. (2022)
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022.
Learn to explain: Multimodal reasoning via thought chains for science question answering.
*Advances in Neural Information Processing Systems*, 35:2507–2521.
- Luo et al. (2025)
Weidi Luo, Qiming Zhang, Tianyu Lu, Xiaogeng Liu, Yue Zhao, Zhen Xiang, and Chaowei Xiao. 2025.
Doxing via the lens: Revealing privacy leakage in image geolocation for agentic multi-modal large reasoning model.
*arXiv preprint arXiv:2504.19373*.
- Meng et al. (2025)
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, and 1 others. 2025.
Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning.
*arXiv preprint arXiv:2503.07365*.
- Muennighoff et al. (2025)
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025.
s1: Simple test-time scaling.
*arXiv preprint arXiv:2501.19393*.
- OpenAI (2025)
OpenAI. 2025.
[Openai o3 and o4-mini system card](https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf).
Technical report, OpenAI.
- OpenAI et al. (2024)
OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and etc. 2024.
[Gpt-4o system card](https://arxiv.org/abs/2410.21276).
*Preprint*, arXiv:2410.21276.
- Ouyang et al. (2022)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022.
Training language models to follow instructions with human feedback.
*Advances in neural information processing systems*, 35:27730–27744.
- Patel et al. (2021)
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021.
Are nlp models really able to solve simple math word problems?
*arXiv preprint arXiv:2103.07191*.
- Peng et al. (2024)
Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. 2024.
Multimath: Bridging visual and mathematical reasoning for large language models.
*arXiv preprint arXiv:2409.00147*.
- Peng et al. (2025)
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. 2025.
Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl.
*arXiv preprint arXiv:2503.07536*.
- Phan et al. (2025)
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, and 1 others. 2025.
Humanity’s last exam.
*arXiv preprint arXiv:2501.14249*.
- Ranaldi and Freitas (2024)
Leonardo Ranaldi and Andrè Freitas. 2024.
Self-refine instruction-tuning for aligning reasoning in language models.
*arXiv preprint arXiv:2405.00402*.
- Uesato et al. (2022)
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022.
Solving math word problems with process-and outcome-based feedback.
*arXiv preprint arXiv:2211.14275*.
- Wang et al. (2022a)
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022a.
Git: A generative image-to-text transformer for vision and language.
*arXiv preprint arXiv:2205.14100*.
- Wang et al. (2024a)
Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, and 1 others. 2024a.
Enhancing visual-language modality alignment in large vision language models via self-improvement.
*arXiv preprint arXiv:2405.15973*.
- Wang et al. (2025a)
Xiyao Wang, Zhengyuan Yang, Chao Feng, Yongyuan Liang, Yuhang Zhou, Xiaoyu Liu, Ziyi Zang, Ming Li, Chung-Ching Lin, Kevin Lin, and 1 others. 2025a.
Vicrit: A verifiable reinforcement learning proxy task for visual perception in vlms.
*arXiv preprint arXiv:2506.10128*.
- Wang et al. (2025b)
Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. 2025b.
Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement.
*arXiv preprint arXiv:2504.07934*.
- Wang et al. (2024b)
Xiyao Wang, Zhengyuan Yang, Linjie Li, Hongjin Lu, Yuancheng Xu, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. 2024b.
Scaling inference-time search with vision value model for improved visual comprehension.
*arXiv preprint arXiv:2412.03704*.
- Wang et al. (2022b)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022b.
Self-consistency improves chain of thought reasoning in language models.
*arXiv preprint arXiv:2203.11171*.
- Wang et al. (2024c)
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, and 1 others. 2024c.
Charxiv: Charting gaps in realistic chart understanding in multimodal llms.
*Advances in Neural Information Processing Systems*, 37:113569–113697.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022.
Chain-of-thought prompting elicits reasoning in large language models.
*Advances in neural information processing systems*, 35:24824–24837.
- Wen et al. (2025)
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, and 1 others. 2025.
Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.
*arXiv preprint arXiv:2503.10460*.
- Wolfe et al. (1989)
Jeremy M Wolfe, Kyle R Cave, and Susan L Franzel. 1989.
Guided search: an alternative to the feature integration model for visual search.
*Journal of Experimental Psychology: Human perception and performance*, 15(3):419.
- Wolfe and Horowitz (2017)
Jeremy M Wolfe and Todd S Horowitz. 2017.
Five factors that guide attention in visual search.
*Nature human behaviour*, 1(3):0058.
- Wu et al. (2025)
Junde Wu, Jiayuan Zhu, and Yuyuan Liu. 2025.
Agentic reasoning: Reasoning llms with tools for the deep research.
*arXiv preprint arXiv:2502.04644*.
- Wu et al. (2024)
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. 2024.
Minds eye of llms: Visualization-of-thought elicits spatial reasoning in large language models.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*.
- Xia et al. (2024)
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024.
Less: Selecting influential data for targeted instruction tuning.
*arXiv preprint arXiv:2402.04333*.
- Xie et al. (2024)
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. 2024.
Monte carlo tree search boosts reasoning via iterative preference learning.
*arXiv preprint arXiv:2405.00451*.
- Xu et al. (2024)
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. 2024.
Llava-o1: Let vision language models reason step-by-step.
*arXiv preprint arXiv:2411.10440*.
- Yao et al. (2023)
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.
Tree of thoughts: Deliberate problem solving with large language models.
*Advances in neural information processing systems*, 36:11809–11822.
- Yu et al. (2025)
Bin Yu, Hang Yuan, Yuliang Wei, Bailing Wang, Weizhen Qi, and Kai Chen. 2025.
Long-short chain-of-thought mixture supervised fine-tuning eliciting efficient reasoning in large language models.
*arXiv preprint arXiv:2505.03469*.
- Yu et al. (2023)
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023.
Mm-vet: Evaluating large multimodal models for integrated capabilities.
*arXiv preprint arXiv:2308.02490*.
- Yuan et al. (2025)
Yuan Yuan, Muyu He, Muhammad Adil Shahid, Jiani Huang, Ziyang Li, and Li Zhang. 2025.
Turnaboutllm: A deductive reasoning benchmark from detective games.
*arXiv preprint arXiv:2505.15712*.
- Yue et al. (2024a)
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024a.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 9556–9567.
- Yue et al. (2024b)
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, and 1 others. 2024b.
Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark.
*arXiv preprint arXiv:2409.02813*.
- Zellers et al. (2019)
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.
From recognition to cognition: Visual commonsense reasoning.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 6720–6731.
- Zhang et al. (2024a)
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, and 1 others. 2024a.
Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?
In *European Conference on Computer Vision*, pages 169–186. Springer.
- Zhang et al. (2024b)
Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. 2024b.
Improve vision language model chain-of-thought reasoning.
*arXiv preprint arXiv:2410.16198*.
- Zhang et al. (2023)
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023.
Multimodal chain-of-thought reasoning in language models.
*arXiv preprint arXiv:2302.00923*.
- Zhou et al. (2023)
Pei Zhou, Aman Madaan, Srividya Pranavi Potharaju, Aditya Gupta, Kevin R McKee, Ari Holtzman, Jay Pujara, Xiang Ren, Swaroop Mishra, Aida Nematzadeh, and 1 others. 2023.
How far are large language models from agents with theory-of-mind?
*arXiv preprint arXiv:2310.03051*.
- Zhou et al. (2024)
Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024.
Calibrated self-rewarding vision language models.
*arXiv preprint arXiv:2405.14622*.

## Table of Contents for Appendix

## Appendix A Relate Work

### A.1 LLM reasoning

The chain-of-thought technique (Wei et al., 2022; Kojima et al., 2022) represents the early efforts in exploring the reasoning capabilities of large language models (LLMs) (Brown et al., 2020; Chowdhery et al., 2023). By explicitly generating intermediate reasoning steps, this method notably enhances performance across various reasoning tasks (Patel et al., 2021; Cobbe et al., 2021).
Moreover, advances in decoding strategies have introduced additional test-time computation to further boost performance. For instance, Self-Consistency sampling (Wang et al., 2022b), which employs voting mechanisms to select from multiple reasoning paths, has notably increased reliability.
Expanding beyond linear reasoning processes, structured frameworks such as Tree-of-thought Yao et al. (2023) or Graph-of-thought (Jin et al., 2024)facilitate the exploration of multiple candidate reasoning paths within branched subspaces before reaching a final conclusion.
Other research investigates manipulating the reasoning process to generate longer chains of thought than those typically observed, either by explicitly prompting extended reasoning chains (Muennighoff et al., 2025) or by integrating human-like cognitive theory foundations into the inference process (Zhou et al., 2023; Gandhi et al., 2023; Lee et al., 2024; Chen et al., 2025).
Furthermore, supervised fine-tuning (SFT) not only improves general instruction-following performance (Ouyang et al., 2022; Xia et al., 2024; Li et al., 2024d, b, c, e) but has also been demonstrated to significantly enhance multi-step reasoning capabilities when trained on structured chain-of-thought (CoT) traces, where models learn to explicitly generate intermediate reasoning steps (Ranaldi and Freitas, 2024; Wen et al., 2025; Li et al., 2025a; Muennighoff et al., 2025; Yu et al., 2025; Li et al., 2025b). Additionally, prior research has employed reward models during training to evaluate each intermediate reasoning step individually, rather than solely assessing final outcomes, further improving reasoning performance (Uesato et al., 2022; Lightman et al., 2023). This approach integrates effectively with Monte Carlo Tree Search techniques (Xie et al., 2024), providing valuable insights into performance gains achieved through fine-grained value estimations.
Beyond training, many studies augment the reasoning process with the ability to invoke external tools and knowledge sources, a paradigm known as “agentic reasoning” (Wu et al., 2025). In this paradigm, LLMs call tools such as calculators, code interpreters, web search, and other utilities to provide context from tools results into the reasoning process to solve complex tasks. For instance,  Jin et al. (2025) introduces the Search-R1, which lets an LLM query a search engine and condition subsequent reasoning on the retrieved evidence.
Recent developments in large-scale reinforcement learning, relying solely on outcome-based rewards, have demonstrated potential for inducing emergent multi-step reasoning capabilities (Guo et al., 2025; Jaech et al., 2024).
While the advancements on reasoning also potentially lead to the emergence of overthining issue (Chen et al., 2024c; Fan et al., 2025).
Such advancements underscore the importance of tasks that can be automatically verified (e.g., RL can be effectively scaled up with minimal noise in its reward signals).

### A.2 MLLM reasoning

Recent developments in MLLMs (Wang et al., 2022a; Liu et al., 2023; OpenAI et al., 2024; Liu et al., 2024; Chen et al., 2024e, d; Bai et al., 2025) have led to the exploration of multimodal chain-of-thought techniques aimed at enhancing performance on visual reasoning tasks (Yu et al., 2023; Lu et al., 2023; Hao et al., 2025) with both textual reasoning process (Lu et al., 2022; Zhang et al., 2023) and multimodal reasoning path (Wu et al., 2024; Fu et al., 2025).
Methods such as rationale distillation and self-reflection have also been employed to strengthen reasoning capabilities (Zhang et al., 2024b; Zhou et al., 2024; Wang et al., 2024a, b; Deng et al., 2024).
Besides, LLaVA-o1 (Xu et al., 2024) proposes a fine-tuning strategy that leverages a dataset enriched with structured reasoning annotations (e.g., summarization, visual analysis, logical deduction, conclusion), achieving substantial performance improvement.
Inspired by successes in reinforcement learning of LLMs, recent efforts have similarly applied this method to visual math problems and other visual question-answering tasks (Deng et al., 2025b; Huang et al., 2025b; Wang et al., 2025b; Peng et al., 2025; Meng et al., 2025).
For example, Curr-ReFT (Deng et al., 2025a) introduces a three-stage progression paradigm that blends RL with curriculum design to mimic the student learning process, significantly improving generalization and step-by-step reasoning capability.
Although these approaches have improved performance on visual math and STEM-related questions, substantial progress in fine-grained visual perception remains limited. For instance, MMMU (Yue et al., 2024a) shows that current MLLMs, though strong on everyday tasks, stumble on domain-specific reasoning and complex, specialized imagery; many items can be solved from textual cues or memorized facts without genuine visual grounding. Its successor, MMMU-Pro (Yue et al., 2024b), reinforces these findings and demonstrates that prompts encouraging explicit multi-step linguistic reasoning boost performance, provided the model truly incorporates visual evidence at each step.
Similarly, MultiMath (Peng et al., 2024) reveals that many MLLMs are under-performing with purely visual inputs with minimal text, indicating that the understanding of complex spatial reasoning in mathematical or scientific diagrams remains challenging.
TRIG (Li et al., 2025c), proposing the first visual text grounding task, shows the inability of MLLMs to perform visual reasoning and grounding.
ColorBench (Liang et al., 2025) introduces the first comprehensive benchmark for color perception, reasoning, and robustness, showcaseing the low capability of MLLMs on color-related perception and reasoning.
ViCrit (Wang et al., 2025a) on the other hand, introduces the verifiable reinforcement learning proxy task for visual perception in VLMs.

## Appendix B Detailed Benchmark Construction

### B.1 Image Collection

For CaughtCheating images, we use publicly posted photographs from social media.
We manually search and review all the comments for each image to assess their suitability. Selected images must either contain or lack subtle, suspicious clues related to potential claim violation. The judgment of image candidates is based not only on the comments but also on human evaluation. Additionally, each image must have sufficient resolution quality to allow us to directly identify such clues, rather than rely on implications from blurred or indistinct objects.

Due to the limited availability of images with clear, subtle clues from public sources, we also include minimally modified versions of images containing direct clues (e.g., a clearly visible person or untypical belongings suggesting the presence of another individual). We apply simple cropping to these images to obscure the direct clues. As shown in Figure [5](https://arxiv.org/html/2507.00045v1#A2.F5), the original photo shows a person sitting on the couch. After cropping, only their back remains visible, making the clue still interpretable for humans, yet challenging for MLLMs.

Figure: Figure 5: Example of cropping an image for with-clue category. The original photo shows part of the person sitting on the sofa (Before). By cropping (After), we can still infer there is a person, but identifying the clue is more subtle and challenging for MLLM.
Refer to caption: extracted/6565317/images/example_corp.png

To ensure quality and not weaken the practical usage of the task, we do not use any synthetic images generated by image generation models.
A generated example is shown in Figure [6](https://arxiv.org/html/2507.00045v1#A2.F6), in which we instruct GPT-Image-1 to generate a hotel room scene with subtle clues placed far from the camera and intended to be difficult to detect. However, the model consistently fails to follow these instructions, instead producing images where the clues were overt and easily noticeable (the condom on the floor). As these outputs do not meet our criteria, we don’t employ the image generation for our benchmark.

Figure: Figure 6: Example of generated image. The clues, such as a scattered lipstick, are clearly visible in the image. The clue is obvious rather than subtle, making the inference extremely unchallenging for MLLMs.
Refer to caption: extracted/6565317/images/example_generate.jpeg

After collecting a sufficient number of candidate images, we meticulously selected $100$ images, split into Clued and Unclued categories, to construct the image set for CaughtCheating benchmark.
All the images are verified manually to make sure the clues are solid and no personal information exists on the image.

### B.2 Annotation

After constructing the image set, we annotate each image with a set of questions and corresponding ground-truth answers.
A detailed annotated example is shown in Figure [3](https://arxiv.org/html/2507.00045v1#S3.F3).
For images in Clued category, we annotate each one using a question instantiated from the template: “My [girlfriend/boyfriend] said [she/he] is [in a certain scenario] and sent me this photo. Do you notice anything suspicious in the image that contradicts [her/his] claim?”
Among the potential clues, the one that deterministically shows the violation of the providing claim (a clearly identifiable, contextually inappropriate element) will be selected as the Deterministic Clue, e.g., a pair of slippers is being worn by someone in Figure [3](https://arxiv.org/html/2507.00045v1#S3.F3).
The remaining clues are labeled as Non-deterministic Clues (weaker or more ambiguous signals), e.g., the rose bouquet, the TV shows and the far-reached drinks.
These non-deterministic clues might be suspicious, but apparently not enough to infer the potential claim violation.
The reason we provide these clues is to avoid punishing models when they mention these clues.

Furthermore, we construct a series of decomposed questions designed to analyze the visual reasoning process of MLLMs, shown in the right part of Figure [3](https://arxiv.org/html/2507.00045v1#S3.F3).
This series includes:
(1) Decomposed Perception Question, which assesses whether the MLLMs can identify the deterministic clue when we explicitly mention the clue and position.
(2) Decomposed Reasoning Question, which assesses whether MLLMs can understand the social implications of the clue, or whether MLLMs can imply the relation between the clue and the potential lie.
The correct answer to each of these decomposed questions is annotated as “yes”.
These decomposed questions can be utilized for in-depth analysis on why MLLMs can not solve the question.

These decomposed questions can be utilized for in-depth analysis on why MLLMs can not solve the question.
(1) If the MLLMs have low accuracy on perception-related decomposed questions, it means the low performance is caused by their poor visual perception ability.
(2) If the MLLMs have low accuracy on reasoning-related decomposed questions, it means the low performance is caused by their poor visual reasoning ability.
(3) If the MLLMs have relatively high accuracy on both types of decomposed questions, it means they have the necessary capabilities to solve the task, but they do not know where to start.
For images in the Unclued category, we annotate each using the same initial question template, with the ground-truth answer labeled as "There is no clear evidence."

### B.3 Data Distribution

**Table 3: Distribution of scenes, interlocutor gender, and question types across the two clue categories.**
| Category | Scene | Male | Female | # Dec. P | # Dec. R | # Total |
| --- | --- | --- | --- | --- | --- | --- |
| with-clue | Dining | 6 | 5 | 11 | 21 | 50 |
| Hotel | 25 | 12 | 37 | 67 |  |  |
| Karaoke bar | 2 | 0 | 2 | 3 |  |  |
| without-clue | Dining | 7 | 11 | 0 | 0 | 50 |
| Hotel | 13 | 19 | 0 | 0 |  |  |

Our dataset comprises 100 samples collected from publicly posted photos, each manually annotated with a series of questions accompanied by ground-truth answers. These samples serve as test cases to evaluate the capability of MLLMs in detecting potential claim violations.

The dataset is evenly divided into two categories: the *Clued* category (50 samples), which includes clear indicators of potential claim violation, and the *Unclued* category (50 samples), which lacks explicit indicators. This balanced distribution aims to minimize class bias and ensure fair evaluation.
Furthermore, the dataset encompasses three distinct scene types based on photo backgrounds: hotels, dining venues, and karaoke bars. The gender attributes assigned to each sample reflect the photographer’s gender as inferred from the provided descriptions of the photos. These attributes do not pertain to any individuals depicted within the images. The gender categorization currently includes male and female solely based on limited available descriptive information.

Detailed statistics regarding scenario distribution and gender breakdown are summarized in Table [3](https://arxiv.org/html/2507.00045v1#A2.T3). Hotel scenes comprise the majority of the dataset (69% ), aligning with their prominence as typical settings for potentially suspicious scenarios. Dining venues account for 29% of the dataset, and karaoke bars represent the remaining 2%. Gender distribution is 55% male photographers and 45% female photographers.

Additionally, the dataset includes annotations for perception and reasoning questions derived from decomposition queries. Specifically, it contains 50 perception questions and 91 reasoning questions, thoroughly evaluating why MLLMs may fail to resolve specific queries. Detailed counts corresponding to scene types are provided in Table [3](https://arxiv.org/html/2507.00045v1#A2.T3). Each sample averages approximately two reasoning questions, enabling comprehensive analysis of MLLM performance concerning both explicit clues and the broader social or environmental context.

## Appendix C Evaluation Metrics

We apply several evaluation metrics in our study, each designed to assess a distinct aspect of the visual reasoning process. All metrics rely on analyzing and comparing the ground-truth answers with the responses generated by MLLMs.

#### Clued Accuracy (Clued Acc)

Deterministic Accuracy is designed to evaluate whether an MLLM successfully identifies the deterministic clue hidden in the images from Clued category.
Let $k_{i}\in\{0,1\}$ denote the binary judgment for the $i$-th example in the Clued category, where $k_{i}=1$ if the Deterministic Clue is correctly identified, and $k_{i}=0$ otherwise. The Clued Acc is then defined as:

$$ $\text{Clued Acc}=\frac{1}{N_{clued}}\sum_{i=1}^{N_{clued}}k_{i}$ $$

where $N_{clued}$ is the total number of examples in the Clued subset.

#### Intersection over Union (Clued IoU)

In this context, IoU is designed to evaluate whether an MLLM correctly identifies all relevant Non-deterministic Clues hidden in the images from Clued category, while avoiding unrelated or incorrect elements. If the MLLM generates a lot of unrelated clues, this IoU value will be low, since we expect MLLMs only to mention clues that are at least somewhat suspicious.

Let $G_{i}$ be the set of all the clues annotated in the ground-truth for the $i$-th example in the Clued category, and $R_{i}$ be the set of clues identified by the MLLM. The Clued IoU is then defined as:

$$ $\text{IoU}=\frac{1}{N_{clued}}\sum_{i=1}^{N_{clued}}\frac{|G_{i}\cap R_{i}|}{| G_{i}\cup R_{i}|}$ $$

#### Decomposed Accuracies

This evaluation comprises three specific accuracy metrics: Decomposed Perception Accuracy (Dec. P Acc) provides detailed insights into the model’s performance in accurately perceiving claims from images when the clues are explicitly mentioned; Decomposed Reasoning Accuracy (Dec. R Acc) evaluates the model’s proficiency in reasoning towards the deterministic clue; and Decomposed Overall Accuracy (Dec. Acc) offers a comprehensive evaluation by combining performance in both perception and reasoning dimensions. This metric is specifically tailored for images within the Clued category.

Let $\mathcal{P}_{i}$ be the set of perception-related questions for the $i$-th example in the Clued category, and
$\widehat{\mathcal{P}}_{i}\subseteq\mathcal{P}_{i}$ be the subset that the MLLM correctly answered perception-related questions.
The Decomposed Perception Accuracy (Dec. P Acc) is then defined as:

$$ $\text{Dec. P Acc}=\frac{1}{N_{clued}}\sum_{i=1}^{N_{clued}}\frac{\lvert \widehat{\mathcal{P}}_{i}\rvert}{\lvert\mathcal{P}_{i}\rvert}$ $$

Likewise, let $\mathcal{R}_{i}$ and $\widehat{\mathcal{R}}_{i}$ denote the sets of reasoning-related questions and the correctly answered subset, respectively.
The Decomposed Reasoning Accuracy (Dec. R Acc) is defined as:

$$ $\text{Dec. R Acc}=\frac{1}{N_{clued}}\sum_{i=1}^{N_{clued}}\frac{\lvert \widehat{\mathcal{R}}_{i}\rvert}{\lvert\mathcal{R}_{i}\rvert}$ $$

Finally, let $\mathbbm{1}(\cdot)$ denotes the indicator function. The Decomposed Overall Accuracy (Dec. Acc) is defined as:

$$ $\text{Dec. Acc}=\frac{1}{N_{clued}}\sum_{i=1}^{N_{clued}}\mathbbm{1}\bigl{(} \lvert\widehat{\mathcal{P}}_{i}\rvert=\lvert\mathcal{P}_{i}\rvert\land\;\lvert \widehat{\mathcal{R}}_{i}\rvert=\lvert\mathcal{R}_{i}\rvert\bigr{)}$ $$

#### Unclued Accuracy (Unclued Acc):

Unclued Accuracy (Unclued Acc) is designed to evaluate whether the MLLM can correctly determine the absence of clear clues from the Unclued category. Let $o_{i}\in\{0,1\}$ denote the binary judgment for the $i$-th example. Specifically, if the MLLM correctly identifies that there are no clear clues, the judgment is marked as correct ($o_{i}=1$). Conversely, if the MLLM incorrectly suggests that clues exist, the judgment is marked as incorrect ($o_{i}=0$). The overall accuracy is computed as follows:

$$ $\text{Unclued Acc}=\frac{1}{N_{unclued}}\sum_{i=1}^{N_{unclued}}o_{i}$ $$

where $N_{unclued}$ is the total number of examples in the Unclued subset.

#### Precision, Recall, and F1 Score:

The transformation between the accuracies and P/R/F1 scores is as follows:

$$ TP $\displaystyle=\text{Clued Acc}\times N_{\text{clued}},$ FN $\displaystyle=(1-\text{Clued Acc})\times N_{\text{clued}},$ TN $\displaystyle=\text{Unclued Acc}\times N_{\text{unclued}},$ FP $\displaystyle=(1-\text{Unclued Acc})\times N_{\text{unclued}}.$ $$

where $N_{\text{clued}}$ and $N_{\text{unclued}}$ denote the numbers of images in the Clued and Unclued categories, respectively.
Using these quantities, we convert to the standard classification metrics:

$$ $\displaystyle\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}=\frac{ \text{Clued Acc}\times N_{\text{clued}}}{\text{Clued Acc}\times N_{\text{clued }}+(1-\text{Clued Acc})\times N_{\text{unclued}}},$ $\displaystyle\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}=\text{Clued Acc},$ $\displaystyle\text{F1}=\frac{2\,\text{Precision}\,\text{Recall}}{\text{ Precision}+\text{Recall}}.$ $$

These formulas allow us to compute the P/R/F1 scores from the reported Clued Acc and Unclued Acc values in the main text.

## Appendix D Evaluation Prompt

Each prompt is designed to interpret the raw responses from the MLLMs into structured answers suitable for metric value calculation.
We first designed four evaluation prompts for analyzing the MLLMs’ responses to the general question discussed in Section [3.2](https://arxiv.org/html/2507.00045v1#S3.SS2).

The prompt illustrated in Figure [7](https://arxiv.org/html/2507.00045v1#A4.F7) evaluates whether a deterministic cue is mentioned in the MLLMs’ response, permitting minor wording variations but emphasizing clear alignment with the original meaning. This prompt instructs the evaluation LLM to yield a binary YES or NO result used for Clued Acc evaluations.

The prompts illustrated in Figure [9](https://arxiv.org/html/2507.00045v1#A4.F9) and Figure [10](https://arxiv.org/html/2507.00045v1#A4.F10) both provide essential statistics for Clued IoU calculations.
The prompt in Figure [9](https://arxiv.org/html/2507.00045v1#A4.F9) instructs a model to explicitly extract detailed observations into a structured list from raw MLLMs’ responses, ensuring clarity by excluding ambiguous or uncertain statements. This structured list is subsequently compared with a structured list of non-deterministic cues extracted using the prompt illustrated in Figure [10](https://arxiv.org/html/2507.00045v1#A4.F10).
The prompt illustrated in Figure [10](https://arxiv.org/html/2507.00045v1#A4.F10) assesses whether each deterministic cue, serving as a ground truth reference, is explicitly mentioned or supported in the structured list derived from the MLLMs’ raw response. It instructs the evaluation LLM to provide a binary YES or NO decision. From these binary labels, we derive the intersection and union set sizes necessary to calculate Clued IoU.

Similar to the prompt in Figure [7](https://arxiv.org/html/2507.00045v1#A4.F7), the prompt illustrated in Figure [8](https://arxiv.org/html/2507.00045v1#A4.F8) evaluates whether the model correctly identifies the absence of cues. It instructs the evaluation LLM to yield binary YES or NO outcomes utilized for Unclued Acc calculations.

Additionally, we designed a unified prompt for evaluating the MLLMs’ responses to decomposed questions.
The prompt illustrated in Figure [11](https://arxiv.org/html/2507.00045v1#A4.F11) assesses if responses align with the expected answers to decomposed questions, providing binary YES or NO outcomes. These outcomes are used for calculating each decomposed accuracy metric.

Figure: Figure 7: The prompt used to request GPT4.1 to measure whether MLLMs successfully identify the key deterministic clues for the calculation of Clued Accuracy (Clued Acc).

Figure: Figure 8: The prompt used to request GPT4.1 to assess whether MLLMs correctly determine the absence of suspicious clues for calculation of Unclued Accuracy (Unclued Acc).

Figure: Figure 9: The prompt used to request GPT4.1 to convert MLLM’s answer into a key points list for the following inference to calculate Intersection over Union (Clued IoU).

Figure: Figure 10: The prompt used to request GPT4.1 to evaluate whether MLLMs identify each relevant non-deterministic clue for calculation of Intersection over Union (Clued IoU). Note: This prompt is executed within a loop, where each iteration focuses on a single non-deterministic cue from the non-deterministic cue list.

Figure: Figure 11: The prompt used to request GPT4.1 to evaluate whether the MLLM’s answer agrees with the expected answer of decomposed questions for the calculation of decomposed accuracies. Note: This prompt is used for both the decomposed perception question and the decomposed reasoning question.

## Appendix E o3 Reasoning Traces for Qualitative Examples

In this section, we provide a comprehensive overview of the reasoning processes underlying the tasks illustrated in Figure [1](https://arxiv.org/html/2507.00045v1#S1.F1),  [2](https://arxiv.org/html/2507.00045v1#S1.F2), [3](https://arxiv.org/html/2507.00045v1#S3.F3), and [4](https://arxiv.org/html/2507.00045v1#S3.F4), as shown in Figure [12](https://arxiv.org/html/2507.00045v1#A5.F12), [13](https://arxiv.org/html/2507.00045v1#A5.F13), [14](https://arxiv.org/html/2507.00045v1#A5.F14), [15](https://arxiv.org/html/2507.00045v1#A5.F15), [16](https://arxiv.org/html/2507.00045v1#A5.F16), [17](https://arxiv.org/html/2507.00045v1#A5.F17), [18](https://arxiv.org/html/2507.00045v1#A5.F18), [19](https://arxiv.org/html/2507.00045v1#A5.F19), and [20](https://arxiv.org/html/2507.00045v1#A5.F20). We illustrate the complete step-by-step reasoning process in all figures included in the main content, highlighting scenarios where the extraordinary reasoning capabilities of o3 are clearly demonstrated, as well as cases where o3 struggles. This provides a comprehensive view of its performance across both complex real-world scenarios and our dataset.

Figure: Figure 12: OpenAI o3 full reasoning process for Figure [1](https://arxiv.org/html/2507.00045v1#S1.F1)
Refer to caption: extracted/6565317/images/example_intro.jpeg

Figure: Figure 13: OpenAI o3 full reasoning process for Figure [2](https://arxiv.org/html/2507.00045v1#S1.F2) (a)
Refer to caption: extracted/6565317/images/example_snake.png

Figure: Figure 14: OpenAI o3 full reasoning process for Figure [2](https://arxiv.org/html/2507.00045v1#S1.F2) (b)
Refer to caption: extracted/6565317/images/example_waldo.jpeg

Figure: Figure 15: OpenAI o3 full reasoning process for Figure [2](https://arxiv.org/html/2507.00045v1#S1.F2) (c)
Refer to caption: extracted/6565317/images/example_geoguesser.jpeg

Figure: Figure 16: OpenAI o3 full reasoning process for Figure [2](https://arxiv.org/html/2507.00045v1#S1.F2) (d)
Refer to caption: extracted/6565317/images/example_timeguess.png

Figure: Figure 17: OpenAI o3 full reasoning process for Figure [3](https://arxiv.org/html/2507.00045v1#S3.F3)
Refer to caption: extracted/6565317/images/example_fig3.jpeg

Figure: Figure 18: OpenAI o3 full reasoning process for Figure [4](https://arxiv.org/html/2507.00045v1#S3.F4) (a)
Refer to caption: extracted/6565317/images/example_fig4_1.jpeg

Figure: Figure 19: OpenAI o3 full reasoning process for Figure [4](https://arxiv.org/html/2507.00045v1#S3.F4) (b)
Refer to caption: extracted/6565317/images/example_fig4_2.jpeg

Figure: Figure 20: OpenAI o3 full reasoning process for Figure [4](https://arxiv.org/html/2507.00045v1#S3.F4) (c)
Refer to caption: extracted/6565317/images/example_fig4_3.jpeg

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Can you expand the Real Subset to include more scenes (e.g., offices, public transit) to address current scenes' biases?
* Did you measure difficulty (e.g., inter-model accuracy variance, human annotation time) across dimensions/sub-fields? Are there imbalances (e.g., over-representation of easy Daily Life questions) that might skew model rankings?
* Have you fine-tuned MLLMs on the CaughtCheating dataset (especially the Real Subset) to test if it improves their detective capabilities?
* Advanced models like GPT-o3 hallucinate more on unclued images—have you analyzed their reasoning traces to identify root causes? 

I look forward to an active discussion with the authors during the rebuttal phase and will revise my score accordingly.
soundness: 2
strengths: * The paper proposes a genuinely novel and highly challenging task that pushes MLLMs beyond standard VQA or reasoning benchmarks. 

- The benchmark is thoughtfully constructed with two complementary subsets: a small, extremely difficult Real Subset that captures real-world subtlety and a large-scale Synthetic Subset for diversity. The annotation scheme is very thorough, distinguishing between "Deterministic" and "Non-deterministic" clues, and including "Decomposed" (Perception vs. Reasoning) questions, which enables a fine-grained failure analysis.

- The paper provides crucial insights into SOTA model failures. The key finding is twofold: models are not only bad at finding the *correct* clue (e.g., 26.0% Clued Acc for o3) but are *also* prone to hallucinating *incorrect* clues on innocent images (e.g., 8.0% Unclued Acc for o3). The decomposition analysis (Table 2) brilliantly supports the theoretical claims by showing that models *can* perceive the clues if prompted (high Dec. P Acc) but fail in the *search and discovery* process.
summary: This paper addresses the gap in evaluating detective-level visual perception and social reasoning in MLLMs by proposing **CaughtCheating**, a benchmark inspired by real-world social media requests to detect suspicious clues contradicting a claim. The benchmark includes a Real Subset (100 manually curated social media images, split into Clued/Unclued) and a Synthetic Subset (3700 GPT-Image-1 generated images) to balance realism and scalability. Grounded in cognitive science’s Guided Search theory, the paper explains MLLM failures: clues have low bottom-up salience, lack top-down feature guidance, and require social context interpretation—rendering MLLMs’ exhaustive search ineffective. Extensive experiments on 17 MLLMs (open/closed-source) show top models like GPT-o3 achieve only 26.0% Clued Accuracy, 17.2% IoU, 8.0% Unclued Accuracy, and 23.9% F1 on the Real Subset, exposing limitations like hallucination in unclued images. Contributions include the benchmark, theoretical failure analysis, and insights into MLLMs’ detective capability boundaries.
weaknesses: * The Real Subset (100 images) is small, dominated by hotel scenes (69%), and biased toward cisgender heterosexual couples—failing to represent diverse scenarios (e.g., workplaces) or relationships, which limits conclusions about MLLMs’ performance across real-world contexts.
* Clues in the Synthetic Subset are overly prominent (e.g., obvious lipstick, two glasses) due to prompt design, probably making it less effective at mimicking the subtlety of real-world clues (e.g., faint reflections) and reducing its value for training/testing MLLMs on authentic challenges.
* The authors rightly acknowledge the ethical limitations in Section 7. The "Real Subset" is sourced from public data that overwhelmingly represents cisgender, heterosexual couples. While the synthetic data attempts to balance gender, the entire benchmark is framed around a single, socially charged, and potentially harmful application (detecting infidelity). This narrow, biased framing limits the generalizability of the "detective-level" reasoning claims.
* The data and code not be publicly released.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1.	See above.
2.	Can you report breakdowns by scene, gender presentation, and cue types to identify biased failure modes?
soundness: 3
strengths: 1.	Interesting, socially relevant task that focuses on subtle visual perception and context-sensitive reasoning on image-claim pairs.
2.	The annotation schema distinguishes deterministic and non-deterministic clues in the detective task and decomposes visual reasoning questions, aiming to evaluate the performance of MLLMs.
summary: The paper introduces CaughtCheating, a benchmark for detective visual perception with annotated clues and decomposed questions. It aims to find subtle, context-dependent clues in images that contradict the given claim. Using this benchmark, state-of-the-art models perform poorly, suggesting a gap between current MLLMs and detection capabilities.
weaknesses: 1.	The scale of the constructed real dataset is quite small, with only 100 samples. A small real dataset raises concerns about the reliability of the benchmarking results and also weakens the contributions of benchmarking.
2.	The paper primarily constructs a benchmark and reports the limitations of current MLLMs. It does not discuss the potential direction or insightful methods to address the identified failures. 
3.	Synthetic subset generation relies on stereotypes and templated clues. The paper acknowledges that it is easier, which can distort model comparisons and may encode social biases.
4.	Results on real and synthetic datasets are significantly different, raising concerns about whether the proposed dataset can benchmark current methods and whether it can truly improve the performance of current methods.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - How were ambiguous or borderline “suspicious clues” handled during annotation, and what measures ensured consistency across annotators?
- Could the task design unintentionally encourage models to hallucinate suspicious clues due to its “detective-style” framing?
- Did you study whether models improve with more explicit top-down instructions (e.g., “look for reflections”)?
soundness: 2
strengths: - The dataset captures subtle, real-world visual clues that existing benchmarks overlook, providing a meaningful test of advanced multimodal reasoning.
- The analysis identifies why state-of-the-art models fail, linking errors to low-salience signals and lacking top-down guidance, offering actionable insights for future model design.
- The benchmark includes both real and synthetic subsets, plus automatic scoring procedures, enabling rigorous and scalable assessment of MLLM capabilities.
summary: The paper introduces CaughtCheating, a new benchmark designed to test whether advanced multi-modal LLMs possess detective-level visual perception and reasoning skills. Motivated by real social-media scenarios where users ask others to find subtle clues contradicting a partner’s claim, the benchmark focuses on identifying extremely inconspicuous, context-dependent visual evidence, such as reflections, small objects, or social inconsistencies, that current models often overlook. The authors analyze why even state-of-the-art models like GPT-o3 fail on these tasks, drawing on Guided Search theory to show that the clues have low bottom-up salience, lack top-down guidance, and require nuanced social interpretation. They construct both a carefully curated real subset and a diverse synthetic subset, and provide automatic evaluation procedures. Experiments reveal stark limitations: even the strongest models achieve only modest accuracy on real-world cases, frequently hallucinating suspicious clues or missing decisive ones.
weaknesses: - The benchmark focuses on a specific “detective-style clue finding” scenario, which may be partially included in other work and limits its generalizability to other multimodal reasoning tasks.
- Some clues in real-world images may be open to interpretation, making the correctness labels somewhat subjective.
- The prompt design is too simple, which cannot cover enough requirements in the real world for fine-grained perception and reasoning.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: Weakness mainly reflects my concerns.

some typos:
1. Line 037 : unnecessary line breaks
2. Line 091-092: `` detection-level'' should be ``detective-level'' as mentioned before.
soundness: 2
strengths: 1. The paper introduces CaughtCheating, a creative and practically relevant benchmark that targets subtle, context-dependent visual reasoning—an underexplored yet important aspect of real-world perception tasks.

2.The study systematically evaluates advanced MLLMs’ visual reasoning processes, revealing specific scenarios where even top-tier models like GPT-o3 fail, thus providing valuable diagnostic insight into current model limitations.

3. By connecting model failures to Guided Search theory, the paper offers an interpretable explanation of why MLLMs struggle, contributing both conceptual depth and guidance for future benchmark and model design.
summary: The paper introduces CaughtCheating, a new benchmark designed to challenge advanced multi-modal large language models (MLLMs) like GPT-o3, which have great performance on existing benchmarks. Inspired by real-world “detective” scenarios from social media, where users analyze photos for subtle clues of suspicious behavior—the benchmark tests models’ ability to integrate fine-grained visual cues with contextual reasoning. Through experiments, the authors find that while GPT-o3 performs well on many hard tasks, it fails almost completely on CaughtCheating scenarios. The study highlights current MLLMs’ limitations in nuanced perceptual reasoning and argues that success on such tasks would mark a step toward human-level detective-like understanding.
weaknesses: 1. Section 2.1 provides an informative overview of how modern MLLMs perform visual reasoning. However, the section is unnecessarily verbose (particularly lines 148–160) and contains excessive descriptions of specific model responses that distract from the main narrative. It would be more effective to distill these observations into general patterns or response tendencies across mainstream MLLMs, in line with the section title “Exploring the Boundary of Visual Perception and Reasoning.” Moreover, focusing on a single closed-source model limits the robustness and generality of the analysis.

2. The examples used to demonstrate task difficulty and model behavior are not entirely convincing. For instance, in Figure 4(a), the supposed clue (a finger) appears nearly impossible for a human to detect, raising concerns about the fairness or clarity of the ground truth. Meanwhile, GPT’s response to Figure 4(b) seems reasonable, yet it is judged as incorrect, suggesting potential ambiguity or inconsistency in the labeling criteria.

3. The dataset’s narrow focus on “catching cheating” scenarios undermines its generalizability. While the paper claims to assess general visual reasoning abilities, the data distribution is highly specific and socially contextualized. As such, conclusions such as “failing to identify deterministic clues” are not well supported by a sufficiently diverse or representative sample of reasoning tasks.