VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?
Authors: Minkyu Kim
Sangheon Lee
Dongmin Park
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-language Models
Multimodal Large Language Models
Comparative Reasoning
Benchmark
Visual Question Answering
Primary area: datasets and benchmarks

Abstract: The ability to distinguish subtle differences between visually similar images is essential for diverse domains such as industrial anomaly detection, medical imaging, and aerial surveillance. While comparative reasoning benchmarks for vision-language models (VLMs) have recently emerged, they primarily focus on images with large, salient differences and fail to capture the nuanced reasoning required for real-world applications. In this work, we introduce **VLM-SubtleBench**, a benchmark designed to evaluate VLMs on *subtle comparative reasoning*. Our benchmark covers ten difference types—Attribute, State, Emotion, Temporal, Spatial, Existence, Quantity, Quality, Viewpoint, and Action—and curate paired question–image sets reflecting these fine-grained variations. Unlike prior benchmarks restricted to natural image datasets, our benchmark spans diverse domains, including industrial, aerial, and medical imagery. Through extensive evaluation of both proprietary and open-source VLMs, we reveal systematic gaps between model and human performance across difference types and domains, and provide controlled analyses highlighting where VLMs’ reasoning sharply deteriorates. Together, our benchmark and findings establish a foundation for advancing VLMs toward human-level comparative reasoning.

Published as a conference paper at ICLR 2026
VLM-SUBTLE How FAR ARE VLMs FROM HUMAN-LEVEL SUBTLE COMPARATIVE REASONING?
Minkyu Kim Sangheon 12 Dongmin Park'
KRAFTON, "KAIST
ABSTRACT
The ability to distinguish subtle differences between visually similar images is essential for diverse domains such as industrial anomaly detection, medical imag- ing, and aerial surveillance. While comparative reasoning benchmarks for vision- language models (VIMs) have recently emerged, they primarily focus on images with large, salient differences and fail to capture the nuanced reasoning required for real-world applications. In this work, we introduce VLM--SbbleBench [. benchmark designed to evaluate VLMs on subtle comparative reasoning. Our benchmark covers ten difference types Sypes-Attribtte, State, Emotion, Temporal, Spatial, Existence, Quantity, Quality, Viewpoint, and Action-and curate paired question-image sets reflecting these fine-grained variations. Unlike prior bench- marks restricted natural image datasets, our benchmark spans diverse domains. including industrial, aerial, and medical imagery. Through extensive evaluation of both proprictary and open-source VLMs, we reveal systematic gaps between model and human performance across difference types and domains, and provide controlled analyses highlighting where VLMs' reasoning sharply deteriorates. To- gether, our benchmark and findings establish foundation for advancing VLMs toward human-level comparative reasoning.
INTRODUCTION
The ability to discern subtle visual differences, ie., minor discrepancies between otherwise similar objects, scenes, situations, is cen- tral to human cognition. enables us to per- form wide range of fine-grained compara- tive tasks, such as recognizing micro-expression changes in daily life (Li al.) 20022), detecting anomalies in manufacturing (Bergmann et al. 2019). assessing subtle variations satellite imagery (Asokan & Anithal (019), and dis- tinguishing disease slages medical imag- ing (Litjens et al.) 2017). Beyond everyday life, the capacity to detect and reason about such subtle differences has expanded human abilities toward higher forms intelligence, underpin- ning advances from scientific discovery com- plex social organization.
VLM-Sahcillecch
ccuurayr
MLL.M-CompBench
VL. BuSble
Typo
Enaks
To Traped
Spent
Actinn
Random Guessing
02
34
DINOv3 Similarity
Figure 1: Comparison of VLM-SSbbleeBench and
MLLM-CompBench with GPT-4o.
Recently, vasion-language models (VLMs) have shown remarkable progress toward artificial general intelligence (AGI), demonstrating promising results in various tasks, such as visual question answer- ing (VQA) and scene description (Zhang et al.) (2024). Yet, most progress has primarily centered on single visual inputs, e.g., an image or a video, while comparative tasks that require comparison over
Equal contribution.
Work done during an internship at KRAFTON.
'Ou dataset and code available at /twpaiiwwwtit/////oigooo/aaaaa/a
ubt eeeenh
com http:///wwoo..accccoo//o/aaaaaaa

---

Published as a conference paper at ICLR 2026
Attribute
Metual
State
Inteery
Emotion
Noreal
Temporal
Corns
Q: Which image shows detter ling aread?
Q: Which image shoue legur
suizr insalation?
Q: la which image cince the panun appeer los inury?
Q
Which from: hoppenad fire?
Ceceeeeeee imagee
Secoal innage
inagge Coneeeeeeee
inege
imnge
image
Quality
Whet differone betwan thow imaga? Blue
imsgged?
imagee
uncrooppourc artifactaf?
innage
B
Quantity
Actil
Viewpoint
Natural
Action
Liarts
O: wlich disction did the cumera roll fuo Eat inage the sccond
Plans Imsage Secord imrage which imagee
builling?
image!
Clackobbe
Figure 2: Example tasks from the VLM-SubtleBench, covering ten difference categories (Attribute, State, Emotion, Temporal, Spatial, Existence, Quality, Quantity, Viewpoint, Action) and six do- mains (natural, game, medical, industry, aerial, synthetic). For each example, the correct answer is highlighted in bold green. Model responses from GPT-5-main, Claude-sonnet are shown beneath each question in order. Some VQA instances are simplified due to space constraints; full versions and additional examples are provided in the appendix.
and Gemini-2.5.
multiple inputs, e.g., two images, have received relatively little attention. However, narrowing this gap is increasingly important, as recent applications often deploy VLMs as agents to perform com- plex tasks involving comparative reasoning across multiple observations, e-g-, self-reflection over previously observed scenes (Hong et al. 2024). To truly serve stuaan-leved surrogates, advancing VLMs' capacity for advanced comparative reasoning is becoming indispensable.
A few benchmarks have been presented to evaluate the comparative reasoning capabilities of VLMs. However, as shown in Figure [ prior benchmarks focus on relatively simple comparisons between fairly dissimilar scenes, e.g., identifying differences in salient features (states) of distinet objects (two lemons), as indicated by the low average subtlety scores of MLLM-CompBened measured by embedding similarity with DINOv3 (Siméoni et al.) 2025). ). As result, they are easily solved by state-of-the-att VLMs, such as GPT-4o. In addition, most prior benchmarks are composed of natural images, and thus fail to assess performance in specialized domains such as industry medical (See T.. Therefore, this calls for new benchmark that can evaluate human-level subtle comparative reasoning across diverse domains and high-diffiullty tasks.
To this end, we present VLM- SuMM-,, a benchmark designed to evaluate human-level com- parative reasoning capabilities of VLMs. As illustrated Figure 2 "LM-SubbleBond comprises 13K triplets of image pairs, questions, and answers, covering 10 representative difference types, i.e., Aurihute, State, Emotion, Temporal, Spatial, Existence, Quality, Quantity, Viewpoint, and Ac- tion. collected from diverse image domains, i.c., Natural, Game, Industry, Aerial, Synthetic, and Medical. Even advanced proprictary VLMs such as GPT-5 and Cemini-2.55prr exhibit significant performance gaps compared to humans, especially difference types that demand spatial, tempo- ral, and viewpoint reasoning.
We conduct systematic studies the performance of open-source and proprietary VLMs, the ef- fectiveness of test-time prompting strategies on comparative tasks, and controlled experiments with synthetic image pairs, which reveal the extent which VLMSs align with human performance and
2

---

Published as a conference paper at ICLR 2026
Table I: Summary of Comparative Reasoning Benchmarks. 'Is Subtle?' indicates whether the benchmark contains image pairs whose DINOv3 similarity averages at least 0.8, meaning the differences are subtle. VLM BuM-uhh is the only benchmark that focuses on subtle comparison, spans diverse domains, and includes both multiple-choice questions and captioning tasks.
Benchmarks
Is Subtle?
# Domains
# Diff. Types
MCQ
Captioning
Birds-to- -o-tddd-WWws (Forbes et al. 2019)
X
Spot-the-Diff Jhamtam et al.. 2018)
/
X
MLLM-CompBencll (Kil et al.
2024
V
x
VLM-SubtlcBench (Ours)
6
10
where their reasoning sharply deteriorates. Our findings show that (1) proprietary VLMs still leave large gaps from human performance, particularly on spatial, temporal, and viewpoint reasoning where even the best model trails humans by over 30 percentage points; (2) simple prompting strate- gies, such as chain-of-thought prompting. grid layouts, and overlapping images, yield only limited improvements; and (3) VLMs are highly sensitive to difficulty factors such object size and count. Together, ILM--SubleBenchh and our experimental studies provide critical insights toward narrow- ing the gap between VLMs and humans in subtle comparative reasoning.
RELATED WORK
Multi-Image Benchmarks for VLMs. While most VLM benchmarks focus on single-image un- derstanding (Liu et al. 2 Ying et al. 2024 Wu et al. 2024), growing body of work evaluates multi-image reasoning. Early efforts such as NLVR2 (Suhr et al. 2019) and Winoground (Thrush et al. 2022) tested whether models can reason over image pairs through sentence verification and compositional matching, respectively. More recently, BLINK (Fu et al. 2024) evaluates core visual perception tasks such as relative depth and visual correspondence, exposing large human-model gaps on low-level visual skills. REMI (Kazemi et al. 2024) focuses on high-level relational reason- ing, such analogy and sequential pattern completion, across multiple images. MuirBench (Wang et al. 2025) provides a comprehensive suite 12 multi-image task types, ranging from visual retrieval to scene understanding. While these benchmarks collectively advance multi-image evalu- ation, none are specifically designed to evaluate comparative reasoning between image pairs, iden- tifying what has changed between two similar images. CeLL-hhomBench (Kil al., 2024 the closest to our work, as it directly evaluates comparative judgments across eight difference types. Yet, its image pairs typically depict different subjects or settings with salient differences, as evidenced by their low average similarity scores (Figure []. In contrast, differences within nearly identical contexts, requiring genuine cross-image comparison (Table [T..
focuses on subtle
Difference Understanding across Domains. Detecting and describing visual differences has been studied across specialized domains. Spot-the-Diff (Thamtani & Berg- Kinn-KKirkattrick (018)) introduced textual descriptions of changes between surveillance frames, CLEVR-Change (Park al. 2019) provided a synthetic testbed for change captioning, and Birds to-ds-to-Woo Words (Forbes et al. 2019) col- lected fineggriieed comparative descriptions of bird photographs. In the medical domain, MIMIC Diff-VQA (Johnson et al. 2019) supports difference-focuue
VQA on chest X-ray pairs, while GeoBench (Lacoste et al. (023)) covers remote sensing tasks including change detection for Earth monitoring. While these works demonstrate the importance of difference understanding, each is confined to single domain and a narrow set of difference types, leaving no unified benchmark that evaluates VLMs' comparative reasoning across diverse visual distributions.
Image Difference Captioning. A related line of work focuses on generating textual descriptions of image differences. Img-Diff (Tiao et al. 2025) constructs contrastive image pairs targeting object- level changes such as replacement and removal, OneDiff (Hu et al., (025) aggregates multiple sources and trains a generalist model covering color, texture, and spatial changes, and DiffTell (Di) et al. 2025) curates a high-quality dataset that further encompasses style and text manipulation. While these efforts advance difference captioning with larger-scale data and dedicated models, they primarily focus natural images and the captioning task alone. Our benchmark complements this line of work by providing both VQA and captioning tasks across ten difference types and six do- mains, enabling more comprehensive evaluation of comparative reasoning.
3

---

Published as a conference paper at ICLR 2026
3 VLM-SUBBLEBENCH
In this section, we introduce VLM--ubbtlBBnchh a benchmark designed to evaluate subtle compar- ative reasoning capabilities of VLMs. VLMM-SubleBench focuses on whether models can reliably identify subtle differences between two highly similar images, a key aspect of comparative visual reasoning. In the following, Section 3.T describes the scope of the benchmark, covering the vi- sual domains and difference categories included. Section 3.2 details the dataset curation process, and Section 3.3 explains how caption annotations were performed to ensure high-quality textual de- scriptions. Section 3.4 presents dataset statistics that highlight the diversity of Benhhh.
3.1 SCOPE OF VIM-SUBTLEBBNCHH
Covered Image Domains. To evaluate whether model possesses human-level subtle compara- tive reasoning across diverse, cognitively demanding tasks, it is essential to cover images from a wide range of domains. Thus, we design VLM SLMM-SubleBenchh
span six representative image do-
mains: Natural Scenes, capturing everyday real- world photographs (Abu-El-Haijj et al.] [2016] Lin
et al. 2014 Souček et al., 2022 Cao et al. 2014 Livingstone & Russo 2018 Kossaifi al. 2017
Gupta et al. 2016 Zhou et al. 2025; Wang et al. 2024 Idrees et al. 201 8 Lin et al. 2025 Game
Environments, simulated yet realistic scenes that test generalization beyond natural images (Abu-
El-Haija et al. 2016 Lin et al. 2025); Aerial Imagery, covering remote sensing and overhead
Views where subtle spatial differences are critical (Liu al.) 2024a] Huang al.] 2022); Indus-
trial Inspection, representing structured settings where -eniiie defects or anomalies need to
be detected (Bergmann et al.) 2019 2022]: Medical Imaging, where diagnostic reasoning often
requires distinguishing subtle changes across visits (Johnson et al.] 2019: Hu et al. [2023); and Syn-
2019,
thetic Primitives, consisting of abstract 2D shapes with varying colors and arrangements on plain backgrounds, which further allows controlled analysis.
Covered Difference Types. We also design
to cover diverse types of differ- ences. Specifically, we follow the categorization proposed in Kil et al. (2024), while extending by adding two new types of differences, Viewpoint and Action. In total, VLM-SSbuleBench encom- passes ten difference types: Attribute captures variations in object properties such as color, size, or shape; State reflects the condition of an object, such whether an apple is peeled; Emotion addresses comparative judgments of facial expressions; Temporal involves identifying which im- age corresponds to later stage of an event; Spatial describes changes in arrangement relative position; Existence refers to whether an object is missing; Quantity handles whether the number of objects differs across images; Quality captures degradations such as blur, noise, or overexposure; Viewpoint reflects changes in camera perspective; and Action denotes differences in human or ani- mal poses or activities. Together, these ten categories establish a comprehensive taxonomy of subtle differences, spanning from low-level visual variations to high-level semantic changes.
3.2 DATASET CURATION
For each difference category, we curate paired images from diverse sources and generate compara- tive mnestion-answer pairs through a mix of rule-based, annotation-driven, and model-assisted meth- ods. Existing datasets with rich ground-trute -tuth labels and metadata enable systematic pairing and QA construction, while synthetic edits and primitives provide controlled settings for specific attributes. Below we briefly summarize the curation strategy for each category. Refer to appendix A for details.
Attribute. We use four sources: MVTEC-AD (Bergmann et al., 2019) for industrial inspection, COCO et 2014) for natural images, MIMIC- -Dff- VQA (Hu et al. 2023) for medical domain, and synthetic primitives. In MVTEC-AD, pairs are formed by selecting anomalies of the same type but with different severity, using pixel-level defect annotations to order defect size. In COCO, we first identify images containing single object using instance segmentation annotations, then use GPT-40 to determine the object's color and suggest a similar but distinct alternative color. We then 2.t--i----lllss-iaaeeeee
apply the image editing model change the object's 's color accordingly, producing minimally different pairs. For the medical domain, leverage LIMIC-Diff- VQA, which provides chest X-ray pairs from different visits of the same patient. We reformulate the original clinical questions into faypellon-frienll forms using GPT-4o (e.g., "lung opacity progression" "Which image shows whiter lungs?"). For synthetic data,
(also known as "nano-banana") to
4

---

Published as a conference paper at ICLR 2026
Image Pair Selection
VQA Annotation
Caption Annotation
GJ Coplar Sianleenne www.enincc
OR
we senallat Svored Ieags
Elicke Thange VTM
maller
Alpxblia Scletio
Thans Palkiy
1130 Q Coceriii
laad Gouraiin
Nabiin sutirr
(e) Source with Ground Truch Aunotation
ciplas
Quality Assurance
kuslin unird laage
relencc higher
Sooeed
ClpyCrr Randic Trame Soletiic
Hamear Araaaiee
Ruda Qu gouctaicn
Romane de lrac.
(b) Source withou Ground Truth Ansottiion
Scllig Frauu
pin Chail
Conaae "Numhe incnasod Cocd gouctezz euliplociaii cannloc
Wilch
Saku comert Filenur? A Yotko cince buoetee snclee lerge Ciay boonse analler Only siar becerme lengn
Fell lnegg
2. Peerhoene
12 Seoadd
Irwgelddong
LL3S4bud QJ Geoveatio
Gemeation
(4)
Figure 3: Data Curation Pipeline of BLM-SubtleBench.
render primitive scenes and apply subtle brightness size modifications single object. QAs then ask which image has a larger defect, which color is stronger, which shape is bigger, or what change occurred.
State. From MVTEC-AD, we focus object breakage cracks. Akin to attribute, image pairs are constructed by sampling images with different levels of damage, with annotations guiding the relative severity. The resulting QAs ask which crack is larger which state shows more breakage. We also include natural domain pairs from Changelt (Souček et al., (2022), where human annotators manually annotated the stode-modi action together with the object states in a set of internet videos. From each video, we sample multiple frames based on the provided state information, and human annotators then select frame pairs that capture the object before and after the state- modifyinn action. The QA pairs then ask which image reflects a greater degree of state modification. For example, if the object is an apple and the action peeling, the QA would ask which image is the apple peeled more?".
Emotion. We draw images from emotional video datasets CREMA-D, RAVDESS, AFEW-VA,
and DAiSEE (Cao al., 2014, Livingstone Russo 2018 Kossaifi al. 2017 Gupta et al., 2016) all of which provide clip-level emotion annotations. From these clips, we randomly sample frames and construct paired examples based on the relative intensity of expressed emotion. The QA pairs ask which image conveys stronger or weaker emotion, based on the annotations.
Temporal. From YT8M (Abu-EI-Haija et al. 2016) and VLM4D (Zhou et al., 2025) video datasets, we randomly select two frames from the same clip. Their temporal order is determined by timestamps, and QA pairs ask which image depicts the earlier event. This task requires models to capture temporal progression rather than static differences, sometimes relying on common-sena knowledge (c.g., boat cutting through water can only move forward, not backward).
Spatial. We use VLM4D, which provides 4D annotations of the object's translational and rota- tional motions in video. From each video, we uniformly sample multiple frames. Human annotators then select frame pairs that visually align with the ground-truth motion annotations. Using these pairs and their associated motion data, we generate QA that ask the spatial changes resulting from the motion. The model is required identify which transformation the object has undergone.
Existence.
For acrial imagery, LEVIR-MCL (Liu al. 2024a) provides image pairs of the same location across years, along with segmentation maps capturing object-level appearance or disap- pearance and human-labeled difference captions. We also construct synthetic settings by rendering
5

---

Published as a conference paper at ICLR 2026
Domain Statistics
Difference Type Statistics
1,580
Game
1,444
1,370
Industry (9,556)
1,120
1,007
1,007
Natural (53.8%)
Acrial (4.9%)
-55
236
R00
5B6
600
182
Syndhetic (24.8%)
214
906
916
465
Medical (2.8%)
Aunbute
State
Emotion
Temporal
Sputia
Existance
Quantity
Quality
Viowpoint
Action
Figure 4: Statistics of the test split of VLM-SubtleBench
primitives and removing one object to create a pair. Additionally, COCO is used to remove objects from natural images, similar attribute. For synthetic data, render primitive scenes containing multiple types of shapes and colors, and then add or remove one instance. In all cases, QAs ask what has disappeared or appeared between the two images, grounded in annotations edit prompts.
Quantity. We combine multiple domains to capture differences in object counts. MVTEC- LOCO (Bergmann et al., (2022) provides annotated anomaly and normal images, enabling com- parisons object multiplicity. UCF-QNRF (Idrees al.) 2018) offers crowded street scenes with dense human annotations, from which we sample pairs with count differences. Aerial imagery datasets (LEVIR-ME UBC (Huang et al. 2022)) allow comparisons of building counts. Synthetic datasets and Gemini-edine edited images (MegaFruit (Wang et al. 2024)) further introduce controlled ob- ject additions. QAs consistently ask which image contains more objects, people, or buildings. For synthetic data, render primitive scenes containing single type of shape and color, and then create quantity differences by adding or removing one instance the shape. The QA pairs then ask which image contains more (or fewer) shapes.
Quality, From YT8M, we extract ten random frames per video and present them human anno- tators, who select the frame with the best visual quality and the one with the worst, based on the presence of blur, noise, overexposure, or compression artifacts. The selected pair forms quality comparison sample. Based on these annotations, we construct binary QA pairs that ask which image has better or worse quality.
Viewpoint. CameraBench (Lin et al. 2025) provides camera-centric and object-centric annota- tions describing translations and rotations. From each video, uniformly sample frames, and human annotators then select pairs of frames that are visually aligned with ground-truth camera annotations (e.g., sufficient visual cues for viewpoint change). Based these pairs, we construct binary QA tasks: for example, if the camera rotated to the right, the question asks "In which direction did the camera move?" with answer choices right left. Similarly, for object-centric annotations. QAs ask whether the camera orbited an object clockwise or counterclockwise.
In case of synthetic data, we render primitive scenes and simulate camera motion by translating the objects or by rotating them clockwise or counterclockwise around the center. From the six possible transformations, we randomly sample four options form multiple-choice QAs, where the model must identify the ground-truth motion.
Action. From YT8M, we sample pairs of frames that capture changes in human object actions and interactions. For each pair, human annotators identify salient entity whose action differs across the two frames and provide (i) the entity description and (ii) short free-form action phrase for each frame. We then generate QA pairs using the template "In which image the (item) faction]:?" and minimally refine grammar with GPT-4o.

---

Published as a conference paper at ICLR 2026
3.3 DIFFERENCE CAPTIONS
In practical scenarios across diverse VLM application domains, the ability to directly describe differ- ences between two images is crucial. To enable such forms of evaluation, we additionally collected human-written captions explicitly highlighting the differences between paired images, thereby com- plementing the comparative QA tasks. We sampled 1,200 random image pairs (10% of the test split) for human caption annotation. Annotators were instructed to identify at least one difference between the two images and write comparative caption using the annotation interface.
3.4 DATASET STATISTICS
Each difference category contains least 1K question-answer pairs, resulting in total of 13K pairs. Every difference type includes data from the natural domain, where foundation models are most commonly applied in practice. We split the dataset into a test set (11.7K) and validation set (1.3K). The test set is used for evaluation, while the validation set is used for fine-tuning models in our experiments. Figure presents the statistics of S.MM-SubteBBenh test set.
EXPERIMENT
4.1 EXPERIMENT SETUP
Models. We evaluate both open-source and proprietary vision-language models. For the open-
source side, we use the Qwen2.5-VL. (Bai et al. 2025) family at three scales (7B, 32B, and 72B), as
well as Ve and LLaVA-One Vision (both 7B). For proprietary models, include GPT- 4o (Achiam et al. 2023), o3, GPT-S-main, GPT-S-thinking, Claude-sonnet-4 (Anthropic) 2025), Gemini-2.5-flash (Team] 2025). and Gemini-2.5-pro. This set spans both non-reasoning models (e.g., GPT-4o, GPT-5-main) and orrieneng-oriented models (e.g., o3,
Prompting Strategies. To better understand the role of prompting, we experiment with several strategies. We adopt the standard Chain-of- Thain-of-Thought (CoT) approach, which encourages models to generate intermediate reasoning before producing final answers (Wei et al., 2022). We further introduce two-step reasoning setup in which the VLM generates responses first step, the VLM is prompted to describe the differences between the two images that are relevant the question; the second stage, the two images, the question, and the output from the first step are provided together to answer the question. We also augment images with grid and instruct the models to parse them sequentially along the horizontal axis (Izadi et al.] 2025). To investigate how models handle multiple images, we test different fusion techniques: (i) horizontally concatenating the two images into single composite input, (ii) creating an overlap image by averaging the pixel values of the two input images and using it together with the original images, (iii) generating grayscale subtraction image by computing the absolute pixel-wise difference, normalizing it by the maximum value to highlight regions of change, and providing along with the original images, and (iv) highlighting regions of interest by retaining pixels with large differences, clustering adjacent pixels, and drawing bounding boxes around at most three of the largest clusters emphasize the main regions of change. Further details of these prompting techniques are provided in Appendix(B.2
stages. In the
Evaluation Metric. We use task-appropriate metrics to evaluate model performance. For multiple-choice questions, performance is measured by accuracy, capturing the proportion of cor- rect answers. For the captioning task, we apply cosine similarity score (CSS) (Reimers & Gurevych, 2019) and ALLMass---judee (Zheng al. (0233 Lin et al. 2025), to assess the quality and relevance of generated captions. See Appendix B.3 for details on the CSS and LLM--as-a-udge evaluations.
4.2 BENCHMARK RESULTS
Multiple-Choice Questions. Table 2 summarizes the performance of proprietary and open-source VLMs VLMM-ubbleBennhh Among proprietary models, GPT-S-thinking achieves the strongest results overall, ranking first in out of the 10 difference types and yielding the highest average accuracy. Other reasoning- -oriented models such as o3 and Gemini also show strong performance, highlighting the advantage of models explicitly designed for reasoning. Within the open-source models, Dwen2.5-VVL-722 stands out with competitive accuracy, in some cases approaching that of
7

---

Published as a conference paper at ICLR 2026
Table Performance of open-source and proprictary vision- Saiiuune models in VLM-SubtleBench. Human evaluation was conducted on randomly selected 10% of the samples.
Model
AT
EM
TM
EX
QN
QL
AC
AVG
Random Guess
35.9
50.0
50.0
50.0
36.6
23.2
48.9
50.0
42.1
50.0
43.3
Human Eval
92.0
93.0
93.0
93.0
95.0
97.0
97.0
99.0
98.0
98.0
95.5
Open-sounce
37.0
51.3
51.8
47.4
37.3
25.6
49.5
48.0
43.7
46.9
43.6
LLaVA-One Vision 7B
41.6
56.8
73.9
48.7
35.5
44.2
54.9
62.7
49.1
60.5
52.0
Qwen2.5-
46.5
63.7
87.8
50.2
39.5
73.8
58.0
70.9
47.5
69.3
59.4
Owen2.5-VI-322B
48.3
64.0
85.3
50.4
43.6
84.2
67.5
72.5
47.4
72.0
62.2
Qwen2.5-VL-72B
53.9
68.9
85.9
49.9
47.8
81.7
67.7
78.4
56.2
74.1
65.4
Proprietary
GPT-4o
51,5
73.6
89.5
52.7
42.4
60.6
58.2
72.4
51.4
76.7
61.6
03
78.0
79.5
92.9
60.4
55.1
82.2
78.2
87.6
64.6
84.8
75.7
GPT-5-main
72.9
78.4
92.7
53.6
50.1
75.4
72.6
84.5
57.5
83.6
71.3
GPT-5-thinking
83.6
80.7
93.1
60.2
59.9
85.4
84.8
68.5
84.9
77.8
Claude-sonnet-4
48.9
64.7
83.3
49.3
48.7
87.5
70.8
53.5
66.3
62.6
Gemini-2.5- flaslah
49.3
72.5
88.4
53.9
40.7
73.2
60.0
77.1
51.8
72.3
62.5
2.5pp
55.3
76.4
89.8
57.6
44.8
79.9
68.0
84.8
60.3
76.8
68.2
proprietary systems. Among the 7B-scale models, Qwen2.5- achieved the highest accuracy, followed by LLaVA-OneVision. while LLaVA-NcXT-7B showed lowest performance.
Across different difference types, VLMs show strong performance emotion, with GPT-5-thinking achieving 93.1% accuracy. In contrast, all models perform weakly on temporal, spatial, and view- point differences, which require common-sense reasoning (e.g., predicting the future position of a person or distinguishing between object and camera motion) and spatial understanding. These findings underscore the need for VLMs to incorporate richer sentialltempoall representations.
Captioning. J the performance of VLMS VI.M-SubuleBench's captioning task. Similar to VQA tasks, GPT-5-thinking achieves the strongest overall performance across all metrics. However, when captions are evaluated with the LLM-as-a-judge metric, its accuracy reaches only 43.0%, leaving a noticeable gap compared to ground-trutt captions. Large open-source VLMs, such as Qwen2.5 Siven2.5-VL-33. VL-32B/72B, perform on par with proprietary models in terms of CSS score, but exhibit substantially lower performance under the LLM-as-a- j-juge evaluation. In particular, Qwen2.5-VV 5-LL- 72B scores 24.3, which significantly behind GPT-5-thinking" 43.0.
4.3 EFFECT OF PROMPTING
Table reports the effect different prompting strategies. Adding reasoning steps before the an- improved performance in 9 out of 10 domains. While such gains intuitive in tasks like temporal that require world knowledge, it particularly interesting that reasoning also boosts per- formance in tasks such atribbte and quality, where success hinges capturing fine-grained visual differences. This suggests that explicit textual reasoning supports not only abstract inference but also subtle perceptual discrimination, consistent with our main finding that models with stronger inherent reasoning achieve higher accuracy. In contrast, the two-step reasoning approach leads to slight decrease in performance. We observe that the model frequently produces intermediate de- scriptions indicating "no difference" in the first stage, which results in incorrect final predictions. The highlighting method yields a modest improvement performance. is particularly effective on datasets with limited variations (e.g., synthetic data); however, its performance declines on datasets exhibiting substantial variations in brightness image quality (e.g., YT8M), where bounding boxes often fail to accurately localize regions of change.
Other strategies generally led to performance drops. particular, concatenating two images into a single input, common setup in prior work (Kil et al. 2024 Jiao al.) 2025), degraded accuracy in 9 out of 10 domains. Overlap and subtract showed mixed effects: they yielded clear gains in spatial and existence tasks where only objects change under fixed views, and in viewpoint tasks where the scene is mostly static with camera movements. However, in other domains these strategies provided little benefit, reflecting their dependence highlighting layout differences between images.
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance of open-source and proprietary vision-language models VLM-SubtleBench captioning.
Attribute (Color)
Astribure (Size)
Spatial
20
Obrecn Siz 8
Model
CSS
LLM-Judge
Open-source
Qwen2.5-VL-7B
0.42
12.7
003
6.1
6.15
120
200
200
Brigtmess
Size Change (56
Trunslatio Viewpoint
Qwen2.5-VL-32B
0.52
22.7
Existence
Quantity
Qwen2.5-VL-72B
0.54
24.3
16
28
28
Proprietary
Obiec Site
GPT-4o
0.54
25.2
8
54
40
35
12
37
03
0.56
38.1
GPT-5-main
0.57
37.2
GPT-5-thinking
0.57
43.0
16
25
Claude-sonnet-4
0.54
24.8
+OOjjetss
A Objjcts
Gemini-2.5-faase
0.50
25.9
Figure 5: Performance heatmaps of GPT-4o on synthetic data under controlled difficulty factors.
Gemini-22 proo
0.52
29.4
Table 4: Effect of prompting strategies and fine-tuning in VIMM-SubdeBench.
Model
AT
ST
EM
TM
SP
EX
QN
QL
VP
AC
AVG
Random Guess
35.9
50.0
50.0
50.0
36.6
23.2
48.9
50.0
42.1
50.0
43.3
Prompting Strategies
GPT-5-main
72.9
78.4
92.7
53.6
50.1
75.4
72.6
84.5
57.5
83.6
71.3
Reasoning
76.5
79.1
91.2
56.1
51.6
80.2
75.8
86.1
57.0
85.4
73.1
Two-Step Reasoning
70.8
79.1
93.4
56.9
47.4
81.3
66.4
83.5
58.0
83.6
71.0
Grid
71.6
77.5
89.1
52.8
51.0
75.8
72.5
82.6
$7.2
84.2
70.6
+ Concat
70.3
77.9
92.2
51.6
44.6
75.5
64.8
81.2
52.3
82.4
68.2
Overlap
69.5
76.7
91.8
52.4
53.6
76.1
69.1
79.0
58.8
83.0
70.2
Subtract
73.8
76.4
91.8
50,9
55.4
78.0
69.8
80.1
60.0
82.0
71.2
Highlight
71.1
75.2
92.0
51.1
54.9
86.5
74.3
77.8
57.3
82.9
71.5
TanniTnnng
Qwen-2.55VLL-7
46.5
63.7
87.8
50.2
39.5
73.8
58.0
70.9
47.5
69.3
59.4
Fine-tuned
62.0
69.1
92.2
52.5
47.0
85.3
77.0
85.9
57.5
75.4
69.5
4.4 CONTROLLED EVALUATION WITH SYNTHETIC DATA
Setup. We leverage synthetic data generation to systematically manipulate task difficulty, which allows precise control over the factors that may cause VLMs to fail. For each difference type, selected two primary factors that strongly influence difficulty and varied them along a controlled axis. Specifically, for attribute, we considered the size of changed objects and the magnitude of variation (brightness shifts in [0, 1] for color, or size-change ratios for scale). For spatial, we manipulated object size and the degree of translation. For existence and quantity, the two were object size and scene complexity, defined as the total number of objects. For viewpoint, we varied camera translation and scene complexity. For each configuration, we generated 100 paired images probe VL.M performance. Evaluation was performed using GPT-4o. Notably, for quantity differences, random guessing achieves 50%, while for the other categories the baseline is 25%.
Results. Our experiments reveal distinct failure modes across difference types. For attribute (color), the decisive factor is the magnitude of brightness change: the model requires shifts of roughly 25% to show strong performance above 70%, while smaller changes, e.g., 5%, lead to mine performance. In the attribute (size) condition, the model depends more on the absolute size of the changed object than on relative scaling, achieving reliable accuracy only when large ob- jects undergo substantial transformations. For spatial differences, accuracy is largely influenced by both translation and object size, with the model responding more strongly to relative displacement than absolute pixel shifts; notably, smaller objects moving larger relative distances are easier to de- tect. In the existence setting, scene complexity emerges as the dominant factor: accuracy is nearly perfect with four or fewer objects but rapidly degrades to below 60% once scenes exceed 32 objects, though larger disappearing objects remain casier to track. Similarly, in quantity, performance re- mains high, nearly 80% in simple scenes with objects, but drops to close 60% in complex scene

---

Published as a conference paper at ICLR 2026
Table Rank correlation of our benchmark and MLLM-CompBench with MMAD and QAG.
Table 6: Downstream performance after fine- tuning each benchmark.
MMAD
QAG
MMAD
QAG
VLM-SubtleBench 0.8424
0.7212
Swen2.5-VL-777
65.0
34.4
MLLM- MLLMM-ComBeench
0.8110
0.7195
VLMMSubuleBench
69.6
35.5
+ MLLM-CompBench
66.3
32.2
containing ten or more objects, approaching the 50% random baseline. Finally, for viewpoint, the model shows an interesting opposite trend: performance improves as scene complexity increases, benefiting from richer visual cues, and stable accuracy requires camera translations of around 160 pixels (which 27% of the image height).
4.5 EFFECT OF FINE-TUNING
To evaluate whether additional supervision can mitigate the comparative reasoning challenge, we fine-tune Qwen2.5-VL--77 using the validation set. Table Dolleppreee presents the results. Fine-tuning yields consistent performance improvements across all difference types. with particularly notable gains in existence, quantity, and quality categories. In contrast, the spatial and temporal dimensions showed more modest gains, suggesting that richer spatial-temporal reasoning rather than in-distribution adaptation may be required for further progress. Despite these improvements, a substantial gap remains compared to GiPT-5-thinking and human performance, indicating that broader data diversity and advanced training method could offer promising future directions beyond the present scope. Details on fine-tuning are provided Appendix B.5
4.6 REAL-WORLD RELEVANCE ANALYSIS
We assess the real-world relevance of VLM-SubtiBBench through correlation and transfer studies on industrial anomaly detection (MMAD @Jang et al.) 2025)) and aerial surveillance (QAG-360K (Li 2024)). both requiring fine- eeeigra visual discrimination. Table reports the rank correla- tions Spearman 1904) between each benchmark and the downstream tasks. Across models, our benchmark shows higher rank correlations with MMAD and QAG-360K than MLLM-CompBende
suggesting that it better captures the comparative cues underlying downstream performance. Model- wise results are summarized in Appendix C.T
To evaluate practical transfer, we fine-tune Owen2.5-VL--77 on validation split of VLM- SubtleBench and compare with an equally sized subset MLLLM-CompBench. Table 6 reports the resulting downstream accuracies on MMAD and QAG-360K. Fine-tuning Benn yields larger gains on both application benchmarks, whereas tine-tuning on MLLM-CompBente negative transfer. These results indicate that the subtle, fine-grained differ-
provides limited ence types in VIMM-SubtleBencc more effectively encode cues for real-world perceptual reasoning.
5 DISCUSSION
Conclusion. We introduce VLM-SubbleBenchh a benchmark for evaluating subtle comparative reasoning in vasion-language models. VL.M-SubtleBench focuses image pairs with subtle changes across ten difference types and six domains, where humans succeed but current VLMs struggle significantly. Our evaluation reveals systematic model-human gaps across difference types and domains, and controlled synthetic studies expose consistent failure modes, positioning VLM- SubtleBench as both rigorous benchmark and a diagnostic tool for future model development.
Broader Impact. Detecting subtle visual changes is core requirement for VLM-based agents operating dynamic environments. In game environments, where LLM agents must interpret com- plex visual states across diverse genres (Park et al.) 2026), perceiving fine-grained changes such as character movements is crucial, and VLM-SubuleBench can serve as a diagnostic tool for such perceptual capability. In robotic manipulation and navigation, detecting minute changes in object pose or state is essential for reliable physical interaction. In GUI agent scenarios, recognizing subtle interface changes such button becoming enabled is critical for executing correct actions.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Sami Abu-El Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaanaarasimhan. Youtube- 8m: A large-scale video classification benchmark, 2016. URL/tttps://axxii. www.ooggggaboaoooo..oe/.....sssa abs abt.//....116......o. 08675
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303 08774, 2023.
Al Anthropic. System card: Claude opus 4 & claude sonnet Claude-4 Model Card, 2025.
Anju Asokan and JJESI Anitha. Change detection techniques for remote sensing applications: A survey. Earth Science Informatics, 12(2):143-160, 2019.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong. Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL htt.::ttps:/aatt .rgg abs//502.... 13923
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9592-9600, 2019.
Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger. Beyond dents and scratches: Logical constraints in unsupervised anomaly detection and localization. Interna- tional Journal of Computer Vision, 130(4))947-96) 2022.
Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390, 2014.
Zonglin Di, Jing Shi, Yifei Fan, Hao Tan, Alexander Black, John Collomosse, and Yang Liu. Difftell: A comprehensive dataset for image difference captioning, 2025. URL https:/ /oppsree/
ew. www.net/ffrrmm?mm d= 6uy
(u j8Dcfk
Maxwell Forbes, Christine Kaeser-Chen, Piyush Sharma, and Serge Belongie. Neural naturalist:
Generating fine-grained image comparisons. arXiv preprint arXiv:1909.04101 2019.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma. and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
Abhay Gupta, Arjun D' D'Cunha, Kamal Awasthi, and Vineeth Balasubramanian. Daisee: Towards engagement recognition in the wild. arXiv preprint arXiv: 1609.01885, 2016.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14281-14290, 2024.
Erdong Hu, Longteng Guo, Tongtian Yue, Zijia Zhao, Shuning Xue, and Jing Liu. Onediff: A generalist model for image difference captioning, 2025. URL http:::/aaxxv..rrgg.. 2407 05645
Xinyue Hu, Lin Gu, Qiyuan An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi. Tatsuya Harada, Ronald M Summers, and Yingying Zhu. Expert knowledge- image difference graph representation learning for difference- medical visual question answering. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4156-4165, 2023.
11

---

Published as a conference paper at ICLR 2026
Xingliang Huang, Libo Ren, Chenglong Liu, Yixuan Wang. Hongfeng Yu, Michael Schmitt, Ronny Hänsch, Xian Sun, Hai Huang, and Helmut Mayer. Urban building classification (ube)-a dataset for individual building detection and classification from satellite imagery. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1413-1421, 2022.
Nam Hyeon- Hyeon-Woo, Moon Ye-Bin Wonseok Choi, Lee Hyun, and Hyun Oh. Vlm's eye ex- amination: Instruct and inspect visual competency of vision language models, 2024. URL bttps://arxizs wttp:///////..////2233
Haroon Idrees, Muhmmad Tayyab, Kishan Athrey, Dong Zhang, Somaya Al-Maadeed, Nasir Ra- jpoot, and Mubarak Shah. Composition loss for counting, density map estimation and localization in dense crowds. In Proceedings of the European conference computer vision (ECCV), pp- 532-546, 2018.
Amirmohammad Izadi, Mohammad Ali Banayecanzade, Fatemeh Askari, Ali Rahimiakbar, Mo- hammad Mahdi Vahedi, Hosein Hasani, and Mahdieh Soleymani Baghshah. Visual struc- tures helps visual reasoning: Addressing the binding problem in vlms. arXiv preprint arXiv:2506.222 244.22666..226.... 146,, 2025.
Harsh Jhamtani and Taylor Kerg-Kirkpattick. Learning to describe differences between pairs of similar images. arXiv preprint arXiv:1808.1055: 2018.
Xi Jiang, Jian Li, Hanqiu Deng, Yong Liu, Bin-Bin Gao, Yifeng Zhou, Jialin Li, Chengjie Wang, and Feng Zheng. Mmad: A comprehensive benchmark for multimodal large language models in industrial anomaly detection, 2025. URL htthstts: //axiiv org/ abs 2410 09453
Qirui Jiao, Daoyuan Chen, Yilun Huang. Bolin Ding, Yaliang Li, and Ying Shen. Img-diff: Con- trastive data synthesis for multimodal large language models. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 9296-9307, 2025.
Alistair EW Johnson, Tom Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chib-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth Berkowitz, and Steven Horng. Mimic-cxr-jpg, large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.0004: 2019.
Mchran Kazemi, Nishanth Dikkala, Ankit Anand, Petar Devic, Ishita Dasgupta, Fangyu Liu, Bahare Fatemi, Pranjal Awasthi, Dee Guo, Sreenivas Gollapudi, and Ahmed Qureshi. Remi: A dataset for reasoning with multiple images, 2024. AtLphttp::/ / arwiiiiiivv///../ org/ abs 2etps/2406.990 09175
Jihyung Kil, Zheda Mai, Justin Lee, Arpita Chowdhury. Zihe Wang, Kerrie Cheng. Lemeng Wang, Ye Liu, and Lei-Lun Harry Chao. Milm-compbench: A comparative reasoning benchmark for multimodal mlms. Advances in Neural Information Processing Systems, 37:28798-2882 2024.
Jean Kossaifi, Georgios Tzimiropoulos, Sinisa Todorovic, and Maja Pantic. Afew-va database for valence and arousal estimation in-the-wild. Image and Vision Computing, 65:23-36, 2017.
Alexandre Lacoste, Nils Lehmann, Pau Rodriguez, Evan David Sherwin, Hannah Kerner. Bjöm Lütjens, Jeremy Andrew Irvin, David Dao, Hamed Alemohammad, Alexandre Drouin, Mehmet Gunturkun, Gabriel Huang, David Vazquez, Dava Newman, Yoshua Bengio, Stefano Ermon, and Xiao Xiang Zhu. Geo-bench: Toward foundation models for earth monitoring, 2023. URL aittps://arxin http://aa.i..../////////00/
Ke Li, Fuyu Dong, Di Wang, Shaofeng Li, Quan Wang, Xinbo Gao, and Tat-Seng Chua. Show me what and where has changed? question answering and grounding for remote sensing change detection, 2024.
23828
Yante Li, Jinsheng Wei, Yang Liu, Janne Kauttonen, and Guoying Zhao. Deep learning for micro- expression recognition: A survey. IEEE Transactions on Affective Computing, (3())2028-2044 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
12

---

Published as a conference paper at ICLR 2026
Zhiqiu Lin, Siyuan Cen, Daniel Jiang, Jay Karhade, Hewei Wang, Chancharik Mitra, Tiffany Ling, Yuhan Huang, Sifan Liu, Mingyu Chen, et al. Towards understanding camera motions in any video. arXiv preprint arXtv:2504.15376. 2025.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey on deep learning in medical image analysis. Medical image analysis, 42: 60-88, 2017.
Chenyang Liu, Keyan Chen, Haotian Zhang, Zipeng Qi. Zhengxia Zou, and Zhenwei Shi. Change- agent: Towards interactive comprehensive remote sensing change interpretation and analysis. IEEE Transactions on Geoscience and Remote Sensing, 2024a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024b.
Steven R Livingstone and Frank Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)e0116391, 2018.
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Dongmin Park, Minkyu Kim, Beongjun Choi, Junhyuck Kim, Keon Lee, Jonghyun Lee, Inkyu Park, Byeong-Ul Lee, Jacyoung Hwang, Jaewoo Ahn, Ameya Mahabaleshwarkar, Bilal Kartal, Pri- tam Biswas. Yoshi Suhara, Kangwook Lee, and Jaewoong Cho. Orak: A foundational benchmark for training and evaluating Ilm agents on diverse video games. In International Conference on Learning Representations (ICLR), 2026.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks, 2019, URL httph:ttt////
10084
Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi. Jamie Tolan, John Brandt, Camille Couprie. Julien Mairal. Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. Dinov3, 2025. URL https: lattps://arxiv. org/
2bs//508... 0104
Tomáš Souček, Jean-Baaee Baptiste Alayrac, Antoine Miech, Ivan Laptev, and Josef Sivic. Look for the change: Learning object states and state-modifying actions from untrimmed web videos. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13956- 13966, 2022.
C. Spearman. The proof and measurement of association between things. American Journal of Psychology, 15(1):72-101, 1904. doi: 10.2307/1412159,
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computationall Linguistics (ACL), 2019.
Gemini Team. Gemini 2.5:
Pushing the frontier with advanced reasoning, multimodality. long context, and next generation agentic capabilities, 2025. URL https: orgg/bs............. 2507. 06261
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic composi- tionality. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion (CVPR), 2022.
13

---

Published as a conference paper at ICLR 2026
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. In Proceedings of the International Conference Learning Repre- sentations (ICIR), 2025.
Yanan Wang, Zhenghao Fei, Ruichen Li, and Yibin Ying. Learn from foundation model: Fruit detection model without manual annotation. arXiv preprint arXiv:2411.16196, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpont foun- dation models on low-level vision. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Kaining Ying, Fanqing Meng. Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang. Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi, 2024. URL https: wwwpvv.iivvv/..////i//.aa./a.wi. org/ abs www./404...2..... 16006
Jingyi Zhang. Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision langualanguuage models for vision tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 46(8):5625-5644 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-ae-a-judge with mt- bench and chatbot arena. Advances neural information processing systems, 36:46595 46623, 2023.
Shijie Zhou, Alexander Vilesov, Xuchai He, Ziyu Wan, Shuwang Zhang, Aditya Nagachandra, Di Chang, Dongdong Chen, Xin Eric Wang, and Achuta Kadambi. Vlm4d: Towards spatiotem- poral awareness in vision language models. arXiv preprint Xrx 02009, 2025.
14

---

Published as a conference paper at ICLR 2026
A
BENCHMARK CURATION DETAIL
Table 7 provides a detailed breakdown of the test split by difference type, domain, and data source.
Table 7: Number of test examples per difference type, domain, and source.
Type
Domain
Source
Natural
COCO
11 114
Industry
MVTEC-AD
300
Attribute
Medical
MIMIC-Dif-VQA
330
Synthetic
Synthetic
700
Natural
Changelt
465
State
Industry
MVTEC-AD
586
Emotion
Natural
CREMA-D. RAVDESS, AFEW-VA VA, DAiSEE
1007
Natural
YT8M
613
Temporal
Natural
VLM4D
292
Game
YT8M
104
Natural
VLM4D
520
Spatial
Synthetic
Synthetic
600
Natural
COCO
98
Existence
Aerial
LEVIR-MCI
394
Synthetic
Synthetic
600
Natural
MegaFruits
330
Natural
UCF-QNRF-ECC
Natural
UBC
133
Quantity
Natural
Changelt
58
Aerial
LEVIR-MCI
Industry
MVTEC-LOCO
234
Synthetic
Synthetic
500
Natural
YT8M
952
Quality
Game
YT8M
Natural
CameraBench
1080
Viewpoint
Synthetic
Synthetic
500
Natural
YT8M
916
Action
Game
YT8M
92
A. ATTRIBUTE
MVTEC-AD (Bergmann al. 2019) data used construct the attribute dataset. Each defect type is categorized as either attribute (e.g., color anomaly, contamination, glue residue) or state (e.g., crack, cut, hole); defect types that cannot be meaningfully compared terms of severity (e.g., missing components, spatial misplacements) are excluded. For each defect type, pairs are formed between two anomaly images of the same type but with different defect sizes. Defect size is measured the percentage of defective pixels relative to the total image area, computed from pixel-level ground-truth masks. We select pairs whose defect sizes differ by at least 0.2%, with up to 15 pairs sampled per defect type. QA pairs then ask which image shows larger smaller defect area.
COCO (Lin et al. 2014) data also used. We first identify images containing single object using COCO instance segmentation annotations, filtering by object-to-image area ratio (0.5-10%) GPT-4o is then prompted with the bounding-box-ameen object's primary color, suggest a similar but distinguishable alternative (c.g.. red - orange), and generate the corresponding QA pair. The image editing model -aw.i----faas---a----eeee
image to simultaneously determine the
15

---

forms using GPT-4o Published as a conference paper at ICLR 2026
mnan--bananal applies the color change. All edited images manually checked by human anno- tators, who verify that the color change matches the intended specification and that the QA pair accurately reflects the applied modification.
MIMIC-DiLI-VQA (Hu et al. 2023), built on the MIMIC-CXR dataset (Johnson et al. 2019), pro- vides chest X-ray pairs from different visits of the same patient along with comparative clinical questions. Since our benchmark focuses on visually perceptible differences independent of medical expertise. reformulate the original clinical questions into layperson-frient (e.g., converting "lung opacity progression" into "Which image shows whiter lungs?"). Annotators then verify the consistency between the reformulated questions and the corresponding image pairs.
Synthetic. We render scenes containing 2-10 shapes (circles, squares, and triangles) with distinct colors on an 800x600 white background. One shape randomly selected, and either brightness adjustment or size modification is applied. For brightness changes, we shift the lightness channel (L) in OKLAB color space by 5-10%, producing a subtly brighter or darker variant of the same hue. For size changes, we scale the selected shape by 15-20%. The resulting QA pairs ask which transformation occurred to which object, with three distractors formed by combining the wrong shape, wrong color, or opposite direction of change.
A.2 STATE
MVTEC-AD pairs constructed Section A.1 are also used for state differences, since some anoma- lies pertain to state rather than attribute.
Changelt (Souček et al., 2022) contains annotations of state- moatteeodiiying actions and resulting object states internet videos. A subset of the videos is manually annotated, and we use this portion of the data. From each video, we sample multiple frames corresponding to either state action, or state2. Based object and action, automatically generate questions. For example, if the object is apple and the action is peeling, the question becomes: In which image is the apple peeled more? Annotators then select frame pairs in which the state-modifying action is more evident in the second frame (Figure
Ouestion:
impg upple pested
Palk
Folder: apple 2gQYIvaDXWO
Selected Pairs
Bekiat /magoo
Nor
Figure 6:
Example annotation interface for sources with ground-truth annotations (Changelt). An- notators select frame pairs from pre-sampled candidates based on the provided labels.
A.3 EMOTION
CREMA-D, RAVDESS, AFEW-VA VA, and DAiSEE (Cao et al. 2014 Livingstone & Russo 2018
Kossaifi et al. 2017 Gupta et al. 2016) are clip-level video datasets annotated for emotion. Specif- ically, CREMA-D and RAVDESS consist of actors speaking sentences with specified emotions and
16

---

Published as a conference paper at ICLR 2026
intensity levels. AFEW-V VA contains movie clips annotated with valence and arousal, while DAiSEE provides short video snippets capturing users' emotions. From these clips, we randomly sample frames and construct paired examples based the relative intensity of expressed emotions. For question generation, we use the labeled emotion categories; in the case of AFEW-VA, which lacks explicit emotion labels, annotators asked to choose the most appropriate emotion depicted in the scene.
A.4 TEMPORAL
VLM4D (hou et al. 2025) is a benchmark designed to evaluate the spatiotemporal reasoning ca- pabilities of video LLMs. It provides video clips depicting exocentric and egocentric movements of people, animals, or objects, along with VQA tasks about their motion. Annotators select frame pairs that reflect the corresponding spatial change. When movement cannot be reversed (e.g., per- son riding a bicycle or running), annotators label the sample as temporal. Only samples labeled as temporal are used form the temporal pairs.
For the temporal subset, we generate questions that ask about the temporal order of the two frames. Specifically, use two templates: "Which frame happened first?" and "Which frame happened later?"
YT8M Abu-EElHaajaa al. 2016) contains diverse collection YouTube videos spanning numer- ous domains. For each video, we segment clips using histogram matching and construct image pairs by randomly selecting frames with cosine similarity below 0.99, thereby avoiding frame extraction from frozen screens. Annotators then select frame pairs whose order cannot be reversed, marking them as temporal pairs (Figure
image pai Annotated Temporal Palr
temporal analysts. annotation.
Image
Image
PREVIOUS
ENTER TOGGLE ANNOTATION
Figure 7: Example annotation interface for sources without ground-truth annotations (YT8M tem- poral). Annotators provide labels (e.g., whether the temporal order irreversible) for each frame pair.
A.5 SPATIAL
VLM4D is used to construct the spatial data. We collect frame pairs following the same procedure described in Section A.4 Samples labeled temporal (i.e., non-reversible motions) used to form the temporal pairs. For the spatial pairs, we use both reversible and non-reversible samples.
We then adapt the original VQA questions into comparative form using GPT-4o. For example, the original question is "Which direction is the horse moving towards?" with the answer "left", reformulate it as "In which image is the horse relatively in left position?" In addition,
17

---

Published as a conference paper at ICLR 2026
also generate counter-directionns variant by flipping the direction (e.g., "right"), in which case the correct answer becomes the earlier frame.
A.6 EXISTENCE
LEVIR-MCI (Liu et al. 2024a) is an aerial dataset for image change detection. A subset of the data includes captions describing changes within the same region. We construct image pairs in which less than 20% of the total area has changed, further filtering for pairs with cosine similarity greater than 0.8. GPT-4o is then employed to paraphrase the captions in order to generate answers and distractors for the before and after images, and to determine whether each question pertains to existence or quantity.
COCO Lin al.) (014) is also used for existence pairs. We select images containing 1-10 objects with individual area ratios between 1% and 5% of the image. For each image. the largest object is chosen as the removal target. GPT-4o is prompted with the bounding-hox-annotacee image to generate a removal instruction and the corresponding QA pair. The image editing model Gemini- f--faah-immage-reeee
then removes the specified object. Human annotators verify that the removal is clean and that the QA pair accurately reflects the change.
Synthetic. We render scenes containing 20-30 background shapes of uniform size (35 pixels) with random types, colors, and rotations on an 800x600 white canvas. One shape is randomly selected as the target and either removed from added the second image, while all other shapes remain unchanged. The resulting QA pairs ask what has appeared or disappeared, with three distractors formed from unchanged background shapes paired with random appeared/disaappaaed labels.
A.7 QUANTITY
MVTEC-LOCO (Bergmann et al., 2022) data are used to construct the quantity dataset. For each product category fe-g..juice bottle, screw bag, pushpins), identify anomaly images with quantity- related defects (e.g., additional or missing screws, nuts, or washers) using the ground-truth defect annotations provided in the dataset's configuration. Each anomaly image is then paired with its visually closest normal image, selected by computing DINOv3 feature cosine similarity between the anomaly and all available normal images. QA pairs ask which image contains more or fewer of the specified item.
Changelt data featuring "pouring" actions are repurposed for quantity differences. Frame pair selection follows the same annotation procedure described in Section A.2 Since pouring increases the amount of liquid in a container, the resulting QA pairs ask "Which image has more {liquid} been poured?", where the liquid type varies across videos (e.g., beer, juice, milk, tea).
LEVIR-MCI data related to quantity are also included. We follow the same annotation method described in Section|.66
MegaFruits, UCF- SCC--NRF-ECCE and UBC (Wang et al. 2024; Idrees et al. 2018 Huang et al. 2022) are datasets containing multiple instances of fruits, people, or buildings. Annotators first draw a bounding box around a selected object in the image. A black box the same size is then overlaid on the image, after which nano-banana is tasked with inpainting the region while ensuring that no new object is introduced. In this way, we obtain modified images in which one or more objects have been removed. The resulting QA pairs ask "Which image has (fewer, more) (object) ?", where the object type depends on the source dataset (e.g.. blueberries, peaches, or strawberries for MegaFruits; people for UCF-QNRF-ECA buildings for UBC).
Synthetic. We render scenes containing 10-20 shapes of a single type and color (size 20-40 pixels) arranged at random non-overlapping positions on an 800x600 white canvas. To create the second image, we either add or remove one instance of the same shape type. QA pairs ask which image contains more (or fewer) of the given shape, with binary answer choices (first image / second image).
A.8 QUALITY
YT8M also used to construct the quality dataset. Many videos in YT8M contain frames vary- ing visual quality due factors such motion blur, sensor noise, overexposure, compression
18

---

Published as a conference paper at ICLR 2026
artifacts. For each video, extract ten random frames and present them to human annotators. Annotators select two frames: one with the best visual quality and one with the worst, judging qual- ity based on the presence or absence of blur, noise, overexposure, and artifacts. The selected pair forms quality comparison sample. For each pair, the question is randomly chosen from one of two forms: "Which image has better quality (less blur, noise, overexposure, artifacts) or "Which image has lower quality (more blur, noise, overexposure, or artifacts)?", with binary answer choices corresponding to the first or second image.
A.9 VIEWPOINT
CameraBench (Lin et al. 2025) provides ground-truth camera movement annotations for video clips. covering a range of motion types such as dolly-in/out, truck-leftrright, pan-leftrrigh, tilt- up/down, pedestal-up/down, and zoom-in/out. For each video, extract ten frames uniformly sampled from the middle 80% of the video (excluding the first and last 10%). Human annotators then select frame pairs that visually reflect the annotated camera movement, ensuring that the chosen frames provide clear visual cues for the specified motion (e.g., sufficient parallax for dolly move- ments visible lateral shift for truck movements).
Based on the selected frame pairs and their associated camera movement labels, we automatically generate binary QA pairs. For each movement type, the question asks about the direction of camera motion (e.g., "In which direction does the camera move from the first image to the second image?") with two answer choices corresponding to opposite directions (e.g.. left/right for truck movements, in/out for dolly movements). When a video contains multiple camera movement labels (e.g., si- multaneous pan-left and tilt-up), multiple QA pairs are generated from the same frame pair, each targeting different movement axis.
Synthetic. We simulate camera motion by applying uniform transformation to all shapes in the scene on an 600 white canvas. For camera translation, all shapes are shifted by 20-80 pixels along a single axis, simulating a camera pan. For camera rotation, all shape positions are rotated by 5-20 degrees around the image center, and each shape's s own orientation is adjusted by the same angle. QA pairs ask the direction of camera motion, with answer choices including both translation and rotation directions distractors.
A. ACTION
YT8M is used to construct the action dataset. We form frame pairs in the same manner as described in Section A.44 For each frame pair, annotators examine the two images and identify salient entity whose action differs across the pair. They then provide (i) the entity description (item) and (ii) a short free-form action phrase for each image (first/secod)), which can capture changes pose, activity, or interactions involving people, animals, or objects.
We generate VQA questions from these annotations. Given an annotated (item, action) from one of the two images, we create the question "In which image is the {item} {action]?" ?" and minimally refine its grammar using GPT-4o.
B EVALUATION DETAIL.
B. PROMPT DESIGN FOR EVALUATION TASKS
Figures 16 the exact prompts used in experiments. Unless otherwise noted, tempera- ture is fixed at 0.5 and the repetition penalty at 1.0.
19

---

Published as a conference paper at ICLR 2026
System Prompt
You are a helpful assistant that answers multiple-choice questions about differences between two images. Your task to carefully analyze both images and identify the main difference between them.
Guidelines:
Unless specified in the options, the difference is described in terms of the second image relative to the first. Respond **nny with the answer letter (A, B, C, etc.). not provide any reasoning explanation.
User Prompt
Question: {question_ text]
Carefully examine the images and choose the best description of the key visual difference.
Options:
{options text]
Figure 8: Standard prompt used in our benchmark.
System Prompt
You are a helpful assistant that answers multiple-choice questions about differences between two images. Your task is to carefully analyze both images and identify the main difference between them.
Guidelines:
Unless specified in the options, the difference described in terms of the second image relative to the first.
Resspon ** **olly** in the following format. The answer should be single letter.
#4# Reasoning
fexplanation the key visual difference between the two images]
##fff Answer
[answer (single letter)]
User Prompt
Question: {question_tess _exxt]
Carefully examine the images and choose the hest description of the key visual difference.
Options:
(options text]
Figure 9: Reasoning prompt used in our benchmark.
20

---

Published as a conference paper at ICLR 2026
System Prompt (1st step)
You are an expert image analyst. Your task is to carefully describe the differences between two images. Do not answer questions about the images yet only analyze and describe what is different between them.
User Prompt (1st step)
Please provide careful and detailed description of the differences between these two images. Focus what changed, what's different, what distinguishes the first image from the second image. Using the description of the differences, you will be asked the following question and you will need to choose one correct answer. However, do not answer the question yet just analyze the differences:
{question text)
{options_texs
System Prompt (2nd step)
You helpful assistant that answers multiple-choice questions about differences between two images. Your task to carefully analyze both images identify the main difference between them. You will be provided with description of the differences to help guide your analysis.
Guidelines:
Use the provided difference description to understand what has changed between the images.
Verify the described difference by examining the images.
Unless specified the options, the difference deseribed
the second image relative the first. Respond ** only* with the answer letter (A, B, C, D, etc.). Do not provide any reasoning or explanation.
User Prompt (2nd step) am showing you two images (first and second).
Description of the differences: (diff' description)
Question: (question_texx text]
Based on images the description differences, choose answer.
Options: (options text]
Figure 10: Two-step reasoning prompt used in our benchmark.
System Prompt
You are a helpful assistant that answers multiple-choice questions about differences between two images. The grid lines added to both images to help you compare the objects better. Your to carefully analyze both images identify the main difference between them.
Guidelines:
Unless specified the options, the difference described of the second image relative the first. Respond ** only** with the answer letter (A, B, C, D. etc.). Do not provide any reasoning or explanation.
User Prompt
Question: (question_texx text]
Carefully examine the images and choose the best description of the key visual difference.
Options: foptions text)
Figure 11: Grid prompt used in our benchmark.
21

---

Published as a conference paper at ICLR 2026
System Prompt
You are a helpful ussistant that answers multipe--hhice questions about differences between two images that concatenated horizontally (first image on left and second image on the right, separated by a black line), Your task is to carefully amalyze both images and identify the main difference between them.
Guidelines:
Unless specified the options, the difference described terms of the second image relative to the first. Respond **only *nly*** with the answer letter (A, B, C, D. etc.). Do not provide any reasoning or explanation.
User Prompt
Question: {question text]
Carefully examine the images and choose the best description of the key visual difference.
Options:
(options text]
Figure 12: Concatenating prompt used in our benchmark.
System Prompt
You helpful assistant that answers multiple-choice questions about differences between two images.
Your task to carefully analyze the images and identify difference between them. I am showing you four images:
1. Original first image
Original second image
3. Highlighted first image (with areas of significant change marked with green boxes, and other areas dimmed)
4. Highlighted second image (with the same areas marked)
The highlighted images help you focus on the most significant differences between the two images. Use them to quickly identify where the changes occur, then examine those areas carefully in the original images.
Guidelines:
Unless specified in the options, the difference is described terms the second image relative to the first.
Focus the green-boxed regions the highlighted images to identify where changes answer letter (A, B, C, D, etc.). not provide any reasoning or explanation.
Respond #nll+
User Prompt
am showing you four images:
1. Original first image
2. Original second image
3. Highlighted first image (green boxes mark significant change areas, other areas dimmed)
4. Highlighted second image (same areas markod)
The highlighted images (3 4) show you WHERE the main differences are located. The green boxes indicate the top 2-3 most significant change regions. Use these to guide your attention, then carefully examine those specific areas in the original images (I and 2) to determine WHAT the difference is.
Question: fquestion_tees
Carefully examine the images and choose the best description of the key visual difference.
Options: (options ]
Figure 13: Highlighting prompt used in benchmark.
22

---

Published as a conference paper at ICLR 2026
System Prompt
You are a helpful assistant that answers multiple-choice questions about differences between two images. Your task is to carefully analyze first and second images and identify the main difference between them, The third image is the overlay of the first and second images. You may use the third image help you analyze the difference between the first and second images.
Guidelines:
Unless specified the options, the difference described of the second image relative the first. Respond *only** with the answer letter (A, B, C, D, etc.). Do not provide any reasoning or explamation.
User Prompt
um showing you three images:
1. First image
2. Second image
3. Overlapped image (50/50 blend first and second images)
Question:
(question text)
Carefully examine the images and choose the best description of the key visual difference of first and second images.
Options: (options text)
Figure 14: Overlapping prompt used in our benchmark.
System Prompt
You are a helpful assistant that answers multiple-choice questions about differences between two images. Your task is to carefully analyze first and second images and identify the main difference between them. The third image is black-and and-white difference map between the first second images, where brighter areas indicate larger differences. You may use the third image to help you analyze the difference between the first and second images.
Guidelines:
Unless specified in the options, the difference is described in temms of the second image relative to the first. Respond **ony*** with the answer letter (A, B, C, etc.). Do provide any reasoning explanation.
User Prompt
um showing you three images:
1. First image
2. Second image
3. Black-and-white difference map between the first and second images
Question:
(question text]
Carefully examine the images choose the best description of the key visual difference of first second images.
Options: (options_tee __ext)
Figure 15; Subtracting prompt used in our benchmark.
23

---

Published as a conference paper at ICLR 2026
System Prompt
You are expert at visual analysis and image comparison. Compare the images and briefly describe what's different between them.
Keep your response concise and direct. Use simple phrases like "In the first image, X, however in the second image, or "X appeared in second image" or first image, X relatively Y". Avoid detailed explanations structured lists.
User Prompt
Describe the differences between the two images
Figure 16: Captioning prompt used in our benchmark.
B.2 VISUAL INPUT CONSTRUCTION AND EXAMPLES
We describe how visual inputs are constructed for prompting, including grid overlays, image con- catenation, image overlap, and image subtraction.
Grid Overlays. A 4x4 grid overlay is generated by drawing black lines with 30% opacity and a line width of 3 pixels both images.
Concatenated Images. Image pairs are concatenated horizontally with a 1-pixel-wide black sepa- rator, forming single composite image that is then used the VQA input.
Overlap Images. Two aligned inputs are blended with equal weights (50% contribution each) in pixel space, producing composite image that visually merges both inputs. The generated overlap image is provided to the VLM together with the original input pair (Figure 17).
Subtraction Images. Subtraction images generated by computing the absolute pixel-wise dif- ference between the aligned inputs, converting the result to grayscale, and normalizing it to highlight regions of maximal change. Specifically, for two images J1, 12 E [0, 2x5]xxxxx x3
1 Σ (1 (1(xx,,,)) G(æ, y) = 3
S(x, v) 255 G(a,y)
max G(u,
where S denotes the grayscale subtraction image. For both overlap and subtraction variants, the generated images are provided to the VLM alongside the original inputs (Figure 18).
Highlight Images. Highlight images are generated by drawing bounding boxes the largest gions of change. Given the pixel-wise difference map G, we obtain a mask of considerable change by computing percentile threshold: letting Tp be the p-th percentile of values in G (we p 90), we define the binary mask
G(r,y) > Tp-
M(r, 9) {o:
otherwise.
Morphological closing and opening are applied M to connect nearby changes and remove noise, producing a cleaned mask with connected components {C²). The clusters {C&} are sorted by area, and the three largest clusters are selected. To avoid highlighting insignificant regions, we retain the second and third clusters only if their areas are least 50% of the area of the largest cluster; otherwise, they are discarded. Highlight images are then constructed by dimming the background (using 0.5) while preserving the original appearance only inside the selected bounding boxes. Finally, we draw green-colored boundary around each box to emphasize the regions of change (Figure 19
B.3 CAPTION EVALUATION METRICS
Cosine similarity score. We measure caption-level semantic agreement using cosine similarity score (CSS) (Reimers& & Gurevych 2019) implemented as cosine similarity between Sentence-
24

---

Published as a conference paper at ICLR 2026
(a) First image
(b) Second image
(c) Overlap result
Figure 17: Example of overlap-image construction. Two aligned inputs are blended with equal weights (50% each) to produce a composite image that visually merges both inputs.
Xire
Xtra
(a) First image
(b) Second image
(c) Subtraction result
Figure 18: Example of subtraction-imant construction. Given two aligned images (41, Iz), the subtraction map S highlights regions with maximal pixel-wise change.
BERT embeddings of the ground-truth caption and the model-generated caption. We compute CSS using a CentenceTransformg model (paraphrase -MiniLM-L6---
LLM-as-a-judde Following (Lin et al. 2025), we use GPT-4o as an evaluator to judge whether the candidate caption matches the reference caption. We prompt GPT-4o with: Reference caption: "(reference)" Candidate caption: "[candidate) Does the candidate caption match the reference caption? Answer Yes No. We treat the ground-truth caption the reference and the model- generated caption as the candidate, and compute accuracy as the fraction of outputs labeled "Yes".
B.4 HUMAN EVALUATION
Human evaluation is conducted by sampling 1,000 examples in total: 100 examples difference type. Since each difference type draws from multiple data sources, we first determine the number of samples per source proportionally to its contribution within the type, then randomly sample that many examples from each source. During evaluation, annotators are asked to answer questions using an interface that displays two images side by side for comparison, with the option to view images sequentially.
B.5 FINE-TUNING SETUP
Training performed on NVIDIA A100 GPUs, each equipped with 80GB memory. We use learning rate of le-5, a per-device batch size of 8, resulting in an effective batch size of 32. The models are trained for epochs, with parameters including vision encoder, projector, and lan- guage model jointly optimized.
For the transferability study (Table 6). we fine-tune Qwen-2.5-VL-7B on our 1.277-sample valida- tion split of SubtleBench and size-matched subset of MLLM-CompBencch for comparison. For MMAD evaluation, all training instances are converted to the standardized MMAD task format, and for QAG-360K, all training samples adapted to standard prompt template. Aside from this dataset-specific formatting, the fine-tuning procedure is identical across all settings, and we use the same hyperparameters described above.
25

---

Published as a conference paper at ICLR 2026
(a) Highlighted first image
(b) Highlighted second image
Figure 19: Example highlight images. Significant regions of change, extracted via percentile thresh- olding and morphological filtering, are emphasized by dimming the background and drawing green bounding-box boundaries.
Table 8: Model-wise accuracy on VIMM-SubtleBench, MLL.M-CompBench, MMAD, and QAG- 360K.
Models
VLM-SubtleBench
MLLM-CompBench
MMAD
QAG-360K
Qwen-2.5-VL-7B
59.4
73.6
65.0
34.4
Qwen-2.5-VL-32B
62.2
74.6
67.6
35.5
Qwen-2.5-VL-72B
65.4
76.9
68.9
41.2
GPT-4o
61.6
75.7
67.7
35.2
03
75.7
86.3
72.9
39.7
GPT-5-main
71.3
83.9
70.6
36.3
GPT-5-thinking
77.8
86.3
73.5
42.1
Claude-sonnet-4
62.6
73.6
70.9
30.7
Gemini-2.5-
62.5
85.2
71.4
36,8
Gemini-2.5-pp
68.2
87.2
72.2
36.3
C ADDITIONAL RESULTS
C.1 CORRELATION ANALYSIS WITH DOWNSTREAM BENCHMARKS
&shos the accuracy of all evaluated models across VI.M-SubiieBench. MLLMMCompBBnch, MMAD, and QAG-360K. For MMAD evaluation, we randomly sample 10% of the test set and evaluate models using this subset. For QAG-360K evaluation, we exclude the change-ratio category due to its continuous answer format and convert the remaining question types into multiple-choice queries following our base prompt template. We then randomly sample 723 validation examples for evaluation.
C.2 ADDITIONAL PROMPTING STRATEGY
We explore an additional prompting strategy based on pure language-only comparison.
For the emotion, existence, and quality simil to MLLM-CompBench (Kil et al. 2024
we ask the following questions for each image:
Emotion: "Describe the emotion expressed in the image in detail and rate its intensity on a scale of 1-10."
Existence: "Carefully list all objects visible in the image, including their approximate locations."
"Analyze the quality of the image and rate it on scale of 1-10, considering blur, noise, overexposure, compression artifacts, and other quality issues."
Quality:
26

---

Published as a conference paper at ICLR 2026
Table Effect of pure language-based comparison for emotion, existence, and quality categories.
Category GPT-5-main
Two-stage Reasoning
Emotion
92.7
87.8
Existence
75.4
63.2
Quality 84.5
84.6
Table 10: Domain-wist wise accuracy (%) of open-source and proprietary VLMs.
Model
Natural
Game
Industry
Aerial
Synthetic
Medical
Random Guess 49.2
50.0
50.0
26.9
29.3
50.0
Open-source
LLaVA-NeXT-7B
48.6
47.9
53.8
35.4
30.0
41.5
LLaVA-OneVisione Vision-7B
59.3
56.0
54.7
62.2
32.4
50.9
Qwen2.5-VL-7B
65.2
61.8
66.3
62.8
43.8
50.3
Qwen2.5-VL--2B
66.1
63.4
64.4
64.6
53.0
54.5
Qwen2.5-VL-72B
69.6
65.1
69.6
64.1
54.8
65.2
Proprietary
GPT-4o
68.4
65.8
71.5
46.9
45.0
62.4
03
77.3
76.4
79.3
71.4
72.5
68.5
GPT-5-main
74.1
72.1
77.9
60.8
63.6
78.8
GPT-5-thinking
77.2
75.5
81.2
70.7
78.8
82.4
Claude-sonnete
64.2
60.3
66.3
74.1
56.3
54.8
Gemini-2.8 2.5-ii22
68.1
66.5
73.7
73.4
43.3
62.4
Gemini-2..0 pro
73.2
71.8
79.7
75.7
50.6
68.8
We then perform VQA using only the generated descriptions, without providing the original images, evaluate the model's ability answer purely from language-hase -aaaa reasoning.
The results for the pure language-based comparison are shown in Table 9 For the emo- tion and existence categories, this method exhibits lower performance, consistent with the findings reported in MLLM-CompBench, Interestingly, performance on the quality category remains nearly identical, suggesting that explicit 1-10 rating prompts may serve as reliable intermediate repre- sentation for comparative judgment in this aspect.
Results.
C.3 DOMAIN-WIST PERFORMANCE ANALYSIS
Table 10 shows domain-wist accuracy for proprictary VLMs. Among proprictary models, o3 and GPT-5- GPT-5-thinking achieve the highest scores in most categories. In particular, they exhibit substan- tial performance gap over other models the synthetic and medical domains, while both show comparatively weaker results on the aerial domain.
C.4 EXTENDED COLOR-SENSITIIC
ANALYSIS
Inspired by (Hyeon- Woo et al., 2024), we extend our synthetic-control analysis to include color- sensitivity axis, probing whether hue-level perceptual biases compound subtle comparative reason- ing failures. We incorporate five representative colors (two green tones and three non-green: blue, red, and magenta) and systematically vary AE (hue shift in OKLAB space), brightness (L channel), size, count, and translation. OKLAB chosen for its perceptual uniformity, where encodes light- ness and (a, b) correspond to green-magenn and flue-yellow opponent axes. To isolate hue effects, AE adjustments are applied by modifying (a, b) while keeping constant, and brightness variation fixes hue while sampling
[0.4, 0.8]. All other setup conditions follow those in Figure 5
Results are presented Figure 20 Consistent with prior work (Hyeon- Woo et al., 2024), models ex- hibit pronounced difficulty in distinguishing green hues, showing significantly lower accuracy than
27

---

Published as a conference paper at ICLR 2026
for red or blue tones. Magenta elicits the most severe degradation (approaching 0%), revealing systematic color-specific weakness in GPT-4o. contrast, brightness variation produces no notable color-dependent gap, suggesting that VLMs relatively invariant luminance when performing comparative reasoning. Cross-factor analyses (color x size/count/ size/coun/veewpoin)) show minimal interac- tion, implying that these tasks largely unaffected by color sensitivity, as hue discrimination itself contributes little to the reasoning objective.
Rrightness
Quanity
Viewpoint
D7
22
30
3
29
21
X
21
22
F7
Higusa
14
031 032
Magstak
Magafoxe
Chorge
Figure 20: Color-sensitivity analysis of GPT-4o under controlled synthetic settings,
C.5 SOURCE VALIDATION AND DISTRIBUTIONAL CONSISTENCY
To examine whether the use of nano- nano-banana may introduce stylistic artifacts or distribution shifts that could confound evaluation, we conduct an analysis on our caption dataset. Specifically, we compare approximately 1K original (non-edited) caption pairs with subsets in which one image of each pair was reconstructed using nano-banana. Evaluation is performed on both VQA accuracy and captioning metrics (CSS and LLM-Judge). Table [] reports the comparison between edited and non-edited pairs. Across all metrics, differences are negligible, indicating that the use of nano-banana editing does not introduce measurable stylistic or distributional bias that could affect evaluation.
Table 11: Comparison between real and nano-boanana-rrcoosttuuuted image pairs on VQA and cap- tioning metrics.
Type
Accuracy
CSS
LLM-Judge
Real
60.6
0.51
26.3
Reconstructed
60.6
0.51
27.3
C.6 EFFECT PROMPTING OPEN-SOURCE VISION-LANGUAGE MODELS
Table 12 reports the effect of prompting strategies open-source model (Qwen2.5-VL-72B) complementing the proprictary model analysis (GPT-5-main) the main text. Overall, we observe similar trends: reasoning slightly improves performance, while concatenation and overlap tend to degrade accuracy. Notably, the subtract strategy yields the strongest gains on spatial tasks, consistent with the proprietary model results.
Table 12: Effect of different prompting strategies in VLM-SubtleBendde
Model
AT
ST
EM
TM
SP
EX
QN
QL
VP
AC
Random Guess
35.9
50.0
50.0
50.0
36.6
23.2
48.9
50.0
42.1
50.0
Qwen2.5-VIL V-L-7B
53.9
68.9
85.9
49.9
47.8
81.7
67.7
78.4
56.2
74.1
Reasoning
50.6
68.0
87.5
51.0
44.4
81.7
65.0
77.1
54.7
75.0
+ Grid
54.9
67.6
85.6
49.8
46.3
83.4
70.1
75.7
54.4
71.1
Concat
55.7
68.0
86.4
50.0
39.6
67.5
61.0
65.4
47.0
73.7
+ Overlap
46.3
63.2
85.3
49.7
48.8
78.7
63.1
72.5
49.7
71.6
Subtract
55.7
64.4
85.6
49.5
51.1
79.8
63.4
63.0
49.9
71.2
28

---

Published as a conference paper at ICLR 2026
Table 13: Performance of proprictary models by data source
Dataset
Random
GPT-4o
GPT-5- main
GPT-5- thinking
GPT-o3
Claude- sonnet-4
Gemini- 2.5-flash
Gemini- 2.5-pro
Attribute
MVTEC-AD
50.0
76.7
79.0
82.0
79.0
66.3
76.3
83.3
COCO
25.0
78.
82.5
86.0
84,2
71.1
69.3
77.2
MIMIC-DIFF-VOR VQA
62.4
78.8
54.8
62.4
68.8
Synthetic
25.0
31.3
66.0
84.4
81.1
35.0
28.3
33.4
State
MVTEC-AD
50.0
67.2
73.5
75.9
74.9
64.0
71.5
75.1
Changelt
50.0
84.5
85.4
65.6
73.8
78.1
Emotion
CREMA-D, RAVDESS, AAEW-VA DAiSE
50.0
89.5
92.7
93.1
92.9
83.3
88.4
89.8
Temporal
YT8M
55.4
63.0
611.4
VLM4D
50.0
52.7
49.3
54.8
53.8
49.0
49.3
50.7
Spatial
VLM4D
50.0
58.5
57.3
62.5
60.4
52.1
56.5
58.5
Synthetic
25.0
43.8
57.7
50.5
45.7
27.0
33.0
Existence
LEVIR-MC MCI
20.0
43.7
59.1
74.6
73.6
79.2
80.5
79.7
COCO
25.0
77.6
86.7
80.6
88.8
83.7
75.5
92.9
Synthetic
25.0
69.0
84.2
93.5
77.8
Quantity
MVTEC-LOCO
50.0
75.6
87.6
93.2
90.6
71.8
75.6
86.8
MegaFruits
50.0
50.9
59.7
66.1
67.0
57.9
51.5
62.77
UCF-QNRF-ECC
50.0
71.2
78.8
50.0
71.2
UBC
50.0
51.9
60.9
59.4
66.9
61.7
52.6
59,4
LEVIR-MCI
20.0
59.2
73.5
69.4
65.3
67.3
73.5
87.8
Changelt
50.0
41.4
58.6
55.2
56.9
50.0
60.3
62.1
Synthetic
65.8
63.2
Quality
YT8M
50.0
72.4
84,5
84.8
87.6
70.8
77.1
84.8
Viewpoint
CameraBench
50.0
56.2
63.8
69.9
68.5
57.6
58.1
64.8
Synthetic
25.0
41.0
44.0
65.4
44.8
38.2
50.4
Action
YTSM
50.0
76.7
83.6
84.9
84.8
66.3
72.3
76.8
C.7 PERFORMANCE PROPRIETARY MODELS BY DATA SOURCE
Table 13 shows the performance of proprietary models by data source. Across models, o3 and GPT- 5-thinking achieved the highest scores in most categories, exhibiting substantial performance gap over other models in the synthetic and medical domains, while both showed comparatively weaker results on the aerial domain. Across domains, models generally performed strongest on natural and industrial imagery, while performance degraded in synthetic and aerial settings.
29


Reviews:
[{"review_id": "ac3qBXAB6O", "reviewer": "Reviewer_Waq7", "summary": "In this paper, the author presented VLM-SubtleBench, a benchmark to evaluate subtle comparative reasoning in VLMs across 10 difference types and 6 visual domains. The benchmark contains more than 11k image-pair QA samples and 1k human-annotated difference captions. In this paper, the authors also claim that existing such benchmarks focus on salient differences and lack domain diversity. They evaluated several current VLMs on the benchmark and found that they struggle with subtle differences compared to human performance.", "questions": "- how subtlety was maintained for the types from frames were takes from videos, to be sure that only the object in question moved.\n- How do authors ensure repetition of the same datasets across categories represents distinct types [weakness 3]?\n- Insights about domain-specific evaluation and evaluating multi-image models on the benchmark [weakness 5].", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- In this paper, they introduced an increasingly relevant capability: subtle visual comparison between images, across multiple domains. It contains various difference types (attribute, temporal, viewpoint, etc.) and datasets beyond natural images (industrial, medical, etc.).\n- It also has a mix of real data and synthetic setups, to show controlled evaluation capabilities.\n- The paper is easy to read, though the figures could use more text to make them clearer.", "weaknesses": "- The dataset seems to be largely a small increment of prior multi-image VLM benchmarks like MLLM-CompBench, ReMI. The claim of novelty in subtlety is only partially convincing.  Subtle differences are defined via embedding cosine similarity (DINOv3), but this does not necessarily guarantee perceptual or semantic subtlety.\n- In Figure 3, it can be seen that when the catcher moved, the other player moved as well. As the paper claims to be fine-grained, I would be interested to know how the authors ensured that in the video, only the object/person in question moved, not the other parts.\n- Repetition of the same datasets across categories. MVTEC-AD reused across “attribute” and “state” categories. \n- “Overlap” and “subtraction” images are described vaguely. It is unclear to me whether pixel-level overlay vs feature-space subtraction is used. This step needs technical clarity and justification. [line 354]\n- No evaluation on two-image native models (e.g., LLaVA Next, Clip).\n- What is domain domain-specific performance difference for each model in the benchmark? Can the author provide some insights on: are the methods bad at medical or really good in synthetic for every type?\n- Typos: line 242, archange to are", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LvEsgp2dc5", "reviewer": "Reviewer_chNu", "summary": "The paper proposes a benchmark to evaluate how well models are at discerning subtle changes between two images in form of visual question answering and captioning. The dataset covers instances from multiple domains (such as game, industrial, medical, etc) and has ten different types of differences (such as temporal, spatial, emotion, etc). Evaluation results are comprehensive including multiple popular open-source and proprietary models such as GPT-5, o3, Claude-sonnet-4, Gemini-2.4-pro along with different prompting strategies. Human evaluation is also conducted and current results indicate a gap between human performance (avg. performance 95.5) and current frontier models (best 77.8 avg performance).", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper focuses on an interesting problem setup of fine-grained changes between two images, and it is interesting how current frontier models struggle at these tasks.\n2. The paper adequately describes the dataset construction process, model evaluation setup, and experimental results. Overall, it is well written.\n3. The evaluation process studies multiple factors that can influence model performance -- such as how to combine the two images when feeding images, different prompting strategies and impact on model performance with diff. controllable percentage of change b/w 2 images.", "weaknesses": "1. This task of subtle difference changes b/w two images has been previously explored in works such as Spot-the-Diff [1], Img-Diff [2] and MLLM-CompBench [3] as noted by authors. The primary novelty seems to be expansion to multiple domains, more question types and combination of multiple choice questions and captioning in a single benchmark. In this regard, novelty is a bit limited.\n\n2. There can be further baselines/prompting strategies considered such as:\n- Calculating regions of interest from the subtraction of 2 images, and then highlighting these regions in the 2 input images (through simple bounding boxes or masks) and feed them to VLM stating regions of interest are highlighted.\n- 2-step reasoning process -- first ask the VLM to describe differences b/w the 2 images with respect to answering the question, and then feed this output in addition to the 2 images and original question.\n\nRelatively minor:\n3. It would also be interesting to study whether this task can simply be solved by training models on a mix of synthetic and real samples as the task itself might be out-of-distribution. But I understand this may be out of scope for current paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q7vllUHFOl", "reviewer": "Reviewer_Qbbf", "summary": "The paper introduces VLM-SubtleBench, a benchmark targeting subtle comparative reasoning across 10 difference types (Attribute, State, Emotion, Temporal, Spatial, Existence, Quantity, Quality, Viewpoint, Action) and 6 domains (Natural, Game, Industry, Aerial, Medical, Synthetic), comprising 11.7k (image-pair, question, answer) triplets. It emphasizes minimally different pairs (e.g., pairs with high DINOv3 similarity) and augments MCQ with a captioning track. Systematic studies span open-source and proprietary VLMs, prompt/fusion strategies, and controlled synthetic stress tests. Human accuracy is ~95.5%, whereas the best proprietary model remains well below this, with particular weaknesses in temporal/spatial/viewpoint categories.", "questions": "- Your Figure 5 synthetic-control study probes failure modes by manipulating low-level factors (e.g., brightness/scale/count/translation). Could you extend this with a color-sensitivity axis inspired by VLM’s Eye Examination [1], which reports consistent green insensitivity across VLMs, to refine the Attribute subset? Concretely, consider hue sweeps with a green vs. non-green contrast and ΔE/brightness steps, and report performance as a function of hue as well as factor interactions (color × size/count/translation). This would test whether known perceptual color deficits compound subtle comparative reasoning failures.\n\n- You clearly motivate the benchmark with high-stakes domains (industrial anomaly detection, medical imaging, aerial surveillance). Could you provide evidence of transfer, e.g., correlations between SubtleBench category scores and downstream metrics on representative datasets in those domains, or a small-scale finetuning study showing measurable gains in real applications?\n\n[1] VLM’s Eye Examination: Instruct and Inspect\nVisual Competency of Vision Language Models, https://arxiv.org/pdf/2409.14759", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Substantive contribution via task definition & data curation. Clearly formalizing subtle comparative reasoning as an evaluation target and curating a benchmark dataset with transparent collection/validation protocols is, by itself, a meaningful research contribution.\n\n- Breadth + diagnostics. Coverage of 10 difference types and 6 domains with controlled synthetic factors (e.g., brightness deltas, object size, translation, object count) supports failure-mode analysis rather than aggregate scores only.", "weaknesses": "- Data generation dependencies. Some Attribute pairs are created with Gemini-2.5 flash image preview (“nano-banana”) editing; Medical questions are refined by gpt-4o. This can introduce stylistic artifacts or distribution shifts that confound evaluation unless carefully audited. Please quantify any such effects (e.g., edited vs. non-edited subsets).\n\n- The paper identifies notable gaps in temporal/spatial/viewpoint (e.g., stable accuracy requiring ~160 px camera translation in synthetic tests), but a more explicit reporting of difficulty curves on natural data would help establish ecological validity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p9jaPLs2z5", "reviewer": "Reviewer_WLtr", "summary": "The paper introduces a new benchmark designed to evaluate whether multimodal LLMs can distinguish subtle differences between similar images (i.e., perform subtle comparative reasoning). The benchmark is comprehensive: it contains 11.7K triplets of image pairs, questions, and answers, covering ten types of differences (e.g., attribute or state) across five image domains, including natural, industrial, and medical. The authors evaluate multiple recent MLLMs, both open-source and proprietary, on this benchmark and show that subtle comparative reasoning remains a significant challenge for current models.", "questions": "- Similar to the study (Table 3) in MLLM-CompBench, what would happen if the models were first asked to analyze two images separately and then compare them using a purely language-based question, instead of being given both images simultaneously?\n\n- Did annotators label all the data? How many annotators were involved in building this benchmark, and what were their backgrounds?\n\n- It seems that only a test set is provided. Do you also have training or validation sets? It would be interesting to see whether fine-tuning on such data could improve performance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The benchmark is comprehensive, encompassing a large number of data points as well as diverse visual types and domains.\n\n- The paper identifies the remaining limitations of recent MLLMs, providing valuable insights into where the research community should focus to further improve these models.\n\n- The paper is well-written and easy to follow.", "weaknesses": "- A more detailed analysis is needed. e.g., why do MLLMs struggle more with certain types of comparisons? Why do some models perform better than others?\n\n- The paper currently reports results but lacks discussion or suggestions on how to improve subtle comparative reasoning in MLLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)