VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMs
Authors: Can Li
Ying Liu
Ting Zhang
Mei Wang
Hua Huang
Venue: ICLR 2026 Poster (2026)
Keywords: Figure-based Mathematical Reasoning
Large Multimodal Models
Mathematical Benchmark
Primary area: datasets and benchmarks
TL;DR: We introduce VisioMath, the first benchmark for evaluating mathematical reasoning with image-based answer options, revealing that current large multimodal models struggle with fine-grained visual distinctions and achieve limited accuracy.

Abstract: Large multimodal models have achieved remarkable progress in integrating vision and language, enabling strong performance across perception, reasoning, and domain-specific tasks. However, their capacity to reason over multiple, visually similar inputs remains insufficiently explored. Such fine-grained comparative reasoning is central to real-world tasks, especially in mathematics and education, where learners must often distinguish between nearly identical diagrams to identify correct solutions. To address this gap, we present VisioMath, a curated benchmark of 1,800 high-quality K–12 mathematics problems in which all candidate answers are diagrams with subtle visual similarities. A comprehensive evaluation of state-of-the-art LMMs, covering both leading closed-source systems and widely adopted open-source models, reveals a consistent decline in accuracy as inter-image similarity increases. Analysis indicates that the dominant failure mode stems from image–text misalignment: rather than grounding reasoning in textual cues, models often resort to shallow positional heuristics, resulting in systematic errors. We further explore three alignment-oriented strategies, spanning training-free approaches and finetuning, and achieve substantial accuracy gains. We hope that VisioMath will serve as a rigorous benchmark and catalyst for developing LMMs toward deeper diagram understanding, precise comparative reasoning, and grounded multi-image–text integration. The code and dataset are available at https://github.com/Nefefilibata/VisioMath.

Published as a conference paper at ICLR 2026
VISIOMATH: BENCHMARKING FIGURE-BASED MATH- EMATICAL REASONING IN LMMS
Can Li1.2.3 Ying L2u Ting Zhang Mei Wang
Hua Huang 1.2,3
1 School of Artificial Intelligence, Beijing Normal University
2 Beijing Key Laboratory of Artificial Intelligence for Education
3 Engineering Research Center of Intelligent Technology and Educational Application, Ministry of Education
ABSTRACT
Large multimodal models have achieved remarkable progress in integrating vi- sion and language, enabling strong performance across perception, reasoning, and domain-specific tasks. However, their capacity to reason over multiple, vi- sually similar inputs remains insufficiently explored. Such fine-grat mre-grained compar- ative reasoning central real-world tasks, especially in mathematics and edu- cation, where learners must often distinguish between nearly identical diagrams to identify correct solutions. To address this gap, we present VisioMath, a cu- rated benchmark of 1,800 high-quality K-12 mathematies problems in which all candidate answers are diagrams with subtle visual similarities. comprchensive evaluation of state-of-the-e LMMs, covering both leading closed-source systems and widely adopted open-source models, reveals a consistent decline in accuracy inter-image similarity increases. Analysis indicates that the dominant failure mode stems from image-text misalignment: rather than grounding reasoning in textual cues, models often resort to shallow positional heuristics, resulting in sys- tematic errors. We further explore three alignment-orienn oriented strategies, spanning training-free approaches and finetuning, and achieve substantial accuracy gains. We hope that VisioMath will serve as rigorous benchmark and catalyst for devel- oping LMMs toward deeper diagram understanding, precise comparative reason- ing, and grounded multi-image text integration. The code and dataset are avail- able at https:::: : /github. com Ceemm/eeefiifi
INTRODUCTION
In recent years, Large Multimodal Models (LMMs) (Chen et al. 2025 OpenAl| 2024 Team 2024a
Wang et al. 2024c; Wu et al. 2024b) have achieved remarkable success across various multimodal tasks. This surge in capabilities is largely attributed to the availability of massive, high-quality vision-and lisionannd-aanguage datasets (Chen al. 2023 He et al., 2023 Kuznetsova et al., 2020 Singla et al. 2024). which enable the training of increasingly capable models. By jointly modeling image
and text modalities, LMMs enable seamless cross-modal reasoning, allowing for the interpretation of complex visual scenes in natural language and vice versa. This integration not only enhances basic
perceptual capabilities but also supports high-level cognitive tasks such as visual recognition (Chen
et al.
2024b; Huang al. 2024 Wang et al., 2024d), logical reasoning (Wang et al. 2024e Wu
ct al.
2024a
Xiao et al. 2024), and context understanding (Zhang et al. 2024a).
With the rapid development of LMMs, designing holistic benchmarks is essential for systematically investigating the capabilities and limitations of these models. Numerous evaluation benchmarks have been proposed, targeting different aspects of LMM performance, including perception, rea- soning, domain-specific tasks, hallucination, and multimodal integration (Huang & 'Zhang [024]] Li et al. 2024c). Among these, multimodal reasoning ability, particularly mathematical reason- ing that requires integrating visual and textual information, has always been a central focus. This form of reasoning presents distinct challenges, requiring not only the understanding of mathematical semantics in text but also the accurate interpretation and synthesis of visual representations.
Cooresponding author.

---

Published as a conference paper at ICLR 2026
Stem without images (option-only)
Stem with multiple images
English
Options:
English
Among the following four figures, which one is the net of'a triangular prism? ()
A
As shown in the figure, there is machine part placed (Figure 1). its front view as shown in (Figure 2). then which one its top view?
Chisese
F E
Chinese
de St
Stem with single image
English
Options:
Given graph the function y- shown in the figure. which of the
Figurel
Figure2
Options:
Jollowing could be the graph of 21x by
10
Chinese
B
Aa Raxt
da Bly
Figure I: Illustrating examples our VisioMath dataset, in which data samples consist of visual answer options exhibiting high visual similarity, and the stem may appear with or without images.
To evaluate multimodal reasoning capabilities, various multimodal mathematical reasoning bench- marks have been introduced (Lu et al. 2024: Zhang et al.] 2024b, Wang et al. 2024b). These benchmarks can be broadly divided into two categories. The first involves single- image scenario, where each problem is paired with single diagram that supplements the text. While effective for assessing basic multimodal understanding, these setups are limited in capturing the complexity of real-world visual reasoning, single image often lacks the richness and inter-image dependencies needed for higher-order comprehension. In response, recent studies have shifted toward the second category: multi-image scenario. These tasks require reasoning across problems with multiple visual inputs. This paper also investigates on multi-image scenario with a particular emphasis on a specific and underexplored setting: reasoning over multiple highly similar images.
In this paper, we examine distinct class of multimodal benchmarks in which all answer choices are presented as images. Our motivation arises from the observation that many real-world mathematical problems, especially in educational settings, present options as diagrams (e.g. geometric figures). Addressing such problems involves more than visual recognition; necessitates comparison of vi- sually similar structures and reasoning about subtle symbolic differences. While recent benchmarks such as CMM-Math-tess (Liu et al. 2024b). Verse-mr (Li et al. 2024b), and MV-Math (Wang ct al. 2025b) have advanced the evaluation of multimodal reasoning by introducing multi-image questions, they often overlook a crucial aspect where reasoning must be grounded in perceptually similar visual features. Our work aims to address this gap and thereby provide evaluation per- spective that specifically targets LMMs' reasoning across closely resembling images.
To achieve that, introduce VisioMath, novel benchmark comprising 1,800 meticulously cu- rated, high-quality mathematics problems. The dataset spans a broad spectrum of K-12 mathematics topics, including geometry, algebraic visualizations, numerical comparisons, and functional pattern recognition, thereby capturing the diversity of real-world curricula. The focus on K-12 mathemat- ies is deliberate: fine-grained comparative reasoning is prevalent in this domain, where students must distinguish nearly identical diagrams to identify correct solutions. This makes VisioMath not only an ideal benchmark for evaluating LMMs' visual-textual grounding capabilities but also di- rectly relevant for improving their potential to support K-12 tutoring and educational applications. Specifically, each problem features diagrammatic answer options, with approximately 50% also in- corporating least one image in the question stem to provide essential visual context. To ensure accuracy and reliability, each question has been independently annotated and cross- calsddvalddted by at least expert annotators. To reduce answer-choice bias in LMMs, we enforce uniform distribu- tion across the four multiple-choice options (A, B, C, D). As shown in Figure I each answer option is distinct diagram differing subtly from the others, requiring fine- grained visual discrimination.
We conduct a comprehensive evaluation on the VisioMath benchmark. Our study encompassed a diverse set of LMMs across various model families and scales, including state-of-the-art closed-
2

---

Published as a conference paper at ICLR 2026
entrance
Image Process
questions
Cropping
Embedding
(inage, sim)
durin
PDF PDF
PDF
Lable
Labli
Manual review
VisioMath
Sim((Q)
duplicates X flawed items
Text Process
J one image per option
Question
HI+H++
Question
{i} JSON
A.
C.
stem
Extract
LATEX
Check
(Structured text)
Figure 2:
Data processing pipeline of VISIOMATH, including text extraction and verification, image cropping, and integration of visual similarity information to construct the final dataset.
source models such GPT-4.1 (OpenAl 2025) and Gemini2.5 Pro (Comanici al. 2025), well as prominent open-source models like Qwen2.5-VL. (Bai et al. 2025). These models span differ- input paradigms, with some restricted single-image input and others capable of processing multiple images simultaneously. We perform a detailed error analysis and find that image-texx text mis- alignment accounts for the largest proportion. This highlight fundamental overlooked limitation in current LMMs: their inability to reliably establish fine-grained correspondences between multi- ple images and distinct textual inputs. tasks such image-opiin option problems, where each figure must be uniquely paired with specific textual option, LMMs often fail to preserve these one-to- one mappings. This weakness indicates that, although LMMs excel single-image reasoning and holistic multimodal understanding, they remain inadequate when tasks demand precise cross-modal alignment across multiple visual-ter. pairs.
We further explore three complementary strategies aimed at mitigating image text misalignment and enhancing multi-image reasoning: consolidating multiple images into a single layout, establishing explicit visual-textual anchors, and fine-tuning with alignment-oriented multi-image chain-of- thought dataset. Notably, such limited Chain-of- (oaooooffToogg((Cooo) fine-tuning data yields a substantial accuracy gain (+12.6%), illustrating the critical role of explicit visual-textual alignment in enabling effective multi-image reasoning. We hope our work will motivate more systematic exploration of methods for enhancing multi-image text alignment in complex reasoning tasks.
In summary, our key contributions are:
VisioMath Benchmark. We introduce VisioMath, the first benchmark specifically de- signed for image-option mathematical reasoning. bridges the gap between traditional multimodal visual question-answeriin benchmarks, providing rigorous testbed for evalu- ating LMMs' diagram understanding and fine- mne-ggrinedd visual reasoning.
Comprehensive Evaluation. We systematically evaluate wide range of tate-of-thhear LMMs, including GPT-4.1 and Gemini2.5 pro, and reveal that even perffrrrngn models struggle with reasoning over visually similar answer options, highlighting critical limita- tion when dealing with complex reasoning requiring multi-image-tex alignment.
Analytical Strategies. We perform detailed error analyses to identify key failure modes, design controlled experiments to validate the critical limitation, and introduce alignment- focused strategies that substantially improve figure-base based reasoning performance.
2 VISIOMATH
Motivation. In mathematics education, multiple-choice questions with diagrammatic answer op- tions are pervasive. These diagrams often exhibit high visual similarity, differing only subtle ge- ometric structures functional eurves. Humans reliably leverage such fine-grained differences through prior knowledge and structured reasoning. In contrast, LMMs typically rely on superficial embedding similarity, making it difficult discriminate between nearly identical options.
Routine for students, this setting remains unexpectedly challenging for LMMs. As illustrated in Figure the four candidate diagrams share almost identical visual styles, yet solving the problem

---

Published as a conference paper at ICLR 2026
Proparscy Probubiliy Denely
Median 0.55
22 on (196) Soid Grersery
Transleantion Gammety
22.5% (400)
40.%%
Probubiiiy Denaty
Froquancy
Frequancc
(7.7%
22.6% (436)
Desrrpiiee Lecmetry
11% (63) Gevnetty
Othere (1%)(CT) Unwweyy Plemoen (Toms
04
A
Question text length
Visual Similarity
(a) Distribution of subjects.
(b) Distribution of question text length.
(c) Distribution of visual similarity.
Figure 3: Detailed statistics of the VisioMath dataset. The figure shows distributions of (a) subject, (b) question text length and (c) visual similarity, highlighting both textual and visual characteristics.
requires aligning the textual description with precise visual interpretation. To capture this ubiquitous but underexplored scenario, we introduce Visio.ath, benchmark explicitly designed to evaluate LMMs' reasoning ability over multiple highly similar image options mathematics.
2.1 BENCHMARK CONSTRUCTION
Building on the motivation introduced above, VisioMath is constructed to faithfully reproduce exam- like scenarios where reasoning hinges subtle visual distinctions. To this end, during construction we follow three design principles, representativity reliability, and high visual similarity. The overall construction pipeline illustrated in Figure 2
Representativity. VisioMath contains 1,800 multiple-choice questions with 8,070 diagrammatic options, collected from Chinese high school and college entrance examinations administered be- tween 2002 and 2023. Using real exam items ensures external validity: the benchmark directly reflects the types of problems students actually face, and its results can generalize real educa- tional scenarios. Each problem is paired with option diagrams as well as stem diagrams (average 4.48), and we intentionally balanced the correct answer distribution across A-D (24-26% each) to eliminate positional bias. The question text length average 61.5 tokens, reflecting moderate linguis- tic complexity as shown in Figure (b). We also present the distribution of subject areas across the dataset Figure (a), offering integrated overview of the benchmark's 's coverage.
Reliability. To ensure that evaluation results reflect genuine reasoning ability rather than spurious cues, we standardize and curate all samples. Question texts were digitized into consistent JSON format, where mathematical expressions were transcribed into LaTeX to guarantee uniform parsing. Answer diagrams preventing layout or formatting artifacts from providing shortcuts. Finally, all items underwent man- ual review eliminate duplicates, low-quality images, and conceptually flawed questions. These steps establish dataset that is reliable to faithfully evaluate the visual reasoning ability of LMMs.
carefully cropped from PDFs enforce a strict image per option rule,
High Visual Similarity. A distinctive property of VisioMath lies in its systematic quantification of visual similarity among answer options. For each question Q, option images are encoded using the Qwen multimodal-embedding.. vl model, and the question-level visual similarity then defined as the minimum pairwise cosine similarity across all encoded images:
Sim Q) = min cos (fxx),
(1)
where f()) denotes the image embedding encoder.
As illustrated in Figure 3 (c), a large proportion of VisioMath problems contain highly similar op- tions, creating fine- grained distinctions that especially challenging for LMMs. Importantly, preserve the full spectrum of similarity levels rather than filtering out low-similarity cases, so that performance be systematically compared under different similarity regimes.
4

---

Published as a conference paper at ICLR 2026
Table Comparison between VisioMath and existing evaluation datasets. Here, EN and CN denote English and Chinese, respectively; FO refers to figure-based options; and Avging indicates the average number of images for each problem.
Datasets
Multi-image problem
Language
&Problems (FO)
#Prrolems
#lmages
Avging
We Math QQio el al. 2024
EN
6500
6500)
MMMU-MMi
2024
EN
540
540
Mast-Vista
La of 2024
EN
6141
6141
Math-Vorse
Kllant
2612
Math Vision
Wang di 21144b
X
EN
3040
3040
et
AT24]
5,929
CMMU-MATH
2024
X
CN
778
778
MahhExplain
Park al 20255
EN
997
997
MathGlance
EN
1,609
1,609
Gaokao-MM-MMath (Zong & Quu
2024
CN
17
80
142
1.78
CMM-Math-iest [Lu
3794
2.26
MathVeree my [b al
2124h
EN
788
6304
8
(Wang ct 302Nb
3.02
VisioMath (OWS))
CN.EN
1800
1800
8070
4.48
2.2 BENCIMARK ANALYSIS
Unique Challenges. VisioMath introduces a set unique challenges that distinguish it from ex- isting multimodal benchmarks. Unlike conventional tasks that pair a single image with text, Vi- sioMath requires reasoning across multiple diagrammatic options simultaneously, transforming the problem into one of comparative visual reasoning that mirrors authentic exam scenarios. Moreover, the benchmark faithfully preserves the presence of highly similar distractors, where candidate dia- grams differ only in subtle geometric symbolic details, thereby testing models' capacity for fine- grained perceptual discrimination. Finally, effective problem solving VisioMath demands precise text-visual alignment, models must ground linguistic conditions such symmetry, monotonicity, or functional transformations in the correct image choice. Collectively, these characteristics elevate rigorous evaluation of figure-based visual reasoning.
VisioMath from simple image recognition
Benchmark Comparison. We compare VisioMath with prior multimodal math benchmarks in Table [ Most existing datasets adopt single-image setting with textual answer options (e.g., We-Math, MMMU, Math-Vise Vista, Math-Verse Math-Vishie Vision). Multi-image formats are rare, and when present, image-based options are either absent or inconsistently represented. For instance, Verse-emmmms mv includes multiple images but no image in options. CMM- and MV-Math contain some image-based based options, yet many are embedded in composite layouts rather than pro- vided as independent visual elements. VisioMath, in contrast, explicitly structures answer options as collections of distinct and semantically meaningful images, thereby supporting more nuanced evaluation of fine-grained visual mathematical reasoning.
EXPERIMENT
Setup. To comprehensively evaluate the performance of LMMs in handling complex visual in- puts, we select a diverse models across different accessibility types and input configurations. Specifically, we include closed-source LMMs representing the current state-of-the-an in commer- cial multimodal systems. addition, we conduct experiments on open-source LMMs that explicitly support multi-image inputs with various model sizes. This broad coverage ensures representative analysis across model capacities and architectures. Finally, we include LMMs that have been specif- ically trained on mathematical corpora and optimized for mathematical QA tasks. All LMMs are evaluated under zero-shot setting to ensure fair and consistent comparison of their generalization capabilities. More details are provided in Appendix A.
3.1 RESULTS
Table reports the comparative performance of various LMMs on VisioMath benchmark, with re- sults disaggregated by the ground-trute (GT) answer position (A-D). The evaluation considers two distinct conditions: (i) question stems presented without images and (ii) question stems accompa- nied by images. For each condition, provide both average accuracy and per-option performance.
5

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison on VisioMath with results categorized based on GT position.
Question stem wlo images
Question stom with images
Models GT position
Avg
Avg
A
B
C
D
Avg
A
B
D
Human Random
91.3
92.3
92.5
95.6
93.8
88.5
89.7
94.4
87.6
87.5
88.0
25.6
25.4
24.0
25.6
23.0
28.6
26.0
22.8
27.6
28.4
25.6
Closed
LMMs
QwenVL.mma max (Bai et
2025
44.1
53.4
35.2
62.6
62.5
50.2
34.1
31.1
34.1
32.8
38.6
GPT4.1 (ppenA (T25)
52.6
61.6
72.4
59.9
60.2
56.1
42.8
54.8
39.3
43.7
31.9
Soed TTiikiingg BytcDance
2024
72.3
85.7
90.3
87.2
82.4
83.9
58.0
71.8
53.7
59.4
Gemini 2.5 Pro Commmici el 2025
80,9
86.2
89.2
84.6
85.2
86.3
75.2
78.8
77.6
75.0
68.6
Open-source LMMs (mulli-tnage (mpot))
L.2.mVL2..528 Chen
24.6
27.1
12.8
25.5
36.3
30.2
21.9
10.3
26.2
15:0
Dwen2.5-VL.-33
- Ban et
instrucl
2025
25.4
26.1
51.0
40.5
14.5
5.9
24.7
18.3
70.1
5.4
4.3
Onevison 7B Yang 21
29.6
35.0
30.2
23.7
22.0
1e-m2-5-VL-7B-insttrr Mai ml
2025
32.7
39.5
30.1
58.1
39.8
29.8
25.3
8.7
28.5
32.4
34.3
Gemma3-27B
[Temm 2025g)
35.3
43.7
67.9
40.1
33.6
38.4
26.2
40.2
24.8
12.3
25.1
Vision-RI
Huang
2025
36.7
43.7
47.4
57.3
38.7
33.7
29.2
24.5
52.3
29.4
10.6
Qwen2.5 VL 7211 instFrctt
[aa et
2025
43.7
53.5
36.2
63.9
61.3
49.8
33.0
29.9
37.8
29.9
35.2
GLM-4.5V
Team et 2025
53.7
69.1
71.9
75,8
68.4
61.2
37.2
46.5
42.5
31.4
26.6
Open-Source LMMs
(math-oriented)
MM. P4M-PRMBB Du al. 2025
31.7
38.4
28.1
43.2
44.9
35.7
24.4
10.8
41.6
35.3
11.6
MM-Eureka- (Meng
1205
37.9
50.9
36.2
62.1
52.7
50.1
24.0
21.1
22.4
27.4
25.6
MMM-Burkka-7B-CPUU (Lu St 2025
39.3
51.0
33.2
54.2
61.3
51.4
26,9
16.2
29.9
39.7
23.7
Table 3 further details the accuracy of LMMs on subsets of the dataset stratified by image similarity levels. The dataset is divided into quartiles based on the degree of visual similarity between images within each question, and model performance is reported separately for each quartile. This analysis aims to evaluate models' fine mne-grained reasoning capabilities under varying visual similarities.
Based on these results, we have following observations.
Observation 1 (Question stems containing images pose greater challenges for LMMs). As shown in Table 2 most LMMs demonstrate noticeably lower performance on questions whose stems in- clude images compared to those with text-only stems, which is trend consistent across nearly all positions. This observation suggests that the inclusion of images in the question stem significantly increases the complexity of the visual reasoning task. Specifically, when both the stem and the op- tions involve visual content, LMMs are required to process and integrate multiple sources of visual information, which likely imposes a higher cognitive load on the model. This indicates that current LMMs still struggle with multi-image reasoning scenarios and highlights potential bottleneck in their capacity for holistic visual understanding.
Observation 2 (Performance degrades under high visual similarity). LMMs exhibit performance degradation on tasks involving high inter-image similarity, as shown in Table 3 For instance, Doubao-1 1.5- Vision achieves 74.9% accuracy in the quartile with the lowest similarity, but this drops to 62.0% in the highest-similarity quartile, a 12.9% decline. This performance gap stems from the increased demands for fine- tme-grained cross-image associative reasoning, which current LMMs in- sufficiently support due to limited visual granularity and reasoning capabilities. Notably, LMMs exhibit strong performance correlations across similarity quartiles: models performing well in low- similarity scenarios tend to retain relative strength under high similarity.
Observation 3 (Distinct failure modes in Human and LMMs). As shown in Table human perfor- mance moderately decreases as visual similarity among candidate diagrams increases, confirming that higher similarity introduces additional perceptual challenges. Second, beyond certain similar- ity threshold, the accuracy plateaus, suggesting that errors at this stage are driven more by concep- tual understanding than by perceptual similarity. This suggests that while high similarity increases perceptual load, humans can still reliably distinguish fine-grained differences through careful ob- servation. In contrast, LMMs frequently fail on perceptually trivial distinctions that humans rarely confuse. as illustrated in the eтror analysis Figure 9 This disparity indicates that current model failures stem largely from inadequate iisual-textual alignment rather than a lack of reasoning depth.
3.2 ANALYSIS
Error Categorization. We conduct a systematic error analysis of GLM4.5V to better understand the limitations of LMMs on VisioMath. From the model outputs, we randomly sample 50 erro

---

Published as a conference paper at ICLR 2026
Table Performance comparison on VisioMath with results categorized based on image similarity.
Models |magg similarity
Avg
[0. (6.0666688
(0.68.0877)
(0.87.0.96)
(0.96.1)
Human
91.3
95.7
91.2
87.6
89.0
Random
25.6
23.6
24.4
27.8
27.1
Closed-source LMMs
QwenVT_-max (Bai
2023
44.1
47.3
50.2
41.3
37.6
GPT-4.1 [OpenA] 2025
52.6
56.4
42.9
45.1
Seed1.6 Tiininn Bavce
2024
72.3
82.4
74.2
66.2
66.4
Gemini 2.5 Pro Comanici of a
2025
80.9
86.2
83.8
76.7
76.9
Open
image input)
Intem 2BB Chen et 2024a
24.6
24.2
28.9
22.7
22.7
Qwen2.5 VL mntricc [Bhh 21
2025
25.4
26.7
27.6
24.4
22.9
R1-Onevison-78 Yang
29.6
32.2
28.9
Qwea2.5-VL-4 NLL-BB------ [5gh Siiiet <t
2025
32.7
33.6
37.8
29.8
29.6
G-mma3-278
Team 20255)
41.2
26.4
Vision-R1-78
Huaing Cf 2025
36.7
46.7
38.9
30.4
30.9
Qwen2 V-
2025
43.7
47.1
50.8
38.0
38.7
GLM-4.5V
Team el al.. 2045
53.7
68.7
59.3
44.2
44.7
Open-sourre LMMs (marl-oriented)
MM-PRM-8B (Du
2025
31.7
37.6
37.1
26.9
25.1
MM. Eureka 70 [enn
2025
37.9
45.6
44.0
29.1
33.1
MM-Fureka-7B8 7B-CDG
2025
39.4
47.8
46.0
30.9
32,9
neous cases and manually inspect their characteristics, and we categorize the errors into four types, with their proportions illustrated in Figure 5 (a) (mode details in Appendix D). Among the iden- tified categories, image-text misalignment account for the largest shares, representing 36% of the errors. Compared to single-image datasets such as MATH- Vision, these proportions are significantly higher. This finding highlights that reasoning over multiple visual contexts introduces substantial challenges, particularly in maintaining consistent semantic alignment across both images and text.
Effect of Option Shufffing. The image text misalignment errors suggest that current LMMs rely heavily on heuristic positional correspondences between options and images. To investigate this, we conducted a controlled shuffling experiment: the image order was kept unchanged, while the textual references to the options were permuted. For example, the original prompt "The last four pictures are respectively the pictures for options A, B, C, and D was modified to "The last four pictures are respectively the pictures for options B, C, D. and A." with the ground-truth truth answers adjusted accord- ingly. By keeping the image order constant, we isolate the effect of image order on performance. Results shown in Figure (b) suggest consistent and clear decline under this manipulation. For instance, Gemini 2.5 Pro accuracy dropped from 80.9% to 72.2% (-8.7%). These findings indicate that existing LMMs struggle robustly capture and align semantic correspondences between textual options and visual content, highlighting the need for improved cross-modal alignment mechanisms in multi-image reasoning tasks.
Rationale for Visual Similarity Metric. We define the visual similarity of a question, Sim(() QQ), using the minimum pairwise cosine similarity (Smin) computed by Qwen nultimodal-embeddin vI. Here we study the effect of the aggregation strategy and the visual encoders.
The Aggregation Strategy. We use Smin to establish striet lower bound for visual discrimination difficulty. Two alternative measures are the mean similarity and the median similarity. Comparing Swin with mean and median similarities reveals highly aligned quartile boundaries (Figure[4(a) and consistent model performance trends-sppeificann specifically, uniform accuracy drop in the highest similarity quartile (Q4) across all methods (Table 4 This confirms Smin is a statistically robust metric for structural dataset grouping.
The Visual Encoders. Unlike CLIP (optimized for natural images) and BLIP (general QA), Qwen is trained extensively on diagrammatic reasoning, making it uniquely suited for capturing fine-grained semantics in math diagrams. Nearest neighbor retrieval experiments (Figure 4 (b)) validate this choice: Qwen maintains strict geometric and topological consistency (e.g.. slope, intercept). Con- versely, BLIP suffers from geometric drift after the top-1 result, and CLIP prioritizes high-level semantics over precise visual topology, rendering both unsuitable for fine-grained differentiation.

---

Published as a conference paper at ICLR 2026
(a) Score distribution of minimum, mean, and median similarity across all problems.
(b) Top-5 retrieval results with Qwen multimodal- embedding-vl, BLIPv2, CLIP.
Figure 4: Visual similarity analysis: (a) distribution of different similarity metrics; (b) retrieval results using different visual encoders.
Table 4:
Performance comparison on VisioMath with results categorized based on image similarity quartiles (Q1-Q4) defined by Minimum (min), Mean (mean), and Median (med) aggregation.
Models Imagee similarity
Q1
Q2
Q3
Q4
min
med
min
med
min
med
med
V-enVLs (Bai
2023
33.3
35.6
35.1
37.8
35.1
35.8
32.4
32.9
32.0
28.2
28.9
GPT-4o (ppen4 6024
53.8
56.0
55.8
50.9
49.5
50.0
40.0
40.0
40.0
39.1
39.1
40.0
Gemini2 Bash lhinking (DeepMind
56.7
Gemini2-flash (DeepMini MIAAT
66.7
69.1
70.4
59.8
56.8
55.1
49.3
49.1
49.5
46.2
45.5
45.5
Doubao-1.5- V.Viiio Teem
2025g
74.9
75.8
77.1
68.2
68 2
65.7
60.4
61.1
61.1
62.0
60.4
61.5
Seed1.6-Thinne T-iiikii (Baiiikinng(BiiDDDDa (ByieDiance
7024
82.4
84.0
82.9
74.2
72.4
74,2
66.2
67.6
66.4
67.6
66.4
66.2
Gemini 2.5 Pro (Comannee al
2025
86.2
89.3
88.6
83.8
82.6
84.0
76.7
76.7
76.7
76.9
76.9
76.9
Gemma3-27BB [Lean]
43.3
29.6
26.2
26.2
Qwen2.5 "1L7225 7.725-mmst Inscrr (Ba et
2025
47.1
50.2
49.3
50.8
46.9
48.4
38.0
39.1
38.2
38.7
38.7
38.9
GL.M44.5 Team 2025
68.7
54.9
46.7
3.3 STRATEGIES FOR PERFORMANCE ENUAACCMEET
Building on the above analysis of LMM limitations, we explore practical strategies to improve multi- image reasoning performance VisioMath. These strategies fall into two cafegories: training-free techniques that leverage structural or labeling cues, and training-based approach that incorporates specialized multi-image reasoning data. Collectively, they demonstrate the potential mitigate vision-text misalignment and enhance cross-figure reasoning.
Strategy (Consolidated single image layout). We first examine whether providing all visual infor- mation single spatial layout improves reasoning. Specifically, option images and stem images concatenated into composite image. As shown in Figure 5 (b). this structural simplification consistently improves performance, suggesting that LMMs struggle to distribute attention effec- tively across multiple independent images. For instance, Seedl. beedd.6-Thinnle T-Tnking achieves accuracy increase from 72.3% to 78.7% (+6.4%) under this setup. The results indicate that co-locating visual information helps LMMs reason
effectively over multiple images.
Strategy 2 (Explicit visual-textual anchors). this strategy, each image is directly associated with its corresponding textual label, either through overlaid embedded annotations. This experimental setting is designed to evaluate whether establishing explicit visual-textual correspondences can en- hance disambiguation and support more accurate decision-making Empirical results shown in Fig- ure (b) demonstrate that this approach yields notable performance gains: for instance, VL- plus improves from 32.9% 42.7% (+9.8%), whereas Gemini 2.5 Pro shows smaller but mea- surable gain of +0.9%. These results indicate that current LMMs continue to struggle with robustly binding textual content to the corresponding visual elements. Importantly, the findings highlight that carefully designed visual-textual anchors can effectively mitigate misalignment errors, offering a practical pathway to improve multimodal reasoning performance.
Strategy 3 (Alignment--rrennted multi-image thuin-of-thought training). To further enhance reason- ing performance, we develop a specialized multi-image CoT dataset explicitly aimed at improv- ing visual-textual alignment across multiple diagrams. Starting from 1,072 multi- problems with an image-caption style prompt to generate
collected online, we first employ preliminary reasoning paths that describe each diagram individually, ensuring localized alignment
8

---

Published as a conference paper at ICLR 2026
Sewwey Longle tosge!
20.0%
Sinsgy 2eoplitrandee
36.0% Iaage Misdiggmmnt
-0.5
+0.0
20.5
34.0% Viekre Resagaition
10.0%
Knowlodac Everr
(a) Distribution of error types.
(b) Performance LMSs actoss Original, Shufiing, and Stralegy Variants.
Figure Illustrating error type distribution and the impact of input data structure on performance.
Table 5: The effect of strategy 3, using alignment- oriented multi mullt--mmaee chain-of thouhht fine-tuning.
Model
Original
Shuffling
Strategy
Strategy
Strategy 3
Swen2.5-VL.33B-inttrue (Bai al.
2025
25.4
23.4 (-2.0)
30.9 (+5.5)
28.8 (+3.4)
38.0 (+12.6)
Qwen2.5-VL.-P7 -i.n----- (Bai et al.
2025
32.7
26.1 (-6.6)
38.1 (+5.4)
42.5 (+9.8)
43.3 (+10.6)
Qwen2.5-VL-77 --7t-----ttt (Bai el al.
2025
43.7
35.6 (-8.1)
47.6 (+3.9)
50.1 (+6.4)
51.4 (+7.7)
IntemmL22.522 (Chen et al. 2024a)
24.6
23.1 (-155
26.3 (+1.7)
27.2 (+2.6)
32.2 (+7.6)
between visual elements and textual explanations. These initial outputs are then refined by DeepSeck V3.1 through CoT Data Generation Prompt, which enforces step-by-step integration of per-image descriptions into a globally coherent reasoning trajectory, tightly binding visual observations to tex- tual inferences. To guarantee reliability, only samples yielding correct final answers are retained, resulting in 500 high-quality multi-image CoT exemplars with explicit visual-textual anchors. We fine-tuning Qwen2.5-VL with various model sizes and Intern VL2.5-22...- and the results Table show that accuracy increases for all models, despite using only a small set of CoT data. Notably, Qwen2.5-VL-3B increases from 25.4% 38.0% (+12.,%), surpassing R1-One Vinionni (29.5%) and Vision-R1 (36.7%). These results highlight that current models are severely constrained by the scarcity of alignment-oriente multi-image CoT training data, and that targeted augmentation with explicit alignment signals can substantially boost figure-based reasoning.
4 RELATED WORK
Multimodal Understanding Benchmarks. Numerous benchmarks have been introduced to eval- uate the capabilities of LMMs. While several multi-image benchmarks, such as Blink (Fu et al.) 2024]. MUIR (Wang et al. 2024a). and MMIU (Meng et al. 2024), have emerged, they primarily assess basic perceptual abilities like caption recognition and object coanttinn-laliingg short in mea- suring deep reasoning. Simultaneously, evaluating the genuine dependency visual information remains a critical challenge. Recent studies have moved beyond simple natural image QA to multi- modal science problems requiring deeper reasoning capabilities: MMSciBench (Ye et al. 2025 and SeePhys (Xiang et al. (0255) introduce the concept of "Vision-Esseniiae Essential" problems to reveal models' reliance on textual shortcuts, while VisAidMath (Ma al. 2025) explores active visual reasoning through auxiliary line generation. Unlike prior works that mostly focus on interpreting single stem or generating aids, Vasho.ath introduces a challenging benchmark centered on comparative visual discrimination via fine-grained, multi-image comparative reasoning over visually similar diagrams.
images, enabling a more comprehensive assessment of models'
Mathematical Reasoning Benchmarks. Various datasets have been proposed to evaluate the mathematical capabilities. Text-based benchmarks such as GSM8K (Cobbe et al. 202 and MATH (Hendrycks al. 2021) are widely used. To evaluate mathematical reasoning requiring vi- sual understanding, such as geometry and function graph analysis, several multimodal datasets have recently emerged, for example, Math- Verse (Zhang al. 2024c). Math-Vista (Lu et al.! 2024), and Math- Vish-Visoonn (Wang et al. 2024b). Nonetheless, as LMMS advance in iaati-iimge reasoning, these single-image- benchmarks increasingly inadequate for evaluating their full capabilities.

---

Published as a conference paper at ICLR 2026
In response, recent research efforts have begun to explore more complex multi-image reasoning narios that better reflect the real-world demands mathematical mroblem-solvingg Despite recent advances, key limitation persists: existing
benchmarks such Math Verse-mr (Li) et al. 2024b) and MV-Math (Wang et al., 2025b) often neglect figure-based answer options, which common in mathematics domain g.. geometry problems with diagrammatic options). This gap underscores the need for new benchmarks that support figure-based multi-image reasoning.
5 CONCLUSION AND LIMITATION
We introduce VisioMath, a benchmark designed to evaluate multimodal reasoning in contexts where answer options consist of multiple, highly similar diagrams. This benchmark fills critical gap in existing evaluation frameworks, which rarely consider the challenges of comparative reasoning across visually confusable candidates. Our experiments reveal that current LMMs perform poorly under these conditions: accuracy declines sharply with increasing inter-image similarity, and fre- misalignment. Controlled shuffling experiments further
quent errors stem from multi-image- show that many models rely on positional heuristics, exposing fundamental weaknesses in their reasoning mechanisms. We further explore alignment-oriented data augmentation and multi-image CoT finetuning, Results demonstrate that these strategies yield substantial gains, even under limited data regimes, indicating that relatively lightweight interventions can enhance L.Ms' capacity for robust visual-textual binding.
While VisioMath provides a rigorous evaluation of multi-image, diagram- flagram-based reasoning mathe- matics, our current benchmark is limited to K-12 math topics. Extending this benchmark to other do- mains, such as physics, engineering diagrams, or chemistry molecular structures, would test LMMs' ability to generalize multi- ialtt reasoning across diverse visual-semantic contexts.
6 BROADER IMPACT
VisioMath highlights critical limitations in current LMMs, particularly in fine-grained visual-text alignment and figure-based visual reasoning. By providing targeted evaluation platform, it can guide the development of more accurate multimodal models, benefiting educational applications, intelligent tutoring systems, and diagram understanding STEM disciplines. However, as with any benchmark, there is a risk of overfitting models its specific structures; care must be taken to ensure that improvements reflect genuine reasoning capabilities rather than dataset-specific heuris- tics. Overall, we envision VisioMath supporting both model development and pedagogical research, fostering AI systems that can more effectively interpret and reason over complex visual information.
ACKNOWLEDGMENT
This work was jointly supported by the National Natural Science Foundation of China (62437001), the Fundamental Research Funds for the Central Universities (2253500001, 2243100004).
ETHICS STATEMENT
This research based on a dataset compiled from publicly available papers from Chinese high school and college entrance examinations administered between 2002 and 2023. All data were sourced and processed in compliance with applicable laws and institutional regulations. During the curation process, we implemented systematic filtering protocol identify and remove any potentially harmful, offensive, or otherwise inappropriate content. Consequently, the final dataset used in this work is considered ethically sound and suitable for academic research.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, provide comprehensive documentation of our method- ology. Section 2.1 details the construction and annotation of our dataset. Appendix A documents the experimental settings, model version, concrete implementations of Strategy and Strategy 2, and all
10

---

Published as a conference paper at ICLR 2026
hyperparameters used for fine-tuning and evaluation. The construction of our CoT reasoning data is described in Appendix B, and the full set prompts for both generation and evaluation is available in Appendix C. Together, these materials enable independent researchers to replicate our evaluation and reproduce the reported results.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, and et al. Phi-3 technical report: A highly capa- ble language model locally on your phone, 2024. URL hattps://arxiv.or///aa 14219
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond, 2023. Jtttps:::
org/ abs/ 2308. 12966
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Siho Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiann Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint Xrvi: 13923, 2025.
ByteDance. Seed1.6 .6 tech introduction, 2024. URL httph:tt/ttsssareeeeeeess www.:eeeearrcc..ubooooooo com seedl_ 6
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions, 2023. URL https:
/ //axxiv.or//abbat 22317 12793
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Yc, Hao Tian, Zhaoyang Liu, Lixin Gu. Xuchui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang. Tan Jiang, Bo Wang, Conghui Hc, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance bound- aries of open-source multimodal models with model, data, and test-time scaling, 2025. URL https://arxivy org/ abs/ 24 412 05271
Zijian Chen, Wei Sun, Yuan Tian, Jun Jia, Zicheng Zhang, Jiarui Wang, Ru Huang, Xiongkuo Min, Guangtao Zhai, and Wenjun Zhang. Gaia: Rethinking action quality assessment for ai- ai-generateed videos, 2024b. Utthttps: /tl:://arree //axxvv.v... org/ abs /2w/22066222 06087
Karl Cobbe. Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1r.iv:21 14.166 2021.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, and Ice Pasupat. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. Bolihttss://aret wwwpvvi.rrr//.o
org/ abs////////// waabs/250..0.m ottpoo//////b///..025555 25612666776661111
Google DeepMind. Gemini 2.0 flash, 2025a.
URL
[technolooiies/gmmnnnii /flas//
Google DeepMind. Gemini 2.0 flash- tiiskihinn 2025b.
URL attps://@epmind.gogg//
technol ogi weggie//gee htggiii///mmii///////// faa.faasshhhhai

---

Published as a conference paper at ICLR 2026
Lingxiao Du, Fanqing Meng, Zongkai Liu, Zhixiang Zhou, Ping Luo, Qiaosheng Zhang, and Wenqi Shao. Mm-prm: Enhancing multimodal mathematical reasoning with scalable step-level supervi- sion, 2025. Uttppss:://axx wttp://www...a//////////// 13427
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. and Angela Fan. The llama 3 herd of models, 2024. SRRL:tts:
ozq abs /20b/ 221883
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive, 2024. URL httph::tp////. /atpss//aaxivv arxiv orq/ /a////b///// 2404 12390.
Team GLM. Chatglm: A family of large language models from glm-1300 130b to glm-4 all tools, 2024.
URL fettps://arxiv.co org/
12793
Conghui He, Zhenjiang Jin, Chao Xu, Jiantao Qiu, Bin Wang. Wei Li, Hang Yan, Jiaqi Wang, and
Dahua Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese abs abtpabbs/300 2308 10755
large models, 2023. URL /ttt/s::///arxxii/
org/
Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, and Hua Huang. Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning. In Kate Larson (ed.), Proceedings of the Thirty-Third International Joint Conference Artificial Intelligence, IJCAI-24, pp. 530-838. International Joint Conferences on Artificial Intelligence Organization, 8 2024. doi: 2224963/jcai.2022499 URL httpps://di....... org/
10. 10.24963/1jcaa http://aaa.222///99/ Main Track.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL
03874
Jiaxing Huang and Jingyi Zhang. A survey on evaluation of multimodal large language models, 2024.
/ wwwp:////aaaaa......m//// 15769
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoningg capability in multimodal large language models, 2025. URL
06749
Zhipeng Huang, Zhizheng Zhang, Zuu Zha, Yan Lu, and Baining Guo. RelationvIm: Making large vision-language models understand visual relations, 2024. URI. htt.hhtsts//axi..rrrr.g abs/ 2 /4003.23..22 12801
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning, 2024. URL URL htttttt/p///xiii/ wtwp///////b/sss//ss/ss/// 01483
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings. Ivan Krasin, Jordi Fost-Tuset. Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari.
The open images dataset v4: Unified image classification, object detection, and visual relation- ship detection scale. International Journal of Computer Vision 128(7):1956 1981, March 2020. ISSN 1573-1405. doi:
URL http: //tp://xxx.o www.xxxoooooooooooooo.oooooooooe org/
10. 020- 01316-2
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better under- standing lasiunn models: insights and future directions, 2024. URL http::tts:///aat org/ abs/ 2408. 12637
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei L., Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024a.
httpps:////xiiii /ww.iiiiiii..///vv http://awwwvvv.g..../.// abs/ 2408. 03326
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Lextteeext Tackling multi-image, video, and 3d in large multimodal models, 2024b.
htt.htss:://aet wrw. wtwp/baa///a//o/o
07895 07895
12

---

Published as a conference paper at ICLR 2026
Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, Ying Tai, Wankou Yang, Yabiao Wang, and Chengjie Wang. A survey on benchmarks of multimodal large language models, 2024c. URL httpsss://aaxiv www.bbs/g/ab/////
2408. 08632
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. wwwpr/aaapeeeeeee httpp//ppprreeeeeeppppp////2222/ e/ F
Curran Associates, Inc.,
2023. URL http:ss//pproeeeddng.....errrr
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jac Lec. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://
llavavvl.gitaae 1o wttp:ooccooc./o.ooo.oooo 1atp:///2222-00-------laaa------ 10-1------aaaaeex.....
Wentao Liu, Qianjun Pan, Yi Zhang, Zhuo Liu, Ji Wu, Jie Zhou, Aimin Zhou, Qin Chen, Bo Jiang, and Liang He. Cmm-math: chinese multimodal math dataset evaluate and enhance the math- ematics reasoning of large multimodal models, 2024b. URL
2409.02833
Zongkai Liu, Fanqing Meng, Lingxiao Du, Zhixiang Zhou, Chao Yu, Wenqi Shao, and Qiaosheng Zhang. Cpgd: Toward stable based reinforcement learning for language models. arXiv preprint arXiv:2505.12504 2025.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-
Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoming of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2024. URL https:
02255
Ruilin Luo, Zhuofan Zheng, Yifan Wang. Yiyao Yu, Xinzhe Ni, Zicheng Lin, Jin Zeng, and Yujiu Yang. Ursa: Understanding and verifying chain-of-thought reasoning in multimodal mathematics. arXiv preprint arXiv:2501.04665 2025.
Jingkun Ma, Runzhe Zhan, Yang Li, Di Sun, Hou Pong Chan, Lidia S. Chao, and Derek F.
Wong. Visaidmath: Benchmarking visual-aided mathematical reasoning, 2025. URL https:
7/arxiv. www.org.aaooo wwt.g/ab///////// 2410.22959
Fanqing Meng. Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models, 2024. URL https://arxiv,
org/abs /208 02718
Fanqing Meng, Lingxiao Du Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule- based reinforcement learning. arXiv preprint 2003::2503.07365, 2025.
Meta. The Ilama 4 herd: The beginning of new era of natively multimodal ai innovation, 2025.
.ew.aaa.mettaammmmmmmmmmaaaaaae. meta. htt.://ooooooooaaaaaaaaaaaaaaaaa
OpenAl. Hello gpt-4o, 2024. URL httpst::://///
OpenAl. Introducing gpt-4.1 in the api, 2025. URL
g-t-4-17
Jaewoo Park, Jungyang Park, Dongju Jang, Jiwan Chung, Byungwoo Yoo, Jaewoo Shin, Seonjoon Park, Tachyeong Kim, and Youngjae Yu. Explain with visual keypoints like a real mentor! benchmark for multimodal solution explanation, 2025. URL https://arxiv: org. abs/
2504. 03197
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong. Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, and Honggang Zhang. We-math: Does your large multimodal model achieve numan--ike mathematical reasoning? arXiv:2407.01288 2024.
arXiv preprint
http:p:://aaxiv..oo http:pp::/.aaa...oor//////222..2 01284
13

---

Published as a conference paper at ICLR 2026
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang. Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-rl: stable and generalizable r1- rl-style large vision- vision-language model, 2025. httthttss//aaat .rg. abs/
2504. 0604.07615
Vasu Singla, Kaiyu Yue, Sukriti Paul, Reza Shirkavand, Mayuka Jayawardhana, Alireza Ganjdanesh, Heng Huang, Abhinav Bhatele, Gowthami Somepalli, and Tom Goldstein. From pixels prose: A large dataset of dense image captions, 2024. URL
10328
Kai Sun, Yushi Bai, Ji Qi. Lei Hou, and Juanzi Li. Mm-math: Advancing multimodal math eval- uation with process evaluation and fine- fine-graing grained classification, 2024. URL https://arxxx
org/abs/ 2404. 05091
Yanpeng Sun, Shan Zhang, Wei Tang, Aotian Chen, Piotr Koniusz, Kai Zou, Yuan Xue, and Anton van den Hengel. Mathglance: Multimodal large language models do not know where to look in mathematical diagrams, 2025. htsp::/::: htkLhttss///axxv.coo http://wxvvv...g.bb.b.
Doubao Team. Doubao-1.5-pro, January 2025a. URL nttps://team.douset com/ zh./ special/doubao 1____r
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text, 2024a. /tt.
05530
Gemma Team. Gemma 3 technical report, 2025b. URL
19786
Qwen Team. Qvq:
To see the world with wisdom, December 2024b. URL https://qweet
github. ic/blog/qvg wet--27---ppeeeee
Qwen Team. Introducing qwen-vl, January 2025c. URL https://gwen..m.iithbb. 1o/
blog/qwen-vil
V Team, Wenyi Hong, Wenmeng Yu, and Xiaotao Gu. Glm-4.5v and glm--.1v-thinking Towards versatile multimodal reasoning with scalable reinforcement learning, 2025. URL https://
arxiv.org/ abs wtt.o///ab////00/// 01006)
Fei Wang, Xingyu Fu, James Y. Huang. Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. Muirbench: A comprehensive benchmark for robust multi-image understanding, 2024a.
http://aaxxvvvo// www.or//////////// abs 2406.040000 09411
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoniing with math-vision dataset. arXiv preprint arXiv:2402. 14804, 2024b.
abs 2402. 14804
Ke Wang. Junting Pan, Linda Wei, Aojun Zhou, Weikang Shi, Zimu Lu, Han Xiao, Yunqiao Yang, Houxing Ren, Mingjie Zhan, and Hongsheng Li. Matheoderrvl: Bridging vision and code for enhanced multimodal mathematical reasoning, 2025a. URL. /ttsss///rxvv.. abs/
2505..1 10557
Peijic Wang, Zhong-Zii Zhi Li, Fei Yin, Xin Yang, Dekang Ran, and Cheng-Lin Liu. Mv-math: Eval- uating multimodal math reasoning in multi-visual contexts, 2025b. URL https://arxiv: azg/abs/2502 20808
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing rision-languee model's perception of the world any resolution, 2024c.
14

---

Published as a conference paper at ICLR 2026
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuam Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvim: Visual expert for pretrained language models, 2024d. htthttt://// /tttpp://arxiv... wwaxiixxvvvvrro..aa.ae..e......e org/
abs 2oss/211..0 03079
Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal rea- soning, 2024e. URL
/ abs/ 06805
Junfei Wu, Qiang Liu, Ding Wang, Jinghao Zhang, Shu Wu, Liang Wang, and Tieniu Tan. Logical closed loop: Uncovering object hallucinations in large vision-language models, 2024a. URL
https: //arxv//xxvvvv owg//oggg//soo.o. hwtgg//b////// ab0/ 2402. 1622
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang. Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie. Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseck-v12: Mixture-of- experts
models for advanced multimodal understanding, 2024b. URL https: 10302
// /aaaxiv.orrgg.oo......aa.......a
Kun Xiang, Heng Li, Terry Jingehen Zhang, Yinya Huang, Zirong Liu, Peixin Qu Jixi Hc, Jiaqi Chen, Yu-Jie Yuan, Jianhua Han, Hang Xu, Hanhui Li, Mrinmaya Sachan, and Xiaodan Liang- Seephys: Does seeing help thinking? benchmarking based physics reasoning, 2025.
URL httttt////::xxii .rgg abs/ 2505 19099]
Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal Ilm logical reasoning benchmark in visual contexts, 2024. /tl.htttss///rxww.orrrr 2200 .02.0.0009733 04973
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng. Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. RI -oneviisso::: Advancing general- ized multimodal reasoning through cross-modal formalization, 2025. /tRppsttts://axii
org aba/ 2503.51111 10615
Xinwu Ye, Chengfan Li, Siming Chen, Wei Wei, and Xiangru Tang. Mmscibench: Benchmarking language models on chinese multimodal scientific problems, 2025. URL. httpt::/aaxxiii
arg/abs/2500 01891
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 9556 9567, Los Alamitos, CA, USA, June 2024. IEEE Computer Society. doi: 10.119//CVPP52233.224..00999
URL http:ttss//ddiiieeccoeooooo http::/doo...eecccooptttterceete orq/
10. 70.1109/CVPR527555 2091100913
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. internln-xcomposer22.. A versatile large vision language model supporting long-contextur input and output, 2024a. URL
https: / arxiv............ www. abs 2407. 03320
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-We Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your muuti--mdal 1lm truly see the diagrams visual math problems?, 2024b. URL https:///aattti org/ abs/
2403 4624
Xiaotian Zhang, Chunyang Li, Yi Zong. Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.122. 12474, 2024c.
15

---

Published as a conference paper at ICLR 2026
Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swiftr scalable lightweight infrastructure for fine-tur ane-ttning, 2024. URL URLhtttss://arxie http:/:://xxxxvvoav/a abs 2408 05517
Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng. Math-puma: Progressive upward mul- timodal alignment to enhance mathematical reasoning, 2024. URL https://arxiin org/
abs/2408. 08640
Yi Zong and Xipeng Qiu. GAOKAO-MM, Chinese human-level benchmark for multimodal models evaluation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACL 2024, pp. 8817-8825, Bangkok, Thailand, Au-
gust 2024. Association for Computational Linguisties. doi: 10. 10.18653/11222...ff.. http:/22.......ss-aae/. aCl.51. URL https: Tectps://aclante /aaaa//aaattcoooooooo///....... 20d.
16

---

Published as a conference paper at ICLR 2026
A EXPERIMENT DETAILS
A.1 EXPERIMENT SETTING
All experiments were conducted on a Linux server equipped with two NVIDIA H800 GPUs (each with 80GB of memory). The Python version used in the experiments was 3.9.20, while the version of vllm library was 0.8.1, respectively. Each model evaluation was performed in zero-shot setting with deterministic decoding (lemperature=O). Due frequent updates and improvements, closed- source models often undergo version changes that can significantly impact evaluation results. Even subtle updates may alter model behavior, performance, or prompt adherence. As such, the results reported in this benchmark are tied to the specific versions used during our evaluation. To ensure transparency and reproducibility. Table 6 lists the exact release dates or version identifiers of all closed-source models evaluated in VisioMath. Readers should be aware that performance discrep- ancies may arise when using newer or older versions the same models.
Specifically, we employ the same prompt template across all models to eliminate prompt-induced variance, and fix the decoding temperature to 0 to promote deterministic outputs. Accuracy serves as the primary evaluation metric, measuring the proportion of correctly answered instances. We utilize GLMM--Flahh (GLM 2024) to extract the options from the responses generated by LMMs. In scenarios where the model fails to produce valid answer, i.e., none of the standard options (A, B, C, or D) can be reliably identified, its response is classified as invalid. Such cases are treated as incorrect predictions in the final accuracy computation.
To ensure a fair comparison, we adopted consistent prompting strategies across the three input types: Original, Strategy 1, and Strategy 2. For Strategy zero-padding. For Strategy 2. we extend each option image by adding 50-pixel-hip strip at the bottom, matching the width of the image, and insert the corresponding option letter (A. B, C, or D) within the strip. Examples of these configurations are illustrated in Figure 6 In Strategy 3, we adopted the Supervised Fine-Tuning (SFT) training strategy on Vw.nVL2..5-.
horizontally concatenated all images with
Using a single H800 GPU and the ms-swift framework (Zhao et al. 2024), we set the batch size to 2, the learning rate to le-4, and the gradient accumulation steps to 4. The training was conducted over 336 steps.
Table 6: Version information or release dates of evaluated closed-source models.
Model
Version (release date)
GPT-4o OpenAl 2024
2024-11-20 2025-04-14
GPT-4.1 OpenA 2025
Gemini2-faase flash (DeepMind) 2025a)
2024-12-11
(DeepMmnd 2025b
2025-01-21
Gemini 2.5 Pro
(Comanici et al. 2025)
2025-06-17
Qwen VL-max [Team 2025c
2025-04-08
Qwen VL-plus (Team
2025c)
2025-01-25
Doubao-1.55 L.sbaao-.5-Viiion--- Vision-pro [Team
2025a
2025-03-28
Seed1.6 6 (ByteDaace
2024)
2025-08-15
GLM4V-plus (GLM
2024)
2025-01-11
A.2 FULL EXPERIMENTAL RESULTS
Due to space limitations in the main text, we report the full evaluation results of various LMMs on the VisioMath benchmark in Table 7 and Table g Moreover, we evaluate the adaptability of models not originally designed for multi-image processing. For these models, we implement an image concatenation strategy, in which all images associated with given question are merged into single composite image. Despite its straightforwardness. this approach exposes critical limitations. Among the models evaluated, the best performer, which is on par with the naive baseline of random guessing. These results underscore a fundamental limitation of single-image LMMs in multi-image contexts: they fail to effectively model relational information across distinct visual inputs. This highlights the need for architectures that explicitly support cross-image representation learning and comparative reasoning.
1.-.v...no---11111 achieves only 24.4% accuracy,
17

---

Published as a conference paper at ICLR 2026
Strategy 1 Consolidated single image layout
Image stitching
Original
Strategy
Explicit visual visual-textual anchors
Option bar
Figure 6: Illustrating the image format of original, strategy and strategy in our experiments.
B MULTI-IMAGE COT FINE-TUNING
This section explains how CoT reasoning data was constructed, including description generation by QwenVI.-Max, refinement by DeepSeek3.1, and filtering strategies. We construct specialized multi-image chain-of-thought (CoT) dataset through structured three-stage pipeline to enhance model performance.
Stage (Problem collection). We crawled 1,072 mathematical single-choice questions that contain more than four pictures from the internet to serve as the raw problem pool.
Stage (CoT Sample Generation). Initial reasoning paths and descriptive captions produced for each problem using QwenVL-Max with an Image-Caption with the original questions, then fed into DeepSeek V3.1 via CoT Data Generation Prompt to generate refined reasoning trajectories and corresponding answers. Samples are subsequently filtered based answer correctness, resulting in 500 high-quality multi-image CoT examples.
Prompt. These outputs, together
Stage 3 (Dataset Expansion). To increase both the scale and diversity of the dataset, Option Shuffling Prompt strategy applied, expanding the dataset from 0.5k to 1.3k samples.
This three-stage pipeline ensures that the final dataset contains both high-quality reasoning examples and sufficient data scale, providing robust foundation for effective model training.
C PROMPT TEMPLATES
We employ the same prompt template across all models to eliminate prompt-induced variance. Specifically, use five types of system prompts in our paper:
Original Answer Prompt: The baseline system instruction that is uniformly appended to
all models prior to evaluation, serving to standardize response format and output scope.
Option Shuffling Prompt: variant of the Option Shuffling Prompt in which the cor- respondence between options and images is completely deranged, designed to test and mitigate the model's reliance on positional priors, and used for synthetic data generation.
Answer Extraction Prompt: A prompt used to guide the LLM in extracting and normal- izing the final answer from the model's output (c.g.. mapping free-form text or reasoning steps discrete options such A/B/C/D).
18

---

Published as a conference paper at ICLR 2026
A prompt that instructs the MLLM to generate concise, comparable textual descriptions and preliminary analyses for each image, serving as a cross-modal representation bridge.
Laage--aption-Stylle Prompt:
CoT Data Generation Prompt: A prompt that integrates the question, image captions, and -iroovdd reasoning trajectories to produce high-quality chain-of- toout rationales and final answers, which can be leveraged for data augmentation and fine-tuning.
The detailed prompt texts are shown in Table
D ERROR ANALYSIS
This section presents a detailed analysis of errors, categorizing them into four types, reporting their distributions, and providing representative examples.
Image-Text Misalignment (36%). These errors occur when GLM-4.5V fails to correctly capture the semantic correspondence between textual options and visual content. For example, in Figure the model misinterprets the relationship between the image and the answer options, incorrectly treating the reference image Option A.
Vision Recognition Error (34%). Vision recognition errors reflect the model's difficulty in accu- rately perceiving visual information. As shown in Figure g GLM-4.5V fails to correctly interpret the shapes of the unfolded cubes corresponding to Options B and C.
Reasoning Error (20%). Reasoning errors arise when GLM-4.5V does not correctly follow logical steps or underlying problem constraints. For instance, in Figure 8, the model incorrectly assumes that the depicted line graph necessarily satisfies the definition of function.
Knowledge Error (10 ).). Knowledge errors occur when GLM-4.5V lacks relevant domain knowl- edge produces outdated/inaccuraal information. For example, in Figure 7 the model erroneously interprets the top view a sphere as a circle with a visible center point.
D.1 VISION RECOGNITION ERROR ANALYSIS
We further conduct detailed analysis vision recognition errors, report their distributions, and provide representative examples.
Fine-Grained Geometric Perception Error (18%). Fine- aenegrriiedd geometric perception errors oc- cur when GLM-4.5V struggles to distinguish subtle quantitative differences between highly similar options. For instance, in Figure the model incorrectly distinguishes between Option and Option C based on the presence of hollow points.
Spatial Topology & Transformation Error (24%). These errors oceur when GLM-4.5V fails to comprehend the mapping between 2D shapes and 3D objects or understand spatial connectivity. A representative example is shown in Figure where GLM-4.5V fails to correctly interpret the left view of the given solid.
Spatial Positional Relation Error (22%). Spatial positional relation errors occur when GLM-4.5V misjudges the relative position of geometric elements within coordinate system. For example, as shown in Figure 10, the model crroneously assumes that the graph of the function lies entirely above the x-axis.
Abstract Global Pattern Recognition Error (36%). Abstract global pattern recognition errors arise when GLM-4.5V identifies local features but fails to integrate them into coherent global geometric pattern or structural layout. As shown in Figure 10 GLM-4.5V fails to determine whether the figure exhibits axial symmetry.
E USE OF LLMs
In this work, large language models (LLMs) were utilized supporting tools to aid in the revision and polishing of certain text segments during manuscript preparation. All model-generated content was thoroughly examined, revised, and refined by the authors to guarantee accuracy and compliance
19

---

Published as a conference paper at ICLR 2026
with academic writing standards. Notably, the study's conceptual framework, methodological de- sign, data analysis, and interpretation of findings were conducted entirely by the authors, without dependence automated systems. The authors bear full responsibility for the originality, validity, and integrity of the research presented in this paper.
Image-Teed Test Misalignment
Kmow Sedge Error
Qucetion: shows
foor
Quediaam drown is the Sgure, the comnssonnds which shapes
A
B
D
Standirdd answer:
Standard answer:
response
observing
determine
isies opt
given figare, analyte top view
dimersional
matches
Option (paneepanndng the secund queat: powcents chai cutline sconales
Outior C (conesponding nding to the third guret
andlo, consitene with the ciaracteristis of the front
elliss, posiiily the top view the
Option Cir (ryinterr The top view of oytindee is cirde (withou centeed but since the peotiem tigune udee comerr doss nor matrh.
which doss mutch the prob figurs,
(eemisheee) The view hemisphere semiciecle diameter line)
cons or arguctior from another angls, not from slew:
Optior jooiresponingg
dinting to the founh figurel isa mangle but
feature
Option O juore:: The tos view of cone is circle jwith certer and includes the projection spen, probiem figure is only ample without the spec, so
the fonn, close regile mangle rather than hooowee tntangle coners
Figure 7: Example enror cases for image-text misalignment and knowledge error.
Ressoning Errur
Vision Recognition Error
Quesien: following
fancton
Questioa: follow figunci camnot felded
Standard asswer:
Stanclard answer:
Medal responss
resporue
Accorcing cefistion fundtoon, every value ot in doman, thene muet uriguely
deterrieed conesponting
carve
fusctin
Ostion B: the cascol airle, there centaa values of e-d that corressond diflcrntt volure tyn3 and y--1), which doen not catieyy eeitythe delnttinn funetion.
Oztion case straight line, each comeespond to unique satalying defection of lunctin.
Option Belloggs "1-4-2
Option Alto "1-4 tyoe your faces the middle aooe
volla
Oution in the cisc Iypicroola, s similrlyy shere cest uos con espording to multtale y valles ltg, comerpards and ym-ll which cons not satisly dertattonn function.
one belcer, which can be folced ints cube.
Option Conmains "taid" poid etrurtue fitur small up foemiing larger squard), forbidden cubs cannet
curve
Resessayy to
salundi would sariefy defiition function. coesidened local ingions with multislc valaos).
functions
citir fuection
probiem
"he arower question
"herelore,
lend, bool
In ueion, only nen on catetie sestteette detiction of function. The arower this question is +juepin boo bss of boxd>
Figure 8: Example error cases for reasoning error and vision recognition error.
20

---

Published as a conference paper at ICLR 2026
Grineed
Transfrrvation
Quastie holed quatar, Paccordiutee be of saaga amorg it proct, which carrcc opocants Laniii
Quurkan peovee suld lobes. Whick
Steddrd
russens
piremere till owi UP4 Landiey y eepuinig sejeedilonss bellrd purkar This aoo aneual yu (rtiraeteree
U, and LAACEM) eocentt
en vin and of euck optinn seperndly
helglt the Vi clmeeee J.F)) Therelorr,
Homeeet
tealee lemaee
Feyaar a Foree
more renomable vattor we A.l. the U.00 ti SPect iitmes
Ira
kreen Bayuae allived
brebueneeg option pornect foodd sataihy thet wher'' sirkk 2/1 decramm
yondl li andly unske))
The lrve be kimet Beyyer allirre
21 true meat This inture hour with cork Dhught bext boo
Figure 9: Example error cases for Fine-Grained Geometric Perception Error and Spatial Topology & Transformation Error.
Spaltal Hrlstios Errer
Qeertin: The of saclory yar
Quotiax
Spaeitl
respoees
any il Comaan The deconiaaoon 10 ghoci (n us tee
U/1
huld' thes huliciin
Outice laow 't's stornci whan riide anvight upddens soit ansymmatic fgare.
vesaaaa in E'1 wa'e s0-011 Witir Va so co wo Maueeeapeeg -o st
see netweer You Vumeee mome brnerul to fia wogg
alddward withr alwwn ea ond eene
palyeamnall to.so 30 Ae repoos
Optes fumo tober ting modiie right <bbggh of be denadeoeeettte
ratorr espraach Vodk annitig armeraee aemeett 1nr i 11 maaee
regat
VV.
mases cacresss, and spprescces (2767
and salces anl
boskiee regics,
shape.
Figure 10: Example enror cases for Spatial Positional Relation Error and Abstract Global Pattern Recognition Error.
21

---

Published as a conference paper at ICLR 2026
Table 7: Performance comparison on VisioMath with results categorized based on GT position.
position
Axg
images
images
D
Humne
91.3
92.3
92.5
95.6
93.8
88.5
R9.7
94.4
87.6
87.5
$8.0
Cloued-ource LMMI
plus
27.9
30.2
26.1
33.5
31.5
27.3
25.4
22.8
26.0
11.1
QwenVL-plos
25.5
OwenVL.max el
4024
44.1
53.4
35.2
62.6
62.5
50.2
34.1
31.1
34.1
32.8
38.6
GPT Open/1l 1
15.9
$4.7
55.6
56.4
$4.7
$2.5
36.5
47.3
30.4
36.3
30.4
GPT4,1
2025
52.6
61.5
72.4
59.9
60.2
56.1
42.8
54.8
39.3
43.7
31.9
[DepMMndd
Gemim22 Bash DeepMini 20753
55.5
65.1
78.1
59.9
65.2
59.6
45.2
57.7
34.5
38.7
47.8
Doubae-1 1.5-Viiton-
Team 202
66.3
75.6
78.6
78.0
76.6
70.2
$6.4
73.4
55.6
48.0
45.4
S
72.3
85.7
87.2
44.6
Comunics
Open-townvr
ZMMSs
(munfi-inugee Impori
despseckv12
Wa 20203
13.5
21.5
15.9
24.1
15.2
2.7
25.6
58.3
27.1
6.9
InlemVV12.5-220
IntemVL2.3/24 Ches 2i P112433
24.5
27.1
12.8
25.5
36.3
30.2
21.9
10.3
26.2
38.2
15.0
Llama3.2-11B Visgit (unny c 2021
25.3
26.2
30.5
24.7
39.8
10.6
24.2
30.2
23.4
26.3
6.3
Ideficst-NH-lllng
Labrerger
10.6
24.6
11.3
Qwen225-VL.3BB11m
Phi3.5-vision AAhdjn al 2144
25.7
25.3
73.5
22.0
14.1
2.4
78.8
13.6
3.4
depseccki2-ss.mm
Ki124
26.6
32.0
42.8
28.5
50.4
28.2
20.8
35.6
12.6
12.3
16.9
Manis Momiis-B--leffee
27.9
17.6
42.1
36.5
32.4
JntemVL...--B-000
4PP
20/44
28.4
30.9
12.8
3500
31.5
42.4
25.6
9.5
27.1
47.1
21.7
Mi
Abdie et
2022
29.3
34,5
40.8
36.1
31.3
31.
23.6
36.5
22.8
15.7
15.9
R1-Onevisan-78
30.2
16.7
IntemVL2..550
Liee 21
29.9
33.1
26.4
31.7
50.4
22.0
26.6
30.7
16.4
40.7
18.4
MicmVLL225-8B-XPPP
20244
30.9
35.9
25.0
48.5
39.5
29.4
25.5
23.7
25.2
34,3
19.3
QyQ-728-Previae
[edm:
26.0
Qwen2 inzcler
Warg
2onn:-5-Vl-7-iinttreel
Bai
20.4
32.7
39.5
30.1
58.1
39.8
29.
8.7
28.5
31.4
34.3
Gemma3-278 Team 20250
35.3
43.7
67.9
40.1
33.6
38.4
26.2
40.2
24.8
12.3
25.1
VisionR1-78
2025
Owen25.VL.3.8-110 3.3-111--1
16.2
Owen2.5.VL.. VL- instret Bal
2123
43.7
33.5
36.2
63.9
61.3
$9.3
33.0
29.9
37.8
29.9
35.2
GB M-4.V
Team et
X22
53.7
69.1
71.9
75.8
68.4
61.22
57.2
46.5
42.5
31,4
26.6
Llams4-MaNNna--111
63.4
Open sonrice LMMs
Evingle imagee (ippc)
LL 4-- (Linetal
30246
30.7
22.6
21.4
8.4
32,4
20.3
15.7
29.5
2.8
23.0
18.4
MinCCMM-225 Alddin
21.0
13.2
34.1
Lan ora m0244
25.0
19.9
73.5
3.1
12.9
01.5
26.3
78.4
2.3
16.7
0.0
LLaVA-13-H 2B Li 20235
23.7
23.6
23.5
19.4
12.5
38.4
23.8
33.6
17.3
14.2
28.5
MAY.R (GEM)
LLAVA-116 STROESSEEEE
2-2
2.9
Upenn Selvice LMMMs [muith ovtiented)
22.9
USRA-PSRIIT
22.8
23.8
14.5
25.5
21.8
26.2
1.5
MM- Farolan
Mong mn 217A
37.9
50.9
36.2
62.1
52.7
50.1
24.0
21.1
22.4
27.4
25.6
HMM-Brrkka-1
7H- PULL (A (EI
2025
39,3
51.0
33.2
54.2
61.3
51.4
26.9
16.2
20.9
39.7
23.7
MM-PRM-BB
MathCadarV1..
Wiir
50.5
MathCodarVL-SS
Walp 2f
31.31
31.5
33.8
24.0
44.3
34.0
31.4
29,0
27.0
35.0
31.9
22.2
PUMA7B Zhusng
AT/4
24.6
24.0
19.9
2,6
38.7
31.3
25.2
11.6
0.4
74.5
17.9
1-ManET
22

---

Published as a conference paper at ICLR 2026
Table 8: Performance comparison VisioMath with results categorized based image similarity.
Imuge similarity
[0.16.0.68]
(0.68,0887)
(0.87,0.96]
87.6
Cloved Sooteee LMM,
GLMiViplu
GGM
2021
21.9
29.6
32.9
23.3
25.8
QueaVL-plus
33.3
31.8
QweaVL_eax Ba 2E
332-
48.1
47.3
30.2
41.3
31.6
GPT-4 Orea-U
7174
45.9
53.8
50.9
40.0
39.1
GPT-4.1 Opstat
56.4
DeopMMind
48.2
Gemini22Mlssh
[hepMinni] STran
33.5
66.7
59.8
49.3
46.2
Doulso-115. 1.5.
27/50
66.3
74.9
68.2
60.2
62.0
Seed1.6
Thinking
Bytebraee
82.4
Commonct
Open sounce LMMS (mmnf--muge Inpor)
DeepSesk
Wa
20144
27.6
InteraVL. 2-22
Chen2rma 21xe10
24.5
24.2
28.9
22.7
22.7
(Tile 2024
25.3
23.1
27.8
26.4
23.6
16elecc--8--lamm
Labrrice
26-9
26.1
Qwen2.5-V1.
AB-instrat
267
27.6
24
22.9
Phi3.5-4ision
Alviin pi all M0:4
25.7
23,6
78.7
27.8
22.9
DeepSesV
TTnto Wi
X124)
26.5
30.7
29.6
24.9
21.3
Minii--8--Mlefee
1m1eraV1..5.4RR
Clenc
31.8
27.3
MPC7 [a 2
20140
28.4
28.2
34.11
26.2
25.1
MiniCPM-a-1.6
Abllin
4774
29.3
34
35.3
244
22.4
R1-Cmevison-78
MiniCPM-V-2..
Abihin 21 P3/6
29.7
30.7
34.9
28.4
24.7
InternV1.2 2.588
Till dF A17am
20.9
32.4
31.8
20.6
26.0
ImteraVL..5-8B-511 5--
LL3 21244
30.9
35.6
37.1
25.8
25.1
Quen2-V1-77B-inmmmn
Waber
35.1
37.8
76.1
Owest.5-VL--B-intup
32.7
33.6
37.8
29.8
20.6
Gemmal-27B Tram 207.5
353
43.3
41.2
29.6
26.4
2025
Qwen2.5-V1.
220-instrac 1531
2025
41.8
50.0
46.2
38.4
32.7
Qwen2.5-V1
Swen2.5-V1-77n-innmmt
77.B-innttrett
(15 al
MBAS
43.7
47.1
50.8
38.0
38.7
G1M-4.5V Tesm al D176
53.7
68.7
59.3
44.2
44.7
Open-wwurce LMMS raingle iiagee (ipp7)
[L
clal
303a
20.7
22.1
21.4
17.8
18.4
MiniCPM-V-2..5
112VA- (aalll 20241
22.7
25.6
v..VA.-..--misppll 77 Linge 202
23.0
21.8
26,0
23.6
20.5
LLAVA-11.5-7B iu
23.7
23.3
25.3
24.9
21.1
LLaVA 6-c-n- 1--.-n1.--00.0--11111
Liu etal,
202
24.4
24.0
26,0
26,0
21.8
Open Opete-sonice sounice LMMM (mmtth- uriontee))
MalhCoderV1. Wimn el
2025g
24.5
24.7
25
26.0
22.2
PUMA79 Zbuand 21 Allss
24.5
26.2
25.8
23.8
22.4
USRA aL
27.4
26.0
31.4
26.3
VLM
2025
MabCoderVLLBB
Want
2025
31,5
33.8
37.1
28.4
26.7
MM PRM 8D Dy b. On 2024
31.7
37.6
37.1
26.9
25.1
MM-Euncke MM--Eurca--7B-
925
37.9
45.6
44.0
29.1
33.1
23

---

respectivelyy the question Published as a conference paper at ICLR 2026
Table 9: This table presents the prompts used for process evaluation and answer generation by various LMMs the VisioMath benchmark.
Phase
Input
Prompt
You are Al wsistant helps me extract the anserss single-chuice quntioas. You be provided with on unswer Your taok find the final option of the model. If the model's snswer is meuningless, cutput Z. You should ouput uppercase lester A, (if they vulid options),
Example
Answer: According to the question description and all related pictures, option is the correct answer. Option A is contrally symmetric figure hecouse its four veriios are all symmerric, while symmetric.
Answer Extraction
Model's redeoss
options
(GLMM-Flash)
Output A
Example
Answert A. Sphare B. Cirole Dise D. Cirele
Ourgut
Example
Answer: (model unswer)
Output:
Please solve single-choice math question. The last four pictures are respoctively the pictures opiios A. B. and Select the currect from B, and based the question desprption
Question
Generation (LMMs)
Diagrams
Option Shaflling
Question
Please solve single choice math question. The last four
Gencration
Diagrams
deserittion and all relevant pictures.
options H.
from
multipp: images question want you answer. you strictly format with theee specific section: SUMMARY. CAPTION and REASONING. To explain further:
In SUMMARY. briefly explain what stess you'll ake to salve the problem. In CAPTION. daseribe
deseription inside tags like cinagela</3 homagel></inagee mage <image2> weteee///////ee/// image2>, eec In REASONING, culfine a bop-by-stpp thonght grocess you would use solve the problem haval on the images.
images, wrapping
Image Caption
<SUBNARY
Generation
you approach preblen....
(LMMs)
Diagrams
</SUNMARY>
<CAETIOU>
<inagel>... </inagel>
cinege25... inage22>>>
</CAPTION>>
<RBASOOTTE>>
[Provide chain thought, loginal explanet ion of the problen. atep-by-step reasoning
This should images.
Question Cuptoo
Please salve this maltsple-hoiice math question and answer in English. The last four images car- respand optinn A. B. and D respectivlly. Based on the quesiton description and all relevant images,
Generation
(DerSSck-V3.1)
from A,
24


Reviews:
[{"review_id": "zDj1H9xqdk", "reviewer": "Reviewer_ABAJ", "summary": "This paper, *“VisioMath: Benchmarking Figure-Based Mathematical Reasoning in LMMs,”* presents a new benchmark designed to evaluate **fine-grained visual reasoning in mathematical contexts**, particularly when **multiple highly similar diagrams** are presented as answer choices.\n\nThe authors construct **VisioMath**, a dataset of **1,800 high-quality K–12 math problems** collected from real entrance exams (2002–2023). Each problem provides **four visually similar diagrammatic options (A–D)**, with some including **images in the question stem**. Such *diagram-based answer formats* are indeed rare among existing math benchmarks, making this a **valuable and innovative dataset contribution**.\n\nThe benchmark evaluates a wide range of **open- and closed-source large multimodal models (LMMs)**, and key findings include:\n\n- **Multi-image reasoning remains a major weakness** — accuracy often drops to near chance level when multiple diagrams must be compared.  \n- **High visual similarity sharply reduces performance** — all models degrade significantly as image similarity increases, indicating weak fine-grained perception.  \n- **Positional bias exists** — models show uneven accuracy across option positions, especially when the question stem includes images.  \n- **Image–text misalignment dominates failure cases** — models frequently rely on superficial spatial cues rather than grounded reasoning.\n\nTo mitigate these issues, the authors explore **three alignment-oriented strategies**:  \n(1) merging all visuals into a **consolidated single layout**,  \n(2) overlaying **explicit textual anchors** on images, and  \n(3) fine-tuning with multi-image chain-of-thought (CoT) samples.  \nThese approaches lead to **notable performance gains**, demonstrating that targeted alignment interventions can substantially improve multimodal reasoning in visually complex mathematical tasks.", "questions": "Overall, this is a very solid and well-executed paper.\nI do have one question: when evaluating on the benchmark, why didn’t the authors include specialized math-oriented multimodal models for comparison?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novel Benchmark Design for Visual Reasoning in Math**  \n   The benchmark uniquely incorporates **both image-based question stems and diagrammatic answer options**, a setting that is rarely seen in prior math benchmarks.  \n   This integration of text and diagrams offers a more realistic and cognitively demanding setup, and I believe such *interleaved visual–textual formats* represent an important future direction for multimodal reasoning benchmarks.\n\n2. **High-Quality, Real-World Data Source**  \n   The dataset is constructed from **authentic K–12 entrance exam problems**, ensuring **pedagogical validity**, **realistic problem difficulty**, and alignment with real-world educational reasoning tasks.  \n   This grounding in real exam materials enhances the dataset’s credibility and practical relevance.\n\n3. **Effective Improvement Strategies**  \n   The paper proposes **three lightweight yet effective alignment-oriented interventions** — layout consolidation, explicit textual anchors, and multi-image CoT fine-tuning.  \n   These strategies are simple to implement but lead to **clear and interpretable performance gains**, demonstrating practical value for improving multimodal alignment in large models.", "weaknesses": "1. **Restricted Evaluation Format (Multiple-Choice Only)**  \n   Every question in VisioMath follows a **four-choice multiple-choice format**, which simplifies the reasoning process and limits evaluation to **discrete answer selection**.  \n   This design makes it difficult to assess open-ended or step-by-step reasoning ability, which is essential for understanding the full reasoning depth of large multimodal models.\n\n2. **Lack of Failure Case Analysis Beyond Accuracy**  \n   As a benchmark paper, the analysis would be more insightful if it included **qualitative examples of failure cases**.  \n   For instance, showing what a model’s **chain-of-thought output looks like when cross-image attention fails** would help readers better understand the specific weaknesses and reasoning breakdowns of current LMMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PPE493oSPY", "reviewer": "Reviewer_9Krr", "summary": "This paper introduces VisioMath, a new benchmark to evaluate the fine-grained, multi-image comparative reasoning of Large Multimodal Models (LMMs). The authors identify a critical gap in existing benchmarks: LMMs are rarely tested on problems common in K-12 mathematics, where all answer options are visually similar diagrams (e.g., graphs, geometric figures).\n\nThe VisioMath benchmark consists of 1,800 high-quality, curated K-12 math problems sourced from real-world exams. Its defining feature is that all answer options are diagrams, and the dataset includes a metric to quantify the (often high) visual similarity between them.\n\nThrough a comprehensive evaluation of state-of-the-art LMMs (e.g., GPT-4.1, Gemini 2.5 Pro, Qwen2.5-VL), the paper demonstrates:\n\n* A consistent decline in accuracy as the visual similarity between answer options increases.\n* The dominant failure mode (36% of errors) is image-text misalignment. Models rely on shallow positional heuristics (e.g., assuming the first image corresponds to option A) instead of true semantic grounding.\n* This reliance on heuristics is proven by a controlled option shuffling experiment, where permuting the text labels (e.g., \"A, B, C, D\" → \"B, C, D, A\") while keeping images in the same order caused a significant drop in accuracy (e.g., -8.7% for Gemini 2.5 Pro).\n\nFinally, the authors propose and validate three strategies to mitigate this: two training-free (consolidating images into a single layout and adding explicit visual-textual anchors) and one training-based (alignment-oriented CoT finetuning), which achieved a +12.6% accuracy gain on Qwen2.5-VL-3B.", "questions": "1. Could you please clarify the discrepancy between the failure of single-image LMMs using concatenation (Table 2) and the success of Strategy 1 (also concatenation) for multi-image LMMs (Fig 4b)? Is this difference due to (a) the models' pre-training (single-image vs. interleaved multi-image), (b) the specific concatenation method used (e.g., zero-padding), or (c) the inherent architectural differences in how these models process a composite image?\n\n2. The +12.6% gain from your alignment-oriented CoT data is very impressive. To demonstrate the scalability of this approach, have you attempted to finetune other models in the Qwen2.5-VL family (e.g., the 7B or 72B versions) to see if this substantial gain is consistent across model scales?\n\n3. What was the rationale for choosing the *minimum* pairwise similarity as your Sim(Q) metric? Have you analyzed how the results would change if you used the *average* or *median* pairwise similarity, which might be more robust to a single outlier option?\n\n4. The gains from Strategy 2 (explicit anchors) vary wildly, from +9.8% for QwenVL-plus to only +0.9% for Gemini 2.5 Pro. What is your hypothesis for this large variance? Does it suggest that more advanced models like Gemini 2.5 Pro are already attempting this alignment (and thus don't benefit much from the explicit hint), whereas weaker models are not?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality and Significance: The paper identifies and rigorously tests a novel, practical, and highly significant problem. The task of \"figure-based option\" reasoning, especially with high-similarity distractors, is a ubiquitous real-world scenario (especially in STEM education) that has been almost entirely overlooked by existing benchmarks.\n* Benchmark Quality: The VisioMath benchmark is meticulously constructed and serves as an excellent diagnostic tool, not just a leaderboard. The curation from real exams ensures representativeness. Key quality steps include enforcing a one-image-per-option rule, balancing the answer distribution to prevent bias, and, crucially, quantifying the visual similarity of options to enable controlled analysis.\n* Analytical Depth: The paper's primary strength is its deep analysis, which goes far beyond reporting accuracy. The option shuffling experiment (Fig 4b) is a simple but brilliant controlled study that provides definitive proof of the model's reliance on positional heuristics. This insight is arguably more valuable than the benchmark itself. The error categorization (Fig 4a), which pinpoints \"image-text misalignment\" as the dominant failure, is also a key contribution.\n* Clarity: The paper is very well-written, with clear and effective figures. Figure 1 instantly communicates the problem, Figure 2 shows the careful data pipeline, and Figure 4 clearly presents the core analytical findings.", "weaknesses": "* Apparent Contradiction in Concatenation Strategy: There is a confusing contradiction in the results. The baseline evaluation (Table 2) shows that single-image LMMs, which use a \"composite image concatenation strategy,\" perform at random-guess levels (e.g., LLaVA-v1.6 at 24.4%). However, \"Strategy 1,\" which also uses a \"consolidated single image layout\" (concatenation), *improves* the performance of multi-image LMMs (e.g., +6.4% for Seed1.6-Thinking). The paper does not adequately explain why concatenation fails so catastrophically for one set of models but serves as an effective enhancement for another.\n* Limited Scope of Finetuning (Strategy 3): The finetuning result (Strategy 3) is very promising, yielding a substantial +12.6% gain. However, this experiment was only conducted on a single, small model (Qwen2.5-VL-3B). This makes it difficult to assess if this is a generally effective strategy or if the gains would be as pronounced on larger, more capable models (e.g., 72B or closed-source models).\n* Benchmark Sourcing: The dataset is sourced exclusively from Chinese high school and college entrance examinations. While the authors translate the problems, this single-source origin may limit the benchmark's global representativity, as it may contain cultural or pedagogical biases specific to that education system.\n* Similarity Metric Rationale: The visual similarity for a question, Sim(Q), is defined as the *minimum* pairwise cosine similarity between options. This choice is not fully justified and could be fragile. For example, a question with three identical options and one complete outlier would receive a very low similarity score, even though it would be a very \"confusing\" set for a model. An average or median pairwise similarity might be a more robust measure of overall inter-option confusability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lAWpFxSrpZ", "reviewer": "Reviewer_bDac", "summary": "This paper introduces VisioMath, a new benchmark designed to address the evaluation gap in the fine-grained comparative reasoning abilities of current Large Multimodal Models when handling multiple, visually similar options. It presents VisioMath, a benchmark comprising 1,800 high-quality K-12 mathematics problems. Its unique characteristic is that all candidate answers are visually highly similar diagrams, compelling models to perform fine-grained visual discrimination and comparative reasoning, rather than simple pattern matching. Through extensive experiments on a series of SOTA models, the paper reveals a common deficiency in current models on this task, with the root cause being primarily image-text misalignment. Finally, the paper explores and validates three effective alignment strategies that significantly improve model performance. Its core contribution lies in providing a new, challenging evaluation tool, while also offering deep insights and practical directions for enhancing the multimodal alignment capabilities of LMMs.", "questions": "same as weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper identifies and addresses a critical problem in the field of LMM evaluation by focusing on visually similar diagrammatic options.\n\n- Through the quantification of visual similarity and an innovative \"option shuffling\" experiment, it uncovers that the core weakness of current LMMs lies in precise image-text alignment.\n\n- It proposes three concrete and viable performance enhancement strategies ranging from training-free methods to lightweight fine-tuning and experimentally demonstrates their effectiveness, offering the community a clear roadmap to tackle this issue.", "weaknesses": "- The definition of visual similarity relies entirely on a single model (Qwen multimodal-embedding-v1). Different models' visual encoders may have varying interpretations of \"similarity.\" A brief discussion or experiment demonstrating the correlation or discrepancy with other metrics (such as CLIP embeddings or DINO scores), or one that further justifies this choice, would make this core metric more robust.\n\n- The fine-tuning experiment for Strategy 3 was conducted on only a single model (Qwen2.5-VL-3B). Applying this method to another model family with a different architecture and observing similar results would provide stronger evidence for the generality of this conclusion.\n\n- The analysis of \"Vision Recognition Errors\" lacks depth. At 34%, this error type represents a significant proportion. Are these errors due to the model's inability to recognize basic geometric shapes (e.g., confusing a triangle with a pyramid), or its failure to understand more abstract symbols (such as the monotonicity of a function's graph)? A more fine-grained error classification would offer greater insight into the limitations of current visual encoders in diagram understanding.\n\n- The dataset is sourced exclusively from Chinese exams. Although mathematics is a universal language, the phrasing of the problems and the style of the diagrams may still carry certain cultural or curricular imprints.\n\n- In the \"option shuffling\" experiment, when faced with misaligned labels, do the models tend to guess randomly, or do they systematically follow the new (but incorrect) positional cues (e.g., always choosing the image corresponding to the new position of label 'A')? This could provide deeper insight into the nature of the heuristic rules the models rely on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CZqgdlT33l", "reviewer": "Reviewer_4GQo", "summary": "The paper introduces VisioMath, a curated benchmark comprising 1,800 K–12 mathematics problems in which all answer options are diagrams that are highly visually similar. VisioMath aims to rigorously evaluate Large Multimodal Models (LMMs) on fine-grained, comparative mathematical reasoning tasks, particularly focusing on challenges where multiple image options differ only subtly. The authors present an in-depth evaluation of various state-of-the-art LMMs in both closed- and open-source families, reveal systematic weaknesses related to image–text misalignment, and propose alignment-oriented strategies that yield meaningful performance improvements. The benchmark, dataset construction pipelines, error analyses, and strategic interventions are all discussed in detail, with extensive experimental results and visualizations supporting the analysis.", "questions": "1. Many of the tables are too dense and hard to read (eye-straining already) or grasp key points e.g. by color highlights or optionally move something into the appendix.\n\n2. Wording can be improved, \"question stem length\" is confusing, \"visual-option\" should perhaps be \"image-option\" because otherwise I can easily understand it as \"visual-optional\" so overall the presentation needs to be improved, a lot of the wording are highly confusing and will benefit significantly from more polishing in phrasing.\n\nIn general I think this design is really nice for testing vision-based reasoning capability for LMMs and the shuffing/strategy experiments are quite nice as analysis. Although the overall presentation could be significantly improve to aid readability of this manuscript I think this paper has merited a OK contribution to this field.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. VisioMath fills an under-explored niche by targeting figure-based multiple-choice mathematical reasoning where distractors are intentionally visually similar.\n2. offers insightfully categorizes failures (image–text misalignment predominates) and connects these with systematic bottlenecks in LMM architectures. The controlled shuffling and ablation study setting is quite nice, offering insights to advance the multimodal reasoning domain\n3. OK coverage of current vision or multimodal models with regards to evaluation and with perturbation setting to gauge the robustness of eval results", "weaknesses": "1. Limited Scope and Generalization: K12 and a quite narrow domain (geometry, algebra, etc.). The conclusions regarding LMMs’ comparative reasoning ability may not generalize to other STEM domains such as physics diagrams or scientific reasoning in general. If the authors want to target K12 for its pedagogical value maybe there should be a clear case made for that in analysis etc.\n\n2. Comparison with related work: more multimodal benchmark that explores vision-dependency of reasoning can be compared and discussed like SeePhys, MMSciBench, VisAidedMath etc. that goes beyond the current limited scope to position this work better in the landscape of multimodal reasoning, the authors could further strengthen this work by comparing the vision-based approach in these related works. The current first para of related work on LMM is really not that necessary and cannot possibly cover all LMMs due to the nature of that field's progress.\n\n3. Human baseline? Are these similarities only confusing for models or for humans as well? I also am a bit confused as to the necessity of classifying open-source LMMs into single vs. multi-image input? If they cannot process multiple images then they cannot process all the options correctly right?\n\nPS the current abstract entry on OpenReview has a entry typo \"Large\" missing an \"L\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)