Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs
Authors: Yanpeng Sun
Shan Zhang
Wei Tang
Aotian Chen
Piotr Koniusz
Kai Zou
Yuan Xue
Anton van den Hengel
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Models
Multimodal mathematical reasoning
Mathematical Diagram Understanding
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Diagrams represent a form of visual language that encodes abstract concepts and relationships through structured symbols and their spatial arrangements. Unlike natural images, they are inherently symbolic, and entirely artificial.  They thus pose unique challenges for Multimodal Large Language Models (MLLMs) distinct from natural image processing. Recent studies have shown that MLLMs often exhibit flawed reasoning and hallucinations when handling diagram inputs.  We investigate here whether these limitations stem from shortcomings in the models' ability to interpret diagrams themselves. To this end, we develop a diagnostic test suite that isolates perception from reasoning. Our systematic evaluation reveals that MLLMs perform poorly on basic perceptual tasks, e.g., shape classification, object counting, relationship identification, and object grounding, with near-zero accuracy on fine-grained grounding. Further analysis shows that weak diagram perception leads to ``blind faith in text", where models rely on textual shortcuts rather than visual understanding (that is, they are $\textit{Math Blind}$). We hypothesize that enabling models to capture the inherent structural properties of diagrams, represented as graphs of primitives and their interrelationships, is essential for improving diagram understanding. Experiments with 7B and 32B MLLMs validate this assumption, with models trained on such representations achieving a +79\% gain on the grounding task. Crucially, these gains transfer to reasoning, achieving 3–4\% cross-suite improvements on four public benchmarks even without additional chain-of-thought reasoning data. Our findings demonstrate that low-level perception supports faithful high-level reasoning in mathematical MLLMs. We provide both methodological frameworks and empirical evidence to guide future research in this direction. Our project page is at \href{https://vi-ocean.github.io/projects/MATHEMETRIC/index.html}{\color{blue}{viocean/\ourMethod}}.

Published as a conference paper at ICLR 2026
MATH BLIND: FAILURES IN DIAGRAM UNDERSTAND- ING UNDERMINE REASONING IN MLLMS
Yanpeng Sun Shan Zhang Wei Tang* Aotian Chen'
Piotr Koniusz
Kai Zou
Yuan Xue Anton van den Hengel 4
1 'Adelaide AIML, SUTD IMPL, "NJUST IMAG, 'OSU, "Data6 UNSW, 'NetMind.ai shan.zhang @ adelaide. edu.au, yanpeng sun @ sutd.edu.sg
ABSTRACT
Diagrams represent form of visual language that encodes abstract concepts and relationships through structured symbols and their spatial arrangements. Unlike natural images, they are inherently symbolic, and entirely artificial. They thus pose unique challenges for Multimodal Large Language Models (MLLMs) dis- tinct from natural image processing. Recent studies have shown that MLLMs of- ten exhibit flawed reasoning and hallucinations when handling diagram inputs. We investigate here whether these limitations stem from shortcomings in the models' ability to interpret diagrams themselves. To this end, we develop a diagnostic test suite that isolates perception from reasoning. Our systematic evaluation reveals that MLLMs perform poorly on basic perceptual tasks, e.g., shape classification, object counting, relationship identification, and object grounding, with near-zero accuracy on fine-grained grounding. Further analysis shows that weak diagram perception leads to "blind faith in text", where models rely on textual shortcuts rather than visual understanding (that is, they are Math Blind). We hypothesize that enabling models to capture the inherent structural properties of diagrams, represented as graphs of primitives and their interrelationships, is essential for im- proving diagram understanding. Experiments with 7B and 32B MLLMs validate this assumption, with models trained on such representations achieving +79% gain on the grounding task. Crucially, these gains transfer to reasoning, achieving 3-4% cross-suite improvements on four public benchmarks even without addi- tional chain-of-thougee reasoning data. Our findings demonstrate that low-level perception supports faithful high-level reasoning in mathematical MLLMs. We provide both methodological frameworks and empirical evidence to guide future research in this direction. Our project page is at Viocean/MATHEMMETRIC
INTRODUCTION
Computer vision has traditionally focused image-based perception tasks such object detec- tion Meng et al. (02T); Zhu et al. (2020), segmentation [i] et al.] (2021); Mishra et al. (2019). and spatial reasoning Cheng al. (2025) Driess al. (2023); Lin et al. (2014). These capabilities form the foundation for higher-level visual reasoning in recent advances of Multimodal Large Language Models (MLLMs) Achiam et al. (202a)): Liu et al. (2023b); (unn et al. (2024). Despite remark- able successes in general vision tasks, current MLLMs face considerable challenges in interpreting mathematical diagrams.
Although both natural images and symbolic diagrams can be represented grids of pixels, they constitute very different forms of information. Images represent samples of the intensity of the real world. Diagrams, despite taking many forms, are uniformly generated by humans as abstract visual representations of concepts characterized by precise geometric structures and symbolic notations Lu et al.; Zhhag et al. (2024a). Images natural collections of signals, where diagrams are artificial collections of symbols. Diagrams are critical to human visual communication across educational contexts, scientific discourse, and STEM problem-solving, but the problem of analyzing symbolic visual information is far broader.
" Core Contribution Project Lead
M Yuan.Xuc@osume..de edu; inton.vandenneenell.. @ @adelaide.eduuau

---

Published as a conference paper at ICLR 2026
Recent benchmarks such Math Vista Lu et al. and Math Verse Zhang et al. (2024a) evaluate math- ematical visual reasoning in MLLMs. These works conflate perception with reasoning, however, because they assess performance complex tasks that combine diagram interpretation and math- ematical reasoning. thus remains unclear whether the performance truly reflects the models' ability to comprehensively understand the symbolic information in diagrams. Perceptual misinter- pretations propagate downstream, leading to faulty reasoning and frequent hallucinations, although the final answer may occasionally still be correct Bai et al. (2024) 1 Jiang al. (2024); Wang et al.
(2224)). Developing models capable of understanding symbolic information in diagrams is a critical milestone in advancing machine intelligence Cromley et al. (2010); de Rijke (1999).
Motivated by the above, we have designed diagnostic benchmark to isolate and rigorously evaluate mathematical perception in MLLMs. MATHEMETRIC features problems that hu- mans can solve "at glance" without extensive reasoning. The benchmark contains 1,198 im- ages and 1,609 carefully designed perception- oriented questions across four distinct task cat- egories: shape classification, object counting, relationship identification, and object ground- ing. These tasks span three core mathemat- ical domains-plane geometry, solid geome- try, and graphical representations (line, bar, and pie graphs) and question formats (multiple- choice, true/false, and form-
LLoVA
41 amall
InternV1.. 5-58
Owen2 V2-22
69T-50
SVV Moth Govppeakk-
We systematically evaluate current MLLMs on MATHEMETRIC investigate key ques-
tions:
G-LLaVA-78
SWz Math-
Do current MILMS genuinely perceive mathematical diagrams? Fig. D presents the performance of eight current MLLMs generic models and two math-specific mod- els els-across across four perception-focce focusse tasks in
Relationship Idencification
Object Grounding
Object Counting
Shape Classification
Performance on MATHEMETRIC reveals that diagram interpretation is challeng- ing for MILMs, particularly in fine-grained grounding tasks that require precise spatial lo- calization. SVE-Math-DecpSeee -7B, trained on GEOMETRIC, significantly outperforms com- parators, validating the structure-aware geometric data design.
Figure 1:
MATHEMETRIC (see Tab. 1 for more MILMM results). Among the generic MILMMs, Qwen2.5-VI--BB Bai et al. (2025) achieves the highest average accuracy. followed by GPT- 4o. But both suffer from severe hallucina- rennosinng to simple questions with un- necessarily long chains-of-then of-thought and gener- ating irrelevant visual content (see Fig. 6). The math-specific SVE-Math--eeepSeek Zhang et al. 2025), with a geometric primitive visual encoder, performs on par with Qwen2.5-VI.-7B in shape classification and relationship identification. However, all models, including Qwen2.5-VIL-77 and DeepSeck-VL22 VL2-Small, which are trained on nearly 2T natural images, consistently underperform on fine-grained bounding box grounding, with accuracy below 20%.
We then quantitatively analyze factors influencing MLLMs' perceptual capabilities $3.3 provid- ing convincing evidence to answer this question. Key findings as follows: (1) models are vulnerable to subtle visual noise and irrelevant distractors, failing to attend to salient objects; (2) when diagram-text conflicts arise, models over-rely on textual information, particularly those with weaker perceptual ability; and (3) models often resort to pattern memorization instead of perceptual understanding, as evidenced by their insensitivity to vertex ordering in shape classification, even though vertex order defines shape identity under formal geometric rules.
Does stronger perceptual ability lead to better reasoning performance? To understand the potential reasons for weak diagram perception in MLLMs, we examine the limitations of exist- ing
training datasets, such as MAVIS Zhang al. (2024b) and AutoGeo Huang et al. 2024), two of the largest in mathematical vision. They often contain ambiguous expressions and obscure structural properties of diagrams (Fig. 2a
As a means of investigation, we construct
2

---

Published as a conference paper at ICLR 2026
AutoGeo
MAVIS
GEOMETRIC
Entiogle ARE
Iou ibaed dalpian
LEDIAREE of ARCOO
IO0, 100
Queetie:: Raveer daar and ocdia dccttption of image of geanati
Quextine: Decafa the iags couciety
Quastier: Plans dacoie ganeatic of Du Raaa zage
shipss @
siewss dectedd shaps
Attreer Fistly doe conagle AnCD
macking reckingle Loull hew, for the abeut RNTU aecoadd
Will
eecarage uene or hole wit th
Anreii The pertign aiecte dtandetttd by bartay vilo All
Secenda, with olge AD aatuica
aloi.. cveting
2871 cade
G aul bang in wnleee DN.. All, and Dirropottelyy
Leletas XT Taads joldd
eqplas the mnliedd SUVTU a the dita adaae cacta the and
jodge drat upuir t100
(h)
Figure 2: Illustration of diagram-caption alignment training datasets, w.r.t, AutoGeo, MAVIS, and our GEOMETRIC (Fig. 2ab with shapes in green, relationships in yellow, box locations in blue, and ambiguity in red. Fig. 2b demonstrates a positive correlation between low-level perception and high-level reasoning tasks, evaluated on MATHEMETRIC and Math Verse. Clear diagram perception leads substantial improvements in mathematical reasoning performance.
GEOMETRIC, a high-quality (geometry-image, text] pairs that encode shapes, attributes, and in- terrelationships as graphs with fine-grained bounding box locations. Training on GHOMETRIC yields strong improvements in perception tasks, achieving +79% on the grounding task (Fig. [ Additional results for Qwen2.5-vL-7B/322B trained on GEOMETRIC are presented in $3.2
Moreover, in the close-up comparison with math-specific MLLMS (Fig. 2b. colored dot points), training with our proposed dataset substantially improves perceptual performance over AutoGeo and MAVIS, by 23.0% and 35.6%, respectively. In contrast, two alternative variants underperform SEEE-Matt-DeepSee-e
with MAVIS diagram-caption pairs (despite being 5 x larger in scale than ours) yielding poorer results due to high uncertainty and the out-of-distributin nature of real-world geometric diagrams.
the baseline
To further examine the impact on reasoning, we fine-tune those three models on the same reasoning dataset MMathV3600 Shi et al.] (2024)) for a fair comparison. Two key observations emerge: (1) our model achieves 28. 1% accuracy on Verse, +~44 gain over others. This is notable be- cause the improvement arises purely from enhanced perception, without additional reasoning data, whereas Multi-Math-7B Peng et al. (2024), trained with large-scale reasoning samples and rein- forcement learning, achieves only 26.9% (2) without clear visual guidance, models privilege verbal reasoning, termed blind reasoning. For example, with AutoGeo or MAVIS, models perform compa- rably to the base model on Math Verse under the same reasoning data, showing that structure- geometric samples provide essential visual understanding for accurate reasoning. Our model accu- rately identifies relevant visual elements, enabling it to generate more valid and faithful reasoning steps (see model responses in Figs. 18).
2
EVALUATION AND TRAINING SUITE DESIGN
2.1 MATHEMETRIC
Existing mathematical visual reasoning benchmarks often conflate perception with higher-level tasks such as numerical calculation and proof generation. As evaluation is based solely on final an- swers, intermediate perception errors remain hidden. Thus, it remains unclear whether MLLMs genuinely perceive diagrams or merely rely on the prior knowledge of powerful LLMs. We intro- duce MATHEMETTIC,, the novel benchmark to evaluate MILMS' perception-demanddee abilities through both quantitative and qualitative analysis across coarse-to-fine granularity levels.
Data Composition. To comprehensively assess the perceptual abilities of MLLMs in mathematical contexts, our evaluation images cover plane geometry (66%), ), solid geometry (20%), and graphs (14%), including lines, bars, and pie charts. To facilitate MLLM evaluation, we formulate all tasks masks-except for bounding box grounding (free-form) as multiple-choice or true/false question- answering problems. In total, contribute 1,609 questions and 1.198 unique images, ensuring an even distribution across the different perception tasks. Detailed statistics for data composition are presented in Tab. of the Appendix.
3

---

Published as a conference paper at ICLR 2026
Qe the shope onject IMMI in the image?
Q: Are there cincic the picture
Q: Whea edotionetty
Q: Pleass porrid bexnding hax coontrune region sentence scalose trinngle
hetween Iiz NP Ins
ingeet
(a) right ttinngle
(a] perpedioatz
(5) oquinteral irangle
tangre
BIX
(c) cligse
(4) tninngo
posilel
Al [199 148, 317, 349
Figure 3: Sampled MATHEMETRIC examples from plane geometry w.r.f. each question-ane answerr (Q&A) type, covering shape classification, object counting, relationship identification, and object grounding (from left to right), The green dotted bounding box shown for illustration purposes only and are not provided as input to the models.
Categorization. The benchmark encompasses shape classification, object counting. relationship identification, and object grounding. Fig. [ shows example illustrations of plane geometry (see $D for additional examples). Key features are as follows:
Shape classification a classic vision task where the model identifies object classes based on attributes, i.e., vertices, material, color, and size. Our dataset includes a diverse set of geometric categories, comprising 16 basic shapes for plane geometry. 3 CLEVR- defined Johnson et al. (2017) objects for solid geometry, and 5 graphical elements as de- fined FigureQA Kahou et al. (2017).
Object counting requires models to determine either the total number of objects in an image or specific shape count, i.e., the number of circles or triangles present.
Relationship identification evaluates models' understanding of 4 spatial and over 10 math- ematical relationships between pairs of geometric primitives.
Object grounding evaluates fine-grained localization by requiring MLLMS to accurately predict the top-left and bottom-right coordinates in the format (x1, yl, x2, y2) for an object within the image. This ensures that models can precisely identify and localize geometric structures based on textual descriptions.
Question&Answer Construction. Herein, we briefly introduce the construction of ques- tion-answer (Q&A) pairs for model evaluation. The detailed construction pipeline is presented in $F of the Appendix, which documents our synthetic data engine for generating plane geome- try diagrams as well as the reformatted versions of the public CLEVR Johnson al. (2017) and FigureQA Kahou et al. (0117) datasets for solid geometry and graph representations. This data engine enables controlled shape generation, relationship modeling, and visual attribute assignment, ensuring diverse and well-balanced datasets across perception tasks.
Fig. 4 describes the entire generation process: 1) Structured annotations for geometric primitives. Inspired by AlphaGeometry Trinh et al. (2024), we use geometric clauses fundamental units, combining basic shapes with mathematically valid interrelationships. Our synthetic data engine samples from pool of 16 shapes (e.g., isosceles triangle, square, rectangle, parallclogram, isosce- les trapezoid. pentagon, circle... ... ellipse) and 10 relations (e.g., parallel, perpendicular... ... tangent) and verifies logical consistency. The outputs are stored structured JSON annotations specify- ing attributes, bounding boxes, and relationships, which are then used both for image rendering with Matplotlib Huanter (2007) and for generating question-answer (Q&A) pairs via template-based pipelines. 2) Q&A generation. From structured JSON annotations, we employ a template-basee based pipeline to generate multiple-choice, truc-false, and free-form questions for basic perception tasks. Correct answers are derived directly from annotations, while plausible distractors are generated to challenge geometric perception and combined with the correct answer into multiple-choice sets. For paired query diagrams, geometric clauses are translated into visual representations using Python code. To increase task difficulty, we apply image augmentations such as Gaussian noise, irregular scribbles, and wedge-shaped symbols for congruent angles or auxiliary lines (see Figs. and [2 in the Appendix). 3) Reformatted public datasets. To cover solid geometry and graphs, we reformat CLEVR Johnson et al. (2017) and FigureQA Kahou et al. (2017) into the same structured format, enabling consistent Q&A generation across domains.
After generating synthetic data and collecting public datasets, we conduct comprehensive review to verify answer accuracy, ensure consistency between questions and diagrams, and confirm relevance the four perception tasks, ensuring high-quality and precise dataset annotations.
4

---

Published as a conference paper at ICLR 2026
Geometric Shape
Q Generation
C Clanses Generator
M.O ey.rrispple
Sample
Exghanniion oqalitend tringle KMOQ
Prereqebikc None
A.B.C.D resiongle E interaation !t (10.
Iccester (APMG)) yf plave (1,
O. Alus
11
Chower
Prereqebitc Mt.
Exglanation: M,
(R. M,0)
Inaid
Dacad
"L.O" Vealiraal sangle neth Navie
Trcaee Pilc por KMO'
(o) 30 o0
Task 3: 3:t0get Greaniing
Counting
Relationship Pool
Add
Or Finee provic the
Chasse unceser (b2V0)
benieiapi (15030 code you 1A244
Replasation OPis exiide 52VD
Q. The vilioeda sincn 11 add lsug
Prevepuiito
K.M.C Q exporiaapte D) Wf nxester (GH3)))
Chenez
(R,3M)
Immage Generation
Explannioer: Consladt X
kat)
1. Theee 204
der Z6 y X2
Prereqeiito R.K.
lealiig 2. putt sactiti perilttu
Figure 4: We synthesize geometric figures by randomly sampling elements from the geometric shape pool and relationship pool, ensuring consistency through verifier that enforces logical constraints based on manually designed rules, fundamental mathematical principles, and prerequisite points. All visual elements are structured and saved in JSON format. Images are rendered using the Matplotlib package, and corresponding Q&A pairs generated using template-basee pipeline.
2.2 PERCEPTION-
TRAINING DATASET (GEOMETRIC)
Evaluation results on MATHEMETRIC reveal that both open-source and closed-source MLLMs struggle to identify relevant visual regions in symbolic and abstract mathematical diagrams, despite their strong performance in theoretical reasoning and numerical computation. This contrasts with human cognitive abilities, where low-level perceptual tasks are typically solved rapidly compared to high-level reasoning tasks. This raises an important question: why do generic MLLMs MLLMs-despic being trained on large-scale visual datasets Struggl to perceive mathematical diagrams? The key reason lies in the domain gap between natural images and mathematical diagrams [Lu et al.) Zhang et al. 2025). Unlike semantically rich natural images (bitmaps), diagrams are defined by precise geometric structures and symbolic notations (vectors). Without superficial patterns or se- mantic priors to exploit, MLLMs fail to generalize Geirhos et al. (2022). This underscores the need for datasets that reflect the uniquely structured, graph-like nature of diagrams. Moreover, learning from explicit structures within the data context would reduce learning complexity and enhance the problem-solving abilities of models Han al. (2025).
We thus develop a structured visual dataset- where the training corpus is built from object attributes and extends to other objects based their relationships-tu to improve visual attention and mitigate MLLMs' reliance on textual shortcuts. In specifc, GEOMETRIC follows a structured format:
First, count (N) prominent object(s) in the image. Next, for shape information, object fattrbu. } is a {shape"), }, and .... Furthermore, I also know the fine nheeggraied bounding box coordinates: the located at (box_cor.)), and .... Finally, let me explain the relationships: the (shape') {ailrbu.*) } (rela. 3) to the [shape') {attrbu.'), J. and .....
{shape") {attrbu."
To further enhance the model's ability to follow instructions, we construct task-specific instruction dataset in multi-turn conversation format. Each question is tailored to specific perception task, with answers presented either in free-form or as a selected option. For example: Q: What is the shape of object (attrbu." }}? A: {shape"). }. See S in the Appendix for additional demonstrations.
Overall, GEOMETRIC contributes MILM training by: (1) providing clear object attributes and their relationships, akin to graph nodes and edges, in training QA pairs; (2) offering fine fine-grained bounding box coordinates of elements, enabling models to systematically learn spatial awareness; and (3) integrating with reasoning- based CoT mathematical visual datasets during self-fine-tuning stage, allowing models to both perceive and reason accurately.
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Generic and Mathematical MLLMs. We evaluate 20 MLLMs on MATHEMETRIC across plane
geometry, solid geometry, and graphs, including closed-source generic models such as GPT-4o and GPT-o1, well as open-source models like LLa 1..5 Liu et al. (2023b), mPLUG-Owl3 Ye et al. (2024) MeenLLM-XComposett Dong et al. (2024), QWenZVL Wang et al. (2024c). Qwen2.5 5LL Bai
al. 2025
DeepSeek- VI.2 Wu et al. (2024). InternVI.2 Chen al. (2024)), Intern VL.2.5 Chen
5

---

Published as a conference paper at ICLR 2026
Table I: Performance comparison of different MILMs on MATHEMETRIC across plane geome- try, solid geometry, and graphs. cls, cnt, grd, and riat represent different question categories: shape classification, object counting, object grounding, and relationship identification, respectively. all indicates the overall accuracy, calculated the ratio of correctly answered questions to the total number of questions in the benchmark, while Avg. denotes the average all score across all subjects.
Aug.
Planc Goouuctry
Solid Geumetry
Gruphe
CAT
grd
off
oat
gil
rlat
of
CM
gof
riat
Authom (ours)
98.5
SW.7
99.3
99.9
1000
99.3
1001
100L0
98.7
96.3
99.8
1001
99.3
10000
Open Souvee Gosers MIZAS
1L
14.2
37.5
35.2
11
321h
118
29.3
40.4
23.5
47.0
60.5
18.1
37.6
61.8
42.6
0.0
45.
mPLUG
Ow
Ye
3:4
78
50.0
36.4
46.7
41.6
3.9
58.5
65.3
95.4
53.5
0.0
62.5
48.2
59.4
77.8
0.0
06.7
2024
94.4
Qwn3-V1
Wane 2
37245
47.
41.2
12.8
64.1
91.0
78.4
81.1
84.1
58.9
Quen V1
Wang 8
728
59.9
42.4
11.2
20.8
17.4
52.0
71.2
97.7
$4.5
77.5
66.1
76.8
98.2
16.1
34.9
Owei25.V
18.5
100.0
Qwen25
54.3
0.0
67.0
68.5
100.0
DaggSckk
Tmy
2004
38
32.6
20.5
45.2
34.4
4.6
32.0
39.0
76.7
32.0
0.0
57.5
29.4
39.1
57.4
0.0
18.2
DoopSeck-V12200
Wo.
7170
16R
51.5
17.
47.
436
12.5
48.5
08.8
7111
60.0
$1.2
76.8
53.
11.3
81.8
IntemVL
De
38.0
0.0
50.5
759
4
8B
50.7
35.0
48.8
36.0
0.0
60.0
63.5
98.5
72.2
70.0
51.4
68.1
77.8
0.0
69.7
IntemV1.
1015c
36R
63.1
44.0
59.9
53.0
1.5
66.0
78.8
98.8
92.8
38.1
12.5
66.5
06.3
3.2
69.
Vissn
Hrane stal
MINT-CVV
Laca et
2025
7B
56.7
39.1
52.7
46.4
2.9
58.0
61.7
96.5
67.0
6.4
61.3
63.3
R2.6
92.6
0.0
93.9
Qwen25.V.
(ours)
78
72.9
78.5
70.7
70.3
62.6
65.0
71.9
97.9
96.2
12.9
70.0
66.2
94.2
06.3
4.8
89.4
Qwen25-VL++ (ours)
Slord-SSwwr Ganene MIL36
53.2
3.1
62.5
72.2
33.2
11.6
0.0
14.0
75.6
52.6
52.3
$1.5
48.0
19.1
20.3
24.4
0.4
21.5
64.7
94.2
79.4
0.0
66.3
(60
03.7
77.8
0.0
69.7
Sowiee Nathomationl MELMS
0.0
90.5
37.5
47.3
59.3
G-LLaVA ef
711
10.3
25.6
27.8
41.2
0.6
38.0
32.5
65.4
39.1
4.85
12.5
33.9
58.0
37.0)
0100
42.4
PUME
Muth Ziaan
2025
LRSA
30.3
0.0
41.2
38.8
54.6
68.5
711
42.1
11.
4010
30-4
1.1
53.0
66.7
61.4
53.0
8.7
13.8
48.6
79.7
574
33
13.8
SVE-MillTTerpee Zhong dt MBS
78
46.6
154
52.4
16.
3.6
$1.0
49.4
77.9
62.9
41.3
55.1
81.3
75.9
697
SVE.Mad-DeepSeek (turs)
$8.4
63.9
96.5
60.7
78.4
et al. (2024a), Vision-R R1 Huang et al.
(2025) and MINT-CoT Chen et al. (2025). Additionally, we assess math- spec-sseeifiic MLLMs, including SVE-Math- Seep-eee Zhang et al. (2025). Math-LLaVA
Shi et al. (2024), G-LLaVA Gao et al. (2023), URSA Luo et al. (2025) and MultiMath [Peng et al. (2024). This comprehensive evaluation provides insights into the diagram perception capabilities of state-of-the-art multimodal models.
Implementation Details. For open- and closed- -souree generic MILMs, we follow official inference settings, including temperature, number of beams, and maximum token length. For open-source mathematical LLMs, we adopt the standard configurations from SVE-Math- -Meat---eeeSeek, setting the temperature to O, the number of beams to 1, and the maximum token length 1024. Choices tracted using predefined rules tailored to each MLLM's output format. We apply GEOMETRIC for and parameter-efficient LoRA training on Qwen2.5- VLL-7B332B. See $ for training implementation details.
full-parameter SFT on SVE-Math-Deep
3.2
MAIN RESULTS
summarizes the performance of current MLLMs on plane geometry, solid geometry Johnson ct al. (2017), and graphs Kahou et al. (2017). Below, we provide an analysis of their performance on MATHEMETRIC
Tab.
Generic MLLMs. General-purpose models trained on diverse datasets, including tables, charts, and documents Chen et al. (2019); Kahou et al. (2017); Yuan al. (2022). as well as visual grounding datasets Shao et al. (2019); You et al. (2024). still perform poorly on mathematical diagram per- ception. In particular, their performance plane geometry remains low, with most models scoring below 45% on average. For solid geometry and graphs, general- purpose models significantly out- perform mathematical MLLMs, due to their exposure to large-scale FigureQA Kahou et al. (001)), CLEVR Johnson et al. (2017), and various chart understanding datasets Yuan et al. (2022). How- ever, they still fail in fine- mee-griined box-level tasks, with most models achieving accuracy at an loU threshold of 0.65, and lag significantly behind human-level perception. The symbolic and structured diagrams require genuine visual understanding rather than superficial pattern recognition. Without this, models fail to identify where look, leading poor perception performance. Notably, scaling

---

Published as a conference paper at ICLR 2026
up model size is neither optimal effective solution, as it provides only marginal gains in perception compared to reasoning benchmarks. For instance, increasing Qwen2VL from 7B to 72B improves top-1 accuracy by 22.3% on Math but only 8.3% on MATHEMETRIC.
Mathematical MLLMs. Models such as MultiMath, Math-LLaVA, and SVE-Math- -ate--eeepSeek fine-tuned from generic MLLMs (i.e,, LLaVA and DeepSeek) using mathematical visual data, achieve strong reasoning performance on Math Verse ZZhang et al.] (2024a). MathVista Lu et al.) GeoQA Gao al. (2023) and MATH-V Wang al. (2024b), but struggle with geometric perception across all three subjects.
Among them, SVE-Math-DeepSeek outperforms other mathemati- cal MLLMs by incorporating primitive visual encoder trained with detection and boundary seg- mentation losses. As shown in Tab. 2 SVE-Math-DeepSeekk an enhanced version trained on GEOMETRIC, and achieves significant gains in plane geometry perception. Notably, even without direct training solid geometry and graphs, it outperforms others due to its improved ability to discriminate relationships and understand spatial configurations. To further validate the positive correlation between perception and reasoning, we fine-tune strong generic Swen2.5-VL-7B/322 on GEOMETRIC. Tab. shows that our dataset yields ~ +4% improvements over WVEE-Math-eeepSeee and ~ +3% gains over Owen2.5-VL-7BB.
Table 2: Performance comparison on math perception and reasoning benchmarks.
Model
MATHEMEETRIC
MothVerse
Matt GeoQA
MATB-V
Cople
QweeES VL Briecall
2025
46.0
59.0
65.7
4912
69.2
76.4
25.1
Valco R1 mitiaag midl
356
66.6
08.4
52.4
73.5
78.
MINT-CAÉ 18:37
2005
39.1
51.7
63.3
67.8
62.1
Pure
Man: LLAV 138 Sh dt
2004
219
44.8
47.3
20.1
45.6
60.7
15.7
G-1 Va--i Grn bicg
256
11.
31.9
17.8
25.6
64.2
SVE-Mrb- Desppesk 18
35.1
24.3
48.7
SVE.Math- Deppeekk7B (ours)
846
54.1
60.7
25.1
51.3
76.2
16.6
Qwen25-VL-77 (omr))
78.5
11.9
68.2
52.6
70.3
79.6
27.3
Q1.025-11-3218
71.1
76.9
across four mathematical reasoning benchmarks. Although we do not employ explicit visual-text integration mechanisms, as done in MINT-CoT model still exhibits automatic adaptation and task transfer from low-level perception to high-level reasoning, demonstrating that these two abilities are complementary, As shown in the qualitative comparisons between the base model and the GEOMETRIC-fin--tun many previously incorrect cases are resolved simply by correcting a key perceptual error, and the resulting improved perception stabilizes multi-step reasoning chains. Another related line of work is ceinfrreeennt-lermiigg based reasoning enhancement, as in Vision-RI Huang et al.] (2025). which incentivizes reasoning ability built large-scale and corss-domain perception-reasoninn datasets, requiring much heavier computational resources. Exploring how to explicitly model visual-text interactions remains a promising direction for understanding how enhanced perception can further support and strengthen reasoning. We provide an in-depth analysis in C
al. (025). our
model (Figs. 116-18).
A431
VL.72-VLL-7B
SVVL-7
30
DespSeek-VL.2. VL.2-Tiy
Venee di pniciin
Deeppeck Smal-
L1-14-1.5-138
060
IntanVi 2:548
945
GPT-4
806
R01 dww. 0077
SVU Math DeepSeek
Oif-trer
SVL-Math- 7-7
V12 Sual)
#CF
5SVE-Mat Das Begdlad--7
Figure 5: We evaluate five key factors influencing Figure 6: Perceptual errors under Chain-of- MLLM perception: object count (# obejet), visual quality, visual distractors, textual distractors, and Shain-of-Thoughh (CoT) reasoning, with close-up
Thought (CoT) vs. direct reasoning. Chain-of- thought (CoT) errors occur when model pro- vides step-by-step explanations yet yields an in- correct answer.
results shown in Tabs 4 and Fig. 6
7

---

Published as a conference paper at ICLR 2026
3.3
ABLATION STUDY
Since mathematical models are trained exclusively on math-domain datasets, they offer a more con- trolled setting for analyzing perception capabilities compared purrpos- MLLMs. There- fore, we choose SVE--ath-DeeppSek as the base model for the below ablation study.
3.3.1 KEY FACTORS INFLUENCING DIAGRAM PERCEPTION
We examine five key factors: the number of objects, object quality, visual distractors, text distractors, and Ohain-of-Thhuggh (CoT) responses. While those factors are by no means exhaustive, they aim to illuminate some of the fundamental perceptual limitations of current MI..Ms, providing insights for practical applications and future improvements. The overall accuracy (all) on plane geometry from MATHEMETRIC across nine MILMS is presented in Fig.
The Number of Objects. Visual perception complexity increases with the number of objects. We observe that as the number of geometric shapes grows, models struggle with accurate object lo- calization, shape classification, and counting. When scaling the number of objects from I to N (N < 5). Qwen-series model's 's performance drops by approximately 12%, the DeepSeek-series by 18%, and GPT-4o by 15%. Our method and its baseline show declines of 6% and 13%.
Table 3: Accuracy (%) on plane geometry per- ception w.t. object quality (w/ or w/o Gaussian noise) and visual distractor injection.
Table 4: Performance of relationship identifica- tion (rlar) on MATHEMETRIC under different textual distractor settings.
MATHEMETRIC
Noise ~ (0000.3) 0.3)
Disuactors w/o
MATHEMETRIC (nlar)
Unrela,
Conflicts
w/o
w/ati
w/At1
w/o
w/xt
w/o
w/srs
Qwen2V1.-78
29.8
33.9
29.8
26.2
Qwen2VL-7B
$3.0
48.5
53.0
24.5
Owen2.5VL-7B
33.9
32. 11.a
33.9
31.0
Qwen2.5VLL-78
52.0
56.0r4.
52.0
30.0,22.0
DeepSeck Tiny
27.9
23.8
27.9
22.6,5.3
DeepSeek V1.2-Tiny
32.0
34.5:2.
32.0
3.5
DeepSeck-VL.2Small
27.9
37.5
27.9
25.9,aa
DeepSeck-VI22 VL--Sce-VL22Small
48.5
42.5,n0
48.5
20.5
LLaVA-v1.5-1333
26.8
31,6r4.8
26.8 29.1+2.3
LL5VA.v1.5-13B
37.5
16.0
IntemVL2.5588
25.6
20.8
25.6
19.6,6.0
IntemVL2.5-8B
60.0
49.0
60.0
18.0,
GPT.4o
[0.9
32.7
28.6
GPT-4o SVE-Math- 7
62,5
63.5-1.0
62.5
38.0
SVE. DW--Math-DeepScck-78 7B
32.8
28.0
32,8
29.2,a.
49.0
15.0 1600
SVE-Math-DeppScck *-7B
80.7
78.3
80.7
80.3.o.4
SVE-Math-DeepSeek 1.7B
96.5
95.0
96.5
82.5
Table 5: Additional results for "blind faith in text" phenomenon. Accuracy (%) on mane-geometry (rlar) under explicit modality-prioritn prompts.
Table Accuracy (%) on plane geometry for shape classification (els) w.r.t. vertex or- dering (clockwise v.s. random).
MATHEMETRIC (riar)
Conflicts
Neutral
Visual-priorty w/an
Text-priority W'at..
MATHEMRTRIC (clg)
Random
Cloekwise
w/o
w/ans
Qwen2V1--78
47.4
47.6
Qwen2VL--7B
53.0
24.5
25.5
25.5
Qwen2.5VL-77
56.7
56.2
Qwen2.5VL-78
52.0
30.0
30.5
28.3
DeepSeek-VEL Vin2-Tnny
45.0
45,2
DeepSeek VL2-TTny
32.0
3.5 3.5,08.5 28.5
11.0
9.5
DeepSeek.VV... V12--mmall
47.1
47.6
DeepSock-
48.5
20.3,ps.0 o
22 02.00.
19.5
LLaVA-v1.5-13B
16.0 ps.
15.0gmm
14.5
LLaVA-v115-7B
29,8
29.0
InternVL2.5-8B
60.0
18.0
17.0 17.0pa
15.5
IntemVV1.2.588B
48.3
48.8
GPT-4o
62.5
38 0.....
39 p
34.0
GPT-46
57.8
58.4
Dr-MaattDDopSck-22
$1.0
15.0 .8.0
15.0 paec
12.5
SVE-Math DecgSook-BB
SVE-Muul-DoyySed-TB
51.2
SVR-Math-DeepSeeks -78
96.5
82.5
87.5gp.0
84.0
SVE-Math-DoepSock .78
74.9
75.8
We draw visual ietractoors-irregue scribbles, wedge-shaped angle mark- ers, and auxiliary lines-to evaluate MLLMs' ability focus on relevant geometric elements (see Fig. 10). Despite explicitly prompting models to ignore distractors (e.g., the input diagram contains visual distractors on the target foreground objects; please ignore them when performing the percep- tion tasks), most are still negatively affected. Close-up results are shown in Tab. 3 All evaluated MILMMs show a performance drop of 2-6% under visual distractors, except -LLVVA-1.--133 which improves by 2.3%. Our model remains robust, showing minimal sensitivity to distractors and better visual focus on geometric primitives.
Visual Distractors.
Object Quality. For visual fidelity analysis, we apply Gaussian noise to degrade object quality. As shown in Tab. 3] with a variance of 0.3, most models degrade, while Qwen2VL, DeepSeek VL2- Small, and LLa VA-1.5 13B unexpectedly improve. This may result from exposure to degraded vi- suals during training, making noisy images better aligned with their learned distribution cean---sspeillll given the lack of clean geometric diagrams in the training data. To enable more comprehensive
especially
8

---

Published as a conference paper at ICLR 2026
FE-MMath-DeepSeed
SVE-Matl-DeepSee DeeepSek
InternVL2.5
2
x+1
18
Since trangle WSR similar trinngle we
X Using the proparty of sumilar crangles, we establish the equation Mit
X Since the triangles are similar, the mioo of the conesponding equal, so R2
Question:
Two triangles are similar. wS 8, and VT is Find WR.
Given that WS and 10. we can caleulite RW sn RW 0422 B
X Subatituting the given values, we get
X Subsiitute the given values: io waree Then, WRt-24 Therstore. the correct auswer is
MMM Solving WR yields Therefore, anewer option D
Choices:
concect
6:6
B:
C:10
D: 12
Figure 7: Response comparisons between SVE-Math- SVE-Matt-DeepSeek -7B. SVE-Math- SVE-Math-DeepSeek- and InterVL.2.588B in Math Verse. More demos are shown in Figs. 16 of the Appendix.
evaluation and mitigate data bias, we introduce stronger noise (variance 0.5/0.8). As distortion in- creases, all models show sharp drops in geometric recognition. Accuracy under noise is shown in Fig. IT with example distortions in Fig. 12
Text Distractors. To examine the influence of textual modality in vision-centered tasks, we evaluate two settings within the relationship identification task. The first introduces irrelevant information unrelated to the target, such foreground/backkroun colors object vertices. The second presents contradictory cues: a diagram with clear visual cues (e.g., two visibly parallel lines) and no ambigu- text; and the same diagram but with contradictory textual distractors added during inference (e.g., the two lines are perpendicular). In both cases, correctness is determined by whether the model's answer matches the ground truth defined by the visual cues in the diagram, irrespective of the textual guidance. The results in Tab. 4 show that providing conflicting knowledge significantly impairs the perceptual ability of all evaluated MLLMs. The impact is especially pronounced in models with weaker perception capabilities such as Intern VL2.5-8B (48.4% in Avg.), while stronger perceptual models such as Qwen2.5-VI. VI.-7B (59.2% in Avg.) are less affected. Our method, which achieves the best perceptual performance, demonstrates greater robustness but still suffers a 14% drop. This highlights that MLLMs tend to over-rely on textual input when inconsistencies arise, exhibiting a "blind faith in text""
These conflict experiments were conducted under neutral settings, where no modality was favored during inference. To further examine whether explicit prompting mitigate this behavior, conducted additional controlled experiments using system prompts that explicitly modulate modal- ity priority: visual-priority prompi: "Carefully examine the diagram and prioritize visual informa- tion. Only use text labels to confirm what you observe visually"; text-priority prompt: "Focus on the textual labels and annotations in the diagram. Use the visual structure support the textual information". The results in Tab. 5) show that even when explicitly instructed to prioritize visual information, the model still exhibits "blind faith in text". although performance is marginally higher than under the text-priority prompt. Addressing this limitation should therefore be an important focus for future research.
Chain-of- Thituggh (CoT) Response. CoT reasoning is designed to enhance step-by-step logical inference, yet its effectiveness in visual perception tasks remains uncertain. Our analysis shows that while Co'I' improves textual reasoning. it does not directly enhance spatial geometric understand- ing. Models incorporating CoT reasoming often struggle with fundamental perception tasks, leading to significant CoT errors. Models often generate excessive yet irrelevant rationale misaligned with the diagram, ultimately resulting in incorrect responses, particularly in Qwen-series models and GPT-4o (over 30% in Fig. C). See for CoT response examples in the Appendix.
3.3.2 DIAGRAM UNDERSTANDING REMAINS CHALLENGING
To further examine the diagram understanding of MILMs, we design vertex-ordering ablation. Vertex order is a strict mathematical constraint: in a consistent clockwise order, vertices must form a closed shape. This ablation tests whether MLLMs rely on genuine geometric understanding or merely statistical pattern matching. We therefore randomize vertex order for each object, and results in Tab show that models are insensitive to this change. This insensitivity suggests that MLLMs do not internalize the geometric rules governing shape formation, but instead depend surface-leve patterns. Enabling models to grasp such geometric constraints remains a challenging yet promising direction for advancing true diagram understanding.

---

Published as a conference paper at ICLR 2026
3.3.3 COUNTING OVERLAPPING OBJECTS Is OVERLY DEMANDING
In our synthetie plane-geometry diagrams, we preserve object separations reduce ambigu- ity in counting, but overlaps remain unavoidable as object density increases. By default, MATHEMETRIC directs models count only prominent objects, ignoring complex overlaps. We ablate this design choice by requiring models to count all potential overlaps. Results show that over- lapping cases are far more difficult than non-overlapping ones (54.8 v.s. 32.8 for Qwen2.5-VL.-75. 88.2 v.s. 67.3 for -VF-Math-DeepSeeee +-7B). See Tab. and $C for detailed analysis and settings.
Table Performance comparison w.r.
Table Performance comparison on math perception and reasoning benchmarks. The symbols and de- note models trained without GEOMETRIC and with a frozen visual encoder during the SFT, respectively.
variants trained on GEOMETRIC and
other mathematical alignment datasets. *
is the pretrained model continuously self-
fine-tuned (SFT) using MathV36l0 and custruction-formatee GEOMETRIC.
MATHEMETRIC
Model
Math Verse
Math GeoQA
Soild Graphs
Model
Plane Soild Graphs
MATHEMETRIC
MathVerse Verse Vatha
Math
SVE Path-ath-aae Seth--DepSeck--- 7BB
SVE.Math- Math-DeepSSaa-78 7B
84.6 54.1
35.4 49.4
55.1
60.7
24.3
28.1
48.7
51.3
76.2
72.8
SVE-Malh- DeepSeek
35,4
19.1
55.1
24,3
48.7
WVE-Mati-DeupSeee 1-78
25.0
72.5
Align
+AiitoGie (A)
18.5
34.7
40.4
17.8
45.0
SVE Math- DeepSeck -B (o)
82.9 53.6
61.2
26.2
50.1
74.6
+MAVIS (0)
6.0
2.5
6.8
GEOMETRIC (2)
41.6
40.2
49.2
19.7
46.2
Qwe-2.5-VL-7B
11.0
69.0
65.7
49.2
68.2
76.1
25.2
Qwen2. V-V.1-8
7X.5
71.9
68.2
52.8
70.3
79.6
SFT
78.1
42.3
54.1
23.3
47.1
Qwen2.5.VL. (+)
45.3
65.0
75.2
84.6
54.1
60.7
28.1
51.3
Swen2.5-VI.'-778
77.3
72.5
68.9
52.0
70.0
78.2
3.3.4 EFFECT OF GEOMETRIC
Building on WVE-Matt-DeepSeede we first train a projector while freezing LLM and visual encoder in the visual-language alignment stage using either AutoGeo, MAVIS, or GEOMETRIC image- caption aligmment datasets. Training with GEOMETRIC yields +6.2% on perception tasks but suf- fer substantial drop when training on other datasets (~~27% in Tab. 7). We then self-fine-tune the three pretrained models using Math and intruction-formatiie GEOMETRIC. Training with GEOMETRIC yields a notable +~44 on Math Verse and Math Vista over EVE-Math-DeepSeee.
whereas other variants underperform on Math Vista.
To better assess the contribution of our perception-orient GEOMETRIC, we conduct ablation studies by removing GEOMETRIC and using only mathematical visual reasoning datasets, such as V360K and MultiMath the SFT stage. Additionally, we evaluate configurations where the visual encoder is frozen during the SFT stage, and only the projector and language model are up- dated. Tab. |
shows that without GEOMETRIC, SFT on reasoning data alone yields only marginal improvements- even performance degradation--compared to the base models. For example, Qwen2.5-VL.+-7B (o) performs 3.9% worse than Qwen2.5-VI--7B on Math Verse. This degradation may stem from overfitting to the reasoning dataset, resulting in excessive reliance textual modal- ity. In contrast, integrating GEOMETRIC into the training process enables the model to learn where to attend visually during reasoning, leading improved overall performance. Moreover, compar- isons between SVE-Math-DeepSeekk and SVE-Math- -eep-See (o), as well as Qwen2.55VLL+-.. and Qwen2.5-VL. VL+ (o), further underscore the importance of optimizing the visual encoder to enhance visual perception.
CONCLUSION
Unlike semantically rich natural images, mathematical diagrams are structured and abstract, defined by symbolic elements and precise relationships. Current mathematical benchmarks often conflate reasoning and perception, making difficult assess true diagram understanding in MLLMs. To address this, we introduce MATHEMETRIC benchmark specifically designed for evaluating diagram perceptual capabilities. While these tasks appear trivial for humans to solve they present significant challenges for current models. Through systematic evaluation across 20 MLLMs and detailed ablations, we identify several critical factors that degrade diagram perception, including text over-reliance, sensitivity symbolic perturbations, and limited fine-grained spatial grounding. Our analyses further highlight the necessity of high-quality, structure- aware training data. Models trained with GEOMETRIC dataset exhibit substantial improvements in perceptual accuracy and demonstrate measurable benefits on downstream reasoning benchmarks, bridging the gap between visual understanding and logical inference.
glance",
10

---

Published as a conference paper at ICLR 2026
5 ETHICS STATEMENT
This work focuses on advancing diagram understanding and exploring cross-suite transfer to mathe- matical reasoning in multimodal large language models (MLLMs). Our datasets are either syntheti- cally generated or derived from publicly available resources. For reused datasets, we adopt CLEVR (CC BY 4.0) and FigureQA (CCO 1.0); all reformatted files (e.g.. JSON conversions and lookup tables) are redistributed under the same licenses as their original sources. For GEOMETRIC, we will release the full package, including
generated data and the code used to produce it, under CC BY 4.0 upon acceptance, ensuring maximal reusability while preserving attribution to (a) the CLEVR and FigureQA authors and (b) our clause engine. This approach follows open-science best practices while safeguarding attribution and respecting licensing terms. Our work does not involve human subjects or sensitive information, and both models and datasets are intended solely to advance education and scientific discovery.
REPRODUCIBILITY STATEMENT
We have taken several steps ensure the reproducibility of our work. Details of model architectures and evaluation protocols are provided in x2 and $ with additional implementation details, train- ing configurations, and hyperparameters included in Appendix $C Our datasets are either publicly available (CLEVR, FigureQA) or synthetically generated; the data generation process and full re- formatting details are described in Appendix $F
We release all code, data, evaluation benchmarks, and model checkpoints under a CC BY 4.0 license to maximize transparency and reusability.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023a.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXhv:2303.0877 0877774,, 2023b.
Lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican. Malcolm Reynolds, et al. Flamingo: visual language model for fow-shot learning. In Advances in neural information processing systems, pages 23716 23736, 2022.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorska Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation based formalisms. In Annual Meeting of the Association for Computational Linguistics, pages 2357-2367, 2019.
Avinash Anand, Raj Jaiswal, Abhishek Dharmadhikari, Atharva Marathe, Harsh Popat, Harshil Mital, Ash- win R Nair, Kritarth Prasad, Sidharth Kumar, Astha Verma, et al. Geovqa: A comprehensive multimodal geometry dataset for secondary education. In 2024 IEEE 7th International Conference on Multimedia Infor- mation Processing and Retrieval (MIPR). pages 102-108. IEEE, 2024.
Zahra Babaiee, Peyman M. Kiasari, Daniela Rus, and Radu Grosu. Visual graph arena: Evaluating visual conceptualization of vision and multimodal large language models. arXiv preprint arXiv:2506.06242 2025. Dataset available at htttt:// /ga. csail edu
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1399 2025.
Zechen Bai, Pichao Wang. Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallu- cination of multimodal large language models: A survey. arXiv preprint arXiv:2404. 18930, 2024.
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 3313-3323, 2022.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang. Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-hased fact verification. arXiv preprint arxiv: 1 1909.02011110. 164, 2019.
11

---

Published as a conference paper at ICLR 2026
Xinyan Chen, Renrui Zhang, Dongzhi Jiang, Aojun Zhou, Shilin Yan, Weifeng Lin, and Hongsheng Li. Mint-cot: Enabling interleaved visual tokens in mathematical chain-of-thoupe reasoning. arXiv preprint arXiv:2506.05331, 2025.
The Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.052777 2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Intervl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Procedings of the IREF/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-24119 2024b.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang. Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. Advances in Neural Information Processing Systems, 37:135062 135093, 2025.
J. G. Cromley, L. E. Snyder-Hogan and U. A. Lubis-Dubas. Cognitive activities in complex science text and diagrams. Contemporary Educational Psychology, 35(1):59 74, 2010.
M. de Rijke. Logical reasoning with diagrams. Journal of Logic, Language and Information, 8(3):387-390, 1999.
Linger Deng. Yuliang Liu, Bohan Li, Dongliang Luo, Liang Wu, Chengquan Zhang, Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, et al. R-cot: Reverse chain-of of thought problem generation for geometric reasoning in large multimodal models. arXiv preprint 24Xx::2400.17885 2024.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internim-xcomposee2:: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401. 16420, 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery. Brian Ichter, Ayzaan Wahid, Jonathan Tompson. Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, pages 8469-8488, 2023.
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, al. G-llava: Solving geometrie problem with multi-modal large language model. arXiv preprint ar3iv:2312. 1370, 2023.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Bren- del. Imagenet-traine cnns are biased towards texture; increasing shape bias improves accuracy and robust- ness, 2022.
Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stee Stay, Dhruv Batra, and Devi Parikh. Making the v vqa matter: Elevating the role image understanding in visual question answering. International Journal of Computer Vision, 127:398-414, 2019.
Haoyu Han. Yaochen Xie, Hui Liu. Xianfeng Tang. Sreyashi Nag, William Headden, Yang Li, Chen Luo, Shuiwang Ji, He, et al. Reasoning with graphs: Structuring implicit knowledge to enhance Ilms reasoning. arXiv preprint arXiv:2501 008445, 2025.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shanhui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint 0xXiv:2503006744 2025.
Zihan Huang, Tao Wu, Wang Lin, Shengyu Zhang, Jingyuan Chen, and Fei Wu. Autogeo: Automating ge- ometric image dataset creation for enhanced geometry understanding. arXiv preprint arXi::449.000339 2024.
John D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science Engineering, 9(3):90-95, 2007.
Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 27036- 27046, 2024.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Gir- shick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceed- ings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 2901-2910, 2017.
12

---

Published as a conference paper at ICLR 2026
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Åkos Kádár, Adam Trischler, and Yoshua Ben- gio. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07:000 2017.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 13299-13300 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen, Evaluating object hallucination in large vision-language models. In The Conference on Empirical Methods in Natural Language Processing, 2023.
Zechao Li, Yanpeng Sun, Liyan Zhang, and Jinhui Tang. Ctnet: Context- based tandem network for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, (4(12):9044-99 2021.
Zhenwen Liang, Tianyu Yang, Jipeng Zhang, and Xiangliang Zhang. Unimath: A foundational and multimodal mathematical reasoner. In Proceedings of the Conference Empirical Methods in Natural Language Pro- cessing, pages 1126-7133, 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. European Conference on Computer Vision, pages 740-755, 2014.
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Hisenschlos. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. In The Annual Meeting Of The Association For Computational Linguistics, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in neural information processing systems, pages 34892 -49916, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216-233, 2024.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference Learning Representations.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-chun Zhu. inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In Annual Meeting of the Association for Computational Linguistics, pages 6774-6786, 2021.
Ruilin Luo, Zhuofan Zheng, Yifan Wang, Xinzhe Ni. Zicheng Lin. Songtao Jiang, Yiyao Yu. Chufan Shi, Ruihang Chu. Jin Zeng. et al. Ursa: Understanding and verifying -f-thughttt reasoning in multimodal mathematics. arXiv preprint arXiv:2501 104686, 2025.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang- Conditional detr for fast training convergence. In Proceedings of the IEEE International Conference on Computer Vision, pages 3651 3651-3666 2021.
Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty. Oer-vqa: Visual question answering by reading text in images. In International Conference on Document Analysis and Recognition, pages 947-952. IEEE, 2019.
Shuai Peng, Di Fu, Liangeai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. Multimath: Bridging visual and mathematical reasoning for large language models. arXiv preprint arXiv:2409.001447 2024.
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1466 1466-1476, 2015.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Ob- jects365: A large-scale, high-quality dataset for object detection. In 2019 IELE/CVF International Confer- ence on Computer Vision (ICCV), pages 8429-8438, 2019.
Wenhao Shi, Zhiqiang Hu, Yi Bin. Junhua Liu, Yang Yang. See-Kiong Ng. Lidong Bing, and Roy Bai Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint 240v 17294, 2024.
13

---

Published as a conference paper at ICLR 2026
Yanpeng Sun, Huaxin Zhang, Qiang Chen, Xinyu Zhang. Nong Sang, Gang Zhang, Jingdong Wang, and Zechao Li. Improving multi-mode large language model through boosting vision capabilities. arXiv preprint arXiv:2410.13733, 2024.
Trieu Trinh, Yuhuai Tony Wu, Quoc Le. He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625:476-483 2024.
Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Milm can see? dynamic correction decoding for hallucination mitigation. In The Thirteenth International Conference on Learning Representations, 2024a.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjic Zhan, and Hongsheng Li. Mcasuring multimodal mathematical reasoning with math-vision dataset. In Advances in Neural Information Processing Systems, pages 95095 95169, 2024b.
Ke Wang, Junting Pan. Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In Advances in Neural Information Processing Systems, pages 95095-95169, 2025.
Peng Wang. Shuai Bai. Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint arXiv:2409. 12191, 2024c.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseck-vi2: Mixture-of-cxperts aision-lagguage models for ad- vanced multimodal understanding. arXiv preprint arXiv:2412.10300 2024.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. In Inter- national Conference on Learning Representations. 2024.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere any granularity. In The Twelfth International Conference on Learning Representations, 2024.
Weihao Yu, Zhengyuan Yang, Linjee Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International Conference on Machine Learning, pages 57730-57754, 2024.
Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-aware network for handwritten mathematical expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4563- 2022.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-disciliine multimodal understanding and rea- soning benchmark for expert agi. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9556 9567, 2024.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-moda 1lm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169 186. Springer, 2024a.
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong. Jiaming Liu, Aojun Zhou, Bin Wei, al. Mavis: Mathematical visual instruction tuning with an automatic data engine. arXiv preprint ar Xiv:2407.087333 2024b.
Guo.
Shan Zhang, Aotian Chen, Yanpeng Sun, Jindong Gu, Yi-Yu Zheng, Piotr Koniusz, Kai Zou, Anton Van Den Hengel, and Yuan Xue. Primitive vision: Improving diagram understanding in milms. In International Conference on Machine Learning, 2025.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2020.
Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng. Math-puma: Progressive upward multimodal alignment to enhance mathematical reasoning, In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 26183-26191, 2025.
14

---

Published as a conference paper at ICLR 2026
Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs
Appendix
A RELATED WORK
A.1 MATHEMATICAL REASONING BENCHMARK
To evaluate MLLMs performance across different domains, various benchmarks Li et et 2 Yu ct al. )202)): Li et al. 2023);; Goyal et al. (2019) Liu et al. (2024) have been proposed, primarily fo- cusing on natural scene understanding. However, benchmarks specifically designed for multimodal mathematical reasoning remain searce.
Early benchmarks such as MathQA Amini et al. (2019), UniGeo Chen et al. Geometrra Lu iet al. (202I), GEOS Seo et al. (2015), and GeoQA++ AAnand et al. (2024) introduced multi- modal mathematical tasks but were limited in scope, often focusing specific subdomains like plane geometry. More recent efforts have sought to provide broader and more diverse evalua- tions. MMMU et al. (2024) assesses multimodal mathematical understanding with an emphasis on symbolic reasoning and word problem-solving. Math Vista Lu et al.] targets geometry-related tasks by integrating both real-world and synthetic diagrams to evaluate visual reasoning. Math- Verse et al. (2024a) expands this by incorporating a wider range of multimodal challenges in- volving charts, graphs, and structured visual content. MATH-V /aag al. (2025) further addresses the limitations of existing benchmarks by curating 3,040 high-quality math problems sourced from real-world competitions. While these benchmarks introduce visual elements, their core focus re- mains on assessing mathematical reasoning. It remains unclear whether the performance on these tasks truly reflects the models' ability to comprehensively understand the symbolic information in diagrams. The ability to accurately interpret mathematical symbols, diagrams, and spatial structures a fundamental component of solving multimodal math problems, yet current benchmarks place limited emphasis this aspect.
A.2 MATHEMATICAL MLLMS
Recent multimodal large language models (MI.LMs) Alayrac et al. (2022 Liu et al. (2023b); (Sun et al. (2024) have made significant strides in vision-language understanding. However, foundation models such as GPT-4V Achiam et al. (2023b), Qwen2-VL Wang et al. (2024c), and Deepseek- VL2 Wu al. 2024), while strong general multimodal tasks, exhibit notable limitations in mathematical symbol recognition, spatial reasoning, and logical deduction reeiiernn them insuf- ficient for rision-based mathematical problem solving.
Recent efforts have introduced math-specific MLLMs for improving mathematical reasoning. Al- phaGeometry Trinh al. (2024) achieves state-of-the-ard results in geometry by leveraging theorem- proving and reinforcement learning. However, relies solely on text-based diagram descriptions, lacking direct image processing. G-LLaVA Gao al. (2023) extends LLaVA with geometric rea- soning capabilities but struggles with complex visual structures and generalization beyond plane geometry. UniMath Liang al. (2023) integrates structured math representations for solving vi- sual word problems but remains focused on symbolic reasoning, limiting its handling of free-form mathematical diagrams. MatCha [iu al. (2023a) specializes chart-based reasoning, extracting quantitative relationships from structured visual data. However, its reliance predefined formats limits adaptability to unstructured mathematical visuals.
Beyond architecture improvements, MAVIS Zhang et al.) (2024b) introduces an automated data en- gine to generate large-scale mathematical visual datasets, reducing annotation costs while ensuring high-quality diagram-caption pairs and problem-solving rationales. However, its diagrams are con- structed by simply combining basic geometric shapes, lacking mathematical relationship constraints such perpendicularity and parallelism. AutoGeo Huang al. (2024) considers special properties of lines, such as midlines and radii, as foundational to many geometric theorems, incorporating these properties into geometric figures. Reverse Chain-of-Thouue (R-Co'T) Deng et al. (2024) introduces

---

Published as a conference paper at ICLR 2026
Table Key statistics of MATHEMETRIC are summarized in Tab. 9a and the subject-task distri- bution is illustrated in Fig. 9b
puadrdatoral
brepesoid
quxttengis
AN Tib
dirlle
segment
scatine trangle
Stutistic
Number
peroagess
úinscales triangle
deg
Total questions
Releten Counting
Free-form questions
Mulliple choice questions
406 (25. 2%)
893 (55.5%)
Dlssuiiatton
Ciasiiictioo
right trangla
eqqullateral trannge
iadccele trapessid
false questions
(19.3%)
Grounding
Unique number of images
1,198
spella Relation
Graphs
rectingle
trapecoiid
Unique number of questions
Unique number of answers
Total questions
1609
380
Sald Ard Aeshing
Classifiation questions
Counting questions
401 (24. (2%.9%
489 (30.4%)
10
Courning
Piame Gemmenry
Relation questions
313 (19.5))
IN
Casuiication
Counthng
Grounding questions
(25.2%%
sphere
Maximum question length
200
cube
Muximum choice number
Maximum answer length
4
oylinerr
Grounding
RASA Retfection
Average question length
Average answer length
1,8
Average choice numbeer
3.5
Average question length
16.09
recicle
Average length
1.21
Average choice number
3.40
(a)
(b)
the Geometry Generation Chain to generate the geometry image and corresponding description. However, current data engines still fail to explore the underlying structures mathematical dia- grams, leading to ambiguous captions and redundant information that negatively impact MLLMs' perception abilities. Consequently, models trained on such datasets perform poorly on perception- demanding tasks, even falling below random guessing on multiple-choice questions. Visual Graph Arena Babaice et al. (2025) corroborates this finding in the context of graph diagrams, where models fail to recognize that differently laid-out graphs represent the same underlying concept. These find- ings underscore the urgency of constructing structured symbolic visual datasets that enable models to both perceive and reason effectively.
B DETAILED DATA STATISTICS OF MATHEMETRIC
To comprehensively evaluate the perceptual abilities of MLLMs in mathematical contexts, MATHEMETRIC includes 1,609 questions grounded in 1,198 unique images. These span three key domains: plane geometry (66%), ), solid geometry (20%), and graphical representations (14%), such as line, bar, and pie charts (see Tab. 99.. The benchmark comprises four core perception tasks: shape classification (489 questions, 30.4%) object counting (401, 24.9%), relationship identification (313, 19.5%), and fine-graine fine-grained object grounding (406, 25.2%). Except for the grounding task, which uses free-form answers (e.g., bounding box coordinates), all other tasks are framed as multiple-choice (55.5%) true/false (19.3%) formats to support scalable and consistent evaluation. By default, ground truth answer is randomly assigned with equal probability. 25% for each choice letter and 50% for true/false. In our benchmark: the proportion of answer A/B/C/D is 26.7% 1.%...... 1.t..%%..%/////// 22:2%% and 48.3 48.3% %1.7% for true/false. As summarized in Tab. 9 the dataset includes 1,514 unique ques- tions and 380 unique answers, with an average question length of 118.0 tokens and answer length of 1.8 tokens. This design supports fine- ime-grained analysis of model performance across diverse percep- tual challenges and subject domains.
C TRAINING IMPLEMENTATION DETAILS
We conduct experiments on both the math-specific WVEE-Mat-DeeepSSe and generic Qwen2.5-VL models (7B and 32B). Both follow two-stage training pipeline: i) alignment training with image- caption pairs and ii) instruction tuning for reasoning and perception.
For SVE-Math. Math-DeepSSek. (GEOMETRIC) while keeping the visual encoder and language model frozen. In the second stage, perform full-parameter self-fine-tuning (SFT) using combination of instructive GEOMETRIC
first train the rision-languse projector using image-caption pairs
2

---

Published as a conference paper at ICLR 2026
samples and M6tK updating the visual encoder, projector, and LLM jointly, For the generic Swwe2.5-VV-77B3222 models, we follow the same first-stage alignment process. To preserve prior reasoning capabilities and mitigate catastrophic forgetting, adopt parameter-effeient LoRA tun- ing and incorporate an additional 300K MultiMath Peng et al. (2024) visual reasoning samples in the second SFT stage.
Training Settings. During alignment training, we use a global batch size of 256, a learning rate of 1e-4, and train for one epoch with maximum sequence length of 2048. In the SFT phase, we reduce the learning rate to le-5 (with a vision-specific learning rate of 2e-6) and maintain the same batch size and sequence length. We use the Adam optimizer without weight decay and apply cosine learning rate schedule. To improve memory efficiency, we employ Fully Sharded Data Parallel (FSDP), gradient checkpointing, and enable BF16 precision. We avoid CPU/GPU offloading to maximize throughput.
Hardware and Runtime, All 7B model training was performed on A100 GPUs (80GB each). Alignment training took approximately 5 hours, followed by 12 hours for full fine-tuning (SVE- Math-DeepSeek) and 16 hours for LoRA fine-tuning (Qwen2.5-VL-7BB For the 32B Qwen2.5-VL model, we used 16 x H100 GPUs (96GB each), with alignment training taking 9 hours and LoRA- based SFT 15 hours.
More Ablations. 1) Herein, assess whether it is necessary to count complex overlap- ping objects. Even humans interpret overlapping objects differently. To ensure more accu- rate conclusions for overlapping object counting, dependently label the same 200 images sampled from our benchmark. The ground truth for these cases is represented vided the same answer, we merged them). We added a system prompt to our counting templates: you are required to count potential overlapping objects. For evaluation, we adopt free-form answer matching- if the model's 's response matches any entry in the annotated answer list, it is considered correct. Top-1 accuracy is then computed based on this criterion, as shown Tab I0
asked three authors manually and in-
list of one to three possible answers (if three annotators pro-
Table 10: Accuracy (%) on non-overlapping vs. overlapping counting. Overlapping objects remain consistently harder for MLLMs.
baanV122 1220NE25--100 OPT SVE.ta
2) Instead of using numeric-coordinate grounding, we adopt vertex lists as an alternative abstrac- tion for grounding planar geometric objects. Vertex-list grounding is conceptually simpler than coordinate-based approaches: rather than predicting multiple continuous numerical values, the model only needs to identify discrete symbolic labels already annotated in the diagram. For ample. Q: Please provide the list of vertex labels described by the sentence: Square. A: ABCD, A comparison between vertex-list grounding and numeric-coordinate grounding is shown in Tab. [I
Table 11: Comparison of vertex-list grounding vs. numeric-coordinate grounding for planar dia- grams.
VL.72VLLTB VL
Qwen25.VL.72 VL-7B
LL.VA.113.78
ImenVV225-32
GPT-40
G G.LLVA 7B
MultMahh 7B
SVE Muth, DespSeck-
SVE Math DespSiek 1.7B
Venes
45.6
49.1
18
42.7
44.8
3.9
33.1
41.7
893
Numerics
18.5
14.2
82.9
3) To prevent models from exploiting vertex-count shortcuts in the shape clasifclas task, we de- sign hard distractors for 75% of the questions. For each ground-truth polygon, at least one distractor is drawn from the same polygon family, ensuring that models cannot rely solely on the number of vertices. For instance, when the correct answer a scalene triangle, distractors are sampled from other triangle types (isosceles, right, equilateral). Likewise, for quadrilaterals, distractors include shapes such as rectangles, squares, or trapezoids (e.g., a rectangle paired with a right trapezoid; see Fig. [9, first row). To further validate that vertex-count shorteuts are not driving model performance, we construct an additional controlled dataset of 200 planar diagrams in which vertex labels are re- placed with purely visual numeric markers (e.g., 1, 2, ...... These markers do not encode polygon
3

---

Published as a conference paper at ICLR 2026
rumbered objac dutis inads lou bancs, What the stope of otjer in irugh?
O: rumbered white dutis muiee blu banes. Can ieeetty the of objet the picaae?
digia imide bis boxos. Whst the ilaape of objoct the image?
eoeeglaaneaal qratriornal th pentlllegmm
60 sallese triangle da equlkoral tringle (o quallangle
(6) right kingge (b) scalase Lingle
expakoeral tringgl
rectangl
paragen
Figure 8: Visualization of samples where objects are annotated using numerical visual markers instead of vertex labels. Zoom in for the best view.
type and therefore eliminate vertex-count cues. Example cases are provided in Fig. and the cor- responding ablation results summarized in Tab [2 These results show diverse behavior across models when switching from vertex labels to visual markers. If vertex-count shortcuts dominated model performance, all models would exhibit similar, substantial drops under the marker condi- tion-but this is not observed. Several models (e.g., Qwwn2.55VLL-77. LLaVA-v1.5-7B. SVE-Math- DeepSeek-7B, and our model) maintain stable performance under both conditions. The observed irregularities instead reflect differences in models' ability to interpret visual markers: for example, GPT-4o handles marker-based diagrams well, whereas Intern VL2.5-38B performs better with vertex labels. These findings confirm that our distractor design is robust and that model performance is not driven by vertex-count shortcuts.
Table 12: Accuracy (%) on vertex-list vs. visual-marker classification for planar diagrams.
Ower2 VL 78 Owen2.5 VL 7B LLAV vL.5
IntemVLL. 359
GPT 40
LLaVA 7B
MultiMath 7B SVE Muth DeepSeck 7B (VE Math DeepSeek 7B
Veries
50.7
48.6
34.6
55.3
53.1
31.6
44.1
43.9
72.6
Makus
47.9
3533
43.3
53.2
Discussion of MINT-CoT and Vision-RI vs. Ours. MINT-CoT and Vision-RI align with our findings that perception is critical for reasoning, yet operate at different scopes:
1) Relation MINT-COT: MINT-CoT explicitly injects interleaved visual tokens into the chain- of-thought, enabling the model to reference visual evidence during reasoning. This represents an important direction for coupling perception and reasoning step-wisely. contrast, our work fo- cuses on improving diagram perception itself, without introducing any specialized mechanism for mixing visual and textual tokens the reasoning process. As a result, the two lines of work are compatible: MINT-CoT studies how visual information is integrated into CoT reasoning; our work studies whether the model can accurately perceive symbolic diagrams in the first place, and how enhanced perception naturally transfers to reasoning.
2) Relation to Vision-R1: Vision-RI aims to incentivize reasoning ability on top of a strong base model using reinforcement learning. A key factor in Vision-R1's success is the quality and diversity of its cold-start data: over 43 curated datasets spanning mathematical diagrams, science and medical figures, general QA images, and aguureunderrsanding datasets. Importantly, during data construc- tion, Vision-RI uses image captions as CoTs, giving the model holistic visual context that supports the development of high-quality reasoning traces. While both works support the idea that percep- tion is a prerequisite for effective multimodal reasoning, Vision-RI focuses primarily on improving reasoning ability, whereas our work focuses on explicitly enhancing symbolic diagram perception. This distinction is crucial: reasoning improvements in Vision-R1 rely heavily on RL strategies and large-scale, high-quality CoT training samples, not on explicit perceptual enhancement.
Overall, we summarize the key points below:
Our work: focuses on improving diagram perception and demonstrates measurable transfer from enhanced perception to downstream reasoning.
MINT-CoT:
interleaves visual tokens within chain-of-thought reasoning to produce more visually aligned inference.
incentivizes reasoning ability through reinforcement learning, built on high- quality, large-scale perception-reasoning training data.
Vision-RI:
4

---

Published as a conference paper at ICLR 2026
Qu What is the depe ofjorr APIP image?
Q: ahope
Q: licenity the type chan image
papp
cbject image?
Shape Classification
(e) konerlee vinngle
cquilatenal quadniasenn
(e) dor liee clart
reciaaple
tapecsed
cylinerr
(c) piegle chart
boc loctool ber clart
vetical
10
Q:
The pietuee contions slupos docuth
busiontal
501 (b1
Gal
(9)
149
blongls many
Thes should
Object Counting
piouuee
(i)
(a) Fabe
Eas
det
objects
OTH, WXH CEN, ORT PIEL, UCB
eb1
(G3.5
60
680
Q: Ploae prosiée the bourding hos conndinaic of the respinn dea centence dosrrhas right imppored orige [166, 393, 351]
Ploas provie bounding coseditic of region
Plzae provie the honding bos countinic of resion fiis sentenes dowrbbe people show
(122, 189, 100,
A:148, 73, 263,
Object Grounding
Please proside shot descrittion this region: [180, 309, ssyg
provide region nétheroyliiderr 102,
Please provde the bounding hos cooedivats of the
region this scrianee desir less legend.
(947, 178
Q: Deserihe where
Q: Is the snall hroon gihe
impurooi COSN aclation
Qc la "Turpooke the line mediun!?
trinngee
(b) right
(a) bottonn
lage sphere image? (a) (h)
Yes (h) No.
(c) upp left
Relationship Identification
(5) botten right
16)
As(a)
Qu
spatid pumplc
VWT, apply
Q: highes
UWA
pemesdicalar
top ridts
Yo
Ohhinggl bioerer
(h) horton right
(t) No
(c) angle moveiin (d) mumsecttun
(e) betor ket (c) top lell
As 66
As
A: (Q)
(8)
(c)
Figure 9: Visualization of sample cases from MATHEMETRIC (a), (b), and (c) correspond to problems related to Plane Geometry, Solid Geometry, and Graphs, respectively.
D VISUALIZATION OF MATHEMETRIC
MATHEMETRIC is specifically designed evaluate prrception-demandee tasks in Multimodal Large Language Models (MLLMs), focusing on four core visual understanding capabilities: shape classification, object counting, object grounding, and relationship identification. These tasks, com- monly studied in classical computer vision, are typically solvable by humans with minimal cognitive effort. Our benchmark spans three key domains Saasiins-planeeee geometry, solid geometry, and mathematical graphs ensure broad coverage of visual representations encountered in educational and scientific contexts. Fig. Drg.vidss illustrative examples from MATHEMETRIC demonstrating the diagram- matic input and corresponding question-answer (Q&A) pairs used for evaluation.
5

---

Published as a conference paper at ICLR 2026
E PROBLEM TEMPLATES
This section introduces the problem templates employed in MATHEMETRIC, with illustrative ex- amples provided in Tab. 13 We adopt template-based generation engine to systematically construct diverse perception-oriented Q&A pairs. Each question is generated by parsing the structured JSON annotations of given image, which include information on shapes, object attributes, spatial posi- tions, and inter-object relationships. These elements are then filled into carefully designed templates (Tab. 13 to produce grammatically correct and semantically meaningful Q&A pairs.
Our template designs are tailored to accommodate diverse subject domains domains-including plane ge- ometry, solid geometry, and graph- graph-based based diagrams- and span spectrum of perceptual complexity, from coarse-level to fine-grained tasks. For example, a coarse-level shape classification template may pose question such as, "What is the shape of the object with {vertices) where the correct answer is directly retrieved from the structured annotations. For fine-grained object localization, we adopt a grounding template similar to that used in Wu et al. (2024), prompting models with: "Please provide the bounding box coordinates of the region this sentence describes: {shape} {vertices}". This allows us to evaluate the model's 's ability to extract precise spatial information and align lan- guage with visual primitives at granular level.
We design three types of questions: multiple-choiee, true/false, and open-ended. Specifelly, for multiple-choice questions, we incorporate plausible distractors to challenge the model's s geometric perception. These distractors are selected from geometric candidate pools, varying from visually similar clearly distinguishable options, ensuring balanced evaluation of fine-grained visual dis- crimination. By leveraging this template engine, MATHEMETRIC ensures consistent question formulation, controlled variation in difficulty, and robust coverage of geometric primitives and their relationships, forming reliable testbed for evaluating diagram perception in MLLMs
F DATASET CONSTRUCTION FOR PLANE/SOLID GEOMETRY & GRAPHS
F.0.1 SYNTHETIC DATA ENGINE FOR PLANE GEOMETRY
Fig. 4 (main paper) describes the entire generation process. Inspired by AlphaGeometry Trinh al. 2024 we use geometric clauses as fundamental units to construct complex plan geometric fig- ures. geometric clause formalized description of basic geometric objects and mathematical relationships, along with their properties or attibutes, a.k.a. prerequisite points. We first construct two geometry substrate pools, one containing 16 different geometric shapes (i.e., isosceles trian- gle, square, rectangle, parallelogram, isosceles trapezoid, pentagon, circle.. ellipse and segment), and the other defining 10 mathematical relationships (i.e., on, intersection, parallel, perpendicular, tangent, ... and reflection). We then randomly sample one more substrates from these pools and pass them through a verifier, which ensures that logically paired shapes and relationships are preserved in the construction of valid geometric images. The verifier makes decisions based on either manually designed rules, fundamental mathematical knowledge, or prerequisite points. For example, parallel lines cannot intersect, and without enough prerequisite points, angle trisec- tion clause is invalid. In general, the chosen relationships are enforced by introducing additional shapes into the figure, ensuring that the specified relationships are accurately maintained and geo- metrically consistent throughout the construction process. We finally save the outputs as structured JSON annotations for image rendering using the Matplotlib package Hunter (2007) and generating question-anss answer pairs via template-basse pipelines.
Image Generation. The ISON annotations define foreground and background styles, i.e., colors sampled from a monochromatic palette, line width, and font size, as well as object shape informa- tion (class names), attributes (vertices labeled with random letters), bounding box locations, and mathematical relationships. Notably, spatial relationships are generated based on the bounding box locations two objects (e.g., top-left, top-right, bottom-left, bottom-right). Similar to the image rendering process in Trinh et al. (2024). we translate geometric clauses into visual representations using Python code. This process first determines coordinates of points defined in each clause for basic shape visualization, then generates crease task difficulty, we apply image augmentations, e.g., Gaussian noise, irregular scribbles, and wedge-shaped symbols for congruent angles or auxiliary lines (see Figs. and 12).
primitives based specified relationships. To in-

---

Published as a conference paper at ICLR 2026
+ Gaussian
Scribble
+ Wedged Shaped
Symbols
+ Auxilaar Line
(a)
(b)
(d)
Figure 10:
Visualizing distractors w.s. adding Gaussian noise, drawing irregular scribbles and incorporating wedge-shaped symbols or auxiliary lines.
564
1211
11.
ompooe despoooke
Figure 11: Perceptual perfromance w.r.A. varying Gaussian noise variance.
Question-Answer Generation. Based on structured annotations, we employ a cemplate-bssed pipeline to generate multiple-choice and true-false questions across four perception tasks. For ex- ample, constructing Q&A pairs for shape classification, we first randomly select an object and its associated attributes from the given figure. We then formulate question by filling placeholders
7

---

Published as a conference paper at ICLR 2026
Figure 12: Visualization of distorted images under varying levels of Gaussian noise.
in our carefully designed templates. For true-false questions, the ground truth answer is directly derived from the ISON annotations. For multiple-choice questions, we generate plausible distrac- tors to challenge the model's 's geometric perception, combining them with the correct answer into set of four choices. Distractors are selected from geometric candidate pools, ranging from visually similar (e.g., equilateral isosceles triangles) to visually dissimilar pairs (e.g.. equilateral triangle vs. circle). Further details on the Q&& generation templates are provided in Tab. [3]
Copyright and Consent: For GEOMETRIC, we will release the entire package including all data and code used to generate samples under CC BY 4.0 on acceptance.
F.0.2 REFORMAT DATASET FOR SOLID GEOMETRY GRAPHS
Copyright and Consent: CLEVR Johnson et al. (211)) is under CC BY 4.0 and FigureQA Kahou et (2017) is under CCO 1.0. All files that we re-publish (JSON reformats and look-up tables) will carry the same licenses their sources.
Solid Geometry. CLEVR Johnson et al. (0077) is a synthetic Visual Question Answering (VQA) dataset containing 3D-rendered objects. Each object in the scene is defined by its 3D position and set of attributes, including size (small or large), shape (cube, cylinder, sphere), material (rubber or metal), and color (gray, blue, brown, yellow, red, green, purple, cyan). However, CLEVR's anno- tation format is incompatible with our template-based pipeline for generating perception evaluation QA pairs. To resolve this, reformat the original annotations into structured ISON, following the same format as the synthetic data process. Specifically, we extract object information from CLEVR's 's scene annotations and organize them into shape information and object attributes. We then leverage an object's 's 3D location and its direction for calculating its 2D top-left and bottom-right coordinates, from which relative spatial relationships are derived.
Once the reformatted JSON annotations are generated, we apply the same template- camplate-based QA gen- eration process as designed for plane geometry, with an additional uniqueness check for shape clas- sification. To ensure clear and consistent labeling, we filter out ambiguous cases where identical attribute combinations correspond to different shapes within the same image. For example, age contains multiple distinct shapes with the same attributes (e.g., a large rubber blue square and large rubber blue cylinder), we exclude classification questions that rely on the attribute combination 'large rubber blue" prevent ambiguity.
im-
Graphs. FigureQA Kahou et al. (2017) is a visual reasoning dataset containing over one million question-answer pairs, grounded in synthetic, scientific-style figures, including line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts. The official annotations capture various rela- tionships between plot elements and evaluate characteristics such maximum, minimum, smooth-
8

---

Published as a conference paper at ICLR 2026
ness, and intersection, all framed binary yes/no questions. Additionally, FigureQA provides numerical data annotations used to generate each figure, along with bounding- box annotations for all plot elements. Similar to CLEVR, we reformat FigureQA annotations compatible with template- based pipelines.
Specifically, the shapes include five graph types defined in the official specification, with each element's ss attribute represented by its unique color. Additionally, we store the bounding box coordi- nates of each foreground element (such lines, bars, and pie slices), along with legends and titles, as ground truth for the grounding task. For relationships, we follow the binary true/false question- answering problems evaluate the model's understanding of geometric and graphical relationships.
G
EXAMPLE ILLUSTRATION OF GEOMETRIC
Figs. 13 15 demonstrate how GEOMETRIC delivers explicit geometric information-ccoveringg covering ob- ject count, shape classification, fine-grained bounding box coordinates, and inter-object relation- preset- in both caption-style and instruction-follovingg conversational formats.
H VISUAL DISTRACTORS
To evaluate the robustness of MLLMs' perceptual capabilities, we introduce set of visual perturba- tions through data augmentation. These include Gaussian noise, irregular scribbles, wedge-shaped symbols, and auxiliary lines, as shown in Fig. [0] The aim is simulate real-world visual am- biguities and assess whether MLLMs can retain accurate geometric understanding under degraded conditions. These controlled distortions provide rigorous benchmark for evaluating the models' ability to extract and interpret mathematical structures from visually complex inputs.
By gradually increasing the Gaussian noise level, we can systematically evaluate its impact on the ability of MLLM to recognize mathematical structures. As shown in Fig. 12 the image transitions from a noise level of 0.1 to 0.8, progressively blurring the geometric features. We observe that as the noise intensity increases, the difficulty of recognizing mathematical structures also rises, posing greater challenges to accurate recognition and reasoning. When the noise level reaches ~ (O, 0.8), the mathematical structures become nearly indistinguishable to the human eye, resulting in sigmificant performance drop on MATHEMETRIC, as illustrated in Fig. I1
I CASE STUDY
we present comparative analysis of model responses across sev- eral Multimodal Large Language Models (MLLMs), including SVE-Math-DeepSekk Zhang et al.) (025), Intern V1.2.5 Chen et al. (0024)), Owen2.5-VI. Bai et al. (2025), and GPT-4o, alongside our enhanced variants: -at-MMat--eepSeee tuning base models (StE-Math- -eeeath-DeeSeekkk and Vwen on GEOMETRIC significantly im- proves response accuracy over their respective basclines. This improvement suggests that our care- fully curated training data effectively enhances the model's mathematical perception, enabling it to better comprehend problem structures, reason through mathematical concepts, and generate more precise answers. For example, in Fig (a). (a), the base model fails to recognize the hypotenuse, while the GEOMETRIC model correctly identifies it and naturally applies the Pythagorean theorem to reach the correct answer. Similarly, in Fig 16 (b), the GEOMETRIC model correctly perceives the diameter, infers the presence of a right triangle. and again applies the appropriate theorem. In Fig I8 (a). both the base model and ours know the relevant theorem the Inscribed Angle Theorem. How- ever, the base model misperceives key visual cues (e.g., tangent lines), leading to hallucinated angle relations and an incorrect final answer. The GEOMETRIC model correctly perceives the tangency and the right-angle structure, enabling it to execute the theorem correctly and arrive at the correct answer. Furthermore, observe that even GPT-4o, strong competitor in the field, produces in- correct responses in certain cases. Upon closer examination, these errors often stem from inaccurate mathematical perception rather than purely computational mistakes. This observation reinforces the notion that an MLLM's s ability to accurately perceive and interpret mathematical content a critical factor in achieving high performance. Overall, our findings highlight the fundamental role of precise
Model Responses. In Figs. 16 18
and Qwen Our evaluation shows that fine-

---

Published as a conference paper at ICLR 2026
perception in mathematical reasoning, akin to its importance in vision and language understanding. perception can lead to substantial gains in accuracy and reliability.
Error Examples. In this section, provide more detailed error examples of GPT-4o, Qwen2.5- VL-7B, and our 7B model. We categorize the errors into two types: Chain-of-Thoughhe (CoT) Errors and Recognition Errors. CoT' errors occur when the model engages in step-by-step reasoning for perception questions but ultimately provides incorrect answers. Recognition errors, on the other hand, arise when the model attempts direct answering without reasoning yet fails produce the correct result. Representative examples for each error type are illustrated in Figs. 19
J LIMITATIONS AND BROADER IMPACTS
Limitations. The proposed MATHEMETRIC focuses on evaluating MLLMs' perceptual capabil- ities in structured, symbolic mathematical diagrams spanning plane geometry, solid geometry. and graphical representations. It enables fine- grained assessment through carefullly designed perception tasks. Our findings clearly show that current models exhibit limited perceptual aerceptualabiiiee ability diagram understanding across all three subjects, but our proposed training dataset, GEOMETRIC, remains limited synthetic and geometry- ambiguity of real-world educational materials (e.g., textbook figures, handwritten notes, or scanned diagrams). future work, we aim to extend our graph-based data construction to solid and graph- ical domains, which requires further investigation into their distinct structural representations. For instance, in graphical images such as line and bar charts, the underlying structure differs signifi- cantly from geometric diagrams. Nodes may correspond to data points, axis labels, bars, while edges may capture trends, groupings, or relational mappings across axes. Developing meaningful node-edge representations these contexts will be essential for enabling accurate visual grounding and interpretation. Moreover, we believe additional tasks, such as visual marker detection, are also critical for comprehensively evaluating MLLMs. For instance, recognizing special diagrammatic markers that represent relationships like parallelism or perpendicularity (e.g., double lines or small squares) is essential for deeper geometric understanding. Similarly, OCR capabilities are important, such as detecting angle measures or textual annotations embedded within diagrams. Expanding the benchmark to cover these aspects would further advance the assessment of MILMs² diagram perception abilities.
content. As such, it may not fully capture the diversity and
Furthermore, although we analyze how perception affects reasoning, our current setup does not ex- plicitly model the interleaved interaction between visual grounding and logical reasoning critical direction for advancing multimodal mathematical reasoning.
critical yet underexplored capability of diagram perception-aae and aims to push the field toward more interpretable and reliable multimodal reasoning. By isolating perception from reasoning, we reveal fundamental limitations in current models and provide a diagnostic tool for future model development. These insights could benefit applications in math education, Al-assisted learning, and cognitive science research. Caution is warranted, however, as over-reliance on synthetic datasets may lead to overfit- ting, and enhanced perceptual accuracy does not inherently ensure reliable reasoning. Furthermore, improved performance on benchmark tasks may not directly translate to robust generalization in real-world scenarios. We advocate for future research that bridges synthetic benchmarks with natu- ralistic data and advances human-aligned evaluation frameworks for visual reasoning systems.
Broader Impacts. This paper sheds light
K USE OF LARGE LANGUAGE MODELS (LLMs)
We used LLMs solely as auxiliary tool for grammar correction and minor language polishing. They did not contribute to research ideation, experimental design, data analysis, or writing of tech- nical content.
10

---

Published as a conference paper at ICLR 2026
Structured Descriptions
Multi-turn Conversation
Image:
Image:
H
Please provide structured geometric information for the image.
Question:
Question: Provide the shape type of object DKLT in this image.
Answer: First, I count prominent objects in the image, including rectangle, isosceles triangle, square. Next, for the shape information, object DPIK is a rectangle, object EHV is an isosceles triangle, object JOYN is square. Furthermore, also know the fine-grained bounding box coordinates: the rectangle DPIK is located at [113, 205, 251, 328], the isosceles triangle EHV is located at [236, 215, 399, 317]. the square JOYN is located [154, 179, 230, 256]. Finally, let me explain the relationships: rectangle DPIK is in the bottom left relative to isosceles triangle EHV; rectangle DPIK is the bottom left relative square JOYN.
Answer: rectangle
Question: Count the number of rectangles in the image.
Answer: 2
Question: Please provide the bounding box
coordinate of the region this sentence describes: rectangle EXHI.
Answer: [261, 143, 364, 251]
Question:
Deseribe the relationship of rectangle EIНX relative to trapezoid FRQO in the image. Answer: bottom right
Figure 13: Sampels of caption-style and instloction-followingg GEOMETRIC.
Structured Descriptions
Multi-turn Conversation
Image:
Image:
Question: Please provide structured geometric information for the image.
Question: Can you identify the type of object
CHPG in the picture? Arequilateral quadrilateral
B:trapezoid Ciright trapezoid D:right triangle
First, count 2 prominent objects in the image, including right triangle, circle. Next, for the shape information, object EVP is right triangle, object E is circle. Furthermore, I also know the fine-grained bounding box coordinates: the right triangle EVP is located at [256, 256, 348, 374], the circle is located [113, 113, 399, 399]. Finally, let me explain the relationships: right triangle EVP is in the bottom right relative to circle E; line AB is parallel line EV.
Answer:
Answer: B
Question: How many trapezoids are there in the image? A:3 B:4 C:1 D:
Answer:
Question: Please provide the bounding box
coordinate of the region this sentence describes: trapezoid CHPG.
Answer: [113, 133, 399, 380]
Question: Does the image show the relationship among angle CHG, angle CHID, angle DHM and angle MHG as "angle trisection"?. A:Yes B:No.
Answer: A
Figure 14: Sampels of caption-style and instruction GEOMETRIC.
11

---

Published as a conference paper at ICLR 2026
Structured Descriptions
Multi-turn Conversation
Image:
Image:
G
Question: Please provide structured geometric information for the image.
Question: Describe the geometric shape of object HKUOQ in the picture.
First, count 2 prominent objects in the image, including I isosceles triangle. circle. Next, for shape information, object JNR is an isosceles triangle, object HRJN is circle. Furthermore. also know the fine-grained bounding box coordinates: the isosceles triangle JNR is located [179, 143, 318, 343], the circle HRJN is located at [169, 143, 399, 369]. Finally, let me explain the relationships: isosceles triangle JNR in the top relative circle HR.N; point F is the reflection of point H with respect to point
Answer:
Answer: pentagon
Question: Identify the total number pentagons
present in this image.
Answer: 1
Question: Please provide the bounding box
coordinate of the region this sentence describes: pentagon HKUOQ.
Answer: [112, 128, 399, 384]
Question: Describe the geometric relationship between line HE and line KO in the image. Answer: perpendicular
Figure 15: Sampels of caption-style and instruction-following GEOMETRIC.
SVE-Math-DeepSeek
SVE-Muth-DeepSeck
InternV1.2.5
12
length hypoteense 12, we can use the Pythagorean theorem find the length the perpondicuttr.
ABC is 10, we can me the
Pythagocvon
right
Sermala for the area of triangle to selve for
trinngle formed by y, and hypotomnee
Question:
Lat's denote the length papesdicallr
X The area of minngle ABCC equal tringle
hygotenose
Find x.
ACD and
the the BCD
The right trisngle has logs and
two sogmenti,
Cholces:
Acconding
Pythagances
height tringle
X Using the Pythagoromn thoorem:
A:2
2:
thzorem, we have 12"h e2, then ycz 2VTT. Henos,
and the height miangle BCD
484,
C:
2. we can write the equation as
option
0.5 a 0. x 5
Then, find
SVe-Math-DeepSeckk
SVE-Math Deeep-D
Intern VL2.5
Since ZACB
X Connect
To find the radius the cirole, Pylhugrrean given points form right eiangle with the diameter of the circle as the typetemsse
tangents
we have OA perpendicular to AC and OB perpendicnaar to DC. Therefore, angle OAC angle
Since
Question:
Be milus the cirele & AB
30"
AC inches, and BC- 15 inches. Find the radias of the circle.
8.5 inchos. Therefor,, the answer &
Diumeter
AB vAC BCT
X
VBP +
The ridius
trinngle AOB in isosceles trinngl..
half of the diameter. So, x
Choices:
option
7.55
B:8
So.
"17, Therefoce, the radius circle
R.S.
1:
is 17. Therefore, the answer is
(b)
Figure 16: Response comparisons between SVE-Math-DeeeSeedk (7B), SVE-Math-DeepSeed (7B), and InterVL2.5 (8B) Math Verse.
12

---

Published as a conference paper at ICLR 2026
Table 13: Examples of problem templates used by MATHEMETRIC on different source data acorss different tasks.
Source
Task
Three randumil chosen examples from hundreds.
What is the shupe of object (yetiees) in the image! Choices: [: (() D.().
you identify the type object [verices] in the pierure? Choices: A:[a] R:(h) (.(.)
Can you identify the type of object the picture? Choices: A:[a] [:(b) C:Jc] D:(d).
What the shape the object in the image? Choices: A:[a] B:(b] C:(c) D:(d).
There (bc) only [num] [slupe] in the picture, right" Choices: A: Yos B:No.
[he) there only (num) [shope) in the picture? Choices: A/Yos B.No
can only see [num] objects the picture, con't you? Choices: A:Yes B:No.
There should be only [num] shapes in the picture, correct? Choices: A/Yes B:No.
Hum many (shupe)s can you lind the picture? Choices: A:(a] B:[b] C:(c)
cut
Pleuse count all the (shupe)s I in the inage. Choices: A:(a] B:(b) C:(c] D:(d].
What the number shapes the picture? Choices: Ax(a] B:(b) C-(c) D:(d).
How muny objects can you idunity in the image! Choises: [a B:(b] C-ie] Drid).
Please identify and select all the [shape]s in the picture. Choices: A:[a] B:(b] Ci(e D:(d).
Plane Geometry
Find and select all the [shape]s Is in the picture. Choices: A:[a] B:(b] Ci(c] D:(d).
Please provide the bounding box coordmate of the region this
deseribe:: (shape] [verbees)
Can the relationship (preposilion) [shupe] in the image be deserited as "(relation) Choises: Yos B:No.
Does the image show the redarionship (peeposition) (shape) as "<""ree "[relation] "? Choices: A:Yes R:No.
[shapel] described as being the [relation] | relative to (shape2] in image? Choices: A:Yes
Is [shopel] said to be in the" "relation) relative to (shapc2) in the image? Choices: A:Yes B.No
rlat
What the relatiooship [prepositionn [shape] the picture? Choices: A: B:(b] C-/c] Dr(d).
Can you identify the relationship [preposition) (shape) in the image? Choices: A:(a] B.(b) Ci(c| D:(d).
What the relative position of { to (shupe)] the image? Choices: Az(a) B:(b) C:(e) D:(d).
What the sputial relationship of shopel to shape2 in the image? Choices: A:(a) (b(b) C:(c) D:(d).
What the shape the [size] [color) object made [matenal) in inage? Choices: Ac(a] [: D:C)
Can you identiy the type of [size] (color) object with [material) miterial in the picture? Cholees A:fa] (b(b) ( D.Ga).
There (be), only [num] [Shape] the picture, rightt Choices: Yes B:No.
{be) there only (mum] (shape) in the picture? Choices: A:Yes B:Na.
There only (num) objocts the picture. right? Choices: A:Yos B:No
Are there only [mmm) shapes in the picture? Choices: A:Yes B:No.
How many [shape] Is can you find the pierure? Choices: A:[a] [:[h] C:(c] D:(d).
Please count
[shape]s Is in image. Choices: Ad[a] B:(b) Ci(c] D:(d).
Count the shapes in the image. Choices: A:[a] B:(b) C:(c) D:(d).
How many shapes can you visually identify the image? Choices: As(a] B:[b] C:(e) D:dd).
How many (size) (color) (material) objects are there in the image? Choices: A:(a) B:(b) C(c))
Soild Geometry
many (size) (volur) (material) objects present
image?! Choices A:(a) B:(b) D.J).
grd
Please provide the hounding hox coondinate of the region this sentence deserihes: (size) [calor] [moterial] (shape)
it correct that [shapel] deseribed heing the [relatio]] relative [shape2] the image? Choice:: Y: B:No
Is (shapel) 1 described as being in the "trelation" rolative to (shape2) in the image? Choices: A:Yes B.No.
(shopel) said be in the "[cciiion relative to (shup22 in image? Choises: A:Yos B.No. Can you confirm In the image, where is [shapel] in relation to [shupe2]? Choices: A:(a] B:(b] C:[e] D:[d].
(shapel)
the "[relation]" relutive to [shupe2) the image? Choices: Yes B/No.
What the relative position of [shupel) to [shupe2] the image? Choices: A:(a] B:(b) Cife) Di(d).
What the spatial relationship of [shapel] to [shape2) in the image? Choices: A:[a] (b Cifel Dr(d).
Describe (shapel) situated relative (shape2) in image. Choices: Ac(a] Bx(b] C:te) De(d).
What type of chart is shown in the image? Choices: Ac[a] H:(b) D:()..
Identify the type chart the image. Choices: A:la] B:(b) Ci(c) Di(d)-
Which of the following best deseribes the chart graph the image? Choices: A:fa) B:(b) C(ec) D:d))
cls
Which
chart the image represent Choices: As(a) B:(b) Ci(e) D:(d).
What kind of data visualization shown in the image? Choices: A-(a] B:[b] (c)c D:(d).
you identify the type of chart the image? Choices: A:(a) B:[b] C:le) D:(d).
[be) there only [mum] [shaps] in the piclure? Choices: Yes B.No.
The picture contains unly (mm)) (shope), ). each representdd by a different color, doen't ii? Choices AcYos B.No.
can only see (num) [shage] with different coloes the pleture, can't you? Choices: A/Yes BoNo
Graphs
ent
How many [shape]s Is are there the image? Choices: A:(a] B:(b] C:(s) D:(d).
Can you count toral number of [shape]s in image? Choices: A:(a] R:(h) C-tc) (:(d).
What is the total number of [shape]s in the picture? Choices: Ac(a] B:(b] C:fe) D:(d).
Plcase provide bounding box coordinate of rogion sontence describes: (color] [shape).
rbi
(question) Choices: A. You B.Nu.
13

---

Published as a conference paper at ICLR 2026
SVE-Mall-DeepSeek
SVE-Math-DeepSert
InternV1.2.5
Since
Tangents drawn extemal
20AP
and of tangency: Therefore, <PDO
circle equal
length. Therefore,
<PAO 90"
2PAB ZPBA.
Question:
Also, A.AOB 2 ZAOC -130..
X Since P- 65", we have 4BOP
X
200B is given 65 Since O the center the circle, <COB saple subeendol
As shown the figure, PA tangent to circle O A and B vespectively. degree
90"
180 65 115". Therefore, 4
130 50", Thus, the arswer
AOB 360" 90" 90
X The inseribed 2CAD half the central angle 400B. Therefore.
(??
Choices:
65
R: 135*
triangle is 180". So, 2PAB + &PD ZP = 1005. Then, &P 100" ZPRA -PAA 1887 65
angles
50"
D: 100"
(a)
SVE-Math-DeppSeek
SVE-Math-DeepSeek
InternVL25
Since tmnents
angle we use propary inscribed angle in a circle The inseribed angle thoorem states that inserihed imple hulf the the central that the
the circle O. we have 201P ZOBP 90".
and PB lingeri O,
we have ZOAP 2OBP 90"
Therefore, Since ZACB in a AOB (the
Therefore, 50" 130".
Question:
As shown the figure, given that angle 50.0, then the size ingle ACB is!
Thus ZACB 1/2 2A0B 65".
inseribed angle half of the
Tharffon, the anwweri
ziven figure. exterier angle tringle ABC, and it is given 50" According to the extetior angle theorem, the opposite
ingle by are).
Choices: 65
Therefore, the correct answer A
B: 60"
C:
Therefore. angle equal sum ZACB and 2ABCC
X Thus, ZACB _ 550
Figure 17: Response comparisons between SVE-Math-Deeppeckk + (7B), SLEE-Math-DeepSSee (7B), and InterVL2.55 (8B) Math Verse.
14

---

Published as a conference paper at ICLR 2026
Qwen2.5-VL-7B+
ChatGPT-4o
Qwen2.5-VIL-7B
Step (Pugeny of Tangent Linck.
X Norice that <CBD 50" is
X Idencify the given information: 2DCD
Since CD tanget cincle st
exsenior ingle tringle AABIC
50", PrrO the center of the circk. AB
point 20CD 905.
Points A, C lie the circle,
is dimeter of the circle 4ACB
Step (Given Angley: Given
50 2ABCC inseribod
night angle.
Question:
50
50"
X Since cCBD 50" cuterice
X Use the Inseriled Angle Thoren: An
The degree of angle A is?
Stpp (Sum Angles Triungl)) trinngle OCD, 20CD 2C0D
the sum of the two opposite
angle stangle ABC, equals
inscribed angle half the measure of
the central angle the wubieends
+4D 180..
interior ingles, le: CCBD
Step (Sabuiiute Known Valuog):
<CAB - -AABB
angle sibtending
BD.
20
Answer:
Step (Caleulite 2COD), 200D
2COD + 50 1807.
X But more importantly. in this
specific cirlle contiguratin. if
will he hois the missame of LRCD.
2B0D = 2BCD = 300-100"
correpending cenml angle 2800
1800 907 50" 47
CD intersect
B,
/ Supp 6 (Inscribed Angle Theorm) Accoeding the Inscibed Angle
you are given that
and
X ZABC is me inseribed angle
subnending are AD. The comrspencing commi angle 2ADD will be twice the
Step (Calculate 2AI: ZA 40"
Theorem, 4A cCDD
is tangent the circle at point B,
then: 2CAB m 2CBD - 50
of LABC: ZAOD ZABC. Frum the previous colculation. ZB0D-100 Therefore, e AACD-100
So 2AACC 2ADD 50" Then 2A- 180 LACB ZABC - 90" 50" 400
(a)
Qwen2.5-VL-7B+
ChatGPT-4o
Qwen2,5-VL--7B
Stap (laseribal Augle Theaman's Since OA OC, 2ACO 2CAO
lu tringle COE, <CAO 22.5
Idennity Key Propentes: Since Junnde of carle ZACB ACB-ssu tby the laserited Angle Theorem). OC is radis of the circle so OC-0A--8
is an angle betwees line segment
22.5%
CA and diameter AB. Trinngle
Step 2 (Angle Calesharinn); ZBOC
COA within circle
As shown in the figure, angle CAD 22.5. OC 6.0, then the length of CD (0?
Question:
Step (Perpendicular Dimmeee
.ZZA
45"
X tigonometry triangle COA, SaIzCAD) OE
dander
maliss.
miius. Given O0060, the radius
Step 4 (Booceless Right Tringge
Thomrn): Since CRI AB, 4CFD
Pripent): In ACRO, 200R 45'
-00
2,296
Sa.
x smm22.5))
siILCAAO) 0,3827
X Place the center O the orgin (0.0) cooninati
Let Orean 0A-0C-6 Sincs 2CA0-22.5".
A"t
X tringle CED cnate stangle), the
Answer: 6v2
trangls.
ACEO is issecles right
lag
the
Coi6coo 22.5 (41877 2.42).
V Step 5 (Calculate CB): OB 43 -3/2
OB 2,256
Since
vertiaal disanes
from to the soxxis, which is
V Sup (Calenlate CD): CD 2CF
simply Bee yeoooninnn of Co
-2 x3
CD 6xin(225)) 6. VE-1
3/2
(b)
Qwen2.5-Vi VL-
ChatGPT-4o
Qwea2.s-VL-7B
v
Sup (Inscribal Angle Thearan)- Accoeding the Inseribed Angle Thoorem, ZAOB ZEACB.
v/ WW are gives a cincle with center
X Giver: 2ACB-65 Since is the canter of the cirde. OA OB. and OC ar ralli of the circle OAB
O, and ciangle ABC inseribed in
655
it. and IP extended
Step (Subetiute known
they
point
making trangle
Find angle
Question:
Step (Chleshation) 2AOB - 130".
ZADB
65".
In circle goonatry furned by chards intersezting
the
angle
v The contrsl angle subtended by AB
equiliiiaal trangle (since
siles me oqual).
Step (Propesty Tangentj:
cutsiće the circle (lkke ZAPB)
the inscrbbed angle subsended by the same are any point on the circunferacce Tharefone ZAOB - "ScAACB 2405 - 130
Since Pi are tanpents
equals the difference between the
cinole O, 1...0AAP-00H 2DMP-20PP-00 S0".
ares they intercepe divided by 2:
Answer:
(Sum Angles
<APB (AD) (D)
S0
Quadrilatzall: quddilate
A0B2, sum intetix
X Dut in this case. since APB subtend the AB, and trnnle
Since X Now consider tringle
angles
the cirole, they form two equal sides of this trangl. Thus, miangl OAP is isosceles with OA-OR.
and OP are both
(Coleulane
ABC inscribed
ZOAP- ZOBP ZAAO.
Step (Sabetiture the known
X COMP220XAZA00-111 180", then 20AP 4 130" - 190". So 20XP 25" Therefore, 4APS -2 ZOPA
Step & (Calenlation) 2P- 50'.
(c)
Figure 18: Response comparisons between Qwen2.5-VL-7B+ (Qwen2.5-VL-7B fine-tuned with GEOMETRIC using LoRA), GPT-4o, and Qwen-2.5-VL-775 in Verse.
15

---

Published as a conference paper at ICLR 2026
Chain-of-Thought (CoT) Errors
Question: Can you explain how line Gl and line OS in the image are related each other?
A. parallel
B. perpendicular
C. on
D. incircle
Ground Truth: A
GPT-4o Answer
Stepl: The lines GI and OS are both chords of the circle. They are not parallel or perpendicular to each other, nor are they tangent or secant lines.
Step2:
Step3: Therefore, the correct choice is C: on
Chain-of-Thought (CoT) Errors
Please ignore any shapes formed by red auxiliary lines. Can you identify the type of object CJLNR in the picture?
Question:
4. isosceles trapezoid
B. equilateral quadrilateral
C. pentagon
D. quadrangle
Ground Truth: C
GPT-40 Answer
Stepl: The object labeled CJLNR in the picture is a quadrangle, which is four-sided polygon. Step2:
The red auxiliary lines do not form any additional sides or angles that would change the classification of the shape.
Step3: Therefore, the correct answer to the question D: quadrangle.
Figure 19; Examples of Chain-of- Thoughh (CoT) Errors in GPT-4o.
16

---

Published as a conference paper at ICLR 2026
Chain-of-Thought (CoT) Errors
Question: What is the shape of object IPYR in the image?
A. right trapezoid
B. isosceles trapezoid
C. equilateral quadrilateral
D. equilateral triangle
Ground Truth: B
GPT-4o Answer
Stepl: To determine the shape of object IPYR, we need to analyze the angles and sides.
Step2: The angles at I and appear to be right angles, and the sides IP and PR are parallel.
Step3: This suggests that IPYR is right trapezoid. So, the answer is A: right trapezoid.
Figure 20: An example Chain-of ofTTuughht (CoT) Errors GPT-4o.
17

---

Published as a conference paper at ICLR 2026
Chain-of-Thought (CoT) Errors
Question: Can you identify the type of object in the picture?
A. isosceles triangle
B. circle
C. scalene triangle
D. right triangle
Ground Truth: D
Qwen2.5-VL-7B Answer
Step1: : The image shows a triangle with three vertices labeled as D, C, and E.
Step2: To determine the type of triangle, we need to look the lengths of its sides.
Step3: If two sides are equal, it is an isosceles triangle. If all sides are of different lengths, it a scalene triangle. If one angle is 90 degrees, it is a right triangle.
Step4: From the image, we can see that the lengths of the sides DE, EC, and DC are not equal, so the triangle is not isosceles or right.
Step5: Therefore, the correct answer is C: scalene triangle
Figure 21: An example of Chain-of Thought (CoT) Errors in Qwen2.5-VL-7B.
18

---

Published as a conference paper at ICLR 2026
Chain-of-Thought (CoT) Errors
Question: Can you count the total number of objects in the image?
A. 3
B.
C.
5.
Ground Truth: A
Qwen2.5-VL-7B Answer
Let's count the objects in the image step by step: Stepl:
There are 5 points labeled: A, J, G, L, and S.
Step2: There LS.
lines connecting these points: AI, AG, JL, and
Step3: So, the total number of objects in the image is 5 points and 4 lines, which makes 9 objects in total.
However, the options provided are A:3, B:4, C:2, and D:5. It seems there might be misunderstanding in the question or the options. Based on the image, the correct answer should be: D:5.
Step4:
Figure 22: An example Chain-of-Thougee (CoT) Errors in Qwen2.5-VLL-7B.
19

---

Lines that Published as a conference paper at ICLR 2026
Chain-of-Thought (CoT) Errors
Question: Can you explain how line GI and line OS in the image are related to each other?
A. parallel
B. perpendicular
C. on
D. incircle
Ground Truth: A
Qwen2.5-VL-7B Answer
To determine the relationship between line GI and line OS in the image, let's analyze the given options:
Step1: A: Parallel Lines that never intersect and are always the same distance apart. B: Perpendicular
intersect at a 90-degree angle. C: On One line is on the other line. D: Incircle A circle that is tangent to all three sides of triangle.
Step2: In the image, line GI and line OS are not parallel, not perpendicular, and not on cach other.
Step3: However, they are both radii of the same circle, which means they are equal in length and emanate from the same center point (O).
Step4: Given the options, the correct answer is: C: On
-haon-tf-Thought Figure 23: An example of Chain-of- Thought (CoT) Errors in Qwen2.5-VL-77
20

---

Published as a conference paper at ICLR 2026
Recognition Errors
Question: How many objects are there in the image?
N
A. 1
B.
C.
W
D.
Ground Truth: D
GPT-4o Answer
C:
Recognition Errors
Question: You can only see 2 objects in the picture, can't you?
A. Yes
B. No
W
C
Ground Truth: B
GPT-4o Answer
A: Yes
Figure 24: Examples of Recognition Errors in GPT-4o.
21

---

Published as a conference paper at ICLR 2026
Recognition Errors
Question: Can you identify the type of object QRUW in the picture?
A. isosceles trapezoid
B. trapezoid
C. pentagon
D. right trapezoid
Ground Truth: B
GPT-4o Answer
I can't identify the object QRUW in the picture.
Recognition Errors
Question: Please provide the bounding box coordinate of the region this sentence describes: right triangle FGK.
Ground Truth: [113, 182, 398, 329]
Qwen2.5-VL-7B Answer bbox 2d: [109, 135, 347, 286), label: right triangle FGK
Figure 25: Examples of Recognition Errors in GPT-4o (top) and Qwwen.55VL-77 (bottom).
22

---

Published as a conference paper at ICLR 2026
Recognition Errors
Question: What is the shape of object AFJP in the image?
A. isosceles triangle
B. equilateral quadrilateral
C. rectangle
D. right trapezoid
Ground Truth: C
Qwen2.5-VL-7B Answer
B
Recognition Errors
Question: Please ignore any shapes formed by red auxiliary lines. What the shape of object ABFT in the image?
A. trapezoid
B. circle
C. square
D. ellipse
Ground Truth:
Qwen2.5-VL-7B Answer
The correct answer is C: square
Figure 26: Examples of Recognition Error in Qwen2.5-VL--B..
23

---

Published as a conference paper at ICLR 2026
Recognition Errors
Question: Can you identify the type of object AHM in the picture?
A. right triangle
B. isosceles triangle
C. rectangle
D. equilateral triangle
Ground Truth: B
M
SVE-Matth-eeppseek Answer
A
Recognition Errors
Please provide the bounding box coordinate of the region this sentence describes: quadrangle GSJY.
Question:
Ground Truth: [243, 186, 343, 350]
SVE-Math-DeepSeekk Answer [287, 159, 399, 297]
M
Figure 27: Examples of Recognition Errors in SVE-Math-DeepSeeet +-7B.
Recognition Errors
Question: What is the shape of object IPYR in the image?
A. right trapezoid
B. isosceles trapezoid
C. equilateral quadrilateral
D. equilateral triangle
Ground Truth: B
SVE-Nath-DeepSeek' Answer
Figure 28: An example of Recognition Errors in SVE-Math-DeepSeek +-7B.
24


Reviews:
[{"review_id": "NWuMkvv37Z", "reviewer": "Reviewer_Y1aj", "summary": "This paper address an important aspect of evaluating MLLMs on visual math problems: Isolating the perception of simple visual concepts from complex mathematical reasoning and problem solving. To this end, the authors present a benchmark that tests basic perception skills like counting, shape recognition, and locating objects, and a training dataset of geometric shapes and relationships to improve the models' perfromance. Their results show while models peform poorly on the benchmark, using the training set their performance improves, and even results in improvements on the other reasoning benchmarks.", "questions": "- Could you please give some details on the size and samples of the training set (GeoMetric)?\n- How do you make sure that the weakness 2 is not happening?\n- Your work resembles a recent work on conceptualisation (Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models, ICML2025). In that works the authors observe two anomalies in perception, the Easier-Worse and  Middle-Score. Did you observe any similar anomalies in your experiments?\n- Do you have an idea to evaluate not only the final answer, but also the models' CoT accuracy on the benchmark? (sometimes the models give the correct answers by chance, with the wrong CoT)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper addresses an important challenge in evaluating the performance of the MLLMs on visual understanding and conceptualization\n- The benchmark is curated with a high quality, the labels are generated in an accurate way (not like some work that use LLMs for sample generation/annotations)\n- The paper also provides a training set for training the models\n- The results of training using this set are strong, with high improvements on their proposed bencmark, but more interestingly, improvement on other reasoning benchmarks (MathVerse and MathVista)", "weaknesses": "- The object grounding task (with. example in Figure 3, rightmost) is not. It is of no surprise that the models are performing so poorly on this task, given the difficult nature of finding the exact pixel coordinates from an image, which is at the same time unnecessary for visual perception and reasoning. A better alternative to this task would have been to ask the models to give the object vertex letters (for instance in the same task: Q: Please provide the List the vertex labels of the object: scalene triangle. A: HVI)\n- The shape classification task can have a shortcut for models: If they see a vertex label of three letters, it is a triangle, so even without the image they can response correctly if only one answer in the answer choices is a triangle, or with a higher probability than random (0.25) if not all answer choices are triangles. \n- Some of the graphs and figures could improve for clarity. For instance, Figure 1 is hard to compare different models. maybe a radar plot could have been more illustrative. Also the colors of Figure 5 could be chosen better. It's quite hard to follow the light brown and yellow colors.\n- While the paper states that the tasks are easy for humans multiple times, there is no human performance baseline provided. Going back to the first point, the grounding task would be impossible for humans only with looking at the images.\n\nI am willing to increase my score if these concerns are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z8kc8uEQKa", "reviewer": "Reviewer_DjMf", "summary": "This paper investigates a critical but underexplored limitation of Multimodal Large Language Models (MLLMs): their inability to perceive mathematical diagrams correctly, leading to reasoning failures. The authors introduce two main contributions: 1. MATHEMETRIC, a diagnostic benchmark that isolates low-level perception (shape classification, counting, relationship identification, grounding) from high-level reasoning across three domains—plane geometry, solid geometry, and graphs. 2. GEOMETRIC, a structured training dataset encoding diagrams as graphs of geometric primitives with attributes and relationships.\n\nThrough evaluations of 18 MLLMs (generic and math-specific), the paper demonstrates that current models fail severely in fine-grained perception (e.g., <20% in grounding) but that training on GEOMETRIC yields substantial perception gains (+79% grounding accuracy) and modest reasoning improvements (+3–4% on MathVerse/MathVista). Finally, the authors argue that perception is a fundamental bottleneck for reasoning and that structure-aware visual pretraining can partially bridge this gap.", "questions": "1. How would your findings change under interleaved visual token reasoning paradigms (e.g., MINT-CoT [1]) where visual tokens are mixed directly with text during CoT generation? Could these newer paradigms render the perception–reasoning separation less meaningful?\n\n2. When textual and visual information conflict, how do you determine which should be “correct”? Have you tried prompting models with explicit modality preference instructions to measure alignment sensitivity in the relative experiments (“blind faith in text”)?\n\n3. The inclusion of recently released reasoning-enhanced models, such as Vision-R1 (Huang et al., 2025), needs to be added to strengthen the study's central claim regarding perception as the primary bottleneck. \n\n4. To enhance the evaluation, it is recommended to incorporate advanced multimodal math benchmarks like MathVision [3].\n\n\nReferences:\n [1] Chen et al., MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning, arXiv:2506.05331, 2025.\n [2] Huang et al., Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models, arXiv:2503.06749, 2025.\n [3] Wang et al., Measuring Multimodal Mathematical Reasoning with MathVision Dataset, NeurIPS 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work introduces a novel and valuable perspective on multimodal reasoning by isolating diagram perception from mathematical reasoning, a distinction often blurred in existing benchmarks like MathVista and MathVerse. The proposed “Math Blind” hypothesis is conceptually fresh and empirically grounded, offering a clear diagnostic lens for assessing visual understanding in symbolic domains. The paper is clearly written, visually well-presented, and effectively highlights a long-overlooked challenge in MLLMs—the perception of structured, symbolic diagrams.", "weaknesses": "- The paper operationalizes perception through four basic CV-style tasks (classification, counting, grounding, relationship). This oversimplifies the rich perceptual hierarchy involved in diagram understanding, which may include topological reasoning, implicit constraints (e.g., angle or symmetry inference), and multi-object composition.\n- The claimed perception-to-reasoning transfer (+3–4%) is small compared to the massive perception improvement (+79%), weakening the causal claim that “low-level perception is the main bottleneck of reasoning.”\n- Recent high-performing multimodal reasoning models such as Vision-R1, MINT-CoT [Chen et al., 2025], and the MathVision benchmark [Wang et al., 2024] are not included.\n- The “blind faith in text” phenomenon is intriguing, yet the paper lacks controlled prompts that explicitly modulate text–image priority (e.g., “prefer textual information” vs. “prefer visual cues”). Without this, the analysis remains observational, not causal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yLKGiAmYtR", "reviewer": "Reviewer_5MW7", "summary": "This paper investigates whether the failures of current Multimodal Large Language Models (MLLMs) in mathematical reasoning stem from limited diagram perception. The authors introduce MATHEMETRIC, a benchmark designed to isolate perceptual abilities (shape classification, object counting, relationship identification, grounding) from reasoning, and a synthetic GEOMETRIC dataset that encodes diagram structure as graphs of shapes and relations. Experiments on 18 MLLMs demonstrate that most models perform poorly on basic perceptual tasks, and that improved perceptual training can yield modest reasoning gains across MathVerse, MathVista, and GeoQA.", "questions": "See Weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. **Timely and relevant topic**: The paper addresses an important and underexplored aspect of multimodal reasoning: diagram perception, aligned with ongoing interest in evaluating the visual grounding of MLLMs.\n\n2. **Benchmark contribution**: The proposed MATHEMETRIC benchmark provides a systematic framework to separate low-level perception from reasoning, which is valuable for diagnosing model bottlenecks.\n\n3. **Synthetic dataset design**: The GEOMETRIC dataset introduces structured, graph-like representations of diagrams that could facilitate learning of spatial and relational structures.\n\n4. **Comprehensive evaluation**: The study covers a broad set of models and tasks, including both open- and closed-source MLLMs, and provides ablation studies on factors such as distractors, noise, and text–image conflicts.", "weaknesses": "1. **Lack of validation for benchmark annotation quality.**\nSince MATHEMETRIC and GEOMETRIC are key contributions, the paper should provide validation or human verification to ensure the accuracy and reliability of the annotated data. Without such validation, it is difficult to assess whether the benchmark truly measures perceptual ability rather than artifacts of synthetic generation.\n\n2. **Unclear causal interpretation of findings.**\nThe central question \"Do current MLLMs genuinely perceive mathematical diagrams?\" is not convincingly answered. The reported evidence (e.g., reliance on text, pattern memorization, robustness to distractors) does not establish causal links between perception and reasoning. The argument that stronger perception leads to better reasoning remains correlational, lacking controlled experimental design or counterfactual analysis.\n\n3. **Ambiguity in defining \"genuine perception\".**\nClaims about \"blind faith in text\" or \"Math Blindness\" are conceptually interesting but not rigorously operationalized. The study could benefit from clearer definitions or measurable criteria for \"genuine diagram perception\".\n\n4. **Potential shortcut effects.**\nIf improved perception data lead to higher reasoning accuracy, it remains unclear whether this reflects better perceptual grounding or merely learning shortcuts in synthetic data. This ambiguity weakens the interpretation of performance improvements as evidence of genuine understanding.\n\n5. **Writing and organization issues.**\nThe writing could be more concise and polished. Certain sections (e.g., section 3.3) contain excessive detail that obscures the main findings, while others (e.g., motivation and discussion) could better connect empirical results to claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lPZ4NrESz2", "reviewer": "Reviewer_4XJK", "summary": "The paper proposes a benchmark called MATHEMETRIC that just tests basic visual understanding, like counting shapes or finding relationships, separate from complex reasoning. They found that even top models fail these simple perception tasks, often just trusting the text blindly. Their fix is a new training dataset, GEOMETRIC, that explicitly teaches models the underlying structure of diagrams. When they trained models on this, perception scores shot up with a 79% gain on grounding. More importantly, this new visual skill transferred, making the models 3-4% better at actual high-level reasoning on other public benchmarks.", "questions": "I have a few required clarifications from the authors:\n\n1) The 3-4% gain on MathVerse -  Could the authors provide more qualitative examples in the appendix showing what kinds of reasoning problems the GEOMETRIC-trained models now solve correctly? Are they all problems that hinge on a single, correctly perceived geometric property, or does the improved perception unlock more complex, multi-step reasoning chains?\n\n2) The text-distractor experiment- While it shows that models over-rely on conflicting text. Did the authors perform the inverse experiment: providing a diagram with clear visual cues (e.g., two parallel lines) but no ambiguous or ambiguous textual information? Do models still fail, or can they reason from the visual-only input? This would further strengthen the \"blind faith in text\" hypothesis.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Primarily, for me, the main strengths of the paper are:\n\n1) Transfer of Learning: The most significant result is the 3-4% improvement on downstream reasoning tasks (Table 2). This demonstrates that fixing the low-level perceptual foundation directly improves high-level reasoning, confirming the paper's central thesis. Achieving this gain without any new reasoning data was a useful finding\n\n2) Isolation of variables: The authors rightly point out that existing benchmarks like MathVista and MathVerse conflate perception and reasoning, making it impossible to diagnose failure. The MATHEMETRIC benchmark is a good diagnostic tool that successfully disentangles these two capabilities.", "weaknesses": "A few weaknesses to point out:\n\n1) The paper's solution dataset is synthetic and focuses on plane geometry. While the authors show some cross-domain generalisation to solid geometry and graphs (Table 2), the gains are far smaller. The \"graph-based data construction\" is not yet extended to these other domains. This makes the solution feel less general than the diagnosis, although the authors mention this in the limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)