VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information
Authors: Ryo Kamoi
Yusen Zhang
Sarkar Snigdha Sarathi Das
Ranran Haoran Zhang
Rui Zhang
Venue: COLM 2025 (2025)
Keywords: vision-language models
TL;DR: We introduce VisOnlyQA, a dataset to evaluate the capability of Large Vision Language Models to perceive geometric information, such as lengths, angles, and shapes, and reveal that they still cannot accurately perceive basic geometric information.

Abstract: Large Vision Language Models (LVLMs) have achieved remarkable performance in various vision-language tasks. However, it is still unclear how accurately LVLMs can perceive visual information in images. In particular, the capability of LVLMs to perceive geometric information, such as shape, angle, and size, remains insufficiently analyzed, although the perception of these properties is crucial for tasks that require a detailed visual understanding. In this work, we introduce VisOnlyQA, a dataset for evaluating the geometric perception of LVLMs, and reveal that LVLMs often cannot accurately perceive basic geometric information in images, while human performance is nearly perfect. VisOnlyQA consists of 12 tasks that directly ask about geometric information in geometric shapes, charts, chemical structures, and 3D shapes. Our experiments highlight the following findings: (i) State-of-the-art LVLMs struggle with basic geometric perception. 23 LVLMs we evaluate, including GPT-4o and Gemini 2.5 Pro, work poorly on VisOnlyQA. (ii) Additional training data does not resolve this issue. Fine-tuning on the training set of VisOnlyQA is not always effective, even for in-distribution tasks. (iii) LLM may be the bottleneck. LVLMs using stronger LLMs exhibit better geometric perception on VisOnlyQA, while it does not require complex reasoning, suggesting that the way LVLMs process information from visual encoders is a bottleneck. The datasets, code, and model responses are provided at https://github.com/psunlpgroup/VisOnlyQA.

Published as a conference paper at COLM 2025
VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information
Ryo Kamoi, Yusen Zhang, Sarkar Snigdha Sarathi Das, Ranran Haoran Zhang, Rui Zhang
Penn State University
(ryokamoi, rmz5227)epsu..em
Abstract
Large Vision Language Models (LVLMs) have achieved remarkable perfor- mance in various vision-language tasks. However, it is still unclear how accurately LVLMs can perceive visual information in images. In particular, the capability of LVLMs to perceive geometric information, such as shape angle, and size, remains insufficiently analyzed, although the perception of these properties is crucial for tasks that require detailed visual understand- ing. In this work, we introduce VisOnlyQA, a dataset for evaluating the geo- metric perception of LVLMs, and reveal that LVLMs often cannot accurately perceive basic geometric information in images, while human performance is nearly perfect. VisOnlyQA consists of 12 tasks that directly ask about geometric information in geometric shapes, charts, chemical structures, and 3D shapes. Our experiments highlight the following findings: (i) State-of- the-art LVLMs struggle with basic geometric perception. 23 LVLMs we evaluate, including GPT--oo and Gemini 2.5 Pro, work poorly on VisOnlyQA. (ii) Additional training data does not resolve this issue. Fine-tuning on the training set of VisOnlyQA is not always effective, even for in-distribution tasks. (iii) LLM may be the bottleneck. LVLMs using stronger LLMs exhibit better geometric perception on VisOnlyQA, while it does not require com- plex reasoning, suggesting that the way LVLMs process information from visual encoders a bottleneck. The datasets, code, and model responses are provided at https: / thub. ccmm/psunllggroup//iiooo/o/oo
Geametry Mativiat)
Geometry Quadrlateral
Geometry iFiuur Mutifistal
Geometry
Geometry
Geometry Multrisaa
DE longer 0.25 0.5
trizngle Trum Falaw?
There quadriiatrrl True Falae?
Theangis BGC gume. dogress
langer
125 dagrees degress
18 0.500
Answer: False
Answer: True X Falre
Answer.
Answer: (b) degrezs
Answon (c) 0.25
Answor: Folse
GPT-44
True
(a)
(sl deginsss degieees
X (e) ouas
True
InternV.2220
(d
(stngie) iFluue from MMMU
Chemistry from NIVMLL
Charts- Chariia)
from (QA1
FicurofionCL
SuarrOLEVR
NH
NH
The attached
oppoalte vertices
hesagaraa structure.
Of and attached rpposire vertices of the heasgonel structioe
(6) 45198 47433 (6)
47077 46189
y-uluu 20087
interseet hear the value-2039
Estoria and Lebanan
times wideer then yellow cubee (e) Ibi 2/50
The cylinder ie
The srale befoween the directins the gray car and motorcydle
Faloe?
Ansove:: Falen
Acous::
68
Ansoee:: (e) 46570
Annwer: Falon Trus
Anewer 6d
Ancoert (e) 90 degreess
GPT-40
Trus
(16, (d)
[3) 47077
3.5
121 135 degrezs
0b
degreess
Figure 1: Examples from 12 tasks in VisOnlyQA and answers from LVLMs. Figures in VisOnlyQA are from existing datasets or generated by us, and all questions are created by us. Questions in this figure are abbreviated. Refer to Appendix H for full inputs and responses.
1

---

Published as a conference paper at COLM 2025
1 Introduction
Large Vision Language Models (LVLMs) have demonstrated significant advancement across a range of challenging vision-language tasks that require expert-level reasoning and knowl- edge (Liu et al., 2024; Chen et al., 2024b; OpenAI, 2024a). However, their ability to perceive visual information in images has not been sufficiently studied (Zhang et al., 2024a; Li al., 2024). Specifically, it remains unclear how accurately LVLMs can perceive geometric in- formation, such as shape, angle, and size, while geometric perception is fundamental to understanding visual information in images and is commonly required in vision-language tasks (Balachandran et al., 2024; Gao et al., 2025; Xing et al., 2025).
A primary obstacle to studying the geometric perception of LVLMs lies in the absence of dataset suitable for analyzing this capability, as in Table 1. (1) Recent popular datasets for evaluating LVLMs, such as MMMU (Yue et al., 2024) and Math Vista (Lu et al., 2024), target tasks that require expert-level reasoning and knowledge. The performance of LVLMs on these datasets largely affected by multiple capabilities and is not suitable for analyzing specific capabilities.
(2) While there exist datasets designed for evaluating LVLMs at per- ceiving visual information in images (Antol et al., 2015; Goyal al., 2017; Li et al., 2024), they often evaluate LVLMs in high-level comprehension tasks, such as scene understanding, which are not suitable for analyzing geometric perception and also do not necessarily require accurate perception of geometric information.
In this work, we propose VisOnlyQA, a new dataset designed to evaluate how accu- rately LVLMs can perceive basic geometric information in images. As in Figure 1, our dataset includes questions that directly ask about basic and common geometric informa- tion (e.g., length, angle, and shape) in diverse scientific figures, including geometric shapes, chemical structures, charts, and 3D shapes. VisOnlyQA has favorable properties for analyzing the capability of LVLMs to perceive geometric information: (1) The questions in our dataset do not involve challenging reasoning or knowledge, enabling us to exclusively evaluate the geometric perception of LVLMs independent of other capabilities.
(2) We use scientific figures to create unambiguous questions that directly ask about geometric information in images, which require accurate geometric perception.
We evaluate 23 LVLMs and observe that state-of-the-art LVLMs, including GPT-4o and Gemini 2.5 Pro, perform poorly in the basic geometric percep- tion tasks in VisOnlyQA (48.8% and 79.0% in accuracy on the Real split), while human performance is nearly perfect (93.5%), as in Figure 2. This result indicates that existing LVLMs often cannot accurately perceive com- mon geometric information in images, such as shape, angle, and size (54.1). In addition, we observe that this limita- tion persists even on simple geomet- ric shapes consisting of only two or three lines (64.2). This finding raises concerns about the faithfulness of LVLMs to visual input in vision-language tasks.
Eval-Real
Geren
Evall Eval-Synthatic
deren
Sythetlogerraat
Ang
Leng!)
Coade
Jena
Gecrve
Sentheticeeeuit
Chemiohy
Dammaiy
Figure 2: LVLMs perform poorly on VisOnlyQA, while human performance is nearly perfect. Ta- ble 5 provides detailed results.
To explore approaches to improve the capability of LVLMs to perceive geometric information, we evaluate LVLMs fine-tuned on the training set of VisOnlyQA. We observe that fine-tuning largely improves performance in some tasks and models, indicating that the lack of training data is part of the reason why LVLMs cannot accurately perceive geometric information. However, at the same time, fine-tuning does not always improve their performance VisOnlyQA, even for in-domain data, and our result shows that task properties and model size largely influence the performance after fine-tuning (54.4). While these findings suggest that enhancing the geometric perception of LVLMs is not straightforward, our experiments also indicate trend with implications for future improvement LVLMS using larger
2

---

Published as a conference paper at COLM 2025
Dataset
Geometric Perception
Require Accurate
Specifically Tangeting Visual Perception
Decoupled Evaluation Geometric Perception
Image Categories
Question Categories
MM-Ver (2024)
X
General Figures
Scene understanding, Math
SEED-Beach (2024)
General Figures
Scene underslanding
CharNiv (20245)
Charts
Math, Information extraction Math Academic exame
MatiVista (2024)
Math, Synthetic
MMMU (2024)
Math, Academic, Charts
ViOOlyQA (ours)
Math, Chemistry, Charts, 3D
Geometric information
Table VisOnlyQA is designed to evaluate the ability of LVLMs to perceive geometric information while removing the influence of other capabilities like reasoning. Popular datasets often evaluate multiple capabilities simultancously, making them unsuitable for analyzing specific capability. Existing datasets for evaluating visual perception often target high-level tasks and do not require the accurate perception of geometric information.
language models exhibit better performance on VisOnlyQA when using the same visual encoders. This is counterintuitive result because our dataset evaluates the perception of geometric information, which does not involve complex reasoning or knowledge. This finding suggests that the way to process visual information encoded by visual encoders is bottleneck to understanding geometric information in images, and strong language models are required for LVLMs to effectively process visual information (54.5).
In summary, VisOnlyQA reveals that current LVLMs still lack the capability to accurately perceive basic geometric information, such as shape, angle, and size, and simply scaling model size or training data is insufficient to fully overcome this limitation.
Related Work
Datasets for evaluating the geometric perception of LVLMs. HallusionBench (Guan et al., 2024) and IllusionVQA (Shahgir et al., 2024) show that LVLMs exhibit poor perception of geometric information in misleading figures, such illusive geometric shapes. Our work, in contrast, focuses on images and questions reflecting common and practical applica- tions. Fu et al. (2024) evaluate LVLMs by assessing the performance gaps when providing figures and text that include identical information (e-g., chess games in text and image representations). Our dataset provides more direct way to evaluate and analyze geometric perception. Gao et al. (2025) reports that GPT4-V suffers from hallucinations when describ- ing geometric shapes. Our dataset offers a more detailed analysis and shows that more recent models struggle with perceiving geometric information, even in much simpler tasks. GePBench (Xing et al., 2025) is a contemporaneous study with a similar motivation while specifically focusing on the relationships between multiple geometric shapes.
Datasets that evaluate multiple capabilities of LVLMs. Popular datasets for evaluating LVLMs often target tasks that involve complex reasoning knowledge, such math- ematical reasoning (Lu et al., 2021a; Chen et al., 2021; 2022; Lu et al., 2024; Gupta et al., 2024), chart understanding (Kafle et al., 2018; Methani et al., 2020; Masry al., 2022), and academic exams (Lu et al., 2022a;b; Yue et al., 2024). While these tasks also often involve the perception of geometric information in images, performance on these datasets is largely influenced by multiple capabilities, making them unsuitable for isolating and analyzing specific capabilities of LVLMs.
Datasets that evaluate the visual perception of LVLMs. Various datasets have been proposed to evaluate the visual perception of LVLMs, and they also often require geometric perception. However, popular datasets (Antol et al., 2015; Goyal et al., 2017; Gurari et al., 2018; Fu et al., 2023; Liu al., 2025; Xu et al., 2025; Li et al., 2024) often do not require accurate perception of geometric information, as they mainly target tasks that ask for high-level understanding of images, such as object recognition and scene understanding. This is potentially due to the difficulty in creating unambiguous questions that directly ask about geometric information on general images. In this work, we target scientific figures because they enable to annotate unambiguous questions about geometric information.
3

---

Published as a conference paper at COLM 2025
Geometry
Chemistry
Charts
3D
Triangle
Quadri- lateral
Length
Angle
Area
Diameter
Shape (s)
Shape (mn)
Extra- ctien
Inter- section
Size
Angle
Total
Eval-Real
100
100
100
100
100
100
50
50
100
100
900
Synt-hettheiic
100
100
100
100
100
100
100
700
Train
10k
10%
10k
10k
10k
10k
10k
20k
Answer
True// False
True/ False
True/
5
5
5 options
True/ Fabse
Select
True/
3
5
Format
False
options
options
Multiple
options
False
options
options
Table 2: Dataset statistics of VisOnlyQA. VisOnlyQA-Eval-Ren includes figures in existing datasets and uuman-annotated questions. VisOnlyQA Eval- Synthhtit and VisOnlyQA- Traiin comprise synthetic figures and automatically generated questions.
Number of Lines
4
Angle between
Geometry
Triangle
Length
50
50
50
50
Geometry
Length
50
Angle
50
50
50
50
(b) Length task with different angles between two lines.
(a) Geometric shapes with different numbers of lines.
Table 3: Statistics of datasets for analysis (Figure 4), which are based on Synthein...
VisOnlyQA Dataset
We introduce VisOnlyQA, new dataset designed evaluate and analyze the capability of LVLMs to perceive geometric information in images, such as shape, angle, and size. Each instance of VisOnlyQA consists of figure, a multiple-choice question, and answer label. As in Figure 1 and Table 2, VisOnlyQA includes 12 tasks on figures in four categories: geometric shapes, chemical structures, charts, and 3D shapes.
What are the favorable properties of our dataset? We design VisOnlyQA to be suitable for analyzing the capability of LVLMs perceive geometric information images. Specifically, our dataset includes questions that directly ask for a precise perception of basic geometric information in scientific figures. (1) This approach prevents questions from demanding challenging reasoning knowledge. As result, the perception of geometric information is the only bottleneck for recent LVLMs when solving tasks VisOnlyQA, and their perfor- mance this dataset is not largely influenced by other capabilities (54.3). This property of our dataset enables a direct eva evaluation of the geometric perception of LVLMs independent of other capabilities. (2) In addition, scientific figures enable us to create unambiguous questions that directly ask about geometric information in images.
How does our dataset help understand LVLM behavior general images? While our dataset targets scientific figures to make unambiguous questions that directly ask about geometric information, targets fundamental geometric information commonly required to understand the details of broad types of images, including real-world images (Xing et al., 2025). As our dataset includes simple and basic tasks involving common geometric perception, poor performance on VisOnlyQA raises concerns about the reliability of LVLMs in real-world vision-language tasks, not only in scientific domains.
3.1 Sources of Figures
VisOnlyQA includes two types of figures: Real and Synthetic. The Real figures are from existing datasets. We use figures in popular datasets to evaluate whether LVLMs truly understand images in those datasets. It also ensures that images are from real-world distributions and not adversarially created. Although we use existing images, all questions in VisOnlyQA are newly annotated. The Synthetic figures are automatically generated. The primary purpose of synthetic figures is to provide large-scale training data to analyze fine-tuned models. In addition, for evaluation, they ensure that there is no bias caused by human annotations because both images and questions are synthetically generated.
4

---

Published as a conference paper at COLM 2025
Scripte
Tamplate
Synthetic Geometry (0conctry Anglol
Image Generation Seripes
Pos tiens of Points 5, de coy
Calculate All Angles
Which the following options is reasorable estimate of the angle EGC?
Rardomnyyseeet anglle surzatle a aptions
Connected Prints (h She (N, fox
randoran
Angle sdc 8.1 dogreed
Answer: Qg) dagteoa
Sizee
Bardorty
The svlinda is X times wder th the yallow cubs. lot 0.5 2
3D (3D-Skm))
Dubbied
peimpti call endent you mater cmatt
atpecix that not tosides
Posttions
yellow cober C2 30.50
Compane
Answer: 1o)
Figure 3: Construction process of synthetic images and questions in VisOnlyQAAEval-Synttee and VisOnlyQA-Tres Train. This process does not involve language models and uses precise metadata, guaranteeing the correctness of generated question-answer pairs.
Real Figures. We use figures in popular datasets: geometric shapes in Math (Lu et al., 2024), which includes Geometry3K (Luu al., 2021a), GeoQA+ (Cao & Xiao, 2022), GEOS (Seo et al., 2015), and UniGeo (Chen et al., 2022), chemistry figures in MMMU (Yue et al., 2024), and charts in ChartQA (Masry et al., 2022) and CharXiv (Wang et al., 2024b).
Synthetic Figures. For geometric shapes, we create new dataset, SyntheticGeome- try, by generating geometric shapes by writing Python scripts based on an open source project (Leeb, 2024) reproducing AlphaGeometry (Trinh et al., 2024). For 3D shapes, use CLEVR (Johnson et al., 2017) and SuperCLEVR (Li et al., 2023c).
3.2 Question Annotation
For Real figures - Human annotation. We manually annotate questions and answers for ten tasks on the Real figures. We provide question templates and instructions to annotators, and each question-answer pair is annotated by one annotator and verified by another annotator. The annotators are PhD students specializing in natural language processing.
For Synthetic figures - Sfarree-Synthttiie questions. We generate synthetic questions using the metadata the synthetic figures and question templates, as in Figure 3. For geometric shapes, use the metadata in SyntheticGeometry, including the positions of points and lines. We write Python scripts to compute the geometric information (shape, length, angle, and area) from the metadata and generate question-answe pairs for five tasks. For 3D shapes, we write Python scripts to generate question-answer pairs about the relative sizes of objects CLEVR (3D-Size) and angles between objects in SuperCLEVR (3D-Angle) using the metadata in CLEVR (positions and sizes) and SuperCLEVR (positions and angles).
3.3 Dataset Design and Statistics
Data split. VisOnlyQA includes three splits: Eval-Real, Eval-Synthetic,, and Train. Eval-Real includes 900 instances for ten tasks on three categories of figures from existing datasets (ge- ometry, chemistry, and charts). Eval-Synthetic and Train include 700 and 70k instances for seven tasks on two categories of synthetic figures (geometry and 3D).
Analysis dataset. In addition to the main dataset, we provide dataset based on Eval- Synthetic for detailed analysis. We create datasets consisting of simple geometric shapes with a different number of lines and the Length task with different angles between two lines, as shown in Table 3 and Figure 4.
Reducing biases. To make label distribution balanced, we instructed the annotators to make an equal number of questions for each option. We also shuffled the options to remove biases caused by the order of the options. In addition, to avoid biases caused by the wording, we include a negative version of the questions for all true false questions (e.g., There is triangle ABC in this figure and There is no triangle ABC in this figure).
Annotation quality. We manually evaluate 100 randomly selected instances from the Eval- Real-Real set and confirm that all cases contain valid questions with correct answers, demon- strating the high quality of our dataset.
5

---

Published as a conference paper at COLM 2025
Human performance. We provide randomly sampled questions to three new annotators (300 instances in total) for the Real split and two new annotators (140 instances in total) for the Synthetic split. The average human performance is 93.5% and 95.0% in accuracy (5), showing that VisOnlyQA QA is easy for humans. We observe that most human errors are due to their mistakes, such misreading the order of symbols, rather than issues in the dataset.
Geometric shapes. Table 4 shows statis- tics of geometric shapes in VisOnlyQA. For the Real figures, we manually annotate 24 figures to get the statistics. For the Syn- thetic figures, we calculate the statistics of all images. Geometric shapes in the Syn- thetic data include more points, lines, and circles, with larger standard deviations.
# Points
# Lines
# Circles
Real Synthetic
5.0 (= 1.3)
5.8 (4: 2.2)
0.2 (4: 0.4)
9.3 3.2))
10.6 (+: 4.5) 0.4 (+ 0.5)
Table Average number (# std dev) of points, lines, circles in geometric shapes in VisOnlyQA.
Experiments
We evaluate 23 open and proprietary LVLMs and five fine-tuned LVLMs on VisOnlyQA, Our experiments aim to answer the following research questions:
RQ1: Can existing LVLMs accurately perceive geometric information? (64.1, 4.2.
RQ2: Does VisOnlyQA evaluate the capability to perceive geometric information indepen- dent of other capabilities, such as reasoning and knowledge? (64.3)
RQ3: Does additional training data improve the geometric perception of LVLMs? ($4.4)
RQ4: Do language models of LVLMs influence their geometric perception? (64.5)
Models: We evaluate 23 LVLMs in 9 model families, including 15 open models: Phi-3.5- Vision (Microsoft, 2024), LLaVA-Next (8B, 34B) (Li et al., 2025), Llama 3.2- (11B, 90B) (Meta, 2024), Molmo (7B-D, 72B) (Deitke et al., 2025), Qwen2-VL (2B, 7B, 72B) (Wang et al., 2024a), InternVL2 (4B, 8B, 26B, 40B, 76B) (OpenGVLab Team, 2024); and propri- etary models: Claude Sonnet 3.5 (Anthropic, 2024), Sonnet 4, Opus 4 (Anthropic, 2025), GPI-4o-mini, GPT-4o (OpenAI, 2024a;b), 1emini-1.5 Flash and Pro (Google, 2024), and Gemini 2.5 Pro (Google, 2025). Refer to Appendix B for details.
Prompts: We evaluate two types of zero-shot prompts: with and without chain-of-though reasoning (Wei et al., 2022; Kojima et al., 2022). Full prompts are in Appendix C.
4.1
LVLMs Cannot Accurately Perceive Basic Geometric Information in VisOnlyQA
Table 5 shows the accuracy of LVLMs on Eval-Real and Eval-Synthetic (with no chain-of- thought). The performance of LVLMs far from perfect on all tasks, with the best average accuracies of 79.0% and 55.4% by Gemini 2.5 Pro on the Real and Synthetic splits, while human performance nearly perfect (93.5% and 95.0%).
Our results show that larger LVLMs exhibit better capability in perceiving geometric in- formation but also indicate that simply scaling model size does not lead to human-level performance. Specifically, even large models perform near-randomly ing Geometry- Quadri Quadrilaters laterall and as Geometry-Angle and 3D-Angle in the Synthetic split. Gemini 2.5 Pro is the only model that achieves high performance on chemistry and chart figures, but still exhibits near-random performance on most tasks involving geometric shapes. This is a cautionary observation, indicating that existing LVLMs still cannot accurately perceive basic geometric information, such as angle, shape, and intersection. Appendix provides examples of model responses.
some tasks, includ- in the Real split, as well
4.2 LVLMs Exhibit Poor Geometric Perception Even on Simple Geometric Shapes
Results in Section 4.1 show that LVLMs exhibit poor geometric perception capabilities. To further analyze this limitation, we create dataset for analysis (Table 3) that includes simple geometric shapes with different complexities and tasks with different difficulties.

---

Published as a conference paper at COLM 2025
Geometry
Chemistry
Charts
Triangle
Quadri- lateral
Diameter
Length
Angle
Area
Shape (8]
Shape (m)
Extraction
Inter- section
Average
Random
50.0
50.0
20.0
20.0
20.0
30.0
50.0
6.2
20.0
50.0
34.2
Phi-3.5 --visinn
48.0
50.0
17.0
17.0
27.0
30.0
54.0
10.0
29.0
50.0
35.6
LLaVA-Next 8B
50.0
50.0
16.0
15.0
26.0
49.0
42.0
4.0
22.0
49.3
33.3
LLaVA-Next 34B
49.0
50.0
30,0
15.0
22.0
44.0
34.0
10.0
35.0
50.0
35.2
11B
1ama 90B
51.0
46.0
14.0
28.0
27.0
48,0
60.0
20.0
35.0
45.0
37.1
MalMo 7B D
49.0
45.0
20.0
11.0
23.0
56.0
40.0
12.0
31.0
48.0
34_1
MolMo 728
44.0
47.0
22.0
25.0
33.0
50.0
48.0
30,0
46.0
52.0
39
Qwen2-VI-22
32.3
Qwen2-VL.22
50.0
50.0
23.0
19.0
34.0
46.0
46.0
16.0
45.0
52.0
38.9
Owen2-VL
44.0
$2.0
27.0
27.0
37.0
61.0
56.0
36.0
$3.0
$3.0
44.4
V1.24V1.224B
180
49.0
54.0
$3.0
38.4
VL2
222m2V12-2BB
44.0
47.0
24.0
22.0
26.0
35.0
58.0
28.0
47.0
46.0
39.3
IntemVV12 40B
43.0
45.0
$2.0
23,0
31.0
57.0
28.0
30.0
61.0
58.0
42.1
IntemV12-76B
44.0
42.0
28.0
34.0
45.0
56.0
60.0
36.0
54.0
46.0
Claude Sonnet 3.5
50,0
47.0
25.0
20.0
33,0
59.0
52.0
40.0
61.0
52,0
43.4
Claude
Claude Opus
47.0
34.0
36.0
60.0
72.0
50.0
49.3
GPT-to-mini
43.0
66.0
26.0
19.0
30,0
58.0
58.0
32,0
40.0
53.0
42.4
GPT-4o
58.0
48.0
27.0
34.0
38.0
69.0
72.0
50.0
46.0
58.0
48.8
Gemini Flash
Gemini 1.5 Pro
47.0
53.0
33.0
40.0
53.0
70.0
62.0
52.0
67.0
53.0
52.6
Gemini 2.5 Pro
66.0
52.0
55.0
39.0
56.0
90.0
92.0
88.0
86.0
72,0
79.0
Human
96.7
90.0
93.3
93.3
86.7
100.0
93.3
93.0
93.3
95.0
93.5
(a) Accuracy VisOnlyQA-Eval
Geometry
Average
Triangle
Quadrilateral
Length
Angle
Size
Angle
Random
50.0
30.0
20.0
20.0
20.0
33.3
20.0
30.5
Phi-3 5-vision
54.0
55.0
15.0
22.0
21.0
39.0
20.0
32.3
LLaVA-Next 8B
81.0
17.0
21.0
19.0
26.0
28.9
LLaVA-Next344
50.0
25.0
24.0
20.0
48.0
32.0
35.7
Llama 3.2 11B
54.0
52.0
31.0
21.0
21.0
32.0
21.0
33.1
Lama 3.2 90B
61.0
36.0
12.0
16.0
20.0
45.0
26.0
33.7
MolMo 7B-D
MolMo
23.0
22.0
18.0
500
27.0
35.1
Owen2-VL-28
50.0
50.0
31.0
23.0
20.0
38.0
23.0
33,6
Qwen2-VL-7B
58.0
39.0
24.0
18.0
22.0
58.0
21.0
37.1
Owen2-VL-728
76.
InternVL2-4B
51.0
21.0
24.0
18.0
57.0
18.0
34.1
InteraVL.2-BB
51.0
57.0
21.0
17.0
23.0
46.0
30.0
35.0
IntemV1122268
51.0
53.0
30,0
23,0
21.0
72.0
25.0
39.3
IntemVL2-400
IntemVVL2-76B
29.0
18.0
22.0
$4.0
27.0
40.4
Claude Sonnet
Claude Sonnet
57.0
$9.0
2800
32
30.0
79.0
28.0
44.7
Claude Opus
50.0
53.0
36.0
21.0
24.0
84.0
26.0
42.0
GPT-4o-minl
GPT-4o
17.0
Gemini 1.5 Flash
51.0
29.0
21.0
19.0
60.0
21.0
36.4
Gemini 1.5 Pro
54.0
57.0
34.0
21.0
40.0
69.0
22.0
42.4
Gomini 2.5 Pro
47.0
74.0
55.4
Human
95.0
95.0
95.0
90.0
95.0
100,0
95.0
95.0
(b) Accuracy on VisOnlyQA-Eval-Syrnnn
Table 5: Accuracy of LVLMs VisOnlyQA-Eva with chain-of-thoughh reasoning. All LVLMs perform much worse than humans and are comparable to random performance in many tasks. Bold font indicates the best model performance in each column.
We evaluate Intern VL2 76B and Gemini 1.5 Pro. First, Table shows that these models consistently exhibit poor geometric perception on geometric shapes with different numbers of lines (i.e., complexity). As shown in Figure 4, even simple geometric shapes that only include two or three lines, LVLMs cannot accurately perceive shape, length, and angle. Second, Table 7 shows that the angle between two lines does not largely influence the performance on the Length task, which compares the lengths of two lines, while we expected that this task would be more difficult when the angle is larger. These results suggest that the current WLMSs face fundamental challenges in geometric perception, regardless of the complexity of the geometric shapes or the difficulty of the tasks.
7

---

Published as a conference paper at COLM 2025
Number of Lines =2
Number of Lines 3
Angle between Two Lines =
Angla between Two Lines 90
What angle figure?
There triangte figure.
longer than
longor
Answer: (b) 45 degrees
Answer:
True
Answer: (b) 0.25
Answer: lal
Gemini 1.5 Pro
[d] 135 degrees
False
0.5 2
IntemVL2-708
(d) 133 degrees
False
Figure 4: Example figures and model outputs for the analysis dataset. LVLMs exhibit poor geometric perception even on very simple geometric shapes.
Lines
Angle
Triangle
Angle
Lengih
34.0
18.0
20.0
20.0
50.0
24.0
22.0
22.0
20.0
22.0
30.0
24.0
42.0
30.0
42.0
44.0
26.0
22.0
44.0
30.0
38.0
Gemini
InterVL12
24.0
36.0
38.0
36.0
(a) InternVL22 76B
(b) Gemini 1.5 Pro
Table 7: Accuracy on the Length task with different angles be- tween two lines.
Table 6: Accuracy of LVLMs on simple geometric shapes.
4.3 VisOnlyQA Evaluates Geometric Perception Independent of Other Capabilities
To verify our claim that VisOnlyQA evaluates the capability to perceive geometric information independent of other capabilities, this section demonstrates that our dataset does not involve reasoning or knowledge difficult for recent LVLMs. If recent LVLMs do not make mistakes in reasoning or knowledge our dataset, we can conclude that the performance of LVLMs on this dataset evaluates the capability to perceive geometric information alone. this section, examine chain-of-thought reasoning of LVLMs for error analysis.
Error analysis in chain-of-thoughh. Chain- of-thought reasoning provides clues to analyzing why LVLMs make mistakes.
GIT-4o HemmV22700 Inte4422-600 Ptera 80 Intem2240
955
We manually annotate errors in chain-
of-thought reasoning by six models on VisOnlyQA-Eval-Reay and provide the results
97%
in Figure 5. We manually annotate error categories for 250 responses (50 responses for each model). Following prior work (Yue et al., 2024; Zhang et al., 2024a), we classify their errors into the following categories. Refer to Appendix for details.
12%
05
94%
Question Understanding Error: LVLMs understand questions incorrectly.
05
91%
Visual Perception Error: LVLMs do not
correctly perceive visual information. In
0%
0%
Genivi 15 Pro
our dataset, this category only involves
97%
errors in geometric perception.
Reasoning Error: Reasoning on per- ceived information includes mistakes.
Quastion Understanding
Viaual Pertzptiin
Insufficint Reasoning
Aadundarc Readoning
Passonin Enrrr
Minor Problems in Reasoning: Reason- ing is insufficient or redundant.
Figure 5: Error categories in chain-of-thougle reasoning by LVLMs VisOnlyQA-Evas Rval-Real. Almost all errors are visual perception errors, verifying that our dataset evaluates the geo- metric perception of LVLMs independent of other capabilities. Each response can include multiple categories of errors.
We observe that almost all errors are visual perception errors, as in Figure 5, verifying that our dataset evaluates the geometric perception of LVLMs independent of other capabilities. Specifically, almost all errors

---

Published as a conference paper at COLM 2025
made by Gemini 1.5 Pro do not involve anything other than visual perception errors, indi- cating that VisOnlyQA can evaluate geometric perception almost entirely independent from other capabilities for future models stronger than Gemini 1.5 Pro. However, at the same time, we need to be cautious when comparing the performances of LVLMs with weaker reasoning capabilities, as up to 10% of their mistakes on our dataset may not involve visual perception errors; if the performance difference between two weak models in our dataset small, we cannot conclude either is better at geometric perception. Still, these errors in other capabilities do not affect our conclusion that existing LVLMs exhibit clear limitations in perceiving geometric information.
Chain-of-thougie does not consistently improve performance. We also observe that chain- of-thought does not consistently improve the performance of LVLMss on VisOnlyQA (Ap- pendix F.1). This result differs from observations on datasets for the visual reasoning tasks, where chain-of-though largely improves performance (Wu et al., 2023; Chen et al., 2024a; Zhang et al., 2024a). This result is consistent with our claim that reasoning is not bottleneck in our dataset for recent LVLMs and does not largely influence the final performance.
4.4 Additional Training Data Does Not Always Improve Geometric Perception
Motivation and hypothesis. We hypothesize that current LVLMs struggle to perceive geometric information due to a lack of training data requiring this capability, consistent with the motivation of prior work (Gao et al., 2025; Xing et al., 2025). To verify this hypothesis, we evaluate LVLMs fine-tuned on VisOnlyQA-
Settings. We fine-tune InternVL2 (4B, 8B, 26B) (OpenGVLab Team, 2024), Qwen2-VL (2B, 7B) (Wang et al., 2024a), and Phi-3.5-Vision (Microsoft, 2024) on each task in VisOnlyQA- Train (7 tasks in total) and evaluate on Eval-Synthetic (in-distribution; figures from the same distribution as the Train data) and Eval-Real coutoof-distibbution). To evaluate the maximum fine-tune each model in a single-task setting 10k training data. In total, we fine-tune seven models independently for each LVLM. We use prompts without chain-of-thought. Refer to Appendix D for detailed settings.
possible performance,
Improvement by fine-tuning depends on task properties. As shown in Figure 6, LVLMs fine-tuned on VisOnlyQA- Train exhibit both positive and negative results in VisOnlyQA. Positive results: All models achieve near- perffce performance in 3D-Size after fine-tuning, and models larger than show large improvement even on the out-of-distribution figures in Geometrys try-tength and Area. This result partially supports our hypothesis that training data for existing LVLMs are insufficient and indicates that our approach of using synthetic training data has the potential to improve the capability of LVLMs to perceive geometric information. Negative results: However, fine-tuned models are still often much worse than human performance, even on in-distribution figures. Specifically, fine-tuning almost does not improve performance in 3D-Angle, e, and we observe relatively small improvements
VisOnlyQA EyanllQeetell-SSthheeieiii (In Distribbuiion Figures)
Thtnggee
Quadriattrral
Angle
Synthecisseemetryy
IisOnlyQA-Eval-Res (Outtof-Distribation Figures)
Acoursy
Ph-3 m--3.5-uston
Intervi 2-B
Owen2-99222
IntervVL.-8B
Qwee2-91-70
Intervii 2-203
Tringgle
Quadnlateral
Angle
Geometry
Figure 6: Accuracy after fine-tuning on VisOnlyQA- Train. We evaluate on VisOnlyQA-Eue Synthetic, which is generated from the same distribution as the training data, and VisOnlyQA- Eval-Real, which includes images from different distributions. The numbers above the bars represent the accuracy after fine-tuning, and the ones inside the bars represent the improvements from the original models (white bars with hatches). Details are in Table 13.

---

Published as a conference paper at COLM 2025
Geometry-Triangle, Quadrilateral, and Angle, even on in-distribution figures. This result indicates that fine-tuning on datasets that require accurate perception of geometric information is not always effective, depending on the properties of target tasks.
Improvement by fine-tuning depends on model sizes. Figure 6 shows that models larger than 7B tend to achieve greater performance gains after fine-tuning. Specifically, mod- els larger than 7B exhibit much larger improvements than smaller models in Geometry- Length and Area. This result suggests that model sizes largely influence their capability to perceive geometric information, even when training data for target tasks is available. Saturation: However, we also observe that InternVL2-261 achieves almost the same perfor- mance as Intern after fine-tuning. It suggests that simply fine-tuning larger models on our datasets will not achieve human performance.
Was our hypothesis supported? Partially, Our results indicate that the insufficiency of training data is one of the reasons why the current LVLMs often cannot accurately perceive geometric information in images. However, depending on target tasks and models, additional training data does not always resolve the issue.
4.5 Larger Language Models Improve the Geometric Perception of LVLMs
InternVL2 4B and 8B, and Qwen2-
VL 2B and 7B, respectively, use the same vision transformer (ViT) within each pair while differing in their language models. We ex- pected the visual encoders to play major role in geometric percep- tion and models using the same VIT to perform similarly on Vi- sOnlyQA, particularly after fine- tuning, since fine-tuning would help models understand tasks, fur- ther reducing the impact of the reasoning capability of language models of LVLMs. However, LVLMs using the same ViT and different language models, and the gaps become larger after fine-tuning. This observation indicates that language models of LVLMs affect the capability to perceive geometric information, and the influence of LLMs of LVLMs is not limited to reasoning or knowledge. This result suggests that language models play a crucial role in processing visual information encoded by ViT, and strong language models are needed even in geometric perception tasks that do not involve challenging reasoning or knowledge.
Original
Fine-tuned
VIT
LLM
Real
Synthetic
Real
Synthetic
InternVL2-4B
304M
3.8B
38.4
34.1
46.0
57.7
VL2-8B
304M
7.7B
40.7
35.0
52.4"
64.6"
Qwen2-VL.23
675M
1.5B
32.3
33.6
43.8
54.6
Qwen2-V1-77 675M
7.6B
38.9*
37.1*
48.2"
65.0"
Table 8: Larger language models improve the per- formance of LVLMs on VisOnlyQA Eval when using the same visual encoders. *: Larger model is bet- ter (p < 0.05, paired bootstrap (Koehn, 2004)).
shown in Table 8, there are performance gaps between
5 Conclusion
This work evaluates the capability of LVLMs perceive geometric information in images, such as shape, angle, and size, and reveals that the current LVLMs still often cannot ac- curately perceive basic geometric information. We introduce VisOnlyQA, a new dataset designed for evaluating the geometric perception of LVLMs independent of other capa- bilities, such as reasoning. Our experiments on VisOnlyQA show a cautionary observation indicating that LVLMs still cannot accurately perceive basic visual information and may not be faithful to the input images in vision-language tasks. We also create a training set of VisOnlyQA to investigate approaches to improve the geometric perception of LVLMs. Our analysis of models fine-tuned on the training data suggests that simply scaling model size or training data does not fully resolve this issue in the perception of geometric information.
10

---

Published as a conference paper at COLM 2025
Reproducibility Statement
In our GitHub repository, provide our VisOnlyQA dataset, code for dataset creation and all experiments, and model responses. The appendix includes details of model access, prompts, and hyperparameters.
Acknowledgment
This work was supported by NSF CAREER Award IIS-2338418. We also thank OpenAl's Researcher Access Program for providing API credits. We appreciate VLMEvalKit for supporting our dataset." We are grateful Kai Katsumata for the valuable discussions and to Xueqing Wu for constructive feedback on our dataset. We appreciate valuable suggestions from anonymous reviewers, including those recommending experiments Section 4.2.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikot aj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zis- serman, and Karén Simonyan. Flamingo: a visual language model for few-shot learn- ing. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 23716-23736 Curran Associates, Inc., 2022. URL
neerue cctpppp.ccpppppppppprreerr hwtpppperfilll//ppeeeeeeeeeeeeee ftape//aa//ppp///22222 pdf.
Anthropic. Claude 3.5 sonnet model card addendum, 2024.
URL https:
//www-cdn www.wwcc..aathriiiic.c. cwm/
Addeendum. .ff.
Anthropic. System card: Claude opus & claude sonnet 4, 2025, URL https::/ hct.cccc/cee24444c4c444eeeeeee 222652226666666443322222222 .ddf.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruy Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceed- ings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint Xiv:2308 12966,, 2023.
Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Lucy, and Safoora Yousefi. Eureka: Evaluating and understanding large foundation models. arv preprint ar Xiv:2409.11066 2024.
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung- Hoon Na (eds.), Proceedings of the 29th International Conference Computational Linguistics, pp. 1511-1520, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https:://aclannhhlloy org/ 2822. col ing-1. 130.
11

---

Published as a conference paper at COLM 2025
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 513513-3555 513-523, Online, August 2021. Association for Computational Linguistics. doi: 10.18653 /v11 .ii..iiinnsasll..... URL
22rgg2201..oo wttp///.....nnniiii.e.ee...ae... findings-and ac1.46.
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. UniGeo: Unifying geometry logical reasoning via reformulating mathematical expression. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3313-3323, Abu Dhabi, United Arab Emirates, December 2022, Association for Computational Linguistics. doi: 10.18653/v v1 (188683/112222emm/// /222222emmmlmmaiin. 218.
URL https: h haaass///l/aaatoooggyyy org/2022. w.ne22.emnlppp... emnlp-man mamin...
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Measur- ing and improving thain-of-thoughe reasoning in vision-language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapler of the Associalion for Compulational Linguistics: Human Language Technologies (Volume Long Papers), pp. 192-210, Mexico City, Mexico, June 2024a. As- sociation for Computational Linguistics. doi: 10.18653/v w1w.:/2..aacl-lllaaaa org/2024
URL
https://aclanthollee
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. ar Xio preprint ar arXiv:2404.16821 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24188-24198, June 2024c.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli Van- derBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kemb- havi. Molmo and pixmo: Open weights and open data for state-of-the vision-languaa models. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), Pp. 91-104, June 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborm, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learnng Representations, 2021. URL https://openreview wtw.reerrriiiieeeeeoorruuumm/mme net/forum2id--Viccod
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, P'an Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. VImevalkit:
An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference Multimedia, MM '24, pp. 11198-11201, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706868. doi: 10.1145 (0.1145/3664647, 3664647.3688520. URL https: //doi. org/10 44445/666447777 3685520.
12

---

Published as a conference paper at COLM 2025
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint ar Xr:::22306.13339 13394, 2023.
Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. Isobench: Benchmarking multimodal foundation models on isomorphic representations. In First Conference on Language Modeling, 2024.
URL https::/ http://wwee/ooooo///dddoodlee
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing HONG, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-LLaVA: Solving geometric problem with multi-modal large language model. In The Thirteenth International Conference on Learning Representations, 2025. URL httph:://ppeeeet /twp///reeww.ee/////eeeeeeeeeeee px1674Wp3C.
Buse Giledereli, Yifan Hou, Yilei Tu, and Mrinmaya Sachan. Do vision-language models really understand visual language? arXiv preprint arXiv:2410.00193, 2024.
Google. Our next-generation model: Gemini 1.5, 2024. URL https: googleeellgoog///////ooo hatpmoologg///gggglleeeee
Google. Gemini 2.5: Our most intelligent ai model, 2025. URL httpps///bog. //oog// gochnology/googlee /eepeeedddeedd//ggeeeeeeee
ch--2222/..
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEF.F. Conference Computer Vision and Patlern Recognition (CVPR), July 2017.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hal- lusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14375-14385, June 2024.
Himanshu Gupta, Shreyas Verma, Ujjwala Anantheswaran, Kevin Scaria, Mihir Parmar, Swaroop Mishra, and Chitta Baral. Polymath: A challenging multi-modal mathematical reasoning benchmark. ar Xio preprint ar arXio:2410.1470, 2024.
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Zihan Huang, Tao Wu, Wang Lin, Shengyu Zhang, Jingyuan Chen, and Fei Wu. Autogeo: Automating geometric image dataset creation for enhanced geometry understanding. IEEE Transactions on Multimedia, 27:3105-3116 2025. doi: 10.1109
Nam Hyeon-Woo, Moon Ye-Bin, Wonseok Choi, Lee Hyun, and Tae-Hyun Oh. VLM's eye examination: Instruct and inspect visual competency of vision language models. Transactions on Machine Learning Research, 2025, ISSN 2835-8856. URL LTL.0///IMMM..22555757722 net/Forummidd-CgWWW2222
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
13

---

Published as a conference paper at COLM 2025
Samira Ebrahimi Kahou, Adam Atkinson, Vincent Michalski, Åkos Kádár, Adam Trischler, and Yoshua Bengio. FigureQA: An annotated figure dataset for visual reasoning, 2018. URL http::
Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. In Al for Math Workshop @ ICML 2024, 2024. URL
Sebastian Koch, Albert Matveev, Zhongshi liang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Marc Alexa, Denis Zorin, and Daniele Panozzo. Abc: A big cad model dataset for geometric deep learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Philipp Koehn.
Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu (eds.), Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp- 388-395, Barcelona, Spain, July 2004. Association for Computa- tional Linguistics. URL https: /fttath//allanthoologg... www.w/g-www./ow. /
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Re- trieval and Language Models, 2022. URL https://openreww/ wwt.:eet//oooooee
iN.
Alexander Kuhnle and Ann Copestake. Shapeworld new test methodology for multi- modal language understanding, arXiv preprint wff:::1044.0411, 2017.
Felix Leeb. https:/ / com /wt.///////oooo//////a/popp http:////doooopaaaoooe.too 2024. URL com/ hwt.feiixxuudoo//llppaaggeeetttt
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying
Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 13299-13308, June 2024.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- Curran Associates, ral Information Processing Systems, volume 36, pp. cc/paper. www.ffillefpppeeeee paper/2023/filee Inc., 2023a. URL httts:://proceediggs.ellle 5
one day. In A. Oh, T. Nau-
and Benchmarks... .d
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun MA, and Chun- yuan Li. L.LaVA-neXT-intellavve Tackling multi-image, video, and 3d in large multimodal models. In The Thirteenth International Conference on Learning Representations, 2025. URL http///ormmmiio00ii00000000
https://openreviee
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 23-29 Jul pwwess/ve2//1iii htt.eeednnng.m.....psss/////////
2023b. URL https: wwwpppeeinn.....oooeeeerppprr.pp mlr.mmrrr
html.
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L. Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14963-14973, June 2023c.
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Eisenschlos. MatCha: Enhancing visual language pretraining with math reasoning and chart derendering. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association
14

---

Published as a conference paper at COLM 2025
for Computational Linguistics (Volume 1: Long Papers), PP. 12756-12770 Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653 10.11663/w1222222..m.. 714. URL https://aclanthhoooyy.o/////////
714.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL https: ///pperreieew.nt//oornmmnnnmmeie nett/forumniddww@H22tti....
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26296 26306, June 2024.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision ECCV 2024, pp. 216-233, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72658-3.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Chun Zhu. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Aunual Meeting of the Association for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Pp. 6774,6786, Online, August 2021a. Association for Computational Linguistics. doi: 10.18653/ ht////11122211////// w1/2022aacl-oon..22222 URL https: http///ccaaatthootco org/2021 acl acl-long. 528.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems, volume 35, pp. 2507-2521, Curran Associates, Inc., 2022a. URL https: httpp:://proceeee http:///pooceeenss.... neureps.cc/pppprr saappe-ffless/e ttlesppaper/0022/ill .ddf.
11332b666ff488544afadd w5w.aa9a-aaaae-aaaaaacc
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URL https: / /wtp///pppeeeee.eeeeeeeee net/ fot///drmmmmmnjjkk Tc Tc_Bc. Bc.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math- ematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, 2024. URL https: http:::///ppereveerr wwwperewwwwwwwwwneeeeeeeeeeeeeee net/forum?id= KUNZEQMWU7.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 2269,227,, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/vi /v1//222..inniii//// ann..iinnnga...11.. URL
org/ 2022. www.o//////....nnn...reo acl.177.
Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq UniChart: A universal vision-language pretrained model for chart comprehension and reasoning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14662-1 16684, Singapore, December
15

---

Published as a conference paper at COLM 2025
2023. Association for Computational Linguistics. doi: URL https::/aalaanthlllgee.oo org/2033. 12023.emmaa. .et./222eemmmpmmain...m 906.
Meta.
Llama 3.2: Revolutionizing edge ai and vision with open, URL httas://ai.meta.coomomm www...c......eeeee
customizable models, 2024.
Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In The IEEE Winter Conference on Applications of Computer Vision (WACV), March 2020.
Microsoft. Discover the new multi-lingual, high-quality phi-3.5 slms, 2024. URL
blog/
OpenAL. Gpt-4 technical report. arXiv preprint 2r0
OpenAL. Hello gpt-4o, 2024a. URL /ttpt/////////oaaaaaaaa////////a
OpenAI. Gpt-4o mini: advancing cost-efficient intelligenne,, 2024b. URL https:://ppeaaii.
com/ indeiddd//iie wwwpiiignetd-ooo---eene -adi-ndvannning cost oitpaancnnnnnsssefefeeiieeiinnne
OpenGVLab Team. Internv12: Better than the best expanding performance boundaries of open-source multimodal models with the progressive scaling strategy, 2024. URL
https://interrwll.gg thu....tttti
Long Ouyang, Jeffrey Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744 Cur- ran Associates, Inc., 2022. URL https: http://///.coooeeenieee..eco nwwrips....... cc/ wwwpppppppaaaaecceeceee filesesfleesppppeetr .df.
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Lluis Mårquez, Chris Bull,son-Burch, and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Pp. 1466-1476, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: (0.18633/v1/115.1 https://aacaantnooogy...r/////-3
URL
Haz Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhattacharjee, Wasi Uddin Ahmad, Yue Dong, and Rifat Shahriyar. Illusion VQA: A challenging optical illusion dataset for vision language models. In First Conference on Language Modeling, 2024. URL https: wwwp:eewwweeforuueeeeee wttp://wwww.neeeoooo/////////a/e
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi A corpus of natural language for visual reasoning. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Compulational Linguistics (Volume 2: Short Papers), pp. 217-223, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1 11/4111221... URL httpttts//cclatthooloooo
org/P17-2034.
Trieu H. Trinh, Yuhuai Wu, Quoc Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995)446-482,, January 2024. ISSN 1476-4687. doi: 1411038//41586-02--0000 941586-923-96747-
URL http: http//dxdddooog//////// 1038/
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arviv preprint 2409.12191, 2024a.
16

---

Published as a conference paper at COLM 2025
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, and Danqi Chen. Charxiv: Charting gaps in realistic chart understanding in multimodal LLMs. In
The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchnarks Track, 2024b. URL htttps://openreview httpp/pprrreiieeeeeeeeeorrrraiia nett:foruumidc=cym@TTkk...
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds)), Advances in Neural Information Processing Systems, 2022. URL https://openreview. net//forum?idc..vjllleesBB
Yifan Wu, Pengchuan Zhang, Wenhan Xiong, Barlas Oguz, James C. Gee, and Yixin Nie. The role of chain-of-thought in complex vision-language reasoning task. ar Xiv preprint wrXiv:2311.09193 2023.
Shangyu Xing, Changhao Xiang, Yuteng Han, Yifan Yue, Zhen Wu, Xinyu Liu, Zhangtai Wu, Fei Zhao, and Xinyu Dai. Gepbench: Evaluating fundamental geometric perception for multimodal large language models. ar arviv preprint ar Xie:2412.220366 2025.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation bench- mark for large vision-language models. IEEE Transactions on Pattern Analysis and Machine doi: 0T000//TPAM.220243550777
Intelligence, 47(3):1877-1893 2025.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles, Caiming Xiong, and Ran Xu. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint ar arXio:2408.088 08887, 2024.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen,eunfeng Tian, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXio:2304.14178, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-vet: Evaluating large multimodal models for integrated capabilities. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview.net hew.:eewnee/fooormmicootttttrss
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9556-9567, June 2024.
Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Yin and yang: Balancing and answering binary visual questions. In Proceedings of the IEEE yang: Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal IIm truly see the diagrams in visual math problems? In Computer Vision - ECCV 2024: 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part VIII, pp. 169-186, Berlin, Heidelberg, 2024a. Springer-Verlle ISBN 978-3-031-73241-6. doi: 10.1007/978-3--0000 177799978--031-2222-- 10. URL https: /tt:::/doiioooo org/ 1007/ 978-3-031-732422 10.
17

---

Published as a conference paper at COLM 2025
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Shanghang Zhang, Peng Gao, and Hongsheng Li. MAVIS: Mathematical visual instruction tuning with automatic data engine. In The Thirteenth International Conference on Learning Representations, 2025. URL https: httt::://ppnreeiiee/ net/forummid= MnJzJ2gvuf.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arViv preprint ar Xiv:2306.1 17107, 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In The Twelffh International Conference on Learning Representations, 2024. URL https: /openr/viee. net/forum?id=
12 tZbq88f27.
18

---

Published as a conference paper at COLM 2025
Table of Contents of Appendix
A Additional Related Work
20
Model Access
B.1 Proprietary Models
21
B.2 Open Models
C Details of LVLM Evaluation
23
D Details of Fine-tuning
24
Details of Chain-of-Thought Error Analysis
F Additional Results
26
F1
Improvements by Chain-of-Thought
26
F2 Improvements by Fine-tuning.
27
F3 Fine-tuning of Different Components of LVLMs
27
G Computational Resources
28
Example Data and Model Outputs
29
19

---

Published as a conference paper at COLM 2025
A
Additional Related Work
Large vision language models. Recent LVLMs often consist of vision transform- ers (ViT) (Dosovitskiy et al., 2021) and large language models (Ouyang et al., 2022; OpenAL, 2023), which are jointly trained on vision language tasks such as image captioning and visual question answering (Alayrac et al., 2022; Li et al., 2023b; Liu al., 2023b; Ye al., 2024). Powered by the multi-modal pre-training on transformers, various open source (Liu et al., 2023b; 2024; Chen al., b024cb;; Bai al., 2023; Zhu et al., 2024; Xue et al., 2024; Mi- crosoft, 2024; Deitke et al., 2025) and proprietary (OpenAI, 2024a; Anthropic 2024; Google, 2024) LVLMs have been developed in recent years. Several studies also propose models for specific applications, such as mathematical reasoning (Zhang et al., 2025), chart under- standing (Liu et al., 2023a; Masry et al., 2023), medical images (Li et al., 2023a), and text-rich image understanding (Zhang et al., 2024b).
Synthetic images for training and evaluating visual perception. In this work, we create synthetic geometric shapes for evaluating and training geometric perception. There is prior work that uses synthetic geometric shapes for evaluating or training geometric reasoning. Geom Verse (Kazemi from a predefined set of shapes and formulas. AutoGeo (Huang et al., 2025) is a large-scale training dataset created by rule-based pipeline. G-LLaVA (Gao al., 2025) uses dataset generated from text-only LLMs to improve performance in geometric problems. There also exist datasets that use synthetic figures in other domains evaluate the visual perception of LVLMs on tasks including visual question answering (Antol et al., 2015; Zhang et al., 2016; Kuhnle & Copestake, 2017; Lu et al., 2021b), chart understanding (Kahou et al., 2018; Kafle et al., 2018), visual reasoning (Suhr et al., 2017), mathematical reasoning (Lu et al., 2021a), diagram understanding (Giledereli et al., 2024), 3D object understanding (Johnson et al., 2017; Koch et al., 2019; Li et al., 2023c), and color distinction (Hyeon-Wov et al., 2025).
al., 2024) is a synthetic evaluation dataset generated
20

---

Published as a conference paper at COLM 2025
B Model Access
This section provides details of the model access and model parameters we use in Section 4.1. For all models, we use a temperature of zero or oo-ssmmle=aaase. The model responses in this paper were collected between October 1, 2024, and March 9, 2025.
B.1 Proprietary Models
OpenAl GPT. We access GPT-4o (OpenAI, 2023; 2024a;b) models via OpenAl API, We evaluate
and apt-44-2024-08-0 with the parameter of detail: high, which make the model to receive high resolution images.*
Anthropic Claude. We access Claude 3.5 Anthropic (2024) and Claude 4 (Anthropic, 2025) Anthropic API, We evaluate -laaude3-55-sonee -------et-2220000ee 4 and
Google Gemini. We access Gemini 1.5 (Google, 2024) and Gemini 2.5 (Google, 2025) via Google C We evaluate gemini-1.5-flash--eee 5--lash-00,, Seninii1.5-ppo--0 and
B.2 Open Models
We evaluate models published on Hugging Face Model Hub.? For Intern (OpenGVLab Team, 2024), Qwen2-VL. (Wang et al., 2024a), and Phi-3.5-vision (Microsoft, 2024), we evaluate the models using code released by the authors. For other models, we evaluate using VLMEvalKit (Duan et al., 2024). Refer to Table 9 for the models we evaluate.
For Qwen2-VL, set max.pixels=1220-2200
9nttps://platform. wwtp://rmm.poeiiiiiooomooo
*nttps://platfoom.openai
http:://///coooooooggl....e com/
htths://tthh com/QwenLM/
Qwen2-VL,
http://ww....t...............ota BCookBook
iPhttps://hugiinggaas
21

---

Published as a conference paper at COLM 2025
Phi-3.5-vision
LLaVA-Next 8B
11aaa
LA 34B
llava.next.yl..m
MolMo 7B-D
Nolne-7B-D-0924
MolMo 72B
nclno-728-8924
Llama 3.2 11B
Llam=332-1BB-issioonnnrrrc
Llama 3.2 90B
Llana-3.2-98--istoon-nnss
Qwen2-VL-2B
Qwen/Qwm2-WW-228-
Qwen2-VL-7B
Qwen2-VL-72B
Dven/Gwen2-V--72B-IB
InternVI.2-4B
Been@vlab/Inteern...
InternVL.2-8B
0gendVV.ab/InteerrW22---8
VL2-26G
openoVLab/1Ineernv222266
InternVL2-40B
openovllab/Interrw.2-48
VL2-77V
Claude Sonnet 3.5
claude-3-----mme---27466622
Claude Sonnet
cl cluub-ssomeet-42222233111
Claude Opus GPT-4o-mini
Lluude-apus-4222255511
cot-4e-mLn1-2224-97-
GPT-4o Gemini 1.5 Flash
gpt-4o-2824-08-B8
gemini-1.5-flashee
Gemini 1.5 Pro
genini-1.9-pro--e
Gemini 2.5 Pro
Table 9: LVLMs we evaluate in this paper. For open models, this table shows model names in Hugging Face VLMEvalKit.
22

---

Published as a conference paper at COLM 2025
C Details of LVLM Evaluation
This section provides details of experiments in Section 4.1 and 4.3.
Prompts. Table 10 shows two types of prompts with and without chain-of-thought use to evaluate LVLMs on VisOnlyQA in Section 4.
Prompt Type
Prompt
qqueston)
w/o thain-of-thought
Your response should only include the final answer (:response type)). Do not include any reasoning explanation in your response.
(question)
w/ chain-of-thought In your response, provide a short explanation or reasoning for your answer. Then, provide the final answer (trespons..tpee..
Table 10: Prompts we use when evaluating LVLMs on VisOnlyQA. (response .type) specifies the format of final answers, such as (a, b, c, d) (True, False)
We extract the selected options from responses from LVLMs using GPT-4o. We instruct GPT-4o with the following prompt, where (response type) is final answers for each task, such as "a, b, c, d, or "True, False".
Postprocessing.
Your task is extract the final answer (selected option) from the response. Your response should only include (response type]. Question: {question} Response: (response}
We use the following prompt for Chemistry-Shapp
Your task extract the final answer from the response. Your response should only include the final answer(s) in a format "a", "a,b", "a,c,d" "a,b,c,"". For example, "(a), (b), (c), (d)" should be converted to "a,,,c,d". Question: {question]
Response: (response)
23

---

Published as a conference paper at COLM 2025
D Details of Fine-tuning
We fine-tune Intern VL2 (4B, 8B, and 26B) (OpenGVLab Team, 2024), Qwen2-VL (2B and 7B) (Wang et al., 2024a), and Phi-3.5-vision (Microsoft, 2024). We use the following parame- ters for our fine-tuning. For other parameters, we use fine-tuning code and hyperparameters provided by the authors, and we fine-tune each model for three epochs. !1
InternVL2
We use a learning rate of 4e-6 for the 4B and 8B models and 2e-6 for the 26B models, which are ten times smaller than the original settings. We find the original learning rate is too large for our dataset and does not decrease training loss effectively in some tasks.
Qwen2-VL
We use the total batch size of 128.
We use warmup_sterp 100 instead of warmup of 0.1 in the original setting because our dataset is relatively small.
ILtern
https://githin httos://githubb.cmmmoenv/ll..a wwt.ee/wwwaeiiiieeeeeeeaeeaiiiin
internv12. 0/2nd_ Qwen2-VL: https://github. http:h/wwwnoiiiiiggg////aaarr/ra
main/examples
httm///wammeee/trriffull///// wwt.ffuul//wee2l.fulllll.l yaml, Phi-3.5-vision: https://gi htt.c/o/cco///c//oo///////////
Phi-3Come http://////eeeee.eieennnnn//////
www.eenennneiiiiiii... finetuning/
24

---

Published as a conference paper at COLM 2025
E Details of Chain-of-Thought Error Analysis
Table 11 shows an example for each error label in the analysis of chain-of-thought reasoning in Section 4.3.
Eroor Label
Image
Model
Response
In the figure, CAEB is not quadrilateral. A quadrilateral must four distinct sides four vertices. However, CAEB the side AE with triangle AEC and triangle AEB, making not closed four-sided figure. Final answer: False.
Question Understanding Error
GPT-4o
8
D
C
The statement is True. While points A, D, and exist in the figure, they not connected by line segments form closed shape. Therefore, triangle ADP does nut exist the figure. Final Answer: The final answer (booxed [True]$
Perception
Gemini 1.5 Pro
Error
B
To estimate
areas triangles QNM and QMI, visually compare their bases and heights. From diagram, it appears that base triangle QNM (which NM) is roughly twice the length of the base of triangle QMI (which ML.. Assuming the heights the triangles are approximately
Reasoning Error
ILtemVL12-76B
same, ratio of their areas would be the square of the ratio of their corresponding sides. ratio
approximately 2:1, ratio the areas would (2:1)2, which 4:1. Therefore, triangle QNM approximately times larger in area than triangle QML. Final answer: (c)
Insufficient Reasoning
D
Gemini 1.5
The figure shows a triangle with three edges and three vertices, which are explicitly connected.
True
The double between hexagonal structures left chemical structure represents double bond. This is cummon way depict double bond chemical structures, where lines represent single bond and one line represents a double bond. Final True.
Redundant Reasoning
IntemVLL2-26
Table 11: Examples for each error label in chain-of-thought error analysis.
25

---

Published as a conference paper at COLM 2025
F
Additional Results
This section provides additional results of experiments in Section 4.
E1 Improvements by Chain-of-Thought
Table 12 shows improvement by chain-of-thouu tfoh-ught reasoning on VisOnlyQA-Eval-Ree.. We do not observe consistent improvement from the setting that does not use chain-of-thought reasoning and only generates the final answer.
Geometry
Chemistry
Charts
Average
Triangle
Quadri- Iateral
Diameter
Length
Angle
Shape (6)
Shape (m))
Extraction
section
Phi-35-visiom
-100
-2.0
LLaVA-Nest 8B
4.0
2.0
1.0
-7.0
-1.0
-2.0
14.00
2.0
0.0
1.8
LLaVA 34B
3.0
1.0
-2.0
5.0
-3.0
0.0
4.0
2.0
-1.0
3.0
0.4
MolMo 7B-D
0.0
1.0
3.0
8.0
3.0
-3.0
-6.0
10.0
-3.0
5,0
188
MolMo 728
-1.0
-1.0
2.
2.0
11.0
1.0
-10.0
4.0
-11.0
0.0
0.0
Owen2.VL 11-22VL--222 22
Owen2-VL-77
2.0
2.0
-5.0
-400
Qwen2-V1 728
3.0
-2.0
5.
-2.0
5.0
1.0
20.0
2.0
-5.0
3.0
2.1
IntemVLL24B 4BB
-2.0
-13.0
8.00
6.0
5.0
0.0
6.0
-2.0
-1.0
4,0
408
VL2-m
InterVV
3.0
2.0
VntemmL2-100
7.0
-1.0
1,0
1.0
11.0
3.0
18.0
8.0
-6.0
+3.0
2.9
Intern VL2-766
3.0
2.0
+1.0
-7.0
1.0
0.0
-6.0
-2.0
-5.0
-2.0
-1.4
Claude Sonnet 3.5
4.0
3.0
5.0
4.0
-2.0
-5.0
32.0
2.0
20.0
10.0
6.2
GPT-4e-mint
-20
14.0
Gemini Flash
3.0
-3.0
-1.0
1.0
4.0
7.0
6.0
Gemini 1.5 Pro
0.0
8.0
2.0
-6.
-5.0
2.0
10.0
4.0
3.0
3,0
1.8
Table 12: Improvement by Chain-of-Thought Reasoning.
26

---

Published as a conference paper at COLM 2025
F2 Improvements by Fine-tuning
Table 13 shows the performance of LVLMs fine-tuned on VisOnlyQA-Train, which corre- sponds to Figure
Geametry
3D
Triangle
Quadii- lateral
Length
Angle
Area
Size
Angle
Average
Random
50.0
50.0
20.0
20.0
20.0
33.3
20.0
Phi-3.5-vision
Fine-tuned
Original
62.0
54.0
50.0
56.0
27.0
15.0
22.0
25.0
29.0
21.0
38.0
87.0
28.0
20.0
32.3
46.0
Qwen2-VLL-28
Fine-luned
Original
69.0
50.0
50.0
41.0
31.0
28.0
23.0
56.0
20.0
100.0
38.0
20.0
23.0
54.6
33.6
VisOnlyQA- Evat-hyythetic
Qwem2-VL--B
Fine-tuned
Original
58.0
59.0
71.0
24.0
18.0
22.0
100.0
98.0
24.0
21.0
65.0
37.1
(In. In-Disiribution)
InternV1.2-44
Fine-tuned
Original
50.0
51.0
21.0
25.0
18.0
57.0
18.0
57.7
34.1
IntennVL2--88
Original
51.0
57.0
21.0
17.0
23.0
46.0
30.0
35,0
IntemmVL2-268
Original
53.0
30.0
23.0
72.0
25.0
39.3
Phi-3.5-vision
Original
17.0
Fine-tuned
50.0
20.0
21.0
21.0
32.0
Qwen2:
Original
Fine-tuned
62.0
63.0
39.0
27.0
28.0
$3.8
VisOnlyQA- Eval-Real (Out-
Qwen2-VL--B
Fine-tuned
Original
55.0
59.0
51.0
33.0
43.0
48.2
of-iisribubbon)
InternVL2--B
Pine-luned
Original
67.0
63.0
30.0
37.0
30.0
17.0
33.0
46.0
InternVL.2-8G
Original
29.0
30.0
27.0
33.2
Fine-luned
69.0
52.0
70.0
26.0
45.0
52.4
IntemnVL.2-268
Original
44.0
47.0
24.0
22.0
26.0
32.6
Fine-tuned
68.0
58.0
70.0
31.0
48.0
55.0
Table 13: Accuracy of LVLMs fine-tuned on VisOnlyQA-Train We evaluate the fine-tuned models EvallyQA-Evll-Syttttie which is generated from the same distribution the fine-tuning data, and VisOnlyQA-Evv Eval-Reat Real, which includes images from different distribu- tions. This table corresponds to Figure 6.
F3 Fine-tuning of Different Components of LVLMs
Experiments in Section 4.4 fine-tune LLMs of LVLMs, which is the default setting of the fine- tuning code by the authors of Intern Table 14, we show the performance of fine-tuning different components of VL.2-8B. These findings Section 4.5, reinforcing the conclusion that the LLM component plays a central role in enabling geometric perception in LVLMs. In addition, fine-tuning the LLM component alone yields comparable performance to fine-tuning all model parameters.
consistent with the analysis in
Fine-tuning
Length (Synthetic) Length (Real)
Area (Synthetic)
Area (Real)
w/o Fine-tuning
21.0
29.0
23.0
27.0
ViT Only
62.0
47.0
46.0
29.0
LLM Only
78.0
70.0
66.0
45.0
All Parameters
83.0
71.0
68.0
42.0
Table 14: Accuracy of AnternVL2-88 with different fine-tuning settings.
27

---

Published as a conference paper at COLM 2025
G Computational Resources
To evaluate and fine-tune open models, we use server with eight NVIDIA A100 SXM4 80GB GPUs. Almost all experiments can be done on four A100 GPUs, but the training of InternVL2-266 requires eight A100 GPUs in the configurations of the training code provided by the InterVL authors. Refer Appendix D for detailed settings of fine-tuning.
28

---

Published as a conference paper at COLM 2025
H Example Data and Model Outputs
Table 15 to 65 show examples from VisOnlyQA and model responses. The examples in these tables are uniformly randomly selected from Eval-Real and Eval-Synthetic.
All instances in our dataset and all responses from LVLMs are provided in our GitHub repository.
29

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
There is no triangle B'AC' the figure. True or False? triangle polygon with three edges and three vertices, which are explicitly connected in the figure.
False
Answer W/o
Model
Answer w/ CoT
InternVL2- 4B
The figure shows triangle with vertices labeled A, and The triangle A'AC' not present in the figure.
False
False.
InternVL2-
In the given figure, there is triangle B'AC AC' formed by points and C'. The vertices B A, and C' are explicitly connected by edges, forming closed figure. Therefore, triangle B'AC' does exist the figure.
False
8B
determine there is triangle B'AC' in the figure, we need to check there are three vertices connected by edges.
InternVL2. 26B
In the given figure, we can see that there are three vertices: B,, and C.. These vertices connected by edges, forming triangle. Therefore, the statement "There is no triangle B'AC' in the figure" is false. Final answer:
False
The figure shows a triangle labeled as B'AC', This triangle has three edges BB',, and A'C") and three vertices (B', and C'), which are all connected. Therefore, the statement "There is no triangle B'AC' in the figure" is False.
InternV1.2 76B
False
Let's analyze the figure:
There's segment from B to point AC, labeled B'. This creates two triangles within the
The image shows triangle vertices labeled
original triangle: ABB' and BB'C, However, there no triangle explicitly labeled as B'AC' this figure.
Claude 3.5
While we can see a triangle furmed by points B A, and C, it's not specifically labeled highlighted as distinct triangle the given diagram.
Folse
Sonnet
Reasoning: The question asks about the existence of triangle B'AC' in the figure. Although the points B', A. and labeled as separate entity the diagram.
form triangle, it's not explicitly drawn
Final answer: True (There no triangle explicitly labeled or identified as B'AC' the figure)
In the figure, triangle (( ('ACCC \) is formed by the vertices \ V), VA \), and VC {.. These points are explicitly connected by edges, forming closed shape with three sides. Therefon, triangle \( (B'CC' \) does exist the figure. Final
GPT-4o
True
False.
The statement "True*
There is triangle BAC with vertices B, A, and C. While points B', C' are marked on segments BA and BC respectively, there are segments drawn connecting to either B' Therefore, triangle B' exists.
Gemini 1.5
False
Pro
Final Answer: The final answer is 5 boved [
Table 15: Examples of dataset and model responses for Geometry-Trianglle (1)
30

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
A
D
E
There is triangle DBC in the figure. True or False?
triangle polygon edges vertices, which are explicitly connected in the figure.
B
C
Model
Answer w/ CoT
Answer w/uCul
The figure shows triangle with three edges and three vertices, which are explicitly connected.
InternVL2-
True
True
False.
The figure provided does not show a triangle DBC. Instead, shows triangle ABC with line segment DE intersecting it. The vertices of the triangle are B, and C, and the line segment DE is drawn inside the triangle, creating two smaller triangles, triangle DBC present the figure.
InternVL2
Folse
ADE and DEC. There
given figure, triangle DBC has three edges (DB, BC, and CD) and three vertices (D) B, and C), which are explicitly connected. Therefore, the statement "There triangle DBC in the figure" is true. True.
triangle polygon
three edges and
InterVLL 26B
True
InternV1.2 76B
The figure shows triangle labeled as DBC, with vertices D) B, and C. connected by edges. Therefore, is valid triangle. True
True
Let's analyze the figure:
1. The image shows shape with three distinet vertices labeled D, and C. These three points are connected by straight line segments.
shape formed by connecting these three points creates closed figure with three sides. This confiuuration meets the definition of triangle: polygon with three edges and three vertices.
Claude 3.5 Sunnet
True
Reasoning: The figure clearly depicts three-sided closed shape with vertices labeled B, and C, which constitutes triangle.
Final answer: True
In the figure, there is triangle labeled M Vtriangle ABC V). The line segment V(DE )) parallel to \ BCC 1) and divides V( Atriangle ABC \) into two smaller triangles, M triangle ADE and (triangle BDE However, there is triangle labeled (triangle DBC \) in the figure.
GPT-4o
False.
Final
False.
The figure shows points D, and connected line segments DB, BC, and CD. These three sugments and the points D, and form triangle. Final Answer: The final answer is 5 bosed [True]
Gemini 1.5 Pro
True
Table 16: Examples of dataset and model responses for Geometry-Triangee (2)
31

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
D
There triangle ACD the figure. True False?
A triangle is polygon with three edges and three vertices, which explicitly connected figure.
False
B
Model
Answer w/ CoT
Answer CoT
figure shows quadrilateral connecting these vertices form closed shape with four sides, which is characteristic of quadrilateral. However, the figure does not explicitly show a triangle ACD.
vertices C,
InternVL2-
True
A triangle is polygon with three edges and three vertices. In the given figure, there triangle ACD explicilly drawn.
Therefore, the answer is True.
False
InternV1..- 8B
In the given figure, a quadrilateral ABCD shown with vertices A, B, C, and D. The lines AD, BC, and AC drawn, forming triangles ACD, ABC, and ABD. Therefore, triangle ACD does exist in the figure.
False
To determine triangle ACD exists in the figure, need to check the three vertices C, and are connected by edges.
1. Vertex A connected vertex by the edge AC. Vertex connected
InternVL2 26B
vertex D by the edge CD. Vertex D is connected to vertex A by the edge DA. Since all three vertices connected by edges, triangle ACD does exist in the figure. Therefore, the statement "There is nu triangle ACD in the figure" is false. Final answer: False
False
The figure shows quadrilateral ABCD with diagonals AC and BD intersecting at point O. The vertices C, and D connected by edges, forming a triangle ACD. Therefore, the statement "There is no triangle ACD in the figure" is False.
InternVL2
False
Let's analyze the figure carefully:
1. We can see four distinct points: A, B, and D. There's point O the center, which appears to be the intersection the diagonals. We can clearly lines connecting to D, A to C, and C to D.
Given these observations:
Claude 3.5 Sunnet
The line AD forms one side of the potential triangle ACD - The line AC forms another of the potential triangle ACD. The line forms the third the
Folse
potential triangle ACD.
These three lines connect the three puints A, and D, forming closed shape with three sides and three vertices. This meets the definition triangle. Therefore, there is indeed triangle ACD the figure. Final
False
figure, points C, are connected by line segments, forming a closed shape with three edges and three vertices. This satisfies the definition of triangle. Therefore, the statement "There is no triangle ACD figure" is False
GPT-4o
False
The vertices A, C, and are connected by line segments AC and AD, forming the triangle
Gemini 1.5 Pro
False
Final Answer: False
Table 17: Examples of dataset and model responses for Geometry-Triangee (3)
32

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
There is quadrilateral ABCO in this figure. True or False?
A quadrilateral is four-sided pollygon having four edges (sides) and four coreers (vertices).
True
B
C
Answer w/o
Model
Answer CoT
The figure shows circle with points A, B, C, and on the circumferrence. The line segments AB, BC, CA, and AO form quadrilateral.
InternV1.2 2-
False
True
False.
Reasoning: The figure shows a quadrilateral ABCO, but it is nut standard quadrilateral. A standard quadrilateral has four sides and four vertices. However, in this figure, point O is the center the circle, and the lines AB, BC, and CA are chords of the circle. The line segment AC diameter of the circle, and point midpoint of AC. Therefore, the figure does nut furm standard quadrilateral but rather eyelic quadrilateral (a quadrilateral inscribed circle) with additional properties due to the circle's center.
InternVL2- 8B
False
A quadrilateral defined four-sided polygon. In the given figure, quadrilateral ABCO is formed by the four vertices A, B, C, and O, and the four edges connecting vertices.
InternVL2- 26B
False
Therefore, the statement "A quadrilateral is four-sided polygon having four four edges (sides) and four comers (vertices)" is true.
True.
The figure shows circle with four points labeled A, and These points connected by lines forming four-sided polygon. Therefore, the figure represents a guadrilateral.
InternVL2- 76B
False
Final answer: True
Let's analyze the figure carefully:
1. The image shows a circle with three distinct points labeled A, B, and C on its circumferencee.
2. There's 's also point O, which appears to be the center of the circle.
each other to point 4. While there are multiple triangles formed (such ABC, ABO, BCO, and ACO), there isn't a four-sided shape visible.
Lines are drawn connecting points A,
Claude 3.5
False
Sonnet
5. quadrilateral would require distinct vertices connected
sides, which
we don't see in this figure.
Explanation: The figure shows
with inscribed triangles, but not contain four-sided polygon. The shape ABCO is not quadrilateral because is connected other points, forming triangles instead four-sided shape.
Final answer: False
In the figure, (AA)) (BV), (CV), and (OV) points. However, the lines triangle ((BC))) ) and three additional line segments ((OO)), (BO)), and (COY)) that meet at point (OO)). This does form closed four-sided polygon, as there are not four distinct sides forming quadrilateral.
GPT-4o
False
Final answer: False.
The figure shows points A, B, C, and O connected by line segments to form closed, four-sided shape. These points represent vertices, and the lines connecting them represent the edges. Thus, the figure represents a quadrilateral. Answer: final answer 5 bexed [True]S
Gemini 1.5 Pro
True
Table 18: Examples of dataset and model responses for Geometry-Quuddrilatterll (1)
33

---

Published as a conference paper at COLM 2025
Image
Questton
Gold Anawer
There quadrillateral COEF in this Figure, True Folse?
A quadrilateral a four-sided polygon having four edges (sides) and four corners (vertices).
Thue
G
E
D
Model
Answer w/ CoT
Answer w/o CoT
The figure shows circle with points E, C, O, and F on its dircmneeeecee. The line segment CO is chard the circle and the line segment OG radius of the circle. The line segment OE radius circle,
radius circle, and line segment OG midius circle. The line segment is radius of the circle, and the line segment OF is radius of the circle. The line segment OD is radius of the circle, and the line segmment OG is radius the circle. The line segment OE radius of the circle, and the segment OF radius
lime segment
the circle The segment
circle. The line segnent OD is radius the circle, and segment OG is radius of the circle. The line segment OE is radius of the circle, and the line segment Of is rudius of the circle. The line segment OD a radius of the circle, and the line OG is radius the circle. The line segment OE is radius of the circle, and the line
segment
radios
circle.
segment
radius
circle,
segment is radius the circle. The line segment OE is midius of the circle, and the line
IntemVL1--1B
segment OF is radius of the circle. The line segment OD a radius of the circle, and the line OG is radius the circle. The line segment OE is radius of circle, and the line radius
x
False
x
segment
circle.
segment
line
segment OG is a radius of the circle. The line segment OE is radius of the circle, and the line segment Of is radius of the circle. The line segment 0D a radius of the circle, and the line segment OG radius the circle. The line segment OE is radius of the circle, and the line segment
radios of circle.
segment
radius
circle,
line
segment OG is radius the circle. The line segment OE is rudius of the circle, and the line
segment OF is radius of the circle. The line segment OD a radius of the circle, and the line
segment OG is radius the circle. The line segment OE is radius of the circle, and the line segment OF radius of circle.
segment
circle,
line
segment OG is a radius the circle. The line segment OE is radius of the circle, and the line segment OF is radius of the circle. The line segment OD a radius of the circle, and the line.
False
The figure provided does not form quadrilateral CDER quadrilateral is defined four-sided form closed lour-sided shape. The points C,
IntemVL2-888
this figur, COFF are nut cormected in a way that forms quadrilateral, Insted, the points are arranged a way that forms a circle with line segment intersecting it. Therefore, the figure does not meet the ertleria for a quadrilateral.
polyyonn,
False
A quadrilateral is defined as a four sided polygon. The figure shows quadrilateral COEF with four sides and four vertices, Therefore, statement "There quadrilateral COEF in this figure" is true. True.
V12-266
False
The figure shows quadrilatenal COFE, which is four-sided polygoes. The vertices are C, O,F, and F, and the sides are CO, OE, EE, and FC, Therefore, the statement Inse.
IntemVV12-76l
True
True
Let's analyze the figure carefully:
The image shows circle with several peints and line segments marked. can see points labeled O
D, 3. However, these points do not form a quadrilteral COIP: C and O are connected by a radius OO is the center of the circle- E and F are points on the circumference - There are no visible line segments connecting these points to form clased siddd shape figure appears show poasibly a chard (CD), but not quadrilateral COEF.
Claude 3.5 Somnet
with various
like diameter (EF), radii, and
False
5. For quadrilateral CDEF to exit, we would need to see four line segments connecting these a clesed shape. which Explanation: While points C, quadrllterrl as they are not connected to create a four-sided closed shape. Finall answer: False
in this dingram.
are present in figure, they do not form a
In the figure, the puints C, O, E, and F are connected, but they do not form quadrilateral. A quadrilateral requires four distinct sides, and in this the lines CO and OF not connected directly to foem clased shape four sides. Instead, COFF forms triangle point O inside the circle. Therefore, the statement is False.
GPT-4o
additional
False
The given figure has points O, E, and F. These four points can be connected to form a fourssided polygon. Therefore, COEF is quadrilateral. Final Answer: The final answer is Sbbosed [TrueJs
Gemini 1.5 Pro
True
Table 19: Examples of dataset and model responses for Geometry-Quadrilatte (2)
34

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
A
P
There quadrilateral CBPQ in this figure. True False?
A quadrilateral is four-sided polygon having four edges (sides) and four comers (vertices).
True
C
B
Model
Answer CoT
Answer W/o
To determine there is quadrilateral CBPQ in the figure, we need to identify if there vertices (comners) and four edges (sides) that form closed shape. In the given figure, we can see the following points: Point A Point B- Point C-
IL2 4B
True
However, there no point Q in the figure. Therefore, there is no quadrilateral CBPQ in this figure.
Final answer: True
False
The figure does not contain quadrilateral CBPQ. A quadrilateral is defined figure, the points C, P. and do not form closed four-sided shape. Instead, the points form a triangle (ABC) and segment (PQ) that intersects circle at point P. Therefore, there no quadrilateral CBPQ in this figure.
InternVI... 8B
four-sided polygon, but
False
A quadrilateral four-sided polygon, and in the given figure, the shape CBPQ has only three sides and three vertices. Therefore, quadrilateral.
InternVL2- 26B
classified
True
True.
The figure shows circle with points C, B. P. and Q, However, the quadrilateral the line segment does not the figure.
InternVL2- 76B
CBPQ formed Therefore, the statement "There is quadrilateral CBPQ in this figure" is true.
Fabse
Let's analyze the figure carefully: 1. We
circle with center C. There's triangle with vertices and what appears tu be P. Puint Q is nut explicitly labeled, but there is point where the line intersects Now, let's consider if we can form quadrilateral CBPQ: C the center of the circle top
Claude 3.5 Sunnet
B is clerrly marked P appears to marked labeled, could be considered as the point where line Al intersects the circle
triangle- Q, while not
X
False
If we connect these four points (C, B, P, and the unlabeled intersection point as Q), indeed form four-sided figure with four vertices.
Therefore, it is possible to identify quadrilateral CBPQ in this figure, even though Q not explicitly labeled.
Final answer: False
In the figure, points B, and Q all connected four sides. A quadrilateral requires four connected edges forming closed loop. Here, the points do form such shape. Therefore, the statement is "True"
closed shape with
GPT-4o
True
The statement "False"*
The points C, B, P, and Q are connected by line segments CE, BP, PQ, and QC, forming closed four-sided figure. This meets the definition of quadrilateral. doesn't matter that some of the lines forming it happen to also be part of other shapes (triangle circle). Final Answer: The final answer is $ boxed (
Gemini 1.5 Pro
False
Table 20: Examples of dataset and model responses for Geometry-Quadrilaterall (3)
35

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Line AP X times longer than CB. Which the following
options is a reasonable estimate of X? You only need to estimate from the visual information and do need do any mathematical reasoning. (a) 0.25 (b) 2 (c) 4 (d) 0.5 (e)
B
Answer C/T
Model
Answer w/ CoT
To estimate the value of X, need to visually compare the lengths of line AP and CB.
1. "Observation": -Line AP the hypotemuse of the right triangle AOB. LOB.eLinne CB the radius of the circle and is perpendicular to line AB point
2. "Comparison"
Since AB is radius of the circle, equal to the radius of the circle, which is a fixed length. AP is the hypotenuse of the right triangle AOB, which longer than the radius
InternVL2- 4B
d
3. "Estimation": Given that AB a fixed length and AP is significantly longer, we can estimate that AP is upproximately 2 times longer than AB. 4. "CCnclusion"" t Based
the visual information, reasonable estimate for the value of (the ratio of AP to CB) is
Therefore, the final answer is 2b)
estimate value \(X \), we can visually compare the lengths of \( (P \)) and V( CB 1) in the given diagram.
1. *Visual Comparison AP \) appears to be about times the length of CB
This is because V. AP extends beyond V CB by significant proportion, making it appear apppear much longer.
InternVL2- 8B
b
"Reusoning" Given the visual estimation, M{ AP \) appruximately times the length of \( (CB \J. This is rugh estimate hased on the relative lengths observed in the diagram.
Therefore, the most reasonable estimate of \(X is:
"Final Answer: (c) 4"
To estimate the value of (XX) V), we need to compare the lengths of ((A)) \) and V( \) visually.
1. Observe the lengths of \( (P \) and V (C \) in the diagram. 2. Estimate the ratio of V AP \) to (CB by visually comparing their lengths.
InternVL2- 26B
(b)
From the visual information, appears that V( AP ()) approximately four times the length of ) CB (.. Therefore, a reasonable estimate for {(X is
The correct
(c)
estimate length CB is chord of the circle. AP extends from the circle's edge at A to point P, which is significantly farther away than From the visual information, it appears that roughly four times the length of CB.
relative to CB, we can visually compare segments.
InternVI.22 76B
(b)
Therefore, reasonable estimate X is Final answer: (c) 4
Looking at the imagu, can circle with its center O and triangle APB where P is point on the circle. Line CB chord of the circle, passing through its diameter. Visually comparing the lengths of AP and CB:
1. AP extends frum point on the circle to a puint well outside it 2. CB is a chord within circle, noticeably shorter than
appears to significantly longer than CB- roughly 3 to 4 times longer.
diameter.
Claude 3.5 Sonnet
Among the given options, the one that best matches this visual estimate is (c) 4. Option (a) 0.25 and (cl) 0.5 would imply AP shorter than CB, which clearly the case. Option (e) would mean they' re equal, which is also not true. Option (b)
is closer but still seems underestimate the difference.
Therefore, based on visual estimation alone, the most reasonable answer is: (c)
From the visual information, line segment \( (AP))) () appears to be approximately twice the length segment \CB (). Therefore, reasonable estimate X(X is Final answer: (b) 2
GPT-4o
Visually, AP appears roughly twice as long the diameter of the circle. CB is chord that close in length to the diameter (although clearly shorter). Therefore, AP appears be roughly twice the length of CB. Final Answer: (b)
Gemini 1.5 Pro
x
b
x
Table 21: Examples of dataset and model responses for Geome -ryeteryy--yt (1)
36

---

Published as a conference paper at COLM 2025
Guld Answer
Image
Question
A
E
Line CE is X times longer than AD. Which of the following estimate
options is a reasonable estimate X? You only from the visual information and do not need to do any mathematical ressonngg (a) 0.5 (b) 1 (c) 2 (d) 4 (c) 0.25
B
D
Model
Answer w/ CoT
Answer CoT
To estimate the value of X, need to visually compare the lengths of CE and AD.
1. "Observation": CE is the chord of the circle. - AD is the radius of the circle. diagram,
much longer than The radius (AD) appears to be about half the length of the diameter, which is twice the length of the radius. CE appears to about twice the length of the radius. Since CE approximately twice the length of AD, estimate that X around
"Comparison"":
InternVL2- 4B
3. "Estimation"
Therefore, the reasonable estimate of 2.
"Final Answer*:: (c).
To estimate the value of VX \) based on the visual information, we need to compare
the lengths of M CE \) and ((AD ()
1. *Visual Comparison V CE appears be about twice the length of (D))
InternVL2- 8B
This is a rough estimate based on the visual proportions in the image.
"Reasoning" If ( CE V) twice as long AD then M( would be Thus, the must ressonable estimate of {(X 1) is
"Final Answer"": (c)
To estimate the value (XX }),, we can visually compare the lengths of M(CE ()) and M AD ()) the given circle diagram.
InternVL2- 26B
From the diagram, it appears that \ CE () approximatell twice the length of V
AD ).. This suggests that ( () around 2.
Therefore, the reasonable estimate for {(X is (c)
So, the final answer is (c) 2.
To estimate the length of CE relative to AD, observe the visual proportions in the circle. Line CE appears be roughly twice the length of AD.
InternVL2- 76B
Given the options: (a) 0.5 (b) (c) (d) (e) 0.25
x
(c)
The most reasonable estimate is (c) 2.
Final answer: (c)
Based the visual information provided in the image, I estimate the relative lengths of lines CE and AD.
Line CE appears to diameter very close diameter of the circle, passes through or very near the center: line AD, on the other hand, is a chard that does not pass through the center and is visibly shorter than CE.
Claude 3.5 Sonnet
Comparing these two lines visually, CE looks to be about twice the length of AD. It's clearly longer, but not dramatically so- certainly not four times as long At the same time, CE is definitely where CE would be equal to or shorter than AD.
than just little bit longer than AD, ruling out the options
Given these observations, the most reasonable estimate for (where CE X times longer than AD) would be 2. Therefore, my answer (c)
From the image, line \ \ appears to be about twice as long as line \ AD (). Visually estimating, Y CE \) significantly longer than \ {), excessively
GPT-4o
Therefore, the reasonable estimate for \(X is The final answer is (c)
Gemini 1.5
CE looks be roughly the same length AD, Therefore, is approximately 1. Final Answer: (b)
(a)
Pro
Table 22: Examples of dataset and model responses for Geometry (2)
37

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
A
D
E
Line BC X times longer than AD. Which of the following
options is a reasonable estimate of X? You only need to estimate from the visual information mathematical reasoning. (a) 1 (b) 0.25 (c) 2 (d) 4 (e) 0.5
do need to do any
B
C
Model
Answer w/ CoT
Answer
CoT
To estimate value X, need visually compare the lengths of BC and AD.
1. "Observation": - AD appears to be about half the length of BC, This is rough visual estimate based
proportions of the triangle.
InternV1.2 2-
2. "Reasoning**. If AD is approximately half the length of BC, then BC is approximately twice the length of AD. Therefore, X, which represents the ratio of BC to AD, approximately 2.
4B
3. "Conclusion"" The reasonable estimate X from the visual information is "Final Answer" (c)
estimate the value VX based on the visual information, we need to compare
the lengths of \ BC \)) and V (AD)) ().
1. "Visual Comparison": V( BC 1) appears to be about twice the length of M(AD) ().
InternVL2-
This is rough estimate based on the visual proportions in the image.
2. "Reasoning"" If ( BC \ is twice as long as ((( (AD) (), then X( would be 2.
Therefore, the reasonable estimate of XX 1s
"Final Answer":: (c)
To estimate the value (X) V, need to compare the lengths of \( BC and )( AD () visually: From the diagram, it appears that ( BC is significantly longer than V (AD 1)). A reasonable estimate for V(X \ would be value that reflects this visual disparity. Option (a) 1 suggests that BC )) equal in length M {), which
visually accurate. Option (b) 0.25 suggests that \ BC is one-fourth the length of M AD J, which also does not match the visual information. Option (c) 2 suggests that M BC )) twice the length of V AD () which is closer to the visual estimate but still not quite accurate. Option (d) 4 suggests that \( (BC)) \) is four times the length of / reasonable estimate. Option
InternV1.. 26B
AD which visually seems
suggests that \BCC !) is half the length of M AD (). which is not visually accurate.
Given the visual information,
most reasonable estimate X(X \)
Final answer: (d) 4
To estimate the length of BC relative to AD, observe the visual proportions in the triangle. The line segment AD appears to he approximately half the length of BC. Given the options:
This would mean AD and BC are equal, which is not visually accurate. (b) 0.25 This would mean RC is four times langer than AD, which seems too long. (c) (-) This would mean BC twice as long as AD, which looks about right. (d) 4- This would mean BC is four times longer than AD, which seems too long. (e) 0.5 This would meun BC is hulf length of AD, which is incurrect.
InternVI.2 76B
(d)
The most reasonable estimate from the visual information is (c)
Final answer: (c)
Looking at the image, we can see a triangle ABC with smaller triangle ADE inside it. The line BC forms the base of the larger triangle, while one of the sides the smaller triangle.
Visually comparing the lengths of BC and AD:
1. BC appears be significantly longer than AD. 2. BC to span almost entire width of the image, while AD is much shorter line segment. 3. AD looks to length BC.
Claude 3.5 Sonnet
1/3 Given these observations, we can estimate that BC approximately 3 to 4 times longer than AD. This means X, which represents how many times longer compared AD, should be around or 4.
be roughly
Among the given options, the closest value to this visual estimate 4. Therefore, the final answer (d)
From visual information, line BC appears be significantly longer than line It looks like BC is about four times the length of AD. Therefore, reasonable estimate is.4.
GPT-4o
d
The final answer is (d)
Gemini 1.5
appears to be roughly twice length AD. Final Answer: (c)
Pro
Table 23: Examples of dataset and model responses for Geome try-Length (3)
38

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
A
Which of the following options is reasonable estimate of the angle CAB the figure? You only need to estimate from the visual information and do not need to du any mathematical reasoning. (a) degrees (b) 45 degrees (c) 135 degrees (d) 10 degrees (e) 180 degrees
B
D
C
Model
Answer w/ CoT
Answer CoT
To estimate the angle CAB, we can visually assess the triangle and the right angle at point D. The angle at point formed by line segments
AB, Since
triangle right triangle with right angle at D, the sum of the angles in triangle is 180 degrees. The right angle at D) is 90 degrees, so the remaining two angles (CAB and BDC) must add up to 90 degrees.
InternVI 2- 4B
Given that angle BDC is right angle, angle CAB must be less than 90 degrees. Among the given options, the only reasonable estimate for angle CAB is degrees. Final answer: (b) 45 degrees
The figure shows right triangle with right angle at point D. The angle CAB is the angle at the top of the triangle, which is opposite the side BC. Given that the right angle at D, and triangle right triangle, angle CAB be one the other two angles in the triangle.
In right triangle,
sum
two non-right angles 90 degrees. Therefore, the
InternVL2- 8B
angle CAB must be less than 90 degrees.
Among the given options: (a) 90 degrees is incorrect because it would make the
b
triangle straight line. (b) 45 degrees remsonable estimate for angle right
triangle. (c) 135 degrees incorrect because it is greater than 90 degrees. (d) 10 degrres is too small and nut reasonalee estimate for an angle 180 degrees is incorrect because it would make the triangle straight line. Therefore, the most reasonable estimate for the angle CAB 45 degrees. Final answer: (b) 45 degrees
right triangle. (e)
To estimate the angle CAB, we the altitude of the triangle ABC, it divides the triangle into two smaller right triangles, $) jensuremati] (triangle) Baaggle/SAAAA $ABD and 5 (eeesuremaahh triangle JACCD, right triangle, the angle opposite the altitude is right angle (90 degrees). Therefore, $ ensuremath (angle))B)BDD JSAD and 5 jensureaat (aangle]SCC JSCAD are both 90 degrees. Since (ennurremath) (angle]SBAC is the angle between the two legs of the right triangle, it must be the remaining angle in the triangle, which is 90 degrees. Therefore, the angle CAB 90 degrees. The final answer is (a) 90 degrees.
concept of similar triangles. Since AD
InternV1 2- 26B
45 degrees
To estimate the angle CAB, observe the triangle ABC and the line AD. The line AD appears bisect the angle BAC into two equal parts. Given that triangle ABC is isosceles (AB AC), the angle A (BAC) is
around 70-80 degrees. Since AD bisects this angle, each of the two resulting angles (BAD and CAD) would be approximately half the original angle.
InternVL2- 76B
(b) 45 degrees
Therefure, the angle CAB (which is one of these bisected angles) would be around 45 degrees.
Final answer: (b) 45 degrees
Based the visual information provided the image, can estimate the angle CAB as follows:
The triangle shown appears be roughly isusceles, with the sides CA and looking approximately equal in length. The angle at the top vertex (angle CAB) seems to less than a right angle (90 degrees), but not by a large amount. Given this visual assessment, the angle CAB looks to be close 45 degrees. It's clearly much larger than 10 degrees and significantly smaller than 90 degrees or anything above that. Among the given options, 45 degrees (option b) is the most reasonable estimate that matchus what we can see the figure. Final answer: (b)
Claude 3.5 Sonnet
The angle Vangle CAB \)) appears to acute angle, it is less than right angle (90 degrees). It is also significantly smaller than 45 degrees. Therefore, the most reasonable estimate from the given options 10 degrees.
GPT-4o
b
Final answer: (d) 10 degrees.
Angle CAB looks less than 90 degrees, and closer to half of 90 degrees. 45 degrees is reasonable estimate. Final Answer: (b)
Gemini 1.5 Pro
b
Table 24: Examples of dataset and model responses for Geometry-Angle (1)
39

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Which following options reasonable estimate of the angle AOD in the figure? You need to estimate from the visual information and do not need to do any mathematical reasoning. 180 degrees (b) 10 degrees (c) 135 degrees degrees (e) 45 degnees
E
d
Answer Coo
Model
Answer w/ CoT
To estimate the angle AOD, need to consider the shape quadrilateral ABCD, The figure shows that ABCD is square, as all sides are equal and all angles are right angles degrees). In squane, the diagonals are equal in length and bisect each other at right angles. Therefore, each diagonal divides the square into two congruent right-angled triangles.
InternVL2-
d
Since the diagonals of square bisect the angles of the square, each angle at the vertices the square divided into equal parts. Thus, the angle AOD half of the right angle at vertex A, which is 90 degrees. angle
Therefore, reasonable estimate Final answer: (e) 45 degrees
degrees.
To estimate the angle AOD the figure,
observe the following: 1. The figure shows square ABCD with diagonals AC and BD intersecting at point square, the diagonals are equal in length angles (90 degrees).
other at right
InternV1.2 2- 8B
Since diagonals square intersect at right angles, the angle AOD 90 degrees. Therefore, the reasonable estimate of the angle AOD is: (d) 90 degrevs
To estimate the angle AOD, we can observe that the figure is rhombus, which means sides equal opposite angles equal. Since diagunals of rhombus bisect each other at right angles (90 degrees), the angle AOD is half of the angle formed by the disgunals at point Given that the diagonals bisect each other right angles, the angle AOD is half 9p degrees, which 45 degrees. Therefore, answer
InternV1.2 26B
degrees
45 degrees.
estimate angle AOD, observe the figure note position points A, and D. The diagonals of the parallelogram intersect point O, and in parallelogram, the diagonals bisect each other. Therefore, angle AOD is one the angles formed by the intersection of the dingonals. From the visual information, angle AOD appears to be right angle because the diagonals
InternVI.2- 76B
(d) 90 degrees
90 degrees a parallelogram. Thus, the reasonable estimate of angle AOD is 90 degrees. answer: (d) degrees
ooking at the figure, we can see that AOD one of the angles formed by the quadrilateral (possibly square rectangle), The
diagonals what appears diagonals intersect at point O creating four angles. From visual inspection, the angle AOD looks angle. A right angle is 90 degrees, and half of that would be 45 degrees. The angle certainly doesn't look like 180 degrees (which would be a straight line), 10 degrees (which would be very larger than a right angle). It also doeen't appear to a full 90 degrees, as it's visibly the quadrilateral.
approximately half right
Claude 3.5 Sonnet
angle), 135 degrees (which would be
d
smaller than the right angles the Therefore, based on visual estimation alone, the most reasonable estimate for angle AOD degrves.
Final answer: (e) 45 degrees
From visual information the figure, angle \ \aaaglee AOD \) appears acute angle, significantly less than 90 degrees. It looks closer to small angle, pussibly anound 45 degrees.
GPT-4o
Therefore, the reasonable estimate for angle (( langle AOD (e) degrees
is:
The angle AOD appears to be close to angle. A right angle measures 90 degrees.
Gemini 1.5 Pro
Final Answer: (d)
Table 25: Examples of dataset and model responses for Geometry-Angle (2)
40

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
A
D
Which following options reasonable estimate of the angle BMN in the figure? You only need to estimate from the visual information and do need do any mathematical reasoning. (a) 90 degrees (b) 135 degrees (c) 180 degrees (d) 45
N
B
degrees (c) 10 degrees
M
C
Answer w/o
Model
Answer CoT
To estimate the angle BMN in the figure, we need to consider the shape and the given angles. figure shows quadrilateral with diagonal, and the angles and C are right angles (90 degrees).
Given
angles at
right angles, sum
angles
maarilaterrll is 360 degrees. Since the other two angles (at vertices B and D) not directly given, we can infer that the angle BMN is likely be part of the remaining angles that up 360 degrees.
InternVL2 4B
Considering the options provided: (a) 90 degrees - This is the angle at vertex A, not BMN. (b) degrees- This a reasonable estimate the angle EMN, large angle but not as large 180 degrees. (c) 180 degrees This would imply degrees This is too small the angle BMN. (e) 10 degrees This is too small for the angle BMN.
straight line, which not the case here.
Therefore, the must reasonable estimate the angle BMN is 135 degrees. Final answer: 135 degrees.
The angle BMN appears intersection of lines BM and MN. This typically represents 90-degree angle in geometric diagrams, Therefore, the most reasonable estimate of the angle BMN 90 degrees.
right angle, as indicated by the small square the
InternVL2- 8B
b
Final answer: (a) 90 degrees
To estimate the angle BMN, we can observe the given geumetric figure. The figure shows right-angled triangle (ABD) interior angles are 90 degrees. Since BMN is part of the rectangle, it is reasonable to estimate that angle BMN is also 90 degrees. degrees
InternVL2 26B
rectangle (ABCD). rectangle,
(b) 135 degrees
Final
estimate angle BMN, observe the figure M, and N. The angle BMN appears be right angle because line BM perpendicular line MN.
relative positions points
InternVL2- 76B
(b) 135 degrees
Therefore, the reasonable estimate of the angle BMN 90 degrees. Final answer: (x) degrees
Looking at the figure, can see that angle BMN is formed by two lines that meet at significantly larger than right angle degress),
point M. This angle appears but it's not straight line (180 degrees).
The angle looks to be roughly halfway between right angle and a straight line. This would put it close 135 degrees, which
of the options provided.
Claude 3.5 Sonnet
It's cleatly much larger than 45 degrees or 10 degrees, and noticeably larger than 90 degrees. However, doesn't extend all the way form straight line, degrees would be too large.
d
visual estimation, most reasonable estimate among given options would be 135 degrees.
Final answer: (b)
The angle VI Jangle BMN \) appears to be an acute angle, as it is smaller than right angle. Visually, seems closer to angle rather than large one. Among the options, 10 degrees the most reasonable estimate for an acute angle.
GPT-4o
d
Final answer: (e) degrees
Angle BMN appears to be slightly larger than a right angle (90 degrees). 135 degrees seems like reasonable estimate. Final Answer: (b)
Gemini 1.5 Pro
Table 26: Examples of dataset and model responses for Geometry-Angle (3)
41

---

Published as a conference paper at COLM 2025
Image
Question
Gold Answer
ADB is times larger in area than ABC: Which of the following options is reasonable estimate? You only need to estimate from the visual information do not any matheratical reasoning (s) (b) 0.25 (c) (e)
D
8
Model
Answer w/ CoT
Answer CoT
To estimate the area ralio of triangle ADB to triangle ADC, we can visually com their sizes. Triangle ADB appears roughly twice large triangle ABC in terms of height base length.
I12em
Given the options: (a) 0.5 (b) 0.25 (c) 1 (d) 4 (
The most reasonable estimate from the visual information (c)
Final answer: (e)
The diagram shows triangle V "rriangle ABC \) with smaller triangle V Vtriangle ADE Viiiide it. The base X DE is part the M (DC)) of Y \tnangle ABC to V DC () is the same for both triangles, as they share the same vertex VAA V..
height from V.A.
From the visual information, it apapeerrs that X (DE 1) approsimately half the length of \/ DC V). Since the height the same for both triangles, the area of / (triangle ADE roughly half the
IntemV11 12-88
area ('triangle ABC {).
Therefore, the area of (( Vtriangle ADB \) approximately half the area of \ ('triamgle ABC AA.. The final answer (a) O.S.
To estimate the area ratio of triangle (ADB)) to triangle M(BCA), we can visuallyy compare the base lengths and the heights of the triangles.
"Bass Comparison" The
(DDV) appears roughly twice length of
(BCC). suggests that the area (ADDB)) will be larper
larger base. "Height Comparison": The height from (AV)) to (DD) appears to be roughly the same as the height from
MA to (RL)) This means the height ratio
I.
VL2-26
Given that the ratio 2 and the height ratlo the rallo of (ADB)) ((BCC) estimateed using the formula for the area of triangle (V /eext (1rac())-(2) (1) yiimes
text ( [base] Vimes Vest (height) W
M Vtext Ares ratio) Veet( (frac() text of) ADB (textt of } ABC) \right)
Ueft( (frac(2)(1) (rightt2 -4 AJ
Thus, the
M(ADB) times larger than the VABCA).
correct answer (d)
To estimate catio bases and heights. 1. *Base Comparison**, The base of triangle ADB (from to B) appears to be roughly twice as long the base of triangle ABC (from to C). "HHeggh Compatison" The height trangle ADB (from the line containing the line containing B and C).
of triangle to triangle ABC, we can visually compare their
appears to roughly the same as the height triangle
IntemVL2--6B
The area of triangle given by the formula V/ test (Area) = (frac(1) (2) times Vtext (base) for both triangles, the ratio the areas will
ABC (from
(d)
times text height) Since the height is be the same Given that the base of ADB approsimately twice the base of ABC, the area ADB will be approsimnately twice the area of ABC.
ralio their
Therefore, the reasonable estimate (e)
Looking the image, we can see triangle ABC with point D the side AC. This creates two triangles the larger original triangle ABC, and smaller triangle ADB within up roughly half or alightly less than
Visually eatimating the areas, triangle ADB ADB appeats of the lotal area of triangle ABC. The question stales that ADB is X times langer in area than ABC, which means we're looking for the inverse of this visual estimate. Given ADB looks about half size ABC, would be the reciprocal approximateely
Claude 3.5 Sonnet
which
Arong the given options, 2 (option e) seems to be the most rossonable estimate based on the visual information provided. Final
(e) 2
From the visual information, Iriangle V Vtriangle ADB appears to be larger than triangle V
langer than M (1, the height from (AA \) to DB () is the same as the height to / (EBC)) (). This suggests that Y (triangle ADB ) significantly
ttrangle ABC V...
GPT-4o
larger than (( (triangle ABC') (). A ressonable estimate is that V (triangle ADB about twice the size of V triangle ABC
Final answer: 2
ADB appears noughly
large ABC, because ADB encompasses
area
Gemini 1.5 Pro
within trisngle AFC, which itself noughly same area as triangle ABC Therefore, ADBB is apprusiznately 2 times langer than ABC. Final Answer: (e)
Table 27: Examples of dataset and model responses for Geometry-Area (1)
42

---

Published as a conference paper at COLM 2025
Image
Question
Gold Answer
D
DSO is times larger area DBC. Which the followting options ressenable estimate? You only need to estimate from the visual information and do not need do any mathematical reaseningg, (a) 0:25 (b) te) (d) 2 0.5
B
Model
Answer w CoT
Answer CoT
To estimate the value of M( 1), we need to visually compare the areas of triangles \[ ((DAO) and
M DBC V..
"OOservatiooo" Triangle Af (DAO appears to noughly
large as triangle (( (DBC
in terms of height and base. 2. "Esttiattoon" I we assume that the height and base of \( DAO !) are approcimately twice those of (DBC') (J then the area of (( DAO \) would be roughly four times that ot \( DBC V)
IntemV11244 2-48
the options: 0.25 (b)
(d) (,)
The most ressonable estimate for XXX \) 4, as aligns with the visual obvervation that V (DAO () is about twice large as V. (DBC ().
"FFial Answer":: (b) 1
To estimate the ratio of the areas of triangles DAO and DBC, we can visually compare their sizes. 1. "Visual Comparison" Trinngle DAO appears signtfcantly larger than traangle DBC. beight triangle DAO from point from point D line AB.
AB much greater than the height of triangle DBC
2. "Reasoning"" area of triangle given by the formula V Vext Area) (fme{1))22) for both heights.
InlemV12--88
times (Jeut(aase [base] \limes (heitheighh) (). Since the base (line AB) is trangles, ratio of their areas directly propoctional
ratio 3. "Fatimation"". From the visual information, triangl DAD's height appper approsimately 4 times the height of triangle DBC. Therefore, the area of triangle DAO is approsimately 4 times the area of Iranglc DBC. Thus, most reasonable estimate is (b)
estimate area two triangles.
triangle DAO triangle DBC, we can visualll compare the arsas the
1. Obsetve the triangles DAO and DBC. 2, Notice that triangle DAD is larger than triangle DBC. 3. Estimate the ratio the areas by comparing the visual sizes the triangles.
IntemVL12-268
From visual information, it appears that triangle DAO approximalely
larger than
triangle DBC.
Therefore, the answer is (b)
To estimate the area ratio of trisngle DAO to triangle DBC, we can visually compare their bases and heights.
"Base Comparison": : The base trinngle DAO (DO) appears to roughly half the length the
base of triangle DBC (DB), "Height Comparison" The height of triangle DAO (AC) appears he roughlly the same as the height of triangle DBC (BO).
IntemVL1--68
Given that the area of triangle is calculated as V/ (frac (1) (2) !times (text(bass (base) (times
(d)
btest [height) V),
base
DBC, and the heights the same,
would
DBC.
Thus, the area of DAO is approximately 0.5 times the area of DBC. Final answer: (e) 0.5
Looking at the image, we can quadrilateral ABCD with a diagonal AC that divides it into two trangles. DAO and DBC.
Visually estimatiin Triangle DAO appears be significantly larger than triangle DBC, It covers more than half of the entire quadrilateral, while DBC covers less than half.
areas:
Claude 3.5 Sonnet
The question states that DAO times larger estimate, we can reasenably conclude that than DBC. Goven this information and the DBC. Among the given options, the value that best represents this relationshsp 2s Therefore, reasoning leads to the answer: (d) 2
From the image, triangle's 3f Vtriangle DAO \) and triangle V/ Urrinngle DBC \)) share the same height from point L 1) line MCC') 5). The base AO of Vtriangle DAO \) appears to shorter than the base BBC () of Vrringll DBC Visually, V. (triangle DAO soems reusonable estimate for the ratio of their areas is 0.5.
GPT-40
size of \ triangle DBC VJ. Therefore,
The final answer (e) 0.5.
The triangle DAO appears to be roughly half the size of triangle DBC. Therefore, DBC roughly beice large DAO, meaning that DAO roughly 0.5 times larger than DBC. Answer: (e)
Gemini 1.5 Pro
Table 28: Examples of dataset and model responses for Geometry-Area (2)
43

---

Published as a conference paper at COLM 2025
Image
Question
Gold Anawer
A
BAM times langer in than BMN, Which following options reasoraable estimate? You only need to estimate from the visual information and do not need to do any matherattica reasoning, [x] 0.25 (b) 4 (c) (d) 0.5 (e)
M
B
N
C
Model
Answer w/ CoT
Answer C/T
estimate value X(X V). we to visually compare areas triangles \( (BAM \) and ( BMN VJ.
1. "Observation" Triangle VI BAM \ visually larger than triangle 1 BMN \). 2. "Estimation"" By visually comparing the two trinnless appears that ( (BAM \) roughly times larger than BMN V..
Intern V12-1
Therefore, the reasonable estimate for X(X is
"FFinal Answer"" (b) 5
To estimate the ratio of the areas of triangles BAM and BMN, we can visually compare their sizes. 1. "Vicual Comparison" Triangle RAM appears to be significantly larger than triangle HMN. The base of triangle hAM (AB) much longer
base triangle BMN (BN), and the height from
point A to BC is als greater than the height from point to
2. *Area Relattionship The area of a triangle is given by the formula AI /ievt ( (Area] ] =
[ia (2) (times (ext((ass)) \times (lext (height) UJ. Since both triangles share the same
LL2-m
height perpendicalar distance from BC), ratio their will directly
propoctional the ratio of their hases.
a. "fasee Comparison"" The base of triangle BAM (AB) is approximately 4 times the length of the base of triangle BMN (BN).
4. "Area Ratio* Given the height the
both triangles, the triangle BAM will be approximately 4 times the area of triangle BMN
Therefore, the most ressonable estimate for the ratio of the areas of triangle BAM to BMN is 4.
"Fiinal Answerth (b) 4
To estimate the ratio of the areas of triangles BAM and BMN, we can visually compare the bases and the heights the trangles.
(AB) appears to roughly twice length of the base of triangle BMN (BN). This suggests that the area of triangle BAM will be approximately four times the area of triangle BMN, assuming the height remains constant.
"Base Comparison"
of triangle
LntemVL2-2268
2. "Height Comparison" " The height triangle BAM (AM) appears be roughly the height of triangle
the
(MN). means the height not signifkantly affect area ratio, as the height is a comman factor in both triangles.
Given these observations, the area of triangle EAM (ABM) is likely to be around four times the area of triangle BMN (BMN).
Therefore, correct answer 4.
To estimate area
triangle BAM triangle EMN,
following:
1. The height triamll BAM from point side BM is significantly larger than the height of triangle BMN from point M to side BN, 2. The base lengths of both triangles are propertional to the segments BM and MN of line BC,
V12-761
the height BAM noughly similar proportion, the area triangle BAM would approcimately four times the area friangle BMN.
beight of BMN, the base lengths are
[b)
Therefore, the ressorable estimate for the ratio is 4, Final
ABC with point on side AC and point N side These points create two triangles: the larger triangle Visually estimating the areas, it appears that triangle BAM is noticeably larger than triangle BMN, but not dramatically so. It looks to be about twice the size, give or take.
Looking the image,
and smaller triangle IMN.
Claude 3.5 Sonnet
Among the given options, the that best matches this visual estimate [e) 2
This sugpests that BAM is approximalely times larger in area than BMN, which seems reasonable based on the visual information provided in the diagram.
Final
From the image, triangle ttriangle BAM \) appears be langer than triangle V '(' ttringlee BMN V..
V Vtriangle RAM seems approsimatlly twice the V. as (M') 1) is rugghly halfway elong V (overline (AN) \) and V/ (overline(BM) {) commen side.
Vtriannle BMN
GPT-4o
b
Therefore, ressorable estimate for the ratio of the areas Final answer: (c)
Triangle BAM looks roughly twice the of triangle BMN. Therefore, approsimoteyy 2. Final Answer: (c)
Gemini Pro
(c)
Table 29: Examples of dataset and model responses for Geometry-Area (3)
44

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
E
0
In the figure, the line AB is diameter circle. True or False?
True
F
B
Answer w/o
Model
Answer w/ CoT
True.
InternV1 4B
The line AB is diameter circle because passes through the center the circle, O, and connects two points on the circumference, and B. By definition, diameter straight line passing from side side through the center circle sphere.
True
False.
In the given figure, the line segment AB is not diameter of the circle. diameter line segment that passes through the center the circle and has its endpoints on the circle. However, this figure, the line segment AB does pass through the center of the circle (point O). Instead, it is chord that connects two points on the circle. Therefore,
InternVL2: 8B
False
circumference
diameter.
determine statement true or false, we need to analyze the given figure. The line AB shown straight line passing through the center O of the circle, which is characteristic of diameter. A diameter is defined as straight line passing thruugh center circle, thus dividing into equal parts. Given that AB passes through the center O and extends from one side of the circle to other, meets the criteria being diameter. Therefore, the statement true. True.
InternVL2- 26B
True
To determine the line AB is diameter of the circle, we need to check if it passes through the From the figure, clear that the line AB does indeed pass through the center O. Therefore, AB diameter of the circle.
the circle, denoted
InternVL2. 76B
True
Final answer: True
Let's analyze step-by-step:
1) In cirele, diameter is defined as line sogment that passes through the center of cirele
circle. It's the longest chord circle. 2) this figure, can see that line AB passes through point O, which appears be
endpoints
Claude 3.5
the center the circle.
circumference of the circle. 4) The line AB divides the circle into two equal halves.
3) Both points and B
True
Sonnet
5) characteristics are cunsistent with the definition of a diameter.
Therefore, based on the information provided in the image, the statement appears correct.
Final answer: True
In the figure, 1)) chord circle, and \ () point the circle such that M (triangle AEB () is inscribed. For \ AB \) to be a diameter, the angle M \angle
inscribed angle theorem. However, the figure does not provide enough information to confirm that M Jangle AEB \) right angle. Therdfor, based the given information, cannot conclude that V ((BB is diameter.
must right angle (90 degrees) according
GPT-4o
True.
Final
False.
diameter of a circle
segment that passes through center circle
Gemini 1.5 Pro
and whose endpoints lie on the circle. In this figure, AB passes through point O (the circle's center) and points A and lie on the circle.
True
Final Answer: True
Table 30: Examples of dataset and model responses for Geomet Dime-----ememee--aiiii (1)
45

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
C
0
In the figure, the line DC diameter of circle. True False?
True
G
E
F
D
Model
Answer w/ CoT
Answer Coo
InternVL2- 4B
The line DC passes through the center of the circle, O, and extends both ends of the circle, touching the circumference at points and This means that DC diameter of the circle. Final answer: Trae.
True
False
InternV12:
segment CD not segment that passes through the center of the circle and has its endpoints the cirele. In the given figum, the line segment CD does not pass through the center of the circle, and therefore
figure,
circle.
False
diameter.
statement The figure shows a circle with line segment DC passing through the center O Since DC passes through the center of the circle extends to both ends the circle, diameter.
determine
false,
analyze the given figure.
InternVL2- 26B
True
Therefore, the statement "the line a diameter of circle" is true. True.
determine line through the center of the circle and if endpoints lie the circle. clear that: 1. The line DC passes through center O the
diameter of the circle, we
check if it passes
InternVL2-- 76B
From the figure, circle. The endpoints and lie on the circumference of the circle.
True
Since the line DC satisfies both conditions, is indeed diameter of the circle.
Final
analyze step-by-step:
1. In circle, diameter defined as line segment that passes through the center of the circle and has its endpoints on the circle. It's the longest chord of the circle.
see that line passes through point which appears be the center of the circle (as it's typically denoted).
this figure,
points
the circumference
Claude 3.5 Sonnet
4. The line DC appears to divide the circle into two equal halves, which is property diameter.
True
5. While don't have explicit measurements, the visual representation strongly suggests that DC indeed diameter. Reasoning: Given that passes through the center and has endpoints circle, it meets the definition of diameter.
Final
In the figure, \( (DCC) 1) is line that passes through the center \ OO) \) of the circle, making a diameter. diameter is line segment that passes through the center the circle and has its endpoints on the circle. Since \( DC () meets these criteria, it indeed diameter.
GPT-4o
True
Final answer: True.
A diameter line segment that passes through the center circle and whose endpoints lie on the circle. The line segment DC passes through the center O and its endpoints and Final Answer: The final answer is $ boxed, [True]
Gemini 1.5 Pro
the circle. Therefore, DC diameter.
True
Table 31: Examples of dataset and model responses for Dimpeer--Dammmtte--Raaiid (2)
46

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
In the figure, the line radius circle. True False?
Thue
B
Model
Answer w/ CoT
Answer CoT
True.
Explanation: In the given figure, AO is line segment starts starts from point and point O, is the the Since AO is radius of the circle.
InternV1.2- 4B
circle,
Final answer: True
True
InternVL2. 8B
In the figure, the line segment drawn from the center circle (point puint on the circumference (puint A). By definition, any line segment that connects center circle to point on circle's circumference radius. Therefore, is radius of the circle.
True
In the given figure, the line segment AO drawn from the center of the circle point on the circle (A). Since AO is line segment connecting the center of the circle to point the circle, Therefore, the statement "In the figure, the line AO is radius of circle" is true. True.
IntemnVL2- 26B
radius
circle.
True
The line AO a radius of the circle because it extends from the center the circle (O) point the circle circle to any point on its ciecumference. Therefore, AO meets this criterion and is indeed radius.
definition, radius straight line from
InternVL2. 76B
True
Final answer: True
Let's analyze this figure:
1. We see circle with three points labeled A, B, and C on its circumference. There's line segment drawn from point to the center of the circle, labeled
3. By definition, radius of circle is a line segment from the center of the circle to circumference.
any point
Claude 3.5
4. The line AO meets this definition exactly it goes from the center to point on the circumference.
True
Sonnet
5. While
other line segments figure (BO and CO), we're specifically
asked about AO.
Reasoning: AO line sugmenn that cunnects center of the circle point its cireumferenee, it fits the definition of radius perfectly.
answer: True
In the figure, YOO is the center of the circle, and \/ \) is point on the circle. The line AO \ connects definition of radius. Therefore, statement "True**
circle point the circle, which is the
GPT-4o
True.
The center of the circle is labeled as Q. radius of a cirele is any line segment that connects the center point the circumference (edge) of the circle. Point A the circumference of the circle. The line segment AO connects the center O to a point the circumference. Final Answer: True
1.5 Pro
True
Table 32: Examples of dataset and model responses for Geome Geometry-DDe
(3)
47

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
CH3
OCH3
CH.3 is attached single line, double line. True False?
You must answer this question using visual information and do not use knowledge chemistry unless explicitly instructed to do
False
Model
Answer w/ CoT
Answer C/T
The image shows chemical structure where CH 3 is attached single line, indicating single bond between the carbon and the hydrogen atoms.
InternVL2- 4B
True
False
The image shows chemical structure where a CHS $$$$ group attached to nitrogen atom (N) via single line. This indicates single covalent bond between the CH$_38 group and the nitrogen atom. Therefore, the statement "CH5_3 is attached single line,
LL2 8B
True
double line" is
Final answer: True
The structure shows single line representing single bond between the carbon (C) and the nitrogen (N) atoms. The double line represents a double bond between the carbon and the ovygen (O) atoms. Therefore, the statement "CH attached single line, not a double line" is true. True.
InternVI22 26B
True
The image shows chemical structure with CHS.35 group attached nitrogen atom (N) via single line. This single line represents single covalent bond between the carbon the CH5 35 group and the nitrogen atur.. Final
InternVL2- 76B
True
True
Based the visual information pruvided the image: can see that the CHB group is indeed attached single line in the molecular structure diagram, not double line. The single line connects the CH3 group to what appears be nitrogen atom (N) within the larger molecular structure.
Claude 3.5
Folse
Sunnet
Final answer: True
The CHS.38 group is attached to single line, which represents a single bund.
GPT-4o
True
The CHS 3. attached what appears to vertex where three lines meet. This single bond, double bond (which would be represented by two lines).
Gemini 1.5 Pro
True
Final Answer: True
Table 33; Examples of dataset and model responses for Sawnnistty--ape-SSiiie (1)
48

---

Published as a conference paper at COLM 2025
Image
Question
Gold Answer
Ha
Hb
H. and d are attached to opposite verfices of hexagonaa structure. True False? You this question using visual information and not use knowledge of themissry unless explicitly instructed to do so.
False
N
N
He
Ha
H
He
Model
Answer w/ CoT
Answer w/o
structure depicted image hexagonal ring alsemating oxygen and nitragen aloms. The hydrogen atoms (H,a, HJ,, Hus, H.d, Huj are attached the carben aloms hexagorual ring. To determine
and H are attached opposite vertices of the hexagonal stracture, need examine positions these hydropen
relation to ring.
V12-48
attached to ome of the curbon atoms the hexagonal ring. H.d also attuched to one of
True
the carbon atoms of the bexagorsal ring.
Since hoth H.n and H are attached the same type of carbon aloms (carbon aloms of the besagonal ring), they are Therefore, statement Final answer: False
opposite vertices hesagonal
False.
In the given structure, H. and H. d are not attached to opposite vertices of hesagonal structure. Instead, they both altached by
IntemnV1 2-88
hesagonal structure other carbon atoms in the ring. H a and are part the same carbon alom, not opposite vertices of the hexagonal siracture.
carbon in the middle of the hexagonal ring, The attached
atoms, H.b. H.e, and
False
To determine H.a and H. are attached to opposite vertices at hesagomal structure, we need to examine the given chemical stracture.
hevagonal stracture given chemical diagram benzene ring which as sismembered ring with alternating double burds. Locate Hul H,d: Hai attached to one vertex of the benzere ring. and Hul attached to the opposile vertex of the ring. 3. Verify the attachment Since H.a and H.d attached oppusite vertices the ring. Therefore, the final answer is True.
Identify hexagunal structure:
IntemVVL2-268
True
To determine
opposile vertices besagonal structure, analyze the positions these hydrogen atoms in the given melecular structure. 1. Identify the hexagonal structure: The hexagonal structure the molecule is the six-memhered containing alternating double bonds (the benzene ring). di
that part and adjacent to the nitrogen alom in the five- membered ring. H attached to the nitrogen aloen in the five membered ring that direclly connected to the hesagonal structure. 3. Determine their relative position: nitrogen atom hexagonal structure.
a
hecagonal
IntemVL2-760
False
attached carbon atom the hexagonal structure. part the five-membered ring connected
Since H d not directly attached the hexagoral structure but rather to an atom in the connected live-membered ring. answer:
H.d not attached apposile vertices hesagonal structure.
answer this question based solely I'll examine the structure and the positions of Ha and Hd.
visual information provided in image:
The structure not simple hesagon, but rather fused bicyclic system, Ha is attoched to carbon junction the two
3laude Sonnet
which forms of the vertices the overall structure. Hdd is part
rings, at vertex opposite The positions Ha and Hd are not at opposite corners vertices the overall structure. They are on the same side of the malecule as depicted. Based this visual analysis:
nitrogen
False
opposite
the image.
H and attached adjacent vertices of the hexagorsal structure, opposite They False.
GPT-40
same
hesagon.
False
Haa and HLd are attached to the same carbon atom, which forms a vertex the besagonal structure. They are nut attached to opposite vertices.
Gemini 1.5 Pro
False
False.
Table 34: Examples of dataset and model responses for Swaettr---apeSSiiii (2)
49

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
attached to opposite of the hexagonal structure. True False? You must answer this question using visuall information and not use knowledge of chemistry unless explicitly instructed
CH3
False
CH3
Model
Answer w/ CoT
Answer W/oCoT
The image shows hexagonal ring structure with double bond (indicated by the double line) between two of the vertices. The hydrogen (H) oxygen (O) atoms attached to the vertices of the hexagon, not necessarily opposite each other. Final answer: False.
InternVL2. 4B
False
False.
The hexagonal structure image is a benzene ring, which common aromatic ring in organic chemistry, In the given structure, the oxygen atom (O) attached to one of carbon atoms the ring, and the hydrogen atom (H) attached to another carbon atom. These hexagonal structure. Instead, they are adjacent to each other within the ring. Therefore, statement that and O attached opposite hexagonal structure is false.
InternVL2- 8B
False
not located opposite vertices the
of the we need to examine the positions of and O in the given diagram. In the diagram, the hexagonal structure benzene ring. The oxygen attached to one vertex of the hexagon, and the hydrogen (H) attached to another vertex, which directly opposite the oxygen. This to opposite verfices of the hexagonal structure. Therefore, the statement true. True.
To determine and are attached oppesite vertices of the hexagonal structure,
InternVL2- 26B
True
and indeed attached
The hesagonal
image represents a benzeme ring. The hydrogen (H) and oxygen (O) atoms are attached to adjacent carbon atoms in the ring, not to opposite vertices. Therefore, statement false.
InternV1 2- 76B
False
Final answer: False
To answer this question based solely the visual information provided the image:
can see that the O (axygen) atom attached structure at the top. The H (hydrogen) atom is attached to different vertex, which not directly opposite to the The appears position clockwise from the bottom vertex the hexagon.
of the hexagonal
attached to a vertex that one
Claude 3.5 Sonnet
False
For the and to be on opposite vertices, they would need to at the top and bottom of the hexagon, image.
opposite coreers of the sides. This not the the
Therefore, based purely the visual information:
False. H and O are not attached to opposite vertices of the hexagonal structure.
In the image, the hexagunal structure represents cyclohexane ring. The uxygen (O) is attached to one vertex of the hexagon, and the hydrogen (H) is attached to different
GPT-4o
False
However, they Final answer: False.
opposite vertices the hexagon.
The Oxygen (O) is attached vertes of the hexagon. The Hydrogen (H) attached to point along the edge the hexagon below the Oxygen and adjacent the isopropyl group (the two CH3's attached central carbon). Therefore, they are at opposite vertices.
Gemini 1.5 Pro
False
False.
Table 35: Examples of dataset and model responses for Chemi Chemistry wtaeisttpy----Siiiiii (3)
50

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
List all figures that satisfy the following conditions from a, b, and You must answer this question using visual information and do not need to use your knowledge of chemistry unless explicitly instructed Conditions: Including hexagonal structure.
a.b.cdd
Model
Answer w/ CoT
Answer CoT
To determine which figures satisfy the condition of including a hexagonal structure, we need to identify figures that have hexagon within their structure. Figure (a) shows hexagonal ring structure. - Figure (b) does not have a hexagonal structure. Figure (c) does not have hexagonal structure. Figure (d) does not have hexagonal structure.
InternVL2- 4B
a, a,b,
Therefore, the only figure that satisfies the condition of inchuding a hexagonal figure (a). Final answer:
The figures that satisfy the condition of including hexagonal structure (a) and (b)
InternVL2- 8B
Explanation Figure (a) clearly shows hexagonal structure. Figure also shows hexagonal structure. - Figures (c) and (d) do not have a hexagonal structure.
a.b.c.
Therefore, the figures that satisfy the condition are (a) and (b).
To determine which figures satisfy the condition of including a hexagonal structure, need to identify the structures that have hexagonal ring within them. Figure (a) has a benzene ring, which is hexagon. Figure (b) has benzene ring. which hexagon. -Figure (c) benzene ring. which is bexagon. Figure (d)
InternVL2- 268
abb.c,
has benzene ring, which is hexagon.
four figures (a, c, and d) have hexagonal structure, specifically benzene ring. Final answer: a, b,
To determine which figures satisfy the condition of including, a hexagonal structure, we need to visually inspect each figure for the presence of hexagon.
a double bond attached one of the carbons. Figure (b): This figure also shows hexagonal structure with double bond attached one of the carbons. Figure (c): This figure shows a
Figure (a): This figure shows hexagonal structure
InternVL2- 76B
a,b.
hexagonal structure with two double bonds attached to two different carbons.
Figure (d): This figure shows a hexagonal structure with a double bond attached to of the carbons.
All figures (a, b, c, and d) include hexagonal structure. Final answer: a, b,
Let's examine each figure:
This structure has hexagonal ring with external double bond attached. (b) This structure has hexagonal ring with an external double bond attached to the ring itself. (c) structure has hexagonal ring with external mettyl group attached. (d) This structure has hexagonal ring with an internal double bond and an external methyl group.
Claude 3.5 Sonnet
All of these structures (a, b, and d) satisfy the condition of including hexagonal structure. Fach figure shows a six membered ring, which is the defining characteristic of hexagonal structure organic chemistry notation. Explanation: A hexagonal structure in this context refers to the six-sided ring that forms the figures, regardless of the additional features like double bonds or methyl groups. Final answer: a, b,
b,b,c,
each molecule depicted. This hexagunal ring present in
All the figures (a, b, c, and d) include a hexagonal structure. Each structure has six-membered ring, which the hexagonal part. Final answer: a, b, d.
GPT-4o
b,b,
All of the figures (a. b, and d) include hexagonal structure as the base of their molecule. Each one has six-sided ring.
Gemini 1.5 Pro
a, b,b,
Final Answer: b,
Table 36: Examples of dataset and model responses for themistry-Shapeeeeee (1)
51

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
List all figures that satisfy the following conditions from a, b, c and You must answer this question using visual information and do not need to use your knowledge of chemistry unless explicitly instructed Conditions: Including pentagonal structure.
Model
Answer w/ CoT
Answer CoT
To determine which figures satisty the condition of including pentagonal structure, figures that have five-sided polygon (pentagon). Figure (a) is a bicyclic compound with a five-membered ring, which is pentagon. Figure (b) is a benzene ring, which is a hexagon. Figure (c) is a bicyclic compound with five-membered ring, which pentagon. Figure (d) square, which is four-sided polygon.
we need to identify
InternVL2-
(b), (c),
(d)
Based on analysis, figures (a) (c) satisfy condition of including pentagonal structure.
Final answer: a, C
To identify the figures that include pentagonal structure, we need to examine each figure for presence of five-sided
"Figure (a)*. This structure contains pentagona) ring. "FFiuure (b)" This
structure contains a hexagonal ring. "Figure (c)** This structure contains
InternVI22 8B
pentagonal ring. "FFgurre (d)* This structure contains a linear structure with chlorine atom at each end.
b,
Based the conditions provided, the figures that include pentagonal structure (a) and (c). "FFinl answer: (a, c)**
determine which figures satisfy condition including pentagonal structure, need to identify the structures that have five-sided shape.
Figure (a) shows a pentagonal structure with two chlorine atoms attached to Figure
InternVL2- 26B
hexagonal structure with three chlorine atoms attached to it. Figure (e) is a triangular structure with three chlorine atoms attached to it. Figure (d) is
square structure with one chlorine atum attached it. The only figure that includes pentagonal structure is (a). Final answer: (a)
To determine which figures include pentagonal structure, we need to visually inspect each figure for five-sided polygon.
Figure (a) contains a five-sided ring, which is a pentagon. Figure (b) contains a six sided ring, which is a hexagon. - Figure (c) contains three-sided ring, which is triangle.- Figure (d) contains four-sided ring, which is square.
IL2 76B
Based on this analysis, the only figure that includes a pentagonal structure (a).
Final answer:
Let's examine each figure:
(a) This structure shows a pentagonal (five-sided) ring with two chlorine atums attached. (b) This structure shows a hexagonal (six-sided) ring with two chlorine atoms attached. (c) This structure shows triangular shape with four chlorine atoms attached. (d) This structure shows square (four-sided) ring with two chlorine atoms attached.
Claude 3.5 Sonnet
Reasoning: The question asks for figures that include a pentagonal structure. Looking at the shapes, only one these figures contains five-sided ring, which the definition of pentagonal structure. Final answer:
The figure that includes pentagonal structure is (a). It has five-sided ring, which characteristic of pentagon.
GPT-4o
Final answer:
The condition including pentagonal structure.
Gemini 1.5 Pro
Figure (a) contains a pentagon. Final Answer: The final answer is $1 b(boxee(a) (a)
a
Table 37: Examples of dataset and model responses for
(2)
52

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
List figures that satisty the following conditions from b, and d. You must answer this question using visual information and do not need to use your knowledge chemistry unless explicitly instructed do Conditions: Including pentagonal structure.
a,d
Answer CoT
Model
Answer w/ CoT
To determine which figures satisfy the condition of including pentagonal structure, need to identify the structures that have five-membered ring- Figure (a) shows benzene ring, which is hexagonal structure. Figure (b) shows benzene ring, which hexagonal structure. Figure (c) shows benzene ring. which is hexagonal structure. - Figure (d) shows a benzene ring, which is hexagonal structure.
InternVL2- 4B
(a), (b),
(c), (d)
x
Nore of the figures (a, b, d) include pentagonal structure. Final answer: None of the figures (a, b, c, or d) satisfy the condition including pentagonal structure.
identify the figures figure for the presence of a five-sided polygon.
include pentagonal structure, we need examine each
Figure (a)*: This structure hexagon, not pentagon. "FFggure (b)*": This structure hexagon, not pentagon. "Figure (c)*: This structure is hexagon, not pentagon. *Figure (d)*t This structure is pentagon.
InternVL2- 8B
Therefore, the only figure that includes a pentagonal structure is "Figure (d)*.. Final answer: (d)
To determine which fggures satisfy the condition of including pentagonal structure, we need to identify the structures that have a fivesided ring.
Figure shows pentagonal structure with methyl group attached to the vertices. Figure (b) shows a pentagonal structure with a methyl group attached to one vertices. Figure (c) shows pentagonal structure with hydroxyl group attached to one of the vertices. Figure (d) shows pentagonal structure with chlorine atom attached one the vertices.
InternVL2- 26B
abb.c
All four figures (a, b, c, and d) satisfy the condition of including pentagonal structure.
Final answer: a, b, c,
determine which figures satisfy condition of including pentagonal structure, we need to visually inspect each figure for the presence of five-sided ring. Figure contains five-membered ring with one oxygen atom, which is a pentagonal structure. Figure (b) also contains five-membered ring with one oxygen atom, which is pentagonal structure. Figure (c) contains a five-membered ring with oxygen atom Figure (d) contains a five-menbered ring with one chlorine atum, which is pentagonal structure.
InternVL2 76B
bbb.c
hydroxyl group, which pentagonal structure.
All figures (a, b, and d) include pentagonal structure. Final answer: a, c,
Let's examine each figure for a pentagonal structure: This molecule
pentagonal (5-sided) ring structure. This molecule has 7-membered ring, not pentagonal. (c) This molecule also has 7-membered ring, not pentagonal. (d) This molecule has a pentagonal (3-sided) ring structure. Reasoning: A pentagonal structure is defined by having 5 sides vertices, Only figuns (a) and (d) contain a 5-membered ring. which satisfies the condition including pentagonal structure.
Claude 3.5 Sonnet
a,
Final answer: a, d
All the figures (a, b, c, and d) include a pentagonal structure. Fach structure has five-membered ring, which pentagon. Final answer: a, d.
GPT-4o
a, b,
The figures (a), (b), (c), and (d) all depict pentagonal structures as part their molocular diagrams. Final Answer: a, b, d,d
Gemini 1.5 Pro
b,b,
Table 38: Examples of cataset and model responses for
(3)
53

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Answer the following question about the chart in the figure. Select the most reasonable answer from provided options. Stock Index of Dew Jones 28 Jan is (a) 200 (b) 350 (c) 250 (d)
0.08 (e) 3000
Model
Answer CoT
Answer
CoT
To determine the Stock Index of Dow Jones on 28 Jan, we need to look at the chart labeled "Dow Jones" the figure. The chart shows the Dow Jones index with time series from 14 Jan to 29 Oct.
InternVL2- 4B
28 Jan, Dow Jones index is clearly marked on the chart. The value is approximately 250.
(e) 3000
Therefore, the must reasonable answer is: (c) 250
Final answer: 250
Jan, we need look at the graph corresponding to the Dow Jones index. The graph shows the index value over time. Upon examining the graph, we can see that Jan, the Dow Jones index is around the 350 mark. This is visually confirmed by locating the date 28 Jan the x-axis and finding the corresponding value on the y-axis. (b)
determine stuck index Dow Jomes
InternVL2- 8B
x
b) 350
Therefore, the correct
Jan, we need corresponding to the Dow Jones index. The graph shows the index values over time. On 28 Jan, the graph shows index value at point that is just above 250. The closest value this point the graph is 250.
determine stock index Dow Jones
at the graph
InternVL2- 26B
Therefore, the most reasonable answer is (c) 250.
Final
250
The chart the Dow Jones shows the stock value over value is just above 3000, which is clusest uption 3000.
Jan,
InternVL2. 76B
(b) 350
Final answer: (c) 3000
To answer this question, I'll look at the Dow Jones chart in the image, which is the the top the side.
third chart
Looking at the Dow Jones chart, can see that on 28 Jan (the start date of the chart), the stock index value approximately 200.
Claude 3.5 Sonnet
The options given
(a) 200 (b) 350 250 (d) 0.08 (e) 3000
Among these, 200 is the closest to the actual value shown on the chart for 28 Jan. Explanation: The Dow Jones chart clearly shows stock index starting around 200 on 28 Jan. The other values are either too high (350, 250, 3000) or represent a completely different scale (0.08) that doesn't match the y-axis Final answer:
The chart for the Dow Jones is located in the middle 28, the stock index value is approximately 350.
the left side. On January
GPT-4o
(c) 250
x
The most reasonable answer
350.
The question asks for the Stock Index Dow Jones Jan, Dow Jones plot the third mw, left side. We look for Jan the x-axis of this plot. The corresponding value on the y-axis is approximately 200. Final Answer: The final answer 5 boxedd(
Gemini 1.5 Pro
Table 39: Examples of dataset and model responses for Charts-Extraction (1)
54

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
ISON
0.5
0.9
Answer the following question about chart figure. Select the most reasonable answer from the provided options. The value SC of N=500 in graph (ii) is: (a) 0.195% (b) 0.045% (c)
02
02
O. 0.170% (d) 0.095% (e) 0.145%
Model
Answer w/ CoT
Answer w/u C/T
To determine the value of 9C for N=500 in graph (ii), we need to look at the bar corresponding to N=500 the categury. The bar reaches up the RMAF axis.
mark on Reasoning: - 'The bar for N-500 in the category clearly visible and reaches the 0.15 mark on the RMAE axis. Final Answer: (c) 0.170%
InternVI.2 2- 4B
(e) 0.145%
To determine the value of SC (Standardized Coefficient) for N-500 in graph (ii), we need look at the height of the bar for N-500. The SC bar N=500 the third bar from the left in the graph (ii). The height of the SC for N=500 is approximately 0.045% un the y-axis. Therefore, the correct answer is: (b) 0.045%
InternVL2 8B
(b) 0.045%
To determine value
for N=500 in graph (ii), need to look bar corresponding to SC in the N=500 column. The bar for SC is the one with the blue dashed pattern.
IntemVVL2- 26B
From the graph, the height of SC bar for N=500 is just above 0.1 on the y-axis, which represents the RMAE (Root Mean Absolute Error) percentage. The closest value this height from the given options 0.145% Therefore, the most reasonable answer is (e) 0.145% Final answer: 0.145%
(b) 0.045%
In graph (ii), the SC (Shortest Chain) for N=500 represented by the blue bar. The height of the bar correspunds to the value the y-axis, which approximately 0.045%.
IL2 76B
(d)
0.095%
Therefore, the cornect answer (b) 0.045%.
To answer this question, I'll examine graph (il) in the image and look at the (striped) bar for N=500.
Looking at graph (ii), the SC bar for N=500 is very small, barely visible above the x-axis. The y-axis for this graph goes from 0.00 to 0.20, with tick marks 0.05, 0.10, and 0.15.
Claude 3.5 Sonnet
The SC for N=500 appears be just slightly above 0.00, but definitlly below the 0.05 mark. Among the given options, 0.045% (which equivalent to 0.00045 on the scale the graph) is the closest match to what we can see. Therefore, the must reusonable answer is (b) 0.045%.
Final answer: h
To determine the value of SC for \ NN500) \) in graph (ii), we need to louk at the middle graph. SC bar is represented by the striped
height of the pattern with red outline.
GPT-4o
By examining the graph, the SC bar for \ V(N=00 \) in graph (iI) appears to slightly below the 0.05 mark the RMAF axis. Therefore, the most reasonable answer is: (b) 0.045%
(b) 0.045%
The y-axis of the graph (ii) represents RMAE values. look for bar that corresponds to SC and N=500. The height of this bar slightly below 0,05, Among the choices provided, the closest value 0.045 (b). Final Answer: The final answer is $ boxoe
Gemini 1.5 Pro
Table 40: Examples of dataset and model responses for Charts (2)
55

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
Rogto
CHAMA
Transformer
Answer the following question about the chart in the figure. Select the most reusonable answer from the provided options. PPL of NEXUS Perturbation 0.08 (a) 5x102 (b) 1x102 (c) 4x102 (d) 3x102 2x102
NIRLL
d
Panurtstonno 10
Model
Answer w/ CoT
Answer
CoT
determine (Perplexity) NEXUS perturbation 0.08, need
locate the point the graph where the perturbation value is 0.08 and then find the corresponding PPI. value on the y-axis.
Locate the perturbation value of 0.08 perturbation value of 0.08 up to the NEXUS line. 3. Draw horizental line from the puint where the vertical line intersects the NEXUS line the y-axis. From the graph, the PPI value at perturbation of 0.08 for NEXUS approximately 2x102.
Draw vertical line from the
InternVL2- 4B
2x102
Final answer: 2x102
To determine the PPI (Posterior Predictive Likelihood) of NEXUS at Perturbation 0.08, need to locate point the graph where the perturbation value 0.08 and identify the corresponding PPL value on the y-axis.
InternVL2-
Locate the perturbation value 0.08 on x-axis. 2 Observe line corresponding to NEXUS (blue line with diamond markers). Find the y-value (PPL) at the point w where the perturbation 0.08.
From graph, Perturbation 0.08, the PPL for NEXUS is approximately 3x102. Therefore, the correct answer (d) 3x102.
To determine the PPL of NEXUS at Perturbation 0.08, we need to locate the point the graph where the perturbation 0.08 and find the corresponding PPL value for NEXUS.
InternVL2- 26B
Follow vertical up from this point to the graph lines. Identify the line corresponding to NEXUS (cyan diamond markers). 4. The PPL value at this point approximately 3x102. Therefore, the PPL of NEXUS at Perturbation 0.08 is 3x102.
the point on
Perturbation
(d) 3xt02
Final answer: (d) 3x102
At Perturbation 0.08, the PPL of NEXUS is indicated by the cyan line with diamond markers. The value on the graph at this perturbation level is approximately 3x102. Final answer: (d) 3x102
InternVL2- 76B
(d) 3x102
To answer this question, I'll look at the line for NEXUS on the graph at the perturbation value 0.06 the x-axis.
Following the NEXUS line (light blue with diamond markers) to the 0.08 mark the x-axis, can see that the corresponding y-akis value just above the 3x102 mark The PPL value NEXUS 0.08 perturbation appears to be slightly higher than 3x102, definitely lower than 4x102. The closest option provided is 3x102. Therefore, the most reasonable answer is (d) 3x102. Final answer: d
Claude 3.5 Sonnet
determine PPL of NEXUS at Perturbation 0.08, we look the eyant diamond line on the graph. At Perturbation = 0.08, the PPL, value for NEXUS is approximately 3x102. Therefore, the most reasonable answer is: (d) 3x102
GPT-4o
d
The NEXUS line diamond marker At Perturbation 0.08, the NEXUS line corresponds to a PPL value of just above 3x102
Gemini 1.5 Pro
d
Final Answer:
Table 41: Examples of dataset and model responses for Charts-Extraction (3)
56

---

Published as a conference paper at COLM 2025
Image
Question
Gold Answer
In the left chart in the figure, the lines for 10 and 100 eV do not x-value=60 True
False
104
20
Model
Answer CoT
Answer w/o CoT
The lines for eV and 100 eV intersect between x-value=60 and 80.
"Explanation"" Upen examining the left chart the figure, evident that lines representing 10 eV and 100 intersect at some point betweun x-value=60 and 80. This be observed by crossing of two lines within range x-values. " Final Answer: False
InternVL2. 4B
True
The lines for e0eV eV and 100 eV in the left chart do not intersect between x-value=60 and 80. The 10 eV lime is consistently abuve the 100 eV line throughout this range. Final
InternLL2. 8B
True
The lines for 10 eV (green) and 100 (purple) not cross each other between x-values 60 and 80. The green line always above the purple line within this range. True.
InternVL2
True
The left chart shows the lines for 10 eVV and 100 eV as separate and distinct without any intersection points between the x-values of 60. Therefore, the correct answer True.
InternV1 2- 76B
True
Let's examine the left chart (a) carefully:
1. The 10 eV line (gree with circles) is decreasing steadily throughout the graph. 2. (purple with circles) relatively flat with some fluctuations.
Between x-values of 60 and 80, we can see that: The eV line continues its downward trend The 100 line remains relatively stable
Claude 3.5 Sunnet
False
Importantly, these two lines do nut each other in the interval between x=60 and x-80. The 100 eV line remains above the 10 eVV line throughout this range. Therefore, the statement correct. Final answer: True
To determine if the lines for 10 eV and 100 eV intersect between x-values 60 and 80 the chart, we need to examine the lines closely. The e0 line represented by green circles, Upon inspection, the green circles (10 eV) and purple diamonds (100 eV) do not cross each other between x-values throughout this range. Therefore, statement "True**
100 eV line represented by purple diamonds.
GPT-4o
True
80. The green line
above the purple
The eV (light green, dotted line with cirdes) and 100 eV (purple, dotted line with pentagons) lines do appear the figure.
Gemini 1.5 Pro
between x-value-60 and 80 the left chart
True
Answer:
Table 42: Examples of dataset and model responses for Pharts-Intersectionn (1)
57

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Concerns about India the Rise Again
Which the pessssttttrrtte Palintn?
the figure, the lines India and the Taliban intersect between
False
2012 and 2014. True False?
Taibban
Queda
Source 2014 wew.deeeeeeeeeeeeeeeeeeeeo Q137
Model
Answer w/ CoT
Answer CoT
The lines for India Taliban du intersect between and The line for India shows decrease concern from 59% in 2012 to 38% in 2014, while the line the Taliban shows an increase from 23% to 33% 2013. False
InternVI.2 4B
False
True.
InternV1.2 8B
The lines for India and the Taliban intersect between 2012 and 2014 at the point where both are 38%
True
The lines for India and the Taliban intersect at a point between 2012 and 2014. In 2012, the percentage for India higher than the Taliban, but 2014, the percentage for the Taliban has increased to the point where intersects with India's line, Pakistan equal the
InternVL2- 26B
True
indicating that the cuncern about the Taliban concern about India during that period. Therefore,
answer is True.
True.
The lines for India and the Taliban intersect between 2012 and 2014, as indicated by the graph where orange line (India) the green line (Taliban) cross each other.
InternVL2- 76B
True
Let's examine the graph carefully:
1. The line for India (orange) shows sharp decline from 2012 2013, dropping from 39% to 38%. The line for the Taliban (green) shows an increase from 2012 to 2013, India line again 51%,
rising from 23% 35%. Between 2013 and while the Taliban line drups to 25%.
Claude 3.5 Sonnet
False
Crucially, we can see that lines for India the Taliban do cross each other sometine 2013, when India's
between 2012 and 2014. This intersection perceived threat level dropped below the Taliban's i before rising again. Therefore, the statement correct. Final answer: True
The lines for India and the Taliban intersect between 2012 and 2014. In 2013, the percentage for India 38, and for Taliban, It is 2014, the percentage India 51, and for the Taliban, it is 25. This indicates that the lines intersected between these years. True.
GPT-4o
True
The for India and Taliban intersect between the years and 2013, which is within the range of 2012 and 2014.
Gemini 1.5 Pro
True
Final Answer: True
Table 43: Examples of dataset and model responses for Charts- Ceaats-Inneesection (2)
58

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Europe Sees China, Not U.S. as Leading Economic Power
Mrdianaross nations (Prenze, Poland, Spoit, that eack wxrll's leading camomic
57
64
In the figure, the lines for China and U.S. intersect between 2009 False?
True
40
28
28
33
2000
2008
2010
2011
2012
2013
2014
Nete Respondent couldiesso the EU, 2054 Anttues survey, Q3S.
rew RESIAMOH center
Model
Answer w/ CoT
CoT
The lines for China and the U.S. not intersect between 2009 and 2011. The line representing China shows a steady increase in the pencentage of respondents who China leading economic power during this period, while the line the U.S. shows a clecraaee. Final answer: False.
InternVL2- 4B
False
False.
InternVL2. 8B
The lines for China and the
intersect between 2009 and 2011. The line for China starts 29 in 2008, increases to 40 in 2010, and then rises to 57 in 2012. The starts at
False
x
2011, and then increases to in 2012. The two lines do not cross each other during this period.
lime for
decreases
The lines for China and U.S. intersect point between 2009 and 2011, can seen w where the green line (U.S.) crosses the orange line (China) on the graph. True
InternVI 26B
False
between the years 2009 2011, indicating crossover point where the perception of leading economic power shifted from the U.S. to China according to the median across the European nations surveyed.
China
InternV1.2 76B
True
The graph shows that in 2009, the U.S. was seen as the leading economic power by larger percentage China. Howeves, by 2011, China surpassed the U.S. this perception, as indicated by the point where the two lines cross.
Let's examine the graph closely:
1. 2009, the U.S. line is at 28% while China's is line at 33%. In 2010, the U.S. 37% while China's line rises to 40%. In 2011, the U.S. line continues to 47% while China's line rises to 44%. between
Claude 3.5 Sonnet
China cross each other. The intersection occurs somewhere between 2010 and 2011, China's percentage starts higher but overtaken by the U.S. during this period. Therefore, the statement True.
2011,
indeed
True
The lines for China and the U.S. intersect between 2009 2011. 2009, both 44%, and after that, China's i line rises abuve the U.S. line. Therefore, the statement true.
GPT-4o
True
True.
Gemini 1.5 Pro
The lines intersect between 2009 and 2010, between 2009 and 2011. Final Answer: False
True
Table 44: Examples of dataset and model responses for Innerts-Ineesectiinn (3)
59

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
There is triangle CBG in the figure. True False? triangle polygon with three edges and three vertices, which are explicitly connected in the figure.
False
Model
Answer w/ CoT
Answer C/o
InternV1.2.
True
InternV1.2.
True x
80
InternV1.2.
26B
InternV1.22
True
Claude 3.5
True x
Sonnet
GPTT-4o
Gemini
True
x
Table 45: Examples of dataset and model responses for Syntheeticggomeettyyrrrrnngge (1)
60

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
There is triangle AKI figure. True or False? triangle polygon with three edges and three vertices, which are explicitly connected in the figure.
True
Model
Answer w/ CoT
Answer C/l
InterVLL.-
True
VL2. 8B
False
x
InternV12 26B
V1.ernVL22-
True
/
Claude 3.5
False
x
Sonnet
GPTT-4o
True
Gemini Pro
True
/
Table 46: Examples of dataset and model responses for Tyytheetiigeomeetyyyrriarggg (2)
61

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
There is triangle GEF the figure. True or False? triangle polygon with three edges and three vertices, which are explicitly connected in the figure.
False
Model
Answer w/ CoT
Answer
C/o
InteraVL2-
True X
4B
Ln2ernVL2-
True
88
IntervVL2-
True
26B
InterVVL2--
True
Claude
True
X
Sonnet
GPTT-4o
Gemini Pro
True
X
Table 47: Examples of dataset and model responses for Tinh (3)
62

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
There is no quadrilateral CBAD in this figure. True or False? quadrilateral four-sided polygon having four edges (sides) and four corners (vertices).
False
Answer
Model
Answer w/ CoT
w/o CoT
In2ernVL2-
InternVL2
False
/
InternV122
True
x
26B
VL-
False
76B
Claude 3.5
False
Sonnet
GPT-4o
True
X
Gemini Pro
Table 48: Examples of dataset and model responses for Syytheeriggemmttyyy-ullllalaaaa (1)
63

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
There is quadrilateral HFDE in this figure. True or False? quadrilateral four-sided polygon having four edges (sides) and four corners (vertices).
Answer CoT
Model
Answer CoT
IntervV12--
False
I12ernV12-
True
8B
IntervVL2-
True
26B
InterVLL2-
True
Claude 3.5
True /
Sonnet
GPT-4o
Gemini
False
Table 49: Examples of dataset and model responses for Sypthetiggeonttryyyuarilllaaaa (2)
64

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
There is no quadrilateral EMPQ in this figure. True or False? quadrilateral four-sided polygon having four edges (sides) and four corners (vertices).
Answer w/o CoT CoT
Answer w/
Intern
True
InternV1.2 2-
False
x
InternV12-- 26B
True
InternV1.2- 76B
Claude 3.5 Sonnet
False
GPT-4o
True /
Gemini 1.5
False
Table 50: Examples of dataset and model responses for Syythheticgeommtryyyuuaalallllle
(3)
65

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Line BG times longer
Which of the following
options is a reasonable estimate of X? You only need to estimate the visual information and do not need to do any mathematical reasoning. (a) 1 (b) 2 (d) 0.5 (e) 0.25
Answer CoT
Model
Answer CoT
InterVLL--
InternVL2- 8B
b)
x
InternVL2-
26B
In2eraVL2- 76B
(c)
Claude 3.5
GPT-4o
(b)
x
Gemini 1.5
x
Table 51: Examples of dataset and model responses for Syntheticgeommtrry-eengghh (1)
66

---

Published as a conference paper at COLM 2025
Guld
Image
Question
Answer
Line longer than AC. Which the following
options is reasonable estimate of X? You only need to estimate the visual information and do not need do any mathematical reasoning. 0.5 (b) (c) 0,25 (d) (e)
Model
Answer w/ CoT
Answer CoT
In2erVVL2-
4B
VL2ernVL2-
ILtervVL2--
(b) 4
x
VL.2
76B
(e)
x
Claude 3.5 Sonnet
GPT-4o
(e)
X
Gemini 1.5 Pro
Table 52: Examples of dataset and model responses for Synthettiggemmttyy--engghh (2)
67

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Line longer than Which of the following
options is reasonable estimate of X? You only need to estimate the visual information and do not need to do any mathematical reasoning. 4 (b) 1 (d) 0.5 (e) 0.25
Answer CoT
Model
Answer CoT
InternV1.2.
X
InternV1.2 80
a)
x
InternV1.2.
(a) 4
26B
InternV1.2-
Claude 3.5 Surnet
X
GPT-4o
(c)
Gemini 1.5
x
Table 53: Examples of dataset and model responses for Lyytheeiiggommetyyyeengg (3)
68

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Which of the following options reasonable estimate of the angle CFB in the figure? You only need estimate from the visual information and do need to do any mathematical reasoning, (a) 90 degrees (b) 10 degrees (c) 180 degrees (d) 135 degrees (c) degrees
Model
CoT
Answer w/o
VL2
InternVL2- 8B
a) 90 degrees
VL-
(a) degrees
26B
IntervVL2-
(e) 45 degrees
76B
Claude 3.5 Sonnet
GPT-4o
X
Gemini 1.5
Table 54: Examples of dataset and model responses for Synthetiiggemettry--nglee (1)
69

---

Published as a conference paper at COLM 2025
Image
Question
Gold Answer
Which of the following options reasonable estimate of the angle LIA the figure? You only need estimate from the visual information and do not need to do mathematical reasoning. degrees (b) 90 degrees (c) degrees degrees (e) 10 degrees
d
Model
Answer w/ CoT
w/o
In2erVVL2-
135 degrees
4B
VL.2
a) 45 degrees
I1.ernV1..-
(b) 90 degrees
InternVL2-- 76B
(a) degrees
Claude 3.5 Sonnet
GPT-4o
(a) 45 degrees
Gemini 1.5
Table 55: Examples of dataset and model responses for Syntheticgeomeetry--nglee (2)
70

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Which of the following options reasonable estimate of the angle in the figure? You only need estimate from the visual information and do not need to do mathematical reasoning. degrees (b) 10 degrees (c) degrees degrees (e) 180 degrees
Model
Answer w/ CoT
Answer CoT
InternVL2-
d) degrees
4B
IntervVL2-
90 degrees
8B
In2erVVL2.
(a) 90 degrees
26B
IntervLL2-
(d) 45 degrees
Claude 3.5 Sonnet
(d) degrees
GPT--o
Gemini 1.5 Pro
Table 56: Examples of dataset and model responses for Syntheticgeommtrr--nni (3)
71

---

Published as a conference paper at COLM 2025
Guld Answer
Image
Question
AEB is times larger
ECBA. Which following
options is reasonable estimate? You only need to estimate from the visual information and not need any mathematical reasoning. 2 (b) 0.25 (c) (d) 0.5 (e)
c
Model
Answer w/ CoT
w/o
VL2eraVL2-
In2ernVL2-
InternVL2-
(a) 2
x
26B
VL.2
76B
(d) 0.5
x
Claude 3.5 Sonnet
GPT-4o
(a)
x
Gemini 1.5 Pro
Table 57: Examples of dataset and model responses for Syntheticgeomeeryy-rer (1)
72

---

Published as a conference paper at COLM 2025
Guld
Image
Question
Answer
EADJ times larger options is reasonable estimate? You only need to estimate from the visual information and not need any mathematical reasoning. 0.25 (b) (c) 0.5 (d) (e)
ABD. Which following
Model
Answer w/ CoT
Answer w/o
IntervVL2-
4B
VL2ernVL2-
ILtermVL2--
(b) 4
x
VL.2
(b) 4
x
76B
Claude 3.5
Sonnet
GPT-4o
x
Gemini Pro
Table 58: Examples of dataset and model responses for Ay (2)
73

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
AEB is times larger
BAC. Which of the following options is a reasonable estimate? You only need to estimate from visual information and not need any mathematical reasoning. 0.25 (b) (c) (d) 0.5 (e)
Answer CoT
Model
Answer CoT
In2erVLL--
InterVL2.-
x
8B
IntervLL--
(b)
26B
InterVLL2-
(b)
Claude 3.5
Sonnet
X
GPT-4o
(c)
Gemini
x
Table 59; Examples of dataset and model responses for Synthetiigeomeery-AAee (3)
74

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
The blue cylinder is X times taller than the red sphere. Which of the following options is reasonable estimate? (a) 2 0.5 (c)
Answer w/o CoT
Model
Answer w/ CoT
V1.2
I1ternV1.2.
b.
InterVLL-- 26B
(c)
/
LL2ernVL2-
(a)
76B
Claude 3.5 Sonnet
GPT-4o
x
Gemini 1.5 Pro
Table 60: Examples of dataset and model responses for 3D-Size (1)
75

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
The cyan cylinder times taller than the brown cylinder. Which of the following options reasonable estimate? (a) (b) 1 (c)
Answer w/o
Model
Answer w/ CoT
V1.2
I1ternV1.2.
(b)
InterVLL-- 26B
(c)
/
LL2ernVL2-
(o)
76B
Claude 3.5 Sonnet
GPT-4o
/
Gemini 1.5 Pro
Table 61: Examples of dataset and model responses for 3D-Size (2)
76

---

Published as a conference paper at COLM 2025
Gold
Image
Question
Answer
The yellow rubber sphere is times wider than the yellow cube. Which of the following options reasonable estimate? (a) (b) 0.5 (c)
Answer w/o
Model
Answer w/ CoT
V1.2
I1ternV1.2.
b)
InterVL2-- 26B
(a)
/
LL2ernVL2-
(+)
76B
Claude 3.5 Sonnet
GPT-4o
/
Gemini 1.5 Pro
Table 62: Examples of dataset and model responses for 3D-Size (3)
77

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Which of the following options reasonable estimate of the angle between the directions the gray motorcycle and the brown airplane in the figure? We define degrees the direction is the same and 180 degrees if the direction is opposite. (a) 180 degrees degrees (c) 0 degrees (d) degrees (e) 90 degrees
Model
Answer w/ CoT
Answer woT
InternVL2-
x
4B
LL2ernVL2-
88
IntervVL2-
(a) 180 degrees
26B
InternV1.22
(e) 90 degrees
Claude 3.5
x
Sonnet
GPTT-4o
Gemini Pro
x
Table 63: Examples of dataset and model responses for 3D-Angle (1)
78

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Which of the following options reasonable estimate of the angle between directions the blue car and the fighter jet in the figure? 180 degrees if the direction is opposite. (a) 90 degrees (b) 135 degrees (c) degrees (d) degrees degrees
define degrees direction the and
Model
Answer w/ CoT
Answer
InternVL2-
x
4B
LL2ernVL2-
135 degrees
88
IntervVL2-
(a) 90 degrees
26B
InternV1.22
(a) 90 degrees
Claude 3.5
X
Sonnet
GPTT-4o
Gemini Pro
x
Table 64: Examples of dataset and model responses for 3D-Angle (2)
79

---

Published as a conference paper at COLM 2025
Gold Answer
Image
Question
Which of the following options is reasonable estimate of the angle between the directions of airplane and the purple motorcycle in the figure? We define degrees if the direction is the same degrees (b) 180 degrees (c) 135 degrees (d) 45 degrees (e) degreus
degrees if direction is opposite. (a)
Model
Answer w/ CoT
Answer w/o
InternVL2- 4B
LL2ernVL2- 88
IntervVL2-
(a) 90 degrees
26B
InternV1.22
(a) 90 degrees
Claude 3.5
x
Sonnet
GPTT-4o
Gemini Pro
Table 65: Examples of dataset and model responses for 3D-Angle (3)
80


Reviews:
[{"review_id": "Efe7YfApMd", "reviewer": "Reviewer_PGab", "summary": "This paper proposes VisOnlyQA, a dataset designed for evaluating the geometric perception capabilities of large vision-language models (LVLMs). VisOnlyQA consists of 12 tasks for various aspects of geometric perception ability. The figures in VisOnlyQA are sourced from existing datasets or synthesized using existing codebases. The authors invested effort in designing the questions. Experimental results indicate that current LVLMs exhibit poor performance in geometric perception. Even after fine-tuning, a gap remains compared to human performance.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Research on the geometric perception capabilities of LVLMs is an important and fundamental problem.\n- The authors tested a wide range of LVLMs (20 models).\n- It's great to see the experiments regarding chain-of-thought to verify if reasoning ability is necessary.", "reasons_to_reject": "- Given that all figures come from either existing works or existing projects, I believe the authors should demonstrate a significant effort in designing the textual questions. However, the current writing does not provide sufficient detail to reflect such effort. Below, I list some potential questions that should be addressed. Additionally, there are likely other aspects the authors should consider to make the proposed dataset more convincing:\n  - What criteria were used to design the questions?\n  - How is the difficulty level of the questions controlled? How to control the difficulty level?\n  - What is the quality of the designed questions? Do they always make sense? I do not see any description of a quality verification process or filtering.\n- Given that the proposed dataset is challenging for most LVLMs, even after fine-tuning, I suggest that the authors break down the questions into several simpler ones to better identify the bottlenecks in LVLM performance. For instance, all examples in Figure 1 seem to be quite complex. What if the figures were simplified and the questions focused on very basic tasks, such as comparing two angles or the lengths of two lines? Do LVLMs truly understand the concept of degrees? It's important to investigate the root cause of their poor performance. Can LVLMs recognize very simple geometric shapes? Perhaps they can, but they might fail at comparison, calculation, or at combining multiple concepts to solve a problem. I believe this work requires deeper insight, rather than merely presenting an evaluation.", "questions_to_authors": "Please see above.", "extra_scores": {}}, {"review_id": "7VHx91sUvX", "reviewer": "Reviewer_EwqX", "summary": "The paper addresses a gap in evaluating how effectively Large Vision-Language Models (LVLMs) perceive geometric information in images, which includes basic properties like shape, angle, and size. The authors introduce \"VisOnlyQA,\" a dataset specifically created to evaluate LVLMs' geometric perception capabilities through 12 distinct tasks involving shapes, charts, chemical structures, and 3D shapes.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed benchmark is challenging for state-of-the-art LVLMs, including proprietary models, while humans achieve nearly perfect performance.\n\n- The authors have conducted extensive experiments to demonstrate the effectiveness of their proposed benchmark.", "reasons_to_reject": "- I wonder whether there is any selection bias in the evaluation setting. Since the answer formats in the proposed benchmark are typically True/False or Multiple-Choice Questions (MCQA), the authors need to clarify if the models tend to favor certain answers (e.g., consistently choosing \"True\" or option (a)) or if they genuinely reason about the provided images and queries.\n\n- This is a minor question: I would like to see additional results from more advanced models, such as Gemini-2.5-Pro and Qwen2.5-VL-72B.\n\n- As shown in Figure 4, there is a substantial portion of perception errors, which is understandable. However, I would appreciate further discussion on addressing these visual perception errors—specifically, what factors other than the LLM's capabilities (discussed in Table 5) might significantly influence performance.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gNnkwBDRLj", "reviewer": "Reviewer_v9QR", "summary": "This paper introduces the VisOnlyQA benchmark, which is designed to test how well LVLMs understand basic geometric properties, without requiring complex reasoning or background knowledge. The authors evaluate a wide range of models and find that even the strongest ones perform poorly on these tasks. They also explore whether adding more training data or using larger models helps, and provide useful findings.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well motivated and easy to follow. It raises an important and specific question about the ability of LVLMs to understand geometric information.\n2. The benchmark is carefully designed to focus only on geometric perception. By reducing the need for reasoning or background knowledge, it allows for a more focused and controlled evaluation.\n3. The experiments include a broad set of both open-source and proprietary models, along with ablation studies that explore different factors.\n4. The results show that many current LVLMs have trouble with basic geometric understanding, and that improvements through fine-tuning or larger model size are limited and vary by task.", "reasons_to_reject": "1. Although the paper shows that models perform poorly, it doesn’t go into detail about why current architectures struggle with geometric perception. A deeper analysis of possible bottlenecks—like transformer attention maps or patch sizes—would strengthen the paper.\n2. It’s unclear how much the vision encoder contributes to the task compared to the language model. One possible reason for the poor performance could be that vision encoders haven’t seen enough geometric data during training. It would help to run separate fine-tuning experiments on just the vision encoder or just the language model to better understand their individual roles.\n3. While the paper does try to balance real and synthetic data, the fine-tuning relies mostly on synthetic examples. It’s not clear if this creates a bias between training and evaluation, or how well the improvements would transfer to real-world tasks.", "questions_to_authors": "1. In the VisOnlyQA-Eval-Real results, InternVL2-26B performs worse than InternVL2-4B, even though it has a larger vision encoder. Could the authors explain this unexpected result?\n2. What is the confidence interval for each reported evaluation score? When the paper claims that performance is close to random guessing on certain tasks, what confidence level is being used to support that claim?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a benchmark for evaluating if vlms can understand geometric information highly relevant to figure understanding. For example, can models reason about angles, geometric relationships between objects, ect.  The dataset involves 12 distinct types of tasks, sourced from a combination of existing sources of images and new synthetic or human annotations on the question side. The paper provides a large analysis, showing there is significant room to improve, both for open source and api models. 

Two reviewers found the paper well motivated, having good breadth, and that the evaluation points out a genuine missing capability in existing systems. The AC strongly agrees, and especially in the contexts where figure understanding is important, this benchmark points out desirable lower level capabilities. 

One review somewhat disagreed about the value of the paper. They felt that if a benchmark finds models significantly lacking, it should be simplified until we have sufficient signal to diagnose the underlying problem. This would indeed be desirable and authors provided some more experiments in rebuttal on simplified data (although not enough to really diagnose why models struggle). The AC agrees this would desirable, but doesn't agree that all benchmarks that point out a significant shortcoming should also diagnose the source of it.
Decision: Accept