GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training
Authors: Renqiu Xia
Mingsheng Li
Hancheng Ye
Wenjie Wu
Hongbin Zhou
Jiakang Yuan
Tianshuo Peng
Xinyu Cai
Xiangchao Yan
Bin Wang
Conghui He
Botian Shi
Tao Chen
Junchi Yan
Bo Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Geometry Problem Solving
Complicated Task Reasoning
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: This paper verifies that formalized vision-language learning is beneficial to learn informative representations for Geometry Problem Solving (GPS), enhancing the interpretability of GPS and the correctness of the solution process.

Abstract: Despite their proficiency in general tasks, Multi-modal Large Language Models (MLLMs) struggle with automatic Geometry Problem Solving (GPS), which demands understanding diagrams, interpreting symbols, and performing complex reasoning. This limitation arises from their pre-training on natural images and texts, along with the lack of automated verification in the problem-solving process. Besides, current geometric specialists are limited by their task-specific designs, making them less effective for broader geometric problems. To this end, we present GeoX, a multi-modal large model focusing on geometric understanding and reasoning tasks. Given the significant differences between geometric diagram-symbol and natural image-text, we introduce unimodal pre-training to develop a diagram encoder and symbol decoder, enhancing the understanding of geometric images and corpora. Furthermore, we introduce geometry-language alignment, an effective pre-training paradigm that bridges the modality gap between unimodal geometric experts. We propose a Generator-And-Sampler Transformer (GS-Former) to generate discriminative queries and eliminate uninformative representations from unevenly distributed geometric signals. Finally, GeoX benefits from visual instruction tuning, empowering it to take geometric images and questions as input and generate verifiable solutions. Experiments show that GeoX outperforms both generalists and geometric specialists on publicly recognized benchmarks, such as GeoQA, UniGeo, Geometry3K, and PGPS9k. Our data and code will be released soon to accelerate future research on automatic GPS.

Published as a conference paper at ICLR 2025
GEOX: GEOMETRIC PROBLEM SOLVING THROUGH UNIFIED FORMALIZED VISION-LANGUAGE PRE-
TRAINING
Renqiu Xia Mingsheng Li2.3,, Hancheng Ye², Wenjie Wu', Hongbin Zhou",
Jiakang 2.a Tianshuo Penga.1, Xinyu Cai², Xiangchao Yan", Bin Wang', Conghui He?, Botian Shi?, Tao Chen Junchi Yan 1.2, Bo Zhang
1 School of Computer Science & Artificial Intelligence, Shanghai Jiao Tong University,
2 Shanghai Artificial Intelligence Laboratory.
3 School of Information Science and Technology. Fudan University.
4 MMLab, The Chinese University of Hong Kong
Equal Contribution, B Corresponding Authors, Project Leader
ABSTRACT
Despite their proficiency in general tasks, Multi- Large Language Models (MLLMs) struggle with automatic Geometry Problem Solving (GPS), which de- mands understanding diagrams, interpreting symbols, and performing complex reasoning. This limitation arises from their pre-training texts, along with the lack of automated verification in the problem-solving Besides, current geometric specialists are limited by their task task-specific designs, making them less effective for broader geometric problems. To this end, we present GeoX, multi-modal large model focusing on geometric understanding and rea- soning tasks. Given the significant differences between geometric diagram-symber and natural image-text, we introduce unimodal pre-training develop a diagram encoder and symbol decoder, enhancing the understanding of geometric images and corpora. Furthermore, we introduce soometry-languagee alignment, an effective pre-training paradigm that bridges the modality gap between unimodal geomet- ric experts. We propose to generate discriminative queries and eliminate uninformative representations from unevenly distributed geometric signals. Finally, GeoX benefits from visual instruction tuning, empowering it to take geometric images and questions in- put and generate verifiable solutions. Experiments show that GeoX outperforms both generalists and geometric specialists publicly recognized benchmarks, such as GeoQA, UniGeo, Geometry3K, and PGPS9k. Our code is available at https://gii thu.. www./olllaaae Innovator Geex
natural images and process.
AnmeratorAAnd-amppll Transformer (GS-Former)
INTRODUCTION
Large Language Models (LLMs) (Touvron et al., 2023a; Ouyang et al., 2022) and their multi- modal extensions (MLLMs) (Liu et al., 2024; Chen et al., 2024; OpenAI, 2023; Anthropic, 2024) have demonstrated exceptional abilities to effectively handle wide range of general domain tasks, such as cross-modal retrieval (Caffagni et al., 2024; Zhang et al., 2023a; Wang et al.; Xia al., 2024a;b), visual question answering (Wu & Xie, 2024; Huang et al., 2025; Ye et al., 2022), and summarization (Bianco et al., 2023; Rotstein al., 2023). With the increasing focus on Artificial General Intelligence (AGI), both LLMs and MLLMs making inroads into specialized domains such as mathematics reasoning (Imani et al., 2023; Wang et al., 2024a;b), demonstrating promising performance improvements.
Plane geometry is a pivotal and unique branch of mathematics that requires the integration of multi- modal data well as knowledge from different scientific fields, such theorem proving (Trinh et al., 2024) and algebraic computation (Faulstich & Oster, 2024). However, developing AI systems to automatically solve geometry problems is challenging due the inherent complexity of both visual

---

Published as a conference paper at ICLR 2025
Two Examples Comparing our GeoX and GVT-4V
Formal vs. Natural Langunge for Geometry- Alignment
Gose 1.
Image
Longuage- wawagg---ggaee ligned
30
recmeee orde
and wich (2a0g) (130)
Therefone in tegle
Hull wis fiesio the we sigevets ii
No: e-st Juss las comment
(2/4) 3 2t (28/2/4) Anou megan 173 (2340es25rang
Asswer: 287
Quathinn Al-5, Ands C-507: find regin.
Our Geeli
V.ON.. Ginti
Geoeetric
56.89
Permel Longuage oa algreed
Line I WXX Viddet 2 insce X
(Gn Benor) (Comasik)) (Gonayllc)
aagnor aguf Wo W.
Baee on Herysss defertaii it
repar
Case 2.
6PT-44:
GeoX Geometric Problem Solving
andk bot aiice lis froe O crumme
oc rouc cb. Thene eos laod CAO would ln will The
Multi
Pretromed
Autenatic
Interpmatble
Facoirg at the comppordd cormal engie Shan S000iii
Questian, The center 4ORO call near, i the point langecy.
correct enluer froe the the nea glive lp
Gaupetticc
Beok
Reuo
Guanatric Siher
Fial
of A 67 Chroes 80.0, 20
Find V. ON 4uhe 9_3
Goomeins
Golver
Question
Langage))
Answeri
Token wrong resporse solution
Token correct final onswer
Taken: meaningless tokens in natural language
Figure I: Highlights of GeoX: 1) Comparison between GPT-4V (OpenAI, 2023) and GeoX: GPT-4V often fails to provide the expected results or solving approaches. Besides, verifying GPT-4V's solutions is labor-intensive, requiring expert knowledge and step-by-step analysis. 2) Comparison between formal and natural (informal) language: Unlike existing works (Gao et al., 2023; Zhang et al., 2024) that use natural language, we advocate for formal language due to its effectiveness and verifiability, making more suitable for geometric tasks. 3) GeoX solves geometric tasks a unified format by taking geometric images and questions as input, generating verifiable program sequences, and performing solving with a solver.
and language modalities. Previous works (Peng et al., 2023; Wu et al., 2024) rely on additional detection models and make decisions based on manually crafted rules, but are often criticized for their complexity (Zhang et al., 2023b). On the other hand, NGS (Chen et al., 2021), Geoformer (Chen al., 2022), and PGPSNet (Zhang et al., 2023c) focus on predicting program sequences, yet they often suffer from poor adaptability due to their speccsppiillzed model designs and limited ability in modeling complex geometric diagrams and problems.
Although MILMMs (Shi et al., 2024; Lu et al.) have made significant progress in multi-modal mathematical reasoning, their performance still lags behind that of specialized geometry models. Notably, they sometimes exhibit an interesting phenomenon where they generate a correct answer accompanied by an incorrect solution process or solving approach, as shown in Fig. 1. Besides, we observe that using natural language to describe geometric diagrams introduces a significant amount of redundant information. In contrast, formal descriptions are more concise and clear, providing necessary information about symbols, shapes, numbers, and their relationships, making them better suited for geometric multimodal pre-training. To this end, we argue that effectively leveraging multimodal information from both visual and textual sources through formalized pre-training is meaningful in mitigating the challenges that MLLMs face when solving geometric problems.
However, combining visual and symbolic information for pre-training boost the ability of GPS is challenging, due to the following two reasons: 1) Large Domain Gap for Geometric Understanding. Prior works (Gao al., 2023; Shi al., 2024) adopt a frozen CLIP ViT (Radford et al., 2021) as the diagram encoder, which is trained on natural images rich in colors and textures. However, geometric diagrams are usually monochrome, composed of elements like lines, shapes, and symbols. exhibiting significant domain discrepancy. 2) Uninformative Representations for Geometric Reasoning. In geometric images, useful information concentrated specific areas, while other regions are uninformative and considered noise. The inability to handle this uneven distribution of geometric information leads to suboptimal performance.
To address these challenges, we propose GeoX, geometry-centric large model that can comprehend geometric problems and solve geometry tasks in a unified formulation. To this end, we propose a formalized training scheme that consists of three progressive stages: unimodal pre- trainingg formalized geometry-languee alignment, and visual instruction tuning. In the first stage, as introduced in Sec. 3.2, we focus on integrating visual encoder with prior knowledge of geometry by masked auto-
2

---

Published as a conference paper at ICLR 2025
encoding. At the same time, train a geometric decoder in an auto-regressive manner to enhance its comprehension of the geometry language, which is interleaved with numbers, symbols, and words. Furthermore, solving geometric problems often requires not just recognizing shapes or symbols but also reasoning their interactions and implications. Thus, as described in Sec. 3.3, we introduce geometry-language alignment, which utilizes formalized descriptions instead of natural language captions, offering a new perspective to effeetively align geomelry-semant features. We present Senerator-and-Samplee Transformer (GS-Former), capable of generating geometry content-aware queries and removing uninformative representations under the guidance of semantic learning. In Sec. 3.4, enable GeoX to generate solutions based on the input geometric problem and image, we adopt end-to-end visual instruction tuning to obtain the ultimate model. Furthermore, in Appendix A, we theoretically explain why the proposed formalized pre-training more effective in GPS tasks.
In Sec. 4, we conduct extensive experiments on four widely recognized benchmarks to evaluate GeoX's ability reasoning complex and diverse geometrie problems, where our approach achieves state-of tftt--rr- results. Insightful analyses and ablation experiments are performed to further validate the effectiveness of our method.
Our contributions can be summarized as follows:
Our study reveals the large potential of formalized visual-language pre-training in enhancing geometric problem-solving abilities. To enable the formalized pre-training, we propose GeoX, aiming to build geometric generalist models by modeling geometrie tasks into unified formulation.
We analyze the unique challenges in the field of geometry problem solving and propose GS-Former, which effectively bridges the modality gap between geometric diagrams and formalized language.
Compared with previous generalist and specialized models, our GeoX achieves competitive performance on GeoQA, UniGeo, Geometry3K, and PGPS9K, further demonstrating GeoX as strong baseline model for solving geometric problems and motivating future research.
2 RELATED WORKS
Multi-modal Large Language Models. The past year has witnessed the notable success of Large Language Models (LLMs) families (Ouyang et al., 2022; Touvron et al., 2023a;b; Team, 2023), showcasing near-human performance across diverse tasks. Concurrently, researchers have made significant efforts to extend the abilities of LLMs in handling visual-related tasks, contributing to the flourishing of Multimodal Large Language Models (MLLMs) (Bai et al., 2023; Achiam et al., 2023; Reid al., 2024). MILMS typically adopt cross-modal projector as the bridge to reconcile the modality gap between visual encoder and L.M,, such as Q-former (Li et al., 2023b) or linear layers (Liu et al., 2024). Although MLLMS have demonstrated impressive performance in conventional vision-language tasks (Han et al., 2024; Xia et al., 2023; Li et al., 2023c), they yield unsatisfactory results when addressing multimodal mathematical problems involving geometric diagrams and symbols. Besides, G-LLaVA (Gao et al., 2023) and MAVIS (Zhang et al., 2024) train LLM on the constructed geometry datasets with descriptions in natural language form. Recently, Chimera (Peng et al., 2024) proposes using the general-expert collaboration masking method to effectively integrate expert knowledge into a general MILMSs However, as illustrated in Fig. 1, these works face two issues: 1) unable provide the answer required, and 2) incorrect solving steps that still result in correct answers. Furthermore, verifying the solving process of MLLMs is extremely costly since it requires human experts from geometric knowledge and step-by- seep-by-step by-step examination. To this end, we propose GeoX, which solves geometric tasks in a unified formulation and predicts verifiable solutions.
Geometry Problem Solving (GPS) is hingsttadding yet challenging task in mathematics, requiring models with the ability to understand geometric elements and reason with logic. Existing automatic systems for GPS fall into two categories: rule-based approaches and neural approaches. Rule-based approaches (Seo et al., 2015; Sachan & Xing, 2017; Lu et al., 2021; Peng et al., 2023; Wu et al., 2024) rely external tools like OCR to parse diagrams into texts, which are then used for logical reasoning based on path search and condition matching. Although these methods have shown satisfactory performance in GPS, they are heavily dependent on manually crafted rules, making them difficult.
3

---

Published as a conference paper at ICLR 2025
to generalize to diverse geometry scenarios. Neural approaches networks to predict solving steps via program sequences, which are then executed by the solver. For example, NGS (Chen et al., 2021) and Geoformer (Chen et al., 2022) introduce auxiliary self-supervised tasks to refine diagram representations, with experiments on GeoQA (Chen et al., 2021) and UniGeo (Chen et al., 2022) demonstrating the effectiveness of their methods. Other methods, such PGPSNet (Zhang et al., 2023c) and LANS (Zhang et al., 2023b), integrate structural and semantic clauses into solving process and utilize specially designed decoders to achieve better performance on both Geometry3K (Lu et al., 2021) and PGPS9K (Zhang et al., 2023c). While these geometry specialists have shown impressive performance, their uniquely designed models for specialized datasets limit their ability to solve broader geometric tasks. In contrast, we introduce the unified formalized vision-language pre- training for general geometric tasks, achieving superior results across diverse benchmarks compared to previous methods on GPS.
3 FORMALIZED VISION-LANGUAGE PRE-TRAINING
3.1 METHOО OVERVIEW
To tackle complicated plane geometry problems, we introduce GeoX, adopting a formalized pre- training scheme consisting of three progressive stages, as illustrated in Fig. 2.
Unimodal Pre-training. Vanilla generalist models (OpenAl, 2023; Anthropic, 2024; Team al., 2023; Bai et al., 2023; Peng et al., 2024; Chen et al., 2024) have poor representation capacity in the geometric domain, due to the significant gaps between formalized data (e.g., informal text descriptions and natural images) and formalized data (e.g.. formal geometric symbols and scientific images). a result, we propose unimodal pre-training Sec. 3.2, aiming to enhance the GeoX's ability to understand geometric diagrams and symbols.
Ceometry-Languaae Alignment. To facilitate the aforementioned pre-trained unimodal models for performing cross-modal alignment, we propose effective Generator-and-Samge Transformer (GS- (GS-Former), which is trained using pairs geometric diagrams and formal language descriptions, as detailed in Sec. 3.3.
End-to-end Instruction Tuning, After geometry-language alignment, the ultimate model is required to generate solutions based on the given geometric problems and images. To this end, we tune GeoX in an end-to-end visual instruction tuning manner (as introduced in Sec. 3.4), boosting its capacity to comprehend geometric problems and generate formal solution programs.
During the inference phase, the solution generated by GeoX is fed into the symbolic solver (Chen al., 2021; Zhang al., 2023c), which performs step-by-step operations predict the final answer.
3.2 UNIMODAL PRE-TRAINING
Geometry Encoder. To mitigate the deficiencies of the existing visual encoders in comprehending geometric images, we collect more than 120K diagrams' from the web and electronic textbooks to equip VIT with prior knowledge of geometry, abbreviated Geo-ViT. Similar He et al. (2022), tune the vision encoder-decoder using the masked auto-encoding scheme, where some patches are masked and the remaining subset fed into the visual encoder, with the original image subsequently reconstructed by a lightweight decoder. In the next stages, we only utilize the pre-trained encoder to represent geometric diagrams.
Symbol Decoder. Considering the capability of LLMs follow users' instructions and handle different tasks, we utilize the decoder-only LLM as our symbol decoder to generate solutions. However, LLMs (Brown, 2020; Touvron al., 2023b) lacks the specialized learning for geometry. To this end, we build 100M-token geometric corpus based on the existing datasets (Chen et al., 2021: 1; Lu al., 2021: Gao al., 2023; Zhang al., 2023c; Chen et al., 2022; Cao & Xiao, 2022), containing a wide range of geometric problems, symbols, theorems, and so on. More details can be found in Appendix E. We choose LLEMMA-7B (Azerbayey et al., 2023) the base model, an open-source language model for mathematics pre-trained on Proof- Pile-2 (Azerbayev et al., 2023), and further fine-tune it on the geometric corpus using a standard ruto-segressive language modeling objective, resulting in Geo-LLM-7B.
typically trained on general text, which
/https:/ //huggi naceee.aaaee.oooo. co/ wtt.w//o/aaaaaaeeeeeeeeeee wtt.d/aaaasssseeueeeeeeeooo
4

---

Published as a conference paper at ICLR 2025
Stage One: Geometry Vision and Langunge Pre-trokting
Stage Two: Geometry Vision Language Aligmment
Pre-trokking 120k geunstry images for Geo-VIT
65-Former
Stage
Tange
SARLURGE Gongler
whide
Gez-
ViT
Geo- Deceder
Geo VIT
Oren Arsenn
Text Foon
Pre-training 100m-tokan GPs for Gea- -eo-1
Teit Briwaddig
Toxl Frrent
Lofggagg
Coprort
O
only
Stage
Three: Geometry Problem Solving Pinc-fure fnc-furning
Semanter ceaatte--gelor splledddd
Geemetry Sample
Frezen model wiihtts
Gou Ennie
Crbeading
Upirne
Tamoie
Trainable eudel weights
Get-- VIT
65- Former
BL-WTT-09
calnediing
Elenent-wise addition
Quenton Teur
Qurr
Graeddiy
Vaiee medaliity
prokinn
scheter
Lengugge medelity
Learnable eiery
Figure 2: Overview of GeoX for training. We present versatile method for automatic geometric problem solving through unified formalized vision-language pre-training, which comprises three progressive stages.
3.3
GEOMETRY-LANGUAR ALIGNMENT
3.3.1 DATA ENGINE
While recent datasets (Gao et al., 2023; Zhang et al., 2024) have made strides in captioning geometric images using natural language, they often result in redundant information, as depicted in Fig. 1. In contrast, approach emphasizes the of formal deseriptions to encapsulate the spatial structural information within geometric images. This information is implicitly represented, not explicitly stated in the problem text. Our curated dataset' focuses on capturing the essence of geometric imagery by detailing the relationships between the most fundamental elements (points) without explicitly annotating higher-level constructs such as squares or triangles, which can either be inferred from the relationships we describe or are directly provided in the problem text.
Our formalized diagram-cappee dataset delves into the spatial relationships a granular level. starting with the basic building blocks of geometric images. We identify and describe the relative positions and connections between points, ensuring that the spatial relationships are accurately represented. These relationships
categorized into two primary types: 1) Collinear Relationship (e.g., line A B C signifies that points A, B, and are on the same line) and 2) Concyelic Relationship (e.g., Mooot 1 lieson A C denotes that points A, B, and C are the same circle with center O). The dataset encompasses 6232 geometric images sourced from the internet, meticulously annotated by team of 10 experts with comprehensive explanations of formalized diagram-caption pairs Appendix C.
period of 200 hours. Moreover, provide concrete examples along
3.3.2 GENERATOR-AND-SANE
TRANSFORMER
With the formalized languaee dataset, GeoX lears unified representation space for geometry and formalized language through the which includes a Geo-aware Query Generator (GQG) and a Multi-Modal Transformer.
-ampler Transformer (GS-Former),
Geo-aware Query Generator. Both Resampler (Alayrac et al., 2022; Li et al., 2023a) and Q- Former (Li et al., 2023b; Dai et al., 2023) extract visual features using set of static query tokens, which are randomly initialized and regarded as model parameters. However, these queries, which remain the same for different diagrams, fail to capture discriminative features unique to individual samples. Thus, we introduce the Geore Query Generator (GQG), which incorporates contextual information to dynamically generate queries.
5

---

Published as a conference paper at ICLR 2025
To be specific, GQG utilizes visual features from the encoder and aggregates contextual information through an attention- based module and pooling operation. The contextual features then are projected and added with learnable queries (Li et al., 2023b), which builds connection between the learnable queries and the geometric content. Our empirical results demonstrate the effectiveness of GQG, resulting in improved performance.
Multi-Modal Transformer comprises Nr. layers, each containing a self-attention block, cross- attention block, and a feed-forwart network. Queries within each layer initially interact with paired formal captions and are then fed into the cross coss-attentioo block to extract visual features.
To handle the uneven information distribution in geometric images as described in Sec. I, we introduce the Semantics mamaatttss-guiedd guided Geometry Sampler (SGS), which dynamically removes uninformative visual representations guided by vision-language alignment. Specifically, SGS tasked with predicting a binary mask M {m]) i E K E N,, with each m E {0, 1} determining whether to retain or discard visual representations. Here, K represents the layer number and N denotes the number of patches. This module receives the previous mask M and visual features inputs, using linear layer to obtain retention probabilities Pt. To enable differentiable sampling from probabilities, we use the reparameterization (Jang et al., 2016) with Gumbel-Softmass
M M C (ounbel-Softmax("))
(I)
where O is the Hadamard product, i and i 1 represents the previous stage and current stage. A notable feature of our GS-Former is its capability to progressively drop noisy and semantically irrelevant features under the guidance of geometric language alignment. This is achieved by initially setting all elements of the decision mask to 1, followed by inserting the SGS block at subsequent layers. Additionally, GS-Former is initialized with weights from pre-trained BERT models (Kenton & Toutanova, 2019), except for the SGS and cross-attention layers, which are initialized randomly.
Inspired by BLIP-2 (Li et al., 2023b), we introduce a multimodal alignment loss Latign to optimize GS-Former, incorporating three training objectives: Geometry- Contrast and Geometry- Text Matching, both designed to align features between geometric diagrams and formal text, along with Geometry Caption Generation, aimed at generating formal captions based on visual information. We further impose sparsification term Eepr into the overall optimization objective to prevent trivial solutions where all mask values m) are set to
Cp = Lutign + AaSpes where Capr
Σ FE FERJENN
|m/|,
(2)
KN
3.4 END-TO-END VISUAL INSTRUCTION TUNING
To enable the model to handle geometry-centric tasks, we continue the training with end-to-end visual instruction tuning, directing the ultimate model to generate solutions. As illustrated in Fig. 2, we feed the diagrams into the pre-trained Geo-Vit together with GS- GS-Former, to obtain the semantically aligned geometry features Fy. Besides, utilize trainable projection head W to project F, into the language embedding space and obtain visual tokens Tg. Geo-I.LM,, serving as a decoder for various geometry tasks, takes both visual tokens Ta and instruction tokens T as input, and generates solutions in an auto-regressive manner. Our training objective is to optimize the GeoX so that the likelihood of the target sequence S = { is maximized given the visual input T, and instruction Tp.
In practice, GeoX is trained using cross-entropy loss L, defined as follows, which optimizes the model predict the (-th token it given preceding token sequences 81,11[1:11]-
C. Σ log P(s:
(3)
4 EXPERIMENTS
4.1 DATASETS, METRICS, AND IMPLEMENTATION DETAILS
Datasets. To assess the effectiveness of GeoX, we conduct experiments on four widely recognized geometry benchmarks: GeoQA (Chen al., 2021), UniGeo (Chen et al., 2022), Geometry3K (Lu

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of various methods the GeoQA benchmark with different accuracy metrics.
Methods
Texal
Angle
Tangth
Methods
Mousc
Total
Angle
Length
Ceneralints
Specialitty
aPPLG44001 Oie ol. 2023)
16.0
16.5
15.9
LLaVA-vi. vi. (Liu et. 2025))Seeeee
29.2
40.5
15.9
LLaVA-w1.5 (Lin all 2024)
20.7
20.9
19.8
FiLM|Percz 2018)
31.7
34.0
29.7
Owen VL (Bai et 2013)
24.4
23.7
24.4
RNiSannoo 2017)
38.0
42.8
32.5
GPT-4V [OpenAI, 2023)
41.4
90.3
42.8
MCAN Ya 20199
59.7
45.0
34.6
Specialias
Top-1
BERT (Kenton Toutsnova, 2019)
10
54.7
65.8
42.1
LLaVAwr1.5
2024es
9.4
14.9
3.2
NGS/Chen 2021)
56.9
69.8
39.
CD8 al. 2021)
46.3
Geofomert(hen
49.1
T5rLinng et 2003)
DPE-NDSSCan Xiao,
47.7
Lang et al., 2023)
SCA-GISSNNng 2023)
50.
(Uurs)
54,9
45.2
GeoX (Oins)
Table 2: Comparison of model performance on UniGeo for geometry calculation and proof problems.
Calculation %) Angle
Proving (%)
Methods
Metric
All
Length
All
Por.
Tr..
Qua.
Con.
Sio..
Generalialy
mPLUG-OwI2 O% 2023)
18.7
18.7
19.1
LLaVA-wL.5
2024)
26.4
Qwen-Vi. (Bai et ol., 2023)
24.4
24.2
25.4
GPT-4V (OpenAl, 2023)
47.9
45,8
51.6
Specialiats
Top-1
LLaVA-v1.5 (Liu et al. 202leer
16.1
19.2
13.1
1.0
0.0
1.1
0.4
0.2
Genformer (Chen et all, 2022)
46.8
57.8
35,0
51.3
13.9
63.8
20.4
56.1
64.0
UniMath (Linng
82.9
UniMath-Flan- TS-basc (Liang et al., 2023)
83.0
GooX (Ours)
54.4
63.1
43.1
97.8
77.8
100.0
95.4
99.5
99.2
Specialiste
LLaVA-v1.5 (Liu et al. 20244Sollver
43.0
51,3
35.3
11.3
0.0
16.2
5.0
2.9
27.5
BERT (Kenton & Toutaneva, 2019)
$2.0
63.1
39.2
48.1
15.4
48.0
31.7
49.5
75.1
NGS (Chen al., 2021)
Top-10
$1.9
63.6
38.8
47.4
11.2
46.9
31.3
48.3
77.6
Gooformer (Chon et al. 2022)
62.5
75.5
48.8
56.4
19.4
69.4
20.4
60.3
75.0
GooX (Ours)
68.6
76.7
58.3
99.5
97.2
100.0
97.7
100.0
100.0
et al., 2021), and PGPS9K (Zhang et al., 2023c). GeoQA comprises 4,998 geometry problems sourced from Chinese middle school exams, including different types of problems, such as angles, lengths, and areas. Following Liang et al. (2023); Gao et al. (2023), we use the English version to maintain linguistic consistency with other datasets. UniGeo features 4,998 calculation problems from GeoQA and 9,543 proving problems from high school textbooks and online resources, providing a comprehensive benchmark for evaluating geometry reasoning abilities. Both Geometry3K and PGPS9K include high-quality diagrams and detailed annotations.
Metrics. We adopt the same evaluation metrics used in previous studies to ensure fair comparability, Following Chen et (2021) and Chen al. (2022), we assess the model's s performance on GeoQA and UniGeo with top-1 and top-10 accuracies. For evaluation on Geometry3K and PGPS9K, we apply three metrics assess the performance of GeoX: Completion, Choice, and Top-3, as introduced in Zhang et al. (2023c). To evaluate MLLMs in solving complex geometry problems, such as Qwen-VL (Bai et al., 2023) and GPT-4V (OpenAI, 2023), we follow LANS (Zhang et al., 2023b) by utilizing Completion (which requires models to provide answers directly) and Choice (which involves selecting from given options).
Implementation Details. We optimize the diagram encoder using MAE VIT-B (He et al., 2022) checkpoints, training it for 800 epochs with a batch size of 256 and an initial learning rate of 6.4e-5. We initialize the symbol decoder with LLEMMA-7B (Azerbayev et al., 2023) weights and train for 5 epochs with batch size of 32 and an initial learning rate of le-6. For geometry-language aligmment, we train the GS-Former for 360 epochs with a batch size of 256 and an initial learning rate of 1c-4. The number of queries in GS-Former is set to 8. Additional details regarding visual instruction tuning can be found in Appendix F. We implement GeoX using PyTorch and conduct experiments more than eight A 110 (80GB) GPUs. During inference, we employ beam search size of 10, consistent with Zhang et al. (2023c) and Chen et al. (2021).
4.2 COMPARISONS WITH STATE-OF-THE-ART METHODS
Performance Comparison with Generalist Models. As to multimodal large models, LLaVA- v1.5 (Liu et al., 2024), mPLUG-OW12 (Ye et al., 2023), Qwen-VL (Bai et al., 2023), and GPT- 4V (OpenAI, 2023) exhibit strong cross-modal reasoning abilities for general tasks. However, when applied to solve geometry tasks, these models are insufficient. Our GeoX significantly outperforms these generalists on various geometry datasets, including GeoQA (Chen et al., 2021), UniGeo (Chen et al., 2022), Geometry3K (Lu et al., 2021), and PGPS9K (Zhang et al., 2023c). As indicated in
7

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparison Geometry3K 3K and PGPS9K.
GoometryåK
PGPS9K Choice
Methods
Completion 1 Choice
Top 31
Completion
Top
Generalists
mPLUG-OwW2 (Ye 2023)
2.2
26.7
3.0
26.4
LLaVA v1.5 (Liu et
2024) 2024)
2,9
22.9
1.8
21.8
Qwen-VL (Bai
2023)
27.5
1.4
24.7
GPT-4V (OpenAl, 2023)
34,8
58.6
33.3
51.0
Specialists
LLaVA- v1.5 (Liu et al., 2024y+Solver
19.7
47.4
31.6
21.6
38.1
35.3
GeoDRL (Peng et 2023)
68.4
NGS (Chen et al., 2021)
35.3
58.8
62.0
34.1
46.1
60.9
Geoformer (Chen
2022)
InterGPS (Lu et al., 2021)
44.6
56.9
PGPSNet (Zhang et al., 2023c)
48.1
70.1
63.7
44.4
57.6
64.8
GeuX (Ours)
58.6
72.5
69.4
52.7
63.3
65.4
Tab. and Tab. 2, GeoX achieves top-1 accuracies of 54.9% and 54.4%, respectively, significantly outperforming the best generalist models. Similarly, on Geometry3K and PGPS9K in Tab. 3, GeoX achieves 58. 6% and 52.7% in Completion, respectively. In comparison. GPT-4V (OpenAI, 2023) achieves 34.8% and 33.3%, while other models such as Qwen- (Bai al., 2023) and LLavV (Liu et al., 2024) perform worse
Performance Comparison with Specialist Models. Compared with geometry specialists such NGS (Chen al., 2021), UniMath-T5 (Liang et al., 2023), Geoformer (Chen al., 2022), DPE-NGS (Cao & Xiao, 2022). and SCA-GPS (Ning et al., 2023), GeoX demonstrates superior performance across GeoQA and UniGeo. Specifically, GeoX surpasses the best geometry specialist by +4.9% and +7.6% on GeoQA and UniGGe-Calcculaton,, respectively. Additionally, our model achieves significant improvements over previous methods on UniGeo--rovving by +14.8% and +43. % Tab. As reported in Tab. 3, our method outperforms SOTA models on Geometry3K and PGPS9K. Notably, previous works (Zhang et al., 2023c;b) require additional image annotations (Diagram GT) as input, which is labor-consuming and contrary experimental settings. To make fair comparison, remove Diagram GT and replicate these methods under the original conditions. Particularly, we fine- tune LLaVA (Liu et al., 2024) with formal language and adopt solvers for problem-solving. consistent with the approach used in GeoX. Extensive results in Tabs. to 3 demonstrate the effectiveness of GeoX, achieving state-of-the-an performance across diverse scenarios.
Besides, should be noted that G-LLaVA-7B (Gao et al., 2023) and MAVIS (Zhang et al., 2024) achieve 64.2% and 66.7% accuracy on GeoQA. However, these models can produce correct results despite errors in the solving process. In contrast, our method treats any process errors as incorrect results. To this end, we introduce a comparable metric, with detailed results provided in Appendix D.
4.3 QUANTITATIVE EVALUATION ON THE GPS TASK OF MATHVISTA
We provide quantitative comparison with the model that performed best on the GPS task in Math Vista (Lu et al.). To this end, we extract the Geometry subset from Math Vista, referred to as Math Vista-GEO. We assess these methods using the same evaluation script Math along with the evaluation strategy introduced in Appendix D. As reported Tab. 4, GeoX more effective in solving geometry tasks.
Table 4: Accuracy scores on testmini of Vista-GGGGG
Methods
Accuracy
GPT-4V (OpenAI, 2023)
54.8
GPT-4o (OpenAL, 2024)
66.1
GeoX (Ours)
72.6
4.4 INSIGHTFUL ANALYSES
Effectiveness of Uni-modal Pre-training. We compare Geo-ViT with CLIP-VIT (Radford et al.,
2021), which has been widely used for GPS in previous studies (Gao et al., 2023). Additionally, evaluate the performance of different language models in solving geometric problems, including ILAMA-2-775 I.EEMA-7BB, and Geo-I.LM-7B. As reported in Fig. 3, compared to general- purpose models or the mathematical model, our pre-trained model demonstrates superior results across various geometry benchmarks.
Effectiveness of Geometry-Language Alignment. As illustrated in Tab. 5, without multi-mon modal feature alignment, the baseline model perform poorly, achieving only 33.1% Completion on Geome- try3K. The introduction of GS-Former significantly boosts performance. Moreover, our results reveal
8

---

Published as a conference paper at ICLR 2025
Geo-VIT
11AMA-2
LEAMA-2
Gen-V
LLaMMA
30
Gee-ILM
Pefformazee
GeameleyKK
.coOA
UniCGo
Geometry3K
Benchmark
Benchmark
Figure 3: Effectiveness of Uni-modal Pre-training. We compare the widely used CLIP-ViT-H and our Geo-Vite ViT-B, along with three LLM models: LLAMA-2-7B, LLEMMA-7B, and our Geo-LLM-7B.
Imagel
Quasitin:
4Fth CNCA <BbC. 10
For aandlliooon ABCo FA parp FC on serp
20r
Geomerryak
POPSSX
Arewer GT:
Anmvvr GT:
Geet Pred:
Asseer predi
SeoX Predi
Sun ant4 20 x 42 360.
FRX Ponn 13 9.4 VO
71
6488
Image:
Questioes
Ieage:
Question:
APl and UP ane tregent to circl O points respectively engle psintt
GeoQA
Co
UniGeD
ayaar Fobo
Answer
50.0
Asswer
8D
Geal Fred:
pred:
Geal Pred:
Answer pred:
180 60
g_mml V_0
Figure 4: Visualization results on four datasets by our GeoX.
that formal language is more effective for GPS than natural language, with +2.9% improvement in Completion on GeOmetry3K.
Table 5: Effectiveness of geometry-languag alignment.
Geomeny3K
PGPS9K
Module
Alignment
Language
Completion
Choice
Top 33
Comgletion
Choice
Top - 37
33.1
$4.0
48.2
31.5
43.6
50.1
48.6
65.7
63.2
42.7
54.3
56.8
GS-Former
Natural Formal
55.7
71.5
67.2
52.2
62.2
67.1
58.6
72.5
69.4
52.7
63.3
65.4
Ablation of Modules in GS-Former. The results in Tab. 6 demonstrate the effectiveness of the Geo-aware Query Generator (GQG) and Semantics-guided Geometry Sampler (SGS) within GS- Former. Adding the GQG improves Completion by +2.4% and +1.0%, while combining both designs yields the best performance. The quantitative results in Appendix further demonstrate GS- Former's effectiveness in capturing valuable information from geometry diagrams, such as lines and symbols.
4.5
CASE STUDY
As shown in Fig. 4, we conduct case study to analyze the capabilities of GeoX. GeoX tries to predict formalized program sequences composed of mathematical variables, constants, and operations, such as summation (Sum), subtraction (g ___mminme nus), perimeter calculation (PRK_Perim) theorem mnnu) etc., which can be compiled and solved by the GPS-solver.
Perim), the Pythagorean
Furthermore, we have conducted the generalization validation of GeoX in a broader scope, including its application geometric problem-solving from natural images. Our GeoX has demonstrated promising performance in these scenarios, indicating the potential for its generalization to even wider fields. We present some visualized examples in Fig. 5.

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study of modules in GS-Former, assessing the contribution of GQG and SGS modules when GS GS-Former is utilized for geometry-formal language alignment.
Goo-aware
Semantics-guided
GeometryKK Choice
PGPS9K Choice
Query Generator
Geometry Sampler
Completion
Top 31
Completion
Top
55.0
70.3
68.3
49.8
59,9
64.6
57.4
71.7
68.1
50.8
62.0
64.3
52.7
Image:
Question:
Imagei
Question
birthday thee siore heght 30 the laterdl surfoce of
wnde leight sn'll tht wich setrrs tire
Birthday Hat
Revolving Door
of button the colene to botrer of curne
Answer GT:
<33)
GeoX
Arswer predi
GeoX Pred:
Answer predi
2.holf N_D
col_cone N
1.73
gougLLmnn NL2 V_D
Imoge:
Question:
Image:
fobkible square Ghean AC-R0-t0en CD-D0-30en, hoble is now lad Hatt to make the hoblettp high from the ground, the angis betweer the
showt figure
rign and Boyran
Eac io the graand paitt an cill any
Foldable Table
Boutbo Weu dhey tel hell you used orlere
Bicycle
this
Answer GT:
24
GeoX Pred:
Answer pred:
GeoX Pred:
Answer predi
9.minus 0_3
doubie N_2
2_divide V.D N_6
Suhinus V_D
giminu V.NN
Figure 5: Four visualized examples of geometric problem in natural images solved by our GeoX.
CONCLUSION, LIMITATIONS, AND FUTURE WORK
In this paper, have proposed GeoX, novel multi-modal large model specifically designed for automatic Geometry Problem Solving (GPS) tasks. GeoX verifies that formalized vision- langugee learning beneficial learn informative representations for automatic GPS tasks. GeoX can produce formalized process descriptions, which enhance the interpretability of GPS and the correctness the solution process. Besides, extensive experimental analyses demonstrate GeoX's general capabilities on multiple geometric datasets.
ACKNOWLEDGEMENT
The research was supported by Shanghai Artificial Intelligence Laboratory, the National Key R&D Program of China (Grant No. 20222D0160104), the Science and Technology Commission of Shanghai Municipality (Grant No. 22021100102), the National Natural Science Foundation of China (Grant No. 92370201 and 62222607), and Shanghai Rising Star Program (Grant No. 23QD 23QD1401000).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXhv:2303.08772 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in neural information processing systems, 153:2311--23133 2022.
Anthropic. The claude model family: Opus, sonnet, haiku. httpss//ww.anthropicccoo 2024.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Mareus McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
Simone Bianco, Luigi Celona, Marco Donzella, and Paolo Napoletano. Improving image captioning descriptiveness by ranking and unn-asee fusion. arXiv preprint (rxxi:2306.11199) 2023.
Tom B Brown, Language models are few-shot learners. arXiv preprint ArXiv:2005. 2020.
Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornaa, Lorenzo Baraldi, and Rita Cucchiara. Wiki-llava: Hierarchical retrieval- retrieval-augmented generation for multimodal 1lms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1818-1826, 2024.
Jie Cao and Jing Xiao. augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational 2022.
Linguistics, pp.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 513-523, 2021.
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 3313-3323, 2022.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far we gpt-4v? closing the gap to commercial multimodal models with open-source suites, arXiv preprint arXiv:2404. 16821, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
Fabian M Faulstich and Mathias Oster. Coupled cluster theory: Toward algebraic geometry formulation. SIAM Journal on Applied Algebra and Geometry. ((1):138-188, 2024.
Keinosuke Fukunaga. Introduction statistical pattern recognition. Elsevier, 2013.
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312. 1 2023.

---

Published as a conference paper at ICLR 2025
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2024.
Yihan Hao, Mingliang Zhang, Fei Yin, and Lin-Lin Huang. PgdpSk: A diagram parsing dataset for plane geometry problems. In 2022 26th International Conference on Puttern Recognition (ICPR), pp. 1769-1769. IEEE, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free nigh-performance model merging. Advances in Neural Information Processing Systems, 37: 37:122741-122769 2025.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05398 2023.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with gumbel-softmax, arXiv preprint arXiv:1611.011 114.01141110 2016.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert:
Pre-training of deep
bidirectional transformers for language understanding. In Proceedings of NAACL-HILT, pp. 4171- 4186, 2019.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.00425, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730- 19730-19742. PMLR, 2023b.
Bootstrapping language-image
Mingsheng Li, Xin Chen, Chi Zhang, Sijin Chen, Hongyuan Zhu, Fukun Yin, Gang Yu, and Tao Chen. M3dbench: Let's instruct large models with multi-modal 3d prompts. arXiv preprint arXiv:2312.10763, 2023c.
Zhihao Li, Yao Du, Yang Liu, Yan Zhang, Yufang Liu, Mengdi Zhang, and Xunliang Cai. Eagle: Elevating geometric reasoning through l1m- tim-empwerred visual instruction tuning. arXiv preprint arXiv:2408. 1197, 2024.
Zhenwen Liang, Tianyu Yang, Jipeng Zhang, and Xiangliang Zhang. Unimath: A foundational and multimodal mathematical reasoner. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp- 71 7126-7133, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. The Twelfth International Conference on Learning Representations.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Cong-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04166 2021.
Maizhen Ning, Qiu-Feng Wang, Kaizhu Huang, and Xiaowei Huang. A symbolic characters aware model for solving geometry problems. Proceedings of the 3 ACM International Conference Multimedia, pp. 7767-7775, 2023.
OpenAl. Gpt-4v, https: /ppeaal....../////////
2023.
12

---

Published as a conference paper at ICLR 2025
OpenAl. Hello http:s//ppeeaa..cc/////////e//e/
2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Shuai Peng, Di Fu, Yijun Liang. Liangcai Gao, and Zhi Tang. Geodrl: A self-learning framework for geometry problem solving using reinforcement learning in deductive reasoning. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13468- 13468-13480, 2023.
Tianshuo Peng, Mingsheng Li, Hongbin Zhou, Renqiu Xia, Renrui Zhang, Lei Bai, Song Mao, Bin Wang, Conghui He, Aojun Zhou, et al. Chimera: Improving generalist model with domain-specific experts. arXiv preprint arXiv:24122 22002223,. 2024.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint wrxiv:2403.05530 2024.
Noam Rotstein, David Bensaid, Shaked Brody, Roy Ganz, and Ron Kimmel. Fusecap: Lever- aging large language models to fuse visual data into enriched image captions. arXiv preprint arXiv:2305.177118 2023.
Mrinmaya Sachan and Eric Xing. Learning to solve geometry problems from natural language demon- strations textbooks. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017), pp. 251-261, 2017.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Advances in neural information processing systems, 30, 2017.
Minjoon Sco, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 conference on empirical methods natural language processing, pp. 14661116664 1476, 2015.
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng. Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.174 17294, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: family of highly capable multimodal models. arXiv preprint arXiv:23/2. 1805, 2023.
InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ana Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /397/, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307. 09288,, 2023b.
13

---

Published as a conference paper at ICLR 2025
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476 482, 2024.
Bin Wang, Zhuangcheng Gu, Chao Xu, Zhang, Botian Shi, and Conghui He. Unimernet: A univer- sal network for real-world mathematical expression recognition. arXiv preprint arXiv:2404. 15254, 2024a.
Bin Wang. Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. Mineru: An open-source solution for precise document content extraction, arXiv preprint arXiv:2409. 18839, 2024b.
Haoqing Wang, Xun Guo, Zhi- ZhngHong Deng, and Yan Lu. Rethinking minimal sufficient representation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16041-160055 2022.
Yabing Wang, Le Wang, Qiang Zhou, Hao Li, Gang Hua, Wei Tang, et al. Multimodal I1m enhanced cross-lingual cross- modal retrieval. In ACM Multimedia 2024.
Penghao Wu and Saining Xie. V Guided visual search as core mechanism in multimodal Ilms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13084-13094, 2024.
Wenjun Wu, Lingling Zhang, Jun Liu, Xi Tang, Yaxian Wang, Shaowei Wang, and Qianying Wang. E-gps: Explainable geometry problem solving via top-down solver and bottom-up up generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
13828-13837, 2024.
Renqiu Xia, Bo Zhang, Haoyang Peng, Ning Liao, Peng Ye, Botian Shi, Junchi Yan, and Yu Qiao. Structchart: Perception, structuring, reasoning for visual chart understanding. arXiv preprint arXiv:2309. 1268, 2023.
Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, et al. Docgenome: An open large-scale scientific document benchmark for training and testing multi-modal large language models. arXiv preprint arXiv:2406. 11633, 2024a.
Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangehao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, et al. Chartx & chartvlm: A versatile benchmark and foundation model for 2024b.
complicated chart reasoning, arXiv preprint arXiv:2402.
Peng Ye, Baopu Li, Yikang Li, Tao Chen, Jiayuan Fan, and Wanli Ouyang. b-darts: Beta-decay regularization for differentiable architecture search. In proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10874-10883, 2022.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou, mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.044 1.04257, 2023.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 628 1-290, 2019.
Bo Zhang, Jiakang Yuan, Botian Shi. Tao Chen, Yikang Li, and Yu Qiao. Uni3d: A unified baseline for multi-dataset 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9253-9262, 2023a.
Ming-LLang Zhang, Zhong- -Zi Li, Fei Yin, and Cheng Liu. Lans: layout-aware neural solver for plane geometry problem. arXiv preprint arXiv:2311. 2023b.
Ming-Liang Zhang, Fei Yin, and Cheng-Lin Liu. A multi-modal neural geometric solver with textual clauses parsed from diagram. arXiv preprint arXiv:2302.11009 2023c.
Renrui Zhang. Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, et al. Mavis: Mathematical visual instruction tuning. arXiv preprint arXiv:2407.0m 08739, 2024.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
The appendix mainly includes the following aspects:
Sec. A: Theoretical analysis on the proposed formalized vision-language pre-training
Sec. B: More visualization results.
Sec. C: Examples of formalized diagram-cappe pairs.
Sec. D: Additional quantitative evaluations.
Sec. E: Data acquisition for geometric corpus.
Sec. F: Implementation details.
Sec. G: Further discussions and analyses.
Codes are released at https: bttps://github. com com//Alphaa-moovvatrr/eeee including the pro- cess of the training and evaluation of GeoX.
A THEORETICAL ANALYSIS
In this section, we theoretically explain why the proposed formalized pre-training benefits more than informal pre-training methods in downstream tasks of geometry problems. First, consider the sufficient representations for the pre-training of the Geometric Problem-Solving (GPS) models, containing the information shared between different modalities of geometry data. The definition of sufficient representations borrowed and extended from the idea Wang et al. (2022), We denote T3 as the target formalized descriptions of samples in the pre- pre-training dataset, while Tinf is denoted as the informal descriptions of samples the pre-training dataset. The representations learned from Ty is denoted zy. while the representations learned from Tiny is denoted Zinf- The downstream task label is denoted as T,, which is a formalized textual sequence that will be fed into the GPS-Solver for verifiable numerical solutions.
Definition 1. (Sufficient Representations) The representations 2u of yı is sufficient for another task y2 if and only if (21 y2) I(y1, y2). where 21.s learned from y1, and yı, y2 are the labels of two different prediction tasks that contain the shared information. I(-)) refers to the mutual information between the two variables.
Definition 2. (Minimal Sufficient Representations) The representations min is minimal sufficient if and only if [221,min 822 min (21,adf Y2)-
Lemma 1. 2y provides more information about the downstream task T than Zinf- That is, I(zy, T) 2 "
Proof. Since both T, and Tinf supervised learning tasks, their learned representations zy and Zinf are both sufficient representations. However, since Tins only contains the semantic informa- tion without structural context that required by the downstream tasks. Therefore, holds that I < [z T), V2r that is sufficient. That is, 2inf is a minimal sufficient representation. As for 2y, it learns information from the formalized description and thus is more related to the downstream tasks. Consequently, we have the relationship between žinf and 2 as follows,
I(2,,T) T) [(inf, T) LITT,TT'Za, 2 T(zmn, T))
ITT,
(4)
The first equation indicates that the mutual information I [[y T') can be decomposed into the minimal mutual information 1 [ T') and the information gap between (TT Z and [ Tlzy). where I ITT T| f) refers to the information about T that can be observed from Ty condition of Zinf- Since Ty contains more formalized information related to T and [(zán Ty) ) s I(zy, Ty), we can get [TT, T 2 I[T,, Fl Consequently, [[2,, T) IZin,; T) holds.
Theorem 1. The upper bound of error rates in downstream tasks using minimal sufficient representa- tions is higher than that using sufficient representations.
15

---

Published as a conference paper at ICLR 2025
Proof. For the downstream tasks, consider the Bayes rate (Fukunaga, 2013) to estimate the lowest achievable error of the classifier. According to the paper (Wang et al., 2022), for arbitrary representations 2, its Bayes enror rate P. satisfies that,
Pe S exp[ H(T) + [2,,
(5)
where (T) represents the entropy of variable T. Since [zz, T) 2 [zinf. T), can be concluded that the upper- bound of fe, is smaller than that of Pe,ing- This indicates that ideally 2, is expected achieve better performance than 2in in downstream tasks.
B MORE VISUALIZATIONS
K
M
15 mm
x
24
30
15 mm
Lines
Rectangulars
Triangulars
Circulars
Figure Attention map of GS-Former on different types of geometric diagrams.
In Fig. 6, present attention maps of GS- GS-Former, which show the model's attention distribution across different regions. The areas with higher brightness indicate regions considered more useful for making decisions. In contrast, darker areas often semantically irrelevant and uninformative, which will be removed by GS-Former This visualization highlights our model's ability to capture pivotal information across complex geometric images, such as lines, rectangles, triangles, circles, etc.
C EXAMPLES OF FORMALIZED DIAGRAM-CAPTION PAIRS
Image
Caption
Image
Caption
Line À B
Line
Line 0
Line
Line 0
Line
Line A
Line
Line C
Line
Line u
Line
Line E
Wodot liegon C
Line B 0
tine
B
Line
Line D C
Line C D
D
Line 0 C Wedot
Line E A
87
Table 7: Four examples of our formalized diagram-caption pairs containing two relationships among points geometry images.
In Table 7, we provide additional examples of descriptions that delineate the collinear and concyelic relationships in geometric images at the granularity of points. It is noteworthy that we adhered to strict grammatical and standardization criteria during annotation process. Specifically, for collinear relationships, the term Line denotes the relationship. and the order of the points is listed from left to right. For concyclic relationships. the symbol \ \oodot signifies the center of the circle, lieson indicates the points on the circumference, and the points are listed in clockwise order.
16

---

Published as a conference paper at ICLR 2025
D MORE EVALUATIONS
Inspired by the Choice metric proposed by Zhang al. (2023c), we introduce an accuracy metric for GeoX to ensure complete fairness when comparing with solver-free methods like G-LLaVA (Gao et al., 2023). Specifically, we observe that even if errors occur in the solving process, solver-free methods can still provide answer (by randomly selecting one from four options), whereas our solver-based approach considers any process errors as incorrect results. To this end, in comparison with solver-free methods as shown in Tab. 8, we define GeoX's ' accuracy by assuming that, when the solution process encounters errors, the model's 's performance is equivalent to randomly selecting from four possible options. We also evaluate our method against solver-free approaches GeoQA (Chen et al., 2021). As shown in Tab. 8, our method outperforms the current state-of-the-an solver-free methods in Top-1 accuracy.
Table 8: Comparison with solver-free geometry specialists on GeoQA. We directly report results using Top-1 accuracy.
Methods
Base LLM
Accuracy
Math-LLaVA (Shi et al., 2024)
Vicuna-1. 1.5-13B
48.1
G-LLaVA (Gao et al., 2023)
LLaMA-2-7B
64.2
MAVIS (Zhang et al., 2024)
MAmmoTH-2-7BB
66.7
EAGLE (Li et al., 2024)
Vicuna- 1.5-7B
67.1
GeoX (Ours)
Geo-LLM LLM-7B
67.4
E DATA ACQUISITION FOR GEOMETRIC CORPUS
Data Sources. We detail the geometric corpus collections used to train Geo-LLM, sourced from a variety of publicly available geometric datasets, including GeoQA (Chen et al., 2021), (Chen+(Chen et al., 2021), UniGeo(Chen et al., 2022), PGDPSK (Hao et al., 2022), PGPS9K (Zhang et al., 2023c), Geometry3K (Lu al., 2021), and G-LLaVA (Gao et al., 2023).
GeoQA (Chen et al., 2021) comprises 4,998 real- world geometry problems sourced from Chinese middle school exams, each annotated with detailed solution processes and human performance metrics. The dataset is organized into three primary categories: angle, length, and other geometric calculations, and is divided into training, validation, and test sets at ratio of 7.0.1.:.1.5.
Geometry3K (Lu et al., 2021) provides 3,002 detailed geometry problems derived from high school textbooks, divided into training, validation, and test sets in a 0.7:0. 1:0.2 ratio. Geometry3K expands on previous datasets (Sco et al., 2015) by including irregular quadrilat- erals, polygons, and additional unknown variables and operator types. Moreover, less than 1% of Geometry3K problems can be solved without diagrams, making it more challenging.
GeoQA+ (Cao & Xiao, 2022) enhances the original GeoQA (Chen et al., 2021) by adding 2,518 newly annotated geometric problems, increasing the total to 7,528 problems with 6,027 dedicated for training. This expanded dataset introduces more complex problems, including area calculations, and raises the difficulty with 27 knowledge points and an average of 2.61 solving steps per problem.
UniGeo (Chen et al., 2022) introduces a comprehensive geometry dataset encompassing both calculation and proof problems, including 9,543 proving problems sourced from educational websites and 4,998 calculation problems from GeoQA (Chen et al., 2021). The proof problems categorized into five sub-tasks (parallel, triangle, quadrangle, congruent, and similarity) with detailed reasoning and expressions. To facilitate unified problem- solving, both proofs and solutions are reformulated into sequence formats, aligning the proving steps with calculation protocols.
PGDPSK (Hao et al., 2022) contains a total of 5,000 images, divided into training, valida- tion, and test sets with a 0.7:0. 1:0.2 split. It encompasses 16 geometric shapes, 5 positional relations, 16 symbol types, and 6 text types. PGDPSK provides detailed annotations, including geometric primitives, symbols, text types, and their relationships.
17

---

Published as a conference paper at ICLR 2025
PGPS9K (Zhang et al., 2023c) consists of 9.022 geometry problems paired with 4,000 unique diagrams, covering 30 problem types from grades 6-12 mathematics curricula. is split into training and test sets, with 8,433 samples for training and 589 for testing. PGPS9K includes detailed annotations for diagrams and solution programs.
G-LLaVA (Gao et al., 2023) is a large-scale multi-modal geometry dataset consisting over 110k question-ansye annswer (QA) pairs, divided into an alignment dataset to provide foundational geometric knowledge and an instruction-tuning dataset to improve the model's soblem-solving abilities. This dataset created with the assistance of GPT-API (Ouyang et al., 2022) using various strategies, including equation solving, value scaling, and sentence paraphrasing.
Data Collection and Filtering. To meet the demands of pre-training for Geo-LLMM, we build up specialized filtering and pre-processing pipeline to construct the geometric corpus. Initially, extract the data only from the training portions from the existing geometric datasets to prevent label leakage. Besides, we use free Translate-API convert Chinese data into English, ensuring language consistency. For each data entry, we remove content unrelated to geometric problems, such as annotation IDs, dates (Lu et al., 2021), I), and sources (Zhang et al., 2023c). Ultimately, we achieve collection of 100 million tokens of data.
F ADDITIONAL DETAILS
Prompts for MILMs. In Tab. 9, we provide examples of how to prompt multimodal large models to reason on geometric problems across two different evaluation modes. Each evaluation mode consists of several components: System Prompt, Diagram, Question, and optionally, Choices. The System Prompt specifies the type of problem the model required solve and the expected output format. The Diagram corresponds to the relevant image, while the Question and Choices are presented in the text. The key difference between the Choice and Completion modes is that Completion requires the model to provide answers directly, while Choice only involves selecting from predefined options.
Evaluation Versions for Generalists. In Tab. 10, we present the model / API versions utilized for the evaluation of generalists reported in Tabs. to 4. These include MLLMs such as mPLUG-Ow12 (Ye et al., 2023), Qwen-VL VL (Bai et al., 2023), LLaVA-v1.5 (Liu et al., 2024). GPT-4V (OpenAI, 2023), and GPT-4o (OpenAI, 2024).
Implementation Details. After unified formal vision-language pre-training, fine-tuned GeoX on each dataset to achieve better performance. The hyperparameters required for end-to-end visual instruction tuning are shown in Tab. 11.
Eval Mode
Prompt
Choice
You are intelligent robot expert solving geometry problems. Please ans- the Question based on the image. You should provide the reasoning process, and then you must give the correct choice in the end based on your reasoning the following form:
System Prompt:
The answer is (A), (B), (C) or (D).
Diagram: The Diagram is < <img> immage id. png</ //ng>>
Question: As shown in the figure, in triangle A B C. it is known that angle A = 80.0, angle B -60.0 D E parallel B C. then the size of angle C E is O.
Choices: (A) 40.0 (B) 60.0 (C) 120.0 (D) 140.0
Completion
System Prompt: You are an intelligent robot expert at solving geometry problems. Please ans- wer the Question based on the image. You should provide the reasoning process, and then you must give the correct answer in the end based on your reasoning in the following form:
c.g., The answer is []
Diagram: The Diagram <img> wwwiiimaeeiia id. png</ /ing> Question: Line m is the perpendicular bisector of XZ, WZ 14.9. Find WX.
mgg//in>>>
Table 9: The prompts used for Choice and Completion modes Multi-modal Large Language Models (MLLMs). To guide MLLMs in reasoning geometric tasks, we adopt two evaluation modes like Zhang et al. (2023b); Choice and Completion.
18

---

Published as a conference paper at ICLR 2025
Model Name
Model API Version
mPLUG-Ow12 (Ye al., 2023)
mplug-owi22llam22270
LLaVA-v1.5 (Liu et al., 2024)
Ilava-v1.5-13b-ht
Qwen-Vi LL (Bai et al., 2023)
Qwen-VL-Chat
GPT-4V (OpenA1, 2023)
gpt-4-isioon-preview gpt-4o-2024-05-13
GPT-44o (OpenAl, 2024)
Table 10: Model API versions used for evaluation across different MLLMs.
Instruction Tuning
GeoQA
UniGeo
PGPS9K
Geometry3K
Training Batch Size
64
Scheduler
Cosine Annealing
Optimizer
AdamW
Warmup Ratio
0.05
0.05
0.05
0.03
Epochs
100
80
45
30
Learning Rate
3e-5
3e-5
6e-5
2e-5
Evaluation Steps
200
400)
200
200)
Table 11: Hyperparameters for end-to-end visual instruction tuning.
We finetune these models on
A100 (80GB) GPUs, respectively.
G FURTHER DISCUSSION
Analysis advanced MILMss Ability Formal Programs Generation. shown in Tab, 4, GPT-4o (OpenAI, 2024) demonstrates the highest accuracy on Math In this section, we delve deeper into the few-shot learning ability of GPT-4o's in generating formalized program sequences, which are then sent to the GPS solver for solving (Chen et al., 2022). Specifically, apply 2-shot in-context learning, providing GPT-4o (OpenAI, 2024) with two examples formal problem-solving along with the complete set of operation functions. Then, GPT-4o tasked with predicting the corresponding solving program when presented with new problems and geometric images. As shown in Fig. GPT-4o (OpenAI, 2024) is capable of predicting simple geometric programs, but for more complex problems, exhibits issues such as predicting only the operation without the variable (e.g., g d_equal
b), incorrect variables (e.g., gougu_minus minus 5.0 V V_ vs gougu_minngnn minuu 5. 0 v_ 1_ in c). or wrong operations (e.g., g_equal vs g s_minns in d). In contrast, GeoX can predict the correct solution these complex and diverse cases.
As shown in the figure, the dinmeter CD) DCF = 20.0, then / BOD is equal to ().
As shown in the figure, in the quadrilttral ABCD, BADD= 1200.
midpoint chord
if you find a point M on BC and CD respectively.
perimeter
the smallest,
then the degree of L AMN ANM is O.
GPT-4o:
g double
GPT-4o:
g_squa g_add 1200 90. 8 minus C_3 1200 g double V_0
GooX:
2. double 20.0
GeoX:
Ground Truth:
double 20,0
Ground Truth:
S C_ 120.0 g double V_0
(e)
(d)
As shown in the figure, it is known that the radins of GO 5 and the chord AB 8.0, then the distunce from the center O to AB
As shown the figure, the light source P directly above the crossbar AB, shadow of AB under light CD, AB CD, AB 2.0, CD S.D, the distance between point is 3.0, the distance between AB and CD is
GPT-4a:
dividd 8.0.0 gongu_ mingu
50V V_2
GPT-46:
g. bill 5.0 2.0 divide 3.D Inst bili 2.0 5.0 3.0 3.g.mnnss 3.0
GeoX:
half gouga_minn V.0
GeoX:
Ground Truth:
g half gouga_minus V.0
Ground Truth:
g_ bili 2.0 5.0 3.0 g_minus 3.0 V_O
Figure 7: Comparison of GPT-4o and GeoX in predicting formalized programs for solving complex geometric problems.
19


Reviews:
[{"review_id": "HOXie3EQhS", "reviewer": "Reviewer_TRSt", "summary": "This paper proposes GeoX, a multimodal large model for geometric problem solving. It consists of several training stages: training a geometry-specific ViT, a geometry-specific LLM, learning cross-modal alignment and instruction tuning. The paper specifies the necessity of using formal language for geometry figures and design a sampler to better capture visual features. Some results of GeoX are reported on geometry benchmarks.", "questions": "Please kindly see the weaknesses above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The design of visual sampler is reasonable and interesting. This extracts more informative regions from the entire images.\n\n2. Using formal language instead of natural language is important for this field.\n\n2. The paper is easy to follow and presents good-quality figures.", "weaknesses": "1. The idea of training math-specific vision encoder and LLM and then aligning them has been introduced in EAGLE and MAVIS. So the training pipeline cannot be viewed as a novel contribution.\n\n2. The evaluation benchmarks are a little narrow. How about the performance of GeoX on geometry problems in MathVista, We-Math and MathScape.\n\n3. How does this geometry-specific model compare to SoTA multimodal large models, such as InternLM-Xcomposer, InternVL2 and QWENVL2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fAc8TW47Bt", "reviewer": "Reviewer_z62m", "summary": "This paper introduces GeoX, a multi-modal large language model (MLLMs) for the geometric problem solving (GPS) task. Previous MLLMs has limitations for understanding of both visual and symbolic information in geometry. GeoX overcomes this limitation by proposing a novel architecture, including: 1) collect a 120K pre-training dataset to train the vision encoder, symbol decoder for the diagram and language understanding; 2) Proposed Generator-And-Sampler Transformer to bridges the modality gap between geometric diagrams and formalized language; 3) With instruction tuning, this model can outperform baselines on 4 different datasets.", "questions": "1. Could the authors provide additional details for the ablation studies and clarify why the numbers differ between tables?\n2. If possible, could the authors include a comparison between Geo-ViT and CLIP to highlight performance differences?\n3. Have the authors finetuned Llava on the GPS dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors found that previous vision foundation models struggled to reason geometric images effectively. They proposed an unsupervised autoencoder model to train a Geo-ViT encoder on a 120K geometry image dataset they curated. This dataset, if released, could provide valuable resources to the community.\n2. The GS-Former is simple but substantially improves model performance.\n3. The study also demonstrates that using formal language significantly outperforms natural language in solving geometric problems.\n4. This model outperforms both generalist and specialist models by a large margin on multiple datasets.\n5. The attention maps show that the model focuses on areas relevant to the question.", "weaknesses": "1. The numbers in Table 5 and Table 6 are inconsistent. For example, on the Geometry3K dataset, the Completion score for the model without GS-Former is listed as 33.1 in Table 5, but appears as 55.0 in Table 6. Please ensure that the settings are consistent across tables.\n2. The authors argue that the CLIP encoder has limitations in processing geometric images, so they proposed a Geo-ViT encoder. However, there are no ablation studies to demonstrate the effectiveness of Geo-ViT compared to CLIP.\n3. The baseline LLava score is notably low, with a Top-1 score of 9.5 on GeoQA. Did the authors finetune the LLava model on the GPS dataset, given that the original LLava fine-tuning dataset lacks GPS-related questions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WAl4dKkafO", "reviewer": "Reviewer_bNTG", "summary": "This paper aims to align geometric representations with natural language descriptions, enhancing the model's ability to interpret and generate descriptions for complex geometric shapes and structures. This approach significantly improves the performance of existing models in accurately generating and interpreting geometric data compared to baseline methods.", "questions": "1.\tWhat specific features are extracted from the geometric data, and how are these features represented in the model? A detailed description of the feature extraction process would clarify this aspect.\n2.\tThe results presented in Table 2 indicate improvements, but could the authors justify the choice of metrics used, i.e., All, Angle and Length? Are there additional metrics that could provide a more comprehensive evaluation?\n3.\tThe paper should discuss how well the proposed model generalizes to unseen data. Are there specific scenarios or types of geometric structures where the model's performance is expected to decline?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tThe introduction of geometry language alignment as a distinct framework represents a fresh perspective in the intersection of geometric representation and natural language processing. \n2.\tThe experimental results demonstrate improvements over baseline methods, providing compelling evidence of the effectiveness of the proposed approach.\n\nOverall, the paper presents a promising approach to geometry language alignment, and with the suggested improvements, it has the potential to make a contribution to the field.", "weaknesses": "The interplay between geometric representation and natural language processing is complex. The paper does not sufficiently address how these two modalities are integrated within the framework, which could be crucial for understanding the overall approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QAY6jsRDRN", "reviewer": "Reviewer_xp8T", "summary": "The authors present GeoX, a multimodal large model designed specifically for tasks requiring geometric understanding and reasoning. Recognizing the distinct nature of geometric diagrams and symbols compared to natural image-text data, the authors introduce unimodal pre-training to develop a dedicated diagram encoder and symbol decoder, which enhances GeoX’s ability to process geometric images and symbolic information. Additionally, the paper proposes a novel geometry-language alignment strategy—an effective pre-training paradigm that bridges the modality gap between the unimodal geometric components. To further improve representation quality, the authors introduce a Generator-and-Sampler Transformer (GS-Former) that generates discriminative queries and filters out uninformative representations from unevenly distributed geometric signals.", "questions": "Since the provided examples are somewhat limited to math problems, I have concerns about the proposed approach's generalization to other geometric problems. I would consider increasing my score if the authors could validate the model on additional cases, such as geometric problems related to natural images, etc.\n\n\n\n==========================\nThank you to the authors for the rebuttal. The results look good, and I’ve increased my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The designed geometric solver significantly reduces hallucinations and incorrect results found in existing VLMs. I appreciate the design of the geometry-language alignment, which uses formalized descriptions instead of natural language captions, providing a new perspective for effectively aligning geometric and semantic features. GeoX demonstrates that formalized vision-language learning is beneficial for learning informative representations in automatic Geometry Problem Solving (GPS) tasks. GeoX can produce formalized process descriptions, enhancing both the interpretability of GPS tasks and the accuracy of the solution process.", "weaknesses": "Regarding the weaknesses identified in the paper, I have the following questions for the authors:\n\n1. Computation Time of the Geometric Solver: What is the specific computation time required for the Geometric Solver? Understanding the time complexity of this component is crucial for evaluating its efficiency in practical applications.\n\n2. Strategies to Alleviate Computational Burden: The entire procedure appears to be quite resource-intensive. Have the authors considered any strategies or methodologies to alleviate the computation cost, training time, and inference time? Exploring optimization techniques or alternative algorithms could significantly enhance the model's usability, especially in real-world scenarios where computational resources may be limited.\n\n3. Performance Comparison with Parameters and Costs: In the performance comparison section, could the authors provide a detailed breakdown of the parameters used in their experiments, along with the associated computation costs? This information would be invaluable for readers to assess the trade-offs between performance and resource requirements when using the proposed model.\n\n4. Ablation Study on Architecture: Is there any ablation study conducted on the architecture used for vision and language pretraining? Analyzing the contributions of different components in the model would help clarify which aspects are most beneficial for achieving the reported performance gains. This could also inform future research directions and potential improvements in model design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)