G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model
Authors: Jiahui Gao
Renjie Pi
Jipeng Zhang
Jiacheng Ye
Wanjun Zhong
Yufei Wang
Lanqing HONG
Jianhua Han
Hang Xu
Zhenguo Li
Lingpeng Kong
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Mathematical Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: Generate mathematical geometry questions from various perspectives and fine-tune a powerful MLLM, advancing its capabilities in geometric problem-solving.

Abstract: Large language models (LLMs) have shown remarkable proficiency in human-level reasoning and generation capabilities, which encourages extensive research on their application in mathematical problem solving. However, current work has been largely focused on text-based mathematical problems, with limited investigation in problems involving multi-modal geometric information. Addressing this gap, we aim to enable LLMs to solve geometric problems by understanding image input. We first identify the limitations of current Multimodal Large Language Models (MLLMs) in this area: they struggle to accurately comprehend basic geometric elements and their relationships. To address these challenges, we leverage the inherent attribute of logical structure compactness in geometric figures, utilizing text-only Large Language Models (LLMs) to curate a comprehensive multimodal geometry dataset. This dataset, named Geo170k, contains more than 170K geometric image-caption and question-answer pairs. Utilizing the Geo170k dataset, we introduce G-LLaVA, a model that demonstrates exceptional performance in solving geometric problems. It significantly outperforms GPT4-V on the geometry task of MathVista benchmark with only 7B parameters.

Published as a conference paper at ICLR 2025
G-LLAVA
SOLVING GEOMETRIC PROBLEM WITH MULTI-MODAL LARGE LANGUAGE MODEL
Jiahui Gao Renjie Pi Jipeng Zhang', Jiacheng Ye², Wanjun Zhong', Yufei Wang',
Lanqing Hong'. Jianhua Han', Hang Xu', Zhenguo Li", Lingpeng Kong"
1 Noah's Ark Lab The University of Hong Kong
The Hong Kong University of Science and Technology sumi lerecennect.. hku. hk, xpi@connect. ust.
ABSTRACT
Large language models (LLMs) have shown remarkable proficiency in human-level reasoning and generation capabilities, which encourages extensive research on their application in mathematical problem solving. However, current work has been largely focused on text-based mathematical problems, with limited investigation in problems involving multi-modal geometric information. Addressing this gap, we aim to enable LLMs to solve geometric problems by understanding image input. We first identify the limitations of current Multimodal Large Language Models (MLLMs) in this area: they struggle to accurately comprehend basic geometric elements and their relationships. To address these challenges, we leverage the inherent attribute of logical structure compactness in geometric figures, utilizing text-only Large Language Models (LLMs) to curate a comprehensive multimodal geometry dataset. This dataset, named Geo 10K,, contains more than 170K geo- metric image-caption and question-answe we introduce G-LLaVA, model that demonstrates exceptional performance solving geometric problems. It significantly outperforms GPT4-V on the geometry task Math Vista benchmark with only 7B parameters.
pairs. Utilizing the Geol 70K dataset,
INTRODUCTION
Large language models (LLMs) exhibit human-like proficiency in reasoning (Wang al. 2022 Wei ct al. (2022) and generation (Ouyang et al. 2022 Touvron et al.] (002)), which encourages extensive research on their application in mathematical problem solving (Fu et al., 2023) Gou et al. 2023 Luo et al. 2023 Yue et al. 2023). These problems often require highly sophisticated and symbolic reasoning capabiliitis, which were considered impossible solve before the of L.MSs.
an intuitive approach to use LLMs for mathematical reasoning problems presented in a tex- tual form. However, considerable portion of such problems requires understanding geometric information. Furthermore, even when certain problems may not explicitly involve geometric data, incorporating geometric-basse methodologies often carries substantial practical implications (e.g.. analytic number theory). With the advent of GPT4-V (OpenAl 2023). Gemini (Google (023), and
numerous multi-modal large language models (MLLMM)) (Bai et al. 2023b Dai et al. 2023 Gao
et al. 2023b; Lai et al.] 2023 Li et al. 2023 Liu et al.) 2023 Zhu et al. 2023). recent work start to
progressively look into employing M...Ms to tackle multi-modal geometric reasoning problems in
mathematics (Lu et al. 2023 Yang al. 2023).
However, we have observed that even the most advanced MLLMs still exhibit limitations in addressing geometric problems, due to challenges in accurately comprehending geometric figures. For instance, as demonstrated Figure GPT4-V often produces inaccurate descriptions for geometric figures. Specifically, the model struggles with understanding the relationships between fundamental elements like points and lines, and fail to accurately interpret elements such as the degrees of angles. The
Equal Contribution.

---

Published as a conference paper at ICLR 2025
GPT4-Y
LLAVALS
The image shows goomeinc figure conssting semoircle night ongled triengle semicinde las center
The image fectures whine cinde wet lime drwn trongh its cemer: The line labeled wals the letter "d" and cstends from the canser of the cincl the else. The ctalle is divided inso Sour oquel puss, with each part laseee with the leners "Ia" ra," "," and "d The circte is also labeled with the number "y" the top left somer. The everal appeorance of the imaage siggests mathenztical or geometric conespe.
borionni segners AC, it seniciille
point indicating the mispoine of the are. The right analed iaagle is focmed by the dazzeter MC of the semeiizle and a verical segitent BDD droopong from point to point us the diemeter AC, The siangles hyposemse is the tadins of the semerel,, and the right ingle is poirt
uemou pat
Deseribe the figure detail.
MinGFF-4
G-LLaVA
image shows mlangle with byposenue length leas length represent the
The diggram senticircle AB Paint
and mmsglee vartice the niangl: The sygutemse labeled sumber 10, and the laps in bhited number 5 and 12) The siungle drawn Nack ink on while bockground.
with
semiciclle and CD drown from point D pount C. Point the senicircie
Figure I: State-of-the-art MLLMs suffer severe hallucination geometric figures, which greatly hinders their abilities on solving geometric problems. In contrast, after the first alignment phase, G-LLaVA demonstrates enhanced ability in interpreting geometric figures compared to LLa
underlying reason may be the fact that these MLLMs are predominantly trained with natural images, which differs significantly from geometric figures.
To address this issue, one direct and effective approach is to enhance current MLLMSs by augmenting them with data containing high-quality descriptions of geometric information. However, a significant challenge is the limited size of the largest publicly available geometric problem dataset, which includes only a few thousand questions, each with a single solution. To collect new geometric datasets, it would introduce prohibitive expenses human experts are hired conduct labelling manually. On the other hand, automatic labelling with MLLMs is infeasible, since even the state-of of-the-art MLLMs (e.g., GPT4-V) face challenges in comprehending geometric diagrams.
In this paper, we first highlight an inherent characteristic of geometric figures termed logical structure compactness. This property allows for the construction high-quality multi-mode datasets using solely text-based LLMs such as ChatGPT. Specifically, logical structure is defined as the geometry elements and their relationships. With a compact logic structure, all the essential information for representing and reconstructing the geometric figure can be encompassed, which is owning to the concise nature of geometric elements (e.g.. circles, triangles, lines). Therefore, upon acquiring expressions containing logie structures information, such as QA pair and logic form (Figure which are prevalent in public geometric datasets, we can use them to represent the geometric figure and input them to text-only LLM,, so that the LLM can interpret figure and produce high-quality annotations.
Specifically, we create two types of data: 1) geometric cross-modal alignment data, which aims at improving the MLLM's understanding of the geometric figures by aligning the image representations with their textual annotations, which also reduces hallucinations; 2) geometric instruction data that empowers the model with stronger geometric reasoning ability by covering more diverse geometric question scenarios. During data generation process, we leverage series of geometric characteristics enhance the quality, diversity, and scale of the training data (as shown in Figure(). We term our generated dataset Geol 70K, which contains around 60,000 geometric image-caption pairs and more than I 110,000 question-answer pairs. This dataset is 28 times larger than GeoQA+ (the previous largest open-source dataset), greatly expanding the coverage of geometric problems.
With our collected Geo170K, we derive G-LLaVA, an MLLM capable of solving geometric problems, surpassing SOTA MLLMs by large margin. Specifically, G-LLaVA-13B outperforms LLaVA- 13B by 27.4 on the Geomtery Problem Solving (GPS) task on restmini split of Math Vista (Lu et al. 2023). In addition, with only G-LLaVA-7BB it is able to surpass the powerful GPT4-V on the geometry problem solving questions. Our code, data, and models are publicly accessible at https: / thub. http://hhhhtoooccm///pp////r/-/e
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Geometry Problem Solving. Geometry reasoning problem is a challenging visual mathematical
reasoning problem. Early efforts by Alvin et al. (2017); [Sachan and Xing (0177); ): |Sachan et al. (2017);
(
Seo et al. (2015) focused on creating datasets through manual efforts. More recent approaches have
introduced enhanced methods and datasets, including GK (Lu et al. 2021). GeoQA (Chen
et al. 2021). GeoQA+ (Cao and Xiao 2022), UniGeo (Chen et al. 2022). UniMath (Liang et al. 2023). , and SCA-GPS (Ning al. al.j 2023). aiming to improve both performance and explainability. However, the scale of current datasets remains limited, and the performance of traditional models in the multi-modal mathematical geometry problem domain remains markedly lower than progress has been made in text-based mathematical problems, particularly when compared to methods that utilize large language models for solving math word problems (Cobbe et al.] [202]] Gou et al. [023]
Lu et al. 2023 Wei et al.) 2022 Zhang et al. 2024bb. More recently, significant progress has been made in multimodal mathematical reasoning. Mathl_LaVA (Shi et al. 2024) constructed a diverse dataset of 360k multimodal mathematical questions by utilizing GPT-4V to augment 40k
seed examples from existing datasets. MultiMath (Peng et al. 2024) employed GPT-40 in multiple rounds of annotation build the comprehensive MultiMath30OK dataset, which spans K-12 levels and includes image captions and step-wise solutions. Additionally, MathPuMA (Zhuang et al.) (024) introduced -ox-tto-image rendering process that converts textual content into visual form, enhancing the diversity of image sources. While these works have significantly advanced the field, our approach differs by primarily focusing on augmenting datasets from the textual side to enhance the MI.LM's understanding of geometry. Without relying on expensive models like GPT-4V or GPT-4o for data annotation, we generate multimodal data using text-only LLMs. This text-centric augmentation serves as complementary method to image-centric approaches, offering greater scalability and cost-effectivenesss By leveraging text-only LLMs, we demonstrate that substantial improvements in geometric reasoning can be achieved, highlighting alternative pathway to advancing MLLMs without extensive reliance on visual data augmentation.
Data Generation via LLM. Bootstrapping data from pre-trained models has long been an active area of research. [Meg et al. (2022); ; Ye et al. (2022a b) generates training data using pre-trained language models such GPT-2 for classification tasks. Gao et al. (2023a) improves the quality of generated dataset via bi-level approach. Recently, automatic data generation becomes more ubiquitous
with the advent of powerful LLMs such as ChatGPT, line of recent works utilize LLM- generated
data to perform instruction tuning (Huang et al. 2024 Li et al. 2024 Peng et al. 2023
Pi et al.
2024b
Taori et al. 2023
Wang
2023
Zhang et al. 2024a).
Multi-Modal Large Language Model. Recent years have witnessed transformative advancements in the development of large language models (LLMs), characterized by series of pioneering
studies (Bai et al. 2022 Brown et al. 2020 Chowdhery et al., 2022] Hoffmann et al. 2022 Ouyang
let al. 2022 Scao et al. 2022 Smith et al. 2022 Touvron et al. (023). These breakthroughs have
significanlly clevated the capabilities of language understanding and generation, showcasing near-
human proficiency across diverse tasks. Concurrently, the success of LLMS has inspired explorations
into vision-language interaction, leading to the emergence of multi-modal large language models (MLLMs) (Bai et al. 22023b] Dai et al., 2023 Dong et al. 2022 Gao et al. (2024a| by Li al. [0233] Liu et al. 2023 2024; Pi et al. 2223a 2024a Su et al. [023] Zhu et al., 2023). These models have exhibited remarkable capabilities in synthesizing detailed descriptions and engaging dialogue based on visual inputs. However, we observe that even the state-of oh-the--rttt MLLMs face challenges in resolving geometric problems using diagrams and figures.
3 OBSERVATION
We observe that state-of-the-art (SOTA) MLLMs, although being adept understanding natural images, have difficulty in comprehending simple geometric figures. In Figure [ we demonstrate that the deseriptions from SOTA MLLMs for geometric figures suffer from severe hallucination. This inadequacy in interpreting geometric diagrams may be one of the major causes of the failure in solving geometric problems.
3

---

Published as a conference paper at ICLR 2025
Geonetric Description Generation Reverse Infermatiin Rocovety
Generate Pair Elenents
QA Pair
Logie Form
0: 0 cincamenthall cinle
Simdtrt A.A.angg//AA.......... Aww.AAACCCTnnnn...........
A.S. A.ACC,...... A110-30000 (, Bhon the sie
PointLisOOiLineeD Lisis, BJ
Imagee Description
At Irangle AOG, DA-UB. ZABO+30": Thensfone
disgrati of trinngls mocitted within rinde O) Prints and
Info Doems
Factual QA pains
ZADI-19II- 22 A110+1701 Therofone, 2 Ae-12 ZAOR
Carz verissa
tringli, they le the arcumference of cincle. The
L.Triangle ABC similar trangle CBD 2. Point D lies the line BC
fize line
Nox AB.
cester of the cicle is murkel as point O.
Logical Siruciure Compaciness
Logical Structure Compactncss
Geometry Characteristic
Geometric Generalizahllity
Scalability
Conditional Uniqueness
Ressoning Diversity
Subotitation Equation Solving
Valee Scating
Re Foesulating Coudition ai Unkcowi
Scalizze Parplizse
Point 3 in line AC, Iiz All I, line AC line di? Line BC is
Q: Poin is line AC, line AB 1. line AC is line BC is? Line BC is
Q: Point B is lice AC, line AB 1. line AU 3. line BC is
Q: Poins is line AC, line AB line AC is line iC A: iC
A: Line BC
AC, line Q: (x- line MC (ym5), line ix?
Q: Point is tine AC; line All is line AC is line ICCss A: The jength of BC uniss, obtained subracting AB unils) from AC unite).
Q: Point is line AC, ABl is line AC 3, line BC 2s What the value of A: Demote All se determine by solving 2*2n5. Therefore, a equals I.
with ABell and segmen longth segment subriacting
A: Denote BC as z. we can build the equation 2ea therefore any y's,, which simpliffie to a-3-1=2 Thos. is
can
segment lengti of units, AC equats unis.
woyment
Genernte using text-only L.M
Multimodal Geometric Alignment
Multimodal
Figure 2: Framework of multi-modal geometric data generation using geometry characteristics.
Specifically, we find GPT4-V has difficulty understanding the basic elements, such as points, lines and angles (e.g., angle B in Figure Db. Furthermore, it struggles to comprehend the spatial relationships between the basic elements. On the other hand, open- open-source MLLMS like LLaVA1. and MiniGPT4 (7B) face even greater challenges in accurately identifying the geometric shapes. This inadequacy in geometric interpretation may be a major cause of the failure in solving geometric problems.
To address this issue, point out that geometric figures enjoy appealing property, which refer to as logical structure compactness. Specifically, logic structure is defined as the geometric elements (shapes) and their spatial relationships. Due to the simplicity of geometric elements such as circles, triangles, and lines, geometric figures can be accurately represented by leveraging logical structure information. This stands contrast to natural images, where such precise reconstruction is not possible. In publicly available geometric datasets, the annotations for question- question-answer answerrw pairs and the logic form representation are precise expressions that capture the underlying key logical structures inherent in geometry figures (see Figure 2 By leveraging on this characteristic, we can utilize these
Table 1: Geometric diagram description generation via information recovery. The description is generated based on the textual QA pair. The upper section shows the QA pair employed to instruct text-only ChatGPT, while the lower section in blue shows the responses generated by ChatGPT.
Geometric Description Generation via Inverse Information Recovery
QA Pair: Question: shown in the figure, circle the circumscribed circle of triangle ABC, and it known that angle ABO 30.0, then the size of angle ACB O Answer: In triangle AOB, OA=OB, angle ABO=30%: Therefore, angle A0B=180 2 angle ABO=120= 200; Therefore, angle 1/B=1e A
Diagram Description:
The diagram consists of a triangle ABC inscribed within a circle, where the circle is denoted as circle O. Points A, B, and C are the vertices of the triangle, and they all lie on the circumference of the circle. The center of the circle is marked point O.
4

---

Published as a conference paper at ICLR 2025
compact representations instead of the original geometric figure as input to text-only LLM, enabling the LLM to comprehend the geometric figure and generate annotations with high quality.
4 GEOMETRIC DATA GENERATION
The key limitations existing datasets (Cao and Xiao 2022 Chen et al. 2021 2022 are threefold: (I) the limited size; (2) the absence of detailed descriptions for geometric images; and (3) the lack of diversity in problem-solving methodologies and reasoning paths. To address these challenges, we leverage the logic structure compactness of geometrie figures construct multi-modal geometry dataset, building upon existing datasets. This dataset includes two parts: an alignment dataset and an Instruction-tuniing dataset.
4.1 GEOMETRIC CROSS-MODAL ALIGNMENT DATA
Geometric Image Description Generation. Inspired by previous works on MLLMs (Liu et al. 2023;
Zhu et al., 2023), detailed image-description pairs are crucial for training MILMs to compre- hend image contents. However, there is lack of image-text paired dataset for geometry problems.
Factual QA Pairs for Basic Elements
Logic Form:
Similar (Triangle (A, B, Triangle (C,
Triangle B,
Triangle (A, B)
Equals (lengthOf (Line (C, B)), 5)
Equals (lengthof (I.ine (C, D)), 3)
Equals (LengthOf (Line (B, D)), 4)
PointLiesOnLine (D, Line (A, Perpenaiculat (Line (A, C), Line (C, B Perpendicull (Line (C, D), Line (A, D)
Phase-1 Generation
Info Items:
The geometric figure have the following information:
Info I: The figure represents a triangle with vertices A, B, and C. Triangle ABC is similar to triangle CBD.
Info 2:
Info 3: Point D lies the line segment AB.
Info 4: Line segment AC is perpendicular line segment CB.
Summary:
The figure depicts triangle ABC and triangle CBD, which is similar triangle ABC. Additionally, triangle ACB is also present in the figure. The lengths the segments CB, CD, and BD 5, 3, and 4 units respectively. Point D lies on line segment AB, and perpendicularity is indicated between the line segments AC and CB, as well as between the line segments CD and AD.
Phuse-2 Generation
QA pairs:
Question: Does the diagram include a triangle formed by vertices A, B, and D?
Answer: No, the diagram does not include triangle formed by vertices A, B, and D. The triangles
present ABC and CBD, and the point D lies the line segment AB.
Question: Does the diagram contain triangle with vertices A, C, and
Answer: Yes, the diagram contains triangle ACB, which is the same as triangle ABC.
3. Question: Is point D the lies on line BC?
Answer: No, point D is not on lines BC. The point D lies the line segment AB.
Question: Is line segment AC perpendicular line segment CB?
Answer:
line segment AC perpendicular to line segment CB.
Table 2: Geometric image description and factual QA pairs for understanding basic elements. The sections in blue display the responses generated by ChatGPT.

---

Published as a conference paper at ICLR 2025
To close this gap, we propose novel approach that leverages the unique logic structure compactness property to generate comprehensive descriptions effectively from labeled question-ansee answer (QA) pairs, as illustrated in Table [ In particular, leverage the strong comprchension abilities of text-only ChatGPT 3.5 to generate image captions given human-labeled textual question-answer (QA) pairs that contain essential information of the logic structure. This process be viewed type of inverse information recovery. The generated descriptions are paired with existing images to form the image-description datasets.
Factual QA Pairs for Basic Elements. We generate QA pairs to facilitate the comprehension of geometric diagrams, focusing on basic geometric elements. The process begins with the interpretation of human-labeled logical forms that capture the entire logic structures of geometric figures on Geometry3k (Lu et al. 202T). In the Phase-1 generation, we employ ChatGPT to convert these logical forms into textual descriptions that cover various geometric elements (e.g.. shapes, lines, and points) and their relationships. In Phase-2 generation, use ChatGPT to produce factual QA pairs given both the logic form and the result from Phase-1. The whole process is shown in Table
The QA pairs may explore the presence of certain geometric elements (e.g., "Are there triangular shapes in the diagram?") or check the accuracy of the relationships described (e.g., "Is point the lies on line BC?"). The answers are "Yes" or "No". along with the corresponding reasoning pathway. This method enables the model to comprehend geometric concepts and interpret the details in geometric diagrams accurately (Table 2 The detailed prompts are shown in Appendix D
Remark. The alignment data is used during the first-stage cross cross-modee modal alignment to enhance the model's s ability to understand geometric diagrams. The effectiveness is shown in Figure]
Table 3: The original example.
Original Example
As shown in the figure, the length of the short arm of the railing at the railway crossing is 1.0 and the length of the long arm is 16.0. When the end of the short arm drops by 0.5. the end of the long Answer: By the property of similar triangles, the lengths corresponding sides are proportional. Since the long arm is 16 times longer than the short arm, the height it rises is 16 times the height the short arm drops, which is 0.5 multiplied by 16, resulting in rise of 8 meters.
Question:
16m
rises
0.5m
Table 4: Synthetic example via value scaling.
Value Scaling
Question: At a railroad crossing, the short arm of the barrier is 2.0 unit long and the long arm is 32 units long. If the short arm drops units, by how many units will the long arm rise?
16m
'h'. In similar triangles, the ratios of their corresponding sides are proportion. h can calculated using the equation 1/h 2/32. Solving for 'h' gives 32 1 /2 which equals 16 units. Therefore. the long arm rises by 16 units.
Answer: Denote the rise of the long
0.5
4.2 GEOMETRIC CROSS-MODAL INSTRUCTION DATA
To improve the model's ability to understand user instructions and provide solutions to geometric problems, we further curate an instruction-tuning dataset based on a series of geometric characteristics, which is enriched with diverse question and reasoning paths.
Value Scaling (VS). We note that geometric QA pairs remain correct if all the lengths in a geometric problem scaled simultaneously by the same factor, which refer to geometric scalability. However, note that this is not the case for quantities like angles. Based on this, we augment length- related problems by scaling all the length values in the QA pairs via the same factor (as shown in

---

Published as a conference paper at ICLR 2025
Table 5: Synthetic example via replacing values to unknown variables and solving the equation.
Variable Substitution and Equation Solving
Given railway crossing barrier with two arms, where the shorter arm has length of 'a' units 1.0) and the longer arm has length of "b' units (h 16.0), determine the elevation 'y' units of the longer arm's end when the shorter arm descends by 'd' units (d 0.5). Express your answer function 'a', "b", and 'd'.
Question:
0.5m
Answer: To find the clevation 'y" of the long arm's ' end, we apply the property of similar triangles and establish the equation Solving for "y" leads to bxd which simplifies to y = 1..0x0.5 Therefore, "y" equals 8.0 units, indicating that the long arm rises by 8.0 units when the short arm descends by 0.5 units.
:. Substituting the given values, we get 16.0"
1.0
Table 6: Synthetic example via re-formulating condition as unknown.
Re-Formulating Condition Unknown
Question: At railroad crossing, the short arm of the harrier unit long and the long arm 16.0 units long. When end of the short arm drops by 0.5, the end of the long arm rises 8 units. What is the value of unknown variahle u? Answer: Denote the short arm of the barrier as variable u. By the property of determine by solving the equation 0.5/8 w/16.
16m
0.5m²
similar triangles, Therefore, u equals
Table 7: Synthetic example via sentence paraphrase.
Sentence Paraphrase
Question: In the illustration, the railing the railway crossing has short arm measuring 1.0 unit in length and long measuring 16.0 units. When the short arm drops by 0.5 units, what the corresponding Answer: The triangles are similar, and their corresponding sides are proportional. 0.5m The long arm is 16 times longer than the short arm, resulting in an 8-meter rise when the short arm drops by 0.5 meters.
16
the long arm?
Table 4J.
When different scalings are applied, the LLM becomes more flexible in handling diverse numerical inputs, which aids in refining the model's reasoning capabilities and generalizability.
Equation Solving (ES). As shown in Table 5 we replace the specific values in the original QA pairs with unknown variables and prompt the LLM to construct the solution solving equation. Such data helps the MILMM generalize its understanding of the problem, which enables it to apply the similar reasoning and solution steps different scenarios. The abstraction of the problem by using variables and equations helps the MILM focus on the underlying mathematical concepts and relationships, rather than overfitting specific values.
Re-Formulating Condition as Unknown (RCU). The inherent logical structures within the dresttio-answer pairs enable the reconstruction of the geometric figures based on the provided information. Consequently, it becomes possible to interchange the known and unknown variables, wherein the condition stated in the question can be considered as the variable, while the unknown variable can be viewed as the condition (Weng et al. 2023 Yu et al. 2023). this way, the MLLM repeatedly exposed to the relationships between the elements. This reinforcement helps the model learn the dependencies and connections between different elements in the geometric problem. An example is shown in Table
Sentence Paraphrase (SP). As shown in Table 7 paraphrase both the questions and answers, exposing the LLM to broader range of phrasing and language variations. This makes the model more robust understanding questions in various forms and generating diverse reasoning paths.
7

---

Published as a conference paper at ICLR 2025
Data Correctness Verification. We implement verification process ensure the correctness of the reasoning paths when generating new question-answer pairs. Specifically, we require Chat- GPT to finish the answer in a specific pattern, marked as "Resul:/RESUULT].. Subsequently, we employ regular expression to extract the answers and compare them to the expected true an- swers. To ensure correctness of our data, only the generated QA samples that associated with the correct answers are retained. For example, in the VS approach, we verify whether the answer is equal to the original answer multiplied by the scaling factor, and in the RCU ap- proach, we check whether the generated answer matches the value provided in the condition.
Table 8: Results on testmini set of Math- Vista (Lu al. 2023) geometry problem solving (GPS) For input, Q represents for question, / represents for image, Le represents for image caption generated by Bard, and 4 represents fo OCR text detected in the image. Baseline results are obtained from Lu et al.] (2023). Human performance and the results surpassing human performance are highlighted in grey. Our results highlighted in blue
EXPERIMENTS
5.1
SETUP
Dataset. We generate the alignment data and in- struction data utilizing training set of GeoQA+ (Cao
and Xiao, 2022) and Geometry3K (Lu et al. 2021). More specifically, the factual QA pairs in the align- ment data are generated using Geometry3K, which contain
logical forms. We eval- uate G-LLaVA on the geometry problems solving (GPS) task (testmini split) of Math Vista (Lu et al. 2023) and test set of GeoQA. Note that the GPS task Math Vista testmini set collected from GeoQA, Geometry3K, GeoQA+, and GEOS (Seo ct al., 2015). To verify the generalization ability of G-LLaV, also conduct evaluation the
Model
Input Heuristics Baseline
Accuracy (%)
Random Chance Frequent Guess
21.6
34.1
Human
Q.J
48.4
Close Source Model
Text-Only L.Mss
Zero-shot ChatGPT
Q
26.9
Zero-shot GPT-4
Q
37.0
2-shot CoT Claude-2
Q
29.8
newly introduced Math Verse benchmark (Zhang
2-shot CharGPT
Q
36.5
al. 2024b).
2-shot CoT GPT-4
Q
44.7
2-shot PoT ChatGPT
Q
30.8
Model. We develop our model based on
2-shot GPT-4
Q
33.2
LLaVA Liu et al. 2023). More specifically.s
Visual-ugmmented L.Ms
2-shot CoT Claude-2
Q.1,,4
31.7
utilize LLAMA-2 Touvron et al. 2023) as the lan-
CoT ChaiGPT
Q.1eJ
29.3
guage model and employ the pretrained vision trans- former Radford et al. (2021) as the vision encoder (ViT). We conduct experiments with both 7B and 13B LLMs. We employ ChatGPT (gpt-3.5-turbe 0613) for data generation. The detailed description of our prompts is provided in Appendix D
2-shot CoT GPT-4
Q.InI
31.7
2-shot PoT ChaiGPT
Q.I..
26.4
2-shot GPT-4
Q.
Multimodal LLMs
Multimodal Bard
Q.J
47.1
SPHINX (V1)
Q.1
23.1
SPHINX (V2)
Q.1
16.4
Gemini Nano
Q.1
21.6
Gemini Nano
Q.J
23.6
Training. We train our model two phases, namely 1) geometric visual-language alignment, and 2) geo- metric instruction tuning. Each phase conducted with the corresponding dataset, respectively. In both phases, we leverage the conventional language modeling loss. More implementation details provided in Appendix
Gemini Pro
Q.I
40.4
Gemini Ultra
Q.
56.3
GPT4-V
Q.J
50.5
Open Source Model
IDEFICS (9B- (B--Instruct)
Q.I
21.1
mLLLG-Owl (LLaMA-7B)
Q.I
23.6
MiniGPT4 (LLaMA-2-7B)
Q.I
26.0
I.I.MA LaMMA-Adapee--V (7H)
Q.
25.5
LLaVAR
Q.I
25.0
Evaluation Metric. We use accuracy the evalu- ation metric. Several prior studies (Cao and Xiao 2022 Chen al. 2021 2022) report results using Top-10 10 accuracy (generating 10 sequences and se- lecting the first sequence that successfully addresses the problem as the prediction). our paper, our experimental results directly report Top-1 accuracy. More details are attached to Appendix
InstructBLIP (Vicuna-7B)
Q.J
20.7
LLaVA (LLaMA-2-138)
Q,
29.3
G-LLaVA-7B
Q.J
53.4
G-LLaVA-13B
Q.J
56.7
5.2 EXPERIMENTAL RESULTS
Main Experiments. We compare MLLMs on testmini split of Math Vista (Lu et al., 2023) bench-
mark Table R. The results shows that, geometric cross-modal alignment and instructing tuning
8

---

Published as a conference paper at ICLR 2025
dataset is effective in improving MLLMs' geometric problem solving ability. Our specific in-domain model G-LLaVA-7B can even surpass the strong GPT4-V on geometrie problems.
Comparison with Conventional Methods. We additionally compare our method with conventional SOTA methods in geometry problem solving do- main. As illustrated in Table our method demon- strates notable improvement in Top-1 accuracy over the existing SOTA techniques. Moreover, our model's top-1 accuracy outperforms the baselines' top-10 10 accuracy, demonstrating significant im- provement in predictive precision.
Table 9: Comparison of model performance with traditional methods on GeoQA.
Model
Input
Accuracy (%)
Random Chance
25.0
Frequent Guess
32.1
Top Accurary
NGS Chen al. 2021
Q.I
56.9
DPE-GT'S Cao and Xio 2022
Q,
62.7
SCA-GPS Niig
Q.I
64.1
Accuracy
Geoformer (Cenn et al
2022
Q,
46,8
Effectiveness of Cross-Modal Alignment. To evaluate the alignment phase's effectiveness, we conducted experiments with and without geometric alignment phase in Table TO The results suggest that the alignment phase enhances the model's 's abil- ity to interpret images, which is also illustrated by the qualitative result in Figure We also observe that releasing the LLM during the alignment phase leads more hallucinations, which consequently degrade its performance.
UniMath (Liang el al. 2023
Q.
G-LLaVA-7B G-LLaVA-13B
Q.1
64.2
Q, 67.0
Table 10: Effectiveness of alignment in the pre- training phase. Top-1 accuracy is reported. AL stands for first-stage moos- alignment.
Model
Input
Accuracy
Random Chance
25.0
Frequent Guess
32.1
Geoformer (Chen et al UniMath (Liang al. 2023
2022
Q
46.8
Q,I
50.0
Gain Introduced by Different Types of Augmen- fation. We demonstrate the performance gains in- troduced by each augmentation strategy, as well as the cumulative number of training data. As shown in Table II each our proposed augmentation strategies is able boost the model performance. Note that as the accuracy grows higher, the extra gain becomes more difficult.
I.LaVA-7B
Q.1
18.7
+ SFTT
Q,1
62.8
+ AL (tunable LLM ) SFT
Q,
60.1
+ AL (fixed LLM + SFT
Q,I
64.2
Table 11: The impact of incorporating data aug- mentation GeoQA. For this ablation study, the training set covers the full GeoQA+ train set.
Performance Across Problem Difficulties. We perform experiments on problems with different dif- ficulties, as shown in Table 12 Specifically, OP represents the number "operations", or reasoning steps that needs to be taken for solving the prob- lem. The results verify that our G-LLaVA con- sistently outperforms baseline models by large margin across various difficulty levels.
Training Data
Data Size
Accuracy
GeoQA+ train set
6027
48.2
w/ SP
75,982
56.1
w/ ES
100,411
59,4
VS
109,787
62.3
w/ RCU
119,563
65.1
Performance Across Different Types of Questions. We compare G-LLaVA with the baselines models on problems with different type of questions, as shown in Table 13 The results suggest that G-LLaVA performs better than the baseline models in various geometric problems such angle, length, and area problems.
Generalization Ability. We also conduct experiment on the newly-introduceed benchmark Math- Verse (Zhang et al. 2024b). For case of evaluation, we conduct evaluation on the multiple-choice (MC) questions. The full results of the restmini collected from Zhang et al. (2024b). According to results in Table 14 G-LLaVA significantly surpasses other open-source MLLMs in the Mathverse
Table 12: Performance of different difficulty
Table 13: Performance on different types ques-
problems on GeoQA.
tions on GeoQA.
Model
OP-1
OP-2
OP=3
OPs=4
Total
Model
Angel
Length
Area
Other
Total
LLaVA-7B
209
22.9
LLaVA-7B
22.2
14.3
LLaVA-13B
19.1
21.3
18.5
24.6
20.3
1LaVA-13B
17.5
23.0
25.5
28.6
20.3
G-LLaVA-7B
77.5
60.8
40.9
G-LLaVA-7B
42,9
G-LLaVA-13B
79.0
64.9
55.5
49.1
67.0
G-LLaVA-133
71.5
61.1
63.8
57.1
67.0

---

Published as a conference paper at ICLR 2025
Table 14: Model performance on Math Verse testmini
Text Dominant
Text Lite
Text Only
Vision Intensive
Vision Dominant
Vision Only
Model
All
testmini
LLaMA-Adapter (ao et al.
2023c
5.7
6.2
5.9
2.7
6.1
4.2
6.1
ImageBind-LLM. LLM Han et al. 2023
9.3
11.4
11.3
11.7
8,9
11.2
3,4
mPLUG-OW12 Ye et 202.3
4.6
6,6
6.3
6.1
6.3
5.6
4.9
MiniGPT-v2 Chen
2023a
11.0
12.1
12.0
11.7
13.1
10.3
7.4
SPHINX-Plus [ao al.
20244b
11.6
14.9
11.6
10.4
LLaVA-NeXT (Lau
al.
2024
10.3
12.8
12.0
9.9
10.7
9.7
6.3
ShareGPT4V (Chen et al.
20233
13.1
16.2
16.2
6.6
15.5
13.8
3.7
VL-Plus (Bai 2023a
11.8
15.7
11.1
14.5
9.0
13.0
10.0
LLaVA-7B (Liu
2023
7.6
8.8
7.6
11.5
7.4
7.4
6.9
G-LLaVA-77B
16.6
20.9
20.7
21.1
17.2
14.6
9.4
testmini-ME
ILaVA-7B
11.0
12.8
13.1
12.8
14.2
13.3
0.0
LLaVA-13B
16.7
22.7
21.1
19.0
18.1
19.3
0.0
G-LLaVA-7B
34.7
42.7
36.7
42.4
31.2
28.2
27.3
G-LLaVA-13B
36.3
43.1
37.4
45.2
32.3
29.4
30.3
benchmark, demonstrating its generalization ability. Our model also outperforms LLaVA-7B (our base model) by margin of 9.0, which highlights the quality of our annotated dataset. Moreover, G-LLaVA not only shows substantial improvements in text-dominant, text-lite, text-only, and vision- and vision-only categories. This verifies G-LLa capability to interpret geometric figures and solve problems effectively.
intensive categories, but also exhibits advancements in
Additionally, we show the results of sub- tasks on Math Vista-GPS multiple-choice questions for easier evaluation in Table [15] which includes GeoQA+, UniGeo, Geome- try3K, and GEOS. We observed significant performance improvement from 23.5% to 47.1% the unseen GEOS task. This advancement confirms that our dataset improves the MLLM's ability to interpret geometric figures and enhances its generalization capabilities.
Table 15: Subtask performance of Math Vista on geom- etry problem-solving (MC).
Model
GeoQA+
UniGeo
Geometry31
GEOS
LLaVA-7B
15.8
6.5
24.2
14.5
23.5
G-LLaVA-7R
85.7
56.5
72.6
40.3
47.1
GPE4-
As shown in the figure All is tangent to O point and the excuded lise oD intersetss crek O point C Conrect BC, BC, ifinll A 16 then angle apal () Choices: A36 B:54 Caur D:27"
aplandal appcars repecsent problem invelving properties
sngles.
given
angle degrees,
angles.
Here's muliss inteneste tangeri
paint hangancy angle This nouss that ungle CBA is deares.
2.Triangle OXB is right iriandee with anple DAB being degrees and angle AOB being 16 degrses Liiing the facl that the smm of lenils in tringle is dlggnsss aed can find zngle ABO
G-LLAVA
Aingl ABO equal degrees minus the sum angles AOM UALL which
190 16-54deggess degnoss
199 190-900 16-54
Since AB is tangent to ciitle point B, thereGre, atgle ABO Since angle A therefore, angle BUX SP. Therefore. acsnding the maathhl angle theoren, angle C angle BOX 27 Tharofoee, the soneet choice
J.nnge
they both
line
equal angle wegnonn angle aegles segennee
A.Angle
deprees.
So, ongle angle ACB, degrees. Therefore, correct chooe is:
Figure 3: Visualization of problem solving by GPT-4-V and G-LLaVA.
CONCLUSION
This paper attempts address the limitations of MLLMs in solving geometric problems. Taking advantage of geometric logic structure compactness, we propose several strategies to enrich the multi- modal geometric data by leveraging text-only LLMs,, resulting in the generated dataset, Geol706 With this dataset, our G-LLaVA outperforms GPT-4-V on the geometric task (GPS) of Math Vista, with as few 7B parameters. We hope our work provides new insights improving multi-modal LLMs' ability to solve geometric problems.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Chris Alvin, Sumit Gulwani, Rupak Majumdar, and Supratik Mukhopadhyay, Synthesis of solutions for shaded area geometry problems. The Thirtieth International Flairs Conference, 2017.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile languale model for understanding, localization, text reading, and beyond, 2023b.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint M.Xiv:2204005880 2022.
Tom Brown Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 1877-19119,,, 2020.
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference Computational Linguistics, pages 1511-1520, 2022.
Jiaqi Chen, Jianheng Tang. Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning. In Chengqing Zong, Xia, Wenjie Li, and Roberto Navigli, editors, Findings of the Association for Computational Linguisties: ACL-IICNLP 2021, pages 113-523, Online, August 2021. Association for Computational Linguistics. doi: 10. 18655//1/20211 findings-acl ac1.46. Bttthtpss:1 : / ac wcl.anoooooooyy..ooooooooooooooh 2021. f Lndings-aci.
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. UniGeo: Unifying geometry logical reasoning via reformulating mathematical expression. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages
2022.
Jun Chen, Deyao Zhul Xianqian Shen Xiang Li, Zechun Liu2 Pengehuan Zhang, Raghuraman Krishnamoorhi22 Vikas Chandra2 Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: Large language model as unified interface for vision-language multi-task learning, arXiv preprint arXiv:2310.09478. 2023a.
Lin Chen, Jinsong Li, Xiao wen Dong, Pan Zhang, Conghui He. Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. ArXiv, abs/2311.12793,, 2023b. URL
hittps://api. semant lescholar..... wttp:://ooooooooooooooee311111.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoiea, and Eric Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL httph/ttss///ssss..omo
2023- 03 1-23--330 30-vv iel-vicume cuna/
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purposs vision-language models with instruction tuning, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. InternIm- xcomposer2: Mastering free-form text-image composition and comprehension in vision- language large model. arXiv preprint arXiv:2401. 16420, 2024.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot, Specializing smaller language models towards multi-step reasoning, 2023.
Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and I. Lingpeng Kong. Self-guided noise-free data generation for efficient zero-shot learning, 2023a.
11

---

Published as a conference paper at ICLR 2025
Jiahui Gao, Renjie Pi, Tianyang Han, Han Wu. Lanqing Hong, Lingpeng Kong. Xin Jiang. and Zhenguo Li. Coca: Regaining awwreeysss multimodal large language models with constitutional calibration, 2024a.
11365
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin. Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model, 2023b.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304. 15010, 2023c.
Peng Gao, Renrui Zhang. Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for family of multi-modal large language models. arXiv preprint arXiv:2402.05933 2024h.
Google. Gemini: A family of highly capable multimodal models. 2023. URL
googleapis ccm/ deepmind wesei-eeiia//// meeeda//gemi.//gmmmmm... 1 report. .di
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang. Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A lool-integrated reasoning agent for mathematical problem solving, 2023.
Jiaming Han, Renrui Zhang. Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, al. Imagebind-Ilmi Multi Multi-modality instruction tuning, arXiv preprint wtXiv:2309003900 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optinat large language models. arXiv preprint 2200v 15556, 2022.
Dong Huang, Guangtao Zeng. Jianbo Dai, Meng Luo, Han Weng, Yuhao Qing. Heming Cui, Zhijiang Guo, and Jie M. Zhang. Effi-code: Unlcashing code efficiency language models, 2024. https://aaxx abs
2410. /t/////b///1...../
10209
Xin Lai. Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXi::208..000 00692, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
Qintong Li, Jiahui Gao, Sheng Wang, Renjie Pi, Xucliang Zhao, Chuan Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Forewarned is forearmed: Leveraging Ilms for data synthesis through failure-inducing exploration, 2024.
16736
Zhenwen Liang, Tianyu Yang. Jipeng Zhang, and Xiangliang Zhang. Unimath: A foundational and multimodal mathematical reasoner. In EMNLP, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li. Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: i
Improved reasoning, ocr, and world knowledge, January 2024.
bloq/ 2024-01 01 01-0000--000000 1lava Slava-nnxt..... /
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In The 59th Annual Meeting of the Association for Computational Linguistics (ACL). 2021.
Pan Lu. Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Rei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with gpt-4y, bard, and other large multimodal models, 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding, arXiv preprint arXiv:2202.04538 2022.
12

---

Published as a conference paper at ICLR 2025
Maizhen Ning. Qiu-Feng Wang, Kaizhu Huang, and Xiaowei Huang. symbolic characters model for solving geometry problems. In Proceedings of the 31 ACM International Conference on Multimedia, MM '23, page 7767-7775, New York, NY, USA, 2023, Association for Computing Machinery. ISBN URL bttps: 011/1......1.11111111111
9798400701085. doi: 10. 145/3351178333612151 3612570
OpenAI. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-2777 27744, 2022.
Baolin Peng. Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023.
Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. Multimath: Bridging visual and mathematical reasoning for large language models, 2024.
00147
Renjie Pi. Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao. Jianhua Han, Hang
Xu, Lingpeng Kong, and Tong Zhang. Detgpt: Detect what you need via reasoning, 2023a. URL ptt:s
7/ //arxiv.om orgg
wwtp/bs/////ooooo//////////aaare abs/ 2305 14167
Renjie Pi, Lewei Yao, Jiahui Gao, Jipeng Zhang, and Tong Zhang. Perceptiongpt: Effectively fusing visual perception into Ilm, 2023b. hRtps: org/ abs/2311.... 06612
Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization, 2024a. URL https://
arxiv wwrivi.orgg/oo abs/ 2403. 0370
Renjie Pi, Jianshu Zhang, Tianyang Han, Jipeng Zhang, Rui Pan, and Tong Zhang. Personalized visual instruction org/ abs/ 2410. 07113
tuning, 2024b. URL
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Mrinmaya Sachan and Eric Xing. Learning to solve geometry problems from natural language demonstrations in textbooks. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017), pages 251 251-261, 2017.
Mrinmaya Sachan, Kumar Dubey, and Eric Xing. From textbooks to knowledge: A case study in harvesting axiomatic knowledge from texthooks to solve geometry problems. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 773-784, 2017.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, François Yon, Matthias Galle, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211 100, 2022.
Minjoon Seo, Hannanch Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1466-1476, 2015.
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See- Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models, 2024. URL
htttps://ark: wttp:///..rr//////////////// 17294
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun
Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. Using deepspeed and megatron train eecaatron-turing nlg 530b, large-scale generative language model. arXiv preprint arXiv:2201. //00/////99, 2022.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction- follow them all, 2023.
Rohan Taori, Ishaan Gulrajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An insiluction-followiing llama model. com/tat.sus wab/ strn paaa 2023.
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint Xiv:2330 13971, 2023.
Xuezhi Wang. Jason Wei. Dale Schuurmans. Quoc Le, Chi, Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 10Xiv2208..1171,22222 2022.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1903, 2022.
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. Large language models are better reasoners with verificaain.. CoRR, abs/2212.09501 2023.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang. Chung- Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v(ision). 2023.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. Zerogen: Efficient zero-shot learning via dataset generation. In Empirical Methods in Natural I Language Processing, 2022a.
Jiacheng Yc, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. ProGen: Progressive zero-shot dataset generation via in-context feedback. In Findings of the Association for Computational Linguisticx: EMNLP 2022, 2022b.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration, 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. 2023.
Jipeng Zhang. Jianshu Zhang, Yuanzhe Li, Renjie Pi, Rui Pan, Runtao Liu, Ziqiang Zheng, and Tong Zhang. Bridge-coder: Unlocking Ilms' potential to overcome language gaps in low-resource code, 2024a. URL
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams visual math problems? arXiv preprint arXiv:2403. 14624, 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models, 2023.
Wenwen Zhuang, Xin Huang. Xiantao Zhang, and Jin Zeng. Math alignment to enhance mathematical reasoning, 2024. URL https::/s///aa www..orggg...................... aws/aa/a 08640
Math-puma: puma: Progressive upward multimodal
A REPRODUCIBILITY STATEMENT
We develop an innovative, automatic, and scalable tata-generation method that leverages a text- only GPT to create multimodal data. Detailed instructions for data generation are available in Appendix D We employed a standardized training protocol to facilitate fair comparisons, with implementation details provided in Appendix E Our code, data, and models are publicly accessible at hthttps://qithhu com wwom/pppiluze C- LLaVA
B ETHICAL IMPACT
Our work aims to enhance the geometric problem-solving ability of MLLMs without introducing any additional ethical concerns resolving existing ones.
14

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXAMPLES
C.I FULL EXAMPLE FOR FACTUAL QA PAIRS GENERATION
The factual QA pairs generation process consists of two phases: 1) Translating the human-labelled logic form into detailed information items and summary of the diagram description. 2) Generating factual QA pairs based on the provided information and summary. We utilize ChatGPT-3.5 as the LLM for data generation. The full example is shown in Table 17 The detailed prompts are shown in Table 20 and Table 21
C.2 ADDTITONAL EXAMPLE FOR EQUATION SOLVING(ES) STRATEGY
We attach additional synthetic examples by using Equation Solving(ES) strategy in Table
D PROMPTS
D.1 PROMPT FOR GENERATING GEOMETRIC CROSS-MODAL ALIGNMENT DATA
The detailed prompt for generating geometrical image description is shown in Table and the prompts for generating factual QA pairs are shown in Table 20land and Table 21
D.2 PROMPTS FOR GENERATING INSTRUCTION DATA
The prompt for Equation Solving (ES), Value Scaling (VS), Re-Formulating Condition Unknown (RCU), and Sentence Paraphrase (SP) are shown in Table 22] Table 233 Table [24] and Table [5]
respectively. "[The Original Full Information]" in Table 24ble224eenote the original question with answers completed based on the ground truth.
E MORE EXPERIMENTAL DETAILS
E. MODEL ARCHITECTURE
We utilize the LLaVA (Liu et al. 2023) architecture for our model. The model mainly consists of a large language model (LLMM) such as Vicuna Chiang et al. (2023), pre-trained vision trans- former Radford et al. (202I) (VIT) as image encoder. addition, a projection layer is required to map the visual features from the image encoder to the same dimension the LLM.
During inference, given an image and a textual instruction, the image encoder first extracts the visual tokens from the image, which then mapped to the dimension of LLM's embedding space via the projection layer. Then, the mapped image features are concatenated with text embeddings to serve as the input the LLM. Subsequently, the LLM begins perform next-token-generation..
E.2 MODEL TRAINING
We train our G-LLaVA in two phases, namely 1) geometric visual- Langual alignment, and 2) geometric instruction tuning. In both phases, we leverage the conventional language modeling loss, which can be formulated as follows:
L(Sur Smn 1)
Σ log St.
F(s Sm,,
(I)
where F represents the model. is the geometric figure; Star and S,n represent the target and input sentences, respectively; S denotes the 1th token of the target output, and L. stands for its length.
15

---

Published as a conference paper at ICLR 2025
GPT4-Y
As shown the figrt,, All the diander the somicicll angle AFCC 9000 poin D the miápoint of an AC, than mode DAB ( A:400 Cloises: A:40" B:50" C:63" C:65" D.50
To salve problen, solation:
apply some peoeetry pencipies sispaby-
seniciii ZACB right because angle suttended by dameter in semorccle always night ingle.
Given
Since ZAC w right angle and ZABC SI" can find usnn the fact that the angles in tringl sim 18tr
<BAC- 1807 -ir s0 407
G-LLAYA
Point D the midpooni of AC, wlich mam - DC and ZDAB 2DAC
becnuse sngles sihended
the angllee
Connecting RD, is shown the diagram Recarse point the midpoast segaenn AC, thet is CD equals AD. Thereffone augle ABD equals ingle degress, which degroes Since
milpon
be of which 45") bossuee AD DD So, <DAC
angle ABC
2255;
angle AB the diameter of the semicitle, angle ADB is 90 degrees. Thersfono, ingle DAB is degnes minus 25 degrees, which degres. Therefore,
2. Sence ZRAC and ZDAB
ZDAB 409 22-
Tharefers, de coenss uower is the 2DABis is However, this uption not listl multiple--aae choics atswers. There night cror the question or in the interpretatius Plases resiow quesiin
providell
information steps
In purtlicogram ABCD, the diegonal AC and BD intersect at point O, ingle DAC 42.0, CBD = 23.0, then angle COD (s
GFT4-Y
-ZDAC-44
-ZCRD 21*
Chosces: A:ll B:63 C:65 1x67
Iapenilclagran ABCD, disgunl bisert aach otler, ZA00 200D <800-
4COB
Since opposic ingle in paralelogmen squal, we have 2DAB 2DAC
ZCDB ZCRN 21"
G-LLVAA
Deng the conecrative anglex in purilehiggen
1807
ZAOD 2DAB 150
strught O,
Since quadnilterrl, ABCD parallelogram, have AD parallel DC. Theredore, BCA equal
Gium 23", ZCOD sabimasing from
degrees Augle angle CBD and angle BCA, which is degrees plus 23 degnes, equal 65 degress. Therefeee, the cornect inswer
calealated sumi
Therefore, <COD 157
Figure 4: Other cases of geometric problem solving by GPT-4-V and G-LLaVA.
E.3
SETUP
Dataset. We generate the alignment data and instruction data utilizing training set of GeoQA+ (Cao and Xiao 2022) and Geometry3K (Lu et al. [2221). More specifically, the contrastive question-answer (QA) pairs the alignment data are generated using Geometry3K, which features human-labelle logical forms. Note that GeoQA+ covers the training set of GeoQA (Chen et al. 2021), and share the same val/test set as GcoQA (Chen et al.) 2021). More details of data split on GcoQA and GeoQA+ is listed in Table 16 Our approach results in 60K alignment data samples, and more than 1 11KK instruction data samples.
We compare MILMs the geometry problems solving (GPS) task testmini split of Math Vista (Lu jet al. (202)), a popular benchmark for visual math reasoning. This benchmark has assessed several leading MLLMS such GPT4-V and Gemini. We also compare our model with traditional in-domain models on the test split of GeoQA following (Liang et al. 2023). Note that the GPS task in Math Vista testmini set collected from four source datasets: GeoQA (Chen et al. 202T), Geometry3K (Lu
et al. 2021
GcoQA+ (Cao and Xiao 2022), and GEOS (Soo et al. 2015
Implementation Details. We employ ChatGPT (gpt-3.5 3000-335-turbb-011333 for data generation. The LLM part of G-LLaVA utilizes ILAAA-2 (Touvron et al. 2023) as the language model and employ the pretrained vision transformer Radford et al. (2021) the vision encoder. We conduct experiments with both 7B and 13B LLMs.
For model training, we found the testmini split of Math Vista contains some examples of Mix- rk of GeoQA+, we remove the samples that also appears in testmini split of Vista. During training, the learning rate is set to 3e We expand the images into squares during training, where the extended background color is set to white. For image augmentation, we set the maximum translation distance to 0.25 of the length of longer side. If not otherwise specified, the models trained for
16

---

Published as a conference paper at ICLR 2025
epoch for cross- cross-modal aligmment and 2 epochs for instruction tuning, respectively. And the batch sizes are set to and 32 per GPUs, respectively. For training G-LLaVA-7B: each run requires 10 hours on 8 A40 GPUs (48G of memory).
F ADDITIONAL CASES GEOMETRY PROBLEM SOLVING
We provide additional cases of geometry problem solving using different models in Figure results show that G-LLaVA can provide accurate reasoning paths and answers.
The
Dataset
Train
Validation
Test
GeoQA= (Cao and Xiao 2022
6027
745
754
GeoQA Chha et al. 2021
3499
745
754
Table 16: Data Split of GeoQA and GeoQA+.
Factual QA Pairs for Basic Elements
Logic Form:
Similar (Triangle (A, C, Triangle (C,
Triangle B,
Triangle (A, B)
Equals (lengthOf (Line (C, B)), 5)
Equals (lingthof (I.ine (C, D)), , 3)
Equals (LengthOf (Line (B, D)), 4)
PointLiesOnLine (D, Line (A, Perpenaiculat (Line (A, C), Line (C, B Verpendicules (Line (C, D), Line (A, D)
Phase-1 Generation
Info Items:
The geometric figure have the following information:
Info I: The figure represents a triangle with vertices A, B, and C. Triangle ABC is similar to triangle CBD.
Info 2:
Info 3: Point D lies the line segment AB.
Info 4: Line segment AC is perpendicular line segment CB.
Summary:
The figure depicts triangle ABC and triangle CBD, which is similar triangle ABC. Additionally, triangle ACB is also present in the figure. The lengths the segments CB, CD, and BD 5, 3, and 4 units respectively. Point D lies on line segment AB, and perpendicularity is indicated between the line segments AC and CB, as well as between the line segments CD and AD.
Phuse-2 Generation
QA pairs:
Question: Does the diagram include a triangle formed by vertices A, B, and D?
Answer: No, the diagram does not include triangle formed by vertices A, B, and D. The triangles
present ABC and CBD, and the point D lies the line segment AB.
Question: Does the diagram contain triangle with vertices A, C, and
Answer: Yes, the diagram contains triangle ACB, which is the same as triangle ABC.
3. Question: Is point D the lies on line BC?
Answer: No, point D is not on lines BC. The point D lies the line segment AB.
Question: Is line segment AC perpendicular line segment CB?
Answer: line segment AC perpendicular to line segment CB.
Table 17: Geometric image description and factual QA pairs for understanding basic elements. The sections in blue display the responses generated by ChatGPT.
17

---

Published as a conference paper at ICLR 2025
Variable Substitution and Equation Solving
Question:
Given railway crossing barrier with two arms, where the shorter arm has length of 'a' units (a 1.0) and the longer arm has length of 'b' units longer arm's end when the
16m
16.0). determine shorter arm descends by 'd' units (d 0.5). Express your answer as a function 0.5m of 'a', 'b', and 'd".
elevation "y" units
Answer:
To find the clevation "y" of the long arm's end, we apply the property similar triangles and establish
the equation _ = 5. Substituting the given values, we get 2.5 I.
Solving for 'y' leads to
16.0
y hxd which simplifies to 16.0x0.6 Therefore, "y" equals 8.0 units, indicating that the long 1.0
rises by 8.0 units when the short arm descends 0.5 units.
Question:
In railway crossing lever system, the lengths of the shorter and longer arms are "s" units (s 1.0) and 'I' units (1 = 16.0) respectively. If the shorter arm lowers by 'deltaS' units (deltaS = 0.5), what is the increase 'deltaL' in the length of the longer arm?
Answer:
The increase 'deltal'' in the longer can be found using the proportionality relationship of similar triangles. The equation be expressed deltaS
deltal Inserting the given values, the equation Solving for 'deltaL' yields dellaL Ixdelta8 which simplifies to
becomes 0.5 1.0
dettaf.
16.0
deltaL = 16.0x0.5 Thus, deltaL is 8.0 units.
1.0
Question:
At railroad crossing, the short arm of barrier 1.0 unit long and long arm is 16.0 units long. the short arm drops 0,5 units, by how many units will the long arm rise? Represent the rise of the long arm as 'h'.
Answer:
The triangles formed by the shorter longer arms as they descend are similar. The rise of the long arm 'h' can be calculated using the property of similar triangles, the equation 0.5/h 1/16. Solving for 'h' gives h = 16 0.5 / 1, which equals 8.0 units.
Table 18: Synthetic example via replacing values to unknown variables and solving the equation.
Prompt for Generating Geometric Image Description
As specialist in geometric mathematics with expertise in multi-modal (vision-language processing, your skill involves reimagining diagrams based the information provided questions and answers. will provide you with a question-answer pair related to geometric diagram, and your task be to mentally reconstruct the corresponding diagram and provide a description without involving calculations or mentioning specific numerical values. To get started, let's consider three examples:
QA pair:
Question: [Sample Question]
Answer: [Sample Answer]
Description:
[Sample Description]
I need a correct and comprchensive description of the diagram, avoiding any specific numerical data such as exact measurements angles. Clearly identify the points within the shapes (c.g., the cireumference of the circle).
should not mention details specific to this given question.
Table 19: Prompt for Generating Alignment Data.
18

---

Published as a conference paper at ICLR 2025
Prompt for Generating Factual QA Pairs for Basic Elements: Phase-1 Generation
You helpful expert in geometric mathematics. will first provide you with few-shots each contain a question and the associated correct answer. Then, you should answer a new question.
Example 1:
Question:
logic_form:
[Sample Logic Form]
Above logical forms for geometric figure. You should describe the information in this figure.
Answer:
The geometric figure has the following information: Info 1: [Sample Info 1]
Info 2: [Sample Info 2]
Summary:
[Sample Summary]
Example
You must response in the following template:
The geometric figure have the following information:
Info 1: <INFO>
Info 2: <INFO>
Info 3: <INFO>
Summary: <SUMMARY>
Table 20: Prompt for generating factual QA pairs for basic elements in Phase- Generation.
19

---

Published as a conference paper at ICLR 2025
Prompt for Generating Contrastive QA Pairs for Basic Elements: Phase-2 Generation
As an expert in geometric mathematics, will provide you with a detailed description a geometric diagram. Your task will be to formulate contrastive questions and answers related to this diagram, such 1) whether the geometry has triangle elements? 2) whether the description of the relationship between point and line correct?
The questions should he directly related to the descrihed diagram and should he answerable based on the provided geometric information without requiring numerical calculations.
logic forms:
[Sample Logic Form]
info_items:
The geometrie figure has the following information:
Info I: [Sample Info 1]
Info 2: [Sample Info 2]
Summary: [Sample Summary]
QA pairs:
pair_
Question: [Sample Question]
Answer: [Sample Answer]
pair
Now I will give you the geometric information. You should give questions and answers related to this diagram. The questions should pertain the specifics of the diagram, and they must be answerable using the given geometric details.
You should answer using the following template:
QA pairs:
pair
Question: <QUESTION>
Answor<ANSWER>
pair
Question: <QUESTION> Answer<AASSEE:
pair_3:
Question: <QUESTION>
Answer<ANSWERS
Table 21: Prompt for generating factual QA pairs for basic elements in Phase-2 Generation.
20

---

Published as a conference paper at ICLR 2025
Prompt for Equation Solving (ES)
As proficient in geometric mathematics, please rephrase both the question and the answer utilizing your advanced knowledge in the field. You should substitute variables with ones such y, Z, any other suitable replacements you can envision to solve question. Furthermore, certain steps, you should attain the solution by solving an equation involving these variables.
Here the examples:
Current question-ansee answer pairs:
Question: [The Original Question]
Answer: [The Original Answer]
New question-ance answer pairs: Rephrased question: [New Question] Rephrased answer: [New Answer]
Please rephrase both question and the answer utilizing your advanced knowledge in the field following the above example. You should end the answer with "Resul:[RESULT"".
Table 22; Prompt for equation solving stategy.
Prompt for Value Scaling (VS)
As specialist in increasing numeric values, your role adjust the numbers in both the question and the answer by proportionally enlarging them, for instance, by multiples like 2 times. Example:
QA pairs:
Question: [The Original Question]
Answer: [The Original Answer]
New QA pairs:
Rewritten [uestion:[New Question]
Rewritten answer: [New Answer]
You should ensure that the solution is accurate. Conclude your response with: 'Therefore, the answer option <CHOICE>' Focus scaling up values, rather than performing calculations. You should the answer with "ResuIt:
You should answer using the following template:
New QA pairs:
Rewritten question: <QUESTION>
Rewritten answer: <ANSWER>
Table 23: Prompt for value scaling strategy.
21

---

Published as a conference paper at ICLR 2025
Prompt for Re-Formulating Condition Unknown (RCU)
As geometric mathematics expert, please rephrase both the question and the answer based on a given question-answer pair. will replace known variable in the question with 'u' create new question, your task solve using other information the previous question-answer pair.
I will also provide you with the full information. You should solve the new question and promise the solution is correct.
Example:
QA pairs:
Previous question: [The Original Question]
Previous answer: [The Original Answer]
Full information:
[The Original Full Information]
New QA pains:
New question: [New Question]
New answer: [New Answer]
You should finish the <ANSWER> in the You should end the answer with "Re- sult:/RESULTI..
You should respond using the following template: New answer: <ANSWER>
Table 24: Prompt for Re-Formulating Condition as Unknown (RCU) strategy.
Prompt for Sentence Paraphrase (SP)
You are an assistant to help me rephrase questions. You should rephrase my questions to different with complete information. Here are some examples:
Question: [The Original Question]
Rephrase the ahove question: [Rephrased Question]
You should help me rephrase new question following the given examples. For each question, you should rephrase 10 questions in the following format:
Question: <RAW MIW_QUUSTION>> Rephrase the above question:
Q1 R..RREEHRASED 2 QE.REPHRASED L QUESTION>
Table 25: Prompt for sentence paraphrase strategy.
22


Reviews:
[{"review_id": "Epqk6S83ds", "reviewer": "Reviewer_1KS1", "summary": "Current Multi-Modal Language Models (MLLMs) struggle to solve geometry problems for lack of comprehension of geometry information. This paper generates a large-scale geometric problem dataset named Geo170K by GPT3.5, which contains more than 170K geometric image-caption and QA pairs. The data is made up of two components: the alignment data and instruction data. The former is generated through logical forms in Geometry3K while the latter augments QA pairs in GeoQA+. Besides, a model based on LLaVa is proposed to be trained on Geo170K. Numerical results show that G-LLaVA outperforms existing traditional methods and MLLMs like GPT4-V on the geometry problem solving (GPS) tasks.", "questions": "1. Will the RCU process generate a question with multiple answers, since sometime the cause and effect are not necessarily sufficient and necessary conditions? And unfortunately, the data correctness verification cannot recognize such case.\n2. In Table 10, the performance gain led by cross-modal alignment seems limited, and why the fixed LLM performs better than tunable LLM?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper curates a large-scale geometry dataset (28 times larger than the biggest existing geometry dataset) containing both detailed description of geometric images and reasoning paths, which may help MLLMs improve their ability in geometric comprehension.\n- Data augmentation techniques like value scaling, equation solving, Re-Formulating conditions as unknown, and sentence paraphrase make it convenient to enlarge the size of dataset and enhance robustness.\n- Extensive experiments validate G-LLaVA’s superiority in geometry problem solving. Experiments on MathVerse demonstrate its generalization ability.", "weaknesses": "- The experiments are all about choice questions. Some works also involve proving[1] and completion[2] questions, which can be more difficult. Experiments on such questions are neglected.\n- In many experiments, the strongest existing model GPT4-V is not included, making the result less convincing.\n- The proposed G-LLaVA selects LLAMA-2 as LLM and a pretrained ViT as the vision encoder. A comparative analysis or experiment on other architecture or pretrained model may help justify this choice.\n\n[1] Chen J, Li T, Qin J, et al. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression[J]. arXiv preprint arXiv:2212.02746, 2022.\n\n[2] Li Z Z, Zhang M L, Yin F, et al. LANS: A layout-aware neural solver for plane geometry problem[J]. arXiv preprint arXiv:2311.16476, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H99kD23um8", "reviewer": "Reviewer_tzwZ", "summary": "This paper presents G-LLaVA, an MLLM focused on geometric problems. The authors propose that the difficulty in geometric problems lies in logical structure compactness. Subsequently, they introduce a new dataset, Geo170K, with geometric cross-modal alignment data and geometric instruction data to enhance the geometric perception ability of MLLM.", "questions": "In Table 11, why does just including SP increase accuracy by 8%? This type of data augmentation doesn't enhance the model's understanding of geometry. Does this imply that the performance gains are due more to an improved understanding of the problem description rather than geometry?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Understanding geometric problems is important for MLLMs.\n2. Presents a dataset with geometric alignment and instruction data.", "weaknesses": "1. The novelty is limited, as it primarily fine-tunes an existing MLLM.\n2. The models compared in Table 14 need to be updated; stronger baselines, such as QwenVL2, InternVL2 should be considered.\n3. Since the authors claim that G-LLaVA enhances geometric understanding from image input, it would be interesting to see the G-LLaVA's  performance improvement when only input $Q$ is provided, in Table 8, 9 and 14.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IadYPwg9NZ", "reviewer": "Reviewer_Nr4G", "summary": "This paper investigates the capabilities of existing MLLMs in geometric problem-solving tasks and introduces the Geo170K dataset, which contains over 170K image-caption and QA pairs. It also proposes training the G-Llava using the proposed dataset and validates its performance on the GeoQA dataset, demonstrating its effectiveness.", "questions": "Please check the previous sections.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work explores an interesting topic, geometric problem solving, and currently, most MLLMs perform poorly on this task.\n2. The expression and organization of this manuscript make it easy for readers to understand.\n3. The dataset proposed in this manuscript is relatively large and includes image-caption and question-answer pairs data.", "weaknesses": "1. Novelty and Technical Contribution: The proposed geometric dataset construction approach relies on the existing logical annotation data and uses ChatGPT for automatic labeling. Besides, the overall model structure and training strategy mainly apply the existing techniques, lacking sufficient innovation. Please explain the differences in model structure or training strategy during the rebuttal phase and whether the authors made customized designs on a geometric domain. No specific considerations for the geometric domain were found.\n2. Data Quality Issues: Since annotations are generated automatically by ChatGPT, is there any hallucination in the dataset? Is there any quality inspection conducted to support the claim of a \"high-quality dataset\"? \n3. Insufficient Evaluation: The paper lacks quantitative comparisons on geometry-task-related benchmarks like UniGeo, Geometry3K, and PGPS9K, commonly used in prior studies [1-4].\n\n[1] Chen J, Li T, Qin J, et al. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression[J]. arXiv preprint arXiv:2212.02746, 2022.\n[2] Liang Z, Yang T, Zhang J, et al. Unimath: A foundational and multimodal mathematical reasoner[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 7126-7133.\n[3] Zhang M L, Yin F, Liu C L. A multi-modal neural geometric solver with textual clauses parsed from diagram[J]. arXiv preprint arXiv:2302.11097, 2023.\n[4] Li Z Z, Zhang M L, Yin F, et al. LANS: A layout-aware neural solver for plane geometry problem[J]. arXiv preprint arXiv:2311.16476, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "91e0rillHL", "reviewer": "Reviewer_UDES", "summary": "The paper is a pioneering work about Geometry Problem Solving using MLLM. The contribution is to provide a dataset and also a baseline using LLaVA.", "questions": "Please refer to the Weaknesses for more details. All in all, the authors are encouraged to append the discussion or experiments about recent progress in both dataset and methods.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "It's a pioneering work on MLLM's mathematical reasoning especially focusing on geometry problem. Compared with its earlier version, the authors added more experiments on recent datasets like MathVerse.", "weaknesses": "The work was a milestone in developing mathematical skills of MLLM and the authors tried to append results on a latest dataset. However, the research in the area moves fast in both dataset construction and methods. \nDespite the updates, the paper is still flawed in\n1. Its targeted problem, i.e., geometry problem solving (GPS), is limited nowadays. The mathematical reasoning of MLLM evolves fast with the corresponding dataset. Now, it also covers the function plots in MathVerse and chart in MultiMath.\n2. The baseline needs a fair comparison. Even though the problem is confined to GPS and methods in comparison is confined to open-source methods, the methods like MathLLaVA, MultiMath and Math-PUMA are encouraged to be compared with.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)