Do Large Language Models Truly Understand Geometric Structures?
Authors: Xiaofeng Wang
Yiming Wang
Wenhong Zhu
Rui Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Geometric Ability Evaluation
Geometric Relationship Identification
Primary area: datasets and benchmarks
TL;DR: This paper introduces a dataset to evaluate large language models’ understanding of geometric structures and proposes a method to enhance their ability to identify geometric relationships.

Abstract: Geometric ability is a significant challenge for large language models (LLMs) due to the need for advanced spatial comprehension and abstract thinking. Existing datasets primarily evaluate LLMs on their final answers, but they cannot truly measure their true understanding of geometric structures, as LLMs can arrive at correct answers by coincidence. To fill this gap, we introduce the GeomRel dataset, designed to evaluate LLMs’ understanding of geometric structures by isolating the core step of geometric relationship identification in problem-solving. Using this benchmark, we conduct thorough evaluations of diverse LLMs and identify key limitations in understanding geometric structures. We further propose the Geometry Chain-of-Thought (GeoCoT) method, which enhances LLMs’ ability to identify geometric relationships, resulting in significant performance improvements.

Published as a conference paper at ICLR 2025
Do LARGE LANGUAGE MODELS TRULY UNDERSTAND GEOMETRIC STRUCTURES?
Xiaofeng Wang, Yiming Wang, Wenhong Zhu, Rui Wang" Shanghai Jiao Tong University (hanyedy, yiming. wang, zwhong714, wangrui12] @ @sjtu.edu.cn
Geomeniic Struetum Bescriptian with Nanural Language
For irtangi IBC, dic inctttintoneee From paint perperdistoon bot are draan the those adles 50 s pencica for to Ac, or perser dloua la bi. and de spppeedissll to the colo
i tlermaar of tru ABC we leon that perperdaaaa UL, SP apersendaele and Us pepperdiata to
dilte to Incomeet Mentifcation Gicometri Relaternhi)) Sendain 44 ddd to jUsmaningii Reasering Thersfor, we cn concide that 00 (oorect Arower]
Duestien Devripttore
Find lengt Eain nouner
or X
Common
Question
Geometric Relationship Identification
Goometric
Problem
Rationalc
Caluulation
Corresponding Geometric Relationships
Ranult Our GromBol
Desiription (ielation I DS the oft tuite orangagg ARC?
because 60 perpercicuoo the
Question Description (lalation between Iradj
UM gencrted Arswer:
Carnol be vol gieon card tarn are inufiiient desaminn whather they are porccccukk X
Quastion Duscription (talation between bwo anghol nos and cc0 coppleeeeen
UM gameuted Answer:
You 4000 and 2000 ar complenaniare lacass trinnll coo op perpandea der to pc.
cean differentiale my understanding of each geometric component
Figure I: The general process of solving geometric problems (Middle). Within this process, identify- ing the geometric relationships is fundamental step, one must first accurately identify the geometric structures, then apply theorems for reasoning or calculation to reach the final answer. Compared to traditional geometric datasets that only assess the accuracy of final answers (Top), we extract the fundamental steps of geometric relationship identification (GRI) to create the GeomRel benchmark, which evaluates whether LLMs truly understand geometric structures (Bottom).
ABSTRACT
Geometric ability is a significant challenge for large language models (LLMs) due to the need for advanced spatial comprehension and abstract thinking. Existing datasets primarily evaluate LLMS on their final answers, but they cannot truly measure their true understanding of geometric structures, as LLMs can arrive at correct answers by coincidence. To fill this gap, we introduce the GeomRel dataset, designed to evaluate LLMs' understanding geometric structures by isolating the core step of geometric relationship identification in problem-solvvne benchmark, we conduct thorough evaluations of diverse LLMs and identify key limitations in understanding geometric structures. We further propose the Geometry (GeoCoT) method, which enhances LLMs' ability to identify geometric relationships, resulting significant performance improvements. Our work is accessible at https//githbb.ccm//bayyyCyCCC
Using this
INTRODUCTION
Mathematical ability has gradually become key benchmark for evaluating large language models (LLMs) (OpenAl 2022 Peng et al.) 2023 Touvron et al.) (202a)). as it can measure whether a model has preliminarily developed rigorous logic and abstract thinking (Sternberg & Ben-Zeev 2012). Geometric ability, as an important branch, demands more spatial reasoning and a profound understanding of geometric diagrams. which presents even greater challenges for models. Recently, many efforts have focused on testing the geometric ability of L.MMs, widely collecting or constructing
"Corresoonding Author

---

Published as a conference paper at ICLR 2025
large-scale datasets comprehensively evaluate whether the models possess strong geometric abilities (Ye et al. 2008 Trinh et al. [024] Zhang & Moshteghi 2024 Zhang et al. 2024a).
Geometric ability relies accurately understanding geometric structures (Linquuist & Shulte 1987)
When humans tackle geometric problems, they usually start by identifying relationships within the structure, then reason and calculate based those relationships to arrive at final answer (Clements & Battista 1992, Buckley et al. 2019). This process is illustrated in the middle of Figure In contrast, LLMs generate answers in an end-to-end mode without explicitly decoupling these steps. They produce a blended rationale that includes the final answer, making it challenging to assess the correctness of their step-by-step reasoning. Consequently, current evaluation methods primarily calculate the accuracy by exactly matching the final answers with reference answers.
However, this risks misrepresenting the LLMs' true geometric abilities. We manually decou- ple the part of the LLM-ggeneraed rationale in- volving geometric relationship for each sample on three popular datasets, and calculate the in- dividual accuracy of this and the final answer, respectively, as shown in Table T Normally, the accuracy of identifying geometric relationships should be higher than the final answer, as both geometric relationship identifications and rea-
Table 1: Accuracy comparisons between the final answer and GRI under three existing datasets with the GPT-3.5-Turbe model. We randomly sample 50 text-only geometry problems from each dataset, and the GRI accuracy is manually verified.
Dataset
Answer
GRI Acc
MATH Hendrycks et al 2021b
0.18
0.22
Genmelry
0.12
PGPS9K Zhanng er
0.32
0.14
soning calculations influence the latter. However, we are surprised to find that on these datasets, the accuracy of geometric relationship identification is significantly lower than the answer accuracy, This indicates that in many cases, LLMs coincidentally produce correct answers despite errors in identifying geometric relationships, as one example illustrated at the top of Figure [T] These findings suggest that, under current datasets with the evaluation paradigm of exactly matching final answers, we are unable to measure whether L.LMs truly understand geometric structures accurately.
To this end, we extract the sub-step of geometric relationship identification (GRI) from mainstream geometric problems and construct dataset called GeomRel. can serve as a minimal module for evaluating model's ability to understand geometric structures. Although it is unable to measure more complex reasoning abilities, the measurement of geometric ability is lossless as it only involves one skill of identifying relationships, which is the foundation of reasoning ability. In Section 2 We present a detailed overview of our GeomRel dataset, including its construction, categorization, and the methods used to generate both basic and advanced versions. Our dataset forms the benchmark for testing their ability to identify and understand geometric structures.
Based this benchmark, we comprehensively evaluate whether current LLMs understand geometric structures in Section Extensive experiments on the benchmark demonstrate that:
Current LLMS perform well in identifying simple geometric relationships but perform poorly in identifying complex structures, especially for Angle-based relationships. The best-performing LLM, GPT-4o, exceeds the Random performance by 48.91% % on the basic GeomRel, but only 20.34% on the advanced GeomRel.
We further investigate how the diversity strategies implemented in GeomRel - such point relabeling and the incorporation of irrelevant information - affect model performances. Our findings suggest that strategically enhancing the complexity of geometric deseriptions can lead to significant performance improvements.
We examine the effects of various prompting techniques, such as Few-Shot (Brown et al. 2020 and Chain-of-Thought (Wei al. 2022 Kojima al.) 2022), on model performances. Our findings indicate that, despite differences in reasoning lengths compared to the original prompts, these techniques do not lead to substantial improvements in geometric identification performance. Additionally, we encounter some unexpected obstacles during the forward reasoning process.
We also fine-tune the open-source model LLaMA3-8B- I-nsstrut using question-answer pairs but find that this does not necessarily improve its understanding of geometric structures.
Finally, in Section 4 to further enhance LLMs' ability in geometric relationship recognition, inspired by the Chain-of- Thain-of-Thought Thought (CoT) technique, we propose the Geometry Chain-of- Chain-of-Thought (GeoCoT) to elicit LLMs identify geometry relationships step by step. The two-stage pipeline first breaks
2

---

Published as a conference paper at ICLR 2025
down geometric structures into points and lines, then precisely extracts relevant information from the breakdown and applies reverse reasoning to overcome reasoning obstacles. This method substantially increases identification accuracy, with an average improvement of 9.15% the basic GeomRel and 14.79% % on the advanced GeomRel in the Few-Shot setting, showing a range of improvements across various domains.
GeomRel: GEOMETRIC RELATIONSHIP IDENTIFICATION BENCHMARK
To examine whether LLMs are capable of understanding geometric structures, we propose the Geomet- ric Relationship Identification (GeomRel) benchmark. We expect that LLMs can accurately identify explicit or implicit geometric relationships according to given geometric structure description.
A geometric relationship is formed by two identical or different geometric elements. Therefore, we first identify the most basic geometric elements and abstract several geometric relations based them conduct finite geometric relationship pool. Next, we gathered geometric scenarios that could give rise to the targeted geometric relationships, starting from definitions and extending to properties. After organizing and supplementing these scenarios, Using this data, we applied rule-based operations, such as condition concatenation, to generate more complex geometric structures. Additionally, we incorporated relatively independent cases, resulting in the advanced dataset. Finally. by adding scenarios where the relationships could not be determined and performing diversity-enhanee distribution of the dataset across the main categories is shown in Table 3 The framework construction process is shown in Figure and we will introduce the details in the following sections.
obtained the basic dataset.
operations, we arrived the complete GeomRel dataset. The
Basic data
Condition pools
Advanced data
point sugament
Lize-based
Line-based
lino A and CD parallel
concatanation
relationsiip
Whal's lae releiionship betweess
segizent AB inside stapes
Cleesfottion
A.B Intersection.....
lines A and C.F? Intersatiion.
Angle-based
Logical chains EF ABB.CDD CD LEF) B, AC
Angle-basel
Geometric Relatonship
What's the velatonshipp
What's
relaiionship bntween
BC and /BBD? Vertical angels
and CRD?
Allermate interrior angeds.
Shape-based based
Additional data
Shape based
inside trproroti BBDD
Is OA a radius al cinde O?
Yes
Yes.
Indelurmnt
?
Adding inslevant & Re labeling Points Increasing Data Diversity
G GeomRel
Figure The GeomRel Framework
2.1 FINITE GEOMETRIC RELATIONSHIP POOL
Before defining the relationship pool, we first identify four fundamental geometric elements: point line 1, angle a, and shape s, following progressive order from zero two dimensions. We define the spaces of point, line, angle, and shape are P. L. A, and S, respectively. They consist of the element S.
space P x
We can theoretically combine any two types of elements (which can be the same) from E to create ten types of relationships (Ca + 4 = 10). However, among these, the point-poine relationship and the relationship between angles and the other three elements have no practical significance, while the shape-shape relationships may involve three-dimensional space, which we will not consider for Therefore, we treat the remaining five as the basic relationship types in the relationship pool, with each relationship type corresponding to several common special relationships (A total of 26 relationships), Further, we categorize them into three main domains: Line-based Angle-based, and
3

---

Published as a conference paper at ICLR 2025
Shape-based relationships. The hierarchical relationships and all specific relationships are shown in which consists of our geometric relationship pool.
Table
Table 2: The geometric relationship pool of our GeomRel dataset.
Domain
Relationship Type
Specific Relationships
Line-Point (< P,P>>
is on I; is i Parallel; Perpendicular; Intersecting
Line-based
Line-Line (< L)
Angle-based
Angle-Angle (< A,A>)
Complementary: Supplementary: Corresponding: Alter- nate Interior; Consecutive interior; Alternate Exterior; Consecutive Exterior; Adjacent/opposite(nn quadrilateral)
Shape-based
Shape-Point (< S,
i insidc/outside/on the boundary of (Polygon, circle) intersects/taageet to/disjoint from s (Polygon, cir- cle); Special segment (Triangle median, Circle ra- dius/diameter/chord, Quadrilateral side/diagonal)
Shape-Line (<8,6>)
2.2
BASIC DATA GENERATION
After constructing the geometric relationship pool, we next generate our relationship identification data. To cover as many geometric situations that involve these relationships as possible, we used a "from definition to properties" approach (Curry 1977). Starting from definitions refers to the funda- mental geometric definitions, which are typically singular and relatively few in number. contrast, deriving from properties involves identifying other geometric elements that exhibit characteristics leading to the same relationship, which often yields multiple cases where the relationship holds. An example are shown in Appendix C.2 This set of data serves as the foundation for our subsequent generation of more complex geometric structures, laying the groundwork for the final preparation of the basic subset in the dataset.
2.3 ADVANCED DATA GENERATION
The advanced data is obtained through series of operations including combination and enhancement built upon the basic data. By pooling these basic data, classifying them and combining them according to the rules we constructed, and then adding additional non-rule generated data and some diversifying operations, we obtained a complete and rich advanced dataset
(Figure 2
We organized geometric de- scriptions from the basic dataset that satisfy the same geometric relationship into subsets defined as condition pool. Each data entry is structured as follows:
condition pools:
Algorithm 1
Merging Geometric Conditions
1: Imput: List of conditions Chain
2: Outpul: merged condition Cmerged
3: Cmerged +- Chain (0)
(Input: ein, Condition ce Output : )out
4: for in Chain[1:] do
5: consist(c, Cmergea)
which is consolidated into a condition pool. For each pool Pj. all conditions within satisfy the ge- ometric relationship R, where pair of elements EEur) are involved. Additionally, since gen- crating augmented data requires more detailed geometric information, for example, conditions
6: 1" Modify the representation of elements in
C to be compatible with Cmerged */ [mmrreeloutput) Cmerged [output] then
7:
if c(input] =
Add c/condition] to Cmerged [condition] 9: Cmerged (oupurt + cloutpur)
8:
10: end if
that satisfy the relationship "point C is on line AB," if A and B are two different points on circle O, we cannot determine the relationship between point C and circle O using the previous relationships. Therefore, we decompose, refine, and supplement some of the relationships previous dataset. In total, 15 condition pools are created for following steps (See Appendix C.3 for examples).
end for
New conditions can be concatenated in the form of conditional chains. Before we concatenate them, we need to determine the logic first. For example, if conditions in pool 1 satisfy the intersection of two lines, and conditions in pool 2 satisfy the parallelism of two lines, then by combining conditions from these two pools, the intersection of two lines, thus combining these logics together. Another example would be if Condition satisfies line segment being inside shape, and Condition 2 satisfies point being
Condition concatenation:
obtain new relationship indicating
4

---

Published as a conference paper at ICLR 2025
inside this line segment, then by combining them, Similar logics exist, and we have created a total of 15 new logics. These logics are formed by chains of two or three geometric conditions. We demonstrate in Algorithm how such conditional chains are combined to form new conditions. Specific example of our concatenation method can be found in Appendix C.4
conclude that point is inside a shape.
Generation of Additional Data: In addition to the data obtained through conditional concatenation, we have also generated some data that relatively independent in terms of geometric structure. Particularly concerning relationships between angles, we generated simple scenarios based on examples from past datasets, and extracted the relationships therein. For instance, in scenarios where two parallel lines intersect a third line, we annotated the relationships between the eight angles formed by the intersection of three lines pairwise, processed them, and integrated them into the dataset.
Disambiguation: Due to the reliance on visual aids for past geometric relationship judgments, ambiguity may arise in purely textual geometric descriptions. We have removed or modified instances where confirmation through textual description alone was not feasible manually. For example, in cases such the intersection of lines AB and CD, where point E lies on line CD, the relationship between point E and line AB (whether collinear) cannot be definitively established when the position of point D as intersection is uncertain. Therefore, chose to exclude such data.
2.4 INDETERMINATE DATA GENERATION
Our geometric relationships are ultimately questioned through multipl-cchice questions. To prevent the blind guessing of results from affecting assessments of geometric capabilities, we have included an option in each question labeled "Cannot be inferred". Simultancously, we have added some similar questions where this option is need to be used, which means the geometric relationships in this question are not clearly defined. By incorporating these specific question, we aim to test the model's s ability to recognize such scenarios and minimize the impact of language model hallucinations.
We generate indeterminate data by removing replacing conditions with irrelevant ones. For example, given the condition "Line AB and CD) are perpendicular to line EF," we deduce that lines AB and CD are parallel. However, changing "CD perpendicular to EF" to "CD perpendicular to GH" prevents us from establishing the final relationship. Ambiguous data generated in advanced scenarios are retained using this option.
2.5 INCREASING DATA DIVERSITY
Adding unrelated information (UI): After establishing the geometric information relevant to relationship determination, enhanced the dataset by incorporating irrelevant information to make the data more realistic and comprehensive. This additional information includes ir- relevant geometric configuration information, such as introducing new points to form new shapes or altering existing geometric relationships to create new ones. Geometric measurement information that does not affect the original geometric structure but adds quantitative details also included. Adding this information enhances the diversity of the dataset and enables evaluation of LLMs' ability to filter information during assessment.
Table 3: Statistics of rela- tion types in the dataset
Cluster
Basic
Advaned
Line-based Angle-based
122
1168
103
275
Shape-based
108
853
Re-labeling Points (RP): In the previous steps, most of the data obtained used conventional alphabetical notation to represent geometric elements, such as quadrilateral ABCD and triangle EFG. To better assess the model's generalization ability, randomly selected and shuffled the notation of these points. For example, the notation of quadrilateral ABCD could be shuffled to quadrilateral DACB, and triangle EFG could be re-labeled as triangle EWG. This process does not alter the geometric structure but introduces variability in the notation.
By randomly sampling proportion of the original data and applying these two operations, we expanded the original dataset by 25%, the final Statistics of the dataset are shown in Table 3
Specific example can be found in Appendix C.5
5

---

Published as a conference paper at ICLR 2025
3 COMPREHENSIVE EVALUATION
In this section, we comprehensively evaluate whether LLMs can truly understand geometric structures through their GRI abilities based on our GeomRel dataset.
3.1 EXPERIMENTAL SETUP
Model and Implementation. We comprehensively evaluate nine LLMs, encompassing both API- based models and open-source models. The API-based models include the GPT series (GPT-3.5- Turbo, GPT-4-Turbo, and GPT-4o) (Wu et al., 2023] OpenAA] (2023), the Qwen series (Qwen1.5,
110B and QwenMax) (Bai al. 2023), , and Claude-3-Ope (Anthropic) 2024). The open-source models include the Llama series LaMMA-2-13B-hhar LIaMA-3-8B-Inan
LlaMA-3-70B-Inse (Touvron et al. 2023b a) are evaluated. For all baselines, we set temperature 5 = We also provide random baseline comparison. The human baseline was derived from benchmark tasks completed by five science and engineering graduate students. Following this, we fine-tuned the LlaMA-3- 8B-Instruct model and conducted detailed study its performance post fine-tuning. Metrics. We evaluate LLMs using large-scale comparisons between model-generaan answers and standard reference answers. By modifying concise prompts, achieve extraction accuracies exceeding 99% across various models. Our assessment extends to evaluating L.MSs' accuracy across diverse domains within the dataset. Additionally, we investigate their performance identifying cases where conditions are insufficient. Specifically, we compute precision, recall, and FI score for the "can't 't be inferred" category. Precision (P) represents the proportion of truly unidentifiable cases among those that the model failed to identify, while Recall (R) indicates the proportion of unidentifiable geometric structures that the model correctly flagged as undeentifiabl. The F1 Fc-score (F1) provides balanced evaluation by tabing the the harmonic mean of P and R.
Prompt Settings. In large-scale model evaluation, we do not include reasoning-guidie text in the prompt (Zero-Shot prompting). While for the default large language model GPT-3.5-Turbo. we also employed other prompting approaches. Specifically, few-shot in-context learning (Few-Shot) (Brown et al. 2020), zero-shot chain-of-thouse
Soro--Sho-Coo) (Kojima et al.) 2023). few-shot chain-of- -fhtuught prompting (Few-Shot-CoT) (Wei et al. (2023) are leveraged to tackle various graph reasoning tasks in the GeomRel benchmark. Task instructions simple and clear for generality, as presented in Appendix E.4
3.2
MAIN RESULTS (TABLE
Table Accuracy performances (%) of different LLMs on our GeomRel dataset.
Line-based
Angle-based
III. Shape-based
Average
Model
Basic
Advanced
Basic
Advanced
Basic
Advanced
Busic
Advanced
GPT-4o
77.87
52.91
66.67
29.00
87.04
58.38
77.86
47.93
GPT-4-Turbo
81.15
53.34
66.67
29.00
80.56
56.98
76. 76.79
46.44
GPrr3.5-Turbo
65.57
46.23
21.75
Qwen1.5-110B
68.85
35.53
59.17
23.75
46.30
35,99
58.11
31.76
QwenMax
69.67
39.90
62.50
23.75
67.59
42.32
66.59
35,32
Claude-3-Oput
75.41
44.35
45.00
20.50
67.59
43.38
62.67
36,08
LLaMA3-70BB-nntnnn
69.67
38.70
34,17
27.25
69.44
40.09
$7.76
35.35
LLaMAS--B-Insstue
63.16
34.14
30.43
32.50
50.92
39.87
48.17
35.17
LLaMA2-13B-CHL
36.84
25.70
30.43
30.00
42.31
25.95
36.53
27.22
Random
28.53
29.43
25.00
20.00
33.33
33.33
28.95
27.59
Human
71.73
39.34
52.86
34.63
90.63
69.41
71.74
47.79
Performances on Basic GeomRel. Except for the earlier BLLMA2-13B-Chhr model, all other models significantly surpass the Random baseline on both the Line-based and Shape-based domains. Notably, the stronger models, GPT-4o and GPT-4-Turbo, achieve over 60% accuracy across all three domains, demonstrating impressive performance. This suggests that L.MSs possess the ability to understand simple geometric structures and exhibit preliminary GRI abilities.
Performances on Advanced GeomRel. We note that performances in the advanced GeomRel significantly lower than in the basic GeomRel across all domains. Except for the GPT family of models, the results advanced GeomRel are nearly Random performance levels. These findings

---

Published as a conference paper at ICLR 2025
suggest that more complex structures pose major challenge for accurate identification by current LLMs, highlighting their limitations in stronger GRI abilities.
Angle-based Relations are Particularly Difficult for LLMs. In comparison to Line-based and Shape-based domains, LLMs demonstrate notably weaker performance in the Angle-based domain. Specifically, the performances in the advanced GeomRel approach Random levels, indicating that Angle-based geometric relations are more difficult for LLMs identify.
Evaluation Reasonableness. As shown in Figure 3 there is a positive correlation between the performance of the basic and ad- vanced datasets in line-based and shape-based tasks, indicating that an improved understanding of these foundational relations also enhances recognition of similar relations in more complex geometric structures. However, for the angle-based based subset, we do not observe a positive correlation, likely due to the inability to comprehend the geometric structures present in the advanced subset.
Lon asost
Angu base Shape caons
Soonnng to
Bias between geometric relations that LLMs subjectively fail to identify and those that are objectively unidentifiable. In Table[5] we observe the model's performance related unrec- ognizable geometric relations. Overall, the F1 performance is consistent with that on the full dataset. The stronger models, GPT-4-Turro and GPT-4o, show a balance between precision and recall, suggesting that they can account for unidentifiable geometric structures without hastily giving a "cannot be inferred" response.
Figure 3:
Accuracy correla-
tions between basic and advanced GeomRel.
In contrast, the Qwen series and Claude-3-Opus models exhibit higher recall but lower precision, indicating a ten- dency provide more uncertain answers. This may be related to additional measures these models take to handle hallucinations, leading to lack of confidence in providing accurate answers. On the other hand, the LLaMA series models lean towards higher precision but lower recall, suggesting a tendency to force inferences even when the problem exceeds their capabilities. Examples are shown in Appendix E.3
Table L.M performances related to unrecognized geometric relations.
Basic
Advanced
Model
R
F1
P
R
FI
QwenMax
0.63
0.78
0.70
0.32
0.60
0.42
Qwenl,5-1100
0.54
0.79
0.64
0.30
0.68
0.42
GPT-3.S-Turoo
0.47
0.43
0.63
0.16
GPT-4-Tubo
0.77
0.79
0.78
0.43
0.50
GPT-4o
0.79
0.80
0.79
0.43
0.48
0.46
LLaMA3-7UB
3luude-3-Opus
0.65
0.87
0.74
0.39
0.56
0.46
LLaMA3-8B 8B
0.30
0,16
0.21
0.45
0.14
0.21
0.32
3.3 ABLATION STUDY ON DATA DIVERSITY OPERATIONS
Line-based
Angle-based
Shape-based
70
70
70
Acountt (N))
Accuray
Acourazy
8
40
-0
30
RP
-U1
Figure 4: Ablation study about data diversity strategies of GeomRel on GPT-3.5-Turbo model.
we design several data diversity strategies enhance the evaluation richness of our dataset. Now. we conduct an ablation study on these strategies to explore whether they truly have an effective differentiating effect the LLMs' GRI abilities.
In Section 2.5
Next, we selected portion the original data containing approximately 300 questions that had not undergone diversification operations, ensuring that the proportions of the subsets remained consistent with the overall dataset. This data is then subjected to three different sets of operations: the first set involved point change (RP), the second set involved adding unrelated information (UT), and the third
7

---

Published as a conference paper at ICLR 2025
set applied point change after adding unrelated information (RP + UI). Along with the original data, these four sets were tested on the G model.
The ablation results are shown in Figure 4 variations in the model performances, despite the core relationships in the geometric descriptions re- maining consistent. Additionally, we find that controlling the complexity of geometric descriptions leads to varying performances, which can be either positive or negative. The same operation demonstrated inconsistent effects across different datasets. For example, applying RP the original lines basic data improved performance, whereas it significantly decreased performance for the lines advanced data. Applying UI to the angles basic data led to a notable performance decline, but doing it to the angles advanced data with subsequent point changes resulted in some improvement.
We observe that different strategies result in significant
Interestingly, although both RP and UI are designed increase problem complexity. RP often yielded better results in over half of the comparisons. This was particularly evident for the basic data. We hypothesize that, in some cases, using more complex descriptions may stimulate LLMs' reasoning abilities, thereby enhancing performance.
3.4 INFLUENCE OF PROMPTING TECHNIQUES
Table 6: Accuracy performances (%) under different prompting techniques. We also report the aver- age token numbers consumed during inference in parentheses. This experiment was also conducted using the GPT-3.5-Turbo model, with the same settings as before.
I. Lise-
II. Angle-based
III. Shape-based
Model
Basic
Advanced
Basic
Advanced
Basic
Advanced
Zero-Shot
65.57 (68)
46,23 (156)
54.17 (64)
21.75 (78)
72.22 (110)
43.26 (180)
Zero-Shot-CoT
60.71 (130)
46.07 (277)
46.82 (130)
21.75 (163)
71.48 (166)
40.39 (311)
Few-Shot
64.87 (28)
45.75 (28)
53.29 (31)
24.30 (31)
72.05 (26)
41.52 (25)
Few-Shot-CoT
70.67 (101)
49.94 (128)
59.20 (79)
30.94 (134)
71.53 (110)
46.68 (151)
We also analyze the prompt techniques used, aiming to explore whether different prompt techniques would have a significant impact on the GRI abilities of LLMs. The results are shown in Table 6
In-Context Learning Can be Counterproductive. We observe that neither Zero-Shott-Col nor Few-Shot techniques improve performance compared to Zero-Shot, with Zero-Shot-CoT even causing significant decline in some domains. Although the "let's think slep by step" prompting increases the length of reasoning. it appears that these additional steps do not meaningfully enhance LLMs' understanding of geometric structures and are likely ineffective in influencing results.
Manually-guided Retionales Improve Limited Performances. When we replace the answers in the Few-Shot examples with human-written reasoning steps, we observe some performance improvements (Few-Shot -> Few-Shot Shot-CoT), especially in the Line-based and Angle-based domains. However, this enhancement is limited, remaining within narrow range. In the shape-based domain of the basic GeomRel, we even see a slight performance decline. These findings suggest that while decomposing the rationale into steps enhance performance, the gains are limited.
Longer Reasoning Length Does Not Mean Higher Accuracy. In the Few-Shot setting, examples consist of straightforward question- answer pairs without additional reasoning steps, leading to the shortest reasoning lengths. No- tably, this omission does not hinder the perfor- mance of LLMs, which remains on par with
Table 7: Reasoning obstacle (in red) in the re- sponse and the correct choice (in green)
Question
Trinngl ABC is right. angled point What is the relationhip between line ABl and line BC?
Amswer chnices
A. Panillel R. Prrpeniicllrr C. Intersacing but not perpendiaatee
D. Cannse be infened
Modid Resporse
Step I: Tiiaggle ABC right angled it B, which meass that angle
ls a right: atgle.
Step 2: In right angled trinnate, the side opposite the right anple
the Zero-Shot performances. In contrast, the
is caled the hypotenmee, and the ocher roo sides are called the legs. Step Line AB is one of the legs of trinngle ABC, and line IBC the hyputemmnn
other three settings incorporate reasoning steps. The Few-Shot-CoT setting exhibits a reasoning length comparable to that of the Zero-Shot set- ting but demonstrates better performance. In contrast, the Zero-Shot-ee longest reasoning length, yields the poorest performance. These observations indicate that reasoning length does not directly correlate with accuracy; rather, the reasoning effectiveness has a more significant impact on performance than the mere reasoning length.
Step Thanfoee, the rehricnship betwern line AB and line RC that they are intersasing but not poppadicadar. The snower is C. Intenasting but not perpendicalts.
setting, despite having the
8

---

Published as a conference paper at ICLR 2025
Reasoning obstacles exist in the forward reasoning of geometric structures. We analyzed the reasoning processes in the Few-Shot-Col group's results and found that reasoning obstacles appeared in all cases where the model failed to solve the problems. In these cases, what seemed to be straightforward conclusion was unreachable for the model. As shown in Table 77, the model was unable to distinguish whether AB and BC were the legs the hypotenuse of right triangle, even though it previously stated that "the side opposite the right angle is called the hypotenuse. Thhi outcome further suggests that large models may lack a conceptual understanding of geometric structures. See more explanations and examples in Appendix
3.5 SUPERVISED FINE-TUNING ON GeomRel
LLMMA3-RD -iI---aaaa
Finstual ELiMA3 Io-laster
To further explore how L.MMs can possess stronger GRI abilities, we try to fine-tune LLMs with our GeomRel. We split it into training, validation, and test sets in a ratio of 6:2:2, and fine-tune the Llama3-8B-Instruuc model.
Anglobased (foai))
(schonzeal)
Figure shows the Zero-Shot results, we observe that the
Atgle besed (adrasced))
buse (bask))
fine-tuned Llama3-88 8B-Instruct does not show significant im-
provement over its original version and even performs worse. In the advanced GeomRel, the fine-tuned performance almost identical to that of the original model, indicating that
Stape-based (boss)) Shape-lased (advanced)
the fine-tuned model with explicit relationship data does not enhance their geometric reasoning capabilities. However, in the basic Geom the results exhibit considerable vari- ability compared to the original model, with improvements in shape data but substantial declines in angles and lines data. This could be attributed to the relatively small size of the basic datasets, causing the test results to exhibit some degree of randomness after splitting. Our preliminary analysis of this data suggests that LLMs do not gain an increased understanding of geometric structures from the provided data, and fine-tuning models on QA-formed data makes difficult to enhance their GRI abilities, We analyzed the reasons for the failure, and in Appendix E.5 results of other models for comparison.
Figure 5: Comparison of Models
we provided the fine-tuning
GEOMETRIC CH
(GEoCOT) METHOD
In Section 3.4 we have found that the impact of using different prompting techniques model performance was quite limited, and the conventional CoT technique failed be effective in identifying geometric relations.
Stage Geometry
Obnuse trianglo cau interier
Geometrr destription trivegle trianglo EPC, paint G the
ABl BC and wildd istemeet inaaa sosiannnai BC
nodd
midppinn
sudpiate juint in be modgaint BC, puice the III,
To explore ways to improve the GRI abilities of LLMs, we propose a novel prompt tech- nique for geometric problems based on CoT (GeoCoTT). The pipeline and example have been illustrated in Figure
Stage Relationship Obscrvation
Loua
Process want
All
model
ABl Print C' the madp iist of BC, All
the Enangie Alxk
Correc Comerreloonntrrorrrr
Ir is fow AR thnn Gs point sethon Als EKC but paen the midgeon of Bil not Conelusion Therdfrn gastzedictoos cn Als, the anreeer 21, No.
Bulling Bedkng
4.1 TWO-STAGE PIPELINE
We first instruct the LLMs to break down the geometric structures, decomposing the geometric information into points and lines. Then, extract the relevant parts from the decomposed information and guide reverse reasoning based on the given question. Pipeline details are as follows.
Figure 6: The pipeline and example of GeoCoT.
Stage 1: Geometry breakdown. In the first stage, we expand the geometric information provided by the textual description. For instance, given a rectangle ABCD, we derive and list secondary conditions such as: AB CD, BC AD, AB is perpendicular to BC, BC is perpendicular to CD, and expansion ensures that the model does not overlook crucial information, thereby reducing the risk of incorrect reasoning. Our previous studies indicate that large models often fail to accurately draw
on. This

---

Published as a conference paper at ICLR 2025
Line (Basic)
based (Advanced)
40-
Accupa (5g)
Acceray
70
60
Sxro-Shot
-Cal
-GroCol Pow-Shot
<ol
-ooCol
Zero-Shot -Dal
-GroCo Few Shor
-Col
-GroCaT
Angle-based (Bask)
Augla-based (Adrunes))
Apurcy (50
Accurayy (N))
40
60
Zero Shot
-Groool Seo-SSott
Col
GeoCol
Zero Shot
GGoCol Tow Sho
Co1
-Goooo
Shape-based (Back)
Shape-hasod (Adrmnca)
60-
Acumay (N)
Accuncy
Stao
.007
-GuoloT Few-Shot
-CuT
-GroCoT
Zcro-Stoi
-0o7
-CocCCo Fow-Shot
-CaT
-KoCaT
Figure 7: Model performances under our GeoCoT and other paradigms (In each subplot, the groups to the left of the dashed line use Zero-Shot, while the groups to the right use Few-Shot).
conclusions based on the provided geometric content. Therefore, by methodically listing potential secondary conditions, we enable the model to consider all relevant information during reasoning.
Stage 2: Relationship observation. In this stage, we re-input the result of stage into the LLMs along with the problem. First, relevant information extracted from the known conditions based on the elements mentioned in the problem. Then, reasoning is performed. To address the obstacles, we guide the model to adopt a reverse thinking (RT) approach. In this approach, the model assumes that certain geometric relationship holds and then works backward, arriving at either consistent or contradictory result, which ultimately informs the final decision.
4.2 RESULTS
We utilize in-context learning and create tailored exam- ples for each domain of GeomRel following the Geo- CoT pipeline. We also develop the Zero-Shot-GGeoCof prompt and conduct experiments using the GPT-335- Turbo. As shown in Figure 7 GeoCo'T exhibits strong performances in both Zero-Shot and Few-Shot settings, achieving an average improvement 9.15% on the ba- sic GeomRel and 14.79% % on the advanced GeomRel under the Few-Shor setting. This reflects significant en- hancements with our GeoCoT across various domains.
Table 8: Component ablation of GeoCOT.
Setting
Accuracy (%)
Basic
Advanced
Zero-Shot- Shot-GeeCoT
65.04
40.64
w/o Stage I w/o RT in Stage 2
62.50
37.93
64.37
38.81
72.55
51.98
w/o Stage 1 w/o RT in Stage 2
63.96
44.58
68.03
42.79
We further investigate the specific impact of each com- ponent in our two-stage pipeline by removing the backward reasoning guidance in the reasoning parts of Stage and Stage and then test their performance. The results are shown in Table g indicating that removing any component leads to a decline in performance. In the Zero-Shot setting, removing Stage even causes the performance to be nearly indistinguishable from the original no-prompt paradigm. Some mechanisms and examples of the approach are shown in Appendix F
5 CONCLUSION
In this paper, we introduced the GeomRel dataset to evaluate LLMs' geometric abilities by focusing on geometric relationship identification. Our findings highlight significant limitations in LLMS' understanding of complex geometric structures, particularly angle-based relationships. While data augmentation and prompting techniques offered limited improvements, our GeoCoT method signif- icantly enhanced performance, improving accuracy in identifying geometric relationships. These findings underline the importance of focusing on geometric structure comprehension and offer new insights for improving L.LMs' performance in this domain.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This paper is supported by the General Program of National Natural Science Foundation of China (62176153) and Ant Group.
REFERENCES
Chris Alvin, Sumit Gulwani, Rupak Majumdar, and Supratik Mukhopadhyay. Synthesis of solutions for shaded area geometry problems. The Thirtieth International Flairs Conference, 2017.
Anthropic. The claude model family: Opus, sonnet, haiku. https://wwwwcdn.cee 1eeba9eelcc00//cccf3333366888883
com/
2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chenggiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang. Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. CoRR, abs/2309. 16609, 2023. doi: 10.48550/ARXIV2330.. 16609. URL https:/tt:://oo. org/10. 48550..//85///ii 2309. 16609
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https //arxiv.org/ absa 2005. 14165
Jeffrey Buckley, Niall Seery, and Donal Canty. Investigating the use of spatial reasoning strategies in geometric problem solving. International Journal of Technology and Design Education, 29:
341-362, 03 2019. doi: 10. 10.1007/s10798-011 011-9446-3.
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 1511-1 2022.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. GeoQA: A geometric question answering benchmark towards multimodal numerical reason- ing. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 513-523, Online, August 2021a. Association for Computational Linguistics. doi: 10. 10.18633/v122021 wwt.22in.....ins-..i.m.amm indings-acl.46. URL landings-acl..en
httts:://aclanthol.l. www.:tttowoooooooooo/o/o2//
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXiv preprint arXiv:2212.02746. 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph. Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavloy, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbent-Voss Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained code, 2021b. URL htttpss//aarrv..rrraa////////
11

---

Published as a conference paper at ICLR 2025
Douglas H Clements and Michael T Battista. Geometry and spatial reasoning. Handbook of research on mathematics teaching and learning: A project of the National Council of Teachers of Mathematics, pp. 420-464, 1992.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichiro Nakano, Christopher Hesse, and John Schulman, Training verifiers to solve math word problems, 2021. URL httph::////:/xii.v.
abbs/2100 14168
Haskell Brooks Curry. Foundations mathematical logic. Courier Corporation, 1977.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021a. URL https://arxiv org/abs/209s 03300
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021b.
Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: systematic evaluation of large models for geometric reasoning. CoRR, abs/2312. 12241, 2023. doi: (0-4850/AXXI.2312.2222
https://::///o http///ii.org//////// 48550//rXiir 2312. 12241
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213, 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. httpst org/ als/ab///222 2205. 11916
Mary Montgomery Lindquist and Albert P Shulte. Learning and Teaching Geometry, K-12. 1987 Yearbook. ERIC, 1987.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Long-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference Natural Language Processing, ACL/ICNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 6774-6786. Association for Computational Linguistics, 2021. doi: 10. 18653/V
URL JRL/https:/// h/tp::://doi..o o 18. wwt..1885//v222211 http://2211cccl11111 acl-oog.. 528
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki. and Chris Callison- Burch. Faithful chain-of-thought reasoning, arXiv preprint arXiv:2301. 13379, 2023.
OpenAl. gpt-3.5-turbo-0122 CoRR, 2022. URL BoRL:nttps///geaaa.cc////gggg/g/
OpenAl. GPT-4 technical report. CoRR, abs/2333.087774, 2023. doi: 104455M/ARXIV.2203..07774
URL htttps://doi.org/11 www.:///......33......
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4. CoRR, abs/2304.03277,, 2023. doi: 1048550/ARXIV2200..033 URL https
//doi.org/1e 4118.4855//axiiv 2304.
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Lluis Mårquez, Chris Callison-Burch and Jian Su (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1666-1476. Lisbon, Portugal, September 2015. Association for Computational . org//15-1177
Linguistics. doi: 10. 1..18633v1/DD... 1711. URL httpss:/ttt/ssclattttt
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats. Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are mul- tilingual tffin-of-thought reasoners. arXiv preprint arXiv:2210.03055, 2022.
12

---

Published as a conference paper at ICLR 2025
Robert J Sterbberg and Talia Ben-Zeev. The nature of mathematical thinking. Routledge, 2012.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023a. doi: 10.48550/ARXIV233000 139771 org/10. 48550/arXiv 2382. 13971.
URI. https://doi
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodrigue7, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https: http:p::/aaxxv.orrr wtw.orggabbb//////// 09288
Trieu Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476 482, 2024.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset, 2024. URL https::s/////ti org/ abs/2402. 14804
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 111iv 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi. Quoe Le, and Denny Zhou. of prompting elicits reasoning large language models, 2023. URL htttps://aaxiv http://rrwii.oooo////////asass// /2tpaaab./aa...... 2201 11903
Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. brief overview of chatgpt: The history, status quo and potential future development. JEEE/CAA Journal of Automatica Sinica, 10(5): 122-1122-11122 136, 2023.
Zheng Yc. Chang-Ching Chou, and Xiao-Shan Gao. An introduction to java geometry expert (extended abstract). In Thomas Sturm and Christoph Zengler (eds.), Automated Deduction in Geometry- 7th International Workshop, ADG 2008, Shanghai, China, September 22-24, 2008. Revised Papers, volume 6301 of Lecture Notes in Computer Science, pp. 189-195. Springer, 2008. doi: 00.10077978-3--4222204----
Jiaxin Zhang and Yashar Moshfeghi. Gold: Geometry problem solver with natural language descrip- tion, 2024.
Jiaxin Zhang. Zhongzhi Li, Mingliang Zhang, Fei Yin, Chenglin Liu, and Yashar Moshfeghi. Geoeval: Benchmark for evaluating Imss and multi-modal models on geometry problem-solving. 2024a.
Ming-Liang Zhang, Fei Yin, and Cheng-Lin Liu. A multi-modal neural geometric solver with textual clauses parsed from diagram. In Proceedings of the Thirty-Secona International Joins Conference Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao. SAR, China, pp. 3374-333 1174--382.11 ijcai.org, 2023. doi: I022496/JCCA1222233366 URL https: //ooi. orgoorg////// 24963/ 2023/37
13

---

Published as a conference paper at ICLR 2025
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-modal Ilm truly the diagrams in visual math problems?, 2024b. URL httt::://aaxii.orr//aaaaaa
14624
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.00 230.:223,, 2022.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625. 2022.
A
RELATED WORK
A.1 EVALUATION OF LLMS ON GEOMETRY PROBLEM SOLVING
Geometric problems represent a significant and challenging aspect of mathematical problems. Several
benchmarks have been developed to assess large language models' abilities in geometric problem-
solving tasks, including GEOS (Seo et al. (2155), GeoShader (Alvin et al., (017). Geometry3K (Lu
cet al. 2021), GeoQA (Chen et al. 202 GeoQA+ (Cao & Xiao 2022), UniGeo (Chen et al. 2022 PGPS9K (Zhang et al. 2023). Geom Verse (Kazemi et al. 2023) and GeoEval(Zhang et al. 2024a).
However, their evaluation primarily focuses on assessing the solutions to problems, without truly addressing the model's 's ability to comprehend geometric structures. The MATHVERSE (Zhang et al., 2024b) has addressed some of these issues by paying attention to the reasoning process but remains insufficient in evaluating large models' deeper understanding of geometric structures, it lacks task decomposition. Our research begins with the fundamental elements of planar geometry and all pairwise relationships between them, constructing specialized dataset for geometric relation recognition to specifically investigate large language models' understanding of geometric structures.
A.2 CHAIN-OF-THOUGHT PROMPTING FOR LLMS
Recently, intriguing chain-of- of-thughht techniques have greatly improved both the reasoning perfor- mance and interpretability of LLMs by decomposing multi-step problems into intermediate steps (Wei)
et al. 2022 Kojima et al. 2022; Zhang et al. 2022; Wang et al. 2022,] Shi et al. 2022 Zhou et al. Lyu al. 2023). These strategies have been widely applied to various mathematical problems. However, as a distinct type of mathematical problem, effectively understanding geometric structures crucial. Yet, no prior work has specifically explored CoT prompting for geometric scenarios. To the best of our knowledge, we are the first to study tf-tuuuhttt prompting for specific geometric problems, guiding L.MSs to generate effective reasoning from geometric descriptions to infer geometric relations.
2022
B DISCUSSIONS AND LIMITATIONS OF THE BENCHMARK
B.1 THE INFLUENCE OF REFERENCE DIAGRAM IN GEOMETRIC REASONING
Researchers have conducted studies on the performance of LLMs in solving geometric problems that involve visual information (Zhang et al. 2024b Wang al. 2024), and finds that the diagrams show some differentiated enhancement effects. However, as shown in Figure 8 the presence of images makes difficult to distinguish whether the model's 's success is due observational skills or spatial imagination. Furthermore, we aim to evaluate the recognition ability of general language models, rather than focusing solely on multimodal models. the early stages of dataset construction, we created an experimental dataset based on the Geometry3k dataset to inquire about geometric relations, using the GPT-4V model to assess differences in performance across various forms of geometric descriptions. As shown in Figure 9 the group using original diagrams (with D (Raw) performed worse than the groups using other textual geometric descriptions.
Additionally, accurate diagram- based data only be obtained through collection, making it difficult to scale, whereas textual data can be more easily expanded. Thus, we ultimately opted for text-based geometric descriptions from rule-based construction perspective.
14

---

Published as a conference paper at ICLR 2025
Description: Line AB and line CD intersects at point E. Q Question: What's relation between ZAEC and <BED?
W With diagram:
Without diagram:
Let me image the diagram,
D
"Obviously, can see in the diagram that they vertical angles." "
C oh, get the result, they are vertical angles."
B
Figure 8: Diagrams vs. Text: Direct observation vs. imagined visualization.
Accrrayy
20
5-
(M1) (NL) O (ML) (F1)
(NJ (Row)
Figure 9: The accuracy of geometric relation recognition on the mini test dataset. Q represents the question, D represents the diagram, NL refers to natural language input, FL indicates formal language, and Raw represents the original image. The diagram data this section is derived from the Geometry3k dataset.
B.2 LIMITED SCOPE OF COLLECTED GEOMETRIC RELATIONS
Since our study is limited to planar geometry, while the definition geometric problems includes solid geometry, have not addressed this aspect due time constraints and the greater complexity of the elements and relations involved in solid geometry. Expanding the geometric construction methods would require significant additional effort. Therefore, the problems in our dataset are restricted to "within a plane." Future work could further explore LLMs' understanding of three-dimensional geometric structures.
C DETAILS OF THE DATASET
C.1 GEOMETRIC RELATIONSHIP CATEGORIES
Line-based Geometric Relationships. Line-based geometric relationships involve only the two most fundamental elements. When it comes to relationships involving points, they are often represented using points on line, such as in the problem of collinear points. Therefore, in our dataset, we have placed these relationships into subset. Given that these involve two basic elements, we have used enumeration to collect all geometric relationships in plane geometry that include only points and lines. This includes relationships between points, between points and lines, and between lines. Specifically, we have recorded 5 sets of relationships: three points being collinear, a point being (or not being) on line, parallelism, intersection, and perpendicularity.
15

---

Published as a conference paper at ICLR 2025
Angle-based Geometric Relationships. Angle-based geometric relationships require a stronger spatial understanding, as angles are formed by the intersection lines. We primarily study the relationships between angles, including the relationships between the four angles formed by the intersection of two lines and the more complex relationships between the eight angles formed when two lines intersect with third line. Additionally, our dataset includes specific angle relationships in particular contexts, such as adjacent angles in polygons. Overall, we have recorded 9 sets of relationships: Complementary, Supplementary. Corresponding, Alternate Interior, Consecutive Interior, Alternate Exterior, Consecutive Exterior and Adjacent/opposite angles in quadrilateral.
Shape-based Geometric Relationships. Shape- Shape-Based Based Geometric Relationships are based on figures and aim to explore the relationships between points and lines (or segments) and geometric figures. The relationships between figures and points primarily involve positional relationships, such as whether the point is inside, outside, or on the boundary of the figure. When related to lines, in addition to intersections, tangency, and disjointedness, this category also includes relationships involving special line segments within figures, such the medians of triangles or the radii of circles. This section includes a total of 12 geometric relationships.
C.2 BASIC DATA GENERATION
Table 9; From Definition to Properties Approach for Establishing Parallelism
Source
Description
Definitions
"In a plane, lines AB and CD never intersect,"
"In a rectangular coordinate system, the slopes of lines AB and CD are equal." "
Properties
"The direction vectors of lines AB and CD are proportional." Quadrillateral ABCD is parallelogram," "I "Lines AB and CD are both parallel to line EF."
As illustrated in Figure 9 we obtain various geometric descriptions that establish the relationship "AB is parallel to CD. Starting from the definition, we obtain two distinct definitions of parallel in different situation; from the properties, identify several fundamental cases in which parallelism can be established.
C.3 EXAMPLES CONDITION POOLS
In Table 10 we present two examples of the condition pool, each containing multiple geometric descriptions.
Table 10: Examples of condition pool
Input
Condition
Output
line segment (output) inside shape (imput)
Triangle ABC
If ABC is triangle, D is the midpoint of side AC
BD
Triangle ABC
If ABC is a triangle, M is the midpoint of side AB
CM
Cirele
center and radius r, If ABCD is square, is the midpoint side BC
If circle
any point on circumference circle
A0
Square ABCD
AE
Rectangle PQRS
If PORS is a rectangle, M the midpoint of side PQ
MS
Trapezoid ABCD
If ABCD is a trapezoid with AB parallel to DIC, and is any point on side AB
CM
line (imput) intersects another line (outpid
AB
Line AB intersects line CD al point E Quadrilateral ABCD with diagonals intersecting at point
CD
AC
AB
Angle ABC acute
BC
AB
Points A, B. and C form an obtuse triangle
BC
16

---

Published as a conference paper at ICLR 2025
C.4 CONCATENATION OPERATION
As shown in Figure 11 based on the theorem that if one line intersects another line, then its parallel line also intersects that line, have integrated conditions to ultimately arrive relatively more complex geometric structure that satisfies the condition of "two intersecting lines".
Table 11: Example of geometric concatenation operation
Data 1 (from pool: two parallel lines):
Input: AB
Condition: Quadrilateral ABCD is parallelogram
Output: CD
Data 2 (from pool. Two infersecting lines).
Input: AB
Condition: In triangle ABC, point D is the midpoint of BC
Output: BC
Concatenated Data (two intersecting lines):
Input: CD
Condition: Quadrilateral ABCD is parallelogram, and in triangle ABE, point F is the midpoint of BE
Output: BE
C.5 DIVERSIFICATION OPERATION OF THE DATASET
Table 12: Diversification of dataset examples
Data Type
Question
Original Data
circle with center intersects AC at points and F, triangle AGE, with GD the altitude from vertex G to side AE, Are lines AC and AD the same line? Answer choices: A. Yes B. No C. Cannot be inferred
UI
at points and F, triangle AGE, with GD as the altitude from vertex G to side AE, and triangle AFG is equilateral, AF= 7. Are lines AC and AD the same line? Answer choices: A. Yes B. No C. Cannot be inferred circle with center intersects KN at points Z and T, triangle KLZ. with LP as the altitude from vertex L to side KZ. Are lines KN and KP the same line? Answer choices:
circle with center intersects
RP
A. Yes B. No C. Cannot be inferred
UI+RP
circle with center intersects KN at points Z and T, triangle KLZ, with LP as the
altitude from vertex L to side KZ, and triangle KTL is equilateral, KT = 7. Are lines
KN and KP the same line? Answer choices: A. Yes B. No C. Cannot be inferred
we increased the complexity of the geometric descriptions and enhanced the diversity of the dataset through Adding unrelated information (UI) and Re-labeling Points (RP).
As shown in Figure 12
D CORRELATION WITH COMMON DATASETS
In Figure 10
compare the performance of various models dataset (including both the basic and advanced subsets) with several widely-used benchmark datasets. We observe a high degree of correlation in performance, particularly with math-related benchmarks, further validating the effectiveness of our dataset.
E EVALUATION DETAILS
E. MODEL HYPERPARAMETERS
Table 13 presents the complete list of hyperparameters applied to the models (including the hyperpa- rameters of the fine-tuning operations) throughout the evaluation phase.
17

---

Published as a conference paper at ICLR 2025
1.00
GeomRd Basic-
0.92
0.88
0.9
0.48
0.73
0.75
Goomiee Advanced.
0.92
0.92
0.71
0.46
0.61
0..50
0.25
Math (MATE)
0.88
0.92
0.83
0.72
0.78
-0,00
GSMHK
0.9
0.71
0.83
0.72
0.91
Cading
0.48
0.72
0.72
0.94
0.50
-0.75
0.61
0.78
Goomie Heals
Goo Advanced
Madh (MATH)
GSMBK
Codtng (HemaErra)
MCC OMIILL
Figure 10: Correlation heatmap of model performances. The benchmarks we compared are MATH Hendrycks al. (0221b) GSM8K (Cobbe et al. (202I), HumanEval (Hendryeks et al.) (021a) and MMLÚ (Chen et al. 2021b). The models used and experiment settings are consistent with those described in Section
3.1
Table 13: Hyperparameters of Each Model
Model Name
Parameters
Comments
GPT-oo
Iemperature" 0.
max Jokens 1024
wession 30V-1o-203--5-11
GPT-4-umbo
"emperature" 0. mmxx tokens": 1024
version "GPT-4-turbo"
GPT:3 S-cumbo
"teppraturre: ": 0, "max tokens": 1024
versine "2pe-
2ge-3.5-rrfoo-0235
Qwenl 5.1
'eepperttree: 0, "max tokens": 1024
"quenl.5-110b-chal
OwenMax
'Iempeature" 0, "nax, tokens": 1074
version "qwen-mmax"
Clande-5-4 -yQus
"temperauure: 0, "max, tokens": 1024
versioe 1llnde---oqus-2244422
LLaMA2- 1ha-Cht
'temperature" 0, "max tokenst 1024
model 'Llama 2-13--2133bb 2-13b-char
LLaMA3-70B MMMMA--70BIInnttte Instrrcct
'temperature' 0. "max_tokens" 1024
model "Llama 3- 3-70B-Instrect
Insttrrrr
"temperatre":
tokens":
"L-amm-3-288-
Qwen2-7B- Insemst
'tempenatrre. 0, "max, tokens": 1024
model 'Qwen2. 2.B Instruct'
LLaMA3-SB Base-FT
"'emperature,; 0, "max tokens": 1024, train batch size: 4, ""inctuning. type"; lora,
model "Llamm- 3-38"
"leaming 1.0e-4,
"temperature" "max, tokens": epachs": "M11": tree
LLaMAG-SB.Iestmcc-
"'finetaning type": 'learningraate" 1.0e44, "num, "mum_train_ "rai__trii__spcchs" 10.0. 'bf16" tree
Qwen2-7B- InssmecttFF
"temperatire" " 0, "max tokens": 1024, "tmiinmnbtttt batch, size lize:: "iinetaning rype": lom,
model "Qwen2.7B-Insteee
"leamning rute: 1.0e-4, "num, araiineppccs""
"bfi6": tree
Qwen2-7B
baaiii_bacc__ieeeeeeee liinttuiing type": "leomnigg rale": 1.0e-1, "nmm, "num_train, traint_cpochhs" 10.0, "bf16: trae
'temperature's "max__okeer""
'Qwen2-7B*
E.2 ZERO-SHOT RESULT DIFFERENT MODELS
Tables 17 21 present examples from different subsets under the Zero-Shot settings LLMs, we can observe the performance differences of models across different subsets. In Table we provide multiple examples from the advanced shape-based subset and summarize the responses three models, where we can observe that GPT-4o demonstrates significantly superior performance.
We conducted additional experiments on the GPT-3.5-turbo model using multiple sampling runs (temperature set 1). Its performance evaluated using pass @k and self-consistency with majority voting over generations (denoted as acc(SC=k)). The results are summarized in Table [4
We observed that pass@ @ increases rapidly with larger k. indicating a growing likelihood of generating the correct answer through multiple attempts. However, the majority voting accuracy (acc(SC=k)) does not improve significantly, suggesting that the model's probability of generating correct answers remains low. This indicates that its reasoning on geometric tasks is inconsistent, leading to dispersed outputs.
18

---

Published as a conference paper at ICLR 2025
Table 14: Results( (%) GPT-3.5-turbo using multiple sampling runs
Line-bassed
II. Angle-based
III. Shape-based
(pass acc(SC=k))
Basic
Advanced
Basic
Advanced
Basic
Advanced
64.29 14.29164.29
42.46 142.66
42,11
22.41 122.41
05.75168..7
38.31 38.31138.31
71.43 60.71
62.30 145.24
15.16
37.95 122.41
93.75 75.00
59.74 38.96
75.00 66.29
72.22 144.44
68.42 142.11
55.17 125.86
100.00 75.00
68.83 143.51
10
75.00 64.29
83.33
145.24
84,21 157.89
70.69 132.76
100.00
75,00
77.92
142.21
E.3
OBJECTIVELY UNIDENTIFIABLE OR SUBJECTIVELY UNIDENTIFIABLE
Table 23 presents the QwenMax's ' Zero-Shot responses, with the four examples corresponding to: unidentifiable relation with a definite model conclusion, unidentifiable relation with no definite model conclusion, identifiable relation with no definite model conclusion, and identifiable relation with definite model conclusion. We observe that the model may incorrectly judge a geometric structure as incomplete, may make arbitrary conclusion about geometric structure where information is indeed insufficient.
E.4 DIFFERENT PROMPTS STRATEGY'S RESULTS ON
In Table 25 we can observe the differences in response lengths across various prompting strategies. In this example, the Zero-Shot group is the only one that produced a correct answer, while all other strategies resulted incorrect answers. This indicates that prompting strategies can sometimes lead to poorer outcomes.
E.5 ADDITIONAL FINE-TUNING RESULTS
In response to further experiments and suggestions, we present updated fine-tuning results for various models on the GeomRel dataset. The results are shown in Table [5
Table 15: Fine-tuming Results for Different Models on GeomRel
Line-based
II Angle-based
III. Shape-based
Model
Basic
Advanced
Basic
Advanced
Basic
Advanced
LLaMA3-8B-
63.16
42.11
52.17
13.04
26.92
38.46
34.14
40.56
32.50
13.75
39.87
40.51
LLaMA3-8B- FT
86.86
71.67
63.16
54.90
93.75
61.69
Qwen2-7B-Instruct
65.55
31.84
53.75
20.71
55.38
34.69
82.14
70.87
78.95
54.44
94.48
68.83
Bwen2-7B-Base-A
89.29
70.48
73.68
51.51
95.70
70.13
GPT-4o
77.87
52.91
66.67
29.00
87.04
53.38
Performance of Fine-tuning: Fine-tuning on LLaMA3-8B-Bas significantly improves performance across all subsets, outperforming GPT-4o in most categories. However, fine- tuning on LLaMA3-8B-Instrue results in lower performance, likely due to the model's tendency to select uncertain responses, such as "cannot be inferred," " when faced with difficult cases.
Comparison of Qwen2-7B Models: Both Owen2-7B-Bass and Qwen2-7B-Instruun demonstrate strong performance after fine-tuning, with the Base model showing slightly better results in most cases. This contrasts with LLaMA models, where the fine-tuned Base model consistently outperforms the fine-tuned Instruct model. The difference may be due to methodologies and content.
variations in instruction-tun
Importance of Model Selection: The results highlight the importance of carefully selecting models for fine-tuning GeomRel. While LLaMA models show substantial improvements after fine-tuning, Qwen models exhibit comparable or superior performance in several categories.
19

---

Published as a conference paper at ICLR 2025
These findings emphasize the importance of both model selection and fine-tuning strategies in achieving optimal performance on the GeomRel dataset.
We conducted further analysis the fine-tuning results of the LLaMA3-8B- model and observed that its marginal gains were due to the model frequently outputting the uncertain answer, "cannot be inferred." Additionally, fine-tuned the LLLMA3-8B-BBR model, which showed substantial improvement, achieving 81.26% and 62.75% accuracy on the Basic and Advanced subsets, respectively, surpassing GPT-40.
To better understand this phenomenon, we analyzed the proportion of "cannot be inferred" responses generated by each model, shown in Table 16
Table 16: Proportion of "cannot be inferred" responses across models.
Model
Basic
Advanced
LLaMA3-88 IB-Inntruct
14.71%
8.21%
Fine-tuned LLLaMA3-B--Isstt
71.53%
67.40%
Fine-tuned L.LMA3-8B--BB
25.03%
21.41
Truth
27.94%
26.49%
The results reveal that the fine-tuned LLaMA3-8B- model heavily leaned toward uncertain answers, with a significantly higher proportion of "cannot be inferred" responses compared to the actual data distribution. In contrast, the fine-tuned LLLMMA-8B--BB model closely aligned with the actual proportions in the dataset.
This discrepancy highlights two critical factors:
The impact of fine-tuning data: The LLaMA3-8B -nstrruu---ttttuctttt model's performance suggests that its initial instruction instrcction-tuiin introduced a bias toward cautious or uncertain answers.
The alignment of model outputs with data distributions: Fine-tuning the LLaMA3 3-B-- Base model led to outputs more consistent with the actual dataset proportions, contributing to its superior performance.
These findings underscore the importance of selecting appropriate base models for fine-tuning, especially for tasks where output distribution plays crucial role in overall accuracy.
F REASONING OBSTACLES AND THE EFFECTIVENESS OF OUR GEOCOT APPROACH
To better understand the forward reasoning obstacles mentioned in Section 3.4 and the effectiveness of our GeoCoT method proposed in Section 6, we attempt to analyze from the perspective of reasoning pathways.
process can pass
process cannot pass
Geometry
Relation
Figure 11:
Forward reasoning process of LLMSs
As shown in Figure I. during the model's 's forward reasoning, when both the problem and the geometric description provided, the model tends generate reasoning process more directly
20

---

Published as a conference paper at ICLR 2025
thereby overlooking some implicit inferences (0 3, O + 4). This leads to the failure in deriving conclusion 7, which in turn prevents the final answer from being reached. The model, relying solely on 5 and 6, arrives at final result that ultimately leads to an incorrect answer. Specific examples of errors made by GPT--355-trbo in the Few-shot-ColT setting in Table 27
related to the problem 2,
process stage poss
process stage pass
Geometry
Relation
Figure 12: Reasoning process with GeoCoT
However, as shown in Figure 12 after applying GeoCoT, in stage 1, we ignore the specific problem and focus solely on the geometric description extract as much explicit and implicit information as possible (0 - 1, 0 + 2, 0 - 0 + 4, 1 - 5, - 6). In Stage 2. we adopt reverse thinking approach by assuming the result and reasoning backward from (8 5, 8 7, - 6, - 3, + 4). We then examine 3 and 4 for consistency or contradiction, allowing us to make judgment on our initial assumption and derive the conclusion (3, + 8). This method effectively overcomes the obstacles encountered in forward reasoning. A comparative example is shown in Figure 7. Specific examples of GPT-3.5-turbo's performance under CoT and GeoCoT are shown in Table 28 observe the differences in their reasoning and the correctness of the answers.
and Table 29
To further support our explanations, we introduced a new metric, Necessary Condition Coverage (NCC), which measures the proportion of necessary intermediate conditions covered during the reasoning process that lead to the correct answer. We evaluated this on 24 randomly selected questions using GPT-3.5-turbo with Few-Shot CoT and Few-Shot GeoCoT methods. The results shown in Table 26
The results demonstrate that our two-stage GeoCoT method improves NCC, particularly for advanced problems. This highlights the effectiveness of decomposing geometry problems and incorporating reverse reasoning to enhance coverage and reasoning quality.
21

---

Published as a conference paper at ICLR 2025
Table 17: Zero-shot results of various LLMs on the basic line-based subset
Question:
Quadrilateral ABCD is rectangle. What is the relationship hetween line AB and line BC?
Answer choices:
A. Parallel
B. Perpendicular
C. Intersecting but not perpendicular
D. Cannot be inferred
Gold Answer:
B. Perpendicular
HPTT3,,5-Trbbu
The relationship between line AB and line BC is A. Parallel.
My choice is A. Parallel.
x
GPT-40:
In a reclangle, all interior angles are right angles (90 degrees). This means that each pair of udjacent sides meets at right angle.
Given that quadrilateral ARCD rectangle:
Line AB und line BC adjacent sides of the rectangle.
Since the interior angles of rectangle are 90 degrees, line AB and line BC must meet at right angle.
Therefore, the relationship between line AB and line BC is that they are perpendicular to each other.
The correct answer is:
B. Perpendicular
V
LLaMA-3-70B-Inss -otttmstutttt
Since ABCD rectangle, we know that it has four right angles (90-degroe angles). Let's analyze relationship between line AB and line BC:
AB is one side of the rectangle, and BC is an adjacent side. In rectangle, adjacent sides are perpendicular to cach other, which means they form right angle (90-degrce angle).
Therefore, the correct answer is:
B. Perpendicular
V
Question:
Line AB intersects circle O at two points, point C and point D. What is the relationship between line AB and line OD?
Answer choices:
A. Parallel
B. Perpendicular
C. Intersecting but not perpendicular
D. Cannot be inferred
Gold Answer:
C. Intersecting but perpendicular
G.T-3,5-Turbo:
Line AB and linc OD are intersecting but not perpendicular.
Choice C. Intersecting but not perpendicular.
GPT-40:
To determine the relutionship between line AB and line OD, let's analyze the given information:
1. Line AB intersects cirele O at two points, C and D. This means that line AB is a secant line to the circle. Point is the center of the cirele. Given that line AB intersects circle points and on the circumference.
infer that line OD radius of circle, where is a point
For line AB to intersect the circle at two points, it must pass through the circle, making it a socant line. The radius OD is perpendicular to the tangent at point D. However, since AB is a secant line and not necessarily a tangent, we need to consider the general case.
In the general case, the relationship between secant line (AB) and radius (OD) that intersects the secant line one of the points of intersection (D) is that they are perpendicular. Therefore, the correct answer is: B. Perpendicular X
MAMMA---70B-Inttruee
When a line intersects a circle at two points, it is called secunt line. In this case, line AB is a secant line that intersects circle O at points and D. Now, let's analyze the relationship between line AB and line OD. Since point on circle the line joining the of the circle (O) to point D (line OD) is a radius of the circle.
The key property here is that a radius is always perpendicular to tangent line at the point of tangency. Since line AB secant line, it can thought as limiting case a tangent line, where the tangent line intersects the cirele at two points instcad of
Therefore, infer that line OD (the radius) is perpendicular to line AB (the secant line) point D.
My choice is: B. Perpendicular
22

---

Published as a conference paper at ICLR 2025
Table 18: Zero-shot results of various LLMs on the basic angle-based subset
Question:
In plane. there quadrilateral ABCD, and point is point in the plane such that AE is perpendicular BC and DE is perpendicular AD. What the relationship between LBED and LAED?
Answer choices:
Vertical angles
B. Supplementary angles
C. None of the above
D. Cannot be inferred
Gold Answer:
Supplementary angles
GPI-3.5-Turbo:
Since is perpendicular
DE perpendicular to AD, can see that LAED and LBED are both right angles. Therefore, the relationship between LBED und LAED is they are vertical angles.
Choice Vertical angles
X
GPT-4o:
determine the relationship hetween BBE/ and AAED, let's analyze the given conditions:
1. is perpendicular BC.
2. is perpendicular
Given these conditions, we can infer that:
-AEB 90"
LAED 90
Since angles are right angles, they are not vertical angles not supplementary (angles that add up to 180 degrees).
Therefore, the correct answer is:
None of above X
LLaMA-3-70R- In0ttnnstutt
Let's amalyze the situation:
AE is perpendicular to BC, and DE perpendicular to AD.
BBD and /ADD are formed by the transversals AE and DE, respectively.
Since AF, and are perpendicular their respective lines, /BED and /AED are alternate interior angles. Therefore, LBED and LAED are equal.
My choice
Vertical angles
Question:
Quadrilateral DEFG is a rectangle with DE=2 and FG=4, What is the relationship between LFGD and LEFG?
Answer choices:
A. Diagonal angles
B. Adjacent angles
C. None of above
D. Cannot be inferred
Gold Answer:
B. Adjacent angles
GPT-335-TTurbe
The relationship between /FGD and EEGG is that they are adjacent angles.
So the correct answer is:
B. Adjacent angles
GPT-4o:
To determine the relationship between LFGD and LEFG in the rectangle DEFG, let's analyze the properties: LFGD is angle vertex G. formed by FG and GD.
LEFG is an angle vertex F, formed by EF and FG.
Since they share the side FG and are adjacent, the conrett relationship is that they are adjacent angles.
Answer choice: B. Adjacent angles
TaMM-3--0B-Instruuce
In rectangle, all angles are right angles. Since DEFG is rectangle:
LFGD 90
-EEFG = 90"
They are not diagonal angles but are adjucent because they share the side FG and vertex F.
So, the correct unswer is:
B. Adjacent ungles
23

---

Published as a conference paper at ICLR 2025
Table 19: Zero-shot results of various LLMs on the advanced angle-based subset
Question:
Lise ABl intereets line CD point E, line GH interseets line CD at point F. GH is paraliel to AB, What the relazionstipp between 2GPC and ABBC?
Answer choices:
Supplemetary angles
Vertical ingles
None
Cannot inforee
Gold Answer:
B Alterntic esterior angles
CPT:35Tuurb::
The relationsipp between 2GFC and ZBEC is they altemate exterioe angles. When purallel lines intersected by transversal, altemate exterior is:
cougraent. Therefore,
exterior angles
GPT--
To desermine the selationship between ZGPC und ZBBCC let's antlyze the given information step by step
1. **t Lines and Intersctionn:+
Line AB insersects line CD point B.
Line GH interizets line CD point
Line paralie! line
**Parallee Linex Tranevenale:**
Gil paralie) jelatonships.
and CD acts as transvesaal intersocting and properties of parallel lines transvenals angle
3. **Angle Relationship:*
4BEC formed the intersection of lines AB and CD.
/CPCC is formed at the intersaction of lines GHH and CID.
4. ~>Using Parallel Line Properties:**
sides of the transverial CD and outside the parallel lines GH and AB. Therefore, they are altemate extetior angles. Glvven this molyyss, the relationship berween / /GPC and /RFC that they altemate exterior angles.
ABEC and ZGPC are
The
angles
LIaMA-3-000-nnii
A nice geometry problem!
Lets analyse the sination:
Line AB intersects line CD point is so ABEC an intesior angle. -I Line GH parallel AR, so (HH AB.
Line GII intersocis line CD
Eo 2GFC also inserior
GHt
conclide
corresanniing angles by lises equal. Spocifically. ZGRU and ZBBC
cornogonding angles.
Comesponding angles are equal, we
/BEC.
Therefure. the relationsip between ZGFC and ZBEC that they are equal.
My chooce is None of the above Isince "equal" is not among the options).
Question:
Quadtilatral TXOF rectangle, TO one diagonals. What the rehtiionhip between ZTOF and ZTOX?
cholces:
angles
Complementary tuples
Supplemmtary angles
Nore of the above
E. Cannot inforred
Gold Answer:
B. Complementary angles
G.T-3:5Turro::
and -TOX they compleeentary angles. This restangle diegonals hiscet each </OFF 2TUX adjacent ungles that right angle. Therefore,, the messures 90 degrees, making them complementary angles.
relationshipp
Choice Complementary angles
GPT:4:
To desermine the relationsiuup between ZTUE and ZTOX rectungle TXOE; let's analyze the properties of rectangle and its diagonals. 1. + Properiis of Rectangk::**
A rectingle has right angles (each degrees).
Opposite
equal and panilll.
disgonals metiingle equal length bisect eoch
"Diagonals Recuigge:***
diogonals intersect, they form two congrmmnt triangles.
Each diuponal divides the reciangle into two right triungles.
3. A'Aglles Formed by the Diagonal:
TOF and / TOX are angles formal by the disgonal TO with the sides of the metangle.
Since TO diagonal, it bisects the rectungle into two congruent right triangles, and euch angle at the interncction of the diagonal with the sides of the rectangle is pari of these right trinngls..
4. "*Kelttooship Between TUF ZTUX: inples
rectingle. the
360 deprees.
diagonal, spliss restungle two right trangles, angles aruund poin: O) must sum degrees because they are supplementary
angles.
Thareforo, /TOF and /TOX are sopplemetary sngles because they add up to 180 degroes.
Answer chosce: C. Supplementary angles"*
x
LLaMA-3-70l-Instruce
Since TXOF a rectnngle we know that has sets of opposite right angles, Let's focus on the right anglos verticns and 2TOP und 2TOX adccent ungles that form line O. We know that the measures adjacent ungles that form straight line 180". they supplementary angles.
Since /TOP /TON adjueent zngles form strightt Supplmeentrry ingles.
180".
24

---

Published as a conference paper at ICLR 2025
Table 20: Zhot Results various LLMs on the basic shape-based subset
Question:
If PIC is triangle, point U is the extension line of side IC, CU IC. Is line segment PU inside triangle PIC?
Answer choices:
A. Yes
B. No
C. Cannot be inferred
Gold Answer:
B. No
GPT-3,5-Turbe
Since point U is the extension line of side IC and CU 1 * IC. this that point U located outside of triangle PIC. Therefore, line segment PU is also outside of tringle PIC. So, the answer B. No.
GPT-4o:
To determine whether line segment PU is inside triangle PIC, let's analyze the given information step by step.
1. **Triangle PIC*# We have triangle with vertices P. I, and C.
2. **Point U the extension of side IC Point U located
the line extending from side such that CU 1 IC. This means U outside triangle PIC because lies on the extension side heyond point
3. **Position PU relative triangle PIC**; Since is outside the triangle, need consider the position of
line segment PU.
U is the extension IC beyond then U not between I and C but rather beyond C. Therefore, line segment PU will extend from point P point U that outside the triangle.
Given that U outside the triangle, line segment PU cannot be entirely contained within triangle PIC. It must pass outside the triangle reach U.
Thus, the correct answer B. No
V
LLaMAA-3-70B-nnstructt
A nice geometry problem!
Let's break down step by step:
1. We know that point U is the extension line of side IC, which means that IU straight line.
2. We also know that = 1
which implies that point C the midpoint of IU.
3. Now, consider the triangle PIC. Since C the midpoint of
draw perpendicular hiscctor from to
IC, which intersects IC its midpoint, say,
4. Since CM perpendicular IC,
conclude that CM is also perpendicular PU (as PU is purt of IU). 5. Now, observe that triangle PMC right triangle with right angle M. We know that PM IC (since P is
inside the triangle PIC), and CM 1/2 (since is the midpoint of IU).
6. Using the Pythagorean theorem in triangle PMC, get: PMF CM²= PC?
7. Since CM + IC. we can substitute this value into the equation: PMP IC= PC..
8. Simplifying the equation, we get: PMP + PC?
PC?.
9. Since PM IC, we can conclude that PMF Adding 1 IC² to hoth sides, get: PMF I0++ IC
10. Simplifying the inequality, get: PC IC?.
11. Since PC side of triangle PIC, know that PC IC (otherwise, PC would be longer than the side IC, which
is possihle triangle).
12. Comhining results from steps 10 and 11, we can conclude that PC7 * IC IC?, which implies that IC.
Now, here's the crucial part: since PC IC, can conclude that point P lies inside the triangle PIC.
Therefore, my answer is:
A. Yes
X
25

---

Published as a conference paper at ICLR 2025
Table 21: Zero-shot Results of various LLMs on the advanced slape-basek subset
Question:
If HNFO is rectangle, point is on the extension line of side HO. HO, triangle OMB is right-angled at with being the altitude drawn from vertex to OB, and point is foot altitude. What is relationship between point and rectangle HNFO?
Answer choices:
A. Inside B. Outside C. Cannot be inferred
Gold Answer:
B. Outside
GPT-3.5-Ture
Point is inside rectangle HNFO.
Choice: A. Inside
x
GPT-4o:
To determine the relationship between point and rectangle HNFO. let's analyze the given information step by step.
1. **HNFO is rectangle**; This means that all angles in the rectangle are right angles, and opposite sides are equal in length.
2. **Point B is on the extension line of side HO. and OB HO This implies that point B collinear with points
H and O, and B is located such that the distance OB is equal to the length of side HO of the rectangle.
3. **Triangle OMB is right-angled at 0+* This means that angle OMB is a right angle (90 degrees).
4. **ME is the altitude drawn from vertex M to OB, and point E the foot of this altitude**; This means that ME
is perpendicular to OB, and E is the point where this perpendicular line intersects OB.
Since OB is an extension of side HO and OB HO, point B lies outside the rectangle HNFO. Since triangle OMB is right-ungled at O. point M must be somewhere such that OM is perpendicular to OB. The altitude ME from M to OB will intersect OB at point E.
Since OB is an extension of side HO and point B is outside the rectangle, the altitude ME from M to OB will also be outside the rectangle. Therefore, point E. being the foot of this altitude, must also lie outside the rectangle HNFO. Thus, the relationship between point E and rectangle HNFO is: Answer choice: B. Outside
LLaMA-3-70B-Instee
A nice geometry problem!
Let's break it down step by step:
1. HNFO is a rectangle, so all its angles are right angles.
2. Point B is on the extension line of side HO, and OB = HO, which means OB is continuation of HO.
3. Triangle OMB is right-angled at O, which means ZMOB = 90".
4. ME the altitude drawn from vertex M to OB, which means ME is perpendicular to OB.
5. Point E. is the foot of this altitude, which means E. lies on OB.
Now, let's analyze the situation:
Since ME is perpendicular to OB, and OB is continuation of HO, ME also perpendicular to HO. This means ME is parallel to FN, since HO and FN are opposite sides of the rectangle HNFO.
As ME is parallel to FN, and E lies on OB, we can conclude that E lies on the extension of FN, which means E lies on the side NF of the rectangle HNFO. Therefore, the relationship between point and rectangle HNFO is that point E lies **nn*** the rectangle.
My answer choice is: A. Inside X
26

---

Published as a conference paper at ICLR 2025
Table 22:
Zero-shot results of various LLMs on the advanced shape-based subset with simplified answers (The correct options are highlighted in red).
Question
GPT-3.5-Turbo
GPT-40
LLaMA-3-70b
II HNFO rectangle, point B the extension line side HO, triangle OMB right-angled with ME being the altitude drawn vertex to OB, and point is foot
Point E inside rect-
Point E. being the
lies the side
angle HNFO.
foot of the altitude,
NF the rectangle
Choice: Inside
must also lie outside the rectangle HNFO. Answer choice: Outside
HNFO.
Answer: A. Inside
this altitude. What is the relationship
between point and rectangle HNFO?
Answer choices: Inside Outside Cannot inferred
GFMZBO is hexagon, point E on the extension line of side OG, GE 1/2 OG, in the acute-angled triangle GDE, DX is the altitude from vertex D to side GE, with X as the foot of the perpendicular. Is point X inside hexagon GFMZBO?
Point is located outside the hexagon GFMZBO. Answer: B. No
Point X must also be outside the hexagon. Answer choice: B. No
Point must lie in- side the hexagon.
Answer: A. Yes
x
Answer choices: A. Yes B. No Cannot he inferred
If WIBUY pentagon, N is the midpoint of side WL, in regular pentagon NAYLD, point D the intersection of diagonals NY and AL, and angle NDL degrees. point D) inside pentagon WIBUY?
The information does give clear indi- cation.
Point D. being part of the larger regular pentagon NAYLD, is likely outside. Answer choice: B.
Point D lies inside the pentagon.
Answer: A. Yes
Answer: Cannot be inferred
No
Answer choices: Yes No
Cannol be inferred
Point A is point on circle (with center O), the line OA intersects circle O at another point B, and line segment AB the diameter of circle O. Additionally, point is point on circle O such that AC is perpendicular to line segment AB. Is OA a diameter of circle O?
OA is also diameter of circle O. Answer: A. Yes
OA is not a diameter itself but radius.
OA a diameter of
circle O, as it passes through the center.
Answer choice: B. No
Answer: A. Yes
Answer choices: A. Yes B. No C. Cannot be inferred
In quadrilateral ABCD, E is the midpoint of side BC, and angle A is
Yes, CE one side of quadrilateral ABCD, Choice: A. Yes
CE is not of the four sides of quadri- lateral ABCD. Choice: B. No
CB is not side of quadrilateral ABCD, Choice: B. No
congruent to angle C. Side AD is parallel to side BC. Is CE one side of
quadrilateral ABCD?
Answer choices: Yes C. Cannot be inferred
triangle ABC, the AD bisects angle BAC, point D side BC, and segment AD perpendicular to side BC, segment BD equal to units. CD the angle bisector of triangle ABC? Answer choices: A. Yes No C. Cannot be inferred
is equal CD, making the an- gle bisector trian- gle ABC. Choice: A. Yes
not the angle bisector triangle
is the angle bisce- tor triangle ABC. Choice: A.
ABC. Choice: B. No
27

---

Published as a conference paper at ICLR 2025
Table 23:
Four cases of Zero-Shot inference results of QwenMax. The correct answers from the options are highlighted in green.
Question:
The non-coononeeooe coincident lincs AB and CD in the coordinate system have equal slopes. What is the relationship between line AB and line EF?
Answer choices:
A. Parallel B. Perpendicular C. Intersecting but not perpendicular D. Cannot be inferred
Response:
If lines AB and CD have equal slopes, this means they both have the same angle of inclination with respect to the x-axis. In a coordinate system, lines with the same slope are parallel to each other because they maintain the same rate of change in y for change in x. Given that lines AB and CD are non-coincident and have equal slopes, they are parallel. The question asks about the relationship between line AB and line EF, However, no information has been provided about the slope or position of line EF in relation to AB or CD. Without any information on line EF's slope or its relation lines AB and CD, cannot infer whether line EF is parallel, perpendicular, intersecting (but not perpendicular) to line AB, Therefore.the answer D. Cannot be inferred. Unrecognizable relation with a definite model conclusion X
Question:
pentagon ABCDE, AB 5. What the relationship between line AB and line EF?
Answer choices:
A. Parallel B. Perpendicular C. Intersecting but not perpendicular D. Cannot be inferred
Response:
Given the information provided. we only know the length of line AB in pentagon ABCDE, which is 5 units. Without any additional details about the angles, positions, or lengths of other sides, including line EF, we cannot conclusively determine the relationship between line AB and line EF. Therefore, the answer D. Cannot be inferred.
Unrecognizable relation with no definite model conclusion
Question:
Quadrilateral ABEF is trapezoid. What is the relationship between line AB and line EF?
Answer choices:
A. Parallel B. Perpendicular C. Intersecting but not perpendicular D. Cannot be inferred
Response: In a trapezoid. by definition, one pair of opposite sides is parallel. Therefore, in quadrilateral ABEF, which is given to be trapezoid, either AB and EF are parallel, or AE and BF are parallel. Since the question specifically mentions line AB and line EF, the correct answer that they are parallel. The answer is A. Parallel.
Recognizable relation with no definite model conclusion X
Question:
CEXIZU is a hexagon, I is the midpoint of side CU, triangle CPI right-angled at C, with PW being the altitude drawn from vertex P to CI. and point W is the foot of this altitude. What is the relationship between point W and hexagon CEXJZU?
Answer choices:
A. Inside B. Outside C. Cannot be inferred
Response: Given the description, we can infer the following geometrie relationships: 1. CEXIZU is hexagon: This tells us the general shape we're re working with but doesn't 't immediately inform the position of W relative to the hexagon. 2. I is the midpoint of side CU: This locates point within the hexagon, specifically on one of its sides. Triangle CPI right-angled at C: This establishes right-angle point C, with points P and defining the other two vertices of the triangle. Since I is on side CU of the hexagon, and C is a vertex of both the hcxagon and the triangle, P must either be inside the hexagon (if the triangle is entirely contained within it) outside (if the triangle extends beyond). 4. PW is the altitude drawn from vertex P CI: An altitude in triangle is perpendicular to the base. this case, as side CT of the triangle. Since I is the midpoint of CU and W is the foot of the altitude from P to CI, W necessarily falls on CI. 5. Point W is the foot of this altitude: This means W is exactly where PW intersects CL. positioning W directly on the line segment that part of the hexagon's 's boundary. Given these points, since W is on the line segment CI. which is an internal line segment of the hexagon, point W must be inside the bexagon CEXJZU. The answer is A. Inside. Recognizable relation with definite model conclusion V
base could considered
28

---

Published as a conference paper at ICLR 2025
Table 24: Comparison LLaMA3-8B-Imm and Fine-tuned LLaMA3-8B- Instrruct
Question:
In triangle ABC, the line AD bisects angle BAC. point D is on side BC, and segment AD is perpendicular to side
BC, segment BD is equal to 2 units. Is CD the angle bisector of triangle ABC? Answer choices:
A. Yes
B. No
C. Cannot be inferred
Gold Answer:
No
GPI-3.5-Turbe
To determine CD the angle bisector of triangle ABC, need consider the properties of perpendicular bisector. Since AD perpendicular BC and bisects angle BAC, know that triangle ADB is congruent to triangle ADC by the angle-side-angle (ASA) congruence theorem. This that BD equal CD, making CD the angle bisector of triangle ABC. Therefore, the answer is: A. Yes
X
GPT-40:
To determine whether CD the angle bisector triangle ABC, need analyze the given information and apply relevant geometric principles.
Given:
1. Triangle ABC with AD bisecting angle BAC.
2. Point D is side BCC
3. Segment AD perpendicular to side BC.
4. Segment BD equal 2 units.
Since AD is perpendicular
BC, triangle ABD triangle ADC right triangles. Additionally, since AD bisects
angle BAC. triangles ABD and ADC congruent by the Angle-Angle-Sid (AAS) congruence criterion (they share angle BAD, angle ADB is 90 degrees, and AD is common).
Because triangles ABD and ADC congruent, follows that:
BD= DC (since corresponding parts of congruent triangles are equal). Given that BD units, follows that DC =2 units well.
Since D is the midpoint of BC and AD is the angle bisector of angle BAC, it implies that CD is also the angle bisector of angle BCA.
Therefore, the correct X
is: Yes
LLLaMA-3-70B-Isstte Instruutt
A nice geometry problem!
Let's analyze the situation:
1. AD bisects angle BAC, so LBAD= LCAD.
2. AD perpendicular BC, ZADB <ADC=
3. BD units.
Now, let's consider the triangles ABD and ACD.
Since AD the bisector of angle BAC, have:
/BAD LCAD LADB ZADC (alternate interior angles).
We also know that LADB and ZADC both 90" (since AD perpendicular BC).
So, triangles ABD and ACD similar by AA similarity (two pairs of equal angles).
Now, since BD = units, we can set up proportion:
BD/ / CD AB ACC
Substituting BD 2, we get: 2/ CD AB / AC.
Since AB and AC the sides of triangle ABC, can conclude that CD is proportional the ratio of sides of triangle ABC. This means that CD is indeed the angle bisector of triangle ABC.
My choice is: Yes
X
29

---

Published as a conference paper at ICLR 2025
Table 25:
Comparison of Prompting Strategies for Geometric Reasoning (The correct option is highlighted in red).
Prompting Strategy
Question: Quadrilateral ABCD is parallelogram, but not a rectangle or rhombus. In regular pentagon BFCGE, point is the intersection of diagonals BC and FG. What is the relationship between line AB and line CE?
Answer choices:
A. Parallel
B. Perpendicular
C. Intersecting but not perpendicular
D. Cannot be inferred
Zero-Shot
Since ABCD is parallelogram, opposite sides are parallel. Therefore, line AB parallel line CD. In regular pentagon BFCGE, diagonals BF and CE intersect at point E. Therefore, line AB and line CE are intersecting but not perpendicular. The correct answer is C. Intersecting but not perpendicular.
Zero-Shot-CoT
First, we know that opposite sides of a parallelogram are parallel. Therefore, AB is parallel to CD and BC is parallel to AD. Next, we can use the fact that the diagonals of regular pentagon intersect at 72 degree angle. Since E is the intersection of diagonals BC and FG, we know that angle BEF is 72 degrees and angle CEF is also 72 degrees. Now, we can use this information to determine the relationship between AB and CE. Since AB and CE are proportional, AB and CE are parallel. The answer is Parallel.
Few-shot
The relationship between line AB and line CE cannot be inferred. The answer is D. Cannot be inferred. X
Few-Shot-CoT
Step 1: Quadrilateral ABCD is a parallelogram, so opposite sides are parallel. Step 2: Regular pentagon BFCGE has diagonals that intersect at point E, which is on line CE. Step 3: Since line CE intersects both diagonal BC and diagonal FG, it must be a transversal to these two lines. Step 4: By the transversal property, if AB is parallel BC, then AB must also be parallel to FG. Step 5: However, we cannot infer whether AB is parallel to BC or not, cannot determine the relationship between line AB and line CE. The answer is D. Cannot be inferred. X
Table Necessary Condition Coverage (NCC) results for Few-Shot CoT and Few-Shot GeoCoT
methods.
NCC (%)
CoT
GeoCoT (Stage 1/ /Sage 2)
Base
84.54
90.76 (81.10 (81.10/9.63)
Advanced
57.14
68.35 (55.28 / 13.07)
30

---

Published as a conference paper at ICLR 2025
Table 27: Reasoning obstacles in the Few-Shot-CoT group. The correct answers from the options highlighted in green, while the reasoning errors the responses are highlighted in red.
Question 1:
Points A, B, C, and D are vertices of convex quadrilateral. Are points A, and C collinear?
Answer choices:
A. Yes No C. Cannot he inferred
Response:
Step I: Points A, B, C, and D are vertices of a convex quadrilateral, so they form closed figure.
Step 2: Collinear points are puints that lie on the same line. To determine if A, B, and C are collineur, need
to check if they lie on the same line.
Step 3: Without any additional information or given relationships between the points, cannot determine if A, and C collinear. The answer is C. Cannot he inferred.
Question
Triangle ABC right-angled at B. What the relationship between line AB and line BC?
Answer choices:
A. Parallel B. Perpendicular C. Intersecting but not perpendicular D. Cannot be inferred
Response:
Step I: Triangle ABC is right-angled
which means that angle B is right angle.
Step 2: In right-angled triangle, the side opposite the right angle called the hypotenuse, and the other two
sides are called the legs.
Step 3: Line AB is one of the legs of triangle ABC, and line BC the hypotenuse. relationship between line AB and line that they are intersecting but not perpendicular. The answer is C. Intersecting but not perpendicular. Question 3:
Step 4: Therefore,
Line EF intersects side AB of triangle ABC at an obtuse angle, circle with center E intersects EF at points D and G. What is the relationship between line AB and line DF?
Answer choices: A. Parallel B. Perpendicular C, Intersecting but not perpendicular D. Cannot be inferred
Response:
Line EF intersects side AB of triangle ABC at an obtuse angle, so we can infer that point is on the same side of line AB as point C.
Step 1:
Step 2: A circle with center E intersects EF at points D and G, so points D, E, and G are collinear.
Step 3: Since puints D and G are on the circle with center E, we can infer that line DE is perpendicular to line EG.
Step 4: Therefore, line DF is perpendicular to line AB. The answer is B. Perpendicular.
Question 4:
If ABC is triangle. point D is on the extension line of side BC, CD 1/2 BC, in isosceles trapezoid AGDF, with AG II DF and AF GD, E is the intersection of AD and GF. What is the relationship between point and triangle ABC?
Answer choices:
A. Inside B. Outside C. Cannot be inferred
Response:
Step I: Point D is on the extension line of side BC, CD 1/2 * BC, so point D is outside triangle ABC.
Step 2: In isnsceles trapezoid AGDE, with AG DF and AF GID, point E. is the intersection of Al) and GF:
Step 3: Since point D is outside triangle ABC and point H is linc AD,, we cannot determine whether point B is inside outside triangle ABC.
Step 4: Therefore, the answer is C. Cannot be inferred.
Question 5:
Line AB intersects line CD point E, and line EF is perpendicular to line CD at point P. What the relationship between LAED and LAEC?
Answer choices:
A. Vertical angles B. Supplementary angles C. None of the above D. Cannot be inferred
Response:
Step 1: Line AB intersects line CD at point
infer that ZAEC and ZAED adjacent angles. Step 2: Line EF is perpendicular line CD point so we can infer that ZAED and ZDEF are complementary angles.
Since ZAED /DEF are complementary angles and ZAEC and LAED are adjacent angles, we cannot determine the relationship between /AED and / AACC. The answer C. None the above.
Step 3:
31

---

Published as a conference paper at ICLR 2025
Table 28: Results of GPT-3.5-turbo with CoT and GeoCoT (The correct option is highlighted in red).
Question:
Quadrilateral ABCD with diagonals intersecting at point line tangent circle at point E. point B the line
AF?
Answer choices:A, Yes B, No C. Cannol be inferred
Zero-shot-Co Response:
1. Since line AC is tangent to circle O at point F, we know that angle AFE is a right angle (tangent line is perpendicular to the radius at the point of tangency).
2. Since the diagonals quadrilateral ABCD intersect at point E, we know that triangles ABE and CDE are simtlar (by
AA similarity).
3. Since triangles ABE and CDE are similar, angle AFB is congruent to angle CED.
4. Since angle AEB is congruent to angle CED, and angle AFE is right angle, we can conclude that angle AEB is also
right
point B lies on the line AF x Therefore, the answer is; A. Yes.
Zero-shot-GeoCof Response:
Stage 1 result:
Quadrilateral ABCD with diagonals intersecting point E:
Draw quadrilateral ABCD with points A, B, C, and D.
Draw the diagonals AC and BD intersecting at point E.
Line AC is langent to circle O at point F:
Draw circle O with center at point E.
Draw line AC intersecting circle al point F such that it is langent to the circle.
Breakdown of the geometty.
Quadrilateral ABCD has diagonals AC and BD intersecting point E.
Line AC is langent to circle O al point F, where circle O is centered at point E.
This creates a relationship between the quadrilateral and the circle, with the langent line touching the circle at point F,
The properties of tangents and circles can be used to find angles and lengths within the geometry of the figure.
Stage 2 result:
Given that line AC is langent circle O at point know that the angle between the tangent line and the radius the point of tangency is 90 degrees. To determine point is on the line AF, can assume that point B is on the line AF and see if it leads to contradiction.
Assume point B is on the line AE:
Since line AC tangent to circle O point E. angle APE is a right angle (tangent and radius are perpendicular). point is line AF, then angle AFB also right angle (since AF tangent circle O).
This would mean that points A, F, and B are collincar.
However, this assumption leads to contradiction because in quadrilateral ABCD, the diagonals AC and BD intersect at point E. If points A. F. and B are collinear, then point E would lic on line AF, which is not possible since E is the intersection of the diagonals AC and BD.
Therefore, the assumption that point on the line AF leads to a contradiction. lence, point B not on the line AF, Therefore, the correct answer B. No.
Few-shot-Col Response:
Step 1: Line AC is tangent to circle O point H. so F is the point of tangeney, We can infer that F line AC. Step 2: Quadrilateral ABCD with diagonals intersecting at point F, so point E is the intersection of diagonals AC and BD. Therefore, point E lies on line AC. Step 3: Since point E lies on line AC and F is the point tangency on line AC, we can conclude that point B is on the linc AF.
Step 4: Therefore, the answer is A. Yes.
Few-shot-GeoCoT Response:
Stage 1 resulf:
Quadrilateral ABCD:
A four-sided polygon with vertices at points A. B, C, and D.
The sides of the quadrilateral are AB, BC, CD, and DA, which connect the vertices.
Diagonals of quadrilateral ABCD:
The diagonals are AC and BD, which intersect point E.
Diagonals intersect inside the quadrilateral at point E, creating two triangles: ABE and CDE.
Line AC tangent to circle O at point F:
Circle O has point of tangency at F, where line AC touches the circle.
Tangent property: Line AC touches cirole O at exactly one point, which point F.
The tangeney implics that
perpendicular to the radius circle at point F (if the radius OF is drawn).
Stage 2 resds:
We want to confirm the relationship between point B and line AF Point B is vertex of quadrilateral ABCD.
If point B lies on line AF, then B. A, and F are collincar. Line AC is tangent to circle O at point F. mcaning F lics on line AC and the point of tangency. There is no indication that point is connected to cirele O or line AC hased on information provided. Quadrilateral ABCD has diagonals intersecting al point which means diagonal AC diagonal BD. Since is vertex of the quadrilatcral, it lies on diagonal BD, but there's no relationship provided that would place B on line AF, so there is contradiction.
a point both
Conclusion: Therefore, point B is not on line AF The answer is B. No.
32

---

Published as a conference paper at ICLR 2025
Table 29: Results of GPT-3.5-turbo with CoT and GeoCoT (The correct option is highlighted in red).
Question:
In triangle ABC, the line AD is perpendicular to side BC, point D is on side BC, the length of side AB is 5 units, and the measure of angle A is 90 degrees. Is BD the altitude of triangle ABC?
Answer choices:A. Yes B. No C. Cannot be inferred.
Zero-shot-Co'' Response:
Since angle A is 90 degroes and line AD is perpendicular to side BC, we can conclude that triangle ABC is right triangle.
Given that the length of side AB is 5 units, we can the Pythagorean theorem find the length of side BC. x. Using the Pythagorean theorem, we have:
Let length of side
(AB)² + (BC)² = (AC)². x
Repeating calculations. the answer is not concluded.
Geoo-S Response:
Stage 7 result:
1. Triangle ABC is a right triangle because angle A is 90 degrees.
2. Line AD is the altitude of triangle ABC because it is perpendicular to side BC.
3. The length of side AB is 5 units.
4. Since angle
90 degrees, triangle ABC is right triangle with right angle A. 5. The length of side AD is the height of the triangle, which is the altitude from A to side BC.
6. The length of side AD can be found using the Pythagorean theorem, where AD2 + BD² = AB
7. Once the length of side AD is found, the area of triangle ABC can be calculated using the formula: Area=
1/2 * base height.
Stage 2 result:
To determine if BD is the altitude of triangle ABC, we can assume that BD is the altitude and see if leads to a contradiction.
Assume BD is the altitude of triangle ABC. This means that AD is perpendicular to BC and BD is perpendic- ular to AC.
Since triangle ABC is right triangle with angle 90 degrees, AD the altitude of the triangle. This implics that BD is not the altitude, as there can only be one altitude in triangle. Therefore, BD cannot be the altitude of triangle ABC. The correct answer B. No.
Answer: B. No. V
Few-shot-Cof Response:
Step I: triangle ABC, the line AD is perpendicular to side BC, so BD is segment that connects vertex of the triangle (B) to the foot of the altitude (D).
Step 2: The length of side AB is 5 units, and angle A is 90 degrees, making triangle ABC right triangle.
Step 3: In a right triangle, the altitude from the right angle connects the right angle to the hypotenuse.
Step 4: Therefore, BD is the altitude. Answer: A. Yes. X
Few-shot-GeoCol Response:
Stage I result:
Triangle ABC: This is right triangle because angle A is 90 degrees. The sides of the triangle are AB, BC, and AC, which intersect at the vertices A, and C. Line AD: This line is perpendicular to side BC, meaning it forms a right angle (90 degrees) with BC at point D. Point D lies on side BC, so AD is an altitude of the triangle.
Point D: This point is located on side BC, and since AD is perpendicular to BC, D is the foot of the altitude
from point A to side BC.
Side AB: The length of side AB is given units.
Stage 2 result:
If BD is the altitude of triangle ABC, then BD must be perpendicular to side AC, as the altitude extends from a vertex and forms right angle with the opposite side. However, the problem states that AD is perpendicular to BC, making AD the actual altitude. In right triangle, there can only be one altitude from a vertex to specific side, so it leads Therefore, the answer is B: No, BD is not the altitude of triangle ABC. Answer: B. No.
contradiction.
33


Reviews:
[{"review_id": "Nz1BdqJsGw", "reviewer": "Reviewer_XLWM", "summary": "This paper investigates whether large language models can truly understand geometric structure and solve geometric problems. It introduces a new benchmark, **GeomRel**, focused on geometric relationships in three categories: line-based, angle-based, and shape-based relationships. To expand the dataset’s depth and complexity, the authors construct advanced and diverse data subsets within GeomRel. Evaluations on this benchmark reveal that even advanced models like GPT-4o outperform Random Guess by only 20.34% on complex tasks in the advanced GeomRel subset. Building on insights from these evaluations, the paper proposes a two-stage pipeline that guides models to decompose geometric structures and perform relationship observation. Experimental results demonstrate the effectiveness of the proposed pipeline in enhancing model performance on geometric tasks.", "questions": "- Could you elaborate on how the benchmark was constructed and clarify how the different relationships, difficulty levels, and diversity augmentation methods were integrated? And for instance do you leverage LLMs to help generate the benchmark or all the questions are based on rules?\n- Could you provide results from fine-tuning on the LLaMA-3-8B base model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper introduces a new benchmark that reveals current LLMs struggle to effectively recognize geometric structures.\n- It demonstrates that even with few-shot prompting or fine-tuning, current LLMs do not perform well in recognizing geometric relationships. To address this, the paper proposes a two-stage pipeline that guides LLMs to decompose and observe geometric structures.\n- Experimental results in the final section show that the proposed two-stage pipeline effectively enhances LLM performance on geometric tasks.", "weaknesses": "- This paper dedicates significant space to describing the rules and various data augmentation methods used in constructing the benchmark. However, the overall process and rationale for construction could be presented more clearly.\n- In Section 3.5, the paper uses the LLaMA-3-8B-Instruct model as the base model for fine-tuning, which is somewhat unconventional, as it is more typical to fine-tune base models on math-related datasets.\n- Including additional experimental metrics commonly used in math domains, such as pass@k, majority voting, or best-of-n, could improve the depth of analysis and provide a more comprehensive evaluation of model performance in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qCazHySzwk", "reviewer": "Reviewer_Nc3K", "summary": "This paper addresses the limitations of large language models (LLMs) in understanding geometric structures, proposing a new benchmark, GeomRel, specifically designed to assess LLMs' ability to identify geometric relationships. The authors identify that while existing datasets mainly measure final answer accuracy, they fail to capture whether LLMs truly understand underlying geometric structures. The GeomRel benchmark isolates the task of geometric relationship identification (GRI) as a foundational skill for geometric reasoning. Using GeomRel, the paper evaluates several LLMs and finds that most perform well on simple geometric relationships but struggle with more complex structures, particularly those involving angle relationships. The authors introduce a new method, Geometry Chain-of-Thought (GeoCoT), which decomposes geometric reasoning into step-by-step relationship identification, significantly improving model performance on GeomRel by over 9% on basic tasks and nearly 15% on advanced tasks.", "questions": "1. It would help the readers better, if the authors can provide the exact versions of proprietary LLMs that was used as well as other hyper parameters set for these models for inferences as well as for the fine tuning experiments.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The GeomRel dataset, which isolates geometric relationship identification as a key step, provides a novel and focused way to evaluate LLMs' geometric understanding. This dataset addresses a unique gap in the field and enables more focused evaluation of geometric reasoning capabilities.\n\n2. The Geometry Chain-of-Thought (GeoCoT) method improves LLMs’ performance in identifying geometric relationships by breaking down problems into reasoning steps. The proposed method significantly improved both basic and advanced variants of the dataset.\n\n3. The authors evaluates a range of LLMs, including both proprietary and open-source models, providing a rounded view of how different models handle geometric reasoning. This extensive evaluation is valuable for identifying model-specific strengths and weaknesses in geometric tasks.\n\n4. The paper investigates multiple prompting methods, including Zero-Shot and Chain-of-Thought (CoT), analyzing their effectiveness in GRI tasks. This analysis highlights that while traditional CoT is less effective for geometric tasks, the GeoCoT adaptation provides usefgul improvements for geometric reasoning.", "weaknesses": "1. Although fine-tuning on GeomRel was attempted, the results only report results of a single model. More experiments exploring different models could clarify which models has performance improvements with fine-tuning.\n2. It is difficult to truly understand the difficulty of the dataset for human intelligence. This can be understood by sampling the dataset and carrying out a systematic human evaluation to report human baselines on these geometrics reasoning problems.\n3. GeomRel, while valuable for evaluating LLMs on basic geometric relationships, covers a limited range of geometric concepts, primarily focusing on 2D relationships involving lines, angles, and shapes. This narrow scope may not fully represent the complexity of real-world geometric tasks and it will further help evaluate reasoning capabilities particularly those that involve 3D relationships, transformations (like rotations and reflections), or coordinate-based reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HhM9TCfZ4W", "reviewer": "Reviewer_8Tsz", "summary": "This paper investigates LLMs abilities to understand geometric relationships and spatial reasoning based on textual descriptions. The authors observe that LLMs often arrive at correct answers without grasping the underlying geometric relationships. To address this, they propose a new task called Geometric Relationship Identification (GRI) and introduce GeomRel, a dataset specifically designed to evaluate models' abilities in GRI. For dataset construction, they create a collection of basic geometric relationships and develop more complex examples by combining and enhancing these foundational relationships.\n\nThe authors conduct extensive experiments across several models and strategies, including zero-shot, few-shot, Chain-of-Thought (CoT), and fine-tuning methods. Their findings indicate that current LLMs have significant limitations in spatial understanding and that most strategies yield limited improvements. They propose a new prompting technique, GeoCoT, which decomposes geometric observations and employs reverse reasoning.", "questions": "* While the advanced split of GeomRel does require managing multiple geometric relationships, it still treats each relationship as relatively independent. Real-world geometric problems, however, often demand that LLMs execute step-by-step calculations in sequence, where each conclusion logically affects the next. \n* Some analyses in Appendix F are particularly insightful, demonstrating the reasoning path and showing how the vanilla model fails while reverse reasoning aids understanding. It would be valuable to provide experimental results to support these explanations.\n*  the paper show marginal gains from fine-tuning. It would be valuable to explore the reasons behind this limited improvement, possibly factors like dataset size, or the nature of the fine-tuning data. \n* In explaining the performance improvements from point relabeling, the authors hypothesize that \"using more complex descriptions may stimulate LLMs’ reasoning abilities.\" This point could benefit from further explanation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper establishes GRI as a new task, shifting focus beyond answer accuracy to intermediate steps in spatial reasoning. \n* The proposed dataset, GeomRel, is valuable, especially with its advanced split that incorporates logical chains, indeterminate cases, and extraneous information, mimicking real-world problem complexity and ambiguity. \n* The paper provides a thorough evaluation across multiple LLMs and reasoning methods, yielding insights into their spatial reasoning capacities and limitations.", "weaknesses": "* For disambiguation, the authors manually reviewed and excluded ambiguous data throughout the data construction process, which may reduce scalability and limit others’ ability to expand the benchmark.\n* The paper’s focus on GRI as an isolated skill may be too narrow, leaving it uncertain if success in GRI tasks will translate to general spatial reasoning or even multi-step problem-solving abilities.\n* The two-stage GeoCoT method, which involves decomposing geometry problems and reverse reasoning, may be challenging to scale to more complex tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)