GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving (2024)

Abstract: Recent advancements in large language models (LLMs) and multi-modal models (MMs) have demonstrated their remarkable capabilities in problem-solving. Yet, their proficiency in tackling geometry math problems, which necessitates an integrated understanding of both textual and visual information, has not been thoroughly evaluated. To address this gap, we introduce the GeoEval benchmark, a comprehensive collection that includes a main subset of 2,000 problems, a 750 problems subset focusing on backward reasoning, an augmented subset of 2,000 problems, and a hard subset of 300 problems. This benchmark facilitates a deeper investigation into the performance of LLMs and MMs in solving geometry math problems. Our evaluation of ten LLMs and MMs across these varied subsets reveals that the WizardMath model excels, achieving a 55.67% accuracy rate on the main subset but only a 6.00% accuracy on the hard subset. This highlights the critical need for testing models against datasets on which they have not been pre-trained. Additionally, our findings indicate that GPT-series models perform more effectively on problems they have rephrased, suggesting a promising method for enhancing model capabilities.1

heading: 1 Introduction
text: Geometry math problems are a key component in assessing the mathematical reasoning skills of K12 students, serving as a critical benchmark for evaluating educational outcomes (Zhang et al., 2023c). The complexity of solving these problems stems from the requirement to interpret both textual and visual information, in addition to applying mathematical reasoning skills. This complexity has made geometry problem-solving a key area of interest for researchers aiming to evaluate the capabilities of
∗ Equal Contribution † Corresponding Author
1The code and data available are at https://github.com/ GeoEval/GeoEval.
AI models in this domain (Chou and Gao, 1996; Ye et al., 2008; Zhang et al., 2023a; Trinh et al., 2024; Zhang et al., 2024; Zhang and Moshfeghi, 2024).
In recent years, several datasets, such as Geometry3K (Lu et al., 2021), PGPS9K (Zhang et al., 2023b), and GeomVerse (Kazemi et al., 2023), have been developed to test the proficiency of AI models in solving geometry math problems. Yet, these datasets often lack a standardized format and sufficient diversity, complicating the assessment of models’ genuine proficiency in geometry problemsolving. Furthermore, these datasets typically focus on one type of geometry problem, such as flat geometry, overlooking other crucial areas like solid geometry. This oversight limits the ability to conduct a thorough evaluation across the full spectrum of geometry problems.
Simultaneously, advancements in large language models (LLMs) and multi-modal models (MMs) have demonstrated significant potential in handling complex reasoning tasks (Chen et al., 2022b; Wei et al., 2022; Zhang et al., 2023d; Chen et al., 2023). This potential has raised considerable interest in testing these advanced models across a variety of tasks, such as math word problem solving (Lu et al., 2023) and physical problem solving (Sawada et al.,
1258
2023). Despite this interest, specific research on evaluating these models’ effectiveness in geometry problem-solving remains scarce. Therefore, it is critical to develop a new, comprehensive benchmark that can effectively assess LLMs and MMs in geometry problem-solving, especially considering the potential exposure of existing public datasets during model training (Sainz et al., 2023). Comparing the performance of current LLMs and MMs on such a benchmark is essential, as it could yield valuable insights that further the development of models capable of tackling complex reasoning tasks.
To prompt research towards assessing LLMs’ and MMs’ proficiency in geometry math problemsolving, we introduce the GeoEval benchmark, a comprehensive collection specifically designed for this task. GeoEval features its Comprehensive Variety, sourced from seven public datasets and formatted uniformly to encompass a wide range of geometric shapes. It includes Varied Problems, covering flat, solid, and analytic geometry to challenge models comprehensively. GeoEval supports Dual Inputs, accommodating both geometric diagrams and textual problem statements, making it suitable for evaluating both LLMs and MMs. To counter the potential overfitting to previously seen datasets, GeoEval introduces Diverse Challenges through backward reasoning, augmented, and hard subsets, each designed to test different aspects of models’ geometry problem-solving abilities. Additionally, GeoEval is annotated with Complexity Ratings, allowing for a fine-grained analysis of model performance across various difficulty levels, thus providing a robust framework for advancing AI capabilities in understanding and solving geometry math problems. Examples of geometry problems from our GeoEval can be found in Figure 1.
In this paper, we conduct extensive experiments using the GeoEval benchmark to evaluate the proficiency of ten LLMs and MMs in solving geometry problems. This includes three LLMs: CodeGen216B (Nijkamp et al., 2023), GPT-3.5 (OpenAI, 2022), and GPT-4 (OpenAI, 2023); two LLMs specialized in mathematics: WizardMath-70B and WizardMath-7B-V1.1 (Luo et al., 2023); and five MMs: llava-7B-V1.5 (Liu et al., 2023), Qwen-VL (Bai et al., 2023b), mPLUG-Owl2 (Ye et al., 2023), InstructBLIP (Dai et al., 2023), and GPT-4V (OpenAI, 2023). The findings reveal that GeoEval forms a challenging benchmark, with both LLMs and MMs struggling to resolve its complexities effec-
tively. Notably, our results indicate that: 1 Models pre-trained on mathematical corpora, such as the WizardMath models, deliver superior performance across various GeoEval subsets (Section 4.3.1), establishing new benchmarks in the field. 2 One advantage of these models is that they implicitly encompass the required mathematical knowledge demanded to solve geometry math problems (Section 4.6). 3 However, we also find that though pre-training on a mathematical corpus is crucial for solving geometry math problems, it may not be enough (Section 4.3.4). 4 Additionally, we observe that GPT series models exhibit enhanced problem-solving efficiency when tackling geometry questions that they have previously rephrased (Section 4.3.4). 5 Further analyses underscore the value of incorporating descriptions of geometric diagrams, which significantly aids LLMs in understanding and solving geometry problems (Section 4.5). 6 Finally, our experiments show that the performance of both LLMs and MMs declines as the problem length and complexity of the problem increases (Section 4.7). Through the GeoEval benchmark, we believe this research provides the first comprehensive quantitative assessment of the latest LLMs and MMs in the domain of geometry problem-solving.

heading: 2 Related Work
text: Numerous benchmarks have been developed to assess the capabilities of LLMs in the the geometry problem-solving task. However, these benchmarks face limitations, such as restricted access, like GEOS (Seo et al., 2015) and GeoShader (Alvin et al., 2017) datasets, or insufficient scale, as seen with GEOS++ (Sachan et al., 2017). Although recent efforts have introduced new benchmarks like Geometry3K (Lu et al., 2021), UniGeo (Chen et al., 2022a), and PGPS9K (Zhang et al., 2023b), they still fall short in offering a uniform format and embracing a wide range of problem types. In response, we introduce the comprehensive and challenging GeoEval benchmark, aiming to advance the evaluation of geometry problem-solving abilities.
Recently, LLMs (Peng et al., 2023; Touvron et al., 2023; OpenAI, 2022) and MMs (Liu et al., 2023; Ye et al., 2023; OpenAI, 2023) have achieved impressive results on complex tasks, attracting research into their performance across specialized tasks. Previous work like MathVista (Lu et al.,
2023) have concentrated on scientific domains, likewise SEED (Li et al., 2023) explores models’ understanding of temporal and spatial relationships. Despite these advancements, there remains a gap in the examination of models’ ability to solve geometry math problems. Through the GeoEval benchmark, we aim to fill this gap by offering a detailed assessment of both LLMs’ and MMs’ abilities to tackle a variety of geometry math challenges.

heading: 3 GeoEval Dataset
text: The GeoEval benchmark is structured into four subsets: GeoEval-2000, comprising 2,000 problems; GeoEval-backward, with 750 problems; GeoEvalaug, containing 2,000 problems; and GeoEval-hard, including 300 problems. The subsequent sections will detail the collection process for each subset, followed by an explanation of the unique features of the GeoEval benchmark.2

heading: 3.1 Data Collection

heading: 3.1.1 Collection from Diverse Data Sources
text: We have compiled a comprehensive collection of public geometry math problem datasets, with a total of 24,912 geometry math problems from sources such as Geometry3K (Lu et al., 2021), PGPS9K (Zhang et al., 2023b), UniGeo (Chen et al., 2022a), GeoQA+ (Cao and Xiao, 2022), GeometryQA (Tsai et al., 2021), as well as geometry problems from the MATH (Hendrycks et al., 2021) and MathQA (Amini et al., 2019) datasets. The first four datasets feature geometry questions that include both problem texts and geometric diagrams, whereas the latter three datasets comprise questions that only contain problem texts. Detailed information about all source datasets is available in Appendix B.
Building on the data gathered, we then selected 2,000 geometry math problems to create our GeoEval-2000 subset. This selection process was guided by the aim to inclusively cover a wide range of basic geometric shapes, ensuring a broad representation of geometry concepts. The distribution of geometric shapes within this subset is further detailed in Appendix C.

heading: 3.1.2 Backward Data Generation
text: In contrast to forward problems, backward problems use the answer from forward problems as a
2Statistics for the GeoEval benchmark are in Appendix A.
starting point, posing a query to determine a specific number that was part of the forward problems but is concealed in the backward problems (Jiang et al., 2023). These types of questions are particularly effective in assessing models’ capability for multi-step reasoning. Following the methodology of previous research (Yu et al., 2023), we selected 750 problems from the GeoEval-2000 subset and created corresponding backward questions. This process involved masking a number, the solution of the forward problems, as "X". The prompt "The correct answer is ansgold. Now please answer what is the value of X?", where ansgold represents the correct answer to the forward problems, is then added. The example of backward problems can be found in Appendix D.

heading: 3.1.3 Augmented Data Generation
text: To evaluate the resilience of current models and mitigate the risk of data leakage that may occur during the pre-training phase, we implement a context learning strategy for rephrasing problems from the GeoEval-2000 subset. Each problem is rephrased into five variant candidates by GPT-3.5 (OpenAI, 2022), ensuring they retain the original problem’s semantic essence while varying in lexical structure. Out of these five alternatives, one is selected randomly to substitute the original problems, forming the GeoEval-aug subset.

heading: 3.1.4 Hard Data Collection
text: While the GeoEval-2000 subset comprises geometry problems from a variety of source datasets, it exhibits a lack of diversity in problem categories, notably in solid geometry and analytic geometry. To enhance the diversity of problem categories, we introduce the GeoEval-hard subset, which includes 300 geometry problems specifically focusing on solid geometry and analytic geometry, providing a broader assessment scope. More details regarding the comparison between the GeoEval-hard subset with other datasets are in Appendix E.
The GeoEval-hard subset sources from the copyrighted collection containing 10,000 geometry math problems created based on templates summarized from the online resources. An initial selection is made using a rule-based engine equipped with a keyword list, targeting solid and analytic geometry problems. This step yields around 3,100 potential problems, identified as the GeoEval-hard-raw subset. Next, a manual review further narrows these down to 300 problems related to solid and analytic
geometry. The cleaning and manual inspection process is documented in Appendix F.

heading: 3.2 Features of GeoEval
text: The GeoEval benchmark is specifically designed to assess the ability of models to resolve geometric math problems. This benchmark features five characteristics: Comprehensive Variety, Varied Problems, Dual Inputs, Diverse Challenges, and Complexity Ratings, with each attribute exemplified in the Appendix G. For an insightful contrast, Table 1 offers a comparative analysis of GeoEval against earlier datasets.
Comprehensive Variety GeoEval consists of a diverse collection of geometry problems sourced from the seven most recent datasets. Therefore, the problems in GeoEval cover a wide range of geometric shapes, offering a comprehensive view of varied geometry math challenges.
Varied Problems The GeoEval benchmark encompasses three distinct categories of geometry math problems, namely flat geometry, solid geometry, and analytic geometry.
Dual Inputs GeoEval features problems in two formats: those accompanied by diagrams and those consisting solely of text. This versatility makes it suitable for evaluating models that process diagrams or text-based inputs.
Diverse Challenges In addition to gathering public datasets, GeoEval also generates its out-ofdistribution dataset aimed at addressing data leak-
age problems. This includes a backward reasoning subset, an augmented subset, and a hard subset, all created by us.
Complexity Ratings GeoEval is equipped with annotations indicating the complexity level for each problem, serving as a guideline to evaluate models’ proficiency in solving these tasks.3

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: In this study, we deliberately select state-of-the-art LLMs and MMs that are widely recognized for their advanced capabilities, including:
• LLMs Specialized in Programming Code: We include CodeGen2-16B model (Nijkamp et al., 2023), which is renowned for its proficiency in understanding and generating programming code, offering insights into its adaptability to solve geometry math problems.
• LLMs with a Focus on Mathematics: This includes WizardMath-7B-V1.1 and WizardMath-70B (Luo et al., 2023), explicitly pre-trained on mathematical corpora. Their inclusion allows for an assessment of models that have been fine-tuned to tackle complex mathematical problems.
• LLMs Designed for a Broad Range of Topics: Models such as GPT-3.5 (OpenAI, 2022)
3Algorithm for classifying complexity is in Appendix H.
and GPT-4 (OpenAI, 2023) exemplify the advanced commercial LLMs engineered to encompass a broad range of topics.
• Multi-Modal Models (MMs) with Diverse Decoders: Given the ubiquity of ViT architecture (Dosovitskiy et al., 2021) as the vision encoder in MMs, we select models that integrate ViT with various LLMs as decoders. This includes llava-7B-V1.5 (Liu et al., 2023) with Vicuna (Peng et al., 2023), Qwen-VL (Bai et al., 2023b) using Qwen (Bai et al., 2023a), mPLUG-Owl2 (Ye et al., 2023) with LLaMA (Touvron et al., 2023), InstructBLIP (Dai et al., 2023) with Vicuna (Peng et al., 2023), and GPT-4V (OpenAI, 2023).
These models are evaluated through a zeroshot approach, utilizing straightforward instruction prompts to directly assess their geometry problemsolving capabilities without further fine-tuning specifically for our benchmark.4

heading: 4.2 Evaluation Metric
text: Building upon the approach by MathVista (Lu et al., 2023), we first input the generated sequence from the model into GPT-4 to extract the target value or option letter. To enhance the precision of our answer extraction, we formulate intricate rules for post-processing the outcomes in cases where GPT4 falls short. Specifically, our extraction pipeline involves two steps: firstly, using a prompt to extract the answer. Secondly, employing regular expressions to extract any remaining answers that couldn’t be obtained from the first step. Please refer to Table 4 and Table 5 in Appendix for the extraction instruction and the constructed samples. This approach has enabled us to attain an extraction accuracy surpassing 97%5, similar to the success rate reported in MathVista (Lu et al., 2023).
The extracted results are compared against the golden answers to determine the final performance metric. Given the model’s intention to produce responses in varying formats, either as the precise answer (for instance, "3.15") or as the corresponding option letter (such as "A"), we regard a prediction as accurate if it either matches the golden answer or the golden option letter.
4Details on the prompt design and the hyper-parameters used for these models are available in Appendix I.
5We assess the accuracy of the extraction by manually reviewing 200 uniformly sampled examples.

heading: 4.3 Experimental Results
text: In this section, we present the accuracy achieved by models on our GeoEval benchmark. Table 2 highlights that models pre-trained on a math-specific corpus tend to outperform others. Furthermore, except for llava-7B-V1.5 and Qwen-VL, multi-modal models (MMs) generally exceed the performance of large language models (LLMs). Notably, InstructBLIP exhibits exceptionally high accuracy scores across all subsets, yet its results raise some concerns, and we have chosen to exclude the InstructBLIP model. The rationale behind this decision is detailed in Appendix J.

heading: 4.3.1 Comparison among LLMs
text: When reviewing the performances of LLMs as detailed in Table 2, it becomes evident that models pre-trained on mathematical corpora demonstrate superior efficacy in solving geometry math problems compared to those trained on general corpora. Specifically, evaluating on all problems of the GeoEval-2000 subset (marked as "A" in the table), WizardMath-70B leads with an accuracy of 55.67%, while WizardMath-7B-V1.1 closely follows with a 54.78% accuracy, outperforming other LLMs. Conversely, GPT-4, GPT-3.5, and CodeGen2-16B report notably lower accuracies, all under 30.00%. Focusing on questions solely based on problem text within the GeoEval-2000 subset (indicated as "T" in the table), GPT-4 emerges as the frontrunner, securing the highest accuracy of 43.86%, with WizardMath models also surpassing the 32.00% accuracy. These findings underscore the enhanced proficiency of models pre-trained on math-specific corpora in tackling geometry math problems, particularly when problems are welldescribed textually, as evidenced by GPT-4’s leading performance.
In the GeoEval-backward subset, WizardMath7B-V1.1 excels with the highest accuracy of 32.66%, closely followed by WizardMath-70B at 28.66%. This significant drop in performance across all LLMs, compared to the GeoEval-2000 results, highlights a collective weakness in backward reasoning capabilities. For the GeoEval-aug subset, WizardMath-7B-V1.1 again tops the leaderboard with an accuracy of 47.75%, with GPT-4 not far behind at 45.75% accuracy. Lastly, within the GeoEval-hard subset, all models, excluding GPT-3.5, exhibit relatively low accuracies, indicating a broad difficulty in addressing the most
challenging solid geometry and analytic geometry problems. To investigate the reason for GPT-3.5 achieves better performance than GPT-4 on the GeoEval-hard subset, we find that GPT-4 tends to generate verbose solutions, often accompanied by code, which causes it to either terminate before solving the problem or enter a self-cycling loop of generating redundant information, failing to provide a final answer. In contrast, GPT-3.5 adopts a more concise approach, consistently producing option letters (e.g., "A") following the reasoning steps as solutions. We believe this concise solution generation strategy contributes to GPT-3.5’s relatively better performance on the GeoEval-hard subset.

heading: 4.3.2 Comparison among Multi-Modal Models
text: Table 2 shows that among the MMs, GPT-4V and mPLUG-Owl2 consistently outperform their counterparts across all subsets. Specifically, within the GeoEval-2000 subset, mPLUG-Owl2 leads with an accuracy of 37.76%, closely followed by GPT-4V at 37.22%, with the remaining MMs falling behind at lower accuracies. Specifically, Qwen-VL and llava-7B-V1.5 achieve accuracies of 25.60% and 12.80%, respectively. When examining problems that only involve texts, GPT-4V achieves a 43.86% accuracy, significantly surpassing llava-7B-V1.5 (21.01%) and Qwen-VL (25.97%).
In the GeoEval-backward subset, mPLUG-Owl2 tops with the accuracy of 35.33%, with GPT-4V following at 26.00% accuracy. This performance shows a notable lack of backward reasoning skills, as illustrated by the diminished results of llava-7BV1.5 and Qwen-VL in this category. Moving to the GeoEval-aug subset, GPT-4V leads with an impressive 45.75% accuracy, with mPLUG-Owl2 in second place with 38.00% accuracy. Both Qwen-VL and llava-7B-V1.5 show comparable performances in this subset. Lastly, within the GeoEval-hard subset, mPLUG-Owl2 demonstrates the highest efficacy with a 22.66% accuracy, closely followed by Qwen-VL and llava-7B-V1.5. Surprisingly, GPT4V records a lower accuracy of just 10.10%, highlighting the challenging nature of the GeoEval-hard subset and the varied capabilities of MMs in addressing the most difficult problems.

heading: 4.3.3 Comparison between LLMs and Multi-Modal Models
text: In the GeoEval-2000 subset, specifically for problems that only include texts, GPT-4’s performance exceeds the top MMs, Qwen-VL, by 17.89%. This is attributed to the MMs’ inability to access geometric diagrams, which likely hinders their comprehension of the problems. Moreover, when evaluating all problems of the GeoEval-2000 subset, WizardMath-70B surpasses the best MMs, QwenVL, by 17.91% in accuracy. However, MMs like
GPT-4V and mPLUG-Owl2 achieve significantly higher accuracy than LLMs not pre-trained on mathematical content. This underscores the value of mathematical pre-training for excelling in geometry problem-solving. Notably, GPT-4V’s accuracy on all GeoEval-2000 problems is 9.27% higher than GPT-4’s, suggesting GPT-4V’s superior capability in solving geometry problems with diagrams.
This pattern persists in the GeoEval-aug subset, where WizardMath-7B-V1.1, a model trained on a mathematical corpus, achieves the highest accuracy at 47.75%. Conversely, mPLUG-Owl2 leads in the GeoEval-backward and GeoEval-hard subsets, with accuracies of 35.33% and 22.66%, respectively. Given that GeoEval-aug rephrases questions from GeoEval-2000, it implies both subsets might have been exposed to the models during their pre-training phase. In contrast, GeoEvalbackward and GeoEval-hard subsets are less likely to have been previously exposed. This suggests that WizardMath-7B-V1.1 excels with familiar geometry math problems, while mPLUG-Owl2 demonstrates a robust capability in tackling unseen geometry problems. This is further evidenced by the low performance of WizardMath models on the GeoEval-hard subset, where both models only achieve an accuracy of 6.00%.

heading: 4.3.4 Analysis on the Best Model
text: Table 2 shows that GPT-4, the leading LLMs, records the highest accuracy on the GeoEval-aug subsets, though it only secures a 27.95% accuracy on the GeoEval-2000 subset. A similar pattern of improvement is noted for the GPT-3.5 model, which sees its accuracy jump from 24.71% on the GeoEval-2000 subset to 41.25% on the GeoEvalaug subset. This improvement aligns with the involvement of GPT-3.5 in generating the GeoEvalaug subset, suggesting that the capabilities of GPT3.5 and GPT-4 in addressing geometry math problems significantly benefit from their use in rephrasing geometry question texts.
While WizardMath-70B and WizardMath-7BV1.1, both pre-trained on a mathematical corpus, demonstrate superior performance on the GeoEval2000 subset, they show a marked decline in accuracy across the other subsets, with the most significant decreases observed on the GeoEval-hard subset. This indicates that although pre-training on a mathematical corpus is crucial for solving geometry math problems, it may not be enough.
In contrast to the significant variances in ac-
curacy observed among LLMs across different subsets, the top-performing multi-modal model, mPLUG-Owl2, maintains relatively stable accuracies with scores of 37.76% on the GeoEval-2000, 35.33% on the GeoEval-backward, and 38.00% on the GeoEval-aug subsets. Additionally, the performance of GPT-4V on the GeoEval-aug subset surpasses its accuracy on the GeoEval-2000 subset, mirroring the trends observed with GPT-4 and GPT3.5, further illustrating the enhanced effectiveness of GPT-series models when engaged in rephrasing the content of geometry questions.

heading: 4.4 Results Across Different Subjects
text: Figure 2 displays the performance of models across various subjects, revealing distinct strengths. The WizardMath-7B model significantly outperforms others in flat geometry problems, such as length and lines. Conversely, in solid geometry problems like cuboids and spheres, GPT-4V surpasses WizardMath-7B, indicating its superior capability in addressing solid geometry questions.

heading: 4.5 Benefit from the Geometric Diagram Descriptions
text: To assess the impact of including geometric diagram descriptions on models’ ability to comprehend geometric diagrams and solve related problems, we selected a sample of 300 questions with geometric diagram descriptions from the GeoEval-
2000 subset. We then evaluated the performance of two models, GPT-4V and WizardMath-7B-V1.1, on these questions, both with and without the use of geometric diagram descriptions, which describe the geometric shapes and relations encapsulated in the diagram. The results in Table 3 indicate that GPT-4V’s accuracy decreases by 5.33% without the diagram descriptions. More significantly, WizardMath-7B’s accuracy falls by 18.73% in the absence of these descriptions. This evidence suggests that supplemental geometric diagram descriptions significantly enhance models’ efficiency in solving geometry math problems, particularly benefiting LLMs.

heading: 4.6 External Constants Required for Solving the Problems
text: In the GeoEval benchmark, certain questions require external constants, such as the value of π, which is not typically included in the problem text. This necessitates models to have pre-existing knowledge to accurately solve these problems. Figure 3 assesses the performance of four models on problems differentiated by the need for external constants, identified through a heuristic approach that classifies problems according to whether their solutions require constants.
Figure 3 shows that the WizardMath-7B-V1.1 model maintains consistent accuracy on the GeoEval-2000 subset, regardless of the requirement for external constants, unlike other models, which perform better on problems without such requirements. This consistency in WizardMath-7BV1.1’s performance is likely due to its pre-training on a math-specific corpus, providing it with the necessary knowledge to resolve geometry math problems effectively. In contrast, models trained
on general corpora may not possess this specialized mathematical knowledge, hindering them from using external constants to solve the problems correctly.

heading: 4.7 Performances According to Different Problem Lengths and Varied Complexities
text: Figure 4 shows how models perform with inputs of different lengths. Performance slightly varies for problems ranging from 80 to 100 characters, but there’s a clear trend of decreasing accuracy as problem length increases. This is expected, as longer questions typically involve more complex geometry math problems, challenging the models more as the length grows. The figure also points out that the WizardMath-7B-V1.1 model is notably more adept at handling longer questions, with GPT4V and GPT-4 showing relatively stable accuracy for increased question lengths. On the other hand, GPT-3.5 and CodeGen2-16B perform less effectively on lengthy questions.
Upon the analysis in Figure 5, similar to the observations made in Figure 4 regarding input lengths, we delve into the models’ performances as they relate to the complexity of geometry math problems. Figure 5 presents the performance of models across varying levels of problem complexity. It is evident
that as the complexity of geometry problems escalates, the accuracy of the models correspondingly diminishes.

heading: 5 Conclusion
text: In this study, we present GeoEval, a benchmark developed to assess the geometry problem-solving capabilities of large language models (LLMs) and multi-modal models (MMs). GeoEval comprises four distinct subsets, each designed to facilitate a thorough evaluation. Through our assessment of ten cutting-edge LLMs and MMs using the GeoEval benchmark, we underscore the critical role of mathematical corpus pre-training for effective geometry problem resolution. This is exemplified by the WizardMath model’s leading performance on the GeoEval-2000 subset, achieving an accuracy of 55.67%. However, the WizardMath model’s challenges with the GeoEval-hard subset suggest a need for enhanced reasoning skills. Additionally, our analysis reveals that GPT-series models exhibit improved performance on geometry problems they have rephrased, pointing to the potential benefits of self-rephrasing in problem-solving.

heading: 6 Limitations
text: This study, while providing significant insights into the capabilities of large language models (LLMs) and multi-modal models (MMs) in solving geometry problems, has several limitations.
One primary constraint is that our evaluation predominantly focuses on quantitative metrics of accuracy, potentially overlooking qualitative aspects of model reasoning and explanation that are crucial for educational applications. The performance of models on the hard subset also highlights a gap in advanced reasoning abilities, suggesting that current LLMs and MMs, including those pre-trained on mathematical corpora, may still struggle with highly complex or novel problem types. In addition, we conduct experiments focusing on testing the models’ ability to recall and effectively utilize knowledge about mathematical constants. However, a comprehensive evaluation of external knowledge utilization is required, such as theorems and principles, which are beyond just mathematical constants. We plan to explore models’ abilities to leverage diverse forms of external knowledge in future work.
Moreover, this work reveals the effectiveness of rephrased problems by GPT-series models and
suggests a specific interaction effect that may not generalize across all types of geometry problems or other LLMs and MMs, indicating a need for broader research to fully understand the implications of rephrasing on model performance.

heading: Acknowledgments
text: This work has been supported by the National Key Research and Development Program Grant 2020AAA0109700, and the National Natural Science Foundation of China (NSFC) Grant U23B2029.

heading: A Statistic Analysis
text: Table 6 presents a statistical breakdown of the GeoEval benchmark. This benchmark encompasses a total of 5,050 geometry math problems, categorized into four subsets: GeoEval-2000 (2,000 problems), GeoEval-backward (750 problems), GeoEval-aug (2,000 problems), and GeoEval-hard (300 problems). Besides the problem text, each problem in the dataset includes at least one of the following: a geometric diagram, a description of the diagram, or both. The majority of the correct answers are numerical values, with a minority comprising expressions, coordinates, or option letters, primarily in the GeoEval-hard subset.

heading: B Source Datasets
text: Table 7 provides details on the source datasets that contribute to the GeoEval-2000 subset, including information on their content and characteristics. Meanwhile, Figure 6 visualizes the proportional contributions of these source datasets to the GeoEval-2000 subset, showcasing the variety and scope of the geometry problems collected from each source.

heading: C Distributions of Different Geometric Shapes
text: Figure 7 illustrates the varied distribution of geometric shapes within the GeoEval-2000 subset, highlighting the diversity of geometry concepts represented in this collection.

heading: D Backward Question Example
text: Figure 8 is an example from the GeoEval-backward subset.

heading: E Comparison between GeoEval-hard subset and other public datasets
text: To thoroughly assess the models’ abilities in grasping concepts of solid and analytic geometry, the GeoEval-hard subset was created to include a diverse range of visual elements, such as threedimensional views, across a spectrum of topics in solid geometry. The distinctions between the GeoEval-hard subset and other publicly available datasets are detailed in Table 8, demonstrating the unique coverage and complexity of the GeoEvalhard subset in comparison.
F Inspection of GeoEval-hard subset
To ensure the GeoEval-hard dataset’s high quality and accuracy, and prevent the LLMs and MMs from recalling or inferring solutions to problems
with similar structures or templates as those seen during training, we made several modifications to the original problems sourced from the GeoEvalhard-raw subset, where these modifications were made to prevent direct recall or plagiarism from the training data, while still preserving the underlying mathematical reasoning required. Specifically, we have adopted these steps:
1. We changed the variable names and numerical values used in the problem statements.
2. We swapped the original question with the provided conditions.
3. We used the original answer as a new condition in the reformulated problem statement.
Finally, we thoroughly analyze the revised reformulated problem to ensure it can be solved. Specifically, we form a team of six reviewers, each holding at least a Master’s degree, to scrutinize every question. This evaluation process is structured in three phases: individual review, swap review, and candidate review. The primary focus lies on two key standards: the completeness and relevance of the geometric diagrams, and the reasonableness of the answers provided.
In the first phase, "individual review", each reviewer is randomly assigned 50 geometry math problems from the GeoEval-hard dataset. Their task is to assess the geometry math problems based on the standards, marking any that fail to meet these standards. During the "swap review" phase, these sets of 50 geometry math problems are exchanged among reviewers for a second evaluation. To ensure unbiased assessment, we hide the results of the initial review. Here, reviewers again highlight geometry math problems not conforming to the standards. The final phase, "candidate review", involves selecting geometry math problems for the dataset based on the outcomes of the first two phases. Geometry math problems unmarked in
both phases are retained, those marked in both are discarded, and those highlighted in only one phase undergo further examination by the entire review team, with the majority decision determining their inclusion.

heading: G Examples from GeoEval Representing Five Features
text: G.1 Comprehensive Variety
Figure 9 presents sample data from the GeoEval2000 subset, illustrating its diversity in terms of data sources.
G.2 Varied Problems
Figure 10 displays examples of three distinct problem types in the GeoEval benchmark: flat geometry, analytic geometry, and solid geometry.
G.3 Dual Inputs
Figure 9 shows that the GeoEval benchmark comprises geometry math problems that contain both diagrams and textual descriptions, as well as problems that include textual descriptions alone.
G.4 Diverse Challenges
Figure 11 showcases examples from the GeoEval2000, GeoEval-backward, GeoEval-aug, and GeoEval-hard subsets, illustrating the diverse challenges within the GeoEval benchmark.
G.5 Complexity Ratings
Every problem in the GeoEval benchmark is annotated with a complexity rating, indicating the level of skill necessary to solve it, as shown in Figure 12.

heading: H Algorithm for Classifying Geometry Math Problems Complexity
text: Algorithm 1 details our methodology for classifying each geometry math problem into distinct levels of complexity.

heading: I Evaluation Details
text: I.1 Model Hyper-parameters Table 9 presents the complete list of hyperparameters applied to the models throughout the evaluation phase.
I.2 Instruction Prompt Used for Evaluating Models
Before employing instruction prompts to steer model responses, we combine the problem texts, diagram descriptions, and choice lists from an example, as depicted in the "Merge" row of Table 10. Following this combination, as illustrated in the "Instruction" row of Table 10, we incorporate instruction prompts into the merged texts and then forward these to the models to generate responses.

heading: J Reason for Removing InstructBLIP from the Comparison
text: As shown in Figure 13, InstructBLIP’s responses on the GeoEval-2000 subset are typically scalar, lacking any intermediate reasoning steps. This suggests that InstructBLIP may have been exposed to GeoEval-2000 questions during its pre-train phase, leading to the memorization of answers. This is supported by the observed performance decline from GeoEval-2000 to GeoEval-aug, which falls from 52.18% to 35.00%. Additionally, InstructBLIP tends to directly generate option letters (e.g., "A") for the GeoEval-hard subset without any reasoning process, resulting in an improbably high accuracy rate of 70.30% for this subset. Consequently, in our subsequent analysis and discussions, we have chosen to exclude the InstructBLIP model.

heading: K Models Performances across Different Data Sources
text: Table 11 shows model performances on the GeoEval-2000 subset according to the different
original datasets. We can observe that WizardMath models still achieve the best accuracy scores on almost all datasets.
GeoEval-2000 Data Sources Models MATH (Geometry) (%) GeometryQA (%) GeoQA+ (%) PGPS9K (%) UniGeo (%) MathQA (Geometry) (%)
CodeGen2-16B 0.36 0.35 0.44 0.18 0.41 0.25 GPT-3.5 0.35 0.31 0.19 0.27 0.23 0.26 GPT-4 0.58 0.74 0.27 0.28 0.27 0.44
WizardMath-7B-V1.1 0.58 0.53 0.59 0.55 0.54 0.35 WizardMath-70B 0.54 0.58 0.62 0.54 0.57 0.35
llava-7B-V1.5 0.26 0.4 0.12 0.15 0.12 0.19 Qwen-VL 0.29 0.46 0.27 0.22 0.32 0.24 mPLUG-Owl2 0.27 n/a 0.29 0.46 0.27 0.0 InstructBLIP 0.0 n/a 0.59 0.48 0.57 0.0 GPT-4V 0.45 0.61 0.34 0.38 0.45 0.38

heading: Future Work
text: We plan to explore models’ abilities to leverage diverse forms of external knowledge in future work. Moreover, this work reveals the effectiveness of rephrased problems by GPT-series models and
suggests a specific interaction effect that may not generalize across all types of geometry problems or other LLMs and MMs, indicating a need for broader research to fully understand the implications of rephrasing on model performance