MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark (2024)

Abstract: Recent advancements in large language models (LLMs) have showcased significant improvements in mathematics. However, traditional math benchmarks like GSM8k offer a unidimensional perspective, falling short in providing a holistic assessment of the LLMs’ math capabilities. To address this gap, we introduce MathBench, a new benchmark that rigorously assesses the mathematical capabilities of large language models. MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills. The benchmark progresses through five distinct stages, from basic arithmetic to college mathematics, and is structured to evaluate models at various depths of knowledge. Each stage includes theoretical questions and application problems, allowing us to measure a model’s mathematical proficiency and its ability to apply concepts in practical scenarios. MathBench aims to enhance the evaluation of LLMs’ mathematical abilities, providing a nuanced view of their knowledge understanding levels and problem solving skills in a bilingual context. The project is released at https://github. com/open-compass/MathBench.

heading: 1 Introduction
text: Mathematical reasoning and problem-solving represent pivotal facets of human intelligence and have captivated the interest of artificial intelligence (AI) research for decades. The capability of machines to grasp, interpret, and address mathematical challenges not only serves as a benchmark for their cognitive prowess but also fulfills a critical role in their deployment across various sectors.
The advent of modern Large Language Models (LLMs) such as OpenAI’s ChatGPT and GPT-4 (Achiam et al., 2023) has marked a significant milestone, showcasing an unparalleled ability to gener-
† Corresponding authors.
ate text that mirrors human-like discourse and to unravel intricate mathematical conundrums (Liu et al., 2023a).
Despite these advancements, the evaluation of LLMs’ mathematical capabilities remains hampered by some inherent limitations of existing benchmarks (GSM8k (Cobbe et al., 2021), MathQA(Amini et al., 2019), etc.). These resources predominantly offer a singular perspective on problem-solving abilities and lack comprehensive difficulty grading. Math (Hendrycks et al., 2021b) attempted to classify high-school math competition problems into varying levels of complexity based on annotators’ subjective evaluations, offering an incomplete picture of mathematical proficiency. Such datasets, while valuable, fall short in encapsulating the full spectrum of mathematical knowledge and overlook the importance of fundamental theory understanding, which is essential for tackling application problems (Upadhyay and Chang, 2017). Those limitations make it difficult to conduct a comprehensive evaluation of LLMs’ math capability (both theory and application) across different levels and disciplines and under a multilingual context.
In response to these challenges, we construct
1 6884
MathBench, a novel and comprehensive multilingual benchmark meticulously created to evaluate the mathematical capabilities of LLMs across a diverse range of difficulties, from basic arithmetic to challenging college-level mathematics. MathBench sets itself apart with a unique five-stage taxonomy, mapped to the educational trajectory from primary school through to college. This mechanism is designed to assess LLMs’ mathematical understanding in breadth and depth. The benchmark incorporates carefully curated questions that cover basic theory knowledge and practical applications. This dual focus enables MathBench to probe and interpret the models’ capabilities from a foundational standpoint. Additionally, MathBench supports bilingual evaluation in both Chinese and English, which facilitates a more nuanced and comprehensive assessment of LLMs’ math capabilities, offering a realistic reflection of the global landscape of mathematical knowledge.
In this paper, we detail the methodology behind the creation of MathBench, including the hierarchical knowledge system that underpins the dataset, the data collection process, and the criteria for question selection. We hope that MathBench can serve as a valuable resource for researchers and developers seeking to advance the mathematical abilities of LLMs and to understand the limitations of existing models in solving diverse and complex mathematical problems.
MathBench features the following contributions:
• We introduce MathBench, a comprehensive
dataset that features a five-level difficulty mechanism with a hierarchical knowledge system. • MathBench includes a wide variety of question types, from fundamental mathematical concepts to practical application in real-world scenarios. • We conduct extensive experiments on MathBench across different models to identify bottlenecks in current LLMs. The provided discussion and analysis are expected to offer new avenues for improving their mathematical capabilities.

heading: 2 Methodology
text: MathBench features a well-crafted difficulty hierarchy and an emphasis on evaluating the theoretical knowledge understanding of LLMs. Sec. 2.1 presents the tiered levels and the corresponding knowledge foundations, explaining the ability taxonomy and design rationale. Sec. 2.2 details the collection process and statistics of MathBench.

heading: 2.1 The Hierarchical Knowledge System
text: In MathBench, we define a knowledge framework with five main stages and three levels in order to obtain fine-grained evaluation results. Among five stages, four stages are mapped to the four main education stages: Primary, Middle, High, and College, while the other stage is named Arithmetic, serving as the foundation of the remaining four stages.1 Each Stage in MathBench is associated with two fine-grained knowledge levels: Subject
1The ‘Arithmetic’ stage evaluates the ability to perform four basic math operations: add, subtract, multiply, divide.
2
Area and Topic, accordingly. As shown in Figure 2, we extend MathBench from the basic stages to a comprehensive range of mathematical concepts and problem-solving skills. Such taxonomy is designed to capture the depth and breadth of mathematical knowledge, from foundational arithmetic to complex, abstract college-level concepts.
Subject Areas include major mathematical disciplines such as Algebra, Geometry, Trigonometry, Calculus, Statistics, Probability, etc.. This categorization allows for a wide range of questions, facilitating an organized approach to covering the diverse areas of mathematics. Within each subject area, we further refine the classification into specific Topics. For example, under Algebra, topics might include Linear Equations, Quadratic Equations, Polynomials, and Functions. The Topic-level granularity ensures that the dataset can provide detailed insights into a model’s understanding and proficiency in specific areas of mathematics.
In MathBench, each question is tagged with metadata indicating its stage (Primary, Middle, High, College, or Arithmetic), subject area, and topic. Such tags enable a fine-grained analysis of models’ performance across different areas of mathematics and allow researchers to identify specific strengths and weaknesses in mathematical understanding.
Moreover, the inclusion of the Arithmetic stage emphasizes the importance of mastering basic math operations, which is the foundation of all subsequent mathematical learning and problem-solving.

heading: 2.2 Data Collection and Statistics
text: With the pre-defined knowledge framework, we primarily collect questions from two perspectives: (a). theoretical knowledge questions, to test the model’s grasp of basic formulas, theories, and their corollaries, which are the foundation for solving mathematical problems; (b). practical application questions, which often require a good understanding of the fundamental theories, reflecting the ability to apply these theories in practice.
Question Format Definition. During the evaluation, some models struggle with open-ended questions and fail to follow instructions and provide plain and concise answers. Therefore, we reformulate questions that could have complex answers2 into the multiple-choice format, typically with four options. During collection and annotation, we ensure the uniqueness of the correct answer and the high confusion-level of distractive options.
Theoretical Knowledge Questions. For theoretical knowledge questions, we collect the definition and detailed corollaries of knowledge points topic by topic from the math textbooks and the Internet. We then transform them to multi-choice questions with high-quality annotations.
Practical Application Questions. On selecting the practical application questions, we primarily consider the following aspects: 1. The question needs to match the corresponding education level; 2. The questions should comprehensively cover the previously defined knowledge taxonomy; 3. The questions should be well-formulated so that LLMs can answer them properly. We primarily focus on stage-based educational exams or competitions. Those questions are comprehensive and representative, offering a certain degree of difficulty gradient, such as ZhongKao, GaoKao in Chinese Math and AMC, SAT in English math. Additionally, we incorporate open-source questions to enhance the diversity and breadth of the questions. We list the sources of questions in MathBench in Table 1.
Quality Screening. To enhance the quality of the MathBench dataset, we implement a semiautomated question filtering process to mitigate issues such as intrinsic question errors and alignment with educational stages utilizing GPT-4, details presented in Appendix A.3.
2All theoretical knowledge questions and practical application questions from middle school to college level
3
Dataset Summary. We curate 3709 questions for the final MathBench, including both Chinese and English languages across five stages with threelevel knowledge taxonomy. This compendium is divided into two distinct sections: MathBench-T, which consists of 2,209 theoretical questions, and MathBench-A, comprising 1,500 questions focused on practical applications. Each question has been subjected to a rigorous semi-automated vetting process. Detailed statistics can be found in the Appendix A.1.

heading: 3 Experiments and Analysis

heading: 3.1 Configuration
text: Evaluation Protocols. We employ CircularEval (CE) (Liu et al., 2023b) and Perplexity (PPL) as our principal evaluation methodology for Chat and Base models respectively. CE systematically assesses an N -option multi-choice question by evaluating it N times, each time permuting the order of the options.
To maintain consistency in evaluations, we standardized the maximum output length to 2048 tokens and employed a greedy decoding strategy for all Large Language Models (LLMs). For openended questions, we utilized a few-shot CoT setting, whereas for multiple-choice questions on Chat models, we implemented a zero-shot CoT approach. In the case of Base models during PPL evaluation, a few-shot setting was adopted. We used OpenCompass (Contributors, 2023) as the evaluation framework for our assessments.
Evaluated Models. Our evaluation encompasses both closed-source commercial LLMs and open-source LLMs, covering more than 20 models. Based on MathBench, we deliver a thorough evaluation of the capabilities of current LLMs. We list all evaluated LLMs below: • Closed-source models: GPT-3.5 and GPT-43from Openai, Qwen-Max4, DeepSeek-V2-API5, GLM46 and Anthropic Claude-3-Opus7. 3GPT-4 version: gpt-4-0125-preview and GPT-4o(GPT-4o-2024-05-13); GPT-3.5 version: gpt-3.5-turbo-0125
4https://help.aliyun.com/zh/dashscope/ create-a-chat-foundation-model?spm= a2c4g.11186623.0.0.581c64d16b7Azw
5https://platform.deepseek.com/ api-docs
6https://open.bigmodel.cn/dev/howuse/ glm4
7https://www.anthropic.com/news/ claude-3-family
• OpenSource LLMs: We evaluate a wide spectrum of LLMs, including Llama3(Touvron et al., 2023), Qwen (Bai et al., 2023), InternLM2 (Team, 2023a), Yi 8, Baichuan2 (Yang et al., 2023), DeepSeek(DeepSeek-AI et al., 2024), Mixtral(Jiang et al., 2024)and ChatGLM3 (Zeng et al., 2022). • OpenSource Math LLMs: Llemma(Azerbayev et al., 2023), MetaMath-llemma(Yu et al., 2023), DeepSeek-Math(Shao et al., 2024), MAmmoTH(Yue et al., 2023) and InternLM2-Math(Ying et al., 2024).

heading: 3.2 Main Results
text: We showcase the principal outcomes of MathBench in Table 2, detailing the application-oriented aspects in (MathBench-A), and the theoretical components in (MathBench-T).

heading: 3.2.1 MathBench-A
text: Among all models evaluated in the MathBench application, GPT-4o (GPT-4o-2024-05-13) achieves the highest overall average score, particularly excelling in the more challenging Middle, High, and College stages. Following GPT-4o, Claude3-Opus and DeepSeek-V2-API outperform in basic arithmetic operations, specifically in the Arithmetic and Primary stages respectively. For opensource LLMs, Qwen1.5-110B-Chat stands out as the best performer, distinguishing itself as the leading player among all open-source models. Additionally, DeepSeek-Math-7B-RL, an LLM designed for mathematical tasks, secures its position as the top open-source model in mathematics, despite its relatively small parameter size.
Among open-source chat models, performances across models with ∼7B, ∼20B, and ∼70B parameter size reveal distinct capabilities: ∼7B Chat Models. InternLM2-Chat-7B and Llama-3-8B-Instruct emerges as the superior model at the ∼7B scale and outperforms other 7B Chat models across all stages. It’s noteworthy that, as the difficulty of problems increases, the gap between Llama-3-8B-Instruct and other models also grows. For instance, on the five stages from Arithmetic to College Math, It outperforms ChatGLM36B by 43.95%, 73.17%, 82.48%, 258.49%, and 723.53%, respectively. The trend indicates that as the difficulty escalates, the performance disparity between models significantly increases since higher-stage math problems often involve more
8https://github.com/01-ai/Yi
4
complex concepts and problem-solving strategies, imposing greater demands on the models’ comprehension and reasoning abilities. All ∼7B models struggle with advanced mathematical problems, indicating a challenge in smoothly resolving complex questions for small-scale LLMs.
∼20B Chat Models. InternLM2-Chat-20B performs the best at the ∼20B scale, followed by Qwen-14B-Chat. Though Yi-34B-Chat has a much larger parameter size, it lags behind other ∼20B models. Similar to ∼7B models, models around ∼20B also struggle with more complex mathematical problems at the High School and College stage.
∼70B Chat Models and Math Models. In the realm of large-scale open-source language models, a significant performance disparity is evident when comparing models of varying sizes. Notably, the Qwen1.5-110B-Chat model demonstrates exceptional proficiency in addressing mathematical application problems. Its performance not only surpasses that of other open-source chat-oriented models but also eclipses the capabilities of numerous specialized mathematical models. Remarkably, it exhibits comparable effectiveness to closed-source
models, such as GPT-4-0125-Preview, in solving application problems (58.4 vs 58.8).
Focusing on models dedicated to mathematical tasks, the DeepSeek-Math-7B-RL model stands out for its adeptness in tackling application-based questions across a spectrum of stages, encompassing basic Primary, High and College math. Remarkably, it outstrips not only its counterparts, but also the substantially larger DeepSeek-67B-Chat model, by a margin of 24.8%. This is particularly noteworthy given that the DeepSeek-Math-7B-RL achieves this superior performance with a model size nearly one-tenth that of the DeepSeek-67B-Chat, underscoring the efficiency and targeted capability of the former in mathematical problem-solving domains.

heading: 3.2.2 MathBench-T
text: In the theoretical segment of MathBench, designated as MathBench-T, GPT-4o consistently achieved balanced and exceptional results across nearly all theoretical stages. Although Qwen-1.5110B-Chat exhibited slightly superior performance in the Primary stage, GPT-4o attained an average theoretical score of 87.0. This score was the high-
5
est among all tested Closed-source Models and Open-source Chat Models. When combined with an application score of 70.9 in MathBench-A, these results indicate that GPT-4o demonstrates both balanced and superior performance in theory and application on MathBench. This underscores GPT4o’s strong grasp of theoretical knowledge and its proficiency in applying such knowledge effectively.
Among other models except GPT-4o, the Qwen series models stood out, with Qwen-Max-0428 and Qwen1.5-110B-Chat ranking just behind GPT4o. Notably, in the theoretical stage of Primary, Qwen1.5-110B-Chat scored the highest among all models with an 93.4 CE score. However, GPT4o’s advantage lies in higher educational stages or perhaps more advanced theoretical stages. For example, in the college-level theoretical knowledge stage, GPT-4o achieved a CE score of 84.1, which is 16.9 points higher than the best open-source math model, Deepseek-Math-7B-RL.
Similar to MathBench-A, InternLM2-Chat-7B demonstrated robust theoretical capabilities at the common 7B stage models. Despite achieving similar effectiveness to Llama-3-8B-Instruct, InternLM2-Chat-7B exhibited a significantly larger lead in the theoretical stage, surpassing the Qwen7B-Chat model by 31.3%. Within the domain of mathematical chat models, Deepseek-Math-7B-RL continued to outperform numerous mathematical models, achieving superior results in both theory and application. Notably, it even surpassed Llama3-70B-Instruct in the theoretical domain.
Overall, in the tests conducted on MathBench, there was not a significant rank change between models in terms of theoretical and application capabilities. That is, models that ranked highly in application capabilities also tended to perform well in theoretical tests, and vice versa.

heading: 3.3 Detailed Analysis
text: With MathBench, we can easily assess the model’s mathematical capabilities at different granularities including education stage, language, subject area, or even specific topics with questions on both theories and applications. Below, we will delve deeper into the evaluation results and discuss about the following questions:

heading: How Models’ Scores on Application Problems
text: Vary Across Stages? Figure 3 presents the average performance of all aforementioned models on application questions in MathBench. Most models
perform reasonably well on Arithmetic and Primary math problems. However, their effectiveness drastically declines when it comes to the Middle stage or above. Such phenomenon suggests that existing models are good at tasks that can be solved through direct computation, pattern recognition, or memorizing basic concepts. However, they showcase inferior performance when solving more complex math problems.
Is There A Gap between Theory Understanding and Application Capabilities? Theories serve as the foundation for addressing the majority of application problems. As illustrated in Figure 5, we present the trend of LLMs in terms of theoretical and application scores across different stages. In the Primary stage, the two scores are highly correlated for most LLMs, with only a few exceptions. Among top-ranked models, Qwen-72BChat demonstrates the best theoretical ability, while
6
Claude-3-Opus demonstrates superior application ability. When it comes to more advanced stages, models require better computational and reasoning capabilities to achieve good application scores. GPT-4 leads in the application track across all stages, while the gap is larger in more advanced stages. For example, comparing to Qwen-72BChat, the difference in theoretical and application scores (Dt, Da) increases from (1.4, 8.7) in the Middle stage to (6.0, 11.7) in the High stage, and finally to (13.5, 23.0) in the College stage. Moreover, from the Middle stage onwards, there is a general trend of decline in both theoretical and application abilities of models. Compared to theoretical scores, the decline in application scores is more serious.
Which Model Performs Better under the Bilingual Scenario? Figure 4 demonstrates the bilingual capabilities of various LLMs on MathBench, indicating the importance of linguistic versatility in mathematical tasks that demand an understanding of nuances in language and math concepts across different languages. Among all LLMs, GPT-4 leads with the highest bilingual score of 67.1, showing a balanced performance between Chinese (65.2) and English (69.0). This demonstrates GPT-4’s advanced bilingual processing abilities. Other models including Qwen-72B-Chat and DeepSeek-Math7B-RL also exhibit significant bilingual capabilities. It’s also noteworthy that among all LLMs evaluated, most of them feature a much larger performance gap between Chinese and English, compared to GPT-4. The detailed result of bilingual test of MathBench can be found in Appendix B.4.

heading: Enhancing Model Proficiency in Fundamental
text: Theories: Initial Explorations In an effort to augment the model’s grasp and application of theoretical concepts in problem-solving contexts, we embarked on exploratory initiatives, focusing primarily on two methodologies: Chain of Thought (CoT) and Knowledge Infusion.
We selectively sampled 200 questions from MathBench, deliberately skewed towards theoretical reasoning and application (with a distribution of 40% application-oriented and 60% theory-centric questions), to perform ancillary experiments on the Qwen-72B-Chat model. The outcomes, delineated in Table 3, elucidate the accuracy achieved through different strategic approaches.
• Straight: Immediate response without pre-
ceding CoT.
• CoT: Response derived post-CoT, serving as MathBench’s standard evaluative criterion.
• Straight-Knowledge: Immediate response, preconditioned by the integration of relevant knowledge points prior to posing the question.
• CoT-Knowledge: Response post-CoT, facilitated by the preliminary inclusion of pertinent knowledge points.

heading: Strategy Accuracy (%)
text: Knowledge points were meticulously curated from academic textbooks and instructional resources. The experimental data suggests a progressive enhancement in efficacy: Straight < CoT < Straight-Knowledge < CoT-Knowledge. This progression evidences the significant impact of both CoT and knowledge point infusion on augmenting model performance for questions heavily reliant on theoretical reasoning or practical application, with their combined utilization yielding the most favorable outcomes.

heading: 4 Discussion

heading: 4.1 Effect of Model Size on Math Capabilities
text: We found that for models of different sizes within the same series, most of them conform to the Scaling Law (Kaplan et al., 2020) on MathBench. For example, Qwen series, MammoTH series, and Yi series have shown steady improvement in their MathBench scores as the parameter size increases, as shown in Figure 6. However, it doesn’t mean that models with small parameter sizes can not achieve good math performance. For instance, DeepSeekMath-7B demonstrates outstanding performance on MathBench and outperforms models with 10x parameters, including DeepSeek-72B and a larger math model MammoTH-70B.

heading: 4.2 Error Analysis
text: In our study, we conduct a comprehensive error analysis on a set of 80 theoretical and 100 application questions random selected from every
7
stages, for models selected across different scales, as illustrated in Figure 7. The error categories are uniformly observed across all evaluated models, indicating common challenges that transcend specific parameter scales. Our selection of models includes GPT-3.5, GPT-4, InternLM2-Chat7B, Qwen-14B-Chat, Qwen-72B-Chat, DeepseekMath-7B-RL and MammoTH-70B. Detailed cases for error analysis can be found in Appendix C.2. Insufficiency of knowledge. For theoretical questions, 78% of model errors are due to misconceptions about mathematical concepts, which notably emerged as a significant concern in several models. Such errors accounted for 49.5% of all mistakes, underscoring a general challenge in grasping fundamental knowledge and terminology. Deficiencies in reasoning. Furthermore, models exhibited shortcomings in logical reasoning, with 33.4% of errors attributed to logically consistent
but flawed reasoning processes. Moreover, errors such as reasoning that deviated from the intended query accounting for 9.6%, underscored the models’ limitations in understanding user intentions and providing pertinent responses. Response length limit. Though statistically not the primary error mode (4.0%), responses that exceeded the token limit shed light on the challenge of reasoning complex tasks within limited length and adhering to given instructions. Other cases. Occasionally, models will generate responses devoid of an explicit reasoning process, obstructing additional scrutiny. Moreover, models endowed with enhanced reasoning capabilities exhibit a greater capacity for critical thinking regarding the options presented, thereby offering alternative answers that transcend the limitations of predetermined choices.

heading: 5 Related Work
text: Solving math word problems through automated methods has been a long-standing concern for re-
8
searchers. This section summarizes seminal studies and delineates key evaluation datasets proposed for assessing mathematical problem-solving approaches, tracing the field’s evolution from its origins to the present day.
Preliminary Mathematical Datasets Previous works proposed datasets such as Alg514 (Kushman et al., 2014), SingleEq (Koncel-Kedziorski et al., 2015), and DRAW-1K (Upadhyay and Chang, 2017) are primarily concentrated on elementary linear algebraic problems. Similarly, datasets like AddSub (Hosseini et al., 2014) and SingleOp (Roy et al., 2015) MultiArith (Roy and Roth, 2016) are exclusively dedicated to fundamental arithmetic operations: addition, subtraction, multiplication, and division. These datasets are very limited both in the form and content of their assessments, focusing solely on a specific small part of basic mathematics.

heading: Benchmarks tailored to specific educational
text: tiers Some benchmarks are designed based on educational levels. Math23k (Wang et al., 2017) collects a corpus of real math word problems for elementary school students. While ASDiv (Miao et al., 2021) expands the textual patterns to encompass most problem types found in elementary mathematics. GSM8K (Cobbe et al., 2021) presents a high-quality collection of elementary mathematical word problems that, on average, require multiple steps to solve and provide solutions in natural language annotations. These datasets mostly focus on elementary mathematics and seldom examine college-level knowledge.
Enriching the diversity of mathematical problem types within benchmarks MathQA (Amini et al., 2019) seeks to categorize problems from AQuA (Ling et al., 2017) into different mathematical domains based on the frequency of mathematical terminology used. Mathematics Dataset (Saxton et al., 2019) expands the subject of mathematics and this dataset covers a broader spectrum of mathematics, including arithmetic, algebra, probability, and calculus. MATH (Hendrycks et al., 2021b) features a higher level of complexity, comprising problems ranging from arithmetic to calculus, and aims at testing models’ capabilities in understanding and solving complex mathematical challenges. While these efforts have enhanced the diversity of the data in certain aspects, they are lacking in diversity in other aspects such as question formulation (Saxton et al., 2019).
Integrating mathematical problems with domain knowledge NumGLUE (Mishra et al., 2022) not only assesses the ability of models to solve mathematical problems given direct computational expressions, but it also designs multiple tasks to comprehensively evaluate the models’ abilities to use other reasoning skills, such as common sense and reading comprehension. Lila (Mishra et al., 2023) is developed through the extension of 20 datasets that cover a broad range of mathematical topics. This dataset exhibits varying degrees of linguistic complexity and features diverse question formats as well as background knowledge requirements. These works inspire us to design more diversified testing scenarios.

heading: 6 Conclusion
text: In summary, MathBench adopts structured approaches to categorize questions by stage and knowledge level. It aims to provide a comprehensive evaluation of LLMs’ mathematical proficiency. By covering a wide range of subject areas and topics across educational stages, MathBench offers a unique resource for researchers and educators interested in advancing the field of mathematical learning and assessment.

heading: 7 Limitations
text: We have developed a comprehensive mathematical evaluation benchmark, MathBench, which includes a detailed knowledge framework and multidimensional, fine-grained mathematical questions. Despite its strengths, the benchmark currently has several limitations, which are summarized as follows:
Data Source: To enhance diversity, some questions were sourced from open-source datasets(∼ 19%). However, these open-source questions may be subject to data contamination, which could compromise the assurance that models have not been exposed to these questions before. In future iterations, we plan to automate the construction of questions across various stages to more effectively test the models’ genuine mathematical capabilities.
Lack of Detailed Reasoning Paths: Given the diversity of questions and time constraints, MathBench currently does not provide detailed reasoning paths for each question. This limitation makes it challenging to unlock the full potential of the questions. Moving forward, we aim to investigate semi-automated methods to offer both natural lan-
9
guage and code-based reasoning approaches for each question, thereby maximizing the value of MathBench’s questions.

heading: 8 Ethical Considerations
text: For our benchmarks, we relied on reference materials and closed-source models that are accessible to the public, thereby avoiding any potential harm to individuals or groups. The data produced by the LLMs underwent a meticulous human selection and processing phase to ensure the protection of privacy and confidentiality. We did not use any personally identifiable information, and all data were anonymized prior to analysis. Additionally, we employed ChatGPT and Grammarly to refine our manuscript’s language.

heading: 9 Acknowledgements
text: We thank the OpenCompass team for their diligent efforts in data collection and validation, which ensured the quality and completion of this work. We extend our special thanks to Jiaye Ge for providing comprehensive support on this project, orchestrating the data collection process, and ensuring the timely delivery of the dataset. We also thank OpenDataLab for providing data sources for part of this work. Our gratitude extends to Mo Li for his suggestions on the modifications of some charts in the paper, as well as Kuikun Liu for the Lagent support. Chuyu Zhang and Jingming Zhuo provided valuable advice on the writing of the paper.
This work was supported by National Key R&D Program of China 2022ZD0161600 and Shanghai Postdoctoral Excellence Program 2022235.

heading: A MathBench Statistics

heading: A.1 Dataset Statistics
text: The detailed statistics of MathBench questions, Table 4 for the data distribution of theoretical and application questions across various stages, Table 5 for fine-grained knowledge levels.

heading: Stage Type English Chinese Total

heading: A.2 Data collection details
text: For self-collected questions in MathBench, We primarily collect through the following methods:
For the Primary stage GSM-X-CN and GSMX-Plus datasets, we semi-automatically generate new questions using GPT-4. Specifically, the construction of the GSM-X-CN Chinese question set involved two steps:
We first translate English questions in GSM8k test set into Chinese using GPT-4, resulting in a Chinese version of GSM8k. We then replace the entity names under the Chinese context while ensuring that the questions’ meanings remained unchanged. This process creates elementary-level questions suitable for Chinese Q&A.
For the GSM-X-Plus dataset, which is in English, we generate new questions by first generating solution code for the original test set questions. We then replace some numeric parameters (taken from the original questions) in the question with multiples of the variable k. By executing the modified solution code, we obtain the new answers. In MathBench, we set k ∈ (2, 10).
For exams such as AMC, GaoKao, ZhongKao, etc., we initially collect relevant questions from the Internet. These questions are then underwent processing and annotation by domain experts. Questions for primary and secondary education levels are handled and annotated by undergraduate students, while questions for university-level exams were processed and annotated by graduate students specializing in mathematics or computer science. The description of knowledge-based questions is provided in Sec. 2.2.
In addition to the self-collected datasets described above, we also incorporate questions from the following open-source datasets: CEVAL (Huang et al., 2023), MMLU (Hendrycks et al., 2021a), Arithmetic-HG, Math401 (Yuan et al., 2023) and SciBench (Wang et al., 2024). We download the data from OpenDataLab (He et al., 2022). All open-source datasets we used are MIT License.

heading: A.3 Quality Screening
text: Given the wide variety of sources and types of questions, we notice that the following issues may affect the benchmark quality: 1. Intrinsic errors in the questions, such as being unanswerable or having multiple correct answers. 2. Questions of low evaluation value, too difficult or too trivial for the intended education stage.
All of the above situations can easily lead to unstable model responses and increased probability of incorrect answers in CircularEval. To address these issues, we employ a novel semi-automated question filtering approach for quality screening.
Specifically, we use GPT-4 to perform Circular Evaluation (CE) on all questions. We then select questions that GPT-4 answered incorrectly 0, 1, or
12
2 times out of four attempts (CE − 0, CE − 1, CE − 2) for manual review to ensure the overall question quality.

heading: B Detailed Experimental Results

heading: B.1 Overall Results
text: The overall experimental results for chat models are shown in Table 6, and Table 7 for base models. We report the average score of theoretical and application questions for stages except Arithmetic, which is predominantly indicative of pure computational prowess.

heading: B.2 Evaluation of Base Models
text: The results for the Base models are presented in Table 8. Consistency in performance is observed between the Base models and their Chat model counterparts, with InternLM2-7B emerging as the optimal model in the 7B parameter range. Qwen14B and Qwen-72B demonstrate superior performance within their respective parameter classes on the MathBench benchmark. For mathematical tasks, Deepseek-Math-7B-Base’s results align closely with those seen in the Chat model evaluations, indicating a significant correlation between the efficacy of Base models and their corresponding Chat models, which leads to similar performance trends across models within the same category.
Notably, ChatGLM3-6B-Base secures the second-highest ranking in the 7B base model evaluation, outperforming several other models, includ-
ing Qwen-7B and Mistral-7B-v0.1. However, this performance is not mirrored in its Chat model variant, ChatGLM3-6B, which is surpassed by Qwen-7B-Chat by 95.2% on MathBench-A and by 104.7% on MathBench-T. This discrepancy in performance between the Chat and Base versions of the model may be attributed to the different finetuning strategies applied during the subsequent tuning phase, which could explain the observed gap in performance.

heading: B.3 Results with Accuracy
text: The detailed accuracy results are presented in Table 9.

heading: B.4 Bilingual
text: The corresponding results is presented in Table 10.

heading: B.5 Detailed Model Performance for Each Topic
text: In Figure 8 indicating average knowledge point performance, it is evident that topics associated with fundamental mathematical skills—such as ’Unit Conversion,’ ’Four Operations,’ and ’Basic Concepts of Equations’—register higher average scores. This suggests that the majority of models exhibit a proficient command of simple and elementary mathematical questions.
Conversely, topics demanding abstract reasoning and intricate computations, like ’Double Integrals,’ ’Mathematical Logic,’ and ’Set Theory,’
13
show lower average scores. Addressing the mathematical queries in these topics may require bespoke model analysis and optimization. It is crucial to pinpoint the source of inaccuracies within these topics, whether it be due to a deficit in reasoning ability or an unstable grasp of the relevant foundational theoretical concepts.

heading: B.6 The Gap between Circular and Accuracy Evaluation
text: A comparison between Circular Evaluation (CE) scores and Accuracy (ACC) scores is illustrated in Figure 9. As model performance improves, the
discrepancy between CE and ACC scores becomes increasingly narrow, suggesting that more powerful models tend to provide more robust and stable answers in mathematical question answering.

heading: B.7 How Models Perform with Code Agent on MathBench
text: We utilize the external code interpreter and follow the ReAct (Yao et al., 2023) protocol in Lagent (Team, 2023b) to evaluate LLMs’ ability in solving mathematical problems of MathBench. The results, as depicted in Figure 10, show the comparison of performance with and without the
14

heading: Models Arith Primary Middle High College Avg.
text: Code Agent on the Theory and Application sections of MathBench. Overall, the inclusion of the Code Agent significantly enhances performance in the Application section, especially in Arithmetic, where it boosts the performance of InternLM27B-Chat by about 64% (from 53.0 to 87.3). This demonstrates that the addition of the Code Agent can substantially improve the model’s basic numer-
15
ical calculation capabilities. However, for more complex problems, such as those in the College level Application section, the Code Agent does not notably improve model capabilities and even slightly degrades performance. For theoretical problems, the Code Agent does not significantly enhance InternLM2-7B-Chat’s performance across various stages on MathBench. This suggests that mathematical theoretical ability, as a crucial foundational skill for models, requires more than just external tools. Instead, it necessitates exploring more effective ways to enhance large language models’ understanding and application of mathematical con-
cepts.

heading: B.8 Reasoning Path
text: Analyzing the reasoning paths of various models across multiple difficulty levels reveals significant performance disparities. We set a brief discussion below and provide more detailed cases for reasoning path analysis in Appendix C.3.

heading: Performance across diverse difficulties. In
text: straightforward scenarios, models swiftly solve the problems with direct reasoning and yield logical outcomes. Yet, complex issues, marked by dense symbols, vast knowledge, and intricate links, neces-
16
sitate broader knowledge navigation, accentuating divergences in deductive strategies.
Reasoning paths of chat models with different parameter sizes. Small-scale chat models strive for logical coherence in mathematics, yet may make mistakes due to knowledge deficiencies, particularly in symbol interpretation and relational understanding. In contrast, large-scale models feature expansive knowledge and nuanced insights, which enhance symbol processing and minimizing knowledge gaps. However, even with substantial parameters, challenges in efficient knowledge management persist, occasionally leading to irrelevant diversions and diminished reasoning efficacy.
Reasoning paths of math models. Specialized math models, despite the smaller parameter sizes, exhibit superior mathematical comprehension and systematic logical reasoning. They excel in applying mathematical knowledge and notation to reason through complex problems.
Superlative deductive navigation of Closedsource models. GPT-4 stands out for its effective reasoning and deep problem comprehension. It engages in logical, coherent, and succinct discussions, adeptly navigate complex reasoning paths, and
manage mathematical symbols effectively. GPT-4 distinctively recognizes problem statement ambiguities, showcasing a detailed and nuanced reasoning process.

heading: C Extra Analysis

heading: C.1 Prompts Demonstration
text: Please refer to the respective prompt block for a detailed demonstration.

heading: C.1.1 English Open-ended test
text: The corresponding prompt is presented in Figure 21.

heading: C.1.2 Chinese Open-ended test
text: The corresponding prompt is presented in Figure 22.
C.1.3 English single choice with reasoning The corresponding prompt is presented in Figure 23.
C.1.4 Chinese single choice with reasoning The corresponding prompt is presented in Figure 24.
17

heading: C.2 Error Types Demonstration
text: Please refer to the respective cases for a detailed error types demonstration.

heading: C.2.1 Misunderstandings of concepts
text: The corresponding case is presented in Figure 12.

heading: C.2.2 Flawed reasoning
text: The corresponding case is presented in Figure 13.
C.2.3 Misaligned with the question The corresponding case is presented in Figure 14.

heading: C.2.4 Exceed max out length
text: The corresponding case is presented in Figure 15.

heading: C.2.5 Responses constrained to Options
text: The corresponding case is presented in Figure 16.

heading: C.2.6 Non-adherence to the prompt
text: The corresponding case is presented in Figure 17.

heading: C.3 Reasoning Paths Demonstration
text: C.3.1 Small-scale chat model The corresponding case is presented in Figure 18.
C.3.2 Large-scale chat model The corresponding case is presented in Figure 19.

heading: C.3.3 Math model
text: The corresponding case is presented in Figure 20.
18
19

heading: Incorrect Model Answer

heading: Incorrect Analysis:
text: 20

heading: Prompt:

heading: Incorrect Model Answer:

heading: Error Analysis:
text: 21

heading: Prompt:

heading: Incorrect Model Answer:

heading: Error Analysis:
text: 22

heading: Prompt:

heading: Incorrect Model Answer:

heading: Error Analysis:
text: 23

heading: Prompt:

heading: Incorrect Model Answer:

heading: Error Analysis:
text: 24

heading: Prompt: Question:

heading: Incorrect Model Answer:

heading: Error Analysis:
text: 25

heading: Prompt:

heading: Incorrect Model Answer:

heading: Error Analysis:
text: 26

heading: Prompt:

heading: Model Answer:
text: 27

heading: Prompt:

heading: Model Answer:
text: 28

heading: Prompt:

heading: Model Answer:
text: 29
30
31
32

heading: Future Work
text: In future iterations, we plan to automate the construction of questions across various stages to more effectively test the models’ genuine mathematical capabilities. Lack of Detailed Reasoning Paths: Given the diversity of questions and time constraints, MathBench currently does not provide detailed reasoning paths for each question. This limitation makes it challenging to unlock the full potential of the questions. Moving forward, we aim to investigate semi-automated methods to offer both natural lan-
9
guage and code-based reasoning approaches for each question, thereby maximizing the value of MathBench’s questions