MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data

## Abstract

Abstract Large language models (LLMs) have significantly advanced natural language understanding and demonstrated strong problem-solving abilities. Despite these successes, most LLMs still struggle with solving mathematical problems due to the intricate reasoning required. This paper investigates the mathematical problem-solving capabilities of LLMs using the newly developed “MathOdyssey” dataset. The dataset includes diverse mathematical problems at high school and university levels, created by experts from notable institutions to rigorously test LLMs in advanced problem-solving scenarios and cover a wider range of subject areas. By providing the MathOdyssey dataset as a resource to the AI community, we aim to contribute to the understanding and improvement of AI capabilities in complex mathematical problem-solving.
We conduct benchmarking on open-source models, such as Llama-3 and DBRX-Instruct, and closed-source models from the GPT series and Gemini models. Our results indicate that while LLMs perform well on routine and moderately difficult tasks, they face significant challenges with Olympiad-level problems and complex university-level questions. Our analysis shows a narrowing performance gap between open-source and closed-source models, yet substantial challenges remain, particularly with the most demanding problems. This study highlights the ongoing need for research to enhance the mathematical reasoning of LLMs.
The dataset, results, and code are publicly available. 1 1 1 https://mathodyssey.github.io/

## 1 Introduction

Large language models (LLMs) have demonstrated exceptional proficiency in mastering human language and handling mathematical problems, including typical routine math problems (OpenAI, ; Touvron et al., ; Reid et al., ). In recent years, several benchmarks related to mathematics have been proposed, such as the GSM8K dataset (Cobbe et al., ), the MATH dataset (Hendrycks et al., ) and so on. Recent LLMs and prompting approaches have addressed these problems with notable success (OpenAI, ; Touvron et al., , ). For instance, GPT-4, using advanced prompting techniques (OpenAI, ), has achieved more than a 90% success rate on GSM8K and 80% on MATH. These achievements indicate that LLMs possess remarkable capabilities in mathematical reasoning.

The quest to improve LLMs’ mathematical problem-solving abilities is not just a demonstration of technological advancement but a crucial step toward developing more general and capable artificial intelligence systems. On the one hand, this endeavor requires datasets that accurately measure and challenge the AI’s mathematical reasoning beyond basic problems. Although their performance is high on datasets like GSM8K (Cobbe et al., ), it remains uncertain how well they handle more complex mathematical challenges, such as those found in university-level courses and competitive high school mathematics. Performance may diminish significantly in these areas. This gap highlights the ongoing need for enhanced mathematical reasoning capabilities in AI, a critical area for assessing cognitive abilities akin to human intelligence.
Moreover, a significant obstacle is that many existing datasets might have been included in the training phases of these models, potentially skewing performance metrics. Prominent examples include STEM-Q (Drori et al., ), GSM8K (Cobbe et al., ), and the MATH dataset (Hendrycks et al., ), which may no longer provide a true test of an LLM’s mathematical capabilities.
On the other hand, high-quality, expert-crafted original problems are scarce. For instance, a study by OpenAI (Davis and Aaronson, ) included only 105 such problems in high school and university-level science and math.

**Table 1: MathOdyssey dataset examples. We demonstrate three distinct levels to challenge various aspects of mathematical knowledge: Olympiad-level, High School, and University-level mathematics. Each example consists of three parts: the problem, the answer, and the reasoning. Note that both GPT-4 Turbo and Llama-3-70B are unable to solve the first Olympiad-level example. See Appendix A for the LLMs’ solutions.**
| Olympiad-level<br>Problem: Let $S=\left\{1,2,\cdots 2024\right\}$, if the set of any $n$ pairwise prime numbers in $S$ has at least one prime number, the minimum value of $n$ is                  . |
| --- |
| Answer: $16$. |
| Reasoning: Taking the 15 numbers $1,2^{2},3^{2},...,43^{2}$. They violate the<br>condition. Furthermore, since S does not contain any non-prime numbers with a minimum prime factor of at least 47 (because $47^{2}>2024$). Set 1 aside, there are only 14 types of non-prime numbers in S, classified by its minimum prime factor. Applying the Pigeonhole Principle, we conclude that n = 16. |
| High School<br>Problem: What are the solutions of the quadratic equation $15x^{2}=2x+8$.<br>$\displaystyle\text{ A) }\,\left\{-\frac{4}{3},-\frac{3}{2}\right\}\,\,~{}~{}<br>\text{ B) }\,\left\{-\frac{4}{5},\frac{2}{3}\right\}\,\,~{}~{}\text{ C) }\,<br>\left\{-\frac{3}{2},\frac{4}{5}\right\}\,\,~{}~{}\text{ D) }\,\left\{-\frac{2}<br>{3},\frac{4}{5}\right\}$<br>Answer: $D$ |
| Reasoning:<br>First move all terms to one side: $15x^{2}-2x-8=0$. Then factor into $(5x-4)(3x+2)=0$. Setting $5x-4$ to zero results in a solution of $x=\frac{4}{5}$ and setting $3x+2$ to zero results in a solution of $x=-\frac{2}{3}$. |
| University-level<br>Problem: Find the limit lim_x→1f(2x2+x-3)-f(0)x-1 given $f^{\prime}(1)=2$ and $f^{\prime}(0)=-1$. |
| Answer: $-5$. |
| Reasoning:<br>Let $g(x)=2x^{2}+x-3$. Since $g(1)=0$, the desired limit equals $\lim\limits_{x\to 1}\frac{f(g(x))-f(g(1))}{x-1}$. By the definition of the derivative and the chain rule and noting that $g^{\prime}(1)=5$, we have<br>$\lim\limits_{x\to 1}\frac{f(g(x))-f(g(1))}{x-1}=f^{\prime}(g(1))g^{\prime}(1)=<br>f^{\prime}(0)g^{\prime}(1)=(-1)(5)=-5.$ |

To directly address these challenges, we introduce the “MathOdyssey” dataset, a rigorously curated collection of 387 mathematical problems for evaluating the general mathematical capacities of LLMs. See examples in Table 1. The MathOdyssey dataset is developed by the GAIC Math organization and features a spectrum of questions from Olympiad-level competitions, advanced high school curricula, and university-level mathematics.
Mathematics professionals, including high-school educators, researchers, and university professors, crafted these problems under the invitation of the GAIC Math organization. Their involvement ensures the dataset not only supports advanced AGI research but also fosters necessary interdisciplinary collaboration.

Furthermore, we open-source the MathOdyssey dataset to facilitate its use in evaluating other LLMs. The dataset has not been used for training by LLMs. We explore its utility in benchmarking the advanced mathematical reasoning abilities of LLMs. By ensuring the originality and confidentiality of the questions, we maintain the integrity and fairness of the assessments, providing a reliable tool for advancing research into artificial general intelligence.

Our contributions are as follows:

- •
We introduce a new mathematical challenge that provides different levels of mathematical problems and covers a wider range of subject areas.
- •
We open source the MathOdyssey benchmark dataset, a meticulously curated collection of mathematical problems spanning various domains and levels, complete with natural language solutions. This dataset is specifically designed to probe the reasoning abilities of LLMs, offering a unique tool for assessing AI performance in complex mathematical reasoning. Each question has an objective answer serving as ‘ground-truth’, allowing for objective evaluation on the LLM outputs. In particular, the Open-Answer problems emphasize the importance of detailed reasoning and solution.
- •
We conduct a comprehensive benchmark analysis using our dataset on both open-source and closed-source LLMs. Our findings reveal that while closed-source models currently lead, open-source models are rapidly catching up, highlighting the competitive landscape of LLM capabilities in mathematical problem-solving.

## 2 Related Work

#### Large Language Models for Mathematics.

Applying large language models (LLMs) to mathematical problems has led to significant strides, though solving such problems remains challenging due to the need for highly complex and symbolic multi-step reasoning capabilities. Both GPT-3.5 and GPT-4 (OpenAI, ) have shown promising reasoning abilities for complex mathematical tasks, such as those in the MATH dataset (Hendrycks et al., ). However, the performance of open-source models, like Llama-1 and Llama-2 (Touvron et al., , ), is still far from satisfactory in this domain. To enhance the mathematical problem-solving abilities of LLMs, prompt-based methods have also been developed (Wei et al., ; Wang et al., ; Zhou et al., ). These methods aim to improve reasoning and accuracy by guiding the models through structured prompts that help in breaking down complex problems into manageable steps.

#### Mathematical Evaluation for Large Language Models.

Evaluating the mathematical capacity of large language models (LLMs) is crucial. Benchmarks such as GSM8K (Cobbe et al., ), which targets middle-school level mathematics, and MATH (Hendrycks et al., ), which focuses on high-school math competitions, have been widely used. For university-level problems, datasets like ProofNet (Azerbayev et al., ) and OCWCourses (Lewkowycz et al., ) are prominent. Additionally, MiniF2F (Zheng et al., ) and AlphaGeometry (Trinh et al., ) provide Olympiad-level problems, while the SAT dataset (Azerbayev et al., ) includes problems from the College Board SAT examination.
These datasets have limitations, particularly at the undergraduate level and above, where they fall short in addressing graduate-level and competition-level difficulties (Frieder et al., ). To address this gap, we introduce the MathOdyssey dataset, a diverse collection of mathematical problems designed to serve as a rigorous benchmark for assessing both open-source and closed-source models. Table 2 highlights the properties of MathOdyssey compared to relevant benchmarks, emphasizing the different levels and the diversity of subject areas and question types in our benchmark. This dataset spans a spectrum of difficulty levels, from high school to advanced university mathematics, highlighting the evolving capabilities and ongoing challenges in LLM mathematical problem-solving.

**Table 2: Comparison of existing evaluation datasets for testing AI in mathematics. These datasets are limited, especially in the availability of high-quality, expert-crafted original problems with varying difficulty levels.**
| Dataset | Year | Description | # of Test |
| --- | --- | --- | --- |
| GSM8k (Cobbe et al., ) | 2021 | 8.5k middle-school level math word problems | 1k |
| MATH (Hendrycks et al., ) | 2021 | 12.5k high-school math competitions | 5k |
| OCWCourses (Lewkowycz et al., ) | 2022 | University-level, MIT’s OpenCourseWare | 272 |
| MiniF2F (Zheng et al., ) | 2023 | Olympiad-level | 488 |
| SAT (Azerbayev et al., ) | 2023 | Figureless questions from SAT | 32 |
| ProofNet (Azerbayev et al., ) | 2023 | University-level, proofs | 371 |
| AlphaGeometry (Trinh et al., ) | 2024 | Olympiad Geometry only | 30 |
| MathOdyssey (this work) | 2024 | High School, University-level, Olympiad-level | 387 |

## 3 MathOdyssey

To evaluate the mathematical reasoning abilities of LLMs, we create the MathOdyssey dataset, a rigorously curated collection designed by professionals from both universities and high schools.
To ensure comprehensive evaluation and promote transparency, we have made the entire MathOdyssey dataset and benchmarking code publicly available.
This allows other researchers to replicate our study, compare methods, and explore new approaches using the dataset.

### 3.1 Data Collection

#### Design Principle.

The motivation behind the design of the MathOdyssey dataset is to establish a new benchmark representing the pinnacle of human intellectual achievement, encouraging researchers to push the boundaries of LLMs’ mathematical reasoning capabilities. To realize this vision, we have curated challenges that epitomize comprehensive levels of math problems. Specifically, our benchmark includes:

- •
Inclusion of diverse levels of math problems: Ensuring a comprehensive understanding and catering to various proficiency levels promotes a well-rounded mastery of mathematical concepts and problem-solving skills. This dataset offers a range of problems, starting from basic concepts and gradually increasing in difficulty to cover advanced topics. This allows for a thorough evaluation of AI capabilities across various levels of high school and university mathematics.
- •
Inclusion of different subject area problems: Enhancing LLMs’ mathematical proficiency by exposing them to a wide range of concepts and techniques, from foundational arithmetic to advanced topics such as algebra, number theory, geometry, combinatorics, and calculus. These diverse subject areas help identify LLMs’ strengths and areas for improvement, encouraging the development of critical mathematical reasoning, problem-solving skills, and a deeper appreciation for the interconnected nature of mathematics. By integrating various mathematical disciplines, researchers can create a more engaging and comprehensive learning environment that prepares LLMs for complex real-world challenges in mathematics.
- •
Provision of objective answers and detailed solutions: The objective answers serve as ‘ground-truth’, allowing for objective evaluation of the LLM outputs. In particular, the Open-Answer problems emphasize the importance of detailed reasoning and solution.
Given the varying difficulty and subject areas of these problems, which may exceed comprehension without a specialized background in mathematics, each problem is accompanied by expertly crafted solutions detailing the reasoning steps involved. These solutions are useful for evaluation and can enhance the assessment of LLMs’ reasoning processes.

#### Human professionals.

The dataset was created by human professionals to ensure high quality. Experts developed a wide range of mathematical problems for the MathOdyssey dataset, featuring a spectrum of questions from Olympiad-level competitions, advanced high school curricula, and university-level mathematics. Mathematics professionals, including high-school educators, university professors, and researchers, crafted these problems. Their involvement ensures the dataset not only supports advanced AGI research but also fosters necessary interdisciplinary collaboration.

A typical problem in the MathOdyssey dataset comprises three components: the problem, the answer, and the reasoning, as detailed in Table 1. The problems are original and not sourced from previous datasets or textbooks. Each problem is accompanied by an answer and a detailed solution that explains the reasoning process used to derive the answer. After creation, the problems undergo independent review by a separate team of researchers with expertise in mathematics. This team assesses the problems and their solutions, eliminating any ambiguous or redundant responses to enhance the set’s validity and reliability. This rigorous process guarantees the quality and dependability of the final problem set.

### 3.2 Dataset Analysis

Figure: Figure 1: Mathematical problems across educational levels. We curate and categorize problems by difficulty and subject area.
Refer to caption: x1.png

To understand the properties of the MathOdyssey dataset, we analyze the questions and answers. Specifically, we explore (i) the difficulty of questions based on the type of reasoning required to answer them, (ii) the subject areas of the problems, and (iii) the diversity of answer types.

#### Difficulty of questions.

In the MathOdyssey dataset, each category is designed to evaluate different facets of mathematical reasoning and problem-solving capabilities, ranging from fundamental high school concepts to complex university-level theories, as summarized in Figure 1. This diverse dataset is structured into three distinct levels to challenge various aspects of mathematical knowledge:

- •
Olympiad-level: It tests advanced problem-solving skills with questions in Algebra, Number Theory, Geometry, and Combinatorics.
- •
High School: Broadening the scope, this category includes problems in Algebra, Geometry, and Pre-Calculus, covering a comprehensive range of high school math concepts.
- •
University-level: Catering to higher education, this segment offers challenges in Linear and Abstract Algebra, Calculus and Analysis, Differential Equations, Probability, and Statistics, suitable for university students.

The MathOdyssey dataset categorizes mathematical problems across different educational levels, helping to understand the distribution and scope of problems included in the dataset.
For Olympiad-level Competition, the categories and their respective percentages are Algebra (21.19%), Number Theory (1.03%), Geometry (6.46%), and Combinatorics (9.56%), totaling 38.24%.
For High School Mathematics, the categories are Algebra (17.83%), Geometry (3.62%), and Pre-Calculus (14.21%), totaling 35.66%.
For University-level, the categories are Linear and Abstract Algebra (6.46%), Calculus and Analysis (6.20%), Differential Equations (3.62%), Probability (5.43%), and Statistics (4.39%), totaling 26.10%.
Three subject areas, Differential Equations, Probability, and Statistics, only appear at the University level.

#### Subject areas of the problems.

The problems encompass a wide range of topics, including Algebra, Number Theory, Geometry, Combinatorics, Pre-Calculus, Linear and Abstract Algebra, Calculus and Analysis, Differential Equations, Probability, and Statistics, as shown in Figure 1. The MathOdyssey dataset encompasses a wide range of subject areas, providing a comprehensive testing ground for the mathematical reasoning and problem-solving capabilities of large language models (LLMs). Algebra problems constitute 21.19% from Olympiad-level Competition and 17.83% from High School Mathematics, making them the most represented areas in the dataset. In contrast, Number Theory problems, with only 1.03% from Olympiad-level Competition, have the lowest representation. Pre-Calculus problems, accounting for 14.21% of High School Mathematics, play a significant role in preparing students for more advanced calculus topics. Other subject areas, including Calculus and Analysis, Linear and Abstract Algebra, Differential Equations, Probability, and Statistics, each contribute around 4% to 8% to the dataset. See Appendix B for examples that help better understand the reasoning required to answer the questions.

Figure: Figure 2: There are three answer-types: True-False questions, Multiple-Choice questions and Open-Answer questions.
Refer to caption: x2.png

#### Diversity of answer types.

The MathOdyssey dataset includes a variety of answer types, providing a comprehensive assessment of the mathematical reasoning and problem-solving capabilities of large language models (LLMs). The distribution of answer types is shown in Figure 2, and it is categorized into three main types: True-False questions, Multiple-Choice questions, and Open-Answer questions.
The distribution of answer types in the MathOdyssey dataset is designed to provide a well-rounded evaluation of LLMs’ mathematical capabilities. With 63.0% of the questions being open-answer, the dataset emphasizes the importance of detailed reasoning and solution generation. Multiple-choice questions, making up 32.8%, help assess the models’ ability to choose correct answers from given options, while true-false questions, at 4.1%, provide a quick check of fundamental understanding.
This diverse mix of answer types ensures that LLMs are tested on various aspects of mathematical problem-solving, from basic validation to complex reasoning and solution generation, requiring an understanding of the concepts.

## 4 Experiments

Our goal is to provide a comprehensive standardized dataset to evaluate LLMs on mathematical reasoning. By comparing different models, our benchmarks highlight their strengths and weaknesses.

### 4.1 Models

We evaluate both open-source and closed-source LLMs. The models tested include GPT-4 Turbo, GPT-4 (OpenAI, ), GPT-3.5 Turbo, Gemini models (Reid et al., ), Claude 3 (Anthropic, ), Llama-3-70B, and DBRX-Instruct (Mosaic, ). All models are tested using chain-of-thought reasoning (Wei et al., ). See Appendix C for details of the baselines and prompts.

### 4.2 Model Evaluation

A key advantage of the MathOdyssey data is that every question has an objective answer, so that it is straightforward to check the correctness by code. Such objective answers avoid subjective judgments from humans, making the evaluation consistent and reliable.

We use GPT-4 to assist in evaluating model accuracy, particularly for open-answer questions. The metric measures the similarity between the predicted and ground truth answers. In the MathOdyssey dataset, various types of questions and answers are included. We employ a prompt-based method to provide scores for evaluation, considering the following criteria:

- •
Mathematical Equivalence: Verify answers based on mathematical equivalence using advanced tools like symbolic computation software to confirm the equivalence of different algebraic or symbolic expressions.
- •
Scoring: Assign a score of ‘1’ for answers that match or are equivalent to the provided solution (exact value, choice label, or correctly rounded numerical approximation). Assign a score of ‘0’ for incorrect answers without providing explanatory feedback.
- •
Handling Multiple Choices: Consider the answer correct if the student correctly identifies the choice that matches the solution. Also, treat the corresponding choice as correct if the student provides the exact value that aligns with the problem’s context.
- •
Numerical Equivalence: Accept numerical answers that are correct to at least two decimal places or more, depending on the required precision.
- •
Symbolic and Algebraic Identities: Recognize and accept equivalent algebraic forms as correct, such as standard mathematical identities.
- •
Trigonometric and Logarithmic Forms: Accept equivalent trigonometric and logarithmic expressions, acknowledging transformations that change the form but not the value.
- •
Comprehensive Evaluation: Encourage the use of computational tools for checking equivalence in cases where expressions are too complex for straightforward visual inspection.

See Appendix D for the requirements and prompts used in the evaluation method.

### 4.3 Results and Analysis

We first report the performance on our mathematical benchmarks, as shown in Table 3. Our observations indicate that the benchmark is challenging for these models, with overall performance below 60%.(^2^22Advanced prompting methods using GPT-4 models in the contest have achieved performance improvements between 60% and 70%.) The Gemini Math-Specialized 1.5 Pro exhibits the highest overall performance at 55.8%, suggesting that specialized training significantly enhances capabilities. GPT-4 Turbo achieves 47.03%, followed by Gemini 1.5 Pro at 45.0%, and Claude 3 Opus at 40.6%, all showing competitive performance. For closed-source models (specifically the GPT series) and state-of-the-art open-source models such as Llama-3-70B and DBRX-Instruct, the results show that the selected open-source models not only surpass the performance of GPT-3.5 but are also approaching the capabilities of earlier versions of GPT-4.

When comparing different levels of mathematical problems for GPT models, we observe that High School mathematics is the easiest category for all models, with GPT-4 models scoring above 70%. Olympiad-level problems are the most difficult, with all models scoring below 11%. Similar trends are seen for Llama-3-70B and DBRX-Instruct, with their performance in the Olympiad-level category being even lower, at less than 10%.

Furthermore, closed-source models, particularly the GPT-4 Turbo, exhibit stronger performance in high school and university-level math, highlighting ongoing advancements in their development. This data underscores the rapid progression of closed-source models in handling increasingly difficult mathematical questions over time. The performance gap between the best closed-source model, GPT-4 Turbo, and the open-source Llama-3 for difficult mathematical problems is notably narrow. For instance, GPT-4 Turbo achieves an overall accuracy of 10.14% in the Olympiad-level mathematics, while Llama-3 achieves 9.46%. This demonstrates that both models, despite notable progress, still face significant challenges in solving these complex problems. However, for other difficulty levels, the gap becomes larger. For example, GPT-4 Turbo achieves 84.78% in high school mathematics, while Llama-3-70B scores only 52.17%, a difference of more than 30%.

**Table 3: Results for different LLMs. We use chain-of-thought reasoning for solving problems. The performance of Gemini 1.5 Pro and Claude 3 Opus are quoted from the Gemini 1.5 report (Reid et al., ). Both GPT-4-Turbo and Gemini 1.5 Pro outperform the other models. For GPT-4-Turbo, we use results based on gpt-4-turbo-2024-04-09. For GPT-4, we use results based on gpt-4-0613. For GPT-3.5 Turbo, we use results based on gpt-3.5-turbo-0125.**
| Model | Olympiad-level | High School | University-Level | Overall |
| --- | --- | --- | --- | --- |
| GPT-4 Turbo | 10.14% | 84.78% | 49.50% | 47.03% |
| GPT-4 | 5.41% | 74.64% | 32.67% | 37.21% |
| GPT-3.5 Turbo | 2.03% | 41.30% | 15.84% | 19.64% |
| Gemini |  |  |  |  |
| -1.5 Pro | - | - | - | 45.0 % |
| -Math-Specialized 1.5 Pro | - | - | - | 55.8 % |
| Claude 3 Opus | - | - | - | 40.6 % |
| Llama-3-70B | 9.46% | 52.17% | 21.78% | 27.91% |
| DBRX-Instruct | 8.11% | 42.75% | 20.79% | 23.77% |

**Table 4: Results for different LLMs across various subject areas. Note that the results are used for evaluating the LLMs by direct comparison and may be improved with different prompting methods.**
| Category | GPT-4 Turbo | GPT-3.5 Turbo | Llama-3-70B | DBRX-Instruct |
| --- | --- | --- | --- | --- |
| Olympiad-level: |  |  |  |  |
| Algebra | 8.54% | 2.44% | 8.54% | 4.88% |
| Number Theory | 0.00% | 0.00% | 25.00% | 0.00% |
| Geometry | 16.00% | 4.00% | 8.00% | 20.00% |
| Combinatorics | 10.81% | 0.00% | 10.81% | 8.11% |
| High School Mathematics: |  |  |  |  |
| Algebra | 88.41% | 39.13% | 44.93% | 39.13% |
| Geometry | 92.86% | 71.43% | 71.43% | 57.14% |
| Pre-Calculus | 78.19% | 36.36% | 56.36% | 43.64% |
| University-level: |  |  |  |  |
| Differential Equations | 71.43% | 28.57% | 50.00% | 28.57% |
| Linear & Abstract Algebra | 44.00% | 16.00% | 24.00% | 28.00% |
| Calculus & Analysis | 62.50% | 16.67% | 20.83% | 12.50% |
| Probability | 14.29% | 9.52% | 0.00% | 4.76% |
| Statistics | 64.71% | 11.76% | 23.53% | 35.29% |

Table 4 presents the results for different LLMs across various subject areas. The results show that GPT-4 Turbo consistently outperforms others across most categories, particularly in High School Mathematics and University-Level subjects. It shows a notable lead in Algebra, Geometry, and Pre-Calculus at the high school level, and Differential Equations, Linear & Abstract Algebra, Calculus & Analysis, and Statistics at the university level.
GPT-3.5 Turbo shows consistent but lower performance compared to GPT-4 Turbo.
Llama-3-70B performs well in certain areas, particularly in Olympiad-level problems. It has the highest score in Number Theory among all models. However, it struggles significantly in Series and Probability.
DBRX-Instruct shows strength in Olympiad-level Geometry but generally lags behind GPT-4 Turbo and Llama-3-70B in other categories.

## 5 Conclusion

We introduce MathOdyssey, a dataset for assessing LLMs’ mathematical problem-solving skills. Our dataset, evaluation methods, and code are openly available. We have shown that while LLMs, both open-source like Llama-3 and DBRX-Instruct, and closed-source such as the GPT series, demonstrate proficiency in routine and moderately difficult mathematics, they struggle significantly with complex Olympiad-level problems. Additionally, we have revealed promising developments; open-source models are beginning to approach the performance levels of earlier GPT-3.5 iterations. Despite this progress, performance on the most challenging questions remains low, highlighting a clear gap that future advancements need to address.

Ultimately, our research underscores the ongoing journey towards achieving human-like mathematical reasoning in AI, with the MathOdyssey dataset serving as a benchmark for catalysing future developments. We are optimistic that continued research will progressively bridge the existing capability gap. In the future, expanding the MathOdyssey dataset to include a wider range of problem types and enhancing metrics to better capture deep mathematical reasoning can yield further insights into LLM capabilities.

Limitation. While the MathOdyssey dataset includes a variety of problems across different levels of mathematics, the questions may not cover all types of mathematical reasoning or problem-solving approaches. This limitation could affect how well the dataset generalizes to other forms of mathematical challenges not represented in your collection.

Future. To address generalizability limitations, future work involves expanding the dataset to include a wider range of mathematical topics and problem types, including those that require visual representations, proofs, or interactive problem-solving.

## Acknowledgements

We would like to extend our sincere gratitude to AGI Odyssey, the NGO responsible for organizing the Global Artificial Intelligence Championships (GAIC) Math 2024. Their dedication and commitment to promoting artificial intelligence education and innovation have been invaluable to the success of this project. Additionally, we appreciate their contribution of resources and support, which have played a significant role in making this initiative possible.

## Appendix A MathOdyssey dataset examples and solutions

Olympiad-level Problem. Let $S=\left\{1,2,\cdots 2024\right\}$, if the set of any $n$ pairwise prime numbers in $S$ has at least one prime number, the minimum value of $n$ is                  .
Answer: 16
Reasoning:
Taking the 15 numbers $1,2^{2},3^{2},...,43^{2}$ violates the condition. Furthermore, since $S$ does not contain any non-prime numbers with a minimum prime factor of at least 47, there are only 14 types of non-prime numbers in $S$, excluding 1. Applying the Pigeonhole Principle, we conclude that $n=16$.
GPT-4 Turbo’s Solution: Answer: 307
Reasoning: The minimum number of elements required such that any subset of these elements from the set $S=\{1,2,...,2024\}$ containing pairwise prime numbers includes at least one prime number is 307. This is based on the count of prime numbers up to 2024 (approximately 306) and the application of the Pigeonhole Principle, which dictates that selecting more numbers than there are primes forces the inclusion of a prime number or a violation of the pairwise prime condition.

High School Problem. What are the solutions of the quadratic equation $15x^{2}=2x+8$?

$$ $\displaystyle\text{ A) }\,\{-\frac{4}{3},-\frac{3}{2}\}\,\,~{}~{}\text{ B) }\, \{-\frac{4}{5},\frac{2}{3}\}\,\,~{}~{}\text{ C) }\,\{-\frac{3}{2},\frac{4}{5} \}\,\,~{}~{}\text{ D) }\,\{-\frac{2}{3},\frac{4}{5}\}$ $$

Answer: $D$
Reasoning:
First move all terms to one side: $15x^{2}-2x-8=0$. Then factor into $(5x-4)(3x+2)=0$. Setting $5x-4$ to zero results in a solution of $x=\frac{4}{5}$ and setting $3x+2$ to zero results in a solution of $x=-\frac{2}{3}$.
GPT-4 Turbo’s Solution: Answer: $D$
Reasoning: The quadratic equation was first rearranged to standard form, coefficients were identified, and the quadratic formula was applied to find the solutions. The solutions were verified by substituting back into the original equation.

University-level Problem. Find the limit

$$ $\lim\limits_{x\to 1}\frac{f(2x^{2}+x-3)-f(0)}{x-1}$ $$

given $f^{\prime}(1)=2$ and $f^{\prime}(0)=-1$.
Answer: $-5$
Reasoning:
Let $g(x)=2x^{2}+x-3$. Since $g(1)=0$, the desired limit equals $\lim\limits_{x\to 1}\frac{f(g(x))-f(g(1))}{x-1}$. By the definition of the derivative and the chain rule and noting that $g^{\prime}(1)=5$, we have

$$ $\lim\limits_{x\to 1}\frac{f(g(x))-f(g(1))}{x-1}=f^{\prime}(g(1))g^{\prime}(1)= f^{\prime}(0)g^{\prime}(1)=(-1)(5)=-5.$ $$

GPT-4 Turbo’s Solution: Answer: $-5$
Reasoning: The limit was calculated by interpreting it as the derivative of a composed function, applying the chain rule, and substituting the given derivative values.

## Appendix B MathOdyssey different subject areas

Table 5 presents MathOdyssey examples spanning various subject areas. These encompass Algebra, Number Theory, Geometry, Combinatorics, Pre-Calculus, Linear and Abstract Algebra, Calculus and Analysis, Differential Equations, as well as Probability and Statistics.

**Table 5: Examples of different subject areas.**
| Subject Area | Example |
| --- | --- |
| Algebra | Let $S=\left\{1,2,\cdots 2024\right\}$, if the set of any $n$ pairwise prime numbers in $S$ has at least one prime number, the minimum value of $n$ is                  . |
| Number Theory | A natural number whose last four digits are 2022 and is divisible by 2003 has a minimum value of                  . |
| Geometry | In a cube $ABCD-A_{1}B_{1}C_{1}D_{1}$, $AA_{1}=1$, $E$, $F$ are the midpoints of edges $CC_{1},DD_{1}$, then the area of the cross-section obtained by the plane AEF intersecting the circumscribed sphere of the cube is                  . |
| Combinatorics | If three points are randomly chosen from the vertices of a regular 17-sided polygon, what is the probability that the chosen points form an acute-angled triangle? |
| Pre-Calculus | In $\triangle\mathrm{ABC},\mathrm{AB}=10\mathrm{~{}cm},\angle\mathrm{B}=90^{\circ}$, and $\angle\mathrm{C}=60^{\circ}$. Determine the length of $\mathrm{BC}$.<br>$\displaystyle\text{A)}\ 10\mathrm{~{}cm}~{}~{}\text{B)}\ 10\sqrt{3}\mathrm{~{}<br>cm}~{}~{}\text{C)}\ \frac{10\sqrt{3}}{3}\mathrm{~{}cm}~{}~{}\text{D)}\ 20<br>\mathrm{~{}cm}$ |
| Linear and Abstract Algebra | Find the solution $[x_{1},x_{2},x_{3}]$ to the following equations<br>$\left\{\begin{array}[]{c}x_{1}+3x_{2}+3x_{3}=16,\\<br>3x_{1}+x_{2}+3x_{3}=14,\\<br>3x_{1}+3x_{2}+x_{3}=12.\\<br>\end{array}\right.$ |
| Calculus and Analysis | Evaluate the following limit:<br>$\lim_{n\to\infty}\left(\sqrt{n^{2}+2n-1}-\sqrt{n^{2}+3}\right).$ |
| Differential Equations | Consider the differential equation $\frac{dy}{dx}=xy$. Find the value of $y(\sqrt{2})$ given that $y(0)=2$. |
| Probability | Suppose that $A$, $B$, and $C$ are mutually independent events and that $P(A)=0.2$, $P(B)=0.5$, and $P(C)=0.8$. Find the probability that exactly two of the three events occur. |
| Statistics | Given the data set $\{3,7,7,2,5\}$, calculate the sample mean $\mu$ and the sample standard deviation $\sigma$. Present the answer as $[\mu,\sigma]$. |

## Appendix C Baselines and prompts

Figure 3 depicts the prompt utilized for guiding Language Models (LLMs) in solving mathematical problems within our experimental framework. This prompt distinctly outlines the system’s role as a math professor, delineating task specifications and the anticipated output format for tackling intricate mathematical challenges.

Figure: Figure 3: Mathematical problem-solving prompts employed by LLMs.
Refer to caption: x3.png

## Appendix D Evaluation

Figure 4 depicts the prompt employed during the evaluation of large language models in our experiments. This prompt defines the system’s role as a math teacher, providing both assessment criteria and the expected output format for grading mathematical problems. We have also made our evaluation code accessible to the public.

Figure: Figure 4: Evaluation prompts.
Refer to caption: x4.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. While the authors mention diverse answer types, there are only three categories presented. Moreover, there seems to be a discrepancy between the data in Figure 2 and that described in line 304. Given that existing mathematical models have undergone extensive pre-training and can effectively comprehend natural language queries, why does the diversity of answer types pose a greater challenge for these models? Is there a variance in accuracy across different question types? Do models exhibit inconsistencies when dealing with various types of questions?
2. The creation details of the problems:
* Please include principles and details of problem creation in an appendix. How is answer correctness ensured? Have cross-validation or sampling methods been employed? What is the accuracy rate?
* How are problems categorized into different difficulty levels?
* Does an individual problem encompass multiple domains simultaneously? How is this handled?
3. For evaluation, the authors directly employed GPT-4 for assessment. How to ensure the accuracy of GPT-4's evaluations, and what is the consistency with human assessments? Furthermore, it is recommended that the authors incorporate rule-based evaluation methods to enhance accessibility.
soundness: 2
strengths: * The paper is well-motivated, as GPT-4o poses a significant challenge to current mathematical benchmarks. The introduction and open-sourcing of high-quality, difficult mathematical problems is a meaningful contribution to the field.

* The dataset features distinct levels of difficulty and sub-domain classifications, which enhance its uniqueness.
summary: The manuscript presents an original and challenging dataset for mathematical problem-solving, encompassing various subjects and difficulty levels. Then the paper conducts comprehensive examinations on both open-source and closed-source LLMs. The findings reveal that while closed-source models currently lead, open-source models are rapidly catching up, highlighting the competitive landscape of LLM capabilities in mathematical problem-solving.
weaknesses: 1. The manuscript lacks coverage of important related work and further clarification on the difference and improvements compared to them.
    * OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
    * Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models
    * OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI
    * Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models

2. There are missing baseline comparisons that are crucial for evaluating the open-source models on the proposed dataset, like Qwen2.5-MATH, DeepSeek-Coder, and so on.

3. It is unclear how the authors ensure that the data has not been previously encountered. If the problems are original, details regarding the creation principles and methodologies should be included in the paper. Additionally, how is the correctness of answers verified? Have the authors conducted cross-validation or sampling tests to ensure reliability? What is the accuracy rate?
4. The conclusions drawn seem predictable and do not provide substantial insights. Are there fine-grained analyses and interesting findings?

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 2
questions: 1. Is there a particular reason that an LLM is used as an evaluator instead of using SymPy or Mathematica directly when needed (for example equivalence of symbolic expressions)?
2. While I have not deep dove into the dataset, the example from Table 1 has implicit or missing steps in the university example to my reading. It seems to have an implicit application of L'Hôpital's Rule before taking the limit and then applying the chain rule. 
Thus, my question is how did you balance implicit vs explicit steps in the reasoning? 
I would expect/hope an LLM to list all steps, including those "obvious" to a human marker, and err on the verbose side; however, I accept this as a personal view/bias and am keen to hear what the dataset prerogative was.
soundness: 3
strengths: + The proposed benchmark should have a unique answer enabling easier verification of the correct answer.
+ The problems are crafted specifically for the benchmark avoiding their presence in the pre-training data for LLMs.
+ The problem space covered in terms of topics and difficulty is wide, allowing the identification of problem areas for further research as well as "solved" areas if a topic saturates.
summary: The authors identify a need for a mathematics benchmark that spans a wider breadth of topics and difficulties. They propose MathOdyssey, a benchmark that includes hand-written and curated high-school, university, and Olympic-level problems. Each problem has a unique expected answer and detailed reasoning to aid LLM assessment. They demonstrate that the benchmark is not saturated since GPT-4 o1-preview archives ~65% overall. Further, their wide coverage of topics enables the identification of problem topics for LLMs enabling researchers to focus on those areas.
weaknesses: - Some of the paper language is hyperbolic, for example, S3.1, "Design Principle." Paragraph, L178: "representing the pinnacle of human intellectual achievement" is very strong language and am uncertain the authors could substantiate such a claim unless it is an opinion. Another example is S3.1 L234: "This rigorous process": The curation process, while I can trust was done rigorously, is not presented in sufficient detail for me to make that assessment, and it would be better to instead tone it down to "This process facilitates the quality and dependability...". The paper in general would benefit from a pass that tones down the hyperbolic language to instead focus on the proposed advancements in an objective tone.
- L256-264 could be replaced with "Fig. 1 shows the detailed information". and L266-268 repeat the same information that is already in the figure and could be pointed to.
- The benchmark claims easy verifiability by code but uses GPT-4 as an evaluator (with the associated errors this induces even if the prompt enables the use of tools).
- The ease of having a unique answer is counter-balanced by the false positives induced by correct-answer-with-wrong-reasoning.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Address the weakness of the paper
soundness: 3
strengths: 1. Release of a novel dataset that will help the community as this dataset has not been used in the training of existing models.
2. Comprehensive benching of different models, highlighting their efficiency in solving different categories/areas of problems.
3. Effective use of GPT-4 for evaluating the accuracy of models. The author employs a prompt-based method and provides scores of various categories.
summary: The paper aims to investigate and understand LLMs’ strong problem-solving abilities. The paper introduces a novel dataset “MathOdyssey”. The dataset includes a diverse set of mathematical programs at three levels: High school, university and olympiad level. Each category has a wide range of different problem areas such as Algebra, Number Theory, calculus etc. The dataset contains a total of 387 data points and has novel problems created by mathematics professionals, including high school educators, university professors and researchers. Each problem is accompanied by its final answer and its reasoning chain. They also did a comprehensive evaluation of the dataset and tested it on both closed and open models. They used GPT-4 to assist in evaluating the model accuracy, as the dataset contained a wide range of answer types (open answer, MCQ, and true-false). The evaluation shows that the closed source model particularly GPT-4, o1 and GPT-4 Turbo shows strong performance in high school and university-level math. For open-source models such as Llama-3, the results show that the selected open-source models only surpass the performance of GPT3.5 but are also approaching the capabilities of the earlier version of GPT-4.
weaknesses: 1. Even though the dataset provides various categories of questions in different areas, the count of individual categories is very small. For example Number Theory – Olympiad-level accounts for only 4 problems, Differential Equations – University-level for 14 problems etc. So do the authors have any plan to extend the count of problems in these areas?
2. Even though the dataset does not use any existing problems, a sanity check for data contamination should be done. Experiments from the paper [1] should be added to ensure no data contamination.
3. Evaluating the dataset using models fine-tuned specifically for solving math problems, such as MathCoder [2] helps show how models trained specifically to solve math problems perform on MathOdyssey.

Reference:

[1] : Golchin, Shahriar, and Mihai Surdeanu. "Time travel in llms: Tracing data contamination in large language models." arXiv preprint arXiv:2308.08493 (2023).

[2] : Wang, Ke, et al. "Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning." arXiv preprint arXiv:2310.03731 (2023).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Questions and Concerns:

- Line 103-105: The authors claim, "The key distinction of our dataset is its expert-driven creation, which minimizes the risk of data contamination." In lines 106-107, they state, "The dataset has not been used for training by LLMs." To substantiate these claims, the authors should conduct a contamination detection analysis, which has not been done.

- The term "open-answers" is mentioned without a clear definition. Can these open-answers be further classified into more fine-grained types, similar to the classification in OlympiadBench [1]?

- Given that GPT-4 is used as the answer judge for all experiments and analyses, it is necessary to conduct human evaluations to determine the reliability of GPT-4 as a judge.

- The evaluation prompt requires output in JSON format. However, some LLMs are fine-tuned to produce answers delimited by "boxed". Have you investigated whether LLMs conform to the expected format?

- Have you considered incorporating difficulty levels like MATH? Some high school problems might be easier than certain university-level problems.

- Results on More Open-Source LLMs: Can you provide results from more open-source LLMs?

Other Issues:

Lack of Appropriate Citations:

- Lines 39-41: "has achieved more than a 90% success rate"

- Lines 44-45: "technological advancement but a crucial step toward developing more general and capable artificial intelligence systems"

- Lines 51-52: "Moreover, a significant obstacle is that many existing datasets might have been included in the training phases of these models, potentially skewing performance metrics."

- Lines 320-322: Citations or links for LLMs are missing.

Inaccurate Statements:

- Lines 47-49: "it remains uncertain how well they handle more complex mathematical challenges, such as those found in university-level courses and competitive high school mathematics." In fact, datasets like MATH and MathBench [2] include competitive high school mathematics and university-level courses, respectively.

- Lines 51-52: "Moreover, a significant obstacle is that many existing datasets might have been included in the training phases of these models, potentially skewing performance metrics." There are, however, many variants of existing datasets that address this issue, such as GSM-1K [3], which also weakens the contribution of this paper because MathOdyssey is limited in size.

- Table 2: The number of examples for GSM8K should be 1319.

[1] He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., ... & Sun, M. (2024). Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.

[2] Liu, H., Zheng, Z., Qiao, Y., Duan, H., Fei, Z., Zhou, F., ... & Chen, K. (2024). MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark. arXiv preprint arXiv:2405.12209.

[3] Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., ... & Yue, S. (2024). A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332.
soundness: 1
strengths: The proposed MathOdyssey dataset is novel and may be somewhat useful to certain researchers. While it introduces a variety of challenging problems, the experimental results provide a rough indication of the performance of closed-source LLMs. Overall, the dataset and findings suggest potential (but limited) usefulness to offer insights into LLM capabilities in mathematical reasoning, albeit with room for more comprehensive analysis.
summary: This paper presents a new dataset called MathOdyssey, which aims to evaluate the reasoning abilities of large language models (LLMs). The dataset consists of 387 problems, including 148 at the Olympic level, 101 at the university level, and 138 at the high school level. The problems cover several subjects with three answer formats: true/false, multiple choice, and open answers.

The authors evaluate LLMs' math reasoning performance on MathOdyssey using GPT-4 as the answer judger in a zero-shot manner, providing it with specific instructions. They conduct their experiments with seven closed-source LLMs and one open-source model, Llama-3-70B. Their findings reveal that the Llama-3-70B model still falls short when tackling more complex problems.
weaknesses: MathOdyssey **offers no clear advantages over existing benchmarks**, which may limit the usefulness and contribution of this paper.

- Compared to existing datasets, MathOdyssey is **limited in size**, containing only 387 problems, whereas datasets like GSM8K and MATH include 1,319 and 5,000 problems, respectively. This limitation might impact the reliability of accuracy in ranking the mathematical reasoning abilities of different LLMs.

- **The "difficulty levels" within MathOdyssey are not well-defined**. Although it claims to cover comprehensive levels of math problems, it includes only three educational stages. In contrast, the MathBench [1] dataset offers a wide range of problems, spanning from primary school to university level. By the way, some datasets define the difficulty level as a rating (e.g. an integer number)

- While the authors claim to have diversified answer types, MathOdyssey **only encompasses three distinct answer types**. OlympiadBench [2], however, incorporates a more fine-grained variety of answer types.

- Although MathOdyssey includes several subjects, **the number of testing examples within each subject is relatively small, with many subjects containing fewer than 10 examples**. This limitation may lead to inaccurate analyses across different subjects.

**The experiments are not comprehensive and compelling**:

- The evaluation process is flawed because the authors use GPT-4 as the judge for answers in a zero-shot manner. However, **it is unclear how often this judgment aligns with human evaluators**. An analysis of judgment errors is necessary, and I recommend considering rule-based matching.

- They **include only one open-source LLM, Llama-3-70B** in experiments, which is not comprehensive. The authors should include more open-source LLMs, including both general-purpose chat models and math-specialized LLMs.

- The results and analysis are not compelling, as the reliability of GPT-4's judgment is uncertain. Additionally, **conducting an error analysis could provide valuable insights into how LLMs get wrong in solving math problems**.


This paper **is poorly written** and either **lacks important details or makes inaccurate claims or claims without proper citations** (see Questions).


[1] Liu, H., Zheng, Z., Qiao, Y., Duan, H., Fei, Z., Zhou, F., ... & Chen, K. (2024). MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark. arXiv preprint arXiv:2405.12209.

[2] He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., ... & Sun, M. (2024). Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008.