HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics
Authors: Jingxuan Fan
Sarah Martinson
Erik Y. Wang
Kaylie Hausknecht
Jonah Brenner
Danxian Liu
Nianli Peng
Corey Wang
Michael Brenner
Venue: ICLR 2025 Poster (2025)
Keywords: math
benchmark
dataset
few-shot learning
reasoning
Primary area: datasets and benchmarks
TL;DR: We introduce a new dataset of difficult graduate-level applied mathematics problems; evaluations demonstrate that current leading LLMs exhibit low accuracy in solving these problems.

Abstract: Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce $\textbf{HARDMath}$, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on $\textbf{HARDMath-mini}$, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate the limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like $\textbf{HARDMath}$ to advance mathematical abilities of LLMs.

Published as a conference paper at ICLR 2025
HARDMATH: A BENCHMARK DATASET FOR
CHALLENGING PROBLEMS IN APPLIED MATHEMATICS
Jingxuan Fan;" Sarah Martinson; Erik Y. Wang. Kaylie Hausknecht," Jonah Brenner, Danxian Liu, Nianli Peng, Corey Wang. Michael P. Brenner School of Engineering and Applied Sciences, Harvard University
ABSTRACT
Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce HARDMATH, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical ap- proximation techniques. These problems demand combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generatee a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed- source LLMs on HARDMATH-MINI, sub-sampled test set of 366 problems, as well 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few- shot Chain-of-Thoowe prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct detailed error analysis gain insights into the failure cases of LLMs. These results demonstrate limitations of current LLM performance on advanced graduate- applied math problems and underscore the importance of datasets like HARDMATH to advance mathematical abilities of LLMs]
INTRODUCTION
Many mathematical equations that arise in practical scientific and engincering problems cannot be solved analytically. Traditional mathematies courses tend to focus equations with exact, analytical solutions, teaching only limited set of techniques for solving them. Similarly, the mathematical reasoning datasets used to benchmark large language models (LLMs) are predominantly restricted to problems of this nature. However, many real-world mathematics problems involve integrals, ordinary differential equations (ODEs), and partial differential equations (PDEs) that do not have closed-form solutions and must be approached with different set of techniques. While numerical solutions offer valuable insights, they often fail to provide intuition behind solutions behavior. key approach in applied mathematics involves finding approximate analytical solutions to complex problems using asymptotic and applied analysis techniques- metthos that are largely underrepresented in existing LLM benchmark datasets. To address this gap, we introduce HA MAATHH a dataset specifically designed to focus on asymptotic reasoning in mathematics. This dataset captures a fundamentally different type of mathematical reasoning compared to other benchmarks and can be useful for evaluating LLMs' abilities to make research-relevant approximations.
HARDMATH consists of 1,466 problems inspired by a graduate-level course asymptotic methods. These problems cover algebraic equations, ODEs, and integrals commonly encountered real- world scientific and engineering contexts, where exact solutions often do not exist. Instead, various asymptotic reasoning techniques are employed to find approximate but accurate solutions to these complex mathematical problems.
A primary motivation for developing HARDMATH is the lack of benchmark datasets targeting the mathematical approximation methods required in many applications. While some recent works
"Equal contribution
Dataset: http:://ggtaaaaccooosmmaaa/////a

---

Published as a conference paper at ICLR 2025
have begun to include university- problems (Liu et al. 2024 most datasets focus grade school- to high school-level mathematics problems (Amini et al. 2019; Hendrycks et al. 2021 Cobbe et al. 202T whose solution methods only involve direct, "clean' calculations. In contrast, HARDMATH MATH targets applied mathematics problems that require approximate analytical solutions; an equally important yet underrepresented aspect of mathematical reasoning. Solving such problems can be challenging even for individuals with highmathematical proficiency, as it requires advanced techniques from calculus, differential equations, and complex analysis. Additionally, computational tools are often needed to analyze the behavior of different terms in each equation and to derive numerical solutions that can serve as benchmarks for the approximations. Given the difficulty of these problems and their prevalence across science and engineering -eeeee researchers may not always have a strong background in advanced mathematics this level difficulty and style of mathematics should be included in L.M evaluations.
Rather than relying on the typical approach of collecting problems from textbooks, standardized tests, or competitions, as seen in most existing datasets, we developed algorithms to automatically generate problems and their step-by-step solutions. We implemented a comprehensive testing methodology to evaluate the mathematical reasoning abilities of leading LLMs in the domain of approximation methods. Our dataset includes a larger set HARDMATH that can be used for model developments (e.g. novel prompting techniques or fine-tuning), well as two test sets HA MARDDMATHMMMN and WORD-PROBI PEA.PRRBB..E-SHHS evaluation accuracy summary and error mode analyses. Our results demonstrate that the performance of current LLMs on these problems is poor, highlighting significant room for their improvement on these challenging asymptotics problems.
RDMATH, which are used to assess LLM performance. We present an
2 RELATED WORK
2.1 MATHEMATICAL DATASETS
LLMs have shown promising capabilities in mathematics. However, evaluating and expanding the full extent of these abilities requires diverse datasets with problems that go beyond basic arithmetic or elementary word problems. Existing benchmarks often focus on these simpler domains, with a gap in addressing graduate-level applied mathematics problems that demand deeper understanding and diverse, multi- analytical skills. Most mathematics datasets for evaluating or training LLMs contain samples that either present the problem directly or within a constructed narrative context. Notable examples of these datasets include MATH (12,500 high school competition-style problems) (Hendrycks et al. 202T), GSM8K (8,500 multistep grade-school problems) (Cobbe et al. 2021), MATHQÁ (37.000 GRE/GMMAT-evel multiple-choice problems) (Amini et [ and ODYSSEY-MATH (387 hand-curated problems across various difficulty levels) (Netmind Al, (002)) While these existing datasets are valuable for assessing LLM math performance in certain areas, most are limited in scope and complexity.
Recent efforts target more advanced problems that are most often manually-sourced Relevant works include JEEBENCH (Arora et al.) 2023) and a subset of the MATHBENCH dataset (Liu et al. 2024), both of which cover some college-level topics including simple ODEs and multivariable caleulus. More advanced-level problems are presented in GHOSTS, which contains a GRAD- TEXT subset-a collection of 130 exercises from graduate-level mathematies textbooks in functional analysis, topology, and probability theory (Frieder et al., 2024) and in A ARB, which features small set of university-level formal mathematies problems from prior qualifying examinations in the mathematics departments at Harvard University and the University of California. Berkeley (Sawada al. 2023). However, these datasets are limited by their size and scalability; datasets created by scraping textbooks or similar resources are generally quite small and difficult to broaden easily. Most of these challenging datasets also focus on abstract, formal mathematics and exclude other forms of mathematical reasoning. Finally, textbook problems are often protected by copyright, which can complicate their public use.
Existing datasets (summarized in Table 1 thus lack the scale and specific focus needed to evalu- ate LLMs on advanced mathematical problems that may be highly useful for scientific research. HARDMATH aims to address these limitations by offering a large collection of challenging applied mathematics problems inspired by a graduate- course on asymptotic methods. It emphasizes problems that require diverse mathematical approaches, numerical calculations, and subjective judg-
2

---

Published as a conference paper at ICLR 2025
Table Comparison of HARDMATH with related datasets. Note that for all the datasets excluding MATH and GSM8K, we report the number of relevant problems at comparable difficulty to our dataset (e.g., THEORY-KNOWLEDGH LELGL-----ELEEEE COLLEGE in MATHBENCH, and GRAD-TEXt and HOLES-IN-PROOFS from GHOSTS.) HARDMATH is the largest graduate-level dataset.
Dataset
Size
Problem Sourcing
Difficulty
MATH (Hendrycks et al. 2021
12.5K
Manual
High School
GSM8K (obbe et al. 20211)
8.5K
Manual
Grade School
MATHBENCH-1 [Ju
2024
632
Manual, Algorithmic
Undergraduate
JEEBENCH (Arora
2023)
236
Manual
High School
GHOSTS [Friddr ct al.
190
Manual
Graduate
ARB (Swadaa et al. 2023
34
Manual
Graduate
HARDMATH (Ours)
1.4K
Algorithmic
Graduate
ment, mirroring the complexity of problems faced by researchers in variety of domains. Code for auto-generating the problems in HARD MARMMATH can be used to generate any number of additional problems, which is a unique and powerful feature for scaling LLM benchmarking and model develop- ments like novel prompting techniques or fine-tuning. A key area of interest in current LLM research developing models that can effectively use external tools. The problems in our dataset are unique because they involve approximate solutions that cannot be formalized using tools like Lean similar software, To excel in this benchmark, LLMs must integrate tool use with sophisticated reasoning. This makes HARDMATH particularly valuable for benchmarking and developing LLMs capable of effective tool use, setting apart from other mathematical datasets.
2.2 RECENT INTEREST IN ADVANCED MATHEMATICAL REASONING
As LLMs continue to improve, there has been growing interest in developing more challenging benchmarks, especially in mathematics. A notable example is the recent open challenge, Humanity's Last Exam, which aims to create the world's most difficult public Al benchmark, requesting questions that "only exceptional individuals can answer correctly," do not involve "straightforward caleulation/- computation," and are written by individuals with PhD-level academic training (Hendrycks & Wang 2024). Similarly, frontier models have been advancing quickly, and many are explicitly focused on quantitative and scientific reasoning, such as OpenAl's recent ol series. In line with our motivation for developing HARDMATH to better track the progress of LLMs, OpenAl argues that "recent frontier models do so well on MATH and GSM8K that these benchmarks are no longer effective at differentiating models" (OpenAl 2024).
3 DATASETS
3.1 HARDMATH DESIGN CHOICES
Here, we detail the HARDMATH dataset, which contains problems on polynomial nondimension- alization, polynomial root- finding, ODEs, integrals, and word problems that contextualize each of these. A sample of a HARDMATH integral problem shown in Box 1. HARDMATH contains four problem classes with seven distinct problem types, well 40 handwritten word problems contextualizing the problem types. The main HA RDMATH dataset, which can be used for model developments like fine-tuning, contains 1,060 problems, and the evaluation dataset HA HARDMATH- MINI, which we use in this paper to benchmark LLM performance, contains 366 problems. Figure [ provides a detailed breakdown of these datasets.
Box I: Sample Integral Problem and Solution
Problem:
Consider the integral 7(e) |55.00 +22.0r 9. 05.00 11.00 dr. Develop analytical formu- las that approximate (c) for different regimes of e.
3

---

Published as a conference paper at ICLR 2025
Pr Jblle lype distribuliori in in HAGDMatl-mini (386 problems)
Prubiemn Lyse distriadion HARSHell (1060 preblems)
Rolymomal
Pnlymonea Poots
17
vely molynomials
Addmeemate
00
16 Als
inttgrals
Ib.7%
Symbel
Nar dm
17.0%
Nonder
inegrals
Nondim
Mondim
Caplare
Lastaci
numaic Sondon
kondim
(a) HARDMATH-MIN dataset
(b) HARDMATH dataset
Figure 1: Breakdowns of the HA HARDMATH-MINI (left) and the HA HARDMATH (right) datasets.
The integral is of the form [(c) So c+P(s) do where P(x) is polynomial. Thus, its value can be estimated as the product between a height and a width. Since the integrand is maximized at 0, the height can be set to 2.
Solution:
For small we define the width the point where the integrand becomes half of its maximum height. This corresponds to solving for æ given P(z)
Applying dominant balance, considering the
P(z) with the smallest degree, the width approximated as (2.0,c) 1/6.0 Therefore, analytical
approximation of the integral for small is I(e) /0:5103
0,8009
an intermediate regime where is large, we also define the width based on term with
1/13.0 Therefore, the analytical approximation of largest degree. The width approximated as (.....)
the integral for large is [(e) 7647
the width the integral exceeds the range integration, we consider one more regime for very large The width is then just the range of integration, so in this regime, the integral can be approximated as $. Therefore, the analytical approximation of the integral for very large is [(c)
56
0.89
0.76
56
Altogether, the solutions small, large, and very large
0.83
00.3
One key commonality between all HARDMATH problems is the use of the Method of Dominant Balance in calculating solutions. This reduces equation only the terms that 'dominate' the behavior the solution and can significantly simplify the equation (Bender Orszag 2013). In addition the Method of Dominant Balance, problems also involve other sophisticated mathematical techniques, such as checks for self-consistency and the use of numerical methods. The combination of these tools captures several key aspects of mathematical modeling, including the combined use of computational and analytical techniques. Additionally, subjective choices about the regimes of solution space to consider, the number of terms include in approximate expressions, and the approximation methods themselves must be made on case-by-case basis with rigorous mathematical justification. Both of these aspects are potentially difficult tasks for existing L.L.s. In Box I, we highlight the methods and regimes relevant to solving a sample integral problem.
3.2 DATASET GENERATION AND VERIFICATION
The dataset generation procedure is outlined in Fig. 2 Code for data generation uses SymPy (Meurer al., 2017). a library for symbolic mathematics, and SciPy, a library for scientific computing (Virtanen et al. 2020). to implement the mathematical procedures required for obtaining approximate, analytical solutions. Problems are generated by combining randomly selected coefficients, functional forms, and initial conditions uniquely defined for each problem (described in Appendix A no duplicate problems are included. Solutions are generated by navigating through set of possible cases during the algorithmic problem-solving strategy. Each mathematical step is embedded in explanatory text so HA MARDMATH solutions match the style and rigor of traditional problem set solutions. The main results for all problems included in boxed environments in the solution explanations
4

---

Published as a conference paper at ICLR 2025
distinguish them from the rest of the text. This follows the formatting convention used in other mathematics datasets designed for LLM benchmarking, such as MATH (Hendrycks et al. 2021
Generate problem. Add coefficients powers, functional forms template polynomiat ODE, integra) SymPy expression.
Apply approximation
Generate appreximate analytical solutions solving expressions using SaiPy.
Problem
No
methods
dominant
Verily solutions are selt- consistamt assumplions valid.
Analytical solutions within
LaTeX format step by- solutions
in database?
Taylor
Expansion.
physical
answers.
arguments)
Figure 2: Flowchart detailing the data generation procedure for HARDMATH problems.
For each problem type, the dataset includes: 1) LeTx--formatee problem statements with prompts, 2) #TTX-formatted solution steps and final analytical answer(s). 3) demonstration of the accuracy the analytical results by comparing with numerical solutions, and 4) metadata descriptors of the problem and solution types. For every problem type, we select evaluation points in each solution regime and calculate the relative error between the analytical solution and the numerical solution at these points. Problems were included in HARDMATH only if their approximate solutions had less than 10% error from the numerically calculated ground-truths. For the polynomial root correction problems, also confirm that the corrections improve the original approximation. While manually verifying cach solution step-by-step is impractical for dataset of this size, our validation process ensures high level of confidence in the accuracy of the solutions provided. Verification of HARDMATH-MIN is performed by plotting analytical solutions against numerical ground truths for a range of values in each regime (see example in Appendix A. This semi-automated step provides an easy means for visual human-verificationn plots will directly reveal whether analytical and numerical solutions correspond in the correct solution regimes.
3.3 PROBLEM TYPES
For all problem types discussed in this section, details regarding parameters used to generate problems and mathematical solution techniques are provided in Appendix A.2
3.3.1
NONDIMENSIONALIZTTON OF POLYNOMIALS
Nondimensionallzzttoon is technique simplify equations by reducing the number of parameters (Evans) 1972). In HARDMATH, the first type of polynomial used for nondimensionalization demonstration contains symbolic coefficients and is of the form
" 02x 03, n1 >> n2 0.
(1)
Nondimensionnalizaton converts this the form cy n² 1. The second type contains numerical coefficients and of the form
+aj t0 + a2x
+ a3: n1 > 12
which can be simplified to ey + nn + given a specific numerical value e.
3.3.2
POLYNOMIAL ROOT-FINDING
Exact formulas exist for quadratic, cubic, and quartic equations, but deriving them for quintic or higher-order polynomials is not possible (Stewart) 2015). HARDMATH includes approximate root-finding examples for higher order polynomials of the form ex =x + (example in Appendix
A.2.2
The goal is to solve for roots in terms of using the method of dominant balance for small and large positive regimes.
3.3.3
POLYNOMIAL ROOT CORRECTION TERMS
The use two-term dominant balances Seaances-uuchtae as in the previous problem pege-neleets terms and introduces error. We calculate correction term 8 to reduce this error. Suppose the true roots
5

---

Published as a conference paper at ICLR 2025
I² of polynomial given ==) x(c) - 6, where is approximation to the root and d is the error term. Plugging the roots x (c) = F(c) + into the polynomial allows one to use Taylor expansion of around to solve for the correction 8. Appendix A.2.3 shows full worked solution.
3.3.4 NONLINEAR ORDINARY DIFFERENTIAL EQUATIONS
We generate nonlinear third-order ODEs for which there do no exist exact analytical solutions and provide approximate formulae for small and large z regimes, where the small & regime near 0 and the large regime typically involves the solution diverging (example in AA.2nd) The method robust for higher-order problems, but for simplicity HARDMATH includes only third-order ODEs.
3.3.5 TRADITIONAL INTEGRALS
We consider integrals of the form T(e) 00 2+P(a) dr, where P(x) is an arbitrary polynomial. HARDMATH provides approximations of each integral in three regimes: small, intermediate, and large c. A full example is in Appendix (A.2.5
3.3.6 LAPLACE INTEGRALS
We consider integrals of the form ((x) = Sn ((())==/((d)) which can be approximated using Laplace's & Method when is very large because the integral's value is dominated by the region around
to (Bender & Orszag 2013). Depending on where the minimum is, the approximation is either
2π
or ((x) g(to)e g(to)e f(
((x) Rs g(to)e== ((())
x))'((()l
See and Appendix A.2.6 for examples of Laplace integral problem with solutions.
3.4 WORD PROBLEMS IN CONTEXT
One motivation for creating HARDMATH is to help LLMs recognize and solve problems where approximation techniques needed. To evaluate how LLMs perform such problems in realistic scenarios, we develop smaller dataset of 40 manually-generated word problems (example in Box 2). Although this dataset is smaller than our hand-verified evaluation set, it is large enough evaluate the effect of additional context in the problem statement on LLM accuracy.
Box 2. Sample Word Problem with Context
The density fish at different points along a certain path in lake can be modeled as (e + x² + - where represents the distance from the shore in kilometers (ranging from to 100 km), and represents environmental factors that affect the fish density. To study the total presence of fish along the path, develop approximate analytical formula for I(e) given below:
100
1
1(c)
de.
c +
3.5 AUTOMATIC FENERATION OF CONTEXT FOR WORD PROBLEMS
Recognizing that manual context crafting for these problems is Lbbor-intensive and lacks scalability, we conduct preliminary experiments on automatically generating contexts for word problems using powerful dlosed-source L.M,, such as GPT-4o. The generation process involves two steps: (1) We create foundational set of mathematical problems and solutions following the methodology in Fig. 2 These problems serve as starting point, capturing essential mathematical formulation without context. 2) We then prompt the LLM to generate real-world contexts to embed these problems and solutions based on a specified domain seed (e.g., physics). To ensure physical plausibility, a secondary verification LLM checks parameter ranges (e.g., energy must be negaaiegt and assigns 0- plausibility score. A score around 0.5 indicates that while no hard violations were found, the verifier cannot fully assess violations of less stringent domain priors Only plausibity scores >0.5 are thus

---

Published as a conference paper at ICLR 2025
considered. Example prompts for both generator and verifier are available in Appendix A.2.7 Table Using ODEs as a demonstration, we batch generate 30 problems with [domain_sset seed] physics. Appendix A.2.8 shows example of the original vs. context-embedded problem side by side, and A.2..
3
Appendix A.2.9 6a demonstrates the set of generated problems has good context diversity, covering different sub-fields physics. Moreover, most problems' plausibility scores >0.5, indicating queried math equations make good sense under generated contexts 6b).
While we recognize limitations in capturing more subtle, Nooainn-speciiie priors, this approach offers a promising step toward automating the generation of applied math problems in real scientific contexts. We plan to refine these methods in future work.
4 EVALUATION
4.1 EVALUATION PROTOCOLS
We conduct evaluations of various LLMs on HARDMATH-MINE carefully curated subset of 366 problems that matches the statistical composition of HARDMATH (Fig. D This smaller dataset is designed to optimize computational resources while retaining a sufficient number of questions to ensure consistent and reliable testing outcomes, thus maintaining the integrity of our evaluation. The evaluation focuses on four distinct problem types: 1) Nondim includes nondimensionalization in symbolic and numerical form; 2) Roots includes polynomial root-finding; 3) ODEs includes nonlinear ODEs; and 4) Integrals includes traditional and Laplace integrals. The input prompt for each problem contains the essential problem setup and detailed description of the question. Additionally, hints specific to each problem type are provided to guide the format of the answer. When few-shot prompting is used, it adds a fixed of paired problem-solution examples from the corresponding problem types. Example prompts can be found in Appendix A.3.1 Table 7
We evaluate model-generan generated responses by scoring them for accuracy using a combined protocol of automatic final answer assessment and procedural LLM-based grading. The automatic assessment follows methodology from Hendrycks et al. (02T), where models are prompted to enclose their final answers using the 1 [boxed ) command (Table Evaluation then compares the model's output within the (boxed } command to the dataset solution. To handle different mathematical expression formats, we implement both SymPy-based equivalence checks and numerical evaluations.
In addition to the standard automatic assessment of final answers, we develop a novel procedural grading approach leveraging LLMs, tailored to the unique evaluation challenges of our dataset: 1) Some problem types require complex, multi-step solution procedures (c.g. determining critical point in Laplace integral approximation) where a single cut-off criterion at the final answer cannot capture the full spectrum of model performance. Thus, grading intermediate steps in the solution procedure is necessary for comprehensive assessment. 2) HA RDMATH targets the models' ability to make human-like abstraction and approximation judgments. Some problem types allow narrow range of solutions rather than a single exact one, as long as the reasoning is self-consistent and the final result falls within certain threshold to numerical ground truth.
Inspired by LLMs' ability to generate consistent ratings for response content and style (Hackl et al., 2023), we employ GPT-4o as a procedural grader. The model is prompted with ground truth answer key and grading rubrics adapted from course grading guidelines for each problem type (example grading prompts in Appendix T.3.2Tabllef. (). We manually verify a subset of grading responses and found that LLM-based grading is closely aligned with human grading. Average score adjustment for each model and problem type is summarized in Appendix A.3.3 Table 6 We implement this procedural grading alongside automatic answer assessment for the problem types Roots, ODEs, and Integrals.
4.2 MODEL CHOICE
We compare the performance of several closed- and open- models on HARDMATH in zero- and few-shot settings with the Chain-of-Thoughh (CoT) (Wei et al. (023) prompting. Closed-source LLMs include GPT-3.5 (Radford et al., 2018 2019; Ouyang et al., 2022). GPT-4 (Achiam et al.
2023) and ol-mini (OpenAl| 2024a), open-source L.MMs include Ilama3 [A] 2024 and Code
7

---

Published as a conference paper at ICLR 2025
We believe this subset of models to be representative of current LLM capabilities. We provide the prompts and hyper parameters for LLMs evaluations in Appendix A.3.4 Table
Meta 2023),
hyper-parameters
4.3 QUANTITATIVE RESULTS
We present the accuracy of the models and prompting settings for each problem type and the combined evaluation set (Table 2 Figure 7) Few-shot CoT prompting significantly boosts performance for all models, with o1-mini and GPT-4 showing the greatest improvement, consistent with (Wei et al.) 2023) (Figure 7a).. Interestingly, although the ol-mini official prompting guide recommends simple prompting over CoT (OpenAl 2024b), we observe fairly large performance increase for all problem types at 5 shot CoT compared to 0 shot. Performance increase with prompting behavior also shows problem type-specific patterns: Figure@demonee demonstrates that performance saturates quickly for harder problem types like ODEs. The varying performance increases among different problem types may be due to different error modes in model answers, which we discuss in the following section. It's notable that ol-mini, though with much smaller parameter size, shows considerably better performance at all tested shot levels, confirming its optimized ability for STEM reasoning (OpenAl, 2024a).
Among closed-source models, ol-mini with 5-shot CoT prompting achieves the highest overall accuracy of 62.3%. GPT-4 at 5-shot CoT scores only 43.8%. Among open-source models, Llama3- 8b with 5-shot CoT prompting achieves the highest overall accuracy of 20.2%. We discuss the performance of these representative models--llmiin GPT-4 and Llama3- on HARDMATH- MINI in comparison with established datasets, including GSM-8K (Cobbe al. 2021), MATH (Hendrycks et al., 2021), and more advanced mathematics datasets like GHOSTS (Frieder et al. 2024).
Llama3-8b achieves a test accuracy of 30.0% on the MATH dataset with 4-shot CoT and 79.6% on the GSM-8K dataset with 8-shot Co'T prompting (AT 2024). Testing Llama3-8b ATR results in an overall accuracy of 20.2% with 5-shot CoT prompting. GPT-4 (gpt-4-turbo-2024400000
reported to achieve 72.2% accuracy on the MATH dataset with O-shot CoT prompting (OpenAI 2024) and 92.0% on the GSM-8K dataset with 5-shot CoT prompting (Achiam et al. 2023). On GOOMINIGHOSTS dataset, which also covers graduate- mathematics, GPT-4 reaches an average score of 4.15 out of 5. We test GPT-4 on our HARDMATH-MINI dataset and obtained an overall accuracy of 43.8% with 5-shot CoT prompting-
Finally, we include results on OpenAl's new ol-mini, which is reported to achieve 90.0% accuracy on MATH-500 with 0-shot CoT (OpenAl| 2024a). Testing ol-mini on HA ADAMAATT-MNN reveals a significant performance increase compared to results other models on some (e.g. Nondim) but not all problem types. Overall accuracy with shot CoT reaches 62.3%, substantially lower compared to performance existing mathematics benchmarks. This indicates that the HA HARDMATH benchmark consists of problems that are still challenging and unfamiliar to even the most performant LLMs developed specifically for advanced reasoning.
4.3.1 EXTENSIONS TO WORD PROBLEMS
To assess LLM's ability to solve similar applied math problems in real science and engineering contexts, we also test GPT-4 (best performing model with stable version) on set of 40 hand-crafted word problems that included mixture of Nondim, Roots, ODEs, and Integrals. We avoided additional prompt engineering, omitting the problem-specifie hints listed Table This evaluation resulted in an overall accuracy of 28.1% 1%. We plan to leverage the automated generation method as a basis to expand the number of word problems for future work.
4.4 FINE-GRAINED RESULTS
In addition to reporting the summarized test accuracy, study the detailed breakdown of model responses at different correctness levels and summarize specific error modes of LLMs solving these challenging applied mathematics questions. This analysis helps us compare performance nuances and understand reasoning paths by model, prompting technique and question type.
We first break down model performance by percentage of correct, partial and incorrect responses (Figure This analysis reveals how few-shot prompting enhances model performance across varying problem types but through different strategies. Overall quantitative results already show that ODES
8

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation Accuracy (percentage) on the HARDMATH evaluation set.
Model
ALL
Nondim
Roots
ODEs
Integrals
Closed-source models
GPT-3.5 shot)
6.04
5.05
17.2
1.39
3.33
GPT-3.5 shot CoT)
14.2
6.11
29.3
6.94
18.2
GPT-3.5 shot CoT)
24.6
24.3
35.0
16.2
23.1
GPT-4 shot)
14.0
6.04
33.7
7.87
14.9
GPT-4 (1 shot CoT)
37.6
36.5
52.8
15.9
40.5
GPT-4 (5 shot CoT)
43.8
48.6
57.3
21.7
41.4
ol-mini (0 shot CoT)
29.8
38.1
24.3
10.2
32.5
ol-mini (5 shot CoT)
62.3
84.5
62.1
30.6
46.5
Open-source models
1lbma3-8b (0 shot)
3.67
0.50
11.5
4.63
2.52
Llama3-8b (5 shot CoT)
20.2
17.9
17.1
12.0
28.1
CodeL 13b (0 shot)
1.94
0,00
8.73
1.85
0.50
CodeL 13b (5 shot CoT)
9.79
8.41
13.1
9.7
9.57
CaT
01-mini
Marma 3b
Nondim
Roots
0DEs
Integrals
Nondm
Roots
0D6s
Integrals
Nendm
Roots
ODEs
Integrals
Parthal
Correct
Figure 3: Percentage correct, partial, and incorrect responses for ol-mini, GPT-4 and Llama3, prompting conditions, and problem types.
are comparatively harder for all models while Nondim problems appear to be the easiest (Figure 7b). For hard problems like ODEs, full correctness is rare. Correctness level analysis shows that models tend to increase partial credit responses with CoT prompting, as they struggle to solve the problems entirely but manage partially address them-iin in this case, starting with the easier small regime solutions. In contrast, for simpler problems like Roots, advanced models like ol ol-mini and GPT-4 get more fully correct responses with increasing CoT shot number, demonstrating the models' ability to understand the approximation reasoning procedure fully (Fig. 33.
Second, we summarize the error modes partial and incorrect responses better understand the model's 'ss reasoning pitfalls. Specifically, we want to dissect how CoT changes model performance on the level of detailed errors. Figure uses GPT-4's responses at vs. shots on problem type Roots analysis example. This illustrates how 5 shot CoT prompting significantly alters the error structure compared to shot. The most common eиror mode inco-inecrreetly setting up dominant balance considering only the leading term- diminiissss substantially. Instead, errors shift to more nuanced issues: 1) setting up correct dominant balances but missing certain cases, or 2) failing to calculate complex roots (examples of those error modes included in Box 3 and Appendix (A.4.2). This shift indicates that CoT prompting improves the model's understanding and application of dominant balance techniques, enabling it move beyond intuitive yet incorrect simplifications. We particularly curious to compare ol -l-mm'''' s eтrоr modes with existing models, given its specialization STEM reasoning. However, in the current evaluation, ol-mini sometimes returns only the final answer without showing intermediate steps, making it difficult to trace the source of errors. We therefore leave this exploration for future work.
3. Grading Response: Roots
Consider the polynomial ((x) ex 1 of the polynomials the limit of small positive and large positive Only single term approximation the root is required.
Question:
+ -1 1. Find approximate expressions for all roots

---

Published as a conference paper at ICLR 2025
66.1%
27.0%
0.0%
6. 6.5%
12.7%
50.8%
27.45 14.4%
correct incorrect dominant balance terms
missing dominant balance cases drop imaginary roots
GPT4 O-shot CoT
GPT4 5-shot CoT
Figure 4: GPT-4 error modes for problem type Roots vs. shot CoT prompting
Grading for Small Positive E: Model Response: For small : [1, 1,, -1) Ground Truth:
For small positive [-v-
- iV
1, 1, -
The response only includes the roots from the balance B C = 0 and completely misses the roots from the balance A + B 0. Therefore, score for small positive is 0.5
5 CONCLUSION
We introduce HARDMATH, a new dataset covering several problem types from an advanced applied mathematics course that can be used to benchmark LLMss mathematical capabilities and perform model developments, including fine-tuning. This dataset consists 1060 examples, and we additionally include 366 human-verified examples in HA HARDMATH-ME "problems context' that we use to evaluate various leading LLMs. HARDMATH is unique in several ways. First, there do not exist large-scale mathematical datasets covering problems of similar difficulty from applied mathematics. Second, HARDMATH'S problems and solutions are algorithmically generated, with automatic numeric validity checks and easy visual means for human- veriannvrriiiiation. meaning that one could produce datasets of arbitrary size using our framework. This feature of HARDMATH especially unique, since most existing mathematical datasets require manual problem-settie setting or curation from existing sources (many of which are not publicly accessible).
and 40 human-verified
Our evaluation highlights that while few-shot CoT prompting significantly improves model perfor- mance, especially for models like ol-mini and GPT-4, the overall accuracy on HARD HARDMATH-MINE problems remains much lower compared to other existing benchmarks. This suggests that our dataset poses unique and challenging tasks that go beyond the boundaries of current L.M capabilities, particularly approximations -oriented mathematical reasoning. These findings emphasize the need for further improvement in LLMs to address hard math problems.
Our evaluation results use HARDMATIE COOMAATH-MILE MINI as a comprehensive test set: however, future work will fine-tune LLMs on the larger HARDMATH improve performance. Additionally, while we have evaluated several frontier models, we plan to extend our evaluations to even more LLMs as they become available. This expanded evaluation should provide more detailed insights into performance disparities across different models, further advancing our understanding LLMs' capabilities in handling complex mathematical reasoning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank the students who participated in the initial stages of the Fall 2023 AM 201 final project. Thanks also the Harvard Medical School Research Computing Consultant Group for their consult- ing services, which facilitated the computational analyses detailed in this paper.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprini arXin:2303.08774 2023.
Meta AL Meta llama 3, 2024. wtt.ttos://a. htt.tt//ss.aaaeaaaaccco/////aaaa cessed: 2024-06-03.
Ac-
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorsk,, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- formalisms. In Proceedings of NAACL-IILT, pp. 2357-2367, 2019.
Daman Arora, Himanshu Singh, et al. Have Iims advanced enough? challenging problem solving benchmark for large language models. Proceedings of the 2023 Conference on Empirical
Methods in Natural Language Processing, pp. 7527-7543, 2023.
Carl M Bender and Steven A Orszag. Advanced mathematical methods for scientists and engineers I: Asymptotic methods and perturbation theory. Springer Science & Business Media, 2013.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4168, 2021. htlllttpss//arx....org////
14668v1
John Evans. Dimensional analysis and the buckingham pi theorem. American Journal of Physics, 40112)21155-1822 1972.
Simon Frieder, Luca Pinchetti, Ryan- Rhys Griffiths, Tommaso Salvatori. Thomas Lukasiewicz. Philipp Petersen, and Julius Berner. Mathematical capabilities of chatgpt. Advances in Neural Information Processing Systems, 36, 2024.
Veronika Hackl, Alexandra Elena Müller, Michael Granitzer, and Maximilian Sailer. Is gpt-4 reliable rater? evaluating consistency in gpt-4's 's text ratings. Frontiers in Education, 8, December 2023. ISSN 2504-284X. doi: (0.3389/feduee22233, 1272229. URL http::tp:// doi wwwp:/doi...ggg.../
3389/ 2023. 1272229
Dan Hendrycks and Alexandr Wang. Submit your toughest questions for humanity's last exam, 2024. URL httths::ssswwww...e safe. wwaeff..ai/bll htt/bll//hhhaaaaat////////////// wast://mmmaintttttttt-sssssas-ee -exam Accessed: 2024-10-01.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Truck Dalasets and Benchmarks. NeurlPS, 2021.
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. Mathbench: Evaluating the theory and application proficiency of Ilms with a hierarchical mathematics benchmark. arXiv preprini 1222:2 2024.
Meta. Code Iaaa: Ai for coding, 2023. URL httphhttt//bbbutt fb. wtt.c//////w//22///2
code 11 lama- ai for- codiig/oonng//// Accessed: 2024-06-03.
Aaron Meurer, Christopher Smith, Mateusz Paprocki, Ondfej Čertik, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Štěpán Roučka, Ashutosh
11

---

Published as a conference paper at ICLR 2025
Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3te103, January 2017. ISSN 2376-5992. doi:
10.7717/peerj-cs.103. 103. URL ottpa//e///ool.ooo////7/////1/1/
Netmind.AL. Odyssey-math.
tree/main ma in 2024. Accessed: April 22, 2024.
OpenAL. Simple evals.
2024. Accessed: 2024-06-03.
OpenAL. Introducing openai opeeeeww 2024.
URL
httpss//openaa..ommmnnkk
intrrediccag--ooeeaal-------e--e Accessed: 20-4-10-01.
OpenAl.
Openai ol-mini, 2024a.
URL
/ttps:://penai.comm
openai-ol htw.:aaai---aaaaaaaa---------aa- reasoning/ Accessed: Septem-
ber 30, 2024.
OpenAl. Reasoning models, 2024b. URL https::/pppaatottt////
wwt.rrmmmppaaa.coommm wtw.lltfomm.peeaaiccomoooooo ccme/ooccco//ccs///// works Accessed: Oct. 1, 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-traine rre-training, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019. URL htlLhhtps//eet htaps:///pps semant
Grq/CorpusID 10:10222:53:10 160025533
Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John Nay, Kshitij Gupta, and Aran Komatsuzaki. Arb: Advanced reasoning benchmark for large language models. arXiv preprint arXiv:2307.13692, 2023.
lan Stewart. Galois Theory. CRC Press, Taylor & Francis Group, Boca Raton, FL., 4th edition, 2015.
ISBN
Version Date: 20150112.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Coumapeaa,, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et Scipy 1.:: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-277 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of of-touht prompting elicits reasoning in large language models, 2023.
12

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
SEMI-AUTOMATED SOLUTION VERIFICATION
Figure provides a visual comparison the numerical and approximate solutions for the Laplace Integral example in Appendix AA2.66 over a large domain. This allows for semi-automated human verification that analytical solutions correspond well with numerical ground truths, method used to verify the problems in HA RDMATH-MINN.
10
1M--Ja1-2200 2.tan Pittl st.4 24kart L.be"ic 164
109
10
105
Numerical Solution
10
[x]
0.57744l1
10a
10
10
10
x
Figure 5: Visual comparison of numerical and approximate analytical solutions sample Laplace integral problem for solution verification.
A.2
IMPLEMENTATION AND METHOD DETAILS GENERATION
The following subsections detail the process used to generate the problems and solutions for each problem type.
A.2.1
NONDIMENSIONALIZATION POLYNOMIALS
The first nondimensionalization sub-type generalized by varying the integer values for the degrees M1 and within the range ( n22 < nn < 10, while keeping 01, 02, > O symbolic. Solutions to these problems express the dimensionless parameter in terms of these three coefficients.
Sample Symbolic Nondimensioonalzzation Problem and Full Solution
Problem: Nondimensionalize the polynomial
01 x + a2 + 03
into one of the form 10 + yd + 1. Express e as a function of aj, ag, and ag.
Solution: We begin with the substitution
43
T.
y
a2
This gives the expression
10
01y
10
ag
03y
+ 03
ag
Divide by the coefficient remaining in front of the constant, leaving us with the nondimen- sionalized polynomial with coefficients in terms 01, 02, and 0z:
a1y
10
es
02
0.
13

---

Published as a conference paper at ICLR 2025
By inspection, that
10
01
42
The second subtype implements integer numerical values for the coefficients ai, 02 that are are randomly chosen from [-10, 10].
Sample Numeric Nondinensioonaizzaton Problem and Full Solution
Problem: Nondimensionalize the polynomial
P(x) = 2x + 8x² +
into a polynomial of the form ey² = y² 1. Solve for c.
Solution:
For now, we ignore the numeric values of the coefficients and instead call them 01. 02, Our polynomial is then:
a1 + 022 + ag.
Use the substitution
a3 I 3/
which gives the expression
3
03 02
a1y²
azy²²
a3.
Divide all terms by the coefficient remaining in front of the constant term, giving the nondimensionalized polynomial with coefficients in terms of a, 02, ag:
a1y²
+1
a3
Substituting the known numeric values for a1 02, ag (using their absolute values as we have already accounted for sign), we get:
25 10y
+ +
1024
From inspection of this nondimensionalized equation, we can now identify
25v10
RJ 0.08.
1024
A.2.2 POLYNOMIAL ROOT-FINDING
As with the nondimennioonalizatoon problems, degrees in the polynomial are randomly generated with maximum order and 0 n2 n1. See full problem and solution below.
Sample Polynomial Root- fhot-finding Problem and Full Solution
Problem: Consider the polynomial
P(x) c 5 1.
Find first order approximations for all roots of the polynomials in the limit of small positive and large positive e.
14

---

Published as a conference paper at ICLR 2025
We begin by equating the polynomial to zero to solve for the roots: P(x) x 0. This problem can be rewritten in the form A + B + C' 0, where: A = ez B x²;;
Solution:
=
This problem has analytical solutions, we find approximate solutions to the roots by considering the three possible dominant balances. For each dominant balance, we find the roots of the resulting equation and evaluate whether each balance is self-consistent for small or large positive
We start with the balance + B O, assuming that |C| negligible when compared to |A] and [B] Solving this for x in terms of then gives us non-zero root:
To verify that these roots are consistent with the assumption that |A|, [B] |C), we substitute these found roots back into the terms A, B, and C and compare their magnitudes. Using this method, we find that it true that these roots valid for small e, while validity for large is false.
Therefore, these roots are valid in the limit of small positive e only.
Next we examine the balance B + C = O, assuming that |A| negligible when compared to [B] and |C]. Solving this for æ in terms of gives non-zero roots:
1 1 x² =0
1
V5
i225
10
1
V5
V-10
2
+
+
4
4
4
4
4
V5 iv 10
2/5
V5
iv
10
2
4
4
To verify that these roots are consistent with the assumption that B |C| > |Al,, we substitute these found roots back into A, B, and C' and compare their magnitudes. Using this method, we find that it is true that these roots are valid for small e, while validity for large e is false. Therefore, these roots are valid in the limit of small positive only.
Finally, we examine the balance A + = 0, assuming that |B] is negligible when compared to |A] and [C]. Solving this for x in terms of gives 6 non-zero roots:
ex + 1=0
(-1
Vii
2
(-1+ V
- (1 31)
(1 + 3i)
2
To verify that these roots are consistent with the assumption that |A]. |C* > BB, we substitute these found roots back into A. B, and C' and compare their magnitudes. Using this method, we find that it is false that these roots are valid for small e, while validity for large e true. Therefore, these roots are valid in the limit of large positive only.
By the Fundamental Theorem of Algebra, a polynomial of degree 6.0 has exactly 6.0 roots. We have found 6.0 roots that are valid in the limit of small positive and 6.0 roots valid in the limit of large positive e. Our method therefore provides a complete solution to the problem, finding the correct number of roots in each regime.
15

---

Published as a conference paper at ICLR 2025
The roots of P(a:) for large positive are
(-1 V
2
(11+ 3i)
Vi)
(1+ v3)
2
2
and the roots of ((x) for small positive c are
1
V5
iv
2/5
10
55
V V-10
10
2
1,
4
4
4
55 1
iv 2
V5
iV
10
2
4
4
4
4
4
A.2.3 POLYNOMIAL ROOT CORRECTION TERMS
The true roots x of polynomial are given by x" (c) = ((c) +6, where E is our existing approximation to the root found in Appendix A.3 and 8 is the error term. This requires us solve
<(T + 8) - ( + 8) + 1
by equating coefficients e terms of the same order, as detailed in the worked solution below.
Sample Numeric Nondimensiionalizttion Problem and Full Solution
Problem: Consider the polynomial
P(x) = Ex - +
Find approximate expressions for all roots the polynomial in the limit of small positive e and large positive Use series expansion to calculate improved formulae for these roots to order i.e. calculate (1) corrections for each root.
Solution:
The root calculation in this problem follow the same method as those demonstrated in the A.3, they has been omitted here. We include only correction term calculations for the sake of brevity.
Note:
We now need to calculate correction terms for these roots to give us better approximations. We consider the ansatz that the root given by + 8, where the correction term 8 the sum of higher order terms of c that we initially neglected in our approximation F. By definition, 8 We plug this ansatz into the polynomial and perform a series expansion in 8. We keep terms only up to (1)) in 8. Then, we set the expression equal to 0 and solve for 6.
Regime 1: valid for small
Root 1:
I
Z
+
Substitute this into P(x) for and equate to
8 8
+ 1
+1
We then expand this expression to get
(1)
8- - 382 V
+ 28 -
+ V
+1
16

---

Published as a conference paper at ICLR 2025
and represent series of ((1) in discarding higher order & terms
28 (:)
+Ve V O.
1
We then solve the expression for the correction 8 ((1), and get
6 2
Root 2: !
V +
I +
Substitute this into P(x) for and equate to 0:
5 + (6 -
We then expand this expression to get
+ 28 + $
-
8 + 362 30
eV
and represent series of (1) in discarding higher order d terms
()) -
- +1=0
25
We can then solve the expression for the correction 5 to O(1), and get
(2)
1
8
2
2
Regime 2: valid for small Root 1:
F 7 5
Substitute this into P(x) for and equate to 0:
c + 1)² 0.
We then expand this expression to get
8 + 38 + 35c 8 + ==
and represent as a series of ((1) in discarding higher order 8 terms
8 (3c 1) + O.
We can then solve the expression for the correction to O(1), and get
8
36
Regime valid for large Root 1:
F 5
17

---

Published as a conference paper at ICLR 2025
Substitute this into P(x) for and equate to
-8 te 8 +
+1=0 0.
We then expand this expression to get
3 (
8 + 35 and represent it series of (1) in 6, discarding higher order 8 terms
30E
8
0
-ー- - 0.
8 3r
-1
We then solve the expression for the correction 5 ((1), and get
8 RJ
3E
8-2((11-3)) 2
Root 2:
I+ō= -- - V3i)
+
2
Substitute this into P(x) for and equate to O:
( 8+ V-E /3i)
(-1 V3i
-8
1 0.
2
We then expand this expression to get
38
3/3i0² e
38c (-2)
83
2
3v 3 3ide (-1)
/ V
8
0
2
2
and represent series of O(1) in discarding higher order 8 terms
(-1)
3 3v3ie
(-+)
3 1 +
8
0.
2
2
We can then solve the expression for the correction ö to O(),, and get
(1 + V31)
8
36 1 3/ (-2) +2
(-E(-1+ 2
Root 3:
- 1 (-1+ V3i)
-
2
18

---

Published as a conference paper at ICLR 2025
Substitute this into P(x)) for and equate to
(-1 V3i)
(-1+ v3i)
-e 6
+1=0 0.
2
2
We then expand this expression to get
30
3v 310e
3 (-2)
8
+
2
3v 3
3ide
(-2)
v
+
0
2 and represent as a series of (1)) in 6, discarding higher order d terms
2
3c 3-2)
3/
(-1)
v V
8
+
0.
2
2
We then solve the expression for the correction 5 0(1), and get
- (1 (1))3))
8 2
36 2- 3/e (- +2
A.2.4 ODES
We generate third-order ordinary differential equations of the form
y" (1())(²") + f2()()')) + fa())"+++ fa(z).
where (I)), fa(x), f4() rational functions with integer coefficients. The initial condi- tions are randomly selected integers from [0, 3]. The dataset excludes problems with a function of x dominant term because of the difficulty of deriving power law expressions in these cases.
Approximate solutions at small x can be derived using a Taylor series expansion (up to the third order) around 0. Solving ODEs in the large æ regime involves determining the two largest terms, assuming divergence at some large x², and solving the dominant balance between these terms to create power law approximation of the form
y(x)= = A(x" x)".
ODE Problem and Solution
Problem: Consider the following third-order ordinary differential equation:
y"
y" 24z + 6r + with initial conditions
+ 1/2
5z 222
+++
2
12r cos (x) + 11
(0) 1.00
y (0) 0.00
y²(0) 0.00
Find analytical expressions that approximate the solution of yxx at small and large x.
Solution:
19

---

Published as a conference paper at ICLR 2025
The dominant balance in the large regime is given by
d
dr
dx
We recognize that the solution of this ODE will diverge at finite x and that divergences typically follow power law of the form
a( v=a(r-x");
where x" is the divergence point. The divergence point can be determined by estimated by examining the numerical solution generated by code.
Plugging in the dominant terms we found previously yields the following equation:
op (p- 2) (p - 1) (x 11.45) = a² 11.45)2p-2
=
After substituting the derivatives, the equation is reorganized collect terms with respect to (xr - -x)). This leads to an equation where the coefficients and powers (x - x*) are equated on both sides. Simplifying the equation gives two separate equations, one for the coefficients and another for the powers (x - x²). There now a system of equations. where the coefficients' equation is
op (p - 2) (p = a²pp
and the powers' equation is:
= 2p 2.
Solving this system of equations provides the values of a and p. A valid solution is identified a and are both nonzero. Here, the solution for a and p is found to be:
a = 6, p p=
With these values, the analytical approximation for the solution large (near the divergence point) is given by
y = -6x - -1.45)
The approximate solution small also be solved used dominant balance, but one take advantage of the initial conditions and form Taylor series instead around x 0, which is given by
5'(0) x²
(0)
v(z) = y(0) + +0(() +
+
2!
3!
Plugging in the initial conditions, we get the following expression at small r::
13
y(+)
180
Thus, with rounding for clarity, the solution given by
v(x) = 1 13
x² -6(r
11.45)
180
A.2.5 INTEGRALS
The polynomial P(z) is randomly generated to consist of up to ten terms, where each term is power function of x with an integer power randomly sampled from and 20 and an integer coefficient sampled from to 10. The integration bound E [0, 100) is also randomly selected. This form ensures that the integral does not oscillate.
The height is approximated the maximum value of the integrand, which is 1. and the width be estimated as the distance over which the integrand decreases from its maximum value by a factor of 2, which implies that the width x obeys the equation
+ P(x) =
# P(x) =c.
2e
20

---

Published as a conference paper at ICLR 2025
In the regime of small c, the term with the smallest degree and are the dominant terms, and in the regime of intermediate e, the term with the largest degree and are dominant. There exists one more solution regime when the width of the integral exceeds the limits of integration, when is "very large." In this case, the integral is approximated by I/e, where T. is the integration range.
See full sample of traditional integral problem in Box 1.
A.2.6 LAPLACE INTEGRALS
Laplace integrals of the form [(x) - J ((()c==f(()))) dt assume that ((t > 0, is never a constant, and has an absolute minimum at a point ty either in the interior of or on the bounds of the interval [a, b].
The set of possible Laplace integrals [xx) in our dataset parameterized by four parameters: the bounds [a, 6]. gt)., ftt). and the sign in front of æ. To generate the dataset, the bounds for each problem were randomly sampled from the [-1, 0.9, ... 0.9, 1], and the sign was uniformly sampled from { 1-1,1]. The functions f(t) and g(t) were generated by randomly selecting linear combination of polynomials up to fifth order and basic trigonometric functions.
Our solution uses SymPy under the hood to find the minima of (() (or the dual annealing algorithm if SymPy fails to return the minima).
Sample Laplace Integral Problem and Solution
Problem: Consider the integral
0.3 (xx)
(-1.6t² - 0.5 sin (t)
+ 1.9)e e 2.5t 2.55- 100--008*+1.0 +1.46)
(2)
0.9
Develop an analytical
formula for [(x) that
accurate
as
x
+
oo.
Solution:
The integral is of the form
((x) - / g(1)e= =f(0) de
where a = 0.9, b= 0.3, g(t) = -1.6 0.5 (t) 1.9, and f(t) - 0.8t + 1.4t?. This means the limit that x - oc. In this limit, the integral will be dominated by the integrand near the maximum of f(t) within the bounds [-0.9, 0.3). So, simplify the integral, will expand the integrand around this maximum. In this case, we can find the maximum of f(t) = -2.5t 0.8 + 1.4t² on the interval analytically, We begin by looking for critical point(s) teris of ft) by solving ft( = - 1 2.4t² + 2.8t = 0 for t. This gives us evaluate f(t) at the critical
use Laplace's method to develop analytical approximation in
that terit = (- 0.66, 0). To find the maximum on this interval, point(s) lerit and the bounds -0.9 and 0.3. We take the / that gives the largest value. Here, this maximum lg = [0.66]] Since the integral is dominated by the value of the integrand near -0.66, we Taylor expand the integrand around this point.
1(x) = [[ (o((0.66) + (-0.66)) +..)
((-(0o-_0.)))(00005)))0 (6+0.66]2 (" ("(60.6))+...) dt (4)
But f'(-0. 0 by definition, so we can remove this term from the exponent. We can then approximate
1(x) I Φ( 9(-0.66)) ((f(-) (0) ("(-6.66)) dt,
(5)
which equals
(o- /" too [tto,00)² (-066))) dt
(6)
21

---

Published as a conference paper at ICLR 2025
We perform the change of variables u
[066666
0.66). rewriting the integral
-(-0.66)e 1 o 0.6011 (b+0.66)
2
x² dl
0..601
(a+0.66)
Since - x., we approximate this as
g( 0(-0.66)e==/(-0066)
2 x (
x²
dt
(8)
Solving the integral and evaluating, we find that
π 0.37
1(x) ~ -1.21
e
(9)
22

---

Published as a conference paper at ICLR 2025
A.2.7
WORD PROBLEM AUTOMATIC GENERATION SETUP
Table 3: Prompts for generating problem and solution contexts and plausibility verification.
Task
Task instruction
Question con-
Rewrite this (question_typel problem by embed- ding it within a plausible real-world (domain_seed, problem scenario, without changing the mathematical question the end.
textualization
Solution con- textualization
We provide above real-world domainn (domain_seed) seed) ques- tion, and its mathematical solution. Generate single introductory sentence before the solution that can connect the solution the real-world (doma (domain_seed) in_seed] context provided in the question.
Verification
We provide above real-world [domain__] _eedd) ques- tion. Review and verify its plausibility within the context of parameter ranges of this domain (e.g. energy value cannot be negative). Identify any inconsistencies or areas needing clarification to ensure the problem is realistic for this domain. Return a single float number as plausibility score (0-1) in LaTeX boxed format \boxed at the end.
A.2.8 WORD PROBLEM AUTOMATIC GENERATION EXAMPLE
Original vs. physics context augmented problem (ODEs)
Original Mathematical Problem: Consider the following third-order ordinary differential equation:
y
+ (1/)²+
3
+
3x
4x
4. 14x
5x + sin(x) 8
with initial conditions at O:
v(0) = 1.00
y (0) = 0.00
y"(0) = 0.00
Find analytical expressions that approximate the solution of y(x:) in the small æ and large regimes.
Real- World Context Problem: In a study of the dynamics of a newly discovered type of fluid, researchers are investigating the behavior fluid flow through a porous medium. The flow is described by third-order ordinary differential equation that models the velocity profile of the fluid, denoted by yz)), into account various factors such as the porosity of the medium, the viscosity of the fluid, and external forces acting on the fluid. The governing equation for the velocity profile is given by:
moves through the medium. The equation takes
y"
y
+ (y')*+ + 4- (4r² + 1) + 5 + sin - 8
y"
3x 4x 2
The initial conditions at the entry point of the medium, 0, are specified follows:
(0) = 1.00
y (0) 0.00
y²(0) = 0.00
23

---

Published as a conference paper at ICLR 2025
The researchers interested in finding analytical expressions that approximate the solution of y(z) in the small z and large regimes. These approximations will help in understanding the initial behavior of the fluid enters the medium and its asymptotic behavior travels further through the medium.
A.2.9 WORD PROBLEM AUTOMATIC GENERATION QUALITY EVALUATION
Thermal Conductivity
Battery Dynamics
Open Channel Flow
Turbulence
Damped Oscillations
Groundwater
Celestial Motion
Rotational Dynamics
Hydrodynamics Fluid Flow
Prequency
Tidal Forces
Asteroid Deflection
Thermal Resistance
Erosion Energy Storage
Gravitational Influence
spray Dynamics
Cooling Systems
Satellite Trajectories
orbital Mechanics
Electromagnetic Induction
0.4
0.5
0.6
0.7
0.8
Plausibility Score
(a) Diversity of context subdomains within the broader physics domain for auto-generated word problems.
(b) Plausibility scores distribution automatically generated word problems.
Figure Diversity and quality assessment of automatically generated word problems. Only problems above plausibility score threshold are included.
24

---

Published as a conference paper at ICLR 2025
A.3
EVALUATION SETUP
A.3.1
PROMPTS FOR RESPONSE GENERATION
Table 4: Problem type specific hints by Question and Answer Type
Question
Answer Type
Task instruction
Type
Nondim- symbolic
SymPy
Please answer the question requiring an answer a SymPy convertible formula containing variables and math operation expressions and provide the final answer, c.g., inside Latex boxed format \boxed ().
Nondim- numerical
Float (2)
Please answer the question requiring floating point num- ber with two decimal places and provide the final value, e.g., 0.80, 3.12, inside Latex box Vboxed }.
Polynomial Roots
SymPy List
Please answer the question requiring Python list contain- ing SymPy convertible formulas of variable e and math operation expressions and provide the final list, e.g., [c ] inside Latex boxed format \boxed }
ODEs
SymPy List
Please answer the question requiring Python list contain- ing SymPy convertible formula of y = f()) and provide the final list, e.g., ly 1- 1², y 6/(2 5)], inside Latex boxed format \boxed
Integrals
SymPy
Please answer the question requiring an answer in a SymPy convertible formula containing formulas of vari- able and math operation expressions and provide the final answer, e.g., x inside a Latex boxed format \boxed().
25

---

Published as a conference paper at ICLR 2025
A.3.2 PROMPTS FOR GRADING
Table 5: LLMM-based grading prompts by Question and Answer Type
Question
Answer type
Task instruction
type
Polynomial
SymPy List
Please take this response [esponse } and this answer key (answer key) and grade the response based on the following criteria: 1) Check both the small and large solutions. 2) For each solution, give full credit if it completely matches the elements in the answer key; give partial credit proportional to the number of matching roots between the response and the answer key; give no credit it is completely wrong. 3) For both partial and no credit briefly state the error reason. 4) Average the scores for the small and large epsilon solutions to obtain a final score between 0 and 1. 5) Give the final grading as float Latex boxed format \boxed ).
Roots
ODEs
SymPy List
Please take this response { (response) } and this solution { (answer key) and grade the response based on the following criteria: 1) Check both the small and large solutions. 2) For small regime solution, only give full credit if matches the formula in the answer key exactly; give no credit if it is doesn't t match the form. For large regime solution, give full credit if it matches the formula in the answer key exactly; give partial credit if it doesn't 't match but the numerical evaluation is not far from solution at this regime; give no credit if neither satisfies 3) Average the scores for the small and large epsilon solutions obtain final score between 0 and 1. 4) Give the final grading as a float Latex boxed format \boxed ).
Integrals (tra- ditional)
SymPy List
Please take this response (response) and this solu- tion (answer key) and grade the response based on the following criteria: 1) Check both the small and large solutions. 2) For each solution, give full credit if it matches the formula in the answer key; give no credit is completely wrong and briefly state the reason for the error. 3) Average the scores for the small and large epsilon solutions to obtain final score between and 1. 4) Give the final grading a float in Latex boxed format \boxed() ).
Integrals
SymPy
Please take this response { (response) } and this solution { [answer key) and grade the response based on the following criteria: 1) Check the large x final solution. 2) Give full credit if it matches the formula in the answer key; give half credit if the (response) ) get the checkpoint where correctly identifies by where f attains its maxi- mum and attempt performing Taylor's expansion around but the final answer is wrong; give no credit it is com- pletely wrong. 3) For both partial and credit briefly state the error reason. 4) Give the final grading as float in Latex boxed format \boxed }.
(Laplace)
26

---

Published as a conference paper at ICLR 2025
A.3.3 GPT GRADING HUMAN VERIFICATION
Model
Roots
ODEs
Integrals
GPT3.5 (0)
0
0
GPT3.5 (1)
0
-0.09
-0.02
GPT3.5 (5)
+0.02
+0.07
+0.02
GPT4 (0)
0
-0.02
0
GPT4 (1)
0
-0.04
-0.02
GPT4 (5)
+0.07
-0.07
-0.15
ol-mini (0)
+0.04
+0.05
0
ol-mini (5)
+0.05
+0.05
Llama3 9-bb (0)
0
0
-0.02
Llama3-a8 8bb (5)
-0.07
-0.02
-0.02
Codellama3-14b (0)
0
-0.02
Codellama3-14b
0
-0.02
Table 6: Average adjusted points using human judgment from GPT-based grading. Rows with score adjustments of 0.1 or more are highlighted in pink.
A.3.4
MODEL PAPER-PARAMETERS
Table 7: Generating parameters for various LLMs.
Model
Generation Setup
GPT-3.5
model gpt-3. 5-turb,, temperature 0, max _ex_tokens 4000
GPT-4
model cpt-4-turbo, temperature
max_tokens 4000
o1-mini
model = ol-mini, temperature 0, max_ = 4000
Llama3
model 11ama3: temperature 0
CodeLlama
model codellama 13b, temperature = 0
A.3.5 COMPUTING RESOURCE
Evaluations of open-source models on HARDMATH are conducted on high-performance compute cluster with a single Tesla V100 GPU (16GB vram). Evaluation one problem type typically takes less than hour.
27

---

Published as a conference paper at ICLR 2025
A.4
EXTENDED EXPERIMENTAL RESULTS
A.4.1
EXTENDED EVALUATION RESULTS
Roots
100
60
80
60
Accuracy (%)
ODE
Nandim
40
0
Integrals
0-shot
1-shot
S-shot
c1-mini
gpt-3.5
wddellam-133
ol-mini
gpt-3.5
codellama-13b
gpt-4
gpt-4
lama3-70 7b
Model accuracy breakdown problem sub- types all models with 5-shot CoT prompting.
(a) Model accuracy increases with shot numbers.
Figure 7: Model evaluation accuracy breakdown by shot number and problem sub-types. (a) evaluation accuracy for all models increases with shot numbers for CoT prompting with ol-mini and GPT-4 showing the most obvious improvements; (b) evaluation accuracy breakdown for all models on all problem sub-types under the 5-shot CoT condition.
Nondimensionalizztion
ODE
100
100
80
80
Accuracy (%)
60
Oni-ssho
few-shot
Ori-shoot
Few-shot
40
40
20
20
shot
Zero
8
Number of Examples in CoT (k)
Number of Examples in CoT (k)
~128B Params (reasoning optimized)
175B Params
8B Params
5008+ Params
13B Params
Figure 8: Model performance scaling upon few-shot prompting (0, 5, 10) for problem type Nondim and ODEs
28

---

Published as a conference paper at ICLR 2025
80-
6P13.5
codellama-
Nondim
Roots
ODEs
Integrals
Nondim
ODEs
Figure 9: Percentage of correct, partial, and incorrect responses for GPT-3.5 and CodeLLlama using CoT.
A.4.2 EXTENDED ERROR MODE ANALYSIS
Problem 136 Grading Response: Roots (error mode: incorrect dominant balance setup)
Consider the polynomial P(x) ex x² 1. Find approximate expressions for all roots of the polynomials in the limit small positive and large positive e. Only a single term approximation the root is required.
Question:
Grading for Large Positive E:
Model Response: For large When is large, the term ea. dominates the polynomial for most except for potentially small æ where It might still be relevant. However, focusing on the dominant term gives: P(x) Rs ex.. Setting this equal to zero for approximation (acknowledging that we' re looking for non-trivial roots of the overall polynomial, but focusing dominance guide this term alone since N = doesn't 't provide roots that satisfy the original polynomial's requirement for non-zero roots.
approximation), suggests that don't 't get meaningful roots from
Reference Solution: Finally, we examine the balance A + C = 0 assuming that B is negligible when compared to A and |C|. Solving this for æ in terms of gives nerozero
(-1 V31) v/2 (1 /3i) V/
roots: ex. - = 0 implies
T
To
2
verify the roots found consistency with our initial assumption, we check that |A [CT [B] holds for this root by substituting found root expression for back into A, B and C' and comparing their magnitudes. Using this method, we find that validity for small is False and validity for large is True.
Grading:
The response wrongly sets the dominant balance as A 0 and does not provide
concrete list of roots for large Therefore, score for large positive e is 0.0
Problem 133 Grading Response: Roots (error mode: drop imaginary roots)
Consider the polynomial P(r) cx + x² 1. Find approximate expressions for all roots of the polynomials in the limit small positive and large positive Only single term approximation to the root is required.
Question:
Grading for Large Positive E:
Model Response: To find the approximate expressions for all roots of the polynomial the limits of small and large e, we will consider the two cases separately. For Large When is large, the term ex dominates the polynomial. Thus, the polynomial can be approximated by ignoring the x? term, leading to cx
P(x) = cz z²
0 Solving for I gives us an approximation for the root: x RJ (1) The other roots in this limit are complex and given the scope, we'll focus on this real root approximation.
Reference Solution: Finally, we examine the balance A+C C 0. assuming that [B] is negligi- ble when compared to |A] and [C]. Solving this for x in terms of gives us 3 non-zero roots:
29

---

Published as a conference paper at ICLR 2025
(-1 - 3i) (-1+ v
C2
implies
z
To verify the
2
roots found for consistency with initial assumption, check that [A], C >> B holds for this root by substituting our found root expression for x back into A, B and C and com- paring their magnitudes. Using this method, we find that validity for small is False and va- lidity for large is True. Therefore, these roots are valid in the limit of large positive only. Grading: The response makes additional assumptions to drop the two other complex roots for large regime without the question asking it to do so. Therefore, score for large positive 0.5
A.4.3 EXTENDED EXAMPLE GPT-40 GRADING RESPONSES
Grading Response: ODEs (incorrect)
To grade the response, we will compare the provided solutions in the response with the ground truth solutions in both the small æ and large regimes.
SMALL D REGIME
Response: u(z) = 5e 18
Ground Truth: y(x) = 5
The response and the ground truth solution for small are significantly different. The ground truth solution is polynomial in z, while the response includes exponential term. They do not match all, so the solution for the small regime is completely wrong.
LARGE & REGIME
Response: y(z) = "Assumed rapid growth form due to y/ term."
Ground Truth: y = \30((z 2.68) + (1.84)
The response does not provide specific analytical expression for the large x regime, while the ground truth does provide a specific form. The response mentions rapid growth due to 8/ but without a clear analytical form, this response doesn't 't match the ground truth solution. This indicates that the solution for the large regime is also completely wrong. Given the evaluation criteria, both solutions do not match or are incomplete compared the ground truth. Therefore, each solution gets no credit. The final grading is 0.0
Box 3: Example grading response for ODEs problem that gets zero point.
Grading Response: Integrals (correc))
The model response approximates the integral for small as
3/4
and for large e as
61/4
81/18 6 17/18
Now, let's compare these results with the ground truth solutions:
30

---

The model's result is Published as a conference paper at ICLR 2025
0.64
For small e:
The ground truth solution is [(e)
The model's result is
e..7
6//4 3/4
Both results for small essentially the same, with:
61/4 ~ 0.64
0.89
For large c:
The ground truth solution is T(c)
0.94
17/18
81/18
Both results for large are essentially the same, with:
81/18 0.89
Since the model's response closely matches the ground truth solutions in both regimes (small and large e) and provides essentially the same approximations, would give full credit 1.0 points.
Box 4: Example grading response for Integrals sub-type simple integral problem that gets full
Grading Response: Integrals (partial)
Answer Key
Setup and Formulation: it 1) correctly identifies the integral form; 2) uses Laplace's method; 3) identifies critical points and uses Taylor expansion around the point of interest; 4) performs change of variables to simplify the integral.
Key Steps and Approximations: it 1) expands the integrand around the minimum of f(t;; 2) approximates the integral by Tocusing on the dominant exponential term; 3) provides final analytical approximation for (xx x + oc.
3. Final Result:
I(x) = 0. 0.89 89
Model Answer
Setup and Formulation: it 1) correctly identifies the integral form; recognizes that the exponential term will dominate for large æ;; 3) focuses on the region where the integrand is most significant (near 0.7).
2. Key Steps and Approximations: it 1) uses the method of steepest descents heuristically without detailed Taylor expansion; 2) approximates by evaluating the polynomial part at t = 0.7 and considering the exponential decay; 3) recognizes the dominant behavior of the exponential term but does not perform detailed variable change or expansion.
3. Final Result:
[(x) 0.011 22266-7777
Comparison
Similarities: 1) understanding of dominance: both the answer key and the student correctly identify that the behavior of the integral for large is dominated by the exponential term; 2) identifying significant points: both recognize that = 0.7 is the critical point where the integrand contributes significantly; 3) approximate behavior: Bbth solutions focus
31

---

Published as a conference paper at ICLR 2025
approximating the behavior of the integral for large x, emphasizing the exponential term's dominance.
Differences: 1) mathematical rigor: the answer key employs more rigorous mathematical approach, including Taylor expansion and change of variables, providing more precise and detailed solution; heuristic approach: the student takes more heuristic approach, focusing on the dominant exponential term without detailed mathematical expansions or variable changes; 3) accuracy of final expression: final expression with specific constants derived from the detailed process, while the student's s result, though on the right track, has a different coefficient due to the heuristic method. would give partial credit 0.5
answer key provides more accurate
Box 5: Example grading response for a Integrals sub-type Laplace integral problem highlighting the error of failing to develop Taylor's expansion.
A.5 MOTIVATION FOR INCLUDING NONDIMENSIONALIZATION PROBLEMS
GPT-4 response solving dimensionalized polynomial root finding problem
Problem: Find approximate analytical expressions for the roots of the polynomial:
a0x2 ( + 02 = 0
different regimes of the coefficients op, 01 and 02
Model Solution:
Regime 1: |apl > |,11. [22]
(Dominance of the =-tterm) In this regime, the polynomial sim-
plifies to:
agæ
Hence, the dominant contribution to the roots arises from balancing age with the constant term 0g:
02
agæ0 +
a2
x ~
ag
This gives five distinct roots corresponding to the 5th roots of unity:
1/5
Tt
=
a2
12nk/5
A: = 0, 1,,,,3,,4
Regime
[a|]
>> lool, |4g]
(Dominance of the linear term) Here, the polynomial reduces to:
0
and the root is approximately:
RJ 0. -1 + a2
Including the next dominant term, we balance - 01 x: with 02:
a1
Regime 3: |a2| >>> logl: |ay| (Dominance of the constant term) In this regime, the constant term dominates, and the polynomial becomes inconsistent unless 0g = 0, which contradicts the assumption Jazl ş 0. Thus, there Regime 4: General Balance of Terms When all terms are of comparable magnitude, we con- sider approximate solutions based on the magnitude of z:
real roots in this regime.
1. For large 2: The -temm dominates. Balancing ag with =ajæ:
1/4
a1
a1æ
32

---

Published as a conference paper at ICLR 2025
2. For small 2: The linear term -j dominates over the other terms. Balancing 2j with ag:
0g 02
01
3. For intermediate z: All terms contribute, and the roots must be found numerically using perturbative methods.
Reference Solution:
Nondimensionalization We begin with the quintic polynomial:
ag a1 + 02 = 0.
To explore how the solution depends on the parameters op, d1, and 02, us write:
02
y,
a1
and substitute this into the polynomial. This substitution yields:
02
y azly 1) 0.
dg
01
Dividing through by d2 the equation becomes:
5
ag
(2
ya
y + 1 0.
a1
Now define the parameter:
Quaz
a}
which simplifies the equation to:
cx + 1 0.
We examine the roots of the equation:
Ex 5
x + +1=0
in two asymptotic limits: 0 and e oc. The aim is match the expressions for the roots in the intermediate regime.
Consider the limit -> The equation involves three terms: ex.", -r, and For the equation to be satisfied, these terms must sum to zero. In the limit - 0. typically two of these terms dominate, forming a distinguished balance.
To discover approximate formulas for the roots, we analyze each possible balance and determine consistency.
Balancing ~
The first possibility is to balance the terms - and 1. This implies, to first approximation:
x
For this balance to be self-consistent, the neglected term ex must satisfy:
ex 1.
Since r = 1, this condition reduces to < 1, which is satisfied as e - 0. Hence, this balance consistent. Balancing x ~ ~1
Consider the balance c -1. This implies:
(-2)
1 1/5
33

---

Published as a conference paper at ICLR 2025
For this balance to be consistent, the neglected term ~ c- must be smaller than the terms we have kept. However, this the case. the limit c - O. the neglected term grows unbounded, while the retained terms remain order unity. Therefore, this balance is not consistent.
Balancing 2º
The only other possible balance is cx I. This balance leads four roots:
1/4
In this case, the size of the neglected term 1 is much smaller than the size of the retained terms, which are of order 0(4-1/4). Thus, this balance is self-consistent.
Final Solution the Quintic Roots The quintic equation:
002 01 + 02
has the following approximate roots depending on the coefficient regime, summarized below:
Root I: R 1,
1/4
ai agay
Roots 2-5:
x Rz
e'
k= 1,2,3.
34


Reviews:
[{"review_id": "nipseUOTdm", "reviewer": "Reviewer_dUZT", "summary": "This work presents a new math dataset that targets on applied mathematics and requires graduate level knowledges. The problems in this dataset demand a combination of mathematical reasoning, computational tools, and subjective judgment. For example, it may need checks for self-consistency and the use of numerical methods.", "questions": "1. The description in Section 3.2 is not clear. For example, the paper presents that `sympy` and `scipy` are used to implement the mathematical procedures required for obtaining approximate, analytical solutions. In Appendix A, it only includes some examples with questions and solutions, and I did not see the role of `sympy` and `scipy`.\n\n2. As mentioned in Weakness 2, besides constructing the benchmark, I think this work has the potential to be used for constructing large scale training dataset, teaching the LLMs to solve such hard math problems with the suggested thoughts in CoT, or generated `sympy` code in PoT. What do the authors think about this direction?\n\n3. As my concern of diversity in the Weakness 1, I also have the same concern on the solution ideas. It seems that many problems share the same solution idea by just randomly changing the numbers in the problem. This may be verified by the large performance increase from 0-shot to 5-shot. As the 5 demonstration examples may have already provide the correct solution path, the LLMs only need to change some numbers. Can the author introduce how many solution strategies are used for each type in your dataset?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper presents a method to generate graduate level math problems (in some pre-defined areas) and corresponding solutions.", "weaknesses": "1. The type of the math problems in this dataset is limited. First, only three main subjectives are discussed, ODEs, integrals and polynomial. Second, most of the problems are related to calculations, and it seems none of them is proof related. For such problems, the existed numerical computational software should be able to solve most of them. In summary, the diversity of this dataset is limited.\n\n2. Since the dataset can be generated with python tools, it is possible to generate more examples for training. I didn't see any experiment about training on such dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hIFRVeO73j", "reviewer": "Reviewer_9Y89", "summary": "This paper proposes a new math benchmark for evaluation of LLMs.", "questions": "Have the authors evaluated larger open-source models such as Llama 3.1 405B?", "limitations": "", "rating": 8, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper makes an important contribution to the LLM community. The LLM community needs a harder math benchmark with the growing capability of the models. I think this benchmark will be useful.", "weaknesses": "I don't see obvious weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mzKV4Bl8DK", "reviewer": "Reviewer_2kK3", "summary": "The author introduces a challenging math dataset focused on asymptotic reasoning, highlighting the use of mathematical approximations to address complex problems that commonly arise in real-world scenarios.", "questions": "Refer to the weakness of paper section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- Introducing a dataset that requires approximate analytical solutions is innovative, addressing a gap in current benchmarks. This approach aligns with many real-world problems and reflects how scientists typically tackle them.\n\n2- The dataset is larger than similar graduate-level math datasets. It can be utilized to develop novel prompting techniques or for fine-tuning models.", "weaknesses": "The authors suggest that their dataset could be utilized for fine-tuning LLMs to enhance their capabilities. However, it would be valuable to see empirical results demonstrating this improvement.\n\nThe authors explore one-shot and five-shot CoT prompting, noting a substantial performance increase across most models. It would be worthwhile to investigate more complex CoT prompting techniques", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CagScDFzGu", "reviewer": "Reviewer_m8U8", "summary": "This paper proposes a dataset of mathematical problems taken from a graduate level engineering course, and evaluates the ability of several LLMs on solving the problems in the dataset. The problems include topics related to polynomials, root finding, integrals, etc. Paper considers some base problems and modifies them randomly to generate a relatively large dataset. The dataset also includes some manually crafted problems. The accuracy of best LLM goes above 62% in solving the problems in the test set of this dataset. The paper goes on to analyze the failures of LLMs.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The topic is interesting and the dataset (at least, a good portion of it) can be useful for the community.\n\nPaper is well written and the experiments are insightful.", "weaknesses": "In my view, the dataset could have consisted of more difficult problems. From what I read in the abstract and introduction, I was expecting a much harder set of problems to be included in the dataset. The relatively high accuracy of the GPT model on the dataset (above 60%) is also indicative that a considerable portion of the dataset consists of problems that are not as challenging.\n\nDataset generation method, described in Section 3.2, is still not completely clear to me, even after reading Appendix A. I think the generation procedures need to be explained in more detail such that one can reproduce the generation method by reading the paper and the appendices. Appendix A merely provides some examples of problems in the dataset which I was hoping to see at least one sample in page 1 or 2, rather than the appendix.\n\nThe topic of non-dimensionalizing of a polynomial does not strike me as interesting as some of the other topics. In the appendix A, paper provides exact formulas for non-dimensionalizing a polynomial - an exact formula does not need numerical methods as the paper suggests are the basis of all problems in the dataset. Quoting the claim from page 1: “most datasets focus on grade school- to high school-level mathematics problems whose solution methods only involve direct, ‘clean’ calculations. In contrast, HARDMATH targets applied mathematics problems that require approximate analytical solutions.”\n\nWhen it comes to the so-called Word problems in the dataset, the paper crafts them manually, which is fine. But, it does not sit well with earlier claims of the paper such as “Rather than relying on the typical approach of collecting problems from textbooks, standardized tests, or competitions, as seen in most existing datasets, we developed algorithms to automatically generate problems and their step-by-step solutions.”\n\nRandomly modifying the initial conditions of an ODE or coefficients of a polynomial can of course be done automatically. However, I think it is not fair to contrast such automation with the manual work that was the basis for some of the benchmarks in the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)