HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics

Authors: Jingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht, Jonah Brenner, Danxian Liu, Nianli Peng, Corey Wang, Michael Brenner
Primary area: datasets and benchmarks
Keywords: math, benchmark, dataset, few-shot learning, reasoning

Abstract:
Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce $\textbf{HARDMath}$, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on $\textbf{HARDMath-mini}$, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate the limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like $\textbf{HARDMath}$ to advance mathematical abilities of LLMs.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730342039222

thread_type:
official_review

<thread_object>
creator: reviewer_m8u8

<content>
summary:
This paper proposes a dataset of mathematical problems taken from a graduate level engineering course, and evaluates the ability of several LLMs on solving the problems in the dataset. The problems include topics related to polynomials, root finding, integrals, etc. Paper considers some base problems and modifies them randomly to generate a relatively large dataset. The dataset also includes some manually crafted problems. The accuracy of best LLM goes above 62% in solving the problems in the test set of this dataset. The paper goes on to analyze the failures of LLMs.

strengths:
The topic is interesting and the dataset (at least, a good portion of it) can be useful for the community.

Paper is well written and the experiments are insightful.

weaknesses:
In my view, the dataset could have consisted of more difficult problems. From what I read in the abstract and introduction, I was expecting a much harder set of problems to be included in the dataset. The relatively high accuracy of the GPT model on the dataset (above 60%) is also indicative that a considerable portion of the dataset consists of problems that are not as challenging.

Dataset generation method, described in Section 3.2, is still not completely clear to me, even after reading Appendix A. I think the generation procedures need to be explained in more detail such that one can reproduce the generation method by reading the paper and the appendices. Appendix A merely provides some examples of problems in the dataset which I was hoping to see at least one sample in page 1 or 2, rather than the appendix.

The topic of non-dimensionalizing of a polynomial does not strike me as interesting as some of the other topics. In the appendix A, paper provides exact formulas for non-dimensionalizing a polynomial - an exact formula does not need numerical methods as the paper suggests are the basis of all problems in the dataset. Quoting the claim from page 1: “most datasets focus on grade school- to high school-level mathematics problems whose solution methods only involve direct, ‘clean’ calculations. In contrast, HARDMATH targets applied mathematics problems that require approximate analytical solutions.”

When it comes to the so-called Word problems in the dataset, the paper crafts them manually, which is fine. But, it does not sit well with earlier claims of the paper such as “Rather than relying on the typical approach of collecting problems from textbooks, standardized tests, or competitions, as seen in most existing datasets, we developed algorithms to automatically generate problems and their step-by-step solutions.”

Randomly modifying the initial conditions of an ODE or coefficients of a polynomial can of course be done automatically. However, I think it is not fair to contrast such automation with the manual work that was the basis for some of the benchmarks in the literature.

questions:
Please see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730556728336

thread_type:
official_review

<thread_object>
creator: reviewer_2kk3

<content>
summary:
The author introduces a challenging math dataset focused on asymptotic reasoning, highlighting the use of mathematical approximations to address complex problems that commonly arise in real-world scenarios.

strengths:
1- Introducing a dataset that requires approximate analytical solutions is innovative, addressing a gap in current benchmarks. This approach aligns with many real-world problems and reflects how scientists typically tackle them.

2- The dataset is larger than similar graduate-level math datasets. It can be utilized to develop novel prompting techniques or for fine-tuning models.

weaknesses:
The authors suggest that their dataset could be utilized for fine-tuning LLMs to enhance their capabilities. However, it would be valuable to see empirical results demonstrating this improvement.

The authors explore one-shot and five-shot CoT prompting, noting a substantial performance increase across most models. It would be worthwhile to investigate more complex CoT prompting techniques

questions:
Refer to the weakness of paper section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702805235

thread_type:
official_review

<thread_object>
creator: reviewer_9y89

<content>
summary:
This paper proposes a new math benchmark for evaluation of LLMs.

strengths:
This paper makes an important contribution to the LLM community. The LLM community needs a harder math benchmark with the growing capability of the models. I think this benchmark will be useful.

weaknesses:
I don't see obvious weaknesses.

questions:
Have the authors evaluated larger open-source models such as Llama 3.1 405B?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730753035542

thread_type:
official_review

<thread_object>
creator: reviewer_duzt

<content>
summary:
This work presents a new math dataset that targets on applied mathematics and requires graduate level knowledges. The problems in this dataset demand a combination of mathematical reasoning, computational tools, and subjective judgment. For example, it may need checks for self-consistency and the use of numerical methods.

strengths:
1. This paper presents a method to generate graduate level math problems (in some pre-defined areas) and corresponding solutions.

weaknesses:
1. The type of the math problems in this dataset is limited. First, only three main subjectives are discussed, ODEs, integrals and polynomial. Second, most of the problems are related to calculations, and it seems none of them is proof related. For such problems, the existed numerical computational software should be able to solve most of them. In summary, the diversity of this dataset is limited.

2. Since the dataset can be generated with python tools, it is possible to generate more examples for training. I didn't see any experiment about training on such dataset.

questions:
1. The description in Section 3.2 is not clear. For example, the paper presents that `sympy` and `scipy` are used to implement the mathematical procedures required for obtaining approximate, analytical solutions. In Appendix A, it only includes some examples with questions and solutions, and I did not see the role of `sympy` and `scipy`.

2. As mentioned in Weakness 2, besides constructing the benchmark, I think this work has the potential to be used for constructing large scale training dataset, teaching the LLMs to solve such hard math problems with the suggested thoughts in CoT, or generated `sympy` code in PoT. What do the authors think about this direction?

3. As my concern of diversity in the Weakness 1, I also have the same concern on the solution ideas. It seems that many problems share the same solution idea by just randomly changing the numbers in the problem. This may be verified by the large performance increase from 0-shot to 5-shot. As the 5 demonstration examples may have already provide the correct solution path, the LLMs only need to change some numbers. Can the author introduce how many solution strategies are used for each type in your dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732404131754

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall response to reviewer

comment:
We thank the reviewers for their time and comments! We are glad that the reviewers acknowledge the need for a harder math benchmark that addresses the growing capabilities of LLMs, and the dataset's potential use for improving future LLMs. The comments that our experiments were well-designed and well-presented are especially appreciated!

We have updated the text of our paper in response to the reviewers’ suggestions and added several new figures/sections. Here is a summary of our main changes.

1. To integrate dUZT’s suggestions regarding the use of SymPy/SciPy and the generation process, we added more text details in **Section 3.2** and a new flowchart in **Fig. 2 (pg 4)** of the revised paper to explain the data generation process visually. We also incorporated **Box 1 (pg 4)** to demonstrate diverse solving strategies used in an example solution. 

2. To address the question that evaluations for some model/conditions reaches relatively high accuracy (~60%), we emphasize that our accuracy metric relies on partial credit, which means that the ~60% accuracy rate is a result of our GPT-grading framework that focuses on the solution process and answer rather than only the final answer. **Fig. 3 (pg 9) in our paper demonstrates that much of the credit given for Roots, ODE, and Integral problems is partial—accuracy scores would decrease significantly without this partial credit method.**

3. On the topic of the diversity and relevance of our dataset, we note that the fields of applied mathematics and mathematical physics **rely** on these problem classes. For example, when referring to asymptotic and approximation methods, a widely-known textbook writes that they “are the most useful and powerful methods for finding approximate solutions to equations, but they are difficult to justify rigorously” [1]. Few datasets cover important graduate-level applied mathematics skills, and none to the scale of HARDMath. 

We also better motivate the inclusion of nondimensionalization, since it is a fundamental technique used in our other approximation methods, which current models fail at (**Appendix A.5, pg 32**).

4. we add a new method for automatically constructing a word problem dataset using LLMs like GPT-4o (please see the new **Section 3.5, pg 6, and Table 3, A.2.7, pg 23**). In this framework, we provide the base mathematical problem and a subject domain, are given a word problem by the model, and then evaluate the topic diversity and “plausibility” of the problem using another LLM (A.2.8, A.2.9, pgs 24-25). We only select problems with a plausibility score above a minimum and show that many realistic and diverse word problems can easily be generated.

We again thank the reviewers and the AC for their time.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors introduce HARDMath, a new benchmark dataset for evaluating LLMs' capabilities for solving graduate-level applied mathematics problems. Types of problems included in this dataset include polynomial non-dimensionalization and rootfinding, nonlinear ODEs, traditional and Laplace integrals, and "word problems" (essentially, the previous types of problems contextualized in a quasi-realistic setting). 

The paper is well written. The benchmark seems reasonable and thoughtfully constructed, and the authors do a reasonable job profiling a handful of existing LLMs. The fact that the questions are algorithmically generated is a plus, and may be helpful for mitigating memorization. One concern is that benchmarks such as this one might quickly become obsolete, especially with the current pace of progress in reasoning-focused LLMs such as the o-series from OpenAI. Looking ahead I'd encourage the authors to think about how to ensure long-term usage of this benchmark by the community.