U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs

Authors: Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov, Sergei Tilga
Primary area: datasets and benchmarks
Keywords: Large Language Models (LLMs), Mathematical Reasoning, Benchmarking, University-Level Mathematics, Multimodal, Automatic Evaluation, Solution Assessment

Abstract:
The current evaluation of mathematical skills in LLMs is limited, as existing benchmarks are relatively small, primarily focus on elementary and high-school problems, or lack diversity in topics. Additionally, the inclusion of visual elements in tasks remains largely under-explored. 

To address these gaps, we introduce **U-MATH**, a novel benchmark of \textbf{1,100} unpublished open-ended university-level problems sourced from teaching materials. It is balanced across six core subjects, with  \textbf{20\% of multimodal problems}. Given the open-ended nature of U-MATH problems, we employ an LLM to judge the correctness of generated solutions. To this end, we release **$\boldsymbol\mu$-MATH**, an dataset to evaluate the LLMs' capabilities in judging solutions.

The evaluation of general domain, math-specific, and multimodal LLMs highlights the challenges presented by U-MATH. Our findings reveal that LLMs achieve a maximum accuracy of only 63\% on text-based tasks, with even lower 45\% on visual problems. The solution assessment proves challenging for LLMs, with the best LLM judge having an F1-score of 80\% on $\mu$-MATH.
    
We open-source U-MATH, $\mu$-MATH, and evaluation code on GitHub.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730444696239

thread_type:
official_review

<thread_object>
creator: reviewer_9qxr

<content>
summary:
The paper introduces U-MATH, a novel benchmark designed to evaluate the mathematical reasoning capabilities of Large Language Models (LLMs) at the university level. It comprises 1,125 unpublished, open-ended problems sourced from actual teaching materials, balanced across six core mathematical subjects, with 20% of the problems requiring image understanding. Additionally, the paper presents µ-MATH, a meta-evaluation dataset aimed at assessing the ability of LLMs to evaluate free-form mathematical solutions. The experiments conducted reveal significant challenges in advanced mathematical reasoning and visual problem-solving, with the best-performing models achieving only 53% accuracy on text-based tasks and 30% on visual problems. The paper also highlights the difficulty LLMs face in assessing solutions, with the highest µ-MATH F1-score being 76%, indicating room for improvement in LLMs’ evaluation capabilities. The datasets and evaluation code are open-sourced to facilitate further research.

strengths:
The paper demonstrates a high-quality collection of problems that are well-balanced across six core mathematical subjects. This ensures a comprehensive evaluation of LLMs across different areas of mathematics.
The problems sourced from actual teaching materials add a layer of authenticity and practical relevance to the benchmark, ensuring that the skills assessed are applicable to real-world academic standards.
The creation of µ-MATH for meta-evaluation is an innovative approach to assessing the ability of LLMs to evaluate mathematical solutions. This adds another layer of complexity and originality to the benchmarking process, focusing not just on problem-solving but also on the assessment capabilities of the models.

weaknesses:
While the inclusion of visual elements in 20% of the problems is a step forward, the remaining 80% are text-based. The paper could benefit from expanding the visual problem set to better assess and train LLMs in multimodal mathematical reasoning, which is increasingly important for real-world applications.

The paper focuses on university-level mathematics, but it is unclear how well the findings generalize to other levels or types of mathematical reasoning. Future work could explore the transferability of the models trained on U-MATH to other mathematical domains.

questions:
Why are there no examples of problems that require visual input?

The accuracy when using LLM as a judge is not provided, especially for higher mathematics problems where answers may be in different forms but are actually equivalent, indicating that it is easier to make mistakes compared to comparing a single form of answer.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647198911

thread_type:
official_review

<thread_object>
creator: reviewer_rzhq

<content>
summary:
The authors introduced a new benchmark dataset, U-MATH, designed to evaluate large language models (LLMs) on university-level math problems. The proposed U-MATH benchmark includes 1,125 college-level math problems collected from real educational materials, covering six core mathematical subjects, with 20% of the problems involving image understanding. Additionally, the paper introduces a meta-evaluation dataset named µ-MATH, aimed at assessing the ability of LLMs to judge the correctness of mathematical solutions.

strengths:
1.U-MATH Benchmark: This is a publicly available dataset of university-level math problems, covering six topics: Pre-Calculus, Algebra, Differential Calculus, Integral Calculus, Multivariable Calculus, and Sequences & Series. A unique aspect of this dataset is its inclusion of open-ended questions that require LLMs to perform multi-step reasoning.
2.µ-MATH Meta-Evaluation Benchmark: This benchmark is specifically designed to test LLMs’ ability to assess the correctness of mathematical solutions. It contains 340 questions selected from U-MATH, accompanied by LLM-generated answers manually labeled as correct or incorrect, aimed at evaluating the capacity of LLMs to act as “judges.”
3.Model Comparison: The paper compares the performance of various LLMs, including general-purpose models, specialized math models, and multimodal models, demonstrating the significant challenges LLMs still face in both text and visual tasks. For instance, the highest accuracy for text-based questions is 53%, while performance on visual questions is even lower, with an accuracy of only 30%.
4.Challenges for LLMs as Math Judges: LLMs perform poorly when evaluating mathematical solutions, with the best-performing LLM judge achieving an F1 score of only 76% on µ-MATH, indicating that there is still room for improvement in this task.

weaknesses:
1.The U-MATH dataset introduced in the paper supplements the current math datasets by addressing college-level gaps, while the µ-MATH meta-evaluation dataset enables assessment of large models’ ability to evaluate college-level math solutions. However, aside from knowing that this training set focuses on university mathematics and includes six subjects, we lack information about the dataset’s question diversity, difficulty, reasoning steps required to solve the problems, and other aspects. Additionally, the dataset’s size may be insufficient.
2.The paper mentions that the dataset has been released but does not provide an access link, so I have no direct way to review the dataset.
3.The experiments in the paper provide valuable insights into the capabilities of current text-based and multimodal LLMs in solving university-level math problems.
4.The paper states that U-MATH aims to promote further research and improve LLMs' ability to handle complex math problems. How is "complex" defined here? Does it refer to higher-grade, more challenging (for humans) knowledge, or does it mean problems requiring more and deeper reasoning steps?

questions:
I don't have further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686834791

thread_type:
official_review

<thread_object>
creator: reviewer_zva3

<content>
summary:
The paper introduces U-MATH, a comprehensive university-level mathematical benchmark designed to evaluate the performance of Large Language Models (LLMs) in solving advanced mathematical problems. The dataset consists of 1,125 problems sourced from university coursework, covering six core topics such as Algebra, Calculus (Differential and Integral), Multivariable Calculus, Sequences, and Series, with approximately 20% of the tasks involving visual components. To complement the U-MATH dataset, the authors also present µ-MATH, a meta-evaluation set for assessing the accuracy and reliability of LLM-based evaluators.

strengths:
S1. The inclusion of university-level problems offers a significant advancement over existing datasets that mainly focus on elementary or high school-level tasks.

S2: By integrating visual tasks alongside traditional textual ones, the dataset challenges LLMs to interpret and reason across multimodal formats.

S3: µ-MATH introduces a novel approach to evaluate LLMs' ability to assess solutions, addressing biases and limitations in current evaluation practices.

weaknesses:
W1: The reliance on LLMs as judges (e.g., GPT-4o) to evaluate free-form answers could introduce biases and inconsistencies, particularly since LLMs may struggle with complex derivations or nuanced interpretations of mathematical expressions.

W2: The µ-MATH set includes LLM-generated solutions, which may limit the diversity and challenge of evaluation due to inherent model tendencies or training biases. This could result in less rigorous meta-evaluation as models may overfit to known patterns or heuristics.

questions:
What measures have been taken to mitigate potential biases introduced by using LLMs as judges for solution correctness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730734578334

thread_type:
official_review

<thread_object>
creator: reviewer_h6kh

<content>
summary:
This paper introduces the U-Math datasets, based on university-level mathematics, addressing the issues of insufficient thematic diversity and a lack of visual information question types in current datasets for evaluating the mathematical abilities of large language models. The U-Math datasets was tested on several large language models, revealing that the highest accuracy for text-based tasks was only 53%, while the highest accuracy for visual tasks was only 30%.

strengths:
1. The paper is well-organized, providing a clear outline of the datasets, experimental setup, and evaluation metrics. The authors explain each component in a structured manner, making it accessible to readers.

2. The datasets include a range of mathematical subjects and problem types, which reflects an effort to cover diverse aspects of mathematical reasoning, though the depth and breadth could still be improved.

3. The introduction of U-MATH and µ-MATH provides additional benchmarks for evaluating LLMs in mathematical tasks, which may offer a reference point for similar studies.

weaknesses:
1. Although the U-MATH datasets consists of 1,125 samples and covers six subjects, the sample size is still too small. Evaluating the mathematical abilities of large models using a limited amount of data is not sufficiently convincing.

2. Although the 340 samples in the µ-MATH datasets have been carefully selected to provide a challenging test, a larger sample size could enhance the representativeness of the evaluation, especially across different topics and problem types.

questions:
1.It is recommended to expand both the U-MATH datasets size and the number of subjects.

2.It is recommended to expand the µ-MATH datasets size.

3.In Table 4, you only use accuracy to present the results. Since the study involves math problems, which are more complex than simple classification tasks, could you consider adding additional evaluation metrics like perplexity or WinoGrande ACC (to assess whether ambiguous problems are correctly identified)? This would give readers a clearer picture of how well the models truly understand and respond to university-level math questions. For more details, you might refer to examples in this paper: https://proceedings.mlr.press/v235/dao24a.html.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731706121695

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank the reviewers for their helpful comments. In this revision, we:
* Improve the presentation by adhering to English style and grammar; we additionally use the premium Grammarly subscription for post-editing;
* Add more examples of the problems to the Appendix A, including visual problems (9qXr);
* Provide distribution of sub-topics for 6 code subjects in Appendix B to showcase the depth and diversity of the benchmark (H6kh, Zva3);
* Add an anonymous download link to U-MATH and µ-MATH datasets in Abstract (rZHQ).

Following valuable feedback, in the next revision we will:
* Increase the number of µ-MATH samples while also extending the diversity of the dataset by including solutions generated by various model families. We aim to enhance the depth and robustness of the benchmark, ensuring it accurately reflects the broader challenges encountered during solution evaluation (H6kh)
* Investigate statistics on the number of solution steps required for solution in addition to the existing number of questions/answers per problem (Zva3);
* Add additional models to the comparison: gpt-4o-mini, Gemini-1.5-Flash, LLaMA-3.2, and Athene-V2 to strengthen both depth and width of our research

The following points are mentioned by several reviewers, so we address them here:
* It is important to note, that the primary goal of this work is to introduce a robust and comprehensive benchmark for post-hoc evaluation of LLMs’ math capabilities in university-level settings. So we do not provide a training set for U-MATH and µ-MATH benchmarks; they are presented solidly for evaluations of LLMs. (Zva3, 9qXr)  
However, reviewers’ interest in exploring the generalizability of models trained on university-level mathematical problems to other educational levels aligns with ours. We also recognize the value of researching new methods to enhance the training process, particularly for advanced mathematics. We plan to investigate these aspects in future work.
* Regarding dataset size, to our knowledge, U-MATH is the largest university-level math benchmark available (Table 1). 
However, we aim to provide an affordable way of testing models; validation of larger sample sizes would increase costs, as demonstrated by the [MathEval dataset](https://github.com/math-eval/MathEval ), which includes 30k samples but authors spent around $58,000 for API calls.  
Moreover, the current sample size of 1,125 yields a confidence interval with the highest margin of error being ±3.1% (at a 95% confidence level, with p=50% of a model being correct) for model accuracy estimation, which we deem statistically robust for a benchmark of this scale, particularly given a gap exceeding 3% between the top-scoring and second-ranking models. The closest dataset OCWCourses yields a ±5.9% Margin of Error with 272 samples. Moreover, 150 samples in each text-only subject split and 225 samples in the visual split provide statistically significant differences between most models in ranking.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732805394121

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responce - v2 revision

comment:
We further update the paper following the reviewer's comments. In this revision, we:
* We **substantially update the µ-MATH dataset**, expanding its size to 1084 solutions generated by a diverse set of models: Qwen2.5 72B, LLaMA-3.1 70B, GPT-4o and Gemini 1.5 Pro. This expansion not only increases statistical robustness, but allows fine-grained analysis of model-judgeing itself to analyse bias (Zva3).
* We also provide more detailed analyzes of judges’ performance, prompt sensitivity, behavior patterns and biases, better showcasing the utility of meta-evaluations.
* We include additional models to comparison: Gemini Flash, GPT-4o-mini, Claude 3.5 Sonnet, LLaMA-3.2 11B/90B, LLaMA-3.1 Nemotron 70B, and Athene-V2 72B, improving depth of our analysis and highlighting ‘further-tuning’ improvement for Nemotron and Athene.
* We update scores of the models using a more standardized prediction prompt used in DeepSeek (A.Liu et.al, 2024) and Qwen-2.5-Math (A.Yang et.al, 2024) models, and a simpler, better-performing judgment prompt, with double the inference token limit.
* We also update the U-MATH dataset, to increase the interlap with µ-MATH problems. This together with changes in prompting setups for prediction and judgment lead to changes in final scores on U-MATH. However, the models’ ranking remains stable, with Gemini being the top performer.
* Also, we improve presentation of U-MATH and µ-MATH results by including additional charts in the paper body and Appendix sections, allowing for a detailed comparison of the models’ performance across various splits.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents U-MATH, a university-level mathematical benchmark containing 1,125 problems for evaluating LLMs, along with μ-MATH, a meta-evaluation dataset for assessing solution correctness. While the paper makes a valuable contribution by addressing the gap in university-level math assessment, significant concerns remain about the limited dataset size, potential biases in LLM-based evaluation, and lack of comprehensive analysis of problem complexity. Although the authors provided detailed responses and planned improvements, core issues around statistical robustness and evaluation methodology were not fully resolved. The dataset's current scope and evaluation approach require substantial enhancement before it can serve as a reliable benchmark for advanced mathematical reasoning capabilities in LLMs.