Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist

## Abstract

Abstract Exceptional mathematical reasoning ability is one of the key features that demonstrate the power of large language models (LLMs). How to comprehensively define and evaluate the mathematical abilities of LLMs, and even reflect the user experience in real-world scenarios, has emerged as a critical issue. Current benchmarks predominantly concentrate on problem-solving capabilities, presenting a substantial risk of model overfitting and fails to accurately measure the genuine mathematical reasoning abilities. In this paper, we argue that if a model really understands a problem, it should be robustly and readily applied across a diverse array of tasks.
To this end, we introduce MathCheck , a well-designed checklist for testing task generalization and reasoning robustness, as well as an automatic tool to generate checklists efficiently. MathCheck includes multiple mathematical reasoning tasks and robustness tests to facilitate a comprehensive evaluation of both mathematical reasoning ability and behavior testing. Utilizing MathCheck , we develop MathCheck-GSM and MathCheck-GEO to assess mathematical textual reasoning and multi-modal reasoning capabilities, respectively, serving as upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K.
We adopt MathCheck-GSM and MathCheck-GEO to evaluate over 26 LLMs and 17 multi-modal LLMs, assessing their comprehensive mathematical reasoning abilities.
Our results demonstrate that while frontier LLMs like GPT-4o continue to excel in various abilities on the checklist, many other model families exhibit a significant decline.
Further experiments indicate that, compared to traditional math benchmarks, MathCheck better reflects true mathematical abilities and represents mathematical intelligence more linearly, thereby supporting our design. Using MathCheck , we can also efficiently conduct informative behavior analysis to deeply investigate models. Finally, we show that our proposed checklist paradigm can easily extend to other reasoning tasks for their comprehensive evaluation. 1 1 1 Data and code can be found here: https://mathcheck.github.io/

## 1 Introduction

The AI community has been placing significant emphasis on mathematical reasoning as a means to explore the upper limits of intelligence in large language models (LLMs) (Achiam et al., ; Team et al., ; Meta, ; Jiang et al., ; Wei et al., ; Trinh et al., ; Romera-Paredes et al., ) and multi-modal large language models (MLLMs) (OpenAI, ; Lu et al., ).
A large number of efforts have been made on how to enhance (M)LLMs’ mathematical reasoning abilities. In pre-training, Wang et al. (); Shao et al. (); Lin et al. (); Zhang et al. () studied the impact of the quality of mathematical corpus; in post-training, Yue et al. (); Yu et al. (); Li et al. () augmented a huge number of synthetic data, and then developed supervised fine-tuning (SFT) for math problem-solving. Recently, Luong et al. () and Sun et al. () explored variants of reinforcement learning (RL) for further improvements.

To guarantee the high mathematical reasoning ability has been reached, it is crucial to fairly evaluate models’ performance.
Current mainstream methods rely on the performance across math problem-solving tasks of varying difficulty levels, such as GSM8k (Cobbe et al., ) of elementary level, MATH (Hendrycks et al., ) of high school level, and TheromQA (Chen et al., ) of university level.
Recently, some mathematical datasets that are more challenging, diverse, and multi-modal have been proposed to enhance the mathematical evaluation (He et al., ; Liu et al., ; Lu et al., ; Zhang et al., ).
However, these current evaluation methods focus on *individual* tasks (most of which are problem-solving) and robustness tests for each problem. In other words, they do not provide comprehensive guidance on whether LLMs really achieve mathematical reasoning ability.
In this paper, we argue that: if a model really understands a problem, it should work robustly across various tasks about this problem. Therefore, it is necessary to evaluate models by multi-tasks with diverse robustness test. Through such investigation, the real reasoning ability of a model can be comprehensively evaluated. As a result, we can also perform detailed behavior tests on models (Ribeiro et al., ).

Drawing motivations from this insight, we introduce MathCheck, a well-designed checklist for testing task generalization and reasoning robustness.
MathCheck includes general mathematical reasoning tasks and diverse robustness testing types to facilitate a comprehensive evaluation of mathematical reasoning ability and reasoning behavior testing.
As shown in Figure 1, horizontally, we examine the task generalization including problem solving, answerable judging, outcome judging, and process judging. Vertically, we test the reasoning robustness through the original problem and its three robustness variants consisting of problem understanding, irrelevant disturbance, and scenario understanding. The data of each cell in the checklist corresponds to a specific type of robustness test and task form.
To facilitate the construction of checklist, we propose an (M)LLMs-driven generation framework to automatically generate this data.
Figure 2 illustrates the MathCheck data collection process, where the seed solving problem is firstly rewritten to its robustness problems, next all generated solving data are utilized to construct other task forms.

Utilizing MathCheck, we propose MathCheck-GSM, a MathCheck dataset generated from GSM8k (Cobbe et al., ). It contains a total of 3,096 high-quality samples consisting of 129 groups checklist matrix, which can be used to evaluate mathematical textual reasoning ability comprehensively.
Besides, acknowledging the community’s focus on multi-modal reasoning capabilities, we further propose MathCheck-GEO to evaluate the multi-modal geometry reasoning ability. Generated from GeoQA (Chen et al., ), UniGeo (Chen et al., ), and Geometry3K (Lu et al., ), it contains a total of 1,440 samples with a checklist matrix of 60 groups.
It is noteworthy that the construction pipeline of MathCheck can be applied to most mathematical datasets to dynamically establish a comprehensive and flexible evaluation benchmark, thereby mitigating data contamination (Zhou et al., ; Zhu et al., ; ).

We conduct extensive experiments on 26 LLMs and 17 MLLMs including different scales, API-base and open source, generalist and mathematical models.
We find that frontier LLMs like GPT-4o continue to achieve superior performance in our MathCheck, but many other model families exhibit a significant decline.
Further experiments indicate that compared to solving original problems which is the paradigm of mainstream benchmark, our MathCheck evaluation aligns more accurately with the genuine mathematical reasoning ability of the model. Utilizing MathCheck, we extensively analyze the models’ behaviors including training on massive solving data, reasoning consistency, performance on different complexity problems and applying different prompting technologies. Finally, we show the potential of applying MathCheck paradigm to other reasoning tasks such as commonsense reasoning and code generation, promoting more comprehensive evaluation of reasoning ability.

## 2 MathCheck

MathCheck is a well-designed checklist that includes general mathematical reasoning tasks and diverse robustness testing types for comprehensive evaluation, as well as a tool to automatically generate a large number of test cases in the manner of checklist.
In our checklist, various mathematical tasks are arranged in rows to assess task generalization, whereas diverse variants of mathematical problems are placed in columns to evaluate reasoning robustness.
We will elaborate on the task types in Section 2.1, problem variants in Section 2.2, and how we construct checklist data in Section 2.3.

### 2.1 Task Generalization

Testing models across different tasks on the same domain not only offers a comprehensive and profound evaluation of their capabilities (Frank, ) but also caters to the practical demands and complexities of real-world applications (Ji et al., ). In MathCheck, we incorporate four math tasks including Problem Solving, Answerable Judging, Outcome Judging, and Process Judging.

Problem Solving.
In this task, we ask the model to solve a given math problem. As the most widely used method to test mathematical reasoning ability in contemporary research (Cobbe et al., ; Hendrycks et al., ), it necessitates the model to analyze the problem, recall and apply appropriate math knowledge, and finally conclude reasoning results.

Answerable Judging.
Given a math problem, models need to determine whether the problem provides sufficient information to answer the question. This task requires the model to analyze the question, then identify the essential conditions required for solving this question, subsequently verify whether these conditions are provided within the problem statement. Previous works utilized it to examine whether the model is a reasoner with critical thinking instead of a random parrot (Li et al., ; Sun et al., ; Ma et al., ).

Outcome Judging.
Given a math problem and one of its solutions, let the model determine whether the final answer of the given solution is correct. Outcome-Judging is a coarse-grained judgment of solutions since the model only focuses on the correctness of the final answer. Researchers often apply the outcome-judging ability of models to verify the correctness of augmented data (Tang et al., ) and provide outcome rewards in reinforcement learning (Luong et al., ).

Process Judging.
Given a math problem along with its wrong solution, the model is required to identify the step where the errors begin. Compared with the outcome-judging, the process-judging task is a more fine-grained judgment on the solution, which demands the model to judge step by step until the wrong step is located. It can help to debug the given wrong solution.

### 2.2 Reasoning Robustness

A model that truly understands the inherent mathematical logic of a problem will exhibit reasoning robustness to diverse variations of this problem (Stolfo et al., ). Motivated by this, we utilize four problem forms including the original problem and its three rewritten variants to examine the reasoning robustness of models.

Original Problem.
It is the seed problem of other reasoning robustness variants. At a minimum functionality test, it can check whether the model has the basic mathematical capabilities when no modifications have been made.

Problem Understanding.
It refers to transforming the original problem into a new one that uses different wording or different sentence structures but does not change the mathematical logic of its original version (Patel et al., ; Zhou et al., ; Li et al., ). It pays more attention to semantic robustness, and aims to examine whether models can correctly reason when dealing with different descriptions of the same mathematical logic.

Irrelevant Disturbance.
It refers to inserting irrelevant conditions that are related to the topic of the original question, but have no impact on the final answer. Previous studies have disclosed that large language models are easily distracted by such perturbations (Shi et al., ). It needs the model to distinguish which conditions are necessary and which are irrelevant to the problem.

Scenario Understanding.
When models comprehend the scenario of a math problem and its underlying logic, they should be able to solve other questions within that scenario (Liu et al., ; Yu et al., ; Zhou et al., ). Therefore, we alter the original question to evaluate whether a model has a comprehensive understanding of the scenario. For example, as shown in Figure 1, we ask the question “the number of blue bolts" instead of “the number of total bolts".

Figure: Figure 2: MathCheck generation pipeline.
Refer to caption: x2.png

### 2.3 Checklist Construction

Creating MathCheck data is a labor-intensive and time-consuming process.
The advent of LLMs has introduced a new level of flexibility and quality to generate mathematical content (Norberg et al., ; Li et al., ). Therefore, we employ (M)LLMs (e.g., GPT-4-Turbo in our experiments) as engines to automatically generate our MathCheck data.
The data construction pipeline is shown in Figure 2. Users first assemble a collection of math problems with labels as seed data. Second, (M)LLMs initially rewrite these problems into their robustness varieties to make up the robustness problem set. Third, each problem in this set will be extended to construct multiple mathematical tasks about this problem. Finally, all data are manually checked to form MathCheck dataset correctly.

Based on the seed data, we automatically generate another three robustness problems as shown in the first column of Figure 1.
Problem Understanding and Irrelevant Disturbance are the tasks of rewriting problems without altering the final answer. Hence, we prompt the model to rewrite our math problems while maintaining the original answer. For Scenario Understanding, we first extract a variable from the problem as a new answer, then prompt the model to change the question based on the extracted variable. Once we obtain the four robustness reasoning problems of the solving task, we rewrite them respectively to construct multiple tasks, including Answerable Judging, Outcome Judging and Process Judging as shown in the corresponding row of Figure 1. For the Answerable Judging task, we prompt the model to eliminate a condition from the original problem which is crucial for solving it to obtain an unanswerable problem. For Outcome Judging task, we ask the model to solve the problem and acquire candidate solutions, then these solutions are labeled (Correct or Incorrect) according to the final answer. For Process Judging task, we apply the solution rewritten ability of (M)LLMs to construct process-judging data. Specifically, given a problem along with its correct solution, we prompt the model to make mistakes from the given steps and results in a wrong answer. In such a way, we can get a wrong solution while its mistake steps remain simultaneously.
All of our prompts are listed in Appendix F.2.

## 3 Experiments

### 3.1 Datasets

We use MathCheck to comprehensively measure the mathematical reasoning ability across textual and multi-modal settings. Consequently, two benchmarks MathCheck-GSM and MathCheck-GEO are introduced.

MathCheck-GSM is a MathCheck dataset generated from GSM8k (Cobbe et al., ). We choose GSM8k as the seed benchmark since
(1) it is most widely used for evaluating mathematical textual reasoning capability.
(2) we aim to determine whether advanced models are genuinely capable of reasoning at the grade school level.
We first collect a test-mini set of GSM8k, which includes 129 problems sampled evenly according to the difficulty(^2^22We define the difficulty according to the number of reasoning steps of its answers (2 steps to 8 steps)). Subsequently, we generate 129 MathCheck style groups, totaling 3,096 high-quality samples by MathCheck. It can be used to evaluate the real mathematical reasoning ability of LLMs on GSM8k-level problems. A group of MathCheck-GSM case problems are listed in Appendix G.1.

MathCheck-GEO is a dataset for geometry problems, which is the representative task for evaluating multi-modal reasoning capability. First, we collect seed geometry problems from GeoQA (Chen et al., ), UniGeo (Chen et al., ), and Geometry3K (Lu et al., ), containing 60 problems in both English and Chinese. Subsequently, we generate 60 MathCheck style groups, totaling 1,440 high-quality samples. Notably, this is the first geometry problem dataset involving answerable, outcome, and process judgment tasks.
MathCheck-GEO gives research community a harder and multi-modal MathCheck style dataset, as well as showing the extensibility of MathCheck.
A group of MathCheck-GEO case problems are shown in Appendix G.2.

All datasets are checked with meticulous manual validation to ensure high quality and reliability. To this end, we recruited three graduate students who underwent training tailored to the requirements of our research. This rigorous verification process not only enhances the quality of our data but also reinforces the validity of our findings. Finally, our automatic data generation pipeline can achieve an average pass rate of 84.61% (Appendix C.2). The detailed data statistics and quality discussion of our checklist are reported in Appendix C.

**Table 1: Model performance on MathCheck-GSM. PS: Problem Solving, AJ: Answerable Judging, OJ: Outcome Judging, PJ: Process Judging, OP: Origianl Problem, PU: Problem Understanding, ID: Irrelevant Disturbance, SU: Scenario Understanding. Each score is the average score of related units. For example, ’All’ means all units, ’PS’ includes solving units on four problem types, ’OP’ includes original problems on four tasks units.**
| Models | All | PS | AJ | OJ | PJ | OP | PU | ID | SU |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Generalist Models |  |  |  |  |  |  |  |  |  |
| O1-preview | 93.2 | 91.3 | 94.0 | 93.2 | 94.1 | 95.6 | 93.4 | 90.5 | 93.1 |
| O1-mini | 92.7 | 93.6 | 95.0 | 88.9 | 93.6 | 95.5 | 94.2 | 91.0 | 90.5 |
| GPT-4o | 92.0 | 95.0 | 95.0 | 90.1 | 87.8 | 94.6 | 91.6 | 92.0 | 89.6 |
| GPT-4o-mini | 87.2 | 90.1 | 89.6 | 88.6 | 80.4 | 88.9 | 89.4 | 85.6 | 85.1 |
| GPT-4-Turbo-20240409 | 90.9 | 93.8 | 95.9 | 87.8 | 86.0 | 93.8 | 90.4 | 90.8 | 88.6 |
| GPT-3.5-Turbo | 61.4 | 73.5 | 64.3 | 48.3 | 59.5 | 65.4 | 64.6 | 60.1 | 55.4 |
| Gemini-1.5-Pro | 86.3 | 88.6 | 89.5 | 87.6 | 75.0 | 88.0 | 90.2 | 85.0 | 82.0 |
| Claude-3.5-sonnet-20240620 | 90.2 | 94.8 | 95.3 | 90.9 | 79.9 | 92.5 | 92.1 | 89.9 | 86.3 |
| Claude-3-opus-20240229 | 83.5 | 81.6 | 92.0 | 78.7 | 81.8 | 86.3 | 85.6 | 81.9 | 80.3 |
| Claude-3-sonnet-20240229 | 75.0 | 77.9 | 88.9 | 65.1 | 68.0 | 76.5 | 77.8 | 73.7 | 71.9 |
| Claude-3-haiku-20240229 | 57.5 | 79.7 | 49.9 | 44.3 | 56.0 | 61.9 | 62.4 | 55.9 | 49.6 |
| Llama-3.1-70B-Instruct | 90.5 | 95.2 | 95.3 | 89.4 | 82.2 | 93.3 | 91.2 | 89.8 | 87.7 |
| Llama-3-70B-Instruct | 84.7 | 90.1 | 87.5 | 84.6 | 76.7 | 87.7 | 86.7 | 84.7 | 79.9 |
| DeepSeek V2 | 82.2 | 86.8 | 82.6 | 82.5 | 76.9 | 85.1 | 84.4 | 83.5 | 75.9 |
| Mixtral 8 x 7B-Instruct | 59.9 | 56.0 | 58.1 | 63.9 | 61.6 | 62.8 | 61.5 | 58.8 | 56.4 |
| Mixtral 8 x 7B-Base | 44.7 | 40.9 | 50.8 | 51.8 | 35.3 | 50.6 | 47.8 | 41.2 | 39.1 |
| Qwen1.5-72B-Chat | 50.6 | 71.1 | 64.2 | 31.9 | 35.1 | 57.0 | 51.1 | 43.6 | 50.6 |
| Phi-3-Medium-4K-Instruct | 72.0 | 89.7 | 70.8 | 63.2 | 64.1 | 77.6 | 78.7 | 71.1 | 60.4 |
| Phi-3-Mini-4K-Instruct | 64.1 | 71.3 | 64.5 | 62.9 | 57.6 | 68.5 | 66.6 | 61.2 | 60.0 |
| Llama-3.1-8B-Instruct | 71.0 | 76.9 | 65.8 | 77.2 | 64.0 | 74.6 | 73.6 | 66.0 | 69.6 |
| Llama-3-8B-Instruct | 64.2 | 68.6 | 61.4 | 64.9 | 61.8 | 67.8 | 68.8 | 62.9 | 57.1 |
| ChatGLM3-6B | 36.5 | 32.6 | 41.7 | 50.1 | 21.7 | 39.7 | 35.9 | 31.3 | 39.1 |
| Mathematical Models |  |  |  |  |  |  |  |  |  |
| DeepSeek-Math-7B-RL | 50.7 | 79.5 | 50.0 | 45.1 | 28.1 | 53.3 | 51.2 | 47.5 | 50.6 |
| DeepSeek-Math-7B-Instruct | 50.2 | 70.0 | 64.8 | 40.4 | 25.8 | 51.6 | 54.4 | 45.8 | 49.2 |
| DeepSeek-Math-7B-Base | 44.0 | 49.8 | 51.5 | 44.0 | 30.8 | 49.0 | 46.0 | 37.0 | 44.1 |
| MetaMath-LLama2-70B | 45.7 | 70.0 | 35.7 | 45.3 | 31.6 | 49.9 | 51.5 | 43.4 | 37.8 |

**Table 2: Model performance on MathCheck-GEO.**
| Models | All | PS | AJ | OJ | PJ | OP | PU | ID | SU |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Generalist Models |  |  |  |  |  |  |  |  |  |
| GPT-4o | 65.3 | 57.5 | 75.5 | 69.5 | 58.8 | 65.2 | 67.0 | 64.3 | 64.8 |
| GPT-4o-mini | 59.0 | 50.8 | 69.8 | 61.4 | 53.8 | 61.9 | 62.0 | 54.1 | 57.8 |
| GPT-4-Turbo-20240409 | 61.7 | 51.3 | 72.3 | 64.0 | 59.2 | 63.2 | 62.9 | 61.7 | 58.9 |
| GPT-4-Vision-Preview | 60.0 | 46.7 | 71.1 | 63.6 | 58.8 | 59.3 | 62.8 | 57.8 | 60.2 |
| Gemini-1.5-Pro | 58.7 | 47.5 | 67.4 | 55.0 | 64.6 | 62.3 | 58.6 | 57.1 | 56.9 |
| Gemini-1.5-Flash | 56.8 | 45.0 | 75.1 | 50.6 | 56.7 | 56.8 | 59.7 | 53.8 | 57.1 |
| Claude-3.5-sonnet-20240620 | 58.7 | 54.2 | 71.0 | 53.0 | 56.7 | 59.9 | 63.8 | 54.3 | 56.8 |
| Claude-3-opus-20240229 | 47.2 | 34.2 | 60.6 | 46.7 | 47.5 | 47.2 | 49.1 | 42.4 | 50.2 |
| Claude-3-sonnet-20240229 | 49.9 | 35.8 | 59.0 | 51.6 | 52.9 | 51.2 | 53.0 | 44.7 | 50.4 |
| Claude-3-haiku-20240307 | 36.7 | 27.9 | 41.3 | 41.7 | 35.8 | 39.2 | 38.8 | 33.3 | 35.4 |
| QWen2-VL-72B-Instruct | 61.4 | 60.0 | 53.1 | 61.3 | 71.3 | 69.0 | 62.4 | 58.0 | 56.4 |
| QWen2-VL-7B-Instruct | 42.1 | 35.8 | 49.4 | 46.4 | 36.7 | 40.9 | 45.6 | 41.7 | 40.0 |
| InternVL-1.5-Chat | 37.6 | 22.1 | 54.9 | 46.8 | 26.7 | 42.9 | 34.8 | 37.3 | 35.5 |
| MiniCPM-Llama3-V-2.5 | 37.3 | 37.5 | 38.1 | 45.0 | 28.8 | 37.4 | 45.0 | 35.2 | 31.6 |
| LLaVA-1.6-Mistral-7B-Instruct | 31.8 | 10.0 | 38.8 | 51.2 | 27.1 | 33.8 | 35.5 | 28.4 | 29.2 |
| Phi-3-Vision-128k-Instruct | 29.6 | 12.9 | 35.0 | 48.6 | 22.9 | 32.6 | 31.8 | 28.2 | 26.0 |
| CogVLM2-Llama3-Chat-19B | 24.6 | 7.9 | 26.4 | 46.3 | 17.9 | 27.2 | 28.0 | 22.4 | 20.9 |

### 3.2 Experimental Setup

To systematically benchmark the mathematical reasoning capabilities of existing LLMs, we include a comprehensive evaluation of 43 models, comprising 26 LLMs and 17 MLLMs. These models are principally divided into two categories: generalist models encompassing both API-based commercial LLMs and open-sourced LLMs (large and small scale), and specialized mathematical models. We use the F1 metric for Outcome Judging and Answerable Judging tasks, and the Acc metric for the other two tasks. The list of selected models and details of evaluation setup can be found in Appendix D.

### 3.3 Main Results

Tables 1 and 2 illustrate the performance of various models on the MathCheck-GSM and MathCheck-GEO, respectively. The leftmost column represents the average performance across all tasks and all question variants. The middle four columns detail the performance on various mathematical reasoning tasks, while the right four columns display performance across different question variants. Consequently, each model is represented by a 4$\times$4 checklist table, which showcases the model’s performance in various dimensions. The details of all checklist tables are further elaborated in Appendix A and B.

On MathCheck-GSM (Table 1), O1-preview and O1-mini exhibit outstanding performance with impressive overall score of 93.2 and 92.7, demonstrates strong effect of extending reasoning thought exploration.
GPT-4o is closely followed with a score of 92.0 and demonstrates top performance on the problem solving task and irrelevant disturbance variants. These results indicate that strong foundational models still possess formidable and robust performance across a variety of mathematical reasoning tasks.
Among the open-source LLMs, LlaMa-3.1-70B-Instruct achieves the highest score of 90.5 and performs excellently across a range of tasks and problem variants. Its performance has significantly improved compared to LLaMA-3 version and surpasses that of GPT-4o-mini. Besides, Qwen1.5-72B-Chat underperforms in tasks other than problem solving, which we suspect is due to its special optimization of the solving task. This phenomenon is also observed across all math-customized models, which tend to be trained on similar mathematical problems and problem-solving processes, resulting in a relatively narrow scope of reasoning capabilities.

On MathCheck-GEO (Table 2), GPT-4o demonstrates the best performance, achieving a top score of 65.3 in the All category. The performance of GPT4-turbo-20240409 and GPT4-Vision-Preview is similar, reaching scores of 61.7 and 60.0, respectively. In particular, the performance of Claude-3-sonnet is slightly superior in visual contexts compared to that of its larger counterpart, Claude-3-opus.
Among the open-source MLLMs, the large-size MLLMs demonstrate surprisingly strong performance, with Qwen-VL-70B attaining 60.4 over the GPT-4-Vision-Preview.
However, the most of small-size MLLMs exhibited poor performance especially in probelm solving,
which suggests that the multi-modal reasoning capabilities of open-source small-size open-source MLLMs still have significant room for improvement.

Figure: Figure 3: Correlation with GSM1k (Zhang et al., ), a dataset that reflects real mathematical reasoning ability. $p$ and $e$ represent the Pearson Correlation Coefficient, and Root Mean Square Error, respectively.
Refer to caption: x3.png

### 3.4 MathCheck Represents Mathematical Intelligence More Linearly

One desiderata of a good mathematical benchmark is to reflect real mathematical intelligence perfectly.
We follow previous works (Zhang et al., ; Huang et al., ) to assess “intelligence" from practical standpoints and use performance on private data (Zhang et al., ) and compression efficiency (Du et al., ; Huang et al., ) as surrogates to assess the genuine mathematical abilities of models. By examining the correlation between MathCheck and these surrogates, we can verify whether our design effectively reflects mathematical intelligence, and how it compares to traditional benchmarks.

Correlation with Private Data. Unlike traditional open-sourced benchmarks, private data is less likely to be contaminated or overfitted, making it an appropriate proxy of genuine mathematical intelligence. We adopt GSM1k (Zhang et al., ), a new private GSM8k-level dataset, to measure the real mathematical reasoning of models. We compare the correlation of model performance between GSM1k and MathCheck-GSM/GSM8k. As shown in Figure  3, the left part illustrates the correlation between GSM8k and GSM1k. It reveals that most LLMs achieve scores up to 80% on GSM8k, with scores concentrated in the top half of the graph. However, on GSM1k, the scores are evenly distributed, indicating that some LLMs, such as deepseek-math-7B-RL, have inflated scores on GSM8k. This suggests that the GSM8k score is not a reliable benchmark for assessing the true mathematical reasoning ability of the models. In the right sub-figure, MathCheck-GSM and GSM1k display a good positive correlation, and some models that do not perform well on GSM1k can be detected by MathCheck-GSM. By comparing the Pearson correlation coefficient and the root mean square error, it shows that MathCheck has a higher correlation coefficient with GSM1k, mitigating bias evaluation caused by overfitting and data contamination.

Correlation with Compression Efficiency.
Compression efficiency has been empirically proven that represent intelligence well (Du et al., ) even linearly (Huang et al., ), well aligned with the belief that compression is closely connected to intelligence (Deletang et al., ). Following Huang et al. (), we use BPC-Loss in Arxiv papers tagged with “Math" to measure compression efficiency as a surrogate. Figure 4 shows the correlation between BPC-Loss and GSM8K/MathCheck-GSM. The left sub-figure reveals that a single traditional benchmark like GSM8K cannot adequately reflect genuine mathematical ability, as indicated by the low Pearson correlation coefficient ($p=-0.822$). Many models, such as the Qwen series, deviate significantly from the regression line. In contrast, the right sub-figure displays the correlation with our MathCheck-GSM, demonstrating that MathCheck-GSM exhibits a significantly better correlation with genuine intelligence, with a Pearson correlation coefficient of $p=-0.915$. Our method shows that many models, such as the Qwen series, have scores on our benchmark that align more accurately with their true mathematical abilities. This shows that our design can represent mathematical intelligence more linearly compared to traditional benchmarks.

Figure: Figure 4: Performance correlation with BPC-loss, which reflects compression efficiency (Huang et al., ). The lower BPC-loss represents the higher compression efficiency.
Refer to caption: x4.png

Figure: Figure 5: Behavior of mathematical models trained on massive solving data.
Refer to caption: x5.png

## 4 Behavior Analysis

MathCheck contains multi-dimensional information for evaluation, therefore we can observe the behaviors of the models on it to help analyze the models.

Behavior of Math Models.
Recently, some works claim that math reasoning ability is greatly improved by training on massive amounts of math solving data, but this may not necessarily be true. To validate whether their mathematical reasoning ability really improves, we examine the behaviors of the math models and their base models on Mathcheck. As shown in Figure 5, compared with the base model, the performance of DeepSeek-Math-7B-Instruct/RL on solving units is greatly improved. However, the performance improvement on other units is limited, or even downward. The same phenomenon can be observed on MetaMath. It implies that training solely on massive solving data is not the right direction to improve mathematical reasoning ability. Instead, training models with high-quality and diverse mathematical data, beyond just solving, should be considered.

Reasoning Consistency.
We analyze the reasoning consistency of generalist models across each unit in MathCheck, and the detailed results are shown in Appendix A and B. We can see most of them show good reasoning consistency since they achieve similar scores on each unit, such as GPT series, Llama-3 series and Mixtral series on MathCheck-GSM and GPT series on MathCheck-Geo. This is an interesting finding as it substantiates our assertion: a model that really understands a problem can robustly work well on multiple related tasks. Meanwhile, we also find that some models perform reasoning inconsistently. For example, Qwen1.5-72B-chat, Claude-3-Haiku and Phi-3-Medium show excellent performance on the solving task but much worse in other units of MathCheck-GSM. On MathCheck-GSM, Internet-VL achieves a high score of 40.0 on the original problem solving but decreases considerably when the problem switches to other robustness variants. These abnormal inconsistency behaviors of generalist models are highly similar to those mathematical models, revealing that they may conduct excessive decoration on original benchmarks.

Behavior on Different Complexity Levels.
We categorize the complexity of problems based on the number of reasoning steps of the original problems, and select representative models of varying sizes for evaluation, as depicted in Figure 7. We can observe that the models’ accuracy on the original problem solving fluctuates and does not show an obvious downward trend as the problems are more difficult. While the score ”All” shows a steady downward trend, it implies that MathCheck better demonstrates the reasoning skills and capabilities required when problems become difficult.

Behavior on Different Prompting Technologies.
We evaluate five prompting techniques including Zero-shot, Few-shot (Brown et al., ), CoT (Wei et al., ), Least to Most prompting (Zhou et al., ), and Plan-and-Solve prompting (Wang et al., ). The results of GPT-3.5-Turbo on MathCheck-GSM are illustrated in Figure 7. Overall, Chain of Thought (CoT) and Plan-and-Solve (PS) in the zero-shot setting demonstrate superior performance, though this is not consistently the case across all tasks and settings. In contrast, the Few-shot prompt generally yields worse results than the Zero-shot prompt. Through detailed analyses, we find that the math reasoning generalization of LLMs is sensitive to Few-shot samples, which inspires us that Zero-shot with advanced prompt techniques (e.g., CoT or PS) may be a better choice in mathematical reasoning tasks.

Figure: Figure 6: Performance on different complexity levels (i.e., reasoning steps) of MathCheck-GSM.
Refer to caption: x6.png

## 5 MathCheck applied to other reasoning tasks

Our proposed benchmark paradigm MathCheck can be adapted to other reasoning tasks beyond mathematical problems. Similarly, it can help other reasoning tasks with more comprehensive evaluations and conduct detailed analysis through model behaviors. We attempt the migration of the MathCheck paradigm in both commonsense reasoning and code generation.

Commonsense reasoning requires LLMs to apply parametric knowledge to reason and solve problems. In this paper, we choose the date understanding task in Big-bench (bench authors, ) as test-bed since it is wildly used to measure commonsense reasoning ability (Wei et al., ). Appendix E.1 shows the case of applying MathCheck to date understanding. Similar to mathematical reasoning, date understanding is a numerical reasoning task, where it can easily utilize variants of each unit in MathCheck. With MathCheck, a simple raw data of date understanding
have various corresponding test cases to examine the reasoning robustness and task generalization, helping us better evaluate model’s understanding of dates and avoiding hallucination. This case demonstrates the high adaptability of MathCheck to commonsense reasoning task especially numerical reasoning.

In addition to commonsense reasoning task, we would like to show the possibility of transforming MathCheck in some real-world reasoning tasks such as code generation. Appendix E.2 demonstrates a case of applying MathCheck to code generation. Unlike numerical reasoning tasks, the adaptation of code generation needs to consider task relevance. It shows that MathCheck has potential for real-world tasks such as agents and robotics application. For such real-world tasks, it is important to have multiple variants to examine reasoning robustness and task generalization, which directly reflects the diversity of environment and user requirements. Meanwhile, we encourage researchers to design more variants towards specific reasoning tasks beyond MathCheck and promote research on general reasoning benchmark paradigm.

## 6 Related Work

Benchmarks of Textual Mathematical Reasoning.
Numerous benchmarks have been proposed to evaluate the mathematical reasoning capabilities of AI models, including (Amini et al., ; Cobbe et al., ; Frieder et al., ). Some datasets, such as the elementary-level GSM8k (Cobbe et al., ). Consequently, more challenging datasets have been introduced, including those at the high-school level (Hendrycks et al., ), university level  (Sawada et al., ; Zheng et al., ) and olympic level (Huang et al., ).
Additionally, to provide a more comprehensive evaluation of mathematical reasoning abilities, numerous benchmarks have been developed that measure the robustness of mathematical reasoning (Li et al., ), including semantic perturbations (Wang et al., ; Zhou et al., ), reverse problem-solving (Yu et al., ; Berglund et al., ), and irrelevant distractions (Shi et al., ; Li et al., ).
Above benchmarks paradigm focus on a single task (typically problem-solving) and can not comprehensively reflect models’ reasoning ability at a given level.
Therefore, MathCheck tries to go for better reasoning benchmark paradigm.

Benchmarks of Visual Mathematical Reasoning. Recently, multi-modal large language models have demonstrated outstanding capabilities in visual-language reasoning tasks (Allaway et al., ; Chen et al., ; Yang et al., ; Team et al., ). Several benchmarks (Lin et al., ; Antol et al., ; Hudson & Manning, ; Marino et al., ; Mobasher et al., ) have been introduced to assess the visual reasoning capabilities of multi-modal large language models across various modalities including abstract scenes, geometric diagrams, graphics, and charts (Lu et al., ; Chen et al., ; ; Masry et al., ; Kazemi et al., ). To integrate various math visual reasoning tasks, Lu et al. () propose MathVista by collecting multiple datasets.
Currently, research on visual mathematical reasoning remains at a preliminary stage,
Our benchmark focuses on the universality of tasks and robustness of question formulation within visual contexts, offering a comprehensive evaluation and testing platform for the research community.

## 7 Conclusion

In this paper, we argue that if a model really understands a problem, it should be able to successfully solve various tasks and variations of that problem. Based on this insight, we introduce MathCheck, a well-designed checklist for testing task generalization and reasoning robustness. To this end, we also propose an automatic tool for efficiently generating checklist for most of math reasoning datasets. Our proposed MathCheck allows the research community to clearly observe model performance across different dimensions, yielding more comprehensive and objective evaluation results. Using MathCheck, we develop MathCheck-GSM for textual reasoning and MathCheck-GEO for multi-modal reasoning. In the experiments, we evaluate massive (M)LLMs and conduct detailed analysis of model behaviors on MathCheck. Subsequently, we reveal that
the evaluation on MathCheck is much closer to the true reasoning abilities than previous benchmark datasets (e.g., GSM8K and Geometry3K). Finally, we also show the potential of applying MathCheck paradigm to other reasoning tasks such as commonsense reasoning and code generation. We hope our practice and observation can constitute a significant stride towards better reasoning benchmark paradigm.

## Appendix A Heatmap of MathCheck-GSM

Figure: Figure 8: Visualized heatmap of MathCheck-GSM - Part 1.
Refer to caption: x8.png

Figure: Figure 9: Visualized heatmap of MathCheck-GSM - Part 2.
Refer to caption: x9.png

## Appendix B Heatmap of MathCheck-GEO

Figure: Figure 10: The visualized heatmap of MathCheck-GEO.
Refer to caption: x10.png

## Appendix C Data Statistics and Quality

### C.1 Overview of Data

Table 3 and Table 4 show the data statistics of MathCheck-GSM and MathCheck-Geo. Table 5 shows the data statistics of each group in MathCheck-GSM and MathCheck-Geo. In each group, since answerable judging and outcome judging are binary-classification tasks, we try our best to include two different labels in these units for fair evaluation.

**Table 3: Data statistics of MathCheck-GSM**
|  | Problem<br>Solving | Answerable<br>Judging | Outcome<br>Judging | Process<br>Judging |
| --- | --- | --- | --- | --- |
| Original Problem | 129 | 258 | 258 | 129 |
| Problem Understanding | 129 | 258 | 258 | 129 |
| Irrelevant Disturbance | 129 | 258 | 258 | 129 |
| Scenario Understanding | 129 | 258 | 258 | 129 |

**Table 4: Data statistics of MathCheck-GEO**
|  | Problem<br>Solving | Answerable<br>Judging | Outcome<br>Judging | Process<br>Judging |
| --- | --- | --- | --- | --- |
| Original Problem | 60 | 120 | 120 | 60 |
| Problem Understanding | 60 | 120 | 120 | 60 |
| Irrelevant Disturbance | 60 | 120 | 120 | 60 |
| Scenario Understanding | 60 | 120 | 120 | 60 |

**Table 5: Data statistics of each group in MathCheck-GSM and MathCheck-GEO**
|  | Problem<br>Solving | Answerable<br>Judging | Outcome<br>Judging | Process<br>Judging |
| --- | --- | --- | --- | --- |
| Original Problem | 1 | 2 | 2 | 1 |
| Problem Understanding | 1 | 2 | 2 | 1 |
| Irrelevant Disturbance | 1 | 2 | 2 | 1 |
| Scenario Understanding | 1 | 2 | 2 | 1 |

### C.2 Effectiveness of GPT-4-turbo Rewriting

In the process of human evaluation, we conduct statistics on the pass rate of MathCheck-GSM rewritten by GPT4-turbo, as shown in Table 6. It can be seen that the rewriting pass rate is high, which reflects the effectiveness of our generation method. The success rate of Problem Understanding and Scenario Understanding is higher than 90%. There is a pass rate of 86.82% in the Irrelevant Disturbance and 81.40% in Wrong Step Rewriting. It provides references when we use MathCheck generation.

**Table 6: Pass rate (%) checked by human annotators for the data generated by GPT4-turbo.**
| Rewriting<br>Type | Problem<br>Understanding | Irrelevant<br>Disturbance | Scenario<br>Understanding | Unanswerable<br>Question Rewriting | Wrong Step<br>Rewriting |
| --- | --- | --- | --- | --- | --- |
| Human<br>Pass Rate | 93.02 | 86.82 | 91.47 | 85.38 | 81.40 |

### C.3 Discussion of data bias generated by GPT

While we acknowledge there are possible self-bias in LLM-rewritten questions, we assert that this bias is acceptable and does not undermine the conclusions or rationality of MathCheck. This is supported by considerations across several dimensions.

Motivations. The motivation behind MathCheck is to establish a paradigm that mitigates benchmark hacking in the evaluation of mathematical reasoning, thereby revealing the genuine mathematical reasoning abilities of language models more comprehensively.
Rewriting is an integral part of the MathCheck pipeline, which can naturally be performed by either humans or LLMs. While we acknowledge that involving experts in the rewriting process might be the fairest approach, the scalability of this method is a significant concern, as noted in several of today’s LLM benchmarks, such as Arena Hard (Li et al., ) and MT-Bench (Zheng et al., ), due to the high associated costs. To enhance scalability and practicality, we opted to use LLMs as the rewriters.
Given that GPT-4 is widely recognized as the most advanced model accessible to the public, we believe that choosing GPT-4 as the rewriter is the closest approximation to the quality of expert human rewriting.

Human-Checked Questions.
In fact, for the data construction which the LLM participates in, we mainly utilize the powerful rewriting ability of LLMs to edit the seed math problem instead of generating a new one from scratch. Moreover, we manually check the generated text to avoid some unnatural generated text.

Experimental Results and Analysis. On one hand, although the data are generated by GPT-4-Turbo in our experiments, they do not bring extra benefits to GPT-Family models to make them obviously outperform others. As shown in Table 1, the performance of Claude-3.5-sonnet is similar with GPT-4-Turbo, and even much better than GPT-4o-mini, which follows the commonsense on these LLMs.
On the other hand, we compare the experimental results on Non-GPT-Rewritten and GPT-Rewritten Questions. In some data constructions where the LLM is not involved, GPT4-family exhibits the same performance ranking as the score ”All”. Specifically,
the samples in Original Problem&Outcome Judging (OP-OJ) belong to Non-GPT-Rewiritten Questions, which are generated based on the rules.
Table 7 shows that the performance ranking on non-LLM-generated data is close to the score ”All” , where GPT-series continues to perform better than other advanced models. All of these results verify that the possible bias to GPT models is acceptable in our MathCheck.

**Table 7: Model performance on Non-GPT-Rewiritten Questions of MathCheck-GSM**
| Models | All | OP-OJ |
| --- | --- | --- |
| GPT-4o | 92.0 | 91.8 |
| GPT-4-Turbo-20240409 | 90.9 | 88.9 |
| Gemini-1.5-Pro | 86.3 | 84.6 |
| Claude-3-Opus-20240229 | 83.5 | 82.5 |
| Llama-3-70B-Instruct | 84.7 | 85.4 |

## Appendix D Evaluation Setup

We conduct evaluations of multiple representative generalist and mathematical models on our MathCheck benchmark.
For MathCheck-GSM, the evaluation models encompass: (a) Generalist models, including proprietary models such as O1-Preview (OpenAI, ), O1-Mini (OpenAI, ), GPT-4o (OpenAI, ), GPT-4o-mini (OpenAI, ), GPT-4-Turbo (Achiam et al., ), GPT-3.5-Turbo (OpenAI, ), Gemini-1.5-Pro (Team et al., ), Claude-3 (Anthropic, ), Claude-3.5-Sonnet Anthropic (), Llama-3(^3^33[https://ai.meta.com/blog/meta-llama-3](https://ai.meta.com/blog/meta-llama-3)), Llama-3.1(^4^44[https://ai.meta.com/blog/meta-llama-3-1](https://ai.meta.com/blog/meta-llama-3-1)), DeepSeek V2 (Shao et al., ), Mixtral 8 x 7B (Jiang et al., ), Qwen1.5 (Bai et al., ), Phi-3 (Abdin et al., ), and ChatGLM3 (Du et al., ); (b) Mathematical models, including DeepSeek-Math (Shao et al., ) and MetaMath (Yu et al., ).
For MathCheck-GEO, we conduct evaluations on generalist models: (a) proprietary models such as GPT-4o (OpenAI, ), GPT-4o-mini (OpenAI, ), GPT-4-Turbo (Achiam et al., ), GPT-4-vision (OpenAI, ), Gemini-1.5-Pro (Team et al., ), Claude-3.5-Sonnet Anthropic () and Claude-3 (Anthropic, ); (b) open-source models including Qwen2-VL (Wang et al., ), InternVL-1.5 (Chen et al., ), Phi-3-Vision (Abdin et al., ), LLaVA-1.6-Mistral-7B-Instruct (Liu et al., ), MiniCPM-Llama3-V-2.5 (Yao et al., ) and CogVLM2-Llama3 (Wang et al., ).

For Problem Solving and Process Judging tasks, we employ accuracy as the evaluation measure. For Outcome Judging and Answerable Judging tasks, we utilize Macro-F1 as the metric. We employ a zero-shot setting for generalist models and a few-shot setting (two-shot) for base models and mathematical models to enhance their ability to follow specific instructions and tasks. All the prompts used for evaluating (M)LLMs are provided in Appendix F.1.

For all the close-resourced models, we utilize the default hyper-parameters, setting the temperature to 0 and the max tokens to 1,024. Similarly, for all open-source models, the parameters are uniformly configured as follows: $do_sample$ is set to False, $max_gen_len$ is set to 512, and the temperature is set to 0.1.

## Appendix E MathCheck Applied to Other Reasoning Tasks

Figure: Figure 11: Case of MathCheck in Date Understanding.
Refer to caption: x11.png

### E.1 Date Understanding

To show that our proposed benchmark paradigm MathCheck can be adapted to other reasoning tasks beyond mathematical problems, we try to transform some representative reasoning task into MathCheck paradigm. We firstly apply it in commonsense reasoning, which requires LLMs to apply world knowledge to reason and solve problems. Specifically, we choose the date understanding task in Big-bench (bench authors, ) since it is a wildly used task to measure commonsense reasoning ability (Wei et al., ).

Figure 11 shows the case of applying MathCheck to date understanding. Similar to mathematical reasoning, date understanding is a numerical reasoning task, therefore it can easily utilize variants of each unit in MathCheck. For example, in Irrelevant Disturbance, we can add some irrelevant date conditions to cause disturbance. In scenario understanding, we can ask for other variables in order to examine whether models have a comprehensive understanding of this date knowledge. This case demonstrates the high adaptability of MathCheck to commonsense reasoning task especially numerical reasoning.

Figure: Figure 12: Case of MathCheck in Code Generation.
Refer to caption: x12.png

### E.2 Code Generation

In addition to commonsense reasoning task, we would like to show the possibility of transforming MathCheck in some real-world reasoning tasks. Specifically, we choose the code generation task due to its high relevance to Text2Sql, agents and robotics. Figure 12 demonstrates a case of applying MathCheck to code generation. Unlike numerical reasoning tasks, the adaptation of code generation needs to consider task relevance. For example, in Scenario Understanding, we can ask models to write the same function in other program languages (Python to Java in our case) in order to examine whether models have a comprehensive understanding of this function requirements. It shows that MathCheck have potential for real-world tasks such as agents and robotics application. Meanwhile, we encourage researchers to design more specific variants towards their reasoning task on MathCheck framework to test reasoning robustness and task generalization.

## Appendix F Prompt List

### F.1 Evaluation Prompt

Figure: 1: Zero-shot Prompt of Problem Solving

Figure: 2: Zero-shot Prompt of Outcome Judging

Figure: 3: Zero-shot Prompt of Process Judging

Figure: 4: Zero-shot Prompt of Answerable Judging

Figure: 5: Few-shot Prompt of Problem Solving

Figure: 6: Few-shot Prompt of Outcome Judging

Figure: 7: Few-shot Prompt of Process Judging

Figure: 8: Few-shot Prompt of Answerable Judging

### F.2 Data Generation Prompt

Figure: 9: Prompt of Problem Understanding Rewriting

Figure: 10: Prompt of Irrelevant Disturbance Rewriting

Figure: 11: Prompt of Unanswerable Question Rewriting

Figure: 12: Prompt to Rewrite Question and Answer into a Declarative Statement

Figure: 13: Prompt to Generate the Wrong Step

## Appendix G Case Problems

### G.1 Case Problems in MathCheck-GSM . Problem Group ID: GSM-54

Figure: 14: Problem Solving - Original Problem

Figure: 15: Problem Solving - Problem Understanding

Figure: 16: Problem Solving - Irrelevant Disturbance

Figure: 17: Problem Solving - Scenario Understanding

Figure: 18: Answerable Judging (Answerable) - Original Problem

Figure: 19: Answerable Judging (Unanswerable) - Original Problem

Figure: 20: Answerable Judging (Answerable) - Problem Understanding

Figure: 21: Answerable Judging (Unanswerable) - Problem Understanding

Figure: 22: Answerable Judging (Answerable) - Irrelevant Disturbance

Figure: 23: Answerable Judging (Unanswerable) - Irrelevant Disturbance

Figure: 24: Answerable Judging (Answerable) - Scenario Understanding

Figure: 25: Answerable Judging (Unanswerable) - Scenario Understanding

Figure: 26: Outcome Judging (Correct) - Original Problem

Figure: 27: Outcome Judging (Incorrect) - Original Problem

Figure: 28: Outcome Judging (Correct) - Problem Understanding

Figure: 29: Outcome Judging (Incorrect) - Problem Understanding

Figure: 30: Outcome Judging (Correct) - Irrelevant Disturbance

Figure: 31: Outcome Judging (Incorrect) - Irrelevant Disturbance

Figure: 32: Outcome Judging (Correct) - Scenario Understanding

Figure: 33: Outcome Judging (Incorrect) - Scenario Understanding

Figure: 34: Process Judging - Original Problem

Figure: 35: Process Judging - Problem Understanding

Figure: 36: Process Judging - Irrelevant Disturbance

Figure: 37: Process Judging - Scenario Understanding

### G.2 Case Problems in MathCheck-GEO . Problem Group ID: GEO-15

Figure: Figure 13: Geometry diagram for geometry problems in group 15.
Refer to caption: extracted/5909712/case_geo_diagram.jpg

Figure: 38: Problem Solving - Original Problem

Figure: 39: Problem Solving - Problem Understanding

Figure: 40: Problem Solving - Irrelevant Disturbance

Figure: 41: Problem Solving - Scenario Understanding

Figure: 42: Answerable Judging (Answerable) - Original Problem

Figure: 43: Answerable Judging (Unanswerable) - Original Problem

Figure: 44: Answerable Judging (Answerable) - Problem Understanding

Figure: 45: Answerable Judging (Unanswerable) - Problem Understanding

Figure: 46: Answerable Judging (Answerable) - Irrelevant Disturbance

Figure: 47: Answerable Judging (Unanswerable) - Irrelevant Disturbance

Figure: 48: Answerable Judging (Answerable) - Scenario Understanding

Figure: 49: Answerable Judging (Unanswerable) - Scenario Understanding

Figure: 50: Outcome Judging (Correct) - Original Problem

Figure: 51: Outcome Judging (Incorrect) - Original Problem

Figure: 52: Outcome Judging (Correct) - Problem Understanding

Figure: 53: Outcome Judging (Incorrect) - Problem Understanding

Figure: 54: Outcome Judging (Correct) - Irrelevant Disturbance

Figure: 55: Outcome Judging (Incorrect) - Irrelevant Disturbance

Figure: 56: Outcome Judging (Correct) - Scenario Understanding

Figure: 57: Outcome Judging (Incorrect) - Scenario Understanding

Figure: 58: Process Judging - Original Problem

Figure: 59: Process Judging - Problem Understanding

Figure: 60: Process Judging - Irrelevant Disturbance

Figure: 61: Process Judging - Scenario Understanding

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: Who are the human annotators that evaluated the quality of GPT-4T rewriting? Are they (a subset of) the authors? (That is implied from C.3, but I think it should be explicitly stated.) How did one determine whether a problem "passed"? I just encourage the authors to add a few sentences to better clarify the human evaluation process.
soundness: 3
strengths: - The paper is clearly written. The examples and figures are illustrative. For example, I can easily understand what the benchmark is about by looking at Figure 1. The sections are structured well.

- The task setup is innovative (i.e., the 4 x 4 matrix). It is valuable to have these different styles and formats of problems in one place for evaluation and comparison. The argument that the 4 x 4 setup is relevant to measuring generalization and robustness (at least to some degree) is sound.

- The benchmark has text only and multi-modal parts, which are of interest to multiple sub-communities in the field.

- The authors studied a comprehensive set of LLMs, covering models that are open and closed, big and small, generalist and specialized. And they studied several different prompting strategies, which we know matter for mathematical reasoning.

- The authors do a good job attempting to ensure data quality and establish benchmark reliability. For example, they show human pass rates w.r.t. LLM-based problem generation, and they show that the accuracy of MathCheck-GSM is highly correlated to that of GSM1k, a putative high-quality private dataset. 

- The authors not only discuss how the evaluation method introduced in this paper can be general (applied to other domains), but they also have already done work in that direction by creating variants of "data understanding" and code generation tasks. This would prompt other researchers to consider how this method may be relevant to their own domains of interest.

Overall, I believe that this paper and dataset are a valuable contribution to the AI for math research community.
summary: This paper presents MathCheck, a benchmark/checklist for testing (M)LLMs' task generalization and reasoning robustness in mathematical problems. It utilizes GSM8k and GeoQA to create new math problems that are of different styles (e.g., irrelevant disturbance) and different types (e.g., process judging). By evaluating 26 LLMs and 17 MLLMs on this benchmark, the authors find that some models show generally robust performance (e.g., strong close models like GPT-4o), while other models (e.g., Qwen1.5-72B-Chat) show performance drop in alternative problem settings (e.g., OJ and ID). Overall for MLLMs there is large room for improvement on the multi-modal subset (GEO) of the benchmark. They include several analyses to support the reliability of the benchmark and show how the main ideas in the paper can be applied to other evaluation domains such as commonsense reasoning.
weaknesses: - One weakness of the benchmark is about the difficulty of the problems. MathCheck-GSM is based on GSM8k, which is a grade-school-level benchmark that the SoTA models perform very well on. And correspondingly they tend to perform well on MathCheck-GSM. It would have been a stronger benchmark if MathCheck also included high-school and beyond problems, so that we could have a better understanding of where and how models like GPT-4o and O1-preview fail. But this is not a major weakness and can be left for future work. MathCheck already has promise for measuring progress in smaller/weaker models and multi-modal models.

- A similar weakness to the above is about the diversity of problems. The seed data only contain two types of problems: simple word problems and geometry problems.

- The related work section needs to be improved/expanded. Only talking about other math benchmarks is not enough. I encourage the authors to add at least two paragraphs. One can be on what people have done to improve mathematical reasoning with LLMs (training, inference, prompting, LLMs + formal tools, etc.)—the broad landscape of AI for math. Another can be similarly spirited benchmarks in other domains. For example, the use of counterfactual tasks in [1] is also for evaluating how robust and general LLMs' problem understanding and reason abilities are. 

[1] Wu et al. (2024) "Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks" NAACL.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: In Figure 6, could the accuracy drop be attributed to the increasing difficulty LLMs faced in generating or reconstructing task generation entries for more complex problems?

Appendix C2 presents pass rates for different rewriting types. Would it be practical to also show the pass rates for different complexity levels? (This may not be necessary if the such questions are omitted in evaluation.)
soundness: 3
strengths: The paper provides a novel approach to evaluating math problems by considering whether the models understand the problems, not just solve them. The dimension of task generalization is innovative, offering various methods to judge problem answers and providing insights into evaluating reasoning ability. The linearity and performance on different complexity levels demonstrate the potential of the new benchmark for comprehensive evaluation of models' math reasoning abilities.
summary: This paper introduces MathCheck, a new mathematical reasoning benchmark for large language models (LLMs). It aims to address the limitations of existing benchmarks, which often focus on evaluating individual tasks rather than LLM models' understanding of the problems. MathCheck incorporates carefully augmented math problems in different dimensions, including generalization and reasoning robustness.
weaknesses: The data generation pipeline with an 86% pass rate, although good, doesn’t sound ideal given the scale of samples. The paper could benefit from briefly discussing in the main text when showing the pass rate how low-quality generated questions are handled when detected.

Nitpicks: Line 431: It would be helpful to include references to the alleged works. Line 463: quote left mark is incorrect.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - What is the pass rate for GPT-4o rewritting for the MATHCHECK-GEO benchmark?
- The data generation pipeline has a pass rate of ~85%. It is not explicit in the paper whether the released banchmark has filtered out the questions that don't pass or left them in.
soundness: 3
strengths: In terms of originality and quality, it introduces a novel evaluation paradigm that goes beyond traditional problem-solving assessments to comprehensively evaluate mathematical reasoning through multiple tasks and robustness tests. While there have been some previous work on creating function variations, this paper goes beyond simply rewritting the question.
The methodology is rigorous, with extensive empirical validation involving 43 models and a careful data generation process that achieves an 84.61% pass rate with manual validation. The authors validate their framework's effectiveness through correlation with private data and compression efficiency metrics, imlying performance on this benchmark is more robustly capturing llm math abilities.
The paper is reasonably clear in its presentation, with well-structured explanations of the framework and comprehensive documentation of methodology.
The paper is moderately siginificant. The paper provides a more reliable way to evaluate true mathematical reasoning capabilities of LLMs and is exstensible to other domains as well. In particular, the ability to evauate llms on areas other than problem solving is very useful to MCTS and LLM-as-a-judge methods.
summary: This paper introduces MATHCHECK, a new evaluation framework for assessing mathematical reasoning capabilities in LLMs. The framework's key innovation is its comprehensive evaluation approach that tests both task generalization and reasoning robustness. The authors develop two datasets using this framework: MATHCHECK-GSM for text based mathematical reasoning and MATHCHECK-GEO for multimodal geometric reasoning. They demonstrate that while some frontier models like GPT-4o maintain strong performance across their checklist, many other models show significant performance degradation compared to their standard problem solving abilties. The authors also provide evidence that MATHCHECK correlates better with true mathematical reasoning abilities compared to other benchmarks. Finally, the authors show that the current paradiem of finetuning models for math probalem solving capabilities leads to stagnation or even degredation in other areas of math reasoning.
weaknesses: Would recommend citing 1+ papers on functional variations benchmarks which programatically generate question variations.

MATHCHECK-GSM seems to be saturated by frontier models, making it hard to get an accurate measure of their relative abilities. Constructing a MATHCHECK benchmark from MATH or another harder benchmark would be ideal.

In Appendix C, it is unclear what is meant by "data statistic". I assume in tables 3 and 4 it is the total count of questions for that variation type, but I'm not sure what table 5 is trying to show.

Recommended:
- Evaluate models on MATHCHECK adapted to other reasoning tasks (i.e. commonsense reasonin and code)

Minor:
- Use (M)LLM and MLLM inconsistently/unclear what the term means
- It would be nice to have tables 1 and 2 in a plot somehow so we can better see how performance compared across evaluation categories.
- The text in Figure 5 is a bit small and in general hard to interpret
   - I would recommend using a bar chart or similar where it is easier to see the performance deltas (this might require aggregating over the rows)
- Figure 6 should have an x-axis label
- In figure 7, the pink and red are hard to differenciate (perhaps remove the color fills and only include the outlines)

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: 1.	A seed problem is expanded to include 4 math tasks and 4 problem varieties (as shown in Fig.1), resulting in a total of 4\*4=16 problems. Therefore, for the 129 style groups in MATHCHECK-GSM, the total number of samples should be 129\*16=2064. However, in Sec. 3.1, the number of samples in MATHCHECK-GSM is reported as 3096. Could you clarify this discrepancy?
soundness: 3
strengths: 1.	The paper studies an important research question from a fresh perspective, evaluating the models’ reasoning abilities across various tasks and problem varieties.
2.	Abundant experiments: A total of 43 models are evaluated on the proposed benchmarks, which serve as comprehensive baselines for further studies.
3.	The paper is well-written and easy to follow.
summary: This paper explores the question that how to comprehensively evaluate LLMs’ mathematical abilities. Considering that if a model understands a problem, it should be applied across different tasks and problem varieties, they introduce MATHCHECK. MATHCHECK is a checklist for testing task generalization and reasoning robustness. For textual and multimodal language models, they develop MATHCHECK-GSM and MATHCHECK-GEO separately utilizing MATHCHECK.
weaknesses: 1.	Although the paper attempts to analyze the mathematical reasoning capabilities of existing models from multiple aspects (including PS: Problem Solving, AJ: Answerable Judging, OJ: Outcome Judging, PJ: Process Judging, OP: Original Problem, PU: Problem Understanding, ID: Irrelevant Disturbance, SU: Scenario Understanding), there are the following shortcomings. On one hand, some models (such as O1-preview) have already achieved very good performance (>90%) on these datasets and have shown extremely strong robustness, indicating that the evaluation methods proposed in the paper may not be challenging enough for the most advanced models currently available. Moreover, the reviewer did not find any new insights proposed by the evaluation method for the improvement of existing models. On the other hand, the goal of the paper is to explore whether models truly possess mathematical reasoning abilities, and the multiple aspects proposed seem insufficient: if one model performs well in all the proposed aspects, does it truly have reasoning abilities?
2.	Given that these benchmarks are generated by LLMs and only verified by humans, how to ensure the diversity across different question groups? For example, with different seed questions, LLMs may tend to include similar irrelevant information when generating the “irrelevant disturbance” variety, leading to a less comprehensive evaluation.

Meta-review: (a) Scientific Claims:
The paper introduces MathCheck, a framework for evaluating mathematical reasoning in LLMs through task generalization and reasoning robustness. Using this framework, they develop MathCheck-GSM and MathCheck-GEO for text and multimodal evaluation respectively, demonstrating that while top models maintain strong performance, many others show significant degradation compared to standard problem-solving benchmarks.

(b) Strengths:
- Novel evaluation approach beyond traditional problem-solving assessment
- Rigorous methodology with 43 models evaluated
- Careful data validation and correction process
- Strong empirical validation through correlation with private data
- Clear presentation and extensible framework

(c) Weaknesses:
- MathCheck-GSM relatively easy for frontier models
- Limited seed data types
- Initial 85% data generation pass rate
- Related work could be more comprehensive

(d) Reasons for Acceptance:
- Significant methodological contribution with comprehensive evaluation paradigm
- Strong empirical validation and careful experimental design
- Framework provides actionable insights for model improvement
- Generalizable approach applicable to other reasoning tasks
- Clear presentation and thorough documentation

Decision: Accept (Poster)