Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning

Authors: Joykirat Singh, Akshay Nambi, Vibhav Vineet
Primary area: datasets and benchmarks
Keywords: math reasoning, mistake detection, dataset, benchmarking

Abstract:
Large Language Models (LLMs) have significantly impacted the field of Math Word Problems (MWPs), transforming how these problems are approached and solved, particularly in educational contexts.  However, existing evaluations often focus on final accuracy, neglecting the critical aspect of reasoning capabilities. This work addresses that gap by evaluating LLMs' abilities to detect and correct reasoning mistakes.  We present a novel dataset, MWP-MISTAKE, containing MWPs with both correct and incorrect reasoning steps generated through rule-based methods and smaller language models. Our comprehensive benchmarking of state-of-the-art models such as GPT-4o and GPT4 uncovers important insights into their strengths and limitations. While GPT-4o excels in mistake detection and rectification, gaps remain, particularly in handling complex datasets and novel problems. Additionally, we identify concerns with data contamination and memorization, which affect LLM reliability in real-world applications. While OpenAI's O1 model demonstrates 90\% accuracy in reasoning and final answers on complex tasks, it still remains weak in mistake detection. Our findings highlight the need for improved reasoning evaluations and suggest ways to enhance LLM generalization and robustness in math problem-solving.

Decision: reject
Reviewer scores: 5, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1729268549927

thread_type:
official_review

<thread_object>
creator: reviewer_tpvn

<content>
summary:
This paper aims to assess the reasoning capabilities of large language models (LLMs) when solving math word problems (MWPs). It emphasizes the importance of evaluating LLMs’ ability not only to solve problems but also to detect and correct errors in their reasoning processes. The authors introduce a novel dataset called MWP-MISTAKE, which contains correct and incorrect reasoning steps generated through rule-based methods and smaller language models (SLMs). The study benchmarks several LLMs, including GPT-4o, GPT-3.5Turbo, Claude-3-Opus, and others, revealing insights about their strengths and limitations in handling MWPs, especially in error detection and correction.

strengths:
* The paper introduces a novel dataset (MWP-MISTAKE) that evaluates LLMs’ ability to detect and correct errors in reasoning, filling a gap in prior research that focused mainly on final answer accuracy.
* The research is comprehensive, benchmarking multiple state-of-the-art models on various tasks and datasets, revealing significant insights about their reasoning abilities.
* The experiments are well-documented, with clear descriptions of tasks and metrics. The inclusion of both rule-based and model-generated errors is a thoughtful addition.
* The findings are relevant to applications where reliable reasoning is crucial, such as educational tools, and provide a foundation for future improvements in LLMs.

weaknesses:
* Overall, the primary contribution lies in evaluating the reasoning abilities of state-of-the-art (SOTA) LLMs in mathematical problem-solving, with a focus on mistake detection, etc. The contribution remains somewhat limited in scope.
* The paper identifies data contamination as a limitation but could provide more detailed mitigation strategies. This weakens the reliability of some results.
* While the study covers multiple datasets, the models show reduced performance on newer, complex datasets like JEEBENCH, suggesting limited generalization.
* Although rectification abilities are discussed, the results indicate that even advanced models struggle with consistent correction, which highlights the need for stronger frameworks for error handling.
* While the paper identifies several challenges, it offers limited practical solutions or recommendations for overcoming the observed weaknesses in mistake detection.

questions:
Refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729773503943

thread_type:
official_review

<thread_object>
creator: reviewer_y941

<content>
summary:
This work presents a dataset MWP-MISTAKE to evaluate LLM’s abilities to detect and correct reasoning mistakes, the experimental results show that current LLM remains weak in mistake correction.

strengths:
- The proposed dataset is helpful to the relevant research community.
- This work proposes potential directions for enhancing LLM capabilities in reasoning.

weaknesses:
- The finding that LLM remains weak in mistake detection/correction [1][2] is not novel.
- The reliability and accuracy of the MWP-MISTAKE dataset are questionable. The correct reasoning steps of MWP-MISTAKE are generated by GPT-4 and manually verified for correctness. However, there is a lack of detailed statistics on manual inspection, and it is not explained how the authors handle it to ensure data quality when the reasoning steps generated by GPT-4 are incorrect.
- The evaluation setup is hard to follow.
- There are some typos in the paper, e.g., “corect” in line 105, “their” in line 154.

[1] Large Language Models Cannot Self-Correct Reasoning Yet.

[2] SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responses

questions:
- The paper mentions that the correct reasoning steps are generated by GPT-4 and manually verified for correctness. I am wondering what proportion of the data has been manually checked? Among these data, what is the proportion of cases where the reasoning steps generated by GPT-4 are completely correct?
- What does “Smaller model reasoning steps” in table 3 mean? Does it mean the incorrect steps are generated by smaller language models?
- The paper evaluates the ability of LLMs to detect the correctness of reasoning steps and generate correct answers, but it does not evaluate the accuracy rate of the generated reasoning steps?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730086776329

thread_type:
official_review

<thread_object>
creator: reviewer_hzef

<content>
summary:
This paper examines the capability of current LLMs to detect and correct errors in mathematical reasoning. It introduces a new benchmark, MWP-MISTAKE, featuring both correct and incorrect reasoning steps generated by rules or smaller models. Experiments reveal that LLMs still have room for improvement in consistently identifying mistakes across simple and complex datasets. The authors note that contamination issues and generalization challenges hinder reliable performance in real-world applications.

strengths:
This paper meticulously develops six crafted rules to assess LLMs' ability to detect reasoning errors, providing valuable insights into their mathematical reasoning capabilities in realistic settings. The authors compare current models in terms of mistake detection and correction, while also exploring potential contamination and memorization issues, thereby deepening our understanding of these models.

weaknesses:
1.	In the figure 1, GPT-3.5 Turbo fails to detect errors while GPT-4o succeeds. Can GPT-3.5 Turbo correctly solve this question?
2.	Regarding the metric (line 186), is the F1 score for step-level or path-level detection performance?
3.	I suggest presenting the results of model performance in solving questions alongside error detection. Are questions that models solve correctly easier for them to detect errors compared to more challenging questions?
4.	In Table 3, is the average computed based on all data samples? The data sizes from different sources are unbalanced. Additionally, compared to GSM8K, why does detection performance decrease on D but increase on SM (e.g., JEEBENCH)?
5.	I disagree with the statement in lines 253-255 that “GPT-3.5 Turbo performs similarly to GPT-4 and even surpasses it on certain datasets like GSM-8K.” The results in Table 4 do not support this as well. In lines 258-262, the authors claim that the performance of Llama-3-8b-finetuned rivals GPT-4 due to domain-specific fine-tuning. Have you tested Llama-3-8b-finetuned for memorization?
6.	In section 4.4, while GPT-4o shows high ROUGE-L, it also has the best performance in mistake detection and question answering. Could this high performance be attributed to contamination? Can you provide qualitative cases to compare these LLMs? I am also curious about the contamination scores for Llama-3-8b-finetuned and O1.
7.	Citation issue? (line 44)
8. Some relevant work in evaluation is overlooked, such as "MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation".

questions:
Please review the weakness section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690026642

thread_type:
official_review

<thread_object>
creator: reviewer_ix9b

<content>
summary:
This paper proposes a dataset for evaluating LLMs’ abilities in detecting and correcting mistakes. The dataset contains math word problems with both correct and wrong reasoning steps, sourced from various datasets including SVAMP, GSM-8K, MATH, MATHBENCH, and JEEBENCH. To systematically generate mistakes, the authors employ two approaches: (1) rule-based techniques, and (2) using small language models as bad reasoners. It provides a comprehensive benchmarking of various LLMs and SLMs, with insights into their capabilities and limitations.

strengths:
**Originality**:
The proposed dataset is designed to evaluate models in identifying and correcting reasoning mistakes, a critical yet underexplored area in LLM performance. Its focus on these reasoning capabilities rather than final answer accuracy bridges the gap in understanding models’ overall problem-solving capabilities.

**Quality**:
The paper evaluates a wide range of models, from large to small models, from open to proprietary models. It also provides thorough analysis from various aspects, including question variations, memorization, etc.

**Clarity**:
Dataset construction and experiment setups are generally clear. By defining different categories of mistakes (rule-based or SLM-based) and different types of reasoning failures (eg, wrong numerical values, shuffled steps), the paper provides a structured assessment that is easy to understand.

**Significance**:
Understanding reasoning errors and their correction is meanful for educational applications and the wider community. The paper also highlights potential data contamination and memorization issues, emphasizing the need for more sophisticated training and evaluation practices.

weaknesses:
- While the authors claim that rule-based mistakes mimic human reasoning errors, my concern is that some of these transformations lack natural coherence. For example, shuffling reasoning steps or numeric values often results in errors that do not reflect authentic human errors in mathematical reasoning, which can be distracting to models. Could this be the reason why models struggle more with rule-based errors compared to SLM-generated mistakes? I would suggest refining the rule-based methods to better mimic typical student errors, which could give a more valid evaluation of models’ ability to detect naturally occurring mistakes. 

- Additionally, deleting reasoning steps raises a question about whether such omissions genuinely impact the final answer’s correctness or reasoning quality. In cases where reasoning steps are deleted but the answer remains correct, it would be insightful to investigate whether models still detect these partial solutions as errors and if such ambiguities lead to lower error detection rates.

- In related work, the authors mention studies suggesting that LLMs struggle to detect their own mistakes. However, the paper focuses only on mistakes generated by smaller models, without examining those generated by models of similar or competitive capabilities. Exploring a model’s ability to detect its own mistakes in its own reasoning would add substantial value to the study, particularly given its focus on evaluating reasoning capabilities. For example, the authors could investigate whether GPT-4 or GPT-4o can detect mistakes it generates, which would offer more insights into the limitations of LLMs’ self-evaluation in more complex scenarios.

questions:
Questions and Suggestions:
- Figure 2 is not referenced anywhere in the paper.
- Lines 208-209: Do you mean Table 2? What is the difference between Table 2 and Table 7?
- Lines 242-243: Why is GPT-4o underperforming in both simpler and more complex datasets? A brief explanation of the possible challenges it faces with these datasets would be helpful.
- A detailed analysis of mistake types would strengthen the study. For instance, it would be valuable to know which types of mistakes models consistently struggle to detect.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732199015049

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We appreciate the thoughtful feedback from all reviewers and the opportunity to address their concerns. 

We present **MWP-MISTAKE dataset**, a human-verified dataset of math word problems with correct and incorrect reasoning chains. It evaluates state-of-the-art (SOTA) LLMs and SLMs on their ability to detect and correct reasoning mistakes. This paper presents a **comprehensive and novel evaluation framework** for reasoning mistakes in LLMs using MWP-MISTAKE. It provides critical insights into detection, correction, contamination, and interdependencies between tasks, all while addressing a timely and underexplored research problem. We believe this work is a significant step forward for advancing reasoning capabilities in LLMs.

**Addressing Reviewer Concerns:** Most concerns raised by reviewers revolved around clarification of experimental setups, metrics, and qualitative analyses. We have provided detailed explanations and will incorporate additional results, such as qualitative case studies, extended contamination experiments, and clarification of metrics in the revised manuscript. None of the concerns undermine the novelty, scope, or contributions of our work.  **We look forward to continued discussion and feedback from the reviewers and area chairs.**

1. **Timely Dataset and Problem**:  The reasoning capabilities of LLMs remain a bottleneck in their adoption for high-stakes tasks requiring mathematical or logical precision. MWP-MISTAKE addresses this gap by providing a high-quality dataset designed to evaluate and benchmark reasoning abilities. This aligns with current research trends, including the release of models like OpenAI's O1, which claim improved reasoning but still exhibit critical limitations in mistake detection, as highlighted by our findings.

2. **Comprehensive Evaluation of Reasoning Abilities**: Our evaluations uncover several key insights:

*   SOTA models frequently flag correct reasoning chains as erroneous, casting doubt on their reliability.
*   Models struggle more with localized, subtle errors (e.g., shuffle and delete reasoning steps) compared to propagated mistakes from SLMs.
*   Mistake detection directly influences the model's ability to rectify errors, revealing an interdependence between these tasks.

3. **Data Contamination and Memorization Challenges**: Data contamination and memorization remain a fundamental challenge in LLM research, especially with black-box models like GPT, Claude, and Gemini, where training data is inaccessible. Our findings show that datasets like GSM8K and MATH have significant contamination, which inflates model performance metrics artificially. Conversely, newer datasets such as JEEBENCH reveal genuine reasoning weaknesses, underscoring the importance of our work for creating contamination-free benchmarks.

4.  **Practical Contributions to Model Evaluation**:

*   Our findings show that models like GPT-4o and O1 achieve strong performance on answering but struggle to reliably detect errors in reasoning chains.

*   The paper introduces qualitative analyses and error categorization (e.g., rule-based versus SLM-generated mistakes) to understand why models fail on certain errors.

*    These insights can guide future research in building models with improved reasoning frameworks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a dataset for evaluating LLMs’ abilities in detecting and correcting mathematical reasoning mistakes. The dataset contains math word problems with both correct and wrong reasoning chains. The proposed pipeline includes perturbations to the original reasoning chains based on (1) rule-based techniques, and (2) using less capable language models.

The reviewers generally found the problem studied to be important, the presentation to be clear, the dataset to be helpful for the community, and that the experiments insightful. 

However, reviewers found several issues regarding the paper, including:  
-The scope of the contributions of the work to be limited and incremental with respect to existing findings in the literature  
-The perturbation based method might not accurately reflect the mistakes made by human (partially addressed in rebuttal)  
-Some of the relevant work might be overlooked
-Evaluation setup needs better clarity 
-While the paper discusses several challenges such as contamination there are limited discussions or practical solutions discussed   

While the author response attempts to address several of these issues, key concerns—such as the overall incremental contribution—remain unresolved.