Subtle Errors Matter: Preference Learning via Error-injected Self-editing

Authors: Kaishuai Xu, Tiezheng YU, Wenjun Hou, Yi Cheng, Chak Tou Leong, Liangyou Li, Xin Jiang, Lifeng Shang, Qun Liu, Wenjie Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Mathematical Reasoning, Preference Learning

Abstract:
Large Language Models (LLMs) have exhibited strong mathematical reasoning and computational prowess, tackling tasks ranging from basic arithmetic to advanced competition-level problems. However, frequently occurring subtle errors, such as miscalculations or incorrect substitutions, limit the models’ full mathematical potential. Existing studies to improve mathematical ability typically involve distilling reasoning skills from stronger LLMs or applying preference learning to step-wise response pairs. Although these methods leverage samples of varying granularity to mitigate reasoning errors, they overlook the frequently occurring subtle errors. A major reason is that sampled preference pairs involve differences unrelated to the errors, which may distract the model from focusing on subtle errors. In this work, we propose a novel preference learning framework called eRror-Injected Self-Editing (RISE), which injects predefined subtle errors into partial tokens of correct solutions to construct hard pairs for error mitigation. In detail, RISE uses the model itself to edit a small number of tokens in the solution, injecting designed subtle errors. Then, pairs composed of self-edited solutions and their corresponding correct ones, along with pairs of correct and incorrect solutions obtained through sampling, are used together for subtle error-aware DPO training. Compared with other preference learning methods, RISE further refines the training objective to focus on predefined errors and their tokens, without requiring fine-grained sampling or preference annotation. Extensive experiments validate the effectiveness of RISE, with preference learning on Qwen2-7B-Instruct yielding notable improvements of 3.0% on GSM8K and 7.9% on MATH.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730066253905

thread_type:
official_review

<thread_object>
creator: reviewer_r3f2

<content>
summary:
This paper proposes a novel approach to enhancing the performance of language models in mathematical problem-solving by introducing noise into correct solutions to create subtly incorrect answers. These correct/incorrect answer pairs are then utilized to fine-tune a DPO (Direct Preference Optimization) model. Empirical evidence demonstrates the effectiveness of this approach in improving the model’s accuracy.

strengths:
1. The paper effectively addresses the challenge of subtle error detection, a common issue faced by large language models (LLMs). The proposed method successfully enhances models, transforming relatively weaker ones into stronger versions.
1. The technique leverages the simplicity of generating incorrect solutions rather than correct ones, making the training process more efficient. By instructing models to produce errors, it harnesses the LLM’s capability to identify likely mistakes based on its own tendencies, leading to a self-improvement mechanism.

weaknesses:
1. The effectiveness of this method heavily relies on prompt engineering. The quality and specificity of the prompts used to generate incorrect answers significantly influence the quality and subtlety of the generated mistakes.
1. As the approach is based on pre-defined templates, it may not capture the full spectrum of potential errors a language model might make, leaving certain blind spots unaddressed.
1. The scalability of the proposed method remains uncertain, as it primarily focuses on generating diverse incorrect answers rather than ensuring diversity in correct solutions. This might limit its applicability in more complex or varied scenarios.

questions:
1. Do you think this method can be extended to other domains where subtle mistake detection is crucial, such as logical reasoning and programming? If so, what adaptations would be necessary?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730590601847

thread_type:
official_review

<thread_object>
creator: reviewer_nbyg

<content>
summary:
This paper presents a model fine-tuning method that aims to solve the math problem of LLMs. Specifically, the paper reuses the existing training paradigm (RLFH or DPO concretely) but pays attention to the training pair generation. The negative sample is generated by prompting LLM with intentional instruction on producing wrong answers (with particular error types described in Section 2.1). The main motivation comes from the hypothesis that the existing fine-tuning solution does not provide targetted training pairs, causing the fine-tuned model to capture subtle errors that are not intended. 

The training objectives are borrowed from previous works, including DPO, step-wise DPO, and negative log-likelihood loss (to stabilize the training). The contribution of this paper is more about how to get better negative samples. Experiments show the proposed approach provides reasonable improvement on small models (7B) but with limited improvement on large models (70B). Compared to the DPO solution previously used, the improvement is also not very significant.

strengths:
The paper is well-written and easy to understand. The majority of the descriptions are with clear equations or demonstrative figure supports. While some contents are not included in the paper, they are well-known in the literature. 

Actively looking for hard negative examples is always a research topic. The paper focuses on generating regularized negatives (from the list of error types) such as enforcing the fine-tuned model to avoid making similar mistakes. The strategy is letting LLM provide the wrong examples as part of forward passing over steps.  

Experiments included multiple LLMs in comparison. While most of them are not intended to solve math problems, it is good to have more things to compare.

weaknesses:
The most obvious weakness of this paper is the lack of solid evidence of motivation. While the paper stated that negative samples randomly generated may be unrelated to error, there is no solid evidence on how it impacts the model's math ability. The research gap has yet to be explicitly visible so far. The authors should provide sufficient insight into how those randomly sampled negatives hurt the performance with solid evidence. I am not very convinced this issue is significant. Considering the limited performance improvement on large models (70B), I am concerned if this is indeed a problem of existing works.

Most of the approaches used in this paper are existing training methods. The novelty of this paper is weak, given it is simply proposing a negative sample generation approach. The majority of the training descriptions are optional, given they are well-known. E.g. DPO, step-wise DPO, or NLL. The authors give good justification for why using prompts to generate the wrong sample is an impactful contribution. 

The last point of concern is the performance improvement of large models. The proposed method does not seem to improve the performance of the large models much. Why is that? It seems not an improvement space problem since the large LLMs still suffer from math problems with many errors. However, even after this fine-tuning, the proposed solution fails to take care of them. Any insight? Is it indicating the proposed solution is going in the wrong direction for solving this problem?

questions:
1. Why does this solution fail to improve the math of large models compared to the models fine-tuned on random negative samples? If someone uses a very simple negative sampling approach by generating random numerical values, is there any performance difference from the proposed solution?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601699574

thread_type:
official_review

<thread_object>
creator: reviewer_kism

<content>
summary:
The paper presents a novel preference learning framework known as eRror-Injected Self-Editing (RISE), which is designed to enhance the mathematical reasoning capabilities of Large Language Models (LLMs). The core contribution of RISE is its innovative approach to error mitigation by injecting subtle, predefined errors into correct solutions, creating hard pairs for training that help the model focus on common mistake patterns.
The framework operates by using the LLM to generate correct multi-step solutions and then intentionally introducing errors into these solutions to form self-edited pairs. These pairs, along with correctly and incorrectly solved samples, are used for subtle error-aware Direct Preference Optimization (DPO) training. The paper reports improvements on mathematical reasoning tasks when RISE is applied to LLMs like Qwen2-7B-Instruct, and shows improvement on the GSM8K and MATH dataset. These results demonstrate the effectiveness of RISE in refining the training objective to target subtle error tokens and in improving the model's ability.

strengths:
1. Originality: The paper introduces RISE, a novel preference learning framework that addresses the subtle error problem in LLMs, which is not commonly seen in other works.
2. Quality: The paper compares RISE with other preference learning methods and shows that it outperforms them, which shows the quality of the approach.
3. Significance: The paper makes a significant contribution to the field of LLMs by providing a method to improve their mathematical reasoning capabilities, which is a critical area for LLM development.

weaknesses:
1. Generalizability and Domain-Specificity: The paper focuses exclusively on mathematical reasoning tasks. It would be beneficial to see how RISE performs in other domains, such as reasoning and coding, where subtle errors also play a significant role.
2. Diversity of Error Types: While the paper addresses common error types in mathematical reasoning, it may not cover all possible error categories. For instance, it might be worth exploring errors related to the interaction of different error types.
3. Experiments: The authors should evaluate the performance of RICE on more open-source models (such as Mistral-7B-Instruct-v0.3, qwen 2.5 series). And the influence of hyperparameter $\alpha$ should also be explored.

questions:
1. The interaction of different error types: Will the interaction of different error types improve the performance?
2. Number of self-edited pairs: Why the performance on GSM8K and MATH both decrease with more self-edited pairs? Please further analyse this pheonomenon to pre-define the number of self-edited pairs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696503264

thread_type:
official_review

<thread_object>
creator: reviewer_crh2

<content>
summary:
This paper proposes a preference learning framework called RISE for DPO training. The key idea is to inject carefully created noise in the correct-incorrect answer pairs to guide the model which mistakes to avoid.

strengths:
The paper studies how to optimize preference learning, a timely and important topic. The proposed method RISE is easy to understand, and showcases empirical performance gains on two commonly used open-source models, qwen and llama.

weaknesses:
My major concern is the scope of the performance evaluation. Across all experiments in this paper, the models are trained on a single dataset extracted from Lai et al 2024. It is not clear if the results are cherry-picked or not. For example, how about other DPO training datasets for Math and other datasets? In addition, it is not clear how RISE affects the models' quality on other tasks. The authors classify the 6 evaluation datasets used in this paper as either "in-distribution" or "out-of-distribution", but they are all indeed math-related questions. It would be more desired to see evaluation on real out-of-distribution datasets, such as physics or logic heavy datasets (e.g., ZebraLogic or ARC corpus).

questions:
See my weakness comment.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732507359589

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by Authors

comment:
Dear Reviewers,
\
\
We sincerely thank you for thoroughly assessing our work and providing us with valuable and constructive feedback.

Over the past few days, we have worked diligently to address your feedback and questions via additional experiments and detailed explanations. If you have any further queries or require clarifications, we would be happy to continue the discussion. We would also greatly appreciate it if you could reconsider the rating of our work based on the responses and updates we have provided during the rebuttal period.

Once again, we greatly appreciate your valuable contributions to improving our work.
\
\
Sincerely,

Authors of Paper #10957
</content>
</thread_object>

</thread>
<thread>
creation date:
1732609479062

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all the reviewers for their insightful and constructive feedback on our manuscript. We have carefully addressed each of the individual comments in the reviews and believe we have successfully responded to most of the concerns raised. Additionally, we have incorporated the suggested experiments, along with their discussions and results, in the revised manuscript.

Below, we provide a brief summary of the updates made in the revision, including: **(1) Core Contributions**, **(2) Strengths**, **(3) Updates During Rebuttal**

---

### **Core Contributions**

1. **Novel Framework**. We propose a novel preference learning framework that leverages the LLM itself to **inject errors** into correct solutions, constructing fine-grained hard pairs designed to **mitigate subtle errors**.
2. **Empirical Analysis**. Our study **identifies common error types** in mathematical reasoning and reveals the potential to improve the stability of the reasoning process by mitigating subtle errors.
3. **Experimental Results**. Through extensive experiments across various models (**Qwen2**, **Qwen2.5**, **Llama-3.1**, and **Ministral**) and tasks (**mathematical reasoning**, **logical reasoning**, and **code generation**), we demonstrate that RISE improves the reasoning capabilities of LLMs, helping LLMs mitigate subtle errors and consistently generate correct solutions.
4. **Transferability of Reasoning Ability across Diverse Domains.** Additional evaluation experiments on logical reasoning (ZebraLogic) and code generation (MBPP and Humaneval) show that our method can effectively generalize reasoning preferences learned from mathematical tasks to other complex reasoning domains even **without further in-domain training**.
5. **Flexible Adaptation to New Reasoning Tasks**. Our RISE framework has demonstrated its effectiveness in optimizing code generation, with **minimal modifications to editing prompts**. It shows that RISE can be flexibly and conveniently adapted to new tasks.

### **Strengths**

1. **Novelty of Method**. ``Reviewer kiSm`` and ``Reviewer r3F2`` agreed that our framework RISE is a novel and effective approach for preference learning.
2. **Significance**. ``Reviewer kiSm`` and ``Reviewer r3F2`` affirm that our study makes a contribution to the field of LLMs by focusing on subtle errors and improving mathematical reasoning ability.
3. **Writing and Presentation**. ``Reviewer CRH2`` and ``Reviewer nbYg`` praised the clarity and readability of our writing and presentation.
4. **Simplicity and Effectiveness**. ``Reviewer r3F2`` recognizes that our framework RISE is simple and effective.
5. **Self-improvement**. ``Reviewer r3F2`` approves that letting LLMs identify likely mistakes and optimizing based on error-injected self-edited pairs can lead to a self-improvement mechanism.

### **Updates During Rebuttal**

1. **Section 3.5**: Add evaluation results and analysis on out-of-domain reasoning tasks (logical reasoning and code generation).
2. **Appendix C**: Add validation experiments on more open-source models (Ministral-8B-Instruct and Qwen2.5-7B-Instruct).
3. **Appendix D**: Add validation experiments on another mathematical training dataset, including 15K problems from the original GSM8K and MATH training datasets.
4. **Appendix E**: Add exploration experiments of the hyperparameter $\alpha$.
5. **Appendix F**: Add exploration experiments of prompt designs, including self-instruct prompts and prompts that introduce arbitrary errors without specifying a particular mistake.
6. **Appendix G**: Add validation experiments on other reasoning tasks, such as code generation.

We believe these additions and clarifications comprehensively address the reviewers' concerns and enhance the overall quality of our manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presented an interesting method of generating hard negative preference pair construction through error-injected self-editing. The proposed method enhances the mathematical reasoning capability of LLMs by subtle error-aware DPO training.

Strength: 
1. A useful method to generating hard negative examples to improve the reasoning capabilities of LLMs.
2. The method appeared to be novel, though the novelty might not be significant (An interesting negative sampling approach).

Weakness:
1. The improvement over existing methods is not very significant.
2. Method is heavily depending on prompt engineering and templates, which could limit its general applicability.

The paper is borderline. My major concern (similar to reviewer nbYg) is that the improvement on larger models is relatively minor (in the noisy region), signaling the method might not scale well. This could be seen in other tasks other than Math as questioned by reviewer CRH2.