Step-Controlled DPO: Leveraging Stepwise Errors for Enhancing Mathematical Reasoning of Language Models

Authors: Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li
Primary area: foundation or frontier models, including LLMs
Keywords: large language model, mathematical reasoning, alignment with relative feedback

Abstract:
Direct Preference Optimization (DPO) has proven effective at improving the performance of large language models (LLMs) on downstream tasks such as reasoning and alignment. In this work, we propose Step-Controlled DPO (SCDPO), a method for automatically providing stepwise error supervision by creating negative samples of mathematical reasoning rationales that start making errors at a specified step. By applying these samples in DPO training, SCDPO can better align the model to avoid reasoning errors and output accurate reasoning steps. Qualitative analysis of the credit assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at identifying errors in mathematical solutions. We then apply SCDPO to an InternLM2-20B model, resulting in a 20B model that achieves competitive scores of 88.5\% on GSM8K and 58.1\% on MATH, rivaling all other open-source LLMs, showing the great potential of our method. The code, models and data are released to inspire future work.

Decision: reject
Reviewer scores: 5, 5, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729423793905

thread_type:
official_review

<thread_object>
creator: reviewer_1hbn

<content>
summary:
The paper introduces Step-Controlled DPO, a novel approach to improve the mathematical reasoning abilities of LLMs by leveraging stepwise errors. Unlike Direct Preference Optimization, which relies on final answers to generate preference feedback, SCDPO focuses on stepwise error generation. By creating solutions that introduce errors at specific reasoning steps, SCDPO provides more granular supervision to the model, allowing for better alignment and accuracy in mathematical reasoning. The method is evaluated on several mathematical reasoning benchmarks, showing competitive results with open-source models.

strengths:
1. The SCDPO approach introduces a unique method for improving mathematical reasoning by targeting stepwise errors, which diverges from the traditional focus on outcome-based supervision in DPO.
2. The paper conducts extensive experiments, demonstrating great performance across several mathematical reasoning benchmarks, including GSM8K and MATH. 
3. This paper provides clear theoretical explanation of SCDPO, which is necessary and provides theoretical support for this method.
4. This paper is well-structured and clear in its presentation.

weaknesses:
1. The paper mentions SCDPO works well on both CoT and Code-integrated solutions, but the results is not demonstrated this comparison. The author needs to add a table that highlights the comparison between these two different types of solutions.
2. This work lacks of the experiments on show generalisation ability of the method. The author could do an evaluation on out-of-distribution data, comparing the results with the in-distribution data.
3. This work is quite similar to another work "Step-dpo: Step-wise preference optimization for long-chain reasoning of llms", but the differences between these paper are not clearly demonstrated. The authors need to elaborate more about the comparison about this.

questions:
Line 466 misses a citation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686464703

thread_type:
official_review

<thread_object>
creator: reviewer_qgoa

<content>
summary:
This work proposes SCDPO, a method leveraging stepwise supervision without human annotation to enhance mathematical reasoning in models. The approach uses a model that is trained in math domains to generate both positive and negative samples. To produce dispreferred samples, the model is prompted with the initial portion of correct reasoning chains, then asked to complete the reasoning at a high temperature setting, leading to potential errors in the final answer. In the SCDPO loss, only the latter part of the reasoning chains is considered. Empirical results demonstrate that models trained with SCDPO outperform those trained with other methods.

strengths:
1. The paper is well-written and easy to follow.
2. The method for generating erroneous steps in SCDPO is both interesting and novel. It provides an effective approach to achieve process-based supervision without human annotation on intermediate steps. This approach allows for error localization within certain steps, which—while not precise—represents a step towards finer-grained, automatic process-based supervision.
3. The experiments are solid and convincing, the proposed method demonstrates strength across different models and benchmarks.

weaknesses:
1. The concept of a “step” in the reasoning chain is not clearly defined in the paper. It would be more clear if the authors could provide a more precise explanation of how they segment steps within a reasoning chain. 
2. Potential positional bias in error generation. The approach uses an ascending temperature strategy to create erroneous data, which may unintentionally introduce a bias where errors occur more frequently in the latter half of the reasoning chain. It would be helpful if the authors could investigate whether the distribution of errors indeed skews toward the end.
Since errors cannot be automatically located within the chain, manually sampling and inspecting some dispreferred solutions could provide insights into any potential biases.

questions:
1. (Following up on Weakness 2) Could the authors provide an analysis of the distribution of the dispreferred data? Specifically:

a. What is the distribution of the step $k$ (referenced in Line 138) that is used as the starting point for generating errors in the resulting dispreferred data?

b. What is the distribution of the "actual error" location within the dispreferred reasoning chains?

2. Line 467: There appears to be an unlinked reference.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694569286

thread_type:
official_review

<thread_object>
creator: reviewer_buso

<content>
summary:
The authors propose Step-Controlled DPO (SCDPO), which, different from DPO, provides fine-grained, step-wise training signals by creating negative samples of mathematical reasoning rationales with errors after a specified step. The authors conduct experiments using InternLM2 and Mistral customized versions, showing better performance across seen and unseen datasets, compared to SFT and DPO baselines. They also did ablations to understand the impact of temperature on negative sample generation and the benefits of integrating naive DPO with SCDPO.

strengths:
1. The proposed methodology provides fine-grained, step-wise training signals, which is missing in most DPO and RFT work done in mathematical reasoning task.
2. The explanation of the 2-step pipeline is straightforward and clear.
3. The authors conduct experiments using various models including InternLM2 and Mistral, showing better and generalizable performance  compared to SFT/DPO baselines and various open-source and closed-source models with zero-shot prompting. They also did ablations to understand the impact of temperature on negative sample generation and the benefits of integrating naive DPO with SCDPO.

weaknesses:
1. In section 4.2, the authors validate the generalizability of SCDOP on three baseline SFT models, but only show performance on the datasets that have been used for SCDOP training. It would be good to also show performance on the datasets that are not used for training, similar to table 1, to better understand its generalizability not only across models but also across datasets.
2. The paper mentions that they determine the correctness of reasoning steps by matching the final answer with the ground truth. However, it is not clear how they do the matching. Since the matching mechanism cannot be perfect as the same answer could be in different expressions, e.g. "increasing from 4 to 5" is equivalent to "increasing by 25%", it would be good to add quantitative or qualitative analysis of the matching strategy. In the other words, the paper currently only contains analysis on false negatives but not on false positives.
3. [Minor point] Typos: 
    a. line 121: "We" -> "we"
    b. figure 1.a, the order of i3 and i2 should be switched 
    c. line 366: 78 -> 78%

questions:
1. Since it's not guaranteed to have errors at the first generated step, as mentioned in the limitation, it would be good to add quantitative and qualitative analysis on which step the model generates error, to understand the granularity of the training signals. Additionally, it would be interesting to see if manipulating the distribution of the step index it generates error would cause any difference in performance.
2. Mathematical reasoning is one of reasoning types. It would interesting to see if it can expand to other reasoning tasks as well.
3. There are many "-" in table 1. Any reason?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730897757951

thread_type:
official_review

<thread_object>
creator: reviewer_b65v

<content>
summary:
The paper presents Step-Controlled Direct Preference Optimization (SCDPO), a method designed to enhance the mathematical reasoning capabilities of large language models (LLMs). SCDPO extends Direct Preference Optimization (DPO) by incorporating step-by-step error supervision data. The negative (dispreferred) samples in SCDPO is generated by model itself by adjusting the model's sampling temperature until reasoning mistakes are introduced at specific steps.  SCDPO was tested on the InternLM2-20B and Mistral-7B, achieving competitive performance on benchmark datasets like GSM8K and MATH.

strengths:
- The motivation of this paper that including step supervision in DPO for solving math problem is straightforward but reasonable.

- The step supervision is efficiently introduced by self-generation of LLMs, reducing the cost of annotation.

- The authors have demonstrated the effectiveness of the proposed method comparing to baselines on some regular benchmarks such as GSM8K and MATH

weaknesses:
- Ths most concern is the lack of many baselines (that are kind of resembling the proposed method) in experiments. There are some previous works, e.g., Step-DPO [1], introduce the idea of step supervision in DPO for enhancing the math capabilities of LLMs. This work does make some differences by using temperature to controll the step errors, the comparison should be made with previous step-supervised SFT/DPO methods to demonstrate the effectiveness.

[1] Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs


- The idea of using softmax temperature to controll LLMs to make errors at certain steps is interesting but kind of limited. The naive problems can be accurately solved by LLM step by step, while the authors carefully craft the decoding temperature to "mislead" the LLMs to generate wrong answer at certain steps. Although the step supervision can be collected between the right and wrong answers, the LLM seems to hardly be improved as it can naively solve the problem and the step supervision just reduces some potential errors it may make from random decoding. In other words, the LLM is naively easier to give the right answer but harder to give the wrong answer (need to tune the temperature), making it doubtful if the LLM can achieve much improvment from such step supervision. Have you tried using a larger LLM to generate the step supervision data to improve smaller LLMs? It may be better than the self-improvement of smaller LLMs.

- The paper writing and structure can be improved. Some important informations are missed in this paper,  for example, it's unclear what the hyperparameters (top-p, top-k) used for generating training data, and how many samples the final training data contains. The cite of  Process Reward Synthesizing in related work is also missed.

questions:
See weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731122080678

thread_type:
official_review

<thread_object>
creator: reviewer_5qbr

<content>
summary:
This paper proposes Step-Controlled DPO, a new data-curating methodology that improves the effectiveness of DPO fine-tuning on reasoning tasks. When DPO is applied to fine-tune LLMs on reasoning datasets, one naive way of curating the dataset is to sample pairs of correct/wrong reasoning chains from LLMs. The authors of this work make the observation that when LLMs generate incorrect reasoning chains, they usually start making mistakes after a specific position; in particular, a reasoning chain is incorrect does not mean that the whole chain is incorrect, and its prefix could potentially be correct. Hence, the authors propose to generate negative examples by taking a correct reasoning chain and resample its suffix starting from some position k, which they refer to as SC-DPO. The authors combine the naively generated data with the data generated from SC-DPO, and conduct DPO fine-tuning. Compared to the baseline where only naively generated data is used, their approach show consistent improvement across different models and datasets.

strengths:
The approach presented is simple, effective and immediately applicable to a wide range of logical/mathematical reasoning benchmarks.

weaknesses:
The contribution overall seems relatively minor: SC-DPO is not really a new fine-tuning objective but instead the original DPO objective combined with a rather specific way of generating negative examples; the empirical gains are also relatively small.

Most of the information from Table 1. is probably not super important, e.g. why do the authors need to compare the performance of InternLM2-SFT-SCDPO against the listed open-source LLMs? how is this comparison related to SC-DPO?

Section 3 mentions MDP without defining what it refers to.

Eq (3) (4) and (6) seem somewhat redundant. E.g. to me Eq. (6) seems to be Eq. (4) expanded to its token-level and Eq. (6) itself is probably enough for describing the approach and explaining the intuition.

questions:
the authors generate positive & negative examples by sampling. For some of the more challenging reasoning problems it is probably impossible to get a single positive example out of the 100 samples. Can the authors provide the statistics on that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732245508576

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Update Summary

comment:
Thanks to all the reviewers for their constructive suggestions and comments. Below, we summarize the details of the revised paper.

1. Evaluation results of 5 out-of-domain datasets on 7B models are added to Table 2. (Reviewer buSo, Reviewer 1hBn, Reviewer 5qbR)
2. Evaluation results of several important baseline models are added to Table 1. (Reviewer buSo)
3. Definition and reference of MDP (Markov Decision Process) is added to Section 3 on page 4. (Reviewer 5qbR)
4. The unlinked reference in line 466 is fixed in the related work. (Reviewer b65V, Reviewer qGoa, Reviewer 1hBn)
5. Eq. (4) and Eq. (6) are combined into one equation in Section 2.2 on page 4. (Reviewer 5qbR)
6. Statistics of the coverage of correct solutions is added to Section 2.1 on page 3. (Reviewer 5qbR)
7. Comparison with Step-DPO is added to Table 4 and Section 4.2 on page 9, as well as the related work section. (Reviewer b65V, Reviewer 1hBn)
8. Hyperparameters used for generating training data, and the number of samples in the final training data are added to Section 4.1 on page 6. (Reviewer b65V)
9. Typos are fixed: "We" is changed to "we" on page 3. The order of i3 and i2 is switched in Figure 1. (Reviewer buSo)
10. Quantitative and qualitative analysis of the location of actual errors and its relation to the starting point of error generation is added to Appendix E. (Reviewer buSo, Reviewer qGoa)
11. Explanation of the concept of a “step” is added to Section 2 on page 3. (Reviewer qGoa)
12. Results of manipulating the distribution of the step index where error generation begins is added  to Table 7. (Reviewer buSo)

The valuable suggestions and insightful comments from the reviewers have been invaluable in improving this project. We sincerely appreciate the time and effort the reviewers and chairs have dedicated to evaluating our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents Step-Controlled DPO (SCDPO), a method that introduces stepwise error supervision into DPO training for improving mathematical reasoning in LLMs. The main idea is to automatically generate negative samples by making the model produce errors at specific steps through temperature adjustment, then using these paired samples for DPO training. The paper demonstrates effectiveness by achieving competitive scores on GSM8K (88.5%) and MATH (58.1%) using InternLM2-20B. The key strengths include: 1) A simple yet effective approach to incorporate stepwise supervision without human annotation; 2) Extensive experiments showing consistent improvements across different models and datasets; 3) Clear presentation and analysis. However, there are several notable weaknesses: 1) The contribution appears relatively minor as it mainly introduces a specific way of data generation rather than a novel training objective; 2) The empirical gains on challenging datasets like MATH are relatively small (0.6% vs naive DPO); 3) Improving LLMs through self-generated data rather than leveraging larger models for data generation presents a fundamental limitation, as the model can only learn from its own generated samples. Given these limitations in technical novelty and empirical results, I recommend rejecting this paper despite its good presentation and systematic evaluation.