Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs

## Abstract

Abstract Mathematical reasoning presents a significant challenge for Large Language Models (LLMs) due to the extensive and precise chain of reasoning required for accuracy. Ensuring the correctness of each reasoning step is critical. To address this, we aim to enhance the robustness and factuality of LLMs by learning from human feedback. However, Direct Preference Optimization (DPO) has shown limited benefits for long-chain mathematical reasoning, as models employing DPO struggle to identify detailed errors in incorrect answers. This limitation stems from a lack of fine-grained process supervision. We propose a simple, effective, and data-efficient method called Step-DPO, which treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Additionally, we have developed a data construction pipeline for Step-DPO, enabling the creation of a high-quality dataset containing 10K step-wise preference pairs. We also observe that in DPO, self-generated data is more effective than data generated by humans or GPT-4, due to the latter’s out-of-distribution nature. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters. Notably, Step-DPO, when applied to Qwen2-72B-Instruct, achieves scores of 70.8% and 94.0% on the test sets of MATH and GSM8K, respectively, surpassing a series of closed-source models, including GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro. Our code, data, and models are available at https://github.com/dvlab-research/Step-DPO .

## 1 Introduction

Figure: Figure 1: Accuracy on the MATH test set across models fine-tuned by Step-DPO and other state-of-the-art models. ^†: reproduced result using our prompt.
Refer to caption: x1.png

Figure: Figure 2: Left: Accuracy of judging preferred or undesirable outputs on the validation set during training. Right: Reward margins between preferred and undesirable outputs on the validation set during training. More details about these experiments are given in the appendix.
Refer to caption: x2.png

Mathematical reasoning is recognized as a critical long-chain reasoning ability in Large Language Models (LLMs). This task is particularly challenging due to the often extensive chain of thought required, which can include numerous reasoning steps. Any error in these steps can lead to an incorrect final answer.

Numerous studies (Yu et al., ; Luo et al., ; Yue et al., ; Liu & Yao, ; Lu et al., ; Li et al., ; Shao et al., ; Xin et al., ; Yue et al., ; Tang et al., ) have proposed various data augmentation techniques during the supervised fine-tuning (SFT) stage to enhance alignment. However, models in the SFT process are prone to hallucinations, resulting in saturated performance. A potential reason for this, as highlighted in Hong et al. (), is that as the probability of preferred outputs increases, so does the probability of undesirable ones. This phenomenon makes the model more likely to make errors in long-chain reasoning. Therefore, it is essential to develop methods to suppress the likelihood of undesirable outputs.

Recently, Direct Preference Optimization (DPO) (Rafailov et al., ) has been proposed for alignment using pair-wise preference data and is popular due to its simplicity. Despite its effectiveness in chat benchmarks (Tunstall et al., ; Zheng et al., ), DPO offers minimal benefits for long-chain mathematical reasoning. As shown in Fig. 2 (left), models using vanilla DPO perform poorly in distinguishing between preferred and undesirable outputs, failing to identify errors in rejected answers. Additionally, Fig. 2 (right) shows that the reward margin (i.e., the gap between the rewards of preferred and undesirable outputs) is limited for models using vanilla DPO and plateaus with further training. These findings indicate that models fine-tuned with vanilla DPO cannot pinpoint detailed errors in incorrect answers, hindering the improvement of reasoning abilities.

In this work, we introduce Step-DPO, where each intermediate reasoning step is treated as the basic unit for preference optimization. As illustrated in Fig. 3, unlike vanilla DPO, which only considers preference optimization between complete answers (i.e., $p(y_{win}|x)$ and $p(y_{lose}|x)$), Step-DPO examines the step-by-step answer (i.e., $y=s_{1},...,s_{n}$) and specifically targets the first erroneous reasoning step. Step-DPO aims to select a correct reasoning step and reject an incorrect one, given a math problem and several initial correct reasoning steps (i.e., maximize $p(s_{win}|x;s_{1},s_{2},...,s_{k-1})$ and minimize $p(s_{lose}|x;s_{1},s_{2},...,s_{k-1})$). This transition allows the model to easily locate erroneous tokens for effective optimization, significantly enhancing long-chain reasoning.

Moreover, we present an effective and economical pipeline to collect pair-wise preference data, resulting in a high-quality dataset for Step-DPO. This dataset contains approximately 10K samples, each consisting of: 1) a mathematical problem, 2) prior reasoning steps, 3) the chosen step, and 4) the rejected step. Our three-step pipeline for dataset construction includes: 1) Error collection, 2) Step localization, and 3) Rectification. Notably, the chosen reasoning step is generated by the model itself, as we find that in-distribution data (i.e., self-generated data) is more effective than out-of-distribution data (e.g., data written by humans or GPT-4) for Step-DPO, as shown in Table 4.

With this curated dataset, mathematical reasoning performance can be significantly boosted with only hundreds of training steps, as demonstrated in Fig. 6. For instance, fine-tuning Qwen-72B-Instruct with Step-DPO results in a model achieving 70.8% accuracy on MATH and 94.0% on GSM8K, surpassing a series of closed-source models, including GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro.

Figure: Figure 3: Comparison between DPO and Step-DPO.
Refer to caption: x4.png

## 2 Related Works

### 2.1 Mathematical Reasoning

Large Language Models (LLMs) have exhibited substantial reasoning capabilities, primarily due to their auto-regressive nature, which allows them to predict the next token based on contextual information. However, these models still struggle with long-chain reasoning tasks, particularly in mathematical contexts. Several prior studies (Yao et al., ; Chen et al., ; Yoran et al., ; Li et al., ; Tong et al., ; Fu et al., ; Zhou et al., ) have attempted to enhance the Chain-of-Thought (CoT) inference framework (Wei et al., ) to address this issue. While these efforts have led to significant improvements in certain tasks, they have not fully mitigated common hallucinations and have limited generalizability across all reasoning tasks.

Another research direction (Yu et al., ; Luo et al., ; Yue et al., ; Liu & Yao, ; Lu et al., ; Xu et al., ; Li et al., ; Shao et al., ; Xin et al., ; Zhou et al., ; Liu et al., ; Ying et al., ; Yue et al., ; Tang et al., ; Mitra et al., ; Yuan et al., ) focuses on various data augmentation techniques, such as rephrasing, extension, and evolution, for supervised fine-tuning (SFT). These methods have significantly enhanced the reasoning abilities of LLMs, but their performance plateaus once the data reaches a certain volume. Additionally, methods like those proposed by Wang et al. (); Liao et al. (); Toshniwal et al. (); Gou et al. () employ external tools, such as Python, to substantially reduce calculation errors.

Other approaches (Azerbayev et al., ; Shao et al., ; Lin et al., ; Ying et al., ; Wang et al., ) involve continued pre-training on extensive, high-quality math-related datasets, which markedly improve mathematical reasoning capabilities. Recent studies (Xu et al., ; Ying et al., ) have explored reinforcement learning to mitigate hallucinations in mathematical reasoning. Works like Lightman et al. (); Shao et al. (); Wang et al. () emphasize the importance of step-by-step verification in reinforcement learning for mathematical problems. However, these methods still rely on the quality of the reward model and require the complex training pipelines of RLHF. Building on this line of research, we propose Step-DPO, a simpler, more effective, and more efficient method.

### 2.2 Reinforcement Learning from Human Feedback

Supervised fine-tuning (SFT) can align models with human preferences. However, as the probability of preferred outputs increases, so does the likelihood of undesirable ones, leading to hallucinations. To generate more reliable outputs, Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., ; Ouyang et al., ) has been introduced for LLM alignment. This approach involves training a reward model with comparison data and then using this reward model to optimize the policy model. The final performance heavily depends on the quality of the reward model, and the training pipeline is quite complex.

To simplify this process, Direct Preference Optimization (DPO) (Rafailov et al., ) was proposed, which directly uses pair-wise preference data for model optimization. This transition significantly streamlines the training pipeline. While DPO has proven effective in chat benchmarks, it offers only marginal benefits for mathematical reasoning. Inheriting the principles of DPO, Step-DPO is specifically designed for long-chain reasoning and has shown significant performance improvements in solving math word problems.

## 3 Step-DPO

In this section, we elaborate on the proposed Step-DPO. First, we present step-wise formulation in Sec. 3.1, a novel approach designed to enhance long-chain reasoning abilities by building on DPO. Next, in Sec. 3.2, we illustrate a pipeline for constructing the step-wise preference dataset for Step-DPO. Both components are essential for achieving the desired performance improvements.

### 3.1 Step-wise Formulation

#### Preliminary.

Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., ) is an effective approach for enhancing the robustness, factuality, and safety of LLMs (Ouyang et al., ). RLHF consists of two training phases: 1) reward model training, and 2) policy model training. However, the final performance of RLHF is highly sensitive to various hyperparameters in both phases, necessitating meticulous tuning.

To avoid this complex training pipeline, Rafailov et al. () proposed Direct Preference Optimization (DPO), which directly uses pair-wise preference data to optimize the policy model with an equivalent optimization objective. Specifically, given an input prompt $x$, and a preference data pair $(y_{win},y_{lose})$, DPO aims to maximize the probability of the preferred output $y_{win}$ and minimize that of the undesirable output $y_{lose}$. The optimization objective is formulated as:

$$ $\displaystyle\begin{aligned} \mathcal{L}_{DPO}(\theta)=-\mathbb{E}_{(x,y_{win} ,y_{lose})\sim D}[\log\sigma(\beta\log\frac{\pi_{\theta}(y_{win}|x)}{\pi_{ref} (y_{win}|x)}-\beta\log\frac{\pi_{\theta}(y_{lose}|x)}{\pi_{ref}(y_{lose}|x)})] ,\end{aligned}$ (1) $$

where $D$ is the pair-wise preference dataset, $\sigma$ is the sigmoid function, $\pi_{\theta}(\cdot|x)$ is the policy model to be optimized, $\pi_{ref}(\cdot|x)$ is the reference model kept unchanged during training, and the hyperparameter $\beta$ controls the distance from the reference model.

#### Our Solution.

While DPO has proven effective in chat benchmarks, it brings only marginal improvements for long-chain reasoning tasks such as mathematical problems, as shown in Fig. 2 and Table 3. This limitation arises because most undesirable answers in these tasks do not contain errors initially; the first error often appears midway through the reasoning process. Rejecting an entire undesirable answer in DPO may also discard preceding correct reasoning steps, introducing significant noise and negatively impacting training.

Analogous to how teachers correct students by pinpointing specific errors rather than dismissing entire answers, our proposed Step-DPO provides more detailed supervision by identifying the specific erroneous reasoning step. This granular focus allows the model to swiftly locate, rectify, and avoid erroneous steps.

Specifically, the answer $y$ can be decomposed into a sequence of reasoning steps $y=s_{1},\ldots,s_{n}$, where $s_{i}$ is the $i$-th reasoning step. As illustrated in Fig. 3, given a prompt $x$ and a series of initial correct reasoning steps $s_{1\sim k-1}=s_{1},\ldots,s_{k-1}$, Step-DPO aims to maximize the probability of the correct next reasoning step $s_{win}$ and minimize the probability of the incorrect one $s_{lose}$. This objective can be formulated as:

$$ $\displaystyle\begin{aligned} \mathcal{L}(\theta)=-\mathbb{E}_{(x,s_{1\sim k-1} ,s_{win},s_{lose})\sim D}[\log\sigma(\beta\log\frac{\pi_{\theta}(s_{win}|x;s_{ 1\sim k-1})}{\pi_{ref}(s_{win}|x;s_{1\sim k-1})}-\beta\log\frac{\pi_{\theta}(s _{lose}|x;s_{1\sim k-1})}{\pi_{ref}(s_{lose}|x;s_{1\sim k-1})})].\end{aligned}$ (2) $$

### 3.2 In-distribution Data Construction

Figure: Figure 4: Data construction pipeline for Step-DPO.
Refer to caption: x5.png

Figure: Figure 5: An example of preference data sample for Step-DPO.
Refer to caption: x6.png

According to the optimization target of Step-DPO, we need to create a corresponding high-quality pair-wise preference dataset. Each data sample should comprise four entries: 1) prompt $x$; 2) initial reasoning steps $s_{1\sim k-1}=s_{1},\ldots,s_{k-1}$; 3) preferred reasoning step $s_{win}$; 4) undesirable reasoning step $s_{lose}$, as shown in Fig. 5. To obtain a high-quality dataset, we propose a data construction pipeline illustrated in Fig. 4, which includes the following three steps.

#### Error collection.

First, we collect a set $D_{0}=\{(x,\hat{y})\}$ of mathematical problems $x$ with ground-truth answers $\hat{y}$. Each mathematical problem $x$ is then used as a prompt to infer answers using the initial model $\pi_{ref}$. Before inference, we add the step-wise Chain-of-Thought (CoT) prefix for prompting, i.e., "Let’s think step by step. Step 1:". This ensures that the model’s inference results are structured into multiple reasoning steps, with each step explicitly starting with "Step i:". Upon completion of inference, we obtain the model answers $y$ for each mathematical problem $x$. We then select instances where the final answer $y$ differs from the ground truth $\hat{y}$, resulting in a dataset of erroneous inference results, denoted as $D_{1}=\{(x,\hat{y},y)|x\in D_{0}\}$.

#### Step localization.

Given that each erroneous inference result is explicitly presented as a sequence of reasoning steps $y=s_{1},s_{2},\ldots,s_{n}$, we proceed to verify the correctness of each reasoning step until we find the first error and record its step number $k$. This process can be done manually or using GPT-4. We select $s_{k}$ as the erroneous reasoning step $s_{lose}$, resulting in a dataset that contains the erroneous steps, denoted as $D_{2}=\{(x,\hat{y},s_{1\sim k-1},s_{lose})|x\in D_{1}\}$.

#### Rectification.

To obtain the corresponding correct reasoning step for each sample in $D_{2}$, we need to sample multiple outputs $y_{cont}$ by inferring the model $\pi_{ref}$ with the prompt $x$ and the preceding correct reasoning steps $s_{1\sim k-1}$. This process is formulated as:

$$ $\displaystyle\begin{aligned} y_{cont}\sim\pi_{ref}(y|x;s_{1\sim k-1}).\end{aligned}$ (3) $$

We retain those outputs where the final answer matches the ground truth. Among the remaining outputs, we select the first reasoning step in $y_{cont}$ as $s_{win}$, resulting in the final dataset $D=\{(x,s_{1\sim k-1},s_{lose},s_{win})|x\in D_{2}\}$. An example of a resulting data sample is shown in Fig. 5.

Notably, some cases may have correct final answers but erroneous intermediate reasoning steps. Therefore, we may need to further filter out samples where $s_{win}$ is incorrect, which can be done manually or by GPT-4. We omit this process in the notations for simplicity, and more details are provided in the appendix.

It is important to note that the data pipeline is user-friendly. In this data pipeline, humans or GPT-4 are only required to locate errors and rank answers, and they do not need to write answers or rectifications by themselves.

We also note that the use of in-distribution data is crucial. When selecting $s_{win}$, we use outputs generated by the model $\pi_{ref}$ rather than answers rectified by humans or GPT-4. Since human or GPT-4 rectified answers $s_{win}^{ood}$ are out-of-distribution (OOD) regarding the model $\pi_{ref}$, the log-probability of outputting $s_{win}^{ood}$ (i.e., $\log\pi_{ref}(s_{win}^{ood}|x)$) is significantly lower than that of an in-distribution (ID) output $\log\pi_{ref}(s_{win}^{id}|x)$. Moreover, it is challenging for the policy model $\pi_{\theta}$ to learn to increase the probability of $s_{win}^{ood}$ due to gradient decay issues (detailed in the appendix). Consequently, adopting self-generated in-distribution data as the preferred answer proves to be a more effective way of aligning with human preferences.

## 4 Experiments

In this section, we first introduce the experimental setup in Sec. 4.1. Then, we present the main results in Sec. 4.2, which include an exhaustive performance comparison. Moreover, we conduct an extensive ablation study in Sec. 4.3. Finally, a few demonstrations are shown in Sec. 4.4 to further understand Step-DPO.

### 4.1 Experimental Setup

#### Network Architecture.

Our experiments are based on various base models, including the Qwen2 and Qwen1.5 series (Bai et al., ), Meta-Llama-3-70B (Touvron et al., ), and deepseek-math-7b-base (Shao et al., ).

**Table 1: Math reasoning performance comparison on MATH and GSM8K across various models. general: general-purpose model. open: open-source.**
| Model | size | general | open | MATH (%) | GSM8K (%) |
| --- | --- | --- | --- | --- | --- |
| GPT-3.5-Turbo | - | ✓ | ✗ | 42.5 | 92.0 |
| Gemini-1.5-Pro (Feb) (Reid et al., ) | - | ✓ | ✗ | 58.5 | 91.7 |
| Gemini-1.5-Pro (May) (Reid et al., ) | - | ✓ | ✗ | 67.7 | 90.8 |
| Claude-3-Opus | - | ✓ | ✗ | 60.1 | 95.0 |
| GPT-4-1106 (Achiam et al., ) | - | ✓ | ✗ | 64.3 | 91.4 |
| GPT-4-Turbo-0409 (Achiam et al., ) | - | ✓ | ✗ | 73.4 | 93.7 |
| GPT-4o-0513 | - | ✓ | ✗ | 76.6 | 95.8 |
| Llama-3-8B-Instruct (Touvron et al., ) | 8B | ✓ | ✓ | 30.0 | 79.6 |
| Qwen2-7B-Instruct (Bai et al., ) | 7B | ✓ | ✓ | 49.6 | 82.3 |
| Llama-3-70B-Instruct (Touvron et al., ) | 70B | ✓ | ✓ | 50.4 | 93.0 |
| DeepSeek-Coder-V2-Instruct (Zhu et al., ) | 236B | ✗ | ✓ | 75.7 | 94.9 |
| Code-Llama-7B (Roziere et al., ) | 7B | ✗ | ✓ | 13.0 | 25.2 |
| MAmooTH-CoT (Yue et al., ) | 7B | ✗ | ✓ | 10.4 | 50.5 |
| WizardMath (Luo et al., ) | 7B | ✗ | ✓ | 10.7 | 54.9 |
| MetaMath (Yu et al., ) | 7B | ✗ | ✓ | 19.8 | 66.5 |
| MetaMath-Mistral-7B (Yu et al., ) | 7B | ✗ | ✓ | 28.2 | 77.7 |
| MathScale-Mistral Tang et al. () | 7B | ✗ | ✓ | 35.2 | 74.8 |
| InternLM-Math-7B (Ying et al., ) | 7B | ✗ | ✓ | 34.6 | 78.1 |
| Xwin-Math-Mistral-7B (Li et al., ) | 7B | ✗ | ✓ | 43.7 | 89.2 |
| MAmmoTH2-7B-Plus (Yue et al., ) | 7B | ✗ | ✓ | 45.0 | 84.7 |
| MathGenieLM-Mistral (Lu et al., ) | 7B | ✗ | ✓ | 45.1 | 80.5 |
| InternLM-Math-20B (Ying et al., ) | 20B | ✗ | ✓ | 37.7 | 82.6 |
| MathGenieLM-InternLM2 (Lu et al., ) | 20B | ✗ | ✓ | 55.7 | 87.7 |
| DeepSeekMath-Instruct (Shao et al., ) | 7B | ✗ | ✓ | 46.8 | 82.9 |
| DeepSeekMath-RL (Shao et al., ) | 7B | ✗ | ✓ | 51.7 | 88.2 |
| DeepSeekMath-RL + Step-DPO | 7B | ✗ | ✓ | 53.2 (+1.5) | 88.7 (+0.5) |
| DeepSeekMath-Base-SFT^† | 7B | ✗ | ✓ | 52.9 | 86.7 |
| DeepSeekMath-Base-SFT + Step-DPO | 7B | ✗ | ✓ | 54.1 (+1.2) | 86.7 (+0.0) |
| Qwen2-7B-SFT^† | 7B | ✗ | ✓ | 54.8 | 88.2 |
| Qwen2-7B-SFT + Step-DPO | 7B | ✗ | ✓ | 55.8 (+1.0) | 88.5 (+0.3) |
| Qwen1.5-32B-SFT^† | 32B | ✗ | ✓ | 54.9 | 90.0 |
| Qwen1.5-32B-SFT + Step-DPO | 32B | ✗ | ✓ | 56.9 (+2.0) | 90.9 (+0.9) |
| Qwen2-57B-A14B-SFT^† | 57B | ✗ | ✓ | 54.6 | 89.8 |
| Qwen2-57B-A14B-SFT + Step-DPO | 57B | ✗ | ✓ | 56.5 (+1.9) | 90.0 (+0.2) |
| Llama-3-70B-SFT^† | 70B | ✗ | ✓ | 56.9 | 92.2 |
| Llama-3-70B-SFT + Step-DPO | 70B | ✗ | ✓ | 59.5 (+2.6) | 93.3 (+1.1) |
| Qwen2-72B-SFT^† | 72B | ✗ | ✓ | 61.7 | 92.9 |
| Qwen2-72B-SFT + Step-DPO | 72B | ✗ | ✓ | 64.7 (+3.0) | 93.9 (+1.0) |
| Qwen2-72B-Instruct (Bai et al., ) | 72B | ✓ | ✓ | 59.7 | 91.1 |
| Qwen2-72B-Instruct ^‡ | 72B | ✓ | ✓ | 69.4 | 92.4 |
| Qwen2-72B-Instruct + Step-DPO ^‡ | 72B | ✓ | ✓ | 70.8 (+1.4) | 94.0 (+1.6) |

**Table 2: Math reasoning performance comparison on compitition-level math problems, i.e., AIME 2024 and Odyssey-MATH. Note that the training data for Step-DPO is the same as before.**
| Model | size | open | AIME | Odyssey-MATH (%) |
| --- | --- | --- | --- | --- |
| Gemini-1.5-Pro (Reid et al., ) | - | ✗ | 2 / 30 | 45.0 |
| Claude-3-Opus | - | ✗ | 2 / 30 | 40.6 |
| GPT-4-1106 (Achiam et al., ) | - | ✗ | 1 / 30 | 49.1 |
| GPT-4-Turbo-0409 (Achiam et al., ) | - | ✗ | 3 / 30 | 46.8 |
| GPT-4o-0513 | - | ✗ | 2 / 30 | 53.2 |
| DeepSeek-Coder-V2-Lite-Instruct (Zhu et al., ) | 16B | ✓ | 0 / 30 | 44.4 |
| Llama-3-70B-Instruct (Touvron et al., ) | 70B | ✓ | 1 / 30 | 27.9 |
| DeepSeek-Coder-V2-Instruct (Zhu et al., ) | 236B | ✓ | 4 / 30 | 53.7 |
| Qwen2-72B-SFT^† | 72B | ✓ | 1 / 30 | 44.2 |
| Qwen2-72B-SFT + Step-DPO | 72B | ✓ | 3 / 30 | 47.0 (+2.8) |
| Qwen2-72B-Instruct (Bai et al., ) | 72B | ✓ | 5 / 30 | 47.0 |
| Qwen2-72B-Instruct + Step-DPO | 72B | ✓ | 4 / 30 | 50.1 (+3.1) |

#### Datasets.

In supervised fine-tuning (SFT), we use augmented mathematical problems from MetaMath (Yu et al., ) and MMIQC (Liu & Yao, ) to infer step-by-step responses with DeepSeekMath, as the SFT data used in DeepSeekMath (Shao et al., ) is not publicly available. After filtering out responses with erroneous final answers, we obtain 374K SFT data. Of these, 299K are used for SFT, and the remainder is used for further Step-DPO training.

In the Step-DPO phase, alongside the remaining SFT data, we also incorporate a subset of AQuA (Ling et al., ). These datasets are processed as described in Sec. 3.2, resulting in 10K pair-wise preference data for Step-DPO.

For evaluation, we use the widely adopted MATH (Hendrycks et al., ) and GSM8K (Cobbe et al., ) datasets. Accuracy in these datasets serves as the evaluation metric. The MATH test set contains 5000 mathematical problems spanning 5 difficulty levels and 7 subjects, including algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus. The GSM8K test set includes 1319 mathematical problems, each with a step-by-step solution and a ground-truth answer. The problems in GSM8K are generally easier than those in MATH. Besides, we also use completition-level problems in American Invitational Mathematics Examination (AIME) (MAA, ) and Odyssey-MATH (Netmind.AI, ) to evaluate the math reasoning capabilities in solving hard problems.

#### Implementation Details.

First, we use the 299K SFT data for supervised fine-tuning on the base models, obtaining the SFT models. We train 7B models for 3 epochs and models larger than 30B for 2 epochs. The global batch size is set to 256, and the learning rate is set to 5e-6. We use the AdamW optimizer with a linear decay learning rate scheduler, setting the warmup ratio to 0.03. DeepSpeed ZeRO3 with CPU offload is used to reduce GPU memory usage during training.

Next, we perform Step-DPO based on the SFT models. For Step-DPO, we train 7B models for 8 epochs and models larger than 30B for 4 epochs. The global batch size is set to 128, and the learning rate is set to 5e-7. The hyperparameter $\beta$ is set to 0.5 for the 72B model and 0.4 for others. We use the AdamW optimizer and a cosine learning rate scheduler, with the warmup ratio set to 0.1.

### 4.2 Results

**Table 3: Performance comparison between DPO and Step-DPO. We use only 5K data for training in this ablation study.**
| Model | Qwen2-7B-SFT | Qwen2-7B-SFT + DPO (5K) | Qwen2-7B-SFT + Step-DPO (5K) |
| --- | --- | --- | --- |
| MATH (%) | 54.8 | 55.0 | 55.8 |
| Model | Qwen2-72B-SFT | Qwen2-72B-SFT + DPO (5K) | Qwen2-72B-SFT + Step-DPO (5K) |
| MATH (%) | 61.7 | 62.5 | 64.1 |

**Table 4: Performance comparison between out-of-distribution and in-distribution data. OOD: out-of-distribution data. ID: in-distribution data.**
| Model | Qwen2-7B-SFT | Qwen2-7B-SFT + Step-DPO (OOD) | Qwen2-7B-SFT + Step-DPO (ID) |
| --- | --- | --- | --- |
| MATH (%) | 54.8 | 55.1 | 55.8 |

#### Applying on open-source instruct models.

Table 1 presents a comprehensive comparison of various models, encompassing both open-source and closed-source models. Notably, Step-DPO can be directly integrated into open-source instruction models, such as DeepSeekMath-RL and Qwen2-72B-Instruct, leading to significant performance enhancements even after their prior RLHF training phase. This indicates that Step-DPO complements RLHF effectively. Specifically, when applied to Qwen2-72B-Instruct, Step-DPO achieves scores of 70.8% and 94.0% on the MATH and GSM8K test sets, respectively, surpassing a series of closed-source models, including GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro.

#### Applying on SFT models.

To further substantiate the efficacy of Step-DPO, we applied it to SFT models. Initially, we performed supervised fine-tuning on the 299K SFT dataset mentioned in Sec. 4.1, resulting in models such as DeepSeekMath-Base-SFT, Qwen2-7B-SFT, Qwen1.5-32B-SFT, Llama3-70B-SFT, and Qwen2-72B-SFT. Step-DPO proved highly effective, yielding significant improvements across various model sizes. Particularly, for models exceeding 70B parameters (i.e., Llama-3-70B-SFT and Qwen-2-72B-SFT), Step-DPO achieved approximately a 3% performance boost on the MATH test set.

Interestingly, larger models exhibited greater performance gains from Step-DPO. We hypothesize that larger models have untapped potential that Step-DPO can exploit. If the performance ceiling is not reached through supervised fine-tuning (SFT), Step-DPO can help models approach their optimal performance.

#### Results on math competition problems.

To further illustrate the superiority of Step-DPO in mathematical reasoning, we evaluated the models on competition-level math problems, specifically AIME 2024 and Odyssey-MATH, as shown in Fig. 2. Despite the increased difficulty of these problems compared to MATH and GSM8K, Step-DPO significantly enhanced performance. On Odyssey-MATH, Step-DPO applied to Qwen2-72B-Instruct achieved 50.1% accuracy, narrowing the performance gap with GPT-4o.

Notably, the models used the same Step-DPO training data for these competition-level problems as for problems of normal difficulty, highlighting Step-DPO’s robust generalization capability.

### 4.3 Ablation Study

To validate the effectiveness of Step-DPO and its data construction process, we conducted an extensive ablation study.

#### DPO vs. Step-DPO.

As discussed in Sec. 3.1, models utilizing vanilla DPO struggle to accurately identify errors in incorrect answers, providing only marginal benefits to mathematical reasoning performance. To verify this, we compared vanilla DPO and Step-DPO in terms of both accuracy in judging preferred versus undesirable outputs (left side of Fig. 2) and the reward margin between them (right side of Fig. 2). We also reported the final mathematical reasoning performance on the MATH test set in Table 3. The results indicated that the benefits of DPO are limited and significantly less than those of Step-DPO. In this experiment, we used only 5K Step-DPO training data and 3K for evaluation in Fig. 2.

#### Out-of-Distribution vs. In-Distribution Data.

The importance of in-distribution data was emphasized in Sec. 3.2. To illustrate its effectiveness, we compared out-of-distribution and in-distribution data in Table 4. Out-of-distribution data was generated by using GPT-4 to correct erroneous reasoning steps in incorrect answers, whereas in-distribution data was generated through the pipeline described in Sec. 3.2. The results in Table 4 underscore the critical role of in-distribution data in enhancing performance.

### 4.4 Demonstrations

Figure: Figure 6: An example of comparison between Qwen2-72B-Instruct and Qwen2-72B-Instruct-Step-DPO.
Refer to caption: x7.png

As shown in Fig. 6, we demonstrate an example of comparison between Qwen2-72B-Instruct and Qwen2-72B-Instruct-Step-DPO. It turns out that Step-DPO does well in correcting minor mistakes in previous models. More comparisons are provided in the appendix.

## 5 Conclusion

In this work, we proposed a simple, effective, and data-efficient method called Step-DPO. Unlike DPO, which compares preferences between holistic answers, Step-DPO uses a single reasoning step as the fundamental unit for preference comparison. This transition enables fine-grained process supervision for LLMs, facilitating the quick localization of errors within incorrect answers. Additionally, we introduced a data construction pipeline for Step-DPO, creating a dataset with 10K preference data pairs. Our results demonstrate the significant improvements achieved by Step-DPO and the 10K dataset, particularly for large models. We hope that Step-DPO will provide new insights into model alignment for long-chain reasoning problems.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to my questions in the weakness section.
soundness: 2
strengths: - The idea of step-wise preference optimization is reasonable - it can guide llm to focus on step-wise correctness. 
- The experiments section contains comparison of many models.
summary: This paper proposes to dpo train the model per each reasoning step. To do so, the authors prepare questions and sampling LLMs to generate reasoning steps, and annotate them to form dpo data triples. Using such data to DPO-train the LLM, it can achieves on-par performance with stoa models on reasoning benchmarks.
weaknesses: - Key information is not clear: in the so-called step localization stage, about how to label the step-wise reasoning, the authors described "This process can be done manually or using GPT-4". However, it is not clear how this is done in detail. Since this is very important in the entire processing, it would be necessary to know more. Specifically, 

   1. how many data is labelled by human, and how many is from GPT-4
   2. For human annotation, how many annotators are hired for each question; what is the question to annotate, e.g., is it a binary classification? Also, since many math questions are not easy, what efforts do the authors take to ensure annotation quality.
   3.  For GPT-4 annotation, what is the prompt? Since model could make mistakes , how to reduce this error risk?
   4.  One alternative (and straightforward) way is to using a small amount of labelled data to fine-tune a llm to be a PRM, and use it to give step-wise supervision siginals.

- In table3, compared to normal DPO, the step-wise DPO's gain is very limited. It's not very convincing to claim the advantage of step-wise  DPO. Also, why only using 5K data for this ablation instead of the entire dataset?
- Similarly in table4, the marginal improvement of in distribution data is hard to conclude it is better.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Are there plans to release the curated dataset for reproducibility? I didn't find any mention of this in the draft, but I may have missed it.
2. Some recent variations on DPO, such as DPO-positive from (Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive, Pal et al., 2024), or ORPO (ORPO: Monolithic Preference Optimization without Reference Model, Hong et al., 2024), can mitigate failure modes of DPO, especially the failure mode of decreasing the likelihood of the preferred completion unintentionally. I would be interested to see how {Step-DPO on your treatment of your data} fares against {these methods on a standard DPO treatment of your selected datasets without the stepwise treatment}.
soundness: 2
strengths: 1. A clearly described procedure for curating DPO training data pairs for mathematical reasoning, with a useful step-partitioning procedure.
2. Step-DPO pushes the performance on certain test sets for several SFT models.
3. The paper's diagrams, charts, and communication clearly describe the contribution and results.
4. Dataset creation instructions are useful for those wishing to replicate or extend these results, or generally create good mathematical reasoning models.
5. Analyses about DPO on in vs. out-of-distribution data are helpful insights.
summary: The authors present Step-DPO, a variation on DPO specifically designed to encourage proper reasoning in intermediate reasoning steps, with applications for many-step math reasoning. The method simply uses a differing intermediate reasoning step as the preferred/rejected completion, with the question and all preceding steps as the context. The authors finetune models with this procedure and demonstrate that it leads to improved mathematical reasoning performance.
weaknesses: 1. The significance of the new method's dominance over existing models/results is not clear in several cases. The lack of meaningful error bars in figures and tables means that some improvements appear possibly marginal in Tables 1, 3, and 4. 
2. In some cases, such as for Qwen-72B-Instruct, the bulk of improvements seems to come from the updated prompting strategy used by the authors, not the Step-DPO method. For example, the abstract concludes with the statement that the Step-DPO-trained Qwen-72B-Instruct surpasses GPT-4-1106, Claude-3-Opus, and Gemini-1.5-Pro on GSM8K and MATH. In fact, it seems it does not surpass Claude-3-Opus on GSM8K. Additionally, Qwen-72B-Instruct already surpassed all of these three models on MATH (and GPT-4-1106 and Gemini-1.5-Pro on GSM8K) after the improved prompting strategy, meaning that the conclusion that Step-DPO allowed for these results seems possibly unfounded.
3. In terms of contribution, it is unclear to me that the framing of Step-DPO as a DPO variant is founded. It seems to me that the formulation in Equation 2 is precisely that of Equation 1, with the exception of changing what is the context and completion. I am not sure why we need a full equation statement of Step-DPO. The paper does mention that "Rejecting an entire undesirable answer in DPO may also discard preceding correct reasoning steps, introducing significant noise and negatively impacting training." (201-203). I agree that there is a major issue with the formulation of DPO for tasks with only some portions that are wrong; it could diminish the likelihood of initial correct reasoning steps. However, as long as the portion of the response before the deviation point is the same, it is effectively part of the context and therefore no gradients will be derived from it. This means that it seems to me that the approach presented in the paper is effectively exactly the same as ordinary DPO, but with the initial portion of both completions identical and the portion of the completion after the deviation clipped after one reasoning step. Mostly, the contribution of this work seems to be pair-preference dataset creation for a targeted purpose.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses 2, 3 and 4 for further clarification.
soundness: 3
strengths: 1. The mathematical capabilities of Large Language Models (LLMs) have garnered significant attention within the research community, highlighting the importance and relevance of this area of study.  
2. The proposal is technically sound, and step-level supervision will undoubtedly bring further improvements to the original trajectory-level DPO.  
3. The overall paper is well-presented and easy to follow.
summary: This work studies the improvement of LLM's mathematical reasoning abilities. The authors propose a step-wise DPO method to help LLMs correct errors during the process. The authors construct a step-wise preference dataset through a data construction pipeline, pointing out that using outputs from GPT-4 or humans in the data generation process may lead to out-of-distribution data and ineffectiveness, suggesting the use of a policy model to assist in data generation. The authors conduct experiments on MATH and GSM8K, demonstrating that step-DPO outperformed the original DPO method.
weaknesses: 1. In terms of results, this work ultimately achieved improvements of 1.4% and 1.6% relative to DPO on MATH and GSM8K respectively using Qwen2-72B. My research area is primarily RL, and I'm new to LLMs. Therefore, I won't make judgments on the significance of the performance improvements in the paper. In this regard, I will refer to other reviewers' opinions later. Additionally, I have some questions about the technical design. 
2. The construction of the dataset is the core foundation of the proposed method, but the entire annotation and generation process is based on GPT-4 and the initial model $\pi_{ref}$. How is the quality of the data and annotations controlled? I understand that the authors mention relying on models rather than human effort can make this process user-friendly, but it also makes the process a black box and uncontrollable. In this process, it seems we can only rely on comparing performance results after the fine-tuning process to determine the effectiveness of the method.  

The current data construction relies on two basic assumptions: (1) There exists a powerful model like GPT-4 that can accurately point out erroneous steps in the reasoning chain. (2) It relies on $\pi_{ref}$ to generate effective step-wise rectification.  
These two assumptions are problematic and are not sufficiently discussed in the paper. The experimental results demonstrate that the proposed data pipeline can indeed help the learning of step-wise DPO. However, what are the underlying principles, and what new insights can it provide to researchers in the field? Discussion of these questions can help further enhance the impact of this work. 

3. If relying on the existence of a stronger model, or even one that is close to a ground-truth model, then this process is not simply about improving the model's reasoning ability but is closer to knowledge distillation.  
4. What kind of improvement can the step-wise rectification based on $\pi_{ref}$ provide? In lines 276-283, the authors mention sampling the referred model multiple times and retaining results where the final answer matches the ground truth. How large is the scope of this data, and does it represent capabilities that the model already possesses but have not been exploited? Moreover, if only mining this part of the capability, the improvement the model can achieve seems to be limited.

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: - I don’t understand why it is “general” after using a math dataset to fine-tune the general models. If the model is specific, the author should report other datasets, for example, in Qwen 2.5-math. If the author claims the models are general, they should be encouraged to discuss the impact of math-specific data fine-tuning [3].

- This paper uses GPT-4 to find the wrong steps. How accurate is GPT-4 at finding the wrong steps?

- Why does the ablation study use only 5k data, while the main experiments use 10k data?



---

[1]. Let’s Verify Step by Step.

[2]. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models.

[3]. https://github.com/QwenLM/Qwen2.5-Math

[4]. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
soundness: 2
strengths: - The method is simple and intuitive.

- The experiments show that Step-DPO can improve the baseline mathematical reasoning ability.
summary: This paper proposes Step-DPO, a new approach aimed at improving the mathematical reasoning abilities of LLMs. The core idea is derived from process supervision [1] and incorporated into DPO.
weaknesses: - The main experiments in Table 1 and Table 2 should include the baseline of naive DPO experiments.

- The author uses different source data for SFT and Step-DPO separately. If we use the whole dataset adopted in SFT and Step-DPO in the SFT phase [2], how would the performance of SFT and DPO compare? This could establish a strong baseline.

- If we construct a paired dataset, the author can use the naive DPO loss function to optimize the LLMs. Therefore, this paper could be regarded as a data construction pipeline for step-level paired data.

- Considering the process supervision proposed in [1] and related works [4], the novelty and data construction methods are limited.

- Accuracy gains are small, many < 1%.

Meta-review: This paper presents Step-DPO, which is a variant of DPO that use intermediate reasoning steps as the preference feedback to the model.

The paper is clear, results show that Step-DPO improves performance, and its ablation analyses on DPO vs Step-DPO and in vs out of distribution data show interesting insights.

However, as pointed out by the reviewers, the empirical improvement is relatively small, the baselines are relatively weak (though stronger ones can be constructed), and there are some inconsistency in experiment settings.

The authors did not provide any rebuttal and the concerns from reviewers are not addressed.

Given these reasons, I recommend rejection.

Decision: Reject