Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation
Authors: Songjun Tu
Jiahao Lin
Xiangyu Tian
Qichao Zhang
Linjing Li
Yuqian Fu
Nan Xu
Wei He
Xiangyuan Lan
Dongmei Jiang
Dongbin Zhao
Venue: COLM 2025 (2025)
Keywords: LLM Reasoning
Iterative Optimization
TL;DR: DPO enables iterative self-improvement for LLMs, achieving RL-level reasoning performance with lower computational cost through preference-based learning and verifiable rewards.

Abstract: Recent advancements in post-training methodologies for large language models (LLMs) have highlighted reinforcement learning (RL) as a critical component for enhancing reasoning. However, the substantial computational costs associated with RL-based approaches have led to growing interest in alternative paradigms, such as Direct Preference Optimization (DPO). In this study, we investigate the effectiveness of DPO in facilitating self-improvement for LLMs through iterative preference-based learning. We demonstrate that a single round of DPO with coarse filtering significantly enhances mathematical reasoning performance, particularly for strong base model. Furthermore, we design an iterative enhancement framework for both the generator and the reward model (RM), enabling their mutual improvement through online interaction across multiple rounds of DPO. Finally, with simple verifiable rewards, our model DPO-VP achieves RL-level performance with significantly lower computational overhead. These findings highlight DPO as a scalable and cost-effective alternative to RL, offering a practical solution for enhancing LLM reasoning in resource-constrained situations.

Published as a conference paper at COLM 2025
Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation
Songjun Tu Tu ,Jihhao Lin Xiangyu Tian A Qichao Zhang**, Linjing Li
Yuqian Fu Nan Xu Wei He?, Xiangyuan Lan Dongmei Jiang Dongbin *
Pengcheng Laboratory School of Artificial Intelligence, University of Chinese Academy of Sciences
Institute Automation, Chinese Academy of Sciences
Wenge Technology Fudan University
[tusongjun2023, zhanggichao201 Jeia.ac.ccm ac. lanxyēpcl.ac.cm
Abstract
Recent advancements in post-training methodologies for large language models (LLMs) have highlighted reinforcement learning (RL) as critical component for enhancing reasoning. However, the substantial compu- tational costs associated with RL-based approaches have led to growing interest in alternative paradigms, such Direct Preference Optimization (DPO). In this study, we investigate the effectiveness of DPO in facilitat- ing self-improvement for LLMs through iterative preference-bat ing. We demonstrate that single round of DPO with coarse filtering significantly enhances mathematical reasoning performance, particularly design iterative enhance-
learn-
for strong base model. Furthermore, ment framework for both the generator and the reward model (RM), en- abling their mutual improvement through online interaction across multi- ple rounds of DPO, Finally, with simple verifiable rewards, our model DPO-VP achieves RL-level performance with significantly lower com- putational overhead. These findings highlight DPO as scalable and cost-effective alternative to RL, offering a practical solution for enhanc- ing LLM reasoning in resource-constrainedd situations. Code available at: https///gthbb.com//U0//////000.
1 Introduction
The field of large language models (LLMs) is evolving at unprecedented pace (Hurst et al., 2024; Xi et al., 2025). With the release of O1 (Jaech et al., 2024) by OpenAl in September 2024, the research focus shifted from pre-training to post-training, This transition was further accelerated in January 2025, when DeepSeek-Ri (Guo et al., 2025) brought post-training methodologies to the forefront of LLM research. Then, several studies have explored reinforcement learning (RL) (Gandhi et al., 2025; Zeng et al., 2025b; Tu et al., 2025b) at scale and supervised fine-tuning (SFT) (Muennighoff et al., 2025; Ye et al., 2025; Fu et al., 2025a) with high-quality long-chain data, unlocking emergent capabilities such as self-checking. reflection, and verification in strong base models (e.g., Qwen2.5 (Yang et al., 2024a)). These advancen advancements highlight the potential of post-training in enhancing LLM reasoning.
Generally, searching self-generated data for training within post-training referred to self-improvement (Yang et al., 2024b). At its core, this approach exploits search and learning to leverage the power of general methods as computational resources increase mccease-resonaal resonating with the principle articulated by Sutton (2019) in The Bitter Lesson. Naturally, search and expansion can be achieved through Monte Carlo Tree Search (MCTS) and RI. MCTS-based test-time scaling and reward model (RM) training were once considered key contributors behind O1 (Zhang et al., 2024a; Cui et al., 2025). But the high inference cost and training complexity gradually pushed MCTS out of the research spotlight. In contrast, RL with
Corresponding authors: Qichao Zhang and Xiangyuan Lan.

---

Published as a conference paper at COLM 2025
verifiable rewards has been empirically shown to be more efficient and promising (Guo et al., 2025; Xie et al., 2025). However, current RL training still requires substantial computational resources for reproduction. For instance, SimpleRL (Zeng et al., 2025b) trains for 1.5 days 4x8 H100 GPUs, while PURE (Cheng et al., 2025) also demands 8 A100 GPUs. Achieving comparable or even superior self-improvement performance with lower computational resources and reduced training time is an ideal goal pursued in this field.
Several methods have been introduced alternatives to RL alleviate computational overhead, such as DPO (Rafailov et al., 2023), KTO (Ethayarajh et al., 2024), and IPO (Azar et al., 2024). By modeling implicit rewards, these approaches remove the dependency on explicit reward and value functions of RL and are collectively known as offline maximum likelihood estimation (MLE) methods. Through multi-round iterative training, offline MLE methods can be approximated as off-policy online RL approaches (Pang et al., 2024). Recently, Swamy et al. (2025) demonstrated the theoretical equivalence between online RL and offline MLE under ideal optimization conditions. While the role of RL in post-training has been extensively studied, the impact of the more streamlined and efficient DPO in post-training remains uncertain.
In this paper, we focus on empirically investigating the improvements brought by DPO in post-training, aiming to provide the LLM community with deeper performance analysis of offline MLE self-improvement paradigms. Our key findings are as follows:
Single-Round DPO with Coarse Filtering Can Enhance Strong Base Models: With sim- ple RM or even outcome labels to filter self-generated datasets, achieve a significant improvement in mathematical reasoning of Qwen2.5 with single-round DPO training.
2. Iterative Improvement of Generator and RM can be Achieved via Multi-Round DPO: By relabeling the RM with online data, the mutual enhancement of the generator and RM can be realized through multiple rounds of DPO. In contrast, traditional RL frameworks typically rely on a fixed RM, making it difficult update the RM in an online manner.
3. Multi-Round DPO Achieves RL-Level Performance with Lower Computational Cost: Our final model DPO-VP, based Owwe2.5-Math-27 achieves comparable average performance to SimpleRL-Zero, PURE-VR and DPO-R1-Zero across 5 challenging bench- marks (48.2 vs 48.8/ 47.7 47.0), while requiring only a single 80GB GPU for training.
2 Methods
The overall iterative DPO training framework is illustrated in Figure 1. We first sample multiple responses from the base model, annotate them using rule-based or reward-based labeling. The policy is then optimized through DPO iterations, alternating between RM updates and policy refinement, enabling continuous self-improvement in reasoning ability.
2.1 Iterative DPO with External Annotation
For each question Qz. the base model To generates set of candidate responses [ri
Responses are evaluated using rule-based outcome labels or reward models (e.g. Process Reward Model, PRM or Outcome Reward Model, ORM) that assign preference scores. Specifically, for ORM, we select the response with the highest score (r)M(r) as r+, and the response with the lowest score as r- For PRM, where each step in response is assigned process reward score, we define the evaluation function f as follows:
(sPRM(()) min (..,. (PMM)))((M((((
(1)
where n represents the total number of steps in response r. Then, the best response + and the worst response r- are selected to construct the refined dataset, which subsequently used to update the policy ni using DPO loss:
L = (((urt)r)) logo (B. (lg π((r Qx)
(118
π(r Qx:) ( [k) , - Qx)
log
(2)
2

---

Published as a conference paper at COLM 2025
Question Response
Sample
Annotation
Respatisiy
DPO
Base Mg
Response
Reject, Resporste
Outcome Label
Groestric sequence
Label Annotation
Stop 2: Lot's analyze the given inforeation
Response
R-1: 4 4:3 X3
11
R-2: R-2: V
Step 37: Therefore, S4 is 4,8
Response
R-n:
X
Step
the 120
Reward Model Ro
Step 11: We have 2, 5,
beta
2M Annotation
Response
R-1:[8.9, 0.33) nin-B22
Rg
R1
R2
Step The terms are dy y 92 my dy
R-2:[1.0, 0, 8l R-n:[0.9,.
Step Find 1,40 S 18--9)
External Annotation & Train RM Annotation Train with
Figure 1: Iterative DPO with Outcome Label and RM-Based Annotation
where of is the sigmoid function and f is a temperature scaling parameter. The entire process is conducted over N iterations, where the policy is progressively refined through repeated optimization. Finally, we obtain the final optimized policy π.N-
2.2 Training Reward Model with Online Generated Data
Furthermore, we can train the reward model using data generated by the online generator. Taking PRM as an example, we leverage external annotators (e.g., DeepSeek- V3 (Liu al., 2024a)) to label the responses sampled from the current policy 7₁. This process produces new round of PRM training data, which is then used to fine-tune the previous PRM R,-1, yielding updated PRM R,. The training loss for updating the PRM is defined as follows:
CPRM(RQ R = (( .r) [- logo (R.(Qx) (R((x()))))
By iteratively updating the reward model and selecting the data for next epoch, we progres- sively improve its ability to assess process-level reasoning. In Section 3.4, we explore how this online training paradigm strengthens the PRM's capability capture and differentiate reasoning quality.
Experiments
3.1 Setup
In this section, we systematically investigate the impact of DPO different base models. To ensure comprehensive analysis, we examine models with varying parameter sizes and architectures, including Qwen2.5-3B (Yang al., 2024a), Qwen2.5-7B (Yang al., 2024a), LLLMA3.1 (Grattafiori et al., 2024), and Qwen2.5-Math-77 (Yang et al., 2024b). The structure of the following sections is follows:
In Section 3.2, we explore the simplest approach improving base model performance by examining the effects of Chain-of- (CoT) prompting and supervised fine-tuning (SFT) on different models, establishing baseline for subsequent experiments.
In Section 3.3, we demonstrate that single round of DPO with coarse filtering is sufficient to significantly enhance Qwen2.5's mathematical reasoning performance through self- improvement.
Section 3.4, we investigate how multi-round DPO and iterative reward model training enable continuous iterative improvement between the generator and the reward model.
3

---

Published as a conference paper at COLM 2025
In Section 3.5, compare the final multi-round DPO results with -tate-of-the-art re- inforcement learning methods and instruct models, analyzing potential differences in reasoning paradigms.
3.2 CoT Prompts Improve Qwen2.5
First, we need to ensure that the base model can align its output with specified step-by-step format. To achieve this, we select problems from the training sets of GSM8K (Cobbe et al., 2021) and MATH (Hendrycks al., 2021) and reconstruct the reasoning paths structured step-by-step format using DeepSeek-V3 (Liu et al., 2024a). The specific prompt can be found in Figure 6 in the Appendix A. This process facilitates the creation of a well-structured SFT dataset. Subsequently, we conduct two epochs of SFT on the base model.
Meanwhile, inspired by Kojima et al. (2022), we ex- plore the of CoT prompt "Please reason step by step with steps separated by "tn""", and put your final answer within \boxed urally guide the base model toward better adher- ence the desired step-by-step format. The aver- age greedy sampling scores across the 4 datasets (GSM&K,
Bamn
35.7
31,0
means to nat-
23.0
(Zhang et al.,
25.
25.
2023) and Minerva-Math (Yang et al., 2024b)) are presented in Figure 2. "Base" denotes the approach where the model generates answers directly using the format " Question: {imput} (n Answer: [output)." And "+SFI" indicates that the dataset was con- structed using two different prompting strategies, followed by fine-tuning on the model.
Qwen2.5-78
Qwen2.5-30
Uama) 3-40
Figure 2: SFT & CoT Comparison
The results indicate that for Qwen2.5, simply providing the CoT prompt is sufficient. Apply- ing SFT, however, results in reduction. This decline is probably attributed to the inferior quality of the GSM8K and MATH datasets. In contrast, for LLaMA3.1, SFT is needed to ensure the model conforms the designated format. Therefore, in the subsequent experi- ments, all training and evaluation processes are conducted using the CoT prompt, with LLaMA3.1 employing the CoT-SFT version.
3.3 Single-Round DPO with Coarse Filtering Improves Qwen2.5
To explore the impact of DPO on enhancing base models, we first train an ORM and PRM for each base model. We generate responses per problem from the GSM8K and MATH training sets with Qwen2.5-7B, then evaluate each reasoning path with DeepSeek-V3 to obtain process-level reward labels, the evaluated prompts can be seen in Figure 7 in the Appendix A. The labels are filtered based on reference answers, retaining those that either match the reference with fully correct process annotations or differ from the reference with errors in process annotations. The filtered data are then used to fine-tune the base model, yielding the PRM. Additionally, we construct preference data by pairing correct and incorrect responses for the same problem, then producing the ORM. Notably, our reward model relies on only tens thousands of in-house data samples, making the training cost relatively low. Thus, it can serve data.
coarse-grained filter for selecting releeimprovement
After training the reward model, we resample 8 responses from the MATH training set using the base model. These responses are then filtered based on different rules to construct round of
DPO dataset with positive and negative sample pairs. Finally, we perform DPO training on this dataset and evaluated various model variants on different datasets, including both in-distribution and out-of-distribution datasets. The results are recorded in Table 1. Specifically:
Outcome Label: Accuracy is determined by the outcome label, then a correct and an incorrect response are randomly selected positive and negative samples, respectively.
4

---

Published as a conference paper at COLM 2025
+RRM & +ORM:
The response with the highest RM score is selected the positive sample, and the one with the lowest score as the negative sample. For PRM, the RM evaluation score is determined by the step with the lowest score.
+PRM with offset: Additionally, high-PRM-score responses with incorrect answers, referred to as offset samples, are selected as negative samples to prevent the generator from exploiting PRM. We rank the responses by PRM scores in descending order, and among the top k samples (where is the number of correct responses), incorrect responses are paired with the highest-scoring correct ones.
Additionally, we compare results with closed-source models, including GPT-4o (Hurst et al., 2024), GPT-ol-min (Jaech et al., 2024), Claude-3.5-Sonnet Anthropic (2024), Qwen2.5- 7B-Instruct Yang et al. (2024a), and Owwn2.5-78-DsV3--FFT the latter being version of Qwen2.5-7B SFT on GSM8K and MATH responses generated by DeepSeek-V3. V3.
In-Distribution Dataset
Out-of-Distrbation Dataset
Models
Methods
greedy-avg
GSMSK
MATH-500
Gaokao-302-EN mives passi8 greedy mv8s passit8
Minerva-Math
govedy
mvdt passes
greedy
msel8 pass8S
greedy
GPT-5o
72.5
96.1
76.6
71.4
15.6
Cland-3.5-Sermet
722
72.2
46.9
GPT-ol-mins
76.7
968
90.0
70.9
46.9
6877
91.4
97.8
76.4
23.4
88.2
64.2
78.7
41.9
20.2
Qwwnn2557B--DVV
625
89.
92.0
96.7
72.6
78.0
57.4
60.0
67.0
76.1
27.9
29:4
51.1
Base
56.7
86.4
92.1
97.2
64.2
74.6
87.8
55.6
61.3
76.6
20.6
18.4
35.3
+ultooee Label
63.0
93.5
97.3
74.2
752
67.6
62.9
76.9
25.0
Qwen225577
+ORM
62.2
89.5
93.4
97.2
73.6
79.6
88.8
61.6
66.2
78.2
23.9
26.8
47.7
4PKM
628
93.3
96.9
74.6
75.4
88.6
61.0
77.4
25.0
20.2
37.5
+PRM with offset
61.0
90.3
93.9
97.5
75.8
78.6
58.9
63.9
65.7
78.7
25,9
23,9
37.9
Improve
+11.6
-1.0
+2.1
Base
49.0
24.7
86.9
95.6
61.2
68.2
61.4
42.1
56.6
71.2
15.0
15.8
31.6
Outcome Label
53.7
87.5
94.6
6388
82.8
52.5
55.6
71.6
15.8
16.9
24.7
Qwen22.5.B
+ORM +PRM
54.1
83.8
89.0
95.6
64.4
69.0
$3.6
51.8
56.6
73.3
13.2
17.3
32.7
53.9
63.2
88.4
95.4
64.4
70.4
83.0
53.0
58.2
71.2
15.1
16.2
32.7
iPRM with offsel
53.9
88.2
95.9
65.2
68.4
62.0
53.2
71.2
15.4
16.2
30.5
Best Improve
+5.1
+9.2
+2.1
+0.3
+3.2
+2.2
-2.2
+12.7
+1.6
+2.1
-2.2
-1.5
+1.1
25.4
58.4
27.6
48.2
19.5
46.2
13.6
14.7
Dulcome Label
25.7
48.6
61.9
81.8
2000
27.4
48.4
19.2
25.7
45.7
15.1
15.8
28.7
LLaMA311-8
ORM
25.4
60.3
30.4
19.0
12.9
PPR
26.1
482
58.9
79.1
20.1
21.2
17.8
22.1
26.0
48.2
11.7
10.0
25.7
PRM with offee
26.5
49.8
62.5
$4.5
21.6
28.6
51.8
21.6
24.9
47.5
12.9
10.3
31.3
Best Impoove
+1.1
+2.4
+1.1
+2.
+0.0
+1.2
-3.6
+2.6
-0.7
+2.0
+15
+1.1
0.1
Table 1: Performance Comparison Across Different Models and DPO Data Filtering Methods. Bold values indicate the best results, and highlighted rows represent the best improvements.
Based on the results in Table 1, we derive the following findings:
1. DPO can rapidly enhance performance, with improvement strongly correlated to the base model's capability. For Qwen2.5-7B, single round of DPO using self- generated data brings the best variant, surpassing Qwenn.5-7B-DsV3-SFF and approach- ing Qwen2.5-7B-Instruuc which has been fine-tuned on large dataset. However, for LLaMA3.1-SB. DPO has little effect on self-improvemmnt.
2. Even coarse-grained filtering significantly boosts performance. Leveraging a reward model for supervision and incorporating additional offset negative samples may improve training efficiency. For Qwen2.5-7B, PRM with offset achieves the highest performance without requiring extra sampling.
3.4 Iterative Improvement of Generator and RM can be achieved via Multi-Round DPO
An interesting question is: can the generator and RM achieve mutual improvement through multiple rounds of DPO? The answer is certainly YES!
Starting with 2wwen..57BB-base we greedily sample responses from the MATH training set and annotate using DeepSeek-V3 to initialize PRM-base. Then, we apply the "DPO+PRM" filtering method from the previous section to select data for training the current generator via DPO. After updating the generator, we continue with online greedy sampling on MATH,
5

---

Published as a conference paper at COLM 2025
annotation, and PRM updates. Just as illustrated in the bottom right of Figure 1, this iterative process allows the generator and RM to evolve together over three epochs.
66
911
20
Groedy Scoee
@:44 Score
22.6
Auarge
UUF-530
2713-2
Gymplictent OreeMalt
(a) Generator Score.
(b) PRM Evaluation.
(c) Test-time Scaling.
Figure 3: Performance Evolution of Generator and PRM Across Multiple DPO Epochs.
We conduct a detailed evaluation of the iteratively trained generator and PRM:
1. Generator Score:
As shown in Figure 3a, we observe continuous performance improve- ment across different datasets. The average evaluation score increases from 62.7 in the first epoch to 65.8 in the third epoch, surpassing the best average score reported in Table
PRM Evaluation: We assess the ability of PRM to identify incorrect steps using Process- Bench (Zheng et al., 2024). Given the imbalance between positive and negative samples in ProcessBench, we compute the weighted-F1 score (Hinojosa Lee et al., 2024) to reflect error detection performance. In addition self-comparison, we benchmark against several baselines:
(a) PRM-800K: A PRM fine-tuned on PRM800K (Lightman et al., 2023) dataset with Qwen2.5-7B.
(b) PURE-PRM-7B: A PRM derived from Cheng et al. (2025), trained with PRMS00K through a two-stage MLP and full fine-tuning.
(c) Qwen2.5--MtthPPRM-BB The SOTA PRM officially released by Qwen (Zheng et al., 2024), trained on large in-house dataset.
The results show that our PRM surpasses PRM-800K and, by the third epoch, outper- And demonstrate that PRM can continuously improve through iterative training with the generator at low cost each epoch requires only 7.5K DeepSeek-V queries. Detailed experimental settings can be found in Appendix B.
forms PURE-PRM-7B, while slightly below (wwen2.-MMtt-PPRM.
3. PRM in Test-time Scaling:
We further analyze the role of PRM in test-time scaling. As shown in Figure 3c, we compare majority vote (MV) (Wang et al., 2023) and PRM@64. We find that PRM@64 outperforms MV in every epoch, and the PRM trained with online sampled data from the current generator further improves over the previous epoch's PRM@64 performance. A detailed performance table is provided in Appendix C.
3.5
Multi-Round DPO with Verifiable Pairs Achieves RL-Level Performance Efficiently
3.5.1 Setup and Main Results
Finally, we conduct multi-round DPO on the advanced Owen2.5-Mathh- aiming to bench- mark against RL-based approaches under the same base model and training data. Inspired by DeepSeek-R1 (Zeng et al., 2025b), we observe that the verifiable reward (VR) is 0/1 discrete variable, indicating whether the output meets predefined correctness criteria. This aligns with the original DPO framework, where correct samples are classified as positive and incorrect ones as negative, forming verifiable pairs (VP) for training. By applying VP-based filtering and gradual temperature scaling (inspired by Xi et al. (2024)), conduct 6 epochs

---

Published as a conference paper at COLM 2025
of DPO. The final model, Qwen2.5577-DPOO-VP is compared against several state-of-the-art RL-based methods:
(a) Qween255-Math-7--Instrre (Yang et al., 2024b): An Instruct model by Qwen with strong
mathematical reasoning capabilities.
(b) IStar-Math-7B (Guan al., 2025): model trained with a large amount of self-evolution MCTS data and filtered with process preference model (PPM).
(c) Eurus-2-7B-PRIME (Cui et al., 2025): A model trained by process RL with implicit process rewards.
(d) Qwenn.5-7B-GRRP--Zero (Shao et al., 2024): A model trained by GRPO with verifiable rewards without any extra SFT data.
(e) Owen222-7B55imnllRR--Zeo Zoo (Zeng et al., 2025a;b): A model trained by PPO with verifiable rewards without any extra SFT data. Zero and Zoo refer to two versions released at different time points.
(f) Qwen2.5-7B-PURE: (Cheng al., 2025): model trained by RL with verifiable rewards and min-form credit assignment without any extra SFT data. Owen2.5-7B-DPO-R R1-Zero (Zhang et al., 2025a): A concurrent work that applied itera- tive DPO without any extra SFT data.
To align with existing RL methods, following Zeng et al. (2025b) and Cheng et al. (2025), we select 8K Level 3-5 questions from the MATH as self-improvement data, and additionally evaluate on Olympiad-level benchmarks, including AMC23, AIME24, and OlympiadBench He et al. (2024). Furthermore, we conduct additional comparisons on out-of-domain datasets to assess generalization performance. In addition, we compare with Qwen2.5-7B- DPO-RI-Zero (Zhang et al., 2025a) that applies iterative DPO on 200K questions (more than 8K questions) from the NuminaMath dataset (Li et al., 2024a).
Moreover, we employ annealed sampling to enhance response diversity. Specifically, the sampling temperature f is set to 0.7 for the first three epochs, increased to 1.0 during epochs 4-5, and further raised to 1.2 the final epoch. This annealing schedule is motivated by the observation that model performance nearly converges after three epochs, and increasing the temperature later stages allows for fine-tuning on more diverse samples, leading to additional gains.
Table 2 presents the pass@1 accuracy of different models, while Table 3 further analyzes training data consumption and GPU usage.
pass&1 acc
MATH500
Minerva-Math
Olympaidbench
AMC23
AIME24
Avg.
Owen2.5-Mat-77B
64.8
15.4
25,6
37.5
16.7
32.0
Owen2.5-Math-7B-nnnlhl
83.2
33.5
38.4
62.5
20.0
47.5
rStar-Math-7B
78.4
47.1
47.5
26.7
Buruu-2-7B-PRIME
39.7
35.6
37.5
Owenz.5-7R-GRRP-ZZee
76.2
32.7
38.1
55.0
43.7
Owen2,5-7B-Simg
78.0
33.1
36,6
60.0
26.7
46.9
Bren2.5-7B-Simple-Rl-..
80.4
39.7
38.8
57.5
26.7
48.7
Qwen2.5-7B-PURS. -
36.8
41.9
60.0
20.0
47.7
Owen2.5-78-DPO-RR-ZZ-
76.8
30.9
37.9
62.5
Qwen2.577-DDOO-VP
74.8
35.3
36.9
67.5
26.7
48.2
Table 2: Pass@1 Accuracy Results Across Different Benchmarks. Bold values indicate the best results, and underline value indicate the second one. All models are fine-tuned based the Owen2.5-Math-7B. Bold models represent those that were adjusted using the self- improvement method with exactly the same prompts. The results with * are from our reproduced, and the results with A are derived from the corresponding technical report.
Table and 3, we derive the following conclusions:
Multi-round DPO can achieve mathematical reasoning capabilities comparable to those of RL-based methods. The final model attains an average score of 48.2 across five
7

---

Published as a conference paper at COLM 2025
Qwve2.5-Math-76
(Star-Math-78
Furne-2-7B. PRIME
Qwen2.5.7B
Qwen25--BB
Owen25-7B-DPO.
Instruct
SimpleRL-Zeru
PURE-VR
Base Model
Qwen2 M-Math-
Q-ath2.-Matth-78
Qwen2 5-Math-7B
Qwen2 Qwen25-Mabbe 5-Math-7B
Qwen2 Qwen25-Mathve 5-Math-BB
Qwen2 Qwen25-Mathr 5-Math-78
(oper-source
(MATH,
inrbouse)
NuminaMath, etc.)
RM Data
618K (in-house)
~KK (in-house)
Doven25-Mark
Fum--227B--FF
Nore
None
(728)
Self-impoov
RL. OKM
MCTS PPM
RL. PRM
RL VK
RL VK
DFU + VX
Method
Self-improv Data
66K
33667M
150K
MATHL
SK MATH
MATHI
GPUs Training
H100 days (at most)
A100 for few days
32 01 days
A100 day
A800 for <1 day
Times
days
Table 3: Data and GPUs Comparison of Different Models.
mathematical reasoning benchmarks, which is similar to the performance of Qwen2.5- Math-7B-Instruct and other RL-based approaches with the some training data conditions. Compared to Swwen2.5-7B-DO-RRI R1-rrro, our model still achieves higher average score. We hypothesize that this may be attributed to the increased response diversity introduced by the annealed sampling strategy.
Multi-round DPO requires significantly fewer computational resource can even be executed on a single GPU. Our sampling and training are conducted on four A800 GPUs using data parallelism. The full pipeline can be run on a single 80GB GPU or lower. In our setup with 4 A800 GPUs, each sampling round took 2-2.5 hours with vLLM (Kwon et al., 2023), training took hour per round, and the full process completed about day. On single GPU, the full process would take around 3 days.
3.5.2 More Analysis
Token Length Dynamics and Accuracy Trends First, we analyze token length variations throughout training. In Figure 4, we illustrate the accuracy progression of the iterative process across different datasets in relation the average token length. The results indicate steady improvement in accuracy, while the token length during inference did not exhibit an initial increase followed by a decline. Instead, it remains relatively stable within consistent range. When comparing different models, we observe that their output lengths are generally similar. Detailed results are provided in Table 7 in Appendix E.1.
MAT15500
Mineroa Math
Olympaidbench
2020
100
Toith Longh
Aum Tatan Leeggih
32
100
Froo-1t
300
20
1.56
1.L
1655 1.66
1.77
1.75
0.57
0.53
3.65
0.76
1.07
10
2.10
2/43
2.54
2.43
2.46-
200
AIME24
72
1030
3400
2000
1200
Ave Takan Las gio
42
800
Takor Lor
12
Pass81
30
Ave Takun
29-1.68
2.75
2.45
185
2.25
2.57
351
4.13
3.13
2,00
2.23
2.27
2.20
402
2.03
200
A/C Mums "Lote"
Avc Nums of "Let's"
Ave Nums of "Let's"
Figure 4: Pass@1 Accuracy, Token Length, and Self-Evaluation Trends Across Datasets.
Self-Reflection and Reasoning Paradigms We do not observe a significant AAha Moment," which aligns with the findings of PURE (Cheng et al., 2025). Even when using CoT prompts, Qwen2.5-Math-75 frequently relies on Python-based solutions, despite not actually execut- ing the code. Additionally, identify recurring self-evaluation pattern, such as "Let's
8

---

Published as a conference paper at COLM 2025
re-evaluate... ...", which is already present in the base model. This suggests that DPO does not inherently introduce self-reflection capabilities but instead strengthens existing tendencies shown in Figure 4. When comparing several different
through reward optimization, models, Table 8 in Appendix E.1 shows that Dwen2.5-Math-77--nnttt fewer occurrences of "Let's", indicating shift in reasoning paradigms.
exhibits significantly
3.6 Ablation Study on the DPO Sampling Noisy
To evaluate the robustness of our DPO framework under noisy supervision, we conduct a controlled label noise ablation. Specifically, we simulate noise in the preference data used to train the Preference Reward Model (PRM) under the PRM with offset setting. Following Tu et al.
Label Ablation on Sampling
90
MATH500
0ass01 Accuracy
80
GSMBK (base)
(2025c), for each positive-negatiio response pair, we ap- ply random flipping probability [0, 1] that reverses the preference direction with probability p. simulating in- creasing levels of annotation noise. For example, p 0.5 corresponds to fully random pairwise supervision, while p 1.0 denotes complete inversion of the labels.
70
We test the impact of this noise injection on the final model performance, measured by pass@1 accuracy af- ter one round of DPO on GSM8K and MATH500 using Qwen2.5-7B. The results are shown in Figure 5. These results clearly demonstrate that model performance de- grades as the preference supervision becomes less accurate. While DPO appears tolerant to small levels of label noise, performance drops significantly under higher noise levels. This provides empirical evidence that maintaining reasonably accurate verifier signals is critical for effective preference optimization.
0.0
0.2
0.4
0.6
0.8
1.0
Ratio p
Figure Label Noise Ablation
Generalization We conduct evaluations on additional Out-of-Domain datasets:
(a) CMATH: A Chinese mathematics dataset.
(b) MMLU_STEM subset of MMLU focusing on science subjects like physics, biology.
(c) HumanEval: benchmark evaluating code generation through Python tasks.
(d) LiveCodeBench: A coding benchmark for real-time code editing and debugging.
(e) RACE: A reading benchmark from English exams testing reasoning.
The first two datasets are evaluated using the Qwen-Math codebase (Yang et al., 2024b), with evaluation parameters consistent with those used in Appendix The remaining three datasets are evaluated via EvalScope using its default configuration settings. The results are summarized Table 4.
pass@1 acc
CMATH
MMLU STEM
HumanEval
Live Code Bnech
RACE
Average
Qwen2.5-Mathh M
64.0
58.3
40.9
5.1
61.6
46.0
Burus-2-7B- PRIME
72.5
42.8
41.5
21.2
62.6
48.1
Qwee225-77-Simpii-Rl-eeer
70.3
51.8
52.7
1een2.577t-Simpl-RR--222
89.0
35.9
53.1
17.9
63.9
52.0
Oiven2.5-7B-PURE-VV
69.5
58.8
43.3
6.4
61.3
47.9
Owen2.5-7B-DPO-vP
50.6
61.2
51.4
Table 4: Performance comparison on Out-of-Domain datasets..
Nearly all methods lead to slight performance improvements on the out-of-doomin (OOD) benchmarks, indicating enhanced generalization capabilities. Our proposed DPO-VP no- tably increases the average OOD score from 46.0 to 51.4, 4, achieving gains comparable to those of RL-based methods. This suggests that DPO fine-tuning on mathematical tasks can also effectively enhance generalization other domains.
https://githin http://gttuuuu.ccmmmmmmaasccco// http./////mllllssop//eaassssoopp

---

Published as a conference paper at COLM 2025
4 Related Works
Self-improvement of LLMs Self-improvement primarily occurs during the training phase, where the model enhances its own performance by evaluating and refining its outputs without relying on external human supervision (Xi et al., 2024; Kumar et al., 2025). It is generally believed that LLMs are incapable of achieving self-improvement without any external information (Huang et al., 2024; Tyen et al., 2024). Consequently, an increasing number of methods leverage external supervision signals help self-improvement, for exam- ple, outcome labels for mathematical programming problems (McAleese et al., 2024; Gao et al., 2024), process reward models (Zhang et al., 2024b; Fu et al., 2025b), value function (Wang et al., 2024a) or LLM-as-a-judge (Li et al., 2024b; Zhang et al., 2025b) frameworks.
Improving Reasoning with DPO Recent works have sigmificantly extended DPO-based reasoning (Xu et al., 2024; Zhang et al., 2025a). Iterative DPO with rigorous theoretical analysis was proposed by Xiong et al. (2024), and further generalized to multi-turn and tool-integrated scenarios (Xiong et al., 2025a). Flow-DPO (Deng Mineiro, 2024) introduced online multi-agent learning for mathematical reasoning, while iterative wnngth-reglarizze DPO (Liu et al., 2024b) enables 7B models to approach GPT-4-level reasoning. Self-play and self-training pipelines have also been explored, demonstrating that DPO and preference- based methods can improve LLMs beyond human-labeled data (Chen et al., 2024b; Singh et al., 2023). Additionally, implicit reward bootstrapping with DPO has shown promise for further alignment without explicit reward models (Chen et al., 2024a). Aya Expanse (Dang et al., 2024) illustrates how such alignment advances benefit multilingual and cross- cultural reasoning. In contrast, our work focuses on data selection strategies for DPO and proposes an iterative training framework for generator-verifie! models, enabling mutual enhancement between generation and reward modeling.
Constructing Efficient Reasoning Models In early 2025, with the release of DeepSeek-RI (Guo et al., 2025), efficient post-training paradigms gained significant attention (Trung et al., 2024; Tu et al., 2025a). SimpleRI-Zero (Zeng et al., 2025b) achieved remarkable performance through RL, training on just 8K challenging mathematical problems with rule-based rewards. Xiong et al. (2025b) integrated reflective vocabulary into generative RM to enable selective RL-based correction. PURE (Cheng et al., 2025) revisited stepwise reward credit assignment in RL. Logic-RL (Xie et al., 2025) refined the penalty function for formatting errors, leading to improved logical reasoning capabilities. From the data utilization perspective, LIMO (Ye et al., 2025) and S1 (Muennighol) et al., 2025) demonstrated that fine-tuning with carefully curated long-chain reflective data can effectively amplify the latent reasoning capabilities. From the reasoning paradigm perspective, CFT (Wang et al., 2025) encouraged models to develop critical thinking during SFT. Our work focuses on achieving RL-based reasoning with significantly fewer computational resources. We demonstrate that, with strong base model, coarse-grained selection is enough for performance enhancement, achieving competitive results with reduced computational overhead. Our overall training framework is more resource-efficient than the widely adopted SFT+RL pipeline, while outperforming concurrent iterative DPO methods.
Conclusion
In this paper, we empirically study DPO for enhancing LLM reasoning, showing that single-round DPO with coarse filtering improves performance, while multi-round DPO enables iterative enhancement of both the generator and RM. Our results demonstrate that multi-round DPO achieves RL-level performance with significantly lower computational costs, requiring only a single 80GB GPU. However, in large-scale deployment and long-term training, DPO lacks exploration and sustained improvement potential. Future research could focus on more exploratory offline self-improvement methods or embrace efficient RL algorithms as pathway to further boost the reasoning bounds of LLMss.
10

---

Published as a conference paper at COLM 2025
Acknowledgements
This work is supported by the Strategic Priority Research Program of Chinese Academy of Sciences under Grant XDA0480303, Young Scientists Fund of The State Key Laboratory of Multimodal Artificial Intelligence Systems ES2P100112, Beijing Natural Science Foundation Xiaomi Innovation Joint Fund L253007, and National Natural Science Foundation of China 62402252.
References
Anthropic. The claude 3 model family: Opus, sonnet, haiku. Technical Peport, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakantham, and Min Lin. Bootstrapping language models with dpo implicit rewards. arv preprint ar Xrxi:24066.9766 2024a.
Wenxiang Chen, Wei He, Zhiheng Xi, Honglin Guo, Boyang Hong, Jiazheng Zhang, Rui Zheng, Nijun Li, Tao Gui, Yun Li, et al. Better process supervision with bi-directional rewarding signals. arviv preprint ar Xii::5033.0611 2025.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine- tuning converts weak language models to strong language models. arXiv preprint ar Stf::2401001335, 2024b.
Jie Cheng, Ruixi Qiao, Lijun Li, Chao Guo, Junle Wang, Gang Xiong, Yisheng Lv, and Fei- Yue Wang. Stop summation: Min-form credit assignment is all process reward model needs for reasoning. ar preprint ar Stfff::250.112775 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. ar Xiv preprint ar xrXie22110.14168 2021.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards. arx preprint arXiv:2502.01456 2025.
John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. Aya expanse: Combining research breakthroughs for a new multilingual frontier. arv preprint ar Xie:2412.004261, 2024.
Yihe Deng and Paul Mineiro. Flow-dpo: Improving 11m mathematical reasoning through online multi-agent learning. arx preprint ar arXio:2410.22304 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint ar 40X::2202..0136, 2024.
Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint ar Xrx:::2506.117 197677 2025a.
Yuqian Fu, Yuanheng Zhu, Jiajun Chai, Guojun Yin, Wei Lin, Qichao Zhang, and Dong- bin Zhao. Rlae: Reinforcement learning-assister ensemble for 1lms. arv preprint arXim:2506.00439 2025b.
11

---

Published as a conference paper at COLM 2025
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Good- man. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arxiv preprint (ttr:2553300100,, 2025.
Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Dayi- heng Liu, Chang Zhou, Wen Xiao, et al. Lim critics help catch bugs in mathematics: Towards better mathematical verifier with natural language feedback. arxiv preprint #rXio:2406.14024, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. ar preprint ar Xix::240..21783, 2024.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXio preprint arXi::2501.0455 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arviv preprint ar Xr1::2501..12948, 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging bench- mark for promoting agi with olympiad-level bilingual multimodal scientific problems. In ACL, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Maria Cristina Hinojosa Lee, Johan Braet, and Johan Springael. Performance metrics for multilabel emotion classification: comparing micro, macro, and weighted fl-scores. Applied Sciences, 14(21):9863, 2024.
Jian Hu, Xibin Wu, Zilin Zhu, Weixun Wang, Dehao Zhang, Yu Cao, et al. Openrlhf: An easy- to-use, scalable and high-performance rlhf framework. ar Xie preprint 2rX11:2405.11143, 2024.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations (ICLR), 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. ar Xiv preprint arXiv:2410.21276, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. ar Xiv preprint arXto:2412.16720, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models zero-shot reasoners. Advances in Neural Information Processing Systems (NeuriPS), 35:22199-22213 2022.
Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming- Yang, Phillip HS Torr, Salman Khan, and Fa- had Shahbaz Khan. Lim post-training A deep dive into reasoning large language models. ar arxiv preprint arXit:2502.21321, 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium Operating Systems Principles, 2023.
12

---

Published as a conference paper at COLM 2025
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024a.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. Self-alignment with instruction backtranslation. In International Conference on Learning Representations (ICLR), 2024b.
Fenrui Xiao Xin He Qi An Zhenyu Duan Yimin Du Junchen Liu Lifu Tang Xiaowei Lv Haosheng Zou Yongchao Deng Shousheng Jia Xiangzheng, Zhang Liang Wen, Yunke Cai. Light-rl: Curriculum sft, dpo and rl for long cot from scratch and beyond, 2025. URL https:/ /ggp:://giie thub. homm/Qhhoo36/////
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In International Conference on Learning Representations (ICLR), 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. ar Xiv preprint 2i4:1241..19437, 2024a.
Jie Liu, Zhanhui Zhou, Jiaheng Liu, Xingyuan Bu, Chao Yang, Han-Sen Zhong, and Wanli Ouyang. Iterative eingthreeullarize direct preference optimization: A case study on improving 7b language models to gpt-4 level. arviv preprint ar drXiv:2406.11817 2024b.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol ol-preview with a 1.5b model by scaling rl, 2025, Notion Blog.
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Lim critics help catch Ilm bugs. arviv preprint ar arXiv:2407.00215, 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arviv preprint arXiv:2501.1933 2025.
Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information 2 Processing Systems (NeurIPS), 37:1 16617-116677, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems (NeurIPS), 36:53728-5377 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXio:2402.03300 2024.
Avi Singh, John Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint ar arXiv:2312.06585, 2023.
Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1):38, 2019.
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. Xiv preprint 2xXi:2503301007, 2025.
Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. In ACI, pp. 7601-7614, 2024.
13

---

Published as a conference paper at COLM 2025
Songjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Dongbin Zhao. Learning when to think: Shaping adaptive reasoning in rl-style models via multi-stage rl. arv preprint ar K10:2205510832, 2025a.
Songjun Tu, Jingbo Sun, Qichao Zhang, Xiangyuan Lan, and Dongbin Zhao Online preference-based reinforcement learning with self-augmented feedback from large lan- guage model. In Proceedings of the 24th International Conference on Autonomous Agents and Multingent Systems (AAMAS), pp. 2069-2077, 2025b.
Songjun Tu, Jingbo Sun, Qichao Zhang, Yaocheng Zhang, Jia Liu, Ke Chen, and Dongbin Zhao. In-dataset trajectory return regularization for offline preference- reinforce- ment learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39,
Pp. 20929-20937, 2025c.
Gladys Tyen, Hassan Mansoor, Victor Cârbune, Yuanzhu Peter Chen, and Tony Mak. Lims cannot find reasoning errors, but can correct them given the error location. In ACL Findings, Pp. 13894-13908, 2024.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https://gr thub. com/ www.woogo.oocaaaaeeeeeeeoooooooe hug.hhggaccc//rrr.... 2020.
Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q": Improving multi-step reasoning for Ilms with deliberative planning, ar Xiv preprint ar arXiv:240..14283, 2024a.
Jun Wang, Meng Fang, Ziyu Wan, Muning Wen, Jiachen Zhu, Anjie Liu, Ziqin Gong, Yan Song, Lei Chen, Lionel M Ni, et al. Openr: An open source framework for advanced reasoning with large language models. ar Xiv preprint ar (to:2400.09671, 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023.
Yubo Wang, Xiang Yue, and Wenhu Chen. Critique fine-tuning: Learning critique is more effective than learning to imitate. arXiv preprint arXiv:2501.17703, 2025.
Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, et al. Enhancing Ilm reasoning via critique models with test-time and training-time supervision. arXiv preprint ar arXie:2411.16579, 2024.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):121101, 2025.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing 11m reasoning with rule-based reinforcement learning, ar Xio preprint ar Kiv:2502.14768 2025.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Proceedings of the 41st International Conference Machine Learning(ICML), pp. 54715-54754, 2024.
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Saleh, al. Building math agents with multi-turn iterative preference learning. In The International Conference on Learning Representations, 2025a.
Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang, Self- rewarding correction for mathematical reasoning. arXiv preprint arXiv:2502. 251.:::1....11.,1,.. 2025b.
14

---

Published as a conference paper at COLM 2025
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? a comprehensive study. arXio preprint ar Xrr::2404..07119 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. ar ar preprint arXio:2412.15115, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint ar Xix::2409.122 2024b.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. Xiv preprint ar Xix::2250..33387 2025.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in lims. Xiv preprint ar arXiv:2502.03373 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint Xix:25003.18882, 2025a.
Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https: / /hhps:://hkuttnllp... notion. wewiiiinnniiteeeeeeeeeeeeeeeeeee site/ssiplerl- -reason,,, 2025b. Notion Blog.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets": Lim self-training via process reward guided tree search. Advances in Neural Information Processing Systems (NeurIPS), 67774735-4777, 2024a.
Hanning Zhang, Jiarui Yao, Chenlu Ye, Wei Xiong, and Tong Zhang. Online-dpo-rl: Unlock- ing effective reasoning without the ppo overhead, 2025a. Notion Blog.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. Xr preprint ar wrXiv:2408.12240 2024b.
Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, and Yeyun Gong. Process-basee self-rewarding language models. arXiv preprint ar 1rXio:2503.03766, 2025b.
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evalu- ating the performance of large language models gaokao benchmark. arXiv preprint arXiv:2305.12474 2023.
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayi- heng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning. ar Xiv preprint ar XXix:2412..06599 2024.
15

---

Published as a conference paper at COLM 2025
A Prompts for DeepSeek-V3
Here we provide the prompts used to construct the SFT dataset and annotate reward labels with DeepSeek-V3 in Figure 6 and Figure 7, which are similar to that in Chen et al. (2025).
SFT Dataset Preprocessing
You expert math examiner, skilled at transforming complex mathematical solution steps into clear formats. Your task is to rewrite the solution in the following format:
1. Separating each step with 'mn'n.
2. the final answer in the final step, warp it with S) in its original place.
3. You can only insert "n'm', instcad of adding or repeating any other words.
<example>
He has 60-20- 200000000000-44--0 kilograms of mangoes left after selling them to the market.
Colby sold 1/2 x = 40 kilograms of mangoes to the community.
Therefore, Colby still has 20x8= <=00====00010000 1a [60)] picces of mangoes. <exxmmpl>>
A step should represent complete statement, structure calculation process. Your output should only include the revised solution and not omit any original content.
freference solution]
Figure 6: The Prompt Template for Dataset Preprocessing for DeepSeek-Ve V3.
Reward Label Annotation
You are an expert math examiner. Your task is to review the student's solution and evaluate each step. Steps wrapped in <step_n>, Mark step correct only if it is based on accurate premises and contributes to solving the problem. Mark it as unnecessary if is logically valid but does not aid in solving the problem. Your judgments should include a very concise analysis of each step and the final judgement.
You must provide your evaluations in JSON format like; ["step_]" ("analysis": "<concise analysis of the step>", "judgement": "=cornec/inoorec//meeees/ "). "step 2": (...)....
Below is the question, reference answer, and student's solution that you need to evaluate, Note that the student's solution does not need to match the reference solution exactly.
[Question] (question)
[Reference
[ReferenceAAnswer]] Answer] [answer]
[Student's Solution]
[solution]
Now, provide your evaluations of Student's Solution in JSON format.
Figure 7: The Prompt Template for Reward Label Annotaion for DeepSeek-Va
16

---

Published as a conference paper at COLM 2025
B Details of Processbench Evaluation
We evaluate the performance of the trained PRM on ProcessBench. Notably, in Process- Bench, the number of incorrect/ correct samples the four datasets are follows: GSM8K: 207/ /193 MATH: 594/4000000 /46; OlympiadBench: 661 /61/ OmniMath: 759/241. At this point, the traditional F1 score is not very effective for handling imbalanced datasets.
Therefore, an alternative approach is to separately define the Fl score for positive and negative samples (Hinojosa Lee et al., 2024). The former is equivalent to the commonly used F1 score, while the latter is slightly modified based on the above formula. Then, weighted F1 score is calculated based on the proportion positive and negative samples. The weighted-F1 score better reflects the model performance. We define positive samples as those labeled as incorrect. The formula for calculating the weighted Fl score is as follows:
x prssiiioe X possitive posiitioe + R positive
TP TP FP"
TP TP FN
F1 positive
P
Prssiiive
R Rossitive
2 Pnegative Rnegative Pneggaive
TN
TN
Fl Fluegative
R FN + TN Rnegatine
Pregative + Rnegative
FP + TN
Assuming the proportion of positive samples and the proportion of negative samples is B, the weighted-F1 score is defined as:
Fl weighted = a X F1 positive + f x F1 Flnegatice
Table 5 records the evaluation scores of the reward model on ProcessBench during the self-evolution process, with its average values corresponding to those recorded in Figure 3b.
GSMBK
MATH
OlympiadBench
OmniMATH
Average
FI-P
FI-N
F1
F1-P
F1-N
F1
F1-P
FI-N
F1
F1-P
FI-N
F1
FI
PRM-800K
71.4
SRM-eeatShepred
80.5
72.5
76.6
77.6
43.6
63.8
78.6
15.6
57.2
83.0
33,6
71.1
67.2
PURE-PRM
86.4
88.4
87.8
74.6
82.4
84.7
67.6
78.9
85.4
62.7
79,9
82.4
83.0
93.3
87.9
78.2
93.6
84.4
72.3
93.8
79.6
70.5
92.5
75.8
81.9
PRM-Epoch-0
37.1
97.9
66.4
62.5
88.2
72.9
77.5
64.3
73.0
88.2
61.8
81.8
73.6
PRM-Epoch-1
50.5
72.9
73.4
84.7
78.0
80.8
61.4
74.2
89.1
55.6
$1.0
76.5
PRM-Epoch-2
61.7
96.9
78.7
78.2
80.3
79.1
83.2
58.4
74.8
90.2
52.3
81.1
78.4
PRM-Epoch-3
73.4
73.6
Table 5: Weighted-F1 Score of Different PRM in Processbench.
https///gtttuu..ccmmmeeeee///sss 17

---

Published as a conference paper at COLM 2025
C Details of Test-time Scaling
Table records the test-time scaling results of the PRM trained with online-generated data from the current model each epoch, compared with the PRM from the previous round. The PRM selection strategy follows the PRM-min-vote approach (Wang et al., 2024b), where the smallest step reward value in the responses is used as weighting factor to adjust the majority vote result, ultimately selecting the final target answer.
MATH500
GSMSK
Gaokao-2023-EN
Minerva Math
Epoch
Methods
816
864
0256
816 864 0256 016 864 0256
@16 @64 8256
Greedy
64.2
86.4
55.6
20.6
Base
MV
77.2 78.4
79.2
93.1
93.8
94.1
63.9
65.7
66.0
23.9
27.2
28.7
PRM-0
77.2 79.4
80.0
93.4 94.1
94.4
63.9
65.5
67.8
24.3
27.2
27.9
Greedy
(+7.6)
90.8
(+4.4)
61.6 (-6.0)
26.5
26.5
MV
80.2 81.4 82.2
94.4 95.0 95.2
65.7
68.6
68.6
27.5
31.6
33.8
Epoch-1
PRM-0
80.8 80.8 824
94.7
95.3
95.5
67.1
70.1
68.9
27.2
32.4
34.6
81.2
95.2
95.1
67.3
70.4
69.6
27.6
33.5
34.6
Improvement
+04 +0.8 +0.2
+0.2 -0.1
-0.4
+0.2
40.3
+0.7
+0.4
+1.1
+0.0
Greedy
(+1.2)
90.9
(+0.1)
63.4 (+1.8)
28.7
(+2.2)
MV
94.5
94.7
66.8 66.5
29.7
Epoch-2
PRM-1
95.0
PRM-2
28.7
Improvement
+0.8 +0.6
+0.0
+0.1
+0.6
+1.0
+1.1
+0.5
+0.4
Greedy
(-20
91.5 (+0.6)
62.6
34.2
(+5.5)
MV
67.8
36.0
Epoch-3
PRM-2
81.2 82.2
82.6
95.1
95.2
67.5
68.9
70.2
36.1
37.8
40.0
PRM-3
81.6
82.6
83.2
94.6
95.3
95.4
67.1
69.1
70.4
36.8
37.9
40.4
Improvement
+0.4
+0.6
+0.6
-0.1
+0.2
+0.2
-0.4
+0.2
+0,2
+0.7
+0.1
+0.4
Table 6: Generator performance comparison across different epochs and methods for various datasets. Bold values indicate the best results.
D Details of DPO and RM Training and Evaluation
In Section 3.2-3.4, data sampling, and evaluation of the generator are based on the Open- RLHF (Hu al., 2024) framework. When filtering DPO data, the sample temperature set to 0.7. All training is performed using full parameter fine-tuning, with an SFT learning rate of 5e - 6, a DPO learning rate of 5e 7, a maximum sequence length of 2048, the coefficient B of 0.1 and training batch size of 256. For ORM and PRM training, we use the TRL (von Werra et al., 2020) framework with a learning rate of 1e 6 and batch size of 256.
In Section 3.5, the DPO training follows the same configuration annealed sampling enhance response diversity. Specifically, for the first 3 epochs, the temperature is set to 0.7; for epochs 4-,, f = 1.0; and in the final epoch, f 1.2.
above but employs
All evaluations are conducted using the Qwen-Math evaluation codebase, with greedy decoding of t = 0.0, generating one output per input, and maximum generation length of 2048 tokens. In addition, we we provide the URLs of self-evaluated models footnotes, including Marus-2-7B-PRIMM and (wwen2.577B-PURE-Y
(ren2.5-7B-Simple-RR--Zee
Swven25-77--Simpl--R---
More details and code can be found at
https://github.comm wom.:uwenLL//wee22555555
https::/huugiingface Co///RIEE-R///ELuu--------------
18

---

Published as a conference paper at COLM 2025
E
More Results of DPO-VP Analysis
E.1
Token Length and Self-Reflection Vocabulary Analysis
We present the detailed token length dynamics, accuracy trends, and the comparison of different models in self-reflection in Table 7 and Table 8.
Avg. Token Length
MATH500
Minerva Math
Olympapaidbench
AMC23
AIME24
Average
577
695
662
748
667
Sien2.5-Math-7B-Bie
641
650
886
911
1164
850
B-rus-2-7B-PRIME
655
822
897
1020
1164
912
775
801
767
952
777
Swwn2:5-7-PURRE-V
626
646
863
850
1050
807
Owen2.5-7-DDFO-vP
654
691
924
886
1251
881
Table 7: Comparison of Average Token Length Across Different Models and Datasets.
Avg. Nums of "Let's"
MATH500
Minerva Math Olympaidbench
AMC23
AIME24
Average
Owen2.5-Math-7B Dwwn2.5-Mathh78--nsste
1.56
0.45
1.90
1.68
2.03
1.52
0.40
0.13
0.67
0.70
0.87
0.55
Furus-2-7B-PRIME
1.50
0.96
2.27
2.40
3.30
2.09
Qwen2.5-7B 5-5----e---
1.49
0.57
1.99
1.93
2.20
1.64
Owcn2.5-78-PURE
0.24
1.52
1.33
Qwen2.5-7B-DPO-vP
1.75
1.07
2.44
2.60
3.13
2.20
Table 8: Average Occurrences of "Let's" Across Different Datasets.
E.2 Comparison with Long-Chain SFT Methods
Recently, reinforcement learning methods built upon long-chain SFT have received increas- ing attention. We compare our approach with two recent methods that perform long-chain SFT using limited data: LIMO (Ye et al., 2025) and S1 (Muennighoff et al., 2025). Since both released models are only available at the 32B scale, we reproduce their settings by applying SFT on our base model, Qwen-2.5-Math-77 using their provided datasets. We follow their default hyperparameters and train for 15 epochs. During evaluation, sample with f 2K tokens). The performance on standard benchmarks is summarized in Table
0.6 and set the maximum response length to 32K tokens (in contrast, DPO-VP uses
Surprisingly, DPO-VP outperforms these limited-datt SFT methods on nearly all evaluated benchmarks. One possible explanation is that, at the 7B scale, SFT with small-scale long- chain data may offer less performance gain compared to its effect on larger models like 32B. Nevertheless, both LIMO and S1 still yield moderate improvements over the base model.
pass@1 acc
MATH500
Minerva
Olympaid
AMC23
AIME24
Average
Owen2.5-Math-7B
64.8
15.4
25.6
37.5
16.7
32.0
Owenn.5-Math-7BB-INN.
70.2
21.0
37.0
45.0
16.7
38.0 (+6.0)
Qwenn25-Math-778-S1
72.0
32.4
37.5
55.0
13.3
42.0 (+10.0)
Owen2.5-78-DPO-VP
74.8
35.3
36.9
67.5
26.7
48.2 (+16.2)
Table 9: Pass@1 Accuracy Results with Long- Sang-Chain SFT Methods.
E.3 Training Dynamics of DPO under Different Label Noise Levels
In addition to final performance, we also tracked the training dynamics of DPO under different noise levels. As shown in Figure 8, lower noise (e.g,, p 0.3) leads to higher training accuracy and more stable reward separation between chosen and rejected samples. In contrast, high-noise setups (e.g., p 0.7 or p 1.0) cause convergence slow-down and reward collapse, confirming the importance of maintaining accurate preference signals during training.
19

---

Published as a conference paper at COLM 2025
Figure 8: Training dynamics of DPO under different label noise levels. Lower noise consis- tently leads to better optimization behavior and more stable reward separation.
E.4 Ablation Study on the DPO Sampling Temperature
In this section, investigate the impact of sampling temperature on DPO performance. Our motivation stems from the observation that fixed low-temperature sampling may lead to insufficient exploration in later DPO iterations, resulting in early performance saturation. In our default setting, we used constant temperature = 0.7 across all DPO rounds. To explore the potential of lightweight exploration, we introduced simple temperature increase strategy: using t = 1.0 for epochs 4-5 and t = 1.2 for epochs 6-7. Table 10 reports the average accuracy across five benchmarks under both temperature settings.
Epoch
Fixed | = 0.7
Ours (temperature 1)
3
44.5
44.5
4
44.6
44.7
5
44.3
46.5
6
46.2
48.2
7
45.5
47.6
Table 10: Comparison of average performance (pass@1 across five benchmarks) under fixed and increased sampling temperatures.
As shown, the increased temperature leads to noticeable performance gains starting from epoch 5, indicating improved sample diversity and preference pair quality. This suggests that even without using formal entropy-based exploration objectives, simple sampling heuristic can effectively encourage diversity in preference selection and improve down- stream learning. We believe this highlights the potential of lightweight exploration strategies in DPO, especially in scenarios where reward supervision is sparse or static.
E.5
Use of Distilled PRM Instead of Fixed Verifier (e.g., DeepSeek-V3)
While large-scale verifiers such as DeepSeek-V V3 (a 671B MoE model) provide high-quality supervision, they computationally expensive to query. Each DPO iteration would require over 7.5K queries per epoch if DeepSeek-Vy V3 were used directly, which is prohibitive for multi-round training. To address this, we distill its supervision into smaller PRM initialized from Qwen2.5-7B, enabling scalable and efficient learning. Table 11 compares three one-shot filtering variants, showing that PRM achieves nearly identical performance to DeepSeek- V3 filtering.
Setting
GSM8K
MATH500
Gaokao
Minerva
Avg
+Outcome Label
89.8
74.2
62.9
25.0
63.0
+Qwen2.5-PRM
90.5
74.6
61.0
25.0
62.8
+DeepSeek-VV
90.3
75.1
61.7
25.6
63.2
Table 11: Performance of one-shot filtering with different verifiers.
20

---

Published as a conference paper at COLM 2025
This result supports the use of lightweight, distilled verifier. Additionally, formulation generalizes to settings where a fixed verifier is available- e.g, we use outcome labels as a special case of verifier supervision in Section 3.5.
E.6 Comparison Process-Level RL Methods
We include additional comparisons to strong RL baselines that use learned reward models, such as PURE-PRM, PPO-based method trained with PRM800K. Results in Table 12 show that our method achieves comparable better performance with less training complexity. We also report results for Murrus---BPRIME and rStar-Math-7B in Table 2, both of which adopt full PRM-based supervision. Their performance does not consistently surpass ours.
Method
GSM8K
MATH500
Gaokao
Minerva
Avg
Ours (3 epochs)
91.5
75,0
65.8
34.2
66.6
PURE-PRM
88.3
79.8
63.9
34.1
66.5
Table 12: Comparison with PURE-PRM (process-level PPO with PRM800K).
E.7 About Using Monte Carlo Rollouts in PRM Training
Rather than training PRMs via Monte Carlo (MC) rollouts, which are costly and require full reward computation, we directly label sampled responses online using a strong verifier. This reduces computation and aligns well with our iterative, feedback-driven optimization.
To verify this design, we compare our online PRM with variant trained on the public Math-Shepherd dataset. On ProcessBench in Table 5, our PRM achieves higher average Fl (73.6 vs. 67. suggesting that strong online supervision offers both quality and efficiency.
This strategy is also aligned with recent work (Chen et al., 2025), which uses bi-directional verifier signals to label process-level trajectories. While MC-based training remains more theoretically rigorous, our framework focuses on co-evolving generator and verifier under verifiable supervision. Therefore, this practical approximation is sufficient to support the findings of this work.
E.8 Evolution of Reasoning in DPO-Optimized Models
To further analyze these differences, we examine reasoning patterns in Appenedix Figure 9. In Swenn..-Math-77B, self-checking and confirmation mechanisms are present but inconsis- tently applied, often leading to repeated questioning rather than definitive verification. In contrast, Qwven..--Math-77-nntttee follows a more standardized CoT approach, ensuring a structured reasoning process. Our model, Qwen2.5-7B- 2...57---DP--II exhibits a distinct reasoning confirming the final answer after a single verification. This behavior likely results from the VP-based filtering process, which refines the model's ability to select and reinforce correct reasoning paths. As a result, the model develops greater confidence in its final outputs, effectively reducing unnecessary re-evaluation while maintaining robustness in problem-solving.
To better illustrate the behavioral differences between models, we present a case study comparing four model variants on mathematical reasoning task involving the apparent magnitude of star cluster (Figure 9). The task requires combining knowledge of logarithmic scaling and magnitude-distance relationships, and serves as strong testbed for verifying the model's ability to compute, reason, and self-correct.
F Discussion: Why Iterative DPO Leads to Improvement
Based on the rigorous theoretical analysis in Xiong et al. (2024), intuitively, iterative DPO leads to improvement because it allows the model to continually refine its policy through
21

---

Published as a conference paper at COLM 2025
Question: If star cluster is made of (405 stars, each of whose absolute magnitude is S-5S, compute the combined apparent magnitude of the cluster if it is located al a distance of $1 'mmahrmm (Mpe]S.
Qwen2.5-Math-7B: Internal Checking, Confirmation and Multiple Answers
To solve this problem, need use concept of apparent magnitude and inverse square light. The apparent magnitude star is given by the formula.....
For V. N 10"4 1)) and M4 m -30 Wymmall m_ [ (cobbindd)))) - -2.5 Wlog (10)(104)) 30 40 ....
Let's confirm this with Python code. in "python'a import
Confirm by Python Code
The combined apparent magnitude of the cluster is ((C
**.....
A high school math club is organizing competition where students are asked to solve a problem involving sequence of numbers.
The surn of the first 10 terms of the sequence is Vi ((/boxe(407221)
Re-answer the Question
The range of the composite funetton V g(x) T((()) W) A Vboxxed (-hinfty, Rinfty)) W/an- The value of V e(2) ()) is W (boxed W
high school maih club is organizing a competition where
Unable to Choose the Correct One
Qwahh2--Math-77 -Instruct: Answer Step by Step the CoT Formal
Qwen2.5-7B-DPO-N
Answer Once and Confirm Once with Python Code
To find combined apparent magnitude of the cluster, need to follow these steps//nin
To solve this problem, we need to
Substituting the valuea''n'ni m_ (Seext(clusserr 'log. (10)(10"4) 20 2.5 Vodot 4 = 20 = 10 Tha combined apparent magnitude of the cluster is V((0%).
20 2.5
1. Understand the relationship between absolute magninude and apporent magnituule **I
2, Caleulate the apporent magnitude of one St
Let's confirm this with Python code. "pythoniniuporr math Given
Trg CoT Reasoning
confirm by Python Code
Therefore:ia Qi m (fttet( cluster)) } - 225 Modot 4 -10 5 -15 AfataThe combined apparent magnitade of the cluster is ((boxed(-155))
The combined apparent magnitude of the cluster is ((1hoxed(10%%).
Get the Wrong Answer
Get the Right Answer
Figure 9: An Example of Comparing Reasoning Patterns Across Different Models.
updated feedback, thereby reducing reward misalignment over time. Additionally, the batch exploration strategy promotes diverse sampling, which enhances generalization and mitigates overfitting to early-stage preferences. The performance gains of iterative DPO on math tasks stem from the inherent self-reflection ability in the base model and the iterative nature of the near-online RL setting
E1
The Emergence of Self-Improvement is Strongly Correlated with the Reasoning Patterns in the Base Model
is widely believed in the industry that improvements in reasoning performance, or the emergence of long CoT, stem from the inherent capabilities of base models, such as self- correction and backtracking. Through RL and appropriate reward function design, these latent abilities can be further activated, leading to the emergence of more complex behaviors (Yeo et al., 2025). Moreover, as the length of reasoning is often positively correlated with answer correctness, linking the reward signal to accuracy encourages the model to generate longer responses with more reflection and backtracking, ultimately improving answer precision.
This is indeed the case in our experiments. We believe that the reason Llama3.1 fails to achieve inff-improveeent lies in the same principle. Compared to Qwen2.5, Llama3.1 does not exhibit emergent behaviors such as reflection and backtracking (Gandhi et al., 2025). As a result, it cannot further enhance itself using self-generated data. However, by first distilling long-chain CoT data to equip the base model with self-reflection capabilities, self-improvement can then be effectively realized (Luo et al., 2025).
E2 Iterative DPO Can Viewed as a High Off-Policy Degree Online RL
Swamy et al. (2025) demonstrated the theoretical equivalence between online RL and offline MLE under ideal optimization conditions. For complex mathematical reasoning tasks, there exists significant Generation- Verifiictinn Gap (GVP), meaning that verifying answer
22

---

Published as a conference paper at COLM 2025
(e.g., checking against outcome labels) is often easier than generating correct answer from scratch. In such cases, RL can effectively narrow the search space by leveraging a two-stage process-retaining only policies that optimize for simple verification function, thereby improving efficiency.
Iterative DPO implicitly constructs similar reward function by filtering self-generated data based on reward signals, sharing the same goal of maximizing cumulative rewards as RL: maxg Bng [Er Rst,, at)]. The key difference is that RL requires the simultaneous deployment of a generator and reward model, computing rewards, performing value iteration, and contrast, iterative DPO decouples these
optimizing the policy during online sampling. steps-sand sampling data at each epoch first, then performing selection and policy optimization afterward. This makes it highly off off-policy RI. approach, as data collection and optimization occur in separate phases.
During the period of conducting this research, we observed that leveraging iterative DPO for model optimization has already become viable low-cost alternative to RL fine-tuning some enterprises and research institutions when extreme performance not the primary goal. We are delighted by this coincidence, as it indirectly validates that we have caught up with the research frontier in the field LLM reasoning. We also extend our gratitude to Zhang et al. (2025a), Liang Wen (2025), and all open-source contributors in the LLM community for their selfless contributions to the open-source ecosystem.
23


Reviews:
[{"review_id": "zdUdpwRMmg", "reviewer": "Reviewer_V4JU", "summary": "This paper makes a compelling case that lightweight, preference-based tuning can match the performance of heavyweight RL approaches in math reasoning tasks—at a fraction of the cost. Starting from a Qwen 2.5-7B baseline, a single application of Direct Preference Optimization (DPO) boosts MATH500 accuracy by 11.6 points over vanilla SFT. More interestingly, repeating DPO while jointly training a reward model leads to further gains, eventually approaching state-of-the-art RL pipelines across five math benchmarks. Impressively, all of this is achieved in under a day on just a few A800 GPUs, suggesting iterative DPO as a highly practical alternative to RL-style training.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Strong empirical results.** The paper shows that even relatively coarse preference signals (positive–negative pairs) are enough toclose much of the gap to RL baselines on GSM8K, MATH500, Gaokao-EN, AMC23, and AIME24. In particular, 11.6 pts on MATH500 is achieved in a single round.\n\n**Joint training of generator and verifier.** The reward model (PRM) is updated with the base model, leading to measurable improvements (e.g., PRM weighted-F1 from 65 to 72) and better inference-time scaling (e.g., PRM@64 outperforms majority vote).\n\n**Efficient pipeline.** The full six epoch training run takes only about 12 hours on 4*A800 GPUs (or about 3 days on 1*A800), which is dramatically cheaper than RL pipelines.", "reasons_to_reject": "**Limited domain.** All evaluations focus on math and exam-style tasks. Although Gaokao-EN includes some open-domain passages, the paper does not demonstrate generalization to other domains such as code generation.\n\n**Incomplete baselines.** While the paper compares DPO against RL and MCTS systems, it does not evaluate against more recent strong SFT or hybrid pipelines (e.g., LIMO, S1). This omission makes it difficult to precisely identify where DPO provides the most value.\n\n**Limited filtering ablations.** The paper explores several filtering methods (Outcome, ORM, PRM ± offset) but does not deeply investigate the effects of temperature scaling, top-k/top-p sampling, or PRM training efficiency vs. accuracy trade-offs.\n\n**Lack of explicit exploration mechanisms.** Although the authors introduce mild annealed sampling, DPO does not incorporate exploration strategies such as entropy regularization or diversity sampling. The diminishing returns after a few rounds suggest further gains may be possible with lightweight exploration incentives.", "questions_to_authors": "1. Have you tried iterative DPO in domains beyond sparse-reward math tasks? For example, how does it perform on tasks like NLI or multilingual QA, where you have dense ground-truth labels instead of binary pass/fail signals?\n\n2. How stable is the method when the reward model is noisy or mis-ranked samples? It would be helpful to see an experiment that injects controlled noise into PRM scores to test how much the performance depends on accurate preference learning.\n\n3. Since DPO doesn’t explicitly encourage exploration, do you observe diminishing returns after a few rounds? Could lightweight strategies like entropy bonuses or diversity sampling help push performance further, without using the full RL?", "extra_scores": {}}, {"review_id": "KKo1Qe1Sj3", "reviewer": "Reviewer_1K3W", "summary": "This paper studies using iterative DPO to fine-tune LLMs on mathematical reasoning tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is easy to follow.\n2. The experiments are conducted on various models and the results support the main claim of the paper.", "reasons_to_reject": "1. The novelty of the paper is limited. The authors show that iterative DPO can be applied to enhance the reasoning abilities of LLMs, which does not introduce new methods or provide new insights.\n2. Applying iterative DPO on reasoning tasks has been explored before, e.g., in [1], it was observed that iterative DPO works for reasoning as long as the prompt data is well chosen.\n3. To make the claims more convincing, the performance in Table 1 should be reported for multiple training runs, given that the difference in accuracies, such as the pass@8 accuracies, is within the range of 1.\n4. When training the PRM, the authors use a stronger model, such as DeepSeek-V3, to provide the supervision signal. The resulting PRM is thus approximately fixed, as a proxy of the stronger model, and does not get stronger with more training iterations. A reasonable training strategy is to use on-policy Monte-Carlo unrolls, such as the ones proposed in [2, 3]. Could the authors comment on why not use this training strategy, which is more aligned with the outcome reward model training?\n\n[1] Dong et al., \"RLHF Workflow: From Reward Modeling to Online RLHF.\"\n[2] Wang et al., \"Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations.\"\n[3] Luo et al., \"Improve Mathematical Reasoning in Language Models by Automated Process Supervision.\"", "questions_to_authors": "See weakness above.", "extra_scores": {}}, {"review_id": "LL2ZW4DyrE", "reviewer": "Reviewer_znvZ", "summary": "This paper explores how DPO can enhance the reasoning abilities of large language models, particularly for mathematical problems. It shows that even a single round of DPO with simple filtering can boost performance for strong base models. The study introduces an iterative framework where both the language model (generator) and a reward model (RM) are improved together over multiple DPO rounds. This iterative process allows them to achieve performance comparable to much more computationally expensive RL methods, but with the advantage of being runnable on a single 80GB GPU, making it a more practical and cost-effective alternative to RL for enhancing reasoning.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-written, presents a clear hypothesis, provides strong supporting experiments, and draws a sound conclusion.\n\n-  They show that multi-round DPO can achieve mathematical reasoning capabilities comparable to state-of-the-art RL-based methods. Even a single round of DPO using self-generated data can significantly enhance mathematical reasoning performance.", "reasons_to_reject": "- While the paper is well-executed, its novelty may be limited, as prior work has already demonstrated the effectiveness of iterative DPO.\n\n- The paper heavily focused on *mathematical* reasoning datasets, making claims about enhancing LLM **reasoning** primarily applicable to the mathematical domain. More clarification on this in the writing would be helpful and less misleading.", "questions_to_authors": "- Relying on \"coarse\" methods (such as filtering based on simple outcome labels or minimum step scores from an RM) might potentially limit the quality or complexity of the preference data used for training. Have you observed this?\n\n- In section 3.2 the authors note that applying SFT for Qwen resulted in a performance reduction, attributing this decline to the \"inferior quality of the GSM8K and MATH datasets\" used for SFT. Do you think the quality of the base datasets being used, from which self-improvement data is derived has an impact here?\n\nMissing citations:\n- Improving Reasoning Ability of Large Language Models via Iterative Uncertainty-based Preference Optimization\n- Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning\n- ISR-DPO: Aligning Large Multimodal Models for Videos by Iterative Self-Retrospective DPO\n- Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier\n- AIPO: Agreement-Aware Iterative Preference Optimization for Length Exploitation Mitigation\n- Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level", "extra_scores": {}}, {"review_id": "Y65kvbWTeW", "reviewer": "Reviewer_AnMH", "summary": "This work presents an empirical study of iterative DPO as a lightweight alternative to RL-based post-training for improving mathematical reasoning. The authors show that (1) a single DPO round with coarse filtering already boosts a strong base model, and (2) a multi-round generator/RM co-training scheme DPO-VP reaches RL-level accuracy while requiring less compute. The manuscript is well-structured, the experiments are extensive, and the resulting insights will be of interest to researchers who seek alternatives. In particular, showing mutual evolution of the policy and reward model through iterative DPO is interesting.\n\nThe findings presented in this paper however is less surprising. Regarding the three key findings that authors listed:\n1. Earlier studies, some cited, have already reported gains from single-round DPO with answer verification or from iterative self-training (e.g., RPO [1], ReST-EM [2], SPIN [3]). Without answer verification, there have also been efforts [4] in leveraging DPO for self-improvement. \n2. To show that iterative DPO achieves online-RL level Performance, I would be expecting a direct comparison with the GRPO algorithm on the same training data, even if done on smaller models like 1.5B or 3B. The current baselines are trained on different datasets and schedules, making the comparison inconclusive.\n\n[1] Iterative Reasoning Preference Optimization\n\n[2] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models\n\n[3] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models\n\n[4] Bootstrapping Language Models with DPO Implicit Rewards", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The manuscript is well-structured, presenting a thorough empirical evaluation of iterative DPO as an efficient alternative. \n2. The experiments are extensive and provides interesting insights. \n3. The focus of this paper on resource-constrained scenarios is important for research community.", "reasons_to_reject": "1. Improvements of DPO and iterative DPO have been found by many previous works. The key finding 1 and 2 are less surprising.\n2. The paper lacks an experiment directly compare iterative DPO with online RL algorithms such as GRPO on the same data (even in a toy setting with smaller models) to substantiate its key finding 3.\n3. Analysis on self-reflective behaviors focused on the occurrences of \"let's\" in figure 4 and table 7, which seem less relevant. It should be more relevant to study the occurrences of \"wait\" and \"double check\".", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes to enhance LLM reasoning ability with DPO-based algorithms. The central contribution lies in showing that iterative DPO, combined with co-evolutionary training of the policy and verifier, can match the performance of reinforcement learning approaches while being more computationally efficient.

The major concerns from the reviewers are about novelty: many iterative DPO methods have been discussed in the existing literature. I also agree with this concern and would like to point out one missing citation [1], where iterative DPO has been carefully discussed and shown as a strong enhancement to the classical DPO.

However, I would also admit that the insights that the reward model can be co-evolved is new to me (and the reviewers). This places the paper at an interesting middle ground between DPO and PPO. In addition, during the rebuttal, the authors have included a wide collection of evaluation benchmarks, which consolidate the position of this work.

The final concern is from Reviewer 1K3W regarding the use of external supervision. I agree that this can be potentially improved or replaced with a real human annotation process, which can be left for future work.

To sum up, I would give an acceptance recommendation for this paper. But I would encourage the authors to include the missing citation and more discussions on the final concerns from Reviewer 1K3W.

[1] Xu et a., Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, https://arxiv.org/abs/2404.10719
Decision: Accept