Step-Controlled DPO: Leveraging Stepwise Errors for Enhancing Mathematical Reasoning of Language Models
Authors: Zimu Lu
Aojun Zhou
Ke Wang
Houxing Ren
Weikang Shi
Yunqiao Yang
Junting Pan
Mingjie Zhan
Hongsheng Li
Venue: Accepted by TMLR ()

Abstract: Direct Preference Optimization (DPO) has proven effective at improving the performance of large language models (LLMs) on downstream tasks such as reasoning and alignment. In this work, we propose Step-Controlled DPO (SCDPO), a method for automatically providing stepwise error supervision by creating negative samples of mathematical reasoning rationales that start making errors at a specified step. By applying these samples in DPO training, SCDPO can better align the model to avoid reasoning errors and output accurate reasoning steps. Qualitative analysis of the credit assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at identifying errors in mathematical solutions. We then apply SCDPO to an InternLM2-20B model, resulting in a 20B model that achieves competitive scores of 88.5% on GSM8K and 58.1% on MATH, rivaling all other open-source LLMs, showing the great potential of our method. The code, models and data are released to inspire future work.

Published in Transactions on Machine Learning Research (11/2025)
Step-Controlled DPO: Leveraging Stepwise Errors for En- hancing Mathematical Reasoning of Language Models
Zimu Lu
husima cuhk, www.kkew....k..k...ooooookk
Multimedio Laboratory (MMLab), The Chinese University of Hong Kong
Aojun Zhou
aojunzhoou@gmaillccom com
Multimedia Laboratory (MMLab), The Chinese University of Hong Kong
Ke Wang Multimedia Laboratory (MMLab), The Chinese University of Hong Kong
nengkk@link.cuhk.eddo
Houxing Ren
rentouuzing@ggaillccm
Multimedia Laboratory (MMLab), The Chinese University of Hong Kong
Weikang Shi Multimedia Loboratory (MMLab), The Chinese University of Hong Kong
shink320@mail..tinnghae edd..c
Junting Pan
Multimedia Loboratory (MMLab), The Chinese University of Hong Kong
Mingjie Zhan
smjdll@gmail.com com
SenseTimm
Hongsheng Li"
hsli@ee culk. edu.bk
Multimedia Laboratory (MMLab), The Chinese University of Hong Kong
CPII under InnoHK Shanghai Al Laboratory
Reviewed OpenReview: https: https://opeerevveau openreview. net/ ptt/formm? ? id= jp1AdIcKTj
Abstract
Direct. Preference Optimization (DPO) has proven effective al improving the performance of large language models (LLMs) on downstream tasks such as reasoning and alignment. In this work, we propose Step-Controlled DPO (SCDPO), method for automatically providing stepwise error supervision by creating negative samples of mathematical reasoning rationales that start making errors at specified step. By applying these samples in DPO training, SCDPO can better align the model to avoid reasoning errors and output accurate reasoning steps. Qualitative analysis of the credit assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at identifying errors in mathematical solutions. We then apply SCDPO to an InternLM2-20B model, resulting in a 20B model that achioves competitive scores 88.5% on GSMSK and 58.1% on MATH, rivaling all other open-source LLMs, showing the great potential of our method. The code, models and data are released to inspire future work.
1 Introduction
Recently, Direct Preference Optimization (DPO; Rafailov et al. (022bb) has emerged as popular choice for aligning large language models (LLMs) with relative leedback to improve the quality of generated text.
*Corresponding authors.

---

Published in Transactions on Machine Learning Research (11/2025)
Prior works Christiano et al. 20233 Pal et al. (2024): Xu et al. (2024) have demonstrated that reinforce- ment learning algorithms and DPO can improve the mathematical reasoning abilities of LLMss making the generated reasoning process more controllable. The final answer to a mathematical problem serves as nat- ural way to judge the quality of the modol's response, since a mathematical problem typically has a single correct answer. As a result, the responses producing the correct final answers are desirable and can serve as the preferred samples, while the ones reaching incorrect final answers are undesirable and can serve as the dispreferred samples.
However, solutions to a mathematical problem can be diverse, with many different reasoning paths arriving the correct final answer and many subtle ways to make mistakes. Determining the preferred and dispreferred responses based on the final answer is coarse and may be inadequate for capturing the intricacies of the sulti-step mathematical reasoning process. Previous studies introduce process supervision Lightman et al.) (2023); but requires large amounts of meticulous and expensive human annotation and only applies traditional RL algorithms.
In this paper, we show how to automatically provide explicit stepwise preference supervision by generating DPO
diverse dispreferred solutions that start making errors at a specific step. We propose (SCDPO), an simple yet effective algorithm that introduces stepwise supervision without necessitating extra human annotation. This approach starts with a model finetuned with question-solution pairs and possessing initial math-solving capabilities, which is used to generate solutions to a set of math problems. We choose the solutions whose final answers match those of the ground truth. We take each of these correct solutions and start generating with the model via modulating the hyperparameter of the model, i.e. increasing the temperature of the final softmax function, from various intermediate steps of that solution, and retain the samples where the final answer is incorrect. In this way, the steps before the intermediate step the same as the original correct solution, while the steps after are the ones with possible errors. During DPO training, the correct solutions are the preferred samples, and they are paired with the wrong solutions generated in this way, with the qucstion and the steps before the intermediate stop as the prompts. These stop-controlled training samples help models learn detailed reasoning abilities and are mixed with naive DPO training dala produced by only checking the final answer, which optimizes the general form of the solution.
Our contributions as follows:
We introduce SCDPO, method that automatically provides explicit stepwise supervision to enhance mathematical abilities of LLMs.
We conduct pilot experiments on chain-of-thought and code-integrated solutions, showing that SCDPO can effectively improve mathematical problen-solving performance of three diffcrent SFT models. We also conduct qualitative analysis of credit assignment of SCDPO.
Using SCDPO, finetune an InternI.M-200B model, which reaches 88.5% on GSM8K Cobbe et al. (2021) and 58.1% on MATH Hendrycks et al. (2021) demonstrating the great potential of our method.
2 Step-Controlled DPO Pipeline
In this section, we introduce Step-Controlled DPO (SCDPO), a pipeline for automatically generating pre- ferred and dispreferred responses to math problems, with annotations of erroncous solving steps, and using these responses in DPO training to enhance the mathematical reasoning abilities of LLMs. Our method con- sists of two stages: step-controlled data generation, and step-aware DPO training. The two stages construct feedback-alignment framework that is both effective and cost-officient.
Initial Model. Our method starts with an initial model, denoted as TSFT, which has been finel und with question-solutian pairs from math datasets such as GSM8K and MATH. When prompted with a math problem g. #SFT is able to generate step-by-step solution, denoted as a. a can be broken down into a sequence of reasoning steps, for example, a .u.....,tm) Here, t; (i 00...,m, m) represents either code reasoning step natural language reasoning step within For Chain-of- Thoughtf-Thhught solutions, the reasoning
2

---

Published in Transactions on Machine Learning Research (11/2025)
Preferred Solutions
Dispreferred Solutions
a.
Iaput 9
lapar R:
Inpat o
Enpet 4
b.
What fifteen more than a quarter of 48?
reference model I.. iift
Rind, need to calesdate gusrter
Pinnt nves cnlecintt quarter 48
6
6 jind
%
"pochen montlyy seatts 48 -0.5 mastu
pyyhho munryyy restt 4072 muttinyy.roaat
see't
4
[ (3n0
(ar))
Naxe 2 calculer dicces
aodd 15 ottaimed
Drny
pythoe
and result mwwtgro renatt www.iit add mioun
+h
winittn,, resut 68++ 15 mungg meat eddioss noukt
4
45s
01
(dis-ac)
(dis-sc)
Step-Controlled Data Generation
Figure Demonstration and example of the step-controlled data generation process. a. Step-controlled data generation. First, a solution reaching the correct final answers is collected, which we denote as a; (pre) Then, erroncous solutions that reach incorrect final answers are generated, starting from intermediate steps of a: ((rre), creating dispreferred solutions (dis-se) (dis-ec) and d:3
d/ (pree) The temperature of the newly generated steps gradually b. An example of pair of preferred and
di2
(dis-sc) These dispreferred solutions share
the steps before the intermediate steps with a; increases with each step to make the generation more erroneous. dispreferred solutions. The dispreferred solution starts making errors after particular intermediate step.
steps are separated by "(n". In code-integrated solutions, the reasoning steps are separated by special tokens as described in Wang et al. 2023a
2.1 Step-Controlled Data Generation
The data we collect is in two parts: naive DPO data Dusive and Step-Controlled DPO data Dsc-
samples, used to optimize the general form of the solution. To create Dnaive we prompt #SFT with math questions in the training sets of GSM8K and MATH. Each question is presented to TSFT multiple times and various solutions are generated, with lemperature of 1. If solution reaches the same final answer as the ground truth, and errors adjustments occur at any of the reasoning steps (we detect these by looking for strings like "error" "apologics"), the solution is seen as preferred, while the solutions that reach answers different from the ground truth are considered dispreferred. To find out the frequency of incorrect solution process reaching the correct final answer, we randomly sampled 87 solutions that reach correct final answers, and found that of the 369 reasoning steps in these solutions, only contain errors, which is a very small percentage (about 0.5%). This demonstrates that, in most cases, correct final answer indicates correct intermediate steps. The solution generation of each question stops when at least one preferred solution and one dispreferred solution are generated, or the number of solutions generated reaches upper limit of 100. We use questions from the training sets of the GSM8K and MATH datasets for solution sampling, and repeated sampling ensures that 99.8% of the questions in the GSM8K training set and 91.8% of the questions in the MATH training set yield at least one positive sample. The resulting data can be expressed as:
Generation of Daive- Dnaive contains pairs of
D Dnaive Ulg, (pre), a ((dit)) ;i 1, Nnaive
3

---

Published in Transactions on Machine Learning Research (11/2025)
(dis) represent the preferred and dispreferred solution to Here, qi denotes the ith question, while (are) and a; the ith question.
Generation of Dsc. In order to generate solutions with stepwise error information for DPO training,
we propose method to automalically generate training data with errors starting to occur at a controlled
step. The process is demonstrated in Fig.
I We first take a preferred solution from Duaive, denoted as ((re)) is random intermediate step within ag (pre) As (pre] = (f( ......k *k+1 ....,.,, ). Here, "k
(pre) ((re)) .(pre))
aj (pre) is correct solution, (pre) ..... Lm; can all be seen as correct steps. As shown in Fig, a, to
((pe))
create solution with errors occurring after step k, we present TSFT with sequence (qi, 40 e(pre) % (
and raise the lemperature of the final soflmax function to affect the generation quality, increasing the occurrence of errors in the following steps. Raising the temperature causes the model performance to become unstable and erroneous, We observe that when the temperature is instantly raised and remains at high value, the model can generate garbled strings as errors accumulate, which does not represent any reasoning mistakes and contains no valuable information. To avoid this, we adopt. a gradually increasing temperature, which initially starts at 1.1, and increases by 0.05 with each generated step, until the generation ends or the temperature reaches 1.4. This sctting empirically reduces the frequency of the occurrence of garbled text, while increasing the error rate and diversity of generated errors. We generate the steps following (gi, (prr)) ,t. . multiple times, until one reaching an incorrect answer is generated. Appending the
generated steps (t( .....fR ((pre)), ) we get dispreferred solution with step-controlled error, denoted Oik
(dis-se) (pro) ....., l k+1 ..... th. ( where the sequence (1641 ,...,,,, ... (diss))) is erroneous. An
(pre) (disise)
example is presented in Fig. b. The resulting data can be expressed as:
(pre) (disuste) i== 1,..., Nsc Dsc ((gi, ((re) ait
Here, q: denotes the ith question, while a, (pre) is the preferred solution, and 0 with step-controlled error that occurs after (pre) Nsc is the number of questions in Dsc, while my is the index of the last step of ai
(dis-sc) the dispreferred solution
(pre)
2.2 Step-Controlles DPO Training
Having collected Dnaive and Dsc, we apply them to DPO training. Drnive serves lo regulate the general form of solutions, while Dsc supervises the model's reasoning on step level. During DPO training, samples in Dnaive and Dsc are mixed together randomly, and the DPO loss is applied to each sample. For samples from D Duaive the loss is applied all steps in the preferred and dispreferred solutions, which can be written
To (a ((((pr)))a))) TSFT (a (pre)
Ta(a (dla) (i)) (qs) Tarr(a (q:) )
Guaive (To;; TSFT) = -E (a . (000)) Duniive log
$
- B log
(1)
(dis)
For a pair of preferred and dispreferred solutions in Dsc where the erroneous steps are generated starting from the kth step of the preferred solution, the preferred solution can be denoted a. (ure), and the dispreferred solution can be denoted as 0 (dis-ac) The first reasoning steps are shared between the pair of solutions. The erroneous steps after the kth step in 0j (dis-sc) denoted as (ffdis-sc) ....,,y, tn, (disss)) while the correct steps
(....,,ere)), in a (pre) after the kth step is denoted as (t(pre) ...,, tm; ((pre)). SCDPO directly contrast between the steps in
(dis-sc) after the kth step, applying the DPO loss only the different steps. a (pre) and ag
4

---

Published in Transactions on Machine Learning Research (11/2025)
To find out how ftrffiis remained, we need to follow these
To find out how many froeiiss remained, we nead to follow these
steps
Itepsi
Start with the initial count of firefies) 3,
11 Start with the initial count of froelies:
dozen more lirefiest Maxi(dozen) =[12
2. lees dozen more froelles: dexee) 12(3)
so 4 times Weod ess han a dozen]) 4 ltimes (12 M]
so V 4 times (eexig(less than a dozen)) 4 Wimmss (12 4)10
number fireflis before two flew away.)
number firellie before two flaw away
4 Sustract because two of the fireflies flew awayy.
4. Subtract because two of the fireflie flew awayy)
Figure 2: Credit assignment of part a solution for a GSM8K problem. Each token colored corresponding to the DPO implicit reward as expressed in Eq. | (darker is higher). The left is the credit assignment of SCDPO, which correctly highlights the crror 4 less than a dozen not times (12 - 4), while the credit assignment of DPO on the right fails to highlight it.
(SC(TT; mSFT)
((X) mi
Σ
To (t) (tffrre)) (t (1fpre))
qi (est las, tes)
To To (f
las, tes)
- - -B (mriauuo)) ia ( DDc log a
log
3 log
(2)
#SFT
TSF (dimne))
lat, tas)
Combining Lnaive and Esc, the final loss function of Step-Controlis DPO is as follows:
LSCDPO
-naive Lsc
(3)
In this way, Luavve optimizes the general form of the solution, while Lse focuses on detailed reasoning steps, thus improving the model's accuracy in solving mathematical problems.
3 Theoretical Explanation of Step-Controlles DPO
Theoretical Insight. In this section, provide some theoretical insights into why SCDPO can effectively enhance the reasoning ability of LLMs. As explained in Rafailov et al. (2024a), the DPO loss can be cast into token-level MDP (Markov Decision Process;
Puterman (1994)). Similarly, we can also interpret DPO Fofelune (ostes)
mott (ostees)
as a step-level MDP. As presented in Eq. 2 B llg (t ((or)) (a.tes) and Glog Tger(t) (1)0)))))) (g,tey)) represent the
reward of a single preferred or dispreferred step. For naive DPO, all steps in the preferred and dispreferred solutions have their rewards affecting the loss. However, many steps in the dispreferred solution are actually correct, as the error often occurs in a later step. Step-Controlles DPO reduces the range of steps, starting from the (k+ + 1)th step, from which the dispreferred steps are more likely to be erroncous due to the raisod sampling temperature. The focus of the optimization is thus cast the errored steps rather than the whole solution, letting the model learn more detailed reasoning abilities.
Qualitative Evaluation of Credit Assignment of SCDPO. We perform qualitative evaluation of credit assigmment on two models trained with SCDPO and DPO respectively. For a sequence of tokens x = (Zo....,Im), Im), where I; is the ith token in the sequence, we denote all the tokens before I; as St, written as s; = (ro.... 2;-1). As introduced in recent research Rafailov et al. (2024a), the DPO implicit reward can be expressed as follows:
r (s; v.) logs(s;(ss) 3 log TSPT TSPT(2;(s) (z:(s.)
(4)
Here r(s;, .r) denotes the DPO implicit reward of token F,, which is the value we visualize as the background color of the token. darker color represents a higher reward value. As demonstrated in Fig. P and Fig. 3 when presented with an incorrect reasoning step, SCDPO more accurately identifies the incorrect tokens compared to DPO. Fig. 2 shows part of solution for GSM8K question. In step 2, the solution incorrectly
5

---

Published in Transactions on Machine Learning Research (11/2025)
When acding or suttracting numbers, the remaindera behave
When adding or subtracting numbers, the remainders bahave accordingly. case, adeinng terms, remainder will bei
sccordingly cur case, acding terme,
remainder will be:
Mox! Remmander sum) = (8+1 Imod
Rexmmaaadd eum) (B +1 Imod
Figure 3: Credit assignment of part of a solution for a MATH problem. Each token is colored corresponding to the DPO implicit reward as expressed Eq. | (darker is higher). The left the credit assignment of SCDPO, which correctly highlights the error as the original question was "Find the remainder when 8- 1008 + 1 is divided by 9", the remainders of the terms 8, 10 and 1 should not be summed, while the credit assignment of DPO on the right fails to highlight the error.
interprets "4 less than dozen" " "4 x correctly highlights "4 (12 4)", while the DPO does not. Fig. shows part of solution for MATH question. The solution sums the terms in the expression when two of the terms should have been multiplied. SCDPO correctly highlights the incorrect solution, while DPO does not. Several other credit assignment analysis examples are shown in Fig. 6 Fig. 7 and Fig. g of Appendix
when it should have been "(12 4)". The SCDPO model
[ respectivvly. These examples show that the stepwise supervision provided in SCDPÓ results a better token-level understanding reasoning errors.
4 Experiments
In this section, first train a 20B model using SCDPO, reaching performance rivaling all other models of similar scale. Then, we perform a comprehensive empirical comparison between SCDPO and DPO on three kinds of Mistral-7B SFT models. We also present ablation studies to further explain the design of increasing temperature during the generation of erroncous steps and combining D Duaive with Dsc during training.
4.1 Main Result
Training Data. We collect solutions for questions in the training set of APE210K Zhao al. (2020) GSM8K and MATH from GPT-4 Code Interpreter. Combining 169K samples from APE210K, 34K from GSMSK and 47K from MATH, we get an SFT dataset of 250K question-solution pairs. The SCDPO and DPO training data collected as described before in Sec. 2.1 During sampling, top-p is set to and top-k is set to consider all tokens. The training data for SCDPO contains 13K samples from GSMSK, 46K samples from MATH, and 29K samples from APF210K.
2024 the foundation model, it has demonstrated Cai et al. 20244) even surpassing larger models such as Mixtral-8x7B Jiang et al. (2024) and Llama 70B Touvron et al. (2023) in some cases. Tn the SFT stage,
Training Scttings. We use InternLM2-205 Cai ct al.
high performance in previous works Lu et al. (022))
we finetune the model with a learning rate of 1.0 x 10 5 for 3 epochs, with a context length of 2048 tokens. In DPO and SCDPO training, we use a learning rate of 1.5 x 10-7 to train the SFT model for 2 epochs, with a context length of 1024 and B3 set to 0.1. The models are trained on 16 NVIDIA A800 80GB GPUs with a batch size of 64. To further demonstrate the effectiveness of SCDPO, we also train two other strong open-source models Qwen-2.5-1.55 1.ne--2---..---nttuuuu
Instruct and 2.ven22.5-7B-Instrru using our method.
Evaluation Datasets. Ten representative mathematical datasets covering wide range of question types and difficulties are used in evaluating the models: GSM8K Cobbe et al. (2021), MATH Hendrycks et al. 22021,, OCWCourses (OCW) Lewkowyez el. al. (2022), Hungarian National Exams (hungarian) Paster (202)), Mathematics Saxton et al. (2019), SVAMP Patel et al. (2021) Simuleq Kushman et al. (2014). APE210K Zhao al. (2020), CMATH Wei al. (2023b) and MGSM-2h Shi al. (2023). The first seven datasets consist of English math questions, while the last three consist of Chinese math questions. The evaluation datasets contain a wide range of problem types, covering mathematical problems from grade-

---

Published in Transactions on Machine Learning Research (11/2025)
English
Chinese
Model
Size
GSM8K MATH OCW Hung- Mathe- SVA- arian
Simul-
APE-
CMA- MGSM-
matics
MP
eq
210K
TH
sh
Proprietary Models
GPT-3.5
80.8
34.1
41
73.8
GPT-4
93.6
53.6
30.1
92
84.2
89.3
GPT-4 Code Interpreter
97.0
69.7
GLM-
91.8
49.0
93.5
89.0
GPT-40
95.0
76.6
Claudo-3.5-Somme
96.4
71.1
SecpSock-V3
671B
89.3
61.6
90.7
90.7
79.8
Open-Souree Models
Qwen2
7B
85.7
52.9
10.7
56
51.4
86.3
83.6
54.2
73.8
58.0
Math-Shepherd
7B
84.1
33.0
12.5
46
36.6
81.8
84.6
45.9
68.8
67.6
DeepSeckMath-RL
7B
86.7
58.8
22.1
55
57.4
86.7
69.6
71.9
87.6
78.4
SVPO
7B
81.7
59.5
34.2
InternLM2-Matth
20B
80.7
54.3
12.9
66
41.1
83.4
65.6
64.3
69.0
58.4
MathGenie
20B
87.7
55.7
23.5
69
85.1
87.3
88.5
ChatGLM3-32B
32B
82.6
40.6
73
89.4
85.6
ToRA
80.7
50.8
5.5
77.9
50.2
53.4
41.2
MAmmoTTH
70B
76.9
41.8
65.4
84.3
51.8
MathCoder
70B
83.9
45.1
74.4
84.9
77.0
InternLM2 Family
InternLM2-SFT
20B
86.4
55.8
21.6
84.0
86.9
91.2
77.1
88.4
74.8
IntemLMM--PPT-DPO
20B
87.0
57.6
25.5
74
85.6
89.7
92.6
78.7
89.9
76.0
IntemLM22SFT--DPO....
20B
88.2
57.5
24.5
73
86.3
88.9
91.1
78.8
89.3
76.0
IntemLM22-ST--CDDPO
20B
88.5
58.1
29.4
78
87.5
90.2
93.6
79.3
90.3
80.4
Qwen2.5 1.5B
1.55B Family
1.5B
73.2
55.2
22.8
69
57.3
84.0
75.7
45.9
76.9
57.6
Owen2.5-1.5BBTImm
1.5B
74.6
56.7
23.6
71
61.2
83.7
84.5
45.4
78.2
60.0
Qwen2.5-15 1.t.5-1..----nntDP..P.
74.3
22.8
61.4
84.9
46.8
78.0
60.4
Qwen2.5 1.st1.5B-InttSSCDD
1.5B
75.7
56.5
24.6
76
64.5
84.0
84.7
57.2
83.8
62.0
Qwen2.5-7BB Family
Qwen2.5-7BBInst.
7B
91.3
75.5
38.6
76.4
88.0
87.9
79.4
88.4
79.2
Qwen2.5-785 DPO
7B
91.3
76.4
40.4
80
80.3
89.3
87.8
78.8
85.2
79.0
Qwen2 5-105-7mtmmmstt
7B
91.4
76.0
37.9
81
76.2
88.3
87.6
78.9
87.1
79.8
Qwen2.5-7B-Instt.ee 10mn255-BB-InttSSCDDP
7B
92.0
76.6
41.2
84
82.3
91.3
88.2
80.5
89.0
79.8
Table 1: Performance open-source and closed-source models on seven English datasets (GSMSK, MATH, OCW, Hungarian, Mathematics, SVAMP, and Simul-eq) and three Chinese datasets (APE-210K, CMATH, and MGSM-zh). All results are obtained with greedy decoding. The best score for cach block is shown bold; the second-best. is underlined.
school level college level, comprehensively evaluating the models' mathematical reasoning abilities. We greedy decoding for all evaluations.
Baselines. We compare our 20B models with powerful closed-source models such GPT-3.5 Brown et al.
2020), GPT-4 OpennAl et al. (2024). GPT-4 Code Interpreter OpenAl et al. (2024) and GLM-4 as well as
open-source models such as MARIO Liao et al. 20224) Qwen2 Yang et al. (2024). Math-Shepheri Wang et al.
2024h, ScaLLM-v2 Nguyen et al.
(0024). DeepSeckMath-PF
Shao et al. (2024). SVPO Chen et al. (0024]
3kywork-13R-MMr Yang et al. (2023a), M2tt [ Ying al. (2024), MathGenie Tu et al.
(0024)
Xu al. (2024), Yi-Chat Yi (2023), ToRA Gou al. (2024). MAmmol TH Yue
ct al. 2023 MathCoder Wang al. 2023a and WizardMath Luo et al. (2023).
https://open. http::/ppo..biimmddm.ccc////aaas https://github... http:///hhuub.ccoo//ttteee////// http://ggthaa.ccm/mmmmm///ettttt

---

Published in Transactions on Machine Learning Research (11/2025)
Method
GSMsK
MATH
OCW
hungarian Mathematics
SVAMP
Simuleq
Mistral-7B-Our
SFT (Baseline)
76.8
43.2
21.7
69.8
81.3
73.9
SFTT-contineed
76.3
43.9
18.8
55
70.3
80.8
74.5
SFT+DPO
78.8
45.
18.4
56
74.8
81.0
74.9
SFT+DPO (d-e)
79.0
45.7
18.0
59
74.4
79.2
73.2
SFT+DPO+SC
80.1
47.7
22.4
61
76.5
82.3
79.0
MetaMath-Miisrra-77
SFT (Baseline)
77.7
28.2
12.5
33
33.9
80.0
68.5
SFTT-contimed
76.8
28.5
13.2
35
33.6
80.3
69.1
SFT+DPO
81.0
28.7
14.0
34
33.8
81.0
68.3
SFT+DPO (a.e)
81.4
29.0
14.7
38
34.3
80.9
70.6
SFT+DPO+SC
81.7
29.3
15.4
35.0
81.6
73.2
MaalhCoder-Mistnnng
SFT (Baseline)
78.1
39.3
12.9
62
70.4
79.4
80.5
SFTT-continued
78.2
40.3
12.5
65
71.2
77.3
80.7
SFT++PO
79.2
42.9
14.3
65
74.9
85.4
81.3
SFT+DPO (a-e)
78.3
41.1
14.7
68
74.9
84.9
82.3
SFT+DPO+SC
80.4
43.4
15.7
70
75.4
85.4
83.1
Table 2: Effect of using Ceet-Coutolleed DPO (SCDPO) on three different SFT models: Mistral-7B-Outt BottaMat--Msstral-a and MathCoder- Mistdrr-isttaa-7B. "(d-c)" denote the DPO bascline using the same amount of data as SCDPO. In all three cases, SCDPO outperforms the starting SFT model, continue pretraining on correct samples, naive DPO, and naive DPO with equal amount of data.
Model
GSMSK
MATH
Mistral-7B-Ours-SP
76.8
43.2
Disttal-7BBOOursSCCDPP (temperature=1.0) Bestral-7B-OursSCIi (temperature= 1.3)
78.6
45.9
80.0
45.9
Miistrl-7B-Ouus-SCDDPP (ascending temperature)
80.1
47.7
Table 3: Pilot experiments of using different temperatures when generating error steps. When temperature
equals 1.0, the errors are not diverse enough. When temperature equals 1.3, the model generates unintelligible strings due to accumulated errors. The design of ascending temperature offers more diversity while avoids generating meaningless errors, resulting in the best performance.
Main Results. Tab. displays main results, well various closed-source and open-source basclines. InternLM2- -W--M--FFT-SCDPOOO achieves score of 88.5% on GSMSK, 78 on hungarian, 87.5% Mathematics, 90.2% on SVAMP, 93.6% on Simuleg, 90.3% on CMATH, and 80.4% on MGSM-zh, surpassing all pre- vious models with published parameters, and obtaining second-best scores among open-source models on APE210K. Our model obtains a score 58.1% on MATH, which close to the best and second-best open- source score of 59.5% and 58.8%. While our model rivals the performance of GPT-3.5 on GSMSK and MATH, and surpasses GPT-4 and GLM-4 on MATH, still underperforms GPT-4 Code Interpreter on GSM8K and MATH, and GLM-4 APE210K.
Compared to MaternLM2-SFTT InternLM2-SFT-SCDD consistently increases the score each of datasets by approximately 2% to 3%. Compared to both IntemLM22SFTT-DPO, which uses the D part of MMterLLM2-SSTT-SCDPP training data, and InteruL.M2-SFT-DDPO (dtaeeuuuu) which uses about the same amount of training data as SEeuLMM-SFFT-SDDDEE performance across all five datascts, highlighting the effectiveness of SCDPO enhancing mathematical problem-solving abilities. SCDPO consistently improves performance across ten mathematical datasets on the Qwen2.5 models, with 1.ven-2.5-7B---strutt-e--
ME-eemLLM-SFF-SSCDDDD consistently achieves the best
achieving scores of 92% GSM8K, 76.6%

---

Published in Transactions on Machine Learning Research (11/2025)
Model
GPQA (%)
SFT
26.9
SFT DPO
7.3
SFT + DPO (e-q) SFT DPO + SC
26.8
28.3
Table 4: Performance of InternLM2-20E variants on the GPQA benchmark.
MATH, and 41.2% on OCW, approaching the performance of the proprictary models. Notably, Qwen- 2.5s..BBl and Qwen-2.5-7B-Instrrue had already been trained with multistage reinforcement learn- ing including offiine DPO and online GRPO (Yang et al. 2024); yet SCDPO continues to improve their performance. This result demonstrates that SCDPÓ can be combined with RLHF-basee tuning methods.
4.2 Effect of SCDPO on Other Subjects
In this paper, mostly focus mathematical problem-solving tasks it is representative reasoning task and can be easily verified. To demonstrate SCDPO's effect other subjects, test our 20B models on GPQA Diamond, a challenging multiple-choice Q&AA datasct featuring difficult questions in biology, physics, and chemistry. As shown in the table below, SCDPO also improves performance on GPQA, demonstrating the potential of our method for reasoning tasks in subjects beyond mathematics. We will explore the effect of SCDPO on other reasoning tasks such as code gencration and theorem proving in future works.
4.3 Comparison using Different Starting 7B Models
We validate the generalizability of SCDOP on three baseline SFT models: Mistral-7B-Oun MetaMath- Mistral-7B, and MathhCoder-Mistral-777 Mistral-7B--Ours is finetuned the 34K GSMSK samples and 47K MATH samples we collected from GPT-4. MetaMatt-MMistral-73 is downloaded from the MctaMath Hugging FaggningFccee repository@ MathhColer-Mistrrl-1BB is finetuned using the MathCodelnstrucl dataset Wang et all 2023a), downloaded from HuggingFacd"" We collect Duaive and Dsc as described in 2.1 using problems from GSMSK and MATH. We compare 4 methods of aligning the starting SFT models: 1. Continue
finetuning the starting SFT model using supervised finetuning with preferred solutions from Duaive (SFT- continued). 2. Doing naive DPO training with Daaive (SFT+DPO). 3. Doing naive DPO training with the same amount of training pairs as the SCDPÓ training, expanded from Dnaive (SFT+DPO(ae)). 4. Doing SCDPO training with Duaive and Dsc (SF++DPO+SC).
The results are shown in Tab. 2 The purpose of SFT+DPO((a)) is to rule out the possibility that the performance gain of SCDPO is the effect of more training samples. SFT-continued shows no obvious gains, likely due to the fact that the models has already been finetuned on many solutions from GSMSK and MATH. As demonstrated in Tab 2 on all three SET baseline models, SCDPO shows superior performance compared to DPO. This can be attributed to SCDPO's more detailed supervision on the reasoning steps of the math solutions, demonstrating the effectiveness of our method.
4.4 Ablation Study
Analysis of the Increasing Temperature Design. We present the result of using different temperature during sampling of erroneous steps in Tab. 3 Originally, we tried sampling for the incorrect solutions at the same temperature as the correct solutions (1.0). However, we observed that generated error steps are less diverse than we hoped. Also, as shown in Fig. g the accuracy decreases with the increase of temperature, as the generation bocomes less stable. We then tried raising the temperature to 1.3, and found that a notable part of the generated solutions contains incomprehensible strings at later steps due to accumulated errors. Finally, we settled on raising the temperature gradually, which enables more diversity while lowering the

---

Published in Transactions on Machine Learning Research (11/2025)
80
GSMBK
60
Accurccy
40
20
Temperature
Figure 4: Accuracy of Mistral-7B-Our (SFT) on GSM8K and MATH when temperature is set at different values.
Data
GSM8K
MATH 80.1%
DSc
79.0%
46.2%
Daiiv + Dsc
47.7%
Table 5: Ablation study of using and not using naive during training. The starting SFT model is Mistral- 7B-Ours.
frequency of generating unintelligible sentences. As shown Tab. 3 this method also performs best in the pilot experiments.
Ablation Study of Duaive and Dsc. To demonstrate the necessity of combining Dusive and Dsc, conduct experiment of using only Dsc in DPO training. The results are presented in Tab. 5 As demenstrated in the table, combining Dnaive and Dsc results in better performance than only using Dsc during DPO training. This is likely because Dnaive helps regulate the general format of the generated solutions.
Comparison with Step-DPO. We also compare our method with Step-DPO Lai al. (2024), a work concurrent with ours, which uses GPT-4 locate erroneous steps. As Step-DPO is in the Chain-of- Chainhof-Thought format, we train SaiaaMatt-Mistral--- using the dataset and code of Step-DPO. As shown in Tab. our method outperforms Step-DPO most datasets without relying any stronger LLMs (e.g., GPT-4), remonstrrating the effectiveness of our approach.
4.5 Data Quality Analysis
Analysis of Generated Errors. In this section, we provide quantitative analysis of the erroneous steps generated. We observed seven main kinds of errors: value misusage, condition misinterpretation, coding er- ror, commonsense error, math concept or understanding error, math calculation error, unintelligible strings.
The errors explained as follows: (1) Value misusage: misusing values in places where another value should have been used. (2) Condition misinterpretation: incorrectly interpreting the meaning or indications of conditions.
(3) Coding error: making mistakes in code snippets that causes errors. (4) Common sense error: misunderstanding common sense. (5) Math concept or understanding error: incorrect recollection or understanding of math concepts. (6) Math calculation error: mistakes when making mathematical cal- culations. (7) Unintelligible strings: generation of unintelligible strings that does not represent meaningful reasoning errors. We randomly sampled 100 incorrect solutions in the training data of SCDPO, and counted the number of each type of error. The result is presented in Fig. 5 As demonstrated in the chart, the reasoning errors generated is diverse, distributed evenly among the different types. Only 4% of the incorrect solutions contain unintelligible strings, demonstrating that the design of gradually increasing temperature can mostly avoid the occurance of meaningless errors.
Error Rate of Intermediate Steps When the Final Answer is Correct. In this section, we discuss the error rate of intermediate steps solutions that reaches the correct final answer. As we mentioned in the main paper, randomly sampled 87 solutions that reach correct final answer, and of the 369 reasoning steps
10

---

Published in Transactions on Machine Learning Research (11/2025)
Method
GSMsK
MATH
OCW
hungarian Mathematics SVAMP
Simmleq
Step-DPO
80.4
29.3
12.5
42
33.4
80.4
72.6
SCDPO (ours)
81.7
29.3
15.4
42
35.0
81.6
73.2
Table 6: Comparison between our method and Step-DPO on MetaMath- Misttttll--
condition misinterpretation
21.0%
coding enoor
misusage
0%
14.0%
4,0%
unintelligible strings
16.0%
0%
Common sense error
14.0%
meth calculation error
math concept or understanding error
Figure 5: Percentage of each type of error in the 100 examples we sampled.
in these solutions, only 2 contain errors, which is a very small percentage (about 0.5%). The 2 erroneous steps are in a question whose answer is to ask for "the number of real solutions". The possible answers to this question is very limited, which facilitates incorrect reasoning steps to happen to reach a correct final answer. We observe that this kind of question is rare the GSM8K and MATH datasets we chose to perform SCDPO on, so in most cases a correct final answer indicates that the reasoning steps leading it highly likely to be correct. We also removed solutions that contain apologies error messages, so there are no incorrect steps that are later revised in the correct solutions retained. As result, the steps before the generated erroneous steps in SCDPO training data are correct with a high confidence.
Distribution of Error Generation Starting Points and Actual Error Step Index. We sampled 100 SCDPO rejected solutions and mamually inspected them. While not all errors occur at the beginning of the generation erroneous steps, most of them occur early in the generation, as the earlier steps often make decisions that have large influence on the correctness of the solution. We present the distribution of the step index where the first actual error occurs in the 100 SCDPO rejected solutions, as well as the distribution of the index of the step where generation of erroneous steps begins. As shown in Tab. the two distributions are closely related, demonstrating that the index of the starting point of error generation have a strong effect on the step index of the actual error. The step index of the actual error is often close to the starting point of the error generation. We also manipulate the distribution of the step index where error generation begins. Specifically, we train the Mistral- 7B-SFT model with SCDPO data where the starting index was either less than or equal to 4, or greater than As shown Tab. g limiting the crror starting point index decreases performance compared to not imposing any such limitation.
5 Related Work
LLM for Mathematical Reasoning. Prior works have explored various methods to euhance mathematical reasoning abilities of LLMs. Prompting methods, such as Chain-of- Chain-offThought WWi et al. (2023a). Tree-of- Thought Yao et al. (2023) PAI Gao et al. (2023), Program-of-Thoughh Chen et al. (2023). and CSV Zhou et al. (2023), use carefully engineered prompts bring out LLMs mathematical skills without changing their parameters. Other works optimize parameters of L.MMs for enhanced mathematical reasoning through
11

---

Published in Transactions on Machine Learning Research (11/2025)
Step Index
2
3
4
5
6
8
9
10
11
12
13
14
Actual Error Distr.
17
21
20
12
8
9
5
3
0
4
0
0
Starting Point Distr.
32.8
23.1
15.3
11.4
6.97
4.72
2.53
1.70
0.77
0.44
0.16
0.08
0.01
Table 7: The distribution of the actual error starting step index in 100 randomly sampled SCDPO rejected solutions and the distribution of the starting point of the crror generation of the SCDPO rejected solutions.
Data
GSM8K
MATH
OCW
Hungarian Mathematics SVAMP
Simuleq
DFT+
80.1
47.7
22.4
61
76.5
82.3
79.0
SFT+DPO+SC (k S 4) SFT+DPO+SC (k 4) 79.6
80.7
46.7
17.7
74.7
82.0
79.2
46.4
17.3
59
74.1
82.3
78.6
Table 8: Comparison of performance on various datasets using different ranges of error generation starting point during training. "k" is the index of the starting step of error generation. Limiting the starting index slightly affects performance compared using the full range of indices.
either pretraining or finetuning. Llemma Azerbayev et al. (2024 and MathPile Wang et al.
2023b continue
pretraining LLMs on large amounts of math-related data, while RFT Yuan et al. (2023) 3, Mammoth Yue
eet al. (2023) MathCoder Wang et al. (2023a). WizardMath Luo et al. (2023), ToRA Gou al. (2024)
MetaMath Yu et al. (2024), MathGLM Yang et al (2023b), and MathGenie Lu et al. (2024) finetune pretrained models on question-solution pairs. These methods effectively improves LLMs ability to solve challenging mathematical problems, demonstrating impressive performance on mathernatical benchmarks such as GSMSK [obbee et. al. (2021), MATH Hendrycks et. al. (2021), etc. Our work builds upon models that have undergone pretraining and finetuning, using DPO to further enhance their mathematical abilities.
Improving Mathematical Reasoning Using Relative Feedback. Reinforcement learning from human (or AT) feedback Christiano et al. (2023): Bai al. (2022) as well as several direct alignment methods Rafailov et al. 2024b); Azar et al. (2023 Zhao et al. (2023); Pal et al. (2024); Ethayarajh et al (2024); Liu et al. (2024] have proven effective on various downstream tasks. Our method make use of DPO Rafailov et al. (2024b, introducing a novel way to construct the DPO training data for beller enhancement of mathematical abilities of LLMs. Previous works using reinforcement learning or direct alignment. methods for improving mathematical reasoning utilize either outcome supervision or process supervision. Outcome supervision such Shao et al. (2024) is simple and use the outcome of solution as supervision signal. Lightman et al.) (2023) found that process supervision offers better performance than outcome supervision, but needs expert and detailed human or Al annotation, which is difficult to acquire. Math-Shepherd Wang et al. 2023a and
Process Reward Synthesizing Jiao et al. (2024) estimate process rewards with multiple decoding rationales at
each step, and train reward model with the synthesized rewards. Other works such as Xie et al.
(2024) Yuan
ctt al. (2024) and Chen et al. (2024) use tree structure to provide fino-grained supervision, often relying on a critique model to decide the correctness of reasoning steps. Concurrent works such as /Setlu et al. (2024) and Lai al. (2024) rely on GPT-4 to synthesize data. Step-DPO (Tai et al. (2024) uses GPT-4 for erroneous step localization, which is less cost-effective. In comparison, our method uses increasing temperature to start generating erroneous steps from intermediate steps of a correct solution, and directly contrast erroncous steps with correct steps, offering a simpler, more cost-effective alternative with high performance, without relying on any stronger LLMs (e.g. GPT-4).
6 Conclusion
In this work, we propose Step-Controlled DPO (SCDPO), a method to automatically introduce stepwise error supervision to the process of DPO training by generating dispreferred samples that start making er- rors at specified step. SCDPO effectively enhances the mathematical reasoning abilities of LLMs. The 20B model trained with SCDPO on both English and Chinese data achieves high scores on 10 representa- tive mathematical datasets, consistently outperforming naive DPO, demonstrating the effectiveness of our method.
12

---

Published in Transactions on Machine Learning Research (11/2025)
References
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general thoorctical paradigm to understand learning from human preferences, 2023.
URL
12036
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q- Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2024.
URL
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueleer, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scolt Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Tawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan, Constitutional ai: Harmlessness from ai feedback, 2022. URL https: //arxii//arxi..../ org/ hts//bbs/2211// 2212. 08073
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internim2 technical report, 2024. URL
17297
Guoxin Chen, Minpeng Liao, Chengxi Li, and Kai Fan. Step-level value preference optimization for mathe- matical reasoning, 2024. URL / org/ abs/ 10858
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Dis- entangling computation from reasoning for numerical reasoning tasks, 2023. URL https://arxxv..rg abs/ 12588
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment. learning from human preferences, 2023. URL httpss://rxxi...t wwtp://///./........//.. 1etp//106.... 03744
Karl Cobbe, Vineet. Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL
14168
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024. URL
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models, 2023. URL https: http://rxiivv.rrr/a////1//11111/
1043
13

---

Published in Transactions on Machine Learning Research (11/2025)
Zhibin Gou, Zhihong Shao, Yeyum Gong, Yelong Shen, Yujiu Yang, Minlic Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving, 2024. URL https:
//axxiv.org/ab//22000 17452
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Sleinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https:
7/arxiv..o aaaxxiv.orgaatie wttg///////..../// 03874
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL https:
hwt.xivoorg/bbs/240000 24000
Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F. Chen, and Shafiq Joly. Learning planning-based reasoning by trajectories collection and process reward synthesizing, 2024. URL https://arxiv.orrs org/
abs/2402.00658
Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. Learning to automatically solve algebra word problems. In Kristina Toulanova and Hua Wu (eds.), Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 271-281 Balti- more, Maryland, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/P1141000 URL
https://aclanttoology http://caaaaaaoooooooo////t/2-20 org/P14-1026
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of lims, 2024. URL https://arxiv http://aaxxiivoo//////////// org/abs/246.. 18629
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models, 2022. URL
2200/ 14858
Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output- a reproducible pipeline, 2024. URL attp:://axxvvoorr//////000... wwg./bs////......o 08190
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Tee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL
org/
abs/ 20050
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization, 2024. URL https: //arxiv org/abs/ abs/
Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of 1lms, 2024. URL https://arxiv: org/ abaeaaassaaa 22///24422.2323 16352
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowcring mathematical reasoning for large language models via reinforced evol-instruct, 2023. URL
wttgg/bbs/30..000 095883
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. Seallms large language models for southeast asia, 2024. URL
httpss//aarxe http://axxii.oo/aa///////// prgg/ba//3122000
14

---

Published in Transactions on Machine Learning Research (11/2025)
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohamnmad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadctt-Shapirg Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufci Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirehner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katic Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Memick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reücchir Nakano, Rajoev Nayak, Arvind Neclakantan, Richard Ngo, Hyconwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambaltista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rac, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Sxymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Ben- jamin Sokolowsky, Yang Song, Natalic Staudacher, Felipe Petroski Such, Natalic Summers, Πlya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelleaa VOss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wicthoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hamnah Wong, Lauren Workman, Sherwin Wu, Jelf Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https:///axxii wttps/wwwwrr//b///33.33 08774.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive, 2024. URL https://arxiv. /bb// 2402. 13228
Keiran Paster Testing language models on a held-out high school national finals exam. https://
natiioalnnaaaaanaaaaaaaaaaaallal hs hatiillsssanaaaaaaaaaeeeaaaaaale finals_nnaaalleell 2023.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems?, 2021. URL htttp::/aarxi..or//bbb///////
Martin L. Puterman. Markov decision processes: Discrete stochastic dynamic programming. In Wiley Series in Probability and Statistics, 1994. URL http:////ppp.eematatccccccoarrra
15

---

Published in Transactions on Machine Learning Research (11/2025)
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to q": Your language model is secretly 12358
a q-function, 2024a. URL
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024b. URL https:
Starxiv.org/ab//233002 18290
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models, 2019. URL https://arxiv, http://axxvvvov..aaa//// wwwg/b///////0o 1904. 01557
Amrith SetJur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. RI on incorrect synthetic data scales the efficiency of Ilm math reasoning by eight-fold, 2024. URL https:
7/arxiv. otgg/ abs/a//s///// 14532
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepsockmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https:///axxiii http://xxxvvx...aaaaa/// orgg/bb//2400/0 223000002200000
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivals, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwando, May 1-5, 2023. OpenReview.neg net, 2023. URL https: http://wwwwe.eetttddddddddddaeee
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Tnan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Tsabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Augela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxxt: httalaaiivviiiraaaaaa wwg/ httg/abs/23001 09288
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in 1lms for enhancod mathematical reasoning, 2023a. URL lhttp:::/ //tt:i/////vvv.//a wttp://...rrraabbb....../
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce 1lms step-by-step without human annotations, 2024. URL https:
Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part i mathpile: billion-token-scale pretraining corpus for math, 2023b. URL https://arxii.......// wrg/ habbs/2312.1111
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou of prompting elicits reasoning in large language models, 2023a. URL https://arxiz: wwwpa/wwwwwgg//////.....11. 11903
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. Cmath: Can your language model pass chinese elementary school math test?, 2023b. URL
wwt.aawwww..rg/b////...... 16636
Yuxi Xic, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P. Lillicrpp, Kenji Kawaguchi, and Michael Shich. Monte carlo tree search boosts reasoning via iterative preference learning. 2024. URL https:
Fttxxv..rr//abs/40000000
16

---

Published in Transactions on Machine Learning Research (11/2025)
Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, Jie Tang, and Yuxiao Dong. Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline, 2024. URL https://arxivvorg// wwwgg//a///......///// 02893
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng Hc, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://arxiln org/ abs/ 220724407.0 2407.10671
Liu Yang, Haihua Yang, Wenjun Cheng, Lei Lin, Chenxia Li, Yifu Chen, Lunan Liu, Jianfei Pan, Tianwen Wei, Biye Li, Liang Zhao, Lijic Wang, Bo Zhu, Guoliang Li, Xuejie Wu, Xilin Luo, and Rui Hu Skymath: Technical report, 2023a.
Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Jinfeng Bai, and Jie Tang. Gpt can solve mathematical problems without a calculator, 2023b. URL https:///axiv orggabs/2309....
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. URL
org/ http//bb/220..1111 10601
Yi. A scries of large language models trained from scratch by developers at 01-ai.
com/
01-ai/Yi 2023.
Huaiyuan Ying, Shno Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, and Dahua Lin. Internlm-math: Open math large language models toward verifiable reasoning, 2024. URL huttps://arxie wrwiiii..rrrbbba abs/ 2402.06331
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024. URL
12284
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruob- ing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing I1m reasoning generalists with preference trees, 2024. URL https://arzii: www.riii..r//bbaa.. 2atp//244.00272
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2023.
URL
01825
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning, 2023. URL :tttps://arxivv abs/ 2309.0560
Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. Ape210k: A large-scale and template- rich dataset of math word problems, 2020. URL https://arxiv.orgee
11506
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback, 2023. URL http:://arriv..og/abbb///// 10425
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Tuo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. Solving challenging math word problems using gpt-4 code interpreter with code-based self- celf-verificatin, 2023. URL /cttss//arxiv.ort abs/ 220//3388..0 07921
17

---

Published in Transactions on Machine Learning Research (11/2025)
Step/4: Calculae the total number of magazines elt after the deg's damage.
Siep/i Calculate the total number of magazines left after the dog's s damage.
lotal magazines total friday salurday magazhees sunday
total magazines total friday, salurday magazines sunday
Adding la match the number of magazines she originaly had
Adding 4 match the number of magazines she originlly had
total
magazines left total magazines damaged magazines magazines_ left
ttal magazines total magazines damaged magazines
Total
Ibbl. magazines
Figure 6: Credit assignment of part of a solution for GSM8K problem. Each token is colored corresponding to the DPO implicit reward as expressed in Eq. 4 (darker is higher). The left is the credit assignment of SCDPO, which correctly highlighted the error the number of damaged magazines (which is 4) should not be first added to and then extracted from "total magazines", while the credit assignment of DPO on the right fails to highlight it.
A Acknowledgements
This project is supported part by the Centre for Perceptual and Interactive Intelligence (CPIT) ltd under the Innovation and Technology Commission (ITCY's InnoHK, in part by National Key R&D Program of China Project 20222D0161100, and in part by NSFC-RGC Project N CUHKK498224 Hongsheng Li is PI of CPIT under InnoHK.
B Limitations
Our work contains the following limitations, and we leave them for future work. Firstly, our work is conducted purely linguistic models, which struggle to solve mathematical problems requiring an understanding of images. Secondly, due to the stepwise attribute of SCDPO, it is not very effective on solution formats consisting pure code. It only works on solutions consisting of natural language chain of thought or interleaved natural language and code. A method to properly enhance pure code solutions needs to be derived, which we leave for future work. Thirdly, as with all language models, our models can potentially generate hallucinations produce misleading solutions, which can have negative effect. Finally, while the data construction method distributionally narrow down the steps likely to be erroneous, it does not indicate the exact slep the error occurs, problem inherent with synthetic process supervision methods. Additionally, our work focuses on mathematical problem-solving, without discussing other reasoning tasks such as code gencration, thcorem proving, etc. We plan to explore them in future works.
C Credit Assignment Analysis Examples
In this section, we present several other credit assignment analysis examples, comparing SCDPO to DPO. Fig. 6 Fig. 7 and Fig. g show examples of part of the solutions of questions taken from GSM8K and MATH datasets, colored with the DPO implicit. reward of each token (darker is higher). As demonstrated in the examples, SCDPO is better than DPO at identifying the errors the reasoning steps.
Effect of SCDPO on Harder Reasoning Benchmarks
We test the Qwen2.5-1.55 1----.55--Innstucc and Qwen2.5-7B-Bi Insssrrruut models trained with SCDPO and naive DPO on AIME24 and AMC23, two harder, competition-leve reasoning benchmarks. As shown the table below, both AIME24 and AMC23, SCDPO notably improves the performance of the 1.5B and 7B models and demonstrates superior performance compared to naive DPO. This highlights the effectiveness of SCDPO on harder reasoning tasks.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Lef's denote the number of pencils Antonio has as
Let's derate the number pencils Antonio has as
According to the problem, Mitchell has x +|6 pencils
According the problem, Mitchell has x + pencis.
30 pencils, up equation:
penciis, set up equatian:
6 30
X XX 5) 30
Solving for
Solving
from sympy import symbols, Eq solve
from sympy imzort symbols, Eq. solve
Figure 7: Credit assignment of part solution for GSM8K problem. Each token colored corresponding to the DPO implicit reward expressed in Eq. (darker is higher). The left. is the credit assignment of SCDPO, which correctly highlighted the error Mitchell has 30 pencils, and Antonio has less pencils than
Michell, which is 30 6, so the introduction of is not needed, and + (x + 6) 30 is incorrect, while the credit assignment of DPO on the right fails to highlight it.
n.) an anthmetic series satisties the given congruance)
problem.
solve
that
an arthhmetic series satisties the given conirrencee
The arthmmetic series starts at and has common difference of The artthmetic series starts at and has common difference of last term) can written
5 ferm) 101] can writtan 1( 1), where Vlk 200]
where 20
Figure 8: Credit assignment of part of a solution for MATH problem. Each token is colored corresponding to the DPO implicit reward expressed in Eq. | (darker is higher). The left is the credit assignment of SCDPO, which correctly highlighted the error 101 cannot be written as 5k where k = 20, while the credit assignment of DPO on the right fails to highlight the error.
Model
AIME24
AMC23
Dwen2.5-1.3B-Innttmne
1/30
10/40
2/30
9/40
1/30
11/40
Qwen2.5-1.35 5-111.------stuuutSSSD-P
2/30
15/40 /40
wwen2.5-77B-Intrruct
3/30
20/10
5/30
19/10 10
4/30
22 /40
6/30
23 /40
Table Performance of Qwen2.5 models on AIME24 and AMC23 benchmarks.
19


Reviews:
[{"review_id": "nZao7pKJzt", "reviewer": "Reviewer_vdmo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Step-Controlled DPO (SCDPO), which aims to provide stepwise supervision compared to standard DPO. To generate negative training samples with stepwise supervision, SCDPO takes the prefix of a correct solution, generates new steps with higher temperatures, and takes those samples with wrong answers. When response pairs are generated in this manner, the training loss is only applied to tokens after the correct prefix. They evaluate their approach on multiple math datasets and SFT models, and demonstrate that their approach can improve over baseline DPO methods.\n\nStrengths:\n\n1. Adding stepwise training supervision is an important direction.\n\n2. The proposed negative sample generation method is simple and not very expensive.\n\nWeaknesses:\n\n1. The effectiveness of the approach is unclear, and the technical novelty is limited. Compared to baseline DPO methods, the performance improvement is minor on most datasets. Meanwhile, benchmarks used in this paper are generally simpler compared to the math benchmarks evaluated by latest reasoning models, such as AIME and other MathArena tasks. The authors should check whether their approach can improve on such more difficult tasks.\n\n2. All training experiments are done on Mistral-7B models, which are not the state-of-the-art opensource models and do not produce long reasoning traces. The authors should use recent opensource reasoning models as bases for their training experiments.\n\n3. Some experimental details and result analysis are unclear. See \"Requested Changes\" section for more details.", "claims_and_evidence_yn": "No", "claims_explanation": "1. As for experimental evidence, compared to baseline DPO methods, the performance improvement is minor on most datasets. Also, datasets used in this work are relatively simple compared to benchmarks evaluated by recent reasoning models.\n\n2. The expeimental setup is not very convincing. All training experiments are done on Mistral-7B models, which are not the state-of-the-art opensource models and do not produce long reasoning traces.", "audience_yn": "Yes", "audience_explanation": "Reasoning with process supervision is an important direction, so the topic of this work can be attractive for people who work on post-training for improving LLM reasoning.", "requested_changes": "1. Add results on harder reasoning benchmarks such as AIME.\n\n2. Add results using recent opensource reasoning models as base models for training experiments.\n\n3. In Table 5, explain why SCDPO performs better than Step-DPO. Is it because error localization is inaccurate for Step-DPO?\n\n4. In Table 6, please explain the meaning of each row. Specifically, given that there are 100 solutions in total, why the percentages can be floating numbers instead of all integers?\n\n5. Please clarify the main implementation differences between Step-DPO and SCDPO. Specifically, the paper notes that \"Step-DPO is in the Chain-of-Thought format\", but I assume that SCDPO is also trained in the chain-of-thought format? More discussion on the experimental setup would be helpful.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rUQp3Pr2rJ", "reviewer": "Reviewer_ub5Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Step Controlled DPO, a variant of DPO that uses step-based losses of preferred and dispreferred samples to improve step-by-step reasoning for Mathematical problems. The paper proposes a method to construct the data using a fine-tuned model. First, a prompt is sampled multiple times with a temperature of 1.0, and preferred (correct) and dispreferred (incorrect) samples to create a naive dataset. The with the preferred sample, they strip out some of the CoT traces till some step $t_k$, increase temperature gradually from 1.0 to 1.4 and generate step-controlled dispreferred samples. This is followed by the proposal of a step-controlled DPO loss, where all tokens after the $t_k$ step are used in the original DPO loss, leaving the first matching portions out of the loss function (which they capture via the original naive dataset + DPO loss). They train the model with a combination of both the losses. This is finally followed up by some quantitative and qualitative analysis of their method to highlight how their method differentiates high-quality preferred tokens from the original DPO loss. They showcase state of art results compared to other baselines such as DPO, DPO w/ equal amounts of data as SCDPO, Step-DPO and other previously trained math models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims made are mostly supported. \n\nClaim 1:  \"a method for automatically providing stepwise error supervision by creating negative samples\":  The authors show how to automatically generate data for stepwise error supervision using existing CoT traces. The one potential drawback to their method is that this will only work on tasks for which clear step-wise traces are possible, making this method only applicable in those settings. \n\nClaim 2: \"Qualitative analysis of the credit assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at identifying errors in mathematical solutions\": The authors use the implicit DPO reward criteria proposed in Rafailov[1] to showcase how traces from SC-DPO trained models are better at assigning higher rewards to more correct tokens and identifies incorrect tokens.\n\nClaim 3: \"ablations highlighting proposed method\": The authors show how their ascending temperature for data generation works better over one-shot temperature increase and also show how not restricting where to cut preferred traces to make SC-dispreferred traces is better. Finally, the implicit reward computed shows how the method handles incorrect tokens better than naive DPO.", "audience_yn": "Yes", "audience_explanation": "- The paper proposes a simple to adopt method to correct the CoT traces of a model using step-control\n- The loss function proposed is simple to implement and can be emprirically tested for a range of different tasks where piecewise CoT steps can be generated. \n- the paper shows both qualitative and quantitative analysis of the used method, highlighting how the method corrects reasoning traces.", "requested_changes": "- Page 7 last line, I think the authors meant to say `MathCoder` and not `Mathcoer` ? \n- Figure 7, the x-axis is very difficult to parse, would recommend trying to rotate the values by say ~45% using the plotting library for coherence.\n- The method uses older models for training and evaluation, which are no longer SOTA. (Mistral / InternLM) - making it difficult to understand how more recent state of art models would benefit from these methods. Does the existing RLHF tuning of those models further prevent adoption of the proposed SC-DPO method? Furthermore, the closed source models also used for comparison are very old. gpt-3.5 or even 4 are not SOTA anymore, making it difficult to judge the impact of the proposed method. I'd recommend using some state of art (non reasoning based) models and try comparing against higher quality models like gpt-4o etc.", "broader_impact_concerns": "- N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "b5igMAf4Dt", "reviewer": "Reviewer_2C6z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "__Summary__\n\nThe paper proposes Step-Controlled DPO (SCDPO), which induces divergence after a chosen step and applies a suffix-only DPO objective. It delivers competitive math-reasoning results with careful ablations.\n\n__Strengths__\n\n1. The method is fine-grained, controllable process supervision via suffix-only DPO.\n\n2. Automatic negative generation from an intermediate step, no external judges, cost-efficient.\n\n3. Consistent gains over SFT/DPO with strong absolute scores and clear ablations.\n\n4. Simple to implement; integrates seamlessly with standard DPO pipelines.\n\n\n__Weaknesses__\n\n1. No guarantee that the true first error occurs after the chosen step $k$; supervision can be misaligned.\n\n2. The method is somewhat similar to the idea of process-level supervision, although more fine-grained. \n\n3. The task is mainly math-centric, it is not clear if for code or other non-maths tasks it will still be applicable.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, experimental results clearly support the effectiveness of the method.", "audience_yn": "Yes", "audience_explanation": "Yes, math reasoning is a hot topic nowadays.", "requested_changes": "Please clearly compare and contrast with process-level supervision methods, to highlight the nolvety of this approach.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)