DAPO : Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization
Authors: Jiacai Liu
Chaojie Wang
Chris Yuhao Liu
Liang Zeng
Rui Yan
Yiwen Sun
Yang Liu
Venue: NeurIPS 2025 spotlight ()
Keywords: large language models
reasoning
alignment
multi-step RL
offline RL
Primary area: reinforcement_learning

Abstract: The role of reinforcement learning (RL) in enhancing the reasoning of large language models (LLMs) is becoming increasingly significant. Despite the success of RL in many scenarios, there are still many challenges in improving the reasoning of LLMs. One key challenge is the sparse reward, which introduces more training variance in policy optimization and makes it difficult to obtain a good estimation for value function in Actor-Critic (AC) methods.  To address these issues, we introduce Direct Advantage-Based Policy Optimization (DAPO), a novel step-level offline RL algorithm  with theoretical guarantees for enhancing the reasoning abilities of LLMs. Unlike response-level methods (such as DPO and GRPO) that the update directions of all reasoning steps are governed by the outcome reward uniformly, DAPO employs a critic function to provide step-level dense signals for policy optimization. Additionally, the actor and critic in DAPO are trained independently, ensuring that critic is a good estimation of true state value function and avoiding the co-training instability observed in standard AC methods. We train DAPO on mathematical and code problems and then evaluate its performance on multiple benchmarks. Our results show that DAPO can effectively enhance the mathematical and code capabilities on both SFT models and RL models, demonstrating the effectiveness of DAPO.

DAPO Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization
Jiacai Liu 1.2
Chaojie Wang 1.V
Chris Yuhao Liu!
Liang Zeng'
Rui Yan
Yiwen Sun?
Yang Liu²
Yahui Zhou
'Skwwork AI
FFudan University
{jiacai liu, chaojie eig.
Abstract
The role of reinforcement learning (RL) enhancing the reasoning of large lan- guage models (LLMs) is becoming increasingly significant. Despite the success of RL in many scenarios, there are still many challenges in improving the reasoning of LLMs.. One key challenge is the sparse reward, which introduces more training variance policy optimization and makes difficult to obtaín a good estimation for value function in Actor-Critic (AC) methods. To address these issues, we introduce Direct Advantage-Bast Based Policy Optimization (DAPO), novel step-level offline RL algorithm with theoretical guarantees for enhancing the reasoning abilities of LLMs. Unlike response-leve methods (such as DPO and GRPO) that the update directions of all reasoning steps governed by the outcome reward uniformly, DAPO employs a critic function to provide step-level dense signals for policy optimization. Additionally, the actor and critic in DAPO ensuring that critic is a good estimation of true state value function and avoiding the co-training instability observed in standard AC methods. We train DAPO on mathematical and code problems and then evaluate its performance on multiple benchmarks. Our results show that DAPO can effectively enhance the mathemati- cal and code capabilities on both SFT models and RL models, demonstrating the effectiveness of DAPO.
trained independently,
Introduction
In the rapidly evolving landscape of artificial intelligence, large language models (LLMs) have emerged as cornerstone of natural language processing (NLP) and beyond. These models, trained vast corpora of text data, have demonstrated
unprecedented ability to understand [16] R)
[3 533 and code generations 23 8). When the token generation process of LLM is modeled as a Markov Decision Process (MDP), it can be naturally optimized and aligned with human preference using reinforcement learning (RL) methods, known as Reinforcement Learning from Human Feedback (RLHF). Despite the success of RLHF in various fields [1]42 43
generate, and reasoning such as solving mathematical problems [6 48
46 6). it still encounters challenges and difficulties in the field of reasoning especially in long Chain-on of- -f---oo-Tto-- Thought (CoT) models. One of the key challenges is the sparsity of rewards 57]. When using LLMs for mathematical problem-solving and code generation, rewards are only assigned the terminal tokens. This implies that the intermediate tokens receives no direct reward, and the optimization direction relies solely on the backpropagation of the reward from the terminal token. Consequently, there are two issues in practical RI. training:
55
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Step A.1 Generation
Step B.1 : Compute A dvantages
Quryo: What the positive ends divssbloby?
Step
States - Concattpp
Coerpe
Ansinna 2AM6/10)
Mext
GENERATOR
D grm
Training Query Set
Qu-Visa))
State Concatiqua))
Joternadiatee Stegs x: 1ac% His numbers that and gi 19, 30/8
Adhantage A=Q
D DADY
Pretrained V
Step A.2: Completion
Step B.2 : Direct A Advantage Policy 0 Optimization
Halt double arswer tig
chack gran
... Fuull arewer 21
Ghallen
Stadee s Action
Optimizee
129
Adhaniage
Nixt Stip Geserration
COMPLETER
So our sumber can... Firal answer 50
DADY
LIM Policy
Te (als)
7
States
Prodict
MC
3
DAPO mún Bp
log
Trd (a(x)
CRITIC
Figure I: Direct Advantage-Basse Policy Optimization (DAPO). The whole training procedure of DAPO consists of two individual stages 1) Critic Training (left). Given the training query set Q for each query q E Q. DAPO uses the generator ggen to generate multiple rollouts from q and derive a dataset of training states denoted as Dgen- For each state in D gen- DAPO uses the completer (often Tert) to sample multiple sub-trajectory from 8 and collects the MC value as the estimation of the true value. Then critic network Va is trained to approximate the value function of the completer. 2) Policy Optimization (right). After extracting multiple next steps {a,);= of each state 8 from the completions in step A.I, DAPO then uses the trained critic to compute the advantages for all actions by vi E [m] : A, = Qi - + Σj= Q; and Q₂ is the predicted value Concat(s, a,). Finally, DAPO fits the policy ratio to the advantage to optimize the generation of reasoning steps.
High training variance induced by response-level algorithms. Response-level algorithms (such as DPO [42), GRPO 48 ReMax [30] and etc) treat the complete response as unified whole and the gradient directions across all reasoning steps within the response are governed by the final reward uniformly. Although response-
algorithms have recently achieved great success, such as Deepseek-R1 [1O], the sparsity of the reward indeed introduces more training variance and addressing the training variance introduced by it would lead to a larger performance improvement. See Section D for detailed discussions.
Bad value estimation in actor-critic methods. When applying standard actor-critic methods [5 [2] [2] to LLMs, due to the sparsity of the reward and vast generation policy, the critic often fails only learn good semantic representation but also provide the fine-grained credit assignment for policy optimization, leading to the poor performance of online actor-critic methods [25 62 such as PPO [46). Besides, these methods train the actor and critic simultaneously and the regression targets of both actor and critic are non non-stationary and change in response to the updates of the other during the training process. This interwoven update mechanism also leads an unstable training process that is prone to collapse (7 12] especially when the critic is not good approximation of the true value function.
In order to overcome these challenges, we introduce Direct Advantage-Band Based Policy Optimization (DAPO), new step-level offline actor-critic method designed enhance LLM's performance in reasoning tasks. In order to reduce the training variance induced by the sparsity of the reward, we use a critic function to provide dense training signals. To ensure that the critic function is well-trained, DAPO trains the critic individually before policy optimization to ensure it is a good approximation of the true state value function. The procedure of DAPO is visualized in Figure We summarize our contributions as follows:
We introduce DAPO, new step-level, offline RL method that learns from all generated samples. DAPO optimizes the generation of the reasoning steps by sampling multiple candidate steps for each intermediate step and updating the policy according to the

---

advantages of each candidate step, Our main theoretical result (see Theorem 3.2 shows that DAPO will produce better policy until the reference policy is already optimal.
We conduct extensive experiments to demonstrate the effectiveness of DAPO. DAPO im- proves the performance on multiple mathematical and coding benchmarks. Our results empirically shows that DAPO consistently outperforms both the response-level baseline GRPO and the actor-critic baseline PPO on multiple base models.
2 Preliminaries
In this section, we introduce the mathematical formulation of RL problem we studied and some important related theoretical results.
RL reasoning. The objective for RL reasoning can be formulated as
max Ezzua (By-a()lz) [r (ar, v)] - 3KK, (Te; =r))
(1)
T
where f is the distribution of training prompt z, y is the response sampled from policy T(|[2),
r(z, v) E {0, } is the binary reward for the correctness of the response, Tz, n are short for TT T-11z), (-(r) respectively, rer is the reference policy, 3 2 is the coefficient of KL regularization.
Equivalency to multi-step decision problem. We use the line break \ as the delimiter for reasoning steps. By doing so, response y can be divided into multiple reasoning steps, i.e. y (oo..., -r- where Vt 2 0a, is the reasoning step and is the total number steps. Thus one can easily show that the objective equivalent to following step-level decision problem:
max E
Betaa (-1oa)
Σ r (8t, 9KT (Te t=0
1. 180
where = Concat(x, do, 11) is the state (context prefix), the reasoning step a, is sampled from policy xT-1sx) (here we use π denote both step generation and response generation policy with a slightly abuse of the notation), Ta,: rf are short for (-118)), ( respectively and the reward function satisfies
r (8), ae) (x - T
L<T-1.
For any state S, we define
E Eytso (att)
Σ (s)) - 3KL (Toz, m:o [so 1=0
(3)
Thus the objective
is equivalent to finding a policy क that maximizes the state value function, i.c., max En-p [V (x)]
(4)
and v=== V as the state value function of
We denote v as the optimal policy of objective optimal policy. We define the state space S the set of all possible states that can be encountered in the response and the action space A as the set of all possible singular reasoning steps. For simplicity of analysis, we introduce the following assumptions this work.
Assumption 2.1. We assume there exists a constant H1 < Do such that for any policy and initial state so E S, the random decision horizon T satisfies T < Hi almost surely. We also assume that there exists constant Hg < oo such that for any singular reasoning step a E A, the token length of is greater than H2 so that the state space S and action space A are both finite.
More value functions. We define the KL-constrained state-action value function as Q3 (8,a) := (s,a) u) + V (s a),, where s o Concat(s, a). The advantage function AA is defined as
(5)
Aî (s, a) == Q7 (s, a) - VJ (s) Blog π (a|s)
(6)
ne,f (als)
We denote Q3 (s. a) == (s, a) +V² (s). For unregularized state value functions, i.e. 3 0, V Q AA instead of V/2" Q8 AB to be aligned with the standard RL literatures.
3

---

Bellman operators T defined as
For arbitrary policy π E II and any function V:8 - I, the Bellman operator
+V log (als)
Ys E E R (11())
Tref (a|s)
Performance difference lemma. Following is the key theoretical analysis tool used in this work. Lemma 2.2. For arbitrary two policies N, î E II and any distribution A(S).
V (p) V (p) Eg [3 (6) V3 [+)]
(8)
where V (p): Eguup
V (s) and the visitation measure d is defined as,
d (8) = o (-1os)
Σ
1 (st s3180
The proof of this lemma is deferred to the Appendix results about MDP on Appendix B
C
We also put some additional theoretical
Remark 2.1. Note that definition of visitation measure, dz is not standard the one in M
33
The reason that the planing horizon problem (2) i.e. T, is random while the constant T
7 333. Thus we not try to normalize d to probability distribution by dividing T directly.
3 Direct Advantage-Based Policy Optimization
In this section, we present the details of DAPO. We only consider the case where binary reward r(s, a) E {0, 1} is assigned in the terminal state since is the most common situation in the LLMs setting. We first present the objective of policy optimization with related theoretical results in Section assuming that one has access the true value function.
3.1 Policy objective
We first derive the policy optimization objective in one single iteration provided with the reference policy ret-- In the perspective of policy optimization, the most important quantity may be the advantage function. Recall Lemma 2.2 The performance difference for any policy II w.r. Ree on arbitrary p E (SS) is measured as
VJ (p) Visee (p) = Exndy
T Vi Riet (8)
Viteot (s)
where I, (H, Wret) can be regarded as the policy improvement at state 8. By the definition of Bellman operator T in (7) and that of function V in (3. one can easily show that
π (a(s)
Is (I,, Feet) = E (1))(11))) Teer (s, Blog
Troe (a|s)
A natural idea that ensures VJ (p) Vire (p) O might be to increase I. (π,, Tret) on many first focus how to increase I, (π, Tref) an
states possible. For ease of understanding, arbitrary state S. Considering parameterized policy Ta, the most directly and easily method may be do multi gradient ascent steps w.r.t I, (TO, Wret), i.e.,
K-1
WK et 0K = 8, + +n Σ VoTo (ToL, Tref);
(9)
k=0
where 77 is the step size. Following lemma gives equivalent form of VeIa (To Teet).

---

Lemma 3.1. we
we define the surrogate function
( (s, log To (als) 2
H.t (0)
E2 Ban-ao nop (11)
Tref (a ()
then Vols (To,, Trot) - H (0).
By Lemma 3.1 the update ( is equivalent
0k = nB
VoH (0x).
(10)
We now turn to the offline policy optimization problem setting for the sake of implementation simplicity and data efficiency. A natural question is that can we implement (TO) approximately in an offline dataset without sampling the on-policy rollouts at each iteration? Our answer is affirmative. We find that (which will be proved latter), just solving the problem
1 E (a) min 2
( A (8, log Tref (a|s)
To (a|s)
2
(11)
where ((A) can be any exploratory sampling distribution, can still yield policy that Ia (mo, Trof) > 0 unless Ther = T-.. Taking the consideration of optimizing multiple states si- multancously (weighted by a state sampling distribution v/s). the objective of DAPO is
( (s, log
To (als) 27
min L(0) = aaaovssovvallie w(1s))
(12)
Trel (als)
Remark 3.1. We give detailed discussion in Section D on how DAPO is specifically designed to improve multi-step reasoning abilities of LLMs, thereby distinguishing it from response-level optimization algorithms such as DPO 42), GRPO [48] and elc.
Notice that the state space S and the action space A are both finite under the assumption 2.1 Following theorem establish the validity of Loss and shows that the solution of (12) better policy w.r.t Tee objective (4).
Theorem 3.2 (Monotonic improvement). Suppose Vs E S.a E (s))(s) > 0, (alss) >
Teef [ner(a () O. Let π be the solution in 12). Then for any state E S, there exists func-
tion Ag- (A) >> [0, +00) such that
V8" (p) VT Vie (n) awdss Dh (vA)2
The equality holds if and only Riee T3.
Remark 3.2. The function X serves as a lower bound of the policy improvement I. (π+ Tyett in each states Since Theorem 3.2 is established for arbitrary exploratory VAA X should be function of U. For the case A (-|s) Tyef (-1s), one can show that
Xs (#rot) = 3 KT. (Tref (.18), n (-18))
by combining (20 and 21 the proof of Theorem 3.2 remains as a great interest for us derive more precise expression of X, and figure out which V should yield biggest lower bound.
Theorem 3.2 jimlies that DAPO can improve the performance until there is room for improvement in the trust region of Tee- Theorem 3.2 also establishes the monotonic improvement property on unregularized state value function. This can be verified by
V=l (n) = 3 (4) + smagt [KL (N (-1s), (-1s)) V (N) 2 Vinee (w) Zot (μ))
Implementation of DAPO. In practice, DAPO can be implemented by minimizing on an offline training dataset D {{Ss,, ai, A.) }.)3 where s; vs a; (A(-)s) and Á, is an estimation of true advantage ae (si, a.). The whole procedure can be found in Algorithm
5

---

Advantage estimation. Recall the definition of Q function in 5 and advantage function in ( For non-terminal states, one has (ss, a) 0 and
A Fiee (s, a) Fee (s, v (s) V (s a) (a [V (s o a')]
Thus in our implementation, sample multiple actions fas fa,)i= for each state from Tur(-Is) and estimate the advantage of each action by
M Σ (s
1 M
Vi E [m] A(ss a.) = Vo (so
(13)
Here Ve V pretrained critic function. Since 13 is fully determined by the critic Vg, thus the training of V plays a crucial role for the optimization of training states. In the following, we introduce the optimization method of Va.
3.2 Critic objective
Generally speaking, the critic optimization problem for any target policy π can be formulated as
min RDDD [C (Vo (+), V (s))),
(14)
here L is sample- wise loss function, D is the distribution of training states. However, since the target V**(s) (s) can not be accessed to in advance, we need to construct the estimate of V as the training samples for the optimization of V4. One can show that
T-1
v (8) =
Σ r (st, ar) 180 = (r (1(88-11) (ST-1, 0r-1 = 1/30 =).
Thus, one can firstly use a behavior policy Tgen (also called generator) to generate a set of training states Then for each training state s, one can use as the completer to sample N sub-trajectories
from 8, (s, (x)a6)) (9) (6)
a } and construct the empirical MC mean,
MCN (s)
1 N
Σ
(ST (1
as the estimation of V (s). which is guaranteed converge VT(() as n - oo by the strong law of large numbers. Notice that the state value (*(s) (s) is probability. Thus we the binary cross-entropy loss our implementation, i.e.,
LYCF (y, Uprea)
(y (og()))a))
y) log(1 Upred))
as the sample-wise loss function L in 14 to avoid gradient diminishing if a MSE loss is applied (which is more common in RL literatures). Finally, the critic optimization objective is
min E,DD [LBCE (MCN (s) Vo (s))
(15)
Remark 3.3 (Should treat Vo PRM?). Its worthy note that the critic training method presented above mostly aligned with (56). However, we argue that the Vş. should be treated as value function other than process reward model. Here are the reasons 1) From the optimization perspective, Vg is trained to approximate the true state value of the completer. 2) From the RL perspective, using V& reward function will deviate from the original RLHF optimization objective and easily lead to a serve reward hacking issue on the number of reasoning steps as shown [13].
4 Experiments
In this section, we conduct DAPO on mathematical and coding datasets individually. The experimental setup is detailed in Section 4.1 and specific conclusions and benchmark results of our experiments
presented in Section 4.2

---

Model
Domain MATH
Out
Domain Olympiad Bench
GSMSK
Minerva MATH
College Math
SFT Models
Myywork-Math-LJama
41.90
61.49
5.51
18.68
24.85
+ DAPO
46.88
67,55 76.06
7,34+1.82
22.38
25.87
Llama-3.1 1.1m--3118B--44 8----nsttrutt + DAPO
49.42
85.29
26.48
16.14
30,91
53.62 +4.20
86.74+1.45
23.54 2.94
20.01 43..88
30.91 +1.62
OpenO1-Llamaa ----mma--B---
52.73
85,99
29.04
19.86
29.10
DAPO
60.33 +7.00
88.77 +2.78
29.42 +0.39
24.44 +44.59
32.12 +-3.02
Qwen2..5-72Biing -iniistuutt
82.90
95.40
46,30
45.45
43.00
+ DAPO
84.70
95.70
50.00 +3370
47.70 +22.5
43.40
RL Models
Qwen2 Math-7B Bath-7B--nsttret Instruct
74.46
89.38
40.07
34.36
41.59
+ DAPO
75.41 +0.95
89.45+0.07
37.51 -2.56
37.03 +2.07
42.23 +0.04
Skywork O1 1i--Opn--llmaa,. Llama3, 1-8B
78.
91.64
26.10
43.11
40,40
DAPO
79.81
11.71
92.81
12:17
1.007
1..65
40.26 0.14
Qwen2.5 Math-7B Instruct
83.42
95.78
40.06
38,96
42.65
+ DAPO
84,86 11:44
96.14
41.56 1.50
41.25 2.229
41.97 0.67
Table 1: Performance of models optimized via DAPO on mathematical benchmarks. We use zero-shot prompting and greedy decoding for all evaluations in the table. Across a wide range of model families and out-of-domain math word problem benchmarks, DAPO achieves steady improvement with only 7.5K training samples from MATH 18
SFT Model
Out of Domain MBPP
HumanEval
HvmanEval+
MBPP+
LiveCodeBench
Llama-3. 1.1.a-...-B-BBBB 1-8B-Inntruet
72.0
66.5
72.0 77.0+5.0
56,9
18.8
DAPO
75.0
68.9+2.4
66.1
20.9
BrenOl-Llama-SB.ae
69.5
61.0
69.8
58.7
16.1
+ DAPO
72.0 +2.5
64.6
+3.6
75.9 +6.1
63.8
+5.1
13.7 -2.4
Table 2: Performance of models optimized via DAPO code benchmarks. We use zero-shot prompting and greedy decoding for all evaluations in the table. DAPO consistently improves performance when starting with both regular and reasoning LLMs, even when only 4K training samples from TACO [29] is used.
4.1 Experimental setup
Training dataset. For all the DAPO experiments on mathematics, we utilize only the 7500 training problems from the dataset MATH [18] to generate the advantage datasets for DAPO training, requiring no additional human annotations. In particular, we only use the questions and the corresponding golden answers in the dataset while the provided solutions
not used for training. For coding experiments, For coding experiments, we subsample the TACO (2) dataset to compile approximately 4,000 competition-leve programming questions derived from real-world scenarios. We utilize its provided unit test cases to evaluate whether a solution is accurate.
Base models. Considering the reproducibility and of DAPO, our experiments are taken over of several open-source language models including general and math-specific models. For general models, we consider Llama-3.1 1--mma-.1--B-IInttttt 1-8B-Instruct [52] BeeeO1--llama-B---e (39), Skywork-O1-Open 101-Opp--Lllmal -lama3. 1-
8B [40], Qwen2.5-722 101.55-77-IInntree
511]
For math-specific models, we consider Seywork-Math-Line
[60]. Notice that Skywork-OI: Open-Llama3.1 18B, Qwen2-Math-25 7 and Qwen2.5-Math, 1aann-5-Matt-7---nnttee In-Inruttt are already trained by
[63]. Swwen2-Math-7B-Inaae
[59] and -aae..--aaa-7-----me
RL algorithms 48). Thus we conduct continue-RL training three models by DAPO.
Benchmarks & Metrics. For mathematical experiments, we evaluate performance on English mathematical benchmarks. In addition to the 5000 test problems from MATH [18, We also add 4 out-of-domain benchmarks, GSM8K 99, Minerva Math (28), Olympiad Bench [17 and College Math [50] to test the performance generalization of DAPO. For coding experiments, we evaluate DAPO on several widely-used benchmarks in code generation, i.e., HumanEyal [7]. HumanFxal+ 22. MBPP
4 MBPP+ [5) and LiveCodeBench (21. All the evaluations are conducted in a zero-shot greedy
7

---

Model
Base
+GRPO
+PPO
+DAPO
Llama-3.1-8B-Instuu
49.42
52,28+2.86
52.41+2.99
53.62 +4.20
OpenO1-Llama-88--ve
52.73
55.63 +2.90
54.12+1.39
60.33 +7.60
Qwen2-Math--7--nsstuee
74.46
74.94+0.46
74.93+0.47
75.41 +0.95
Qwen2.5-Matth-7B-nnsmr 83.42
84.33+0.91
83.76+0.31
84.86 +1.44
Table 3: Comparison with baseline methods on the 5K test samples from MATH (18.. DAPO
consistently outperforms both the initial model and model optimized with GRPO and PPO.
sampling (i.e. temperature 0) with cot prompt template and a maximum amount of 2048 newly generated tokens (except 4096 for OpenO1-Llama-88 -----.------.-11111 and Skywork-O1 Oaaer-OOO-Ope--ee 01--000-------.a......
DAPO implementations. For each DAPO experiment, first the base model to generate 32 solutions for each training problem and then 6 solutions are selected out of them while making the correct solutions and wrong solutions as balanced possible. For each selected solution, use one line break n to segment the solution steps and use the base model as the completer to do 16 completions for each reasoning step. Then the MC estimations training. The training methodology of critic is presented in Section 3.2 In our implementation, the critics are fine-tuned for one epoch on Qwen2.5-Math -aan----Maa--7-IIstrru for mathematical experiments and
constructed for the critic
for coding experiments, with learning rate of 5e-6 and batch size 512. After that, we extract the next steps (action) for each intermediate step (state) in the completion datasets and compute the advantage using the critic for DAPO training. For all experiments, we use the global batch size 2048, learning rate 5e-7 and 3 0.01 or 0.02. We also summarize some useful tricks applied in all of the DAPO training in Appendix FT
1...--oddr-7--nnnn
Baseline methods. We compare the performance of DAPO with PPO as the actor-critic method baseline and GRPO as the response-level method baseline on mathematical tasks using the 7.5K problems from MATH [18] as the training queries. We conduct experiments on four base models -0. 1,wen22-ath-77 -.-maa--B-.....00002-Maat----- -Instrrtutt and Qwen2.5-Mathh Instruct. We train PPO and GRPO until the training accuracy curve converges. The more training details of our baseline methods are in Appendix F2
4.2 Main results
Math. Our results, presented in Table Table 3 and Table 4.2 demonstrate that DAPO consistently enhances the performance of the base model across all tested models on the in-domain benchmark Llama-3.1 1----..11-8----rreee OpenO1-Llanm achieve 50.54%, 53.62%, 60.27%, 84.55% greedy decoding 1-8B
MATH [88, through DAPO training, Skywork-Math-LAn 1B-v0.1, and 2wwen.55-72B-insstte
accuracy respectively. For RL models, Qwen2-Math Bn-tIBtttunttttSS Salywor--1--pee-eee and Qwcn2.5-Math-75 In-srtruct achieve 76.40%, 79.81 79.81%,and ,,nnd 84.86% greedy decoding accuracy respectively, indicating that DAPO can further enhance the performance even the base model are already trained by RL methods previously. Table 3able shows that DAPO has a greater improvement than PPO and GRPO various base models in our experiments. In Figure 3 we present the accuracy curve on the MATH TEST during the training process of DAPO. As can be clearly observed, DAPÓ steadily enhances the accuracy throughout the training until it stabilizes.
Code generation. Our results for the code generation task are shown in Table 2 DAPO improves the on multiple wildly DAPO has increased by 3.0% and 2.4% respectively on HumanEval 7 and HumanEval+ [22). 1. by 5.0% and 9.2% respectively on MBPP 4 and MBPP+ (5), and by 2. % on LiveCodeBench 21 For OpenO1-Llamee Baanna--B--Ol 8B-v0.1, DAPO has increased by 2.5% and 3.6% respectively on HvmanEval 7 and HvmanEval+ [2 by 6.1% and 5.0% respectively on
coding performance of both ---aaa-..--8--Inssse 1-8-- and BpenO1--lama-8B--V used benchmarks. For ---ma-3.1-8B-Instteee
MBPP 4 and MBPP+ [5]. and decreased by 2.4% on LiveCodeBench 21 Overall, these results strongly demonstrate the effectiveness of DAPO.
On the iterative DAPO. As suggested by our theoretical results (Theorem 3.2). there still remains the room for performance improvement if Tref is not the optimal policy. Thus we conduct two iterative and Qwen2-Math - At iteration 2, we use
DAPO experiments on Skywork-Math-LJ. the model optimized by DAPO at iteration 1 as Nee. Our experiment results are summarized in Table

---

4.2 Table 10 We can conclude from both tables that DAPO can further improves the performance
of Skywork Skywork-Math- Laath-lama&a and Ma----aahh---ttteee
Model
Base
+DAPO iterl
+DAPO iter2
Skywork-Math-Lie
41.90
46.88
46.88
+4.98
50,54 +8.64
Owen2-Math-77B BB-Instuuct
74.46
75.41
+0.95
76.40
Table Performance of iterative DAPO on 5K test samples from MATH [8 Both Skywork-
Llama and Qwen2-Math-7B -B-nstructtt show score improvement with additional iterations.
Ablation studies of hyperparameters. We conduct some preliminary ablation studies the different components of DAPO. For the choice of KL coefficient 3. we test different / in the experiments Skywork-Mathh-Lian
and the results reported in Table 4.2 It can be seen that suitable choice of 3 yields best improvement. We find 3 = 0.01 generally performs well on various models. Regarding the number of completions used for MC value estimation, we
and 16 in
Model \\a
Base
8 0.002
==
0.02
8 0.05
B 0.1
Skywork-Math-Llama
41.90
44.52
46.88
46.70
45.56
44.50
Table 5: Performance of DAPO on 5K test samples from MATH
18 with different
the experiments on Qwen2- Matt2-MathhBB-- Maath-7B-Instruc Instruct and the results are reported in Table 4.2 We find that the difference not significant between n 8 and 16 in our experiment. However, make sure a larger performance gain as possible, we use n = 16 in all our experiments.
Model \
n
Base
72
12 = 16
Qwen2-Math- Baaahh--BInntruct
74.46
75.30
75.41
Table 6: Performance of DAPO on 5K test samples from MATH [88 with different number of completions for MC value estimation.
On the computation cost of DAPO. One major limitation that may hinder the efficient training-time scaling of DAPO is the high computation cost, particularly during the critic pre-training stage. We leave it as future work for us to find a more efficient and scalable method for critic training. Please refer Section E for more discussions. That being said, given similar computation resources budget to the response-level baseline method, i.c. GRPO, we find that DAPO still outperforms GRPO in our experiment. We run GRPO experiment Meta-Llama-3. 1.taaaa..1---0 1-8BB-Instruu to more than 1000 training steps. The training and test curves can found in Figure can observed that the model exhibits serve overfitting after around 300 steps and the test performance begins drop quickly. In contrast, as shown Figure 3] DAPO improves the test performance steadily without exhibiting overfitting, ultimately achieving better performance gains compared to GRPO. This validates the effectiveness of granular policy optimization enabled by estimating step-level advantages.
Conclusion
In this work, we propose an offline step-level RLHF method called Direct Advantage-Base Policy Optimization (DAPO), which aims to optimize the generation of reasoning steps. DAPO significantly improves performance on both mathematical and coding benchmarks, demonstrating effectiveness. Compared with response-leve algorithms, DAPO leverages the critic function for more fine ane-grained policy optimization. Compared with standard actor-critic methods, DAPO separates the training of the actor and critic into two distinct stages, stabilizing the RL training process while obtaining good value function estimation. As we discussed before, the main limitation of DAPO is the high computation cost, indicating need to find more efficient implementation method. Besides, it also remains a great interest to see whether using larger set of bigh-quality training queries would yield greater performance improvements.

---

References
[1] Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1 2021.
[2] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402. 14740, 2024.
[3] Hessa Abdulrahman Alawwad, Areej Alhothali, Usman Naseem, Ali Alkhathlan, and Amani Jamal. Enhancing texthook question answering task with large language models and retrieval augmented. arXiv:2442.055128 2024.
[4] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and et al. Program synthesis with large language models. arXiv:2108.0773: 2021.
[5] Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and et al. Program synthesis with large language models. urXiv:2108.07732 2021.
[6]
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kermion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield- Doddss, Danny Hemnandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05880 2022.
[7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021.
[8] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. ArXiv, abs/2304.05 02800 2023.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110. 146.6688 2021.
[10] AA, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi. Xiaokang Zhang. Xingkai Yu, Yu Wu, Z.F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li. J.L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen. Qiushi Du Ruiqi Ge, Ruisong Zhang, Ruizhe Pan. Runji Wang. R.J. Chen, R.L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu. Shunfeng Zhou, Shuting Pan. S.S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W.L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X.Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y.K. Li, Y.Q. Wang, Y.X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong,
10

---

Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y.X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z.Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability Ilms via reinforcement learning, urXiv:2501.12445 2025.
[11] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv: 2304.0677, 2023.
[12] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause, editors. Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmüssann Stockholm, Sweden, July 15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1592-1591. PMLR, 2018.
[13] Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. 1rXiv:2410.15115, 2024.
[14] Chengxi Li Guoxin Chen, Minpeng Liao and Kai Fan. Step-level value preference optimization for mathematical reasoning. arXiv:2406. 10858, 2024.
[15] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmmsssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865 PMLR, 2018
[16] Ji-Eun Han, Jun-Seok Koh, Hyeon- Tae Seo, Du-Seong Chang, and Kyung-Ah Sohn. Psydial: Personality-basee synthetic dialogue generation using large language models. arXiv:2404.00930, 2024.
[17] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv: 44008, 2024.
[18] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
[19] Jian Hu, Reinforce++: A simple and efficient approach for aligning large language models. vrxiv:2501.03262v1 2025.
[20] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasooerr-zero: An open source approach to scaling up reinforcement learning the base model. arXiv:2503.244294 2025.
[21] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando LezamaLezama, Koushik Sen,, and lon Stoica. Livecodebench: Holistic and contami- nation free evaluation large language models for code. arXiv:2403.0797. 2024.
[22] Yuyao Wang Jiawei Liu, Chunqiu Steven Xia and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. 2rX1v:2305.01220, 2023.
11

---

[23] Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao. Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv:2310.06770, 2023.
[24] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2503.0149) 2025.
[25] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
[26] Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang. Xiangru Peng, and Jiaya Jia. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. wxXi:2246.188629 2024.
[27] Deep Reinforcement Learning and the Deadly Triad. Hado van hasselt, yotam doron, florian strub, matteo hessel, nicolas sonnerat and joseph modayil. arXiv: 026448, 2018.
[28] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-388 3857, 2022.
[29] Rongao Li, Jie Fu, Bo-Wen Zhang. Tao Huang. Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv:2312.14852 2023.
[30] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference on Machine Learning, 2023.
[31] Weibin Liao, Xu Chu, and Yasha Wang. Tpo: Aligning large language models with multi-branch multi-step preference trees. wxXiv2240..1285.4 2024.
[32] Guanlin Liu, Kaixuan Ji, Renjie Zheng, Zheng Wu, Chen Dun, Quanquan Gu, and Lin Yan. En- hancing multi-step reasoning abilities of language models through direct q-function optimization. arXiv preprint arXiv:2410.09302, 2024.
[33] Jiacai Liu, Wenye Li, Dachao Lin, Ke Wei, and Zhihua Zhang. On the convergence of projected policy gradient for any constant step sizes. 1rXiv:2311.01104. 2024.
[34] Jiacai Liu, Wenye Li, and Ke Wei. Elementary analysis of policy gradient methods. arxiv:2404.03372, 2024.
[35] Yuliang Liu, Junjie Lu, Zhaoling Chen, Chaofeng Qu, Jason Klein Liu, Chonghan Liu, Zefan Cai, Yunhui Xia, Li Zhao, Jiang Bian, Chuheng Zhang, Wei Shen, and Zhouhan Lin. Adaptivestep: Automatically dividing reasoning step through model confidence. arXiv:2406.06552. 2025.
[36] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. arXiv:2503.207831 2025.
[37] Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Sstep-controlled dpo: Leveraging stepwise error for enhanced mathematical reasoning. arXiv:2407.00788, 2024.
[38] Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision. arXivv24006006592, 2025.
[39] O1-OPEN. Open-source ol. https://huggingfacc.ccm http://...........ooooooo/ooooco OPEN 2025. 01-30.
Accessed: 2025-
[40] Skywork ol Team. Skywork-ol open series. ber 2024.
Novem-
12

---

[41] Zehan Qi, Xiao Liu, lat Long long, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. We- brl: Training LIM web agents via self-evolving online curriculum reinforcement learning. vrXiv:2411.02333 2024.
[42] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[43] Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Ghesh- laghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, and Bilal Piot. Offline regularised reinforcement learning for large language models alignment. urXiv:2405: 2405//9705//9077 2024.
[44] Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4341 13500. PMLR, 2018.
[45] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arxiv:1506.000 1506.02438, 2020.
[46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. (TXXi:1707.00347,000 2017.
[47] Amrith Sethur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. myXiv:2410.08144 2024.
[48] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Haowei Zhang Xiao Bi, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. WxXiv:2402.03300 2024.
[49] Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, pages 1057-1063, 1999.
[50] Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21, 2024. OpenReview.net, 2024.
[51] Qwen Team. Qwen2.5: A party of foundation models, September 2024.
[52] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and al. Qwen2.5-matt technical report: Toward mathematical expert model via self-improvement. arXiv:2302. 13971, 2023.
[53] Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie Hc, Shuicheng Yan, and Bo An. Q*: Improving multi-step reasoning for Ilms with deliberative planning. arXiv:2406.142: 2024.
[54] Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. Offline reinforcement learning for Ilm multi-step reasoning. 4rX1v:2412.16145. 2024.
[55] Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
13

---

[56] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shephert Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 9426-9439, 2024.
[57] Albert Wilcox, Ashwin Balakrishna, Jules Dedieu, Wyame Benslimane, Daniel Brown, and Ken Goldberg. Monte carlo augmented actor-critic for sparse reward deep reinforcement learning from suboptimal demonstrations. Advances in neural information processing systems, 55:2254-2267,25 2022.
[58] Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P. Lillicrap. Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv:2405.0045/, 2024.
[59] An Yang, Baosong Yang, Binyuan Hui, Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
[60] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, and et al. 1wwn2..5-mate technical report: Toward mathematical expert model via self self-improvemennt arXiv:2409.12122 2024.
[61] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
[62] Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret. arXiv:2503.01491, 2025.
[63] Liang Zeng, Liangjun Zhong, Liang Zhao, Tianwen Wei, Liu Yang. Jujie He, Cheng Cheng, Rui Hu, Yang Liu, Shuicheng Yan, Han Fang, and Yahui Zhou. Skywork-math: Data scaling laws for mathematical reasoning in large language models the story goes on. arXiv:2407.08345
2024.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatinn right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ) In general, answering "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist".
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims match our theoretical results (see Theorem 3.2) in Section 3 the
experimental results in Section 4 and the discussion Section D
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification: We discuss the main limitation, i.c. high computational cost, in Section where we also discuss the potential approaches address this issue.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The assumptions are stated in Section 2 and all proof are included in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: we provide detailed setting of our experiments in Section 4 and Section The main code of our methods is included in our supplementary material.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes)
Justification: We use the open-source training dataset and the main code of our methods included in our supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ phiips.cc//uulic//uuu
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
17

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed experimental setting in Section and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our evaluate our experimental results with greedy decoding, i.e. using temper- ature Thus there is no statistic random error in our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide sufficient information on the computation cost in Section
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our research confirms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
It is difficult to discuss the potential societal impacts of our paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
19

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited all used datasets in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
20

---

Answer: [NA]
Justification:
Our paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Our method only involves standard usage of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standaandard components.
Please refer to LLM policy (https://neur)) for what should or should not be described.
ct/ http:///.....o..eeeeeeee////2/20
21

---

A Related work
By Theorem
B.2
the the optimal policy 7i has the analytic solution given by
b (vlx)
Teof (u/z) B (r(2:)) Vret (zz)
Vx, y log
(16)
where
exp 3 (z, w/) }}}
Vref (x) == log By
In order to solve T.g DPO 42 use the Bradley-Terry model to parametrize the reward model r and
infers the optimal policy directly from pair data without accessing to Vier. DRO 45 takes straight way and solves the optimal policy directly by
[(Dg wo (ylor) Trof (y/a²)
(r (x, Viot (x) (o))) 21
min Ex.9
one-step However, when it comes to the multi-step RLHF problem 4 the decision horizon T is generally larger than one. In this case, by Theorem B.2 the optimal policy T3 satisfies
(a|s) Frof (als8) ( (r(,,))+ (6) V3 (s))
log
(17)
where is the current state, a is the action, s' f(s a) is the successive state and V is the optimal value function of T defined in (3) that can be accessed in advance. Several policy gradient ascent [49] based methods are applied to solve the optimal policy. PPO use GAE [45) to estimate the policy gradient and a clip trick to keep the training process stable. Considering the computation efficiency,
(30 remove the critic component and use the sampling rewards to adapt the DPO algorithm the step-level setting, developing a series of methods known as step-DPO. These methods also uses the BT model to parametrize the reward model and infer the optimal policy from (17) without the knowledge of V However, these methods still requires the collection of massive pairwise trajectory data. DQO 32 takes the standard actor-critic route like PPO and solves the optimal policy using the Soft Actor-Critic (SAC) Method. WebRL [41] directly solves equation (1)) by using MSE loss and replacing V* with the value function of some known behavior policy OREO [54]
GRPO 48), RLOO 12]. Remax
estimate the gradient. 26 37
also proposes an offline
actor-critic methods for LLM multi-step reasoning. (58) and TPO 31 uses MCTS and implicit
reward respectively to provide dense signals for policy optimization.
47 use the advantage function
of an additional prover policy μ as a PRM in the online policy gradient. In the below, we give
holistic comparison between DAPO and other methods in Table
Method
DPO
GRPO
PPO
DQO
OREO
588
DAPO
Step-wise optimization
X
Step signal provider
NA
NA
critic
critic
critic
MCTS
critic V
Learn from listwise samples
V
V
V
interwoven update of actor-critic
NA
NA
NA
v
Table 7: Comparison between DAPO and other methods
A significant difference between DAPO and other actor-critic methods is that DAPO decouples the training of the actor and critic into two distinct stages. This separation offers the following advantages during the actor training stage: Enhanced training stability. The non-stationary critic along with the interwoven update between actor and critie bring more instability to the training process. DAPO address this issue by removing the critic in the actor training stage. Substantially reduced memory usage. During the actor training stage, DAPO only loads the actor model and the advantage dataset, leading to significantly lower GPU memory consumption compared to PPO, DQO, and OREO.
22

---

B
More on MDP
Bellman Optimality Operators
We define the Bellman optimality operator TB as
Vs E S (s) max
[TV] (s)
Rell
Lemma B.I. Given any function : IR,
{ { (r
ss E S (s) = Blog (1
exp
(18)
Optimal policy the following we give the optimality conditions of the multi-step RL. Theorem B.2 ( a policy II satisfies
vs E S V (s) (T)
then = z... Besides, the optimal policy F has the unique analytic form of
3 (als)
Vs E A.E.SEEAAl
(Q% (8, - V (s)
Reef (a|s) 3
C
Proofs
C.1 Proof of Lemma B.1
By the definition of TB. for any state s E S, we define
(aa) Tee (als)
T (-1s) arg max Easp VV Blog PEA(A)
One can easily find that
Neef (als) exp 18 (s, v(((s,))))
Vs E S, (als)
Eae
exp { +
and we omit the proof of it here. Then
TaV (8) -Tg (s)
r (s. a) (((s,a) - 3 log Ty (als)
E Baway (-18)
Trer (als)
exp (r +
(
(8, (V (((s,a) Blog
exp ( 1 (r +V((s,)))))))
Eai 8)
exp (r(s. (((6.))++((+..)1))
Bogg Eantrct(is))
C.2 Proof of Lemma 2.2
Proof. Recall the definition of state value functions in Then has
T-1
Vī (p) Vž (p)
E
Σ
(r (st, ar)
3KL (क (-1st), Teet (-1s:))) ~ - -V3 (p)
23

---

(
(s( - 3KL (क (-1st), Tref (-1+)))) 30 ~ p
(V3 (5t) V3 (sc+1)) =p
(r (s), a.) 3KL (1t (-1s2) Tref (+1s²) + V (sc+1) V3 (si)) P p)
(11)
(
(T VØ (5)) Vă (st))lso P
Note that under assumption 2.1
the state space S is finite. Thus
T-1 V (st) - Vz (si)
V3 (p) V (p) (a
E1ss I1(s==s) 180 ~ P
(yIeNa
Σ
TT (st) -V V3 (st) = st|s0 ~p
868
T-1
(yseNaa
Σ
TT V (s) V33 (s)) = =s||so ~p
LES t=
T-1
(xIeNaa
Σ
V (s) V3 (s)
Σ
I(s s3|800 P
LES
(T B (8) V3 (s)) (xteNoa
I (s: = 831800
#ES
(a)
p (8) (T V (8) V (x)
#ES
E Saddy
[ V (s) 3 (s)
where (a) leverages the definition visitation measure dz
C.3 Proof of Lemma 3.1
Proof. Recall that
1
( A (s) - llog EanTro (16)
(als) 27
H (mo, Tret)
Nee (als)
A direct computation yields that
3 A Foat (s,a) log (als)
1
2
VoHth (To, Tret)
E Banso, (-1x)
Vo
Jree (a|s)
[ Tret (s, - -log no (als)
no (als)
-EENTO (11))
V
A
Fret (s,
log Fref (a|s)
Jeel (a)ss)
3
-EE (1s)
[ Feet (s, log
To (als)
log To (als)
n neef (a|s)
Note that
Tg (a|s)
I, (NO, Tret) = Banso(1)))
A (s, a) - Blog
ne (a s)
24

---

Then
A (s, 33 log 2 (a
9Ig (TO; Tref)
Vo VoBoroo()
Tref (a|s)
(als)
(aw
Volog no (als)
A (s, a)
log
Traf (a|s)
log (alss Tref (a|ss)
Va Arte (s,
na (a|s)
V Vglog log To (als)
( (s,a) log
Trof (al 8)
(
V log (a|s)]
1 A Tha (s, - loog
ле (als)
B (B
Vo log To (als)
Fref (als)
where (a) is due to
(1- Vo log Ta (a|s)] = Vo Volanma [1]= Vo -1) =0
Thus it's easy to verify that VoIg (Toz. Tret) o H (ne, Fret) and the proof is completed.
C.4 Proof of Theorem 3.2
Proof. Notice that Vs Ssss(s) O. Then it's obviously that for any state S,
[ TT (8,
log (a) 27
m (-18) arg DEA(A)
aarginn min E Barea(le)
Tret (a|s)
Since Ys E S (als( > Tmrlals) 0, one has Vs E S, E A, (als) > 0 otherwise the loss becomes infinity. For ease of notation, we define
p (a)
1 Aoa (s,
Ugaa = log
and A = 8
Tref (als)
Then {
(15 log
# (a|s)
A is the solution of the following optimization problem:
L. (us) 1 Eanuu())s)) [A us,a)²
min (. ERIAI
Σp(a)= p(aa= = Ea lexp (uaatl 1.
s.t
aEA
Denote the Lagrangian function as
L. (Ua, A) L. (us) + X (Banzne( |exp {uaaJl -1).
Since it's convex problem, by KKT conditions, there exists a Xラ IR such that (ut A?) is the solution of the equations:
L. (ut 2;)= =0 Bawmoa (exp (utaal =
(19)
Notice that
1 (Ar, unaa)² +
Tree (als) (als)
V u. Les (Ws, A) Vn, Baves(-)))
exp (u
2
Thef (-1s)
=VA (+|s) 2
Ag + X
exp {us}
VAA (+1s)
25

---

Plugging it into 19 yields that
Fzef (als)
Va E A: A
u3aa = 22
(a|s) exp {uta]
(20)
A direct computation yields that
x Bawweer
eexp { [ugaa]
(a) Eawes())))
1 ref (als)
exp (utaa
v (a|s)
= -aawa(l1s)
As,a Uz,a]
where (a) due to (20). We now to show that 22 O. Notice that
1
x (als)
E (a [As.a uxtal
A Tee (als)
log
3
Tref (a|s)
Tref (als)
log
x (als)
KL (Tref (-|s),
(-1s)
(21)
Thus X; < 0, one has Va A: Asa uta < due to (20) and
Bawxie [Ac Ug O,
which contradicts to 21 Recall the definition of (T, neet). Thus
+ (als)
I. (Tt Teet)
= (awor((1e))
A (s, -33 -3log Nret (a|s)
B Barx (+1s)
Assa - -
(a) B Euro (-1a) (-16)
\ Tref (als)
eep { {u
MA (a|s)
π (a|s)
B Xr East I (+1s)
VA (als)
π (als)
B >3 (-aosss)ss)
VA (als)
+ (als)
B X2 (anusst)))
VAA (als)
2 B X
(v(A))
(22)
where the last inequality is due to
2 -\> - (als) AaEAA
n (als)
E (als)
Basea((s))
=
V/A (a]s
VA (a
Plugging 22 into Lemmal 2.2 directly yields that
(n) Visee (n)) = Fuzt [T, (I,, Tret)) 2 de+ ()s (vA)) 20.
We now to show that V (4) = Viver (μ) holds if and only if mee is already the optimal policy. Notice that
(n)= Toof (4) vs V Ao (MA) = 0 vs S A: uta As.a = 0.
Thus plugging into
21 yields that
KL (Tre (t1s), (-1s)
26

---

and As,a E A: Aa (s, B log
(als)
Hence for any state E S.
[aar(a|s)
To Va (s) B log (a
exp
r (s, a) V (f (s(
Blog (aw
exp
Ted (s,a) a)
B log (a
exp
Tee (s,
1
1
B log (a
exp
Tor (s,
/= (8)
3
B log (aw
exp
V (8)
Viel (s)
V (s)
which means Vira is the fixed point of the Bellman optimality operator thus neef is the optimal policy by Theorem B.2
D Discussion I: connection between DAPO and multi-step reasoning.
In this section, we give detailed discussion on the connections between DAPO and multi-step reasoning and show that DAPO indeed achieves step-level fine-grained policy optimization and reduces the training variance.
Response-level policy gradient methods suffer from high training variance. (PG) methods updates the parameter by applying gradient ascent w.r.t the objective 4 directly and generally takes the form of
Policy gradient
0 - + VoV (p).
For simplicity of analysis, assume 3 O. By policy gradient theorem [.].
TH (w) EE Beter
Σ Ve log no (aelse) TT (st, at)
(23)
Current response-level PG methods mainly focus the estimation of advantage in 23 Sup- pose n responses .1....; Yn are sampled from TT(-1z) for each prompt æ. We denote y/ (i) (
Concat( x, a aT. -.-1),, st (6) Concat(æ, (1) (!..,,(")) where T; is the number of reasoning
steps in response yt. and A!" the estimation of advantage AT (s) (i) a(()). We summarize the
construction A of different methods in Table can be observed that all methods use r(æ, y:)
Methods
Advantage estimation A
Sign of 1/0
Does 4(9 vary with
non negative when r(z,
REINFORCE
r(x.y)
non-positive when r(2.3)
No
non-negative when r(z. and
REINFORCE+ h9
r(a,y) with global batch normalization
No
positive when r[2,6)
(x,, pv) - r (x. 3) where is sampled greedy policy
non negative when r(z, M)) and
RcMax 30
No
non con-pooitive r(2,6)
48
<(xad-gg where Enm r (1))6) VRU-p)"
negative when r(z, M.)
and
non-posiin con-positive when [f.
ORZ 200
((.m) - V. (49)
non-negative when r(2, non positive when r(a,).) 0
No
DR.GRPO 56
r(x, - B. where Ey i (x.
non non-negative when r(z, v.) 1 and
non-positive when r(2,3.)
No
Table 8: Summary of advantage estimation of current response PG methods.
27

---

as the MC estimation of state-value function Thus these methods generally try to increase the probability of all the reasoning steps if the response y is correct and decrease the probability all the reasoning steps if y is incorrect. However, this update pattern indeed introduces high training variance since the sign of the true advantage value, i.e. A (8t, at), may mismatch the final reward of the trajectory. Following is a concrete example. The trajectory shown in Figure Figure contains 14 reasoning steps and a correct trajectory sampled from Llama-3. ---iinstruct The values and advantages of each reasoning step are also reported on the right-hand side of the figure. There is noticeable drop in value (aceuracy) in step 11, decreasing from 0,94 to 0.41, resulting in an advantage of -0.53. This means the model is more likely to make mistake starting from step 11. A well-directed policy update may reduce the likelihood of step 11. rather than increasing it, even if the final answer is conrect. Thus, there exists responses that yield correct answers while containing reasoning steps from which the model is likely to make mistakes, and vice versa. Thus, using the same advantage estimation for all reasoning steps in the entire response introduces high training variance.
Question
Reasoning Steps
Value
Advantage
Foyfnd the volumme of restonguiar box, noad maitppi ilr lagell wibli, fwly/h..
0.41
What is the vollume,
cubic inches, of
Stop Let's call the longot, wittt, and hoight of the box and
regpcsiblly We given wreer of the fion wa fishowes
0.53
rectangular box, whose
facess have ureaas square inches, 16 square inches square Inches?
Step
The fieee with aria zu has diverraions I we.
0:53
Slegs - The with wmee ab hoes cimermaionn se
0.81
Stop 5: The facee area area tas dimerstionn i
0.55
6t Si, we have the following equations from the given aroas.
0.53
7
0.84
0.31
Stop Th volume can mustrppty all three equations
lugetherr to gest
0.84
st (lw))
0.87
0.03
Stop zo: This stmplihees to (twh) 2304.
0:94
of skdes V2304.
0:41
Sleps fuch V144
0.59
Stop 13: Iwit 12 - 48.
Slego og The volume is Waoo cublec inches.
Figure 2: A reasoning trajectory sampled from 1iiia-..---B--nstttue with the true state value and advantage reported in the right hand side.
Step-level fine-grained policy optimization via DAPO. Even though the objective optimized by response-level algorithms can be reformulated to a multi-step decision problem (2) equivalently, it by no means implies that these methods can achieve step-level optimization as illustrated above. For DAPO, it can be directly observed from (12) that, given the previous reasoning steps (state s), DAPO optimizes the distribution of the next reasoning step generation (action depends the step-level advantage estimation other than the final reward. Thus DAPO indeed enables fine-grained step-wise learning by optimizing the actor using the advantage datasets. Our main theoretical result (Theorem 3.2p further shows that updating the actor's 's probability in the direction of the advantage can indeed yield policy improvement if the advantages are well-estimated.
E Discussion II Computational cost of DAPO
In this section, we provide detailed analysis of the computational cost of DAPO and discuss potential approaches for efficient training-ting time scaling. Consider actor model Mactor with N Nactor
28

---

non-embedding parameters is optimized by DAPO an offline advantage dataset D with batch size Bactor over Katorr training steps. For the critic pre-training, we first generate n initial responses from which responses are selected. For each reasoning step in these filtered responses, we then produce m completions for MC value estimation. Then critic model Meriic with N model non-embedding parameters is optimized on the MC value dataset with batch size Baniti over Kaitii training steps. To simplity the analysis, we assume both the response length and the number of reasoning steps per response remain constant at L and 'I' respectively (each reasoning step is of length LíT"). Following (24. we estimate inference computational cost as mode / and the training computational cost can be estimated by 6/Noodel-aain where Ninter and Nuain inference and training tokens respectively. We report Niter and Nuain for our DAPO implementation in Table
Computational cost N Medde (2N + 6.N (rain)
Step
Ninfer
Nirain 0
Generation
T2 L nmf. (T 1)
2n, Nacior
Completion
nmL (T -1) model
Critic Optimization
0
nT . Beritie Keritic
6n B Kartti Ammti
model
Making Advantage Dataset
nmL 0
2nm -
Actor Optimization
L/T Bactor
6L/T Bactor Kactor Nasker
Table 9: Computational cost estimation of DAPO.
Suppose Nactor
NTeritic
Nmoddel- By Table the total computational cost of DAPO is
Nuodel L (2m1 + nm (T 1) 2nmm + 6 BeTtBeetteKeiittee T Bactor ))
I.
L +nI BK
NNodel O LTnm+
where B = max Battor: Beritic] and K == max Kactor; Keritic). In our implementation n. 32, n 6, m = 16, T 20, 2048.
Possible approaches to efficient training-time scaling. Suppose the length of reasoning step Lstep == + is constant. Then the computation cost c Nmodel Lstep O (I + nT BK) The computational cost e may explode due square term T the response length increased. potential approach to mitigate this issue is not to do MC value estimation for all reasoning steps as in [38. Another possible approach to find an efficient and effective method for dividing reasoning steps, ensuring that the total number of reasoning steps T does not increase too rapidly as in [5
F
More details of experiments
F1
DAPO
Model
In Domain MATH
Out of Domain Olympiad Bench
GSM8K
Minerva MATH
College Math
Skywork-Math--LLnm
41.90
61.49
5.51
18.68
24.85
DAPO iter
46.88 6-.88
67. 67.55+6.00
7.34
22.38 +3.70
25,87 +1.03
DAPO iter 2
8.66
69.00 69.04 7.556
8.09
23.86 #5.18
27.20 #2.35
Qwen2-Math-TT -i--Mattt--B--ttreett
74.46
89.38
40.07
34,36
41.59
DAPO iter 1
75.41 +0.95
89.4 $9.45++0.0
37.51
37.03 +2.67
42.23 +0.64
DAPO iter 2
76.40 +19.4
89.30
38.25
37.64 +3.27
41.16 -0.43
Table 10: Full Mathematical benchmark results of iterative DAPO using zero-shot greedy inference.
29

---

Useful training tricks
Following are some useful tricks applied in all of the DAPO experiments.
State-wise learning. Notice that for each training state (intermediate step) 8, there are multiple actions (next step) for DAPO training. Rather than shuffling all state-action pairs, we try to place different actions from the same state within the same batch for DAPO training in order to provide contrastive gradients.
Learning uniformly on approximate action space. Suppose we have n actions {a; 1 for each training state s. Although the true action space A extremely large for LLM, we consider the unique actions of {a=);=1 as the approximate action space denoted as
Åg. After computing the advantage for each action, only add the unique actions to the advantage dataset for DAPO training, which means the training action distribution is (A(+))) Unif( A.) in our implementation.
Learning on the states with large advantage gap. Considering that the approximation error between the prediction of critic network and the true state value, we only learn those states that have large advantage gap to make sure the actions with higher advantage are actually better than those actions with lower advantage. We define the advantage gap
A == max A As,,,) min A (s, az).
arE
ayEл,
In implementation, only learn those states with A, >= 0.1
F.2
Baseline methods
Hyperparameter
Value
actor learning rate
5e-7
critic learning rate
9e-6
KL coefficient
0.01
discount factor 7
GAE factor
0.95
actor training epoch
critic training epoch
rollout max new tokens
2048
(4096 for OpenOO--Llam--BB-v..]
rollout temperature
rollout batch size
1024
global train batch size
1024
Table 11: The hyperparameters employ train PPO
30

---

MATH TEST Accuracy Absolute Improvement During Training
SkyworkcMath-ee Math-Lima
7.60%
OpenO Blam
Skywork Opar-Laama, 1-88
7.00%
7.02%
6.96%
Qown2 5-B--mel---hattrrtt
QwenZ 5-725iisttut
45%
6.50%
5. 5.06%
4.18%
4.30%
Score
3.70%
3.33%
3.24%
2.84%
2,18%
80%
1.71%
1.53%
1. %
25%
0.86% 99%
99%
0.90%
0.78%
1.14%
0.51%
0.00%
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Num Trained State-Artip Pairs (Million)
MATH TEST Accuracy Relative Improvement During Training
16.07%
16
14 80%
14.8%%
14.7256
14.31%
14
63%
13.745
13.00%
12
30.53%
8.6%
8.71%
Score
7.82%
7388%
7.12%
7.28%
A-17%
90%
6.29%
6.41%
6.2
5.43x8.6255 6115
4.5
3.54%
2.34%
Meteellama-31 1-88--nstruct
OpenO1 LLama 89 vo.1
Skywerk 01 Open--penmaee 1-48
Owen2.5 728 instruct
Owen25
Num of Trained State-Action Pairs (Million)
Figure Accuracy curve on MATH TEST during training process. x the accuracy of base model on MATH TEST and be the accuracy after DAPO training. Absolute improvement refers to 2 and relative improvement refers to y-2
1-z
31

---

Hyperparameter
Value
query batch size
64
group size
16
KL coefficient
0.01
rollout max new tokens
2048
(4096 for OpenO1-Llama--B--
rollout temperature
global train batch size
1024
actor learning rate
le-6
Table 12: The hyperparameters we employ to train GRPO
Training Accuracy
MATH TEST Accuracy Improvement During Training
0.
Llame 3.1 Intrus!
Liana lallaai
m
MMM
-2
0.-
.000
400
6:00
80.0
10,00
2.0
400
600
800
300.5
30e
Figure 4: Performance of -la.aaa..-----nniiiiii. optimized via GRPO. The model exhibits serve overfitting after around 300 steps. Left: Accuracy on training batch during RL process. Right: Performance improvement (in percentage points) on MATH 5K test problems.
G Pseudocode of DAPO
32

---

Algorithm Direct Advantage-Basel Policy Optimization
1: Input: Training problems D, initial actor Tea initial critic Vou. maximum iteration number T,
KL coefficient B.
2: for iteration i 0, 1..,, T - do
3: [Step A.1 Generation]
4: Dgan f 0, gegen <- Te. Tief nT,
5: for problem
do
6: Sample multiple responses from the generator, i.e. G(x) {リ} iid
7: For each response E G(z), split it into multiple reasoning steps (e.g. by using the line
break (m) so that y - (ag: ..y 0r-1)) 1), and the construct the states
S (y) - {Concat (z, ap, ar) 1
8:
Add the states derived from all responses into D gens i.e., D Dgen Dgen + UyEa(a) (v)) Stop generation when certain criteria met (e.g., when the computation budget reached).
9:
10:
end for
11:
[Step A.2 Completion]
12: Deom + 0, DMC f 0.
13:
for state E Dgen do
14:
Sample multiple completions C(s) +- {5} N <er(-Is) using the context prefix.
15:
Reward each completion and compute the MC value as
MC (s) (s)
Σ rs.
SEC(S)
16:
Ddd ((s, MC Ms))) D add ((s, C (s))).
17:
end for
18:
Update the critic (starting from Φ,) by optimizing 15 MC value dataset DMC, i.. 0++1 + arg min Dxce [CBE (MC (s), Va (s))
19:
[Step B.1 Compute Advantages]
20:
Dadv f
21:
for (s, C(s)) E D do
22:
action set A(s) + 0.
23:
For each completion C(s), extract the first reasoning step from ý and add it into A(s).
24:
Compute advantage for each action A(s) using optimized critic, i.e..
Va A()) a(s Vou.i (soa)
Σ aEA(())
Vou (so
A
25:
Add the estimated advantages into Dauv. i.e.,
Dudv + Dadv + (s,,,A(s,a) : :QE
26:
end for
27:
[Step B.2 Policy Optimization]
28: Update the actor (starting from 0.) by optimizing 12 on the advantage dataset Davy, ie.,
(
(1)(s)
log To (als) 27
A arg min (a,,,A)--aa
Feel (a|s)
29:
end for
30: Output: Last iterate actor Te, last iterate critic V
33


Reviews:
[{"review_id": "v8JxNI695q", "reviewer": "Reviewer_8sNP", "summary": "This paper introduces Direct Advantage-Based Policy Optimization (DAPO), a new reinforcement learning algorithm designed to improve the reasoning capabilities of large language models (LLMs). DAPO addresses two key challenges in applying RL to LLM reasoning: high training variance in response-level algorithms and poor value estimation in actor-critic methods. The method works by first training a critic function independently to estimate state values, then using these estimates to compute advantages for optimizing the policy at each reasoning step. Unlike existing response-level methods that apply uniform updates across all steps based on final rewards, DAPO provides step-specific training signals through the critic. The authors validate DAPO on mathematical and coding tasks, demonstrating consistent improvements over baseline methods like GRPO and PPO across multiple model architectures. The paper provides theoretical guarantees showing that DAPO produces better policies until reaching an optimal reference policy, along with detailed empirical analysis of its computational requirements and implementation considerations.", "questions": "- What is the step by step algorithm of DAPO?\n- Can you show ablation results with MSE loss for the critic as is typically used in literature?\n- What is the stopping criteria for the DAPO training process?\n- How much is the computational cost in terms of training time, FLOPs? Which machines were used?\n- It has been mentioned that \"only 4K samples\" are used in training? How small is that dataset compared to related works?\n- Why does performance drop in College Math and LiveCodeBench?", "limitations": "Limited discussion of failure cases or scenarios where the method might not work well", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Strong theoretical foundation with clear proofs and guarantees for policy improvement\n- Novel approach to addressing sparse rewards by using step-level advantages instead of response-level signals\n- Comprehensive empirical validation across multiple model architectures and benchmarks\n- Clear improvement over baseline methods (GRPO and PPO) shown in Tables 1-3\n- Ablation studies on hyperparameters and implementation considerations\n\nWeaknesses:\n- Critic training requires generation from base models and segmenting the generated text. It is unclear if this step is generalizable across models and (reasoning) problems.\n- No analysis of performance with weaker base models - all experiments use strong foundation models that are already trained with RL\n- Computational cost analysis is buried in Appendix E and lacks clear explanation of practical implementation requirements\n- Missing step-by-step algorithm description for implementing DAPO in practice\n- Empirical results in Figure 3 show fluctuations in performance, contradicting theoretical claims of monotonic improvement\n- Nit: Poor paper organization with key figures (like Figure 3) referenced in main text but placed in appendix\n- Nit: Technical terms like \"cot\" and \"PRM\" are not defined", "was_revised": true, "final_justification": "Author responses did not change my assessment", "rebuttal": "Thank you for reviewing our paper and providing helpful comments. Let us address your concerns in the following.\n\nRegarding the `Weakness` section:\n\n- From our perspective, this step can be easily extended to other models. The potential obstacle preventing its generalization to other tasks may lie in how to segment the generated text into distinct steps. In our work, we use the newline character $\\backslash$n to segment the text. This method is not necessarily optimal for long chain-of-thought scenarios or other tasks (such as agentic RL). We have discussed this breifly in lines 628-633.\n\n- Our experiments were **not** conducted exclusively on stronger models after RL training. As shown in Table 1, we tested our method on four SFT models and three RL models, covering both stronger and weaker variants. For the weaker SFT model like Skywork-Math-Llama, our algorithm was applied and achieved performance improvement. Table 4 further demonstrates that iterative DAPO can further enhance this model’s performance optimized by DAPO. Our approach also yielded notable improvements on stronger RL models, such as Qwen2.5-Math-7B-Instruct, demonstrating the effectiveness of DAPO.\n\n- Thanks for providing helpful feedback regarding the computational cost analysis. We would like to include a detailed analysis of training FLOPs of DAPO in our experiments in revised version.\n\n- Thanks for your feedback regarding the algorithmic description of DAPO. While the current implementation steps are outlined in Figure 1 and Lines 205–216, we acknowledge that a more granular, step-by-step specification would strengthen practical reproducibility. To address this, we will add a detailed pseudocode description of DAPO in the appendix of the revised version.\n\n- The performance fluctuations observed in Figure 3 **do not contradict** our theoretical claim in Theorem 2. We wish to clarify that the monotonic improvement property (established in Theorem 3.2) specifically compares the converged DAPO-optimized policy against the reference policy --- not adjacent gradient steps in the training stage. As evidenced in Figure 3, all  policies after DAPO training with adaquate gradient steps consistently outperform the initial reference policy. This alignment with theory holds true across all tested models. The transient fluctuations within the training process are attributable to stochastic training data.\n\n- We appreciate your helpful suggestions regarding the paper's writing and will make appropriate adjustments accordingly.\n\nRegarding the `Question` section:\n\n- The step-by-step algorithm is already detailed in Figure 1 and Lines 205–216. Briefly: during critic pretraining, we first use the base model to generate multiple solutions for each training problem. Each selected solution is segmented into states (using $\\backslash$n as the delimiter in this work). For every state $s$ (intermediate step), we perform multiple completions using the base model as the completer. We then construct Monte Carlo mean rewards as training labels for $s$ and optimize the critic network using BCE loss (Line 169). After critic pretraining, we extract next-steps (actions) for each state by segmenting completions into individual steps. Advantages are computed using the pretrained critic (Lines 152–158). This advantage dataset then optimizes the model through DAPO loss (Equation 12; Lines 149–151). **For clearer algorithmic specification, pseudocode for DAPO will be provided in the appendix of the revised manuscript.** \n\n- As stated in line 170, we use BCE loss to accelerate the critic's training. While we did not empirically compare MSE and BCE losses, our choice of BCE loss was theoretically motivated: Consider the critic objective in Equation (14). The gradients of sample-wise loss functions are given by\n\\begin{align*}\n\\nabla_\\phi \\mathcal{L}\\{MSE}\\left(V\\phi(s),V^\\pi(s)\\right) &= \\left(V^\\pi(s)-V_\\phi(s)\\right)\\nabla_\\phi V_\\phi(s)\n\\end{align*}and\n\\begin{align*}\n\\nabla_\\phi \\mathcal{L}\\text{BCE}\\left(V\\phi(s),V^\\pi(s)\\right) &= \\dfrac{\\left(V^\\pi(s)-V_\\phi(s)\\right)}{V_\\phi(s)\\left(1-V_\\phi(s)\\right)}\\nabla_\\phi V_\\phi(s).\n\\end{align*}\nThis shows that the BCE loss gradient is equivalent to the MSE loss gradient with an adaptive sample-wise learning rate $\\frac{1}{V_\\phi(s)(1-V_\\phi(s))} \\ge 4$. This accelerates critic training, particularly when $V_\\phi(s)\\rightarrow 0$ or $1$.\n\n- We stop the DAPO training process when the batch mean of absolute log ratio, i.e. $\\hat{E}_{\\left( s,a \\right)} \\left[ \\left| \\log \\frac{\\pi_{\\theta} \\left( a|s \\right)}{\\pi_{\\text{ref}} \\left( a|s \\right)} \\right| \\right]$, converges, indicating policy convergence.\n\n- We use A100 GPUs for both generation and training. As discussed in Section E, the total FLOPs consumed during the entire training stage can be approximated by $2N_\\text{model}(N_\\text{inference} + 3N_\\text{training})$. In our DAPO experiment on Llama-3.1-8B-Instruct, $N_\\text{inference} \\approx 1.65 \\times 10^9$ and $N_\\text{training} \\approx 9.68 \\times 10^7$. This total computational cost is roughly equivalent to 1k steps of online RL training in GRPO with a batch size of 1024 and a maximum of 2048 new tokens\n\n- The TACO dataset originally contains 26,443 questions. After filtering, only 4k questions were retained for RL training, which is much smaller than the original dataset. This is why we emphasize that there are only 4k samples.\n\n- The performance drop in College Math and LiveCodeBench may be due to limited training samples which lack enough diversity."}, {"review_id": "vX6oOwZR6L", "reviewer": "Reviewer_SP6p", "summary": "The paper introduces DAPO, a step-level offline RL method designed to enhance the multi-step reasoning capabilities of LLM in tasks like mathematical problem-solving and code generation. Unlike response-level RL methods that rely on sparse final rewards, DAPO addresses high training variance by using a critic function to provide dense, step-level training signals. The method trains the critic independently before optimizing the policy, avoiding the instability of traditional Actor-Critic methods. Experiments on mulyiple benchmarks demonstrate DAPO's superior performance.", "questions": "1. The paper mentions Step-DPO but does not include a direct performance comparison in the experiments. Could you add a comparison with Step-DPO to highlight DAPO's superiority?\n\n2. DAPO separates the training of the actor and critic into two stages to improve stability. Could this lead to the critic being unable to adapt to changes in the actor's policy? How does DAPO solve this problem?", "limitations": "N/A", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength\n\n1. DAPO introduces a step-level offline reinforcement learning (RL) approach, addressing the sparse reward problem in multi-step reasoning tasks. \n2. The paper provides a rigorous theoretical foundation,  proving monotonic policy improvement under KL-regularized objectives\n3. Extensive experiments demonstrate DAPO’s effectiveness across multiple mathematical and coding benchmarks\n\nWeakness\n\n1. DAPO's performance heavily relies on the quality and diversity of the training dataset. The paper does not sufficiently discuss how to handle cases where the dataset lacks coverage for certain states or actions, which is a common challenge in offline RL.\n2. Generating multiple sub-trajectories for critic training and sampling multiple actions for advantage estimation may introduce significant computational costs. For researchers or institutions with limited resources, generating high-quality training states and sub-trajectories could be a bottleneck, especially for larger-scale models or more complex tasks.\n3. While DAPO compares favorably to PPO and GRPO, it does not benchmark against other recent offline RL methods, such as step-DPO [1].\n4. DAPO's experiments primarily focus on mathematical reasoning and code generation, which have clear correctness rewards, but the paper does not validate DAPO's effectiveness in other tasks (e.g., open-domain question answering).\n\n[1]Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs", "was_revised": true, "final_justification": "The authors’ rebuttal provided clear responses to my main concerns, especially regarding the separation of actor and critic training and the rationale for not including step-DPO as a baseline. While issues of data coverage and computational requirements remain, these are common limitations in offline RL. The theoretical contributions and empirical results are solid. Considering the clarifications, I have raised my score.", "rebuttal": "Thank you for reviewing our paper with helpful comments! We address your concerns point by point below. \n\nRegarding the `Weaknesses` section:\n\n- We acknowledge that the performance of DAPO relies on data coverage, as it is a typical offline RL method and lacks online exploration. However, as you pointed out, this is a common challenge in offline RL. Theoretical analyses of offline RL also frequently assume adequate data coverage. Thus, addressing the data coverage problem thoroughly is beyond the scope of this work. That being said, our experiments show that merely using trajectories generated by the reference model as the empirical training distribution suffices to yield measurable improvements. This suggests that data coverage may not pose a severe problem for achieving performance improvements in the trust region of reference policy.\n\n- Regarding the computational resource demands raised as a weakness, we acknowledge that pretraining the critic is resource-intensive. A brief discussion of this limitation is already explicitly included in Section E and lines 257–267. We also address potential approaches to reduce the computation cost for critic pretraining in lines 628–633. **It should be noted, however, that achieving higher performance improvements generally requires generating more samples and consequently demands greater computational resources. Our DAPO provides a valid method for attaining such improvements when adequate training resources are available**. As shown in Figure 3, DAPO consistently enhances model performance throughout actor training. Table 3 further demonstrates that DAPO delivers superior performance gains over baseline methods **even under comparable computational budgets**(see discussion in lines 257–267).\n\n- Note that Step-DPO is a preference learning algorithm that requires the pairwise preference dataset $\\mathcal{D}$, and the curation of $\\mathcal{D}$ dominates its performance. **Consequently, conducting a fair performance comparison between DAPO and Step-DPO is challenging**. Therefore, we chose PPO and GRPO, instead of Step-DPO, as our baseline methods.\n\n- For RL training on tasks lacking clear correctness rewards, a reward model is often utilized. **However, reward hacking frequently occurs in such scenarios. To prevent interference from reward hacking when using reward models and to validate the effectiveness of the RL algorithm itself, we focus DAPO's experiments on mathematical reasoning and code generation**. These tasks are verifiable and provide intrinsic correctness rewards, thus minimizing the likelihood of reward hacking. Furthermore, as theoretically guaranteed (see Theorem 3.2), it is reasonable to anticipate that DAPO will achieve performance improvements on the training distribution, even for tasks without clear correctness rewards.\n\nRegarding the `Questions` section:\n\n- The reason why we do not include step-DPO for performance comparison is mentioned above. **The main superiority of DAPO over step-DAPO  is that DAPO eliminates the need to curate a high-quality preference dataset to provide optimization signals for intermediate steps**, as producing such a dataset is often challenging or equally computationally expensive. Our experiments demonstrate that performing simple Monte Carlo sampling from the reference model achieves measurable improvements. Furthermore, for a given input context prefix, DAPO can learn directly from multiple samples, whereas step-DPO requires pairwise data. \n\n- While actor-critic methods often update the actor and critic simultaneously, our core theoretical result (Theorem 3.2) shows that DAPO's policy optimization against the **fixed** value function of the reference model guarantees performance improvement. **Therefore, the critic's inability to adapt to the actor's policy changes within a single iteration is not problematic**. It is worth noting that we also conducted experiments with iterative DAPO. In this approach, the actor model optimized by DAPO becomes the new $\\pi_\\text{ref}$, and the critic is updated to estimate the value function of this new reference model for further policy optimization in the next iteration. The results (lines 243-248) show that adapting the critic to the actor's changes leads to greater performance gains.\n\n**We sincerely hope our responses have addressed your concerns and will positively influence your assessment of our paper.**"}, {"review_id": "Xm6ZuoRBPI", "reviewer": "Reviewer_xRdV", "summary": "Paper \"DAPO: Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage-Based Policy Optimization\" proposes a novel policy optimization method for multi-step reasoning. It proposes dense step-level training signals by computing an advantage function at each intermediate step using a separately trained critic. Experiments on mathematical and code benchmarks show that DAPO consistently improves performance across multiple models (SFT and RL-pretrained), outperforming both PPO and GRPO baselines.", "questions": "1. The DAPO objective seems pretty interesting and novel. I saw similar squared loss in the IPO paper [1]. Could the authors discuss the relation of the proposed work and this reference?\n2. Equation (9) should plus the gradient since we are doing gradient ascent? Also for Lemma 3.1 and (10), the signs should be positive?\n3. Is the multi-step capability of the proposed framework mainly comes from the critic model V? That is, V is serving as the intermediate signal of the multi step states and actions? If so, I'm curious if the cross entropy training of V in line 169 (which is not new) could be improved since the current loss is still only making use of the outcome reward?\n4. It is really astonishing to see the severe overfitting of GRPO in Fig. 4 for MATH dataset. Could the author comment on why DAPO could avoid such overfitting. Also would PPO and other algorithm also result in similar overfitting?\n5. The paper uses dense RL notation that may not be easily digestible for a mixed NLP/ML audience. For example, input prompt could be denote as x and output continuation as y; while input, or input truncated with unfinished continuation could be denoted as state s, and the next chunk of continuation could be denoted as action a. I think this could lead to less confusion.\n6. Typos: Line 190 \"... reproducibility and of DAPO...\"", "limitations": "See questions section.\n\nNo potential negative societal impact.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: The work proposes a well-motivated solution using step-level optimization and separate critic training. The new surrogate loss could still lead to a monotonic improvement theorem (Theorem 3.2). DAPO consistently improves performance across in-domain (MATH) and out-of-domain benchmarks (GSM8K, Minerva, Olympiad, etc.).\n\nWeaknesses (Please respond directly to the questions section): Training the critic is expensive due to massive rollout sampling. The accuracy of advantage estimates relies entirely on the quality of the critic.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper with helpful comments. Below we address each of the points you raised in `Questions` section one by one.\n\n- The squared loss studied in IPO is \n\\begin{align}\n\\min_{\\theta} E_{\\left(x,y_{w},y_{l} \\right) \\sim \\mathcal{D}} \\left[ \\left( \\log \\left( \\frac{\\pi_{\\theta} \\left( y_{w}|x \\right)}{\\pi_{\\text{ref}} \\left( y_{w}|x \\right)} \\right) -\\log \\left( \\frac{\\pi_{\\theta} \\left( y_{l}|x \\right)}{\\pi_{\\text{} \\text{ref}} \\left( y_{l}|x \\right)} \\right) -\\frac{1}{2\\beta} \\right)^{2} \\right],\n\\end{align}\n    while the squared loss studied in DAPO is \n    $$\n    \\underset{\\theta}{\\min} \\frac{1}{2}E_{s\\sim \\nu _{\\mathcal{S}},a\\sim \\nu _{\\mathcal{A}}\\left( \\cdot |s \\right)}\\left[ \\left( \\frac{1}{\\beta}A^{\\pi _{\\mathrm{ref}}}\\left( s,a \\right) -\\log \\frac{\\pi _{\\theta}\\left( a|s \\right)}{\\pi _{\\mathrm{ref}}\\left( a|s \\right)} \\right) ^2 \\right],\n    $$\n    where $\\beta$ is the coefficient of KL regularization. **Although both objectives utilize the squared loss, they exhibit several fundamental differences.** For instance: 1) IPO is a preference learning algorithm requiring pairwise data, while DAPO is not. 2) IPO is a response-level algorithm as it optimizes the response probability directly, whereas DAPO is a step-level algorithm optimizing only the next-step distribution. However, the shared application of squared loss enables both algorithms to modulate the degree of policy adaptation through the KL regularization term $\\beta$. Generally, a smaller $\\beta$ yields a more conservative learning approach with lower performance gains. Conversely, a larger $\\beta$ facilitates more significant policy updates (potentially improving performance) but increases the risk of distribution shift, demanding broader data coverage.\n\n- Thanks for pointing out the typo. The sign should be positive in Equation (9). We will fix it in the revised revision. The negative sign in (10) is correct since Lemma 3.1 shows that $\\nabla_\\theta \\mathcal{I}_s(\\pi_{\\theta_k},\\pi_{\\text{ref}})=-\\beta \\nabla_\\theta\\mathcal{H}^k_s(\\theta_k)$.\n\n- **The multi-step capability originates from the critic model $V$, necessitating critic pretraining in DAPO to ensure $V$ provides accurate intermediate signals for policy optimization**. Note that we only consider the binary outcome reward in this work and the value function defined in Equation (3) is a probability. Thus we use the binary cross-entropy loss in critic pretraining. Given the value function's definition (Equation (3)), we find it challenging to make fundamental improvements to critic pretraining merely through training loss design. Note that the critic pretraining objective (see Equation (14)) involves two elements: (1) the training state distribution $\\mathcal{D}$ and (2) the state-wise loss function $\\mathcal{L}$. For more efficient critic pretraining, we suggest further investigation into designing $\\mathcal{D}$. Related discussion also appears in lines 628-633.\n\n-   **We attribute the overfitting of GRPO observed in Figure 4 primarily to the high training variance of response-level advantage estimation.** As discussed in Section D, some responses yield correct answers despite containing flawed reasoning steps that may lead the model to make mistakes. Consequently, using response-level advantage estimation in RL training can overfit to such responses and harm OOD performance. Our PPO experiment on Qwen2.5-Math-7B-Instruct exhibits similar overfitting. We find that the standard error of the critic output decays to zero. This indicates that given response $y$, the state value predictions for tokens in $y$ output by the critic model converge to some constant $c$, causing the critic to fail in providing meaningful intermediate signals. In this scenario, the GAE($\\lambda$) estimator used in PPO satisfies $A_t \\rightarrow \\lambda^{T-t-1}(r(x,y)-c)$, effectively reducing PPO to a response-level algorithm like GRPO. DAPO pretraining the critic using Section 3.2's procedure ensures accurate intermediate signals for policy optimization, yielding more robust performance improvements.\n-  Thanks for your helpful advice.\n- Thanks for pointing out this typo; we'll fix it."}, {"review_id": "bW92NyiLfn", "reviewer": "Reviewer_j5UX", "summary": "This paper explores the role of reinforcement learning in enhancing the reasoning capabilities of large language models and highlights key challenges, particularly the issue of sparse rewards. Sparse rewards increase training variance in policy optimization and hinder accurate value function estimation in Actor-Critic methods. To address these limitations, the authors propose DAPO, a novel step-level offline RL algorithm with theoretical guarantees for improving LLM reasoning.", "questions": "* 1. This method is designed for discrete spaces, can it be extended to continuous data spaces?\n* 2. Could you provide insights into how the coefficient of KL regularization $\\beta$ affects the experimental results, since this is a critical parameter in offline RL?\n* 3. Regarding the sampling process in Section~3.2, could you clarify (1) whether each trajectory $\\tau_i$ contains only single states $s$, and (2) the precise meaning of $s_{T_i-1}$ in the Monte Carlo mean calculation?", "limitations": "As is mentioned in the weakness section above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n\n* The proposed DAPO algorithm introduces a principled step-level optimization approach using a pretrained critic, effectively addressing sparse reward and critic training challenges in LLM reasoning tasks.\n* For the proposed method, the authors provide comprehensive proofs, ensuring the theoretical completeness and feasibility.\n* Comprehensive experiments show consistent gains across mathematical and coding benchmarks.\n\n**Weakness:**\n\n* This method appears to require extensive adoption during the pre-training of the critic, placing excessively high demands on both resources and complexity.\n* Additionally, since the adopted offline method and advantage estimation do not fully address the out-of-distribution problem in offline RL, this approach heavily relies on the coverage of the pre-training data.", "was_revised": true, "final_justification": "The authors have addressed the majority of my concerns in a thoughtful and constructive manner, and I will maintain my score.", "rebuttal": "Thank you for reviewing our paper and providing helpful comments.\n\n**About the `Weakness` part**\n- Regarding the computational resource, we acknowledge that pre-training the critic is resource-intensive. A brief discussion of this limitation is explicitly included in Section E and lines 257–267. We also address potential approaches for more efficient critic pre-training in lines 628–633. Nevertheless, the computational investment is justified: As shown in Figure 3, DAPO consistently improves model performance throughout actor training. Table 3 further demonstrates that DAPO delivers superior performance gains over baseline methods even under comparable computational budgets (see discussions in lines 257–267).\n- Concerning reliance on data coverage, this is a common challenge in offline RL methods. Theoretical analyses of offline RL (e.g., [1]) frequently assume adequate data coverage. As DAPO is an offline method without online exploration, it reasonably requires sufficient data coverage to achieve performance gains. Importantly, our experiments show that merely using samples generated by the reference model as the empirical training distribution suffices to yield measurable improvements.\n\n[1] Kishan Panaganti, Zaiyan Xu, Dileep Kalathil and Mohammad Ghavamzadeh. Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage. Arxiv.2310.18434\n\n**About the `Questions` part**\n- Our method can be extended to MDPs with continuous state and action spaces, as the Performance Difference Lemma (see line 96) and the definition of visit probability (see line 98) can be generalized to the continuous setting. However, handling continuous data spaces requires additional technical details. To avoid technical complications, we focus solely on the discrete setting, which also aligns with the data space of LLMs.\n- The results of our ablation study on the KL coefficient $\\beta$ are detailed in lines 249–253. Generally, a smaller $\\beta$ yields a more conservative policy learning and lower performance gains. Conversely, a larger $\\beta$ enables more aggressive policy learning (potentially improving performance) but risks higher distribution shift, demanding greater data coverage. Thus, a moderate $\\beta$ achieves the optimal trade-off.\n- (1) Each trajectory may contain multiple states. For example, denoting a trajectory as $\\tau = (x, a_0, \\ldots, a_{T-1})$, where $x$ is the prompt, $a_i$ is the token generated at index $i$ and $T$ is the response length. We\n use the line break$\\backslash$n to segment the response $y=(a_0,...,a_{T-1})$ into multiple steps. Assuming there are $k$ steps in total, the token sequence of the trajectory $\\tau$ truncated at each $i$-th step can be regarded as a state. Thus, this process yields $k$ states in total.  (2) Each state $s$ (a context prefix) serves as the starting point for sampling $N$ sub-trajectories $(\\hat{y}_1, \\ldots, \\hat{y}_N)$. We then compute the group mean reward  of these sub-trajectories as the critic training label of input state $s$. This procedure explains the Monte Carlo mean calculation mentioned."}]

Author rebuttal:

Meta-review: The reviewers found DAPO to be a technically solid and well-motivated contribution. They consistently highlighted the novelty of introducing step-level optimization with an independently trained critic, strong theoretical guarantees of monotonic policy improvement, and compelling empirical validation across multiple mathematical and coding benchmarks. The method was seen as an effective way to address the sparse reward and high-variance issues in response-level RL, with clear performance improvements over PPO and GRPO. The authors’ rebuttal successfully clarified questions about critic training, computational costs, and connections to related methods (e.g., Step-DPO, IPO), and most reviewers expressed satisfaction that their main concerns were addressed.

The main points of disagreement centered on practical limitations: the heavy reliance on data coverage, high computational cost of critic training, and uncertainties about generalization to tasks beyond math/code reasoning. Some reviewers also noted missing baselines and the lack of broader task validation. However, these issues were considered common challenges in offline RL rather than fatal flaws, and did not outweigh the paper’s contributions. Overall, I find this an insightful and well-executed paper, and support its acceptance and NeurIPS.
Decision: Accept (spotlight)