DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Authors: Qiying Yu
Zheng Zhang
Ruofei Zhu
Yufeng Yuan
Xiaochen Zuo
YuYue
Weinan Dai
Tiantian Fan
Gaohong Liu
Juncai Liu
LingJun Liu
Xin Liu
Haibin Lin
Zhiqi Lin
Bole Ma
Guangming Sheng
Yuxuan Tong
Chi Zhang
Mofan Zhang
Ru Zhang
Wang Zhang
Hang Zhu
Jinhua Zhu
Jiaze Chen
Jiangjie Chen
Chengyi Wang
Hongli Yu
Yuxuan Song
Xiangpeng Wei
Hao Zhou
Jingjing Liu
Wei-Ying Ma
Ya-Qin Zhang
Lin Yan
Yonghui Wu
Mingxuan Wang
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Reasoning
RL
Primary area: deep_learning
TL;DR: The state-of-the-art open-sourced LLM RL algorithm, dataset and infrastructure

Abstract: Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the **D**ecoupled Clip and **D**ynamic s**A**mpling **P**olicy **O**ptimization (**DAPO**) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.

DAPO: An Open-Source LLM Reinforcement Learning System at Scale
Qiying Yu
Zheng Zhang", Ruofei Zhu'
Yufeng Yuan'
Xiaochen Zuo
Yu Yue
Weinan Dai4 Tiantian Fan
Gaohong Liu't Juncai Liu
Lingjun Liu 2 Xin Liu
Haibin Lin Zhiqi Lin Bole Ma Guangming Sheng!3; Yuxuan Tong Chi Zhang
Mofan Zhangl: Ru Zhang Wang Zhang Hang Zhu Jinhua Zhu 18 Jiaze Chen
Jiangjie Chen Chengyi Wang' Hongli Yu4 Yuxuan Song124 Xiangpeng Wei
Hao Zhou24t Jingjing Liu24 Wei-Ying Ma Ya-Qin Zhang?*
Lin Yan Yonghui Wu! Mingxuan WanglAt
Equal contribution. $ Equal engincering contribution.
ByteDance Seed
Institute for Al Industry Research (AIR), Tsinghua University
The University of Hong Kong
SIA-Lab of Tsinghua AIR and ByteDance Seed
Abstract
Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such in OpenAI ol blog and DeepSeek RI technical report), thus the community still struggles to reproduce their RL training results. We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm, and fully open- source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RI. success. In addition, we open-source our training code, which is built on the verl framework, along with carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research large-scale LLM RL.
80
50-
DAPO
DeepSeek R1-Zero
60-
2024
30-
DAPO avg
DAPO passs@32
DAPO cans@33
0.
2000
4000
6000
8000
10000
Training Steps
Figure I: AIME 2024 scores of DAPO on the Qwen2.5-320 base model, outperforming the previous SoTA SeepSeee--R-ZZeo---e update steps.
using 50% training steps. The x-axis represents the gradient
Correspondence to zhouhuo@ www.ssinghua..d......mm wangmingxuan. 89 bytedance.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Test-time scaling such OpenAT's ol [1] and DeepSeek's R1 [2] brings a profound paradigm shift to Large Language Models (LLMs) [3, 4, 5, 6, 7]. Test-time scaling enables longer Chain-of- Thought thinking and induces sophisticated reasoning behaviors, which makes the models superior in competitive math and coding tasks like AIME and Codeforces.
The central technique driving the revolution is large-scale Reinforcement Learning (RL), which elicits complex reasoning behaviors such as self- rerificifiiato and iterative refinement. However, the actual algorithm and key recipe for scalable RL training remains a myth, hidden from technical reports of existing reasoning models [1, 2, 8, 9, 10, 11]. this paper, we reveal significant obstacles in large-scale RL training and open- open-source a scalable RL system with fully open-soureed algorithm, training code and dataset that provides democratized solutions with industry-level RL results.
We experiment over Qwen2.5-32B [12] as the pretrained model for RL. In our initial GRPO run, we achieved only 30 points on AIME a performance significantly below DeepSeek's RL (47 points). thorough analysis reveals that the naive GRPO baseline suffers from several key issues such as entropy collapse, reward noise, and training instability. The broader community has encountered similar challenges in reproducing DeepSeek's results [13, 14, 15, 16, 17, 18, 19] suggesting that critical training details may have been omitted in the RI paper that are required to develop an industry-level,, large-scale, and reproducible RL system.
To close this gap. release an open-source state-of-the-as system for large-scale LLM RL, which achieves 50 points on AIME 2024 based on Qwen2.5-32B model, outperforming previous state-of- the-art results achieved by DeepSeek- R1 11--erro-e -ere-QQeen33 [2] (47 points) using 50% training steps (Figure 1). We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm, and introduce 4 key techniques to make RL shine in the long-CoT RL scenario. Details are presented in Section 3.
1. Clip-Higher, which promotes the diversity of the system and avoids entropy collapse;
2. Dynamic Sampling, which improves training efficiency and stability;
3. Token-Level Policy Gradient Loss, which is critical long- Mon RL scenarios;
4. Overlong Reward Shaping, which reduces reward noise and stabilizes training.
Our implementation is based on verl [20]. By fully releasing state-of-the-an RL system including training code and data, we aim to reveal valuable insights to large-scale LLM RL that benefit the larger community.
2 Preliminary
2.1 Proximal Policy Optimization (PPO)
PPO [21] introduces a clipped surrogate objective for policy optimization. By constraining the policy updates within proximal region of the previous policy using clip. PPO stabilizes training and improves sample efficiency. Specifically, PPO updates the policy by maximizing the following objective:
(0) =
(o 9,0 Ac, clip Thau (or q, 0ct))
To (0,0z)) Te (o: q, 22e)
1- e, 1,1 |
(I)
min
where (g. a) is question-answer pair from the data distribution D, is the clipping range of importance sampling ratio, and A, an estimator of the advantage at time step Given the value function V and the reward function R, Å is computed using the Generalized Advantage Estimation (GAE) [22]:
AG AGAB(Y
Σ (1
(2)
where
6, Ri + V(si), 05 759,55
(3)

---

Clpeltghet Higher
dip-llglerr Higher
0.35
Generation Entropy
0.8
AIME avg
0.30
0.5
0.25
0.4
0.20
0.3
0.2
0.00
0.0
2000
2000
2500
Step
Step
(a) Accuracies on AIME
(b) Entropy of actor model.
Figure The accuracy on the AIME test set and the entropy of the actor model's generated probabilities during the RL training process, both before and after applying Clip-Higher strategy.
2.2 Group Relative Policy Optimization (GRPO)
Compared PPO, GRPO eliminates the value funetion and estimates the advantage in group- relative manner. For specific question-answer pair (g, a), the behavior policy To.s samples group of G individual responses {o; {o:) Then, the advantage of the i-th response is calculated by normalizing the group-level rewards [Ra)
mean (R:)
A. std RR: (R,)
Similar to PPO, GRPO adopts clipped objective, together with directly imposed KL penalty term:
JaRPO (8)
E (o)
+ Art) 1 D (Tol )
(5)
min (ri,(0)A clip (o)) -
where
r (0) = To(oi g. cr.)
(6)
Te (oi, g, 0i,<t)
It is also worth noting that GRPO computes the objective at the sample-level. To be exact, GRPO first calculates the mean loss within each generated sequence, before averaging the loss of different samples. As we will be discussing in Section 3.3, such difference may have an impact the performance of the algorithm.
2.3 Removing KL divergence
The KL penalty term is used regulate the divergence between the online policy and the frozen reference policy. In the RLHF scenario [23], the goal of RL is to align the model behavior without diverging too from the initial model. However, during training the long- reasoning model, the model distribution can diverge significantly from the initial model, thus this restriction is not necessary. Therefore, we will exclude the KL term from our proposed algorithm.
2.4 Rule-based reward modeling
The of reward model usually suffers from the reward hacking problem [24, 25, 26, 27, 28, 29]. Instead, we directly use the final accuracy a verifiable task as the outcome reward:
1, -1,
is ss_eeuvvvaeen((j..
R(ú. v)
otherwise

---

where y is the ground-trun truth answer and ÿ is the predicted answer. This is proved to be an effective approach activating the base model's reasoning capability, as shown multiple domains such as automated theorem proving [30, 31, 33), computer programming [34,35, 35, 36, 37], and mathematics competition [2].
3 DAPO
We propose the Decouple Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm. DAPO samples a group of outputs {o: for each question q paired with the answer a, and optimizes the policy via the following objective:
TDA =
1o.1
min (O), clip (ri()ri((),. Flow: + Enigh A..))
(8)
i=l
los|
1=1
<<
(oz is (s_eeuvvalent((ao))) <G,
where
(0) To(os, 9. 0s,ct)
A = R, mean ( (R.)
(9)
(oLe 01,4))
std( ( [ )/-
The full algorithm can be found in Algorithm 1. In this section, will introduce the key techniques associated with DAPO.
3.1 Raise the ceiling: Clip-Higher
In our initial experiments using naive PPO [21] or GRPO [38], we observed the entropy collapse phenomenon: the entropy of the policy decreases quickly as training progresses (Figure 2b). The sampled responses of certain groups tend to be nearly identical. This indicates limited exploration and early deterministic policy, which can hinder the scaling process.
We propose the Clip-Higher strategy to address this issue. Clipping over the importance sampling ratio is introduced in Clipped Proximal Policy Optimization (PPO-Clip) [21] to restrict the trust region and enhance the stability of RL. We identify that the upper clip can restrict the exploration of the policy, where making an 'exploitation' token more probable much easier yet the probability of an unlikely "exploration" token is too tightly bounded to be uplifted.
Concretely, when & = 0.2 (the default value of most algorithms) and Å.,. > 0 (the system tries to increase the probability). consider actions with probabilities Th (o: q) = 0.01 and 0.9. The upper bounds of the increased probabilities (o(o; ; q) are 0.012 and 1.08, respectively (Fo,, (1 + e)). This implies that 'exploitation' tokens with higher probability (e.g., 0.9) are not constrained get even extremely larger probabilities like 0.999. Conversely, for low-probability 'exploration' tokens, achieving a non-trivial increase in probability is considerably more challenging. Empirically, also observe that the mean probability of up-clipped tokens is low: Ta (o q) < 0.2 (Figure 3a). This supports our intuition that the upper clipping threshold indeed restricts the probability increase of low low-probability 'exploration' tokens, thereby potentially constraining the exploration of the system.
Adhering the Clip-Higher strategy, decouple the lower and higher clipping range as êtow and Ehigh. as highlighted in Equation 10:
(8) DT..j)>D.(o..E.. (o
lo:l
min (0) clip (O), Slow, 1 + Shigh A:t.)
(10)
1= lo:|
s.t. 0
{o: |is_equivalent(a,
at)3 <G
We increase the value Ehigh to leave more room for the increase of low-probability tokens. As shown in Figure 2, this adjustment effectively enhances the policy's entropy and facilitates the generation of more diverse samples. We keep flow as is, because increasing will suppress the probability of these tokens to 0, resulting in the collapse of the sampling space.

---

0.35
Samples with avg@ 100%
Mean Up jp-Clipeed Probiity
0.6
0.39
0.25
0.4
0.3
0.15
0.2
6.1
0.10
2000
6000
adoo
Step
Step
(a) Mean up-clipped probability.
(b) The proportion of samples with an accuracy of 1.
Figure 3: The mean up-clipped probability as well the ratio of prompts with accuracy=l.
3.2 The more the merrier: Dynamic Sampling
Existing RL algorithm suffers from the gradient-decreasing problem when some prompts have accuracy equal 1. For example for GRPO, if all outputs (o,) of particular prompt correct and receive the same reward, the resulting advantage for this group is zero. A zero advantage results in zero policy gradients, shrinking the magnitude and increasing the noise sensitivity of the batch gradient, thereby degrading sample efficiency. Empirically, the number of samples with accuracy equal 1 continues increase, shown Figure 3b. This means that the effective number of prompts in each batch keeps decreasing, which can lead to larger variance in gradient and dampens the gradient signals for model training.
To this end, we propose to over-sample and filter out prompts with the accuracy equal to and illustrated in Equation 11, leaving all prompts in the batch with effective gradients and keeping consistent number of prompts. The sampling cost for each batch dynamic. Before training, we keep sampling until the batch is fully filled with samples whose accuracy is neither 0 nor 1.
(O) =
G lull
min (ri,(())Ala, clip (0), Elows 1 Ehigh Ar.t)
(1I)
E i=
los|
t=1 t=1
s.t. << for iis_equivalen((ao)) <G
Note that this strategy does not necessarily impede training efficiency, because the generation time typically dominated by the generation of long-tail samples if the RL system synchronized and the generation stage is not pipelined. Besides, we find that with dynamic sampling the experiment achieves the same performance faster as shown in Figure
3.3 Rebalancing act: Token-Level Policy Gradient Loss
The original GRPO algorithm employs a sample-level loss calculation, which involves first averaging the losses by token within each sample and then aggregating the losses across samples. In this approach, each sample is assigned an equal weight in the final loss computation. However, find that this method loss reduction introduces several challenges in the context of long-CoT RL.
Since all samples are assigned the same weight in the loss calculation, tokens within longer responses may have disproportionately lower contribution the overall loss, which can lead to two adverse effects. First, for high-quality long samples, this effect can impede the model's 's ability to learn reeleatn patterns within them. Second, we observe that excessively long samples often exhibit low-quality patterns such as gibberish and repetitive words. Thus, sample-level loss calcula- tion, due its inability to effectively penalize those undesirable patterns long samples, leads to an unhealthy increase in entropy and response length, shown in Figure 4a and Figure 4b.
5

---

1.5
3400
tollen lows
3.0
Mean Response Length
Generatioo inneration Entropy
4000
2.5
3002
2.5
2000
2000
2000
4000
Step
Step
(a) Entropy of actor model's s generation probabilities. (b) Average length of actor model-generated responses
Figure 4: The entropy of the probability distribution the actor model, as well as the changes in response length.
We introduce a Token-level Policy Gradient Loss in the long-CoT RL scenario to address the above limitations:
(0) E ( Ma
1..
min (rr,(0)AAe,, clip (r)) - Slow 1 Shigh A
(12)
Σ 1= 0
s.t. 0 < for iss ess_eqvivaent((a.o) <G
In this setting, longer sequences can have more influence on the overall gradient update compared to shorter sequences. Moreover, from the perspective individual tokens, if particular generation pattern can lead to increase decrease in reward, it will be equally prompted or suppressed, regardless the length of the response in which appears.
3.4 Hide and seek: Overlong Reward Shaping
In RL training, typically set maximum length for generation, with overlong samples truncated accordingly. We find that improper reward shaping for truncated samples can introduce reward noise and significantly disrupt the training process.
By default, we assign a punitive reward to truncated samples. This approach may introduce noise into the training process, sound reasoning process can be penalized solely due its excessive length. Such penalties can potentially confuse the model regarding the validity of reasoning process.
To investigate the impact of this reward noise, we first apply an Overlong Filtering strategy which masks the loss of truncated samples. We find that this approach significantly stabilizes training and enhances performance, as demonstrated in Figure 5.
Furthermore, propose Soft Overlong Punishment (Equation 13), length-aware penalty mech- anism designed to shape the reward for truncated samples. Specifically, when the response length exceeds the predefined maximum value, define punishment interval. Within this interval, the longer the response, the greater the punishment it receives. This penalty is added to the original rule-based correctness reward, thereby signaling the model avoid excessively long responses.
(0: (Trmes 1- Loate
18) Tomax Toguche
Riengt (1) =
(13)
Lenxx
Leache
<
[y]
<
Lmmxx
3.5 Dataset transformation
Our dataset sourced from the web and official competition homepages through a combination of web scraping and manual annotation. The answers of math dataset typically come in variety of formats,

---

wit overiing lltering overlong
svenking fiaerig wl everlong fittring
0.30
Generation Entropy
AIME avg
0.20
0.10
0.05
2000
1000
2000
Step
Step
(a) Performance on AIME.
(b) Entropy of actor model.
Figure 5: The accuracy of the actor model on AIME and the entropy of its generation probabilities. both before and after applying Overlong Reward Shaping strategy.
Algorithm DAPO: Decoupled Clip and Dynamic sAmpling Policy Optimization
Input initial policy model To; reawrd model R; task prompts D; hyperparameters 6low; Chigh
for step = 1.....M do
2: Sample batch D, from D
Update the old policy model Te.ia f- Te
4: Sample G outputs {o:] 1G
(-1g) for each question q DD 5: Compute rewards {ra) for each sampled output o, by running R
Filter out o, and add the remaining to the dynamic sampling buffer (Dynamic Sampling
Equation (11))
buffer size n& < N:
8: continue
9: For each o; in the buffer, compute dre for the 1-th token of o (Equation (9))
10: for iteration = .... do
11:
Update the policy model Ty by maximizing the DAPO objective (Equation (8))
Output n
such as expression, formula and number, which makes it challenging to design comprehensive rules to parse them. To provide accurate reward signals using rules and minimize errors introduced by formula parsers, inspired by AIME, we select and transform the answers into integers, which are
easy to parse. For example, if the original answer is expressed in the form of atvb we instruct the LLM modify the question that the expected answer becomes + b + c. After selection and transformation, we obtained the DAPO-Math- 17K dataset, which consists of 17K prompts, each paired with an integer as the answer.
Experiments
4.1 Training details
In this work, we focus specifically on mathematical tasks to evaluate algorithm, which can be readily transferred to other tasks. We adopt the verl framework [20] for training. We use naive GRPO [38] our baseline algorithm and estimate advantages using group reward normalization.
For hyper- paremeerr,,t,ss, we utilize the Adam W [39] optimizer with a constant learning rate x 10-6 incorporating linear warm-up over 20 rollout steps. For rollout, the prompt batch size is 512 and we sample 16 responses for each prompt. For training, the mini-batch size is set to 512, i.e., 16 gradient updates for each rollout step. For Overlong Reward Shaping, we set the expected maximum length as 16,384 tokens and allocate additional 4,096 tokens as the soft punish cache. Therefore, the maximum number of tokens for generation is set to 20,480 tokens. As for the Clip-Higher

---

AIME avg@33
02-
0.1
wi Dynamic Sampling
Dynamic Sampling
4000
6000
8000
Step
Figure 6: The training progress before and after applying dynamic sampling baseline setting.
Table I: Main results of progressive techniques applied to DAPO
Model
A AIME24 avgd
DeepSeek-R1-Zee---Nvee-3..
47
Naive GRPO
30
+ Overlong Filtering
36
Clip-Higher
38
+ Soft Overlong Punishment
41
Token-level Loss
42
Dynamic Sampling (DAPO)
50
mechanism, set the clipping parameter Slow to 0.2 and éhigh to 0.28, which effectively balance the trade-off between exploration and exploitation. For evaluation AIME, we repeat the evaluation set for 32 times and report avg@ 32 for results stability. The inference hyperparameters of evaluation are set to temperature 1.0 and topp 0.7.
4.2 Main results
Experiments on AIME 2024 demonstrate that DAPO has successfully trained the Qwen-32B model into a powerful reasoning model, achieving performance superior to DeepSeek's 's experiments on Qwen2.5-32B using the RI approach. In Figure observe substantial improvement of perfor- mance on AIME 2024, with accuracy increasing from near 0% to 50%. Notably, this improvement is achieved with only 50% of the training steps required by
We analyze the contributions of each training technique in our methodology, as detailed in Table 1. The observed improvements demonstrate the effectiveness of these techniques in RL training, each contributing several accuracy points in AIME 2024. Notably, given the vanilla GRPO setting, only 30% accuracy can be reached by training from Qwen2.5-32B base model. For token-level loss, although it brings less performance improvement, we find it enhances training stability and makes the length increase more healthily.
When applying Dynamic Sampling, although more data needs to be sampled due to the filtering out of zero- zero-grade gradient data, the overall training time is not significantly affected. As shown in Figure 6, although the number of sampling instances increases, the model's convergence time is even reduced, due to fewer training steps required.
4.3 Training dynamics and Case study
Reinforcement learning large language models an intrinsically complex system challenge characterized by the interdependence of various subsystems. Modifications to any single subsystem can propagate through the system, leading to unforeseen consequences due to the intricate interplay among these components. Even seemingly minor changes in initial conditions, such variations in

---

04
Reward Scooeo
Response
3020
-32
1002
1000
4000
5020
2200
2005
5020
4202
50dz
Training Steps
Steps
(a) Mean response length.
(b) Reward score.
0.70
0.65-
Mean Probabiitty
0.61
Entropy
3.55
0.
3.5
070
0.9
2002
4030
4603
Training Steps
Training Steps
(c) Generation entropy.
(d) Mean probability.
Figure 7: The metric curves of response length, reward score, generation entropy, and the mean probability of DAPO, which show the dynamics of RI. training and serve essential monitoring indicators to identify potential issues.
data and hyperparameters, can amplify through iterative reinforcement learning processes, yielding substantial deviations outcomes. This complexity often confronts researchers with dilemma: even after meticulous analysis and well-founded expectations that modification will enhance specific aspects of the training process, the actual results frequently diverge from the anticipated trajectory. Therefore, monitoring of key intermediate results during experimentation is essential for swiftly identifying the sources of discrepancies and, ultimately, for refining the system.
4.4 Ablation study
We conduct ablation studies for the key hyperparameter of our proposed Clip-Higher technique, Shigh- The results are shown Table 2. The default value of the Shigh is 0.2. We can find that applying Clip-Higher can always achieve a better performance. When the Shigh is set to 0.28 or 0.3, we can achieve 10 points better than the baseline. The ablation experiments are for about 3K steps.
Table 2: Validation results of GRPO w/ Clip-Higher
Shigh
0.2
0.25
0.28
0.3
0.4
AIME24
AIME24mgaa2
32
28.4
30.3
41.8
40.3
37.2
Conclusion
In this paper, we release fully open-sourced system for large-scale LLM RL, including algorithm, code infrastructure, and dataset. The system achieves state-of-the-art large- scale LLM RL perfor- mance (AIME 50 using Qwen-32B pretrained model). We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm, and introduce 4 key techniques to make powerfully effective and efficient in the long-CoT RI. scenario. Additionally, by open-sourcing the training code and dataset, we provide the broader research community and society with practical access scalable reinforcement learning solution, enabling all to benefit from these advancements.

---

References
[1] OpenAl. Learning to reason with Ilm,, 2024.
[2] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 1rX94 2025.
[3] OpenAL. GPT4 technical report. arXiv preprint arXiv:3303.0877// 2023.
[4] Anthropic. Claude 3.5 sonnet, 2024.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1177-1901, 2020.
[6] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[7] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 1oXiv:4412.193377 2024.
[8] XAI. Grok beta the age of reasoning agents, 2024.
[9] Google DeepMind. Gemini 2.0 flash thinking. 2024.
[10] Qwen. Qwq-32b: Embracing the power of reinforcement learning, 2024.
[11] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms arXiv preprint arXiv:2501.12590 2025.
[12] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.151 151111152.111 2024.
[13] Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, et al. An empirical study on eliciting and improving rl like reasoning models. arXiv preprint XXXv:25533.0444. 2025.
[14] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung- Yeung Shum Xiangyu Zhang. Open-reasoneer-zero: An open source approach to scaling reinforcement learning the base 2025.
model. https://gi thub. com/ Open/Open--ae Reasoner- Zewo/Opeeeeeeee Aaero//ppn-Reessneer
[15] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03220 033262 2025.
[16] Ganqu Cui, Lifan Yuan, Zefan Wang. Hanbin Wang, Wendi Li, Bingxiang Hc, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455. 2025.
[17) Jung Hyun Lee, June Yong Yang, Byeongho Heo, Dongyoon Han, and Kang Min Yoo. Token- supervised value models for enhancing mathematical reasoning capabilities large language models. arXiv preprint arXiv:2407.124 12863, 2024.
[18] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Imm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
[19] Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret. arXiv preprint 20010
10

---

Curran [20] Guangming Sheng. Chi Zhang, Zilingfeng Ye. Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rlhf framework. arXiv preprint arXiv:2409. 19256, 2024.
[21] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
[22] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation, 2018.
[23]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages Associates, Inc., 2022.
[24] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016.
[25] Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforce- ment learning with a corrupted reward channel, 2017.
[26] Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of ai ingenuity, 2020.
[27] Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions reinforcement learning: causal influence diagram perspective, 2021.
[28] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022.
[29] Lilian Weng. Reward hacking in reinforcement learning. lilianweng io, Nov 2024.
[30] Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving, 2020.
[31] Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476 482, 2024.
[32] Trieu Trinh and Thang Luong. Alphageometry: An olympiad-leve ai system for geometry, 2024.
[33] AlphaProof and AlphaGeometry Teams. Ai achieves silver-medal standard solving international mathematical olympiad problems, 2024.
[34] Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 14:2111--1122 2022.
[35] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. 2023,
[36] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, 2023.
[37] Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, and Gabriel Synnaeve. Rlef: Grounding code Ilms in execution feedback with reinforcement learning, 2025.

---

[38] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.034: 03300, 2024.
[39] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Conference on Learning Representations, 2019. 2019.
12

---

A Training Dynamics
The Length of Generated Responses a metric closely related to training stability and performance, as shown in Figure 7a. The increase in length provides the model with larger space for exploration, allowing more complex reasoning behaviors to be sampled and gradually reinforced through training. However, is important to note that length does not always maintain continuous upward trend during training. In some considerable periods, it can exhibit a trend of stagnation even decline, which has also been demonstrated in [2]. We typically use length in conjunction with validation accuracy as indicators to assess whether an experiment is deteriorating.
The Dynamics of Reward during training has always been one of the crucial monitoring indicators in reinforcement learning, as shown in Figure 7b. In the majority of our exper- iments, the trend of reward increase is relatively stable and does not fluctuate or decline significantly due to adjustments in experimental settings. This indicates that, given reliable reward signal, language models can robustly fit the distribution of training set. However, we find that the final reward on the training set often exhibits little correlation with the accuracy on the validation set, which indicates overfitting to the training set.
The Entropy of the Actor Model and Generation Probability related to the model's exploration capability and are key metrics that we closely monitor in our experiments. Intuitively, the model's entropy needs to be maintained within appropriate range. An excessively low entropy indicates that the probability distribution is overly sharp, leading to loss of exploration capability. Conversely, an excessively high entropy is often associated with issues of over-exploration such as gibberish and repetitive generation. For the generation probability, the situation exactly the opposite. As demonstrated in Section 3. by applying the Clip-Higher strategy, we effectively addressed the issue of entropy collapse. In subsequent experiments, we find that maintaining a slow upward trend in entropy conducive to the improvement of model performance, shown in Figure 7c and Figure 7d.
B Dataset Transformation
Here is example of data transformation, where demonstrate the original problem, the trans- formed problem, and the model outputs during the transformation process (the original is in markdown text, which has been rendered for readability).
To ensure the model can complete this task with high accuracy, our goal to have it perform comprehensive reasoning while avoiding hallucinations. We achieve this by providing a clear thinking framework that encourages thorough reasoning. Specifically, we employ Chain Thought (CoT) reasoning along with four well-defined steps: extract the answer format, rewrite the problem statement, solve the modified problem and provide an integer as the final answer. For each step, we provide few-shot solutions detailed guidelines guide the model's reasoning.
Through this approach, we have observed that in most cases, the L.LM can generate reformulations with both format and quality that are satisfactory.
13

---

Original Problem
Let and be real numbers such that x² + y² -2r 16y 113 0. Determine the smallest possible value of z. Answer: 11-2v6
Transformed Problem
Let x and y be real numbers such that x² + y²² 22r - 166 + 113 = 0. Determine the smallest possible value of The original answer in the form k myn, where k m, and are integers. Please find the value of + m + n1. Answer: 19
Model's output (Markdown Text)
Transformation Thought Process
The original answer the problem is an algebraic expression 1-26. We need transform this into an integer form. First, note that the answer consists of an integer part and square root term. According the given rule, we replace the integer and square root components with undetermined coefficients. The original problem uses and variables, so we choose other letters as undetermined coefficients. We express the answer in the form myn and require the student to find k + + n.
Answer Format myn
Transformed Problem
Let æ and y be real numbers such that x² + y/2 22r 16y + 113 0. Determine the smallest possible value of The original answer in the form k myn, where k m, and n are integers. Please find the value + m + n.
Answer Calculation for the Transformed Problem The original answer is: 11 2v6.
The required format is: k m./n.
Matching the coefficients:
11, m 2, n = 6.
The final result is: + m + n + 2 19.
Final Answer
19
14

---

C
Case study
Question:
Given that the base ABC of the tetrahedron S ABC is an equilateral triangle, the projection H of point on the face SBC is the orthocenter of ASBC, the dihedral angle H AB C' is 30", and SA 2, find the volume of the tetrahedron. The answer is in the form so give the value of + m.
Response:
Let consider the given problem: We need determine
Next, let S have coordinates (Fo, yo. 20). Since SA 2, the distance formula gives
/ (xa)² (yo)² + (zu)² ) so we can write x yō = 4. Now, remember that is the orthogonal projection of
However, wait moment, let's rethink about the dihedral angle involving planes in more thoughtful geometric way. Consider the plane 01 = ABC, which lies entirely on the xy coordinate plane (as 0). Let the plane 02 SBC. The point A projected perpendicularly to plane a2 lands on H. The line AB
Table 3: Emergence of Reflective Behavior Reinforcement Learning
During the RL training process, we observe an interesting phenomenon: the reasoning patterns the actor model evolve dynamically over time. Specifically, the algorithm not only reinforces existing reasoning patterns that facilitate correct problem-solving but also gradually gives rise to entirely new modes of reasoning that were initially absent. This finding reveals the adaptability and exploration capability of RI. algorithms and offers new insights into the learning mechanisms of the model.
For example, in the early stages of model training, there was virtually no occurrence of checking and reflecting on previous reasoning steps. However, training progresses, the model exhibits distinct behaviors of reflection and backtracking, as shown in Tables and 4. This observation sheds light on further exploration into interpreting the emergence reasoning abilities during RL, which we leave for future research.
D Limitations
This paper tackles the problem of LLM RL to elicit complex reasoning behaviors. However, only single-turn RL is considered, while multi-turn RL is much more powerful and closer to the real-world applications, which worth exploring. Besides, for math problems, the paper focuses only on problems with numerical answers, while more challenging mathmatical problems like proof questions are not considered. We leave multi-turn RL and proof scenario as future work.
15

---

Question:
Among the 900 residents of Aimeville, there 195 who diamond ring, 367 who own a set of golf clubs, and 562 who own a garden spade. In addition, each of the 900 residents owns bag of candy hearts. There 437 residents who exactly two of these things, and 234 residents who own exactly three of these things. Find the number of residents of Aimeville who own all four of these things.
Response:
Step by Step Solution:
1. Let's define the following variables:
Let A be the set of residents who own diamond ring.
Let be the set of residents who own a set of golf clubs.
Let C' be the set of residents who own garden spade.
Let D be the set residents who own a bag of candy hearts. Since all 900 residents own a bag of candy hearts, DI 900.
Let a, denote the number of residents who own exactly out of the items.
Let's use the principle of inclusion-exclusion for the union of 4 sets A, C,C.D.
04 219
54.75.
Since 04, the number of people owning all four items, must be a whole number, our current approach needs to be reconsidered in terms of the combinatorial counting of people owning different combinations of the items.
Instead of directly using the inclusion exclusion principle, we can use a different approach
inclusion-exclusion
involving the number people owning combinations of the items A, B, C, and D. Let x; be the number of people owning exactly item among A, B, C (excluding the item D which everyone owns).
Then the total number of people owning exactly k items among the A, B, C, and D can be broken down:
Table A case of reflective behavior emergence.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the abstract and introduction, we claim that the proposed method deals with some important issues of large-scale L.L RI and achieves state-of-the-an performance
AIME 2024. Major evidence shown in Figure and Table Section 4 provides further details the experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations discussed in Section D.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
This paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All information needed for reproduction can be found in Section 3, Section 4, and Appendix B. The training code and dataset will be fully open-soureed.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibilityy may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The training code and dataset are fully open-sourced and attached in supple- mental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbccccccceecce hepollcc/giieee Codeeubbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training details listed in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All validation accuracies are averaged over 32 trials (avg@32). We also provide pass @32 and cons @:32 in Figure 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: This paper did not provide information about the computer resources needed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htt:://neuurppe cc Apppllic/Elliilpeeee det ines?
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and the research conducted in the paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work focuses on LLM RL algorithms. Thus, there is no foresecable societal impact of it.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
20

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The data used in our work is simply math questions paired with answers. And the model is trained for solving math problems. So there no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our RL framework is built on verl under Apache-2.0 License. Refer to abstract and introduction for more details.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our new assets include open-source training dataset, which is well docu- mented the paper and released in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: This paper proposes an algorithm for LLM RL.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp///een...///eeee cc/Confereecees/2225 /LLM) for what should or should be described.
23


Reviews:
[{"review_id": "0KlnvUOA8g", "reviewer": "Reviewer_ADmi", "summary": "This paper dives into the recipe of large-scale RLHF for LLMs. The authors point out that while many works such as o1 or DeepSeek R-1 do include their RLHF methods in their reports, they all hold out many details, preventing the open source LLM community from reproducing their results. Hence, the authors introduce DAPO, an RL algorithm designed to overcome common issues like instability and entropy collapse in training large language models for complex reasoning tasks. The authors present four novel techniques, i.e., Clip-Higher, Dynamic Sampling, Token-Level Policy Gradient Loss, and Overlong Reward Shaping, that collectively enhance training efficiency and performance. By applying DAPO to a Qwen2.5-32B model, they achieve a state-of-the-art 50% accuracy on the AIME 2024 math benchmark, outperforming prior work in fewer training steps. The authors also commit to the full open-sourcing of the algorithm, training code, and a new 17K-prompt math dataset to improve reproducibility and foster further research in the community.", "questions": "1. For methods except for Clip-Higher, they are generalizable since they are more like engineering improvements than hyperparameter tuning. For Clip-Higher, it looks like this is pure parameter tuning to balance exploration and exploitation on math tasks. It's not clear if it works well in general.\n\n2. In Sec 3.3, i.e., Token-Level Policy Gradient Loss, the authors mentioned that long samples can be either of high-quality, or of \"low-quality patterns such as gibberish and repetitive words\". How does the proposed token-level sampling, which effectively increases long samples' loss weight as a blanket policy, tackle both scenarios at the same time?\n\n3. In Sec 3.4, i.e., Overlong Reward Shaping, I don't quite get why we should by default \"assign a punitive reward to truncated samples\" (L151), which causes this problem in the first place. As far as I can tell, RL methods having a maximum length doesn't necessary mean that samples got truncated due to excessive length gets extra punishiment as long as the discounted reward computation traces back the whole sequence. Also, if one desires longer sequences, wouldn't increase maximum length just solve the problem?", "limitations": "I would argue that the paper lacks novelty in general. The proposed methods sound more like a better implementation of GRPO rather than a new RL method. However, I do agree that a good implemtation is very important and often very critical in RL.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* All techniques developed by the authors are clearly explained with a good motivation.\n* The DAPO system achieves a 50% accuracy on the AIME 2024, surpassing naive GRPO by 20 points with only half amout of training.\n* Ablation studies are well presented to justify the proposed methods.\n\nWeaknesses:\n* Lack of novelty. The paper sounds more like a good tech report showing off parameter tuning and smart engineering efforts rather than an algorithmic contribution.\n* Lack of evaluation on other forms of tasks. It's not clear if the proposed methods would still work for other domains such as code generation or standard VQAs.", "was_revised": true, "final_justification": "Thanks the authors for additional remarks. I am satisfied that the evaluation of the proposed method is no longer an issue. Hence, I decide to raise my rating.", "rebuttal": "We thank reviewer ADmi for the positive comment on our paper, and address all concerns and questions below.\n\nQ1: Lack of novelty\n\n- **Effective tricks are important to make RL actually work, especially in large-scale** **LLM** **training.** Our methods achieve a significant improvement of 20 points on AIME (the core benchmark adopted by OpenAI-O1, DeepSeek R1) and surpass DeepSeek-R1's RL. To date, few other GRPO-based methods can surpass DeepSeek-R1's performance purely using RL in the zero setting. We achieved this, and the effective algorithm details are the keys to making this happen.\n- **Achieving top-tier performance with a simple method is inherently a kind of innovation.** Our conceptually simple method surpasses DeepSeek-level RL performance. Few open research have reached DeepSeek-level RL performance. Not only have we accomplished this, but we have also made all our detailed recipes, code, and data fully open-source, promoting transparency and reproducibility.\n- Clip-Higher: it goes beyond \"hyperparameter-tuning\" and encodes crucial algorithm insights.\n  - \"Clip-Higher\" is a novel approach not previously well-known or utilized by the broader research community. Specifically, few people know that **Clip-Higher can effectively balance exploration and exploitation**. Besides, Clip-Higher’s approach to independently setting upper and lower clip bounds is novel, where traditionally both clip bounds are identical. Despite its simplicity, it demonstrates exceptional effectiveness. We anticipate that our clear demonstration of its efficacy will inspire its adoption and broader use within the research community.\n  - Pure \"hyperparameter-tuning\" has a vast search space and cannot lead to Clip-Higher. Our identification of Clip-Higher is a targeted, insight-driven adjustment rather than random exploration or just \"tuning\". RL systems contain hundreds of hyperparameters that can be tuned. To name a few, just for the entropy problem, temperature, temperature scheduler, entropy bonus weights, entropy loss weights, kl loss weights (kl helps entropy), epsilon-greedy sampling ratios, top-k/top-p threshold... These are just \"hyperparameter tuning\". Clip-Higher is different from these simple tuning and emerges as both novel and remarkably effective, making it an elegant and powerful solution.\n- Additionally, a major contribution of our work is clearly **identifying critical issues in applying RL to** **LLMs** and providing **practical solutions to make the successful application of RL possible**. Our focused analysis, e.g. about entropy, will likely inspire further research in this important area.\n\nQ2: Lack of evaluation and generality\n\n- We further evaluate the 32B model on additional math benchmarks, where DAPO consistently outperforms GRPO. Additionally, we conduct code experiments on LiveCodeBench, demonstrating that DAPO’s effectiveness extends beyond math tasks.\n\n|                  | AIME24 | MATH500 | AMC  | Minerva Math | Olympiad Bench | LiveCodeBench |\n| ---------------- | ------ | ------- | ---- | ------------ | -------------- | ------------- |\n| DAPO Qwen2.5-32B | 52     | 81      | 88   | 34           | 51             | 62            |\n| GRPO Qwen2.5-32B | 30     | 62      | 72   | 17           | 34             | 50            |\n\n- We evaluated DAPO on three additional backbones, two 7B parameter dense models (Qwen2.5-7B and Qwen2.5-Math-7B) and a 200B parameter MoE model with 20B active parameters. DAPO consistently outperforms baselines across all model sizes and architectures.\n\n|        | GRPOQwen2.5-7B | DAPOQwen2.5-7B | GRPOQwen2.5-Math-7B | DAPOQwen2.5-Math-7B | GRPOIn-house 200A20B MOE | DAPOIn-house 200A20B MOE |\n| ------ | -------------- | -------------- | ------------------- | ------------------- | ------------------------ | ------------------------ |\n| AIME24 | 14             | 20             | 28                  | 36                  | 60                       | 80                       |\n\nQ3: Clip-Higher is purely hyperparameter tuning on math tasks and perhaps not generalizable.\n\n- Our analysis is about the entropy dynamics of the RL process and no specific data type assumption is made. Principlely, this is a general RL trick and can generalize. We conduct a quick experiment and show that on coding tasks, clip-higher is also effective. Besides, we argue that clip-higher together with its superior performance encode algorithmic insights and the related details can be found in Q1.\n\n|                                    | LiveCodeBench |\n| ---------------------------------- | ------------- |\n| DAPO Qwen2.5-32B (only Clip-Higer) | 56            |\n| GRPO Qwen2.5-32B                   | 50            |\n\nQ4: How token-level loss tackle both scenarios.\n\n- The RL training objective has two types of effects: it either encourages (increases likelihood) or punishes (decreases likelihood). Positive samples are encouraged, while negative samples are punished. When applying token-level loss, longer samples naturally receive greater weight; thus, positive samples are encouraged more strongly, and negative samples face increased punishment, significantly reducing their likelihood. This means token-level loss can naturally tackle both scenarios.\n\nQ5: Why is a punitive overlong reward by default? If one desires longer sequences, wouldn't increase maximum length just solve the problem?\n\n- Regarding your second point, autoregressive LLM generation is typically the primary time bottleneck of the whole system. Therefore, in practice, it is necessary to impose a maximum generation length during the rollout phase to ensure efficiency.\n- The default punitive strategy for overly long responses is common practice in verifier-based LLM RL, exemplified by DeepSeek R1. Such methods use accuracy rewards, giving positive feedback only if the response is correct and is validated via robust, rule-based verification (see Sec 2.2.2 of DeepSeek R1). Overlong samples cannot pass such rule-based verification, because they cannot give a correct answer."}, {"review_id": "sE5TcX7oPB", "reviewer": "Reviewer_fAi3", "summary": "The paper introduces DAPO, an open-source RL training techniques in LRMs. The authors propose four key techniques: Clip-Higher, Dynamic Sampling, Token-Level Policy Gradient Loss, and Overlong Reward Shaping. Their implementation achieves impressive results on the AIME 2024 benchmark, scoring 50 points with the Qwen2.5-32B base model.", "questions": "- Why doesn't Figure 4 correlate entropy collapse with model performance metrics? This connection would provide critical insight into how entropy dynamics affect reasoning capabilities.\n- The paper emphasizes the epsilon lower bound's importance in Clip-Higher but provides no experimental analysis of its effects.\n- Additionally, in Figure 6, the baseline and proposed method are trained for different numbers of steps, with the baseline appearing nearly converged - how does this represent a fair comparison of both methods at their optimal performance points?", "limitations": "Yes", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n- The authors provide fully open-source code with straightforward, easily reproducible techniques. \n- The paper is well-written and easy to understand.\n\n### Weaknesses\n- The evaluation relies primarily on AIME 2024 test set, which contains only 30 problems. This narrow benchmark scope creates significant uncertainty about whether the method's success is genuinely robust or merely coincidental. Testing on a single model backbone (Qwen2.5-32B) further compounds this concern about reproducibility across different model architectures.\n- The four proposed techniques appear to be incremental adjustments rather than fundamental methodological advances:\n1) The Clip-Higher approach is essentially parameter tuning of existing clipping mechanisms\n2) The Dynamic Sampling method addresses a known issue (all-zero or all-one samples providing little training signal) but doesn't solve the underlying exploration challenge\n3) Both the Token-Level Policy Gradient Loss and Overlong Reward Shaping techniques are essentially simple penalties or balancing mechanisms for sentence length rather than novel approaches to the core reinforcement learning problems\n- The paper fails to provide comprehensive information about the computational resources required for training. This omission significantly hinders reproducibility efforts, as other researchers cannot accurately assess the hardware requirements or computational budget necessary to implement this approach.", "was_revised": true, "final_justification": "Based on the current description, the contribution on the methodology side mainly comes from the dynamic filtering and the clip-higher strategy. Consequently, I maintain that **the novelty is not sufficiently strong in its current form**, despite the soundness of combining many tricks to boost the GRPO performance. Currently, I maintain my current rating as the reasons to reject outweigh the reasons to accept, with **the main limitations being the strength of the novelty claim and the credibility of empirical evidence**.", "rebuttal": "We thank reviewer fAi3 for reviewing our paper, and address all concerns and questions below.\n\nQ1: Evaluation of more datasets and model backbones.\n\n- We further evaluate the 32B model on additional math benchmarks, where DAPO consistently outperforms GRPO. Additionally, we conduct code experiments on LiveCodeBench, demonstrating that DAPO’s effectiveness extends beyond math tasks.\n\n|                  | AIME24 | MATH500 | AMC  | Minerva Math | Olympiad Bench | LiveCodeBench |\n| ---------------- | ------ | ------- | ---- | ------------ | -------------- | ------------- |\n| DAPO Qwen2.5-32B | 52     | 81      | 88   | 34           | 51             | 62            |\n| GRPO Qwen2.5-32B | 30     | 62      | 72   | 17           | 34             | 50            |\n\n- We evaluated DAPO on three additional backbones, two 7B parameter dense models (Qwen2.5-7B and Qwen2.5-Math-7B) and a 200B parameter MoE model with 20B active parameters. DAPO consistently outperforms baselines across all model sizes and architectures.\n\n|        | GRPOQwen2.5-7B | DAPOQwen2.5-7B | GRPOQwen2.5-Math-7B | DAPOQwen2.5-Math-7B | GRPOIn-house 200A20B MOE | DAPOIn-house 200A20B MOE |\n| ------ | -------------- | -------------- | ------------------- | ------------------- | ------------------------ | ------------------------ |\n| AIME24 | 14             | 20             | 28                  | 36                  | 60                       | 80                       |\n\nQ2: Novelty & Clip-higher is hyperparameter tuning\n\n- **Effective tricks are important to make RL actually work, especially in large-scale** **LLM** **training.** Our methods achieve a significant improvement of 20 points on AIME (the core benchmark adopted by OpenAI-O1, DeepSeek R1) and surpass DeepSeek-R1's RL. To date, few other GRPO-based methods can surpass DeepSeek-R1's performance purely using RL in the zero setting. We achieved this, and the effective algorithm details are the keys to making this happen.\n- **Achieving top-tier performance with a simple method is inherently a kind of innovation.** Our conceptually simple method surpasses DeepSeek-level RL performance. Few open research have reached DeepSeek-level RL performance. Not only have we accomplished this, but we have also made all our detailed recipes, code, and data fully open-source, promoting transparency and reproducibility.\n- Clip-Higher: it goes beyond \"hyperparameter-tuning\" and encodes crucial algorithm insights.\n  - \"Clip-Higher\" is a novel approach not previously well-known or utilized by the broader research community. Specifically, few people know that **Clip-Higher can effectively balance exploration and exploitation**. Besides, Clip-Higher’s approach to independently setting upper and lower clip bounds is novel, where traditionally both clip bounds are identical. Despite its simplicity, it demonstrates exceptional effectiveness. We anticipate that our clear demonstration of its efficacy will inspire its adoption and broader use within the research community.\n  - Pure \"hyperparameter-tuning\" has a vast search space and cannot lead to Clip-Higher. Our identification of Clip-Higher is a targeted, insight-driven adjustment rather than random exploration or just \"tuning\". RL systems contain hundreds of hyperparameters that can be tuned. To name a few, just for the entropy problem, temperature, temperature scheduler, entropy bonus weights, entropy loss weights, kl loss weights (kl helps entropy), epsilon-greedy sampling ratios, top-k/top-p threshold... These are just \"hyperparameter tuning\". Clip-Higher is different from these simple tuning and emerges as both novel and remarkably effective, making it an elegant and powerful solution.\n- Additionally, a major contribution of our work is clearly **identifying critical issues in applying RL to** **LLMs** and providing **practical solutions to make the successful application of RL possible**. Our focused analysis, e.g. about entropy, will likely inspire further research in this important area.\n\nQ3: Dynamic sampling addresses a known issue but doesn't solve the underlying exploration challenge.\n\n- We argue that few LLM reasoning RL papers pointed this out or addressed this problem. We are the first to provide an effective solution to address this problem, which is shown to speed up training and improve performance.\n- The exploration challenge is addressed by Clip-higher. Clip-higher presents a clearly effective solution to this challenge. (shown in Figure 2, it raises entropy and performance). Dynamic sampling is not designed to address the exploration challenge.\n\nQ4: Token-Level PG Loss and Overlong Reward Shaping are not novel approaches to the core RL problems\n\n- We acknowledge that these two techniques are not novel. However, the goal of our paper is not to propose an \"entirely new method for the core RL\". Instead, we aim to demonstrate how RL can be successfully applied to significantly enhance LLM reasoning practically, surpassing state-of-the-art DeepSeek-R1 level RL performance. Through our experiments, we identify them as crucial components for achieving this level of performance and transparently share these findings with the research community. Note that the community continues to face challenges in replicating DeepSeek-R1-level RL results, underscoring the practical value of our contributions.\n\nQ5: Information of computational resources required for training\n\n- The computation resource cost is 128 H100 GPUs for one week. \n\nQ6: Correlation between entropy collapse and model performance:\n\n- We show the correlation in Figure 2. We can clearly see that for the baseline trial, the entropy collapses to its minimum at around 750 steps, and the peak performance is achieved also at round 750 steps.\n\nQ7: Epsilon lower bound's importance is emphasized but no experimental analysis\n\n- Our proposed method is Clip-Higher and is about the epsilon upper bound. The epsilon lower bound is not our focus and we just keep it as it is.\n\nQ8: In Figure 6, the proposed method and baseline are trained for different number of steps, where the baseline is converged.\n\n- Our method surpasses the converged baseline using significantly fewer training steps, clearly demonstrating its superior efficiency and performance."}, {"review_id": "00MXqRD95w", "reviewer": "Reviewer_Q4hn", "summary": "This paper, DAPO, explores the use of Reinforcement Learning (RL) to enhance the reasoning capabilities of Large Language Models (LLMs). The work is motivated by the observation that previous studies often fail to adequately detail the specific methodologies that contribute to their models' success. The primary contribution of DAPO is the introduction of several optimizations to the standard GRPO baseline, which are claimed to yield significant performance improvements.\n\nThese optimizations include:\n\n*   **Asymmetric Clipping:** Utilizing separate low and high clipping parameters within the GRPO loss function.\n*   **KL Divergence Removal:** Eliminating the KL divergence term from the GRPO loss.\n*   **Dynamic Sampling:** During training, filtering out samples within a batch that have already reached a consensus, as these samples do not contribute to the advantage calculation.\n*   **Token-wise Loss Aggregation:** Calculating the loss at the token level directly, rather than first aggregating it per sample and then across the batch.\n*   **Soft Overlength Penalty:** Excluding hard-truncated samples from the loss calculation and applying a graduated penalty to samples as they approach the maximum length limit.", "questions": "- Could you provide more details on the data source? This information is crucial for assessing potential data contamination and for enabling other researchers to fully understand the experimental setup.\n- Could you offer a more detailed justification for your experimental design, particularly for the training protocol and ablation studies? For instance, how many experimental runs were conducted to validate the utility of each technique, and what were the specific criteria for terminating a training run?\n- Could you support the claim that removing KL divergence helps? I can't find any quantitative result about that in the paper.", "limitations": "I think limitations could be better addressed in the paper, see Weaknesses.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "#### **Strengths**\n\n1.  The paper pragmatically tackles an important and challenging problem. It methodically tests intuitive ideas for improving reasoning performance and supports its findings with corresponding ablation studies.\n2.  The paper's commitment to making effective RL training techniques for LLMs more accessible is commendable and a significant benefit to the research community.\n3.  The choice of AIME, a challenging mathematical reasoning benchmark, strengthens the paper's contributions, especially given the significant performance gains reported.\n4.  The use of standard metrics like avg@32, cons@32, and pass@32 ensures comparability with other work in the field. However, the paper would be improved by explicitly defining these metrics for readers less familiar with this area.\n\n#### **Weaknesses**\n\n1.  The evaluation is limited to a single benchmark (AIME). The authors also appear to have tuned hyperparameters directly on this benchmark's evaluation score, which raises concerns about the generalizability of the proposed techniques.\n2.  The paper lacks a comprehensive discussion of its limitations. Aside from a brief paragraph in Appendix D, the broader constraints and potential negative results of the approach are not sufficiently addressed.\n3.  Given the paper's goal of promoting reproducibility, the omission of details regarding computational resources (e.g., hardware, budget, runtime) is a notable weakness. This is particularly concerning as the authors answered \"No\" to the corresponding checklist question 8 about compute.\n4.  The reliance on a single random seed for most ablations and plots is a significant limitation. While the high cost of these experiments is understandable, the paper should explicitly acknowledge it and weaken the claims accordingly, or provide additional data points.\n5.  The authors appear to have misinterpreted Question 16 of the checklist, which asks about the *use* of LLMs in preparing the manuscript, not whether LLMs are the paper's *topic*.", "was_revised": true, "final_justification": "I believe the paper addresses an important problem and contributes to democratizing practical techniques for improving LLM RL training by committing to open-source. While the proposed improvements are not particularly novel or exotic, they provide a straightforward way to boost performance. These insights, in practice, can save substantial GPU-hours and thus hold significant value for many practitioners. The authors have also addressed the limitation that the original evaluation relied on only a single dataset. In addition, they have committed to clarifying the compute resources used and have justified the use of a single seed due to computational cost constraints. For all these reasons, I maintain my original score and recommend acceptance of the paper.", "rebuttal": "We thank reviewer Q4hn for the positive comment on our paper, and address all concerns and questions below.\n\nQ1: More Evaluation and Generality\n\n- We further evaluate the 32B model on additional math benchmarks, where DAPO consistently outperforms GRPO. Additionally, we conduct code experiments on LiveCodeBench, demonstrating that DAPO’s effectiveness extends beyond math tasks.\n\n|                  | AIME24 | MATH500 | AMC  | Minerva Math | Olympiad Bench | LiveCodeBench |\n| ---------------- | ------ | ------- | ---- | ------------ | -------------- | ------------- |\n| DAPO Qwen2.5-32B | 52     | 81      | 88   | 34           | 51             | 62            |\n| GRPO Qwen2.5-32B | 30     | 62      | 72   | 17           | 34             | 50            |\n\n- We evaluated DAPO on three additional backbones, two 7B parameter dense models (Qwen2.5-7B and Qwen2.5-Math-7B) and a 200B parameter MoE model with 20B active parameters. DAPO consistently outperforms baselines across all model sizes and architectures.\n\n|        | GRPOQwen2.5-7B | DAPOQwen2.5-7B | GRPOQwen2.5-Math-7B | DAPOQwen2.5-Math-7B | GRPOIn-house 200A20B MOE | DAPOIn-house 200A20B MOE |\n| ------ | -------------- | -------------- | ------------------- | ------------------- | ------------------------ | ------------------------ |\n| AIME24 | 14             | 20             | 28                  | 36                  | 60                       | 80                       |\n\nQ2: Details regarding computational resources\n\n- The computation resource cost is 128 H100 GPUs for one week. \n\nQ3: More discussion of limitations & A single random seed is a limitation.\n\n- Thank you for kindly raising this point. We will expand our discussion on limitations as follows:\n  - The outcome reward currently provides minimal guidance on the reasoning process, resulting in inefficient and prolonged reasoning chains. Future research could explore more detailed process-based reward designs to potentially shorten reasoning length.\n  - The main experiments were conducted using only a single random seed due to the substantial computational resources required for large-scale LLM RL experiments. We acknowledge this as an important limitation.\n  - Regarding safety, the existing verifier-based RL focuses solely on answer correctness without supervising model output safety. Future work could address this by incorporating both outcome verification and a general safety reward model.\n\nQ4: Misinterpretation of checklist question 16:\n\n- Thank you for pointing out this oversight :). We will correct it and notice it next time.\n\nQ5: Could you provide more details on the data source?\n\n- We crawl all questions from the AOPS (https://artofproblemsolving.com/) website and conducted a thorough decontamination of evaluation sets.\n\nQ6: How many experimental runs were conducted to validate the utility of each technique, and what were the specific criteria for terminating a training run?\n\n- Given the high computational cost, we performed only one run per technique for validation. Training runs were terminated upon reaching performance saturation.\n\nQ7: Could you support the claim that removing KL divergence helps?\n\nWe run comparison experiments on Qwen2.5-Math-7B for 3000 steps. The results show w/o KL achieves better performance. Besides, the w/ KL setting already saturates, yet the w/o KL setting achieves better performance and the performance has not saturated.\n\n|             | AIME24 |\n| ----------- | ------ |\n| GRPO w/o KL | 28     |\n| GRPO w/ KL  | 26     |"}, {"review_id": "JkfO9xzhy1", "reviewer": "Reviewer_ZAAH", "summary": "This paper proposes several technical tricks to improve GRPO and collects a new dataset for training. These tricks are insightful and effective (in their experiments). The writing is easy to follow. However, their experimental comparisons are not sufficient.", "questions": "1, While the authors solely rely on AIME2024 to assess the performance of their method, I believe that a more comprehensive evaluation would benefit from comparisons across additional benchmarks, such as AMC, MATH500, Minerva Math, and OlympiadBench.\n\n2, The authors conduct experiments with only Qwen2.5-32B. How about Qwen2.5 models in other sizes?\n\n3, The authors claim that their method outperforms the previous SOTA DeepSeek-R1-Zero-Qwen-32B using 50% training steps. My concern is that not all steps are equal. The authors do not mention hyper-parameters like dataset size, batch size and gradient accumulative steps. Therefore, such comparison of training steps can be unfair.\n\n4, The authors propose to remove KL divergence in section 2.3. But they do not include the ablation study of removing KL divergence in Table 1.\n\n5, In my understanding, $\\epsilon_{high}$ restricts the increase of r when A > 0, while $\\epsilon_{low}$ restricts the decrease of r when A < 0. I recommend the authors to add this in third paragraph because it is not so straightforward to understand why the authors only mention the situation of A > 0. \n\n6, I agree with the idea of filtering out prompts with the accuracy equal to 1 and 0. In my opinion, they exclude part of samples from calculating gradients, but these samples are involved in generating completions. Therefore, when comparing training efficiency in Figure 6, I think the authors should also compare the number of samples for which the model generates completions.\n\n7, For token-level policy gradient loss, in Equation 12, the length of each completion $\\left| o_i \\right |$ might vary among different batches as $\\left| o_i \\right |$ will increase with the training steps. As a result, the weight of each token will be different among different batches. How about setting $\\left| o_i \\right |$ to a proper constant?", "limitations": "Please refer to the questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "strengths:\n\n1, This paper focuses on GRPO which is a cutting-edge research direction.\n\n2, This paper proposes insightful and effective technical tricks to improve GRPO.\n\nweaknesses:\n\n1, The experimental comparison are insufficient. \n\n2, Some expressions can be improved to enhance clarity. The detailed comments can be found in questions.", "was_revised": true, "final_justification": "I am satisfied with the methods proposed in this paper. Although it looks like multiple tricks, I think useful tricks are important in RL area. But I think reviewer fAi3 's concern \"with the unexpected low performance of the GRPO-32B baseline\" should be paid attention to.", "rebuttal": "We thank reviewer ZAAH for the positive comment on our paper, and address all concerns and questions below.\n\nQ1: Lack of Evaluation:\n\n- We further evaluate the 32B model on additional math benchmarks, where DAPO consistently outperforms GRPO. Additionally, we conduct code experiments on LiveCodeBench, demonstrating that DAPO’s effectiveness extends beyond math tasks.\n\n|                  | AIME24 | MATH500 | AMC  | Minerva Math | Olympiad Bench | LiveCodeBench |\n| ---------------- | ------ | ------- | ---- | ------------ | -------------- | ------------- |\n| DAPO Qwen2.5-32B | 52     | 81      | 88   | 34           | 51             | 62            |\n| GRPO Qwen2.5-32B | 30     | 62      | 72   | 17           | 34             | 50            |\n\n- We evaluated DAPO on three additional backbones, two 7B parameter dense models (Qwen2.5-7B and Qwen2.5-Math-7B) and a 200B parameter MoE model with 20B active parameters. DAPO consistently outperforms baselines across all model sizes and architectures.\n\n|        | GRPOQwen2.5-7B | DAPOQwen2.5-7B | GRPOQwen2.5-Math-7B | DAPOQwen2.5-Math-7B | GRPOIn-house 200A20B MOE | DAPOIn-house 200A20B MOE |\n| ------ | -------------- | -------------- | ------------------- | ------------------- | ------------------------ | ------------------------ |\n| AIME24 | 14             | 20             | 28                  | 36                  | 60                       | 80                       |\n\nQ2: Not equal comparison w/ DeepSeek-R1-Zero-Qwen-32B\n\n- Our dataset size and batch size are presented in Section 4.1. We acknowledge that the comparison cannot be fairly apple-to-apple because DeepSeek didn't reveal any training details or recipes, while just know them train for 10000 steps from DeepSeek R1's paper. But our end2end system performance beats DeepSeek-R1-Zero-Qwen-32B.\n\nQ3: KL Divergence\n\n- We run comparison experiments on Qwen2.5-Math-7B for 3000 steps. The results show w/o KL achieves better performance. Besides, the w/ KL setting already saturates, yet the w/o KL setting achieves better performance and the performance has not saturated.\n\n|             | AIME24 |\n| ----------- | ------ |\n| GRPO w/o KL | 28     |\n| GRPO w/ KL  | 26     |\n\nQ4: More clarification about $\\epsilon_{high}$ and A > 0\n\n- Thank you for your suggestion! We will edit and update our paper to clarify it.\n\nQ5: Consider the number of samples generated when comparing training efficiency in Figure 6\n\n- That's an insightful question! Below, we provide a comparison of training efficiency in terms of both generated samples and end-to-end wall time:\n  - Number of Generated Samples: The baseline achieves 42 points after 6080 training steps, while the dynamic sampling approach achieves 43.75 points in just 2320 steps. On average, dynamic sampling generates batches twice the size of the baseline per step. Thus it effectively uses fewer total samples (2320 × 2 = 4640 batches vs. 6080 batches) while achieving superior performance.\n  - Wall Time: With 128 GPUs, the baseline takes approximately 4 days to reach 42 points. In contrast, dynamic sampling reaches 43.75 points in about 2 days, significantly improving efficiency.\n- We have another insight to share, about this substantial reduction in wall time. Practically, sample generation is often bottlenecked by long-tail samples, and increasing the number of samples introduces less additional overhead. Our generation engine uses the Orca scheduler [3], which replaces finished samples with new samples dynamically. Thus, during the generation of the longest sample, several samples could have been generated. In our setting with 128 GPUs, increasing the RL batch size threefold (from 512 to 1536) resulted in only a 1.5x increase in generation wall time.\n\n|                     | AIME score | Num. of generated samples                             | Wall Time (128 H100) |\n| ------------------- | ---------- | ----------------------------------------------------- | -------------------- |\n| Baseline            | 42         | 6080 (step) * batch                                   | 2 days               |\n| w/ dynamic sampling | 43.8       | 2320 (step) * 2 (dynamic times) * batch= 4640 * batch | 4 days               |\n\nQ6: Using a constant as the denominator to normalize loss across batches.\n\n- Empirically, our concurrent work Dr. GRPO [1] proposed to use a constant as the loss denominator and they use the number of maximum tokens. We have tried that and it doesn't work, it even hurt the performance.\n\n|        | GRPOQwen2.5-7B | Dr GRPOQwen2.5-7B | DAPOQwen2.5-7B |\n| ------ | -------------- | ----------------- | -------------- |\n| AIME24 | 14             | 16                | 20             |\n\n- Theoretically, this operation is not grounded and arguable. In the original PPO paper [2], the objective (Equation 1 and 6 of [2]) is expectation over time step t (actions or tokens in the context of LLM). Thus, for each minibatch, we should weigh the loss by dividing it by the number of tokens. This is expectation over time step t.\n\n[1] Understanding R1-Zero-Like Training: A Critical Perspective (https://arxiv.org/pdf/2503.20783)\n\n[2] Proximal Policy Optimization Algorithms (https://arxiv.org/pdf/1707.06347)\n\n[3] Orca: A Distributed Serving System for Transformer-Based Generative Models (https://www.usenix.org/conference/osdi22/presentation/yu)"}]

Author rebuttal:

Meta-review: The paper introduces DAPO, a practical recipe for large-scale RLHF in LLMs. It proposes four techniques—Clip-Higher, Dynamic Sampling, Token-Level PG Loss, and Overlong Reward Shaping—to improve GRPO training stability and efficiency. Applied to Qwen2.5-32B, DAPO achieves higher accuracy on AIME 2024, surpassing prior work. The code, data and models are all open-source.

This paper has clear exposition of practical RL training improvements, strong empirical results on AIME and additional math/coding benchmarks and significant efficiency gains (fewer steps, better stability). Full open-sourcing commitment enhances reproducibility and community impact.

However, the novelty is somewhat limited, as most techniques resemble parameter tuning or known adjustments. Baseline GRPO results appear weaker than expected, raising questions on fairness of comparisons. Evaluation is heavily math-centric, broader tasks and multiple seeds are missing. And the compute costs and limitations initially underreported. The reviewers raised or maintained the score during the rebuttal.

Despite limited novelty, DAPO makes a useful, reproducible, and empirically strong contribution to RL for LLM reasoning. It can be accepted as a practical paper.
Decision: Accept (poster)