CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models
Authors: ZhiHang Lin
Mingbao Lin
Yuan Xie
Rongrong Ji
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Reasoning Models
Efficient Training
Primary area: deep_learning
TL;DR: This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models based on Group Relative Policy Optimization (GRPO).

Abstract: This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models based on Group Relative Policy Optimization (GRPO). GRPO, while effective, incurs high training costs due to the need to sample multiple completions for each question. Our experiment and theoretical analysis reveal that the number of completions impacts model accuracy yet increases training time multiplicatively, and not all completions contribute equally to policy training---their contribution depends on their relative advantage. To address these issues, we propose CPPO, which prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates. Additionally, we introduce a dynamic completion allocation strategy to maximize GPU utilization by incorporating additional questions, further enhancing training efficiency. Experiments show that CPPO achieves up to $7.98\times$ speedup on GSM8K and $3.48\times$ on Math while preserving or even enhancing the accuracy compared to the original GRPO. We release our code at https://github.com/lzhxmu/CPPO.

CPPO: Accelerating the Training of Group Relative Policy Optinization-Based Reasoning Models
Zhihang Lin Mingbao Lin?, Yuan Xie
Rongrong Ji
Key Laboratory of Multimedia Trusted Perception and Efficient Computing,
Ministry of Education of China, Xiamen University, 361005, P.R. China
SShnghai Innovation Institute, China
Rakuten,, Singapore
'East China Normal University, Shanghai, China
Izhedu @ foxmail. linmb001 @ @outlook.com, yxie wss.ecnu.edu.cm riji amu.edu.cn
Abstract
This paper introduces Completion Pruning Policy Optimization (CPPO) to acceler- ate the training of reasoning models based on Group Relative Policy Optimization (GRPO). GRPO, while effective, incurs high training costs due to the need to sam- ple multiple completions for each question. Our experiment and theoretical analysis reveal that the number of completions impacts model accuracy yet increases train- ing time multiplicatively, and not all completions contribute equally to policy training-their contribution depends issues, we propose CPPO, which prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates. Additionally, we introduce a dynamic completion allocation strategy to maximize GPU utilization by incorporating additional questions, further enhancing training efficiency. Experiments show that CPPO achieves up to 7.98x speedup on GSM8K and 3.48x on Math while preserving the original GRPO. We release our code at https:/piitub.com//axxx///CPC
their relative advantage. To address these
enhancing the accuracy compared
Introduction
Recently, there has been surge in the development and application of advanced reasoning models, with models such as OpenAl-ol [T]. Deepseek-RI (7). and Kimi-k1.5 25 being prime examples. These models exhibit remarkable capability in complex reasoning tasks, such mathematics, coding, and scientific reasoming through step-by-step inference and reflection.
Reinforcement learning has been proven to be effective method for training reasoning models. Deepseek-RI R1 7 demonstrates that reasoning patterns can be effectively elicited through rule-based reinforcement learning. It employs Group Relative Policy Optimization (GRPO) (21) which differs from Proximal Policy Optimization (PPO) 20 by estimating the baseline directly from group scores, eliminating the need for critic model. However, this necessitates sampling group of completions for each question, rendering the training process computationally expensive. Subsequently, GRPO computes the reward for each completion using a based reward function and calculates the relative advantage of each completion. To ensure training stability, GRPO also calculates the ratio of the predicted probabilities of the policy model, reference model, and old policy model for group of completions as part of the policy objective function, further increasing the training overhead of reinforcement learning. The substantial training overhead of GRPO limits training efficiency and scalability. Improving the training efficiency is an important and practical problem.
Prooecc Leader
'Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The computational expense of GRPO training primarily stems from its core design: generating large group of completions per prompt for intra-group comparison, which makes the training process computationally expensive. Moreover, the forward computation of GRPO scales by a factor of (3x) completion number. is natural to question whether the contribution of each completion to the reinforcement learning process is equal. In Sec. we find that the contribution of each completion is related to its relative advantage. In other words, the contribution of each completion to the policy model training is not equal. This insight inspires us to accelerate GRPO by pruning completions.
In this paper, we propose Completion Pruning Policy Optimization (CPPO) to accelerate Group Relative Policy Optimization (GRPO). Given that each completion's 's contribution to the reinforcement learning process varies significantly and is closely related its relative advantage, our CPPO prunes completions based on advantage, thereby accelerating the reinforcement learning process. Specifically, the policy model initially samples a group of completions for each question. Subsequently, the relative advantage of each completion is computed via the reward function. CPPO then prunes completions with low absolute advantage values, retaining only those with high absolute advantage for loss computation. This process considerably reduces the number of completions needed for training, thus speeding up the training process. Moreover, we observe underutilized GPU resources due to completion pruning, leading to resource waste. To tackle this, we introduce dynamic completion allocation strategy that fills each device with completions from new questions, fully utilizing GPU resources and further enhancing training efficiency.
We have conducted experiments on multiple challenging benchmarks and models of different scales to evaluate CPPO's effectiveness. Specifically, we train the Qwen-2.5 series models [29]. such as 2.5....5-...---neee
and Qwen-2.5- 2.5-7...--- on math datasets including Math { and GSM8K 4 The results demonstrate that CPPO achieves up to 7.98x speedup on GSM8K and 3.48x on Math while preserving or even enhancing the accuracy compared to the original GRPO.
2 Related Work
Large Scale Reasoning Models. Large Language Models (LLM) 2 24 2 have made impressive progress in various natural language processing tasks. Recently, researchers have continued boost the performance of large language models in reasoning tasks, such as mathematics [.,) coding [12] and scientific reasoning [1). Snell er al. (23 use dense, process- based verifier reward models and adaptively update the model's response distribution based on the test-time prompt to enhance reasoning ability. rStar-Math (6) proposes a self-evolved deep thinking approach that significantly boosts the math reasoning capabilities of small LLMs. OpenAl-ol [I]] uses large scale reinforcement learning to train a reasoning model that can solve complex reasoning tasks, achieving state-of-the-ar performance multiple benchmarks. However, the training details of OpenAI-ol have not been released, making it difficult to replicate and expand the reasoning model.
Reinforcement Learning. Recently, DeepSeek-R I 77] has incentivized the reasoning capability of large language models through Group Relative Policy Optimization. Inspired by DeepSeek-Re -R'Seekk-''ss 's success, Lo (28) adopts the REIFORCE++ algorithm to enhance the training efficiency and stability of ruse-based reinforcement learning. Hu et al. [10] demonstrate that the vanilla PPO algorithm, without KL divergence constraint, is sufficient to scale up both response length and benchmark performance on reasoning tasks. Nevertheless, these reinforcement learning algorithms universally require multiple completions for each question, resulting in substantial computational costs. There is an urgent need to accelerate the training of reinforcement learning algorithms.
Inference Acceleration for Reasoning Models. The enhancement of model inference capabilities is often accompanied by increased computational overhead and longer response times. Recent works have attempted to accelerate the inference process of reasoning models through efficient Chain of Thought (CoT) methods. TokenSkip [27] proposes a controllable CoT compression method that improves reasoning efficiency by selectively skipping less important tokens while preserving critical ones, thus achieving a balance between efficiency and accuracy. Kang et al. [3] utilize a compressor to condense an original longer CoT into a shorter one while maintaining key information and interpretability. Although numerous works focus on inference acceleration, the acceleration of reasoning model training remains an underexplored area.

---

76.3%%
7155% 20006
1000
Canplanes Nanbir
Complation Nanber
Figure I: Completion number vs. (left) accuracy and (right) training time. Experiments are conducted using Qwen2.5-1.5B-Instru 299
on GSM8K 4
3 Method
3.1
Preliminary
Group Relative Policy Optimization. GRPO 7 foregoes the critic model that is typically the same size the policy model and estimates the baseline from group scores instead. Specifically, for each question 9 sampled from the dataset distribution P(Q). GRPO generates G completions {oi, 0g. -. oc using the old policy model Ta.a And then GRPO optimizes the policy model by maximizing the following objective:
[0:
To (oloi,ll,,00 0i, Ais
(O) EEp-P(O)aa) To (ol0)
min
Oi
Toaia (0s, Os,ct)
clip( T0
=5DK (TTO Tres [ 1}
c, c, c)4,
(1)
To Te (onelg 0x,20))
where
DeL Tres| Tnaf(Oi,el 0i,ce)
log Traf(on.rly, 05.)))
(2)
7o(or,el9,0;
To To(0i,19,00,1 0i,st)
Here, c and B are hyperparameters. Tref the reference model which is usually the initial model before reinforcing learning. And A; is the advantage computed using a group of rewards (ri, ,,, ra} corresponding to the completions within each group:
A T; -mean (oraan((1,,0)) (ri,r,,...,ra))) rra))
std((r1, ra))..
Rule-based Reward Function. Instead of training an additional reward model for reward computa- tion, GRPO employs rule-based reward system that consists of components:
Rformat (ox) Raccuracy(O))
Here, the format reward Riommar (o:) ensures that the output adheres to the expected structure, while the accuracy reward Raceurccy(0) (o:) prioritizes correctness with higher reward for accurate responses. The specific reward functions are presented in Appendix A
Analyzing Completion Impact on Policy Training. From Eq. T.. GRPO's training overhead scales linearly with the number of completions sampled per question. This arises from the necessity of calculating predicted probabilities for the policy, reference, and old policy models over all completions. For instance, in DeepSeek-Math (21), using 64 completions requires 192 forward passes per question (64x3), incurring significant computational costs. This raises two critical questions: (1) How does the number of completions affect policy model accuracy? Does increasing completions always enhance performance? (2) Do all completions in a group contribute equally to training?
To address the first question, conduct an ablation study on GSM8K A using Qwen2.5-1.5B.
Instruct (29]. Results Figure show that model accuracy improves with more completions, but training time grows multiplicatively. This indicates diminishing returns performance gains as

---

training costs increase. Crucially, reducing completions to cut costs risks degrading reasoning capabilities, making it impractical.
For the second question, we investigate whether completions contribute uniformly to training effec- tiveness. Our comprehensive analysis in See. S.c.3.2 reveals that completion contributions are highly variable, with some samples providing significantly more training signals than others. These findings motivate the development of strategies to identify and prioritize high-value completions, potentially improving training efficiency without compromising model performance.
3.2 Completion Contribution Analysis
To measure the contribution of each completion to the policy model training, we first compute the derivative of the policy objective function in Eq. (I with respect to the model parameters 0 as:
To (or,lg, oi,st) As
VoJenro (0) =B -aopp000))P(p-PP000000 (1)0
0
03
Toni (0t, 11 0i,
Tref (0ne19, 0r,ct)
Tres (0 0c,ct)
3
Vo
V log
To (0,, 01,(()
(0t,e4., 05,21)
Vono (0 0i,c)) A; Te
05
Tre/ (or,l9, 01 VOTO (0,, o1,c()
VoTo (0r,el9, 01,51))
oy (oi,19, o,))))
To (0;,e 14,1111000 01,c()
no (05.19..0 01,
(5)
B P100000)000))
Ag
05
Morrd (0i 01,
Tres (0r,1 0i,ce)
Vono (or,cl9, 01
+
Tyy (0r,el 0c,cc)
To (o,,rlg (i
l0.1
Te (o1,rlg, 0i, A;
B100 Te [1m)
Wo (0i Oi,ct)
Advantage weighted probubility ratio
Trel (ou.lg
+ +
To (oi,elq, oi.ce) (o)t))
V o log we (o,, 01,ct)
Policy model gradient
KL divergeee constraint
We analyze the derivative components stressed in Eq. (5). (1) Advantage-wwiggted probability ratio el. A; directly ties the contribution of each completion to its advantage. This term term Noata (OL. (.0.))))
incentivizes the policy to prioritize actions with higher rewards, as the advantage function quantifies how much a given action improves expected returns relative to the baseline. By amplifying high- advantage completions and suppressing low-advantage ones, this term guides policy optimization to-
ward reward-aligned reasoning patterns. (2) KL divergence constraint term B (0..0.0..) - -1) 7o(00.19.00..e)
enforces stability by penalizing deviations from the reference model Tref- However, this constraint is not inherently designed to shape the policy's 's reasoning patterns but rather ensures smooth updates during training. (3) Policy model gradient term Vo log Te (o,,ale oi,<e) represents the gradient the log-probability of the policy's predicted action with respect to the model parameters 6.
Recent work by Hu et al. [0] demonstrates that removing the KL divergence constraint does not impair the trained model's reasoning ability, as the policy's 's core reasoning patterns are primarily driven by the reward-aligned advantage term. Motivated by this insight, we approximate the policy objective's 's derivative as:

---

Completions
Rewards
Advantages
101
A 0.5
Prompts
Reward
Fi
Old
02
System
Tard
A2
Policy Model
09
Ag -05
Policy Model
Update
00
A,
High Absolute
Advantages
Per-token Probability
Completion Pruning
Gradient
Old Policy
Model
D
Completions
Policy Model
DKL
00
Reference
Model
Figure 2: Overview of Completion Pruning Policy Optimization (CPPO). After obtaining the advan- tages, only completions with high absolute advantage retained for the forward of the policy model, reference model, and old policy model. D is the probability ratio of the policy to old policy models.
VoJenro (a)
(on.lg. o,
}. (6)
A Advonaage (Prior-forward)
V log Toe (o
Taa (oizlq, 01
Policy model gradion (Post-forword)
Probabiliity ratio (Post.forward)
focusing on the reward-driven learning signal while decoupling the KL regularization constraint.
To better understand this formulation, decompose the advantage-weighted probability ratio term into the Probability ratio term and the Advantage term. For a completion that significantly contributes to the policy update, all the new three components in Eq. (6) must be non-negligible A near-zero zero value in any these components would render the overall contribution minimal or nonexistent.
From computational timing perspective, these components can be categorized as: (1) The probability ratio and policy model gradient are post-forward information, meaning they can only be computed after the policy's forward computation. (2) The advantage term, however, represents prior-forward information that can be calculated before the policy's 's forward computation.
Given our objective to accelerate GRPO training, we focus on leveraging this prior-forwatt informa- tion. By evaluating the advantage term before the forward computation, can make an informed decision about whether to process completion through the policy model. Specifically, if the absolute value of the advantage for completion is small or insignificant that be practically treated as if it were zero without causing any significant difference in the outcome, we prune that completion from the batch. This selective processing ensures that only completions with high absolute advantage proceed to the forward computation and gradient update stages.
3.3 Completions Pruning Policy Optimization
As shown in Figure we propose the Completions Pruning Policy Optimization (CPPO) algorithm to accelerate the training process of group relative policy optimization. Compared to GRPO's optimization objective Eq. (T). our CPPO introduces selective condition that only includes completions exhibiting sufficiently high advantage. The CPPO objective is formulated as follows:
5

---

G
1o,1
To(os,19, 0z.())
(cppo(0)= = PQ) (an
min
Ar.
6
Tooia (0 01,c))
To 9,
(7)
clip(
1I ( Tres) 1.1+60 +)
Till (0i 41 01,ct)
s.t.
1.12 7,
where 7 is predefined threshold that ensures only completions with an absolute advantage above retained in the gradient update. It should be noted that when the ratio x0.00__9..0..c))
THera (ot,e 9.91,ct)
so and A; < O, aaag (or. g.0r,ce) > 1 + and A, > O. the clip function is activated. This action effectively nullifies the policy model gradient term Eq. (6). equivalent to pruning all completions.
Settings. multi-GPUs training scenario, we observe that the number of completions with significant advantages varies across devices. In such cases, the overall training efficiency is bottlenecked by the device processing the largest number of completions- phenomenon referred as the bucket effect. To mitigate this, for each GPU, we retain only the completions with the largest absolute advantage for each question, where
Unifying Single/MMutiile-GGPUG
k = LG x (1 11-11 P)J.
(8)
where P E (0, 1] denoting the pruning rate. The modified CPPO under this strategy is:
lodl
na Tolonelq, As
(C) = P())
min
Toue (Di,t 0i,
iEI
(9)
clip( To(on,lg, 0z,t) Teat (o,el9,
e, c, c,1+))A
DDK (Toll {J Js.
where the summation is taken only over the index set I corresponding to the k completions with the highest absolute advantage values, i.e.
{i E [l,..., G} |A, is among the top k values}.
(10)
we analyze that completions with high absolute advantage values, either having correct format and correct answer an incorrect format and incorrect answer, provide the clearest training signals. Partial correct completions with small absolute advantages contribute minimally or may mislead the policy model. Removing these completions from the training process can enhance training efficiency without compromising model performance.
In Sec. 4.2.2
The key distinction between CPPO and GRPO is that CPPO does not use all completions for the forward computation of the policy model, reference model, and old policy model. Instead, by retaining only those with high absolute advantages for the gradient update, CPPO significantly reduces the computational overhead during forward passes, thereby accelerating the training process.
3.4 Parallel Processing through Dynamic Completion Allocation
In this section, we introduce a novel dynamic completion allocation strategy to further optimize the training efficiency of CPPO. Conventional approaches, such as those employed GRPO, face inherent limitations due to GPU memory constraints. Specifically, single device can process maximum of B questions per batch, with each question generating G candidate completions. After pruning, the total number of retained completions per device reduces to B x k. resulting in suboptimal GPU utilization and underleveraged parallel computing capabilities.
To address this inefficiency, we dynamically allocate pruned completions from additional questions into the device's processing pipeline, as illustrated in Figure This strategy ensures that each device operates at full capacity by continuously populating its memory with high-quality completions derived from both the original and newly introduced questions. Critically, all newly incorporated completions undergo the same rigorous pruning process maintain consistency.

---

GPU
GPU
GPU
04
Completion Pruning
Completion
Allocation
Datasets
LLM
of
0,
oz
New Completions after Pruning
Pruned Completions
Retained Completions
Figure 3: Illustration of dynamic completion allocation for parallel processing. After pruning completions, completion allocation incorporates important completions from new questions. The o represents the m-th completion of the n-th question.
The benefits of this approach twofold. First, maximizes GPU utilization by fully exploiting the device's s parallel computing potential. Second, it enables each device to process a larger number of questions per batch, thereby reducing the total number of training steps required to achieve convergence. This dual optimization boosts training efficiency while maintaining training quality. The CPPO algorithm can be found in Appendix(B]
4 Experiments
4.1 Experimental Settings
Training Details. We implement CPPO on the Open RI [5] and verl [22] frameworks, utilizing the vLLM inference library [14] for efficient completion generation. Qwen2.5-1, 1.5......tnstt and Qwen2.5-7BBInntruue are trained on two and four GPUs (each with 80GB memory), respectively. We set 0.2 and 0.04 in Eq. (7), batch size to 16, number of epochs to 1, and learning rate to 1 x 10-6, The policy model temperature 1, group size is 16, and the maximum completion length 1024. The prompt templates for CPPO can be found in Appendix[[]
Evaluation Details. We evaluate the performance on multiple benchmarks with different difficulties. including Math [8]. AIME2024 [18] AMC2023 [17], and GSM8K [4] We use vLLM [14 to accelerate the evaluation process. The evaluation batch size is set 10. We use greedy decoding to generate completions for Math and GSM8K. For AIME2024 and AMC2023,
set the temperature as 0.6 and use completions for each question. use Pass@1 accuracy as the evaluation metric.
4.2 Main Results
We evaluate CPPO by training models of different seales on GSM8K and MATH 8 GSMSK contains 8.5K grade-school math problems, while MATH includes 7.5K competition- problems. For the relatively simpler GSMSK dataset, we use Qwen2.5-1.5B ----srutttt for the more challenging MATH dataset,
Fach model is evaluated the corresponding test subset. To further assess out-of ciat-f--distrbbutiin reasoning ability, we test 2ove2255-7-IInstrul on
Drren2.5-7B-Intrruue
AMC2023 [17] and AIME2024 [18. these benchmarks too difficult for Additional results on larger models and different backbones are provided in Appendix and Analyses of stability, convergence, and case studies are presented in Appendix and
4.2.1 Performance Comparison
Training on GSM8K. As shown in Table T CPPO demonstrates clear advantages over GRPO in both accuracy and acceleration ratio. Notably, CPPO achieves comparable or even higher accuracy

---

Table 1: Comparison between GRPO and CPPO on GSM8K test subset. We train Qwen2.5-1.5B
Instruct on the GSM8K training subset three times independently to calculate the mean and standard deviation, and the number of retained completions after pruning denoted by G (1 P.
Group Size (G)
Pruning
Training Time (s)
Accelcrate Ratio
Method
Accuracy (%)
Rate (P)
Swen2.5-1.5BB-Insst
55.72
GRPO
16
0.00%
16
77.38
0.28
23500.33 + 130.49
1.00
CPPO
16
50.00%
78.15 + 0.37
12862.33 78.68
1.83 + 0.01
CPPO
16
75.00%
4
78.76 4 0.25
7436.00 1 232.98
3.16 - 0.08
CPPO
16
87.50%
2
80.01 + 0.38
4516.33 + 237.46
5.22 = 0.31
CPPO
16
93.75%
78.99 + 1.01
2946.00 94.44
7.98 = 0.23
Table 2: Comparison of GRPO and CPPO on the MATH test subset, as well as on out-of-distribun benchmarks AMC 2023 and AIME 2024. We train Qwen2.5-7B: ----strrr-n on the MATH training dataset three times independently to calculate the mean and standard deviation, and the number of retained completions after pruning is denoted by A |G x (1 P)J-
Group Size (G)
Pruning Rate (P)
Training Time (s)
Accelerate Ratio
AMC
AIME
Method
k
Accuracy (%)
2023
2024
Qwen2.5-7B-Instruue
55.20
25.62
5.00
GRPO
0.00%
16
75.26 1: 0.09
33795.00 + 80.18
1.00
46.88
5.83
CPPO
16
50.00%
8
76.01
+ 1.03
20129.00
298.90
1.68 = 0.02
53.12
10.00
CPPO
16
75.00%
4
76.55
+: 0.83
13067.00 = 81.26
2.59 + 0.02
49.38
6.67
CPPO
87.50%
2
75.95
0.55
9722.00 78.87
3.48 4 0.03
46.25
8.33
CPPO
93.75%
74.65 1
+ 1.31
7608.00
542.36
4.46 + 0.29
45.00
5.83
than GRPO across various pruning rates. At pruning rate of 87.50%, CPPO attains an accuracy of 80.01% %, surpassing GRPO's 77.38% by 2.63%.
For efficiency, CPPO greatly accelerates training. At pruning rate of 93.75% it achieves an acceleration ratio of 7 x. The specdup stems from the completions pruning and the completions allocation. Completions pruning reduces computational overhead by discarding less important completions, while the completions allocation strategy maximizes the use of freed memory and leverages the GPU's parallel processing capabilities. As a result, CPPO processes more questions per batch and reduces the total number of training steps required. These results demonstrate that CPPO not only maintains or improves accuracy but also significantly enhances training efficiency, making practical and effective solution for large-scale reasoning model training.
Training on MATH. Table CPPO can well scale to larger models, achieving up to 3.48x acceleration on the MATH without sacrificing accuracy. For instance, at pruning rate of 87.5%, CPPO attains 75.9 95% accuracy, outperforming GRPO (75.26%) while cutting training time by 3.48x.
Furthermore, evaluation on the AMC2023 and AIME2024 benchmarks confirms that CPPO, despite training only on high absolute advantage completions, preserves the model's generalization ability on out-of out-of-distrionn diistributiion tasks. Thus, CPPO not only matches or even surpasses GRPO in enhancing reasoning capabilities but also well reduces training time, making a more efficient alternative.
4.2.2 An In-depth Analysis of CPPO's Higher Accuracy
Results on Sec. 4.2.1 indicate that CPPO sometimes achieves better performance at higher pruning rates on the GSM8K and MATH datasets. For example. CPPO with 75% pruning rate achieves 78.76% accuracy on GSMSK and 76.55% accuracy on MATH, compared to 78.15% and 76.01% accuracy with a 50% pruning rate, respectively. To rule out the possibility that this improvement is merely due to the increased number of questions processed per training step, which is enabled by the completion allocation strategy, compare CPPO with GRPO under the same number of questions per training step, as shown in Figure
The key difference is that CPPO first generates group of completions and retains only the top completions for gradient update, whereas GRPO directly generates completions for update. Despite this, CPPO consistently outperforms GRPO, demonstrating that its accuracy gains stem

---

82
78
77.20%
GRPO
GRPO
78:81 %
CPPO
CPPO
78
Accurcy (%g
75.20%
75.20
74.40
Accuncy
74.00
74,92 16
73,20
74
72
70
70
Figure 4:
Evaluation accuracy comparison. Left: 1reee...-1..bInnstii on GSM8K test subset. Right: Owenn.5-77-Instrruct on MATH test subset. Here, k denotes the retained completion quantity by CPPO (or generated by GRPO), and represents questions per training step.
not from processing more questions per step but from the higher quality of retained completions. The quality of completions plays a crucial role training. CPPO selectively retains high absolute advantage completions from a larger pool, whereas GRPO updates the model with directly generated completions, which may vary quality. This aligns with our completion contribution analysis in Sec. which highlights that completions with high absolute advantages contribute more effectively to training. GRPO with a group size 16, both high and low absolute advantage completions are used for training. In contrast, CPPO with pruning rate of 37.50% trains exclusively on high- advantage completions- yet still achieves superior performance.
To better understand this, we categorize completions into four types: (1) Correct format and correct answer Guides the model generate accurate completions. (2) Incorrect format and incorrect answer Helps the model avoid incorrect completions. (3) Correct format and incorrect answer May mislead the model into generating partially correct responses. (4) Incorrect format but correct answer Similarly, may introduce noise in learning. Specific examples of the four types of completions can be found in [ppendix]]
The first two types are high-quality completions that provide clear training signals. The latter two types, however, are low-quality completions, as their small positive advantage values can mislead the model and introduce noise. Unlike GRPO, which trains on all completions indiscriminately. CPPO filters out these low-quality completions through completions pruning, leading more efficient learning and better overall performance. As more low-quality completions are removed (pruning rate: 0.00% - 87,50%), performance improves, as shown in Table Table[2 However, an excessively high pruning rate can also discard high-que high-quality completions, reducing training effectiveness. This is evident in CPPO's performance decline a 93.75% pruning rate, as shown Table and Table[2]
4.3 Generalizing CPPO to Other Reinforcement Learning Algorithms
CPPO reduces training cost by pruning low-quality completions. Therefore, CPPO can be generalized other group relative policy optimization based algorithms such DAPO (3) and DR.RR [16]
As shown in Table CPPO can be combined with DAPO and Dr. GRPP to further improve training speed and accuracy, demonstrating the strong generalizability of CPPO.
Table 3:
Comparison between different reinforcement learning algorithms the GSM8K test subset. We train Qwen2.5-1.55 1-..2.55-1.BB-Inttrre on the GSM8K training subset, and the number of retained completions after pruming & |Gx (1- All experiments are conducted on the verl framework. which supports various RL algorithms.
Group
Pruning Rate (P)
Training Time (s)
Accelerate Ratio (x)
Method
h
Accuracy (%)
Size (G)
Owen2.5-1.5B-Innttuue
55.19
GRPO
16
0.00%
78.01
7741
1.00
CPPO
16
50.00%
8
78.92
5192
1.49
DAPO
16
0.00%
78.01
3800
1.00
DAPO + CPPO
16
50.00%
8
78.01
2134
1.78
Dr.RRPO
16
0.00%
16
79.45
7991
1.00
Dr.RRP + CPPO
16
50.00%
8
80.14
5122
1.56

---

Table Evaluation CPPO with different pruning metrics on GSM8K. "Largest" prune completions with the highest/lowest absolute advantages, while advantage values. "Random" denotes random pruning.
use raw
Pruning Meric
Group Size
Pruning Rate
Accuracy
Qwen2.5-1.5B Instructt
55.72%
Largest*
16
50.0%
73.32%
Smallest*
16
50.0%
76.83%
Largest
16
50.0%
74.23%
Random
16
50.0%
76.98%
Smallest
16
50.0%
77.67 %
Table 5: Ablation study on the key components of CPPO. Experiments are conducted on Math g using Qwen2.5-7B-Instruct [29].
Group Size
Pruning Rate
Accelerate Ratio
Method
Accuracy
Time
GRPO
16
0.0%
75.20%
33902s
1.00x
Completion Pruning
16
50.0%
75.80%
27547s
1.23x
+ +Completion Allocation
16
50.0%
75.20%
20550s
1.65x
4.4
Ablation Study
Ablation Study Pruning Metrics. The analysis in Sec. reveals that completion's 's impact policy model training is tied to the absolute value of its advantage- a higher absolute value provides stronger training signals. Based on this insight, we adopt the absolute advantage value the pruning metric, removing completions with the lowest absolute advantages. As shown in Table "Smallest" achieves the best performance, while "Largest" performs the worst, with "Random" falling in between. Additionally, "Smallest*" and "Largest*", which prune completions based on raw advantage values rather than absolute values, perform worse than "Smallest", confirming that absolute advantage values are more effective pruning metric. The results align with our analysis in Sec. 3.2 and further validate the effectiveness of pruning based on absolute advantage values.
blation Study on Key Modules. As shown in Table by discarding unimportant completions, the completion pruning module improves the training efficiency by 1.23x. By fully leveraging the benefits brought by completion pruning and the GPU parallel computing capability, the completion allocation strategy further improves the training efficiency 1.65x.
5 Limitations and Future Work
CPPO does not reduce the time required for generating completions. When completion generation dominates the overall training time, the speedup of CPPO may be reduced. However, CPPO can benefit from inference accelcration methods [3] 15], which are orthogonal to it, to improve training efficiency further. Due to the limited GPU resources of the academic community, we only evaluate the effectiveness of CPPO on relatively small-scale models (less than 14B) and math datasets, including Math and GSMSK. In the future, we plan to: (1) evaluate CPPO on larger models and more tasks. (2) optimize the completion generation time to boost training efficiency further.
6 Conclusion
In this paper, we proposed Completion Pruning Policy Optimization (CPPO) enhance the training efficiency of GRPO-based reasoning models. By selectively pruning completions based on their relative advantages with suitable pruning rate, CPPO reduces computational overhead without compromising model performance. Additionally. our dynamic completion allocation strategy fully leverages the benefits completion pruning and GPU parallelism, further boosting training speed. The results demonstrate that CPPO achieves up to 7.98x speedup GSM8K and 3.48x on Math while sometimes preserving or even enhancing the accuracy compared to GRPO. These findings highlight CPPO as a practical solution for optimizing reasoning model training at lower cost.
10

---

Acknowledgments
This work was supported by the National Science Fund for Distinguished Young Scholars (No.62025003), National Science Fund for Excellent Young Scholars (No. 62222602), the Na- tional Natural Science Foundation of China (No. U21B2037, No. U22B2051, No. U23A20383, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural Science Foundation of Fujian Province of China (No. 2021J06003, No.2022106001).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report, 2023.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 2023.
[3] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Jean-Baptinn Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013/18 2023.
[4] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21100 14168, 2021.
[5] Hugging Face. Open rl: A fully open reproduction of deepseck-rl, January 2025.
[6] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang, rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking. arXiv preprint WxXv:2501.045/9, 2025.
[7] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501. 12544....19,,,.. 2025.
[8] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurIPS, 2021.
[9] Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models. arXiv preprint arXiv:2501.033222 2025.
[10] Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, and Heung- Shum Xiangyu Zhang. Open-reasoner-zero: An open source approach to scaling reinforcement learning on the base model, 2025.
[11] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, al. Openai ol system card. arXiv preprint arXiv:2412.16721 2024.
[12] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodehench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 2024.
[13] Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter chain-of- thought without compromising effectiveness. arXiv preprint arXiv:2412. 1664, 2024.
[14] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, 2023.

---

[15] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint 10X7: 2024.
[16] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl rl-zero-like training: A critical perspective. arXiv preprini arXiv:2503.20783, 2025.
[17] Mathematical Association of America. American mathematics competitions, 2023.
[18] Mathematical Association of America. American invitational mathematics examination, 2024.
[19] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirami, Julian Michael, and Samuel Bowman, Gpqa: A graduate-Ivva google- proof q&a benchmark. In COLM, 2024.
[20] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.063:3 2017.
[21] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint Xiv:22402.033000 2024.
[22] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rihf framework. arXiv preprint arXiv: 19256, 2024.
[23] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.033: 04 2024.
[24] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- -aan-Baptsste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, et al. Gemini: A family of highly capable multimodal models, 2023.
[25] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llm. arXiv preprint arXiv:2501. /2599, 2025.
[26] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models, 2023.
[27] Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable thain-of-thought compression in llms. arXiv preprint arXiv:2502. 2025.
(28) Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502. 010225522.7774... 14768, 2025.
[29] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 2024.
[30] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system scale. arXiv preprint arXiv:2503. 14476, 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction accurately reflect this paper's contributions and
scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
Please refer to Appendix ??.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to Sec. We also submit code, which will be released upon publication, in supplementary material for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: We submit code, which will be released upon publication, in supplementary
material for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cmm ec/
public/guidess Sowpees/oooeuuubassssoooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.cc/ppl wwwppbblccgguiicuceccccccoeooopp httgggidddee//eebbbiiinnne htt.SSbbbiiiiooooiiiii.oooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Sec.4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars main experimental results in Table and Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification; There is no negative societal impact of the work performed. is an algorithm optimization paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the assets used this paper are properly credited and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: the new assets introduced this paper are well documented and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpp/////euups..... wwwparippssccooooeeeeeee cc/Conferencess///////e for what should or should be described.
19

---

Appendix
A
Specific Design of the Reward Function
In this paper, define the reward components for the GSM8K dataset as follows:
1, if follows the correct format,
Riormar (o:)
0, otherwise.
2, o, directly matchs the correct answer;
R Raceuracy (0t)
1.5 if matchs the correct answer after regular parsing,
0, otherwise.
And the reward components for the Math datasets are defined as follows:
1, if o follows the correct format, Riormal (0;)
0, otherwise.
2, if of directly matchs the correct answer,
Raccuracy (0:)
0, otherwise.
B Algorithm
We provide the algorithm for our Completion Pruning Policy Optimization (CPPO) in Algorithm| For dynamic completion allocation, we adopt more efficient implementation. Specifically, Sec. (e4. of the main paper, we describe completion allocation after completion pruning to provide more intuitive explanation our method. However, in the algorithm and our code, we perform completion allocation before completion pruning. This is because the number of completions to allocate can be pre-computed based on the pruning rate p. Thus, we first sample batch of 6/(1 - p) questions from the dataset 2> and then sample G' completions for each question. This modification does not affect the final results but improves the efficiency of our CPPO, as it benefits from parallelization inference optimization of vLLMs by sharing prefixes [14]
Algorithm 1 Completions Pruning Policy Optimization
Input initial model n datasets D; datasets size N; batch size b; hyperparameters e, 8, in pruning rate p: Group size G
1: Policy model mT +- T0
2: Reference model Tref 4- Tт
3: M +- (N x (1 pb)/b
4: for step 1, ... M do
Update old policy model ne.u <- me 6: Dynamic Completion Allocation According to Pruning Rate
7:
Sample a batch of b/(1 p) questions Dd from D
Sample completions {or] Te. 9) for each question 9 Dr
9:
Compute rewards fr, (r.}e, for each sampled completion via Eq. 4 in the main paper
10:
Compute advantages
of o, according Fq. (3) in the main paper
Completion Pruning
12:
Retain l + G x (1 - p) completions with the highest absolute advantages for each question
13:
for CPPO iteration I.... do
14:
Update the policy model ma based on the selected completions (Eq. 9
Output Te
C Templates of Completion Pruning Policy Optimization
In this section, we provide the templates of our Completion Pruning Policy Optimization for GSM8K 4 and Math [8] datasets. The templates are shown in and Table
20

---

Table 6:
The templates of our CPPO for GSM8K dataset. Question will be replaced by the question in the dataset.
conversation between User and Assistant. The user asks a question, and the Assistant solves The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> <lanswer> tags, respectively, i.e., <think> reasoning process here </think>>ncanssee>>>> answer here </answer>. User: Question. Assistant:
Table The templates our CPPO for Math in the dataset.
g dataset. Question will be replaced by the question
A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. And the answer should be of the following format: "Therefore, the final answer is: \boxed| ANSWER). I hope it is correct." (without quotes) where ANSWER just the final number or expression that solves the problem. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively. i.c., <think> reasoning process here </hink>>n<aasswe>>>>> answer here </answer>. User: Question. Assistant:
D Experimental Results on Larger Models
We conduct experiments on larger models, specifically Qwen2.5-14B 14B-Instruct, and present the results in T The results show that our CPPO can also accelerate the training of larger models by to 2.83x without compromising accuracy, demonstrating the scalability and generalizability of CPPO.
Table 8: Comparison of GRPO and CPPO on the MATH test subset. We train Qwen2.5- Swee2..5114B- 1-125----1--- on the MATH training dataset, and the number of retained completions after pruning is denoted by
k G x (1 P.
Group Size
Pruning Rate
Accelerate Ratio
Method
Accuracy
Time
1eem2..5-4B-Inntrree
67.80%
GRPO
16
0.00%
16
77.60%
33942s
x.00
CPPO
16
50.00%
79.40%
21375s
1.59
CPPO
16
75,00%
79.40%
15385s
2.21x
CPPO
16
87.50%
78.00%
11997s
2.83x
CPPO
16
93.75%
76.00%
11247s
3.02x
E CPPO Results on GSM8K with Qwen2..-7B-Instruet
We additionally conduct experiments with Qwen2.5-7B 5.---BBIInsttctt on the GSM8K dataset. As shown in Table GRPO improves accuracy by 8.37% the baseline, which is less than the 21. 21.66% improvement observed for Qwen2.5-11 1..55-155-IIntrrue in Table This because GSM8K relatively easy for the 7B model, which already achieves high initial accuracy (83. (3%.0 leaving limited room for improvement. Nevertheless, CPPO still delivers up to 4.67 speedup without loss of accuracy, demonstrating its robustness.
Experimental Results on Different LLM Backbones
conduct experiments on the Llama series models and present the results Ta The results show that CPPO can also accelerate the training of Llama models without compromising accuracy and achieving a significant speedup of up to 3. 3.1 This demonstrates the generalizability of CPPÓ across different LLM backbones.
21

---

Table Comparison of GRPO and CPPO on the GSM8K subset. We train Qwen2.5-7B-Instruu on the GSM8K training dataset, and the number of retained completions after pruning is denoted by G (1 P.
Group Size (G)
Pruning Rate (P)
Training Time (s)
Accelerate Ratio (x)
Method
k
Accuracy (%)
Qwen2.5-7B-Instee
83.00
GRPO
16
0.00%
16
91.37
19294
1.00
CPPO
16
50.00%
8
91.59
12177
1.58
CPPO
16
75.00%
4
91.82
7037
2.74
CPPO
16
87.50%
2
92.04
4975
3.88
CPPO
16
93.75%
1
92.04
4128
4.67
Table 10: Comparison between GRPO and CPPO on the GSM8K test subset. We train Llama-3.. 2- IB-Instruct on the GSM8K training subset, and the number of retained completions after pruning is denoted by k G (1 P. Experiments are conducted on the verl [2] framework.
Group Size (G)
Pruning Rate (P)
Training Time (s)
Accelerate
Method
k:
Accuracy (%)
Ratio (x)
1aama-3.2-IB-Insttuee
46.55
GRPO
16
0.00%
16
62.32
13487
1.00
CPPO
16
50.00%
8
62.55
8362
1.61
CPPO
16
75.00%
62.62
4310
3.13
G
Comparison with Other Reinforcement Learning Algorithms
As shown in Table we compare our CPPO with other reinforcement learning algorithms using the GSM8K test subset the verl 22 framework, which supports various reinforcement learning algorithmss CPPO achieves the best accuracy of 78.92% with training time of 5192s, outperforming RÉINFORCE+ and PPO (00 without KL divergence.
Table 11:
Comparison between different reinforcement learning algorithms on the GSM8K test subset. We train Qwenn.5511.5B-Insmmut the GSM8K training subset, and the number of retained completions after pruning is |G x (1 - P)J. All experiments are conducted on the verl framework.
Group Size (G)
Pruning Rate (P)
Training Time (s)
Method
k
Accuracy (%)
Qwenn.55-1.BBInnsrmee
55.19 19
GRPO REINFORCE+
16
0.00%
16
78.01
7741
16
0.00%
16
78.54
9043
PPO w/o KL CPPO
1
0.00%
1
74.45
5465
16
50.00%
8
78.92
5192
H
Stability and Convergence
We plot the reward curves in Figure and 6 during training on both GSM8K and MATH datasets. Overall, the reward curves provide evidence that CPPO preserves GRPO's training stability while improving convergence speed. The results show that the reward curves of CPPO do not crash or experience drastic fluctuations, which is crucial for stable training. These results suggest CPPO's robust and stable training. Moreover, the reward curves of CPPO show a clear upward trend, reaching higher reward values more quickly than GRPO. This faster increase reward values indicates that CPPO converges more rapidly.
22

---

2.5
2.0
Reward
GRPO
1.5
CPPO (50.00%)
CPPO (75.00%)
CPPO (87.50%)
1.0
CPPO (93.75%)
1000
2000
3000
4000
5000
6000
7000
Steps
Figure 5: Reward comparison during training (1 epoch) between GRPO and CPPO on GSM8K
dataset.
2.8
2.6
2.4
Reward
2.2
GRPO
CPPO (50.00%)
2.0
CPPO (75.00%)
CPPO (87.50%)
1.8
CPPO (93.75%)
500
1000
1500
2000
Steps
Figure 6: Reward comparison during training (1 epoch) between GRPO and CPPO on the Math
dataset.
23

---

I
Case Studies
we provide case studies illustrate the differences between com- pletions generated by GRPO and CPPO. We observe that CPPO can maintain or even improve the reasoning ability of the trained model by pruning completions with low absolute advantages during training. Notably, CPPO significantly reduces the training time, achieving up 7.98x acceleration on GSM8K datasets. Furthermore, in some cases where GRPO fails to provide the correct answer, CPPO with high pruning rate succeeds, demonstrating its effectiveness in enhancing model performance.
As shown Tables 12
Table 12:
The comparisons of completions generated by GRPO and CPPO. The question is sampled from the GSM8K [4] dataset. We use 1-552..5-1.BBInttmeee (29 model with different training methods to generate completions. The "/" and "x" denote correct and wrong response, respectively.
Question:
Ian has a board that 40 feet long. He decides to make a cut so he can have two pieces. The longer piece is 4 times longer than the shorter piece. How long is the longer piece?
GRPO Response /:
<think> Let's denote the length of the shorter piece as L. According to the problem, the longer piece 4 times longer than the shorter piece, so the longer piece's ' length is 4L. The total length of the board is the sum of the lengths of the two pieces, which is 40 feet. Therefore, we have the equation: feet. Since the longer
L 4L 40. Simplifying this, we get 5L 40. Solving for L, we get piece is 4 times longer than the shorter piece, the longer piece's length is 4 x 8 = 32 feet. </think> <answer> 32 </answer>
CPPO (50.00%) )) Response /:
<think> Let's denote the length of the shorter piece z. Then the longer piece would be 4x. The total length of the board is the sum of the lengths of the two pieces, so x + 4r 40. This simplifies to
5x 40. Solving for x. get 8. Therefore, the longer piece is 4 x 8 32 feet. </think>
<answer> 32 </answer>
CPPO (75.00% ). Response /:
<think> Let's denote the length of the shorter piece as x. Then, the length of the longer piece is 4x. The total length of the board is the sum of the lengths of the two pieces, so we have the equation:
I 1 4x = 40, This simplifics to 5x = 40. Solving for x, we get x Therefore, the length of the longer piece is 4x 4 x 8 = 32 feet. </think>
<answer> 32 </answer>
CPPO (87.5% Response v:
<think> Let's denote the length of the shorter piece as x feet. According to the problem, the longer piece is 4 times longer than the shorter piece, so the longer piece is 4z feet. The total length of the board is the sum of the lengths of the two pieces, which 40 feet. Therefore, we can write the 40. Simplifying the left side, we get 5x = 40. To find x, we divide both sides by This means the shorter piece is feet long. The longer piece times the length the shorter piece, so it is 4 x 8 = 32 feet. </think>
equation: x + 4x so x
<answer> 32 </answer>
CPPO (93.75%) Response /:
<think> Let's denote the length of the shorter piece According to the problem, the longer piece is 4 times longer than the shorter piece, so the length of the longer piece is 4x. The total length of the board is the of the lengths of the two pieces, we can write the equation: x + 4x 40. Combining like terms, we get 5z = 40. To find æ, we divide both sides by 5: x = 8. So, the length of the shorter piece is 8 feet. The longer piece is 4 times the length of the shorter piece, so it is 4 x 8 = 32 feet. Therefore, the longer piece is 32 feet long. </think> <answer> 32 </answer>
24

---

Table 13: The comparisons of completions generated by GRPO and CPPO. The question is from
the GSM8K [4] dataset and generated by Qwen2.5-1.55 1.52255-1.B--nttrrur Instruct 29 The "V" denotes a correct response while the "x" indicates a wrong response.
Question:
Lee used to be able to run the 400-meter hurdles two seconds faster than Gerald would run the 400-meter hurdles. But Gerald changed his diet, which improved his speed by 10%. If Lee runs the 400-meter hurdles in 38 seconds, how fast can Gerald, with his improved diet, run the 400-meter hurdles, in seconds?
GRPO Response
<think> Let's break down the problem step by step. Lee used to be able to run the 400-meter hurdles two seconds faster than Gerald. This means that if Lee runs in 38 seconds, Gerald would run in 38 + 2 = 40 seconds. Gerald's s diet improved his speed by 10%, so we need to calculate his new speed. If Gerald's original speed was S seconds, his improved speed would be S x 1.1. We know that Lee's speed is 2 seconds faster than Gerald's original speed, we can set up the equation: 38 S 2. Solving for S, we get 5 = 38 + 2 = 40 seconds. Now, we can calculate Gerald's improved speed: 40 1.1 44 seconds. </think>
<answer> 44 </answer>
CPPO (50.00%) Response X:
<think> Let's break down the problem step by step. First, we need to determine how long used to take Lee to run the 400-meter hurdles. Since Lee used to be able to run it two seconds faster than Gerald, and Lee now takes 38 seconds, we can find out how long used to take Lee to run the hurdles. Let's denote the time it took Lee to run the hurdles before the diet change as T seconds. Since Lee now takes 38 seconds and he used to be two seconds faster, we have this equation, we get T = 40 seconds. This means that before the diet change, Lee took 40 seconds to run the 400-meter hurdles. Now, we need to find out how long it took Gerald to run the 400-meter hurdles before the diet change. Since Lee used to be able to run two seconds faster than Gerald, we have TGrall = 40 = 38 seconds. This means that before the diet change, Gerald took 38 seconds to run the 400-meter hurdles. Now, we need to find out how fast Gerald can run the 400-meter hurdles with his improved diet. Since his diet improved his speed by 10%, we need to calculate 1 100% of his original speed. Let's denote the original speed of Gerald as S meters per second. Since he took 38 seconds to run the 400-meter hurdles before the diet change, we have =
= 38. Solving
400 meters per second.
"3
With his improved diet, his speed is now 1.1 x S. Substituting the value of S, we get 1.1 x meters per second. Simplifying this, we get 440 </answer>
400 410
38
38
11.57 meters per second. </think>
<answer> 11.57
CPPO (75.00% ) Response X:
<think> First, we need to determine how long it used take Lee run the 400-meter hurdles. Since
Lee used to be able to run it two seconds faster than Gerald, and Lee runs it in 38 seconds, we can calculate the time used take Gerald to run it. The time it used take Gerald is 38 + 2 = 40 seconds. Next, we need to calculate how fast Gerald can run the 400-meter hurdles with his improved diet. His speed improved by 10%, so we need to find 110% of his original time. The original time for Gerald is 40 seconds, so the improved time is 40 x 1.1 = 44 seconds. </think> <answer> 44 </answer>
CPPO (87.5% Response v::
<think> Let's break down the problem step by step. First, we need to determine Gerald's original time to run the 400-meter hurdles. We know that Lee is two seconds faster than Gerald, and Lee runs the 400-meter hurdles in 38 seconds. Therefore, Gerald's original time is 38 2 = 40 seconds. Next, we need to calculate Gerald's improved speed after his diet. Gerald's diet improved his speed by 10%. To find his improved time, we need to calculate 10% of 40 seconds and subtract it from 40 seconds. 10% of 40 is 40 x 0.10 = 4 seconds. So, Gerald's improved time is 40 4 = 36 seconds. </think> <answer> 36 </answer>
CPPO (93.75%) Response /:
<think> Lee used to be able to run the 400-meter hurdles two seconds faster than Gerald would run the 400-meter hurdles. So, if Lee runs in 38 seconds, Gerald used to run in 38 + 2 = 40 seconds. Gerald's diet improved his speed by 10%. To find out how fast Gerald can now, we need to calculate 10% of 40 seconds and subtract it from 40 seconds. 10% of 40 seconds is 0.10 x 40 = 4 seconds. So, Gerald's new time is 40- 4 36 seconds. </think> <answer> 36 </answer>
25

---

Illustration of Different Completion Types
We provide examples different completion types generated by Qwen2.5-1.5B-Insttne
2V in
Ta These types are as follows: (1) Correct format and correct answer Guides the model to generate accurate and well-structured completions.
(2) Incorrect format and incorrect answer Helps the model learn to avoid generating entirely incorrect completions. (3) Correct format but incorrect answer Highlights cases where format is correct, but answer incorrect, potentially misleading the model. (4) Incorrect format but correct answer Demonstrates cases where the answer correct, but the format is incorrect, which may introduce noise during training.
Table 14: Illustration of completions from Qwen2.5-1. 5-1.....--.B 22 using a question from GSM8K [. The "v" ("X") denotes correct (incorrect) formats or answers.
Question:
The gauge on a water tank shows that the tank 1/3 full water. To fill the tank, 16 gallons water added. How many gallons of water does the tank hold when full?
Completion Type 1:
<think> Let's denote the full capacity of the tank by C gallons. The gauge shows that the tank is 1/3 full which means is 1/3 gallons. If add 16 gallons to the tank, it will be filled completely. So. C 16 C 2/3 (since 1/3 1/3 + 1/3 is equal to I, and 1/f + 1/f + 1/f is equal to 3/f from 3/2 24 gallons. </think> <answer> The tank holds 24 gallons when full. </answer>
geometry). Simplifying this equation, we get
Format Reward:
Accuracy Reward: /
Advantage: 2.01
Completion Type 2:
<think> Let's denote the total capacity of the tank C gallons. According the problem, when the tank is 1/3 full, it contains 1/3 C gallons of water. When 16 gallons are added to this, the total
becomes full, which means C 16 3 C. This equation helps find the total capacity C. </think>
Format Reward: X
Accuracy Reward: X Advantage: -1.26
Completion Type 3:
<think>We can set up an equation based on the information the problem. Let x be the total number of gallons the tank holds when full. </think>
<answer> X </answer>
Format Reward: /
Accuracy Reward: X
Advantage: 0.04
Completion Type 4:
<think> Let's reason through this step by step: </think>> 1. We know that the tank was 1/3 full. 2. To find out the total capacity of the tank when it's full, we need to multiply the amount already in the tank by (since full tank = 3/3 full tanks). 3. We also know that adding 16 gallons filled the tank to its full capacity. 4. So, 16 gallons equals the difference between the full tank and the current amount,
this information to find the current amount in gallons. 5. If 16 gallons are added to 4/3 of the total capacity to get the full capacity, then the current amount must be 3/4 of the full capacity. 6. Let's call the total capacity of the tank C (it's whole number, in gallons). So, the current amount the tank = (3/4)C (1/3C) 2/3C, and we know that 2/3C 16, 8. Therefore 2/3 * C= 16, 9. So C. must be 16 3/2, 10. Which equals 24 gallons. <answer> The tank holds 24 gallons when full. </answer>
Rearranging 3/4 to look like 3C/4, and knowing that (3C/4) = C
Format Reward: X
Accuracy Reward: / Advantage: 0.69
26


Reviews:
[{"review_id": "0lmOMmy9kq", "reviewer": "Reviewer_o2Dn", "summary": "This paper introduces Completion Pruning Policy Optimization (CPPO), a method to accelerate the training of reasoning models that use Group Relative Policy Optimization (GRPO). The key insight is that not all completions contribute equally to policy training—their contribution depends on their relative advantage. CPPO prunes completions with low absolute advantages before forward computation, thus reducing computational overhead. The authors also introduce a dynamic completion allocation strategy to maximize GPU utilization. Experiments on GSM8K and MATH datasets using Qwen2.5 models show speedups of up to 8.32x on GSM8K and 3.51x on MATH while maintaining or improving accuracy.", "questions": "I think this paper proposed a good intuitive solution, but the limited evaluation scope might not justify the benefit that this algorithm can bring. If the authors can demonstrate the robustness of this algorithm in a large variety of settings, it would help establish the value of this technique. For example:\n\n- can you include comparison with other completion selection strategies?\n- how does the method perform with larger models beyond 7B parameters?\n- how to set hyperparameters for different settings? If different tasks want to adopt this technique, how do they choose $k$, $b$ and the pruning rate?", "limitations": "See questions", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nThe paper addressed a computational limitation in GRPO training, which needs to process multiple completions per question and scales training time multiplicatively. The problem formulation is clear and well-motivated. The proposed CPPO algorithm is intuitively motivated by decomposing the policy objective derivative and observing that advantage can be computed prior-forward, and this already existing information can be used to perform pruning. Paper also introduced a dynamic completion allocation strategy to maximize GPU utilization for practical mutli-GPU setting.\n\nTwo models are finetuneing with CPPO and evaluation on two datasets AMC and AIME shows good accuracy improvements over GRPO with low quality completions being removed during the training process. The analysis section provided some intuitive explanations why the proposed algorithm is working well.\n\nWeakness\n\nThe core contribution of this paper is adopting a heuristic for selecting a subset of data points during training process. While some experiments show that this is effective, the insight that high-advantage completions contribute more to training is quite intuitive, and pruning low-advantage completions is a natural response to this observation. I feel it lacks some fundamental algorithmic innovation.\n\nThe evaluation section can use more experiment settings, such as testing CPPO's advantage with larger models (currently only <=7B parameters) as smaller models in general do not suffer as much in training inefficiency. There are many other math reasoning benchmark datasets, this paper only evaluated on two. There also do no seem to be comparisons made with other acceleration methods for RL training.", "was_revised": true, "final_justification": "I appreciate the authors' answer to my questions and effort to conduct additional experiments to demonstrate the robustness of the approach for larger model and additional benchmark datasets. The results comparing and combining DAPO and CPPO is good to include in the main paper to provide additional insights for readers to understand the difference in these acceleration algorithms. Overall, I think the proposed method has its practical value and does bring additional benefits in training speed while maintaining comparable accuracy.", "rebuttal": "Thank you for your valuable feedback, including remarks such as \"The problem formulation is clear and well-motivated\" and \"a good intuitive solution.\" We give our point-by-point responses to your comments below:\n\n**Q1: Algorithmic innovation.**\n\n**A1:** With all due respect, we believe our work presents significant algorithmic innovation. GRPO is a widely used and influential reinforcement learning algorithm, but its substantial training overhead limits efficiency and scalability. Improving training efficiency is an important and practical problem. To our knowledge, this is the first study to accelerate GRPO by pruning completions, supported by rigorous theoretical analysis. In Appendix B, we provide detailed derivations, and in Section 3.2, we analyze the gradient of the GRPO policy objective to clarify the role of each term. Our analysis demonstrates that **not all completions contribute equally to the policy gradient**, and the contribution of each completion depends on its absolute advantage value. Based on this insight, we propose CPPO and dynamic completion allocation to speed up GRPO training by a considerable margin. All the theoretical analyses, insights, and methods are introduced for the first time in this paper. The novelty and theoretical soundness of our work are also recognized by reviewers ZKXt and kqfe.\n\n**Q2: More benchmark datasets.** \n\n**A2:** We evaluate models trained with GRPO and CPPO on the MATH training dataset and test them on several out-of-distribution benchmarks, including GPQA:Diamond, Olympiad_bench:zh, Agieval:gaokao-mathqa, MMLU:college_mathematics, and MathQA. As shown in Table 1, CPPO achieves comparable average performance to GRPO across these datasets, demonstrating that CPPO can accelerate GRPO training without compromising the model’s generalization ability. While we agree that evaluating our method on more benchmark datasets would be beneficial, limited academic computational resources restrict us to experiments on representative and high-quality datasets such as GSM8K, MATH, AMC, and AIME to validate the effectiveness of our method. We will continue to evaluate our method on additional benchmarks and tasks in future work.\n\nTable 1: Comparison between GRPO and CPPO on additional benchmark datasets. We train Qwen2.5-7B-Instruct on the MATH training dataset and evaluate the model on various out-of-distribution benchmarks using lighteval. The number of retained completions after pruning is given by $k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$.\n\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|GPQA:Diamond (\\%)|Olympiad_bench:zh (\\%)|Agieval:gaokao-mathqa (\\%)|MMLU:college_mathematics (\\%)|MathQA (\\%)|Average (\\%)|\n|-|-|-|-|-|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|-|-|-|33.46 $\\pm$ 2.46|38.23 $\\pm$ 1.38|45.58 $\\pm$ 2.66|26.00 $\\pm$ 4.41|33.47 $\\pm$ 0.86|35.35|\n|GRPO|16|0.00%|16|33.71 $\\pm$ 2.55|40.97 $\\pm$ 1.40|49.00 $\\pm$ 2.67|26.00 $\\pm$ 4.41|34.30 $\\pm$ 0.87|36.80|\n|CPPO|16|50.00%|8|34.60 $\\pm$ 2.70|42.74 $\\pm$ 1.41|47.59 $\\pm$ 2.67|27.00 $\\pm$ 4.46|34.71 $\\pm$ 0.87|37.33|\n\n**Q3: Comparison with other acceleration methods for RL training.**\n\n**A3:** Our work aims to accelerate the GRPO algorithm, which is widely used and influential in reinforcement learning. Due to significant differences between GRPO and other reinforcement learning algorithms, such as PPO, it is not appropriate to compare our method with acceleration techniques designed for other algorithms. To our knowledge, this is the first work to accelerate GRPO by pruning completions. Therefore, we only compare our method with CPPO variants in Table 3 of the main paper. Recently, a concurrent work[1] has also sought to accelerate GRPO, but it focus on different aspects and is orthogonal to our work. As shown in Table 2, CPPO can be combined with it to further improve training time and accuracy. We will include these results in the camera-ready version.\n\nTable 2: Comparison between different reinforcement learning algorithms on the GSM8K test subset. We train Qwen2.5-1.5B-Instruct on the GSM8K training subset, and the number of retained completions after pruning is $k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$. All experiments are conducted on the verl framework, which supports various RL algorithms.\n\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|Accuracy (\\%)|Training Time (s)|Accelerate Ratio|\n|-|-|-|-|-|-|-|\n|Qwen2.5-1.5B-Instruct|-|-|-|55.19|-|-|\n|GRPO|16|0.00%|16|78.01|7741|1.00|\n|CPPO|16|50.00%|8|78.92|5192|1.49|\n|DAPO|16|0.00%|16|78.01|3800|1.00|\n|DAPO+CPPO|16|50.00%|8|78.01|2134|1.77|\n\n[1] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025).  \n\n**Q4: Comparisons with other completion selection strategies.**\n\n**A4:** As shown in Table 3 of the main paper, we compare our CPPO with other completion selection strategies. \"Random\" refers to randomly pruning completions. \"Largest\"/\"Smallest\" prune completions with the highest/lowest absolute advantages, respectively. \"Largest*\"/\"Smallest*\" prune completions with the highest/lowest advantages, which is equivalent to pruning completions with the highest/lowest rewards, since higher rewards correspond to higher advantages according to Eq.(3) of the main paper. The results in Table 3 of the main paper demonstrate that our CPPO, which prunes completions with the smallest absolute advantages, outperforms all other completion selection strategies.\n\n**Q5: Experimental results on larger models beyond 7B.**\n\n**A5:** As requested, we conduct experiments on larger models, specifically Qwen2.5-14B-Instruct, and present the results in Table 3. The results show that our CPPO can also accelerate the training of larger models by up to 2.83x without compromising accuracy, demonstrating the scalability and generalizability of our method. We would like to evaluate our method on even larger models, but due to limited academic computational resources, we can only conduct experiments on Qwen2.5-14B-Instruct at this time. We will include these results in the main paper.\n\nTable 3: Comparison of GRPO and CPPO on the MATH test subset. We train Qwen2.5-14B-Instruct on the MATH training dataset, and the number of retained completions after pruning is denoted by $k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$.\n\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|Accuracy (\\%)|Training Time (s)|Accelerate Ratio|\n|-|-|-|-|-|-|-|\n|Qwen2.5-14B-Instruct|-|-|-|67.80|-|-|\n|GRPO|16|0.00%|16|77.60|33942|1.00|\n|CPPO|16|50.00%|8|79.40|21375|1.59|\n|CPPO|16|75.00%|4|79.40|15385|2.21|\n|CPPO|16|87.50%|2|78.00|11997|2.83|\n|CPPO|16|93.75%|1|76.00|11247|3.02|\n\n**Q6: How to set hyperparameters for different settings? If different tasks want to adopt this technique, how do they choose $k$, $b$, and the pruning rate $P$?**\n\n**A6:** Assuming the group size is $G$ and the pruning rate is $P$, the hyperparameters are set as $k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$ and $b = G / k$. Therefore, it is only necessary to choose appropriate values for $G$ and $P$ for different tasks. A larger group size $G$ generally improves performance but increases training cost, while a higher pruning rate $P$ provides greater speedup but may degrade performance if set too high. We recommend selecting $G$ and $P$ based on available computational resources and task performance requirements. In most cases, setting $G$ to 16 and $P$ to 50% achieves a good balance between performance and speedup.\n\nWe hope our responses address your concerns. If you have any further questions, please feel free to contact us. Thank you again for spending time reviewing our work and providing valuable feedback."}, {"review_id": "lpxpOD5Ksp", "reviewer": "Reviewer_zbVH", "summary": "This paper introduces Completion Pruning Policy Optimization (CPPO), which prunes low‐advantage completions to reduce the number needed for policy updates and dynamically reallocates freed GPU capacity to new questions, thereby maintaining or improving model accuracy while significantly speeding up training . Experiments on GSM8K and MATH benchmarks demonstrate up to **8.32×** and **3.51×** speedups respectively, with accuracy equal to or exceeding GRPO .", "questions": "1. **Threshold Sensitivity**: How sensitive is CPPO’s performance to the choice of γ? Have you evaluated multiple threshold settings to identify an optimal or adaptive strategy?\n\n2. **Integration with Inference Acceleration**: In scenarios where completion generation dominates, have you explored combining CPPO with methods like TokenSkip or speculative sampling to further reduce wall‐clock time?\n\n3. **Generality to Larger Models and Tasks**: Could you extend CPPO’s evaluation to models > 7B parameters and non‐math reasoning benchmarks to validate scalability and robustness?\n\n4. **Robustness of Results**: Have you conducted multiple independent runs to compute error bars or statistical significance measures (e.g., t‐tests) for the reported accuracy and speedup?", "limitations": "The paper acknowledges that CPPO **does not reduce completion generation time** and that evaluations are **limited to models under 7B parameters and math benchmarks**, suggesting future work to integrate inference acceleration techniques and broaden the evaluation to larger models and diverse tasks .", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* **Theoretical Motivation**\n  The authors rigorously analyze completion contributions and show that advantage correlates with training signal, underpinning the pruning strategy (Sec. 3.2, lines 117–123).\n\n* **Dynamic GPU Utilization**\n  The dynamic completion allocation mechanism leverages GPU capacity post‐pruning to process additional questions, maximizing throughput (Sec. 3.4, lines 176–183).\n\n* **Empirical Effectiveness**\n  CPPO achieves up to **8.32×** acceleration on GSM8K and **3.51×** on MATH without sacrificing—and often improving—accuracy, as shown in Table 1 and Table 2 (lines 219–227; 229–235).\n\n* **Comprehensive Ablations**\n  Detailed studies on pruning metrics (“Smallest” vs. “Largest”) and component contributions (pruning vs. allocation) validate design choices and quantify their individual impact (Table 3 and Table 4, lines 266–274).\n\n## Weaknesses\n\n1. **Pruning Threshold Selection (Eq. 7, Line 154)**\n   The choice of the absolute‐advantage threshold $\\gamma$ is fixed without justification or sensitivity analysis. Could the authors consider an **adaptive threshold** (e.g., via cross‐validation or reinforcement learning) [1] to balance pruning aggressiveness and performance? \n\n2. **Completion Generation Overhead (Limitation Sec. 5)**\n   CPPO accelerates forward and backward passes but does not address the time spent generating completions, which can dominate total training time. Integrating inference acceleration methods such as **TokenSkip** [2] or **Eagle** [3] could further reduce end‐to‐end training time.\n\n3. **Narrow Evaluation Scope (Limitation Sec. 5)**\n   Experiments are confined to math datasets and models under 7B parameters. Broader validation on larger models (> 7B) and diverse reasoning tasks (e.g., commonsense QA, code generation) would strengthen claims of generality\n\n4. **Lack of Statistical Significance**\n   The reported accuracies in Tables 1 and 2 lack error bars or multiple‐seed evaluations, making the result variability unclear. Including **confidence intervals** over repeated runs would bolster confidence in the observed speed/accuracy tradeoffs. \n\n\n[1] Gu, Jiahua, et al. \"Adaptive Threshold-Triggered Heuristic-assisted Deep Reinforcement Learning for Energy-efficient and QoS-guaranteed 5G RAN Slice Migration.\" GLOBECOM 2024-2024 IEEE Global Communications Conference. IEEE, 2024.\n\n[2] Xia, Heming, et al. \"Tokenskip: Controllable chain-of-thought compression in llms.\" arXiv preprint arXiv:2502.12067 (2025).\n\n[3] Li, Yuhui, et al. \"Eagle-2: Faster inference of language models with dynamic draft trees.\" arXiv preprint arXiv:2406.16858 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for spending time reviewing our work. Please kindly see our responses to your comments below.\n\n**Q1: Threshold Sensitivity: How sensitive is CPPO’s performance to the choice of $\\gamma$? Have you evaluated multiple threshold settings to identify an optimal or adaptive strategy?**\n\n**A1:** Eq. (7) is only an intermediate step in deriving our final policy objective function. Final policy objective function is formally defined in Eq. (9) and Eq. (10) of the main paper. We do not use a fixed threshold $\\gamma$ in our final policy objective function. Since the range of advantage values varies across tasks and questions, using a fixed threshold $\\gamma$ for completion pruning is unsuitable. During multi-GPU training, we observed that the number of completions exceeding a fixed threshold differs across devices, resulting in overall efficiency being limited by the device with the most completions (the \"bucket effect\"). Therefore, for each GPU, CPPO retains the $k$ completions with the largest absolute advantages for each question, rather than using a fixed threshold. More details can be found in Section 3.3 of the main paper.\n\n**Q2: Integration with Inference Acceleration: In scenarios where completion generation dominates, have you explored combining CPPO with methods like TokenSkip or speculative sampling to further reduce wall-clock time?**\n\n**A2:** We would like to clarify that inference acceleration methods are orthogonal to our work. Even in scenarios where completion generation dominates, our CPPO can achieve a considerable speedup, such as 2.83x (Table 2 with the 14B model). Inference acceleration is a popular research topic, and we believe that combining CPPO with more advanced inference acceleration methods can lead to further improvements in training speed and efficiency, which will also be our future focus.\n\n**Q3: Generality to Larger Models and Tasks: Could you extend CPPO’s evaluation to models > 7B parameters and non-math reasoning benchmarks to validate scalability and robustness?**\n\n**A3:** As requested, we conduct experiments on larger models, specifically Qwen2.5-14B-Instruct, and present the results in Table 2. The results show that CPPO can accelerate the training of larger models by up to 2.83x without compromising accuracy, demonstrating the scalability and generalizability of our method. While we aim to evaluate our method on even larger models, limited academic computational resources currently restrict us to Qwen2.5-14B-Instruct. Additionally, we evaluate our method on more benchmark datasets in Table 3. Specifically, we train Qwen2.5-7B-Instruct on the MATH training dataset and evaluate the model on non-math QA datasets (GPQA:Diamond) and code generation tasks (LiveCodeBench). The results show that both CPPO and GRPO, when trained on math datasets, achieve improved performance on GPQA. However, due to the large domain gap between math and code generation, performance on code generation tasks is slightly reduced for both methods, with CPPO exhibiting less degradation. This demonstrates that our method can accelerate GRPO training while maintaining the model’s generalization ability. Validating rule-based reinforcement learning algorithms on math tasks is common practice in this field [1][2], so we also choose math tasks to evaluate our method in this paper. Evaluating CPPO on more tasks, such as code generation and multimodal tasks, will be our future work, as these require substantial computational resources and engineering effort.\n\nTable 2: Comparison of GRPO and CPPO on the MATH test subset. We train Qwen2.5-14B-Instruct on the MATH training dataset, and the number of retained completions after pruning is denoted by $k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$.\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|Accuracy (\\%)|Training Time (s)|Accelerate Ratio|\n|-|-|-|-|-|-|-|\n|Qwen2.5-14B-Instruct|-|-|-|67.80|-|-|\n|GRPO|16|0.00%|16|77.60|33942|1.00|\n|CPPO|16|50.00%|8|79.40|21375|1.59|\n|CPPO|16|75.00%|4|79.40|15385|2.21|\n|CPPO|16|87.50%|2|78.00|11997|2.83|\n|CPPO|16|93.75%|1|76.00|11247|3.02|\n\nTable 3: Comparison between GRPO and CPPO on additional benchmark datasets. We train Qwen2.5-7B-Instruct on the MATH training dataset and evaluate the model on various out-of-distribution benchmarks using lighteval. The number of retained completions after pruning is given by $k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$.\n\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|GPQA:Diamond (\\%)|LiveCodeBench|\n|-|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|-|-|-|33.46 $\\pm$ 2.46|13.99 $\\pm$ 1.88|\n|GRPO|16|0.00%|16|33.71 $\\pm$ 2.55|13.46 $\\pm$ 1.80|\n|CPPO|16|50.00%|8|34.60 $\\pm$ 2.70|13.78 $\\pm$ 1.83|\n\n[1] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025).  \n[2] Liu, Zichen, et al. \"Understanding r1-zero-like training: A critical perspective.\" arXiv preprint arXiv:2503.20783 (2025).  \n\n**Q4: Robustness of Results: Have you conducted multiple independent runs to compute error bars or statistical significance measures (e.g., t-tests) for the reported accuracy and speedup?**\n\n**A4:** GRPO experiments are computationally expensive and time-consuming, so we do not include error bars in our main paper, following common practice in the community [1][2]. As requested, for more rigorous evaluation, we conducted multiple independent runs for the Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct models and report the results in Tables 4 and 5. The results demonstrate the stability and robustness of CPPO.\n\nTable 4: Comparison between GRPO and CPPO on the GSM8K test subset. We independently train Qwen2.5-1.5B-Instruct on the GSM8K training subset three times to calculate the mean and standard deviation, and the number of retained completions after pruning is $k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$.\n\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|Accuracy (\\%)|Training Time (s)|Accelerate Ratio|\n|-|-|-|-|-|-|-|\n|Qwen2.5-1.5B-Instruct|-|-|-|55.72|-|-|\n|GRPO|16|0.00\\%|16|77.38 $\\pm$ 0.28|23500.33 $\\pm$ 130.49|1.00|\n|CPPO|16|50.00\\%|8|78.15 $\\pm$ 0.37|12862.33 $\\pm$ 78.68|1.83 $\\pm$ 0.01|\n|CPPO|16|75.00\\%|4|78.76 $\\pm$ 0.25|7436.00 $\\pm$ 232.98|3.16 $\\pm$ 0.08|\n|CPPO|16|87.50\\%|2|80.01 $\\pm$ 0.38|4516.33 $\\pm$ 237.46|5.22 $\\pm$ 0.31|\n|CPPO|16|93.75\\%|1|78.99 $\\pm$ 1.01|2946.00 $\\pm$ 94.44|7.98 $\\pm$ 0.23|\n\nTable 5: Comparison of GRPO and CPPO on the MATH test subset. We independently train Qwen2.5-7B-Instruct on the MATH training dataset three times to calculate the mean and standard deviation, and the number of retained completions after pruning is denoted by $k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$.\n\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|Accuracy (\\%)|Training Time (s)|Accelerate Ratio|\n|-|-|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|-|-|-|55.20|-|-|\n|GRPO|16|0.00\\%|16|75.26 $\\pm$ 0.09|33795.00 $\\pm$ 80.18|1.00|\n|CPPO|16|50.00\\%|8|76.01 $\\pm$ 1.03|20129.00 $\\pm$ 298.95|1.68 $\\pm$ 0.02|\n|CPPO|16|75.00\\%|4|76.55 $\\pm$ 0.83|13067.00 $\\pm$ 81.26|2.59 $\\pm$ 0.02|\n|CPPO|16|87.50\\%|2|75.95 $\\pm$ 0.55|9722.00 $\\pm$ 78.87|3.48 $\\pm$ 0.03|\n|CPPO|16|93.75\\%|1|74.65 $\\pm$ 1.31|7608.00 $\\pm$ 542.36|4.46 $\\pm$ 0.29|\n\n[1] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024).  \n[2] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).\n\nWe hope that our responses can well address your concerns. If you have any further questions, please feel free to contact us. Thank you again for spending time reviewing our work."}, {"review_id": "QRSiaIVfwS", "reviewer": "Reviewer_kqfe", "summary": "This paper proposes Completion Pruning Policy Optimization (CPPO), a method designed to improve the training efficiency of reasoning models built upon Group Relative Policy Optimization (GRPO). The authors theoretically and empirically demonstrate that not all completions contribute equally to policy updates, with their usefulness correlated to relative advantage. CPPO addresses this by pruning completions with low absolute advantages, thereby reducing computational overhead without sacrificing performance. Furthermore, the authors propose a dynamic completion allocation strategy to optimize GPU usage. Experiments on GSM8K and Math datasets show that CPPO achieves substantial speedups (up to 8.32×) while maintaining or improving model accuracy.", "questions": "1. Can this algorithm generate to other RL variants?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**:\n\n1.\tAccelerating GRPO-style reinforcement learning methods is an important research direction.\n2.\tThe authors present a clear motivation through the experiment in Figure 1, and the other experiments are also thorough and well-executed.\n3.\tThe paper does show impressive speedup ratios, with up to 8.32x on GSM8K and 3.51x on Math, which is a clear practical improvement for GRPO training. This directly addresses a known bottleneck in GRPO, which is computationally expensive due to multiple completions per question. \n4.\tThe core ideas of pruning low-advantage completions and dynamically allocating remaining GPU resources are genuinely novel and seem well-motivated by the analysis of completion contributions. This intelligent filtering of training signals could lead to more efficient learning.\n\n**Weakness**:\n\n1. The experiments are conducted only on the Qwen2.5 series, making it difficult to assess whether the proposed method generalizes to different LLM backbones.\n2. As a highly engineering-oriented work, I strongly recommend that the authors release their code, as it would be greatly beneficial to the community.", "was_revised": true, "final_justification": "The acceleration effect is significant. Acceleration of RL is an important direction.", "rebuttal": "We extend our gratitude to your valuable and encouraging feedback on our work, such as \"genuinely novel\", \"well-motivated\", and \"well-executed\". Please kindly see our responses to your comments below.\n\n**Q1: The experiments are conducted only on the Qwen2.5 series, making it difficult to assess whether the proposed method generalizes to different LLM backbones.**\n\n**A1:** We conduct experiments on the Llama series models and present the results in Table 1. The results show that CPPO can also accelerate the training of Llama models without compromising accuracy and achieving a significant speedup of up to 3.13×. This demonstrates the generalizability of CPPO across different LLM backbones. We will include these results in the camera-ready version.\n\nTable 1: Comparison between GRPO and CPPO on the GSM8K test subset. We train Llama-3.2-1B-Instruct on the GSM8K training subset, and the number of retained completions after pruning is\n$k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$. All experiments are conducted on the verl framework, which supports various RL algorithms.\n|Method|Group Size($G$)|Pruning Rate($P$)|$k$|Accuracy|Training Time|Accelerate Ratio|\n|-|-|-|-|-|-|-|\n|Llama-3.2-1B-Instruct|-|-|-|46.55%|-|-|\n|GRPO|16|0.00%|16|62.32%|13487s|1.00×|\n|CPPO|16|50.00%|8|62.55%|8362s|1.61×|\n|CPPO|16|75.00%|4|62.62%|4310s|3.13×|\n\n\n**Q2: As a highly engineering-oriented work, I strongly recommend that the authors release their code, as it would be greatly beneficial to the community.**\n\n**A2:** Thank you for your suggestion. The code is included in the supplementary material. We will also release our code publicly upon acceptance of the paper.\n\n**Q3: Can this algorithm generalize to other RL variants?**\n\n**A3:** CPPO reduces training cost by pruning low-quality completions. Therefore, CPPO can be generalized to other group relative policy optimization based RL algorithms such as DAPO[1] and Dr.GRPO[2]. As shown in Table 2, CPPO can be combined with DAPO and Dr.GRPO to further improve training speed and accuracy, demonstrating the strong generalizability of CPPO.\n\nTable 2: Comparison between different reinforcement learning algorithms on the GSM8K test subset. We train Qwen2.5-1.5B-Instruct on the GSM8K training subset, and the number of retained completions after pruning is\n$k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$. All experiments are conducted on the verl framework, which supports various RL algorithms.\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|Accuracy (\\%)|Training Time (s)|Accelerate Ratio|\n|-|-|-|-|-|-|-\n|Qwen2.5-1.5B-Instruct|-|-|-|55.19|-|-|\n|GRPO|16|0.00%|16|78.01|7741|1.00|\n|CPPO|16|50.00%|8|78.92|5192|1.49|\n|DAPO|16|0.00%|16|78.01|3800|1.00|\n|DAPO+CPPO|16|50.00%|8|78.01|2134|1.78|\n|Dr.GRPO|16|0.00%|16|79.45|7991|1.00|\n|Dr.GRPO+CPPO|16|50.00%|8|80.14|5122|1.56|\n\n[1] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025).  \n[2] Liu, Zichen, et al. \"Understanding r1-zero-like training: A critical perspective.\" arXiv preprint arXiv:2503.20783 (2025).  \n\n\nWe hope our responses address your concerns. If you have any further questions, please let us know. Thank you again for your time and valuable feedback."}, {"review_id": "4PzD1Q3aMZ", "reviewer": "Reviewer_ZKXt", "summary": "This paper introduces CPPO, a novel RL method to train language models for reasoning tasks. The method builds on top of GRPO, and adjusts it to only use a subset of completions with high absolute advantages, reducing computation required to process a sample. To compare this method against GRPO, Qwen 2.5 (1.5B and 7B) models are trained and evaluated on mathematical reasoning datasets. The results show that CPPO reduces training time and certain configurations of CPPO outperform GRPO in accuracy.", "questions": "1.How do you compare in speedup and accuracy against other mentioned RL algorithms (REINFORCE++, PPO without KL divergence)?\n\n2.Why did you choose to use different models for different datasets? Can you provide some data showing it wouldn't be reasonable to do so?\n\n3.What was the stopping criterion for training?\n\n4.High pruning rates with the Qwen 7B model don't result in increase in accuracy across MATH and AMC2023 datasets, yet you claim that \"CPPO achieves better performance at high pruning rates\". Can you explain this, as these two seem contradictory?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. CPPO is a novel method that tackles a significant problem of training efficiency of GRPO. The authors provide clear motivation and reasons behind their modifications to the algorithm.\n2. Authors provide extensive algorithm description and code to replicate results.\n3. Comparison is done with relevant benchmarks in the mathematical reasoning space.\n4. The train time speedup, while retaining accuracy, is impressive, positioning CPPO as a viable alternative to GRPO.\n\nWeaknesses\n1. Other RL algorithms, including the ones mentioned in the related work section, are not compared against. Authors mention these methods can have substantial computational cost, however it is unclear how much CPPO improves the speed and accuracy  of training compared to this broader space of RL algorithms.\n2. The chosen metric is pass@1, which can be highly variable, but the variance of this metric is not captured. Additionally, evaluated models differ between datasets. Combined, this undermines the authenticity of the results. Ideally authors would evaluate the algorithm across the same models and capture the variance of results.\n3. The authors claim \"CPPO reduces computational overhead without compromising model performance\", however high pruning rate configurations of the CPPO-trained 7B Qwen model have lower accuracies on MATH and AMC compared to GRPO. The claim should be amended.\n4. The stopping criterion for training is not discussed in the 4.1 section.\n5. In figure 4 it is not clear how this demonstrates that accuracy stems from higher quality completions.", "was_revised": true, "final_justification": "The authors answered reviewers' questions well and added additional evaluations, as requested. The additional results add robustness to the performance evaluation of the method. Authors also improved clarity on multiple subjects raised by reviewers. I suggest to accept this paper.", "rebuttal": "We sincerely appreciate your encouraging feedback on our work, including comments like \"CPPO is a novel method that tackles a significant problem of training efficiency of GRPO\", \"clear motivation\", and \"The train time speedup, while retaining accuracy, is impressive, positioning CPPO as a viable alternative to GRPO.\"  Please see our point-by-point responses to your comments below.\n\n\n**Q1: How do you compare in speedup and accuracy against other mentioned RL algorithms (REINFORCE++, PPO without KL divergence)?**\n\n**A1:** As shown in Table 1, we compare our CPPO with other reinforcement learning algorithms using the GSM8K test subset on the verl[1] framework, which supports various RL algorithms. CPPO achieves the best accuracy of 78.92% with a training time of 5192s, outperforming REINFORCE++ and PPO without KL divergence. We will include these results in the camera-ready version.\n\n\nTable 1: Comparison between different reinforcement learning algorithms on the GSM8K test subset. We train Qwen2.5-1.5B-Instruct on the GSM8K training subset, and the number of retained completions after pruning is\n$k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$. All experiments are conducted on the verl framework.\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|Accuracy (\\%) |Training Time (s)|\n|-|-|-|-|-|-|\n|Qwen2.5-1.5B-Instruct|-|-|-|55.19|-|\n|GRPO|16|0.00%|16|78.01|7741|\n|REINFORCE++|16|0.00%|16|78.54|9043|\n|PPO w/o KL|1|0.00%|1|74.45|5465|\n|CPPO|16|50.00%|8|78.92|5192|\n\n\n[1] Sheng et al. HybridFlow: A Flexible and Efficient RLHF Framework\n\n**Q2:  Ideally, authors would evaluate the algorithm across the same models and capture the variance of results.**\n\n**A2:**  As requested, for more rigid evaluation, we choose to conduct multiple independent runs for the Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct models and report the results in Tables 2 and 3. The results demonstrate the stability  and robustness of our CPPO.\n\nTable 2: Comparison between GRPO and CPPO on the GSM8K test subset. We independently train Qwen2.5-1.5B-Instruct on the GSM8K training subset three times to calculate the mean and standard deviation, and the number of retained completions after pruning is $k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$.\n\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|Accuracy (\\%)|Training Time (s)|Accelerate Ratio|\n|-|-|-|-|-|-|-|\n|Qwen2.5-1.5B-Instruct|-|-|-|55.72|-|-|\n|GRPO|16|0.00\\%|16|77.38 $\\pm$ 0.28|23500.33 $\\pm$ 130.49|1.00|\n|CPPO|16|50.00\\%|8|78.15 $\\pm$ 0.37|12862.33 $\\pm$ 78.68|1.83 $\\pm$ 0.01|\n|CPPO|16|75.00\\%|4|78.76 $\\pm$ 0.25|7436.00 $\\pm$ 232.98|3.16 $\\pm$ 0.08|\n|CPPO|16|87.50\\%|2|80.01 $\\pm$ 0.38|4516.33 $\\pm$ 237.46|5.22 $\\pm$ 0.31|\n|CPPO|16|93.75\\%|1|78.99 $\\pm$ 1.01|2946.00 $\\pm$ 94.44|7.98 $\\pm$ 0.23|\n\nTable 3: Comparison of GRPO and CPPO on the MATH test subset. We independently train Qwen2.5-7B-Instruct on the MATH training dataset three times to calculate the mean and standard deviation, and the number of retained completions after pruning is denoted by $k = \\left\\lfloor G \\times (1 - P) \\right\\rfloor$.\n\n|Method|Group Size ($G$)|Pruning Rate ($P$)|$k$|Accuracy (\\%)|Training Time (s)|Accelerate Ratio|\n|-|-|-|-|-|-|-|\n|Qwen2.5-7B-Instruct|-|-|-|55.20 |-|-|\n|GRPO|16|0.00\\%|16|75.26 $\\pm$ 0.09|33795.00 $\\pm$ 80.18|1.00|\n|CPPO|16|50.00\\%|8|76.01 $\\pm$ 1.03|20129.00 $\\pm$ 298.95|1.68 $\\pm$ 0.02|\n|CPPO|16|75.00\\%|4|76.55 $\\pm$ 0.83|13067.00 $\\pm$ 81.26|2.59 $\\pm$ 0.02|\n|CPPO|16|87.50\\%|2|75.95 $\\pm$ 0.55|9722.00 $\\pm$ 78.87|3.48 $\\pm$ 0.03|\n|CPPO|16|93.75\\%|1|74.65 $\\pm$ 1.31|7608.00 $\\pm$ 542.36|4.46 $\\pm$ 0.29|\n\n\n**Q3: The authors claim \"CPPO reduces computational overhead without compromising model performance.\"; However, high pruning rate configurations of the CPPO-trained 7B Qwen model have lower accuracies on MATH and AMC compared to GRPO. The claim should be amended.** \n\n**A3:** Thank you for pointing this out. We will revise the claim to \"CPPO reduces computational overhead while maintaining comparable model performance at an appropriate pruning rate. An excessively high pruning rate may lower the model performance.\"\n\n\n**Q4: The stopping criterion for training is not discussed in the 4.1 section.**\n\n**A4:** In our experimental settings, we use the same stopping criterion for all methods for fair comparison. Specifically, we set the training epoch to 1, and the training will stop after the model has been trained on the entire training set once. We will add this to Section 4.1 of the main paper.\n\n**Q5: In Figure 4, it is not clear how this demonstrates that accuracy stems from higher quality completions.**\n\n**A5:** In Figure 4, we compare the accuracy of GRPO and CPPO with different question numbers $b$ and retained completion numbers $k$ per training step. The key difference between CPPO and GRPO is that CPPO first generates a group of $G$ ($G > k$) completions and then retains only the top $k$ completions per training step, whereas GRPO directly generates $k$ completions per step. As a result, the completions retained by CPPO are of higher quality, since they are selected from a larger pool based on their absolute advantages. In contrast, the completions retained by GRPO are generated without selection and are more likely to include low-quality completions. The results in Figure 4 show that CPPO achieves higher accuracy than GRPO with all different settings of $b$ and $k$, demonstrating that the improved accuracy is due to the higher quality of completions retained by CPPO.\n\n\n**Q6: Why did you choose to use different models for different datasets? Can you provide some data showing it wouldn't be reasonable to do so?**\n\n**A6:** We choose different models for different datasets for two main reasons: (1) Demonstrating generalizability: Using various models across datasets shows that CPPO works for both small and large models, and on different datasets, highlighting its robustness. (2) Reasonable experimental settings: GSM8K contains 8.5K grade-school math problems, which are relatively simple for larger models like Qwen2.5-7B-Instruct (as shown in Table 3, its GSM8K accuracy is already 83%). MATH, on the other hand, includes 7.5K competition-level problems that require a stronger reasoning ability. Therefore, we use Qwen2.5-1.5B-Instruct for GSM8K and Qwen2.5-7B-Instruct for MATH to ensure the experiments are meaningful and challenging for each model.\n\nTable 3: The initial reasoning capabilities of different models.\n\n|Model|GSM8K Accuracy|MATH Accuracy|\n|-|-|-|\n|Qwen2.5-1.5B-Instruct|55.72%|48.40%|\n|Qwen2.5-7B-Instruct|83.00%|55.20%|\n\n\n\n**Q7: High pruning rates with the Qwen 7B model don't result in an increase in accuracy across MATH and AMC2023 datasets, yet you claim that \"CPPO achieves better performance at high pruning rates\". Can you explain this, as these two seem contradictory?**\n\n\n**A7:** Thank you for highlighting this issue. We will revise our claim to: \"CPPO sometimes achieves better performance at higher pruning rates on the GSM8K and MATH datasets. For example, CPPO with a 75% pruning rate achieves 78.81% accuracy on GSM8K and 77.20% accuracy on MATH, compared to 77.67% and 75.20% accuracy with a 50% pruning rate, respectively.\" We will also clarify that CPPO with excessively high pruning rates, such as 93.75%, may influence the model performance negatively on some datasets, such as MATH and AMC2023.\n\n\nThank you again for spending time reviewing our work and providing valuable feedback. We hope our responses address your concerns. If you have any further questions or suggestions, please feel free to let us know."}]

Author rebuttal:

Meta-review: This paper presents an approach to reduce training time for RL runs with LLMs. This is done by dynamically allocating compute to sampling and training. The idea is to remove some completions that were not providing high magnitude advantages totally from training. The experiments shown on simplistic benchmarks or only on some models show gains in reducing training time.

The reviewers were generally positive about the paper, and liked it after the rebuttal. However, I think that a limitation of this paper is in not having a clear way to show benefits on multiple base models, for instance would the strategy help everywhere?
Decision: Accept (poster)