Title: CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Reasoning Models

Efficient Training
Abstract: This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models based on Group Relative Policy Optimization (GRPO). GRPO, while effective, incurs high training costs due to the need to sample multiple completions for each question. Our experiment and theoretical analysis reveal that the number of completions impacts model accuracy yet increases training time multiplicatively, and not all completions contribute equally to policy training---their contribution depends on their relative advantage. To address these issues, we propose CPPO, which prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates. Additionally, we introduce a dynamic completion allocation strategy to maximize GPU utilization by incorporating additional questions, further enhancing training efficiency. Experiments show that CPPO achieves up to $7.98\times$ speedup on GSM8K and $3.48\times$ on Math while preserving or even enhancing the accuracy compared to the original GRPO. We release our code at https://github.com/lzhxmu/CPPO.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CPPO, a novel RL method to train language models for reasoning tasks. The method builds on top of GRPO, and adjusts it to only use a subset of completions with high absolute advantages, reducing computation required to process a sample. To compare this method against GRPO, Qwen 2.5 (1.5B and 7B) models are trained and evaluated on mathematical reasoning datasets. The results show that CPPO reduces training time and certain configurations of CPPO outperform GRPO in accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. CPPO is a novel method that tackles a significant problem of training efficiency of GRPO. The authors provide clear motivation and reasons behind their modifications to the algorithm.\n2. Authors provide extensive algorithm description and code to replicate results.\n3. Comparison is done with relevant benchmarks in the mathematical reasoning space.\n4. The train time speedup, while retaining accuracy, is impressive, positioning CPPO as a viable alternative to GRPO.\n\nWeaknesses\n1. Other RL algorithms, including the ones mentioned in the related work section, are not compared against. Authors mention these methods can have substantial computational cost, however it is unclear how much CPPO improves the speed and accuracy  of training compared to this broader space of RL algorithms.\n2. The chosen metric is pass@1, which can be highly variable, but the variance of this metric is not captured. Additionally, evaluated models differ between datasets. Combined, this undermines the authenticity of the results. Ideally authors would evaluate the algorithm across the same models and capture the variance of results.\n3. The authors claim \"CPPO reduces computational overhead without compromising model performance\", however high pruning rate configurations of the CPPO-trained 7B Qwen model have lower accuracies on MATH and AMC compared to GRPO. The claim should be amended.\n4. The stopping criterion for training is not discussed in the 4.1 section.\n5. In figure 4 it is not clear how this demonstrates that accuracy stems from higher quality completions.\n\nQUESTIONS:\n1.How do you compare in speedup and accuracy against other mentioned RL algorithms (REINFORCE++, PPO without KL divergence)?\n\n2.Why did you choose to use different models for different datasets? Can you provide some data showing it wouldn't be reasonable to do so?\n\n3.What was the stopping criterion for training?\n\n4.High pruning rates with the Qwen 7B model don't result in increase in accuracy across MATH and AMC2023 datasets, yet you claim that \"CPPO achieves better performance at high pruning rates\". Can you explain this, as these two seem contradictory?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Completion Pruning Policy Optimization (CPPO), a method designed to improve the training efficiency of reasoning models built upon Group Relative Policy Optimization (GRPO). The authors theoretically and empirically demonstrate that not all completions contribute equally to policy updates, with their usefulness correlated to relative advantage. CPPO addresses this by pruning completions with low absolute advantages, thereby reducing computational overhead without sacrificing performance. Furthermore, the authors propose a dynamic completion allocation strategy to optimize GPU usage. Experiments on GSM8K and Math datasets show that CPPO achieves substantial speedups (up to 8.32×) while maintaining or improving model accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**:\n\n1.\tAccelerating GRPO-style reinforcement learning methods is an important research direction.\n2.\tThe authors present a clear motivation through the experiment in Figure 1, and the other experiments are also thorough and well-executed.\n3.\tThe paper does show impressive speedup ratios, with up to 8.32x on GSM8K and 3.51x on Math, which is a clear practical improvement for GRPO training. This directly addresses a known bottleneck in GRPO, which is computationally expensive due to multiple completions per question. \n4.\tThe core ideas of pruning low-advantage completions and dynamically allocating remaining GPU resources are genuinely novel and seem well-motivated by the analysis of completion contributions. This intelligent filtering of training signals could lead to more efficient learning.\n\n**Weakness**:\n\n1. The experiments are conducted only on the Qwen2.5 series, making it difficult to assess whether the proposed method generalizes to different LLM backbones.\n2. As a highly engineering-oriented work, I strongly recommend that the authors release their code, as it would be greatly beneficial to the community.\n\nQUESTIONS:\n1. Can this algorithm generate to other RL variants?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Completion Pruning Policy Optimization (CPPO), which prunes low‐advantage completions to reduce the number needed for policy updates and dynamically reallocates freed GPU capacity to new questions, thereby maintaining or improving model accuracy while significantly speeding up training . Experiments on GSM8K and MATH benchmarks demonstrate up to **8.32×** and **3.51×** speedups respectively, with accuracy equal to or exceeding GRPO .\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n* **Theoretical Motivation**\n  The authors rigorously analyze completion contributions and show that advantage correlates with training signal, underpinning the pruning strategy (Sec. 3.2, lines 117–123).\n\n* **Dynamic GPU Utilization**\n  The dynamic completion allocation mechanism leverages GPU capacity post‐pruning to process additional questions, maximizing throughput (Sec. 3.4, lines 176–183).\n\n* **Empirical Effectiveness**\n  CPPO achieves up to **8.32×** acceleration on GSM8K and **3.51×** on MATH without sacrificing—and often improving—accuracy, as shown in Table 1 and Table 2 (lines 219–227; 229–235).\n\n* **Comprehensive Ablations**\n  Detailed studies on pruning metrics (“Smallest” vs. “Largest”) and component contributions (pruning vs. allocation) validate design choices and quantify their individual impact (Table 3 and Table 4, lines 266–274).\n\n## Weaknesses\n\n1. **Pruning Threshold Selection (Eq. 7, Line 154)**\n   The choice of the absolute‐advantage threshold $\\gamma$ is fixed without justification or sensitivity analysis. Could the authors consider an **adaptive threshold** (e.g., via cross‐validation or reinforcement learning) [1] to balance pruning aggressiveness and performance? \n\n2. **Completion Generation Overhead (Limitation Sec. 5)**\n   CPPO accelerates forward and backward passes but does not address the time spent generating completions, which can dominate total training time. Integrating inference acceleration methods such as **TokenSkip** [2] or **Eagle** [3] could further reduce end‐to‐end training time.\n\n3. **Narrow Evaluation Scope (Limitation Sec. 5)**\n   Experiments are confined to math datasets and models under 7B parameters. Broader validation on larger models (> 7B) and diverse reasoning tasks (e.g., commonsense QA, code generation) would strengthen claims of generality\n\n4. **Lack of Statistical Significance**\n   The reported accuracies in Tables 1 and 2 lack error bars or multiple‐seed evaluations, making the result variability unclear. Including **confidence intervals** over repeated runs would bolster confidence in the observed speed/accuracy tradeoffs. \n\n\n[1] Gu, Jiahua, et al. \"Adaptive Threshold-Triggered Heuristic-assisted Deep Reinforcement Learning for Energy-efficient and QoS-guaranteed 5G RAN Slice Migration.\" GLOBECOM 2024-2024 IEEE Global Communications Conference. IEEE, 2024.\n\n[2] Xia, Heming, et al. \"Tokenskip: Controllable chain-of-thought compression in llms.\" arXiv preprint arXiv:2502.12067 (2025).\n\n[3] Li, Yuhui, et al. \"Eagle-2: Faster inference of language models with dynamic draft trees.\" arXiv preprint arXiv:2406.16858 (2024).\n\nQUESTIONS:\n1. **Threshold Sensitivity**: How sensitive is CPPO’s performance to the choice of γ? Have you evaluated multiple threshold settings to identify an optimal or adaptive strategy?\n\n2. **Integration with Inference Acceleration**: In scenarios where completion generation dominates, have you explored combining CPPO with methods like TokenSkip or speculative sampling to further reduce wall‐clock time?\n\n3. **Generality to Larger Models and Tasks**: Could you extend CPPO’s evaluation to models > 7B parameters and non‐math reasoning benchmarks to validate scalability and robustness?\n\n4. **Robustness of Results**: Have you conducted multiple independent runs to compute error bars or statistical significance measures (e.g., t‐tests) for the reported accuracy and speedup?\n\nLIMITATIONS:\nThe paper acknowledges that CPPO **does not reduce completion generation time** and that evaluations are **limited to models under 7B parameters and math benchmarks**, suggesting future work to integrate inference acceleration techniques and broaden the evaluation to larger models and diverse tasks .", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Completion Pruning Policy Optimization (CPPO), a method to accelerate the training of reasoning models that use Group Relative Policy Optimization (GRPO). The key insight is that not all completions contribute equally to policy training—their contribution depends on their relative advantage. CPPO prunes completions with low absolute advantages before forward computation, thus reducing computational overhead. The authors also introduce a dynamic completion allocation strategy to maximize GPU utilization. Experiments on GSM8K and MATH datasets using Qwen2.5 models show speedups of up to 8.32x on GSM8K and 3.51x on MATH while maintaining or improving accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe paper addressed a computational limitation in GRPO training, which needs to process multiple completions per question and scales training time multiplicatively. The problem formulation is clear and well-motivated. The proposed CPPO algorithm is intuitively motivated by decomposing the policy objective derivative and observing that advantage can be computed prior-forward, and this already existing information can be used to perform pruning. Paper also introduced a dynamic completion allocation strategy to maximize GPU utilization for practical mutli-GPU setting.\n\nTwo models are finetuneing with CPPO and evaluation on two datasets AMC and AIME shows good accuracy improvements over GRPO with low quality completions being removed during the training process. The analysis section provided some intuitive explanations why the proposed algorithm is working well.\n\nWeakness\n\nThe core contribution of this paper is adopting a heuristic for selecting a subset of data points during training process. While some experiments show that this is effective, the insight that high-advantage completions contribute more to training is quite intuitive, and pruning low-advantage completions is a natural response to this observation. I feel it lacks some fundamental algorithmic innovation.\n\nThe evaluation section can use more experiment settings, such as testing CPPO's advantage with larger models (currently only <=7B parameters) as smaller models in general do not suffer as much in training inefficiency. There are many other math reasoning benchmark datasets, this paper only evaluated on two. There also do no seem to be comparisons made with other acceleration methods for RL training.\n\nQUESTIONS:\nI think this paper proposed a good intuitive solution, but the limited evaluation scope might not justify the benefit that this algorithm can bring. If the authors can demonstrate the robustness of this algorithm in a large variety of settings, it would help establish the value of this technique. For example:\n\n- can you include comparison with other completion selection strategies?\n- how does the method perform with larger models beyond 7B parameters?\n- how to set hyperparameters for different settings? If different tasks want to adopt this technique, how do they choose $k$, $b$ and the pruning rate?\n\nLIMITATIONS:\nSee questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}