Tricks or Traps? A Deep Dive into RL for LLM Reasoning
Authors: Zihe Liu
Jiashun Liu
Yancheng He
Weixun Wang
Jiaheng Liu
Ling Pan
Xinyu Hu
Shaopan Xiong
Ju Huang
Jian Hu
Shengyi Huang
Siran Yang
Jiamang Wang
Wenbo Su
Bo Zheng
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models Reasoning; Reinforcement Learning; Reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning (RL) for LLM reasoning has rapidly emerged as a prominent research area, marked by a significant surge in related studies on both algorithmic innovations and practical applications. Despite this progress, several critical challenges remain, including the absence of standardized guidelines for applying RL techniques and a fragmented understanding of their underlying mechanisms. In addition, inconsistent experimental settings, variations in training data, and differences in model initialization have led to conflicting conclusions, obscuring the key characteristics of these techniques and creating confusion among practitioners when selecting appropriate techniques. This paper systematically reviews widely adopted RL techniques through rigorous reproductions and isolated evaluations within a unified open-source framework. We analyze the internal mechanisms, applicable scenarios, and core principles of each technique through fine-grained experiments, including datasets of varying difficulty, model sizes, and architectures. Based on these insights, we present clear guidelines for selecting RL techniques tailored to specific setups and provide a reliable roadmap for practitioners navigating the RL for the LLM domain. Finally, we show that a minimalist combination of two techniques can unlock the learning capability of critic-free policies with a vanilla PPO loss. The results demonstrate that our simple combination consistently improves performance, surpassing strategies such as GRPO and DAPO.

Published as a conference paper at ICLR 2026
TRICKS OR TRAPS? A DEEP DIVE INTO RL FOR LLM REASONING
Zihe Liu" 1,2, Jiashun L.u Yancheng He* Weixun Wang Jiaheng Liu', Ling Pan Xinyu Hu Shaopan Xiong Ju Huang'. Jaan Hu Shengyi Huang", Siran Yang'. Jiamang Wang', Wenbo Su', Bo Zheng'
Allibaaa Group BBejjing Jiaotong University
'Hong Kong University of Science and Technology "Nanjing University PPeking University "OpenRLHF "CleanRL.
ABSTRACT
Reinforcement learning (RL) for LLM reasoning has emerged as prominent research area, marked by a surge in algorithmic innovations and applications. How- ever, critical challenges remain, including the lack of standardized guidelines and fragmented understanding of underlying mechanisms. In addition, inconsistent experimental settings, training data, and model initializations often lead to con- flicting conclusions, obscuring key characteristics and creating confusion among practitioners. This paper systematically reviews widely adopted RL techniques through rigorous reproductions and isolated evaluations within a unified open- source framework. We analyze the internal mechanisms, applicable scenarios, and core principles of each technique through fine-grained experiments, includ- ing datasets of varying difficulty, model sizes, and architectures. Based these insights, we present clear guidelines for selecting RL techniques tailored to spe- eific setups and provide a reliable roadmap for practitioners navigating the RL for the LLM domain. Finally, we show that a minimalist combination of two tech- niques can unlock the learning capability of critic-free policies with a vanilla PPO loss. The results demonstrate that our simple combination consistently improves performance, surpassing strategies such as GRPO and DAPO.
Qwen3-4B-Baal Easy Data
Normolization
Betch-leve
Ratio Clip
Grouprieva leval
Advancage Clip
Pccuresy
35 30
w/o std
Reward
Locol std
How to choese tricks
DAPO (5 techniquesi Lite PPO techriques)
GRPO
Value Clip
Giobal std
Overlong
Langth
Qwen3-8B-Bas Base, Hard Data
Reward
Loss
50
Short
Difficulty
45
Loss Agoi egation
Achaantgge
Outa Filtering
Error Max
40
Right Nin
Sequence- iovl
Accuracy
35
Difficuty
token-leve
30
GRPO
DAPO (5 techniques) sechniques)
Model
25
Figure 1: Left: The proliferation of RL optimization techniques, coupled with diverse initialized models and data, has raised barriers to practical adoption. Right: We establish detailed application guidelines via dissecting internal mechanisms of widely-used tricks, and introduce Lite PPO, a minimalist two-technique combination that enhances learning capacity in critic-free policies with vanilla PPO loss. The average accuracy is calculated across six mathematical benchmarks.
Equal Contribution.
Corresponding Weixun Wang <weixun, @taoban.com>.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent breakthroughs in large language models (LLMSs) such as OpenAl ol (Wu et al. 2024) and DeepSeck RI (Shao et al. 2024) have positioned reinforcement learning (RL) as a key driver in unlocking advanced reasoning capabiliiiess in LLMs. This is particularly evident in challenging reasoning tasks like mathematical problem solving (He et al., 2025b) and code generation (Zhuo et al., 2025), where RL has demonstrated the potential to elevate LLM performance beyond what pre-training alone can achieve. Such an emerging trend has sparked widespread interest within the research community in the direction of "RL for LLM" (or RL4LLM).
However, this rapid progress is shadowed by a lack of usage guidelines (Huang et al. 2024b) and
in-depth understanding of underlying mechanisms. For instance, GRPO (Shao al. 2024 advocates group-level normalization for stability, while REINFORCE+ (Hu et al. (025)) favors batch-level. Moreover, GRPO incorporates variance in normalization, yet Dr. GRPO (Liu et 2025b) explicitly recommends removing variance normalization to prevent bias. Such contradictory and chaotic phenomena underscore the fragmented understanding and inconsistent recommendations within the RL4LLM community. A likely cause for the above phenomenon is that the experimental settings, training data, and initialization of the existing work all have significant differences, which may also cause deviations in the summary of the conclusions.
Apart from the confusion caused by the intrinsic differences of similar techniques, the numerous and seemingly orthogonal techniques, including Normalization, Clip, and Overlong Filtering, also increase the complexity of algorithm application in practice. Practitioners face non-trivial challenges in identifying effective combination from wide range of techniques to unlock the learning capacity of LLMs in specific scenarios. These ambiguities have naturally triggered a key requirement of practitioners:
What scenarios are the existing techniques respectively suitable for? Is there a simple and generaliz- able combination that be used to enhance policy optimization?
Following established RL analysis practices (Andrychowicz et al. 2020 Engstrom et al. 2020 Huang al.) 2024b), we systematically evaluate popular RL techniques through reproducible experiments on a unified open-source framework. Our comprehensive setup spans datasets of varying difficulty, different model sizes, and multiple model types, supported by over 160 independent RL training experiments to ensure robust and statistically meaningful conclusions. We also delve into the theoretical foundations. implementation, and recommended applications of each technique, as summarized in Figure [ Our findings show that most RI. methods are highly sensitive to factors like model type, data distribution, reward design, and hyperparameters. Notably, we demonstrate that combining just two techniues--advanttgee normalization (group mean, batch std) and token- level loss aggregation-is sufficient to maximize the potential of critic-free policies with vanilla PPO loss, outperforming mainstream RL4LLM approaches that rely extra components. Our key contributions are:
1. Removing the standard deviation when reward distributions are highly concentrated enhances the stability and effectiveness of model training. 1.4.1.2
2. Group-level mean and batch-level standard deviation enable further robust normalization.
($4.1.3
3. Clip Higher promotes high-quality exploration for aligned models. ($4.2.1)
4. There appears to be "scaling law" between the performance and the upper bound of the clipping on the small-sized model. ( (4....3
5. Compared sequence-level loss aggregation, token-level aggregation effective on base models but shows limited improvement on aligned models. ($4.3.1)
6. Overlong filtering enhances accuracy and clarity for short-to-medium reasoning tasks but provides limited benefits for long-tail reasoning. ($4.4.1)
7. Two techniques may unlock learning capacity in critic-free policies based vanilla PPO loss. 5
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES
A variety of practical techniques have been introduced to stabilize optimization, reduce variance, and accelerate the convergence of LLMs on reasoning tasks. We categorize commonly used techniques follows: (1) Baseline Design. Baselines are crucial reduce variance in the estimation of policy gradients, with recent advances including using the group-mean reward (Shao et al. 2024) or computing the baseline for each sample as the average gradient estimate from other samples in the group (Ahmadian et al. 2024 Kool et al. (219)). (2) Clipping Strategies. Clipping controls excessive policy updates by constraining rewards, advantages, or ratios, and the Clip Ratio Higher method enhances exploration by relaxing the upper bound in PPO's s ratio clipping (Yu al., 2025). (3)
Normalization Strategies. Normalization rewards or advantages stabilizes gradient magnitudes,
with representative methods including Batch-level (Hu et al., 2025). Group-level (Shao al. 2024
Ahmadian et al. 2024), and Reward Shift without Standard Deviation (Luu et al. 2025b). (4) Filtering Strategies. Filtering excludes uninformative undesirable samples before gradient computation, including Overlong Filtering for excessive lengths Yu et al. 2025), Error Max Clip Mask and Right Min Clip Mask for extreme correctness errors, and Difficulty Mask exclude samples outside larget difficulty range (Yu et al., [2025] Zhang et al., 2025 Chu et al., 2025). (5) Loss Aggregation Granularity. The formulation of loss aggregation determines each token's 's contribution to the overall objective, with common approaches including Sequence-level Loss and Token-level Loss, the latter computes per-token advantages mitigate length bias. (6) Additional Loss Functions. Auxiliary losses can complement the primary objective and regularize training. KL Loss (Yu et 2025 Liu al. 2025b) constrains divergence from reference policy, while SFT Loss (Zhang & Zuo (2025) incorporates supervised fine-tuning objectives to preserve alignment. (7) Reward Design. Shaping the reward function can guide desired output properties, with common strategics including Length Penalty to discourage overly long outputs, Formatting Reward to promote structured outputs, and Accuracy Reward that combines correctness with output length.
EXPERIMENTAL DESIGNS
3.1 EXPERIMENTAL. SETUP
Training Algorithm: We utilize the open-sourced ROLL framework] (Wang et al.) 2025). an efficient and scalable platform specifically designed for reinforcement learning optimization in LLMs, to conduct all experiments. In addition, we adopt the PPO loss (Schulman et al. (017)) with advantage values computed using the REINFORCE algorithm (Sutton et al.] 1999) as the unified RL baseline. To ensure consistency with prior research, we set the global batch size 1024 by using a rollout batch size of 128 and sampling 8 responses per prompt. with maximum response length of 8192 tokens. The learning rate is set to 1e 6. For text generation, use top_p value of 0.99, top value of 100, and a temperature of 0.99.
To comprchensively evaluate reinforcement learning (RL) techniques across parame- ter scales, our experiments cover two model sizes: Qwen3-4B and Qwen3-8B. For each model size, we include both non-aligned pre-trained versions (Qwen3-4B-Bat and Qwen3-8B-Ban (B-Base) and aligned versions, enabling assessment RL gains from different initialization conditions.
Base Models:
Training Datasets: To ensure reproducibility and fairness, we exclusively use open-source datasets for training, including Mi-ppleRL-Zoo-Deee -Zoo-Dtaa (Zeng et al. 2025) and Deepmath (He et al.) (025b). To compre- hensively examine how problem difficulty (c.g., Easy, Medium, and Hard; detailed in Appendix [B.3) affects the performance of the RL technique, we randomly sample 5, 000 entries from the datasets. Figure visualizes the difficulty across the training dataset assessed by GPT-4o (Hurst et al. 2024
0-1
6-8
Easy
Medium
Hard
Percentage (%)
Figure 2: Number of correct responses under
8 rollout iterations across datasets.
Open RI. framework: https:://gg thub. com/al waabbbbbbbaae /OO.I.
3

---

Published as a conference paper at ICLR 2026
Owen3-4B-BBl
Qwen3-8B. Beeen--B-BBase BBBse
Qwen3-4B
Qwen3-88
63
Accuracy (%6
25
30
62
20
25
61
61
60
200
400
200
Easy
Medium
Figure 3: Test accuracy over training iterations is compared for different models trained on different datasets. The shaded regions represent the oscillation amplitude mean + cadd_multpllirr x std), with curves smoothed using an I 1-step moving window and exponential smoothing (a = 0.8).
Evaluation Benchmark: All the experiments are conducted on six math datasets: MATH- 500 (Hendrycks al. 2021), OlympiadBench (He et al., 2024), MinervaMath (Lewkowycz et al. 2022), and subsets of standardized examinations (A1ME24-25, AMC23). These datasets span broad complexity spectrum from basic arithmetic to competition-level mathematics, enabling comprehensive evaluation of reasoning capabilities.
3.2 BASELINE RESULTS
Impact of Data Difficulty Training Dynamics We investigate how data difficulty influences the training dynamics of Qwen3 models. Specifically, we analyze the training convergence patterns through accuracy trajectories and generalization gaps. across three tiers of complexity (Easy, Medium, Hard). The detailed learning curves in Figure show that the model exhibits markedly different accuracy trajectories across training sets of different difficulty levels. When focusing on the differ- ences in learning efficiency between the unaligned Base model and the aligned model under the same experimental setting, the aligned models exhibited substantially higher initial accuracy, but additional learning yielded only modest gains, with accuracy improving by roughly 2%. Additional details regarding the baseline's 's training accuracy and response length can be found in Appendix
ANALYSIS
4.1 NORMALIZATION
Advantage normalization is standard technique for stabilizing RL training in language models by reducing gradient variance (Zheng et al. (023), yet implementations vary. GRPO (Shao al.) 2024) and RLOO (Ahmadian et al. 2024: Kool al. 2019) group-level normalization to promote intra-context competition, while REINFORCE+ (Hu et al. 2025) adopts batch-level normalization mitigate overfitting and reward hacking in low-diversity settings. Formally, given prompt x with K sampled responses and corresponding rewards {r and denoting as the reward of the i-th response in a rollout batch of N prompts with K responses each, the group-level and batch-level normalized advantages are:
Fkk mean ((r))e)
batch Tj mean Nok
A
(I)
4.1.1 ADVANTAGE NORMALIZATION IS SENSITIVE TO REWARD MECHANISMS
To systematically evaluate the impact of advan- tage normalization on PPO variants with a value function using the Monte Carlo return target, we conducted experiments under a unified training framework, exploring three settings: normal- ization, batch-level normalization, and group- level normalization. When analyzing the per- formance in Figure it can concluded that both advantage normalization techniques can significantly influence the model's convergence speed, performance stability, and final outcomes.
Qwen3-4B-ban
Qwen3-8B-base
Accuracy (96)
20
200
403
600
800
1000
250
450
600
800
1000
None)Easy)
Gatch-level(Eas])
Group-levellEasy)
Nona(Hard)
Batch-leval(Hard)
Cruup-tevelHHan))
Figure 4: Accuracy comparison of Base models with different normalization techniques cross Easy and Hard datasets.
4

---

Published as a conference paper at ICLR 2026
Specifically, on both model sizes, group-level normalization consistently achieves stable training dynamics and higher final performance compared to both batch-level normalization and no normal- ization. Batch-level normalization exhibits high sensitivity to reward distribution skew, often leading to performance collapse under an imbalanced batch situation, where a few outlier samples dominate the advantage estimates.
4.1.2 IMPACT OF THE STANDARD DEVIATION TERM IN ADVANTAGE NORMALIZATION
Takeaway
Removing the standard deviation when reward distributions are highly concentrated (e.g.. easy training dataset) enhances the stability and effectiveness of model training.
We found when model responses within a prompt group yield highly similar rewards, e.g., when the responses are almost all correct or all incorrect, the resulting standard deviation becomes ex- tremely small. In such cases, dividing by this small standard deviation during normalization can excessively amplify gradient updates, causing the model to overemphasize tasks of extreme difficulty, a phenomenon similar to "difficulty bias" (Liu et 2025b).
Standard Deviation
OlympiadBench
AIME25
12
Accuracy
60
Accuracy(%)
0.36
16
w/ std (Easy) std (Hard) wio std (Hard)
w/ std (Easy) (Fasy) wr std (Hard) std [Hard]
0.30
0.24
Easy
Hard
200
650
400
650
Figure 5: Left: Standard deviation variations during training on easy and hard datasets. Middle and Right: Test accuracy with and without batch-level standard deviation across easy and hard dataset.
To test whether the standard deviation term is the critical factor driving differences in normalization performance, we employ the batch-level calculation, which exhibited unstable performance in the previous section, calculate the mean of advantage, and conduct ablation experiments on the standard deviation term. This can be formalized as:
Ard
((ry)
(2)
Our experiments reveal that, when training on easy data, the policy quickly converges consistent behaviors, resulting in a highly concentrated reward distribution and rapid decline in standard deviation. In this scenario, using standard deviation-bage based advantage normalization can cause the denominator to become extremely small, excessively amplifying reward and advantage values. This leads to abnormally large gradients, destabilizes training, and may even cause gradient explosions. These findings empirically demonstrate that the standard deviation term plays crucial role in advantage normalization.
In summary, experiments and analysis underscore that, in scenarios where reward distributions are highly concentrated, omitting the standard deviation from advantage normalization effectively prevents abnormal gradient amplification, thereby improving the stability and robustness of model training. However, for tasks characterized by inherently higher reward variance, either normalization approach is generally sufficient to maintain stable optimization.
4.1.3 RECONSTRUCT ROBUST NORMALIZATION TECHNIQUE
Takeaway
Calculating the mean at the local (group) level and the standard deviation at the global (batch) level enables more robust reward shaping.
5

---

Published as a conference paper at ICLR 2026
highlights the critical role of stan- dard deviation advantage normalization. This raises the question: is there more robust and effective combination of mean and standard de- viation for reward shaping? To explore this,
Section 4.1.2
Owen3-4B
Qwen3-8B-base
Accurrsy (%)
-2 1.
[Eary] (Easy) Local snd (Hard]
(Basy)
(Easy)
adopted the stable group-level mean calculation method demonstrated with two approaches for computing the standard deviation; local (group-level) and global (batch- level). We then evaluated the performance of these combinations across two model sizes.
Local snd (hard)
section 4.I.I paired
Global std (Hardd)
Giobal sid (Hand)
Figure 6: Accuracy comparison of base models with different standard deviation calculation.
The results, presented in Figures 6 reveal that global-level calculation exhibits clear advantage. We attribute this to the batch-level standard deviation providing stronger normalization by effectively reducing gradient magnitudes, thereby preventing excessive policy updates. This approach aligns more effectively with the biased reward signals common in sparse rewards and coarse-grained advantage fitting, resulting in more stable and robust learning behavior. Furthermore, results support a claim from Hu et al. (2025) that batch-level normalization, or even subtracting the local mean and dividing by the batch standard deviation in certain scenarios, performs better.
experimental
4.2 CLIP-HIGHER
The Clip mechanism, while improving PPO training stability (Huang al. 2024a can cause issues LLM-based text generation by overly suppressing eww-prbbbbility tokens Yu al. 2025). leading to entropy collapse-wile where model output becomes overly deterministic and loses diversity (Jin) al. 2024). This reduction entropy reinforces high-probabilltyy patterns and shrinks exploration, impaiing performance tasks requiring novel reasoning.
(DAPO(() = (rr,: Stows 1 Ehigh).
Here, greater upper bound Ehigh enables exploration for low- wwwprobability tokens, mitigating entropy collapse. However, there's ' limited analysis on when and how to set this upper bound effectively. In this section, address these gaps with targeted experiments and recommendations.
Qwen3-48-Bal
Qwen3-8B-Ban
Qwen3-4B
Qwen3-88
0.30
0.24
Entropy
0.20
0.25
0.30
0.21
0.20
0.35
0.10
0.16
0.05
0.10
upper clip=0.2
upper clip=0.28
Figure 7: Entropy comparison across different models with Clip-Higher. A higher clip upper bound can mitigate the entropy drop in aligned models.
4.2.1 IN WHICH SETTINGS SHOULD WE CLIP HIGHER
Takeaway 3
For models with stronger fundamental reasoning abilities, increasing the clip higher parameter is more likely to facilitate exploration of better solution paths.
As illustrated in Figure experimental results indicate that the impact of increasing the upper clipping bound Ehigh is model-deeenndet. For the base models, adjusting the upper clipping value yields minor effects policy entropy. In contrast, aligned models exhibit a markedly different response: raising the upper clipping bound notably slows the entropy collapse, leading to consistent performance improvements in downstream evaluation metrics.
This disparity is mainly due to the base models' low clipping rate (around 0.003), which results in minimal policy updates and limited exploration because of their simple policy expressiveness. As

---

Published as a conference paper at ICLR 2026
result, increasing the clipping upper bound has little effect learning outcomes. In contrast, aligned models demonstrate improved reasoning and generalization abilities due to advanced post-training techniques (Yang et al. 2025). Figure [8 in Appendix D illustrates that, compared to base models, aligned models initially have fewer high-probability tokens. Therefore, a higher clipping upper bound reduces token probability disparity and mitigates entropy collapse. It expands the range of permissible policy updates, promoting more diverse action sampling and enhanced exploration during training. This mechanism maintains higher entropy while increasing the likelihood of finding optimal solutions, as shown by improved evaluation metrics.
4.2.2 ANALYZING THE EFFECTIVENESS OF CLIP-HIGHER FROM A LINGUISTIC PERSPECTIVE
Takeaway
Traditional clipping may constrain innovative reasoning structure generation. Clipping higher allows the model to explore broader range of discourse reasoning structures.
Building on our token-level demonstration of Clip-Higher's behavior in section 4.2.1 we now analyze its impact reasoning logic through token-level linguistics. As shown in Figure 20 in Appendix D an upper bound of 0.2 imposes strict constraints on policy updates, primarily affecting connective tokens like therefore", if"" and "but" by limiting large probability deviations. This leads to fewer opportunities for the model to generate innovative or diverse logical structures. By increasing the upper bound to 0.28, the model gains more flexibility, resulting in fewer tokens being clipped and a shift in clipping focus toward nigh-frequency function words such as "is", "the", and " This encourages richer reasoning paths while ensuring sentence stability through selective clipping of common function words.
4.2.3
How TO SET THE UPPER BOUND FOR ADVANTAGE CLIPPING
Takeaway
There appears to be a "scaling law" between the performance and the upper bound of the clipping on the small-sized model, which does not exist on larger models.
shows that Clip-Higher improves aligned models. While most works use the de- fault clip upper bound of 0.28 (Yu al., 2025). we believe that different models have differ- ent preferences for this parameter. To verify this conjecture, we uniformly evaluate different clip upper bounds ranging from 0.2 to 0.32 on two model sizes. As shown in Figure 8 the 4B model improves progressively with higher bounds, peaking at 0.32. In contrast, the 8B model performs best at 0.28, with no consistent gain beyond that.
Section 4.2.1
Qwen3-48
Qwen3-88 BB
725
21.5
upper clp=0,23
upper xpp-0.32
232-2322
Figure 8: Test accuracy of aligned models (trained on medium data) with various clipping upper bounds.
4.3 LOSS AGGREGATION
The strategy of loss aggregation directly determines the contribution of each sample or token the overall gradient during optimization (Liu et al., 2025a). Common strategies include token-level and sequence-leve aggregation. The sequence-level aggregation adopted by GRPO (Shao et al. 2024) first averages the loss across all tokens within each sample, then averages these per-response losses across the batch, thereby assigning equal weight to each response regardless of its length. However, Yu et al. (2025) highlights flaw in this method: longer responses possess diminished influence per token on the total loss, hindering the model's 's ability to learn effectively from longer, complex responses. This reduce the model's 's capacity to learn from long, complex answers, and may bias optimization toward brevity, since shorter correct responses receive larger gradient updates, while longer incorrect responses are insufficiently penalized (Liu et al. 2025b).
7

---

Published as a conference paper at ICLR 2026
4.3.1
DOES TOKEN-LEVEL LOSS AGGREGATION SUIT ALL SETTINGS?
Takeaway 6
Compared to sequence-level calculation, token-level loss proves to be more effective on Base models, while showing limited improvement Instruct models.
To systematically evaluate loss aggregation strategies, we compare sequence-level and token-level loss aggregation on both base and aligned versions of Qwen3-8B, As illustrated in Figure() token-level loss improves convergence and peak accuracy for base models by balancing token contributions, particularly on challenging data. contrast, this advantage disappears in aligned models, where sequence-level aggrega- tion achieves better convergence speed and final performance. Analysis suggests that aligned models already exhibit stable reasoning, making fine-grained token weighting unnecessary harmful. In these cases, sequence-level aggregation better preserves the structure and consistency of high-quality, aligned outputs. These findings high- light that optimal aggregation is model-dependent: token-level suits base models; response-level is preferable for instruction-tuned ones.
Qwen3-8B-Base
wwn33-88
Accuracy 196) 12
10-
62
61
250
750
1000
1.50
300
650
650
MeddSegg -8" Hed//kornioss
Hartt/eig 005
Hand/harr-ioss
Figure 9: Accuracy comparison between sequence- level loss and token-level loss. Results are reported on both Medium and Hard Datasets.
4.4 OVERLONG FILTERING
To improve efficiency, LLMs often use fixed maximum generation lengths for truncation (Chen et al. Team al. 2025). However, this may truncate multi-step reasoning early in training, causing incomplete outputs to be mislabeled as negatives and introducing harmful noise. Overlong filtering (Yu et al. 2025) mitigates this by masking rewards over-length responses, preserving robustness and reasoning quality (He et al. (2025a). Yet, its sensitivity to mask thresholds remains underexplored,
2025:
leaving optimal settings unclear.
8k
16k
20k
2.8
70
18 68
3.6
Responss Lengthi
Accuracy(%)
16
2.4
3.0
60-
2.0
2.4
55
1.2
1.6
50
1.0
52
1.2
48
48
0
200
400
600
800
0
200
400
600
800
200
400
600
800
w/o overlong filtering (Acc)
w/ everlong filtering (Acc)
w/o overlang filtering (Res)
w/ overlong filtering (Ros)
Figure 10:
Total test accuracy and response length of Qwen3-8B- Owee3-8B-Base over training iterations under different maximum generation lengths.
4.4.1
WHEN TO USE THE OVERLONG FILTERING
Takeaway 7
Overlong filtering shows limited effectiveness on long-tail reasoning tasks; however, it can enhance the accuracy and clarity of responses in medium and short-length reasoning tasks.
The results in Figure 10 shows that overlong filtering improves learning at a short threshold (8k), but benefits diminish at 20k. Response length analysis reveals the cause: under 20k, filtered models generate longer outputs than the vanilla policy; at Sk, responses become shorter. As Figure [1 (Left) shows, in the 20k setting, frequent clipping occurs due to repetition non-termination. signs of degenerate generation indicating the mask removes unproductive outputs. In contrast, the 8k threshold also filters extended but valid reasoning, encouraging conciseness and reducing verbosity.
8

---

Published as a conference paper at ICLR 2026
1.0
wio overiong filtering (Reward-1)
NW
overlong filtering (Reward=1)
Repeat Rabo
0.8
Repeat Ratio
Weo overlong filtering (Reward-0)
overlang fitbring (Roward-0)
0.7
8k (Reward=1)
0.5
20k (Rewardm1)
0.2
8K (Reward=0)
(Reward-D)
0.1
0.3
5.0
100
250
300
350
200
4.0
600
860
1.000
1200
1400
1600
Figure 11: Left: Repeat ratios for correct (reward = 1) and incorrect (reward 0) generations under varying maximum generation lengths. Right: Repeat ratios truncated samples with vs. without overlong filtering. Repetition rate statistics are provided in Appendix E.1
As illustrated in Figure II (Right), during RL training, the proportion of "repetitive but unable to terminate" overlong samples increases, indicating degraded EOS modeling and leading output redundancy and termination failures. With the overlong mask, this proportion drops and remains low, enabling the model to better distinguish "generation completed" from "truncated" samples, avoiding invalid learning. More importantly, the mechanism helps policies accurately model termination, preventing them from mistakenly penalizing unfinished outputs as negative examples.
A SIMPLE COMBINATION: LITE PPO
48-Ba5e, Easy
48-Base, Hard
8B-Base, Easy
8B-8a5e, Hard
50
45
Accuracy
40
30
15
20
GRPO
-
DAPO
- Lite
PPO
Figure 12: Test accuracy of non-aligned models trained via three RL methods, i.e., Lite PPO (ours),
GRPO (Shao al. 2024) and DAPO Yu et al. 2025).
Building on the in-depth mechanism analysis, we derive two practical guidelines for non-aligned models: (i) For small and medium-sizen base models, advantage normalization (Section (41.3) significantly boosts performance by transforming sparse rewards into robust signals via group- mean and batch-std normalization. (ii) Token-level loss aggregation is also highly effective, shown in particularly for base model architectures.
Section 1.
Therefore, integrate both techniques, called Lite PPO, into non-aligned models that the vanilla PPO loss without the critic. As shown in Figure 17, Lite PPO outperforms GRPO and DAPO, technique-heavy method with Group-level Normalization, Clip-Iligher, Overlong Reward Shaping, Token-level Loss, Dynamic Sampling. Specifically, Lite PPO exhibits stable upward trend on non-aligned models, while other policies collapse after peaking. This advantage arises from the normalization in Takeaway 2, which mitigates interference from homogeneous reward distributions in mixed datasets. Additionally, this gain stems from adopting token-level loss aggregation, which more effective for base models.
6 CONCLUSION
We present systematic evaluation of RL techniques for LLMs under unified framework, addressing fragmentation in methodology and practice. By analyzing normalization, clipping, and filtering, we reveal that simplicity outperforms complexity: Lite PPO, combining only two core techniques, surpasses heavily engineered algorithms. Our findings highlight the importance of context-award design and challenge the trend of over-complication in RL4LLM. We provide actionable guidelines for technique selection and advocate for standardized, reproducible practices that balance theoretical soundness with practical efficiency.

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. In this study, no human subjects or animal experimen- tation was involved. All datasets used, including Easy, Medium and Hard datasets, were sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. We have taken care to avoid any biases or discriminatory outcomes in our research process. No personally identifiable infor- mation was used, and no experiments were conducted that could raise privacy or security concerns. We are committed to maintaining transparency and integrity throughout the research process.
REPRODUCIBILITY STATEMENT
We have made extensive efforts ensure the reproducibility of our work. The baseline models we evaluated are detailed in Section 3.2 and Appendix D. We provide not only the specific models and API versions, but also the exact sampling parameters we used. The evaluation metrics are described in Section 3.1. We will release our benchmark dataset and evaluation code upon paper acceptance to facilitate reproduction and future research by the community.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce-style optimization for learning from human feedback in llms. In Lun- Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 12248-12267. Association for Computational Linguistics, 2024, doi: www.oo......oooooooooooooo..o..o 11886653vlll /t0////2/22222/2///// acl- Long. 662
URL
Marcin Andrychowicz. Anton Raichuk, Piotr Stanczyk. Manu Orsini, Sertan Girgin, Raphaël Marinier, Léonard Hussenot, Matthieu Geist, Olivier Pietquin, Marein Michalski, Sylvain Gelly, and Olivier Bachem. What matters in on-policy reinforcement learning? A large-scale empirical study. CoRR, 0bb/2006.05990,, 2020. URL htt.::ttttst.ttitsee wwwp:////////oooooooooo...ssssss abs 2006. 05990
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-ml: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506. 13585, 2025.
Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. GPG: A simple and strong reinforcement learning baseline for model reasoning. CoRR, abs/2504.02246, 2025. doi: 10.48550/ ARXIV.2504.02566. httthttt: httpp:://ab...oog10...555/////// 25w.::ii...5....222
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters deep policy gradients: A case study on PPO and TRPO. CoRR, Abs/2005.12729 2020. UtLhttss:: arwii.iiiv.. org. / /abs/////// /2005.2005.. 1220005.1270 12729
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 1 11-16, 2024, pp. 3828-3850. Asso- ciation for Computational Linguistics, 2024. doi: hhttps://dol 1tt3////.....///22222/3
URL
211
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et Skywork open reasoner technical report. arXiv preprint arXiv:2505.22311 2025a.
Zhiwei He, Tian Liang. Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song. Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath-1033 1033: A large-scale, challenging, decontaminated, and verifiable mathematical
10

---

Published as a conference paper at ICLR 2026
dataset for advancing reasoning. CoRR, abs/2504.11456, 2025b. doi: 10.48550/ARXIV25500 11456. URL
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, ew net/ /
2021. URL hwt.thtttppppeneeeee
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient efficict rIhf algorithm with robustness to both prompt and reward models, 2025. URL WWlLhtttps://rxxv..oom org/bbs
abs 2501
03262
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances 2020.
in Neural Information Processing Systems,
Nai-Chieh Huang, Ping-Chun Hsieh, Kuo-Hao Ho, and I-Chen Wu. Ppo-clip attains global optimality: Towards deeper understandings of clipping. In Michael J. Wooldridge, Jennifer G. Dy. and Sriraam Natarajan (eds.), Thirty-Eighn AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty- Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances Artificial Intelligence, EAAI 2014, February 20-27, 2024, AAAI Press, 2024a. doi: 10. 10.1609/AAAI. V38111 1E05111.299155 1.29154. 609 aasi http://aaa..v.11111.. ww8.111111... 29154
Vancouver, Canada, pp.
URL https:: wtt.oi..oggg... www.//////............//////.///
Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The N+ implementation details of RLHF with PPO: A case study on tl:dr summarization.
CoRR, abs/2403, 17031, 2024b. doi: 10448550/ARXV:2243.... 17031. URL https://doi.. /rg/ 10. 45544850//aa http/arxiivv 2203 17031
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21277 2024. URL/Bttps://rxxxwoorrg http:/bb/2211.12222
Ruinan Jin, Shuai Li, and Baoxiang Wang. On stationary point convergence of ppo-clip. In The
Tweifth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: http::://ppeeeeviiii net forum?id uznKICpR
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free!, 2019. WTRL/http::///
rev ew /tt/foouune dd d=rl 1gGGL.DE
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Sanmi Koyejo, S. Mohamed. A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.). Advances Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL
nips ec paper www.efiileeeeeeee htmm
/hash/
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. CoRR, abx/2505.248644 2025a. doi: 10448550/ARXIV.255 2446444 URL httpps:///oi..... org/ ww.448550aaii.222 2tt.:iii.250..2555..
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding ri-zero-like training: A critical perspective. CoRR, 2ttps//25002203,,,. 2025b.
doi: 10.4850/ARXX1.250320777 1 20783
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimovv Proximal policy optimization algorithms. CoRR, abs/1707.0633 2017. URL htttp://aaxiv.org///eo 1707. 06347
11

---

Published as a conference paper at ICLR 2026
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018. URL httpaapppp//...../oooo
abs/1506. 02438
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li. Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv: 2402.03300, 2024. URL
https: org/ wwwpgg//////// /240 03300v3
Richard S. Sutton, David A. McAllester, Satinder Singh, and Yishay Mansour. Policy gra- dient methods for reinforcement learning with function approximation. In Sara A. Solla, Todd K. Leen, and Klaus-Robert Müller (eds.), Advances in Neural Information Process- ing Systems 12, (NIPS Conference, Denver, Colorado, USA, November 29 December 1999], pp. 1057-1063. The MIT Press, 1999. URL 1713 p113-policy-o gradiertddtnee methods for rein -eapteenttteerrannn with function approximatic
ni nips. wcppaerr/pppppppppppppppp
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Sealing reinforcement learning with Ilms. arXiv preprint arXiv:2501.12599 2025.
Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, Zichen Liu, Haizhou Zhao, Dakai An, Lunxi Cao, Qiyang Cao, Wanxi Deng. Feilei Du, Yiliang Gu, Jiahe Li, Xiang Li, Mingjie Liu, Yijia Luo, Zihe Liu, Yadao Wang, Pei Wang. Tianyuan Wu, Yanan Wu, Yuheng Zhao, Shuaibing Zhao, Jin Yang. Siran Yang. Yingshui Tan, Huimin Yi, Yuchi Xu, Yujin Yuan, Xingyao Zhang. Lin Qu, Wenbo Su, Wei Wang, Jiamang Wang, and Bo Zheng. Reinforcement learning optimization for large-scale learning: An efficient and user-friendly scaling library. CoRR, abs/2506.06///// 0222, 2025. doi: 10.48550/ARXIV. 2506.06122. 122. http:tts:
org / 10. 48550 www.so/arxiv...o 2011
Siwei Wu, Zhongyuan Peng, Xinrun Du, Tuney Zheng, Minghao Liu, Jialong Wu, Jiachen Ma, Yizhi Li, Jian Yang, Wangchunshu Zhou, Qunshu Lin, Junbo Zhao, Zhaoxiang Zhang, Wenhao Huang. Ge Zhang, Chenghua Lin, and Jiaheng Liu. A comparative study on reasoning patterns of openai's ol model. CoRR, abs/2410. 13639, 2024. doi: 10. 10048550/ARXIV.2210..556 URL
https: org/10..oo.ooooooooooooo.o.r.1.. 48550 201550/axii10 arxiv 13639
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https: /ttps:://axivv.... org/ abs /25/255555 09388
Qiying Yu, Zheng Zhang, Ruofei 7hu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang. Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open-source LLM reinforcement learning system scale. CoRR, abs/ 14466 2025. doi: 10.48550/ARXIV. 14476
3503.11476. URL
org/ 10.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. CoRR, abs/2503. 1882, 2025. doi: 10.48550/ARXIV.2500:5 18892. URL phttps://dle org/10 78550/arXiv 220...550... 18892
Jixiao Zhang and Chunsheng Zuo. GRPO-LEAD; A difficulty-wwar reinforcement learning approach for concise mathematical reasoning in language models. CoRR, abw/2504.00696. 2025. doi: 10. 48550/ARXIV.2504006666 htRt:ttss:: // www.oo........ 4 48550/ rxiv. 2504 09696
12

---

Published as a conference paper at ICLR 2026
Xiaojiang Zhang, Jinghui Wang. Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, Shimiao Jiang, Shiqi Kuang, Shouyu Yin. Chaohang Wen, Haotian Zhang, Bin Chen, and Bing Yu. SRPO: A cross-doman implementation of large-scale reinforcement learning on L.M.. CoRR, abs/2504. 14286, 2025. doi: 10448550ARXIV.250.. 14286. 2504. 14286
URL
orrg/.orgg// 48550/
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua. Wei Shen, Binghai Wang. Yan Liu, Senjie Jin, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi. Nuo Xu, Wenbin Lai, Minghao Zhu, Haoran Huang, Tao Gui, Qi Zhang, and Xuanjing Huang. Delve into PPO: Implementation matters for stable RLHF. In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. URL https::/ppentevier net forumm id- www.iiEmiiii....
2023.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, and al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.neh, 2025, URL https //openreview.oo net
13

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
Large Language Models (LLMs) were used to aid in the writing and polishing of the manuscript. Specifically, used LLM to assist in refining the language, improving readability, and ensuring clarity in various sections of the paper. The model helped with tasks such as sentence rephrasing, grammar checking, and enhancing the overall flow of the text.
important to note that the LLM was not involved in the ideation, research methodology. or experimental design. All research concepts, ideas, and analyses were developed and conducted by the authors. The contributions of the LLM were solely focused on improving the linguistic quality of the paper. with no involvement in the scientific content or data analysis.
The authors take full responsibility for the content of the manuscript, including any text generated or polished by the LLM. We have ensured that the LLM-generated text adheres ethical guidelines and does not contribute to plagiarism or scientific misconduct.
B DETAILED EXPERIMENTAL. SETUP
B.I PARAMETERS
We employ ROLL, user-friendly and efficient open-source reinforcement learning framework, to implement our pipeline. Subsequently. the key parameters observed during the training process are presented as follows. See our code config file for more details the parameters.
B.2 PROMPT
In this work, we incorporate the following instruction into the system prompt to encourage the model to better demonstrate its reasoning process: "Please reason step by step, and put your final answer within \boxed()." This setting is designed to guide the model to perform step-by-step reasoning and explicitly present the final answer in the form of \boxed( ), thereby enhancing the clarity and readability of the output.
B.3 TRAINING DATASETS
To ensure reproducibility and fairness, exclusively use open-source datasets for training, including SimpleRL-Zoo-De (Zeng et al. (2025) and Deepmath (He et al. (025b). To comprehensively examine how problem difficulty affects the RL technique's 's performance. we randomly sample from the datasets, removing an excessive proportion of examples whose ground-truth label is simply "True" or "False" This adjustment addresses the ostensible positive phenomenon, where models produce correct binary answers from erroneous reasoning chains, thereby introducing noisy supervision that compromises training quality (please refer to Appendix B. for case studies).
Easy Data : We randomly sample 5, 000 entries from SimpleRL Roo-Data- Sata-Easy, which consists of problems drawn from GSM8K and MATH-500-levell.
Medium Data: We select the 5, 000 easiest examples from the DeepMath-103t dataset, based on their assigned difficulty annotations.
Hard Data: We randomly sample 5, 000 entries from DeepMath- with sampling probability proportional to each entry's assigned difficulty level.
14

---

Published as a conference paper at ICLR 2026
seed: 42
max _ax_steps: 500
save save_steps: 20
logging_steps: waal_steps: 1
rollout_batch_siee size: 128
prompt_length: 1024
response_length: 8000
ppo_ epochs: 1
adv estimator: "reinforce" init kl coef: 0.0 asyne_generate_level
actor_train:
training_args:
learning_rate: 1.0e-6
weight_decay:
ver..awiie_itrchh.atttoessseee 4
gradddet_sccmmmlttiiooosteees 32
warmup_ratio: 0.1
warmup_steps: 50
num_train_epachs: 50
actor actor_infed infer:
generating max_new_tokens: seresponse_eength)
top_p: 0.99
top_k: 100
num :ummsbems::
temperature: 0.99
num nun_rettrr_sequencee,
C DETAILED EXPERIMENTAL RESULTS
C.1 BASELINE
Overview of training accuracy and response length
Qwen3-4B-Ban
Qwen3-8B.a -BB-asee
Qwen3-4B
Qwen3-8B
96
96
Accuraytt)
25
8g
Accuracy(%)
80
60
80
45
78
30
30
72
360
600
900
1200
300
600
900
1200
2000
400
600
800
200
400
600
800
Step
Step
Step
Step
Response Lengthik)
Resparse Longth(k)
Response LengthK)
Response LengthK)
6.0
8.5
2.4
3.0-
1.8
360
600
940
1200
300
600 Step
900
1200
200
400
600
800
200
400
600
800
Step
Step
Step
Easy
Figure 13: Test accuracy and response length of four model variants: Qwen3-44-BBss Twen3-8B-Base 3-4B, and Qwen3-8B across different data difficulty.
15

---

Published as a conference paper at ICLR 2026
Test accuracy of Base models
Math500
OlympiadBench
AMC23
Minerva Math
AIME24
AIME25
44
28
20
Qwen3-8B Qwen3
12
24
24
16
600 920
1200
303 900 1200
300
500
1200
300 500 900 1200
300 600 905 1200
300 500 1250
Math500
OlympladBench
Minerva Math
AIME24
12-
Accuracy (N6)
35
45
10
28
20
25
500
1200
300
1209
300
200 Stop
300 1200
200 1200
Stop
Stop
Stop
Step
Stop
Easy
Medium
Hard
Test accuracy of Aligned models
Math500
OlympladBench
AMC23
Minerva Math
AIME24
AIME25
Qwen3-4B
Accuracy
43
42
66
82
01
600
800
200
450
600
450
600
200
600 600
200 400 850
200 400 600
OlympladBench
Qwan3-88
Accuracyly Sourr
92-
48
39
30-
43
48.
37
200
400
800
400
Stop
Stop
Step
Stop
Stop
Stop
Easy
Medium
Hard
Figure 14: Middle 2 rows: Accuracy over training iterations of Base models. The first row presents results of Qwen3-4B-Base The second row shows results of Owen3-88-Base Bottom 2 rows: Accuracy over training iterations of aligned models. The first row presents results of Qwen while the second row shows results of Qwen3-8B.
C.2 RESULTS OF ALIGNED MODEL.
Following analysis in section 4.2.1 that Clip Higher provides stronger benefits aligned models, we instantiated variant of LitePPO that combines mixture normalization and Clip Higher, trained on both the Easy and Hard datasets.
As shown Tables [ and 2 our simple recipe significantly improves the reasoning ability of the aligned model and consistently outperforms DAPO, particularly when trained on the hard dataset. This confirms that LitePPO remains efficient for strong, instruction-tuned models.
MATH-500
OlympiadBench
MinervaMath
AIME24
AIME25
AMC23
Average
DAPO
90.57
64.82
43.43
47.92 49.17
37.54
79.69
60.66
Lite PPO
91.43
66.01
44.72
39.81
81.88
62.17
Table 1: Results using Qwen3-8B (aligned model) trained on the Easy dataset.
MATH-500
OlympiadBench
MinervaMath
AIME24
AIME25
AMC23
Average
DAPO
93.02 94.77
69.15 71.12
45.73 46.60
52.50 49.58
39.17 47.50
85.31 92.19
64.17
Lite PPO
66.96
Table 2: Results using Qwen3-8B (aligned model) trained on the Hard dataset.
16

---

Published as a conference paper at ICLR 2026
C.3 VALIDATING METHOD EXTENSIBILITY
We have conducted additional experiments Llama3-8B, recorded the average score among six benchmarks,i.e. AIME 24, AMC23, GSM8k, MATH-500, Minerva Math, OlypiadBench. The results in Table show that, compared GRPO and DAPO, our Lite PPO demonstrates the best average performance, indicating the portability of our method..
MATH-500
OlympiadBench
AIME
AMC23
GSM8K
Minerva Math
Average
GRPO
19.8
8.3
3.7
18.2
77.3
8.5
22.63
DAPO
25.1
7.6
4.6
18.4
77.1
9.2
23.67
Lite PPO
27.3
9.7
9.1
17.9
79.6
9.6
25.53
Table 3: Results on Llama3-8B.
We have conducted additional experiments evaluate the generalization performance of our method. Specifically, we test the Qwen3-8b- Ba--aee trained by LitePOO on several out-of-domain reasoning tasks, including coding (LCCBv5,, LCB-v6), Interdisciplinary QA (GPQA), and Language Understanding (MMLU-Pro)) The results in Table 4 show that Lite PPO achieves the pass @ score of 25.08 on LCB-v5, 19.71 on LCB-v6, 46.63 on GPQA, and 62.38 on MMLU-Pro, demonstrating our method's excellent generalization ability.
LCB-v5
LCB-v6
GPQA
MMLU-Pro
GRPO
22.94
18.28
42.49
61.84
DAPO
24.01
19.00
45.08
60.60
LitePPO
25.08
19.71
46.63
62.38
Table 4: Pass@ score on other reasoning modalities.
D CASE STUDY OF CLIP HIGHER
We present a comparison of token distributions between the base model and the aligned model.
As shown in Figure 18 compared to the base model, the aligned model has very few preferred tokens with high probability in the initial stage. Therefore, a higher clipping upper bound can effectively bridge the probability gap between tokens and alleviate the entropy collapse. For these models, raising the upper bound expands the permissible range of policy updates, which turn facilitates more diverse action sampling and enhances exploratory behavior during training. This mechanism preserves higher entropy while simultancously increasing the probability of identifying optimal solutions, as evidenced by improved evaluation metries.
Building on our token-level demonstration of Clip-Higher's behavior in section 4.2. we now analyze its impact on reasoning logic through token-level linguistics. As illustrated Figure 20] setting an upper bound to 0.2 imposes stringent constraints on policy updates by limiting substantial probability deviations for individual tokens. Under these stricter conditions, our analysis reveals that clipping predominantly affects connective tokens such as "therefore", "if", and "but". These tokens frequently appear at the beginnings sentences, serving as key semantic markers or transition words within dialog generation. Such connectors often introduce new directions in reasoning. However, their probability ratios between updated and old policies frequently exceed clipping thresholds, triggering aggressive suppression in PPO optimization. While this traditional clipping ensures stability in the overall token distribution, may restrict the model's capacity to generate innovative or diverse argumentative reasoning structures by limiting flexibility in the use of discourse-level connectives.
17

---

Published as a conference paper at ICLR 2026
4B-Base model with different standard deviation calculation
Olympla dBench
Accuracy
S. 3
40
8
36
40
10005000000
505
Math500
Olympia dBench
AMC23
Minerva Math
AIME
AIME25
55-
14
Accuracy(5) 8 St
32-
20
10
Hard
24
30
500
1650
250 750 3000
250
500
1000
250
500 1900
750
250
750 1050
Step
Step
Step
local std
global std
8B-Base model with different standard deviation calculation
Math500
OlympiadBench
AMC23
Minerva Math
AIME24
AIME25
16-
Accuracylts
72
50
32-
16
12
10
300 820 900 1200
300 630
900 1200
MDD
1200
600 950 1200
330 600 900 1200
300 900
1203
OlympiadBench
Minarva Math
AIME24
AIME25
Accuracy(
78
16-
72-
32
20
300 900
1205
300 650 1200
300 M60
1200
600
530
900 1200
300 400 1205
Step
Step
Step
Step
Step
Step
global std
Figure 15: Accuracy comparison of Base models with different standard deviation calculation. Top rows: Accuracy of Qwen3-4B-Bas with different standard deviation calculation. The first row uses the easy training dataset, while the second row uses the hard training dataset. Bottom rows: Accuracy comparison of - with different standard deviation calculation. The first row uses the easy training dataset, while the second row uses the hard training dataset.
OlympiadBence
76
64
24
Medium
Accurecy
60
46
10
96
31
40
38
Math500
OlympiadBane anchh
46.
12
Accurocy (85)
60
10
24-
Harder
48
23
20
Step
Step
Step
sequence- level loss
token-levor lass
Figure 16: Test accuracy of sample-level loss and token-level loss on medium and extremely hard datasets.
E OVERLONG FILTER
E.1 REPEAT RATIO
To further investigate the mechanism by which the overlong filter the aligned model, adopted a rule- based approach to efficiently identify whether overlong samples are caused by the inability control the end-of-sequence (EOS) token, resulting in repetitive generation without termination.
18

---

Published as a conference paper at ICLR 2026
Qwen3-4B-Base model
Math500
OlympiadBench
AMC23
Minerva Math
AIME24
AIME25
Accuracy (86)
40
26
20.
32
Easy
4D-
20-
250
500 750
250
505
750
250
300 750
250 500 750
250 500 750
250 500 750
Math500
OlympiadBem ench
AMC23
Minerva Math
AIME
AIME25
36
Accuracy (%6
Hard
32
28
20-
30
100 230
103
160 730 300
200
300
400
100
350
400
150 200
400
100 200
300
Step
Step
Step
Step
Step
Step
GRPO
DAPO
Lite PPO
Qwen3-8B-Bae model
Math500
DiympladBench
AMC23
Minerva Math
AIME24
AIME25
Accuracy (%)
78
16
Easy
24
20
230 500 750
230 500 730
253
300
730
250 500 755
230 5:0 750
230 300 730
Math5500
OlympiadBench
AMC23
Minerva Math
AIME24
AIME25
Accuracy (36)
36
56
28-
20
150 100 450 600
350 105 150 650
0
150
100 Step
850 630
150
330
450 R00
150 300 450 600
150 300 450 600
Step
Step
Step
Step
Step
GRPO
DAPO
Lite PPO
Figure 17: Test accuracy of non-aligned models trained via three RL methods, i.c., Lite PPO (ours).
GRPO (Shao et al. 2024) and DAPO Yuu et 2025).
Swen3-4B-Bass
Qwen3-4B
clp-0.20 upper clp-0.28
dlip-0.28
30
Praportion (9)
40
40
10,0.3)
[0.3.066]
(0..885) [0a50 95) (0.95,0.99 (0.99.1.0)
10.0.30
[0.3.0.8]
[0.6,088) (0.85.0.95)
(0.95.0.90] (0.99110)
Probability Intervals
Probability Intervals
Figure 18: Predicted probability distributions of Qwen3-weee -BB Base (left) and Qwen3-44 4B (right) under two clipping upper bound E {0.200 0.28).
Specifically, trace backward from the truncation point to locate repeated content. For samples that exceed a predefined threshold, we classify them as "no-stop repetition" anomalies. By calculating the ratio of repeated samples all overlong samples, known the repeat ratio, quantify the model's capability at the current step to model termination behavior in sequence generation.
E.2 EXAMPLES OF OSTENSIBLE POSITIVE PHENOMENA
As demonstrated in Figure I1 in the main text, observe that models with weaker capabilities tend to continue generating content aimlessly even after correctly reasoning and providing the correct answer, until exceeding the output length limit. Such false positives, although receiving reward
19

---

Published as a conference paper at ICLR 2026
Qwen3-8B-o
Qwen3-BB
1.00
100
upper clipm0.20
upper clp-0.20
upper dlpp-0.28
upper cip-0.28
Proportion (%)
60-
60
20
20
33
(0,0.3)
[0.3.0.6)
(0.60085)
(0.85.0.55)
(0.35.0.59)
(0.39.1.0)
(0.0.3)
10.3.0.61
(0.6.0885)
10.25,0.55)
(0.95,0.99]
[0.99.1.0)
Probability Intervals
Probability Intervals
Figure 19: Predicted probability distributions of Qwen3 -BB Base (left) and Qwen3-88 8B (right) under two clipping upper bound E {0.20, 0.28J.
Dusston Poim 5V60775 mibport EovetrwsAays if poit 548 conntrakin ROUSS, sun countratio 5257
tee clue
sigh cloa
Therefore
high clip=0.20
mast
mabue about from A
sone MECMINE
that
le vasee sn mus
1os ver main inner commennti MOSEEM
Dut
poat seat
maybc
high clip=0.28
Alleratielly
Walt
Web
High cip-- 20
high clip-0.20
IC J.
155 Numberik)
Figure 20: Left: A case study under the same prompt across various clipping upper bounds. Right: The trigger differences of various upper bounds at the top 20 tokens with the highest clip frequencies.
of 1 through rule-based evaluation, introduce noise into the model during training. We present representative case for illustration, as shown in Figure 21
F DETAILED PRELIMINARIES
F.1 PROXIMAL POLICY OPTIMIZATION (PPO)
Proximal Policy Optimization (Shhllmn et al. 2017) is a widely used actor-critic algorithm grounded in the policy gradient framework. improves the stability of policy learning by optimizing clipped surrogate objective that restricts the divergence between the new and old policies during training. The PPO objective is:
(P) = E, P(Q)), o (O14)
lol
To(orla, 0ct) To.sa (or oct) A,, clip
(ro 1 1-6, ltc) At). To.a (orlg. o2e)
(4)
min
where Ta and Te. denote the current and old policy models, respectively. q and o represent the sampled question and output sequence, with or the 1-th token in is clipping hyperparameter
20

---

Published as a conference paper at ICLR 2026
angle 1theta \trac22/p155)
equetion V. #1oce(2(ktette) 2'cos((thetal
First, use the double-ang identity for cosine, which states that V 1oos(2)theta) 21cos"2(
V::
into givan equation:
M (1coos2\/teta) Rloce((thetta + ai2/coo"a((tneta) - f) (theea) 1.
Therefore, the initial equation M 21hce(2 21cce( (theta) = holds true for V Atheta Mrra([2pp)(S) V. Thus, the anewer is: |[ \boxod(o). U
We have just shown step by stop that the aquation 21cos/2/thetal (thtt)) 1 0 holds true V (thota (free22pp((5) V. with final affirmation that V. (boked(o) Va the appropriate enswer.
Do you have any other questions? Vusee
Next, that (oos((/rac(22p hww.mmm/mmmniinlllllnnne
root of the quartic polynemial V x14 x*3 6x12 -
www.aa/iwwwiiiiiiiiiinnnnnnnnnne www.://iiiiiiiimmmmmnnnnnnn.nnni Inininininnin in Inintnininin in Inininininin in lninkenlllennnninin
Figure 21: An ostensible positive case, which cannot be terminated after the answer is given the end of inference.
for stabilizing updates. A, is the advantage at step 1, typically estimated via Generalized Advantage Estimation (GAE) (Schulman et 2018). The objective encourages the new policy improve advintege-weighted probabilities while constraining changes within trust region.
F.2 GROUP RELATIVE POLICY OPTIMIZATION (GRPO)
Group Relative Policy Optimization (GRPO), proposed in DeepSeekMath (Shao al. 2024). elimi- nates the value function (critic) and instead estimates the advantage by normalizing rewards within a group of sampled responses for the prompt. Specifically, for prompt x with G responses and associated rewards {ri) i=1 the group-normalized advantage given by:
A mean (frige) std((r))
(5)
The effectiveness of the above normalization method can be understood from the perspective of reward shaping. By emphasizing the differences among candidate outputs for the same prompt, it effectively preserves the reliability of the gradient signal, even in sparse reward settings (Hu et al.) 2020). Instead of adding KL penalty to the reward, GRPO directly regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss. The overall surrogate objective is:
JGRPO (0) E, /// P(Q), (o0)-11aan (01g)]
lail min (rir(0) Ace, clip (ri,(0), 1 1-e, 1+6) Art)) 9DKL. (mo Troe]
G
(6)
where (0)
Tan (+) and are hyper-parameters, and DKr. denotes the KL divergence between the learned policy and a reference policy Rref.
Ta mn..o.lgg.o..c
F.3 DECOUPLED CLIP AND DYNAMIC SAMPLING POLICY OPTIMIZATION (DAPO)
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) Yu et al. 2025) is recent RL method designed to address the unique challenges in LLM reasoning. For each question with gold answer a. DAPO samples a group of G outputs {o, ) from the old policy, computes their rewards, and maximizes the following surrogate objective:
21

---

Published as a conference paper at ICLR 2026
(D) [(q,a)),, (ool)) f [[)]
G lod
min (0i)(0) clip (ri,e(0), Clow hight) A.:)]
21=1 i= t=1
(7)
where A is the group-normalized advantage. In addition, DAPO decouples the upper and lower clipping ranges (Cows (high) better support exploration, dynamically filters out samples where all responses are correct or incorrect, aggregates losses at the token level, and applies special reward shaping for overlong or truncated responses.
22


Reviews:
[{"review_id": "rUBWurzab4", "reviewer": "Reviewer_rKt2", "summary": "This paper presents an empirical analysis of components of efficient RL pipelines for LLMs (in particular GRPO- and DAPO-style techniques) on both non-aligned and aligned Qwen3 models. The authors study the impact of data difficulty, advantage calculation and normalization, clipping strategies, loss aggregation granularity, and reward shaping via overlong filtering, all under a unified PPO-based setup. Based on these observations, they propose LitePPO for non-aligned models, which uses group-level mean and batch-level standard deviation for advantage normalization, together with token-level loss aggregation. Experiments on Qwen3 4B and 8B non-aligned models evaluated on Math500, OlympiadBench, AMC23, Minerva Math, AIME24, and AIME25 show that LitePPO outperforms GRPO and DAPO.", "questions": "1. For Figures 3, 4, 6, 8, 9, 10, and 12, could you clarify precisely what the y-axis “accuracy” refers to? Is it the average accuracy across all six evaluation benchmarks, or a subset, or a particular held-out split from the Easy / Medium / Hard training datasets themselves?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Systematic and fairly extensive analysis of the effect of data difficulty, advantage normalization (including std/no-std variants), clipping strategies, loss aggregation, and overlong filtering on RL performance for LLMs.\n\n- Evaluation covers both non-aligned and aligned model variants and two parameter scales (4B, 8B), which makes the conclusions more convincing than single-model studies.\n\n- The paper is clearly written and well structured; the individual “takeaways” for each component are easy to follow and practically useful.\n\n- The proposed LitePPO recipe is simple yet effective.", "weaknesses": "- Statistical robustness / variability:\n\n    - It is not clear how many random seeds are used; Appendix C suggests a single seed (seed=42). If this is the case, the results are potentially sensitive to randomness.\n\n    - Showing mean and variance over multiple runs (e.g., multiple seeds) would make the empirical conclusions stronger, especially in Figure 12. Even if re-running all experiments is too expensive, clearly stating the number of runs and acknowledging this limitation would help.\n\n- Minor typo:\n    - Line 388: “As illustrated in As shown in Figure 9...”, remove one of “As illustrated in / As shown in”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SifMUmX10l", "reviewer": "Reviewer_G1m5", "summary": "This paper conducts a systematic evaluation of implementation choices for RL with LLMs, covering advantage-normalization variants, clipping with a higher upper bound, loss aggregation at the token vs. sequence level, and overlong-response filtering, within a unified PPO-style framework. Experiments are run on math-reasoning benchmarks using Qwen3-4B/8B (both base and aligned variants). From these studies, the authors distill seven empirical takeaways and introduce a minimalist recipe, Lite PPO, which pairs group-mean with batch-std advantage normalization and token-level loss under a vanilla PPO objective without a critic. On base models, Lite PPO achieves consistent gains over GRPO and DAPO across several math datasets.", "questions": "- What motivated restricting experiments to Qwen3? Could you report at least a final-recipe run on another family (e.g., Llama or Mistral) to assess portability?\n- Can you apply the Lite PPO recipe to aligned models and compare against strong aligned baselines to clarify its utility when the goal is the best final system?\n- Several techniques and experimental patterns resemble DAPO. Could you delineate, in methodological terms, how your approach differs (objective, normalization, clipping, loss aggregation), and provide ablations isolating the incremental contribution beyond DAPO?\n- The higher-clipping “scaling law” appears to rest on a single family with limited sweep points. Do you view this as a size-dependent trend rather than a law? If not, what additional evidence (broader sizes, denser sweeps, statistical tests) supports the stronger terminology?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Organizes scattered RL techniques into a coherent, condition-aware evaluation (base vs. aligned; easy vs. hard), yielding concrete and readable takeaways.\n- Provides careful ablations and useful diagnostics (e.g., entropy and ratio behavior; token-level analyses) that improve interpretability of clipping and aggregation choices.\n- Delivers a simple, reproducible recipe (Lite PPO) that reduces complexity yet performs strongly on base models, offering immediate practical value.", "weaknesses": "- The contribution of this paper is mainly an empirical synthesis of known implementation choices rather than an algorithmic or theoretical advance; no new RL objective or formal analysis is introduced, which constrains the paper’s originality compared to prior work\n- If the practical objective is the strongest final model, the focus on base models leaves uncertain whether the proposed recipe yields meaningful gains for aligned/instruction-tuned models that start from stronger baselines.\n- Evidence is confined to math reasoning; presenting the work as a general “roadmap” for RL with LLMs risks over-generalization without results on other reasoning modalities (e.g., logical, strategic, open-ended).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UMsCc0omq2", "reviewer": "Reviewer_qNU9", "summary": "This paper presents a systematic and rigorous evaluation of reinforcement learning (RL) techniques for improving reasoning capabilities in large language models (LLMs). The authors address the current fragmentation in RL4LLM methodologies by reproducing and analyzing popular RL \"tricks\"—such as normalization, clipping, loss aggregation, and filtering—under a unified framework. Through extensive experiments across diverse model sizes (Qwen3-4B/8B), architectures (base vs. aligned), and dataset difficulties (easy/medium/hard), the study offers actionable insights into the mechanisms and applicability of each technique. A key contribution is the proposal of Lite PPO, a minimalist combination of two techniques (group-mean + batch-std normalization and token-level loss aggregation), which outperforms more complex methods like GRPO and DAPO in critic-free policy optimization.", "questions": "How might your guidelines adapt to non-mathematical tasks? Are there techniques whose effectiveness is highly domain-dependent?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Comprehensive and Reproducible Evaluation: The paper leverages a unified open-source framework (ROLL) and over 160 independent experiments, ensuring robust and statistically meaningful conclusions. The ablation studies (e.g., standard deviation removal in normalization, clip-bound scaling laws) are particularly insightful.\n\n2. Practical Guidelines: The authors translate empirical findings into clear, scenario-specific recommendations (e.g., token-level loss for base models, sequence-level for aligned models), addressing a critical need for standardization in RL4LLM.\n\n3. Minimalist Innovation: Lite PPO demonstrates that simplicity can outperform heavily engineered methods, challenging the trend of over-complication and offering an efficient baseline for future work.", "weaknesses": "1. Experiments are confined to Qwen-family models and mathematical reasoning tasks. While math is a common benchmark, broader validation on diverse domains (e.g., code generation, commonsense reasoning) would strengthen claims of generalizability.\n\n2. While Lite PPO is promising, more ablation is needed to disentangle the contributions of its two components (normalization vs. loss aggregation) across all settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D346pcPnkX", "reviewer": "Reviewer_MsEF", "summary": "This paper shares a systematic review of variations of experimental settings in LLM-RL training to provide a unified guideline for practitioners. The review includes analysis on advantage normalization, PPO clipping and loss aggregation. In each category, the paper provides empirical results with different model sizes and dataset difficulty and a practical recommendation based on the result. Finally, the paper proposes Lite PPO, a combination of two techniques (group-mean/batch-std advantage normalization and token-level loss aggregation), improves performance of non-aligned LLM models.", "questions": "- Computation cost is also another important dimension when people select techniques. Do authors have any insight around this? I imagine that the most of techniques analyzed in this paper won't impact the computing performance though.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and comprehensive. Contributions are clearly stated in introduction and supported by experiments. Addressing the lack of a standard guideline in the LLM-RL field is important to allow practitioners to understand choice of techniques. From this point of view, this paper tackles an important problem in this domain. Overall, the experimental results are well conducted and the proposed Lite PPO algorithm is an natural extension to GRPO based on the findings provided in this paper.", "weaknesses": "Although there isn't obvious flaws in the paper, there are comments to improve the quality of the analysis further.\n- In Section 4.2.3, explanation of why 8B model doesn't have \"scaling law\" of the upper bound clipping parameter strengthens this analysis. Analysis on trends of LLM's outputs might help explain this.\n- In Section 4.4.1, there is a lack of explanation around what leads to the experimental results shown in Figure 10 and 11. What learning dynamics influences these result? This analysis seems vital to understand the choice of overlong filtering.\n\n\nMinor grammatical errors:\n- Line 388, `As illustrated in As shown in` -> `As illustrated in`", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)