Escaping Policy Contraction: Contraction-Aware PPO (CaPPO) for Stable Language Model Fine-Tuning
Authors: Dun Yuan
Di Wu
Xue Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Policy Contraction
Proximal Policy Optimization
Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement learning from human feedback (RLHF) with proximal policy optimization (PPO) is widely used but often yields less diverse outputs than supervised fine-tuning, suggesting an effect in which the policy’s support contracts during on-policy optimization. We formalize this “policy contraction” with the Support Retention Ratio (SRR)—the share of SFT completions that retain non-negligible probability under the RL policy—and additionally track token-entropy, Kullback–Leibler (KL) divergence to the reference, and repetition. We propose Contraction-Aware PPO (CaPPO), a minimum-norm multi-gradient update that co-optimizes reward, entropy, and KL, paired with a controller that steers exploration toward a target token entropy. On HH-RLHF, Summarize-from-Feedback, and UltraFeedback with Qwen2-7B, Qwen2.5-14B, Mistral-7B-Instruct, and Llama-3-8B-Instruct, CaPPO increases win rate by 2 to 4 points over PPO and improves diversity, gaining 0.2 to 0.3 higher SRR. The gains persist under decoding sweeps and are robust to reward scaling and critic variance. Treating reward, diversity, and stability as first-class objectives, CaPPO mitigates contraction without sacrificing alignment performance.

Published as a conference paper at ICLR 2026
ESCAPING POLICY CONTRACTION: CONTRACTION- AWARE PPO (CAPPO) FOR STABLE LANGUAGE MODEL TINE-TUNING
Dun Yuan
Di Wu
School of Computer Science
Department of Mechanical, Industrial and Aerospace Engineering Concordia University Montreal, QC, Canada di www.wuuccccco. wwuconncordaa.ca
McGill University
Montreal, QC, Canada
dun. yuan@malaammm mwgi 1llgl11,,. 11.ca
Xue Liu
Mohamed bin Zayed University of Artificial Intelligence
Abu Dhabi, UAE
School of Computer Science
McGill University
Montreal, QC, Canada
xue. 1 1iuuuunn Luummbzuai... acc.e
ABSTRACT
Reinforcement learning from human feedback (RLHF) with proximal policy op- timization (PPO) is widely used but often yields less diverse outputs than su- pervised fine-tuning, suggesting an effect in which the policy's 'ss support con- tracts during on-policy optimization. We formalize this "policy contraction" with the Support Retention Ratio (SRR) share of SFT completions that retain non-negligible probability under the RL policy-and additionally track token- entropy, Kullback-eeibler (KL) divergence to the reference, and repetition. We propose Contraction-Aware PPO (CaPPO), minimum-norm multi-gradient up- date that co-optimizes reward, entropy, and KL, paired with a controller that steers exploration toward target token entropy. On HH-RLHF, Summarize- from-Feedback, and UltraFeedback with Qwen2-7B, @wenn.55-4B, Mistral-7B- Instruct, and PPO and improves diversity, gaining 0.2 to 0.3 higher SRR. The gains persist un- der decoding sweeps and are robust reward scaling and critic variance. Treating reward, diversity. and stability as first-class objectives, CaPPO mitigates contrac- tion without sacrificing alignment performance.
CaPPO increases win rate by to 4 points over
INTRODUCTION
Reinforcement learning from human feedback (RLHF) (Christiano al., 2017 has become stan- dard paradigm for adapting supervised language models to human preferences. A typical pipeline trains preference-basel reward model and then optimizes the policy with Proximal Policy Op- timization (PPO) (Schulman et al., 2017) while regularizing toward a supervised reference via Kullback-Leible (KL) penalty to stabilize training and curb reward hacking (Stiennon al. 2020 Ouyang et al., 2022). This recipe underpins influential alignment systems in instruction following and dialogue, and annotation costs have been reduced through Al-assisted variants (Glaese et al., 2022 Bai et al. 2022b Lee et al. 2023). Recent studies further suggest RL-based post-training can enhance language understanding and strengthen reasoning-centte models (Hu et al., [0224] Guo eet al.] 2025). Despite these successes, practitioners frequently observe decline in output diver- sity during on- on-policy fine-tuning: per-token entropy falls, repetitions increase, and many plausible alternatives under the reference model receive negligible probability.

---

Published as a conference paper at ICLR 2026
We term this phenomenon policy contraction: the policy's 's support progressively narrows as proba- bility mass concentrates on shrinking subset of completions even as reward or win rate improves. Conventional diversity proxies, including Self-BLEU (Zhu et al.; 2018) and Distinct-n, offer useful signals but are decoding-sensitivy and can be noisy (Holtzman et al. 2019 Li et al. 2015 Celikyil- maz al. 2020). To prohe support loss more directly, we use the Support Retention Ratio (SRR): the fraction of supervised completions that retain non-negligible angthhnorrmalize log- oggliiklihood under the trained policy fixed threshold. SRR measures how much of the supervised set remains likely under the trained policy. It counts the fraction of supervised completions whose length-norman normalized oollikelihood exceeds a percentile threshold set from the reference. This metric is independent of decoding and comparable across prompts.
This perspective clarifies how different post-training choices shape distributional behavior. RL--free preference-oppimizztion methods such as Direct Preference Optimization (DPO) and Kahneman- Tversky Optimization (KTO) streamline optimization and often preserve more of the SFT distri- bution, but they operate off-policy and do not directly address on-policy distributional drift that emerges RLHF (Rafailov et al.] 2023 Ethayarajh et al.] (024). PPO variants that target bet- ter credit assignment or critic robustness, like VinePPO and group- normalized approaches used for reasoning, raise performance but may still emphasize a small set of high-reward modes with- explicit diversity control (Kazemnejad et al., 2024 Mrouch 2025). Recent analyses in the verifiable-rewar setting suggest that reinforcement learning can predominantly reweight solution- bearing modes rather than expand them, which aligns with the contraction picture (Yue al. (025). At the same time, more general evidence indicates that RL can improve language understanding in NLU tasks (Hu al., 2024) and enable strong reasoning systems (Guo al. 2025), underscoring that addressing contraction is distributional problem, not a purely capability problem.
We address contraction by elevating diversity control from an auxiliary penalty to a core training objective. Standard PPO implicitly prioritizes reward maximization, using entropy and KL regu- larization as secondary terms with fixed or manually tuned weights. This scalarization makes the optimization brittle: when the reward scale is large, or critic estimates are noisy, entropy rapidly collapses and the policy contracts onto narrow set of completions. Once entropy falls below a certain level, exploration becomes ineffective, repetitions rise, and the probability mass shifts dis- proportionately to a small subset of high-rewart outputs. These dynamics explain why PPO-trained models often exhibit rising repeat rates, declining support retention, and left-shifted pogobabbbyyy histograms, even when alignment reward improves. Controlling this collapse with static entropy coefficients manually adjusted KI. weights is unreliable and sensitive dataset, backbone, and scale.
To overcome these limitations, we propose Contraction-Awan PPO (CaPPO). CaPPO reinterprets reward, entropy, and KL-to-referrece as peer objectives and computes parameter updates with a minimum norm multi gratiitr descent procedure that approximates a Pareto- improving step (Parisi
ett al. 2014 Van Moffaert et al. 2013 Momma et al. 2022 Liu & Vicente, [002]]. This avoids
brittle scalarization and ensures that progress on reward does not come at the expense of collapsing entropy or uncontrolled KL drift. In addition, CaPPO introduces an entropy mntropy-scheduling controller that monitors per-token entropy and dynamically adjusts the effective entropy coefficient: injecting exploration pressure when entropy collapses and relaxing it when entropy is sufficient. Together, these two components yield method fully compatible with PPO's elipped surrogate, which directly mitigates contraction. Empirically, CaPPO restores support retention and improves diversity met- rics while matching or exceeding task reward and win rate across different models and datasets. Its improvements are consistent across seeds, reward scales, and critic variances, highlighting robust- ness and confirming that treating entropy and KL as first-class objectives is essential for preserving diversity and accuracy in RLHF.
The main contributions of this work are as follows:
We verify the existence of policy contraction in PPO-based RLHF and introduce SRR as direct, decoding-agnostti measure of support preservation.
We present CaPPO with entropy scheduling drop-in extension that balances reward maximization with KL control and entropy maintenance through a pareto-motivated multi- objective update.
2

---

Published as a conference paper at ICLR 2026
We empirically verify contraction and show that CaPPO mitigates it across Qwen, Llama, and Mistral backbones on HH-RLHE Summarize-froe wee---fooo---aeee and UltraFeedback, im- proving SRR and diversity at competitive higher win rates. We also discuss connections to overtraining and post-training adaptability.
The remainder of the paper proceeds as follows. Section reviews RLHF foundations, RL-ffree preference optimization, degeneraaion/ddiersiiyy metrics, and multi-objective optimization. Section quantifies contraction using sampling- -baed diversity, entropy/KL trajectories, and SRR with log- probability histograms of SFT completions. Section 4 details CaPPO and the entropy-scheduling controller, including optimization geometry and practical solvers. Section 5 reports main results, ablations, robustness to reward scaling and critic variance, and decoding-indepentt independence analyses, before Section limitations and future directions.
2 RELATED WORK
RLHF and PPO. Modern alignment pipelines pair preference- rewards with PPO under KL regularization to reference model (Schulman al. 2017 Christiano et al., 2017 Stiennon jet al. 2020; Ouyang et al. 2022 Glaese et al. 2022). This recipe improves instruction following and dialogue while mitigating reward hacking via KL-to-reference anchoring. Constitutional Al and RLAIF reduce human labelling by distilling guardrails and using AI feedback, respectively, yet retain preference/RL phase (Bai et al. 20226 Lee al. 2023). Control-theoretic views of KL-regularized RL explain how anchoring shapes exploration and stability under information con- straints (Galashov et al., 2019). Recent evidence shows RL-based post-training can also strengthen NLU (Hu et al. (2024) and enable state-of-the-an reasoning systems (Guo et al.) (025). Our work focuses on a distributional side effect of this appline-policy contraction by measuring support loss directly and proposing contraction-awar modification of PPO that optimizes reward, entropy, and KL as peer objectives.
Reasoning with RL. Reasoning workloads surface weaknesses in token/value credit assignment and critic stability. VinePPO addresses step-wise credit propagation for multi-step reasoning, im- proving RL training quality under PPO (Kazemnejad et al. (024). In parallel, GRPO-style critic- free training with group normalization has seen broad adoption in reasoning systems; its effective loss and dynamics can be written as KL-regularized contrastive objective with success amplifica- tion guarantees (Mroueh (025)) and large-scale instances demonstrate strong math/code gains (Guo) al. 2025). However, recent analyses indicate that such RL procedures often reweight solution- bearing modes rather than expand the underlying support, improving pass@1 while narrowing the reachable set at larger sample budgets (Yue al., 2025). This view consistent with our find- ings: standard PPO can raise reward win rate even entropy declines, repetition rises, and the log-probability mass concentrates on fewer completions.
RL-free alignment. RL-free preference optimization aims to stabilize and simplify post-training by sidestepping on-policy updates. DPO matches the optimal RLHF policy under a specific reward parameterization via a simple preference-classffccue alignment as prospect-theoretic optimization to better capture asymmetric human preferences (Etha-) yarajh al. 2024); and active preference optimization improves sample efficiency by selecting in- formative prompts (Das et al. (024). These methods often preserve more of the SFT distribution and reduce engineering overhead. However, they operate off-policy and do not directly control the on-policy distributional drift that emerges during RLHF training.
loss (Rafailov et al. 2023); KTO reframes
Diversity and multi-objective methods. Text generation is prone to degeneration and mode col- lapse under likelihood-centric training and decoding (Holtzman et al. 2019), motivating diversity- oriented objectives (c.g.. promoting dissimilar responses) and evaluation metrics such as and Distinct-n (Li et al. 2015 Zhu et al.) 2018 Celikyilmaz et al. 2020 Montahaei et al. 2019) Recent analyses in SFT emphasize preserving support via reverse- KL. with entropy, which can im- prove generalization and test-time scaling (Li al.] (024).. Balancing reward maximization with KL anchoring and entropy maintenance is inherently multi-objective; classical MORL offers policy- gradient methods and scalarization critiques (Parisi et al. 2014 Van Moffaert et al. 2013), while
3

---

Published as a conference paper at ICLR 2026
Pareto-stationan multi-gran gradient updates provide principled directions in ML and RL (Momma et al.
2022 Liu & Vicente [024]] Lin et al. 2022] Liu et al. [2025] Kang et al. (024).. Related contempora- neous observations
overtraining suggest that heavily pre-trained models can be more complex to
adapt (Springer et al. 2025). reinforcing the need for post-training procedures that preserve support rather than further narrow it.
POLICY CONTRACTION
Background and motivation. Preference-based fine-tuning with PPO has repeatedly been ob- served to narrow a model's output distribution: as optimization proceeds, probability mass con- centrates few high-reward modes, token-level entropy falls, and many plausible alternatives under the SFT/reference policy receive vanishing probability. Prior analyses discuss reward over- optimization, KL regularization effects, and decoding sensitivity, but a clean, decoding-agnosti diagnosis of support loss has been lacking. This section establishes that contraction appears across models and prompts under controlled conditions.
Setup and principles. We aim to quantify support loss independently of decoding heuristics and reward-scale artifacts. We therefore report multiple random seeds, sweep standard decoding settings in separate robustness check, and fix reward scaling where noted. We probe contraction using three complementary diagnostics:
(V1) Sampling-basee diversity. We sample it completions for each prompt and compute Self- BLEU, Distinct-n, and n-gram repetition rate.
(V2) Entropy and KL trajectories. We track length-normalized per-token entropy and forward KL to the reference over PPO iterations:
H Eg, (to)
bul log To(y xt)]
KL Teet)-
Contraction indicated by sustained entropy decline fixed or rising KL.
(V3) Support Retention Ratio (SRR). We estimate
[ull log Toly 1 r].
SRR(T)
Ex
Pr youtie (-1x)
the fraction of SFT completions whose length- normaliee log-probability under the PPO policy exceeds a fixed threshold т.
Distributional diagnostics on verification sets. Under PPO, entropy drops while forward KI. the reference rises-a contraction signature. Entropy scheduling recovers entropy and reduces repetition with only modest KI. change. Repetition is reported as % of repeated bigrams; entropy is in nats/token; KL is in nats.
Table 1:
Model
Dataset
Sctup
Repeat-2
Entropy
KL. to ref
Llama3-8B
ShareGPT-IK
SFT-only
21.8
3.88
PPO
24.5
3.42
0.45
+ Entropy
18.0
3.76
0.43
Qwen2-7B
AlpacaEval
SFT-only
21.1
3.90
+ PPO
22.7
3.50
0.46
+ Entropy
18.7
3.77
0.44
Qwen2-7B ShareGPT-IK
SFT-only
20.0
3.84
+ PPO
23.1
3.33
0.45
+ Entropy
17.1
3.70
0.43
We run (VI)-V22 on ShareGPT-IK and AlpacaEval: for external validity with preference data, (V3)
reported on ShareGPT-IK Alpa,aEval, and HH-RLHF. Table I shows that PPO reduces entropy while KL rises, for example 3.88 + 3.42 and 3.90 - 3.50 nats/token KL increases to zs 0.45-0.46 nats, consistent with contraction. Entropy scheduling reverses the entropy dip and lowers repetition with only modest KI. change.
4

---

Published as a conference paper at ICLR 2026
Table 2: Test of policy contraction. Fraction SFT completions retained above threshold SRR increases substantially with entropy scheduling.
Dataset
Setup
SRR@
ShareGPT-IK
PPO
0.37
AlpacaEval
+ PPO
0.39
H1I--LHIF
PPO
0,41
ShareGPT-IK
Entropy
0.58
AlpacaEval
+ Entropy
0,60
HH-RLHF
Entropy 0.62
Qualitative density shift. Log-pree prg-probability histograms over SFT-sampled completions show post- PPO mass concentrating on fewer sequences, with a heavier left tail (near-zero probabilities) on many previously feasible modes. Entropy scheduling broadens the histogram and increases SRR, indicating improved support preservation.
Across ShareGPT-IK and AlpacaEval, we observe consistent entropy declines and low SRR under PPO. Diversity-awar control ($4) restores entropy, reduces repetition, and increases SRR, motivat- ing the multi-objective design, which will elaborated next.
METHODOLOGY
We aim mitigate policy contraction in PPO-based RLHF, where the policy's 's support narrows over training, reducing entropy and diversity. We introduce two complementary components: (i) CaPPO, which treats reward, entropy, and KL. as peer objectives and selects a Pareto-improving update each step; and (ii) an entropy-scheduling controller that stabilizes exploration by steering token en- tropy toward a target. Throughout, mee denotes the SIT reference policy, mo the trainable policy, and R(x, v) preference-model score. Sequence log- ligeliilisoods are length-normalized unless otherwise stated.
4.1 RLHF SETUP AND PPO SURROGATE
Given prompts ε, we sample completions y from Te. score them with R, and diffuse sequence reward across tokens for advantage estimation: rt = R(x, w)/lul, with advantages A, via GAE. The clipped PPO surrogate on the reward component is
Te(ai st) reward (0) E. min (e, clip(p:(0), -,,1) pc(
C PPO
(1)
To (as st)
4.2 CONTRACTION-AWAAL PPO (CAPPO)
Standard practice augments equation T with fixed coefficients on entropy and KL. Such scalarization brittle:
overweighting reward accelerates contraction by eroding entropy and drifting from Rut while overweighting regularizers suppresses reward learning. CaPPO instead searches at every step for descent direction that simultaneously respects reward improvement and support preservation by solving small multi-objective problem.
We define three maximization objectives
(,(0) CPP (
Jc( ((T)), Ju(() -KL(mo( (+) ),a(- |x)),
with gradients = Jr, ge = Je, and Iu = V.a. A point is Pareto stationary if the origin lies in the convex hull of these gradients:
0 E cot (O)) v.e(0"), )
(2)
We approximate equation 2 by finding the minimum-norm convex combination of gradients:
min x
| + Aege + Augaull
و = Σ Aegr, A - 8 nog.
(3)

---

Published as a conference paper at ICLR 2026
Because objectives differ in scale, precondition with diagonal metric P-1/2 (e.g., Adam's sec- ond moment on or a Fisher diagonal from Tet), solve equation B in the preconditioned space, and map back. A simpler alternative is unit-length normalization of 9t. For three objectives, the problem reduced to a small quadratic program; we use two or three rogjected-gradien or Frank-Wolfe steps and check edge solutions. A constrained view makes the connection clearer:
max J,(0) s.t. (L(Toll S qu. H(TT) 2 Ce.
(4)
where the Lagrange multipliers correspond to adaptive mixing weights. A primal- dual update nat- urally enforces these constraints, while a guarded line search ensures reward progress without ex- cessive entropy loss or KI. growth. When gradients conflict, lower bounds on mixing weights are raised according to their cosine similarity, discouraging collapse.
Algorithm 1 CaPPO with Pareto mixing (RLHF)
1: Initialize 0, optimizer, À < 01,0, 8 + Bo, t H.
2: for each iteration do
3: Collect on- on-policy rollouts with Te: compute advantages and rewards from R: estimate token
entropy H4 and DKL (TTe
4:
mean (H)) h 3 &- clip(3 exp(na(H, f)), Aeins Pmax) Jr (olrro((); 9e + VV, H (Ta): Da < VoDKI.
A - (1-p) p1
5:
6:
9i p-1/2g, for i E kl); Gi 0.
7:
X arg mingee GA
8:
9 + EiA.
9:
Find largest 77 such that (KI( (Tet) I EKL and H (To-ná) 2 Hmin
10: 0 + 0 + n7
end for
4.3 ENTROPY SCHEDULING
Entropy scheduling complements CaPPO by stabilizing exploration through simple feedback con- troller the entropy weight. We track the length-normalized sequence entropy
H, = Ex, (-00((11)))) log ( æ))
H, = (1 H, + H,
(5)
where He is an Exponential Moving Average (EMA) with smoothing a (0. We adapt the entropy coefficient / toward a time-varying target (tugert) via clipped proportional update:
Bi+1 clip (mins Be + (t)ga(t) H. 8maxx.
(6)
where 17 is stepsize and (Bmin, 3max) bound the entropy weight. In practice, Harger can be cho- sen as a fixed constant, scheduled decay (to shift from exploration to exploitation gradually), or adaptive value defined support. We compute H, on the same minibatches used for PPO updates, averaging over prompts and sampled completions with lengths |y| excluding special tokens. To dampen noise, H, is main- tained bias-coreected EMA, and clipping ensures that the entropy term stays within stable range (roughly 5-20% of the total surrogate magnitude at initialization). Optionally, small integral term can be added to the update further reduce oscillations, though disable it by default.
EMA of the reference policy entropy plus small offset preserve
4.4 THEORETICAL PERSPECTIVE
CaPPO can be analyzed as stochastic approximation scheme for multi-objective policy optimiza- tion. Its update corresponds to the minimum-norm element of the preconditioned gradients' convex hull, which ensures convergence to Pareto-stationary points under standard Lipschitz continuity, In particular, if g the CaPPO update, then for any feasible descent direction d, we have
(g. d) >
Vd conef Gr. Des ga))

---

Published as a conference paper at ICLR 2026
which formalizes that descent direction can improve one objective without violating another.
A complementary trust-region perspective interprets CaPPO as a constrained optimizer with en- tropy floors and KL caps. The associated Lagrangian dynamics adaptively tune multipliers Au Aes yiclding an update of the form
ac+1 02 + no VJ((0)) ARa VKL(0,) + (e,0(00))++,,VHHH VH.
(8)
with projected dual steps ensuring feasibility. Finally, entropy scheduling can be understood as proportional-ntteral controller that regulates the error Hurrett Hr, ensuring bounded entropy tracking while remaining compatible with the fast multi-gradient updates.
5.0
1.2
0.62
Forward to nnforance
1.0
4.5
-0.62
0.8
4.0
0.60
3.5
CaF90 EsEopy
0.59
3.0
Intrayy
caro 12
2.5
250
1250 Training stops
-5.5
Longth-normalized log log-llkellhood
(a) Entropy training steps.
(h) Log-likelihood distribution shift.
Figure 1: Policy contraction and mitigation. PPO training reduces entropy and crodes support for reference completions, while CaPPO stabilizes entropy and preserves larger fraction of the refer- ence distribution.
These perspectives establish CaPPO principled trust-region variant of PPO that preserves sup- port while continuing to maximize reward. Empirically, this theoretical picture is borne out in Fig- ure D la shows that PPO suffers entropy collapse at similar KL to the reference, whereas CaPPO stabilizes entropy; ibilutttrates the consequence at the sample level, where CaPPO shifts probability mass rightward and retains reference completions that PPO discards.
5 RESULTS
We evaluate PPO and our proposed CaPPO under a standard RLHF setup on three widely used datasets: (i) Anthropic HH-RLHF (helpfulnes/harmlessnsss preferences) (Bai et al. 2022a). (ii) (Stiennon et al.] (020), and (iii) UltraFeedback for general helpfulness preferences (Cui et al. (023). We compare four base models: Qwen2-7B (Team)
OpenAl caamaarieefrom-Feeddbbe
2024), Qwen2.5-14p (Yang et al. 2025), Mistral-7B-Instruu (Jiang et al.) 2023), and Ilama-3-
8B-Instruct (Grattatiori al. 2024).
Overall performance. reports preference win rates relative to SFT across the three RLHF datasets and four model backbones. Several consistent observations emerge. PPO provides clear improvement over SFT in all cases, raising the win rate by 7-15 points depending and datasct. For example, Qwen2.5-14B on HH-RLHF climbs from 50.0 to 65. 1+0.7, while Llama- 3-8B UltraFeedback increases from 50.0 to 0.84 These gains confirm the effectiveness of on-policy RL in extracting the reward-model signal. However, the pattern also shows that PPO im- provements plateau relatively narrow margin, suggesting that further win-rate gains difficult without additional intervention. CaPPO consistently lifts performance beyond this plateau, deliver- ing 2-4 points higher win rate than PPO across all backbones and datasets. Importantly, these gains are not confined to a single model or dataset but appear consistently across Qwen, Llama, and Mis- tral families, indicating that the contraction-awe updates generalize broadly rather than exploiting a particular training recipe.
model size
Table 4 then situates CaPPO among alternative optimization strategies. Off-policy methods such as DPO, IPO, ORPO, KTO, and RRHF preserve high SRR (up to 0.92) but underperform in win rate (62 65%).. On-policy baselines like PPO, VinePPO, and GRPO push win rate higher (67-71%) but at the cost of contraction, reducing SRR to the 0.46-0.62 range. CaPPO stands out by combining the
7

---

Published as a conference paper at ICLR 2026
strengths of both: it reaches the highest co-highest win rate, lowers redundancy, increases lexical diversity, and recovers support retention.
Table also provides macro-averaged evidence of how these methods shape distributional sup- port. PPO-trained models show the characteristic contraction signature- iignature-elvvatee Self-BLEU (0.47-0.49), suppressed Distinct-2 (0.16-0.18) and SRR collapse below 0.45. CaPPO reverses these effects across all four backbones, lowering Self-BLEU by 0.13-0.15, raising Distinct-2 by 0.08-0.09, and improving SRR by 0.29-0.34. These gains establish that CaPPO not only achieves higher alignment reward but also broadens the effective support of the trained model, counteracting the narrowing induced by PPO alone.
Table 3: Preference win rate compared SFT on three RLHF datasets across four base models. Means + std over three seeds. SFT is 50.0 by definition.
Dataset
Model
SFT
+ PPO
PPO Entropy
CaPPO
Qwen2-7B
50.0
62.8 + 0.7
64.3 + 0.7
66.4 t 0.6
Qwen2.5-14P
50.0
65.1 + 0.7
66.8 0.6
69.0 0.5
HH-RLHF
Mistral -B--nssuutt
50.0
60.4 + 0.8
61.9 0.8
64.1 0.7
Llama-3-8h Instruct
50.0
63.5 + 0.7
65.0 + 0.7
67.2 + 0.6
Qwen2-7B
50.0
57.6 +: 0.9
58.9 +: 0.8
61.3 + 0.7
Qwen2.5-144 1-14B
50.0
60.2 0.9
61.5 0.8
63.9 0.7
Summarize-fron Fredb-Fee
Mistral-7B -B-nsltuutt
50.0
55.8 + 1.0
57.1 + 0.9
59.6 + 0.8
Llama-3-80 1--8B-Inntreet
50.0
58.1 + 0.9
59.5 + 0.8
62.0 + 0.7
Qwen2-7B
50.0
61.4 + 0.8
63.2 + 0.7
66.0 + 0.6
Qwen2.5-
50.0
64.6
0.6
UltraFeedback
Mistral-7BB 7B--Instruct
50.0
59.7 + 0.9
61.3 + 0.9
64.0 + 0.8
Llama-3-Sh Instruct
50.0
62.4 + 0.8
64.1 + 0.8
66.9 0.7
Table Comparison additional baselines (Qwen2-7B: macro-average across HH-RLHF, Summarize-frome fra-z-e-rro--Feebbaee UltraFeedback). Off-policy methods use the same preference pairs; on-
policy methods follow public recipes. Means + std over three seeds. Best per column in bold. Meth-
ods: DPO (Ralailov et al. 2023). IPO (Garg et al. 2025), ORPO (Hong et al., (024). KTO (Etha-
yarajh al. 2024). RRHF Yuaan et al. 2023). VinePPO (Kazemnejad et al. 2024). GRPO Shao
et al.
2024)
Method
Win rate
Self-BLEU
Distinct-2
SRR
SFT
50.0
0.39
0.22
0.90
DPO (off-policy
62.8 + 0.8
0.40
0.23
0.88
IPO (off-policy)
63.8 + 0.8
0.40
0.23
0.89
ORPO (off- policy)
63.3 + 0.8
0.41
0.22
0.86
KTO offf-poliy)
61.9 0.9
0.41
0.22
0.85
RRHF (off-policy rank)
64.7
0.38
0.24
0.92
PPO
67.4 0.7
0.48
0.17
0.55
VinePPO
08.6+00.
0.45
0.19
0.62
GRPO
71.0 0.6
0.37
0.24
0.70
PPO + Entropy
68.8
0.42
0.20
0.66
CaPPO
7..2
0.33
0.27
0.82
Ablations. Fixed versus adaptive entropy coefficients HH-RLHF (Table show that adaptive scheduling substantially improves SRR and Distinct-2 at comparable win rates. Summarize- from-Feedbade (Table 7 scalarization sweeps are brittle, whereas CaPPO's Pareto multi- multi-gradient updates achieve uniformly better SRR-win rate trade-offs. Under reward scaling and altered boot- strapping horizons on UltraFeddback (Table 8, CaPPO exhibits lower variance across seeds, indi- cating robustness to critic noise.
8

---

Published as a conference paper at ICLR 2026
Table 5: Diversity and support metrics (macro-averaged across the three RLHF datasets). Self-
BLEU indicates redundancy; Distinct-2 and SRR capture diversity and support preservation.
Dataset
Model
Self-BLEU
Distinct-2
SRR
PPO
CaPPO
PPO
CaPPO
PPO
CaPPO
Qwen2-7B
0.48
0.34
0.17
0.26
0.43
0.74
Qwen2.5-141 1-BB2
0.47
0.33
0.18
0.45
0.76
Macro over RLHF
0.49
0.35
0.16
0.25
0,41
0.72
Llama-3-8 3-8B-M88 11-88-Innstin Instruct
0,48
0.34
0.17
0.26
0,44
0.75
Table 6: Ablation on entropy scheduling. update with the same controller.
CaPPO combines the minimum-norm multi-objective
Variant
Win rate
Self-BLEU
Distinct-2
SRR
PPO (fixed .9)
63.4
0.49
0.17
0.43
PPO (adaptive (3)
65.1
0.42
0.21
0.59
CaPPO (ours)
67.8
0.35
0.27
0.74
Table 7: Scalarization sweep vs. CaPPO.
Method
Win rate
A Win
Distinct-2
AD2
SRR
ASRR
PPO (A==0.0)
57.2
0.0
0.16
0.00
0.41
0.00
PPO (Aæ=0.1)
58.3
0.19
+0.03
0.50
+0,09
PPO (A.=0.33
58.0
+0.8
0,22
+0.06
0.57
+0.16
CaPPO (Pareto)
61.9
+4.7
0.24
+0.08
0.73
+0.32
Table 8: Robustness in terms of seed variance under reward scaling.
Method
Win rate std
A Win std
SRR std
ASRR std
PPO
0.0
0.030
0.000
PPO + Entropy
1.1
-0.3
0.024
-0,006
CaPPO (ours)
0.8
-0.6
0.017
-0.013
Throughput and memory overhead. We report end-to-end training throughput thousands of tokens per second aggregated over 8 A00 80GB, and peak GPU memory, at matched KL budgets and identical batch/sequence shapes. Throughput is averaged over the last 60% of updates after 100-step warmup; we report mean + and std over three seeds. We also show the ratios to PPO (higher better for throughput; lower better for memory). CaPPO adds two additional objective gradients and tiny three-variable QP; GRPO uses group sampling with K >11, which scales compute and buffers with K.
Across I, {256, 512, 1024, 2048), CaPPO attains 92-94% of PPO throughput (mean 93.0%) with approximately 3.1% more peak memory on average; the throughput gap narrows from 92.2% at L-256 to 94.0% at L=2048. PPO+Entropy reaches 96-98% of PPO throughput with approxi- mately 1.1% memory overhead. GRPO runs at 84-88% PPO throughput with 6-7% higher peak memory. These trends match the expected costs: CaPPO adds two extra objective gradients and small three-variabee QP per update, while GRPO's group sampling increases compute and buffers.
Takeaways. Across datasets and model backbones. PPO improves the win rate but consistently induces policy contraction, reflected in lower entropy, reduced SRR, and higher Self-BLEU, CaPPO counteracts this effect, delivering to points higher win rates while also improving diversity, with gains of 0.06 to 0.10 in Distinct-2 and 0.20 to 0.30 in SRR. These improvements arise from the combination of adaptive entropy scheduling, which balances exploration and exploitation, and Pareto multi-objective updates, which avoid the brittleness of scalarization.

---

Published as a conference paper at ICLR 2026
Table Throughput and peak memory matched KL budgets. Throughput is reported kTok/s.
The rightmost subcolumn shows the ratio to PPO at the same sequence length.
Throughput
Peak memory
Method
Seq len
KTok/s
% of PPO
GB
% PPO
PPO
256
34.8 + 0.5
100%
46.1 + 0.4
100%
PPO + Entropy
256
33.7 + 0.5
96.8%
46.6 0.4
101.1%
GRPO
256
29.2 0.6
83.9%
49.3 0.6
107.0%
CaPPO (ours)
256
32.1 + 0.4
92.2%
47.6 0.5
103.3%
PPO
512
31.2 + 0.5
100%
55.2 0.5
100%
PPO Entropy
512
30.0 0.4
96.2%
0.5
101.1%
GRPO
512
27.4 27.4+00. 0.5
87.8%
0.6
106.5%
CaPPO (ours)
512
28.8 18.8:00 0.4
92.3%
0.5
102.5%
PPO
1024
23.9 0.4
100%
0.6
100%
PPO Entropy
1024
23.2 0.4
97.1%
66.6
101.1%
GRPO
1024
2.4
87.9%
7.72
106.5%
CaPPO (ours)
1024
0.3
93.7%
0.6
103.6%
PPO
2048
0..9
100%
78.6
100%
PPO + Entropy
2048
1.3
98.0%
0.8
101.0%
GRPO
2048
1.3
87.9%
83.3 1
106.0%
CaPPO (ours)
2048
14.0 0.3
94.0%
80.9 0.8
102.9%
CONCLUSION
We investigated policy contraction in PPO-based RLHE, effect where the policy's support pro- gressively narrows during on-policy fine-tuning. We provided both diagnostic tools and training- time remedies. We combined conventional degeneration metrics with entropy and KL trajectories on the diagnostic side. and introduced SRR as direct, decoding-agnostic measure of support over- lap with the SFT reference. These probes consistently revealed entropy collapse, rising repetition, and left-shifted log- log-probability distributions under PPO, even when reward improved.
To address this, we proposed two complementary interventions. First, entropy scheduling lightweight controller that dynamically adjusts the entropy coefficient to steer token entropy to- ward target, thereby preventing runaway collapse. Second, CaPPO, which reformulates PPO as multi-objective problem, treats reward, entropy, and KL-to-reference peer objectives and up- dates along Pareto-ing pareto-improving directions. These methods stabilize exploration, preserve support, and improve diversity while matching or exceeding PPO's win rate across benchmarks. Empirically, CaPPO mitigates contraction on different datasets and base models. achieves higher performance with robustness reward scaling, critic variance, and decoding settings. These results demonstrate that support preservation and accuracy are not in fundamental conflict when diversity is elevated to first-class training objective.
Limitations and future work. Our approach introduces modest additional computation from the entropy controller and multi-objective mixing step, and its success depends on the fidelity of the reward model. SRR requires threshold selection, which we mitigate with length normalization and percentile- rules, but further refinements are warranted. Future work includes theoretical anal- ysis of convergence and stability of Pareto updates in trust-region RL and combining training time CaPPO with inference-time diversity or reasoning controllers.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022a.
10

---

Published as a conference paper at ICLR 2026
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08073. 2022b.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey, arXiv preprint arXiv:2006. 147999 2020.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeddbckk: Boosting language models with scaled ai feedback. arXiv preprint 23.1v:2310.01377 10.01377, 2023.
Nirjhar Das. Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. arXiv preprint 100:2402.11500, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.013006 2024.
Alexandre Galashov, Siddhant M Jayakumar, Leonard Hasenelever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojeiech M Czamecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized rl. arXiv preprint urXiv:1905.011 2019.
Shivank Garg, Ayush Singh, Shweta Singh, and Paras Chopra. Ipo: Your language model secretly preference classifier. arXiv preprint arXiv:2502.1614 16182, 2025.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 14375, 2022.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21777 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning, arXiv preprint UrXiv:2501.12945 2025.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration. arXiv preprint
09751,, 2019.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint 27X1v:2403.07661, 2024.
Bokai Hu, Sai Ashish Somayajula, Xin Pan, and Pengtao Xie. Improving the language under- standing capabilities of large language models using reinforcement learning. arXiv preprint arXiv:2410.1 1020, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https: //axxiv.org/aaaet aa////bs/2331 06825
Shida Kang, Kaiwen Li, and Rui Wang. survey on pareto front learning for multi-objective optimization. Journal of Membrane Computing, pp. 1-7, 2024.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Refining credit assignment in training of 1ls.. arXiv preprint arXiv:2410.01677 2024.

---

Published as a conference paper at ICLR 2026
Harrison Lee. Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:339.00026; 2023.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promotiin objective function for neural conversation models. arXiv preprini
2015.
Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. Preserving diversity in supervised fine-tuning of large language models. arXiv preprint arXiv:2408. 16673, 2024.
Xi Lin, Zhiyuan Yang. Xiaoyuan Zhang, and Qingfu Zhang. Pareto set learning for expensive multi- objective optimization. Advances in neural information processing systems, 35:19231-19247 2022.
Erlong Liu, Yu-Chang Wu, Xiaobin Huang, Chengrui Gao, Ren-Jian Wang, Ke Xue, and Chao Qian. Pareto set learning for multi-objective reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39. pp. 18789-18797, 2025.
Suyun Liu and Luis Nunes Vicente. The stochastic multi-gradient algorithm for multi-objective optimization and its application to supervised machine learning. Annals of Operations Research, 339(3):1119) 19-11489 2024.
Michinari Momma, Chaosheng Dong, and Jia Liu. A multi-ohjectivemultitass learning framework induced by pareto stationarity. In International Conference on Machine Learning, pp. 15895- 15907. PMLR, 2022.
Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. Jointly measuring diver- sity and quality text generation models. arXiv preprint (KKiv:/9040397// 2019.
Youssef Mroueh. Reinforcement learning with verifiable rewards: Grpo's effective loss, dynamics, and success amplification. arXiv preprint arXiv:2503.0665 066699 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744 2022.
Simone Parisi, Matteo Pirotta, Nicola Smacchia, Luca Bascetta, and Marcello Restelli. Policy gra- dient approaches for multi-objective sequential decision making. 2014 International Joint Conference on Neural Networks (IJCNN, pp. 2323-2330. IEEE, 2014.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53741 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06777, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03330. 2024.
Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. Overtrained language models are harder to fine-tune. arXiv preprint XwX.::03:3339000060 2025.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3022 2020.
Qwen Team, Qwen2 technical report. arXiv preprint arXiv:2407.10671,2 2024.
12

---

Published as a conference paper at ICLR 2026
Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforce- ment learning: Novel design techniques. In 2013 IEEE symposium on adaptive dynamic pro- gramming and reinforcement learning (ADPRL), pp. 191-199. IEEE, 2013.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang. Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL absg/abs//20 https://arxx http://xxx....///////222222222.2
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses align language models with human feedback without tears. arXiv preprini arXiv:2304.05332, 2023.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504. 22.2504..3377.. 13837, 2025.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 1097-1100, 2018.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1
EXTENDED METHODOLOGY AND REPRODUCIBILITY
This appendix provides implementation details, hyperparameters, evaluation protocols, and addi- tional analyses to facilitate exact reproduction and a deeper understanding of our results.
A.1.1 EXPERIMENTAL SETUP
Models. The policy Ty is initialized from an SFT checkpoint and paired with frozen reference Tet (the same SFT weights unless otherwise noted). We train with PPO using the clipped surrogate (Eq.[i the main text). Unless specified, we share the transformer body between policy and value head; the value head is linear projection over the final hidden state.
Tokenization and sampling. We use the base tokenizer accompanying the SFT model. For roll- outs, decoding uses temperature T E [0.7, 1.0) and top-p [0.90, 0.95] unless otherwise noted. We fix (T, p) and the random seed for comparability for evaluation-time generations used by diversity metrics.
Rewarding. We use scalar reward head or an external reward model for preference-style data. For single-output tasks, r(z, y) is obtained from the reward model; for multi-turn prompts, aggregate token-level rewards by length-normalized averaging.
Advantages and normalization. We compute GAE with (7, A) E [0.95, 0.999) [0.90, 0.98] and normalize advantages to zero mean and unit variance within each minibatch. Returns may be whitened at the episode level.
Optimization. We use AdamW with decoupled weight decay, linear warmup, and cosine decay. Gradient clipping is applied according to the global norm prior to the PPO update. Gradient check- pointing is enabled for long contexts.
A.1.2 DATASETS AND PREPROCESSING
Primary benchmark. The main experiments HH-RLHF for RL fine-tuning. Prompts filtered for length (e.g., |z| < Lmax) and normalized (whitespace and Unicode cleanup).
Held-out prompts for verification. To measure the squeezing effect out of distribution, we main- tain a disjoint held-out set Xheld from the same source. All verification metrics in App. A.1.5 are computed Kheis-
Preprocessing. We prepend system instruction template during training and evaluation when appropriate. We construct (yt ,y from the rollout pool via top-k reward ranking per prompt for reward models requiring pairwise inputs.
A.1.3 HYPERPARAMETERS
SCHEDULES
Global PPO and optimization hyperparameters. Table 10 lists core training hyperparameters. Ranges denote values explored during tuning; selected values per run are recorded in the code re- lease.
Entropy scheduling.
The controller follows Eqs. 5
6). Table I1
summarizes the schedule.
KL tracking. report forward KL( ne mer) on rollout tokens and maintain an EMA with coef- ficient 0.1 for trend plots (not required for training).
A.1.4 CAPPO: SOLVER AND IMPLEMENTATION DETAILS
Objectives and gradients. We optimize three objectives with gradients 91 PP = VoH(Fo) ga Vo( -KL( me))). Each gi may be normalized by [|9:lla lalla before mixing.
14

---

Published as a conference paper at ICLR 2026
Table 10: PPO and optimization hyperparameters.
Hyperparameter
Range/ Value
Notes
PPO clip
[0.1, 0.3]
Trust-region strength
Num PPO epochs
[1. 4]
Minibatch passes per batch
Minibatch size
[64, 512]
Tokens or sequences
Batch size (tokens)
64k, 512k
Accumulated across devices
Discount
[0.55. 0.999
Return discount
GAE A
0.900. 0.98
Bias/variance trade-off
LR (policy/value)
[le-6, 2e-
Cosine decay after warmup
Weight decay
[0.0, 0.1
Decoupled (Adam W)
Grad norm clip
[0.5, 1.0
Global norm Model-dependent
Context length L
[1k, 8k]
Table 11: Entropy scheduling hyperparameters.
Hyperparameter
Range / Value
Notes
EMA coefficient a
[0.05, 0.25]
Smoothing Hy Step on entropy gap Clamping for stability
Adapt rate r
[1e-3, 5e-2]
Buin Bunxx
0, [0.2, 1.0
Target type
Fixed / Decay/ Adaptive median of गe
As in main text On current batch
Fixed Ho
Decay Y
[0.90, 0.999]
Over 7 updates Exploration slack
Adaptive margin 5
[0.0, 0.5]
Combination rule (minimum-norm point).
Solve
3 X
min X
s.t. A 0, Σ
via a short projected gradient on the simplex (5-10 iterations) or a Frank-Wolfe step on conv 82, }- Using the Gram matrix Gij 9. 9j. the objective equals X GX. The three- objective QP is negligible cost.
Trust region and clipping. We retain PPO clipping the reward objective while mixing gradi- ents. This preserves robustness even when 92 or g3 momentarily dominate.
Stability tips. We use small ridge el G near the singularity, freeze X for K steps (e.g., K-4) to reduce oscillations, and damp the mixed direction with momentum E [0.5, 0.9).
A.1.5 SQUBEZING-EFFECE INDICATORS AND VERIFICATION
Per-token entropy. Compute H, = E2,y~ool log ( æ)] with length normalization. Report EMA-smoothed trends and raw values.
Repeat-n. For W1:1y/
# {unique n-grams in y} # (all n-grams in y)
Repeat-n(y) =1
Repeat-n = E [Rpeeat-nny))
We use n E (2,3,4) 4).
Self-BLEU and Distinct-n.
Generate K samples per prompt:
# junique n-grams across Y
Self-BLEU
| Σ Σ BLEU (x))(+)), Distinct-n
#total n-grams across Y
(EY(
KL drift. Track (T(To|| on the held-out per iteration.
15

---

Published as a conference paper at ICLR 2026
SRR. Let (x) be K completions from Tiet- Define
SRR,
1
Σ K
Σ
[lel log TO (2) (2)))
|
y
with 7 chosen by a reference percentile or peef kovef-
probbabbbillty histograms. Histogram Tol log Toly Λ)) for y ~ tee over fixed prompt set; left shift and peaking at low likelihood signal shrinking support.
Support overlap (optional).
Approximate via merged top-k pool (x)):
SO(z)
Σ
minfme(y r),, Teery x)}, SO Ez[SO(z)]-
(E)
Algorithm 2 Policy Contraction Test and SRR
1: Input: Held-out prompts X, Tht snapshots {Tates J. samples-per-prompp K, threshold rule r(-)
2: for X do
3: Sample (IF from Tre collect Hiee; Oref
4: end for
5: Set T;
6: for each snapshot t do
7: Evaluate
log- likelihoods and compute SRR(2)
8: end for
9: Output: (SRR()) and histograms
Training dynamics. We track per-token entropy and SRR over 50k RLHF updates (Figure 2 Under PPO, entropy declines from 4.2 to 2.2 nats and SRR from 0.88 to 0.65, indicating policy contraction. CaPPO stabilizes entropy near 3.6 nats and preserves SRR around 0.92.
Entropy vs. Updates
SRR vs. Updates
0.95
CaPRO
CsPPD
PPg
0.90
PPG
Per-tolnn entrayy (tans)
3.75
3.50
3.25
support netenttion
0.80
0.75
0.70
2.50
2.25
10000
30000
50000
20000
40000
PPO updates
PPO updates
Figure 2: Training dynamics under RLHE. Left: per-token entropy over 50k updates. PPO stcadily reduces entropy, indicating narrowing exploration, while CaPPO maintains higher, stable entropy level. Right: SRR over the same training horizon. PPO progressively loses support from the ref- erence distribution, with SRR dropping below 0.65, whereas CaPPO consistently preserves more than 70% of the SFT support. Together, these trajectories highlight how CaPPO counteracts policy contraction by balancing reward learning with entropy and KL objectives.
Decoding robustness. To rule out decoding artifacts, evaluate at top-p E (0.8, 0.9) and tem- perature E {0.7, 1.0.. CaPPO retains higher Distinct-2 and SRR across settings while maintaining win (Figure
16

---

Published as a conference paper at ICLR 2026
Duenly Dldte ueitigg
Figure 3: Decoding sweeps under RLHF. Results averaged over Qwen2-7B, Qwen2.5-14B Mistral- 7B, and Llama-3-8B on HH-RLHF, cemmmarrze-from-eeebbeae and UltraFeedback. Each plot varies top-p {0.8, 0.9) and temperature {0.7, 1.0). Left: win rate remains high across set- tings, with CaPPO matching or exceeding PPO. Middle: Distinct-2 is consistently higher under CaPPO, showing improved lexical diversity. Right: SRR is substantially higher with CaPPO, in- dicating stronger support preservation across decoding parameters. These results demonstrate that CaPPO's improvements are robust decoding choices rather than artifacts of a particular sampling setup.
A.1.6 ABLATIONS
Entropy variants. Compare: fixed 3, monotone decay 3r Buy and controller based adap- tive Bt. Report final performance and He.
Objective coupling. Compare scalarized PPO (fixed weights reward/entropy/KKL) against CaPPO (Pareto mixing). Probe gradient normalization and freezing À across steps.
Verification sensitivity. Grid SRR thresholds т, number of SFT samples K, and for Repeat-n/Diisinnt-r; verify conclusions remain consistent.
A.1.7 COMPUTE, TIME, AND MEMORY
Environment. All runs were executed on single node with 8xA100 80GB GPUs connected by NVLink. Mixed precision and activation checkpointing were enabled uniformly across methods. Throughput reported as thousands of tokens per second aggregated over 8 GPUs. Tokens are counted after sequence packing and include both prompt and completion tokens actually processed by the training loop. We count tokens touched by the actor as well as those evaluated by auxiliary components (critic and reward), so the metric reflects end-to-end RLHF work rather than generated tokens only.
Timing excludes process start-up and dataloader warm-up. The reporting win- dow begins only after the optimization loop reaches steady state; measurements are averaged over multiple seeds. Wall-clock time includes forward and backward passes, optimizer updates, gradient synchronization, and reward/critic evaluation. Peak memory is the maximum device-resident allo- cation observed on any GPU during the reporting window, using the framework's ''s built-in memory counters (allocated and reserved). Results are reported per configuration the peak across GPUs.
Instrumentation.
Fairness controls. To isolate algorithmic overhead, we keep the reference policy on device for all methods, use identical batch and sequence shapes, and apply the same precision, checkpointing policy, sequence packing, and dataloader settings. Offloading, sharding, compilation flags, and kernel choices are held fixed unless explicitly stated otherwise.
A. EXTENDED SCALARIZATION SWEEPS
We report compact grids for scalarized PPO under KL/entropy matching. Runs are grouped within narrow KL and token-entropy bands relative to the CaPPO reference for each setup. Metrics are macro-averaged and use the same frozen reward model in the main text.
Across KLeentropy-matche bands, increasing 3 improves SRR and lexical diversity with small win-rate changes, while CaPPO remains ahead on SRR and Distinct-2 comparable reward.
17

---

Published as a conference paper at ICLR 2026
Table 12: Scalarized PPO a sweep (Summatize-from-Fodbbbee KL/H matched).
Method (o)
Win rate
Distinct-2
SRR
Scalarized PPO (0.0)
57.2
0.16
0.41
Scalarized PPO (0.1)
58.3
0.19
0.50
Scalarized PPO (0.3)
58.0
0.22
0.57
CaPPO (ref)
61.9
0.24
0.73
Table 13: Entropy coefficient scheduling (HH-RLHP; KL/H matched).
Method
rate
Distinct-2
SRR
PPO (fixed ()
63.4
0.17
0.43
PPO (adaptive 3)
65.1
0.21
0.59
CaPPO (ref)
67.8
0.27
0.74
A.2 USAGE OF LLMS
We used LLM- based assistants to support writing. In this paper, we employed it only for grammar and writing style polishing. All content was manually reviewed and verified by the authors. In line with the conference policies, we explicitly disclose this usage and acknowledge that the authors bear full responsibility for the paper's accuracy and integrity.
18


Reviews:
[{"review_id": "Ei1IAmUSts", "reviewer": "Reviewer_YLyS", "summary": "PPO-based RLHF often narrows the model’s output distribution (“policy contraction”): entropy drops, repetition rises, and many SFT-feasible completions get near-zero probability.\nTo mitigate this, the authors propose Contraction-Aware PPO (CaPPO), which:\n1. Treats reward, entropy, and KL-to-reference as peer objectives in a multi-objective optimization framework.\n2. Uses a minimum-norm multi-gradient update to ensure Pareto-improving steps without brittle scalarization.\nIn addition, the authors introduce Support Retention Ratio (SRR) to quantify how much of the SFT support survives RL training, complementing entropy and KL.\n\nFor experiments, the authors compare off-policy methods (DPO/IPO/ORPO/KTO/RRHF) and on-policy baselines (PPO/VinePPO/GRPO) on HH-RLHF, Summarize-from-Feedback, and UltraFeedback with Qwen2-7B/2.5-14B, Mistral-7B-Instruct, and Llama-3-8B-Instruct.\nCaPPO lifts win rate by ~2–4 points over PPO and raises SRR by ~0.20–0.30, while improving Distinct-2 and lowering Self-BLEU.", "questions": "- While the paper reports win rate as the primary performance metric, it is unclear how a “win” is determined in the experiments. Which reward model is used to decide the winner? If GPT-4 is used, which prompt is employed?\n- How to evaluate $\\lambda$ values in line 5 of Algorithm 1?\n- Intuitively, it isn’t obvious that CaPPO should outperform other on-policy methods while also improving diversity (in Table 4), since one might expect a trade-off between performance and diversity. Could you provide an intuitive explanation for this result?\n- It is unclear how the entropy-scheduling controller is applied in CaPPO, as described in lines 79–80, because Algorithm 1 does not include a $\\beta$ term.\n- As I understand it, CaPPO employs Equation (3) as the main training objective. Then, could you clarify why you chose the multi-objective update in (3) over a simpler linear scalarization (i.e., PPO + $\\alpha$·KL + $\\beta$·Entropy) with scheduled coefficients $\\alpha$ and $\\beta$ as in §4.3?\n  1. Does (3) reduce sensitivity to $\\alpha$, $\\beta$ tuning and reward scale or provide more reliable Pareto behavior?\n  2. Without explicit $\\epsilon$-bounds as in (4), how do you ensure KL and entropy remain within desired ranges (e.g., guards, line search, or dual-style updates)?\n  3. What are the wall-clock and compute trade-offs compared to scheduled scalarization?\n  4. Could you include ablations that sweep  $\\alpha$, $\\beta$ schedules to match similar KL/H targets and report win-rate, SRR, and stability differences?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- CaPPO outperforms other on-policy algorithms in both diversity and performance, as measured by win rate.\n- SRR is a reasonable metric for measuring diversity.", "weaknesses": "- How to evaluate win-rate is not clear.\n- How to implement CaPPO in practice & its wall-clock time is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uMTS7wOiCu", "reviewer": "Reviewer_e29o", "summary": "This paper tackles the problem of diversity collapse due to RL training algorithms in LLMs. Authors call this phenomenon *policy contraction*, implying the distribution of responses shrinks after RL training towards fewer high rewarding sequences. The authors systematically analyze the effects of different RL training methods by measuring policy contraction using measures like Self-BLEU, Distinct-2 (distinct bigrams) and Support Retention Ratio (SRR). SRR is a newly introduced metric which measures the fraction of SFT model competions whose length-normalized log probabilty under the RL tuned policy is greater than some fixed threshold. Using all three measures, authors find that preference-based off-policy methods retain SRR levels of SFT, however take a hit in Self-BLEU and Distinct-2 metrics. On the other hand on-policy policy gradient methods like PPO, Vine-PPO, PPO + entropy, etc. deteriorate in all three diversity measuring metrics.\n\nTo mitigate this policy contraction, they propose Contraction-Aware PPO (CaPPO) which implements reward maximization, entorpy and KL regularization as multi-objectives instead of additive. At the core, their method maintains 3 gradient vectors, one for each objective, and calculates a minimum-norm convex combination of gradients. They first precondition the gradients (to account for different scales) via dialgonal metric (second moment of Adam optimizer) and then solve the quadratic problem in this preconditioned space. Then the multiplers for this optimization method are found via projected-gradient or Frank-Wolfe steps. \n\nFor more robust comparison with PPO, they also introduced an adaptive entropy scheduling which tries to maintain the exponential moving average entropy of the policy close to the target entropy.\n\nIn the experiments they compare CaPPO against a variety of other basline preference based methods and PPO counterparts, including PPO enhanced w/ their adaptive entropy sechduler. Overall, they found their method consistently matches or execeeds the performance of the baseline methods while retaining relatively higher diversity and SRR compared to PPO counterparts.", "questions": "- Interestingly, in Table 4, GRPO shows much less degradation in diversity metrics compared to PPO. Do the authors have any intuition for this obersvation? Does the group sampling in-turn enhances the sampling diversity of the RL training? \n  - I am also curious, how would GRPO mixed with entropy or entropy scheduling behave in comparison to CaPPO.\n- Typo in the intro: KTO refers to \"Kahneman-Tversky Optimization\"", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Very clear motivation and empirical grounding to justify the problem\n- Interesting and theoretically sound multi-objective solution to the problem of policy contraction in RL training\n- Detailed experimental analysis, baseline comparison and ablation studies to demonstrate the efficacy of their method over the other RL baselines.", "weaknesses": "- I would have preferred to see a detailed comparison of the throughput hit when using a more complex optimization method such as CaPPO compared to PPO. How much more memory and time overhead does it incur? Will the additional overhead become a blocker for large LMs in long sequences tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kWpaeQm9dD", "reviewer": "Reviewer_KwhW", "summary": "The paper presents Contraction-Aware PPO (CaPPO), a practical improvement to PPO for RLHF. The authors point out that standard PPO tends to collapse a model’s output diversity as training progresses—they call it \"policy contraction\". CaPPO solves this by balancing reward, entropy, and KL-regularization as equal objectives, using a simple multi-gradient update instead of manual weighting. It also adds an adaptive entropy controller that keeps exploration at a healthy level. The idea is easy to follow, the motivation makes sense, and the experiments show clear benefits in both stability and diversity without hurting alignment quality.", "questions": "Please read the weaknesses section and answer the questions. Thank you!", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper does a great job of turning the vague observation that PPO reduces diversity into a formal, testable concept. The Support Retention Ratio feels like something people will actually adopt for diagnosing policy collapse.\n\n2. The multi-objective reformulation of PPO is both theoretically justified and surprisingly lightweight — it’s clear the authors thought about practical integration rather than proposing a totally new framework.\n\n3. The entropy controller is well-motivated by the observed entropy trajectories; it’s an elegant way to stabilize exploration without resorting to arbitrary tuning.", "weaknesses": "1. The paper admits that SRR depends on a chosen threshold but never tests how sensitive results are to that choice. A short analysis or plot of SRR versus threshold would make the metric feel more reliable.\n\n2. The core multi-gradient idea is interesting, but the paper doesn’t show how the mixing weights change or whether gradients conflict during training. Some visualization or discussion would make the method easier to understand.\n\n3. The claim that contraction is unique to PPO or RL methods isn’t fully demonstrated. Off-policy baselines like DPO and KTO are only compared at the endpoint, not throughout training. Showing their entropy or SRR over time would make the argument more solid.\n\n4. The paper calls CaPPO’s overhead “a small constant factor” but provides no runtime or memory measurements. Since the method adds multiple gradients, a quick comparison of compute cost with PPO would be important for judging practicality.\n\n5. It’s not clear what higher SRR actually means in practice beyond diversity metrics. The paper assumes preserving SFT support is good but doesn’t show that it improves model helpfulness, safety, or calibration.\n\n6. The contraction analysis feels surface-level. Figure 1 shows entropy dropping, but there’s no look into *when* or *why* that happens, or which prompts are most affected. More insight into these dynamics would strengthen the explanation. (Maybe a figure to show this?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)